Fix momentum and adaptive learning to be activated after each mini-batch.