Revert "Fixed learner being late under small load"

This reverts commit 8edca43cf86e10fd4ba0901e40ee96bbdd962b9a.

It does not seem to fix the issue.