Fix the remaining time jumping ahead (fixes #1)