Fix race condition in restoring the previous timer

Can't set to a negative number