Fix thread race with time going backwards