Don't reset timestamp offsets to prevent time jumps