Fix buffer modification check

It used to grab the next key, because the normal mode reactivation
when executing the on_next_key would trigger a timestamp check again.