Give up on accurate delayed switch-back

Mojang changed how things work so we can't accurately tell what state
something's in anymore :(