Don't update the game speed until touch up.

If this feels bad, we can put it back in, but it is slow to update on
some devices.