Made sure to relock the mouse if you click onto the game anywhere.