Also do not mess with the mouse in exit_grabber

/cc https://github.com/blueyed/awesome-cyclefocus/issues/6