Revert some previous changes to mouse handling

Hopefully it should fix the issue at least one user was having...