Fix mousewheel not working

Needed to revert some recent changes I made to mouse handling, but some
optimizations I made are still there