Further optimizations to mouse handling