Derp

Turns out keyDown and keyUp DO get called for mouse buttons, and all I
needed to do was return from the mouse event if the button clicked was
not attack or use item. Next up: fixing rebinding of attack and use item
key for skills (maybe...)