Remove key modifiers in KeyboardInput

Needs testing on x11 and cocoa