Ignore consumed keyboard modifiers

See commit. Not sure if keymap should be free’d.