Only handle arrow keys when no modifiers are pressed.