Fix unintentional unselection with Ctrl+Up/Down in toggleselect mode