Selection by non vim-commands doesn't track correctly

For example if you use shift+v to enter visual line then hit up/down arrow you can expand the selection, but if you select with the mouse then hit up/down it clears the selection and moves the caret instead.

You can also get similar behavior by the Expand selection to Brackets (cmd+shift+m on OSX) command.