Revert "Enable mouse in normal, visual, insert etc. modes"

This reverts commit c592d57f451d7c4687676e68f37758e78ddc9cae.