Logically it's the job of the mode. Normal mode switches to other modes on key presses, so should other modes do.