Fix surprising switch-to-self behavior.

"Natural" switch-by-mouse had the side-effect of causing mouse-
triggered switches that didn't result in a "real" switch causing
the mouse pointer to jump to the opposite side of the screen (this
fixes that).