Don't reload history on semi-random mouse events, fix #470.

:mouse: