Ignore time spent not scrolling when edge scrolling.

This prevents big jumps from occuring when the mouse leaves one side
of the window, and re-enters on another side of the window.

Fixes #431.