Take advantage of event capture

Now once you start scrolling, buttons and anything else will magically cancel (unselect).
Tune the max click delta

Five pixels was pretty darn small to begin with, and as a capture threshold is kind of buggy. Up it to 15.