Really make sure the scroll bar doesn't fire events

We get jaggy scrolling when the scroll bar events
compete with the pan events