Don't allow dragging of the map when the click starts in a button

Fixes #1869