Prevent clicks when swiping, don't swipe on map

This involves some complex (and in many cases, hacky) manipulation of
the event system, but hey! It works!