Only prevent ghost clicks when touch events are in use

Pointers or mouse events will be used otherwise and pointers don't need their clicks busted.