Only prevent default event handling on valid swipe