Swipe is cancelled if reversing swipe direction