Swipe: Keep swiperight when tearing down swipeleft and vice versa

Closes gh-7353
Fixes gh-7351