Swipe: Keep swiperight when tearing down swipeleft and vice versa

Fixes gh-7351