Swipe removal fixes

Makes swipe removal cleaner

Does not fix Jesse's swipe problems - those will most likely require a major reworking of how we implement swipe. Trello card is made, will be looked at before official releases.