Always enable panning for SidePanel

The swipe gesture overlaps with the tableview but users can always
swipe the other direction if they want to display the delete button.