Prevent flickering of background color if the cell is only swipable from one direction.