Removes the pan gesture recognizer when not needed