Only remove the gesture recognizer owned in this class.