Don't allow panning if inside the active input