Do not drag if we are already scrolling.