Do not drag if we are already scaling.