Fix remove fling being triggered while scrolling