Handle drag end only if drag has been started.