Fix corner case that could cause getting stuck in drag mode (#1910)