Shifting deltas only when mouse is moved