Fix erratic behaviour when dragging back from offset