Factor out position updates from scale/direction updates to improve performance when dragging, see #35