Refactor manual user drag override to only calculate once per draw request