If we click on the handle don't set the new position