Roll back threshold of contact point refinement