Skip iteration for empty inlier sample set