detect ending points for filtering out points outside the concave hull