filter vertices leading three vertices to be colinear, and check reflex vertex in ear clipping algorithm