polygon: check if a coordinate is on the edge

This is a fix in which a coordinate which was an on edge was only detected
correctly in some directions.

Signed-off-by: Thorsten Liebig <Thorsten.Liebig@gmx.de>