handling tessellator error and pumping out error status

You are returning before outline is deleted. If there's an error you need to cleanup. Best solution is to wrap TESStesselator in a class and cleanup in the destructor. You can also move the improved Tesselate function to that class (along with the rest of the libtess API).