DouglasPeuckerMod: fix potential integer underflow