operations: fix a bunch of inconsistencies in mosaic.c

And yet another bug about the alpha channel in the color-averaging case
operations: Bug 677275 - Mosaic Filter produces ugly artifacts

Use a correct segment-point distance.