Bug 677275 - Mosaic Filter produces ugly artifacts

Fix this by using a real point-segment distance.
Some artifacts remains though, they must come from somewhere else.

(cherry picked from commit 4caa5a113fe49e57a3861d3dab666da1dd38f2e5)