Fixing a bug in the complex filling algorithm

There was a problem when edges were exactly aligned with the to be
filled position. Due to numerical instabilities and a typo this caused
some positions not to be filled.