Simplify plane cache code

While at it improve the speed of the ColorPixelHarmonizer a bit.