Went back to the old, slow way of looking up colors; unfortunately, IM's remap isn't good enough to use.
Improved Colour::FindClosest performance; reduced benchmark time from 3.5s to 1s, using bins. Woohoo!