After a bit of tweaking with hue, got a reasonably good looking segmentation method. Will now try to classify these segmentations using segment matching method.
Now got the algorithm running on the entire data set, with a 0.3 recognition rate - which is much better than random but still not good by any stretch. The segmentation algorithm seems to be pretty good, as it still gives 100% for lupin and luffy. But trickier characters like Rei seem to introduce outliers all over the place. Classification will need to be robust to those. Furthermore I'll identify exactly what provokes characters to be misclassified, I suspect that segments which are cut in multiple parts in some image may be a big source of error. As the number of segment is small, it might be practical to consider not just 1 to 1 relations between segmentations, but 1 to k for some small k as well - the combinatorial explosion should still be manageable.