Changed the similarity function to a symmetric one. Now yields a 35% recognition rate with leave one out cross validation. The method seems to mostly confuse characters with similar color schemes. I see 2 ways to improve classification:
- a better classification method which is more robust to outliers, using K-nn with larger K or some semi-supervised spectral method.
- a similarity function which takes into account how well the matching preserves image structure.