# A Survey of Handwritten Character Recognition with MNIST and EMNIST
## 2 MNIST Database
### State of the Art  
Because MNIST has been widely used to test the behavior of many implementations of classifiers, there has been an effort to publish some rankings in the past, using MNIST as a benchmark. Most of these rankings, as well as the established literature uses the \"test error rate\" metric when referring to performance over MNIST. This metric is the percentage of incorrectly classified instances. One of the earliest rankings was published by [PERSON] et al. [2] themselves, which includes references up to 2012. This website provides a taxonomy of classifiers and points out whether each work performed data preprocessing or augmentation (addition of new instances to the training set resulting from distortions or other modifications of the original data). In this ranking, we can verify that the early machine learning approaches used by [PERSON] et al. [4] included linear classifiers (whose error rate ranges from 7.6 to 12%), K-nearest neighbors approaches (K-NN, ranging from 1.1 to 5%), non-linear classifiers (about 3.5%), support vector machines (SVM, from 0.8 to 1.4%), neural networks (NN, from 1.6 to 4.7%) and convolutional neural networks (CNN, from 0.7 to 1.7%). It is remarkable that data augmentation leads to better results, in particular, the best error rate achieved using a convolutional neural network with no distortions and no preprocessing in [PERSON] et al. [4] was 0.95%. From the different works gathered in [PERSON] et al. 's ranking, those based in convolutional neural networks outperform the other techniques. However, there are some classical machine learning techniques which are still able to provide competitive error rates (in this paper, we will consider \"competitive\" those techniques achieving an error rate under 1.0%). For instance, [PERSON] et al. [5] achieved 0.63% and [PERSON] et al. [6] achieved 0.54% and 0.52% using K-NN, [PERSON] and [PERSON] [7] achieved 0.87% using boosted stumps on Haar features, [PERSON] et al. [4] achieved 0.8% and [PERSON] and [PERSON] [8] attained results from 0.56 to 0.68% using SVM. When using non-convolutional neural networks, [PERSON] et al. [9] achieved 0.9% and 0.7% respectively using a 2-layer neural network with MSE and cross-entropy loss functions. [PERSON] and [PERSON] [10] achieved 0.83% using a deep convex network without data augmentation or preprocessing. Very interesting results were attained by [PERSON] et al. [11] (0.39%) using a committee of 25 neural networks and [PERSON] et al. [12] (0.35%) using a 6-layers neural network (it is worth mentioning that the reproducibility of this result has been put into question by [PERSON] [13] in his blog). On the other hand, works based on convolutional neural networks attained a much better average performance (in fact, the worst result reported in [PERSON]'s ranking was 1.7%). [PERSON] et al. [4] combined different convolutional architectures along with data augmentation techniques, obtaining error rates ranging from 0.7 to 0.95%. [PERSON] et al. [14] attained between 0.54% and 0.83% using a trainable feature extractor along with SVMs, and [PERSON] et al. [15] reported an error rate of 0.59% using a similar  
Figure 1: Example of the MNIST database. technique consisting on a CNN to extract sparse features and SVMs for classification. [PERSON] et al. [9] obtained error rates between 0.4% and 0.6% using CNN with cross-entropy loss function and data augmentation. [PERSON] et al. [16] reported results between 0.39% and 0.6% using a large CNN along with unsupervised pretraining, and some years later [PERSON] et al. [17] reported a test error of 0.59% with a similar approach technique and without data augmentation. The best results in this ranking are those obtained by [PERSON] et al. [18], which reported an error rate of 0.35% using a large CNN, and 0.27% [19] and 0.23% [20] using committees of 7 and 35 neural networks respectively, using data augmentation in all cases. In 2015, [PERSON] et al.