# A Survey of Handwritten Character Recognition with MNIST and EMNIST
## 2 MNIST Database
### State of the Art
[21] proposed an approach using the 'extreme learning machine' (ELM) [22] algorithm to train shallow non-convolutional neural networks, and they agree on the fact that data augmentation with distortions can reduce error rates even further. In their work, [PERSON] et al. compare their results with other previous ELM and selected non-ELM approaches, updated with some CNN-based works as of 2015. The most outstanding ELM-based results achieved test error rates of 0.97% in the work by [PERSON] et al. [23], and ranged from 0.57 to 1.36% in the proposal by [PERSON] et al. [21]. However, most interesting results included in the comparison by [PERSON] et al. [21] are not those provided in that work, but those from previous works where convolutional neural networks were used. For example, [PERSON] et al. [24] used CNNs with a generalization of dropout they called DropConnect, and reported an error rate of 0.57% without data augmentation and as low as 0.21% with data augmentation. [PERSON] and [PERSON] [25] proposed the use of stochastic pooling achieving an error rate of 0.47%. [PERSON] et al. [26] described the _maxout_ model averaging technique, attaining a test error rate of 0.45% without data augmentation. In 2015, [PERSON] et al. [27] described \"deeply supervised nets\", an approach by which they introduce a classifier (SVM or softmax) at hidden layers, reporting a result of 0.39% without data augmentation. A more recent ranking, updated as of 2016, has been made available by [PERSON] [3] in his GitHub's page. This ranking includes many of the works reviewed so far, as well as other with very competitive results. For example, [PERSON] et al. [28] explore and optimize data augmentation, and attain a test error rate of 0.23% in the MNIST database using a convolutional neural network. A very interesting result is reported by [PERSON] and [PERSON] [29], where an error rate of 0.24% (the best result found as of 2017 in the state of the art without data augmentation) was obtained by using a network-in-network approach with a maxout multi-layer perceptron, an approach they named MIN. Also, very competitive performances without data augmentation were reported by [PERSON] et al. [30] when using gated pooling functions in CNNs (0.29%), by [PERSON] and [PERSON] [31] when using a recurrent CNN (0.31%), by [PERSON] and [PERSON] [32] when using CNNs with normalization layers and piecewise linear activation units (0.31%) and competitive multi-scale convolutional filters (0.33%) [33], or by [PERSON] [34] using fractional max-pooling with random overlapping (0.32%). Additional works where state-of-the-art results were obtained without data augmentation include those by [PERSON] and [PERSON] [35], who reported a test error rate of 0.37% using a fast-learning shallow convolutional neural network, [PERSON] et al. [36], who achieved 0.39% using convolutional kernel networks, [PERSON] et al. [37], who explored multi-loss regularization in CNNs obtaining an error rate of 0.42%, and [PERSON] et al. [38] used so-called convolutional \"highway\" networks (inspired by LSTM recurrent networks) to achieve an error rate of 0.45%. [PERSON] et al. [39] reported an error rate of 0.47% using a network-in-network approach, where micro-neural networks are used as convolutional layers. [PERSON] et al. [40] used convolutional layers for feature extraction and two fully connected NN for classification, attaining an error rate of 0.62%. [PERSON] and [PERSON] [41] designed a network that performed wavelet transform convolutions to the input and a generative PCA (principal component analysis) classifier to obtain an error of 0.43%. [PERSON] et al. [42] proposed a variation of a CNN where the first layer applies Gabor filters over the input, obtaining an error of 0.68%. Also, [PERSON] et al. [43] explored the application of limited-memory BFGS (L-BFGS) and conjugate gradient (CG) as alternatives to stochastic gradient descent methods for network optimization, achieving 0.69% error rate. Finally, [PERSON] et al.