# On the Brittleness of Handwritten Digit Recognition Models

Authors: Alexander K. Seewald
Journal: ISRN Machine Vision
 (2012)

---

## Abstract

Handwritten digit recognition is an important benchmark task in computer vision. Learning algorithms and feature representations which offer excellent performance for this task have been known for some time. Here, we focus on two major practical considerations: the relationship between the the amount of training data and error rate (corresponding to the effort to collect training data to build a model with a given maximum error rate) and the transferability of models' expertise between different datasets (corresponding to the usefulness for general handwritten digit recognition). While the relationship between amount of training data and error rate is very stable and to some extent independent of the specific dataset used—only the classifier and feature representation have significant effect—it has proven to be impossible to transfer low error rates on one or two pooled datasets to similarly low error rates on another dataset. We have called this weaknessbrittleness, inspired by an old Artificial Intelligence term that means the same thing. This weakness may be a general weakness of trained image classification systems.


---

## Body

## 1. Introduction

Intelligent image analysis is an interesting research area in Artificial Intelligence and also important to a variety of current open research problems. Handwritten digits recognition is a well-researched subarea within the field, which is concerned with learning models to distinguish presegmented handwritten digits. The application of machine learning techniques over the last decade has proven successful in building systems which are competitive to human performance and which perform far better than manually written classical AI systems used in the beginnings of optical character recognition technology. However, not all aspects of such models have been previously investigated.Here, we systematically investigate two new aspects of such systems.(i)
Essential training set size, that is, the relation between training set size and accuracy/error rate so as to determine the number of labeled training samples that are essential for a given performance level. Creating labeled training samples is costly and we are generally interested in algorithms which yield acceptable performance with the fewest number of labeled training samples.(ii)
Dataset-Independence, that is, how well models trained on one sample dataset for handwritten digit recognition perform on other sample datasets for handwritten digit recognition after comprehensive normalization between the datasets. Models should be robust to small changes in preprocessing and data collection, but this has not been tested before.1For the first aspect, we have found that all three datasets considered here give similar performance relative to absolute training set size. This indicates that the quality of input data is similar for these three datasets. A relatively small number of high-quality samples is already sufficient for acceptable performance. Accuracy as a function of absolute training set size follows a smooth asymptotic behavior, in which low error rates (below 10%) are reached quite fast, but very low error rates are reached only after sustained effort.For the second aspect, we were surprised to notice that none of the considered learning systems were able to transfer their expertise to other datasets. In fact, the performance on other datasets was always significantly worse or unacceptably high.This may point to a general weakness of present intelligent image analysis systems. We have named this weaknessbrittleness in AI terminology, which seems to us the most appropriate term.Small differences in preprocessing methods which have not been documented in sufficient detail (except perhaps in [1]) may be responsible for this effect. Another explanation might be that idiosyncrasies of the specific dataset used for training are learned as well and hamper the generalization ability of the underlying learning algorithm. This effect is observed independently of learning algorithm or feature representation.A more detailed documentation of preprocessing methods and classification systems in the form of Open Source code would be needed to investigate how to build more robust learning system for this domain, and possibly for intelligent image analysis systems in general.


## 2. Related Research

Reference [1] described the full preprocessing of dataset DIGITS and noted early results of the experiments within this paper.Reference [2] also reported extensive experiments, noting that simpler convolutional neural networks than the one used by [3] might suffice. However, since they did not contribute code for their learning and preprocessing systems, we chose the implementation from [4] which is freely available for research purposes.Reference [5] reported a comprehensive benchmark of handwritten digit recognition with several state of the art approaches, datasets, and feature representations. However, they analyzed neither the relationship of training set size versus accuracy/error nor the dataset-independence of the trained models, which are our two main contributions.Reference [3] introduced convolutional neural networks into handwritten digit recognition research and demonstrated a system (LeNet-5) which can still be considered state of the art.Reference [6] notes open research questions and also proposes contributing open source software for standard preprocessing methods. However, neither do they offer their own extensive code as open-source, nor do they note the dataset-independence issue we have noted here, although it is clearly a major issue in the practical application of such systems.


## 3. Experimental Setup

### 3.1. Datasets

We used two well-known (USPS, MNIST) and one relatively unknown (DIGITS) dataset for handwritten digit recognition. The relatively unknown dataset was created by ourselves, so we had complete control and documentation over all preprocessing steps, documented in [1].The US Postal (USPS) handwritten digit dataset is derived from a project on recognizing handwritten digits on envelopes [7, 8]. The digits were downscaled to 16 × 16 pixels and scaled without distortion (i.e., retaining the aspect ratio; 1 : 1 scaling). The training set has 7291 samples, and the test set has 2007 samples. Figure 1(a) shows samples from USPS. For the experiments in  Section 4.1,  we used USPS as is; for Section 4.2, we used a reformatted version which mimics MNIST preprocessing by rescaling and shifting center of gravity.(a) USPS, (b) MNIST.
(a)(b)The MNIST dataset, one of the most famous in digit recognition, is derived from the NIST dataseta and has been created by LeCun et al. [3]. According to this paper, the digits from NIST  were downscaled to 20 × 20 pixels and centered in a 28×28 pixel bitmap by putting center of gravity of the black pixels in the center of the bitmap. It has 60,000 training and 10,000 test samples. Figure 1(b) shows samples from MNIST. It can be seen that MNIST has about 1% segmentation errors (e.g., column 4, row 4 is a badly segmented four)2. However, this cannot explain the performance differences between the systems reported in Section 4.2 as those comparisons not including MNIST fare as poorly as those that do.The DIGITS dataset was created in 2005, based on samples from students of a lecture given by the author. Each student contributed 100 samples, equally distributed among the digits from 0 to 9. The complete preprocessing is described in [1]. Students were given the choice to withhold the samples, to allow usage of the samples by the author of this paper and to allow usage by anyone (i.e., public domain). 37 students opted for the latter option. These were randomly distributed into training and test (19 training, 17 test), yielding 1,893 training and 1,796 test samples after minor cleanup. The dataset can be freely downloaded from the authors website, http://alex.seewald.at/.Figure2(a) shows the digit dataset with preprocessing optimized to improve classification accuracy (viz., arbitrary aspect ratio—i.e., the digit is scaled to fill available space; blurring with blur = 2.5, no deslanting3), and Figure 2(b) shows the same samples reformatted in a format easier to recognize for humans (1 : 1 scaling, blur = 0.5). Mitchell filter downsampling with integrated Gaussian blurring was used in both cases.(a) digits (b=2.5, arb.scal.), (b) digits (b=0.5, 1 : 1 scaling).
(a)(b)


### 3.2. Feature Sets

We considered two feature sets.(i)
Pixel-based, that is, the pixel grayscale values in eight bit precision (0 = white, 255 = black to be compatible with MNIST) in the order from top left to bottom right (784 numeric features).(ii)
Gradient-based, that is, a 200-dimensional numeric feature vector encoding eight direction-specific 5×5 gradient images (same feature order as for above images). This was one of three top-performing representations in [5] and is called e-grg in their paper. We reimplemented e-grg from scratch and validated on MNIST train/test with 1-NN, yielding a test error rate of 1.29% versus the 1.35% reported in their paper with identical preprocessing.Additional feature sets could have been considered, but we felt that these two sets would be sufficient for the purpose of this paper.


### 3.3. Classifiers

We considered a variety of classifiers in three groups. All of the classifiers except convNN were taken from WEKA [9].


#### 3.3.1. Instance-Based Learning

Initial experiments indicated that—probably because of the high number of classes—a simplek-NN nearest neighbor classifier with k=1 performed best. For the gradient-based representation, only euclidean distance was considered. For the pixel-based representation, we additionally considered the well-known template matching method normalized correlation coefficient, and tangent distance, which can cope with small template distortions. We used the GPL implementation by [10] and validated on their freely available samples. Both distance measures were reimplemented in Java for use with WEKA.


#### 3.3.2. Support Vector Machines

We also considered polynomial and RBF kernel support vector machines (SVM, see, e.g., [11]) classifiers, since these classifiers were two of the three best-performing methods according to [5].From earlier experiments, we already knew optimized parameter settings for this classifiers on the DIGITS training set. We extended these experiments and determined similar optimized parameter settings fore-grg on the same dataset. These were used to train all other datasets. A slight positive bias for DIGITS and low-bias learning algorithms (such as RBF and polynomial SVM) may be present. We also considered a linear kernel SVM with default settings (C=1).


#### 3.3.3. Convolutional Networks

The de facto standard for handwritten digit recognition is the convolutional network (convNN, e.g., leNet-5) by [3]. As WEKA does not include a convolutional network learning algorithm, and there is also none available by the author of the mentioned paper, we had to resort to using the nonscriptable version of the training algorithm by [4]. A set of manual experiments was done to validate the implementation versus MNIST, and we did extensive experiments for Section 4.2. Due to the nonscriptable version, we could not run learning curves on this system. The CPU time would in any case have been exorbitantly high at around 1,500 hours (2 CPU months) for learning curves on all three datasets.Training was done using the following parameters, as these proved to give the best results on the original MNIST training/test sets (according to [4], validated by us),(i)
initial learning rate: 0.001,(ii)
minimum learning rate: 0.00005,(iii)
rate of decay for learning rate, applied every two epochs until minimum learning rate was reached: 0.79418335,(iv)
run with elastically deformed training inputs for at least 52 epochs,(v)
run with non-deformed training input for exactly 5 polishing epochs with a learning rate of 0.0001.


## 3.1. Datasets

We used two well-known (USPS, MNIST) and one relatively unknown (DIGITS) dataset for handwritten digit recognition. The relatively unknown dataset was created by ourselves, so we had complete control and documentation over all preprocessing steps, documented in [1].The US Postal (USPS) handwritten digit dataset is derived from a project on recognizing handwritten digits on envelopes [7, 8]. The digits were downscaled to 16 × 16 pixels and scaled without distortion (i.e., retaining the aspect ratio; 1 : 1 scaling). The training set has 7291 samples, and the test set has 2007 samples. Figure 1(a) shows samples from USPS. For the experiments in  Section 4.1,  we used USPS as is; for Section 4.2, we used a reformatted version which mimics MNIST preprocessing by rescaling and shifting center of gravity.(a) USPS, (b) MNIST.
(a)(b)The MNIST dataset, one of the most famous in digit recognition, is derived from the NIST dataseta and has been created by LeCun et al. [3]. According to this paper, the digits from NIST  were downscaled to 20 × 20 pixels and centered in a 28×28 pixel bitmap by putting center of gravity of the black pixels in the center of the bitmap. It has 60,000 training and 10,000 test samples. Figure 1(b) shows samples from MNIST. It can be seen that MNIST has about 1% segmentation errors (e.g., column 4, row 4 is a badly segmented four)2. However, this cannot explain the performance differences between the systems reported in Section 4.2 as those comparisons not including MNIST fare as poorly as those that do.The DIGITS dataset was created in 2005, based on samples from students of a lecture given by the author. Each student contributed 100 samples, equally distributed among the digits from 0 to 9. The complete preprocessing is described in [1]. Students were given the choice to withhold the samples, to allow usage of the samples by the author of this paper and to allow usage by anyone (i.e., public domain). 37 students opted for the latter option. These were randomly distributed into training and test (19 training, 17 test), yielding 1,893 training and 1,796 test samples after minor cleanup. The dataset can be freely downloaded from the authors website, http://alex.seewald.at/.Figure2(a) shows the digit dataset with preprocessing optimized to improve classification accuracy (viz., arbitrary aspect ratio—i.e., the digit is scaled to fill available space; blurring with blur = 2.5, no deslanting3), and Figure 2(b) shows the same samples reformatted in a format easier to recognize for humans (1 : 1 scaling, blur = 0.5). Mitchell filter downsampling with integrated Gaussian blurring was used in both cases.(a) digits (b=2.5, arb.scal.), (b) digits (b=0.5, 1 : 1 scaling).
(a)(b)


## 3.2. Feature Sets

We considered two feature sets.(i)
Pixel-based, that is, the pixel grayscale values in eight bit precision (0 = white, 255 = black to be compatible with MNIST) in the order from top left to bottom right (784 numeric features).(ii)
Gradient-based, that is, a 200-dimensional numeric feature vector encoding eight direction-specific 5×5 gradient images (same feature order as for above images). This was one of three top-performing representations in [5] and is called e-grg in their paper. We reimplemented e-grg from scratch and validated on MNIST train/test with 1-NN, yielding a test error rate of 1.29% versus the 1.35% reported in their paper with identical preprocessing.Additional feature sets could have been considered, but we felt that these two sets would be sufficient for the purpose of this paper.


## 3.3. Classifiers

We considered a variety of classifiers in three groups. All of the classifiers except convNN were taken from WEKA [9].


### 3.3.1. Instance-Based Learning

Initial experiments indicated that—probably because of the high number of classes—a simplek-NN nearest neighbor classifier with k=1 performed best. For the gradient-based representation, only euclidean distance was considered. For the pixel-based representation, we additionally considered the well-known template matching method normalized correlation coefficient, and tangent distance, which can cope with small template distortions. We used the GPL implementation by [10] and validated on their freely available samples. Both distance measures were reimplemented in Java for use with WEKA.


### 3.3.2. Support Vector Machines

We also considered polynomial and RBF kernel support vector machines (SVM, see, e.g., [11]) classifiers, since these classifiers were two of the three best-performing methods according to [5].From earlier experiments, we already knew optimized parameter settings for this classifiers on the DIGITS training set. We extended these experiments and determined similar optimized parameter settings fore-grg on the same dataset. These were used to train all other datasets. A slight positive bias for DIGITS and low-bias learning algorithms (such as RBF and polynomial SVM) may be present. We also considered a linear kernel SVM with default settings (C=1).


### 3.3.3. Convolutional Networks

The de facto standard for handwritten digit recognition is the convolutional network (convNN, e.g., leNet-5) by [3]. As WEKA does not include a convolutional network learning algorithm, and there is also none available by the author of the mentioned paper, we had to resort to using the nonscriptable version of the training algorithm by [4]. A set of manual experiments was done to validate the implementation versus MNIST, and we did extensive experiments for Section 4.2. Due to the nonscriptable version, we could not run learning curves on this system. The CPU time would in any case have been exorbitantly high at around 1,500 hours (2 CPU months) for learning curves on all three datasets.Training was done using the following parameters, as these proved to give the best results on the original MNIST training/test sets (according to [4], validated by us),(i)
initial learning rate: 0.001,(ii)
minimum learning rate: 0.00005,(iii)
rate of decay for learning rate, applied every two epochs until minimum learning rate was reached: 0.79418335,(iv)
run with elastically deformed training inputs for at least 52 epochs,(v)
run with non-deformed training input for exactly 5 polishing epochs with a learning rate of 0.0001.


## 3.3.1. Instance-Based Learning

Initial experiments indicated that—probably because of the high number of classes—a simplek-NN nearest neighbor classifier with k=1 performed best. For the gradient-based representation, only euclidean distance was considered. For the pixel-based representation, we additionally considered the well-known template matching method normalized correlation coefficient, and tangent distance, which can cope with small template distortions. We used the GPL implementation by [10] and validated on their freely available samples. Both distance measures were reimplemented in Java for use with WEKA.


## 3.3.2. Support Vector Machines

We also considered polynomial and RBF kernel support vector machines (SVM, see, e.g., [11]) classifiers, since these classifiers were two of the three best-performing methods according to [5].From earlier experiments, we already knew optimized parameter settings for this classifiers on the DIGITS training set. We extended these experiments and determined similar optimized parameter settings fore-grg on the same dataset. These were used to train all other datasets. A slight positive bias for DIGITS and low-bias learning algorithms (such as RBF and polynomial SVM) may be present. We also considered a linear kernel SVM with default settings (C=1).


## 3.3.3. Convolutional Networks

The de facto standard for handwritten digit recognition is the convolutional network (convNN, e.g., leNet-5) by [3]. As WEKA does not include a convolutional network learning algorithm, and there is also none available by the author of the mentioned paper, we had to resort to using the nonscriptable version of the training algorithm by [4]. A set of manual experiments was done to validate the implementation versus MNIST, and we did extensive experiments for Section 4.2. Due to the nonscriptable version, we could not run learning curves on this system. The CPU time would in any case have been exorbitantly high at around 1,500 hours (2 CPU months) for learning curves on all three datasets.Training was done using the following parameters, as these proved to give the best results on the original MNIST training/test sets (according to [4], validated by us),(i)
initial learning rate: 0.001,(ii)
minimum learning rate: 0.00005,(iii)
rate of decay for learning rate, applied every two epochs until minimum learning rate was reached: 0.79418335,(iv)
run with elastically deformed training inputs for at least 52 epochs,(v)
run with non-deformed training input for exactly 5 polishing epochs with a learning rate of 0.0001.


## 4. Results

In this section, we will show the full results from our experiments.


### 4.1. Essential Training Set Size

This section is concerned with analyzing the relationship between training set size and recognition accuracy, depending on dataset and learning algorithm.


#### 4.1.1. Pixel-Based Features

Figure3 shows the results for instance-based learning, and Figure 4 shows the results for SVM learning. Both experiments were run on pixel-based features from the three datasets. The test set was fixed while the training set was downsampled to the absolute number of training examples shown as X. The Y-axis shows the accuracy on the test set. Additionally, downsampling was randomized ten times, and the standard deviation over these ten runs is shown as error bars.Abs. training set size versus test set accuracy for pixel-based features on MNIST, USPS, and DIGITS (IBk variants).
(a)(b)(c)Abs. training set size versus test set accuracy for pixel-based features on MNIST, USPS, and DIGITS (SVM variants).
(a)(b)(c)When using instance-based learning, the three datasets perform remarkably similar. Only for the tangent distance variant, DIGITS performs noticeably worse. We presume this is due to the collection of this dataset, where digits had to be  written into a regular grid, which forced a very uniform orientation. As tangent distance was constructed to compensate for non-uniform orientations—wihch is not needed here—the additional degrees-of-freedom of this method may have led to overfitting on this dataset, resulting in inferior performance.When using SVM learning, the picture is similar, albeit less clear. Only for the polynomial variant do we see very similar behavior on the three datasets. For the other two variants, some differences appear. Especially, MNIST performs very badly with the RBF kernel variant. We presume that this is due to the high number of variance in MNIST, and the higher number of parameters for the RBF kernel, such that the amount of training data is no longer sufficient for stable parameter estimation. Also, parameters were optimized for the DIGITS training set, and this may have led to some overfitting4. The polynomial kernel has the best accuracy here, closely followed by the linear kernel. Because of the nonscriptable version for convNN and its long training times, we could not test it here.


#### 4.1.2. Gradient-Based Features

In a second step, we analyzed gradient-based features. Since pixel-based features are a very imprecise way to encode information about handwritten digits, we chose to use direction-specific feature maps which were previously found to work best (see Section3.2).Figure5 shows the results with instance-based learning. Here, we only used IBk with one nearest neighbor as the other two distance measures are inappropriate for non-pixel-based data. We again observe similar behavior for all datasets, at a slightly higher level of accuracy than for the pixel-based features and the same learning algorithm. Clearly, adding relevant background knowledge in the form of tangent distance or normalized correlation distance measures is more helpful to improve IBk than this alternative feature representation.Figure 5
Abs. training set size versus test set accuracy for gradient features on MNIST, USPS, and DIGITS (IBk variants).Figure6 shows the results with SVM learning. SVM results are clearly improved throughout over all datasets. Also, we see a clear ordering for the right half of each figure: MNIST performs better than USPS, and USPS performs better than DIGITS. For SVM learning, the alternative feature set representation improves the results quite distinctly. Note also that just for 1,800 samples, we can have an error of 2% on the MNIST testset, which is quite good considering that the best published results are at around 0.5% and use orders of magnitude more training data. A linear SVM with unprecedented processing speed is only slightly worse. It might be that the higher accuracy of these systems has enabled us to see the hardness of the dataset—like the harder part of MNIST, DIGITS consisted of data contributed by (university) students. USPS would then be between both datasets in terms of sample complexity.Abs. training set size versus test set accuracy for gradient features on MNIST, USPS, and DIGITS (SVM variants).
(a)(b)(c)The shape of all learning curves is remarkably similar and might be estimated with just a few data points. They seem to depend on the learning algorithm, the feature representation, and to a lesser extent on the specific dataset in question (e.g., dataset complexity, sample distribution, or other factors).


### 4.2. Dataset-Independence

All previous results mean nothing if the task has not really been solved. So, as it is clear that—small differences between the datasets notwithstanding—all these datasets deal with the writer-independent recognition of handwritten digits and were created by disjunct sets of writers (which were also properly distributed between training and test set), we estimated the quality of each model by testing it on the other datasets. First, we converted both DIGITS and USPS into MNIST format by centering each digit in a28×28 image, equalizing the histograms via nonlinear gamma correction.5 For DIGITS, we estimated center-of-gravity from the original 300 dpi black-and-white images; for USPS, we estimated c-o-g from the grayscale images after thresholding at 50% (128). In both cases, 1 : 1 scaling was used (i.e., the aspect ratio was retained) (Figure 7).(a) USPS, (b) DIGITS, both reformatted to MNIST format.
(a)(b)First, we trained on each training set in turn and tested on the other two sets. Note that the training and test sets are of different size, so for example, MNIST builds a model from 60,000 samples while DIGITS just builds a model from about 1,800. According to the results from the previous section, we would expect a range of about one order of magnitude (best versus worst) in error rates on the test set corresponding to the training set, with MNIST better than USPS and USPS better than DIGITS. This is exactly what we observed. Surprisingly, the performance on the other test sets is much worse.This time, we also tested LeCun’s original convolutional neural network model as reconstructed by [4]. The results from training on the complete training set with the training method described in part in [3] yielded error rates of 0.74% on MNIST-test, 1.72% on the full USPS dataset (i.e., train and test combined), and 8.74% on the full DIGITS dataset. The error rate is significantly higher for both datasets: about twice as high on USPS and more than ten times higher on DIGITS6. We also trained convNN with the same method and similar settings both for DIGITS and USPS.Table1 shows the results for pixel-based features, and Table 2 shows the results for gradient-based features. Both are showing error rates estimated on the respective test sets and the average error of the two other sets divided by the error on the test set corresponding to the training set the model was trained on. What is immediately apparent is that no combination of learning algorithm, feature representation and dataset to train on was able to transfer the usually good results of its own test set to the other test sets without a significant loss in accuracy. A small ratio such as less than 2.0 is only obtainable for unacceptably high error rates. The lowest error rate on any test set of 3.48% is achieved for convNN trained on MNIST and tested on USPS. This still has more than twice this error rate (8.24%) on DIGITS, and about five times the error rate on the MNIST test set (0.74%).Table 1
Dataset independence for pixel-based features, each dataset separately.
ClassifierTrained onTested onAvg. error versus own testsetMNISTDIGITSUSPSIBk1 euclideanMNIST3.0919.2117.495.94xIBk1 euclideanDIGITS36.2216.5952.722.68xIBk1 euclideanUSPS28.4155.015.337.83xIBk1 NCCMNIST2.8317.6513.705.54xIBk1 NCCDIGITS32.4214.1444.592.72xIBk1 NCCUSPS26.0651.174.588.43xIBk1 TDMNIST1.5113.535.636.34xIBk1 TDDIGITS25.8810.0237.773.18xIBk1 TDUSPS10.5136.473.646.45xSVM linearMNIST6.8334.9716.243.75xSVM linearDIGITS31.5716.0945.542.40xSVM linearUSPS40.6463.256.537.95xSVM polynomialMNIST1.2716.2011.5610.93xSVM polynomialDIGITS30.0511.4747.683.39xSVM polynomialUSPS44.7874.334.4313.44xSVM RBFMNIST4.3153.3420.788.60xSVM RBFDIGITS51.5033.7460.091.65xSVM RBFUSPS81.0589.987.3711.60xconvNNMNIST0.748.243.487.92xconvNNDIGITS21.435.7330.04.49xconvNNUSPS4.2527.563.085.16xTable 2
Dataset independence for gradient-based features, each dataset separately.
ClassifierTrained onTested onAvg. error versus own testsetMNISTDIGITSUSPSIBk1 euclideanMNIST1.2912.085.987.00xIBk1 euclideanDIGITS21.917.2937.624.08xIBk1 euclideanUSPS10.3033.073.496.21xSVM linearMNIST1.3412.925.636.92xSVM linearDIGITS19.765.1230.544.91xSVM linearUSPS14.6239.373.348.08xSVM polynomialMNIST0.478.074.4313.30xSVM polynomialDIGITS17.813.6724.865.81xSVM polynomialUSPS14.6839.032.799.63xSVM RBFMNIST0.578.304.2811.04xSVM RBFDIGITS17.754.0625.465.32xSVM RBFUSPS14.8940.032.799.84xSecond, we chose to also test combining two datasets and testing on the remaining dataset. We downsampled the larger training dataset to the size of the smaller training set and combined them, shuffling the results to prevent order effects. The same test sets as previously were used. This time, we computed the error of the remaining completely unseen dataset divided by the average of errors for the two seen datasets (i.e., those whose training set was part of the dataset pool). Again, convNN was trained on the same data.Tables3 and 4 show the results for pixel-based and gradient-based features. Again, we see that the error of the datasets who were part of the training is usually much higher than the one on the completely unseen dataset. This is not true for IBk with tangent-distance, where both for MNIST-DIGITS and USPS-DIGITS the error on DIGITS testset is higher than the one on the completely unseen dataset. This might be due to the small size of the DIGITS test set, which increases the variance of error estimates computed on this dataset. The same happens for SVM linear on USPS-DIGITS and SVM RBF on MNIST-DIGITS and USPS-DIGITS.Table 3
Dataset independence for pixel-based features, two datasets combined.
ClassifierTrained onTested onError versus avg. of own testsetsMNISTDIGITSUSPSIBk1 eucl.MNIST-DIGITS9.2318.3225.561.86xIBk1 eucl.MNIST-USPS5.9725.895.284.60xIBk1 eucl.USPS-DIGITS22.2722.056.981.53xIBk1 NCCMNIST-DIGITS7.8113.6421.672.02xIBk1 NCCMNIST-USPS4.7424.394.535.26xIBk1 NCCUSPS-DIGITS18.9515.536.981.68xIBk1 TDMNIST-DIGITS4.3411.4110.811.37xIBk1 TDMNIST-USPS2.6516.093.645.12xIBk1 TDUSPS-DIGITS9.6114.094.531.03xSVM linearMNIST-DIGITS10.6221.1021.921.38xSVM linearMNIST-USPS12.2743.108.274.20xSVM linearUSPS-DIGITS20.3723.839.671.22xSVM poly.MNIST-DIGITS4.968.8516.542.40xSVM poly.MNIST-USPS2.6622.163.896.77xSVM poly.USPS-DIGITS14.569.975.231.92xSVM RBFMNIST-DIGITS12.6034.5831.191.32xSVM RBFMNIST-USPS13.6071.665.637.45xSVM RBFUSPS-DIGITS39.8947.387.031.47xconvNNMNIST-DIGITS3.214.006.571.82xconvNNMNIST-USPS1.2511.852.745.94xconvNNUSPS-DIGITS7.035.794.881.32xTable 4
Dataset independence for gradient-based features, two datasets combined.
ClassifierTrained onTested onError versus avg. of own testsetsMNISTDIGITSUSPSIBk1 eucl.MNIST-DIGITS3.587.419.871.80xIBk1 eucl.MNIST-USPS1.9815.593.345.86xIBk1 eucl.USPS-DIGITS9.517.804.141.59xSVM linearMNIST-DIGITS3.434.7911.012.68xSVM linearMNIST-USPS2.2315.143.395.39xSVM linearUSPS-DIGITS7.506.244.381.41xSVM poly.MNIST-DIGITS1.843.516.582.46xSVM poly.MNIST-USPS0.9110.752.546.23xSVM poly.USPS-DIGITS5.944.452.941.61xSVM RBFMNIST-DIGITS1.883.516.982.59xSVM RBFMNIST-USPS1.0011.532.446.70xSVM RBFUSPS-DIGITS6.224.572.841.68xThe better gradient-based feature representation is probably responsible for preventing such outliers in the second tables, as more stable models are learned. This time, SVM polynomial and SVM RBF give the best performance (averaged over the completely unseen test datasets’ error rates), closely followed by convNN which uses pixel-based features. Still, this translates to an error of 5.94%, 10.75%, and 5.94% for MNIST, DIGITS, and USPS, which is at least an order of magnitude higher than the best results for handwritten digit recognition (reported on MNIST).


## 4.1. Essential Training Set Size

This section is concerned with analyzing the relationship between training set size and recognition accuracy, depending on dataset and learning algorithm.


### 4.1.1. Pixel-Based Features

Figure3 shows the results for instance-based learning, and Figure 4 shows the results for SVM learning. Both experiments were run on pixel-based features from the three datasets. The test set was fixed while the training set was downsampled to the absolute number of training examples shown as X. The Y-axis shows the accuracy on the test set. Additionally, downsampling was randomized ten times, and the standard deviation over these ten runs is shown as error bars.Abs. training set size versus test set accuracy for pixel-based features on MNIST, USPS, and DIGITS (IBk variants).
(a)(b)(c)Abs. training set size versus test set accuracy for pixel-based features on MNIST, USPS, and DIGITS (SVM variants).
(a)(b)(c)When using instance-based learning, the three datasets perform remarkably similar. Only for the tangent distance variant, DIGITS performs noticeably worse. We presume this is due to the collection of this dataset, where digits had to be  written into a regular grid, which forced a very uniform orientation. As tangent distance was constructed to compensate for non-uniform orientations—wihch is not needed here—the additional degrees-of-freedom of this method may have led to overfitting on this dataset, resulting in inferior performance.When using SVM learning, the picture is similar, albeit less clear. Only for the polynomial variant do we see very similar behavior on the three datasets. For the other two variants, some differences appear. Especially, MNIST performs very badly with the RBF kernel variant. We presume that this is due to the high number of variance in MNIST, and the higher number of parameters for the RBF kernel, such that the amount of training data is no longer sufficient for stable parameter estimation. Also, parameters were optimized for the DIGITS training set, and this may have led to some overfitting4. The polynomial kernel has the best accuracy here, closely followed by the linear kernel. Because of the nonscriptable version for convNN and its long training times, we could not test it here.


### 4.1.2. Gradient-Based Features

In a second step, we analyzed gradient-based features. Since pixel-based features are a very imprecise way to encode information about handwritten digits, we chose to use direction-specific feature maps which were previously found to work best (see Section3.2).Figure5 shows the results with instance-based learning. Here, we only used IBk with one nearest neighbor as the other two distance measures are inappropriate for non-pixel-based data. We again observe similar behavior for all datasets, at a slightly higher level of accuracy than for the pixel-based features and the same learning algorithm. Clearly, adding relevant background knowledge in the form of tangent distance or normalized correlation distance measures is more helpful to improve IBk than this alternative feature representation.Figure 5
Abs. training set size versus test set accuracy for gradient features on MNIST, USPS, and DIGITS (IBk variants).Figure6 shows the results with SVM learning. SVM results are clearly improved throughout over all datasets. Also, we see a clear ordering for the right half of each figure: MNIST performs better than USPS, and USPS performs better than DIGITS. For SVM learning, the alternative feature set representation improves the results quite distinctly. Note also that just for 1,800 samples, we can have an error of 2% on the MNIST testset, which is quite good considering that the best published results are at around 0.5% and use orders of magnitude more training data. A linear SVM with unprecedented processing speed is only slightly worse. It might be that the higher accuracy of these systems has enabled us to see the hardness of the dataset—like the harder part of MNIST, DIGITS consisted of data contributed by (university) students. USPS would then be between both datasets in terms of sample complexity.Abs. training set size versus test set accuracy for gradient features on MNIST, USPS, and DIGITS (SVM variants).
(a)(b)(c)The shape of all learning curves is remarkably similar and might be estimated with just a few data points. They seem to depend on the learning algorithm, the feature representation, and to a lesser extent on the specific dataset in question (e.g., dataset complexity, sample distribution, or other factors).


## 4.1.1. Pixel-Based Features

Figure3 shows the results for instance-based learning, and Figure 4 shows the results for SVM learning. Both experiments were run on pixel-based features from the three datasets. The test set was fixed while the training set was downsampled to the absolute number of training examples shown as X. The Y-axis shows the accuracy on the test set. Additionally, downsampling was randomized ten times, and the standard deviation over these ten runs is shown as error bars.Abs. training set size versus test set accuracy for pixel-based features on MNIST, USPS, and DIGITS (IBk variants).
(a)(b)(c)Abs. training set size versus test set accuracy for pixel-based features on MNIST, USPS, and DIGITS (SVM variants).
(a)(b)(c)When using instance-based learning, the three datasets perform remarkably similar. Only for the tangent distance variant, DIGITS performs noticeably worse. We presume this is due to the collection of this dataset, where digits had to be  written into a regular grid, which forced a very uniform orientation. As tangent distance was constructed to compensate for non-uniform orientations—wihch is not needed here—the additional degrees-of-freedom of this method may have led to overfitting on this dataset, resulting in inferior performance.When using SVM learning, the picture is similar, albeit less clear. Only for the polynomial variant do we see very similar behavior on the three datasets. For the other two variants, some differences appear. Especially, MNIST performs very badly with the RBF kernel variant. We presume that this is due to the high number of variance in MNIST, and the higher number of parameters for the RBF kernel, such that the amount of training data is no longer sufficient for stable parameter estimation. Also, parameters were optimized for the DIGITS training set, and this may have led to some overfitting4. The polynomial kernel has the best accuracy here, closely followed by the linear kernel. Because of the nonscriptable version for convNN and its long training times, we could not test it here.


## 4.1.2. Gradient-Based Features

In a second step, we analyzed gradient-based features. Since pixel-based features are a very imprecise way to encode information about handwritten digits, we chose to use direction-specific feature maps which were previously found to work best (see Section3.2).Figure5 shows the results with instance-based learning. Here, we only used IBk with one nearest neighbor as the other two distance measures are inappropriate for non-pixel-based data. We again observe similar behavior for all datasets, at a slightly higher level of accuracy than for the pixel-based features and the same learning algorithm. Clearly, adding relevant background knowledge in the form of tangent distance or normalized correlation distance measures is more helpful to improve IBk than this alternative feature representation.Figure 5
Abs. training set size versus test set accuracy for gradient features on MNIST, USPS, and DIGITS (IBk variants).Figure6 shows the results with SVM learning. SVM results are clearly improved throughout over all datasets. Also, we see a clear ordering for the right half of each figure: MNIST performs better than USPS, and USPS performs better than DIGITS. For SVM learning, the alternative feature set representation improves the results quite distinctly. Note also that just for 1,800 samples, we can have an error of 2% on the MNIST testset, which is quite good considering that the best published results are at around 0.5% and use orders of magnitude more training data. A linear SVM with unprecedented processing speed is only slightly worse. It might be that the higher accuracy of these systems has enabled us to see the hardness of the dataset—like the harder part of MNIST, DIGITS consisted of data contributed by (university) students. USPS would then be between both datasets in terms of sample complexity.Abs. training set size versus test set accuracy for gradient features on MNIST, USPS, and DIGITS (SVM variants).
(a)(b)(c)The shape of all learning curves is remarkably similar and might be estimated with just a few data points. They seem to depend on the learning algorithm, the feature representation, and to a lesser extent on the specific dataset in question (e.g., dataset complexity, sample distribution, or other factors).


## 4.2. Dataset-Independence

All previous results mean nothing if the task has not really been solved. So, as it is clear that—small differences between the datasets notwithstanding—all these datasets deal with the writer-independent recognition of handwritten digits and were created by disjunct sets of writers (which were also properly distributed between training and test set), we estimated the quality of each model by testing it on the other datasets. First, we converted both DIGITS and USPS into MNIST format by centering each digit in a28×28 image, equalizing the histograms via nonlinear gamma correction.5 For DIGITS, we estimated center-of-gravity from the original 300 dpi black-and-white images; for USPS, we estimated c-o-g from the grayscale images after thresholding at 50% (128). In both cases, 1 : 1 scaling was used (i.e., the aspect ratio was retained) (Figure 7).(a) USPS, (b) DIGITS, both reformatted to MNIST format.
(a)(b)First, we trained on each training set in turn and tested on the other two sets. Note that the training and test sets are of different size, so for example, MNIST builds a model from 60,000 samples while DIGITS just builds a model from about 1,800. According to the results from the previous section, we would expect a range of about one order of magnitude (best versus worst) in error rates on the test set corresponding to the training set, with MNIST better than USPS and USPS better than DIGITS. This is exactly what we observed. Surprisingly, the performance on the other test sets is much worse.This time, we also tested LeCun’s original convolutional neural network model as reconstructed by [4]. The results from training on the complete training set with the training method described in part in [3] yielded error rates of 0.74% on MNIST-test, 1.72% on the full USPS dataset (i.e., train and test combined), and 8.74% on the full DIGITS dataset. The error rate is significantly higher for both datasets: about twice as high on USPS and more than ten times higher on DIGITS6. We also trained convNN with the same method and similar settings both for DIGITS and USPS.Table1 shows the results for pixel-based features, and Table 2 shows the results for gradient-based features. Both are showing error rates estimated on the respective test sets and the average error of the two other sets divided by the error on the test set corresponding to the training set the model was trained on. What is immediately apparent is that no combination of learning algorithm, feature representation and dataset to train on was able to transfer the usually good results of its own test set to the other test sets without a significant loss in accuracy. A small ratio such as less than 2.0 is only obtainable for unacceptably high error rates. The lowest error rate on any test set of 3.48% is achieved for convNN trained on MNIST and tested on USPS. This still has more than twice this error rate (8.24%) on DIGITS, and about five times the error rate on the MNIST test set (0.74%).Table 1
Dataset independence for pixel-based features, each dataset separately.
ClassifierTrained onTested onAvg. error versus own testsetMNISTDIGITSUSPSIBk1 euclideanMNIST3.0919.2117.495.94xIBk1 euclideanDIGITS36.2216.5952.722.68xIBk1 euclideanUSPS28.4155.015.337.83xIBk1 NCCMNIST2.8317.6513.705.54xIBk1 NCCDIGITS32.4214.1444.592.72xIBk1 NCCUSPS26.0651.174.588.43xIBk1 TDMNIST1.5113.535.636.34xIBk1 TDDIGITS25.8810.0237.773.18xIBk1 TDUSPS10.5136.473.646.45xSVM linearMNIST6.8334.9716.243.75xSVM linearDIGITS31.5716.0945.542.40xSVM linearUSPS40.6463.256.537.95xSVM polynomialMNIST1.2716.2011.5610.93xSVM polynomialDIGITS30.0511.4747.683.39xSVM polynomialUSPS44.7874.334.4313.44xSVM RBFMNIST4.3153.3420.788.60xSVM RBFDIGITS51.5033.7460.091.65xSVM RBFUSPS81.0589.987.3711.60xconvNNMNIST0.748.243.487.92xconvNNDIGITS21.435.7330.04.49xconvNNUSPS4.2527.563.085.16xTable 2
Dataset independence for gradient-based features, each dataset separately.
ClassifierTrained onTested onAvg. error versus own testsetMNISTDIGITSUSPSIBk1 euclideanMNIST1.2912.085.987.00xIBk1 euclideanDIGITS21.917.2937.624.08xIBk1 euclideanUSPS10.3033.073.496.21xSVM linearMNIST1.3412.925.636.92xSVM linearDIGITS19.765.1230.544.91xSVM linearUSPS14.6239.373.348.08xSVM polynomialMNIST0.478.074.4313.30xSVM polynomialDIGITS17.813.6724.865.81xSVM polynomialUSPS14.6839.032.799.63xSVM RBFMNIST0.578.304.2811.04xSVM RBFDIGITS17.754.0625.465.32xSVM RBFUSPS14.8940.032.799.84xSecond, we chose to also test combining two datasets and testing on the remaining dataset. We downsampled the larger training dataset to the size of the smaller training set and combined them, shuffling the results to prevent order effects. The same test sets as previously were used. This time, we computed the error of the remaining completely unseen dataset divided by the average of errors for the two seen datasets (i.e., those whose training set was part of the dataset pool). Again, convNN was trained on the same data.Tables3 and 4 show the results for pixel-based and gradient-based features. Again, we see that the error of the datasets who were part of the training is usually much higher than the one on the completely unseen dataset. This is not true for IBk with tangent-distance, where both for MNIST-DIGITS and USPS-DIGITS the error on DIGITS testset is higher than the one on the completely unseen dataset. This might be due to the small size of the DIGITS test set, which increases the variance of error estimates computed on this dataset. The same happens for SVM linear on USPS-DIGITS and SVM RBF on MNIST-DIGITS and USPS-DIGITS.Table 3
Dataset independence for pixel-based features, two datasets combined.
ClassifierTrained onTested onError versus avg. of own testsetsMNISTDIGITSUSPSIBk1 eucl.MNIST-DIGITS9.2318.3225.561.86xIBk1 eucl.MNIST-USPS5.9725.895.284.60xIBk1 eucl.USPS-DIGITS22.2722.056.981.53xIBk1 NCCMNIST-DIGITS7.8113.6421.672.02xIBk1 NCCMNIST-USPS4.7424.394.535.26xIBk1 NCCUSPS-DIGITS18.9515.536.981.68xIBk1 TDMNIST-DIGITS4.3411.4110.811.37xIBk1 TDMNIST-USPS2.6516.093.645.12xIBk1 TDUSPS-DIGITS9.6114.094.531.03xSVM linearMNIST-DIGITS10.6221.1021.921.38xSVM linearMNIST-USPS12.2743.108.274.20xSVM linearUSPS-DIGITS20.3723.839.671.22xSVM poly.MNIST-DIGITS4.968.8516.542.40xSVM poly.MNIST-USPS2.6622.163.896.77xSVM poly.USPS-DIGITS14.569.975.231.92xSVM RBFMNIST-DIGITS12.6034.5831.191.32xSVM RBFMNIST-USPS13.6071.665.637.45xSVM RBFUSPS-DIGITS39.8947.387.031.47xconvNNMNIST-DIGITS3.214.006.571.82xconvNNMNIST-USPS1.2511.852.745.94xconvNNUSPS-DIGITS7.035.794.881.32xTable 4
Dataset independence for gradient-based features, two datasets combined.
ClassifierTrained onTested onError versus avg. of own testsetsMNISTDIGITSUSPSIBk1 eucl.MNIST-DIGITS3.587.419.871.80xIBk1 eucl.MNIST-USPS1.9815.593.345.86xIBk1 eucl.USPS-DIGITS9.517.804.141.59xSVM linearMNIST-DIGITS3.434.7911.012.68xSVM linearMNIST-USPS2.2315.143.395.39xSVM linearUSPS-DIGITS7.506.244.381.41xSVM poly.MNIST-DIGITS1.843.516.582.46xSVM poly.MNIST-USPS0.9110.752.546.23xSVM poly.USPS-DIGITS5.944.452.941.61xSVM RBFMNIST-DIGITS1.883.516.982.59xSVM RBFMNIST-USPS1.0011.532.446.70xSVM RBFUSPS-DIGITS6.224.572.841.68xThe better gradient-based feature representation is probably responsible for preventing such outliers in the second tables, as more stable models are learned. This time, SVM polynomial and SVM RBF give the best performance (averaged over the completely unseen test datasets’ error rates), closely followed by convNN which uses pixel-based features. Still, this translates to an error of 5.94%, 10.75%, and 5.94% for MNIST, DIGITS, and USPS, which is at least an order of magnitude higher than the best results for handwritten digit recognition (reported on MNIST).


## 5. Conclusion

We have shown that relatively small amounts of training data are sufficient for state-of-the-art accuracy in handwritten digit recognition, and that the relationship between training set size and accuracy follows a simple asymptotic function.We have also shown thatnone of the considered learning systems are able to transfer their expertise to other similar handwritten digit recognition datasets. The obtainable error rates are even in the best case far less than what has been reported on single datasets. This indicates that systems learn significant non-task-specific idiosyncrasies of specific datasets or not sufficiently well-documented preprocessing methods and do not yet offer stable dataset-independent performance. Thus present systems can be considered brittle in AI terminology, albeit at higher performance level than previous classical AI systems.More work is needed to determine how to resolve this weakness. As a first step, we propose a more detailed documentation of preprocessing methods and classification systems in the form of Open Source code for further work in the field, a more comprehensive sharing of both data and methods among active research groups, and focussing specific efforts towards building more robust learning systems. An investigation into specific preprocessing choices and their effect on accuracy would be highly desirable and a major step to building systems with truly stable dataset-independent performance.



---

*Source: 834127-2011-11-30.xml*