Title: DEEP ERROR-CORRECTING OUTPUT CODES

Abstract: Existing deep networks are generally initialized with unsupervised methods, such as random assignments and greedy layerwise pre-training. This may result in the whole training process (initialization/pre-training + fine-tuning) to be very timeconsuming. In this paper, we combine the ideas of ensemble learning and deep learning, and present a novel deep learning framework called deep error-correcting output codes (DeepECOC). DeepECOC are composed of multiple layers of the ECOC module, which combines multiple binary classifiers for feature learning. Here, the weights learned for the binary classifiers can be considered as weights between two successive layers, while the outputs of the combined binary classifiers as the outputs of a hidden layer. On the one hand, the ECOC modules can be learned using given supervisory information, and on the other hand, based on the ternary coding design, the weights can be learned only using part of the training data. Hence, the supervised pre-training of DeepECOC is in general very effective and efficient. We have conducted extensive experiments to compare DeepECOC with traditional ECOC, feature learning and deep learning algorithms on several benchmark data sets. The results demonstrate that DeepECOC perform not only better than traditional ECOC and feature learning algorithms, but also state-ofthe-art deep learning models in most cases.

1. INTRODUCTION:

Error correcting output codes (ECOC) are an ensemble learning framework to address multi-class classification problems (Dietterich & Bakiri, 1995) . The work by (Zhong & Liu, 2013) shows that the ECOC methods can also be used for feature learning, in either a linear or a nonlinear manner. However, although sophisticated coding and decoding strategies are applied (Escalera et al., 2010; Zhong et al., 2012; Zhong & Cheriet, 2013) , the learnability of ECOC is limited by its singlelayer structure. Therefore, to exploit the advantages of the ECOC framework, such as supervised ensemble learning and effective coding design, it's necessary to combine its ideas with that of deep learning.

In recent years, many deep learning models have been proposed to handle various challenging problems. Meantime, desirable performances in many domains have been achieved, such as image classification and detection, document analysis and recognition, natural language processing, and video analysis (Hinton & Salakhutdinov, 2006; Krizhevsky et al., 2012; Szegedy et al., 2014; Simonyan & Zisserman, 2014; Zhang et al., 2015; Wang & Ji, 2015; Hong et al., 2015) . Among others, (Hinton & Salakhutdinov, 2006) presents the ground-breaking deep autoencoder that learns the weight matrices by pre-training the stacked restricted Boltzmann machines (RBMs) and fine-tuning the weights using gradient descent. It delivers much better representations of data than shallow feature learning algorithms, such as principal components analysis (PCA) (Jolliffe, 1986) and latent semantic analysis (LSA) (Deerwester et al., 1990) . In order to boost the traditional autoencoder and prevent the "overfitting" problem, (Vincent et al., 2008) introduces the denosing autoencoder that corrupted the data with a random noise. Recently, most of the research focuses on deep convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which greatly improves the state-of-the-art in the areas of object recognition, unsegmented handwriting recognition and speech recognition (Krizhevsky et al., 2012; Graves et al., 2009; Sak et al., 2014) . However, existing deep networks are generally initialized with unsupervised methods, such as random assignments and greedy layerwise pre-training. In the case of random initialization, to obtain good results, many training data and a long training time are generally used; while in the case of greedy layerwise pre-training, as the whole training data set needs to be used, the pre-training process is very time-consuming and difficult to find a stable solution.

To overcome the limitations of both traditional ECOC methods and deep learning models, and meanwhile, take advantages of both of them, in this paper, we propose a novel deep learning model called deep error-correcting output codes (DeepECOC). DeepECOC are composed of multiple stacked E-COC modules, each of which combines multiple binary classifiers for feature learning. Here, the weights learned for the binary classifiers can be considered as weights between two successive layers, while the probabilistic outputs of the combined binary classifiers as the outputs of a hidden layer or new representations of data. On the one hand, the ECOC modules can be learned layer by layer using the given supervisory information, and on the other hand, based on the ternary coding design, some classes of data are automatically neglected when training the binary classifiers, such that the weights are learned only using part of the training data. Hence, the supervised pre-training of DeepECOC is in general very effective and efficient. We have compared DeepECOC with traditional ECOC, feature learning and deep learning algorithms to demonstrate the effectiveness and superiority of DeepECOC. The results are reported in Section 4.

The rest of this paper is organized as follows: In Section 2, we give a brief overview to related work. In Section 3, we present the proposed model, DeepECOC, in detail. The experimental results are reported in Section 4, while Section 5 concludes this paper with remarks and future work.

2. RELATED WORK:

Traditional ECOC framework has two steps: coding and decoding. In the coding step, an E-COC matrix is defined or learned from data, and the binary classifiers are trained based on the ECOC coding; in the decoding step, the class label is given to a test sample based on a similarity measure between codewords and outputs of the binary classifiers. The widely used coding strategies include one-versus-all (OneVsAll) (Nilsson, 1965) , one-versus-one (OneVsOne) (Hastie et al., 1998) , discriminant ECOC (DECOC) (Pujol et al., 2006) , ECOC optimizing node embedding (E-COCONE) (Escalera et al., 2006) , dense and sparse coding (Escalera et al., 2009; Allwein et al., 2001) , and so on. Among them, the OneVsAll, OneVsOne, dense and sparse coding strategies are problem-independent, whilst the DECOC and ECOCONE are problem-dependent. Generally, the length of the codeword by the OneVsAll, OneVsOne, DECOC and ECOCONE coding designs is related to the number of classes, but that by the dense and sparse coding design is relatively flexible. In this work, we design the structure of DeepECOC based on the properties of each coding strategy. The commonly used binary ECOC decoding strategies are the Hamming decoding (Nilsson, 1965) and Euclidean decoding (Hastie et al., 1998) . For ternary ECOC decoding strategies, the attenuated Euclidean decoding (Pujol et al., 2008) , loss-based decoding (Allwein et al., 2001) , and probabilistic-based decoding (Passerini et al., 2004) are widely used. Currently, the state-of-the-art ternary ECOC decoding strategies are the discrete pessimistic beta density distribution decoding and loss-weighted decoding (Escalera et al., 2010) . In this work, for the simplicity of back propagation, we directly add a Softmax layer at the top of DeepECOC for the decoding. Note that, although many sophisticated coding and decoding strategies have been proposed in recent years (Escalera et al., 2010; Zhong et al., 2012; Zhong & Cheriet, 2013) , the learnability of ECOC is limited by its single-layer structure. To further exploit the advantages of ECOC, such as supervised ensemble learning and effective coding design, it's necessary to combine its ideas with that of deep learning. In the literature of deep learning, there is some work that attempts to construct a deep architecture with multiple feature learning methods (Hinton & Salakhutdinov, 2006; Trigeorgis et al., 2014; Yuan et al., 2015; Zheng et al., 2015; 2014) . For instance, deep autoencoder is built up by RBMs (Hinton & Salakhutdinov, 2006) , and deep semi-NMF combines multiple steps of matrix factorization (Trigeorgis et al., 2014) . Similarly, deep CNNs and RNNs can also be considered as deep models that learn the new representations of data layer by layer (Krizhevsky et al., 2012; Graves et al., 2009; Sak et al., 2014) . The success of these existing models demonstrate that deep networks are beneficial to the representation learning tasks, especially for the large scale applications. However, as discussed in the previous section, existing deep learning models are generally initialized with unsupervised methods, such as random assignments and greedy layerwise pre-training, which result in a long training time of the deep models. In this work, we propose the DeepECOC model, which is based on the stacked ECOC modules. When pre-training DeepECOC, the ECOC modules can be learned with the available supervisory information. Intuitively, as this manner of supervised pre-training has deterministic objective, the learned value of the parameters will be very close to the best local minimum on the solution manifold. Experimental results shown in Section 4 also demonstrate this fact.

3. DEEP ERROR-CORRECTING OUTPUT CODES (DEEPECOC):

In this section, we first introduce the traditional ECOC framework, which is the important building block of DeepECOC. Then we present the learning procedures of DeepECOC in detail.

3.1. THE ECOC FRAMEWORK:

Error correcting output codes (ECOC), which combine multiple binary classifiers to solve multiclass classification problems, are an ensemble learning framework. The ECOC methods in general consist of two steps: coding and decoding. In the coding step, the ECOC coding matrix M ∈ {-1, 1} C×L (binary case) or M ∈ {-1, 0, 1} C×L (ternary case) is first defined or learned from the training data, where each row of M is the codeword of a class, each column corresponds to a dichotomizer (binary classifier), L is the length of the codewords (the number of binary classifiers), C is the number of classes, symbol '1' indicates positive class, '-1' indicates negative class, and '0' indicates that a particular class is not considered by a given classifier. Then, the binary classifiers (dichotomizers) are trained according to the partition of the classes in the columns of M. Fig. 1 shows two coding matrices encoded with the one-versus-all (binary case) and one-versus-one (ternary case) coding strategies. The matrix is coded using several dichotomizers for a 4-class problem with respective codewords {y 1 , . . . , y 4 }. The white girds are coded by 1 (considered as positive class by the respective dichotomizer h j ), the dark girds by -1 (considered as the negative class), and the gray girds by 0 (classes that are not considered by the respective dichotomizer h j ). In the decoding step, the test data are predicted based on an adopted decoding strategy and the outputs of the binary classifiers.

In order to take the probabilistic outputs of the base classifiers as new representations of data, we adopt linear support vector machines (linear SVMs) as the binary classifiers (dichotomizers), which solve a quadratic programming problem min w,b,ξi

EQUATION

where w and b are the coefficients and bias of the binary classifier, y i ∈ {+1, -1}, ξ i 's are the slack variables, and N is the number of the training data. The discriminant function can be expressed as

EQUATION

This problem can be solved as,

EQUATION

EQUATION

where α i 's are the non-negative Lagrange multipliers, N sv is the number of support vectors and SV is the set of support vectors. The dual form of Problem (1) can be written as

EQUATION

where k(x i , x j ) = x T i x j is the linear kernel function, λ is a constant number, and α = {α 1 , . . . , α N } is the vector of Lagrange multipliers. Replacing the linear kernel function with a nonlinear kernel, such as the Gaussian kernel

EQUATION

we can learn a nonlinear SVM, where σ is the parameter for the Gaussian kernel function. The discriminant function of SVMs with a nonlinear kernel can be written as

EQUATION

Applying a decoding strategy on the outputs of the binary classifiers, the ECOC framework can be used for multi-class learning, while applying the sigmoid function on the values of the discriminant function, ECOC can be used for feature learning (Zhong & Liu, 2013) . This is also the foundation of the DeepECOC model.

3.2. DEEPECOC:

To combine the advantages of ECOC and deep learning algorithms, we build the DeepECOC architecture as follows

EQUATION

where the first step makes the clean input x ∈ [0, 1] d partially destroyed by means of a stochastic mapping x ∼ q D (x | x). In the corrupting process, we set a parameter called denoising rate ν.

For each input x, a fixed number νd of components are chosen at random, and their value is forced to 0, while the others are left untouched. This operation makes the model more robust and prevent the overfitting problem in most cases (Vincent et al., 2008) . Subsequently, the "corrupted" data are taken as inputs for the DeepECOC model. W 1 and b 1 are the weight matrix and bias vector learned from the first ECOC module. The output of the first hidden layer is denoted as

EQUATION

where s(•) is the sigmoid activation function s(x) =1 1+e -x . From the second layer to the (n -1)th layer, we use the stacked ECOC modules to learn the weight matrices and biases, which can be considered as weights between two successive layers of a deep network. Similarly, we use the output of the (k -1)-th layer as the input of the k-th layer,

EQUATION

Here, h k can be viewed as an activation output and a new representation of the input datum x.

For example, if we adopt the OneVsAll coding strategy for one layer of the ECOC module, we first define the coding matrix M C×C , where C is the number of classes. Then, we can train C SVM classifiers to obtain the weight matrix W = {w 1 , . . . , w i , . . . , w C } and the bias b = {b 1 , . . . , b i , . . . , b C }. Next, we calculate the output of the first layer by using Eq. ( 9). Subsequently, we repeat this process layer by layer to build the DeepECOC model. It's obvious that, if we adopt different coding strategies, we can get different kinds of DeepECOC architectures.

For the last layer of DeepECOC, we employ the softmax regression for the multi-class learning. Its cost function is defined as

EQUATION

where I(x) is the indicator function, I(x) = 1 if x is true, else I(x) = 0. y i is the label corresponding to x i . It's easy to compute the probability that x i is classified to class j,

EQUATION

Taking derivatives, one can show that the gradient of J(w) with respect to w is,

EQUATION

After the pre-training step, we use back propagation (Rumelhart et al., 1988) to fine tune the whole architecture. Moreover, we also employ a technique called "dropout" for regularization (Hinton et al., 2012) . When a large feedforward neural network is trained on a small training set, dropout generally performs well on the test set. The basic idea of dropout is that each hidden node is randomly omitted from the network with a probability of β. In another view, dropout is a very efficient way to perform model averaging with neural networks. Through these processes, we finally obtain the DeepECOC model, which is robust and easy to be applied to multi-class classification tasks.

Note that, compared to existing deep learning algorithms, DeepECOC have some important advantages. Firstly, unlike previous deep learning algorithms, DeepECOC are built with the ECOC modules and pre-trained in a supervised learning fashion. Secondly, if we adopt ternary coding strategies, due to the natural merit of ECOC, the weights can be learned using only part of the training data. Thirdly, in contrast to the learning of the weight matrices in previous deep learning models, the binary classifiers in each ECOC module can be learned in parallel, which may greatly speed up the learning of DeepECOC.

4. EXPERIMENTS:

To evaluate the effectiveness of the proposed method, DeepECOC, we conducted 4 parts of experiments. In the first part, we compared DeepECOC with some deep learning models and single-layer ECOC approaches on 16 data sets from the UCI machine learning repository 1 . In the second part, we compared DeepECOC with traditional feature learning models, some deep learning models and single-layer ECOC approaches on the USPS handwritten digits2 , and tested DeepECOC with different number of hidden layers. In the third part, we used the MNIST handwritten digits3 to further demonstrate the effectiveness of DeepECOC for handwritten digits recognition. Finally, the CIFAR-10 data set4 was used to demonstrate the effectiveness of DeepECOC on image classification tasks. For all the data sets, the features were normalized within [0, 1]. In the following, we report the experimental results in detail.

4.1. CLASSIFICATION ON 16 UCI MACHINE LEARNING REPOSITORY DATA SETS:

The detail of the UCI data sets are shown in Table 1 . In these experiments, we compared DeepECOC with autoencoder (AE) (Hinton & Salakhutdinov, 2006) , denoising autoencoder (DAE) (Vincent et al., 2008) and single-layer ECOC approaches (Single) (Escalera et al., 2010) . We built DeepECOC with the ECOC optimizing node embedding (ECOCONE) coding method (Escalera et al., 2006) .

Here, since we initialized ECOCONE with 3 different coding methods, i.e. one-versus-one, oneversus-all and DECOC, DeepECOC had 3 variants. In addition, the state-of-the-art linear loss-weighted (LLW) decoding strategy was used for ECOCONE. Finally, a structure with 3 hidden layers was adopted for DeepECOC, which had 0.1 denoising rate and 0.1 dropout rate:

EQUATION

For the fine-tuning process, we used the stochastic gradient descent algorithm. The learning rate and epoches from different data sets are described in Table 3 . The autoencoder and denoising autoencoder's architectures are as same as DeepECOC with ECOCONE initialized by one-versus-one.

For single-layer ECOC approaches, we chose the best results shown in (Escalera et al., 2010) as our compared results. For all DeepECOC models, we used support vector machines (SVMs) with RBF kernel function as base classifiers. The parameters of SVMs were set to default (Chang & Lin, 2011) .

Table 2 shows the average classification accuracy and standard deviation on 16 UCI data sets. Except on the OptDigits data set, DeepECOC achieved the best results compared with autoencoder, denoising autoencoder and single-layer ECOC approaches. In fact, on the OptDigits data set, DeepECOC achieved comparative result with single-layer ECOC approaches. Among others, DeepECOC with ECOCONE (initialized by one-versus-one) coding strategy obtained the best results on 9 data sets, while DeepECOC with ECOCONE (initialized by DECOC) coding strategy obtained the best results on 5 data sets. From the mean rank values, we can see that DeepECOC with ECOCONE (initialized by one-versus-one and DECOC) strategy far surpass other compared methods. The USPS handwritten digits data set includes 7291 training samples and 2007 test samples from 10 classes. The size of the images is 16 × 16 = 256. Our experiments on this data set were divided into 2 parts. Firstly, we compared DeepECOC with two traditional feature learning models (principal components analysis (PCA) (Jolliffe, 2002) and marginal Fisher analysis (MFA) (Yan et al., 2007) ), autoencoder (AE), denoising autoencoder (DAE), LeNet (LeCun et al., 1998) , PCANet (Chan et al., 2015) and single-layer ECOC approaches. Here, PCA is an unsupervised method, MFA is a supervised method. For MFA, the number of nearest neighbors for constructing the intrinsic graph was set to 5, while that for constructing the penalty graph was set to 15. For DeepECOC, we also used 3 coding design methods in this experiment. We used batch gradient descent for the fine-tuning process, the batch size was set to 100, the learning rate was set to 1, the number of epoch was set to 40000, the denoising rate, and dropout rate were set to 0.1. We also used SVMs with RBF kernel and default parameters as base classifiers. For single-layer ECOC approaches, we adopted ECOCONE (initialized by one-versus-one) as coding design method and linear loss-weighted (LLW) decoding strategy. For the LeNet model, we used 2 convolutional layers, two pooling layers and two fully connected layers. The kernel size of the convolutional layers and pooling layers was set to 2 × 2, the stride was set to 1, the number of nodes of the first layer was set to 200, the epoch was set to 8000, the initial learning rate was set to 0.001, learning rate policy was set to "inv", and the momentum was set to 0.9. For the PCANet model, we used two PCA-filter stages, one binary hashing stage and one blockwise histograms. The filter size, the number of filters, and the block size were set to From Fig. 2 (a), we can see that DeepECOC with ECOCONE (initialized by one-versus-one) coding strategy achieved the best result than other methods include traditional feature learning models, existing deep learning methods and single-layer ECOC approaches.

k 1 = k 2 = 3, L 1 = L 2 = 4,

In the second part, we evaluated DeepECOC with different number of hidden layers. We used 2 to 6 hidden layers in our experiments. The parameter settings were as same as the first part. shows the experimental results. We can see that DeepECOC obtained the best result when using 3 hidden layers. When the number of hidden layers is less than 3, the effectiveness of DeepECOC increases with the increasing of the number of hidden layers. Along with the number of hidden layers continues to grow, the effectiveness of DeepECOC decreases.

4.3. CLASSIFICATION ON THE MNIST DATA SET:

MNIST handwritten digits data set has a training set of 60,000 examples, and a test set of 10,000 examples with 784 dimensional features. We designed 2 architectures for autoencoder, denoising autoencoder and DeepECOC. The first architecture was 784 -Z 1 -Z 2 -Z 3 -10, where Z i was the number of hidden neurons designed based on some ECOC coding strategies. We designed this architecture because we wanted to make autoencoder and denoising autoencoder had the same structure with DeepECOC. The second architecture is 784-500-500-2000-10. This architecture was used in (Hinton & Salakhutdinov, 2006) . In order to make DeepECOC adapt to this structure, we used the dense and sparse coding design methods that can control the codeword length. Note that, the dense and sparse coding design methods are totally random and data-independent. The denoising rate and dropout rate were set to 0.1, the batch size was set to 100, the learning rate was set to 0.01, and the number of epoch was set to 80000. For LeNet model, we adopted the parameters as same as (LeCun et al., 1998) . For PCANet model, we used two PCA-filter stages, one binary hashing stage and one blockwise histograms. In the PCANet, the filter size, the number of filters, and the block size were set to k 1 = k 2 = 8, L 1 = L 2 = 7, and 7 × 7, respectively. For the purpose of reducing computational cost, we attempted to extract features of the data using an efficient local binary patterns algorithm. As a result, the representations with dimensionality 36 and 256 were adopted and the data were normalized to [0, 1] as well, called LBP-CIFAR10 (36) and LBP-CIFAR10 (256). We also used 3 hidden layers for all deep learning methods. The learning rate was set to 0.1, and the epoch was set to 4000. For the LeNet model, we used 2 convolutional layers and two fully connected layers without pooling layers. The kernel size was set to 2 × 2, the stride was set to 1, the number of node of the first fully connected layer was set to 64, the epoch was set to 4000, the initial learning rate was set to 0.01, learning rate policy was set to "inv", and the momentum was set to 0.9. For the PCANet model, we used two PCA-filter stages, one binary hashing stage and one blockwise histograms. In the PCANet, the filter size, the number of filters, and the block size were set to k 1 = k 2 = 3, L 1 = L 2 = 4, and 7 × 7, respectively. The classification accuracy are reported in Table 4 .

From Table 4 , we can easy to see that DeepECOC achieved the best results. Moreover, DeepECOC with ECOCONE (initialized by one-versus-one) coding strategy achieved the better results than autoencoder and denoising autoencoder, LeNet and PCANet. Hence, we can conclude that, Deep-ECOC are a general model to handle different real world applications and achieves desirable results in most cases.

:

http://archive.ics.uci.edu/ml/

http://www-i6.informatik.rwth-aachen.de/∼keysers/usps.html

http://yann.lecun.com/exdb/mnist/

http://www.cs.toronto.edu/∼kriz/cifar.html

CONCLUSIONIn this paper, we propose a novel deep learning model, called deep error correcting output codes (DeepECOC). DeepECOC extend traditional ECOC algorithms to a deep architecture fashion, and meanwhile, brings new elements to the deep learning area, such as supervised initialization, and automatic neglecting of part of the data during network training. Extensive experiments on 16 data sets from the UCI machine learning repository, the USPS and MNIST handwritten digits and the CIFAR-10 data set demonstrate the superiority of DeepECOC over traditional ECOC, feature learning and deep learning methods. In future work, we will further exploit the learnability of DeepECOC on large scale applications.

Figures and Tables:

Figure1: Two coding matrices encoded with the one-versus-all (binary case) and one-versus-one (ternary case) coding strategies.

and 7×7, respectively. The experimental results are shown in Fig. 2(a).

Figure 2: (a) Classification accuracy obtained on the USPS data set. Here, DeepECOC(1)∼ Deep-ECOC(3) are 3 variant of DeepECOC with the ECOCONE coding design initialized by one-versusone, one-versus-all and DECOC respectively. (b) Classification accuracy with different numbers of hidden layers on the USPS data set.

Fig. 3(a)  andFig. 3(b)  show the experimental results on 2 architectures. We can see that DeepECOC are comparative with existing deep learning methods on the second architecture and outperform

Details of the UCI data sets (T: training samples; A: attributes; C: classes).

Table content: <table><tr><td>Problem</td><td>of T</td><td>of A</td><td>of C</td><td>Problem</td><td>of T</td><td>of A</td><td>of C</td></tr><tr><td>Dermatology</td><td>366</td><td>34</td><td>6</td><td>Yeast</td><td>1484</td><td>8</td><td>10</td></tr><tr><td>Iris</td><td>150</td><td>4</td><td>3</td><td>Satimage</td><td>6435</td><td>36</td><td>7</td></tr><tr><td>Ecoli</td><td>336</td><td>8</td><td>8</td><td>Letter</td><td>20000</td><td>16</td><td>26</td></tr><tr><td>Wine</td><td>178</td><td>13</td><td>3</td><td>Pendigits</td><td>10992</td><td>16</td><td>10</td></tr><tr><td>Glass</td><td>214</td><td>9</td><td>7</td><td>Segmentation</td><td>2310</td><td>19</td><td>7</td></tr><tr><td>Thyroid</td><td>215</td><td>5</td><td>3</td><td>Optdigits</td><td>5620</td><td>64</td><td>10</td></tr><tr><td>Vowel</td><td>990</td><td>10</td><td>11</td><td>Shuttle</td><td>14500</td><td>9</td><td>7</td></tr><tr><td>Balance</td><td>625</td><td>4</td><td>3</td><td>Vehicle</td><td>846</td><td>18</td><td>4</td></tr></table>

Classification accuracy and standard deviation obtained by DeepECOC and the compared approaches on 16 UCI data sets. Here, DeepECOC(1)∼ DeepECOC(3) are 3 variant of DeepECOC with the ECOCONE coding design initialized by one-versus-one, one-versus-all and DECOC respectively. The best results are highlighted in boldface.

Table content: <table><tr><td>Problem</td><td>Single</td><td>AE</td><td>DAE</td><td>DeepECOC (1)</td><td>DeepECOC (2)</td><td>DeepECOC (3)</td></tr><tr><td>Dermatology</td><td>0.9513</td><td>0.9429 ±0.0671</td><td>0.9674 ±0.0312</td><td>0.9702 ±0.0354</td><td>0.9779 ±0.0208</td><td>0.9747 ±0.0318</td></tr><tr><td>Iris</td><td>0.9600</td><td>0.9600 ±0.0562</td><td>0.9333 ±0.0889</td><td>0.9600 ±0.0535</td><td>0.9267 ±0.1109</td><td>0.9533 ±0.0383</td></tr><tr><td>Ecoli</td><td>0.8147</td><td>0.7725 ±0.0608</td><td>0.8000 ±0.0362</td><td>0.8529 ±0.0403</td><td>0.8824 ±0.0626</td><td>0.9118 ±0.0636</td></tr><tr><td>Wine</td><td>0.9605</td><td>0.9765 ±0.0264</td><td>0.9563 ±0.0422</td><td>0.9875 ±0.0264</td><td>0.9813 ±0.0302</td><td>0.9688 ±0.0329</td></tr><tr><td>Glass</td><td>0.6762</td><td>0.6669 ±0.1032</td><td>0.6669 ±0.0715</td><td>0.7895 ±0.0788</td><td>0.7368 ±0.1140</td><td>0.7562 ±0.0879</td></tr><tr><td>Thyroid</td><td>0.9210</td><td>0.9513 ±0.0614</td><td>0.9599 ±0.0567</td><td>0.9656 ±0.0513</td><td>0.9703 ±0.0540</td><td>0.9608 ±0.0518</td></tr><tr><td>Vowel</td><td>0.7177</td><td>0.6985 ±0.0745</td><td>0.7101 ±0.0756</td><td>0.7475 ±0.0901</td><td>0.6010 ±0.0627</td><td>0.6863 ±0.0788</td></tr><tr><td>Balance</td><td>0.8222</td><td>0.8036 ±0.0320</td><td>0.8268 ±0.0548</td><td>0.9137 ±0.0412</td><td>0.8333 ±0.0318</td><td>0.9167 ±0.0312</td></tr><tr><td>Yeast</td><td>0.5217</td><td>0.5641 ±0.0346</td><td>0.5891 ±0.0272</td><td>0.5959 ±0.0599</td><td>0.5494 ±0.0434</td><td>0.5697 ±0.0462</td></tr><tr><td>Satimage</td><td>0.8537</td><td>0.8675 ±0.0528</td><td>0.8897 ±0.0304</td><td>0.8961 ±0.0480</td><td>0.8360 ±0.0390</td><td>0.9077 ±0.0555</td></tr><tr><td>Letter</td><td>0.9192</td><td>0.9234 ±0.0547</td><td>0.9381 ±0.0641</td><td>0.9532 ±0.0341</td><td>0.9247 ±0.0352</td><td>0.9501 ±0.0563</td></tr><tr><td>Pendigits</td><td>0.9801</td><td>0.9831 ±0.0123</td><td>0.9886 ±0.0034</td><td>0.9908 ±0.0031</td><td>0.9866 ±0.0107</td><td>0.9899 ±0.0075</td></tr><tr><td>Segmentation</td><td>0.9701</td><td>0.9584 ±0.0317</td><td>0.9596 ±0.0211</td><td>0.9711 ±0.0286</td><td>0.9584 ±0.0163</td><td>0.9711 ±0.0233</td></tr><tr><td>Optdigits</td><td>0.9982</td><td>0.9785 ±0.0101</td><td>0.9856 ±0.0088</td><td>0.9867 ±0.0096</td><td>0.9848 ±0.0123</td><td>0.9911 ±0.0091</td></tr><tr><td>Shuttle</td><td>0.9988</td><td>0.9953 ±0.0012</td><td>0.9976 ±0.0014</td><td>0.9988 ±0.0021</td><td>0.9983 ±0.0018</td><td>0.9993 ±0.0010</td></tr><tr><td>Vehicle</td><td>0.7315</td><td>0.6987 ±0.0521</td><td>0.7348 ±0.0454</td><td>0.7561 ±0.0480</td><td>0.6908 ±0.04321</td><td>0.7195 ±0.0148</td></tr><tr><td>Mean rank</td><td>4.0938</td><td>4.8750</td><td>3.9375</td><td>1.7500</td><td>3.9375</td><td>2.4063</td></tr></table>

Details of the learning rate η and epoch on the UCI data sets.

Table content: <table><tr><td>Problem</td><td>η</td><td>Epoch</td><td>Problem</td><td>η</td><td>Epoch</td></tr><tr><td>Dermatology</td><td>0.1</td><td>2000</td><td>Yeast</td><td>0.01</td><td>4000</td></tr><tr><td>Iris</td><td>0.1</td><td>400</td><td>Satimage</td><td>0.01</td><td>4000</td></tr><tr><td>Ecoli</td><td>0.1</td><td>2000</td><td>Letter</td><td>0.01</td><td>8000</td></tr><tr><td>Wine</td><td>0.1</td><td>2000</td><td>Pendigits</td><td>0.01</td><td>2000</td></tr><tr><td>Glass</td><td>0.01</td><td>4000</td><td>Segmentation</td><td>0.01</td><td>8000</td></tr><tr><td>Thyroid</td><td>0.1</td><td>800</td><td>Optdigits</td><td>0.01</td><td>2000</td></tr><tr><td>Vowel</td><td>0.1</td><td>4000</td><td>Shuttle</td><td>0.1</td><td>2000</td></tr><tr><td>Balance</td><td>0.1</td><td>4000</td><td>Vehicle</td><td>0.1</td><td>4000</td></tr><tr><td colspan="3">4.2 CLASSIFICATION ON THE USPS DATA SET</td><td/><td/><td/></tr></table>

Classification accuracy obtained on the LBP-CIFAR10 data set. The best result for each scenario is highlighted in bold face. on the first architecture. In addition, DeepECOC with both two architectures outperform the single-layer ECOC approaches.4.4 CLASSIFICATION ON THE LBP-CIFAR10 DATA SETThe CIFAR-10 dataset is a relative large scale data set which consists of 60000 32×32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.

Table content: <table><tr><td>Problem</td><td>AE</td><td>DAE</td><td>LeNet</td><td>PCANet</td><td>DeepECOC(1)</td><td>DeepECOC(2)</td><td>DeepECOC(3)</td></tr><tr><td>LBP-CIFAR10 (36)</td><td>0.3501</td><td>0.3678</td><td>0.3256</td><td>0.2569</td><td>0.5089</td><td>0.4517</td><td>0.4752</td></tr><tr><td>LBP-CIFAR10 (256)</td><td>0.4352</td><td>0.4587</td><td>0.3221</td><td>0.2569</td><td>0.5588</td><td>0.4589</td><td>0.5224</td></tr></table>

Peer review (iclr2017):
The paper proposes a greedy supervised layer-wise initialization strategy for (deep) multi-layer perceptrons. Layer weights are initialized by training linear SVMs for binary classification where the binary targets are constructed as error correcting codes (ECOC, including one-vs-all, one-vs-one and others). The thus pertained model (together with a softmax output layer) is then globally fine-tuned by backdrop with dropout. Note that as a heuristic greedy supervised layer-wise initialization strategy this work is very similar to the author’s other ICLR submission: « Marginal Deep Architectures: Deep learning for Small and Middle Scale Applications ». The two works differ in the supervised initialization strategy employed. While layer-wise initialization strategies are worthy of further exploration, the paper doesn’t convey any insight as to what makes a better strategy. Experimental results are not sufficiently convincing by themselves alone to support a mostly incremental work; in particular I remain unconvinced that competing methods received full proper hyper-parameter tuning of their own. Results showing accuracies as bar graphs make it hard to read-off precise accuracies, and one cannot easily compare with known state-of-the-art performance references on benchmark problems (such as MNIST). CIFAR10 performance seem far from state-of-the-art. Explanations are unnecessarily detailed for standard algorithms (e.g. SVMs) and not sufficiently for aspects specific to the approach such as lesser known ECOC schemes. One important aspect remains unclear regarding the use of SVMs. Did you use linear SVMs as stated in section 3.1 (« In order to take the probabilistic outputs of the base classifiers as new representations of data, we adopt linear support vector machines (linear SVMs) as the binary classifiers ») or kernel SVMs as mentioned later « For all DeepECOC models, we used support vector machines (SVMs) with RBF kernel function». In the latter case, the paper lacks a description of how learned RBF-kernel SVMs are transferred to a deep network layer (does it yield 2 layers the firrst of which would be a large RBF neural layer?) Also in this case of kernel SVMs the computational cost is likely to skyrocket and the method will have scaling issues. Is this the reason why the method is too expensive to use on CIFAR10 from scratch, and prompts doing LBP first? If you used linear SVMs, did you use an efficient implementation specific to linear SVMS (as opposed to generic kernel SVM code with a linear kernel?). Finally for image datasets, a visual comparison of learned filters could help provide some qualitative insight.