Title: Deep Error-Correcting Output Codes
Authors: Guoqiang Zhong
Yuchen Zheng
Peng Zhang
Mengqi Li
Junyu Dong
Venue: ICLR 2017

Abstract: Existing deep networks are generally initialized with unsupervised methods, such as random assignments and greedy layerwise pre-training. This may result in the whole training process (initialization/pre-training + fine-tuning) to be very time consuming. In this paper, we combine the ideas of ensemble learning and deep learning, and present a novel deep learning framework called deep error-correcting output codes (DeepECOC). DeepECOC are composed of multiple layers of the ECOC module, which combines multiple binary classifiers for feature learning. Here, the weights learned for the binary classifiers can be considered as weights between two successive layers, while the outputs of the combined binary classifiers as the outputs of a hidden layer. On the one hand, the ECOC modules can be learned using given supervisory information, and on the other hand, based on the ternary coding design, the weights can be learned only using part of the training data. Hence, the supervised pre-training of DeepECOC is in general very effective and efficient. We have conducted extensive experiments to compare DeepECOC with traditional ECOC, feature learning and deep learning algorithms on several benchmark data sets. The results demonstrate that DeepECOC perform not only better than traditional ECOC and feature learning algorithms, but also state-of the-art deep learning models in most cases. 


Review discussion:
1. In your experiments, can you clarify that you use the same training procedure for DeepECOC and DAE, do you first pre-train and then fine tune the network with dropout for both methods ?
2. Why do you use SVM as the binary classifier and not another method, such as (regularized) logistic regression as this would matches your subsequent use of a sigmoid? How was the C parameter of SVM set?
3. Clarifications for MNIST results: Figure 3 is difficult to read a table would be much better.
Also accuracies for the non-convolutional architectures seem much worse than what is typically reported with stacked DAE for instance. (>98.7%). Howcome?
Also what are Sparse, Dense, Single?
Many thanks for your review!
1. We used same pre-training+finetuning procedure for the training of both DeepECOC and DAE. For the dropout regularization, we used it in DeepECOC but not DAE.
However, we have compared the results of DAE with dropout and without dropout. Since denoising can be considered as a regularization method (dropout is also), DAE with dropout and without dropout perform similarly.
2. SVMs are widely used in traditional ECOC methods.
However, (regularized) logistic regression is used much less.
So, we used SVMs in the proposed DeepECOC model. Here, even we use (regularized) logistic regression as binary classifiers of the ECOC modules, the sigmoid activation functions are still necessary. For the C parameter in SVMs, we used the default value set in the LIBSVM toolbox.
3. Fig. 3 shows the results obtained by different nets with the same structure. We can change it into a table in next version of this paper. In Fig. 3(a), we first design the deep architecture using stacked ECOC modules, and adopt this architecture for other compared deep networks. We can see that DeepECOC performed much better than the compared methods. In Fig 3(b), we use a state-of-the-art architecture for all the methods. We can see that DeepECOC performed similarly with other deep learning methods, and significantly better than single layer ECOC. Here, we focus on comparing related deep architectures but not achieving the best results on MNIST. The results may be a little lower than the highest reported in the literature. For Sparse, Dense and Single, they refer to DeepECOC initialized with random sparse coding, DeepECOC initialized with random dense coding and single layer ECOC (traditional ECOC), respectively.
The paper proposes a strategy to pre-train the successive layers of a multi-layer perceptron for classification tasks. It consists in training successively each layer to predict an ECOC corresponding to the classification problem. The first layer predicts this ECOC from the input data and each successive layer takes as output its preceding layer to predict another ECOC. Different regularization strategies are used during training (input noise, Dropout). The MLP is then fine-tuned using SGD. Comparisons are performed with baselines on different datasets.
This is a preliminary work. The idea might be valuable but it should be pushed further.
Concerning the writing, sections concerning well known notions (e.g. SVM) could be suppressed. Since ECOC are central to this contribution, a more detailed description of the different ECOC strategies would be helpful. The experiments do not compare the proposed model with state of the art classifiers. The experimental conditions should be made more precise, e.g., it is not clear whether the regularization strategies have been used for all the NN architectures or only for the ECOC based ones.
Finally note that there are several papers that try to learn intermediate representations for classification problems, see e.g. Samy Bengio, Jason Weston, David Grangier: Label Embedding Trees for Large Multi-Class Tasks. NIPS 2010: 163-171
Many thanks for your comments!
Firstly, there are very few papers to consider supervised layer-wise pre-traing and deep ensemble learning.
Hence, from the perspective of machine learning, this paper is not only a preliminary work.
Secondly, thanks for pointing out the writing problems. We'd like to revise this paper accordingly.
Thirdly, we have compared the proposed method, DeepECOC, with deep autoencoder (AE), stacked denoising autoencoder (DAE), LeNet and PCANet. Since DeepECOC is generally applied to inputs of the vector form, we think the comparison results are sufficient to show its advantages.
Fourthly, the experimental conditions are applied to all the compared methods.
Finally, thanks for the suggested NIPS paper.
In fact, we have more interests to learn end-to-end deep architectures.
Error correcting output coding is well established supervised learning approach. Stacking several layers of ECOC seems a natural way of extending the framework to deep learning.
The main motivation of the authors here is to reduce the sample complexity of internal binary classifiers by using ternary coding and to learn discriminative representations at every layer thanks to the local supervision. The main idea of this work is interesting.
However, the presentation can be improved and several sections (about SVMs for example) can be shortened.
I am not convinced by the advantage of this approach compared to end-to-end training of neural networks of other layer-wise training of deep architecture.
In particular, when the codes at every layer can are randomly generated the induced binary problems can be arbitrarily difficult and this could lead to learnability issues hence affecting the quality of the features.
While this problem is counterbalanced by the decoding scheme (and error-correction) at the prediction layer, it is not clear how it should be dealt with in the intermediate layers.
In addition, other approaches such as stacked RBMs or Autoencoders learn regularities capturing information about the distribution of the data at every layer.
In comparison, it is not clear what the learned representation by an ECOC layer is unless it is supervised (One versus Rest).
Overall, though interesting, I am not convinced by the proposed approach and the experimental section does not help mitigate the impression.
More insights on the learned internal representations, and experiments using standard datasets could help.
Many thanks for your review!
Firstly, thank you for pointing out the writing problems. We will improve our paper accordingly.
Secondly, please just consider ECOC as a building block in DeepECOC, like RBMs in deep autoencoders. The advantages using ECOC are: 1) its learning is supervised (we proposed a supervised layer-wise pre-training method); 2) for the pre-training of some weights, according to the ECOC coding, some classes of data may not be used; 3) we can use random dense or random sparse ECOC coding design, which is equivalent to random initialization of the network structure (the weights are still learned based on the binary classsifiers in the ECOC module).
So sorry for the misleading! It's really difficult to clarify so many concepts in one paper.
The paper proposes a greedy supervised layer-wise initialization strategy for (deep) multi-layer perceptrons. Layer weights are initialized by training linear SVMs for binary classification where the binary targets are constructed as error correcting codes (ECOC, including one-vs-all, one-vs-one and others). The thus pertained model (together with a softmax output layer) is then globally fine-tuned by backdrop with dropout.
Note that as a heuristic greedy supervised layer-wise initialization strategy this work is very similar to the author’s other ICLR submission: « Marginal Deep Architectures: Deep learning for Small and Middle Scale Applications ». The two works differ in the supervised initialization strategy employed.
While layer-wise initialization strategies are worthy of further exploration, the paper doesn’t convey any insight as to what makes a better strategy. Experimental results are not sufficiently convincing by themselves alone to support a mostly incremental work;
in particular I remain unconvinced that competing methods received full proper hyper-parameter tuning of their own. Results showing accuracies as bar graphs make it hard to read-off precise accuracies, and one cannot easily compare with known state-of-the-art performance references on benchmark problems (such as MNIST). CIFAR10 performance seem far from state-of-the-art.
Explanations are unnecessarily detailed for standard algorithms (e.g. SVMs) and not sufficiently for aspects specific to the approach such as lesser known ECOC schemes.
One important aspect remains unclear regarding the use of SVMs. Did you use linear SVMs as stated in section 3.1 (« In order to take the probabilistic outputs of the base classifiers as new representations of data, we adopt linear support vector machines (linear SVMs) as the binary classifiers »)
or kernel SVMs as mentioned later « For all DeepECOC models, we used support vector machines (SVMs) with RBF kernel function». In the latter case, the paper lacks a description of how learned RBF-kernel SVMs are transferred to a deep network layer (does it yield 2 layers the firrst of which would be a large RBF neural layer?) Also in this case of kernel SVMs the computational cost is likely to skyrocket and the method will have scaling issues. Is this the reason why the method is too expensive to use on CIFAR10 from scratch, and prompts doing LBP first? If you used linear SVMs, did you use an efficient implementation specific to linear SVMS (as opposed to generic kernel SVM code with a linear kernel?).
Finally for image datasets, a visual comparison of learned filters could help provide some qualitative insight.
Many thanks for your review!
Firstly, this paper and the other ICLR submission about MDA are quite different, in the aspect of motivation and experiments.
Secondly, in this paper, we extend traditional ECOC to a deep model and solve the problem that previous deep learning methods only initialized randomly or using unsupervised pre-training. This is the main contribution of our work. We think the reviewer may pay more attention to the experiments but neglect our main contribution.
Thirdly, although we used CIFAR-10 for comparison, please note that the experimental settings are different from the regular ones. Here, we apply the compared methods on the LBP features of the images.
Finally, sorry for the typos in Section 3.1.
In fact, we use RBF SVMs in all the experiments. We extract the LBP features of images is because some deep learning methods, such as autoencoders, denoising autoencoders and DeepECOC, can only be applied on vector inputs. For the efficient learning of SVMs, it's not in the scope of this paper.

Meta-review:
role: PC
data: comment: The reviewers unanimously recommend rejecting this paper.

Decision: Reject