Venue: ICLR 2017 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The paper proposes a strategy to pre-train the successive layers of a multi-layer perceptron for classification tasks. It consists in training successively each layer to predict an ECOC corresponding to the classification problem. The first layer predicts this ECOC from the input data and each successive layer takes as output its preceding layer to predict another ECOC. Different regularization strategies are used during training (input noise, Dropout). The MLP is then fine-tuned using SGD. Comparisons are performed with baselines on different datasets.

This is a preliminary work. The idea might be valuable but it should be pushed further. Concerning the writing, sections concerning well known notions (e.g. SVM) could be suppressed. Since ECOC are central to this contribution, a more detailed description of the different ECOC strategies would be helpful. The experiments do not compare the proposed model with state of the art classifiers. The experimental conditions should be made more precise, e.g., it is not clear whether the regularization strategies have been used for all the NN architectures or only for the ECOC based ones.

Finally note that there are several papers that try to learn intermediate representations for classification problems, see e.g. Samy Bengio, Jason Weston, David Grangier: Label Embedding Trees for Large Multi-Class Tasks. NIPS 2010: 163-171

review_title: null

final_score: rating: null
review_content: 1. In your experiments, can you clarify that you use the same training procedure for DeepECOC and DAE, do you first pre-train and then fine tune the network with dropout for both methods ?
2. Why do you use SVM as the binary classifier and not another method, such as (regularized) logistic regression as this would matches your subsequent use of a sigmoid? How was the C parameter of SVM set? 
3. Clarifications for MNIST results: Figure 3 is difficult to read a table would be much better. Also accuracies for the non-convolutional architectures seem much worse than what is typically reported with stacked DAE for instance. (>98.7%). Howcome? Also what are Sparse, Dense, Single? 


review_title: Clarifications

final_score: rating: 3: Clear rejection
review_content: The paper proposes a greedy supervised layer-wise initialization strategy for (deep) multi-layer perceptrons. Layer weights are initialized by training linear SVMs for binary classification where the binary targets are constructed as error correcting codes (ECOC, including one-vs-all, one-vs-one and others). The thus pertained model (together with a softmax output layer) is then globally fine-tuned by backdrop with dropout.

Note that as a heuristic greedy supervised layer-wise initialization strategy this work is very similar to the author’s other ICLR submission: « Marginal Deep Architectures: Deep learning for Small and Middle Scale Applications ». The two works differ in the supervised initialization strategy employed. 

While layer-wise initialization strategies are worthy of further exploration, the paper doesn’t convey any insight as to what makes a better strategy. Experimental results are not sufficiently convincing by themselves alone to support a mostly incremental work; in particular I remain unconvinced that competing methods received full proper hyper-parameter tuning of their own. Results showing accuracies as bar graphs make it hard to read-off precise accuracies, and one cannot easily compare with known state-of-the-art performance references on benchmark problems (such as MNIST). CIFAR10 performance seem far from state-of-the-art.

Explanations are unnecessarily detailed for standard algorithms (e.g. SVMs) and not sufficiently for aspects specific to the approach such as lesser known ECOC schemes. 
One important aspect remains unclear regarding the use of SVMs. Did you use linear SVMs as stated in section 3.1 (« In order to take the probabilistic outputs of the base classifiers as new representations of data, we adopt linear support vector machines (linear SVMs) as the binary classifiers ») 
or kernel SVMs as mentioned later « For all DeepECOC models, we used support vector machines (SVMs) with RBF kernel function». In the latter case, the paper lacks a description of how learned RBF-kernel SVMs are transferred to a deep network layer (does it yield 2 layers the firrst of which would be a large RBF neural layer?) Also in this case of kernel SVMs the computational cost is likely to skyrocket and the method will have scaling issues. Is this the reason why the method is too expensive to use on CIFAR10 from scratch, and prompts doing LBP first? If you used linear SVMs, did you use an efficient implementation specific to linear SVMS (as opposed to generic kernel SVM code with a linear kernel?).

Finally for image datasets, a visual comparison of learned filters could help provide some qualitative insight.

review_title: Review

final_score: rating: 3: Clear rejection
review_content: Error correcting output coding is well established supervised learning approach. Stacking several layers of ECOC seems a natural way of extending the framework to deep learning. 
The main motivation of the authors here is to reduce the sample complexity of internal binary classifiers by using ternary coding and to learn discriminative representations at every layer thanks to the local supervision. The main idea of this work is interesting. However, the presentation can be improved and several sections (about SVMs for example) can be shortened. 

I am not convinced by the advantage of this approach compared to end-to-end training of neural networks of other layer-wise training of deep architecture. In particular, when the codes at every layer can are randomly generated the induced binary problems can be arbitrarily difficult and this could lead to learnability issues hence affecting the quality of the features. While this problem is counterbalanced by the decoding scheme (and error-correction) at the prediction layer, it is not clear how it should be dealt with in the intermediate layers. In addition, other approaches such as stacked RBMs or Autoencoders learn regularities capturing information about the distribution of the data at every layer. In comparison, it is not clear what the learned representation by an ECOC layer is unless it is supervised (One versus Rest).

Overall, though interesting, I am not convinced by the proposed approach and the experimental section does not help mitigate the impression. 
More insights on the learned internal representations, and experiments using standard datasets could help.

 
review_title: review

Meta-review: The reviewers unanimously recommend rejecting this paper.
Decision: Reject