Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper studies the common experimental finding that low level features trained end-to-end in a deep model converge (get "locked in place") earlier than higher level features, which may result in problematic undertraining. The focus of the study is not on skip connections, but really on getting adequate training in deeper networks. They posit a "good classifier hypothesis" where, once a deep network converged, they fix the top layers (the "good classifier") and train only the lower ones. They propose a "top-down training strategy" to search where to make the cut for the "top layers" of the "good classifier", based on the validation set.


 (+) The experimental results seem encouraging and supporting the author's claim (consistently improve over baseline on WSJ and CHiME-4).
 (-) No WER (not even without a language model) results on WSJ make it harder to (i) compare to other work (is it just that in this case the authors didn't optimize properly in the first place?), (ii) compare the relative gains between with and without the method in WER.
 (-) For an experimental (no theorem) optimization paper, there should be experiments on at least another domain. And in particular one would have expected more analysis of the experimental optimization results.
 (-) (minor) There is no discussion of the link with target propagation or other synthetic gradients.

Overall, I think this could be an interesting paper, but more work is needed to prove the effectiveness of the method, and to analyze experimentally in more details some of the claims from this paper.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: 
=========================
Update review
After reading the authors response I would like to keep my score as is.
I still see many unclear statements, and most importantly I feel that more analysis of the proposed method should have been done here. 
=========================

This paper proposed a Top-Down method for neural networks training based on the good classifier hypothesis. In other words, after obtaining a classifier that performs well on the test set, keep fine-tuning / re-learning the data representation.
The authors provide character error rate results for the task of Automatic Speech Recognition using WSJ and CHiME-4 datasets.

Although being an interesting research idea, several issues in this paper make it not yet ready for publication at ICLR. 

First, the paper is poorly written; there are many claims the authors are making without providing experiments/proofs/citations. 
For example: "...since the feature extractor learns more slowly, then potentially the classifier may overfit the feature extractor before the feature extractor is able to learn much about the underlying pattern of the data...". 
Or: "...We suggest that the reason for this is that when all layers are trained on the noisy dataset jointly, the middle layers overfit the bottom-most layers much faster than the bottom-most layers are able to learn input features..."

Next, since there is no theoretical/mathematical explanation of the proposed approach, I expect the authors to run an analysis on the results to better understand the effect of using such an approach. For instance, under which settings this method is most efficient? In what layer should  I start the fine-tuning? Is it better to reinitialize the bottom layers or fine-tune them? Does the proposed approach applicable to different domains? i.e. vision/nlp/other speech/signal processing tasks?  Does the proposed approach applicable to different models or only for the proposed one?

Lastly, although it is not the main point in this paper since all results are reported on ASR, did the authors tried to compute WERs too? That way, people can compare results with other ASR models. The baseline seems relatively weak, at least in Table 1.

Minor comments: 
The complexity of the algorithm is written to be O(n). However, this assumes training the model takes O(1) or did I miss something?
Can the authors provide more details/insights regarding the delta differences in Table 1? Did the authors use the same initializations? Did the authors try different ones?
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This work proposed a mechanism to freeze top layers after supervised pre-training, and re-initialize and retrain the bottom layers. For a model with n layers, when a separation index i is specified, the approach define layer 1~i as bottom layers and i+1~n as top layers. The proposed process enumerate all i from 1 to n-1, compute resulting validation errors respectively, and then pick the i with lowest validation error. The algorithm exhibited significant improvement on WSJ and some minor improvement on CHiME-4.

This work provides some new insight for training ASRs and the observations provide further data points for understanding the training behavior. The layer freezing trick however is relatively well-known, and thus leaving the novelty of the proposed idea to be limited at what layers they choose to freeze.

In algorithm 1 it describes the mechanism as having two loops while it really only needs one loop. The author mentioned they used a simplified version later in the text, and I’ll suggest to update the algorithm block to make it clearer.
review_title: Official Blind Review #2

Meta-review: The paper proposes a top-down approach to train deep neural networks -- freezing top layers after supervised pre-training, then re-initializing and retraining the bottom layers. As mentioned by all the reviewers, the novelty is on the low side. The paper is purely experimental (no theory), and the experimental section is currently too weak. In particular:
- Experiments on different domains should be performed.
- Different models should be evaluated.
- Ablation experiments should be performed to understand better under which conditions the proposed approach works.
- For speech recognition, WER should be reported - even if it is without a LM - such that one can compare with existing work.

Decision: Reject