Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes linear over-parameterization methods to improve training of small neural network models. The idea is simple -- each linear transformation in a network is overparameterized by a series of linear transformation which is algebraically equivalent to the original linear transformation. Number of experiments are conducted to show the effectiveness of the approach.

The proposed method is a simple application of over-parameterization to improve neural network model training. The motivation is clear and the proposed method is clearly presented. The paper is easy to understand and follow. Great analyses on the training behavior and generalization ability are conducted. Given the simplicity of the method, this could be a standard way of training small neural network model if the effectiveness of the method is observed more widely.

These are some concerns on the paper:

1) The effectiveness of the approach is not necessarily significant in all experiments. For example, in Table 1, ExpandNet-FC and ExpandNet-CL were not effective. The same trend is observed in Table 2 for 400 epochs. Given that only ExpandNet-CK improves the performance, we could conclude that some intrinsic property of CK is important than over-parameterization. The good results for 90 epochs in Table 2 may mean linear over-parameterization yields faster convergence as suggested by Arora et al. 2018.

2) The comparisons are not extensive. For example, we do not see Init for all models and "w/ KD" for ShuffleNetV2 in Table 2. Table 2 has "N/A". Table 3 and 4 do not have results with "w/ KD".  Knowledge transfer methods should be the baseline of the paper.

Minor comments:

It will be interesting to see the results of the models used for Init.

It might be interesting to conduct experiments with a big model and see if we do not have any gains.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper is extremely interesting and quite surprising. In fact, the major claim is that using a cascade of linear layers instead of a single layer can lead to better performance in deep neural networks. As the title reports, expanding layers seems to be the key to obtain extremely interesting results. Moreover, the proposed approach is extremely simple and it is well explained in Section 2 with equations (1) and (2). This paper can have a tremendous impact in the research in deep networks if results are well explained.

However, in its present form, it is hard to understand why the claim is correct. In fact, the model presented in the paper has a major obscure point. Equation (1) and (2) are extremely clear. Without non-linear functions, equations (1) and (2) describe a classical matrix factorization like Principal Component Analysis. Now, if internal matrices have more dimensions of the rank of the original matrix, the product of the internal matrices is exactly the original matrix. Whereas, if internal matrices have a number of dimensions lower than the rank of the original matrix, these matrices act as filters on features or feature combination. Since the authors are using inner matrices with a number of dimensions higher than the number of dimensions of the original matrix, there is no approximation and, then, no selection of features or feature combinations. Hence, without non-linear functions, where is the added value of the method? How the proposed method can have better results. 
There are some possibilities, which have not been explored:
1) the performance improvement derives from the approximation induced by the representation of float or double in the matrices. The approximation act as the non-linear layers among linear layers.
2) the real improvement seems to be given by the initialization which has been obtained by using the non-linear counterpart of the expansion; to investigate whether this is the case, the model should be compared with a compact model where the initialization is obtained by using the linear product of the non-linear counterpart of the expanded network. If this does not lead to the same improvement, there should be a value in the expansion.
3) the small improvement of the expanded network can be given by the different initialization. In fact, each composing matrix is initialized randomly. The product of a series of randomly initialized matrices can lead to a matrix that is initialized with a different distribution where, eventually, components are not i.i.d.. To show that this is not relevant, the authors should organize an experiment where the original matrix (in the small network) is initialized with the dot product of the composing matrices. The training should be done by using the small network. If results are significantly different, then the authors can reject the hypothesis.
If the authors can reject (1), (2) and (3), they should find a plausible explaination why performance improves in their experiments.
review_title: Official Blind Review #3

Meta-review: The paper develops linear over-parameterization methods to improve training of small neural network models. This is compared to training from scratch and other knowledge distillation methods. 

Reviewer 1 found the paper to be clear with good analysis, and raised concerns on generality and extensiveness of experimental work. Reviewer 2 raised concerns about the correctness of the approach and laid out several other possibilities. The authors conducted several other experiments and responded to all the feedback from the reviewers, although there was no final consensus on the scores.

The review process has made this a better paper and it is of interest to the community. The paper demonstrates all the features of a good paper, but due to a large number of strong papers, was not accepted at this time.
Decision: Reject