Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes a method to modify the computing requirements of a trained model without compromising the accuracy and without the need for retraining (with new requirements). To this end, the algorithm focuses on factorizing the models and uses a 2-branch training process to train low-rank versions of the original model (to minimize the accuracy drop). 

Different from other approaches, the algorithm claims to exploit the importance of each layer when reducing the compute. 



Comments:
- Figure 2 and related numbers are slightly misleading. The paper focuses on CNN while these numbers and the figure is for FC. M changes significantly when using convolutions. It would be great to clarify this all over the text as M increases signigicantly when using (at least) 3x3 convolutions. 

- One missing thing for me is taking into account the latency rather than the number of parameters. While factorization may reduce the number of parameters (considering the rank is sufficiently low), the number of layers and therefore data movements increases and so does the latency. Some analysis on this can be found in [1] where the paper trains a network with low-rank promoting regularization. I missed having [1] and other similar approaches in the related work and how the proposed method compares to those directly promoting low-rank solutions. 

- The approach would be sounded if the algorithm does not need to work on the factorized version of the layer. That would bring direct benefits to inference time. 

- On the algorithmic side, it is not clear to me how the "2-branches" are trained and what parameters are shared. This seems to involve more compute, right? How is this better than aiming at the lowest-rank possible?

- The complexity-based criterion is interesting although only uses FLOPS as a proxy. How would this translate in practice when the latency is not directly represented by the FLOPS (given the parallelization). 


- During the learning, it is not clear to me how the process is implemented and the scalability of this approach. The paper suggests computing SVD per iteration is infeasible. How many iterations are between SVD? and how results are reused. How this is different from [1] where the authors used truncated SVD to promote low-rank every epoch?

- I need clarification on the need of training full-rank and low-rank (end of page 4). If full-rank does not actually provide better accuracy (see [1]), then, why do we need to rely on that?

- Section 3 focuses on works relying on retraining. It would be nice to see how the proposed method compares to those not considering retraining to improve accuracy. 

- Not clear to me what is the take-home message with Figure 4. Is the resulting rank per layer enough to represent a big compression? Why not compressing the last layer. the compression is limited as the factorization does not promote sparsity when combining the basis (Sparsity on V). Thus, the size after factorization is the same as the original. 

- The BN correction does not seem to contribute. Experiments suggest: our method is effective. Not very appealing as a contribution. 


Minor details:
- x is used as a single input (page 2) and as an entire dataset (page 3)?

- How do we set the \alpha values?





[1] Compression-aware training of DNN, Alvarez and Salzmann. NeurIPS 2017
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper proposes to reshape the weights of the layers of deep neural networks and parametrize them with a low-rank matrix decomposition (SVD). The rank is optimized using two criterion (error-based and complexity-based). Since the decomposition is applied post-hoc, the authors propose to correct the parameters of the batch norm analytically. The authors propose to jointly optimize a loss on the full network and the low-rank version. Experiments are done on CIFAR 10 an 100 with a VGG-15 and ResNet-34 architecture.                                                                                           

Some of the ideas are interesting and would be worth developping further. However, the paper in the current state cannot be accepted for the following reasons: (1) the novelty is low, this very type of decomposition is already widely studied (2) the paper is not clear as to what the contributions are, and why they are justified, theoretically or empirically, (3) the review and comparison with the state-of-the-art is lacking and (4) the experimental setup is simplistic and not convicing. (5) overall the paper is imprecise.


Main comments

* Applying SVD to the matricized weights of deep neural networks is not new. Actual contributions need to be separated from existing works.
* The related word needs to be reviewed. Many references are missing. In particular, the proposed method could be considered as a special case of tensor based methods.  
* The references that *are* listed in the related work are not properly reviewed: the authors aim to not compare with them claiming that they require re-training. Lebedev et al provide a method that works both for end-to-end training or post-hoc, by applying tensor decomposition to the trained weights. Fine-tuning is optional and done to recover performance.
* How was the complexity-based criterion obtained? Why use both M and P, since M includes P? How do the proposed criterion compare to simple measures, e.g. explained variance?
* The authors should compare with other compression techniques: layer-wise compression (e.g. Lebedev et al, Speeding-up convolutional neural networks using fine-tuned cp-decomposition, ICLR 2015) or full network compression (e.g. Kossaifi et al, T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensor, CVPR 2019).

Missing experiments

* The proposed learning loss needs to be compared with the original one to demonstrate any potential performance improvement. Currently, it is not clear whether it is actually helping. In other words, there should also be a comparison with \lambda = 1 or 0.
* Does the proposed loss have an effect on the selected rank? On the actual rank of the weights? On the distribution of the eigenvalues?
* The BN correction needs to be experimentally motivated: since the network is trained with a loss that incorporates the low-rank network, is that needed? Does the propose loss affect performance? How does performance change with and without that BN correction?
* Experiments on CIFAR 10-100 is not sufficient to be convincing, the authors should ideally try a more realistic, large scale dataset, e.g. ImageNet.
* VGG-15 is not convincing to show-case compression, as more than 80% of the parameters are in the fully-connected layer
* The authors assume that the SVD decomposition of the weights does no change significantly at each step: is there any empirical evidence supporting this assumption? This most likely depends on the experimental setup (batch-size, learning rate, etc).

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper propose to compress deep neural network with SVD decomposition, which however has been published 6 years ago in this paper to decompose FC layers:
Xue, Jian, Jinyu Li, and Yifan Gong. "Restructuring of deep neural network acoustic models with singular value decomposition." Interspeech. 2013.
For convolutional layers, Tucker decomposition, which is high-order SVD, is apparently a better choice. (Kim et al. (2016)) They should at least compare their method with this one.

In their deduction of full-rank-low-rank model joint training,  W_r^{(l)} = U_r^{(l)}{U_r^{(l)}}^T W^{(l)} is used without any explanation.  Since U_r^{(l)}{U_r^{(l)}}^T = I_r, W_r^{(l)} will be first r rows of W^{(l)} or first r cols of W^{(l)}.  It can not be treated as approximation of W^{(l)}.  In other words, the foundation of their training is wrong, which makes their experimental results unconvincing.

They conduct their experiments with CIFAR-10/100, which is too small for VGG-15 and ResNet-34.  A larger dataset would be better.

The writing of this paper is not good. For example:
1. In proposition 2.1, what does "y" represent.
2. For Error-based criterion, what is its difference with selecting rank according to singular values?
3. What is P^{(i)} and M^{(i)} in complexity-based criterion?

In conclusion, i will give a weak reject.
review_title: Official Blind Review #3

Meta-review: The proposed paper presents low-rank compression method for DNNs. This topic has been around for a while, so the contribution is limited. Lebedev et. al paper in ICLR 2015 used CP-factorization to compress neural networks for Imagenet classification; in 2019, the idea has to be really novel in order to be presented on CIFAR datasets. The latency is not analyzed. 
So, I agree with reviewers.
Decision: Reject