Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes a one-shot approach to compress already trained neural networks using low-rank compression. The authors propose a new scheme of applying low-rank: instead of applying it directly to a layer, authors suggest partitioning the filters into K blocks, and apply low-rank to each of these blocks separately. Haven defined this scheme, authors then propose a one-shot alternating-optimization algorithm to select ranks and to partition matrix columns/filters into K blocks. Once ranks and partitions are determined, the matrices are decomposed and finetuned. Optionally, this entire compress=>finetune process can be repeated multiple times. Authors present numerical results where the suggested approach almost always wins, however some of the comparisons are questionable. In terms of the algorithm itself, the details of alt.opt are missing, as well as theoretical analysis of the proposed compression scheme is missing.
main_review: The paper is well written, and the method's main idea is presented well; the literature review is up to date. The authors present an impressive amount of experimental validations, however, there are several limitations of this work stemming from its main contributions. To remind the reader, there are two main contributions of this paper:
   1) a new low-rank compression scheme
   2) a simple one-shot alternating optimization
which in combination allows to achieve a quick and accurate (based on the results) compression. Unfortunately, both of these contributions lack a substantial analysis and might actually lead to a subpar (theoretically) performance when compared to a regular low-rank compression. Let me explain my analysis next.

**Issue 1** Analysis of a new compression scheme. The authors, in the "multiple subspaces" paragraph (line 82-92) propose to partition (slice) the columns of the matrix and apply j-rank compression to a clustered subspaces and then combine them. For instance, assuming $k=2$ partitioning, if we have a matrix of size $m\times n$, authors propose to split it into $m \times a$ and $m \times b$ (with $a+b=n$) matrices and apply $j$-rank approximation to both matrices. This allows to achieve a compression ratio of $\frac{mn}{2mj + 2j(a+b)}$. In a sense, this suggested scheme is equivalent (in required parameter size) to a $kj$-rank decomposition ($2j$ in our example). However, if we were to apply $kj$-rank-decomposition immediately, by Eckart-Young-Minsky theorem, we would have a minimal approximation error out of all $kj$-rank decompositions available. 

Here is an important question, whether the authors' suggested scheme achieves better than regular SVD approximation when keeping all parameters ($k,j$) fixed? The authors' bound derived by the same Eckart-Young-Minsky theorem does not answer this question, thus I have written a MATLAB script to empirically quantify the approximation errors achieved by the authors' method and by regular SVD.  For a randomly sampled matrices (from a Gaussian), with $k=2$ partitions, $kj$-rank SVD achieves a better approximation error than the author's suggested scheme. This might suggest that author's scheme is inherently less powerful. Certainly, more investigation is needed. (I'm attaching the script below)


**Issue 2** The proposed Algorithm 1 lacks details. The algorithm 1, is being dumped on the reader without a sufficient number of details. First of all, how do OpimalRanks and OptimalSubspaces work? Since details are lacking, it is hard to understand whether this consists of convergent iterations (i.e., each step reduces the objective?). Supplemental materials lack the necessary details as well.


Beyond these two issues, there are other important ones:
**Issue 3** Fairness of some of the experimental evaluations. In section 3.1 authors compare to a bunch of baselines, out of which comparison to L-rank seems unfair. The L-rank method requires running an entire pipeline of training, and not of the "one-shot" compression method. However, authors seem to run in in a "one-shot" setting, in which the method unsurprisingly yields bad results. This is not an apples-to-apples comparison.  At the same time, in table 2,  in the apples-to-apples setting, the quoted L-rank result is on par with the author's method.

**Issue 4** Relevance of theorem 1? Authors spend sufficient time on stating and proving the theorem 1, however, its results are not being used later in the algorithm (at least the details are missing).


**Overall**, my current evaluation is between "reject" to "borderline; tending to reject".

Below is the Matlab script to quickly assess the proposed compression scheme.  To verify my experimental conclusions mentioned above (that it has worse approximation error when compared to regular SVD when evaluated on randomly sampled $100\times50$ matrix  with $k=2$ and $j=20$) you can invoke the function as `analyze_proposed_scheme(randn(100,50), 20)`. These findings urge a more in-depth analysis of the scheme's performance.
```matlab
function analyze_proposed_scheme(X, r)
    m = size(X,2);
    authors_min_error = Inf;
    split_at = Inf;
    for s=r:m-r
        a = X(:,1:s);
        [u1,s1,v1] = svd(a);
        
        b = X(:,s+1:end);
        [u2,s2,v2] = svd(b);
        scheme_approx = [u1(:,1:r)*s1(1:r,1:r)*v1(:,1:r)' u2(:,1:r)*s2(1:r,1:r)*v2(:,1:r)'];
        curr_err = norm(X-scheme_approx, 'fro');
        if curr_err < authors_min_error
            authors_min_error = curr_err;
            split_at =s;
        end
    end
    authors_min_error
    
    [u,s,v] = svd(X);
    % the proposed scheme is equivalent in parameter size to kj-rank
    % decomposition, here we have k=2, and j=r
    X_svd = u(:,1:2*r)*s(1:2*r,1:2*r)*v(:,1:2*r)';
    
    svd_error = norm(X-X_svd, 'fro')
end
```
limitations_and_societal_impact: I do not believe that the paper has a societal impact that should be addressed.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a new layer-wise compression technique to compress neural networks. The new compression method applies channel slicing method to divide convolution layer into k subsets, followed by using SVD method (similar to Denton et al. (2014) method) to compress each layer. The paper further proposes an automatic layer-wise decomposition selector (ALDS) to compress the whole networks with the proposed techniques by optimizing global network compression (equation (8)). Experiment results on Cifar10, ImageNet and VOC compared to the other ow rank method show that the proposed ALDS has a better prediction performance at the considered compression ratio. 
main_review: This paper is an extension work of Denton et al. (2014) method, which involves new proposed techniques, including channel slicing, SVD approximation, ALDS algorithm, and iterative fine tuning. The algorithms seem to be some adhoc changes compared to Denton et al. (2014) method, particularly when the slicing method is just random slicing. The experiment results, without comparing to the benchmark tensor compression method, might be difficult to convince the effectiveness of the algorithm. 

Some questions to the methods used in this paper are
- Channel slicing methods (line 88-90) in this paper seem to be a random partition of slice into k groups. How is the performance of channel slicing method compared to clustering algorithm based slicing method in model compression? How to select k when applying the compression method in practice?
- The paper conduct experiment on variance of datasets using bench mark backbone models. However, convolution layer is a tensor and known to be redundant in parameters, and used to be compressed by low rank methods, such low rank tensor methods (Kim et al, 2015, Novikov et al. 2015). Slicing the matrix into parts leveraged the tensor structure of convolutions layers. Without comparing the algorithm with other tensor compression method, it might be difficult to tell the performance gains.
- Could the author helps to clarify more details about how to find the optimal subspaces k based on layer budget b (Algorithm 1, line 7)?
- In line 81, the paper claims after SVD decomposition of convolution layer, one convolution layer becomes two small convolution layers. Are there any non-linear transformation considered between the two convolution layers?
limitations_and_societal_impact: Not aware of any negative social impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper propose a novel tensor decomposition pipeline. The propose automatic layer-wise decomposition (ALDC) method comprises two parts: local layer composition and global compression ratio optimization. For local layer decomposition, the weight tensor is first divided into groups and the optimization is within the group. For global compression ratio optimization, the author propose to calculate the relative error bound for each layer and then allocate a layer-wise computation budgets. The experiments are conducted on a various of datasets on image classification and segmentation. The proposed ALDC algorithm is verified over a wide range of network backbones.
main_review: Strong aspects:
1. The paper is well organized and the method is easy to follow. 

2. The proposed method is well justified theoretically.

3. The experiments are comprehensive on the selected backbones and datasets. The results are promising.

4. More importantly, the proposed method do not need training of the network. This might be the main advantages over the NAS algorithms.

Weakness:
1. The selected backbones are all large and redundant. Experiments on efficient/mobile backbones (e.g. MBV2/EfficientNet/MobileNeXt) will make the proposed algorithm more convincing.

2. The main metric of the proposed method is based on the number of parameters. As decomposing a weight tensor will result a deeper network, I am concerned with the run time latency of the proposed method.

3. The reference format used by the authors is not correct. 

I will increase my score if the author could address my concerns sufficiently.


limitations_and_societal_impact: It is better to include some broader impacts in the paper.
review_title: null

Meta-review: The paper suggests a network compression scheme using low-rank approximation of linear layers (either convolutional, or dense).  So far not much new here, except that this paper deals with the combinatorially hard problem of optimizing the matrix decomposition (compression) parameters in all layers simultaneously, so as to minimally sacrifice accuracy.  They suggest an EM-like strategy for solving this optimization problem.

There has been a considerable amount of discussion between the reviewers and the authors, including exchange of code snippets, which is rare.  After reading through this exchange, I became more inclined to accept the paper.  I also think that the optimization problem discussed by the paper does not get enough attention in the network compression literature, and the approach the authors suggest is very reasonable.  Therefore, I choose to recommend acceptance in spite of the average “borderline/reject” score average.

Decision: Accept (Poster)