Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a survey of methods for enforcing low-rankness and sparsity in neural network weights and proposes SPARK, an alternating algorithm for creating low-rank and sparse weight tensors from a pre-trained network. Baseline method accuracies are retained or slightly improved while parameter count is reduced by a large factor.
main_review: Overall the ideas and overview presented seem solid.

However, several omissions make me worry about how well this contribution is situated in the literature.

1. The term Spark is well established in the sparsity literature and defined as something else entirely (see e.g. https://www.pnas.org/content/pnas/100/5/2197.full.pdf). I would heavily encourage the authors not to create more confusion about terms than there already is, especially when working in the same field, and rename this method to something else.

2. Arguably the seminal contribution in the most recent neural network era pertaining to low-rankness of weights is https://arxiv.org/abs/1404.0736 . The fact that it is not mentioned makes me worry about how many other references are missing and potential comparisons. E.g. why is the method of Yu, Liu, Wang, Tao not transposed to the architectures evaluated here and compared in the table?

3. The sparsity literature is vast and includes some extremely relevant contributions, all of which have been omitted. The following can almost be drop-in replacements for the alternating algorithm described in the paper:
- http://proceedings.mlr.press/v28/richard13.pdf
- https://arxiv.org/abs/1206.6474
- https://arxiv.org/abs/1111.1133
This list is certainly not exhaustive and it will not do to simply cite these papers. There are more in the references they list and the ones listed are too similar not to be compared.

Contrary to the algorithm presented, the above contributions lend themselves to convergence analysis and guarantees due to their convexity.

summary_of_the_review: Possibly solid contribution, but high uncertainty on the originality of the contribution due to lack of references to the surrounding literature.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel approach for 'model compression': reducing the size and computational cost of a neural network model by converting weight matrices to (a) be sparse and (b) low-rank.  While prior works have considered both sparsity (i.e., pruning) and low-rank-ness before, the proposed method utilizes both simultaneously: approximating the weight matrix as a sum of two matrices, one that is low-rank and one that is sparse. This leads to an improved performance-compression trade-off: indeed, in some cases, this approach seems to have a regularizing effect and actually improves the accuracy of the model.
main_review: ### Strengths

- This is a well-written and well-motivated paper. The algorithm is introduced and derived systematically. I especially appreciated the thorough discussion and analysis in Section 3.
- The experiments are convincing and impressive.

### Weaknesses

These are relatively minor:

- While the paper appears to do a reasonable job of covering related work, it would have been good to have pointers in Section 4.2 for how the projection steps are similar to or differ from prior low-rank/sparse compression methods.
- There are only two compressed versions reported for each architecture/benchmark. I realize that running the optimization+training is computationally expensive. But it would've been good to have a more complete picture of the performance-compression curve. For example, for Resnet-20, there are only models with >= 70% compression --- would've been nice to know how performance degrades as you push for lower target rank/sparsity. Again, this is not a requirement for publication (because these experiments are expensive), but I'd recommend that --- if possible --- authors consider doing a more thorough exploration for Resnet-20 on CIFAR-10 (the lightest model+benchmark).
- One of the things that's a little hard to glean is which is better at maintaining performance: pushing for lower rank or  higher sparsity. Right now the algorithm requires both to be provided as input, but one can imagine doing a 'meta'-optimization on top to set these parameters to achieve the highest performance for a target FLOPs budget. While that would be a separate algorithm and out of the scope of this paper, it would be helpful to give readers a bit more intuition on which approximation causes a higher performance loss (and if it varies from layer to layer).
- Finally, it is interesting to observe that for Resnet-20 and -56, the cheaper models actually lead to higher accuracy. I assume this is because of a regularizing effect, but perhaps the compressed models also have better optimization behavior. It would be nice to also report training set performance to verify this.
summary_of_the_review: Overall this is a great paper, and I believe clearly meets the bar for ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors argue and propose to compress the neural networks using an additive combination of TT decomposed and sparse tensors/matrices. To make this happen, authors formulate an optimizaiton problem and solve it using the ADMM framework. Authors provide compression results that are interesting (improved accuracy for a given compression rate), however, the method has an important shortcoming that requires setting the compression parameters by hand for each layer (ranks, sparsities) that limits its practical applicability. Additionally, authors miss a large body of related work on: a) additive combinations of compressions b) low-rank and tensor decomposition methods.

main_review: 
Authors argue and empirically demonstrate that an additive combination of compression is a good choice for the neural network compression. Then authors formulate an optimization problem and solve it using alternating directions of multipliers which has three steps: step over nn parameters, step over compression weights, and step over multipliers. Authors justify usefulness of this scheme using small empirical studies as well as by showcasing compression results. By itself, this would have been a good contribution if executed perfectly, however there are several shortcomings that prevent me from accepting the paper.

1. Justifications for the claims. To justify the usefulness of the additive combinations, authors perform several empirical studies over weights of the pretrained NNs (ResNet-20, fig2) and show that L+S scheme is much better than any other compounding (S->L or L->S). While this is an interesting empirical study, I would not be so sure to generalize it over all possible networks and use it as a trump card. In other words, an empirical study on a single network should not be generalized for all networks by saing "...L+S is the best choice...". Additionally, authors do not report the rank+sarsity settings used for this experiment, which further questions validity of such studies.  
Here is one practical argument againts L+S or any additive compression scheme: the overall compression ratio of any additive combination scheme is limited by the compression ratio of the worst term. For any L+S scheme, simply doing just L or S would get higher compression (but worse approximation error). Clearly, there is a compression-error interplay  between using additive scheme vs using any single scheme, which should be studied more formally to make any long standing claims.

2. The formulation and the choice of hyperparamters. The fundamental practical issue for the proposed optimization problem is the fact that users need to give all ranks and cardinality constrains per layer, which involves on selecting over combinatorial number of different settings (rank per each layer $times$ cardinality per each layer)^(number of layers).

3. Missing details on reported quantities. Authors report overall compression ratio, however, do not report how these values were computed. While compressed storage of tensor weights is straightforward to obtain, the storage of sparse matrices might recuire different amount of bits depending on how they are being saved to disk. Please report clearly how the measures are being computed.

4. Authors omit mentioning/comparing to a vast literature of low-rank, pruning, and additive combinations literature. Some missing low-rank/tensor decomposition works:
- [L1] Accelerating Very Deep Convolutional Networks for Classification and Detection (IEEE TPAMI 2016)
- [L2]  GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking, NeurIPS2018
- [L3] Automated Multi-Stage Compression of Neural Networks (ICCV Workshops 2019)
- [L4] Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer (CVPR 2020)
- [L5] Factorized Higher-Order {CNNs} with an Application to Spatio-Temporal Emotion Estimation (CVPR 2020)

Relevant pruning works:
- [P1] “Learning-Compression” Algorithms for Neural Net Pruning (CVPR 2018)
- [P2] Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach (CVPR 2020)

Relevant additive combinations work:
- [A1] More General and Effective Model Compression via an Additive Combination of Compressions (ECML 2021)
- [A2] Handbook of Robust Low-Rank and Sparse Matrix Decomposition. Applications in Image and Video Processing (CRC Publishers 2016)
- [A3] Compressing by Learning in a Low-Rank and Sparse Decomposition Form (IEEE Access 2019)

Importantly, many compression approaches use variation of ADMM (e.g., L4, P1, P2, above or the work of Ma et al. (2019)). More specifically, additive combinations for model compression have been studied in [A1], and generally, such a combination approach is very well studied in image processing field. See for example the reference [A2]. Authors should mention these works and factor paper's contribution into the existing literature in a more rigorous way.
summary_of_the_review: My rating of the paper is based on the following issues:

1. Weakly justified claims
2. Practical difficulties of the proposed formulation
3. Missing details
4. Literature review and positioning
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new DNN compression technique. It consists in approximating the weights of a trained DNN by the sum of a low-rank and a sparse tensor. This is done by adding sparsity and low-rank constraints to the usual loss, the optimization problem being solved with ADMM. Experiments and comparisons with state of the art show the effectiveness of the technique.
main_review: I think the paper addresses an important problem and brings an interesting contribution, even though the ideas (low-rankness and sparsity) are quite straightforward. The key elements of the method are clearly discussed and supported (even too heavily in my opinion). The proposed technique is challenged against state of the art for three common architectures on CIFAR-10 and ImageNet.

In my opinion this is an interesting contribution and I support its publication. But I feel that the paper could have been more convincing if providing more extensive experiments, especially because the novelty is somewhat limited (the paper combines already existing techniques, even if this is done cleverly). I detail here some criticisms that could be addressed to improve the contribution.

1. Authors state that they exhaustively analyze the "design knobs and factors" on how to optimally combine low-rankness and sparsity for DNN compression. And this appears to be one of the main contribution of the paper. While I agree with the three asked questions, I think that the answers could be better supported. For instance, the answer to Question 2 compare SVD and TT on a single experiment, and only in terms of approximation error. Intead, I was expecting more experimental settings, more various decompositions (e.g., Tucker), and other metrics such as the final impact on the network's performance (that might not been exactly correlated to the approximation error, as remarked by the Authors). Similarly, for answer to question #3, I agree that using the loss instead of the approximation error might be preferable. But this should be supported by various experiments. And the Authors do not compare the regularized and the constrained formulations.

2. Experiments and comparison with state of the art. Even though the experiments and the baselines seem to assess the interest of the proposed technique, their could be more experiments and comparisons (in sections 5.2 and 5.3 or in Appendix), in order to show the impact of the proposed "design knobs" in the final result, or analyze the parameters, in particular the sparsity level and ranks. The computational load should be properly discussed and given for all the techniques, especially because it could be a drawback of the proposed technique.

3. Minor comments
- Section 1 & 2 contain several repetitions and redundancies
- Figure 3: the way the dimensions are reshaped before SVD is not clearly explained
- Answer to question #3: I would say that the constrained and penalized formulations are very similar, and even equivalent for a given Lagrangian parameter. What makes them really different is the way they are parameterized, and it is often easier to set a constrain parameter than a penalty weight. Could this be better discussed?
- Optimization: why not considering proximal gradient instead of ADMM?

summary_of_the_review: Summary Of The Review
- Fairly good paper, but overall quality could be improved
- Significant contribution, but might be slightly oversell
- Convincing experimental validation but limited to only 3 experiments, with few additionnal results
review_title: null

Meta-review: The paper proposes a neural network compression technique based on sparse and low-rank approximations. The paper received mixed reviews, with one accept, one reject, and two borderline accepts. Most reviewers have appreciated the effort conducted for the evaluation. Three reviewers are nevertheless worried about the limited novelty and two of them found the positioning in the literature unclear with many missing references. In particular, one reviewer makes a strong case against the accceptance of the paper.

The authors have made a significant effort to address the issues raised by the reviewers with a very long rebuttal. The area chair has read in details the responses, the points raised by the reviewers, and the paper itself. He/she tend to agree with the issues raised by the reviewers about the positioning of the paper in the literature and the missing baselines. The rebuttal was very helpful and addresses some of the concerns. There are still some remaining issues
 - the discussion about related work is relegated to an appendix. Yet, it is critical for positioning the paper and a discussion within the main paper would be more appropriate.
 - there is no assessment of the statistical significance of the results. Hyper-parameters are fixed to some ad-hoc values and it is unclear what the effect of different hyper-parameter choices is upon the method and other baselines.
 - for reproductibility purposes, providing code with the submission would be very helpful, especially given the empirical nature of the contribution.

Overall, this is a borderline case, which, unfortunately, would require additional work before being ready for acceptance.
Decision: Reject