Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel framework for compression-aware training of neural networks. The proposed method uses norm constraints, for two types of pruning (1) convolutional filter pruning (2) low-rank matrix decomposition, expressed via updates of the Stochastic Frank-Wolfe (SFW) algorithm efficiently. 
strength_and_weaknesses: Pros: 
- The proposed framework is interesting and beneficial to the community. The authors provide sufficient intuition and motivation behind the use of the sparsity-inducing norm constraints and how they can be effectively realized via SFW. The presentation is clear and the math is easy to follow. 
- All the claims are supported well by empirical studies on benchmark datasets. The baselines seem sensible; although I must admit that I'm not too familiar with the related works in the compression-aware setting

Comments on the robustness study:
One of the interesting sections in the paper is the study on the robustness of the pruned model. The experimental study and the authors' discussion on the benefits of using the rescaled learning rate are insightful, especially at higher compression rates. With gradient scaling, the authors are able to show the 1/\sqrt(T) convergence result (FW gap) for SFW. However, they still collectively don't provide enough convincing arguments for the robustness claims, in my opinion.
clarity,_quality,_novelty_and_reproducibility: The writing is clear and the ideas are easy to follow. The authors have agreed to release the code if accepted. 
summary_of_the_review: Overall, I think this is a good paper. The ideas presented are novel and are well-supported by ample experimental evidence and some theoretical results. The proposed method and discussions are relevant and useful to the community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a constrained optimization framework based on a versatile family of norm constraints and the stochastic FrankWolfe (SFW) algorithm. The proposed method apply on benchmark image-classification architectures and datasets, and it yields competitive results, often outperforming existing compression-aware approaches. 
strength_and_weaknesses: Strength:

The proposed method applied on benchmark image-classification architectures and
datasets outperform the existing compression-aware approaches.  In the case of low-rank
matrix decomposition, the proposed method can require much less computational resources
than nuclear-norm regularization based approaches by requiring only a fraction of
the singular values in each iteration.

Weakness:

The novelty of this paper is very limited. This paper basically combines 
the existing k-support norm regularztion (https://arxiv.org/pdf/1204.5043.pdf, 
https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7952587) and the existing stochastic 
Frank-Wolfe methods (https://arxiv.org/pdf/1607.08254.pdf). 
clarity,_quality,_novelty_and_reproducibility: The writing can significantly be improved. 
The novelty of this paper is very limited. This paper basically combines 
the existing k-support norm regularztion (https://arxiv.org/pdf/1204.5043.pdf, 
https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7952587) and the existing stochastic 
Frank-Wolfe methods (https://arxiv.org/pdf/1607.08254.pdf). 
summary_of_the_review: The proposed method applied on benchmark image-classification architectures and
datasets outperform the existing compression-aware approaches.  In the case of low-rank
matrix decomposition, the proposed method can require much less computational resources
than nuclear-norm regularization based approaches by requiring only a fraction of
the singular values in each iteration. 

Some Questions:

1. Why the proposed method can require much less computational resources
than nuclear-norm regularization based approaches by requiring only a fraction of
the singular values in each iteration ? It would be great if the authors would detail it.

2. In the experimental results, we hope to see the efficiency of the proposed methods. 
Please give the results on test accuracy vs time.

3. There will be a strict upper limit of 9 pages for the main text of the submission, 
with unlimited additional pages for citations (https://iclr.cc/Conferences/2023/CallForPapers). 
This paper maybe not fit for this request.




--------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------
The authors still did not solve my main concern- the limited novelty of this paper. 
So I support to reject this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this work, the authors adopt SWF (stochastic Frank-Wolfe) algorithm to perform compression-aware training. Prelimary experiments demonstrate the efficacy of the proposed approach. 
strength_and_weaknesses: The authors adopt a classic optimization algorithm (SFW) to perform structure compression-aware training. Before this work is accepted, several concerns should be addressed:
(i)	The first concern is about the novelty. This work heavily depends on the unstructed compression-aware training by extending the unstructured pruning to structured pruning setting. The novelty is discounted. 
(ii)	The authors claims that the adopted SFW methods achieve significant speedup compated with nuclear-norm regularization based approach. In general, for nuclear norm regularized optimization, several SVD-free methods have been proposed [1]. We require the authors compare their proposed approach with more advanced optimization methods for solving nuclear norm regularized problem to show its effecgtiveness for a fair comparision. Furthermore, the real speedup ration should be reported to show the efficacy of the proposed algorithm. 
(iii)	The current experiments restrict to CIFAR-10, CIFAR-100, Tiny-ImageNet. In general, ImageNet-1K is a standard benchmark to verify the effiectiveness of the compression methods. We recommend the author provide more experents on ImageNet-1k. 
(iv)	About Theorem 5.1. Why does the term E[g*||Grad{L}||] indicate the convergence of the proposed approach? THe authors should gives more discussions on this convergence criteria. 


[1] SVD-free Convex-Concave Approaches for Nuclear Norm Regularization, IJCAI, 2017.

clarity,_quality,_novelty_and_reproducibility: This paper is well writen
summary_of_the_review: See the comments above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes the SFW algorithm to solve a constrained optimization framework. The proposed method can result in well-performing models that are robust towards convolutional filter pruning as well as low-rank matrix decomposition. Experiment results also show that the proposed method has better “accuracy vs sparsity” performance than existing approaches. 
strength_and_weaknesses: Strength:
Overall, the paper is well-organized and easy to follow. This paper studies two optimization problems: convolutional filter pruning and low-rank matrix decomposition. For the first problem, the author uses the group-k-support norm ball to constrain the optimization problem to compress the model instead of the existing k-sparse approach. To solve this problem, the author proposes to use the SFW algorithm. For the second problem, the spectral-k-support norm is used. Experiments results on those two problems show the advantage of the proposed method. Specifically, the accuracy of the proposed method is more robust than the existing methods when compressing the model.

This work also empirically shows that the robustness of SFW can largely be attributed to the usage of the gradient rescaling of the learning rate. To justify the usage of gradient rescaling theoretically, the convergence of SFW with batch gradient dependent step size in the non-convex setting is established.

Weaknesses:
1.  In the abstract, the author said that compression-aware training could obtain state-of-the-art dense models which are robust to a wide range of compression ratios using a single dense training run while also avoiding retraining. But I believe the proposed framework still needs to run several times independently to obtain models with different compression ratios. That is, to obtain a model with different compression ratios, you have to run SFW another time from the beginning. Therefore, I am not sure why this method can avoid retraining.
  
2. I notice that In Figure 2, the performance of ABFP is comparable to or even better than SFW. Could the author show the advantage of SFW compared to ABFP?

3. Although Theorem 3.1 shows the convergence of SFW with batch gradient dependent step size in the non-convex setting.  However, the assumption of the objective function seems to be too strong (L-smooth and L-Lipschitz). I believe neither of the two studied problems would satisfy these assumptions. Please correct me if I am wrong.

clarity,_quality,_novelty_and_reproducibility: Quality and clarity are good. Originality is minor.
summary_of_the_review: The experiment result is convincing. I also like the idea of combing the SFW and the group-k-support norm constraint together, but I didn’t find much novelty in the theory and algorithm respects.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: (a) Summary: The authors propose a new method for "compression-aware" training. This refers to a growing body of work which aims to train neural networks (NN) such that if the network weights were iteratively pruned post-hoc without any re-training, then performance would degrade slowly and gracefully. Conceptually this is achieved by updating the weights in a sparsity-aware manner. The contribution of this paper is to adapt the well-known (stochastic) Frank-Wolfe algorithm with (a) structured/group sparsity constraints, and (b) low-rank constraints for NN training. Experimental results show improvements over competing approaches that only use vanilla k-sparse constraints during training. 

(b) Strengths: The paper is generally well-written. The idea of group-sparse constraints for training convnets is natural. The experiments are well constructed.

(c) Weaknesses: As pointed out by some reviewers, the central issue with the paper is novelty: the idea of using Frank-Wolfe for compression-aware training has been already proposed and validated thoroughly by Miao et al (ICLR '22). This significantly weakens the message of the current paper (starting from its title onwards). 
To be clear, the authors do make a non-trivial contribution since they employ group-sparse/low-rank constraints, but Frank-Wolfe is such a general framework that merely changing the atomic set is somewhat marginal in my viewpoint. Algorithmically, this means that the linear minimization oracle needs to be adapted, but oracles for group-sparsity and low-rankness have long been established, so there too the conceptual innovation is somewhat marginal.

Overall recommendation: Reject. This is a good paper, which unfortunately, in the light of previous similar (published) approaches, does not meet the high bar set for acceptance. 
justification_for_why_not_higher_score: The paper is generally of high-quality, but may require a careful revision (in the way the ideas are presented) to more clearly distinguish itself  from existing previous works.
Decision: Reject