Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a specialized optimizer for optimizing the DNN to have good compression ability, i.e. the optimized model can be pruned/quantized to smaller models which have good accuracy.

The paper is based on a formulation related to the sharpness aware minimization, by adding a compression related constraint projection on the inner "sharpness-aware" gradient ascent step.

Experiments result show that the proposed method can train a model which not only performs good on dense model but also on sparse models.
strength_and_weaknesses: Strength
- The proposed method can train a single set of model weights which performs well on multiple compression rate, i.e. sparsity ratios.
- The formulation is a good extension of sharpness-aware optimization to the direction of optimization with constraints.
- The authors also considered and discussed the computation complexity of the algorithm.
- The experiments result show that the dense model results can be even better compared to training a standalone dense model.

Weaknesses
- Although the paper derives the method from the view of optimizer, the updating rule is almost the same as using SAM optimizer on supernet/one-shot NAS methods such as once-for-all network[1], bignas[2]. The novelty of this method is not very significant from this point of view.
- It's better to have some discussion and comparison to the method which directly combines SAM with once-for-all network, e.g., is there any difference in the actual updating rule and how much difference in the results.

[1] Cai, Han, et al. "Once-for-all: Train one network and specialize it for efficient deployment." arXiv preprint arXiv:1908.09791 (2019).

[2] Yu, Jiahui, et al. "Bignas: Scaling up neural architecture search with big single-stage models." European Conference on Computer Vision. Springer, Cham, 2020.
clarity,_quality,_novelty_and_reproducibility: This paper is well written and easy to understand.

The derivation and formulation in this paper are in good quality.

The novelty of this paper is not significant, as it's very close to once-for-all and other supernet based methods in essence. (see "Strength And Weaknesses" for more details.)
summary_of_the_review: The paper provides an intact derivation of extending SAM optimizer to the context of constrained optimization and especially model compression.

The experiments validate the proposed algorithm on different sparsity ratios and also dense model.

From the updating rule, the proposed algorithm is the almost the same as using SAM on once-for-all network, so the novelty of this paper is not very significant.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method to train a neural network, CrAM (and CrAM+), which results in a trained network that can be compressed (by pruning or quantization) while maintaining accuracy. The method is developed based on plugging compression in the SAM’s inner maximization step; the perturbed parameters are obtained by taking the ascent step (for maximization) and applying some compression on them. The authors attempt to provide theoretical justification for such an algorithm based on robust optimization literature. The authors present details on some extensions including batch norm tuning or SAM+multi, and further, experimental results on Imagenet as well as SQuAD for different comparisons.

strength_and_weaknesses: Strength
* provide a range of experiments making comparisons on different scenarios, baselines, and domains
* develop theory based on the mini max framework, by which compression interpreted as perturbation can be effectively performed as robust optimisation
* N:M sparsity pattern is nice in practice adding it is definitely plus

Weakness
* CrAM, the basic version of the method, does not work well, whereas the provided theoretical justification is supposed to indicate CrAM should be enough, not CrAM+. The theoretical justification itself is only weakly justifying the method. For instance treating the mask operator and straight-through estimator the way the author does is very weakly justified.
* overclaim for generality of the method. present as if it works for any compression methods, but mostly focuses on top-k based on magnitude of parameters. a quantization scheme is provided in the appendix but not enough to claim for compression in general.
* far below the sparsity level compared to existing methods. present some effectiveness up to 90%, but considering the sparsity literature it is far below. for 95% sparsity it is a level at which the method does not work.
* coated with various experiments, yet fails to concentrate on the central part. Although they conduct many interesting experiments, they are often quite partial point-by-point based and far from being comprehensive.
* The only effective result is that CrAM/CrAM+ can give a network that is one-shot prunable. However, this is not considered enough given the fact that this method requires additional hyperparameter tuning such as the choice of sparsity levels during training, that there exist other one-shot training (and pruning afterwards) methods without requiring SAM framework or double the forward-backward computations.

clarity,_quality,_novelty_and_reproducibility: * The paper directly borrows the SAM framework and tests a few different algorithmic changes in the order of compression operations. In that sense it is hard to say the idea is original.
* In my view writing should improve in terms of organization, focus, and clarity.

summary_of_the_review: The paper explores an interesting combination of SAM and sparsity, but importantly lots of parts in the paper is not quite tight and needs further exploration.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to have a compression-aware minimizer that can compress dense DNN into a sparse one, and most importantly, without much performance drop. Inspired by the sharpness-aware minimizer, the authors leverage the flat minima to find the weights' coordinates that are stable to the perturbation. With the price of doubled computational overhead, the proposed compression-aware minimizer can prune dense models in a one-shot to 70-80% sparsity with a small drop in accuracy. The authors also give theoretical proof to guarantee the convergence of the proposed CrAM. The experiments on the ImageNet dataset verify the effectiveness of the proposed CrAM. The paper is well-organized and easy to follow. 

strength_and_weaknesses: Pros:
1. The paper is well-written and well-organized. It is easy to follow and read. 
2. It is inspiring to leverage the flat minima for the model pruning task. This paper encourages the application of sharpness-aware training.
3. The methodology and appendix E are good for readers. The derivation is sufficient; the theoretical proof makes the paper complete. 
Cons:
1. It would be better if the authors could have more experiments—for example, some parameter studies; some model pruning baselines. (One should be enough for the rebuttal period)  
2. Appendix E is important; it would be better not to be placed the last part of the appendix. 
3. There are too many variants of CrAM in the experiments. It is better to give a universal parameters set of CrAM. 
clarity,_quality,_novelty_and_reproducibility: The quality of this paper is good. Because the motivation, the addressed problem, the derivation of the solution, and the experiments are clear and easy to follow. The novelty of this paper is okay, as it proves the convergence of updating for their proposed CrAM. 
summary_of_the_review: 
To conclude, this paper is a good research paper. It is well-written, and the proposed method is sufficient in derivation and effectiveness. It would be better if some more experiments could be added.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new compression-aware minimizer dubbed CrAM that modifies the optimization step in a principled way, which results in dense models trained via CrAM can be compressible post-training, in a single step, without significant accuracy loss. Experimental analysis on image classification and language modelling shows that CrAM models can be pruned one-shot at a wide range of sparsity levels, while resulting in sparse models that are competitive with existing gradual pruning methods. Also, CrAM produces sparse models which perform well for transfer learning and works for semi-structured pruning patterns.
strength_and_weaknesses: Strength.
1) CrAM is simple to implement as part of a regular training loop and has a single scaling hyper-parameter.
2) CrAM enables training a dense model, which can later be compressed to different target levels, with minimal or no recalibration.
3) To derive the CrAM update, the paper provides the theoretical analysis to justify the choices made in designing the training method.
4) Sufficient Experiments demonstrate the wide range of application, including one-shot pruning, quantization, semi-structured N:M sparsity patterns and transfer learning.

Weaknesses.
1) For most of the comparisons, the paper choose the SAM as the target, such as Table 2,3,4 etc, which is not suitable because it trains models that potentially converge to flatter minima for better generalization rather than compressible models. It is more appropriate to compare with compression-aware C-SAM.
2) Compared with the gradual method DPF, CrAM performs worse at high sparsity, such as 90% and 95%. The paper explains that CrAM was not explicitly trained for such high sparsity. However, in the table 7, compared with C-SAM, CrAM still not perform very well at 80% and 90% sparsity with the same experimental configurations. It is better to clarify more about the phenomenon.
3) In the paper, the computation cost is not specific, such as twice as many forward-backward passes or the cost of BNT is minimal. It is better to show the cost explicitly in the comparison tables, such as the training time for single iteration.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. The idea of CrAM is somewhat novel and supported by theoretical analysis, but the experimental results are not significant compared with C-SAM. And the paper provides the details of algorithm and hyperparameters such that readers should be able to reproduce the main results.
summary_of_the_review: The idea of CrAM is overall novel and supported by theoretical analysis. And it is the first carry over the sharpness-aware idea to the task of obtaining compressible models. Although the experimental results are not significant compared with the closed works, the paper is likely to have modest impact within the subfield. Hence, I recommend for acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a compression-aware minimizer (CrAM and CrAM+) to compress the dense Deep Neural Networks (DNN) with preserved performance. The presented method is inspired by the inner maximization step (parameters perturbation) of Sharpness-aware minimization (SAM) to filter out the stable parameters. Experimental analysis well supports the efficacy of CrAM and CrAM+. 

Strength:
-Reasonable method design and clear writing
-A good formulation and extension of SAM. 
-Clear writing and extensive experiments. 

Weakness:
-Need more discussion about the sparsity level and generalization of CrAM.
-The additional computational overhead of CrAM (2X).

justification_for_why_not_higher_score: To some extent, the proposed method is a good extension of the SAM on the network compression task. The performance is good, and the derivation is reasonable. However, this paper is not inspiring enough to the community. The spotlight paper of ICLR needs more novelty and exploration.
justification_for_why_not_lower_score: As all the reviewers gave acceptance scores to the paper, the proposed method is practical in the compression task. There are a few good extensions of SAM work; this paper is a good extension of SAM. Therefore, this paper could be published in ICLR.
Decision: Accept: poster