Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a unified ViT compression framework that assembles pruning, layer skipping, and knowledge distillation as one. By adding the sparsity loss and the resources loss into the objective, ViT models can be stably trained at a high sparse ratio. By introducing the knowledge distillation loss, the compressed model can maintain performance with around 50% of the original FLOPs. Experimental results in ImageNet with DeiT models prove the effectiveness of the proposed method.
main_review: Strengths

* Model compression is well studied in the fields of CNNs in CV and Transformer models in NLP. But such techniques still need to be adopted and verified in visual Transformers.

* The paragraphs in this paper are clear and easy to understand.

* Experiments in ImageNet with DeiT models prove the effectiveness of the proposed method. Compared with baseline models and recent Transformer compression methods, UVC can obtain larger compression rates, resulting in higher accuracies.

Weaknesses

* The novelty of the paper is quite limited. It is like a combination of existing methods, such as Slimming, BigNAS.

* The whole process of UVC is consists of compression training and post-training. I suspect that the comparison between different methods is not aligned, and post-training leads better performance.

* I suggest that UCV can be verified on more Transformer architectures, such as Swin/T2T/CvT/CeiT, showing the generalization ability.

* All tables are unmarked, which are hard to follow.


summary_of_the_review: Overall, UVC has made a good attempt at model compression for visual Transformer models. But the novelty is limited. This method needs to provide a more rigorous comparison, and at the same time, it needs to be verified on more visual Transformer architectures to prove its effectiveness and generalization.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: ViT compression is a new sub-field just being studied this year. With a few previous methods exploring pruning or data-dropping separately, this paper is the first to integrate multiple compression methods as one joint optimization.  
main_review: Motivation and Overall Structure:
While a few prior arts already considered compressing ViTs, this paper stands out with its elegant formulation as a resource-constrained optimization problem, that is solved from end to end. It looks more synergistic and less ad-hoc. 

I also like the rigorous optimization in this paper. Different from previous ad-hoc methods that need tune more hyperparameters, UVC can compress with one global budget control, and jointly coordinate different compression forms automatically under the hood.

Model and Algorithm: 
Besides the common distillation loss, one of the authors’ main new ideas is to enforce mixed-level group sparsity: the head dimension level, the head number level, and the block number level. The authors point out the rationale as: when pruned to same ratios, finer-grained sparsity (i.e., pruning in smaller groups) is unfriendly to latency, while coarser-grained sparsity (i.e., pruning in larger groups) is unfriendly to accuracy. The mixed-level group sparsity can hence more flexibly trade-off between latency and accuracy. 
Particularly, including layer-skipping as a compression means is novel, and to my best knowledge not considered by previous joint compression works yet. The authors also present good rationale why directly skipping layers in ViT might be acceptable: due to the observed layer collapse in many trained ViTs. 
For the algorithm part: while I am familiar with primal-dual or ADMM, I got lost when reading the ~2 pages of optimization derivation in section 3. Specifically, while laying out all details, it is difficult to follow what main algorithm novelty the authors intend to claim: is just this formulation idea novel? Or the overall primal-dual algorithm being novel? Or the solution to some subproblem novel?  

Experiment and Analysis: 
One biggest issue I have had - I am not convinced by the comparison with the current SOTA method of patch slimming. It shows UVC can compress to higher ratios, but meanwhile suffer from notably more accuracy losses. If I just look at DeiT-Tiny PatchSlimming (0.7 FLOPs) versus UCV (0.6 FLOPs), I’m under the impression that UVC drops accuracy more quickly. 

While the authors argued UVC can also be seamlessly extended to include token reduction, leaving this vaguely as “future work” cannot directly support the current work’s merit over prior arts. Could the authors report a more aligned apple-to-apple comparison with PatchSlimming?

More questions regarding experiments: Why not reporting DeiT-small? And why not more baselines for DeiT-Tiny with token? Also, the choice of target FLOPs looks a bit arbitrary to me, and different methods are not fully aligned. Could the authors clarify how they’re chosen, or whether they’re cherry picked?

Writing:
The paper’s writing is overall polished. BUT the algorithm part (page 5-6, Algorithm 1) is unnecessarily lengthy, I think the readability would be better if many here were moved to supplementary. Also, all tables in the paper are not numbered – very weird and hard to refer to.

summary_of_the_review: This seems to be an overall well-executed paper, with reasonable amounts of novelty and potential. However, the authors need clarify on a few raised questions, on which my final rating will depend.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This is an interesting work trying to assemble three effective techniques for pruning, layer skipping, and knowledge distillation. 
main_review: The authors’ efforts to formulate and solve ViT compression as an end-to-end budget-constrained optimization is appreciated. Directly skipping layers is a fresh idea as a compression means and seems particularly to fit ViT due to its homogenous block structure and common feature collapse. This makes a brand-new angle to compress ViTs. The formulation and algorithm are correct to my best knowledge. UVC only requires specifying an overall resource budget and can automatically optimize the composition of different techniques. It is easier to use than ad-hoc alternatives while being very flexible. UVC experimentally outperformed a few very recent baselines (e.g, surpassing SViTE of NeurIPS’21), accompanied by sufficient empirical dive-in with ablation studies and visualization. 

The novelty is okay but not so great. Combining two or three compression means is well studied in CNN compression. For UVC’s specific combination, the distillation loss is a rather straightforward plug-in, and isn’t the same technically interesting as the other two in joint optimization.
The performance isn’t clearly better than Patch Slimming, mostly due to the current mismatched FLOPs. If the authors cannot show better than it in a fair setting, then I suggest the authors to demonstrate they can be combined so UVC’s empirical benefits can still stand.  
A major caveat in experiments is that the authors never reported their due ablation experiments. A clearly missing baseline is: what if you apply three compression methods sequentially, such as first pruning then skipping layer then distilling (or in whatever other reasonable orders)? This missing is crucial as no existing experiment can directly support joint optimization is essential
There is no DeiT-small in table and I don’t understand why. The compression ratios chosen to display are very “sparse” (kinda understandable, since training VITs is heavy). How will UVC perform if increasing compression ratios further?
(Optional) it would further strengthen the paper if more strong ViT variants can be reported such as Swin Transformer or T2T.

---
The authors have well addressed my major concerns. I am glad to increase my score to 6. 
summary_of_the_review: While this joint optimization is a reasonably novel effort in the (somehow niche) ViT compression field, I cannot endorse this paper further given several critical experiments are missing. I’m willing to revisit my rating if the authors can provide the above questioned results.
review_title: null

Meta-review: This paper receives positive reviews. The authors provide additional results and justifications during the rebuttal phase. All reviewers find this paper interesting and the contributions are sufficient for this conference. The area chair agrees with the reviewers and recommends it be accepted for presentation.
Decision: Accept (Poster)