HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruning

Authors: Tianyi Chen, Xiaoyi Qu, David Aponte, Colby Banbury, Jongwoo Ko, Tianyu Ding, Yong Ma, Vladimir Lyapunov, Kazuhito Koishida, Ilya Zharkov, Luming Liang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Structured Pruning, Training, AutoML

Abstract:
Structured pruning is one of the most popular approaches to effectively compress the heavy deep neural networks (DNNs) into compact sub-networks while retaining the original network performance. The existing methods suffer from multi-stage procedures along with significant engineering efforts and human expertise. The Only-Train-Once series (OTOv1-v3) has been recently proposed to resolve the many pain points by streamlining the workflow. However, the built-in sparse optimizers in the OTO series, i.e., the Half-Space Projected Gradient (HSPG) family, have limitations that require hyper-parameter tuning and the implicit controls of the sparsity exploration, consequently requires intervening by human expertise. To further address such limitations, we propose a novel Hybrid Efficient Structured Sparse Optimizer (HESSO). HESSO could automatically and efficiently train a DNN within a single run to produce a high-performing sub-network. Meanwhile, it is almost tuning-free and enjoys user-friendly integration for generic training applications. To address another common issue of irreversible pruning performance collapse observed in some DNNs, we further propose a novel Corrective Redundant Identification Cycle (CRIC) to plug into HESSO for reliably identifying indispensable structures. We numerically demonstrate the efficacy of HESSO and its enhanced version HESSO-CRIC on a variety of applications ranging from computer vision to natural language processing, including large language model. The numerical results showcase that HESSO can achieve competitive performance to varying state-of-the-art benchmarks and support most DNN architectures. Meanwhile, CRIC can effectively prevent the irreversible performance collapse and further enhance the performance of HESSO on certain applications.

Decision: reject
Reviewer scores: 6, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1730291000564

thread_type:
official_review

<thread_object>
creator: reviewer_nvgl

<content>
summary:
This paper proposes HESSO, a novel optimizer designed for structured pruning of DNNs. HESSO aims to automatically and efficiently train any DNN to produce high-performing sub-networks with minimal need for hyper-parameter tuning, enhancing user-friendliness. Additionally, this paper introduces an extension of HESSO called HESSO-CRIC to address issues related to the irreversible performance collapse that sometimes occurs in pruned models. HESSO and HESSO-CRIC are evaluated across various applications, including computer vision and natural language processing, where it demonstrates competitive performance compared to existing baselines.

strengths:
1. The paper is well-written, with clear organization and rigorous logic, making it understandable to readers.
2. HESSO can automatically and efficiently train and prune DNNs with any architecture, providing competitive performance, which holds broad significance in the field of neural network pruning.
3. The authors conducted a comprehensive analysis of how approximation errors of saliency score can lead to misidentification of indispensable structures in pruned DNNs, resulting in performance collapse. This part is well-organized, highly interesting, and very insightful.

weaknesses:
1. I believe the main issue with this paper is the lack of many necessary comparison baselines, or the existing comparison baselines are outdated.
1) Section 4.3: In the CoFi [1] method, BERT-Base achieves a lossless F1 score on the SQuAD dataset with a 70% reduction in parameters (Figure 2 in CoFi). However, in the HESSO method, BERT-Base shows a drop of about 4% in F1 score with approximately a 50% reduction in parameters.
2) Section 4.4: HESSO seems to be inferior to the ATO [2] method, as ATO achieved a TOP-1 accuracy of 76.07% with a 61.7% reduction in FLOPs, only a -0.06% drop compared to the baseline. However, under the same FLOPs, the accuracy of HESSO is below 76%.
3) Section 4.6: The authors only compare with the SliceGPT method, lacking comparisons with some important structured pruning baselines for LLMs, such as LLM-Pruner (with LoRA fine-tuning) [3], LoRAPrune [4] and LoRAShear [5].
Due to the lack of the above baselines, I have concerns about the effectiveness of the HESSO method.
2. It seems that HESSO is simply improved version of OTO v3. As far as I know, OTO v3 also proposes a hybrid training scheme that applies different update mechanisms to important and redundant variable groups. HESSO does not seem to introduce any significant innovations but rather represents a minor technical improvement over OTO v3.
3. Different saliency scores likely result in varying approximation errors. The authors should demonstrate whether the CRIC mechanism effectively corrects each type of approximation error. Therefore, the authors should show the impact of different saliency scores in HESSO-CRIC on the final accuracy.
4. The phrase "can not be recovered given user resources" in Definition 3.1 is confusing, as it is evident that models with different parameters require significantly different training resources. The authors should clarify the definition of "user resources" more explicitly.

questions:
1. In line 215, the authors state, “The selection of the saliency score in HESSO is flexible and can be tailored to different purposes.” I would like to know how the authors select the saliency score for different tasks in Sections 4.2-4.6. 
2. In lines 503-507, the authors state, “Since both HESSO and HESSO-CRIC … without requiring tensor parallelism.” Does this mean that the HESSO method does not support models larger than 3 billion parameters? In addition, since both HESSO and HESSO-CRIC utilize full gradient information of LLMs, I would like to know the training cost of HESSO and HESSO-CRIC on LLMs.
[1] Structured Pruning Learns Compact and Accurate Models. ACL 2022 Main.
[2] Auto-Train-Once: Controller Network Guided Automatic Network Pruning
from Scratch. CVPR 2024.
[3] LLM-Pruner: On the Structural Pruning of Large Language Models. NeurIPS 2023.
[4] LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning. ACL 2024 Findings.
[5] LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery. ArXiv.
[6] OTOv3: Towards Automatic Sub-Network Search Within General Super Deep Neural Networks. ArXiv.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730342292786

thread_type:
official_review

<thread_object>
creator: reviewer_sytw

<content>
summary:
The manuscript presents a novel approach to deep neural network (DNN) training and structured pruning through Hybrid Efficient Structured Sparse Optimize (HESSO). The topic is highly relevant in the current AI landscape, where deploying efficient models in resource-constrained environments is critical. HESSO stands out by simplifying hyper-parameter setup and offering a progressive pruning strategy that enhances user-friendliness. In addition, the authors also discuss the limitations of the proposed method and the approximation errors of salience scores.

strengths:
1. This paper is well-motivated, and the topic is highly relevant in the current AI landscape, where deploying efficient models in resource-constrained environments is critical. 
2. The manuscript provides a well-structured theoretical framework for HESSO, which helps justify the design choices made in the method.
3. The authors conduct extensive experiments in an array of tasks.

weaknesses:
1. While HESSO aims to simplify hyper-parameter tuning, the manuscript does not provide sufficient evidence to support the claim of reduced sensitivity. A comparative analysis of tuning efforts required for HESSO versus traditional methods is needed.
2. The scalability of HESSO to larger models is not well demonstrated. A table or figure listing the computational cost of applying HESSO to models across different model sizes should be provided.

questions:
1. How does HESSO manage the trade-off between sparsity and performance? Is there a recommended range for optimal sparsity levels?
2. Is there a risk that the method could lead to overfitting, especially during the pruning phase? Maybe evaluation in held-out datasets are needed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690117764

thread_type:
official_review

<thread_object>
creator: reviewer_qbt9

<content>
summary:
The paper addresses the challenge of deploying large deep neural networks by proposing **HESSO**, a new optimizer that enables automatic one-shot joint training and structured pruning for various DNN architectures and tasks. HESSO simplifies the hyper-parameter setup and employs a progressive pruning strategy to explicitly control sparsity exploration, enhancing usability. It optionally integrates a Corrective Redundancy Identification Cycle mechanism to accurately identify redundant groups, minimizing the risk of irreversible performance degradation caused by pruning indispensable structures.  The authors have conducted experiments on various type of tasks and models.

strengths:
1.By simplifying hyper-parameter tuning and explicitly controlling sparsity exploration, HESSO enhances usability, making it accessible to practitioners without deep optimization expertise.

2. The optimizer is designed to be applicable to any DNN architecture and task, increasing its potential impact across various domains and applications.

weaknesses:
1. As a paper on structured pruning, it does not compare with state-of-the-art methods on ImageNet, such as OFA, EagleEye ResRep, etc. The comparisons are too simplistic, making it impossible to see the advantages of the proposed method. Furthermore, in experiments on large models, it only compares with SliceGPT, which is not tailored for structured pruning. More mainstream baseline methods like LLMPruner need to be included for comparison. Overall, the experimental section significantly dampens my enthusiasm for this paper.

2. The authors' discussion of related work in the field is rather lacking and not sufficiently grounded. The entire introduction, methodology, and experimental discussions focus only on OTO-based papers[4,5,6]. However, as far as I know, there are many papers on automatic structure search for structured pruning, including OFA[1], EagleEye[2], etc. I suggest that the authors provide a more comprehensive coverage of the field, which would effectively enhance the quality of this paper.

[1] Cai, H., Gan, C., Wang, T., Zhang, Z. and Han, S., 2019. Once-for-all: Train one network and specialize it for efficient deployment. In ICLR, 2020.

[2] Li, B., Wu, B., Su, J. and Wang, G., 2020. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. In ECCV, 2020.

[3] Ding, X., Hao, T., Tan, J., Liu, J., Han, J., Guo, Y. and Ding, G., 2021. Resrep: Lossless cnn pruning via decoupling remembering and forgetting. In ICCV, 2021.

[4] Chen, T., Ji, B., Ding, T., Fang, B., Wang, G., Zhu, Z., Liang, L., Shi, Y., Yi, S. and Tu, X., 2021. Only train once: A one-shot neural network training and pruning framework. In NeurIPs, 2021.

[5] Chen, T., Liang, L., Ding, T., Zhu, Z. and Zharkov, I., 2023. Otov2: Automatic, generic, user-friendly. In Arxiv, 2022.

[6] Chen, T., Ding, T., Zhu, Z., Chen, Z., Wu, H., Zharkov, I. and Liang, L., 2023. OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing Operators. In Arxiv, 2023.

questions:
Please see the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691190097

thread_type:
official_review

<thread_object>
creator: reviewer_ozcz

<content>
summary:
This paper proposes Hybrid Efficient Structured Sparse Optimizer (HESSO), a structured pruning algorithm. In addition, for reliably identifying the indispensable structure, the authors introduce Corrective Redundant Identification Cycle (CRIC). The experimental results show that HESSO with CRIC plugged in can perform well on various types of models, e.g., image classification, object detection and large language models.

strengths:
1. This paper is well written and organized.
2. This paper provides thorough experiments, by evaluating the performance of HESSO on various tasks and comparing with many other pruning methods.
3. It is demonstrated that HESSO is able to achieve strong and impressive results with a considerable pruning ratio.

weaknesses:
See questions.

questions:
1. What is the computational cost of employing HESSO? I am wondering whether it will significantly increase the computational memory and flow training speed.
2. Seems CIRC can be used as a generic module to identify the indispensable structures. Can it be applied to other pruning algorithms and what is the performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731460098404

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank all reviewers for their careful reviews and insightful feedbacks.

comment:
Dear reviewers and ACs,

We greatly thank all the reviewers for the careful review and for providing insightful and constructive feedbacks. We believe that incorporating these valuable suggestions could significantly improve the quality of our paper.  We are also happy to see that the significance and practical impacts of the work, writing quality, structural logics, experiment broadness, and theoretical boundness have been positively recognized by the reviewers. 

At present, we are working on revising the manuscript and will tackle all raised concerns and questions within this discussion period, such as including more recent baselines and more comparisons. 

Look forward to our further discussion.

Thanks all again!

Best,

Authors of Paper 3237
</content>
</thread_object>

</thread>
<thread>
creation date:
1732009397423

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response (part 1)

comment:
We greatly thank all reviewers for their insightful feedback and constructive suggestions, which have significantly improved the quality of our work. Below, we summarize the main contributions of this work, explain the intent behind our initial experimental designs, and address the shared concern regarding the inclusion of more baselines and comparisons.

### **Contributions of this work**

The contributions are two folds. 
 - **HESSO (the first white-box structured-pruning-aware training optimizer)**: We introduce HESSO, the first white-box structured-pruning-aware training optimizer designed for generic applications and deep neural networks (DNNs) to our knowledge. HESSO is simple and effective. In comparison to prior works, such as black-box optimizers like HSPGs or bi-level optimization schemes like ATO, HESSO offers several key advantages:
     - **Hyperparameter tuning-free**: We provide a general pruning aware training recipe, used across all of our experiments. It requires no expertise in sparse optimization, making it easy to apply across a wide range of applications.
     - **Broad applicability**: Its generic nature allows it to be used in diverse applications and DNNs, demonstrating its utility for practical use cases.
- **CRIC (Corrective Redundancy Identification Cycle)**: Current popular saliency-score-based pruning algorithms sometimes fail to preserve the performance of pruned DNNs due to the existence of indispensable structures. We identify the root cause as the approximation error inherent in existing saliency scores. We propose CRIC, a novel method to reliably identify redundant structures even for the DNNs wherein indispensable structures exist. CRIC comes with theoretical guarantees regarding its efficiency and accuracy, ensuring robust pruning results.  

### **Experimental design intent: Broadness and Representativeness**

Our goal was to develop a generic white-box pruning-aware training optimizer. To validate its broad applicability, we designed experiments covering five distinct applications, i.e., super-resolution, image-classification, object detection, question and answering, and autoregressive LLM. For each application, we selected challenging counterparts to increase representativeness. Many of these selected applications have been used as main benchmarks in prior domain-specific publications, further ensuring the rigor and relevance of our evaluation.

### **Main issues raised during reviews: increasing the Depth of specific experiments.**

We agree with the reviewers that delving deeper into ImageNet and large language models (LLMs) would better align our work with standard pruning literature. To address this, we conducted additional experiments:
- **ImageNet**: We included comparisons with suggested benchmarks to showcase HESSO's performance more comprehensively.
- **LLMs**: We expanded comparisons to include LLM-Pruner, LoraPrune, and LoraShear, which used Lora either in pruning or fine-tuning. The significant margin of HESSO-CRIC largely comes from the CRIC and the full gradient information to more adequately learn and transfer knowledge. However, Lora has been shown easy to be saturated to the underfit DNNs, e.g., pruned DNNs. 

> |Method|Grad Type|Iterative Pruning|Hybrid Training|Dynamic Recover|
>|--|--|--|--|--|
> |LLM-Pruner|Full grad for pruning and Lora for fine-tuning|❌|❌|❌|
> |LoraPrune|Lora for pruning and fine-tuning|✅|❌|❌|
> |LoraShear|Lora for pruning and fine-tuning|✅|✅|✅|
> |**HESSO-CRIC**|**Full grad for pruning-aware training**|✅|✅|--|

> **Table 1 Structurally Prune Phi2-2.7B**
> |Pruning Ratio|Method|Avg Perf|Pruning Ratio|Method|Avg Perf|Pruning Ratio|Method|Avg Perf|
> |--|--|--|--|--|--|--|--|--|
> |20%|SliceGPT|60.22|25%|SliceGPT|57.11|30%|SliceGPT|44.53|
> ||LLM-Pruner|46.75||LLM-Pruner|45.00||LLM-Pruner|44.17|
> ||LoraPrune|50.83||LoraPrune|48.83||LoraPrune|46.92|
> ||LoraShear|51.81||LoraShear|48.95||LoraShear|47.45|
> ||**HESSO-CRIC**|**60.67**||**HESSO-CRIC**|**58.74**||**HESSO-CRIC**|**55.62**|


> **Table 2 Structurally Prune over MobileNet Super-Network**
> |Method|Num of Params (M)|MACs (M)|Top Acc-1 (%)|
> |--|--|--|--|
> |MobileNetV2|3.4|300|72.0|
> |MobileNetv3-Large|5.4|219|75.2| 
> |OFA # 75|5.81|230|76.9| 
> |EagleEye|--|~284| 70.9|
> |**HESSO**|5.60|220|78.2|
> |**HESSO-CRIC**|5.71|225|**78.6**|

We will provide the revised manuscript and responses to each individual question shortly this week.

Look forward to our further discussion.

Yours,

Authors
</content>
</thread_object>

</thread>


Metareview: