HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruning

## Abstract

Abstract Structured pruning is one of the most popular approaches to effectively compress the heavy deep neural networks (DNNs) into compact sub-networks while retaining the original network performance. The existing methods suffer from multi-stage procedures along with significant engineering efforts and human expertise. The Only-Train-Once series (OTOv1-v3) has been recently proposed to resolve the many pain points by streamlining the workflow by automatically conducting (i) search space generation, (ii) structured sparse optimization, and (iii) sub-network construction. However, the built-in sparse optimizers in the OTO series, i.e. , the Half-Space Projected Gradient (HSPG) family, have limitations that require hyper-parameter tuning and the implicit controls of the sparsity exploration, consequently requires intervening by human expertise. To further address such limitations, we propose a novel Hybrid Efficient Structured Sparse Optimizer (HESSO). HESSO could automatically and efficiently train a DNN within a single run to produce a high-performing sub-network. Meanwhile, it is almost tuning-free and enjoys user-friendly integration for generic training applications. To address another common issue of irreversible pruning performance collapse observed in some DNNs, we further propose a novel Corrective Redundant Identification Cycle (CRIC) to plug into HESSO for reliably identifying indispensable structures. We numerically demonstrate the efficacy of HESSO and its enhanced version HESSO-CRIC on a variety of applications ranging from computer vision to natural language processing, including large language model. The numerical results showcase that HESSO can achieve competitive performance to varying state-of-the-art benchmarks and support most DNN architectures. Meanwhile, CRIC can effectively prevent the irreversible performance collapse and further enhance the performance of HESSO on certain applications. The code is available at https://github.com/microsoft/only_train_once .

## 1 Introduction

Large deep neural networks (DNNs) have successfully powered a variety of applications (Ji and Chen, , Zhou et al., , Zhu et al., ). However, their typical significant time and space complexities make inference expensive and restrict deployment in resource-constrained environments. Consequently, how to compress the full DNN to the greatest extend while preserving the performance becomes essential in the many industrial and academic AI deployment pipelines. There are various model compression techniques including but not limited to pruning (Chen et al., , , Fang et al., ), knowledge distillation (Ko et al., ) and quantization (Han et al., ), which have been well developed in the past decades.

Structured pruning typically serves as the foremost technique to produce an optimal sub-network from a pre-defined full DNN by identifying and removing redundant structures (Gale et al., , Han et al., , Chen et al., , , Fang et al., , Wang et al., ). Classical pruning methods focus on conducting a multi-stage procedure, requiring significant engineering efforts and expertise to manually build pruning search space, identify redundant structures, construct sub-network, and fine-tune to recover lost knowledge. To alleviate the human engineering burden, recent works (Chen et al., , , Fang et al., ) have proposed pruning dependency graph to automate the pruning search space and sub-network construction. OTOv1-v2 (Chen et al., , ) further unify these multi-stage components together, requiring only a single training run to directly get a compact sub-network without the need of further fine-tuning. Specifically, they rely on (Dual) Half-Space Stochastic Gradient Descent (D)HSPG methods to train and prune simultaneously and have introduced a rigorous theoretical version AdaHSPG+ (Dai et al., ).

Figure: Figure 1: Automatic any DNN joint training and structured pruning experience achieved by the pruning mode of OTO (Chen et al., , , ) along with the proposed HESSO and its enhanced HESSO-CRIC optimizer. The procedure could be applied onto varying DNN and applications, and seamlessly integrated into any training pipeline to directly produce a compact pruned sub-network without need of further fine-tuning.
Refer to caption: x1.png

|  | DHSPG | HESSO | HESSO-CRIC |
| --- | --- | --- | --- |
| Efficiency | $\filledstar\filledstar$ | $\filledstar\filledstar\filledstar$ | $\filledstar\filledstar\smallstar$ |
| Tuning-Free | $\filledstar$ | $\filledstar\filledstar\filledstar$ | $\filledstar\filledstar\filledstar$ |
| User-Friendliness | $\filledstar$ | $\filledstar\filledstar\filledstar$ | $\filledstar\filledstar\filledstar$ |
| Performance | $\filledstar\filledstar\filledstar$^† | $\filledstar\filledstar\smallstar$ | $\filledstar\filledstar\filledstar$ |
| ^† Under sufficient hyper-parameter tuning efforts. |  |  |  |

Although OTOv1 and OTOv2 have significantly advanced the ease of use in DNN joint training and structured pruning, they still face challenges related to the complexity of the built-in (D)HSPG methods (Chen et al., , , , ). Specifically, these methods often require substantial hyper-parameter tuning for different downstream applications and DNN architectures. Furthermore, the sparsity explorations are implicitly controlled by (D)HSPG, which diminishes the practical convenience and usability of these approaches.

Meanwhile, many modern pruning and neural architecture search methods rely on saliency scores to identify redundant structures. However, they often suffer from performance degradation due to mistakenly identifying indispensable structures as redundant. This degradation can sometimes be irreversible due to architectural design constraints, transparency of training datasets, and the significant training resources required, posing practical challenges for their use.

To overcome these pain-points, we naturally ask, i.e., how to get a joint training and pruning optimizer which is ease-to-use, reliable, high-performing, generic and applicable onto any tasks and DNN architectures.

In this work, we address this question by proposing HESSO: the Hybrid Efficient Structured Sparse Optimizer for automatic one-shot any DNN training and structured pruning. Compared to the HSPG family, HESSO offers several advantages. First, HESSO significantly simplifies the hyper-parameter setup, providing considerable practical convenience. Second, HESSO employs a progressive pruning strategy to explicitly control the sparsity exploration, making it user-friendly. Third, HESSO optionally incorporates a novel Corrective Redundancy Identification Cycle (CRIC) mechanism, so-called HESSO-CRIC, which more accurately identifies redundant groups, thereby minimizing the risk of irreversible performance collapse caused by pruning indispensable structures.

We now summarize our main contributions as follows.

- •
Efficient Hybrid Training and Pruning Optimizer HESSO. We propose an efficient and easy-to-use optimizer, HESSO, to enable automatic joint structured pruning and training for various model architectures and applications. HESSO progressively identifies redundant groups through flexible saliency score estimations and utilizes a hybrid training schema to effectively transfer knowledge from redundant groups to important ones, thereby maintaining the performance of the pruned model. Compared to the DHSPG in OTOv2 (Chen et al., ), HESSO explicitly controls sparsity exploration, significantly reduces the need for hyper-parameter tuning, and is underpinned by theoretical foundations that ensure practical performance across various downstream applications.
- •
Corrective Redundancy Identification Cycle. We propose a novel Corrective Redundancy Identification Cycle (CRIC) to significantly improve the accuracy of redundancy identification. CRIC addresses the approximation errors often associated with popular Taylor-based saliency scores, thereby reducing the risk of mistakenly identifying indispensable groups as redundant. CRIC employs a voting mechanism and measures the saliency scores of each group candidate using a multi-sampling approach towards the origin. CRIC is pluggable into HESSO and helps to ensure reliable model performance by providing a more accurate assessment of group significance.
- •
Numerical Experiments. We validate the efficacy of HESSO across a variety of applications and model architectures. Specifically, we evaluate its performance on high-level computer vision tasks such as image classification and object detection, low-level vision tasks like super-resolution and diffusion generation, as well as natural language processing tasks including large language models. The numerical results demonstrate that HESSO performs competitively, and in many cases, exceeds the state-of-the-art benchmarks, offering significant practical convenience. Additionally, CRIC effectively mitigates the issues of irreversible collapse in pruned models, especially in challenging cases, further showcasing its utility.

## 2 Related Works

#### Automatic Structured Pruning.

Structured pruning aims to compress deep neural networks (DNNs) by removing unnecessary structures while maintaining performance (Han et al., , Wen et al., ). The general procedure typically involves: (i) training a full model; (ii) identifying and removing redundant structures to construct a slimmer DNN based on various criteria (Lin et al., , Wen et al., , Li et al., , Zhuang et al., , Chen et al., , , chen2021orthant, Chen et al., , Gao et al., , Zhuang et al., , Meng et al., , Yang et al., , Zhou et al., , van Baalen et al., , Frankle and Carbin, ); and (iii) retraining the pruned model to recover any accuracy lost during pruning, with or without knowledge distillation. These methods often require a complex and time-consuming process, involving multiple training iterations and significant domain knowledge to manually handle each step.

Recent methods, such as OTO (Chen et al., , , ) and DepGraph (Fang et al., ), have made progress in automating the structured pruning process for general DNNs. OTO offers a one-shot joint training and pruning framework that can seamlessly integrate into various training processes to produce high-performing sub-networks in a single run. DepGraph, on the other hand, focuses on pruning and needs to be integrated with a multi-stage training pipeline. While these automated approaches have significantly improved user convenience, end-users still face challenges with hyper-parameter tuning and the sparse optimization expertise required to calibrate OTO’s built-in HSPG family (Chen et al., , Dai et al., ). Furthermore, some DNNs contain indispensable structures, the pruning of which leads to irreversible performance degradation. Identifying these critical structures remains an open problem that is often handled manually on a case-by-case basis, complicating practical use. In this work, we tackle these pain points to propose an efficient, tuning-free, and user-friendly joint training and pruning optimizer HESSO along with its enhanced version HESSO-CRIC to reliably identify indispensable structures to ensure the performance.

#### Knowledge Transfer.

To retain the performance of the pruned sub-network, HESSO-(CRIC) incorporates a knowledge transfer mechanism through a hybrid training schema. This approach differs from prior methods, which explicitly use knowledge distillation from unpruned models to preserve information in pruned models. Existing techniques typically require expensive computations that involve both pruned and unpruned models, whether by processing logits (Lagunas et al., ) or the hidden activations of intermediate layers (Xia et al., , Ko et al., ). In contrast, our approach preserves knowledge without incurring such computational costs. Another related work, SliceGPT (Ashkboos et al., ), also aims to preserve computational invariance. The knowledge transfer in HESSO-CRIC similarly seeks to maintain computational invariance but does so by focusing on preserving objective function levels. However, SliceGPT is restricted to the transformer architectures and requires manually injecting additional layers. HESSO-CRIC is architecture-agnostic, efficient and user-friendly, demonstrating both scalability and versatility.

## 3 HESSO

Given a target DNN which variables and architecture to be optimized, HESSO formulates a constrained structured sparsity optimization problem upon the parameter groups $\mathcal{G}$ as (1).

$$ ${\displaystyle\mathop{\operator@font{minimize}}_{\bm{x}\in\mathbb{R}^{n}}}\ f( \bm{x}),\ \ \text{s.t.}\ \text{Cardinality}\{g\in\mathcal{G}|[\bm{x}]_{g}=0\}=K,$ (1) $$

where we seek to yield group sparsity over the prunable variables with the target sparsity level as $K$. The parameter groups can be pruning zero-invariant groups determined through the pruning dependency graph analysis or other general group formats (Chen et al., , ).

During the optimization process, HESSO begins with a warm-up phase, where the variables are trained using gradient descent or its variants, such as SGD, Adam (Kingma and Ba, ), or AdamW (Loshchilov and Hutter, ). The purpose of the warm-up stage is to collect gradient information and guide the DNN into a relatively favorable region for convergence. Following this, HESSO performs progressive pruning by periodically identifying redundant parameter groups based on predefined saliency scores. Throughout the progressive pruning phase, HESSO gradually forgets the knowledge in the redundant groups while the remaining important groups continue training, thereby facilitating the transfer and recapture of knowledge. We refer to this approach as hybrid training, where distinct training strategies are applied to different groups. Finally, once all redundant groups are identified and projected onto zero, the remaining important groups continue to be trained until final convergence. The main procedure is outlined in Algorithm 1.

Figure: (a) Trace graph of target DNN.
Refer to caption: x2.png

Figure: Algorithm 1 HESSO: Hybrid Efficient Structured Sparsity Optimizer

### 3.1 Saliency Score

After warming up $T_{w}$ steps in Algorithm 1, HESSO has typically collected reasonable information regarding the gradient and the iterate. It then starts to identify redundant groups upon the target group sparsity level $K$ to partition the groups $\mathcal{G}$ into important group set $\mathcal{G}_{I}$ and redundant group set $\mathcal{G}_{R}$, i.e., $\mathcal{G}_{I}\bigcup\mathcal{G}_{R}=\mathcal{G}$ and $|\mathcal{G}_{R}|=K$.

Figure: Figure 3: HESSO uses saliency scores to periodically identify redundant groups $\mathcal{G}_{R}$ from the group set $\mathcal{G}$ and marks the remaining groups as important groups $\mathcal{G}_{I}$. A knowledge transfer mechanism is proceeded by employing hybrid training strategies onto $\mathcal{G}_{R}$ and $\mathcal{G}_{I}$. In particular, the variables in $\mathcal{G}_{R}$ are progressively projected onto zeros after gradient descent. The important variables are kept training via gradient descent to migrate the impact of redundant project onto the objective function.
Refer to caption: x5.png

HESSO achieves it by periodically measuring the importance of each parameter group $g\in\mathcal{G}$. To begin, we initialize the important group set as the whole group set $\mathcal{G}_{I}\leftarrow\mathcal{G}$, and the redundant group set as empty $\mathcal{G}_{R}\leftarrow\emptyset$.
Given a pre-defined pruning periods $P$, we identify $\widehat{K}\leftarrow K/P$ important groups to designate as redundant during each period. The redundant groups are the ones with bottom-$\widehat{K}$ saliency scores.

$$ $\begin{split}\mathcal{G}_{R}&\leftarrow\mathcal{G}_{R}\bigcup\mathop{{\text{ Bottom-}\widehat{K}}}_{g\in\mathcal{G}_{I}}\ \text{SaliencyScore}([\bm{x}]_{g} ,[\nabla\!f(\bm{x})]_{g})\\ \mathcal{G}_{I}&\leftarrow\mathcal{G}_{I}/\mathop{{\text{Bottom-}\widehat{K}}} _{g\in\mathcal{G}_{I}}\ \text{SaliencyScore}([\bm{x}]_{g},[\nabla\!f(\bm{x})]_ {g})\end{split}$ (6) $$

The selection of the saliency score in HESSO is flexible and can be tailored to different purposes. By default, the following categories are considered:

Magnitude. The importance of a parameter group can be determined by its magnitude. We further normalized against all the current important instances, mapping the score into the range
$[0,1]$. Heuristically, a group of variables with lower magnitude—implying they are closer to zero—typically contributes less to the model output. Therefore, such groups are often considered less important and more likely to be pruned.

$$ $\text{score}_{\text{mag}}([\bm{x}]_{g})\leftarrow\left\|[\bm{x}]_{g}\right\|_{ 2},\ \text{score}_{\text{mag}}([\bm{x}]_{g})\leftarrow\text{score}_{\text{mag} }([\bm{x}]_{g})/\sum_{g\in\mathcal{G}_{I}}\text{score}_{\text{mag}}([\bm{x}]_{ g}).$ (7) $$

Average Magnitude. While considering the overall magnitude can be useful, it may introduce bias by disproportionately favoring groups with more parameters, marking them as more important. To address this potential bias, the average magnitude is also considered. This metric measures the average parameter magnitude within each group, providing a normalized assessment that accounts for the number of parameters in each group. Consequently, the algorithm can more fairly compare groups of different sizes and prevent the overrepresentation of larger groups.

$$ $\text{score}_{\text{avg-mag}}([\bm{x}]_{g})\leftarrow\left\|[\bm{x}]_{g}\right \|_{2}/|\sqrt{|g|}|,\ \text{score}_{\text{avg-mag}}([\bm{x}]_{g})\leftarrow \text{score}_{\text{avg-mag}}([\bm{x}]_{g})/\sum_{g\in\mathcal{G}_{I}}\text{ score}_{\text{avg-mag}}([\bm{x}]_{g}).$ (8) $$

Cosine Similarity. Another criterion for determining group importance is the cosine similarity between the projection direction of parameter group and the negative gradient direction of the objective function. It can be calculated as the cosine similarity between $-[\bm{x}]_{g}$ and the negative gradient $-[\nabla\!f(\bm{x})]_{g}$, followed by a normalization to map onto a common scale. This metric evaluates whether projecting a group of parameters onto zero (i.e., moving towards the origin along the negative parameter direction) aligns with a descent direction for the objective function. A descent direction is expected to decrease the objective function value, suggesting that pruning group of parameters onto zero may not significantly regress model’s performance. As a result, such groups are more likely to be marked as redundant.

$$ $\begin{split}\text{score}_{\text{cosine}}([\bm{x}]_{g},[\nabla\!f(\bm{x})]_{g} )&\leftarrow[\bm{x}]_{g}^{\top}[\nabla\!f(\bm{x})]_{g}/(\left\|[\bm{x}]_{g} \right\|\left\|[\nabla\!f(\bm{x})]_{g}\right\|),\\ \text{score}_{\text{cosine}}([\bm{x}]_{g},[\nabla\!f(\bm{x})]_{g})&\leftarrow \text{score}_{\text{cosine}}([\bm{x}]_{g})/\sum_{g\in\mathcal{G}_{I}}\text{ score}_{\text{cosine}}([\bm{x}]_{g}).\end{split}$ (9) $$

Taylor Importance. To further quantitatively approximate the effect of projecting the parameter group $[\bm{x}]_{g}$ onto zero on the objective function, we can employ the Taylor expansion. Taylor expansion could estimate the impact of small changes in the parameters on the function value, allowing us to consider varying orders of Taylor importance. In particular, the first-order Taylor expansion provides a linear approximation of the objective function around the current parameter point. The impact of setting $[\bm{x}]_{g}\to\bm{0}$ can be estimated by the dot product of the gradient and the change in parameters. This approximation helps identify groups whose removal will likely decrease the objective function.

$$ $\begin{split}\text{score}_{\text{Taylor-1st}}([\bm{x}]_{g},[\nabla\!f(\bm{x})] _{g})&\leftarrow|f(\bm{x})-f(\bm{x}|[\bm{x}]_{g}\to\bm{0})|\approx|[\bm{x}]_{g }^{\top}[\nabla\!f(\bm{x})]_{g}|,\\ \text{score}_{\text{Taylor-1st}}([\bm{x}]_{g},[\nabla\!f(\bm{x})]_{g})& \leftarrow\text{score}_{\text{Taylor-1st}}([\bm{x}]_{g},[\nabla\!f(\bm{x})]_{g })/\sum_{g\in\mathcal{G}_{I}}\text{score}_{\text{Taylor-1st}}([\bm{x}]_{g},[ \nabla\!f(\bm{x})]_{g}).\end{split}$ (10) $$

The second order Taylor importance is based on the second-order Taylor expansion. It includes the Hessian matrix, capturing the curvature of the objective function. This approximation considers not only the gradient but also the second derivative, providing a more accurate estimate of the impact of setting $[\bm{x}]_{g}\to\bm{0}$.

$$ $\begin{split}\text{score}_{\text{Taylor-2nd}}([\bm{x}]_{g},[\nabla\!f(\bm{x})] _{g})&\leftarrow|f(\bm{x})-f(\bm{x}|[\bm{x}]_{g}\to\bm{0})|\approx[\bm{x}]_{g} ^{\top}[\nabla\!f(\bm{x})]_{g}+\frac{1}{2}[\bm{x}]_{g}^{\top}[\nabla\!^{2}f( \bm{x})]_{g}[\bm{x}]_{g},\\ \text{score}_{\text{Taylor-2nd}}([\bm{x}]_{g},[\nabla\!f(\bm{x})]_{g})& \leftarrow\text{score}_{\text{Taylor-2nd}}([\bm{x}]_{g},[\nabla\!f(\bm{x})]_{g })/\sum_{g\in\mathcal{G}_{I}}\text{score}_{\text{Taylor-2nd}}([\bm{x}]_{g},[ \nabla\!f(\bm{x})]_{g}).\end{split}$ (11) $$

### 3.2 Hybrid Training in HESSO

After identifying the redundant groups in Section 3.1, the next step is to project these groups onto zero and transfer their knowledge to the important groups, ensuring the pruned model maintains its performance. This is achieved through a hybrid training schema.

For the redundant groups $\mathcal{G}_{R}$, we progressively and uniformly push their parameters towards zero. This process is detailed in line 3-4 in Algorithm 1 and decipted in Figure 3. The goal is to ensure that the parameters in the redundant groups become zero after $T_{p}$ steps. During this penalization process, there is a risk of forgetting the knowledge contained in the redundant groups, which may manifest as a degradation in the objective function’s value. To mitigate this, we employ a standard optimization method, such as vanilla SGD or its variants like Adam, on the important groups $\mathcal{G}_{I}$. This step aims to continue optimizing the objective function $f$ and preserve the model’s performance despite the pruning of redundant groups. By maintaining the optimization of the important groups, the knowledge lost from the redundant groups can be transferred and compensated for, ensuring that the pruned model remains effective.

Compared to DHSPG in OTOv2 (Chen et al., ), HESSO offers more explicit control over sparsity exploration. The pruning process in HESSO is regulated by the pruning periods $P$ and the period length $T_{P}$, which determine the pace and extent of the pruning procedure. This structured approach simplifies the pruning process, making it easier to manage and understand.

Contrarily, DHSPG involves significant hyper-parameter tuning to adjust parameters related to sparsity exploration. This tuning often requires domain-specific knowledge, as the appropriate settings can vary depending on the particular application or dataset. This requirement can make DHSPG more complex and less accessible, particularly for practitioners without extensive expertise in hyper-parameter and sparse optimization.

As a result, HESSO is generally easier to use and more adaptable to various applications, as it reduces the need for extensive tuning and specialized knowledge. This makes it a more user-friendly option for achieving structured sparsity in models, allowing for more straightforward and consistent application across different tasks and domains.

### 3.3 Limitations of HESSO

Although HESSO could tackle most DNNs and applications, it may sometimes yield unsatisfactory results when the target DNN possesses certain indispensable structures. The definition of indispensable structure is present as follows.

###### Definition 3.1 (Indispensable Structure) .

Given a deep neural network $\mathcal{M}$, a minimally removal structure is called indispensable if removing it from $\mathcal{M}$ would cause performance measurement degrades significantly.

The origin of indispensable structures varies. One reason may be due to architectural design issues where certain layers in $\mathcal{M}$ play more critical roles than others and are very sensitive to any modifications, as exemplified by a low-level vision benchmark in Section 4.2. Another reason could be the learning strategy. For instance, in large language models (LLMs), it has been observed that knowledge is unevenly distributed across different layers (Chen et al., ). Removing any of these structures could result in an irreversible collapse of the DNN’s performance. However, current saliency score identification in HESSO sometimes fails to identify these indispensable components accurately. The underlying reason is that these saliency scores, as discussed in Section 3.1, are designed to approximate the impact of projecting groups of variables to zero. Such approximations, like the commonly used Taylor importance scores, are typically accurate when the iterate is close enough to the origin point. However, during realistic training and pruning, this requirement is usually not met, resulting in unexpected performance issues.

###### Definition 3.1 (Indispensable Structure) .

### 3.4 Corrective Redundancy Identification Circle

To address the limitations discussed in Section 3.3, we propose a novel Corrective Redundant Identification Cycle (CRIC). This method aims to more reliably identify redundant structures within the target DNN, even when indispensable structures are present. The CRIC mechanism can be seamlessly integrated into HESSO, enhancing its ability to accurately discern which parts of the model can be pruned without compromising performance.

Figure: Algorithm 2 Corrective Redundant Identification Cycle (CRIC)

To mitigate the issue of false positive redundant predictions caused by the approximation error of Taylor expansion, CRIC measures the saliency score of redundant group candidates multiple times along the projection to the origin. Unlike the greedy approach in HESSO, CRIC incorporates a corrective cycle mechanism. This mechanism iteratively promotes groups as redundant and tracks the outlier groups. The cycle terminates when the redundancy prediction is deemed reliable, i.e., no outlier appearance is detected. The final output is a set of redundant groups $\mathcal{G}_{R}$ with the bottom-K overall saliency scores. This approach significantly reduces false positive redundant identifications and addresses the failure cases of HESSO, as demonstrated numerically in Section 4.

In Algorithm 2, we utilize a violating group set $\mathcal{V}$ to track outlier or violating groups, which are more redundant or deviate from the current redundant group prediction. $\mathcal{V}$ is initialized with the group set having the bottom-K saliency scores (see line 4). A historical set $\mathcal{H}$ is also used to track groups whose saliency scores have been fully exploited through multiple sampling along the projection to the origin. This set is initialized as empty $\emptyset$, as shown in line 5.

When the violating set is fairly large, i.e., $|\mathcal{V}|>\mathcal{T}$ with $\mathcal{T}$ as a predefined terminating tolerance, we progressively project these violating groups onto zero. By default, saliency score sampling points are uniformly distributed along the projection process. Groups with lower importance scores that have not been visited in $\mathcal{H}$ are added to a newly constructed violating set $\hat{\mathcal{V}}$ for the next corrective cycle. The corrective cycling algorithm continues until violating instances rarely appear, i.e., $|\mathcal{V}|\leq\mathcal{T}$, see line 6.

Theorem 3.2 guarantees that CRIC terminates within a finite number of iterations, preventing endless loops. Furthermore, Corollary 3.3 provides an upper bound on the number of cycles required by CRIC, ensuring a practical and efficient pruning process.

Figure: Algorithm 3 HESSO with CRIC

###### Theorem 3.2 .

The corrective redundancy identification cycle (Algorithm 2) terminates within a finite number of steps for any terminating tolerance $\mathcal{T}$.

###### Corollary 3.3 .

Given the terminating tolerance $\mathcal{T}$, the CRIC terminates with no more $(|\mathcal{G}|-K)/\max{}\{\mathcal{T},1\}$.

Once the corrective cycles terminate, the saliency scores obtained are deemed reliable. At this point, the redundant set $\mathcal{G}_{R}$ is constructed based on these reliable saliency scores, as indicated in line 20. This set of redundant groups is then returned for further use, such as hybrid training in HESSO, as detailed in Algorithm 2. For simplicity, the HESSO variant that utilizes CRIC for identifying redundant groups is referred to as HESSO-CRIC throughout the paper (outlined in Algorithm 3). This naming convention helps distinguish the variant from the original HESSO method, emphasizing the integration of the corrective cycle mechanism to enhance the reliability of the pruning process.

###### Theorem 3.2 .

###### Corollary 3.3 .

## 4 Numerical Experiments

We numerically demonstrate the efficacy of HESSO across a wide range of applications, from low-level vision tasks such as super-resolution (Zhou et al., ), to high-level vision tasks
object detection (Shi et al., ), as well as natural language processing tasks such as question answering (Rajpurkar et al., ) and the popular foundational large language models (Ding et al., ). The architectures used in these experiments encompass a variety of CNN benchmarks (Chen et al., ) and transformers (Vaswani et al., ). These experiments involve training either from scratch or using a pre-trained checkpoint (when available) to validate the versatile application of HESSO-(CRIC).

In our experimental setup, we begin by directly applying HESSO due to its simplicity and resource efficiency, as it is capable of handling most applications and scenarios. If the direct application of HESSO encounters limitations, we then switch to HESSO-CRIC. It is important to note that HESSO-CRIC requires maintaining part of the model’s variables, resulting in slightly higher memory consumption. Across these experiments, HESSO-(CRIC) consistently achieves competitive, and in some cases, surpasses state-of-the-art results. Given its generality and ease of use, HESSO-(CRIC) stands out as one of the leading optimizers for joint training and pruning.

### 4.1 Recommended Experimental Setup

We recommend the following hyperparameter configurations for HESSO and HESSO-CRIC across varying applications and DNN architectures. For the target DNN to be trained and compressed, end-users likely already have a well-established training pipeline that enables the DNN to achieve high performance. To ensure ease of use, we suggest inheriting the hyperparameters in HESSO and HESSO-CRIC from the baseline training schema wherever there is overlap, such as in optimization variants and first- and second-order momentums.

This inheritance strategy should also be applied to other hyperparameters related to the training pipeline, such as training steps and learning rate schedules, though some slight adjustments may be needed for some applications. Specifically, adjustments may be needed due to the hybrid training process. We recommend beginning pruning at 1/10 of the total training steps and completing progressive pruning over another 1/10 of the total training steps. Because of the hybrid training stage, the learning rate schedule might require modification to ensure the DNN is sufficiently trained at a reasonably high learning rate after reaching the target group sparsity level.

Additionally, HESSO and HESSO-CRIC support training either from scratch or from a pre-trained checkpoint. For better performance and faster convergence, we recommend starting from a pre-trained status if such a checkpoint is available. We summarize the recommended hyperparameter selections and training strategies in Table 1. Remark that better hyperparameter setups or training strategies may exist for specific domain tasks to achieve superior performance.

For the remainder of the manuscript, we conduct the experiments according to the above recommenced criteria, unless otherwise specified. Additionally, all experiments were conducted on an NVIDIA A100 GPU with 80GB of memory.

**Table 1: Recommended hyper-parameters and training strategies for HESSO and HESSO-CRIC.**
| Hyper-parameter | Type | Recommended Setup |
| --- | --- | --- |
| \rowcolor<br>Gainsboro!50<br>Optimizer variant | HESSO-(CRIC) | Inherit as the baseline optimizer. Currently support {SGD, Adam, AdamW}. |
| \rowcolor<br>Gainsboro!50<br>Group sparsity | HESSO-(CRIC) | Set upon the target pruned model size. If all variables could be pruned, the pruned model size could be approximately equal as quadratic of the density level. In addition, a randomly pruned model could be obtained by OTO’s APIs. |
| \rowcolor<br>Gainsboro!50<br>First-order momentum | HESSO-(CRIC) | Inherit as the baseline optimizer’s first-order momentum. |
| \rowcolor<br>Gainsboro!50<br>Second-order momentum | HESSO-(CRIC) | Inherit as the baseline optimizer’s second-order momentum. |
| \rowcolor<br>Gainsboro!50<br>Weight-decay | HESSO-(CRIC) | Inherit as the baseline optimizer’s weight-decay. |
| \rowcolor<br>Gainsboro!50<br>Initial learning rate | HESSO-(CRIC) | Inherit as the baseline optimizer’s initial learning rate. |
| \rowcolor<br>Gainsboro!50<br>Salience Score Criteria | HESSO-(CRIC) | By default equally considering the scores in Section 3.1. |
| \rowcolor<br>Gainsboro!50<br>Start pruning step | HESSO-(CRIC) | Set up as 1/10 of total training steps. |
| \rowcolor<br>Gainsboro!50<br>Pruning steps | HESSO-(CRIC) | Set up as 1/10 of total training steps. |
| \rowcolor<br>green!10<br>Pruning periods | HESSO | Empirically suggest to set as 10. |
| \rowcolor<br>red!7<br>Sampling steps | HESSO-CRIC | Empirically suggest to set as 10. |
| \rowcolor<br>yellow!10<br>Learning rate scheduler | Training | Inherit as the baseline training, yet might need adjustments in some application to ensure the model after reaching target group sparsity is sufficiently trained under relatively large learning rate. |
| \rowcolor<br>yellow!10<br>Total training steps | Training | Inherit as the baseline training and adjust upon the learning rate scheduler. |
| \rowcolor<br>yellow!10<br>Start training from scratch or<br>pretraining checkpoint | Training | Both are supported. For better performance, recommend to start from pretraining checkpoint if available. |

### 4.2 Super Resolution

We first selected the popular CARN architecture (Ahn et al., ) for the super-resolution task with a scaling factor of two, referred to as CARNx2. The benchmark DIV2K dataset (Agustsson and Timofte, ) was used for training, while Set14 (Zeyde et al., ), B100 (Martin et al., ), and Urban100 (Huang et al., ) datasets were employed for evaluation. Initially, we utilized OTO’s pruning dependency analysis to identify minimally removable structures and partitioned the trainable variables into pruning-zero-invariant groups. However, directly applying DHSPG or HESSO often led to significant performance degradation that was not reversible. This issue arose due to the architectural design, where the penultimate convolutional layer in CARN plays a crucial role in producing satisfactory visual results, making it a indispensable structure. Pruning minimally removable structures from this layer caused the remaining filters to fail in generating reasonable visual outcomes. The saliency score estimation in DHSPG and HESSO lacked the ability to avoid pruning such essential structures, resulting in irreversible performance collapse.

In OTOv2’s paper (Chen et al., ), we manually excluded these indispensable structures from pruning. However, this manual identification is time-consuming and requires expert knowledge. To address this, we directly applied HESSO-CRIC to CARN and observed that it reliably identified these crucial structures as important groups, leading to a successfully pruned model. Specifically, following (Agustsson and Timofte, ), we used Adam as the optimization variant in DHSPG, HESSO, and HESSO-CRIC, leveraging both first- and second-order momentum to estimate the gradient. As shown in Table 2, when manually excluding indispensable structures, both DHSPG and HESSO significantly reduced FLOPs and parameters by approximately 33% to 80%, with negligible PSNR degradation. HESSO-CRIC achieved a better trade-off between FLOP reduction and PSNR, as demonstrated by exhibiting the frontier curve under varying pruning ratios. Furthermore, it can automatically identify indispensable structures, preventing irreversible performance collapse.

**Table 2: Structurally pruning CARNx2.**
| Optimizer | Exclusion of | Group Sparsity | FLOPS | # of Params | PSNR |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Dispensable Structure | Set14 | B100 | Urban100 |  |  |  |  |
| Baseline | – | – | 100% | 100% | 33.5 | 32.1 | 31.5 |
| \cdashline1-8 DHSPG | Manual | 50% | 24.3% | 24.1% | 33.2 | 31.9 | 31.1 |
| DHSPG | No | 50% | ✗ | ✗ | ✗ | ✗ | ✗ |
| \cdashline1-8 HESSO | Manual | 20% | 66.9% | 66.8% | 33.5 | 32.1 | 31.7 |
| HESSO | Manual | 30% | 50.8% | 50.6% | 32.3 | 32.0 | 31.5 |
| HESSO | Manual | 40% | 40.0% | 39.7% | 33.3 | 32.0 | 31.4 |
| HESSO | Manual | 50% | 30.8% | 30.5% | 33.2 | 31.9 | 31.1 |
| HESSO | Manual | 60% | 19.4% | 18.0% | 33.1 | 31.8 | 31.0 |
| HESSO | No | 50% | ✗ | ✗ | ✗ | ✗ | ✗ |
| \cdashline1-8 HESSO-CRIC | Automatic | 20% | 66.9% | 66.8% | 33.5 | 32.1 | 31.8 |
| HESSO-CRIC | Automatic | 30% | 53.4% | 53.2% | 33.4 | 32.1 | 31.7 |
| HESSO-CRIC | Automatic | 40% | 40.4% | 40.1% | 33.3 | 32.0 | 31.4 |
| HESSO-CRIC | Automatic | 50% | 28.7% | 28.4% | 33.3 | 32.0 | 31.3 |
| HESSO-CRIC | Automatic | 60% | 18.1% | 17.7% | 33.2 | 31.9 | 31.1 |
|  |  |  |  |  |  |  |  |

Figure: (a) Low resoluted image.
Refer to caption: extracted/5847239/img_051_SRF_2_LR-min.png

### 4.3 Question and Answering

**Table 3: Structurally pruning Bert on SQuAD.**
| Method | Group Sparsity | # of Params | F1-score |
| --- | --- | --- | --- |
| Baseline | 100% | 88.3% |  |
| \cdashline1-4 ProxSSI (Deleu and Bengio, ) | – | 83.4%^† | 82.0% |
| HSPG (Chen et al., ) | – | 91.0% | 84.1% |
| HSPG (Chen et al., ) | – | 66.7% | 82.0% |
| DHSPG | 10% | 93.3% | 87.7% |
| DHSPG | 30% | 80.1% | 87.3% |
| DHSPG | 50% | 68.3% | 86.2% |
| DHSPG | 70% | 55.0% | 83.8% |
| \cdashline1-4<br>HESSO | 10% | 94.78% | 87.20% |
| HESSO | 30% | 84.33% | 86.72% |
| HESSO | 50% | 73.88% | 86.46% |
| HESSO | 70% | 63.34% | 85.50% |
| HESSO | 90% | 53.0% | 84.25% |
| \cdashline1-3 HESSO-CRIC | 10% | 94.78% | 87.48% |
| HESSO-CRIC | 30% | 84.32% | 87.10% |
| HESSO-CRIC | 50% | 73.88% | 86.50% |
| HESSO-CRIC | 70% | 63.44% | 85.96% |
| HESSO-CRIC | 90% | 53.0% | 84.10% |
| ^† Approximate value based on   (Deleu and Bengio, ). |  |  |  |

Later, we compare HESSO-(CRIC) with DHSPG, HSPG, and a representative proximal method (ProxSSI) for pruning a transformer model Bert(Vaswani et al., ), evaluated on the SQuAD question-answering benchmark(Rajpurkar et al., ). It is important to note that proximal methods have been standard algorithms for solving sparse optimization problems for decades. However, recent research has shown that they are not effective at exploring sparsity while maintaining model performance in deep learning applications (Dai et al., ).

Following previous work (Chen et al., ), we excluded the embedding layer from pruning. As shown in Table 3, HESSO, HESSO-CRIC, and DHSPG perform competitively on this task in terms of parameter reduction while maintaining F1 scores. However, DHSPG only achieves these results after extensive hyper-parameter tuning, which is neither efficient nor convenient. While HSPG performs comparably, its tendency to penalize all variables toward zero severely restricts the optimization search space, leading to suboptimal performance. Proximal methods, contrarily are not comparable with the HESSO and HSPG families.

### 4.4 Object Detection

Next, we tested HESSO on the popular YOLO (Redmon et al., ) object detection model using the COCO benchmark dataset (Lin et al., ). Table 4 presents the structural pruning results for YOLOv5l (Jocher et al., ) using HESSO and HESSO-CRIC. It is important to note that we selected YOLOv5l to facilitate comparisons with other existing benchmark methods. Starting from a pretrained YOLOv5l model, we applied HESSO and HESSO-CRIC with a target group sparsity of 30%, resulting in a sub-network containing 49% of the original parameters. This allows for direct comparison with benchmarks that retain 50% of the model’s parameters. The results show that a single run of HESSO and HESSO-CRIC achieved significantly higher Mean Average Precision (mAP) compared to other pruning approaches, which often require more complex, multi-stage procedures.

**Table 4: Structurally pruning Yolov5l on COCO.**
| Method | # of Params | $\text{mAP}_{0.5}$ | $\text{mAP}_{0.5:0.95}$ |
| --- | --- | --- | --- |
| Baseline | 100% | 66.31% | 47.71% |
| \cdashline1-4 HFP (Enderich et al., ) | 50% | 63.5% | 43.4% |
| TCFP (Jeon et al., ) | 50% | 61.8% | 42.7% |
| HESSO (30% group sparsity) | 49% | 63.1% | 44.4% |
| HESSO-CRIC (30% group sparsity) | 49% | 63.1% | 44.5% |

Figure: (a) Pretrained YOLOv5l.
Refer to caption: extracted/5847239/bus_base.jpg

### 4.5 Large Language Model

Finally, we evaluated HESSO-(CRIC) on large language models (LLMs). Since both HESSO and HESSO-CRIC utilize full gradient information, we focused on LLMs with fewer than 3 billion parameters, such as the representative Phi-2-2.7B(Microsoft, ), to ensure that a single 80GB GPU is sufficient, without requiring tensor parallelism (Ding et al., ). Our experimental setup followed that of LoRAShear (Chen et al., ).

We observed that without conducting a knowledge distribution analysis and manually skipping certain layers from pruning, as LoRAShear (Chen et al., ) did, HESSO often led to an irreversible performance collapse. This is because knowledge in LLMs is unevenly distributed across layers due to the learning strategy. The saliency scores calculated upon the pretraining weights may fail to identify essential structures, making it difficult to differentiate between indispensable components and those that could be pruned. As a result, pruning such critical structures severely degrades the model’s performance, making recovery with limited resources nearly impossible.

In contrast, HESSO-CRIC was able to automatically bypass these crucial structures, enabling effective and successful pruning. We then compared its results with SliceGPT (Ashkboos et al., ) across several popular benchmarks using the LM-Evaluation-Harness (Gao et al., ). Our findings indicate that HESSO-CRIC consistently outperforms SliceGPT at varying pruning ratios, with performance improvements becoming more pronounced as the pruning ratio increases. We have the perspective that the hybrid training scheme in HESSO serves a similar purpose by achieving computational invariance between pruned sub-network and full DNN, maintaining the same objective function value during progressive pruning. Ours is architecture agnositic and easy to apply onto various applications. However, SliceGPT requires significant manual efforts to introduce addition auxiliary structures and restricts onto transformer architecture. Furthermore, HESSO-CRIC offers the added advantage of a more reliable identification mechanism, excluding indispensable structures and better preserving performance, particularly at higher pruning ratios.

**Table 5: HESSO-CRIC over Phi-2-2.7B.**
| Pruning Ratio | Method | BoolQ | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Baseline | Phi-2-2.7B | 83.30 | 79.11 | 73.82 | 75.77 | 80.05 | 54.18 | 51.40 | 71.09 |
| \cdashline1-10<br>Ratio $=20\%$ | SliceGPT (Ashkboos et al., ) | 68.56 | 74.16 | 61.22 | 67.56 | 70.20 | 41.04 | 38.80 | 60.22 |
|  | HESSO-CRIC | 69.67 | 74.37 | 62.27 | 66.54 | 72.30 | 41.44 | 38.20 | 60.67 |
| \cdashline1-10<br>Ratio $=25\%$ | SliceGPT (Ashkboos et al., ) | 63.70 | 71.49 | 57.72 | 66.46 | 65.86 | 38.99 | 39.80 | 57.71 |
|  | HESSO-CRIC | 67.06 | 73.77 | 58.51 | 65.18 | 70.66 | 38.60 | 38.00 | 58.74 |
| \cdashline1-10<br>Ratio $=30\%$ | SliceGPT (Ashkboos et al., ) | 38.17 | 61.04 | 42.05 | 60.38 | 50.80 | 28.07 | 31.2 | 44.53 |
|  | HESSO-CRIC | 67.61 | 72.14 | 53.11 | 62.75 | 62.74 | 34.81 | 36.20 | 55.62 |

## 5 Conclusion

In this work, we have introduced HESSO, a novel Hybrid Efficient Structured Sparse Optimizer, designed to address the challenges of structurally pruning deep neural networks (DNNs) while maintaining their performance. HESSO streamlines the pruning process by incorporating a progressive pruning strategy guided by saliency scores, a hybrid training approach that transfers knowledge from redundant structures to essential ones, and a user-friendly interface suitable for a wide range of training applications. Additionally, we proposed the Corrective Redundant Identification Cycle (CRIC), a mechanism that mitigates the risk of irreversible pruning-induced performance collapse by reliably identifying and preserving indispensable structures within the network.

Our extensive numerical experiments across various domains, including computer vision and natural language processing, demonstrate that HESSO achieves competitive performance to state-of-the-art benchmarks, adapting to diverse architectures. Furthermore, the integration of CRIC into HESSO not only prevents performance degradation but also enhances the optimizer’s capability to deliver superior results in scenarios where traditional pruning methods falter.

Overall, HESSO and its enhanced version, HESSO-CRIC, represent a significant advancement in the field of structured pruning, offering a robust and versatile solution for optimizing deep neural networks with minimal human intervention. These contributions pave the way for more efficient and scalable model compression techniques, potentially leading to broader adoption in real-world applications where resource constraints are critical.

## Appendix A Appendix

Proof of Theorem 3.2

###### Proof.

The statement is equivalent to that the violating cycle line 6-19 in Algorithm 2 terminates within finite number of steps. For convenience, we denote $\mathcal{V}_{l}$ as the violating set at $l$th cycle. The statement then becomes that there exists an $L<\infty$, such that $\mathcal{V}_{L}=\emptyset$. We now prove it as a two-step fashion.

At first, we show that the violating set at $l$th loop $\mathcal{V}_{l}$ is disjoint to those at all previous loops $\{\mathcal{V}_{i}\}_{i=0}^{i=l-1}$. This is true since the $\mathcal{V}_{l}$ is constructed excluding elements in the $l-1$th historical set $\mathcal{H}_{l}$

$$ $\mathcal{V}_{l}\leftarrow\widehat{\mathcal{V}}_{l-1}\ /\ \mathcal{H}_{l-1},$ (19) $$

and $\mathcal{H}_{l-1}$ is the union of previous violating set $\mathcal{H}_{l-1}=\bigcup_{i=0}^{i=l-1}\mathcal{V}_{i}$. Therefore, $\mathcal{V}_{l}$ is disjoint to all violating sets $\{\mathcal{V}_{i}\}_{i=0}^{i=l-1}$.

Secondly, we prove on contraction. Suppose there exists no an $L<\infty$, such that $\mathcal{V}_{L}=\emptyset$. Since $\mathcal{V}_{l}$ is disjoint with $\{\mathcal{V}_{i}\}_{i=0}^{i=l-1}$, it implies that $\mathcal{V}_{l}$ must include previously unseen and new element from $\mathcal{G}$. Consequently, the historical set $H_{l}=\bigcup_{i=0}^{i=l}\mathcal{V}_{i}$ will have infinite number of elements as $l$ tends to $\infty$, i.e.,

$$ $\lim_{l\to\infty}|H_{l}|=\infty.$ (20) $$

However, (20) contradicts that the historical set $H_{l}$ is a subset of group partition set $\mathcal{G}$, and the cardinality of $\mathcal{G}$ is finite. Therefore, we conclude the corrective redundancy identification cycle always terminates within a finite number of steps.
∎

###### Proof.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What is the computational cost of employing HESSO? I am wondering whether it will significantly increase the computational memory and flow training speed.
2. Seems CIRC can be used as a generic module to identify the indispensable structures. Can it be applied to other pruning algorithms and what is the performance?
soundness: 3
strengths: 1. This paper is well written and organized.
2. This paper provides thorough experiments, by evaluating the performance of HESSO on various tasks and comparing with many other pruning methods.
3. It is demonstrated that HESSO is able to achieve strong and impressive results with a considerable pruning ratio.
summary: This paper proposes Hybrid Efficient Structured Sparse Optimizer (HESSO), a structured pruning algorithm. In addition, for reliably identifying the indispensable structure, the authors introduce Corrective Redundant Identification Cycle (CRIC). The experimental results show that HESSO with CRIC plugged in can perform well on various types of models, e.g., image classification, object detection and large language models.
weaknesses: See questions.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please see the weakness part.
soundness: 1
strengths: 1.By simplifying hyper-parameter tuning and explicitly controlling sparsity exploration, HESSO enhances usability, making it accessible to practitioners without deep optimization expertise.

2. The optimizer is designed to be applicable to any DNN architecture and task, increasing its potential impact across various domains and applications.
summary: The paper addresses the challenge of deploying large deep neural networks by proposing **HESSO**, a new optimizer that enables automatic one-shot joint training and structured pruning for various DNN architectures and tasks. HESSO simplifies the hyper-parameter setup and employs a progressive pruning strategy to explicitly control sparsity exploration, enhancing usability. It optionally integrates a Corrective Redundancy Identification Cycle mechanism to accurately identify redundant groups, minimizing the risk of irreversible performance degradation caused by pruning indispensable structures.  The authors have conducted experiments on various type of tasks and models.
weaknesses: 1. As a paper on structured pruning, it does not compare with state-of-the-art methods on ImageNet, such as OFA, EagleEye ResRep, etc. The comparisons are too simplistic, making it impossible to see the advantages of the proposed method. Furthermore, in experiments on large models, it only compares with SliceGPT, which is not tailored for structured pruning. More mainstream baseline methods like LLMPruner need to be included for comparison. Overall, the experimental section significantly dampens my enthusiasm for this paper.

2. The authors' discussion of related work in the field is rather lacking and not sufficiently grounded. The entire introduction, methodology, and experimental discussions focus only on OTO-based papers[4,5,6]. However, as far as I know, there are many papers on automatic structure search for structured pruning, including OFA[1], EagleEye[2], etc. I suggest that the authors provide a more comprehensive coverage of the field, which would effectively enhance the quality of this paper.

[1] Cai, H., Gan, C., Wang, T., Zhang, Z. and Han, S., 2019. Once-for-all: Train one network and specialize it for efficient deployment. In ICLR, 2020.

[2] Li, B., Wu, B., Su, J. and Wang, G., 2020. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. In ECCV, 2020.

[3] Ding, X., Hao, T., Tan, J., Liu, J., Han, J., Guo, Y. and Ding, G., 2021. Resrep: Lossless cnn pruning via decoupling remembering and forgetting. In ICCV, 2021.

[4] Chen, T., Ji, B., Ding, T., Fang, B., Wang, G., Zhu, Z., Liang, L., Shi, Y., Yi, S. and Tu, X., 2021. Only train once: A one-shot neural network training and pruning framework. In NeurIPs, 2021.

[5] Chen, T., Liang, L., Ding, T., Zhu, Z. and Zharkov, I., 2023. Otov2: Automatic, generic, user-friendly. In Arxiv, 2022.

[6] Chen, T., Ding, T., Zhu, Z., Chen, Z., Wu, H., Zharkov, I. and Liang, L., 2023. OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing Operators. In Arxiv, 2023.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How does HESSO manage the trade-off between sparsity and performance? Is there a recommended range for optimal sparsity levels?
2. Is there a risk that the method could lead to overfitting, especially during the pruning phase? Maybe evaluation in held-out datasets are needed.
soundness: 3
strengths: 1. This paper is well-motivated, and the topic is highly relevant in the current AI landscape, where deploying efficient models in resource-constrained environments is critical. 
2. The manuscript provides a well-structured theoretical framework for HESSO, which helps justify the design choices made in the method.
3. The authors conduct extensive experiments in an array of tasks.
summary: The manuscript presents a novel approach to deep neural network (DNN) training and structured pruning through Hybrid Efficient Structured Sparse Optimize (HESSO). The topic is highly relevant in the current AI landscape, where deploying efficient models in resource-constrained environments is critical. HESSO stands out by simplifying hyper-parameter setup and offering a progressive pruning strategy that enhances user-friendliness. In addition, the authors also discuss the limitations of the proposed method and the approximation errors of salience scores.
weaknesses: 1. While HESSO aims to simplify hyper-parameter tuning, the manuscript does not provide sufficient evidence to support the claim of reduced sensitivity. A comparative analysis of tuning efforts required for HESSO versus traditional methods is needed.
2. The scalability of HESSO to larger models is not well demonstrated. A table or figure listing the computational cost of applying HESSO to models across different model sizes should be provided.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: 1. In line 215, the authors state, “The selection of the saliency score in HESSO is flexible and can be tailored to different purposes.” I would like to know how the authors select the saliency score for different tasks in Sections 4.2-4.6. 
2. In lines 503-507, the authors state, “Since both HESSO and HESSO-CRIC … without requiring tensor parallelism.” Does this mean that the HESSO method does not support models larger than 3 billion parameters? In addition, since both HESSO and HESSO-CRIC utilize full gradient information of LLMs, I would like to know the training cost of HESSO and HESSO-CRIC on LLMs.
[1] Structured Pruning Learns Compact and Accurate Models. ACL 2022 Main.
[2] Auto-Train-Once: Controller Network Guided Automatic Network Pruning
from Scratch. CVPR 2024.
[3] LLM-Pruner: On the Structural Pruning of Large Language Models. NeurIPS 2023.
[4] LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning. ACL 2024 Findings.
[5] LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery. ArXiv.
[6] OTOv3: Towards Automatic Sub-Network Search Within General Super Deep Neural Networks. ArXiv.
soundness: 2
strengths: 1. The paper is well-written, with clear organization and rigorous logic, making it understandable to readers.
2. HESSO can automatically and efficiently train and prune DNNs with any architecture, providing competitive performance, which holds broad significance in the field of neural network pruning.
3. The authors conducted a comprehensive analysis of how approximation errors of saliency score can lead to misidentification of indispensable structures in pruned DNNs, resulting in performance collapse. This part is well-organized, highly interesting, and very insightful.
summary: This paper proposes HESSO, a novel optimizer designed for structured pruning of DNNs. HESSO aims to automatically and efficiently train any DNN to produce high-performing sub-networks with minimal need for hyper-parameter tuning, enhancing user-friendliness. Additionally, this paper introduces an extension of HESSO called HESSO-CRIC to address issues related to the irreversible performance collapse that sometimes occurs in pruned models. HESSO and HESSO-CRIC are evaluated across various applications, including computer vision and natural language processing, where it demonstrates competitive performance compared to existing baselines.
weaknesses: 1. I believe the main issue with this paper is the lack of many necessary comparison baselines, or the existing comparison baselines are outdated.
1) Section 4.3: In the CoFi [1] method, BERT-Base achieves a lossless F1 score on the SQuAD dataset with a 70% reduction in parameters (Figure 2 in CoFi). However, in the HESSO method, BERT-Base shows a drop of about 4% in F1 score with approximately a 50% reduction in parameters.
2) Section 4.4: HESSO seems to be inferior to the ATO [2] method, as ATO achieved a TOP-1 accuracy of 76.07% with a 61.7% reduction in FLOPs, only a -0.06% drop compared to the baseline. However, under the same FLOPs, the accuracy of HESSO is below 76%.
3) Section 4.6: The authors only compare with the SliceGPT method, lacking comparisons with some important structured pruning baselines for LLMs, such as LLM-Pruner (with LoRA fine-tuning) [3], LoRAPrune [4] and LoRAShear [5].
Due to the lack of the above baselines, I have concerns about the effectiveness of the HESSO method.
2. It seems that HESSO is simply improved version of OTO v3. As far as I know, OTO v3 also proposes a hybrid training scheme that applies different update mechanisms to important and redundant variable groups. HESSO does not seem to introduce any significant innovations but rather represents a minor technical improvement over OTO v3.
3. Different saliency scores likely result in varying approximation errors. The authors should demonstrate whether the CRIC mechanism effectively corrects each type of approximation error. Therefore, the authors should show the impact of different saliency scores in HESSO-CRIC on the final accuracy.
4. The phrase "can not be recovered given user resources" in Definition 3.1 is confusing, as it is evident that models with different parameters require significantly different training resources. The authors should clarify the definition of "user resources" more explicitly.