OTOv2: Automatic, Generic, User-Friendly
Authors: Tianyi Chen
Luming Liang
Tianyu DING
Zhihui Zhu
Ilya Zharkov
Venue: ICLR 2023 poster (2023)
Keywords: Model Compression
One Shot
Automatic
Generic
User-Friendly

Abstract: The existing model compression methods via structured pruning typically require complicated multi-stage procedures. Each individual stage necessitates numerous engineering efforts and domain-knowledge from the end-users which prevent their wider applications onto broader scenarios. We propose the second generation of Only-Train-Once (OTOv2), which first automatically trains and compresses a general DNN only once from scratch to produce a more compact model with competitive performance without fine-tuning. OTOv2 is automatic and pluggable into various deep learning applications, and requires almost minimal engineering efforts from the users. Methodologically, OTOv2 proposes two major improvements: (i) Autonomy: automatically exploits the dependency of general DNNs, partitions the trainable variables into Zero-Invariant Groups (ZIGs), and constructs the compressed model; and (ii) Dual Half-Space Projected Gradient (DHSPG): a novel optimizer to more reliably solve structured-sparsity problems. Numerically, we demonstrate the generality and autonomy of OTOv2 on a variety of model architectures such as VGG, ResNet, CARN, ConvNeXt, DenseNet and StackedUnets, the majority of which cannot be handled by other methods without extensive handcrafting efforts. Together with benchmark datasets including CIFAR10/100, DIV2K, Fashion-MNIST, SVNH and ImageNet, its effectiveness is validated by performing competitively or even better than the state-of-the-arts. The source code is available at https://github.com/tianyic/only_train_once.

Published as a conference paper at ICLR 2023
OTOv2: AUTOMATIC, GENERIC, USER-FRIENDLY
Tianyi Chen; Luming Liang, Tianyu Ding, Ilya Zharkov Microsoft Redmond, WA 98052, USA
Zhihui Zhu
The Ohio State of University Columbus, OH 43210, USA 2hu 344004400su.. edd
ftiachen, lulian, tianyuding, charkov//emccrroooft.ccom
ABSTRACT
The existing model compression methods via structured pruning typically require complicated multi-stage procedures. Each individual stage necessitates numerous engineering efforts and wider applications onto broader scenarios. We propose the second generation of Only-Train-Onee (OTOv2), which first automatically trains and compresses gen- eral DNN only once from scratch to produce a more compact model with competi- tive performance without fine-tuning. OTOv2 is automatic and pluggable into var- ious deep learning applications, and requires almost minimal engineering efforts from the users. Methodologically, OTOv2 proposes two major improvements: (i) Autonomy: automatically exploits the dependency of general DNNs, partitions the trainable variables into Zero-Invariant Groups (ZIGs), and constructs com- pressed model; and (ii) Dual Half-Space Projected Gradient (DHSPG); a novel optimizer to more reliably solve structured-sparsity problems. Numerically, we demonstrate the generality and autonomy of OTOv2 on variety of model archi- tectures such as VGG, ResNet, CARN, ConvNeXt, DenseNet and StackedUnets, the majority of which cannot be handled by other methods without extensive handcrafting efforts. Together with benchmark datasets including CIFAR 1EFAR10/100 10/100, DIV2K, Fashion-MNIST, SVNH and ImageNet, its effectiveness is validated by performing competitively or even better than the state-of-the-arts. The source code is available https: /ttp//gttttubbtb com/ tianyic/onnnol httpe//oolyyaailalaeaaaaaaaaaaae once.
from the end-users which prevent their
INTRODUCTION
Large-scale Deep Neural Networks (DNNs) have demonstrated successful in a variety of applica- tions (He al., 2016). However, how deploy such heavy DNNs onto resource-constrained envi- ronments is facing severe challenges. Consequently, in both academy and industry, compressing full DNNs into slimmer ones with negligible performance regression becomes popular. Although this area has been explored in the past decade, it is still far away from being fully solved.
Weight pruning is perhaps the most popular compression method because of its generality and ability in achieving sigmificant reduction of FLOPs and model size by identifying and removing redundant structures (Gale et al., 2019; Han et al., 2015; Lin et al., 2019). However, most existing pruning shown in Figure 1, which has apparent limitations: (i) Hand-Craff and User-Hardness: requires significant engineering efforts and expertise from users to apply the methods onto their own scenarios; (ii) Expensiveness: ducts DNN training multiple times including the foremost pre-training, the intermediate training for identifying redundancy and the afterwards fine-tuning and (iii) Low-Generality: many methods are designed for specific architectures and tasks and need additional efforts to be extended to others.
methods typically proceed a complicated multi-stage procedure
To address those drawbacks, we naturally need DNN training and pruning method to achieve the
Goal. Given a general DNN, automatically train it only once to achieve both high performance and slimmer model architecture simultaneously without pre pre-training and fine-tuning.
To realize, the following key problems need to be resolved systematically. (i) What are the removal structures (see Section 3.1 for a formal definition) of DNNs? (ii) How to identify the redundant
"Corresoonding Author. Partially supported by NSF grant CCF-2240708.

---

Published as a conference paper at ICLR 2023
Midel Relualurs
OTO+2
OTÖvl Low
Others High
Mucki
Sim Misld
Training Cost Autonomy
Low
OTOv1
Fre--Friendliness
x
x
Generality
Parti
BSPO
Sia Mulk
Automatic
User Secr-Priendly
Generic
OTOON
Hfigh Performance
Figure OTOv2 versus existing methods.
removal structures? (iii) How effectively remove redundant structures without deteriorating the model performance to avoid extra fine-tuning? (iv) How to make all the above proceeding automat- ically? Addressing them is challenging the manner of both algorithmic designs and engineering developments, thereby is not achieved yet by the existing methods to our knowledge.
To resolve (i-iii), Only-Train-Onee (OTOvI) (Chen al., 2021b) proposed a concept so-called Zero- Invariant Group (ZIG), which is a class of minimal removal structures that can be safely removed without affecting the network output if their parameters are zero. To jointly identify redundant ZIGs and achieve satisfactory performance, OTOv1 further proposed a Half-Space Projected Gradient (HSPG) method to compute a solution with both high performance and group sparsity over ZIGs, wherein zero groups correspond to redundant removal structures. As a result, OTOvl trains full DNN from scratch only once to compute a slimmer counterpart exhibiting competitive performance without fine-tuning, and is perhaps the closest to the goal among the existing competitors.
Nevertheless, the fundamental problem (iv) is not addressed in OTOv1, i.e., the ZIGs partition is not automated and only implemented onto several specific architectures. OTOv1 suffers lot from requiring extensive hand-crafting efforts and domain knowledge to partition trainable variables into ZIGs which prohibits its broader usages. Meanwhile, OTOv1 highly depends on HSPG yield lution with both satisfactory performance and high group sparsity. However, the sparsity exploration of HSPG is typically sensitive the regularization coefficient thereby requires time-consuming hyper- parameter tuning and lacks capacity to precisely control the ultimate sparsity level.
To overcome the drawbacks of OTOv1 and simultancously tackle (i-iv), we propose Only-Train-Once v2 (OTOv2), the next-generation one-shot deep neural network training and pruning framework. Given a full DNN OTOv2 is able to train and compress from scratch into a slimmer DNN with sig- nificant FLOOPs and parameter quantity reduction. In contrast to others, OTOv2 drastically simplifies the complicated multi- stage procedures; guarantees performance more reliably than OTOv1; and is generic, automatic and user-friendly. Our main contributions are summarized as follows.
Library Usage
from only.train.oree import CTO
Ganerall DNN model
oto - (modell
optimizer ozo.chapg
Train as normal
optimizer step()
oto.compress
Infrastructure for Automated DNN One-Shot Training and Compression. We propose and develop perhaps the first generic and automated framework to compress general DNN with both excellent performance and substantial complexity reduction in terms of FLOPs and model car- dinality. OTOv2 only trains the DNN once, neither pre-training OTOv2 is user-friende user-friendly and easily applied onto generic tasks as shown in library usage. Its suc- cess relies the breakthroughs from both algorithmic designs and infrastructure developments.
fine-tuning is necessity.
Automated ZIG Partition and Automated Compressed Model Construction. We propose a novel graph algorithm to automatically exploit and partition the variables of a general DNN into Zero-Invarian Groups (ZIGs), i.e., the minimal groups of parameters that need to be pruned together. We further propose a novel algorithm to automatically construct the compressed model by the hierarchy of DNN and eliminating the structures corresponding ZIGs as zero. Both algorithms are dedicately designed, and work effectively with low time and space complexity.
Novel Structured-Sparriity Optimization Algorithm. We propose novel optimization algo- rithm, called Dual Half-Space Projected Gradient (DHSPG), to train a general DNN once from scratch to effectively achieve competitive performance and high group sparsity in the manner of ZIGs, which solution is further leveraged into the above automated compression. DHSPG for- mulizes a constrained sparse optimization problem and solves it by constituting a direction within the intersection of dual half-spaces to largely ensure the progress to both the objective convergence and the identification of redundant groups. DHSPG outperforms the HSPG in OTOv1 in terms of tuning, and more reliably controlling sparsity.
enlarging search space, fewer
2

---

Published as a conference paper at ICLR 2023
Experimental Results. We apply OTOv2 onto variety of DNNs (most of which have struc- tures with complicated connectivity) and extensive benchmark datasets, including CIFAR10/100, DIV2K, SVNH, Fashion-MNIST and ImageNet. OTOv2 trains and compresses various DNNs simultaneously from scratch without fine-tuning for significant inference speedup and parameter reduction, and achieves competitive even sate-of-the-ar results on compression benchmarks.
2 RELATED WORK
Structured Pruning. To compute compact architectures for efficient model inference and storage, structured pruning identifies and prunes the redundant structures in full model (Gale et al., 2019; Han et al., 2015). The general procedure can be largely summarized as: (i) train full model; (ii) identify and remove the redundant structures to construct a slimmer DNN based on various criteria, including (structured) sparsity (Lin et al., 2019; Wen et al., 2016; Li et al., 2020b; Zhuang et al., 2020; Chen et al., 2017; 2018; 2021a; 2020a; Gao et al., 2020; Zhuang et al., 2020; Meng et al., 2020: Yang et al., 2019), Bayesian pruning (Zhou et al., 2019; Louizos et al., 2017; van Baalen et al., 2020), ranking importance (Li et al., 2020a; Luo et al., 2017; Hu et al., 2016; He ct al., 2018a; Li al., 2019; Zhang al., 2018), reinforcement learning (He et al., 2018b; Chen et al., 2019), lottery ticket (Frankle & Carbin, 2018; Frankle at., 2019; Renda et al., 2020), etc.; (ii)) (iteratively) retrain the pruned model to regain the accuracy regression during pruning. These methods have to conduct a complicated and time-consuming procedure to trains the DNN multiple times and requires a good deal of domain knowledge to manually proceed every individual step. OTOv1 (Chen et al., 202 2021b) is recently proposed to avoid fine-tunne ttnnig and end-to-end train and compress the DNN once, whereas its automation relies on spending numerous handcrafting efforts on creating ZIGs partition and slimmer model construction for specific target DNNs in advance, thereby is actually semi-automated.
Automated Machine Learning (AutoML). OTOv2 fills into vital gap within AutoML domain regarding given an general DNN architecture, how to automatically train and compress it into a slim- mer one with competitive performance and significant FLOPs and parameter quantity reduction. The existing AutoMI. methods focus on (i) automated feature engineering (Kanter & Veeramachaneni, 2015), (ii) automated hyper-ypraaeee aaperpparameter setting (Klein al., 2017), and (iii) neural architecture search (NAS) (Elsken et al., 2018). NAS searches DNN architecture with satisfactory performance from a prescribed fixed full graph wherein the connection between two nodes (tensors) is searched from a pool of prescribed operators. NAS itself has no capability to slim and remove redundancy from the searched architectures due the pool being fixed and is typically time-consuming. As a result, NAS may serve prerequisite step to search target network architecture as the input to OTOv2.
OTOV2
OTOv2 has nearly reached the goal of model compression via weight pruning, which is outlined in Algorithm 1. In general, given a neural network M to be trained and compressed, OTOv2 first automatically figures out the dependencies among the vertices to exploit minimal removal structures and partitions the trainable variables into Zero-Invariant Groups (ZIGs) (Algorithm 2). ZIGs (G) are then fed into a structured sparsity optimization problem, which is solved by a Dual Half-Space Projected Gradient (DHSPG) method yield a solution DHHSPG with competitive performance as well as high group sparsity in the view of ZIGs (Algorithm 3). The compressed model M is ul- timately constructed via removing the redundant structures corresponding to the ZIGs being zero M significantly accelerates the inference in both time and space complexities and returns the iden- tical outputs to the full model M parameterized as dusse due to the properties of ZIGs, thus avoids further fine-tuning M The whole procedure is proceeded automatically and easily employed onto various DNN applications and consumes almost minimal engineering efforts from the users.
Algorithm Outline of OTOv2.
1: Input. An arbitrary full model M to be trained and compressed (no need to be pretrained).
2: Automated ZIG Partition. Partition the trainable parameters of M into G.
3: Train M by DHSPG. Seek a highly group-sparse solution DusPG with high performance.
4: Automated Compressed Model M Construction. Construct a slimmer model upon T
5: Output: Compressed slimmed model M
3

---

Published as a conference paper at ICLR 2023
3.1 AUTOMATED ZIG PARTITION
Background. We review relevant concepts before describing how to proceed ZIG partition automat- ically. Due to the complicated connectivity of DNNs, removing an arbitrary structure or component may result in an invalid DNN. We say a structure removal if and only if the DNN without this com- ponent still serves as a valid DNN, Consequently, a removal structure called minimal if and only if it can not be further decomposed into multiple removal structures. A particular class of minimal re- moval structures-thal produces zero outputs to the following layer their parameters being zero- are called ZIGs (Chen et al., 2021b) which can be removed directly without affecting the network output. Thus, each ZIG consists of minimal group of variables that need to be pruned together and dominates most DNN structures, e.g.. layers as Conv, Linear and Mult HeHHHeAttt en. While ZIGs exist for general DNNs, their topology can vary significantly due to the complicated connec- tivity. This together with the lack of API poses severe challenges to automatically exploit ZIGs in terms of both algorithmic designs and engineering developments.
Algorithm 2 Automated Zerro-Invarian Group Partition.
1: Input: A DNN M to be trained and compressed.
2: Construct the trace graph (E. V) of .M.
3: Find connected components C over all accessory, shape-dependent joint and unknown vertices.
4: Grow C till incoming nodes are either stem shape-independent joint vertices.
5: Merge connected components in C if any intersection.
6: Group pairwise parameters of stem vertices in the same connected component associated with parameters from affiliated accessory vertices if any as one ZIG into G.
7: Return the zero-invariant groups G.
Algorithmic Outline. To resolve the autonomy of ZIG partition, we present novel, effective and efficient algorithm. As outlined in Algorithm 2. the algorithm essentially partitions the graph of DNN into a set of connected components of dependency, then groups the variables based on the affiliations among the connected components. For more intuitive illustration, provide small but complicated DemoNet along with explanations about its ground truth minimal removal structures (ZIGs) in Figure 2. We now elaborate Algorithm 2 to automatically recover the ground truth ZIGs.
Graph Construction. In particular, we first establish the trace graph (E, V) of the target DNN, wherein each vertex in V refers to specific operator, and the edges in & describe how they con- nect (line 2 of Algorithm 2). We categorize the vertices into stem, joint, accessory or unknown. Stem vertices equip with trainable parameters and have capacity to transform their input tensors into other shapes, e.g., Conv and Linear. Joint vertices aggregate multiple input tensors into a single output such as Add, Mul and Concat. Accessory vertices operate a single input tensor into single output and may possess trainable parameters such as BatchNorm rm and ReLu. The remain- ing unknown vertices proceed some uncertain operations. Apparently, stem vertices compose most of the DNN parameters. Joint vertices establish the connections cross different vertices, and thus dramatically bring hierarchy and intricacy of DNN. To keep the validness of the joint vertices, the minimal removal structures should be carefully constructed. Furthermore, ing input shape dependent (SD) if requiring inputs in the same shapes such as Add, otherwise being shape-independent (SID) such Concat along the channel dimension for Conv layers as input.
call joint vertices be-
Construct Connected Components of Dependency. Now, we need to figure out the exhibiting depen- dency across the vertices to seek the minimal removal structures of the target DNN. To proceed, we first connect accessory, SD joint and unknown vertices together if adjacent to form a set of con- nected components C (see Figure 2c and line of Algorithm 2). This step is to establish the skeletons for finding vertices that depend on each other when considering removing hidden structures. The underlying intuitions of this step in depth are (i) the adjacent accessory vertices operate and are subject to the same ancestral stem vertices if any; (ii) SD joint vertices force their ancestral stem vertices dependent on each other to yield tensors in the same shapes; and (iii) unknown vertices introduce uncertainty, hence finding potential affected vertices is necessary. We then grow C till all their incoming vertices are either stem or SID joint vertices and merge the connected components if any intersection as line 4-5, Remark here that the newly added stem vertices are affiliated by the accessory vertices, such as Convl for BNI-Re and Conv3+Conv2 for BN2 |BN3 in Figure 2d. In addition, the SID joint vertices introduce dependency between their affiliated accessory vertices and incoming connected components, e.g., Concat -BN4 depends on both Convi-BA -NN1-ReLu and Conv3+Conv22 -o222nonoo2eoeeee BN3 since BN4 normalizes their concatenated tensors along channel.
4

---

Published as a conference paper at ICLR 2023
Iepon
Coner
BNI
Rellan
Inpor
Conel
BNI
Rellan
BN2
BN2
Ouipur
Linear22
Linear
Conv3
Conv2
Outper
Libear2
Linearl
Cona3
Conv2
BN3
BNB
AxpPon
AvgPon
Conea
Comeat
(a) DNN to be trained and compressed.
(b) Accessory and chape-dependent vertices.
Inpon
Conel
BNI
Rellai
Inpor
Conel
BN1
Rellii
DN2
BN2
Linear22
Lineari
Conv3
Conv2
Outpe
Libear2
Linear1
Cona3
Conv2
BN3
BN3
AvyPoui
Conve
AvyPoo
DN4
(c) Connected components.
(d) Connected components dependency.
R
8
9
Ka
b
8
7
9i
R,
W.
(e) Ievor-nvvriiatt Groups.
Figure 2: Automated ZIG partition illustration. K and b, are the flatten filter matrix and bias vector of Convi, where the jth row of K represents the jth 3D filter. 7/ and B are the weighting and bias vectors of BNi. W. and bu; are the weighting matrix and bias vector for Lineari. The ground truth ZIGs G are present in Figure 2e. Since the output tensors of Conv2 and Conv3 added together, both layers associated with the subsequent BN2 and BN3 must remove the same number of filters from K2 and K3 and scalars from b2, b3, 72, Ya, .2 and Bg to keep the addition valid. Since BN4 normalizes the concatenated outputs along channel from Convl- BN1-ReLu and Conv3+Conv2--B BN2 BN3, the corresponding scalars in 74. B1 need to be removed simultaneously.
Form ZIGs. Finally, form ZIGs based the connected components of dependency as Figure 2d. The pairwise trainable parameters across all individual stem vertices in the same connected com- ponent need to be first grouped together as Figure 2e, wherein the parameters of the same color represent one group. Later on, the accessory vertices insert their trainable parameters if applicable into the groups of their dependent stem vertices accordingly. Some accessory vertex such as RN4 may depend on multiple groups because of the SID) joint vertex, thereby its trainable parameters Yt and 34 need to be partitioned and separately added into corresponding groups, e.g., 71. 3] and 37. In addition, the connected components that are adjacent to the output of DNN are excluded from forming ZIGs since the output shape should be fixed such as Linear2. For safety, the con- nected components that possess unknown vertices are excluded as well due to uncertainty, which further guarantees the generality of the framework applying onto DNNs with customized operators.
Complexity Analysis. The proposed automated ZIG partition Algorithm is series of customized graph algorithms dedicately composed together. In depth, every individual sub-algorithm is achieved by depth-first sersc- recursively traversing the trace graph of DNN and conducting step-spor ree-specific operations, which has time complexity as O(IV+ | and space complexity as O [V) in the worst case. The former one computed by discovering all neighbors of each vertex by traversing the adjacency list once in linear time. The latter one is because the trace graph of DNN is acyclic thereby the memory cache consumption up to the length of possible longest path for an acyclic graph as Vl. Therefore, automated ZIG partition can be efficiently completed within linear time.
3.2 DUAL HALF-SPACE PROJECTED GRADIENT (DHSPG)
Given the constructed ZIGs & by Algorithm 2, the next step is to jointly identify which groups are redundant to be removed and train the remaining groups to achieve high performance. To tackle it, we construct a structured sparsity optimization problem and solve via novel DHSPG. Compared with HSPG, DHSPG constitutes dual-half hall-haaf-spaee direction with automatically selected regulariza- tion coefficients to more reliably control the sparsity exploration, and enlarges the search space by partitioning the ZIGs into separate sets avoid trapping around the origin for better generalization.
5

---

Published as a conference paper at ICLR 2023
Target Problem. Structured sparsity inducing optimization problem is natural choice to seek group sparse solution with high performance, wherein the zero groups refer to the redundant struc- tures, and the non-zero groups exhibit the prediction power to maintain competitive performance to the full model. We formulate an optimization problem with a group sparsity constraint in the form of ZIGs 9 as (1) and propose novel Dual Half-Space Projected Gradient (DHSPG) to solve it.
minimize f(x), s.t. Card E G|[ 0} K,
#ER*
where is the target group sparsity level. Larger K indicates higher group sparsity in the solution and typically results in more aggressive FLOPs and parameter quantity reductions.
Related Optimizers and Limitations. To solve such constrained problem, ADMM converts it into min-max problem, but can not tackle the non-smooth and non-convex hard constraint of sparsity without hurting the objective. thus necessitates extra fine-tuning af- terwards (Lin et al., 2019). HSPG OTOv1 (Chen al., 2021b) and proximal methods (Xiao & Zhang, 2014) relax it into a non- constrained mixed f regularization problem, but can not guar- antee the sparsity constraint because of the implicit relationship be- tween the regularization coefficient and the sparsity level. In addi- tion, the augmented regularizer penalizes the magnitude of the tire trainable variables which restricts the search space to converge to the local optima nearby the origin point, e.g., of in Figure 3. However, the local optima with the highest generalization may lo- cate variably for different applications, and some may stay away from the origin point, e.g., 22 zg in Figure 3.
Iole
Pi Figure 3: Local optima x* E R² distribution over the ob- jective landscape.
Algorithm Outline for DHSPG. To resolve the drawbacks of the exist- ing optimization algorithms for solv- ing (1), we propose novel algo- rithm, named Dual Half-Space Pro- jected Gradient (DHSPG), stated as
Algorithm 3 Dual Half-Space Projected Gradient (DHSPG)
I: Input:
initial variable æn E IR", initial learning rate Ogp- warm-up steps Tw. half-space project steps Th, target group sparsity K and ZIGs G.
2: Warm up T,, steps via stochastic gradient descent.
Algorithm 3, with two takeaways.
3: Construct Gp and Gmp given G and K as (2).
4: for Te,, Tw + T, + 2...... do
Partition Groups. To avoid always trapping in the local optima near the origin point, we further partition the groups in G into two subsets: one has magnitudes of variables being pe- nalized Gp. and the other does not force penalize variable magnitude
5: Compute gradient estimate V/(x;) or its variant.
6: Update [#t+1]g.,p as [x. - a0 (Z:))l,.
7:
Select proper Ag for Gp.
8: Compute (ă,-1lg, via subgradient descent of
9:
if 2 Th, then
10:
Perform Half-Space projection over [ă1+2lØ,-
Gnp. Different criteria can be applied
11:
Update [#c+i]e, + [ar+1le,
here construct the above partition based on salience scores, e.g., cosine- similarity cos (9z) between the pro- jection direction [x],,, and the negative gradient or its estimation - -[Vf())) Higher cos- similarity over 9 E G indicates that projecting the group of variables y onto zeros is more likely to make progress to the optimality of ʃ (considering the descent direction from the perspective of optimiza- tion). The magnitude over [x), then needs to be penalized. Therefore, we compute Gp by picking up the ZIGs with top-K highest salience scores and Gmp as its complementary as (2). To compute more reliable scores, the partition is proceeded after performing Tap warm-up steps as line 2-3.
12:
Update 04++1
13:
Return the final iterate THHSPG
Gp (Top-K) arg max salience-score(g and Gmp {1, ... ,n) gEg
Update Variables. For the variables in Gup of which magnitudes are not penalized, we proceed vanilla stochastic gradient descent or its variants, such as Adam (Kingma & Ba, 2014), i.e., [#1+1]e.,, + [ ar (V(E))) For the groups of variables in G, to penalize magnitude, we seek to find out redundant groups as zero, but instead of directly projecting them onto zero as formulate a relaxed non-constrained subproblem as (3) to gradually reduce the magnitudes without deteriorating the objective and project groups onto zeros if the projection serves descent direction during the training process.
ADMM which casily destroys the progress the optimum,

---

Published as a conference paper at ICLR 2023
minimize (b(()),,)= := ((z)ee)
+
Σ \o ||12||1| DE9p
(3)
i where Ag is a group-specific regularization coefficient and needs to be dedicately chosen to guarantee the de- crease of both the variable magnitude for 9 as well the objective f. In particular, we compute a negative subgradient of v as the search direction [d(x)]g, == -[V[(,))s, - Lgeg, /g(=)g// max max(1[=)]gll2> ) with 7 as a safeguard constant. To ensure [d(a))l,, as descent direction for both I and || [d([]]] needs to fall into the intersection between the dual half-spaces with nor- mal directions as [V and - l[ for any g E O, as shown in Figure 4. In other words, [d(x))],, (- [=)], and [d((z)] [1, [- are greater than It further indi- cates that Ag locates in the interval (Amin.g Amax..) <
[d( 69
16
at
[[]](())))ll)) (- ( -l if cos (0g)<
Figure 4: Search direction in DHSPG.
(6,) otherwise can be an arbitrary positive constant. Such >g brings the decrease of both the objective and the variable magnitude. We then compute trial iterate [it+1lg, + [2, a, [x)),, via the subgradient descent of v as line The trial iterate fed into the Half-Space projector (Chen et al., b021b) which outperforms proximal operators to yield group sparsity more productively with- out hurting the objective as line 9-10. Remark here that OTÔvI utilizes global coefficient À for all groups, thus lacks sufficient capability to guarantee both aspects for each individual group in accordance.
Convergence and Complexity Analysis. DHSPG converges to the solution of (1) DuSPG in the manner of both theory and practice. In fact, the theoretical convergence relies on the the construction of dual half-space mechanisms which yield sufficient decrease for both objective f and variable magnitude, see Lemma 2 and Corollary in Appendix C. Together with the sparsity recovery of Half-Space projector (Chen et al., 2020b, Theorem 2), DHSPG effectively computes a solution with desired group sparsity. In addition, DHSPG consumes the same time complexity O(n) as other first-order methods, such SGD and Adam, since all operations can be finished within linear time.
3.3 AUTOMATED COMPRESSED MODEL CONSTRUCTION
In the end, given the solution DHSPG with both high performance and group sparsity, we now automatically construct compact model which is manual step with un- avoidable substantial engineering efforts in OTOv1. In general, we traverse all ver- tices with trainable parameters, then re- move the structures in accordance with ZIGs being zero, such as the dotted rows of K, R2, K3 and scalars of b2, 71, G1 illustrated in Figure 5. Next, we erase the redundant parameters that affiliate with the removed structures of their incoming stem vertices to keep the operations valid, e.g., the second and third channels in Is are removed though Os is not zero. The automated algorithm is promptly com- plete in linear time via performing two passes of depth-first-search and manipulating parameters to produce a more compact model M Based on the property of ZIGs, M" returns the same inference outputs as the full M parameterized as THSPG thus no further fine-tuning is necessary.
Conv1
Conv3
Concat
BNI
Couv2
R
6
>i
7
3
st
9e
95
Sa
Figure 5: Automated compressed model construction.
g= (91. 92. 95 and [=bhspolosugsUp = 0.
4 NUMERICAL EXPERIMENTS
We develop OTOv2 to train and compress DNNs into slimmer networks with significant inference speedup and storage saving without fine-tuning. The implementation details presented in Ap- pendix A. To demonstrate its effectiveness, we first verify the correctness of automated ZIG parti- tion and automated compact model construction by employing OTOv2 onto variety of DNNs with
7

---

Published as a conference paper at ICLR 2023
complicated structures (see the visualizations in Appendix D). Then, compare OTOv2 with other methods on the benchmark experiments to show its competitive (or even superior) performance. In addition, conduct ablation studies of DHSPG versus HSPG on the popular super-resolution task and Bert (Vaswani et al., 2017) on Squad (Rajpurkar et al., 2016) in Appendix B. Together with autonomy, freendisens and generality, OTOv2 arguably becomes the new state-of-the-art.
Sanity of Automated ZIG and Automated Compression. The foremost step is to validate the cor- rectness of the whole framework including both algorithm designs and infrastructure developments. We select five DNNs with complex topological structures, i.e., StackedUnets, DenseNet (Huang al., 2017), ConvNeXt (Liu et al., 2022) and CARN (Ahn et al., 2018) (see Appendix for de- tails), as well as DemoNet in Section 3.1, all of which are not easily to be compressed via the ex- isting non-automatic methods unless with sufficient domain knowledge and extensive handcrafting efforts. Remark here that StackedUnets consumes two input tensors, and is constructed by stack- ing two standard Unets (Ronneherger et al., 2015) with different downsamplers and aggregating the corresponding two outputs together. To intuitively illustrate the automated ZIG partition over these complicated structures, we provide the visualizations of the connected components of dependency in Appendix D. To quantitatively measure the performance of OTOv2, we further employ these model architectures onto a variety of benchmark datasets, e.g., Fashion-MNIS. (Xiao et al., 2017), SVNH (Netzer et al., 2011), CIFARTO/ 10/100 (Krizhevsky & Hinton, 2009) and ImageNet (Deng al., 2009). The main results are presented in Table 1.
Compared with the baselines trained by vanilla SGD, under the same amount of training cost, OTOv2 au- tomatically reaches not only the com-
Table I: OTOv2 on extensive DNNs and datasets.
Backend
Dxtaset
Methol
FLOP%
Aol Params
Top:
DemoNee
Pushion MINIST
Baseline
100%
$0.55%
DemoNet
Fushion MNIST
OIOv2
24.0%
23.3%
84.3%
Stackeddnnes
SVNH
Hoseline
100%
100%
96.8%
petitive performance but also remark- able speed up in terms of FLOPs and parameter quantity reductions. In particular, the slimmer DemoNet and StackedUnets computed by OTOv2 negligibly regress the top-1 accuracy by 0..2% but signif- icantly reduce the FLOPs and the number of parameters by 73.6%-83.00 Consistent phenomena also hold for DenseNet121 where the slimmer architecture is about times more efficient than the full models but with competitive accuracy. OTOv2 works with TIMM (Wightman, 2019) to effec- tively compress ConyNeXt-Ting which shows its flexibility to the modernized training tricks. The success of OTOv2 on these architectures well validates the sanity of the framework.
StackedUnets
SVNH
OTOv2
26.4%
17.0%
94.7%
DesseNet121
CIFARIOO
Roseline
100%
100%
77.0%
DesseNet121
CIFAR100
OTOv2
20.5%
26.7%
75.5%
Con-Ncii-Thy
ImageNet
Roseline
100%
100%
82.0%
CowNeei--Tiy
ImageNet
OTOv2
Benchmark Experiments. The secondary step is to demonstrate the effectiveness of OTOv2 by comparing the performance with other state-of-the-arts on benchmark compression experiments, i.e. common architectures such as VGG16 (Simonyan & Zisserman, 2014) and ResNet50 (He et al., 2016) as well as datasets CIFARIO (Krizhevsky & Hinton, 2009) and ImageNet (Deng al., 2009).
VGG16 on CIFAR10. We first consider vanilla VGG16 and a variant referred as VGG16-BN that appends batch normalization layer after every convolutional layer. OTOv2 automatically exploits the minimal removal structures of VGGI6 and partitions the trainable variables into ZIGs (see Fig- ure 14 in Appendix D). DHSPG is then triggered over the partitioned ZIGs to train the model from scratch to find solution with high group sparsity. Finally, slimmer VGG16 is automatically con- structed without any fine-tuning. As shown in Table 2, the slimmer VGG16 leverages only 2.5% of parameters to dramatically reduce the FLOPs by 86.6% with the competitive top-1 accuracy to the full model and other state-of-the-art methods. Likewise, OTOv2 compresses VGG16-BN to main- tain the baseline accuracy by the fewest 4.9% of parameters and 23.7% of FLOPs. Though SCP and RP reach higher accuracy, they require significantly 43%- 102% more FLOPs than that of OTOv2.
Table 3: ResNet50 for CIFAR10.
ResNet50 on CIFAR10. We now conduct experiments to compare with few representative automatic pruning methods such as AMC and ANNC. AMC establishes rein- forcement learning agent to guide a laser-wise compression, while only achieves autonomy over few prescribed specific models and requires
Method
FLOPs
# Params
Top-1 Acc.
Baseline
100%
100%
93.5%
AMC (He 2018h)
60.0%
93.6%
ANNC (Yang at,, 2020)
50.0%
95.0%
Prune (1 ym et al., 2019)
30.0%
93.1%
N2NSKip (Sharaa
2020)
10.0%
94.4%
OTOv1 (Chen et al., 2021b)
12.8%
8.8%
94.4%
OTOv2 (90% group sparsity)
2.2%
1.2%
93.0%
OTOv2 (80% group sparsity)
7.8%
4.1%
94.5%
multiple-stage training costs.
Simple pruning methods
8

---

Published as a conference paper at ICLR 2023
Table 2: VGGI6 and VGGI6-BN for CIFARIO. Convolutional layers are in bold.
Method
BN
Architecture
PLOPS
Params
Top Acc.
Baseline
100%
SBP (Ncklyudov et al, 2017)
31.1%
5.9%
91.0%
BC (1 juizoss et 2017)
38.5%
5.4%
91.0%
RBC
ct.,,, al, 2019)
3.9%
RBP (Zhou all,
28.6%
2.6%
91.0%
01Odl (Chen et ol. 2021b)
21-45-82-1100
16.3%
2.5%
91.0%
OTO,2 (855 group (parsity
2.5%
91.0%
Büscline
TOT/E
100%
95.27
BC al, 2016)
65.8%
37.0%
93.19
Hinge et 20200)
95.6%
SCP (Kang Han, 2020)
33.8%
7.0%
93.8%
OTOvl (Chen et al., 2021b)
12.56.92-123-182-1223 25..
26.8%
5,5%
93.3%
2022))
93.9%
CPGCN (D) Jang & Yang, 2022) OTO-2 (80% group sparsity)
26.99
6.9%
93.1%
1451-77-122-188116---22 92----1116-9----------- 101-1-1--------888-11----- 107-183
23.7%
4,9%
93.2%
such as ANNC and SFW-pruning (Miao ct al., 2021) do not construct slimmer models besides merely projecting variables onto zero. OTOv2 overcomes all these drawbacks and is the first to realize the end-to-end autonomy for simultancously training and compressing arbitrary DNNs with high performance. Furthermore, OTOv2 achieves the offe--of-theeatt results this intersecting ResNet50 on CIFARIO experiment. In particular, as shown in Table 3, under 90% group sparsity level, OTOv2 utilizes only 1.2% parameters and 2.2% FLOPs reach 93.0% top-1 accuracy with slight 0.5% regression. Under 80% group sparsity, OTOv2 achieves competitive 94.5% accuracy to other pruning methods but makes use of substantially fewer parameters and FLOPs.
ResNet50 on ImageNet. We finally employ OTOv2 to ResNet50 on ImageNet. Similarly to other experiments, OTOv2 first automatically partitions the trainable variables of ResNet50 into ZIGs (see Figure 11 in Appendix D), and then trains it once by DHSPG to automatically
78
Group-188 (2013))
Reellep (2021)
76
CBN--0 (2019)
40% group sparsity
RP (3022)
SUP (2018
50% group spanity
construct slimmer models without fine-tuning. We report a performance portfolio under vari- ous target group sparsities ranging from 40% to 70% and compare with other state-of-the- art methods in Figure 6. Remark here that more reliably controlling the ultimate sparsity level meet various deployment environments is a significant superiority of DHSPG to the
Top-1 Accuracy
Hinge (2020)
GNN-RL (2023)
74
SCP (2020)
OTOv2
CP (2017)
RREP (2019)
60% group sparsity
DDS 25 (2018)
REP (2019)
ThinNet (2017)
HSPG. An increasing target group sparsity re- sults in more FLOPs and parameter quantity re- ductions, meanwhile sacrifices more accuracy. It is noticeable that OTOv2 roughly exhibits Pareto frontier in terms of top-1 accuracy and FLOPs reduction under various group sparsities. In particular, under 70% group sparsity, the slim- mer ResNet50 by OTOv2 achieves fewer FLOPs (14.5%) than others with a 70.3% top-1 accuracy which is competitive to SFP (He et al., 2018a) and RBP (Zhou et al., 2019) especially under 3x fewer FLOPs. The one with 72.3% top-1 accuracy under group sparsity as 60% is competitive to CP (He et al., 2017), DDS-26 (Huang & Wang, 2018) and RRBP (Zhou al., 2019), but 2-3 times more efficient. The slimmer ResNet50s under 40% and 50% group sparsity achieve the accuracy milestone, i.e. around 75%. both of which FLOPs reductions outperform most of oftte-of-the-ard ResRep (Ding et al., 2021), Group-HS (Yang et al., 2019) and GBN-60 (You et al., 2019) achieve over 76% accuracy but consume more FLOPs than OTOv2 and are not automated for general DNNs.
70
70% group suanity
30
50
90
FLOPs Reduction (%)
Figure 6: ResNer50 on ImageNet.
5 CONCLUSION
We propose OTOv2 that automatically trains general DNN only once and compresses it into more compact counterpart without pre-training or fine-tuning to significantly reduce its FLOPs and parameter quantity. The success stems from two major improvements upon OTOv1: 1: (i) automated ZIG partition and automated compressed model construction; and (ii) DHSPG method to more reliably solve structured-sparsity problem. We leave the incorporation with NAS future work.

---

Published as a conference paper at ICLR 2023
REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. Proceedings the IEEE conference on computer vision and pattern recog- nition workshops, 2017.
Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. Fast, accurate and lightweight super- resolution with cascading residual network. In Proceedings of the European conference com- puter vision (ECCV), pp. 352-268, 2018.
Arseny. Onnx2torch: an onnx to pytorch converter. onnx2torch, 2022.
Junjie Bai, Fang Lu, Ke Zhang, et al. Onnx: Open neural network exchange. bettps:/ggithh com/onnx/ onnx, 2019.
Jianda Chen, Shangyu Chen, and Sinno Jialin Pan. Storage efficient and dynamic flexible runtime channel pruning via deep reinforcement learning. 2019.
Tianyi Chen, Frank E Curtis, and Daniel P Robinson. reduced-space algorithm for minimizing ( l--regularizdd convex functions. SIAM Journal on Optimizatiion, 27(3):1583-1610, 2017.
Tianyi Chen, Frank E Curtis, and Daniel P Robinson. Farsa for f1-regularized convex optimization: local convergence and numerical experience. Optimization Methods and Software, 2018 8.
Tianyi Chen, Bo Ji, Yixin Shi, Tianyu Ding, Biyi Fang, Sheng Yi. and Xiao Tu. Neural network compression via sparse optimization. arXiv preprint arXiv:2011.04866 2020a.
Tianyi Chen, Guanyi Wang, Tianyu Ding, Bo Ji, Sheng Yi, and Zhihui Zhu. A half-space stochastic projected gradient method for group-sparsity regularization. arXiv preprint arXiv:2009. /2078, 2020b.
Tianyi Chen, Tianyu Ding, Bo Ji, Guanyi Wang, Yixin Shi, Jing Tian, Sheng Yi, Xiao Tu, and Zhihui Zhu. Orthant based proximal stochastic gradient method for _1 1-regularized optimization. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML 2020, Ghent, Belgium, September 14-18, 2020, Proceedings, Part III, pp. 57-73. Springer, 2021a.
Tianyi Chen, Bo Ji, Tianyu Ding, Biyi Fang, Guanyi Wang, Zhihui Zhu, Luming Liang, Yixin Shi, Sheng Yi, and Xiao Tu. Only train once: A one-shot neural network training and pruning framework. In Advances in Neural Information Processing Systems, 2021b.
Tristan Deleu and Yoshua Bengio. Structured sparsity inducing adaptive optimizers for deep learn- ing. arXiv preprint 102.0360,,,,,69, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scal scale hi- crarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icce, 2009.
Yuan Cao Di Jiang and Qiang Yang. On the channel pruning using graph convolution network for convolutional neural network acceleration. 2022.
Xiaohan Ding, Tianxiang Hao, Jianchao Tan, Ji Liu, Jungong Han, Yuchen Guo, and Guiguang Ding. Lossless cnn channel pruning via decoupling remembering and forgetting. Proceedings of the IEEE International Conference on Computer Vision, 2021.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient obbti-obeective neural architecture search via lamarekian evolution. arXiv preprint arXiv:1804.00 090808, 2018.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.088
2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv: /9033016/1 2019.
10

---

Published as a conference paper at ICLR 2023
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574. 2019.
Shang-Hua Gao, Yong-Qiang Tan, Ming-Ming Cheng, Chengze Lu, Yunpeng Chen, and Shuicheng Yan. Highly efficient salient object detection with 100k parameters. In European Conference Computer Vision, pp. 702-721. Springer, 2020.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 15 15.000149, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv: 1808. 06866, 2018a.
Yihui Hc, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Yihui He, Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration mobile devices. In Proceedings of the European Conference Computer Vision (ECCV), pp. 784-800, 2018b.
Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4705 2017.
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from trans- formed self-exemplars. In Proceedings of the IEEE conference computer vision and pattern recognition, 2015.
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pp. 304-320, 2018.
Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. In International Conference on Machine Learning, pp. 5122-51221 5131. PMLR, 2020.
James Max Kanter and Kalyan Veeramachaneni. Deep feature synthesis: Towards automating data science endeavors. In 2015 IEEE international conference data science and advanced analyt- ics (DSAA), pp. 1-10. IEEE, 2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14226614/ 2014.
Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast bayesian op- timization of machine learning hyperparameters on large datasets. In Artificial intelligence and statistics, pp. 528-536- PMLR, 2017.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Bailin Li, Bowen Wu, Jiang Su, and Guangrun Wang. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. In European Conference Computer Vision, pp. 639 654. Springer, 2020a.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 1608 08 2016.
Yawei Li, Shuhang Gu. Christoph Mayer, Luc Van Gool, and Radu Timofte. Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 8018-8027, 2020b.
11

---

Published as a conference paper at ICLR 2023
Yawei Li, Kamil Adamczewski, Wen Li, Shuhang Gu. Radu Timofte, and Luc Van Gool. Revisit- ing random channel pruning for neural network compression. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 191-201, 2022.
Yuchao Li, Shaohui Lin, Baochang Zhang, Jianzhuang Liu, David Doermann, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Exploiting kernel sparsity and entropy for interpretable enn compres- sion. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp- 2800-2809, 2019.
Shaohui Lin, Rongrong Ji, Yuchao Li, Cheng Deng, and Xuelong Li. Toward compact convnets via structure-sparsity regularized filter pruning. IEEE transactions on neural networks and learning systems, 31(2):774-588, 2019.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 1 11966-11999 1196,, 2022.
Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learning. In Advances in neural information processing systems, pp. 3288-3298, 2017.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5066, 2017.
Sangkug Lym, Esha Choukse, Siavash Zangeneh, Wei Wen, Sujay Sanghavi, and Mattan Erez. Prunetrain: fast neural network training by dynamic sparse model reconfiguration. In Proceed- ings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-13, 2019.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEFE International Conference on Computer Vision. ICCV 2001, volume 2. pp. 416 423. IEEE, 2001.
Fanxu Meng, Hao Cheng, Ke Li, Huixiang Luo, Xiaowei Guo, Guangming Lu, and Xing Sun. Pruning filter in filter. arXiv preprint arXiv:2009. 14410, 2020.
Lu Miao, Xiaolong Luo, Tianlong Chen, Wuyang Chen, Dong Liu, and Zhangyang Wang. Learning mouiing-friendly networks via frank-wolfe: One-shot, any-sparsity, and no retraining. In Interna- fional Conference on Learning Representations, 2021.
Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Structured bayesian pruning via lormalrmal multiplicative noise. In Advances in Neural Information Processing Sys- tems, pp. 6775 6784, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2.
Junghun Oh, Heewon Kim, Seungjun Nah, Cheeun Hong, Jonghyun Choi, and Kyoung Mu Lee. Attentive fine-gre ain--grineed structured sparsity for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17673-17682, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32. 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy I Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:16000 1606.02200 2016.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint 20Xx:2003300288, 2020.
12

---

Published as a conference paper at ICLR 2023
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-2411 Springer. 2015.
Avinash Sharma, Arvind Subramaniam, and N2nskip: Learning highly sparse networks using neuron-to-neuron skip connections. In BMVC, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409. 1556, 2014.
Mart van Baalen, Christos Louizos, Markus Nagel, Rana Ali Amjad, Ying Wang, Tijmen Blankevoort, and Max Welling. Bayesian bits: Unifying quantization and pruning. arXiv preprint arXiv:2005.07093, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai L.. Learning structured sparsity in deep neural networks. arXiv preprint arXiv:1608.03666 03160008.66,,, 2016.
Ross Wightman. Pytorch image models.
pytorch- ietrh-iimme-mmdee s, 2019.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms, 2017.
Lin Xiao and Tong Zhang. A proximal stochastic gradient method with progressive variance reduc-
tion. SIAM Journal on Optimization, 24(4):2057-2075, 2014.
Haichuan Yang. Shupeng Gui, Yuhao Zhu, and Ji Liu. Automatic neural network compression by
sparsity- quantization joint learning: A constrained optimization- based approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1178-2188 2020.
Huanrui Yang, Wei Wen, and Hai Li. Deephoyer: Learning sparser neural network with differen- tiable scale-invariant sparsity measures. arXiv preprint vrXivv1908. 190X::9990000999, 2019.
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma. and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. arXiv preprint 179.v:1909.08174. 2019.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In International conference on curves and surfaces. Springer, 2010.
Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. A systematic dnn weight pruning framework using alternating direction method of multipliers. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 184-199, 2018.
Yuefu Zhou, Ya Zhang, Yanfeng Wang, and Qi Tian. Accelerate enn via recursive bayesian pruning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3306-3315, 2019.
Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuron-level structured pruning using polarization regularizer. Advances in Neural Information Processing Systems, 33, 2020.
13

---

Published as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
A.1 LIBRARY IMPLEMENTATION
The implementation of the current version of OTOv2 (February, 2023) depends on Pytorch (Paszke et al., 2019) and ONNX (Bai et al., 2019) which is open industrial standard for machine learning interoperability and widely used in numerous Al products of the majority of top-tier industries (see the partners in ht hoss://onne //onnx. a /). In particular, the operators and the connectivity of general DNN are retrieved by calling the ONNX optimization API of Pytorch, which is the first step to establish the trace graph of OTOv2 in Algorithm 2. The proposed DHSPG is implemented as an instance of the optimizer class for Pytorch. The ultimate compact model construction in Section 3.3 implemented by modifying the attributes and parameters of the vertices onnx models according to DDHSPG and ZIGs. As result, OTOv2 realizes an end-to-end pipeline to automatically and conveniently produce compact model that meets inference restrictions and can be directly deployed onto product environments. In addition, the constructed compact DNNs in onnx format can be converted back into either torch or tensorflow formats if needed by open-source tools (Arseny, 2022).
A.2 LIMITATIONS OF BETA VERSION
Dependency. The current version of OTOv2 (February, 2023) depends on the ONNX optimization API in Pytorch to obtain vertices (operations) and the connections among them, i.e., (E, V) in line 2 in Algorithm 2. It is the foremost step for establishing the trace graph of DNN for automated ZIG partition. Therefore, the DNNs that do not comply with this API are not supported by the beta version of OTOv2 yet. We notice that Transformers sometimes have incompatibility issue for its position embedding layers, whereas its trainable part such as encoder layers does not. The current limitation is in the view of engineering perspective and would be resolved following the active and rapid developments of the ONNX and PyTorch community driven by the industry and academy.
Unknown Operators. For sanity, we exclude the connected components that possess uncer- tain/unknown vertices for forming ZIGs in Algorithm This mechanism largely ensures the gen- erality of the automated ZIG partition onto general DNNs. But the ignorance over these connected components may miss some valid ZIGs thereby may leave redundant structures be unpruned. We will maintain and update the operator list which currently consists of 31 (known/certain) operators to better exploit ZIGs.
A.3 EXPERIMENTAL DETAILS
We conducted the experiments on one NVIDIA A100 GPU Server. For the experiments in the main body, we estimated the gradient via sampling mini-batch of data points under first-order momen- tum with coefficient as 0.9. The mini-batch sizes follow other related works from {64, 128, 256). All experiments in the main body share the same commonly-use from 10 and periodically decay by 10 till 10 4 every pperiod epochs. The length of decaying period T periiod depends the maximum epoch, i.e., 120 for ImageNet and 300 for others.
learning rate scheduler that start
In general, we follow the usage of Half-Space projector in (Chen et al., 2021b) to trigger it when the learning rate is first decayed after pprriod epochs. Gp and Gup are constructed after Tperiad 2 warm up epochs empirically in our experiments. To compute the saliency score, we jointly consider both cosine-similarity and magnitude of each group g E G. For the groups g E Gp which magnitudes need set Xy in Algorithm
to be penalized, not need to be adjusted, i.e., cos (0,) 2 O. Note that A = 10-3 is the commonly used coefficient in the sparsity regularization literatures (Chen et al., 2021b; Xiao & Zhang, 2014). Otherwise, computed the Amin.g := (8,) (x) llz and Amax.o := [[ cos (0,) and set Ag by amplifying Amin.g by 1.1 and projecting back to Amax,g if exceeding.
X A 10 if the regularization coefficient does
B EXTENSIVE ABLATION STUDY
In this appendix, we present additional experiments to demonstrate the superiority of DHSPG over finding local optima with higher performance than HSPG. As described in the main body, the main
14

---

Published as a conference paper at ICLR 2023
advantages of DHSPG compared with HSPG in OTOv1 are (i) enlarging search space to be capable of finding local optima with higher performance any, and (ii) more reliably guarantee ultimate group sparsity level. The later one has been demonstrated by the experiments of ResNet50, where DHSPG can precisely achieve different prescrihed group sparsity level to meet the requirements of various deploying environments. In contrast, HSPG lacks capacity achieve specific group sparsity level due to the i relationship between the regularization coefficient À and the sparsity level. The former one will be validated in this appendix. In depth, one takeaway of DHSPG is to separate groups of variables, then treat them via different and specifically designed mechanisms which greatly enlarge the search space. However, HSPG applies the same mechanism update all variables, which may easily result in convergence to the origin and may be not optimal. In addition, we also provide the runtime comparison in Appendix B.3.
B.1 SUPER RESOLUTION
We select the popular model architecture CARN (Ahn et al., 2018) for super-resolution task with the scaling factor of two. As (Oh et al., 2022), we use benchmark DIV2K dataset (Agustsson & Timofte, 2017) for training and Set14 (Zeyde et al., 2010). B100 (Martin et al., 2001) and Ur- ban100 (Huang et al., 2015) datasets for evaluation. Similarly to other experiments presented in the main body, OTOv2 automatically partitions the trainable variables of CARN into ZIGs (see Figure in Appendix D). Then we follow the training procedure in (Agustsson & Timofte, 2017) to apply the Adam strategy into DHSPG, i.e. utilizing both first and second order momentums to compute gradient estimate line 5 in Algorithm 3. Under the same learning scheduler and total number of steps as the baseline, we conduct both DHSPG and HSPG to compute solutions with high group sparsity, where set target group sparsity as 50% for DHSPG and fine-tune the regularization co- efficient for HSPG as the power of 10 from 10 3 to 10 to pick up the one with significant FLOPs and parameters reductions with satisfactory performance. Finally, the compact CARN models are constructed via the automated compressed model construction Section 3.3. We report the final results in Table 4.
Table 4: OTOv2 under DHSPG versus HSPG on CARNx2.
PSNR
Method
Optimizer
FLOPS
of Params
Set14
B100
Urban100
Baseline
Adam
100%
100%
33.5
32.1
31.5
OTOv2
HSPG
35.5%
35.4%
33.0
31.6
30.9
OTOv2
DHSPG
24.3%
24.1%
33.2
31.9
31.1
Unlike the classification experiments where HSPG and DHSPG perform quite competitively, OTOv2 with DHSPG significantly outperforms OTOv2 with HSPG on this super-resolution task via CARN by using 46% fewer FLOPs and parameters to achieve significantly better PSNR on these benchmark datasets. exhibits a strong evidence to show the higher generality of DHSPG to enlarge the search space rather than restrict it near the origin point to fit more general applications.
B.2 BERT ON SQUAD
We next compare DHSPG versus HSPG on pruning the large-scale transformer Bert (Vaswani et al., 2017), evaluated on Squad, a question-answering benchmark (Rajpurkar et al., 2016). Remark here that since Transformer are not reliably compatible with PyTorch's 's ONNX optimization API at this moment, they can not enjoy the end-to-end autonomy OTOv2 yet. To compare two optimizers, we apply DHSPG onto the OTOv1 framework, which manually conducts ZIG partition and constructs compressed Bert without fine-tuning. The results are reported in Table 5.
Based on Table 5, it is apparent that DHSPG performs significantly better than HSPG and Prox- SSI (Deleu & Bengio, 2021) by achieving 83.8%-87.77 7% Fl-scores and 74.6%-80.0% exact match rates. In constrast, HSPG and ProxSSI reach 82.0%%84. 84.18..1 $44.1% F1-scores and 71.9%-75. 0% exact match rates. The underlying reason of such remarkable improvement by DHSPG is that DHSPG enlarges to search space away from trapping around origin points by partitioning groups into magnitude- penalized or not and treating them separately. However, both ProxSSI and HSPG penalize the mag- nitude of all variables and apply the same update mechanism onto all variables, which deteriorate
15

---

Published as a conference paper at ICLR 2023
the performance significantly in this experiment. The results well validate the effectiveness of the design of DHSPG to enlarge search space for typical better generalization performance.
Table 5: Numerical results of Bert on Squad.
Methodi
Params
Exact
Inference SpoecUUp
Dascline
100%
81.0%
55.3%
1x
OTOOE DUISPG
ProxSSI (Deleu Rengin, 2021)
72.3%
OTOv1 + DIISG (10% group spursity)
93.3%
80.0%
87.7%
1.1x
ONTON HISPO (M0))
010v1 DHSPG (50% group sparsity)
80.1%
79.4%
87.3%
1.2x
OTOV1 + DHSPG (50% group sparsily)
68.3%
78.1%
86.2%
1.3x
Praditi (pla))
010v1 DHSPG (70% group sparsity)
55.0%
74.6%
83, 8%
1.4X
HSPG
OTOv1 HSPG
66.7%
71.9%
82.0%
1.3x
Approsimati value based on the group sparsit reported (Delou & Bengio, 2021).
Roductio (55)
B.3 RUNTIME COMPARISON
We provide runtime comparison of the proposed DHSPG versus the optimization algorithms used in the benchmark baseline experiments. In particular, we calculate the average runtime per epoch and report the relative runtime in Figure 7. Based on Figure 7, we observe that the per epoch cost of DHSPG is competitive to other standard optimizers. In addition, OTOv2 only trains the DNN once with the similar amount of total epochs for training the baselines. However, other compression methods have proceed multi-stage training procedures including pretraining the baselines by the standard optimizers, thereby are not training efficient compared to OTOv2.
1.5
DHSPG
Adam
SGD
Adam W
1.0
0.5
CIFARIO VGG16
ImageNet ResNet50
DIV2K CARNx2
Squad Berl
Figure 7: Average runtime per epoch relative comparison.
C CONVERGENCE ANALYSIS
In this appendix, we provide the rough convergence analysis for the proposed DHSPG. This paper is application- track and mainly focuses on deep learning compression and infrastructure but not the theoretical convergence. Therefore, for simplicity, assume full gradient estimate each iteration. More rigorous analysis under stochastic settings will left as future work,
Lemma 1. The objective function! f satisfies
f(x+ad(x)) f(x)) f(z))((()))))))))))))))))))))f (a- La²
(a-0aaa-
La²
|V](2))]
Σ X²+ -La 1)a Σ Xg cos (0,) (V)(()))) nEG,
9EG,
(4)
Proof. By the algorithm,
- [V(())) [2)
[x] Ag
if E 9p. otherwise.
d(x)=
|lbll
(5)
16

---

Published as a conference paper at ICLR 2023
We can rewrite the direction [d(x)] for Gp as the summation of two parts,
[d(x)], - dd(x) + d((e)
(6)
where
[d(x)]
[V(()),, o, and [((c)) - -111>
[
(0g -90))
Ag sin (0g).
(7)
||
Consequently,
2 [(()))
[d(a))]
2
(d(xr)
[+]
[>))l>
23 sin" (8gy
|<>]||]
(8)
[x=]] ||| |
+ 21v X
sis" (0g)
||و|||||
[I[ 11 + 23 cos (0g) 2Xg [[] cos (8g)
Ag cos (0,)]²
and
[d(z)lg [[V Ag cos (fg) [f(())] := - (9)
[x)
By the descent lemma, we have that f(x + ad(x))
La
<f(x) + (V) T(xx)) +
2 |ld(x))1
(V/(x)) 9mp (d(x)]e.., + [=] + La² [[((x) S..p 11 +
La²
=f(x)
[[d(xr)]s, 12
2
La²
La
=f(x)
Vf(x))l 1 *+a(Vf(=)1g [d(x)) g, +
|[d(x)) l1
2
=f(x)
La²
|V][ 1 +a|V/(x)) (d(x) d(x))
La
(d(x) + ddxx)
2
6p
2
5,
La²
Laa
=f(x)-
a- 2
[f(x)]oo.. 1² *aalV(x))
[d(x))
+
[d(x))
9,
2
90 12
+ La
(d(x))
(d(x)]
+
La²
[d(z))
2
Op
9,
2
1
(a-
Taz
[
+ Ag cos (0g)
=f(x))
[
2
Vf
[V]()))]
2
[V](2)],
DEG.
La
+ La
+
Σ
X2 sin? (0g)
(I]V 11 + Ag cos (0,))²
2
2
DE8p
DEG,
La²)
[f(e)))e 12 ---(0 a (e-La)) ||[ 12 12-0 a Σ Ag cos (0,) (=V]l gEG,
=f(x))
a
2
La²
Σ (sin2 + cos2 (0,)) I.a2 Σ X [|[[(())]]|| cos (0,)
+
2
gEgp
gEgp
La?)
La Σ
=f(x)
a 2
[Vf(2))1
22 (La 1)a Σ Ag cos (0,) [[]][
9EG,
9EG
17

---

Published as a conference paper at ICLR 2023
Lemma 2. Suppose 2. and f is L-smooth, then there exists some positive Xg E (Amin.g Amax.g) for any g E Gp such as
f(x + ad(x)) & f(x) (a- (a-La²)
La [f(())len
1
(10)
2
Proof. Based on Lemma and a S 1 I. we have that f(x + ad((:))
La²
Laa
Sf(x:)
(a- 2
[f(x))]
Σ
X² + (La - 1)a Σ Xo cos (0₂) [[V
2
HEGr
8E9
(a La²
Sf(x)
VT
+
Σ
9)
2
Sap
2
gEpp
(11)
where we denote
La
La²
h( h()g..)
(La 1)a cos (0₂) [IV/ Ag- d
[[T/()))
(12)
2
2
We can see that for any g E Gp. then 1((,,) g) < 0 if and only if the following holds
(1 - La)a (0,) [I + V(L La) cos² (0₂) [I[ + 2La² (a - وو2 [V/(2)]
Ag S
La
(13)
Next, we need show for the group y that requires X, adjustment, the (Amin.g: g: Zg) is valid interval, i... 2g 2 Amin.g To show it, combining with Amin.g = cos (0,) [[V we have that
(La - 1)admin.s + / La) min + 2La (a- Lod
/min.// cos (0,)
2
La²
(La- - 1)Amin.g + Amin.g VI L²a² tan" (0,) + 2La tan (0,)
La
(La 1)Amin.g + Amin.a V- (La 1)22 tan" (0,) + tan (0,) +
(14)
La
(La 1)>min Amin.g
La
Amin.g
where the second last inequality holds since < 22a$1/1 1/L, then
V-(La 1)2 tan? (0,) + tan" (0,)
Then, we need to show that ên is greater than 0. There are two cases to be considered:
cos 0g < O: then 2g 2 Amin.g - cos (0,) [V] > 0.
cos fg 2 O: follows < a < 1/L and (13) that Ag > 0.
Thus for X, E (Amin.s: min(Amnx..p) )gt), h(Ag Ag E (Amin.gs Amux.g) so that h(>g.9) S 0. Finally, the proof is complete we choose Ag satisfies the above for any E Gp-
((Ag.g) g) < 0. Consequently, there exists some positive
fxx + ad(x)) < f(x)
(a- La²
)
||18((0))
+ Σ h(),9)
2
DEG,
(15)
(a La²
||[[())))
S f(x)
2
18

---

Published as a conference paper at ICLR 2023
Lemma 3. For any E Gp. if0 < a 2/ [d(=)] [ 11
then the magnitude of the variables satisfies
1| + ad a)) (l)aalll 1all2 |1
(16)
And i [1]] [d(w]] [[d(1)) for E (0, 1), then
I| + ad(x))l || 2 + (w=2 2w) ||| cos² (0g).
(17)
Proof.
|| ad(x))ll
[2
[xl1 [V](2)) + 23
|||ا||||
(18)
||>>||| 12 2a(x=] (V](x)) Ag
[00 g
+ a
|
+>g
2
||اا|م|||
= ||0]91 112 + t(a),
where
[ + [f(x)]
[ g
2
t(a) 2a(x1} [V](2)] + X
X
Aaa -2a (19)
||اا|||||
and
[a)]
12
A [[(2))]
+ A/
(20)
|| [
12
B [-|] [V)(z)) + Ag
[x]
>
(21)
[[=]]
l1
note that B > 0 because of the selection of An-
[[ [d(=]a then ((a) < 0. [[(
Consequently, have that if 0 <
Finally, if a = w [m] [d(=))
w B
for w E (0, 2), then
11
B
B
B²
((x) = 2
BB
(w2 2w)
(w=
2w) || cos²
cos² (0,), (22)
A2
A
A
which completes the proof.
[20]] [d(=))
Corollary 1. Suppose a = wing [(d(=)), for some w E (0,1) and cos 2 for
g E Gp n g and some positive (0, 13. Then there exists E (0, 1) such that
ad(x))g, 1 (1 (1->²) ||و|,
(23)
Proof. The result can be complete via summing (17) over G, and combining with selection.
D ZIG ILLUSTRATION
this appendix, for more intuitive illustration, we provide the visualizations of the connected com- ponents of dependency for the experimented DNNs throughout the whole paper. They are con- structed by performing Algorithm 2 to automatically partition ZIGs. Due to the large-scale and in- recommend zoom in for reading greater details (under 900%-16009 zoom-
tricacy of graphs, in scale via Adobe PDF reader). The vertices marked as the same color represent one connected component of dependency. See the figures starting from the next pages.
19

---

Published as a conference paper at ICLR 2023
Figure 8: CARN.
20

---

Published as a conference paper at ICLR 2023
IEE
Figure 9; DemoNet.
21

---

Published as a conference paper at ICLR 2023
Figure 10: DenseNet121 121.
22

---

Published as a conference paper at ICLR 2023
Figure 11: ResNet50.
23

---

Published as a conference paper at ICLR 2023
Figure 12: StackedUnets.
24

---

Published as a conference paper at ICLR 2023
Figure 13: ConyNeXt-Tiin -inyne
25

---

Published as a conference paper at ICLR 2023
(a) VGG16.
(b) VGG16-BN.
Figure 14: VGG16 and VGG16-BN.
26


Reviews:
[{"review_id": "CKtjEGUp1c", "reviewer": "Reviewer_eYjt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents OTOv2, an improved version of Only-Train-Once (OTOv1) framework. It introduces two major improvements over OTOv1, including automated zero-invariant groups (ZIGs) partition and a new optimizer called Dual Half-Space Projected Gradient\n(DHSPG). Experiments are conducted on several small image classification datasets and ImageNet. ", "summary_of_the_review": "comments above. ", "strength_and_weaknesses": "Strength:\n1. Compared with the multi-stage model compression procedures, the proposed framework is simple and easy to use. \n2. OTOv2 shows better performances than previous methods on several small image classification datasets and ImageNet.  \n\nWeakness:\n1. Lack of ablation study experiments. It is unclear to me how many improvements the proposed DHSPG bring. Adding ablation study experiments on ImageNet with different model architectures and target sparsities will help resolve this concern. \n2. It is unclear whether the proposed method is effective on recent state-of-the-art models. For example, [1] shows that ResNet50 can achieve 80+ ImageNet top1 accuracy with an improved training setting. Adding additional results on these recent state-of-the-art models will help resolve this concern.  \n\n[1] ResNet strikes back: An improved training procedure in timm", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6K5vaLooaPk", "reviewer": "Reviewer_iQEU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a set of methods for inducing group sparsity in DNN architecture. Their contributions build on an existing system Only-Train-Once (OTO) by automating the identification of groups that must be pruned together (referred to as zero-invariant groups) and improving on the optimization method, half-space projected gradient descent (HSPG),  by splitting the parameter groups into penalized and non-penalized sets and tuning per-group regularization coefficients.", "summary_of_the_review": "Overall, I would like to see the authors improve the clarity of the text and the comparisons between their technique and prior work. ", "strength_and_weaknesses": "Strengths\n- The authors empirical evaluation considers many different tasks and many previously published works.\n- The automated detection of constraints that must be respected in order to maintain a valid DNN architecture after group pruning is impactful for lower the cost applying neuron-pruning methods to DNNs.\n\nWeaknesses\n- I found it hard to assess the results of experiments in section 4. For each experiment the authors include many prior results in tables. Each technique offers a different tradeoff in terms of accuracy lost to achieve a given level of compression. I believe it would be much easier to understand the results of these two-dimensional comparisons if they were shown in a scatter plot. If OTOv2 is more effective, I would expect it to represent the Pareto frontier of the solutions across a range of compression levels. I am not sure that the reported data is sufficient to demonstrate this. For example, the results in Table 4 show OTOv2 achieving a maximum top-1 accuracy of 75.4%, which is as much as 1% off the highest accuracy reported in the table. I think the experiments could be made to be more effective if the authors focused on thorough analysis of one or two benchmarks rather than incomplete analysis of many.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2LFRkg1nM4", "reviewer": "Reviewer_F5Jh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed OTOv2, which is a follow-up work of OTOv1. OTOv2 makes structured pruning more automatic, generic and user-friendly by addressing several problems in OTOv1. Specifically, they proposed an algorithm to automatically find Zero-Invariant Group for arbitrary DNNs and an optimizater to address the sparse optimization problem. Their empirical results is on par with other sota pruning methods, but requires less human efforts.", "summary_of_the_review": "In conclusion, this paper designs a model pruning system OTOv2. OTOv2 can do ZIG partition automatically for arbitrary neural architectures, optimize the sparse model using a novel optimization algorithm DHSPG and produce the pruned model without finetuning. In my opinion, this work makes clear contribution to the model compression community.", "strength_and_weaknesses": "Strength:\n- This paper built a model compression system to minimize the human effort put into weight pruning.\n- The technical details are well-explained.\n- The proposed DHSPG algorithm can satisfy different sparsity requirement without finetuning.\n- The empirical results are better or on par with sota pruning works.\n\nWeakness:\n- The experiments are still limited to popular standard networks including VGG/ResNet/DenseNet, which have been already very well-studied and known to have large redundancy. To support the claim that the proposed system can work on arbitrary DNN, it would be interesting to see how it does on modern architectures like RegNet/Swin/ConvNeXt.\n- It would be great to show some training speed benchmarks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster