Adaptive Depth Networks with Skippable Sub-Paths
Authors: Woochul Kang
Hyungseop Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: adaptive networks
training efficiency
inference efficiency
efficiency
acceleration
inference acceleration
convolutional neural networks
CNN
Vision transformer
vit
transformer
Primary area: deep_learning_architectures
TL;DR: We introduce a novel training method for adaptive depth networks that can provide flexible accuracy-efficiency trade- offs in a single network.

Abstract: Predictable adaptation of network depths can be an effective way to control inference latency and meet the resource condition of various devices. However, previous adaptive depth networks do not provide general principles and a formal explanation on why and which layers can be skipped, and, hence, their approaches are hard to be generalized and require long and complex training steps. In this paper, we present a practical approach to adaptive depth networks that is applicable to various networks with minimal training effort. In our approach, every hierarchical residual stage is divided into two sub-paths, and they are trained to acquire different properties through a simple self-distillation strategy. While the first sub-path is essential for hierarchical feature learning, the second one is trained to refine the learned features and minimize performance degradation if it is skipped. Unlike prior adaptive networks, our approach does not train every target sub-network in an iterative manner. At test time, however, we can connect these sub-paths in a combinatorial manner to select sub-networks of various accuracy-efficiency trade-offs from a single network. We provide a formal rationale for why the proposed training method can reduce overall prediction errors while minimizing the impact of skipping sub-paths. We demonstrate the generality and effectiveness of our approach with convolutional neural networks and transformers.

Adaptive Depth Networks with Skippable Sub-Paths
Woochul Kang" Department Embedded Systems Incheon National University Yeonsu-gu, Incheon, South Korea, 22012 wchkangeinu, kr
Hyungseop Lee Department of Embedded Systems Incheon National University Yeonsu-gu, Incheon, South Korea, 22012 hhss0927@ing ac.kr
Abstract
Predictable adaptation of network depths can be an effective way to control infer- ence latency and meet the resource condition of various devices. However, previous adaptive depth networks do not provide general principles and formal explanation why and which layers be skipped, and, hence, their approaches are hard to be generalized and require long and complex training steps. In this paper, we present a practical approach adaptive depth networks that applicable to both convolutional neural networks (CNNs) and transformers with minimal training ef- fort. In our approach, every hierarchical residual stage divided into sub-pate sub-paths, and they are trained to acquire different properties through a simple self-distillation strategy. While the first sub-path is essential for hierarchical feature learning, the second one is trained to refine the learned features and minimize performance degradation even it is skipped. Unlike prior adaptive networks, our approach does not train every target sub-network exhaustively. At test time. however, we can connect these sub-paths in a combinatorial manner select sub-networks of various accuracy-efficiess rationale for why the proposed training method can reduce overall prediction errors while minimizing the impact of skipping sub-paths. We demonstrate the generality and effectiveness of our approach with both CNNs and transformers. Source codes are available at https///iihbb.com/wwhaaaag.
trade-offs from single network. We provide formal
1 Introduction
Modern deep neural networks such as CNNs and transformers [ provide tfte-off-hhe-rt performance at high computational costs, and, hence, lots of efforts have been made to leverage those inference capabilities in various resource-constrainee devices. Those efforts include compact architectures 2 network pruning 4 5 weight/activation quantization [6], knowledge distillation 77] name a few. However, those approaches provide static accuracy-efficiene trade-offs, and, hence, is infeasible to deploy one single model meet devices with all kinds of resource-constraints.
There have been some attempts to provide predictable adaptability to neural networks by exploiting the redundancy in either network depths [8,), widths [I) both [3]
[3]. However, one major difficulty with prior adaptive networks is that they are hard to train and require significantly longer training time than non non-adaptive networks. For example, most adaptive networks select fixed number of nub-networks of varying depths or width, and train them iteratively, mostly through self-distilling knowledge from the largest sub-network (also referred this exhaustive self-distillation takes long time and can generate conflicting training objectives for different parameter-sharing sub-networks, potentially resulting in worse performance [15 [6].
the super-nef) (10) 11] [3 14]. However,
Moreover, unlike addttaadaptttion networks, there is no established principle for adapting depths since the impact of skipping individual layers has not been formally investigated.
CCoresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

super-net (largest)
adaptive depih
Adaptive depth
train sub-paths using self-distillation
Individual networkss
base-net (smallest)
depth (FLOPs)
(a) Training time
(b) Test time
(c) depth vs. accuracy
Figure 1: (a) During training, every residual stage of a network is divided into two sub-paths. The layers in every second (orange) sub-path optimized to minimize performance degradation even if they are skipped. (b) At test time, these second sub- paths can be skipped in a combinatorial manner, allowing instant selection of various parameter sharing sub-networks. (c) The sub-networks selected from a single network form a better Pareto frontier than counterpart individual networks.
In this work, we introduce an architectural pattern and training method for adaptive depth networks that is generally applicable to various networks, e.g., CNNs and transformers. In the proposed adaptive depth networks, every residual stage is divided into two sub-paths and the sub-paths are trained have different properties. While the first sub-paths learning, the second sub-paths are optimized to incur minimal performance degradation even if they are skipped.
mandatory for hierarchical feature
In order to enforce this property the second sub-paths, we propose a simple self-distillation strategy, in which only the largest sub-network (or, super-ner) and the smallest sub-network (or, base-ner) are exclusively used as a teacher and a student, respectively, as shown in Figure (a). The proposed self-distillation strategy does not require exhaustive training of every target sub- sub-network, resulting in significantly shorter training time than prior adaptive networks. However, at test time, sub-networks with various depths can be selected instantly from single network by connecting these sub-paths in a combinatorial manner, as shown Figure[]-(b) Further, these sub-networks with varying depths outperform individually trained non-adaptive networks due to the regularization effect, shown in Figure (c).
In Section 3 we discuss the details of our architectural pattern and training algorithm, and show formally that the selected sub-paths trained with our self-distillation strategy are optimized to reduce prediction errors while minimally changing the level of input features. In Section [4] we empirically demonstrate that our adaptive depth networks outperform counterpart individual networks, both in CNNs and vision transformers, and achieve actual inference acceleration and energy-saving.
2 Related Work
Adaptive Networks: In most adaptive networks, purameter-sharinn sub-networks are instantly selected by adjusting either widths, depths, or resolutions X I0
19. For example, slimmable neural networks adjust channel widths of CNN models on the fly for accuracy- efficiency trade-offs and they exploit switchable batch normalization to handle multiple sub-networks adaptive depth networks have been proposed for language models to dynamically skip some the layers during inference 13 g However, in these adaptive networks, every target sub-network with varying widths or depths need to be trained explicitly, incurring significant training overhcads and potential conflicts between sub-networks.
17
B
15
18
12
10 20 [I]. Transformer-bast
Dynamic networks [21 are another class of adaptive networks that exploit additional control networks or decision gates for input-dependent adaptation of CNN models 22 and transformers 27
Dynamic Networks:
23 24 25
30 In particular, most dynamic networks for depth- adaptation have some kinds of decision gates at every layer (or block) that determine if the layer can be skipped [1 22
26
28
29
32 33 These approaches are based the thought that some layers can be skipped on 'easy" inputs. However, the learned policy for skipping layers is opaque to users and does not provide formal description of which layers can be skipped for given input. Therefore, the network depth cannot be controlled by users predictable manner to meet the resource condition of target devices.

---

skip
True
bass
h 5-1
Fi
Fu241
hip.!
Palco
expar
Mandatory sub-path
Skippable sub-path
Figure 2: Illustration of residual stage with two sub-paths. While the first (blue) sub-path is mandatory for hierarchical feature learning, the second (orange) sub-path can be skipped for efficiency. The layers in the skippable sub-path are trained to preserve the feature distribution from hf to h super using the proposed self-distillation strategy. Having similar distributions, either h or h super be provided input h to the next residual stage. In the mandatory sub-path, another set of batch normalization operators, called skip-aware BNs, exploited the second sub- -abbppath is skipped. These sub-paths are building blocks construct sub-networks of varying depths.
Residual Blocks with Shortcuts: Since the introduction of ResNets [34,, residual blocks with identity shortcuts have received extensive attention because of their ability to train very deep networks, and
have been chosen by many CNNs 35| 36) and transformers 11 37 38) Veit et al. [39] argue that identity shortcuts make exponential paths and results in an ensemble shallower sub-networks. This thought is supported by the fact that removing individual residual blocks test time does not significantly affect performance, and it has been further exploited to train deep networks [40, 41). Other works argue that identity shortcuts enable residual blocks perform iterative feature refinement, where each block improves slightly but keeps the semantic of the representation of the previous layer 42 43
Our work build upon those views on residual blocks with shortcuts and further extend them for adaptive depth networks by introducing a training method that enforces the properties of residual blocks more explicitly for skippable sub-paths.
Adaptive Depth Networks
We first present the architecture and training details of adaptive depth networks. Then, we discuss the theoretic rationale for how depth adaptation can be achieved with minimal performance degradation.
3.1 Architectural Pattern for Depth Adaptation
In modern hicrarchical networks, such ResNets (34] and Swin transformers 388, there are typically
4 to consecutive residual stages In a network with N,. stages, the s-th (s = 1.....N.)) stage consists of L (typically L E [3, 6)) identical residual blocks that transform input features hf additively to produce the output features h" as follows:
h + Fbb + + (11/2(21/2) + Ft.241(h/, 1/22 (22/2+1).... Fz = h
(I)
Piase
#kkpppable
While a block with a residual function Fg (f= 10=1,..... I.) learns hierarchical features as traditional
compositional networks [44) previous literature 44 42 demonstrates that a residual function
also tend to learn function that refines already learned features at the same feature level. If residual block mostly performs feature refinement while not changing the level of input features, the performance of the residual network is not significantly affected by dropping the block at test time 44 41 However, in typical residual networks, most residual blocks tend refine features while learning new level features as well, and, hence, random dropping of residual blocks at test time degrades inference performance significantly. Therefore, we hypothesize that if some selected residual blocks can be encouraged explicitly during training to focus more on feature refinement, then these blocks can be skipped to save computation marginal loss of prediction accuracy at test time.
To this end, we propose an architectural pattern for adaptive depth networks, in which every residual stage is divided into two sub-paths, or F Pi and F= ipippeeee as Equation I and Figure 2 We train these two sub-paths to have different properties (Section 3.2p. While F Piase trained to learn feature representation hj (= 1)/++1) with no constraint, the second sub-path F skippable is constrained
For vision transformers, residual blocks/stages refer encoder blocks/stages.

---

to preserve the feature level of ba and only refine it to produce h hauper (= h Since layers in F base perform essential transformations for hierarchical feature learning, they cannot be bypassed during inference. In contrast, layers in F skippable can be skipped for efficiency since they only refine base. F skippable is skipped, then hase becomes the input to the next residual stage. Therefore,
N (=E C(N,, k))) sub-networks with varying accuracy-efficieng trade-offs can be selected from single network by choosing whether skip F skippable (s 1, ,,..., N,) not (Table c
This architectural pattern is agnostic to the type of residual blocks. The residual function F can be convolution layers for CNNs and self-attention + MLP layers for transformers. Table Bli Appendix A.1 provides details on how this pattern is applied to both CNNs and transformers.
3.2 Training Sub-Patte with Self-Distillation
Preserving the feature level of base in F skippabble implies, more specifically, that two feature repre- sentations hiu and Super have similar distributions over training input X. Algorithm T shows training method, in which hluse and h hupe are encouraged to have similar distributions by including Kullback- Leibllr (KL) divergence between them, or Dr (happer D | (lb ). in the loss function.
Algorithm Training algorithm for an adaptive depth network M. The forward function of M accepts an extra argument, 'skip', which controls the residual stages where their skippable sub-paths are skipped. For example, the smallest sub-network, or base-net, of M is selected by passing "skip=/True, True, True, Truej' J' when the total number of residual stages, denoted by N,, is 4.
1: Initialize an adaptive depth network M
2: for iri- 1 to Riters do
3: Get next mini-batch of data x and label y
4:
optimizer.zero g_rodlrrrd
5: Souper. hauper M. forward(xxx skip=[False, ... False])
forward pass for super-net
6:
losszuper = criterion(yy super)
7:
losssuper backward()
8:
Sbase, hbase M. Mfooworr(xx, skip=[True, .... Truel)
b forward pass for base-net
9:
losstase = -
N₂ Dx (. ( [1baaee)] D (9 super |1 (8ass))
[] is optional
10:
beesbase-backaaan optimizer.ster step()
self-distillation of skippable sub-paths
11:
12:
end for
In Algorithm [ the largest and the smallest sub-networks of M, which are called super-nei and the base-net, respectively, are exploited. In steps 9-10, the hierarchical features (and outputs) from the super-net and the base-net are encouraged to have similar distributions through self-distillation. Self-distillation has been extensively used in prior adaptive networks [I] 13). However, their primary goal is to train every target sub-network. For example, on each iteration, sub-networks are randomly sampled from a large search space to act as either teachers or students [. which takes significantly longer training time and can generate conflicting training objectives for different parameter-share sub-networks [1] 16]. In contrast, our method in Algorithm focuses on training sub- paths by exclusively using the super-net net
teacher and the base-net as a student. By focusing on sub- sub-paths, the training procedure in Algorithm T is significantly simplified and avoids potential conflicts among sub-networks. At test time, however, various sub-networks can be selected by connecting these sub-paths. The effect of this self-distillation strategy is investigated in Section 4.4
Lastly, due to the architectural pattern of interleaving the mandatory and the skippable sub-pathi minimizing D (5 super (Bsse) also minimizes DKL DKzthsuperl/// [/bbae///bbse) [baase) implicitly. The ablation study in Section 4. that omitting the term E ( Super (hjase) in lossbase has a minor effect on the performance. This implicit approach is useful when the extraction of intermediate features is tricky.
3.3 Analysis of Skippable Sub-Paths
Formal Analysis: DEL DKz(heupe (beuper (hass) in the loss function lossbase can be trivially minimized if residual blocks in F F skippable Iearn identity functions, or h + F Pkippabbee (hfase) hease

---

However, since the super-net is jointly trained with the loss function loss super. the residual functions in F skippable cannot simply be an identity function. Then, what do the residual functions in F skippable learn during training? This can be further investigated through Taylor expansion [43]. For our adaptive
depth networks, a loss function L used for training the super-net can be approximated with Taylor expansion follows:
C ((huuee)) [ibouse F ckippabbe (biaee)}
(2)
C(bbase FL/2+1 (hi//+1) + ... FL- 10-1(h2- (h2-1) F
oC(ht.
C(bbase + (L/2+1(hi, +... F2-11hh (h+)
(FF(()))
(4)
Thū.
In Equation the loss function is expanded around h2 or h +... + FL-1 Only the first order term is left and all high order terms, such as Fz2 (b2,))²² 2006 are absorbed in (F (b;)).
The high-order terms in OF2 can be ignored if Fz (h has small magnitude. In typical residual networks, however, every layer is trained to learn new features with no constraint, and, hence, there no guarantee that L'r, (hī) have small magnitude. In contrast, in our adaptive depth networks, the residuals in F skippable are explicitly enforced to have small magnitude through the proposed self- distillation strategy (refer to Figure for empirical evidence). As result, the terms in O(F2 (h₂)) can be ignored for the approximation. If we similarly keep expanding the loss function around hj (j = L/2 + I.... L) while ignoring high order terms, we obtain the following approximation:
aC(h,) Oh
C(hSuper) se C(baase) +
Σ Fy(h)) j=L/241
(5)
In Equation 5 minimizing the loss L(h during training drives Fy(h)) -1=L/1 1/2 in 1...,
the negative half space of ac(h3) to minimize the dot product between F2(h)) and aC(h;) This
implics that every residual function F effect to gradient descent:
skippable is optimized to learn a function that has a similar
OC(h)) Th L/=
F,(h)) =
(6)
In other words, the residual functions in the skippable sub-paths reduce the loss C(h during inference while preserving the feature distribution of hg
iteratively
Considering this result, we can conjecture that, with our architectural pattern and lelf-distillation strategy, layers in F skippable learn functions that refine input features h haase iteratively for better inference accuracy while minimally changing the distribution of h haase
We can estimate how much the distribution of input h is transformed by residual function F by measuring || [|F())] h/ll2 at each residual block. Figure illustrates (12))]/2/111 every residual block the baseline ResNet50 and our ResNe150-ADN that is trained according to Al- gorithm D Our ResNee50-ADN exhibits greater transformation than ResNet50 in the mandatory sub- paths (blue areas) and less transformation in the skip- pable sub-pane nat-paths (orange areas). This result demon- strates that our self-distillation strategy in Algorithm efffetively trains the skippable sub-paths to min- imally change the input distribution. As result, the blocks in skippable sub-paths can be skipped with less impact on performance.
Empirical Analysis:
stage 1
stage
stage
stage 4
ResNet50
ResNott50-ADN
[Fthlll: /|bbb
0.8
0.4
0.2
14
Indox of resiuaal blocks
Figure 3: [[F(h) ||2 at residual blocks. In ours, skippable sub-paths (orange areas) minimally change the distribution of input h.
ImageNet validation dataset is used for this experiment.
5

---

3.4 Skip-Aware Batch/Layer Normalization
Originally, batch normalization (BN) 45 was proposed to handle internal covariate shift during training non-adaptive networks by normalizing features. In our adaptive depth networks, however, internal covariate shifts can occur during inference in mandatory sub-paths if different sub-networks are selected. To handle potential internal covariate shifts, switchable BN operators, called skip-aware BNs, are used in mandatory sub sub-paths. For example, at each residual stage, two sets of BNs are available for the mandatory sub-path, and they are switched depending on whether its skippable sub-pat uub-path is skipped or not.
The effectiveness of switchable BNs has been demonstrated in networks with adaptive widths 46 and adaptive resolutions [47]. However, in previous adaptive networks, N sets of switchable BNs required in every layer to support N parameter-sharing sub-networks. Such a large number of switchable BNs not only requires more parameters, but also makes the training process complicated since N sets of switchable BNs need to be trained iteratively during training. In contrast, in our adaptive depth networks, every mandatory sub-path needs only two sets of switchable BNs, regardless of the number of supported sub-networks. This reduced number of switchable BNs significantly simplifies the training process as shown in Algorithm|] I Furthermore, the amount of parameters for skip-aware BNs negligible. For instance, in ResNet50, skip-aware BNs increase the parameters by 0.07%.
Transformers 38
exploit layer normalization (LN) instead of BNs and naive replacement of LNs to BNs incurs instability during training [48). Therefore, for our adaptive depth transformers, we apply switchable LN operators in mandatory sub-paths instead of switchable BNs.
4 Experiments
We use networks both from CNNs and vision transformers as base models to apply the proposed architecture pattern: MobileNet V2 (35) is a lightweight CNN model, ResNet (34] is a larger CNN model, and VIT (37] and Swin-T 38 are representative vision transformers. All base models except ViT have hicrarchical stages, each with 2 ~ 6 residual blocks. So, according to the proposed architectural pattern, every stage is evenly divided into sub-paths for depth adaptation. ViT does not define hierarchical stages and all residual encoder blocks have same spatial dimensions. Therefore, we divide 12 encoder blocks into groups, resembling other residual networks, and select the last encoder block of each group as a skippable sub-path. Details are in Table[5jin Appendix [ These models are trained according to Algorithm [ For self-distillation, only the final outputs from the super-net and the base-net, or ÿ super and bhase respectively, are used since exploiting intermediate features for explicit self-distillation has marginal impact on performance (Section
We use the suffix 'ADN'' to denote our adaptive depth networks. A series of boolcan values in parentheses denotes a specific sub-network used for evaluation; each boolean value represents the residual stage where its skippable sub-path is skipped. For example, ResNe150-ADN (FFFF) and ResNet50-ADN (TTTT) correspond to the super-net and the base-net of ResNet50-ADN, respectively.
4.1 ImageNet Classification
We evaluate our method on ILSVRC2012 dataset [49 that has 1000 classes. The dataset consists of 1. 1.28M training and 50K validation images. For CNN models, we follow most training settings in the original papers 34 35
except that ResNet models are trained for 150 epochs. ViT and Swin-T are trained for 300 epochs, following DeiT's training recipe [50, 51]. For Swin-T-ADN, we disable stochastic depths [40] for the mandatory sub-paths since the strategy of random dropping of residual blocks conflicts with our approach to skipping sub-paths. For fair comparison, our adaptive depth networks and corresponding individual networks are trained in the same training settings.
The results in 4 (a) show that our adaptive depth networks outperform counterpart individual networks even though many sub-networks share parameters in single model. Further, our results with vision transformers demonstrate that our approach generally applicable and compatible with their state-of-the-an training techniques such as DeiT's training recipe [50]. We conjecture that this performance improvement results from effective distillation of knowledge from he super to h at each residual stage and the iterative feature refinement skippable sub-paths, shown in Equation 6

---

Model
Params (M)
FLOPs
Acc
Resnet50 (individua))
(G)
(%)
Resnet50 indivigualt
Resnetzo-ADD
ResNet50-ADDN (FFFF)
25.58
4.11
77.6
6I-./16-ADN
ResNet50-ADN (TTTT)
2.58
76.1
SwinTTADN
ResNet50
25.56
4.11
76.7
ResNet50-Baas
17.11
75.0
MbV2-ADN (FFFFF)
0.32
72.5
3.53
MbV2-ADN (TTTTT)
0.22
70.6
MbV2
3.50
0.32
72.1
MbV2-Base
2.98
0.22
70.2
ViT-b/I6-ADN (FFFF)
17.58
81.4
86.59
VIT-b/16-ADN (TTTT)
11.76
80.6
ViT-b/16
86.57
17.58
81.1
DepGrapht
VIT-b/16-Baa Base
67.70
11.76
78.7
Dr-RestNet00-
Sp
Swin-T-ADN (FFFF)
4.49
81.6
WiPPuning 0.31311
Swin-T-ADN (TTTT)
28.30
2.34
78.0
D
Swin-T
28.29
4.49
81.5
Ada-DeiT-S.
Swin-T-BR -win-T-Base
15.34
2.34
77.4
2.0
3.0
3,5
4.0
14 orLOrs
(a) Results on ImageNet
(b) Pareto frontiers of our networks
Figure 4: (a) Results on ImageNet validation dataset. Networks with the suffix "-Base' have the same depths as the base-nets of corresponding adaptive depth networks. (b) Pareto frontiers formed by the sub-networks of our adaptive depth networks. ResNet50 (individual) and ResNet50 (KD individual) are non-adaptive networks having same depths as the sub-networks of ResNet50-ADN.
Table 1:
Our base-nets are compared with state-of-the-ar efficient inference methods. f denotes static pruning methods, denotes andthaadaptation networks, and denotes input-dependent dynamic networks. While these approaches exploit various non-canonical training techniques, such as iterative retraining, our base-nets instantly selected from adaptive depth networks without fine-tuning.
Model
FLOPs
[FLOPs
Acc@1
GReg-1
55
2.8G
33%
76.1%
DepGraph (56
2.0G
51%
75.8%
DR- ResNet50 2.0) 47
2.3G
44%
75.3%
ResNet50-ADN (TTTT)
2.6G
37%
76.1%
AlphaNet-0.75+ |
0.21G
34%
70.5%
MbV2-ADN (TTTTT)
0.22G
32%
70.6%
WDPruning- 0.3-11 57
2.6G
78.3%
X Pruner [58]
3.2G
28%
80.7%
Ada-DeiT-S 27
2.3G
77 77.3%
SPViT-Swin-
3.3G
27%
80.1%
Swin-T-ADN (TTTT)
2.3G
48%
78.0%
Figure (b) shows Pareto frontiers formed by selected sub-networks of our adaptive depth networks: Table 6 in Appendix A.2 shows the performance of all sub-networks. In Figure (bb)) and Table D. several state-of-the-art efficient inference methods and dynamic networks our base-networks. The result demonstrates that our adaptive depth networks show comparable performance across range of FLOPs. Figure (b), it should be noted that individual ResNets trained with knowledge distillation has worse performance than individual ResNets. As reported previous works, successful knowledge distillation requires a patient and long training (52). and straightforward knowledge distillation using ImageNet does not improve the performance of student models (53] (54) In contrast, our ResNet50-ADN trained with the proposed self-distillation strategy achieves better performance than counterpart ResNets. This demonstrates that the high performance of adaptive depth networks does not simply come from distillation effect.
compared with
4.2 Training Cost
One of the main advantages of our adaptive depth networks is the significantly lower training effort needed compared to previous adaptive networks. A Algorithm only one additional forward and backward pass of the smallest sub-network, or the base-net, is required for self-distillation at each

---

ResNet500ADD
Swin-T-ADN
Vel. accuracy 1983
Vel. accureci 19)
80
75
RosNet50 (4.1 GRLOPS)
Swin-T
FFFF (4.1 GFLOPs)
70
FFFF (4.5 GFLOPS)
TFFF (3.9 GFLOPS)
65
TFFF (4.1 GFLOPS)
TTFF (3.4 GFLOPs)
60
TTFF (3.7 GFLOPs)
TTTT (2.5 GFLOPs)
TTTT (2.3 GFLOPS)
80 egooch
120
140
50
100
150
250
epoch
Figure 5: Validation accuracy of net-nnsworks our adaptive depth networks during training. Many nub-networks of varying depths become available from single network even though most of them are not explicitly trained.
Table 2:
Training time (1 epoch), measured on Nvidia RTX 4090 (batch size: 128). AlphaNet" is configured to have similar FLOPs to MbV2 and only adjusts its depth to select sub-networks.
Model
epoch (min)
of sub-nets
Res.NetS0
32.9
ResNNct5--Base
22.7
ResNet50-ADN (ours)
54.7
2 depths
MSDNet [8]
67.1
9 depths
S S-ResNet50 [10]
82.5
4 widths
MhV2
13.1
Mb Base
10.0
MbV2-ADN (ours)
22.5
2 depths
AlphaNet"
11]
230.5
216 depths
training iteration. For most CNNs and transformer networks, the original training settings, such as training schedules and hyperparameters, can be used with minimal changes. However, shown in Figure 3] many sub-networks of varying depths become available from a single network by connecting sub-paths in combinatorial manner even though they are not trained explicitly.
Table that, at every training epoch, our adaptive depth networks require a comparable amount of time training sub-paths, rather than sub-networks, by exploiting only the super-net and the base-net. In contrast, the compared adaptive networks require much longer training time than ours since they have explicitly apply self-distillation to all target sub-networks. For example, on every training iteration, AlphaNet [ randomly samples sub-networks from its search space for self-distillation. Although the prior works we compared may seem outdated, they remain relevant and representative because there has been little progress in improving the training cost of adaptive networks.
separate networks combined. This is because our training method trains
supports significantly larger number of sub-networks, our primary goal is not to maximize the number of sub-networks. Instead, our objective is to provide better performance Pareto with a few useful sub-networks, as illustrated in Figure{[ Since the classification performance of sub-networks does not always scale proportionally with their FLOPS, most sub-networks end up being ineffective. For example, although ResNet50-ADM has 2 sub-networks in its search space, T Appendix A.2 shows that some shallower sub-networks outperform deeper sub-networks.
While AlphaNet II
4.3 On-Device Performance
While the Parcto frontiers formed by sub-networks demonstrate theoretic performance, inference acceleration in actual devices is more important in practice for effective control of inference latency and energy consumption.
Figure (a) (a) shows the performance on Nvidia Jetson Orin Nano. The inference latency and energy consumption of ResNet50-ADN is compared SeResOet50, a representative width-adaptation network. The result shows that depth-adaptation of ResNet50-ADN is highly effective in accelerating inference speeds and reducing energy consumption. Although our ResNet50-ADN has a limited FLOPs adaptation range, reducing FLOPs by 38% through depth adaptation reduces both inference latency and energy consumption by 35% In contrast, even though S-ResNet50 can reduce FLOPPs by up to 93% by adjusting its width, it only achieves up to 9% acceleration in practice.

---

26
ResMet50 llatency)
ResNet50 (energy)
26
accurscy 0%
22
ard 0
Ietency
20-
0.30
73
18-
mandatory = skippsble
16
ResANeS5-AAM (atency)
mandatory > stipabble
14
(energy)
mandetory <akippable
1.5
GFLOPs
GRLOPs
(a) On-device performance
(b) Effect sub-path lengths
Figure 6: (a) Inference latency and energy consumption of adaptive networks, measured on Nvidia Jetson Orin Nano (batch size: 1) (b) Pareto frontiers of three ResNet50-ADNs, each trained with vary- ing ratios between mandatory and skippable sub-paths. Total number of blocks remains unchanged.
4.4 Ablation Study
Table 3: Ablation analysis with ResNet50-ADe and ViT-b/32-ADN, Applied components are checked.
4 and f in parentheses are comparisons to non-adaptive individual networks. By default, only the outputs, or ý super and Vbase: are used for self-distillatio.. The last row with double check marks shows the results when both intermediate features and outputs are used for self-distillation.
self- distllation.
skip-aware BN&/LNs
ResNet50-ADN Acc @ I (%)
ViT-b/32-ADN Acc@ (%)
FFFF
TTTT
FFFF
TTTT
75.2% (L 1.5%)
72.2% () 2.8%)
75.7% (10.2%)
74.1% (1 0.3%)
76.1% () 0.6%)
74.9 % (} 0.1%)
76.4% (10.5%)
74.3% (10.5%)
V
76.6% () 0.1%)
75.1% (1 0.1%)
76.0% (10.1%)
74.3% (10.5%)
V
77.6% (t 0.9%)
76.1% (1 1.1%)
76.6% (* 0.8%)
74.3% (10.5%)
Vv
77.3% (t 0.6%)
76.2% (1 1.2%)
We first investigate the influence of two key components of the proposed adaptive depth networks: (1) self-distillation of sub-paut paths and (2) skip-aware BNs/LNs. When our self-distillation method is not applied, the loss of the base-net, lossbase: in Algorithm I is modified to criterion(y, $tase). Table 3 shows the results. For ResNet50-ADM. when neither of them is applied, the inference accuracy of the super-net and the base-net is significantly lower than con-adaptive individual networks by 1.5% and 2.8%, respectively. This result shows the difficulty of joint training sub-networks for adaptive networks. When one of the components is applied individually, the performance is still slightly worse than individual networks'. When both self-distillation and skip-aware BNs are applied together, ResNet50-ADN achieves significantly better performance than individual networks, both in the super-net and the base-net. The last row, with double check marks, shows that exploiting intermediate features as well softmax outputs for self-distillation has minor impact on performance.
Table 4: Comparison of self-distillation strategies. Our approach (in bold) uses exclusively the super-net and the base-net teacher and student, respectively.
sub-nets
ResNet50 ADN Acc@1 (%)
ViT-b/32-ADN Ace@ (%)
Teacher Student
FFFF
TFFF
TTFF
TTTF
TTTT
FFFF
TFFF
TTFF
TTTF
TTTT
FFFF TTTT
77.6
77.1
76.5
76.0
76.1
76.6
76.0
75.5
74.5
74.3
FFFF
Random
77.1
76.7
76.4
75.5
74.8
75.2
74.7
73.9
72.9
71.1
Random TTTT
75.5
75.5
75.4
75.0
74.9
72.0
72.0
71.9
71.8
71.7
Random
Random
75.4
75.2
75.2
74.9
74.6
70.8
70.7
70.6
70.4
70.3
Our self-distillation approach in Algorithm exploits only two sub- networks exclusively teacher and a student. Specifically, the super-nets, or FFFF, acts as the the teacher and the base-nets, or TTTT, becomes the student. The purpose of this strategy not train only those two sub-networks, but rather train skippable sub-paths in way that minimally modifies
Saff-Distillation Strategies:

---

the feature distribution, as demonstrated in Figure 3 To investigate the effect of self-distillation strategies, we conduct an experiment in Table every training iteration, rather than exclusively using FFFF and TTTT sub-networks for self-distillation, we randomly sample either a teacher, student, or both from 24 sub-networks. These randomly sampled sub-networks are trained explicitly through self-distillation. However, shown in Table all sub-networks, both from ResNet50-ADX and ViT-b/32-ADN, perform significantly better when our self-distillation strategy is applied for training. Even though most of our sub-networks (such as TFFF, TTFF, and TTTF) are instantly constructed at test time without explicit training, they still outperform their counterpart sub-networks trained explicitly through random sampling. This result demonstrates that our method of training is sub-pat -aths is more effective than training target sub-networks themselves.
Lengths of Mandatory Sub-Paths: In Figure (b), we investigate the impact of varying the ratio of lengths between the mandatory and the skippable sub-paths. (Details are in Table 7 If mandatory sub-paths become shorter than skippable sub sub-paths, shallower sub-networks can be selected since more layers can be skipped. However, the result Figure shows that this configuration (shown in blue line) significantly degrades the performance of all sub-networks. Since every sub-network shares parameters mandatory sub-paths, low inference capability of shallow mandatory sub-paths affects all sub-networks. This implies that maintaining certain depths in mandatory sub-patho is crucial for effective inference. Conversely, further increasing the length of mandatory sub-paths (shown in green line) does not further improve performance and only reduces the range of depth adaptation.
Conclusions
We propose practical approach to adaptive depth networks that can be applied to both CNNs and transformers with minimal training effort. We provide a general principle and formal explanation on how depth adaptation can be achieved with minimal performance degradation. Under this principle, our approach can avoid typical exhaustive training of target sub-networks and instead focus on optimizing the sub-paths of the network to have specific properties. At test time, these sub-paths can be connected a combinatorial manner to construct sub-networks with various accuracy-efficieme trade-offs from single network. Experimental results show that these sub-networks form better Pareto frontier than non-adaptive baseline networks and achieve actual inference acceleration. We anticipate that these advances will facilitate the practical application of adaptive networks.
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for their constructive comments and suggestions. This work was supported by the National Research Foundation of Korea (NRF) Grants Funded by the Ministry of Science and ICT under Grant NRF-202RRIFJA1772211
References
[1] Vaswani, A., N. Shazeer, N. Parmar, et al. Attention is all you need. In Conference on Neural Information Processing Systems (NeurIPS), vol. 30. 2017.
[2] Howard, G., M. Zhu, B. Chen, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 170.:::700.044661 2017.
[3] Han, K., Y. Wang, Q. Tian, et al. Ghostnet: More features from cheap operations. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). 2020.
[4] Han, S., H. Mao, W. J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In International Conference on Learning Representations (ICLR). 2016.
[5] Liu. Z., H. Mu, X. Zhang, al. Metapruning: Meta learning for automatic neural network channel pruning. In International Conference on Computer Vision (ICCV), pages 2305. 2019.
[6] Jacob, B., S. Kligys, B. Chen, et al. Quantization and training of neural networks for efficient integer-riibmetticonly inference. In Conference on Computer Vision and Pattern Recognition (CVPR). 2018
10

---

[7] Hinton, G., O. Vinyals, J. Dean. Distilling the knowledge in a neural network. arXiv preprint 1rX253150..02531, 2015.
[8] Huang, G., D. Chen, T. Li, et al. Multi-scale dense networks for resource efficient image classification. In International Conference Learning Representations (ICLR). 2018.
[9] Fan, A., E. Grave, A. Joulin. Reducing transformer depth demand with structured dropout. In International Conference on Learning Representations (ICLR). 2020.
[10] Yu, J., L. Yang, N. Xu, et al. Slimmable neural networks. In International Conference on Learning Representations (ICLR). 2019.
[11] Wang, D., C. Gong, M. Li, Alphanet: Improved training of supernets with alpha-divergence. In M. Meila, T. Zhang, eds., International Conference Machine Learning (ICML), vol. 139 of Proceedings of Machine Learning Research, pages 10760-10771. PMLR, 2021.
[12] Wan, C., H. Hoffmann, S. Lu, et al. Orthogonalized sgd and nested architectures for anytime neural networks. In International Conference Machine Learning (ICML), pages 9807-9817. PMLR, 2020.
[13] Hou, L., Z. Huang, L. Shang. et al. Dynabert: Dynamic bert with adaptive width and depth. Conference on Neural Information Processing Systems (NeurIPS), 33, 2020.
[14] Touvron, H., M. Cord, M. Oquab, et al. Co-training 21 submodels for visual recognition. In Conference on Computer Vision and Pattern Recognition (CVPR). 2023.
[15] Li, H., H. Zhang, X. Qi, et al. Improved techniques for training adaptive deep networks. In International Conference on Computer Vision (ICCV), pages 1891-1900. 2019.
[16] Gong, C., D. Wang, M. Li, et al. NASVit: Neural architecture search for efficient vision transformers with gradient conflict aware supernet training. In International Conference on Learning Representations (ICLR). 2022.
[17] Hu, H., D. Dey, M. Hebert, et al. Learning anytime predictions in neural networks via adaptive loss balancing. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pages 3812-3821. 2019.
[18] Zhang, L., J. Song, A. Gao, et al. Be your teacher: Improve the performance of convolu- tional neural networks via self distillation. In International Conference on Computer Vision (ICCV), pages 3713-3722. 2019.
[19] Beyer, L., P. Izmailov, A. Kolesnikov, al. Flexivit: One model for all patch sizes. In Conference on Computer Vision and Pattern Recognition (CVPR). 2023.
[20] Yu, J., T. S. Huang. Universally slimmable networks and improved training techniques. In International Conference on Computer Vision (ICCV), pages 880--881. 2019.
[21] Han, Y., G. Huang, S. Song, et al. Dynamic neural networks: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence, (4(11)7746-74566 2022.
[22] Wu, Z., T. Nagarajan, A. Kumar, et al. Blockdrop: Dynamic inference paths in residual networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 8817-8826. 2018.
[23] Li, C., G. Wang, B. Wang, et al. Dynamic slimmable network. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 8607-8617. 2021.
[24] Guo, Q.. Z. Yu, Y. Wu, et al. Dynamic recursive neural network. 2019 JEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5142-5151. 2019.
[25] Li, Y., L. Song, Y. Chen, et al. Learning dynamic routing for semantic segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR). 2020.
[26] Yang, L., Y. Han, X. Chen, et al. Resolution adaptive networks for efficient inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2020.
[27] Meng, L., H. Li, Chen, et al. Adavit: Adaptive vision transformers for efficient image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 12299- 12308. 2022.
[28] Yin, H., A. Vahdat, M. Alvarez, et A-VIT: Adaptive tokens for efficient vision transformer. In Conference Computer Vision and Pattern Recognition (CVPR), pages 10809-10818, 2022.

---

[29] Fayyaz, M., S. A. Koohpayegani, F. R. Jafari, et al. Adaptive token sampling for efficient vision transformers. In S. Avidan, G. Brostow, M. Cissé. G. M. Farinella, T. Hassner, eds., European Conference on Computer Vision (ECCV), pages 396-414, 2022.
[30] Heo, B., S. Yun, D. Han, et al. Rethinking spatial dimensions of vision transformers. In International Conference on Computer Vision (ICCV). 2021.
[31] Figurnov, M., M. D. Collins, Y. Zhu, et al. Spatially adaptive computation time for residual networks. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1039- 1048. 2017.
[32] Veit, A., S. Belongie. Convolutional networks with adaptive inference graphs. In European Conference on Computer Vision (ECCV), pages 3-18. 2018.
[33] Wang, X., F. Yu, Z.-Y. Dou, et al. Skipnet: Learning dynamic routing in convolutional networks In European Conference on Computer Vision (ECCV), pages 409-424. 2018.
[34] He, K., X. Zhang, S. Ren, et al. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[35] Sandler, M., A. Howard, M. Zhu, et al. Mobilenetv2: Inverted residuals and linear bottlenecks. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 4510-4520. 2018.
[36] Tan, M., Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning (ICML), pages 6105-6114, 14. PMLR, 2019.
[37] Dosovitskiy, A., L. Beyer, A. Kolesnikov, et al. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations (ICLR). 2021.
[38] Liu, Z., Y. Lin, Y. Cao, et al. Swin transformer: Hierarchical vision transformer using shifted windows. In International Conference on Computer Vision (ICCV). 2021.
[39] Veit, A., M. J. Wilber, S. Belongie. Residual networks behave like ensembles of relatively shallow networks. Conference on Neural Information Processing Systems (NeurIPS), 29:550- 558, 2016
[40] Huang, G., Y. Sun, Z. Liu, et al. Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV), pages 646-661. Springer, 2016.
[41] Xie, Q., M.-T. Luong, E. Hovy, et al. Self-training with noisy student improves imagenet classification. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 10687-10698. 2020.
[42] Greff, K., R. K. Srivastava, J. Schmidhuber. Highway and residual networks learn unrolled iterative estimation. In International Conference Learning Representations (ICLR). 2016.
[43] Jastrzebski, S., D. Arpit, N. Ballas, al. Residual connections encourage iterative inference. In International Conference on Learning Representations (ICLR). 2018.
[44] Simonyan, K., A. Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In 3th International Conference on Learning Representations (ICLR). 2015.
[45] loffe, S., C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), pages 448 456. PMLR, 2015.
[46] Yu, J., T. Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint, 2019.
[47] Zhu, M., K. Han, E. Wu, et al. Dynamic resolution network. In Conference on Neural Information Processing Systems (NeurlPS), vol. 35, pages 10985-10998. 2021.
[48] Yao, Z., Y. Cao, Y. Lin, et al. Leveraging batch normalization for vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 413-422. 2021.
[49] Russakovsky, O., J. Deng, H. Su, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)211-252,22 2015.
[50] Touvron, H., M. Cord, M. Douze, et al. Training data-efficient image transformers & distillation through attention. arXiv preprint, 2020.
12

---

[51] PyTorch. Image classification reference training scripts in pytorch, 2023.
[52] Beyer, L., X. Zhai, A. Royer, et al. Knowledge distillation: A good teacher is patient and consistent. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 10925- 10934. 2022.
[53] Zagoruyko, S., N. Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In 5th International Conference Learning Representations, JCLR, 2017.
[54] Cho, J. H., B. Hariharan. On the efficacy of knowledge distillation. International Conference on Computer Vision (ICCV), pages 4794-4802. 2019.
[55] Wang, H., C. Qin, Y. Zhang, et al. Neural pruning via growing regularization. In International
Conference Learning Representations (ICLR). 2021.
[56] Fang, G., X. Ma, M. Song, et al. Depgraph: Towards any structural pruning. In Conference Computer Vision and Pattern Recognition (CVPR), pages 16091-16101. 2023.
[57] Yu, F., K. Huang, M. Wang, et al. Width & depth pruning for vision transformers. Proceedings of the AAAI Conference Artificial Intelligence, 36(3):3143-311
2022.
[58] Yu, L., W. Xiang, X-pruner: explainable pruning for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2023.
[59] Kong, Z., P. Dong, X. Ma, et al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, T. Hassner, eds., European Conference on Computer Vision (ECCV), pages 620-640. 2022.
[60] Ren, S., K. He, R. Girshick, et Faster r-enn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[61] He, K., G. Gkioxari, P. Dollár, et al. Mask r-cnn. In International Conference on Computer Vision (ICCV), pages 2961-2969, 2017.
[62] Lin, T.-Y., P. Goyal, R. Girshick, et al. Focal loss for dense object detection. In International Conference on Computer Vision (ICCV). 2017.
[63] Lin, T.-Y., Dollar, R. Girshick, et al. Feature pyramid networks for object detection. In Conference Computer Vision and Pattern Recognition (CVPR). 2017.
[64] Selvaraju, R. R., M. Cogswell, A. Das, et al. Grad-cam: Visual explanations from deep networks via gradient-base based localization. In International Conference on Computer Vision (ICCV), pages 618-626. 2017.
13

---

A Appendix: Detailed Settings and Evaluation Results
A.1 Detailed Architectures
number of mandatory blocks
number of skippable blocks
number number
total blocks
mandatory blocks
skippable blocks
blocks
Stage
2
stage
stage
2
2
2
4
stage
stage
3
3
stage
stage
2
stage
(a) ResNet50-ADN
(b) Vit-b/16-ADN
number of mandatory blocks
number of skippable blocks
total blocks
number of mandatory blocks
number of skippable blocks
total blocks
stage
stage 2
2
stage 3
2
stage
stage 4
2
2
stage
stage 5
2
stage
stage 6
2
stage
stage 7
(d) Swin-T-ADN
(c) MbV2-ADN
Table 5: Each stage of base models is evenly divided into two sub-paths; the first is mandatory and the other skippable. Since ViT does not define hierarchical stages, 12 identical encoder blocks divided into stages, resembling other residual networks for vision tasks.
A.2 Performance of Sub-Networks
netwnetwork
FLOPs (G)
Acc@ I (%)
sub-network
FLOPs (G)
Acc@I (%)
FFFF
4.11
77.6
FFFF
17.58
81.4
TFFF
3,90
77.1
TFFF
16.20
81.1
FTFF
3.68
76.5
FTFF
16.20
81.0
FFTF
3.46
75.6
FFIF
16.20
80.6
FIFT
3.90
76.7
FFFT
16.20
81.2
TTFF
3,46
76.5
TIFF
14.67
80.9
TFTF
3.24
75..3
TFTF
14.67
80.4
TFFT
3.68
76.4
TFFT
14.67
80.9
FTTF
3.02
75.8
FTTF
14.67
80.5
FIFT
3,46
75.9
FIFT
14.67
80.9
FFTT
3.25
75.6
FFTT
14.67
80.6
TTTF
2.80
75.9
TTTF
13.21
80.5
TIFT
3.24
75.8
TIFT
13.21
80.7
TFTT
3.02
75.3
TFTT
13.21
80.5
FTTT
2.80
76.0
FTTT
13.21
80.6
TTTT
2.58
76.1
TTTT
11.76
80.6
(a) RGNN
(b) ViT-b/16-ADN
Table 6: FLOPss and ImageNet validation accuracy of sub-net networks.. Only super-net (or, FFFF) and base-net (or, TTTT) are trained explicitly. Sub-networks in the middle can be selected at test time without explicit training. The highest accuracy in each group is shown in bold.
14

---

A.3
Varying the Ratio of Sub-Path Lengths in ResNet50-ADN
number of mandatory blocks
number of skippable blocks
total blocks
number mandatory blocks
number skippable blocks
total blocks
Stage
2
stage
stage
3
4
stage
stage
2
4
6
stage
stage
2
3
stage
(a) of mandatory < skippable
(b) mandatory of skippable
Table 7: The configurations of ResNe150-ADD with different proportions between mandatory and skippable sub-paths. Total number of blocks at each stage remains unchanged.
A.4 Varying the Ratio of Sub-Path Lengths in Vit-b/16-ADN
Each stage of Vit-b/16-ADN has encoder blocks and, by default, we select every last encoder block as skippable sub-path. Therefore every stage has two mandatory blocks and 1 skipable blocks as shown in Table (b). Figure (a) shows different configuration where every last two blocks of the stages become skippable. With this configuration in Figure (a), we can select much smaller sub-networks. For example, the smallest sub-network, or the base-net, of Vit-b/15-ADD has only 4 mandatory blocks and it requires 5,82 GFLOPs. However, the result in Figure (b shows that this configuration significantly degrades performance of all sub-networks. As demonstrated with ResNet50-ADN in Figure the low inference capability of shallow mandatory sub-paths affects all sub-networks. This result again shows that maintaining certain depths mandatory sub-paths is crucial for effective inference.
number of mandatory blocks
number of skippable blocks
total blocks
stage
stage 2
2
ahppsbi
cappstse
stage 3
2
10
stage
2
GELOPE
(a) # of mandatory < of skippable
(b) Pareto frontiers of Vit-b/16-ADN
Figure 7: (a) The configuration of Vit-h/16-ADN with longer skippable sub-paths. (b) Pareto-frontier when different length ratios between the mandatory and the skippable sub-paths are applied.
B Appendix: More Experiments and Analysis
B.1
Object Detection and Instance Segmentation
Table 8: Object detection and instance segmentation results on MS COCO dataset.
Individual Networks
ResNet50-ADN (ours)
Detector
Backbone
FLOPs
Box AP
Mask AP
Box AP
Mask AP
Daster-RCNI
ResNet50
207.07G
36.4
37.8
60
ResNet50-Bass
175.66G
32.4
34.0
RCNN
ResNet50
260.14G
37.2
34.1
38.3
34.1
61
ResNet50-Base
228.73G
32.7
29.9
34.1
31.2
RetinaNet
ResNet50
151.54G
36.4
37.4
62
ResNet50-Basse
132.04G
31.7
35.2
In order to investigate the generalization ability our approach, we use MS COCO 2017 datasets on object detection and instance segmentation tasks using representative detectors. We compare
15

---

individual ResNet50 and our adaptive depth ResNet50-ADN as backbone networks of the detectors. For training detectors, we use Algorithm T with slight adaptation. For object detection, the interme- diate features hase and h suger pyramid networks (FPN) [63], and, hence, a wrapper function not required to extract intermediate features. All networks are trained on train2017 for 12 epochs from ImageNet pretrained weights, following the training settings suggested in [63]. Table 8 shows the results on va12017 containing 5000 images. Our adaptive depth backbone networks still outperform individual static backbone networks in terms of COCO's ' standard metric AP.
1..N,) ) can be obtained directly from backhone network's 's feature
B.2 Visual Analysis of Sub-Paths
To investigate how our training method affects feature representations in the mandatory and the skippable sub-paths, we visualize the activation 3rd residual stage of DesNeet0-AAD. using Grad- CAM [64] The 3rd residual stage of ResNec50-ADN has 6 residual blocks and the last three blocks are skippable. In Figure (a), the activation regions of original ResNet50 changes gradually across super-net, manifests
all consecutive blocks. In contrast, in Figure (b). DEsNeL50-ADN(FFFT) very different activation regions in two sub-paths. In the first three residual blocks, we can observe lots of hot activation regions in wide areas, suggesting active learning of new level features. In contrast, significantly less activation regions are found in the skippable last three blocks and they are gradually concentrated around the target object, demonstrating the refinement of learned features. While (OSNeE5S-AD((TITTT), or base-net, shares the parameters with the super-net net in the first mandatory blocks, their activation regions very different from the super-net's. This is because while the super-net and the base-net share parameters in the non-skippable mandatory blocks, they use different batch normalization operators in the mandatory sub-paths. Further, in Figure (c), we can observe that the final activation map of the base-net is very similar to the super-net's 's final activation map in Figure (b). This implies that they have similar distributions for the same inputs, as suggested in Section
Irpar Image
Mboc
bleck
blcck
Meck
black
S
(a) Original ResNet50
C
(b) ResNet50- (FFFF)
(c) ResNet50-ADN (TTTT)
Figure 8: Class Activation Maps of the 3rd residual stages of ResNet50s. (a) Original ResNet50's activation regions change gradually across all blocks. (b) In ResNet50-ADX (FFFF), the first blocks have extensive hot activation regions, implying active learning of new level features. In contrast, the skippable last blocks have far less activation regions and they are gradually refined around the target. (c) Even though parameters are shared, the activation map of base-net is very different from super-net's since they use different batch normalization operators.
16

---

NeurIPS Paper Checklist
Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 4.3 Figure (a) highlights both the advantage and a limitation of our approach. While the compared adaptation method, c.g., width scaling, provides wider FLOPs adaptation range, our approach adapting depths has a more limited adaptation range. As shown in Figure (b). this limitation stems from the fact that maintaining certain depths in mandatory sub-paths is crucial for effective inference. To achieve further FLOPs scaling, we may need to explore synergies with other adaptation dimensions, such as adjusting network widths [10] and input resolutions [26], along with our depth adaptation method. We leave this our future work.
In Section 4.2 we mention that the prior adaptive networks we selected for the comparison of training time may seem outdated. While we acknowledge the importance of recent literature, there are limited works that closely align with our specific approach. For instance, research on adaptive networks in the context of CNNs is not very active recently. Research specifically addressing the reduction training time for adaptive networks is even further scarce. This scarcity of recent research may be attributed to the impracticality of prior adaptive networks, which is characterized by lengthy and complex training procedures and relatively low inference performance. This impracticality of prior adaptive networks is the major motivation for our research. For vision transformers, there are some studies dynamic networks and our paper also cites some of these relevant studies in Table [] However, their emphasis tends to be on improving accuracy rather than reducing training time. We believe that the scarcity of recent prior research in this area implies that our study introduces a novel direction in this field.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Section 3.3 provides formal analysis our work.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide source codes and pretrained weights as supplementary material. The source codes includes default settings, e.g., hyperameters, used in the evaluation. Further, we provide more detailed experiment settings in Appendix A.I
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide source codes and pretrained weights as supplementary material. They will be available in github shortly for open access.
6. Experimental Setting/Details
17

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Section B.I and Appendix provide details of the evaluation. Further the source codes includes default settings, e.g., hyperameters, used in the evaluation.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We ImageNet and COCO datasets for experiments. Training models with these large datasets requires too much computing resources to obtain meaningful errorbars.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
In the experiments, we mention what kinds of computing resource is used. For example, in Section mention that RTX 4090s, 3090s were used for training. In Section
Justification:
4.3
we mention that Nvidia Jetson Orin Nano was used to evaluate on-device performance.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp///eerrrps..c.//aal///aaad
Answer: [Yes]
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We can hardly think of negative societal impact of our work since the goal of this work is make deep learning training and inference more energy-efficient Our approach not only accelerates model development but also contributes to sustainability efforts.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
18

---

Answer: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
19


Reviews:
[{"review_id": "BeU9zA6x0V", "reviewer": "Reviewer_dirV", "summary": "The paper provides a training methodology to develop small deployable subnetworks (also high performing) when training a single large network. The key claim of this paper is that due to their innovative training techniques the smaller subnetworks learns better feature and proves the point with reasonable baselines.", "questions": "Can this method work in medium size LLMs where distillation could be tricky?", "limitations": "Yes, the authors have tried to address the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Some of the key strengths of this paper is written below.\n\na) The results provided in this paper looks strong with a lot of baselines.\n\nb) The depth adaptation analysis in sec 3 looks a bit hand wavy but tries to provide some prospective, which in general is a good practice.\n\nc) The writing and presentation of this paper is very clear and concise.", "weaknesses": "Some major weaknesses of this paper are below.\n\na) The claim made in the paper \" To the authro's knowledge, this is the first approach ...\" is a bit of an overclaim. In prior CNN literature there are some works that performs subnetwork distillation. \n\nb) In sec 3.2 the authors claim that h_{base} learns compact representation. I would request the authors to quantity this more concretely. \n\nc) The results in Figure 4 a) are a bit tricky to evaluate as author's models are trained with distillation and the baseline models are trained with regular training. How do we know if the high accuracy is due to distillation or due to the proposed technique. The baseline has to be fixed.\nd) More baseline comparisons are required with other techniques such as Matformer (https://arxiv.org/abs/2310.07707), Inheritune (https://arxiv.org/abs/2404.08634) etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "* **Weakness a)** *The claim ... is a bit of an overclaim.* \n\nThank you for your positive feedback and valuable suggestion. \n\nWe appreciate the opportunity to clarify our claims. We acknowledge that there have been prior works in the CNN literature that perform sub-network distillation. Our intention was to highlight the effectiveness of our self-distillation strategy that focuses on training sub-paths, not sub-networks. \nTo address your concern, we will revise the statement to more accurately reflect the novelty of our work in the context of prior studies.  \n\nRevised statement: *“To the best of our knowledge, while prior CNN literature has explored sub-network distillation for adaptive networks, this approach uniquely provides a principle for training sub-paths for predictable depth adaptation. This principle allows us to avoid typical exhaustive training of target sub-networks and instead instantly construct sub-networks of varying depths from specifically trained sub-paths.”*\n\nWe hope this revision clarifies our contribution and addresses your concern. \n\n* **Weakness b)** *Authors claim...$h_{base}$ learns compact representation. I would request the authors to quantity this more concretely.* \n\nWe mention the *compactness* of $h_{\\text{base}}$ because sub-networks, such as ResNet50-ADN(TTTT), achieve higher classification performance (e.g., by 1.1%) than equivalent ResNet50-Base, as shown in Figure 4-(a). We hypothesize that this performance improvement results from compressing the knowledge from $h_{\\text{super}}$ to $h_{\\text{base}}$ through the self-distillation process.\n\nQuantifying the compactness of representation is challenging, but we might attempt it by measuring the *cosine similarity* between feature representations $h_{\\text{super}}$ and $h_{\\text{base}}$. We can infer that the greater the similarity, the more knowledge has been transferred from $h_{\\text{super}}$ to $h_{\\text{base}}$. The following table shows the cosine similarity between $h_{\\text{base}}$ and $h_{\\text{super}}$ measured during forward passes of 1000 ImageNet validation data. At every residual stage, our approach demonstrates much higher similarity between $h_{\\text{base}}$ and $h_{\\text{super}}$, implying that more knowledge has been transferred to $h_{\\text{base}}$.\n\n|  | Stage 1 |  Stage 2| Stage 3  | Stage 4 | \n|---|---:|---:|---:|---:|\n| ResNet50-ADN (ours) | 0.98 | 0.96| 0.81 | 0.87| \n| ResNet50 (Pytorch pretrained) | 0.91 | 0.82| 0.67 | 0.81| \n\nAdditionally, in Figure 8 (Appendix B.2), we visualize $h_{\\text{base}}$ and $h_{\\text{super}}$ using Grad-CAM. \nThese visualized images also show high similarity between $h_{\\text{base}}$ and $h_{\\text{super}}$.\n\n* **Weakness c)** *...How do we know if the high accuracy is due to distillation or due to the proposed technique....* \n\nIn Figure 4(b), we illustrate the impact of applying knowledge distillation (KD) to equivalent networks. For instance, ResNet50 (KD individual) represents the performance when equivalent individual networks are trained with KD, using a PyTorch pretrained ResNet50 as the teacher network. Contrary to common belief, the naïve application of KD does not enhance performance. In fact, following the same training schedule (150 epochs for ResNets), KD results in worse performance compared to ordinary training using target labels.\n\nThis finding aligns with prior work, such as [52][53][54], which also indicates that achieving positive results with KD on ImageNet is very challenging. To obtain positive outcomes with KD, an extended training schedule, a right combination of teacher/student and optimization techniques are required. For example, [52] achieves state-of-the-art ResNet50 performance with KD on ImageNet by employing a 1200-epoch training schedule.\nIn contrast, our ResNet50-ADN trained with the proposed self-distillation strategy consistently achieves better performance than counterpart ResNets. This demonstrates that the high performance of adaptive depth networks does not simply come from distillation effect.\n|  |  Acc@1 |\n|---|---:|\n|ResNet50-ADN (FFFF) | 77.6 | \n|ResNet50-ADN (TTTT) | 76.1 |\n| | |\n|ResNet50 (individual) | 76.7 |\n|ResNet50-Base (individual) | 75.0 |\n| | |\n|ResNet50 (KD individual) | 75.1 |\n|ResNet50-Base (KD individual) | 73.8 |\n\n* **Weakness d)** *More baseline comparisons are required ... such as Matformer, Inheritune* \n\nThank you for suggesting related works. We reviewed both papers and found that Matformer shares a similar goal with our approach. Both Matformer and our method train a universal model from which smaller sub-networks can be extracted without additional training. We believe that Matformer’s nested FFN architecture could complement our work by providing more fine-grained adaptation of FLOPs and parameters within transformer blocks.\n\nHowever, the Matformer paper does not include detailed performance for sub-networks, such as FLOPs versus accuracy, and the source code is not available to reproduce the results. Therefore, it is challenging to include Matformer as a baseline. \nConsequently, we will compare Matformer in the Related Work section.\n\n* **Question)** *Can this method work in medium size LLMs where distillation could be tricky?*\n\nWe have not applied our approach to large language models (LLMs) as we currently lack the resources to train them. However, our results demonstrate that our depth adaptation approach is effective for Vision Transformers (ViTs), which utilize transformer encoders. Although not included in the paper, we recently applied our approach to DETR [R1], an object detection network with transformer decoders, and achieved depth adaptation without any loss of mean Average Precision (mAP).\nWhile we have not yet had the opportunity to apply our method directly to LLMs, we believe it can be effective since LLMs also consist of transformer encoders/decoders.\n\n[R1] Carion et al., “End-to-End Object Detection with Transformers”, ECCV 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QS3Bkg67RL", "reviewer": "Reviewer_phTU", "summary": "The paper presents adaptive depth networks. During training, the network is trained like a “super-net” that contains all the paths. During inference, the skippable networks can be skipped in devices that have limited resources. The whole framework is useful in real-world applications that require various accuracy-efficiency trade-offs.", "questions": "Please refer to the second point in weakness.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation is clear. Proposing such a framework is useful in real-world applications.\n- The authors conduct extensive experiments, including CNNs and Transformers.\n- The experiments show that the proposed method is effective. Even on the TTTT setting, the performance drop is acceptable.", "weaknesses": "- There seem to be several solutions with ideas similar to those in this paper, such as [13]. Although there may be differences in the approaches, all of these methods can achieve the same goals in this paper. In my opinion, it would be better to include a comprehensive analysis of the differences and performance. If the proposed method could stand out in performance among the methods, it would make the proposed method very competitive among these methods.\n\n- A recent study [R1] found that with longer schedules, the ResNet-50 model can achieve about 80% accuracy on ImageNet. Considering that we can choose to give smaller models a longer training schedule to achieve better results, is there a chance to use a longer schedule to make the Base model achieve better results and avoid using an adaptive network?\n\n\n[R1] ResNet strikes back: An improved training procedure in timm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "* **Weakness a)** *There seem to be several solutions with ideas similar to those in this paper, such as [13]. ... it would be better to include a comprehensive analysis of the differences and performance. If the proposed method could stand out in performance among the methods, it would make the proposed method very competitive among these methods.*\n\nThank you for your insightful and encouraging suggestion. We focused our approach on vision tasks due to the simplicity of the model architectures and training steps, which effectively highlight the advantages of our proposed method. However, as you rightly pointed out, the field of NLP has seen significant advancements recently, and there are some notable works such as [13] that shares the same goal. For broader impact, we need to compare our work to those works in NLP. \nOur method, which is applicable to transformer-based models in vision tasks, can indeed be extended to NLP models such as BERT and GPT. However, NLP models typically require more sophisticated training steps, including pretraining and fine-tuning, as well as substantial computational resources. We plan to explore these applications in a follow-up paper, where we will also investigate transformer-specific adaptation techniques.\n\n* **Question)** *A recent study [R1] found that with longer schedules, the ResNet-50 model can achieve about 80% accuracy on ImageNet. Considering that we can choose to give smaller models a longer training schedule to achieve better results, is there a chance to use a longer schedule to make the Base model achieve better results and avoid using an adaptive network?*\n\nAs noted in [R1], the performance of ResNet can be enhanced by employing a longer training schedule and recent augmentation techniques, such as CutMix. Our ResNet50-ADN model benefits from these recent training techniques as well. \n\nFollowing table shows the results when we applied the PyTorch v2 training script (https://github.com/pytorch/vision/issues/3995)  to train ResNet50-ADN and individual networks. Using the PyTorch v2 training script, ResNet50-ADN (FFFF) and ResNet50-ADN (TTTT) achieve 80.44% and 78.78% top-1 accuracy on ImageNet-1K, respectively. The equivalent individual networks, ResNet50 and ResNet50-Base, achieve 80.44% and 78.17% top-1 accuracy, respectively.\nWhile ResNet50 (FFFF) and ResNet50 perform equally well, our smallest sub-network ResNet50 (TTTT) outperforms the equivalent ResNet50-Base by 0.62%. This is the result without active hyperparameter tuning, indicating that there is a chance for further performance improvement for our models. \n\nThese results demonstrate that our approach can be effectively combined with recent training techniques.\n|   |   Acc@1 | \n|---|---|\n|ResNet50-ADN (FFFF) | 80.44%|\n|ResNet50-ADN (TTTT) | 78.79%|\n||\n|ResNet50 (individual) | 80.44%|\n|ResNet50-Base (individual) | 78.17%|\n\n[R1] ResNet strikes back: An improved training procedure in timm.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zv7SaqhNDD", "reviewer": "Reviewer_RAbE", "summary": "The submission presents an approach to adaptive depth networks, where each hierarchical residual stage is divided into two sub-paths and they are trained to acquire different properties:  the first sub-path is essential for hierarchical feature learning, the second one is trained to refine the learned features and minimize performance degradation even if it is skipped. In addition, a formal reason of why the proposed training method can reduce overall prediction errors while minimizing the impact of skipping sub-paths is also provided. Experimental results on ImageNet classification are provided to prove the effectiveness.", "questions": "I don't have specific questions to this submission.", "limitations": "No problem in limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "There are 2 major innovations in this submission:\n1. Training Sub-Paths with Self-Distillation\n2. Skip-Aware Batch/Layer Normalization\nIn the ablation study, both of these two innovations show improvements.\nThe strength of this submission is that it does not only present the ideas but also analytically proved the effectiveness.", "weaknesses": "The overall contribution of the paper is solid. The only thing I would suggest is to expend the experiments to 1 more low-level vision task, for example Single-Image Super Resolution, image denoising and so on. Since usually the decoder part of such low-level vision task is computational heavy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses)** *The overall contribution of the paper is solid. The only thing I would suggest is to expend the experiments to 1 more low-level vision task, for example Single-Image Super Resolution, image denoising and so on. Since usually the decoder part of such low-level vision task is computational heavy.*\n\nThank you for your positive feedback and valuable suggestion. We appreciate your recognition of the solid contribution of our paper.\nWe understand the importance of demonstrating the effectiveness of our approach across various vision tasks. Although we are not very familiar with Single-Image Super Resolution and image denoising, we acknowledge that our approach can be highly effective for computationally intensive tasks. \n\nGiven the short rebuttal period, it is challenging to include the experimental results at this stage. However, we will strive to incorporate experiments on these tasks in the final version of the paper. We believe this will provide a more comprehensive evaluation of our method’s performance and computational efficiency.\n\nThank you once again for your insightful feedback. We look forward to enhancing the overall impact of our paper with these additional experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k9JctkRSCp", "reviewer": "Reviewer_gteR", "summary": "The authors propose an easy-to-train supernet, in which you can then adaptively change the network depth by disabling some blocks (skippable sub-paths) and using skip connections instead, while always keeping mandatory sub-paths. They found that the length of mandatory and the skippable sub-paths should be the same, for maximum efficiency in terms of accuracy and computational complexity. \nDuring training there are 2 forward-backward passes:\n1. Training the whole supernet/teacher (FFFF) network as usually for task objectives (ImageNet classificaiton)\n2. Self-distillation intermidiate feature maps from supernet/teacher (FFFF) to the base/student (TTTT) network using KL-divergence-loss (where is base/student (TTTT) is a sub-network of supernet/teacher (FFFF) )\n\nSwitchable LayerNorm or BatchNorm operators are used in mandatory sub-paths.\nThis is the first approach to adaptive networks that provides a general principle and theoretic basis for supporting predictable depth adaptation with minimal performance degradation.", "questions": "Does your approach achieve higher accuracy than  intermediate feature maps distillation: if you train the Supernet first, then freeze it and distill intermediate feature maps from the frozen Supernet to the separate randomly initialized Basenet? So maybe your approach is not only more flexible than intermediate feature maps distillation, but also more accurate?", "limitations": "Training and testing are carried out on different splits of the same dataset, which is an In-domain comparison, while for real-life applications Out-of-domain is much more important, when training and testing on very different datasets, where completely different approaches may be the best.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The resulting base/students (TTTT) network achieve higher accuracy than a separately trained networks with the same flops, depth and  structure.\nThe resulting supernet/teacher (FFFF) network achieves higher accuracy than a separately trained networks with the same flops, depth and  structure.\nIn Figure 4 (b), the proposed approach lies on the Pareto optimality curve, outperforming all other state-of-the-art approaches in terms of accuracy and computational complexity (Flops), except for SpViT-Swin-Ti, which is more accurate and has fewer flops.\nThe approach is very simple and in most cases it shows state of the art results. Controled experiments for different variants of the proposed approach are presented.\nIn Figure 5: (b) this is shown that the inference latecny of the proposed approach drops more than that of the S-ResNet50 approach with the same reduction in computational complexity.\nIt is tested on several widely used networks ResNet50, Swin-T and ViT-b/32, and show that it can be applied to both CNNs and transformers with minimal training effort.\nCompared to methods of distillation, quantization, etc. this method is more flexible when, after training at test time, you can quickly increase the speed (at the expense of a decrease in accuracy) or increase the accuracy (at the expense of a decrease in speed) in a certain range, because at test time, sub-networks with various depths can be selected instantly from a single network.", "weaknesses": "In Figure 4 (b),  SpViT-Swin-Ti is more accurate and has fewer flops than the proposed approach Swin-T-ADN.\n\nThere is no Out-of-Domain (OoD) comparison when very different datasets are used for training and testing, which shows the robustness of the model and the extent to which it can be applied to real-world problems.\nMost of the charts and tables compare the computational complexity (Flops), with the exception of Figure 5(b), while Latency (ms) is more important for real-world problems.\nIt would be better to have as many charts as possible with Out-of-Domain accuracy vs Latency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "* **Weaknesses a)**\n*In Figure 4 (b), SpViT-Swin-Ti is more accurate and has fewer flops than the proposed approach Swin-T-ADN.*\n\nThank you for your positive feedback and valuable suggestion. \n\nWhile our work outperforms many state-of-the-art approaches, some efficient networks, notably SpViT-Swin-Ti, surpass some of our sub-networks on the Pareto front. SpViT-Swin-Ti excels because it better exploits transformer-specific features. For instance, SpViT-Swin-Ti dynamically prunes tokens, significantly reducing computation by eliminating less important tokens.\nIn contrast, our depth adaptation approach is architecture-agnostic, making it applicable to both CNNs and transformers. We believe that our approach and SpViT-Swin-Ti are complementary. For example, SpViT-Swin-Ti could be trained to achieve better accuracy-efficiency trade-offs by skipping some encoder/decoder blocks, as suggested by our work.\n\n* **Weaknesses b)**\n*There is no Out-of-Domain (OoD) comparison ... It would be better to have as many charts as possible with Out-of-Domain accuracy vs Latency.*\n\nOut-of-domain (OoD) generalization in deep learning is crucial for real-world applications, as deployment data often differs from training data. However, achieving effective OoD generalization is challenging, and we are not aware of standardized benchmarks. We greatly appreciate the reviewer’s insights and would be grateful if references or benchmarks could be provided. This would significantly assist us in enhancing our approach for real-world applications.\n\n* **Question)**\n*Does your approach achieve higher accuracy than intermediate feature maps distillation: if you train the Supernet first, then freeze it and distill intermediate feature maps from the frozen Supernet to the separate randomly initialized Basenet? So maybe your approach is not only more flexible than intermediate feature maps distillation, but also more accurate?*\n\nThe reviewer asked about the effectiveness of applying knowledge distillation (KD) from a super-net, such as ResNet50-ADN(FFFF), to a separate ResNet50-Base model.\n\nFirstly, our observations indicate that the naïve application of KD is not effective when using large datasets like ImageNet. In Figure 4(b), we illustrate the impact of applying KD to equivalent networks. Contrary to common belief, the naïve application of KD does not enhance performance. In fact, following the same training schedule (150 epochs for ResNets), KD results in worse performance compared to ordinary training using target labels. For instance, ResNet50-Base (KD individual), which is trained using pytorch pretrained ResNet50 as a teacher, achieves only 73.8% Acc@1, which is 1.2% lower than ResNet50-Base trained without KD. This finding aligns with prior work, such as [52][53][54], which also indicates that achieving positive results with KD on ImageNet is very challenging. To obtain positive outcomes with KD, an extended training schedule and a right combination of teacher/student and optimization techniques are required.\n\nNext, we conducted an additional experiment to investigate the effectiveness of exploiting ResNet50-ADN(FFFF), which was trained using our self-distillation strategy, as a teacher to train a separate ResNet50-Base model. The results show that ResNet50-Base trained using ResNet50-ADN(FFFF) as a teacher achieves 76.30% Acc@1, which is 0.2% higher than our subnetwork ResNet50-ADN (TTTT). \nThis result demonstrates that the right combination of teacher and student is crucial for effective knowledge distillation. We conjecture that since our self-distillation process enforces ResNet50-ADN (FFFF) to produce intermediate features (and logits) compatible with ResNet50-ADN (TTTT), the knowledge was more effectively transferred to ResNet50-Base, which has the same architecture to ResNet50-ADN(TTTT). This is an interesting result and requires further investigation.\n\nThe following table summarizes the evaluation results.\n| Teacher | Student | Student Acc@1 |\n|---|---|---:|\n|ResNet50 (Pytorch pretrained) | ResNet50-Base |  73.8% |  \n|ResNet50-ADN(FFFF) | ResNet50-Base |  76.3% |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

We sincerely thank all reviewers for their positive feedback and constructive suggestions. 
We have made every effort to address each question and suggestion in detail.
Specifically, we conducted three additional experiments to respond to the reviewers’ questions and suggestions.

1.	**Quantifying compactness of $h_{base}$**: To quantify the compactness of $h_{base}$, we measured *cosine similarity* between $h_{base}$ and $h_{super}$ at every stage. The result shows that our models manifest higher similarity between two feature representations. Through this, we can infer that more knowledge was transferred from $h_{super}$ to $h_{base}$ with our self-distillation strategy. 
2.	**Effect of a longer training schedulep**: One reviewer asked whether training a smaller network, such as ResNet50-Base, with a long training schedule [R1] could achieve better results and avoid the need for an adaptive network. To address this question, we applied the PyTorch v2 training script (https://github.com/pytorch/vision/issues/3995) to train ResNet50-ADN and individual networks. The results show that ResNet50-ADN(FFFF) and ResNet50-ADN(TTTT) achieve 80.44% and 78.78% top-1 accuracy on ImageNet-1K, respectively. The equivalent individual networks, ResNet50 and ResNet50-Base, achieve 80.44% and 78.17% top-1 accuracy, respectively. While ResNet50 (FFFF) and ResNet50 perform equally well, our smallest sub-network ResNet50 (TTTT) outperforms the equivalent ResNet50-Base by 0.62%. These results demonstrate that our ResNet50-ADN model benefits from these recent training recipes as well.
3.	**Effectiveness of ResNet50-ADN(FFFF) as a teacher**: We investigated the effectiveness of using ResNet50-ADN(FFFF) as a teacher to train a separate ResNet50-Base model. The results show that ResNet50-Base trained with ResNet50-ADN(FFFF) as a teacher achieves higher accuracy than our subnetwork ResNet50-ADN(TTTT). In contrast, applying knowledge distillation (KD) to train ResNet50-Base using a vanilla ResNet50 as a teacher does not yield positive outcomes. We conjecture that since our self-distillation process enforces ResNet50-ADN(FFFF) to produce features compatible with ResNet50-ADN(TTTT), the knowledge was more effectively transferred to ResNet50-Base.


Despite those efforts, due to the short rebuttal period, some suggestions remain for follow-up work. In particular, we received valuable suggestions about applying our approach to different tasks and domains. We recognize their importance for broader impact and will try to accommodate these suggestions in the final version of the paper and in follow-up works.
We deeply appreciate the reviewers’ efforts and insights.

Best regards,

[R1] Wirghtma, et al, “ResNet strikes back”, 2021

Meta-review: The paper presents valuable contributions to the design of adaptive depth networks and tests the approach on both convolutional and transformer networks. After rebuttal, all the reviewers were positive about the paper, and the AC agreed with the rating.
Decision: Accept (poster)