Specialization of Sub-paths for Adaptive Depth Networks
Authors: Woochul Kang
Venue: Submitted to ICLR 2023 (2023)
Keywords: convolution neural network
anytime network
adaptive network
accuracy-efficiency trade-offs
imagenet
coco

Abstract: We present a novel approach to anytime networks that can control network depths instantly at runtime to provide various accuracy-efficiency trade-offs. While controlling the depth of a network is an effective way to obtain actual inference speed-up, previous adaptive depth networks require either additional intermediate classifiers or decision networks, that are challenging to train properly. Unlike previous approaches, our adaptive depth networks require virtually no architectural changes from baseline networks. Instead, we introduce a novel training method that enforces some sub-paths of the baseline networks to have a special property, with which the sub-paths do not change the semantic level of input features, but only refine them to reduce prediction errors. Those specialized sub-paths can be skipped at test time, if needed, to save computation at marginal loss of prediction accuracy. We first formally present the rationale behind the sub-paths specialization, and based on that, we propose a simple and practical training method to specialize sub-paths for adaptive depth networks. While minimal architectural changes and training efforts are required, we demonstrate that our approach significantly outperforms non-adaptive baselines in various tasks, including ImageNet classification, COCO object detection and instance segmentation. Further, we show that the smallest sub-networks of our adaptive depth networks achieve competitive model compression effect compared to recent state-of-the-art techniques.

Under review as a conference paper at ICLR 2023
SPECIALIZATION OF SUB-PATHS FOR ADAPTIVE DEPTH NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
We present novel approach to anytime networks that can control network depths instantly at runtime to provide various accuracy-efficiency trade-offs. While con- trolling the depth of network an effective way obtain actual inference speed-up, previous adaptive depth networks require either additional intermediate classifiers or decision networks. that are challenging to train properly. Unlike previous approaches, our approach requires virtually no architectural changes from baseline networks. Instead, we propose a training method that enforces some sub- paths of the baseline networks to have special property, with which the sub-paths do not change the level of input features, but only refine them to reduce prediction errors. Those specialized sub-paths can be skipped at test time, if needed, to save computation at marginal loss of prediction accuracy. We first formally present the rationale behind the sub-paths specialization, and based on that, propose a simple and practical training method to specialize sub-paths for adaptive depth networks. Our approach is generally applicable residual networks including both convolution networks and vision transformers. We demonstrate that our approach outperforms non- -on-adaptive baseline residual networks in various tasks, including ImageNet classification, COCO object detection and instance segmentation.
INTRODUCTION
Modern deep neural networks provide state-of-the-arr performance at high computational costs, and, hence, lots of efforts have been made to leverage those inference capabilities in resource-constrained systems, such as autonomous vehicles. Those efforts include compact architectures (Howard et al..
2017, ZhangZ et al. 2018: Han et al., (020). network pruning (Han et al.) 2016: Liu et al. (019). weight/activation quantization (Jacob et al. 2018), knowledge distillation (Hinton et al. 2015). to name a few. However, those approaches provide static accuracy-efficiens trade-offs that are often tailored for worst-case scenarios, and, hence, the lost accuracy cannot be recovered even if more resources become available.
Adaptive networks such as anytime networks (Huang et al. 2018 Yu et al. 2018 Wan et al. 2020
attempt provide runtime adaptability deep neural networks by exploiting the redundancy in either depths or widths, as shown in Figure ] or resolutions (Yang et al., 2020a). Dynamic networks (Wu et al.) 20188 Li al. 2021 2020 Zhu al.] 2021) add additional control logic the backbone network for cepputddpendende adaptation. However, these adaptive networks usually require auxiliary networks, such as intermediate classifiers or decision networks, which are challenging train properly. Further, since adaptive networks have multiple sub-networks, embedded in single neural network, training them incurs potentially conflicting training objectives for the sub-networks, resulting in worse performance than non-adaptive networks (Li et al. 2019).
In this work, we introduce a novel approach to anytime networks that executable in multiple depths to provide instant runtime accuracy-efficeency trade-offs. Unlike previous adaptive depth networks, our approach does not require additional add-on networks or classifiers, and, hence, can be applied to modern residual networks easily. While maintaining the structure of original networks, we train several sub-paths, or a sequence of residual blocks, of the network to have a special property, that preserves the level of input features, and only refines them to reduce prediction errors. At test time, these specialized sub-paths can be skipped, if needed, for efficiency at marginal loss of accuracy as shown in T

---

Under review as a conference paper at ICLR 2023
0000 000000
- 102000
Figure 1: Anytime networks with (left) early-exit branches, (middle) adaptive widths, or channels, and (right) specialized sub-paths (ours). Dashed layers (or blocks) and channels can be skipped for instant accuracy-efficiency trade-offs at runtime.
The proposed sub-paths specialization is achieved by enforcing sub-networks with different depths to produce features with similar distributions for every spatial dimension. In Section []] we formally discuss the rationale behind the sub-paths specialization and introduce a simple and practical training method for sub-paths specialization. In most previous adaptive networks, the total training time linearly proportional the number of supported sub-networks, and resolving potential conflicts between netwoks is an important problem. In contrast, our approach does not try to resolve potential conflicts while jointly training many sub-networks. Instead, our training method exploits only two sub-networks for sub-paths specialization, and, at test time, those specialized sub-patha are exploited selectively to build many sub-networks of various depths. Therefore, the total training time is greater than training two individual networks. Further, our approach with sub-path specialization do not exploit specific properties of convolution neural networks (CNNs) or vision transformers, and, hence, is generally applicable to residual networks, including both CNNs and recent vision transformers.
In Section demonstrate that adaptive depth networks with sub- paths specialization outper- form counterpart individual networks, both in CNNs and vision transformers, and achieve actual inference acceleration in multiple tasks including ImageNet classification, COCO object detection and instance segmentation. To the best of authors' knowledge, this work is the first general approach adaptive networks demonstrating consistent performance improvements for both CNNs and vision transformers.
RELATED WORK
Adaptive Networks:
Anytime networks (Larsson et al. 2017 7 Huang et al.] 2018 Hu et al. 2019
Wan et al. 2020) and dynamic networks (Wu et al. 2018:] Li al. [021] Guo et al. [019:] Li et al.
2020 Yang et al. 2020a) have attracted lots of attention for their runtime adaptability. Most any time
networks have multiple classifiers that are connected to intermediate layers (Huang et al. 2018 Li
et al. Fei et al. 2022). Training multiple classifiers is challenging task and many anytime
networks (Li et 20199/Zaan et al.) [019 Wan al.] 20200] Huang et al., 2018 Hu et al. 2019) exploit knowledge distillation to supervise intermediate classifiers using the last, or the best, classifier.
Slimmable neural networks can adjust channel widths for adaptability and they exploit switchable
batch normalization to handle multiple sub-networks with single shared classifier (Yu et al. 2018
Yu & Huang 2019b). While some dynamic networks (Li et al. 202T) extend anytime networks simply by adding input-conditioned decision gates at branching paths, few dynamic networks (Wu et al. 2018 Veit & Belongie| 2018; Wang et al.) 20188)) extend residual networks by applying block-level decision gates that determine if the block can be skipped. The latter approach is based on the thought that some blocks can be skipped on easy inputs. However, in these dynamic networks with adaptive depths, no formal explanation has been given why some blocks can be skipped for given input. Therefore, users have no control over the depth of the sub-networks.
Since the introduction of ResNets (He et al. (0166), residual blocks with shortcuts have received extensive attention because of their ability to train very deep networks, and have been chosen by many recent deep neural networks (Sandler et al., 2018 Tan & Le 2019 Vaswani et al. 2017). Veit et al. (2016) argue that identity shortcuts make exponential paths and results in an ensemble of shallower networks. This thought is supported by the fact that removing individual residual blocks at test time does not significantly affect performance (Huang et al.) 2016
Residual Blocks with Shortcuts:
Xie et al. 2020). Other works argue that identity shortcuts enable residual blocks to perform iterative feature refinement, where each block improves slightly but keeps the semantic of the representation of the previous layer (Greff et al. 20166) Jastrzebski et al. 20188). Our work build upon those views on residual blocks with shortcuts. We further extend them for adaptive depth networks by introducing
2

---

Under review as a conference paper at ICLR 2023
novel training method that exploits the properties of residual blocks more explicitly for sub sub-paths specialization.
SUB-PATHS SPECIALIZATION FOR ADAPTIVE DEPTH NETWORKS
In this section, we first formally discuss the rationale behind the sub-paths specialization. Then, the details of training of sub-paths specialization is discussed.
3.1 MOTIVATION AND OVERVIEW
In typical residual networks such as ResNets, the s-th residual stage consisted of L identical residual blocks, which transform input features ha additively and produce the output features h" as shown in Equation [ While block with a residual function F learns higher level features as traditional compositional networks (Simonyan Zisserman 2015), previous literature (Jastrzebski et al. 2018 Greff et al. 2016) demonstrates that residual function also tend to learn a fuuccion that reñines already learned features at the same feature level. The first and the second functions, respectively, are called feature learning and feature refinement. If residual block mostly perform feature refinement while not changing the level of features, the performance of the residual network is not significantly affected by dropping the block at test time (Huang et al. 2016 Xie et al. 2020). However, in typical residual networks, most residual blocks tend to learn both functions, and, hence, random dropping of residual blocks at test time degrades the performance significantly. If some residual blocks can be trained to focus on one function while the remaining blocks are encouraged to focus the other function, then we can safely skip blocks specialized to feature refinement computation at marginal loss of prediction accuracy, needed for efficiency.
featuue
To this end, in approach, residual functions of residual stage are partitioned into two sub-paths, Frsse and F skiippatee. as in Equation I and, during training, they are encouraged to focus more on respective functions, i.e., learning new level features and refining the learned features.
h--1 + Fi F2(hi) + +... + Fx (bx-1) 1)+ Fk+1(hk) + + Fr = h super (1)
base
Fakippable
Since the last half residual functions, or F skippabbee is supposed to preserve the level of input features, F ekkippable be skipped for efficiency and the intermediate features, himser can be used as the input to the next stage. When the higher inference accuracy is required, F kipppabee can be applied to hosse produce the same level, either hi or h puper stage with little change in the feature distribution.
refined features h Super Since the feature representations hāose and h super be provided an input h² to the next (s 1)-th residual
Figure 2 shows residual stage, in which the last half blocks are specialized preserve the level of input features. Since each such residual stage provides two alternative internal paths, if there are N. residual stages with specialized sub-paths, 2%V sub-networks with different accuracy-efffciency trade-offs become available at test time (Section 4.3)
3.2 SUB-PATHS SPECIALIZATION
Preserving the level of input features implies that two features representations hiase and h. happer have similar distributions over input X. We can enforce this during training by minimizing the Kullback-I eibber (KI.) divergence berween the two feature representations:
DKL (haperr ( +
It is noteworthy that enforcing Equation 2 also has the distillation effect of transferring the knowledge from h hupeer to h h Therefore, at each residual stage, he is expected to learn more compact representation from h Super- As a consequence of Equation we can conjecture that residual functions in F skipppable trained have small magnitude:
Exex [K++1 (h;c) +... Fz (h, -
(3)
3

---

Under review as a conference paper at ICLR 2023
skip=Truue
h
F.
Fk
h
F
K+1
h
bane
Ssspor
Shared sub-path (mandatory)
Specilaized sub-path (skippable)
Figure 2: Illustration of a residual stage with specialized sub-path. A residual stage is split into two sub-paths. While the first (blue) sub-path mandatory for sub-networks, the last (yellow) sub-path can be skipped. During training, the last sub-path is encouraged to refine input feature representation see while minimally changing its distribution. When the specialized sub-path is skipped, different set of batch normalization operators, called skip-aware BNs, are used in the shared sub-path. The shortcuts within residual blocks are not shown for brevity.
Then, what do the residual functions in skippable learn during training? This can be further investi- gated through Taylor expansion (Jastrzebski et al. 2018). Any loss function L used for training can be approximated with Taylor expansion as follows:
((per) = [(h + (k+1(hx) + +... + Fh-1(h;,22) FL (hū
(4)
Oc(h2-1) Oh
[haase + (k+1(bx) +...
+...+ + F2-11hh FL (h)
+ O-)
(5)
Oc(h3-1)
2 =L(h +
(6)
Th
j=K-1
In Equations 6 the loss function is iteratively expanded around h (j= K +1,.,, L). In
Equation note that only the first order term F(h) @C(h)/Oh is left and all high order terms, such as F2 (h) (h)) are absored in ((). These high order terms in O()) can be safely ignored in final Eguation 6 since Fs in F Eskippable have small magnitudes, as shown in Equation B Thus, in Equation minimizing the loss L during training optimizes F; (j = K + 1,... L) in the half space to minimize the dot product between F3 and ab 1), This implics that every residual 8h 0
of 0c
Sh
function in F skippable is optimized to learn a function that has similar effect to gradient descent:
(y ~ @c(h)-1
(j = kü=K,,,...
conjecture that residual functions in F skipppabee (1) minimally change the distribution of input features hy but (2) refine them iteratively minimize the loss for better inference accuracy. In other words, the skippable sub-path is specialized for refining input features for better inference accuracy while preserving its feature level. Therefore, F skippaale or the specialized residual blocks, can be skipped without causing much changes of the feature distribution to the next residual stages (Sections and 4.4).
Considering these results in Equations 2 and together,
3.3 TRAINING ADAPTIVE DEPTH NETWORKS
To enforce Equation we propose a training method shown in Algorithm I in which two sub- networks, or the super-net and the base-net, are exploited in single training loop for sub-paths specialization. The super-net and the base-net are the largest and the smallest sub-networks, respec- tively, of the proposed adaptive depth network M. For example, the base-net skips every specialized sub-paths in M. In contrast, the super-net executes all residual blocks of M without skipping.
In steps 5 and 6, the forward passes of the super-net and the base-net are executed for the same input ac. The second argument, skip, of the model M indicates in which residual stage its specialized sub-path is skipped. In ResNets with 4 residual stages, 16(= 24) sub-networks become available by varying the skip argument. For example, we choose the base-net of a ResNet by setting the second argument to skip=/Trrue, True, True, True]. During the training, wrapper function is used to obtain intermediate features from every residual stage as well as the outputs of the sub-networks. For instance, during the forward pass of the base-net, intermediate features from N. residual stages, or N. l and output ÿtase are obtained.
bhase baeeet -r.
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Training with sub-paths specialization. N₂ is the number of residual stages,
1: Initialize an adaptive depth network M
2: for iile.., do
3: Get next mini-batch of data a: and label y
4: optimizer.zero_pee _rrrd_)grr(
5: [øsuper Dsueer] M (a, skip=/False, ... False])
>> Forward pass for the super-net of M
6: (ybase, bbase] = (xx skip=/True, -ruue)
b. Forward pass for the base-net of M
7: loss criterion supeer) +
(1-a){Es= (1
Dxr. (b (upper [ ) D D (Qsuper Dbase) )
8: loss. )
9: optimizer.stepp
10: end for
In step 7, we modify the loss by adding Kullback Leibler (KL) divergence between haase and Rsupers given Equation 2
as an regularization term:
Ns Σ
loss a criterionly, Dsuper) +
D a) DEL (h (haaae) + Dx (Dsuper (base)
(8)
D Juper and Phase are included in the regularization term since they are last features from the sub- networks. The hyperparameter a controls the strength of the regularization term.
explicitly trained either jointly sequentially (Yu & Huang 2019b), and, hence, the total training time is proportional to the number of supported sub- networks. In contrast, our approach does not train all sub-networks jointly sequentially. Instead, in Algorithm D only two sub-networks are exploited during training for sub-paths specialization, and, hence, the total training time is greater than training sub-networks individually. However, test time, the specialized sub-paths can be exploited selectively to construct 2%, sub-networks of various depths. We demonstrate this result in Section 4.3
In previous anytime networks, all sub-networks
3.4 SKIP-AWARE BATCH NORMALIZATION FOR SHARED SUB-PATHS
In our adaptive depth networks, each residual stage expects the distribution of input features is not affected by the choices of internal paths its previous residual stages. However, the shared sub-paths inside each residual stage still need adaptability to handle different sub-networks. Originally, batch normalization (BN) (Ioffe & Szegedy (2015) was proposed to handle internal covariate shift during training non-adee -daattaeive networks by normalizing features. In adaptive depth networks, however, internal covariate shifts occur in shared sub-paths when different sub-networks are used. To handle this internal covariate shifts, switchable BN operators, called skip-aware BNs, used in shared sub-paths. For example, at each residual stage, different BN operators are used by the layers in the shared sub-path when specialized sub-path skipped. The effectiveness switchable BNs has been demonstrated in neural networks with adaptive widths (Yu & Huang 2019a) and adaptive resolutions (Zhu al. 202T), and we apply them for our adaptive depth networks. The amount of parameters for skip-aware BNs is negligible. In ResNet50, akip-aware BNs increase the parameters by 0.07%.
4 EXPERIMENTS
To demonstrate the effectiveness of our approach, we conduct experiments on ImageNet classification and COCO object detection and instance segmentation benchmarks. Three representative residual networks both from CNNs and vision transformers are chosen as base models; MobileNet V2 (Sandler) al. (2018) is lightweight CNN model, ResNet (He et al. 2016) is a larger CNN model, and ViT (Dosovitskiy et al. 2021) is representative vision transformer. For CNN models, every residual stage is split equally into two sub-paths and its second sub-path is skippable. Since the vision transformer (ViT-b) has residual stages, 12 consecutive encoder blocks are divided into 4 groups, resembling CNN models, and the last encoder block of each group is used as skippable sub-path. Since vision transformers exploit layer normalization instead of batch normalization (Yao et al. (021)) we apply switchable layer normalization operators in shared sub- nat- instead of switchable BNs. During training, every skippable sub-path is specialized according to Algorithm T. We use '-ADN' to
5

---

Under review as a conference paper at ICLR 2023
denote our adaptive depth networks. Since our adaptive depth networks have many parameter-sharre sub-networks in a single model, we indicate which sub-network is used for experiments in parenthesis, e.g., -ADN (super-net). For our sub-networks, boolean values in the parenthesis, e.g., (TTFF), also used to indicate which residual stages the sub-path is skipped. For example, -ADN (base-net) equivalent to -ADN (TTTT).
4.1 IMAGENET CLASSIFICATION
We evaluate our method on ILSVRC2012 dataset (Russakovsky et al. 2015) that has 1000 classes. The dataset consists of 1. M.28M training and 50K validation images. For CNN models, we follow most training settings in the original papers (He et al.) 2 Sandler et al., 2018), except that ResNet models are trained for 150 epochs. For ViT-b, we follow DelT's training recipe (Touvron al., 2020; pyt 2022). The hyperparameter is set to 0.5 for all models. For fair comparison, individual networks are trained in the same training settings of their corresponding adaptive depth networks.
Table 1:
Results on ImageNet. Models with '-Base' have the same depth the base-net of corre- sponding adaptive depth network. Latency is measured on an RTX 3090 device (batch size = 64).
Baseline
Model
Params
FLOPs
Acc@ 1
Acc@
Latency
ResNet50-ADD (super-net)
4.11G
77.6%
93.7%
43.2ms
25.58M
ResNet50-ADN (base-net)
2.58G
76.1%
93.2%
27.6ms
ResNet50 (individual network)
25.56M
4.11G
76.7%
93.2%
43.2ms
ResNet50- Bass (individual network)
17.11M
2.58G
75.0%
92.3%
27.6ms
Early-exiting branches
4.11G
75.2%
43.2ms
ResNet50
Zhhng et al. 2019 2022
28.82M
3.52G
72.8%
39.0ms
ei al. 2019
2.05G
58.5%
27.4m
Slimmable widths
4.1G
76.0%
43.2ms
1.0x, 0.75x, 0.5x widths
25.6M
2.0G
75.6%
33.8ms
Yu ct al. 2018 Yu & Huang
2019a)
1.0G
74.0%
21.2ms
MV2-ADN (super-net)
0.32G
72.7%
90.8%
15.5ms
3.72M
MV2-ADN (base-net)
0.25G
70.7%
89.8%
13.2ms
MobileNet
MV2 (individual network)
3.50M
0.32G
72.1%
90.3%
15.5ms
V2
MV2-Base (individual network)
2.99M
0,25G
70.7%
89.6%
13.2ms
MutualNet (Yagg et al. 2020b)
0.32G
72.9%
AlphaNet-0.75x (Wang al. 2021a
0.21G
70.5%
ViT-b/32-ADN (super-net)
2.95G
76.4%
92.7%
39.5ms
88,25M
ViT-h/32-ADN (base-net)
2.00G
73.2%
91.4%
26.7ms
VIT-b/32
ViT-h/32 (individual network)
88.22M
2.95G
75.9 %
92.5%
39.5ms
ViT-b/32-Base (individual network)
69.33M
2,00G
73.1%
90.8%
26.7ms
The results in Table show that our adaptive depth networks consistently outperform individual counterpart networks even though many sub-networks share parameters in single network. We conjecture that these improvements result from distilling the knowledge from h h'uper to h base at each
residual stage and the iterative refinement at the skippable sub-paths, shown in Equation 7 In Table I ResNet50-ADD ADN is also compared with two representative anytime networks (Zhang et al. 2019 Yu &
Huang 2019ab that can control the depths and the widths, respectively. Our adaptive depth networks achieve better performance than these competitors. In particular, unlike competitors, the reduction of FLOPs of ResNet-ADN is closely translated into actual acceleration. For instance, when the FLOPs of ResNet50-AAD (base-net) is reduced by 37.3%, its inference latency is also similarly reduced by 36.2%. In contrast, although S-ResNet50 (0.75x) requires 22% less FL.OPs than ResNet50-ADD. (base-net). it shows 22.4% longer inference latency in practice. Since S N-ResNet5O's network depth is not changed by reducing widths, it still needs to perform similar number of GPU kernel invocations in practice. In early-exiting anytime networks, effective depths of network can be reduced. But as shown in Zhang et al. (2019)* 's work, sub-networks of early-exiting networks manifest significantly low inference accuracy since they only exploit low-level features. In MobileNetV2, our MV2-ADN compared with recent adaptive networks that control network widths and resolutions. While these networks exploit recent training techniques such AutoAugment (Cubuk al. 2019) and input

---

Under review as a conference paper at ICLR 2023
resolution variations (Wang al. a MV2-ADN achieves similar performance without such bells and whistles. Finally, our result with ViT-b/32-ADN demonstrates that our approach is generally applicable to residual networks including vision transformers.
Table 2: Comparison of state-of-the-art efficient inference methods for ResNet 50 on ImageNet. denotes static pruning methods, denotes new filter design, denotes dynamic networks.
Model
Params
FLOPs
JFLOPs
Acc@)
Acc@5
(Liehenwein et al. 2020
20.9M
3.7G
10%
75.9%
92.8%
Versatile- Wang
201 8b
11.0M
3.0G
27%
74.5%
91.8%
Dynamic Slimmable (Li
3.1G
24%
76.6%
ResNet50-ADN (TTFF)
25.6M
3.4G
17%
77.0%
93.4%
GReg-1 1 Wang et al. 2021b)t
2.8G
33%
76.1%
Chost-ResNeiJO-22 (Han et al.
2020
13.0M
2.2G
46%
75.0%
92.3%
DR DR-ResNet50 (a=2.0) Zhu et al. 2021
30.5M
2.3G
44%
75.3%
92.2%
ResNet50-AADN (base-net)
25.6M
2.6G
37%
76.1%
93.2%
In Table two sub-networks of ResNet50-ADN compared with several state-of-the-ard efficient inference methods. Both sub-networks ResNet50- ADN achieve very competitive results compared to many oftte-of-thh-arr methods. Some of these sate--of-the-arr methods require unusual training techniques to provide accuracy-efficiency trade-offs. For example, in dynamic slimmable networks (Li et 202T). sub-networks are randomly sampled at every batch iteration until all sub-networks are converged. In contrast, our approach exploits only two sub-networks, or super-net and base-net, during training according Algorithm [ Further, unlike static pruning methods, since the sub- networks of our approach share parameters in a single model, they can adapt their depths instantly at runtime. The result with ResNet50-ADA (base-net) also demonstrates that non-skippable sub-paths of our adaptive depth networks learn compact feature representations effectively while skippable sub-paths are enforced to preserve the level input features during training.
4.2
ABLATION STUDY
Table 3: Ablation analysis with ResNet50-ADN, Applied components are checked with / mark.
Sub-paths specialization
Skip-aware BNs
Acc@
Super-net net
Base-net nese-net
75.2% (4 1.5%)
72.2% () 2.8%)
76.1% (L 0.6%)
74.9 % (4 0.1%)
76.6% (L 0.1%)
75.1% (+ 0.1%)
V
77.6% (f 0.9%)
76.1% 1% (+ 1.1%)
We conduct ablation study on ImageNet classification to investigate the influence of two key com- ponents of the proposed adaptive depth networks: (1) sub-paths specialization and (2) skip-aware BNs. When our sub-path specialization is not applied, the loss in Algorithm is modified to loss d (criterion(y) @super) + criterion(y, ÿbase)). The results are shown Table When neither of them is applied, the inference accuracy of the super-net and the base-net is significantly lower than that of individual networks by 1.5% and 2.8%, respectively. This result shows the difficulty of training multiple sub-netwe nub-networks in anytime networks. During training, uncoordinated training sig- nals from the super-net and the base-net decrease the effect of training, resulting in worse performance than individual networks. When one of the two components is applied individually, the performance still slightly worse than individual networks'. Finally, when both aub-paths specialization and skip-awa aware BNs applied together, ResNet50-ADN achieves significantly better performance than individual networks, both in the super-net and the base-net. This result demonstrates that both sub-paths specialization and skip-aware BNs essential for the proposed adaptive depth networks.
7

---

Under review as a conference paper at ICLR 2023
Dist blw. BN mears
77.5
76.5
75.3
(Supernet)
(Wase-net)
15
25
sub networks of varying depths
shared layers
(a) Acc@ various sub-networks
(b) Distances btw. skip-aware BNs in shared sub-paths
Figure 3: (a) Performance while varying the depth of ResNe150-ADN, The labels under each bar boolean values indicating the residual stages that skip their specialized sub-path. (b) The distances between the means of skip-aware BNs in the shared sub sub-paths of ResNet50-ADN. Bars with different colors belong to different residual stages.
Input Image
block
biock #2
Block 43
block x4
block #5
Block #6
(a) Original ResNet50
(b) ResNe150-ADN (super-net)
(c) ResNet50-ADN (base-net)
Figure 4: Class Activation Map images of the 3rd stages of ResNet50s. (a) Original ResNet50's activation regions change gradually across all blocks, indicating that every block performs similar functions. (b) In ResNet50-ADN (super), the non-skippable first 3 blocks have extensive hot activation regions, implying active learning of new level features. In contrast, the skippable last blocks have far less activation regions and they are gradually refined around the target. (c) The base-net' net's final activation map is very similar to the super-net's, but has less hot activations around the target object.
4.3 VARYING NETWORK DEPTHS AT TEST TIME
One important advantage of sub-paths specialization is that each residual stage's choice of internal paths does not affect the following residual stages. Therefore, each residual stage can independently make decision about skipping its specialized sub-path. This implies that our adaptive depth networks can support 2. sub-networks at test time, when N, residual stages are given. Figure (+((a) shows the performance of ResNet50-ADM when its depth is varied test time. Among them, only the super-net and the base-net are used for training in Algorithm [ The other sub-networks are built by applying different combinations of skipping decisions at test time. Although these sub-networks are not trained explicitly. their performance does not drop significantly. Instead, they show gradual degradation of performance as the depth of sub-networks becomes gradually shallower.
4.4 ANALYSIS OF SUB-PATHS SPECIALIZATION
The premise of sub-paths specialization is that the feature distribution from each residual stage is not much affected whether its specialized sub-path skipped or not; i.c, Dx (b h (uperr - 0. We
verify this by inspecting the input distribution of the next residual stages. Figure (b)b) shows the
8

---

Under review as a conference paper at ICLR 2023
distances between the means of switchable BNs in the shared sub- of ResNet50-ADN, In Figure ()))), the distance at every residual stage's 's first layer is close to zero. This demonstrates that their previous residual stages produce feature distributions of the same means regardless of whether the specialized sub-path is skipped or not. Figure (b) also shows that the shared layers except the first layers of each residual stage have very different input distributions, demonstrating the necessity of skip-aware BNs in the shared sub-paths.
To further verify that the non-skippable sub-paths and the skippable sub-paths, respectively, focus more on learning new level features and refining learned features. we visualize the activation of 3rd residual stage of ResNet50-ADN using Grad-CAM (Selvaraju et al. (017)) in Figure 4 The 3rd residual stage of ResNet50-ADM has six residual blocks and the last three blocks are skippable. In Figure (a), the activation regions of original ResNet50 changes gradually across all consecutive blocks. This implies that all blocks are involved in gradual learning of new level features well as the refinement of the learned features. In contrast, in Figure Fig ResNet50-ADD manifests very different progressions of activation regions in two sub-paths. In the first three residual blocks, we can observe lots of hot activation regions in wide areas, suggesting active learning of new level features. In contrast, significantly less activation regions are found in the skippable last three blocks and they are gradually concentrated around the target object, demonstrating the refinement of learned features. Further, in Figure (c), we can observe that the final activation map of the ResNe150 (base-net) is very similar to the super-net's final activation map in (b), implying that they are the same feature level suggested in Equation 2 However, the final activation map of the base-net has less hot activation regions around the target object than super-net's, potentially resulting in lower inference accuracy than super-net's more refined features.
(super-net)
4.5 OBJECT DETECTION AND INSTANCE SEGMENTATION
Table 4: Object detection and instance segmentation results on MS COCO dataset.
Individual Networks
ResNet-ADN (ours)
Detector
Backbone
FLOPs
Box AP
Mask AP
Box AP
Mask AP
Faster- RCNN
ResNet50
207.07G
36.4
37.8
(Ren et al.
2017)
ResNet50-Base
175.66G
32.4
34.0
Mask RCNN
ResNet50
260.14G
37,2
34.1
38.3
34.1
He et al.
2017)
ResNet50-Base
228.73G
32.7
29.9
34.1
31.2
In order to investigate the generalization ability of our approach, we use MS COCO 2017 datasets on object detection and instance segmentation tasks using representative detectors. We compare individual ResNets and our adaptive depth ResNets-ADN as backbone networks of the detectors. For training of detectors, intermediate features he and h hagzer (s
use Algorithm with slight adaptation. For object detection, the
1..N can be obtained directly from backbone network's s
feature pyramid networks (FPN) (Lin al. 2017) and, hence, wrapper function is not required to extract intermediate features. All networks are trained on trainnn0 2017 for 12 epochs from ImageNet pretrained weights, following the training settings suggested in (Lin al. (017). Table 4 shows the results on vall va1201 222107 containing 5000 images. Our adaptive depth backbone networks significantly outperform individual static backbone networks in terms of COCO's s standard metric AP.
CONCLUSIONS
We present a novel approach to adaptive depth networks that is generally applicable residual networks. including both convolution networks and vision transformers. Unlike previous approaches, our adaptive depth networks do not require auxiliary add-on networks or classifiers to control network depths. Instead, our approach train sub-pathe of a single network to have special property, with which the sub-paths can be skipped for efficiency at marginal loss of prediction accuracy. We discuss the theoretic rationale behind this sub-paths specialization, and present a simple and practical training method. While our training method uses only two sub-networks, it makes various sub-networks with different depths available at test time. Through extensive experiments with both convolution networks and vision transformers, we demonstrate that the proposed adaptive depth networks outperform individual static networks while achieving actual inference speed-up.
9

---

Under review as a conference paper at ICLR 2023
6 REPRODUCIBILITY
For reproducibility, we provide source codes and the links to the pretrained weights as the supple- mentary material. Code and models will be available in authors' public repository. All programs for training and evaluation run on PCs equipped with two or four Nvidia RTX 3090 GPUs and an Intel i9-10900X CPU @3.7GHz..
REFERENCES
Image
classification
reference
training
scripts 2022.
in
PyTorch,
Ekin D. Cubuk. Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V. Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
Zhengcong Fei, Xu Yan, Shuhui Wang, and Qi Tian. Deecap: Dynamic early exiting for efficient image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12216-12226, June 2022.
Klaus Greff, Rupesh K Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. In International Conference on Learning Representations (ICLR), 2016.
Q. Guo, Z. Yu, Y. Wu, Liang, Qin, and Yan. Dynamic recursive neural network. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 5142-5151, 2019.
Kai Han, Yunhe Wang. Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruming, trained quantization and huffman coding. In 4th International Conference Learning Representations (ICLR), 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint (XX.::1503005311 2015.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: / 704.04861, 2017.
Hanzhang Hu, Debadeepta Dey, Martial Hebert, and J Andrew Bagnell. Learning anytime predictions in neural networks via adaptive loss balancing. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3812-3821, 2019.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference computer vision, pp. 646-661. Springer, 2016.
10

---

Under review as a conference paper at ICLR 2023
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens der Maaten, and Kilian Weinberger. Multi-scale dense networks for resource efficient image classification. International Conference on Learning Representations, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetii-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pultern Recognition (CVPR), June 2018.
Stanislaw Jastrzebski, Devansh Arpit. Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. In International Conference on Learning Representations, 2018.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals, 2017.
Changlin Li, Guangrun Wang. Bing Wang, Xiaodan Liang, Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8607-8617, 2021.
Hao Li, Hong Zhang, Xiaojuan Qi, Ruigang Yang, and Gao Huang. Improved techniques for training adaptive deep networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 189 1-190,, 2019.
Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang, and Jian Sun. Learning dynamic routing for semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Lucas Liehenwein, Cenk Baykal, Harry Lang, Dan Feldman, and Daniela Rus. Provable filter pruning for efficient neural networks. In International Conference on Learning Representations, 2020. URL htttts:://ppeeeeeeeeet
/ forum? -it.dddd-BxxxxDDDD
Tsung- Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo. Xin Yang, Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3296-3305, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, Jun 2017.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Kann-CChihh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4510 2018.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEFE international conference on computer vision, pp. 618-626 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3th International Conference Learning Representations (ICLR), 2015.
11

---

Under review as a conference paper at ICLR 2023
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6114- PMLR, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data- image transformers & distillation through attention, 2020. URL
https:/ /tappss://axiv..../// org/ abs/ 2012. 12877
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, V. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, volume 30, 2017.
Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-18, 2018.
Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances in neural information processing systems, 29:550-558, 2016.
Chengcheng Wan, Henry Hoffmann, Shan Lu, and Michael Maire. Orthogonalized sgd and nested architectures for anytime neural networks. In International Conference on Machine Learning, pp. 9807-9817. PMLR, 2020.
Dilin Wang, Chengyue Gong, Meng Li, Qiang Liu, and Vikas Chandra. Alphanet: Improved training of supernets with alpha-divergence. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10760-10771, PMLR, 18-24 Jul 2021a.
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In International Conference on Learning Representations, 2021b. URL httpttts//opeeeeiiieee
net feet/Eozen wtw.eoorumiiid..im.
Is nPA
Xin Wang. Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the European Conference Computer Vision (ECCV), pp. 409-424, 2a.
Yunhe Wang, Chang Xu, XU Chunjing, Chao Xu, and Dacheng Tao. Learning versatile filters for efficient convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 1608-1618, 2018b.
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths residual networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8817-8826. 2018.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687-10698, 2020.
Le Yang. Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang. Resolution adaptive networks for efficient inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020a.
Taojiannan Yang, Sijie Zhu, Chen Chen, Shen Yan, Mi Zhang, and Andrew Willis. Mutualnet: Adaptive convnet via mutual learning from network width and resolution. In European Conference on Computer Vision pp. 299-315. Springer, 2020b.
Zhuliang Yao, Yue Cao, Yutong Lin, Ze Liu, Zheng Zhang, and Han Hu. Leveraging batch normal- ization for vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 413-422, 2021. doi: 10.
Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers, 2019a.
12

---

Under review as a conference paper at ICLR 2023
Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1803-1811, 2019b.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference on Learning Representations, 2018.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation, In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3713-3722, 2019,
Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self-distillatio:: Towards efficient and compact neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4388- 4403, 2022. doi: 10. N0.1099/PAMM1.2013336111
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848-6856, 2018
Mingjian Zhu, Kai Han, Enhua Wu, Qiulin Zhang, Ying Nie, Zhenzhong Lan, and Yunhe Wang. Dynamic resolution network. In Advances in Neural Information Processing Systems, volume 35, pp. 10985-10998, 2021.
13


Reviews:
[{"review_id": "fcsJCcnFE34", "reviewer": "Reviewer_qCMH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work develops a method to reduce computational costs of a deep residual network without substantial detriment to its prediction accuracy. Specifically, it considers the second half of the layers in each residual block as disposable, i.e. one could skip those layers at inference time to reduce the runtime. To demonstrate this, the work experiments with minimising the KL-divergence between the output of a residual block and the features produced only by the first half of the layers. The experiments on image classification, object detection and instance segmentation indicate that network efficiency can be almost doubled only at a moderate loss in the prediction accuracy.", "summary_of_the_review": "This work achieves reduction in model efficiency with tolerable loss in accuracy. These results are encouraging. However, the experimental protocol requires more elaboration, much like the technical aspects of the presentation.", "strength_and_weaknesses": "Strengths \n- Parameter redundancy in neural networks is well known empirically, and it is certainly a natural idea to exploit it by learning to skip groups of subsequent layers.\n- The implementation of the approach seems sufficiently simple.\n- There is some effort in providing formal justification to motivate the approach (Eq. 2-5).\n- The scope of the experiments is good and encompasses classification, instance segmentation and object detection in addition to the ablation study.\n\nWeaknesses\n\n- Overall, the demonstrated empirical improvements are very incremental. For example, on image classification they are somewhat comparable to Wang et al. 2021 (c.f. Tab. 2).\n- The baseline does not seem sufficiently strong, in fact 2-3% lower than that reported in the original work (He et al., 2015). A short training schedule may actually disadvantage the baseline, since it has a larger number of parameters. (For the proposed method the convergence may happen faster, since due to the KL-divergence the number of training parameters is implicitly reduced).\n- Isn’t the “super-net” the exact copy of the baseline, but with additional KL-divergence term? Why would the baseline have lower classification accuracy?\n- There is generally lack of technical rigour which leaves some room for ambiguity and impedes the flow of the presentation. For example, Eq. 2-5 equate the expectation of an n-dimensional vector with a scalar value, which does not seem meaningful. Moreover, the implementation of these constraints (if one follows them somewhat intuitively) is specifically with the KL-divergence. I believe the equations should reflect that. Another example is Eq. (5): is it assumed to follow from Eq. (4), or is it a standalone assumption?\n\nSomewhat minor:\n- Inconsistent notation, e.g. h_super in Eq. 1 (with or without parenthesis?); same for h_base.\n- Eq. 7 seems sloppy (what is in O(.), partial derivative vs. total derivative).\n- the special treatment of the 23 blocks in ResNet-101 seems to be rather contrived.\n- I do not see much support in the KL-divergence to \"preserve the semantic level of input features\".\n\nFurther notes:\n- Table 1: Please provide details of the evaluation protocol, e.g. what data and criterion was used to select the best model for evaluation?\n- Fig 4c does not appear very convincing: the features learned by the base network are somewhat reminiscent of the “super-net”.\n- Table 3 unclear: if neither of the two components are used, this should be the baseline setting, shouldn’t it?\n- The implementation seems to use a hyperparater T set to 4 in the KL-divergence terms. It is not mentioned in the paper. What is its use?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_Q60B_Pbr9", "reviewer": "Reviewer_zGdd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to achieve anytime networks by controlling the network depth during runtime. The method focuses on the typical ResNet-like architectures and proposes to divide the residual blocks in each stage into two parts that are responsible for feature learning and feature refinement, respectively. A new training method is proposed to achieve the above goal, which is implemented as minimizing the difference between the super network and the base network in both final predictions and pooled intermediate features. Experimental results show that the proposed can achieve better trade-offs than the baselines and previous works.\n", "summary_of_the_review": "This paper presents a new method to achieve controllable depth during runtime. However, the proposed method highly resembles previous work AlphaNet. Besides, the lack of experiments makes the paper not very convincing. As a result, I lean towards rejecting this paper.", "strength_and_weaknesses": "Strengths:\n\n- The discussion on the two primary functions of the residual blocks is interesting.\n- The paper is well-organized and easy to follow.\n\nWeaknesses:\n- The novelty of this paper is questioned. The overall design highly resembles previous work like AlphaNet[1]. The base network in this paper is actually a simplified version of the sampled sub-network in AlphaNet. Besides, while AlphaNet considers multiple scaling dimensions including network depth and width, this paper only considers network depth. As for the training method, both the AlphaNet and this paper adopt KL divergence to minimize the gap between the sub-network and the super-network. \n\n- Some of the theoretical parts lack insight and are somewhat redundant. For example, by some simple substitution and simplification, one can easily show that Equation (2)-(4) are duplicated. Besides, Equation (6)-(8) are just trivial Taylor expansion of the loss function, which is not strongly related to the proposed method.\n\n- The experiments are not sufficient. According to the paper, the method can be applied to any architectures that are built with residual blocks, including the recent prevailing Vision Transformers [2]. I highly recommend that the authors should perform more experiments using ViT as their baseline to show the effectiveness of the proposed method.\n \n\n[1] Wang, Dilin, et al. \"AlphaNet: improved training of supernets with alpha-divergence.\" International Conference on Machine Learning. PMLR, 2021\n\n[2] Dosovitskiy, Alexey, et al. \"An image is worth 16x16 words: Transformers for image recognition at scale.\" arXiv preprint arXiv:2010.11929 (2020).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "014Nkbu_9H", "reviewer": "Reviewer_ipWP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a method to train a network that can run at different depth at test time. The author claimed that a residual block could be considered to perform two functions (1) learning new features (2) refine features. The refinement stage/layers do not change feature semantics thus can be skipped. The proposed method is evaluated on multiple backbones and is demonstrated to achieve better performance than individually trained networks.", "summary_of_the_review": "I am not fully convinced by the claims made in the paper. The experimental evaluation is also not complete and missing some references.", "strength_and_weaknesses": "Strength\n1. The paper showed another way to achieve adaptive depth network, that is dropping layers in every stage rather than just the last stages/layers as in previous works.\n2. The method achieves better performance than individually trained networks.\n\nWeakness\n1. I am not fully convinced by the claim that 'A block with a residual function F is supposed to perform two primary functions (1) learning new higher level features and (2) refining already learned features at the same semantic level'. The paper didn't provide many evidences. The only one maybe Fig.4. But to me, in Fig 4(b), I think the first three blocks didn't change much which the 3rd, 4th, 5th blocks changed a lot, which is against the author's claim.\n2. In Table 1, the author only compare the proposed method with some very early works such as slimmable networks. I suggest the author to add some more recent works such as [1, 2]. It seems that the proposed method didn't outperform these works.\n3. How did the author build the baseline models ResNet29 and ResNet35. These models are not used in the original paper.\n4. The training cost should also be reported since it needs multiple forward pass.\n5. I am not sure I understand how the proposed method works. The loss KL(h_{super}|h_{base}) will pull these two features close. The minimum loss will be reached when the skippable layers are optimized to be zero mapping. Thus I am not sure how the network learns.\n\nRefereces\n\n[1] Li, Changlin, et al. \"Dynamic slimmable network.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[2] Yang, Taojiannan, et al. \"MutualNet: Adaptive ConvNet via Mutual Learning from Different Model Configurations.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-HY4uBSY0i-", "reviewer": "Reviewer_X8ta", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel adaptive depth network by inducing a training strategy without the need of additional intermediate gate/classifier. It use a skip-aware BN, with a cost of 0.07% parameter increasing, but reduces the inference cost and achieve better results comparing with the non-adaptive baselines.", "summary_of_the_review": "Good motivation and direction of research. Simple adaptive skip-paths strategy is proposed, while stronger experiments are needed ", "strength_and_weaknesses": "Pro:\n1) The method proposed by the paper to slim a residual network looks reasonable, it changed from a slimmable channels to slimmable sub-paths. It demonstrates in imagenet coco that a wider subnetwork performs better than deeper but thinner network based on resnet-like basenetworks [resnet50/101, mobilenetv2]. \n\n2) It includes several theoretical side evidence for using skip subpaths rather than skip channels. e.g. [Caramazza & Coltheart 2006], that looks convincing for me. \n\n\nCons:\n1) The baselines compared in the paper looks out dated, therefore is less convince for the generalization of the proposed strategy. \nI think the author should also demonstrate the effectiveness over stronger baselines for example: \n\nVit-like networks, [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale]\nSwin-transformers [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows]\nCvt networks [CvT: Introducing Convolutions to Vision Transformers]\n\nThey induces stronger results on all these tasks, it might be better we have baselines for them. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject