Adaptive Depth Networks with Skippable Sub-Paths

Authors: woochul  kang
hyungseop  lee
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 11:34:02

Abstract: Predictable adaptation of network depths can be an effective way to control inference latency and meet the resource condition of various devices. However, previous adaptive depth networks do not provide general principles and a formal explanation on why and which layers can be skipped, and, hence, their approaches are hard to be generalized and require long and complex training steps. In this paper, we present a practical approach to adaptive depth networks that is applicable to various networks with minimal training effort. In our approach, every hierarchical residual stage is divided into two sub-paths, and they are trained to acquire different properties through a simple self-distillation strategy. While the first sub-path is essential for hierarchical feature learning, the second one is trained to refine the learned features and minimize performance degradation if it is skipped. Unlike prior adaptive networks, our approach does not train every target sub-network in an iterative manner. At test time, however, we can connect these sub-paths in a combinatorial manner to select sub-networks of various accuracy-efficiency trade-offs from a single network. We provide a formal rationale for why the proposed training method can reduce overall prediction errors while minimizing the impact of skipping sub-paths. We demonstrate the generality and effectiveness of our approach with convolutional neural networks and transformers.

Main paper: 
 introduction: 
 Modern deep neural networks such as CNNs and transformers [1] provide state-of-the-art performance at high computational costs, and, hence, lots of efforts have been made to leverage those inference capabilities in various resource-constrained devices. Those efforts include compact architectures [2, 3] , network pruning [4, 5] , weight/activation quantization [6] , knowledge distillation [7] , to name a few. However, those approaches provide static accuracy-efficiency trade-offs, and, hence, it is infeasible to deploy one single model to meet devices with all kinds of resource-constraints.There have been some attempts to provide predictable adaptability to neural networks by exploiting the redundancy in either network depths [8, 9] , widths [10, 11] , or both [12, 13] . However, one major difficulty with prior adaptive networks is that they are hard to train and require significantly longer training time than non-adaptive networks. For example, most adaptive networks select a fixed number of sub-networks of varying depths or width, and train them iteratively, mostly through self-distilling knowledge from the largest sub-network (also referred to as the super-net) [10, 11, 13, 14] . However, this exhaustive self-distillation takes long time and can generate conflicting training objectives for different parameter-sharing sub-networks, potentially resulting in worse performance [15, 16] . Moreover, unlike width-adaptation networks, there is no established principle for adapting depths since the impact of skipping individual layers has not been formally investigated. In this work, we introduce an architectural pattern and training method for adaptive depth networks that is generally applicable to various networks, e.g., CNNs and transformers. In the proposed adaptive depth networks, every residual stage is divided into two sub-paths and the sub-paths are trained to have different properties. While the first sub-paths are mandatory for hierarchical feature learning, the second sub-paths are optimized to incur minimal performance degradation even if they are skipped.In order to enforce this property of the second sub-paths, we propose a simple self-distillation strategy, in which only the largest sub-network (or, super-net) and the smallest sub-network (or, base-net) are exclusively used as a teacher and a student, respectively, as shown in Figure -1-(a). The proposed self-distillation strategy does not require exhaustive training of every target sub-network, resulting in significantly shorter training time than prior adaptive networks. However, at test time, sub-networks with various depths can be selected instantly from a single network by connecting these sub-paths in a combinatorial manner, as shown in Figure 1 -(b). Further, these sub-networks with varying depths outperform individually trained non-adaptive networks due to the regularization effect, as shown in Figure 1-(c ).In Section 3, we discuss the details of our architectural pattern and training algorithm, and show formally that the selected sub-paths trained with our self-distillation strategy are optimized to reduce prediction errors while minimally changing the level of input features. In Section 4, we empirically demonstrate that our adaptive depth networks outperform counterpart individual networks, both in CNNs and vision transformers, and achieve actual inference acceleration and energy-saving. 
related work: 
 Adaptive Networks: In most adaptive networks, parameter-sharing sub-networks are instantly selected by adjusting either widths, depths, or resolutions [8, 10, 11, 17, 13, 15, 18, 12, 19] . For example, slimmable neural networks adjust channel widths of CNN models on the fly for accuracyefficiency trade-offs and they exploit switchable batch normalization to handle multiple sub-networks [10, 20, 11] . Transformer-based adaptive depth networks have been proposed for language models to dynamically skip some of the layers during inference [13, 9] . However, in these adaptive networks, every target sub-network with varying widths or depths need to be trained explicitly, incurring significant training overheads and potential conflicts between sub-networks.Dynamic Networks: Dynamic networks [21] are another class of adaptive networks that exploit additional control networks or decision gates for input-dependent adaptation of CNN models [22, 23, 24, 25, 26] and transformers [27, 28, 29, 30] . In particular, most dynamic networks for depthadaptation have some kinds of decision gates at every layer (or block) that determine if the layer can be skipped [31, 22, 32, 33] . These approaches are based on the thought that some layers can be skipped on 'easy' inputs. However, the learned policy for skipping layers is opaque to users and does not provide a formal description of which layers can be skipped for a given input. Therefore, the network depth cannot be controlled by users in a predictable manner to meet the resource condition of target devices. h L/2+1 s F L/2 F 1 F L F L/2+1 h L s … True False Figure 2:Illustration of a residual stage with two sub-paths. While the first (blue) sub-path is mandatory for hierarchical feature learning, the second (orange) sub-path can be skipped for efficiency.The layers in the skippable sub-path are trained to preserve the feature distribution from h s base to h s super using the proposed self-distillation strategy. Having similar distributions, either h s base or h s super can be provided as input h s to the next residual stage. In the mandatory sub-path, another set of batch normalization operators, called skip-aware BNs, is exploited if the second sub-path is skipped. These sub-paths are building blocks to construct sub-networks of varying depths.Residual Blocks with Shortcuts: Since the introduction of ResNets [34] , residual blocks with identity shortcuts have received extensive attention because of their ability to train very deep networks, and have been chosen by many CNNs [35, 36] and transformers [1, 37, 38] . Veit et al. [39] argue that identity shortcuts make exponential paths and results in an ensemble of shallower sub-networks. This thought is supported by the fact that removing individual residual blocks at test time does not significantly affect performance, and it has been further exploited to train deep networks [40, 41] . Other works argue that identity shortcuts enable residual blocks to perform iterative feature refinement, where each block improves slightly but keeps the semantic of the representation of the previous layer [42, 43] . Our work build upon those views on residual blocks with shortcuts and further extend them for adaptive depth networks by introducing a training method that enforces the properties of residual blocks more explicitly for skippable sub-paths. 
adaptive depth networks: 
 We first present the architecture and training details of adaptive depth networks. Then, we discuss the theoretic rationale for how depth adaptation can be achieved with minimal performance degradation.In modern hierarchical networks, such as ResNets [34] and Swin transformers [38] , there are typically 4 to 7 consecutive residual stages. 2 In a network with N r stages, the s-th (s = 1, ..., N r ) stage consists of L (typically L ∈ [3, 6] ) identical residual blocks that transform input features h s 1 additively to produce the output features h s , as follows:EQUATIONWhile a block with a residual function F ℓ (ℓ = 1, ..., L) learns hierarchical features as traditional compositional networks [44] , previous literature [43, 42] demonstrates that a residual function also tend to learn a function that refines already learned features at the same feature level. If a residual block mostly performs feature refinement while not changing the level of input features, the performance of the residual network is not significantly affected by dropping the block at test time [40, 41] . However, in typical residual networks, most residual blocks tend to refine features while learning new level features as well, and, hence, random dropping of residual blocks at test time degrades inference performance significantly. Therefore, we hypothesize that if some selected residual blocks can be encouraged explicitly during training to focus more on feature refinement, then these blocks can be skipped to save computation at marginal loss of prediction accuracy at test time.To this end, we propose an architectural pattern for adaptive depth networks, in which every residual stage is divided into two sub-paths, or F s base and F s skippable as in Equation 1 and Figure 2 . We train these two sub-paths to have different properties (Section 3.2). While F s base is trained to learn feature representation h s base (= h s L/2+1 ) with no constraint, the second sub-path F s skippable is constrained to preserve the feature level of h s base and only refine it to produce h s super (= h s L+1 ). Since layers in F s base perform essential transformations for hierarchical feature learning, they cannot be bypassed during inference. In contrast, layers in F s skippable can be skipped for efficiency since they only refine h s base . If F s skippable is skipped, then h s base becomes the input to the next residual stage. Therefore, 2 Nr (= Nr k=0 C(N r , k)) sub-networks with varying accuracy-efficiency trade-offs can be selected from a single network by choosing whether to skip F s skippable (s = 1, ..., N r ) or not (Table 6 ). This architectural pattern is agnostic to the type of residual blocks. The residual function F can be convolution layers for CNNs and self-attention + MLP layers for transformers. Table 5 Get next mini-batch of data x and label y 4:optimizer.zero_grad()ŷsuper , h super = M.f orward(x, skip=[False, ..., False]) ▷ forward pass for super-net optimizer.step() 12: end for In Algorithm 1, the largest and the smallest sub-networks of M, which are called super-net and the base-net, respectively, are exploited. In steps 9-10, the hierarchical features (and outputs) from the super-net and the base-net are encouraged to have similar distributions through self-distillation. Self-distillation has been extensively used in prior adaptive networks [11, 10, 13] . However, their primary goal is to train every target sub-network. For example, on each iteration, sub-networks are randomly sampled from a large search space to act as either teachers or students [11] , which takes significantly longer training time and can generate conflicting training objectives for different parameter-sharing sub-networks [15, 16] . In contrast, our method in Algorithm 1 focuses on training sub-paths by exclusively using the super-net as a teacher and the base-net as a student. By focusing on sub-paths, the training procedure in Algorithm 1 is significantly simplified and avoids potential conflicts among sub-networks. At test time, however, various sub-networks can be selected by connecting these sub-paths. The effect of this self-distillation strategy is investigated in Section 4.4.Lastly, due to the architectural pattern of interleaving the mandatory and the skippable sub-paths, minimizing D KL (ŷ super ∥ŷ base ) also minimizes D KL (h super ∥h base ) implicitly. The ablation study in Section 4.4 shows that omitting the term Nr s=1 D KL (h s super ∥h s base ) in loss base has a minor effect on the performance. This implicit approach is useful when the extraction of intermediate features is tricky. EQUATIONEQUATIONEQUATIONIn Equation 4, the loss function is expanded around h s L , or h s base + ...+ F L-1 (h s L-1). Only the first order term is left and all high order terms, such asF L (h s L ) 2 • ∂ 2 L(h s L ) 2∂(h s L ) 2 , are absorbed in O(F L (h s L )).The high-order terms in O(F L (h s L )) can be ignored if F L (h s L ) has a small magnitude. In typical residual networks, however, every layer is trained to learn new features with no constraint, and, hence, there is no guarantee that F L (h s L ) have small magnitude. In contrast, in our adaptive depth networks, the residuals in F s skippable are explicitly enforced to have small magnitude through the proposed selfdistillation strategy (refer to Figure 3 for empirical evidence). As a result, the terms in O(F L (h s L )) can be ignored for the approximation. If we similarly keep expanding the loss function around h s j (j = L/2 + 1, ..., L) while ignoring high order terms, we obtain the following approximation:L(h s super ) ≈ L(h s base ) + L j=L/2+1 F j (h s j ) • ∂L(h s j ) ∂h s j (5)In Equation 5, minimizing the loss L(h s super ) during training drives F j (h s j ) (j = L/2 + 1, ..., L) in the negative half space of ∂L(h s j ) ∂h s j to minimize the dot product between F j (h s j ) and ∂L(h s j ) ∂h s j . This implies that every residual function in F s skippable is optimized to learn a function that has a similar effect to gradient descent:EQUATIONIn other words, the residual functions in the skippable sub-paths reduce the loss L(h s base ) iteratively during inference while preserving the feature distribution of h s base . Considering this result, we can conjecture that, with our architectural pattern and self-distillation strategy, layers in F s skippable learn functions that refine input features h s base iteratively for better inference accuracy while minimally changing the distribution of h s base . . This result demonstrates that our self-distillation strategy in Algorithm 1 effectively trains the skippable sub-paths to minimally change the input distribution. As a result, the blocks in skippable sub-paths can be skipped with less impact on performance.Originally, batch normalization (BN) [45] was proposed to handle internal covariate shift during training non-adaptive networks by normalizing features. In our adaptive depth networks, however, internal covariate shifts can occur during inference in mandatory sub-paths if different sub-networks are selected. To handle potential internal covariate shifts, switchable BN operators, called skip-aware BNs, are used in mandatory sub-paths. For example, at each residual stage, two sets of BNs are available for the mandatory sub-path, and they are switched depending on whether its skippable sub-path is skipped or not.The effectiveness of switchable BNs has been demonstrated in networks with adaptive widths [46, 11] and adaptive resolutions [47] . However, in previous adaptive networks, N sets of switchable BNs are required in every layer to support N parameter-sharing sub-networks. Such a large number of switchable BNs not only requires more parameters, but also makes the training process complicated since N sets of switchable BNs need to be trained iteratively during training. In contrast, in our adaptive depth networks, every mandatory sub-path needs only two sets of switchable BNs, regardless of the number of supported sub-networks. This reduced number of switchable BNs significantly simplifies the training process as shown in Algorithm 1. Furthermore, the amount of parameters for skip-aware BNs is negligible. For instance, in ResNet50, skip-aware BNs increase the parameters by 0.07%.Transformers [1, 38] exploit layer normalization (LN) instead of BNs and naive replacement of LNs to BNs incurs instability during training [48] . Therefore, for our adaptive depth transformers, we apply switchable LN operators in mandatory sub-paths instead of switchable BNs. 
experiments: 
 We use networks both from CNNs and vision transformers as base models to apply the proposed architecture pattern: MobileNet V2 [35] is a lightweight CNN model, ResNet [34] is a larger CNN model, and ViT [37] and Swin-T [38] are representative vision transformers. All base models except ViT have hierarchical stages, each with 2 ∼ 6 residual blocks. So, according to the proposed architectural pattern, every stage is evenly divided into 2 sub-paths for depth adaptation. ViT does not define hierarchical stages and all residual encoder blocks have same spatial dimensions. Therefore, we divide 12 encoder blocks into 4 groups, resembling other residual networks, and select the last encoder block of each group as a skippable sub-path. Details are in Table 5 in Appendix A.1. These models are trained according to Algorithm 1. For self-distillation, only the final outputs from the super-net and the base-net, or ŷsuper and ŷbase respectively, are used since exploiting intermediate features for explicit self-distillation has a marginal impact on performance (Section 4.4).We use the suffix '-ADN' to denote our adaptive depth networks. A series of boolean values in parentheses denotes a specific sub-network used for evaluation; each boolean value represents the residual stage where its skippable sub-path is skipped. For example, ResNet50-ADN (FFFF) and ResNet50-ADN (TTTT) correspond to the super-net and the base-net of ResNet50-ADN, respectively.We evaluate our method on ILSVRC2012 dataset [49] that has 1000 classes. The dataset consists of 1.28M training and 50K validation images. For CNN models, we follow most training settings in the original papers [34, 35] , except that ResNet models are trained for 150 epochs. ViT and Swin-T are trained for 300 epochs, following DeiT's training recipe [50, 51] . For Swin-T-ADN, we disable stochastic depths [40] for the mandatory sub-paths since the strategy of random dropping of residual blocks conflicts with our approach to skipping sub-paths. For fair comparison, our adaptive depth networks and corresponding individual networks are trained in the same training settings.The results in Figure 4 -(a) show that our adaptive depth networks outperform counterpart individual networks even though many sub-networks share parameters in a single model. Further, our results with vision transformers demonstrate that our approach is generally applicable and compatible with their state-of-the-art training techniques such as DeiT's training recipe [50] . We conjecture that this performance improvement results from effective distillation of knowledge from h s super to h s base at each residual stage and the iterative feature refinement at skippable sub-paths, shown in Equation 6. 6 in Appendix A.2 shows the performance of all sub-networks. In Figure 4-(b) and Table 1 , several state-of-the-art efficient inference methods and dynamic networks are compared with our base-networks. The result demonstrates that our adaptive depth networks show comparable performance across a range of FLOPs. In Figure 4-(b ), it should be noted that individual ResNets trained with knowledge distillation has worse performance than individual ResNets. As reported in previous works, successful knowledge distillation requires a patient and long training [52] , and straightforward knowledge distillation using ImageNet does not improve the performance of student models [53, 54] . In contrast, our ResNet50-ADN trained with the proposed self-distillation strategy achieves better performance than counterpart ResNets. This demonstrates that the high performance of adaptive depth networks does not simply come from distillation effect.One of the main advantages of our adaptive depth networks is the significantly lower training effort needed compared to previous adaptive networks. In Algorithm 1, only one additional forward and backward pass of the smallest sub-network, or the base-net, is required for self-distillation at each training iteration. For most CNNs and transformer networks, the original training settings, such as training schedules and hyperparameters, can be used with minimal changes. However, as shown in Figure 5 , many sub-networks of varying depths become available from a single network by connecting sub-paths in a combinatorial manner even though they are not trained explicitly.Table 2 shows that, at every training epoch, our adaptive depth networks require a comparable amount of time as training two separate networks combined. This is because our training method trains sub-paths, rather than sub-networks, by exploiting only the super-net and the base-net. In contrast, the compared adaptive networks require much longer training time than ours since they have to explicitly apply self-distillation to all target sub-networks. For example, on every training iteration, AlphaNet [11] randomly samples sub-networks from its search space for self-distillation. Although the prior works we compared may seem outdated, they remain relevant and representative because there has been little progress in improving the training cost of adaptive networks.While AlphaNet [11] supports a significantly larger number of sub-networks, our primary goal is not to maximize the number of sub-networks. Instead, our objective is to provide better performance Pareto with a few useful sub-networks, as illustrated in Figure 4-(b ). Since the classification performance of sub-networks does not always scale proportionally with their FLOPS, most sub-networks end up being ineffective. For example, although ResNet50-ADN has 2 4 sub-networks in its search space, Table 6 in Appendix A.2 shows that some shallower sub-networks outperform deeper sub-networks.While the Pareto frontiers formed by sub-networks demonstrate theoretic performance, inference acceleration in actual devices is more important in practice for effective control of inference latency and energy consumption. The result shows that depth-adaptation of ResNet50-ADN is highly effective in accelerating inference speeds and reducing energy consumption. Although our ResNet50-ADN has a limited FLOPs adaptation range, reducing FLOPs by 38% through depth adaptation reduces both inference latency and energy consumption by 35%. In contrast, even though S-ResNet50 can reduce FLOPs by up to 93% by adjusting its width, it only achieves up to 9% acceleration in practice. We first investigate the influence of two key components of the proposed adaptive depth networks: (1) self-distillation of sub-paths and (2) skip-aware BNs/LNs. When our self-distillation method is not applied, the loss of the base-net, or loss base , in Algorithm 1 is modified to criterion(y, ŷbase ). Table 3 shows the results. For ResNet50-ADN, when neither of them is applied, the inference accuracy of the super-net and the base-net is significantly lower than non-adaptive individual networks by 1.5% and 2.8%, respectively. This result shows the difficulty of joint training sub-networks for adaptive networks. When one of the two components is applied individually, the performance is still slightly worse than individual networks'. When both self-distillation and skip-aware BNs are applied together, ResNet50-ADN achieves significantly better performance than individual networks, both in the super-net and the base-net. The last row, with double check marks, shows that exploiting intermediate features as well as softmax outputs for self-distillation has minor impact on performance. Self-Distillation Strategies: Our self-distillation approach in Algorithm 1 exploits only two subnetworks exclusively as a teacher and a student. Specifically, the super-nets, or FFFF, acts as the the teacher and the base-nets, or TTTT, becomes the student. The purpose of this strategy is not to train only those two sub-networks, but rather to train skippable sub-paths in a way that minimally modifies the feature distribution, as demonstrated in Figure 3 . To investigate the effect of self-distillation strategies, we conduct an experiment in Table 4 . In every training iteration, rather than exclusively using FFFF and TTTT sub-networks for self-distillation, we randomly sample either a teacher, a student, or both from 2 4 sub-networks. These randomly sampled sub-networks are trained explicitly through self-distillation. However, as shown in Table 4 , all sub-networks, both from ResNet50-ADN and ViT-b/32-ADN, perform significantly better when our self-distillation strategy is applied for training. Even though most of our sub-networks (such as TFFF, TTFF, and TTTF) are instantly constructed at test time without explicit training, they still outperform their counterpart sub-networks trained explicitly through random sampling. This result demonstrates that our method of training sub-paths is more effective than training target sub-networks themselves.Lengths of Mandatory Sub-Paths: In Figure 6 -(b), we investigate the impact of varying the ratio of lengths between the mandatory and the skippable sub-paths. (Details are in Table 7 .) If mandatory sub-paths become shorter than skippable sub-paths, shallower sub-networks can be selected since more layers can be skipped. However, the result in Figure 6 -(b) shows that this configuration (shown in blue line) significantly degrades the performance of all sub-networks. Since every sub-network shares parameters of mandatory sub-paths, low inference capability of shallow mandatory sub-paths affects all sub-networks. This implies that maintaining certain depths in mandatory sub-paths is crucial for effective inference. Conversely, further increasing the length of mandatory sub-paths (shown in green line) does not further improve performance and only reduces depth adaptation. 
conclusions: 
 We propose a practical approach to adaptive depth networks that can be applied to both CNNs and transformers with minimal training effort. We provide a general principle and a formal explanation on how depth adaptation can be achieved with minimal performance degradation. Under this principle, our approach can avoid typical exhaustive training of target sub-networks and instead focus on optimizing the sub-paths of the network to have specific properties. At test time, these sub-paths can be connected in a combinatorial manner to construct sub-networks with various accuracy-efficiency trade-offs from a single network. Experimental results show that these sub-networks form a better Pareto frontier than non-adaptive baseline networks and achieve actual inference acceleration. We anticipate that these advances will facilitate the practical application of adaptive networks. 
 
 Appendix: A. For vision transformers, residual blocks/stages refer to encoder blocks/stages.ImageNet validation dataset is used for this experiment. 


Decision: The paper presents valuable contributions to the design of adaptive depth networks and tests the approach on both convolutional and transformer networks. After rebuttal, all the reviewers were positive about the paper, and the AC agreed with the rating.

Reviews: review: paper_summary: paper_summary: The paper provides a training methodology to develop small deployable subnetworks (also high performing) when training a single large network. The key claim of this paper is that due to their innovative training techniques the smaller subnetworks learns better feature and proves the point with reasonable baselines.
strength_weakness: strength_weakness: Some major weaknesses of this paper are below.

a) The claim made in the paper " To the authro's knowledge, this is the first approach ..." is a bit of an overclaim. In prior CNN literature there are some works that performs subnetwork distillation. 

b) In sec 3.2 the authors claim that h_{base} learns compact representation. I would request the authors to quantity this more concretely. 

c) The results in Figure 4 a) are a bit tricky to evaluate as author's models are trained with distillation and the baseline models are trained with regular training. How do we know if the high accuracy is due to distillation or due to the proposed technique. The baseline has to be fixed.
d) More baseline comparisons are required with other techniques such as Matformer (https://arxiv.org/abs/2310.07707), Inheritune (https://arxiv.org/abs/2404.08634) etc.
questions: questions: Can this method work in medium size LLMs where distillation could be tricky?
limitations: limitations: Yes, the authors have tried to address the limitations.
paper_summary: paper_summary: The paper presents adaptive depth networks. During training, the network is trained like a “super-net” that contains all the paths. During inference, the skippable networks can be skipped in devices that have limited resources. The whole framework is useful in real-world applications that require various accuracy-efficiency trade-offs.
strength_weakness: strength_weakness: - There seem to be several solutions with ideas similar to those in this paper, such as [13]. Although there may be differences in the approaches, all of these methods can achieve the same goals in this paper. In my opinion, it would be better to include a comprehensive analysis of the differences and performance. If the proposed method could stand out in performance among the methods, it would make the proposed method very competitive among these methods.

- A recent study [R1] found that with longer schedules, the ResNet-50 model can achieve about 80% accuracy on ImageNet. Considering that we can choose to give smaller models a longer training schedule to achieve better results, is there a chance to use a longer schedule to make the Base model achieve better results and avoid using an adaptive network?


[R1] ResNet strikes back: An improved training procedure in timm.
questions: questions: Please refer to the second point in weakness.
limitations: limitations: N/A
paper_summary: paper_summary: The submission presents an approach to adaptive depth networks, where each hierarchical residual stage is divided into two sub-paths and they are trained to acquire different properties:  the first sub-path is essential for hierarchical feature learning, the second one is trained to refine the learned features and minimize performance degradation even if it is skipped. In addition, a formal reason of why the proposed training method can reduce overall prediction errors while minimizing the impact of skipping sub-paths is also provided. Experimental results on ImageNet classification are provided to prove the effectiveness.
strength_weakness: strength_weakness: The overall contribution of the paper is solid. The only thing I would suggest is to expend the experiments to 1 more low-level vision task, for example Single-Image Super Resolution, image denoising and so on. Since usually the decoder part of such low-level vision task is computational heavy.
questions: questions: I don't have specific questions to this submission.
limitations: limitations: No problem in limitations.
paper_summary: paper_summary: The authors propose an easy-to-train supernet, in which you can then adaptively change the network depth by disabling some blocks (skippable sub-paths) and using skip connections instead, while always keeping mandatory sub-paths. They found that the length of mandatory and the skippable sub-paths should be the same, for maximum efficiency in terms of accuracy and computational complexity. 
During training there are 2 forward-backward passes:
1. Training the whole supernet/teacher (FFFF) network as usually for task objectives (ImageNet classificaiton)
2. Self-distillation intermidiate feature maps from supernet/teacher (FFFF) to the base/student (TTTT) network using KL-divergence-loss (where is base/student (TTTT) is a sub-network of supernet/teacher (FFFF) )

Switchable LayerNorm or BatchNorm operators are used in mandatory sub-paths.
This is the first approach to adaptive networks that provides a general principle and theoretic basis for supporting predictable depth adaptation with minimal performance degradation.
strength_weakness: strength_weakness: In Figure 4 (b),  SpViT-Swin-Ti is more accurate and has fewer flops than the proposed approach Swin-T-ADN.

There is no Out-of-Domain (OoD) comparison when very different datasets are used for training and testing, which shows the robustness of the model and the extent to which it can be applied to real-world problems.
Most of the charts and tables compare the computational complexity (Flops), with the exception of Figure 5(b), while Latency (ms) is more important for real-world problems.
It would be better to have as many charts as possible with Out-of-Domain accuracy vs Latency.
questions: questions: Does your approach achieve higher accuracy than  intermediate feature maps distillation: if you train the Supernet first, then freeze it and distill intermediate feature maps from the frozen Supernet to the separate randomly initialized Basenet? So maybe your approach is not only more flexible than intermediate feature maps distillation, but also more accurate?
limitations: limitations: Training and testing are carried out on different splits of the same dataset, which is an In-domain comparison, while for real-life applications Out-of-domain is much more important, when training and testing on very different datasets, where completely different approaches may be the best.

Comments: title: Thank you Authors for the Rebuttal
comment: Thank you once again for your valuable suggestions and understanding.
* **Comments a) and b)**

We appreciate your insight. After considering your feedback carefully, we realized that some of our initial statements might be misleading. Specifically, our initial claim that ‘$h_{base}$ learns a compact representation from $h_{super}$’ can be misleading. Since, in our approach, every sub-network shares parameters and $h_{super}$ is not fixed, the knowledge transfer is not uni-directional from $h_{super}$ to $h_{base}$, but rather bidirectional. As demonstrated in Section 3.3, the goal of our self-distillation strategy is to encourage $h_{base}$ and $h_{super}$ to learn similar feature distributions, rather than unidirectional knowledge distillation. Since $h_{base}$ and $h_{super}$ have similar distributions, the selected layers of every residual stage become skippable with minimal performance loss. 

In the revision, we will remove the claim that is hard to quantify and potentially misleading. 
In light of this, we will also rephrase our contribution as follows:

*Our approach uniquely introduces a principle for training selected sub-paths to be skippable with minimal performance loss. This principle allows us to avoid typical exhaustive training of target sub-networks and instead instantly construct sub-networks of varying depths from specifically trained sub-paths.*

* **Comment c)**  *I do not agree; KD has shown great results with Imagenet-1k and ImageNet-21K...*

We appreciate your feedback and agree that knowledge distillation (KD) is a crucial technique for developing efficient networks. Previous works, such as DEIT [1] and TinyVIT [2], have demonstrated its effectiveness.
In Figure 4-b, we compare our work with KD to illustrate that our positive results are not solely due to distillation effect. To ensure a fair comparison, we tried to maintain identical training settings for both our approach and KD. Our objective was not to claim that our method is superior to KD in general.

In the revision, we will include efficient KD approaches, such as DEIT [1] and TinyVIT [2], in Table 1 as state-of-the-art (SOTA) baselines.

* **Comment d)** *... but the authors didn't say anything about the Inheritune paper.*

Inheritune shares some similarities with our approach as it leverages a few transformer blocks from a larger language model (LM) to train smaller models. However, unlike our method, Inheritune trains separate smaller models using the larger LM, making it not directly comparable to our approach. Additionally, a quantitative comparison between Inheritune and our method is not feasible since they are applied to different domains. We may consider Inheritune as a baseline if our work is extended to large language models (LLMs). 

* **Final Comment)**  *Again I re-iterate this is a good work but loose ends needs to be tightened with reasonable baselines and claims.*

Thank you once again for your valuable suggestions and comments. 

They have given us an excellent opportunity to reflect on our work, considering both its strengths and weaknesses. We will update our revision based on your feedback.
Fair. I don't require you to add new comparisons in this short rebuttal period. I tend to accept the submission even without it.
We sincerely appreciate your positive feedback and questions. 

In this experiment, we explored two scenarios for knowledge distillation (KD) of ResNet50-Base: (1) using only logits, and (2) using both logits and intermediate features. As shown in the table below, both scenarios yield slightly better performance compared to our self-distillation method. While the difference between (1) and (2) is marginal, there may be a chance to enhance the performance of (2) with further hyper-parameter tuning, such as applying different KD temperatures for different layers. However, we have not yet explored this path.

This finding highlights the importance of selecting the right teacher-student combination for effective knowledge distillation. For instance, KD using ResNet50 (PyTorch pretrained) as a teacher did not yield positive results.

| Teacher                                 | Student          | Student Acc@1 | Note |
|-----------------------------------------|------------------|---------------:|-------:|
| ResNet50(FFFF)       | ResNet50(TTTT)   | 76.1          |  our approach |
| ResNet50 (PyTorch pretrained), logits only | ResNet50-Base    | 73.8          | |
| (1) ResNet50-ADN(FFFF), logits only     | ResNet50-Base    | **76.3**          | |
| (2) ResNet50-ADN(FFFF), logits + intermediate features | ResNet50-Base    | 76.2          | |

Thank you again for your comments, and please let us know if you have any further questions.
a) The authors' has just updated the statement from subnetwork to subpath. But haven't differentiated it why these two are conceptually different. I have highlighted cause "later layers distilling knowledge to initial layers or a subset of  layers" is not novel. Please clarify how these are conceptually different.

b) If the compactness of representation is difficult to prove then authors should refrain from making such claims. I could not understand the table; a self contained caption is needed. What are the values? what are the stages? how are they trained? how the evaluation is done? A layerwise confusion matrix would have been a better substitute to establish this correlation.

c) I do not agree; KD has shown great results with Imagenet-1K and Imagenet-21K for instance look at DEIT [1] and TinyVIT [2]. I guess the author's are comparing with their version of KD. I request the authors to find the SOTA KD for such a setting. I acknowledge the table shown by the authors but I would request the authors to cite papers for their KD baselines.

d) I acknowledge the Matformer discussion but the authors didn't say anything about the Inheritune paper.

Question- Fair!! such an experiment would take time and rebuttal period is too short.

Overall, this is a good work but some of my concerns that I have raised remain unanswered/ambiguously answered. I would retain my score. Again I re-iterate this is a good work but loose ends needs to be tightened with reasonable baselines and claims.

[1] https://arxiv.org/abs/2012.12877

[2] https://arxiv.org/abs/2207.10666
Thanks to the authors for the answers and clarifications.
Based on the answers, I leave the article rating 7: Accept.

"The results show that ResNet50-Base trained using ResNet50-ADN(FFFF) as a teacher achieves 76.30% Acc@1, which is 0.2% higher than our subnetwork ResNet50-ADN (TTTT). "
Did you use only output logits/pseudo-labels or also feature map tensors from **intermediate layers** (for example, after each downsampling layer) for distillation in this case?