Slicing Vision Transformer for Flexible Inference
Authors: Yitian Zhang
Huseyin Coskun
Xu Ma
Huan Wang
Ke Ma
Stephen Xi Chen
Derek Hao Hu
Yun Fu
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision Transformer
Flexible Inference
Primary area: machine_vision
TL;DR: Use a single ViT to represent multiple smaller variants

Abstract: Vision Transformers (ViT) is known for its scalability. In this work, we target to scale down a ViT to fit in an environment with dynamic-changing resource constraints. We observe that smaller ViTs are intrinsically the sub-networks of a larger ViT with different widths. Thus, we propose a general framework, named Scala, to enable a single network to represent multiple smaller ViTs with flexible inference capability, which aligns with the inherent design of ViT to vary from widths. Concretely, Scala activates several subnets during training, introduces Isolated Activation to disentangle the smallest sub-network from other subnets, and leverages Scale Coordination to ensure each sub-network receives simplified, steady, and accurate learning objectives. Comprehensive empirical validations on different tasks demonstrate that with only one-shot training, Scala learns slimmable representation without modifying the original ViT structure and matches the performance of Separate Training. Compared with the prior art, Scala achieves an average improvement of 1.6% on ImageNet-1K with fewer parameters.

Slicing Vision Transformer for Flexible Inference
Yitian Zhang 1.2 Huseyin Coskun' Xu Ma Huan Wang²
Ke Ma
Xi (Stephen) Chen' Derek Hao Hu Yun Fu? Snap Inc. "Northeastern University Meta
Abstract
Vision Transformers (ViT) known for its scalability. In this work, we target to scale down VIT to fit in an environment with chhamic-changing resource constraints. We observe that smaller ViTs are intrinsically the sub-networks of a larger VIT with different widths. Thus, we propose general framework, named Scala, to enable a single network to represent multiple smaller ViT' with flexible inference capability, which aligns with the inherent design of VIT to vary from widths. Concretely, Scala activates several subnets during training, introduces Isolated Activation to disentangle the smallest sub-network from other subnets, and leverages Scale Coordination to ensure each sub-network receives simplified, steady, and accurate learning objectives. Comprehensive empirical validations different tasks demonstrate that with only one-shot training, Scala learns slimmable representation without modifying the original VIT structure and matches the per- formance of Separate Training. Compared with the prior art, Scala achieves an average improvement of 1.6% ImageNet-I K with fewer parameters. Code is available here.
Introduction
Vision Transformers (ViTs) g9 are renowned for its scalability and various avenues 41 7 have been explored to scale up ViT models. To tailor ViTs to on devices with limited resources, some recent progress [36] 35 utilize knowledge distillation [I3 to scale down ViT. Particularly, DeiT [29 introduces two smaller variants of DeîT-B: DeiT-Ti and DeïT-S which have been widely used in lnouureellmmited applications. Although these small ViTs exhibit enhanced efficiency, they lack the flexibility to implement customized adjustments that accommodate dynamically changing resource constraints in real-world scenarios, c.g., the computation budget of mobile phones depends on the energy level (low-power mode) and number of running apps. Consequently, the standard Separate Training (ST) protocol trains models with different sizes separately to provide spectrum of options with diversified performance and computation. ST requires repetitive training procedures to produce multiple model choices, and the challenge is amplified for foundation models [26] 24 17]. From users' perspective, they are only offered limited model choices, that might not cater to all scenarios.
Analyzing the architectures ViT-Ti/S/B, we observe that these ViTs are the same architecture with the only difference in the number of embedding dimensions (we ignore the difference in the number of heads as it does not impact the overall model size), indicating smaller ViT' are intrinsically the nub-networks of larger model with different widths (see Fig. I This suggests that a large ViT can be transformed to represent small models by uniformly slicing the weight matrix at each layer. Given width ratio r, adjust the size of the network by this single hyperparameter, allowing a single ViT to represent multiple small variants with the weights of those sub-networks shared in nested nature, e.g., VIT-B (r=0.25) equals ViT-Ti and VIT-B (r=0.5) corresponds to VIT-S. In this manner, we empower ViTs for flexible inference capability, and we aim to slice a VIT within a broad slicing
Work done when Yitian was intern at Snap Inc.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

75
VIE-II
Block
Block
Block
70
Block
Block
65
VIE-S
60
Block
US Net
13.3
-20 -38 -18
VIT-B
Block
Block
Block
+33
+2,6 11.1
Scala
45
0.2 0.3 0.4
0.5
0.6
0.7
0.8 0.9 1.0
conventional VIT structure
(i) irreguiar sileing
(l0) uniform slicing
Width Ratio
Figure 1: Illustration of different means slice the VIT architecture. Irregular slicing [2] 38 4 results in uncon- ventional structures while uniform slicing [37] aligns with the inherent design of VIT to vary from widths.
Figure 2: The available uniform slicing method US-Net (37] lags behind Sepa- rate Training (ST) remarkably on ViTs. Performance gaps with ST are shown.
bound and fine-grained slicing granularity so that the diversity and number of sub-networks can be ensured for higher flexibility. This problem is non-trivial as fully training all the sub-network within a constrained budget is nearly infeasible. Consequently, it is quite challenging for these subnets to match the performance of separate training.
Although various approaches have delved into slicing deep networks for flexible inference, the problem we target to resolve, i.e., uniformly slicing ViTs within a large slicing bound and fine-grained slicing granularity, is intrinsically different from others in three perspectives: (1) slicing strategy; as in NAS usually slice through multiple shown Fig. (i), the supernet training techniques 22 38 4
dimensions with a small slicing bound, resulting in irregularities in model architectures and minor computational adjustment space. (2) slicing granularity: recent width slicing approaches [14] [8] either slice specific portions of the network or utilize a considerably large slicing granularity, leading to a limited number of models produced. (3) network architecture: US-Net [37] shares similar vision with us but it has only demonstrated success in the CNN architecture.
It is crucial to note the fundamental differences between slicing CNN and ViT: (1) vanilla small ViTs such as TTT/TB/S/B, are inherently designed to vary based on widths, aligning with our approach. Conversely, many CNNs are structured to vary from depths, like ResNet-18/34 [12]. and slicing them by width brings unconventional architectures. (2) slimmable CNN necessitates calibration [39 37 for each sub-network pre-inference due Batch Normalization [15]. unlike slimmable ViTs that can be directly utilized for evaluation. (3) the transformer architecture 33 has wider applications than
CNN in this era, e.g., MAE M.. CLIP (26). DINOv2 (44, LLMs B1 32 D. Nevertheless, ViTs have much less image-specifer inductive bias than CNN and their slimmable ability remains unclear. As shown in Fig. 2 we empirically implement US-Net on ViT-S and observe substantial performance gaps at most width ratios compared to ST, indicating that the available solution of uniform slicing does not work well on the transformer architecture.
To investigate the underlying causes of this phenomenon, we conduct analyses in Sec. 3 which are briefly summarized in two folds: (1) ViTs display minimal interpolation ability, indicating that the optimization of intermediate subnets falls notably short compared to Separate Training (ST); (2) sustained activation of the smallest sub-network poses negative effect on other subnets, which affects the overall performance as their weights are shared in a nested nature. To resolve these issues, we propose a general framework, named Scala, to enforce ViTs to learn slimmable representation. Specifically, we propose Isolated Activation to disentangle the representation of the smallest sub- network from other subnets while still preserving the lower bound performance. Besides, we present Scale Coordination to ensure each subnet receives simplified, steady, and accurate learning objectives. In this manner, the slimmable ViT can be transformed into multiple smaller variants during inference and match the performance of ST.
Compared ST which trains all the subnets individually, Scala reduces the storage and training costs remarkably since the weights of smaller VITs are shared with the full model and we only need one-shot training without extending the training duration. Further, Scala has a very large slicing bound and fine-graing grained slicing granularity, enabling diverse sub-network choices during evaluation. In this way, the delivered system can make tailored adjustments that accommodate dynamically changing resource constraints in real-world scenarios, promising the application on edge devices. Compared with the prior art SN-Net (25 which supports flexible inference on ViTs, Scala clearly outperforms it under different computation with fewer parameters. Moreover, Scala matches the

---

performance of ST on various tasks without modifying the network architecture, demonstrating its generalizability and potential to replace ST as a new training paradigm. The contributions are summarized as follows:
A Although slicing ViTs exhibits multiple advantages, we provide detailed analysis and practical insights into the slimmable ability between different architectures (Sec. 3 and Tab. 2 and find slicing the ViT architecture to be the most challenging problem.
We propose a general framework Scala to enable ViT' to learn slimmable representation for flexible inference. We present Isolated Activation to disentangle the representation of the smallest subnet and Scale Coordination to ensure each subnet receives simplified, steady, and accurate signals.
Comprehensive experiments different tasks demonstrate that Scala, requiring only one- shot training, outperforms prior art and matches the performance of ST, substantially reduces the memory requirements of storing multiple models.
2 Related Work
Scaling Up ViTs. Like Transfromer
33 in NL.,, scalability and performance improvements in
ViT's [9] have been a central focus of recent research. Specifically, strategies have been explored to scale the depth of VIT [46 30) and it is scaled to even larger sizes with almost 2 billion parameters and reaches new state-of-the-ar results [4]]. Afterward, ViTs have been scaled up to 4 billion [5) and 22 billion 7 parameters with extraordinary performance and enormous costs.
Scaling Down ViTs, The advent of ViTs has also sparked interest in scaling down these models. Techniques such knowledge distillation [13 have been explored to reduce the VIT model size [36 [35. For example, DeiT [29] presents smaller ViTs with 5M parameters. Additionally, researchers have explored quantization methods [22 21 to further compress ViTs for deployment edge devices. Unfortunately, these static models cannot make customized adjustments for resource- changing environments in real scenarios.
Slimmable Neural Network. The derivation of multiple smaller models from a single network has been previously explored but most works focus on the CNN structure. Slimmable Networks [39 37
and its variants 34 I train a shared network which adapts the width to accommodate the resource constraints during inference. Later, this idea is adapted into two-stage NAS methods 2 38 for
supernet training. The supernet is scaled at multiple dimensions with a small computation change, in contrast to work where only scale the width dimension with large slicing bound. SN-Net (25 is a recently proposed method that constructs a supernet with several pre-trained models and inserts linear layers to build dynamic routes for flexible inference. Recently, several of these techniques have been extended to Transformer architecture [14] 18], while they either scale part of the network or the slicing granularity is large which means they could only deliver very few models in the end. Differing from the previous works, our method is the first work to scale the ViT structure with large slicing bound and small slicing granularity which is intrinsically a more challenging problem.
3 Revisiting Slicing in Vision Transformer
Due to the excessive costs of constantly activating all the sub-networks during training, the sandwich rule is proposed in US-Net (37) to train the slimmable network at the smallest width, largest width, and 2 random intermediate widths in each iteration that the performance of the lower bound and upper bound are guaranteed. Although the intermediate sub-networks are optimized less frequently compared to Separate Training (ST), US-Net manages achieve comparable performance with ST on the CNN architecture. To have a better understanding of the distinction between CNN and ViT, we apply US-Net to MobileNetV2 28). CNN, and DeiT-S 29 a ViT, but constantly activate four sub-networks with the width ratio of (0.25, 0.5, 0.75, 1.0) at each iteration. Subsequently, we evaluate the pre- pre-trained models both inbound [0.25, 1.0] and outbound (0, 0.25) unseen width ratios to evaluate their interpolation and extrapolation abilities, respectively. Shown in Fig. 3 CNN exhibits moderate interpolation and extrapolation capabilities by achieving acceptable performance at previously unobserved widths during training. In stark contrast, ViT fails entirely at unseen widths, suggesting that optimizing larger sub-networks does not directly benefit the performance of smaller ViTs, even though their weights are shared in nested nature.
3

---

60
70
50
40
Acc (%)
30
60
20
Se-Ne(CCNN)
55-
10
US Net(VVT)
50
A
US Net
Avg
+1,8
US-Net*
0
40.2
0.2
0.3
0.4
0.5
Width Ratio
Width Ratio
Figure 3: Evaluating US-Net over CNN and ViT at unseen width ratios to examine the interpola- tion (denoted and extrapolation (denoted +) abilities on ImageNet-
Figure 4: We train US-Net VIT-S without con- stantly activating the smallest subnet (denoted as *) and observe average performance gain of 1.8% at other width ratios on ImageNet-IK.
We analyze the results from the expected training epochs for each sub-network. Let X represent the total number of networks to be delivered, wherein X - intermeddite sub-networks are included, and according to the sandwich rule, two of these are randomly sampled during each iteration. Formally, the expected training epochs for the intermediate networks can be expressed as:
2
where y is the number of training epochs for the full model and suggests that the optimization of most sub-networks falls notably short compared ST. As ViT has demonstrated minimal interpolation ability at unseen widths compared to CNN, each sub-network within the slimmable ViT requires optimal utilization of every training iteration achieve satisfactory performance. Nevertheless, the smallest subnet is constantly activated during training according to the sandwich rule and we hypothesize that the over-emphasis the smallest sub-network, often exhibits the worse performance, may increase the training difficulty of other subnets as their weights are shared in a nested nature. To validate we implement US-Net (37 on DeiT-S without constantly activating the smallest sub-network. Fig- verifies our hypothesis showing an accuracy drop at the smallest subnet but significant performance improvement at other width ratios.
4 Scala
We first introduce the training and inference paradigms of Scala. Then, we describe Isolated Activation which disentangles the smallest subnet from other sub-networks while maintaining the lower bound performance. Further, we present Scale Coordination to ensure each subnet receives simplified, accurate, and steady learning objectives. Without any modification to the architecture, we deliver general framework Scala which could be easily built on existing methods.
4.1 Framework
Our goal is to build general framework that makes VIT F (-) slimmable, i.e., the delivered network can be transformed into different small variants for flexible inference. First, we introduce hyperparameter to denote the width ratio of the sub-network F" (.). Based
analysis in
Fig. 3] ViTs have minimal interpolation ability. which suggests that all subnets have to be individually optimized to achieve decent performance. Following the sandwich rule [37]. sample the smallest largest (1 = 1), and random intermediate width ratios mi, m2 at each iteration during training. The corresponding sub-networks are: Fa (.), Ft (.), pm. (-) and Fm2 (.). and accumulate the gradients those subnets at each iteration. At the inference stage, the network F (.) is evaluated at an arbitrary width ratio that has been optimized during training by adjusting r.
4.2 Isolated Activation
Iluustrate Sec. constant activation of the smallest sub-network F (.) ensures own accuracy at the cost of other subnets' performance. This is dilemma as there is significant accuracy drop of F (-) if we do not constantly activate (see Fig- 4b. otherwise, the performance of other sub- networks are severely limited. To alleviate this issue, propose Isolated Activation to disentangle

---

the representation of F (+) from other sub-networks while still constantly activating it. It not only ensures the performance of the lower bound but facilitates the optimization of other subnets as well.
Formally, given the learnable weight E RC.xC,XHXW of a random layer in ViT (Co. Cj stands for the output, input channel number, H, W represents the height and width of the convolution kernel, H = W = 1 for fully connected layers), the weight of F" (-) where o is selected as:
0" 0 [: (r x Co), (r C.)]
(2)
In contrast, the smallest sub-network F (-) is activated as:
0 = 0 ( (s x Co) - (s x :]
(3)
where we slice the weights in a reverse direction so that we disentangle the representation of F (-) from other sub-networks. With this simple but critical design, we not only ensure the performance of F (-) with constant activation, but also alleviate the negative effects it brings.
4.3 Scale Coordination
We present the training strategy of Scala in this section. We follow the setting of DeiT 29 to train the full model F' (-) with knowledge distillation and introduce distillation token for knowledge transfer between sub-networks. As our goal is to scale down given network F (-) to multiple smaller variants, we simply choose the pre-trained model itself Fo (.) as the external teacher for the full network F' (-) to facilitate training. To optimize the sub-network. at different scales, present the Scale Coordination training strategy, which is composed of three techniques: Progressive Knowledge Transfer, Stable Sampling, and Noisy Calibration, to ensure that each subnet receives simplified, accurate, and steady learning objectives.
Progressive Knowledge Transfer. Given an input image v, the activated sub-network F (.) produces two predictions:
Pele: Pdace F" (v;; 0"),
where Pala and Pais denote the prediction generated by the classification and distillation head, respectively. As we activate multiple sub-networks: F (-), Irt (-), Fm (+) and Fm (+) at each iteration during training, our idea is to utilize the predictions of the larger network to facilitate the optimization of smaller subnets.
Given the sorted width ratio list R = progressively distill the knowledge of the larger network into the smaller one:
[s, mı, m2, !]. we utilize the KI. divergence
19 loss to
K
CkL = Σ
Pais log
Patts Palcs
(5)
where K represents the number of classes, r' = [ (r) 1] and (r) denotes the index of r in R. Instead of using Plas as the optimization target for all smaller networks, we ensure each subnet receives simplified learning objective as small subnet have large capacity gap compared to F (-) (e.g.. Fl (-) is almost 16 times larger than F (-) if = 0. 0.25) and minimizing their KI. loss complicates the optimization process and leads inferior performance. With Progressive Knowledge Transfer, we simplify the optimization objective for small sub-networks by utilizing F (.) and FT2 (.) as the teacher assistants to fill the gap between Fl (-) and F (-) and train them a one-shot manner.
Stable Sampling. As the knowledge is gradually transferred from the larger network to the smaller one, the two intermediate networks serve as the bridge to connect F' (-) and F (-), as F (.) the student of F (+) and Fmu (-) is the teacher of F (.). Therefore, we need carefully control the width ratios my and m2 to prevent the obvious model capacity variation.
Concretely, we introduce the slicing granularity and the number of networks X we can deliver (including the full model) with a single ViT denoted as:
X (1-b) s)
+1.
(6)
Then, we divide the slicing bound B- (e.))
into two smaller ones:
ss m
B₂ (m, I)
B
e
5

---

where m = (ل+ي) and mú, my will be the random integer sampled from the uniform distribution B1. B2, respectively. Thus, my and mg are defined as:
2_
mi x e, m2 = mi2 X
(8)
and we ensure the model capacity gap between the four networks is stable and secure the learning objective for each subnet is steady.
Noise Calibration. Although all the subnets receive guidance from larger networks, a notable issue is that the predictions from the teacher are not always accurate, sometimes even noisy, especially at the early training stage. To avoid the noisy signal dominating the optimization direction, first calculate the Cross-Entropy loss by:
=
Σ
Un log (Pais),
R=1
where ÿk represents the one-hot label for class k. Then, we calibrate the noise by combining the KL divergence loss and Cross- Cross-Entropy loss:
L = = X CKL
(10)
where À is hyperparameter used balance the two losses and empirically let À in implementations. By doing so, we mitigate the negative effects brought by the noisy predictions of the teacher model and ensure each subnet is guided by the accurate learning objectives.
5 Experiments
We validate Scala with the plain ViT structure DeiT [29]. We first analyze of the main property of Scala and compare our method with the state-of-the-ar method SN-Net [25] and Separate Training (ST) at larger scale. Moreover, we examine the transferability of Scala and its application on Semantic Segmentation. Finally, we provide ablations to validate the efficacy of our designs.
5.1 Experiment Settings
All the object recognition experiments are carried out on ImageNet-IK [ We follow the training recipe of DeiT [29] and conduct the experiments on V100 GPUs. For Scala, we set 0.25, I = 1.0, and = 0.0625 so that we could enable a single ViT to represent 13 different networks
(X 13) with a large slicing bound (i.e., F (-) is almost 16 times larger than 5 (-)).
Table I: Comparison with scaling baseline methods AutoFormer 4. US-Net 37 and Separate
Training under different width ratios r. The best results are bold-faced.
0.25
0.50
0.75
1.00
Method
Param
Acc1.
GFLOPs
Acc1.
GFLOPs
Acc1.
GFLOPs
Acc1.
GFLOPs
AutoFForner
22M
50.8%
0.4
65.6%
1.3
69.5%
2.7
69.8%
4.6
US-Net 37
22M
52.7%
0.4
60.1%
1.3
66.9%
2.7
73.2%
4.6
Separate Training
43M
45.8%
0.4
65.1%
1.3
70.7%
2.7
75.0%
4.6
Scala
22M
58.7%
0.4
68.3%
1.3
73.3%
2.7
76.1%
4.6
5.2 Proof-of-Concepu
In this part, we conduct experiments over DeiT-S 29 for 100-epoch training to prove the concept.
Comparison with scaling baselines. We compare Scala with multiple scaling baselines, including: (I) AutoFormer 4: we apply this ViT-based supernet training method into our setting scale through width; (2) US- [37]: : the prior work that obtains similar performance with ST over the CNN structure; (3) Separate Training (ST): we repetitively train the model with different widths from scratch and evaluate them individually. Tab. showws that AutoFormer lags behind Scala remarkably as we target scale wider range. US-Net shows significantly worse performance compared to ST which indicates that scaling down ViT is a more challenging problem compared to the CNN architecture. Nevertheless, Scala achieves better performance compared ST at all width ratios with one-shot training, reducing the storage costs of saving multiple models observably.

---

75
75-
70-
65
65
ACC
60
0.35
0.50
635
1.0
ST
0.25
55-
1199
-2
926
411
Scala (X=13) 13)
+129
113
136
+11
Scala (s-0.25)
50
1400
153
(X-7)
50
Scala (s-0.50)
4380
-69
41.7
Scala (X=4)
+45
+.9
Scala (s=0.75)
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0 4.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0 4.5
GFLOPs
GFLOPS
Figure Comparisons of Scala with different slicing granularity and Separate Training (ST). Improvements over ST are shown.
Figure 6: Comparisons of Scala with different slicing bounds and Separate Training (ST). Im- provements over ST are shown.
84
82
77
76
72
72
67
68
+..0
-13
-0.
62
-68
+14
+0.
+02
Uniformer S (ST)
Uniformer XS (ST)
64
57
Uniformer (Scala)
Uniformer XS (Scala)
00.0
0.0
0.5
1.0
1.5 2.0 2.5 3.0 3.5 4.0 GFLOPs
0.4
0.8
1.2
1.6
GFLOPs
Figure 7: Comparisons of Scala and Separate Training (ST) over the CNN-VIT hybrid architec- ture Uniformer-S 20 Improvements over ST shown.
Comparisons of Scala and Separate Training (ST) over lightweight model Uniformer- XS (20] with token pruning. Improvements over ST shown.
Figure 8:
Slicing Granularity and Bound. Fig. 5 shows the results of various slicing granularity First, Scala outperforms ST with different and the advantage small width ratios is more obvious, which promises its application on edge devices. Moreover, it is shown that less fine-grained granularity results in better overall performance with the same slicing bound as the expected training epochs for intermediate subnets increase correspondingly. We further conduct experiments with different while fixing I and to study the effect of the slicing bound. Fig. shows that smaller bounds lead to markedly better performance and it further verifies that slicing through large bound is intrinsically more difficult, which distinguishes Scala from the supernet training methods
38 4 in NAS.
Application on Hybrid Structures. We experiment Scala on the CNN- VIT hybrid architecture
Uniformer-S 20). As Uniformer contains Batch Normalization (BN) [5] which cannot be directly evaluated after slicing because of normalization shifting [39], we calibrate the statistics of BN before inference following (37). Shown in Fig. 7 Uniformer-S is scaled down to 13 different variants with better performance compared to ST, demonstrating the generalization ability of Scala. However, performing BN calibration each width ratio requires considerable extra effort. This highlights the benefit of VIT, as Layer Normalization (LN) allows direct evaluation without additional operations.
Application Lightweight Structures. We further validate Scala lightweight structure Uniformer-XS 220 which integrates the design of token pruning and train these methods for 150 epochs. Shown in Fig. Scala still matches the performance of ST and exhibits significant advantage at small width ratios, which promises its application on edge devices with limited budget.
Fast Interpolation of Slimmable Representation. Training models with different slicing granularity from scratch is time-consuming and here we show that the slimmable representation of certain granularity be scaled to others with small amount of training epochs. Specifically, we train the model with the original for 70 epochs and decrease the value e in the last 30 epochs to deliver more sub-networks for higher inference flexibility. Fig. shows the results of fast interpolation are similar to those trained from scratch and the newly appeared sub-networks are quickly interpolated to achieve decent performance. We further increase for sub-netwee nebworsssks with higher performance and the phenomenon shown in Fig. iS similar to down interpolation. Besides, we observe that the accuracy of abandoned sub-networks gradually decreases but they maintain the performance
great extent.
Slimmable Ability across Architectures. We examine the slimmable ability of different architectures in Tab. by applying Scala on different architectures and evaluating these networks unseen width

---

75-
75
75
71
Acc (96)
65
Acc (36)
65
1123
127
67
1126
+35
[=1,00000
8
115
+43
-12
55
55
50
Stale (X-25)
-- --0.55625
50
Scala IX-11
59
--2.875
Scal (X-13:25)
F-0.4002
Stala (X-13-7)
T-0.4375
45
45
15 Epoch
GFLOPs
GFLOPs
Epoch
(a) Results of w/ and w/o down interpolation.
(b) Accuracy change of the introduced subnets.
(a) Results of w/ and w/o
(b) Accuracy change of the abandoned subnets.
up interpolation.
Figure 9: Down Interpolation. We reduce the slicing granularity in the last 30-epoch training of DeiT-S (29] on ImageNet- KK to deliver more sub-networks for higher inference flexibility.
Figure 10: Up Interpolation. We increase the slicing granularity in the last 30-epoch training of DeiT-S (29] on ImageNet- IK to deliver fewer sub-networks for higher performance.
Table 2: Slimmable ability examination over different architectures on ImageNet- IK under various width ratios. The interpolated results are shown blue color.
Top-1 Acc. (%)
Architecture
Model
0.40625
0.4375
0.46875
0.50
0,53125
0.5625
0.59375
VIT
DeïT-S
1.7
66.4
1.6
68.3
1.9
70.0
1.9
CNN-ViT
Uniformer-S
30.4
73.5
27.8
75.5
32.6
77.0
37.5
CNN
MobileNet
58.8
60.4
60.7
61.6
61.8
64.3
64.4
ratios to explore the interpolation ability. CNN exhibits very strong interpolation ability as the performance at unseen widths lies in the range of trained width ratios. In contrast, CNN-ViT and VIT suffer from remarkable performance decreases different extents and VIT achieves almost zero accuracy which further validates that the problem we target to solve, i.e., slicing ViT, is the most challenging one.
5.3 Comparisons over Extended Training
In this section, we perform training DeiT-B 29 for 300-epoch training following the standard protocol on ImageNet-1K g to compare with the state-of-thhe-ar.
Comparisons with state-of-the-art. SN-Net [25] is state-of-the-art work that supports flexible inference on ViT. Specifically, it utilizes several pre-trained models (e.g.. DeiT-Ti/S/B) to construct supernet and inserts additional layers to build dynamic routes for flexible inference. Shown in Tab. 3 we empirically compare Scala with SN-Net over DeiT-B following the standard 300-epoch training protocol [29], Scala obtains similar performance with SN-Net at large width ratios and clearly outperforms it at small computational budgets. Besides, SN-Net has to preserve the parameters of multiple models and additional layers, while Scala only needs to keep the weights of the full network. When adopting the stronger teacher network [27] as SN-Net does, Scala outperforms SN-Net with an average improvement of 1.6% across all width ratios.
Comparisons with Separate Training. In Tab. 4 we compare with ST on DeïT-B 29 with longer training process, i.e., 300-epoch training, where and DeiT-B, respectively. Scala exhibits a clear advantage at r = 0.25 and matches of ST except 0.50 due to significantly less training time. When X 7, we can achieve similar performance at r = 0.50 with 40% training epochs of ST. Further reducing X to 4, resulting in the constant activation of the two intermediate networks, allows to consistently outperform ST at all width ratios. This substantiates the effectiveness of Scala and the slimmable representation.
0.25, 0.50, 1.00 corresponds to DeiT-Ti, DeiT-S
performance
Table 3: Comparison with SN-Net 225 over DeiT- B [29] I K. O & denotes utilizing
Table 4: Comparison with Separate Training (ST) over DeiT-B 29 on ImageNet- K under different width ratios denotes the expected training epochs of each model.
DeiT-B (2), RegNetY- 16GF model to facilitate training.
27 as the teacher
Method
Param
0.25
0.375
0,50
Top-1 Apc. (%)
0.625
0.75
0.875
1.00
Method
Parun
Acci
Accl.
Arzi
Acc1
SN Net 23
118M
70.6
74.8
79.5
79.3
81.3
81.9
$1.9
S7
163M
72.2%
79.9%
300
81.0%
300
$1.8%
Scala
Sesiia
86M
96M
75.4
75.3
77.2
76.9
79.3
79.7
80.5
80.9
81.2
81.8
82.2
81.6
K2.9
82.0
Scala (x+13)
Scala (X-4)
Scala (C+7)
8684
8684
863
75.6%
75.336
75.3%
80.9%
79.7%
70.3%
300
120
55
81.4%
81.2%
81.9%
300
120
82.0%
$2.2%
$2.0%

---

5.4 Transferability
To assess the transferability of Scala, we employ DeïT-B (2 as the backbone for 300-epoch pre-training ImageNet-1K KK and leverage the foundation model DINOv2-B [24] the teacher network to inherit good behaviors. Our study aims address two key questions:
Whether the slimmable representation can be transferred to downstream tasks? As depicted in Fig. Ta Scala consistently outperforms Separate Training (ST) across all width ratios, despite the intermediate sub-networks being trained for approximately 55 epochs. After that, we conduct linear probing on video recognition dataset UCF101 with 8 evenly sampled frames and average their features for the final prediction. For the classification head added on Scala, we make it slimmable to fit the features with various dimensions and follow the same training protocol as in object recognition. In Fig. TTb two notable observations emerge: (1) Scala consistently outperforms ST across different width ratios on the UCF101 dataset, implying the great transferability the slimmable representation; (2) Scala retains its slimmable ability when applied to new task and exhibits promising performance across a wide slicing range (10~141 GFLOPs), promising its application on other downstream tasks.
83
84
81
79
80
77
78
75
100
ST
76
0.25
0,50 0.75
1.00
73
Scala
Scala
74
3
9
15 18
10
40
100
130
GFLOPs
GFLOPs
(a) Pre-training ImageNet-1K.
(b) Linear probing UCF101 with sampled frames.
Figure 11: Transferability of Seala. We first conduct pre-training on ImageNet- IK with the help of foundation model DINOv2-B [24]. Then conduct linear probing video recognition dataset UCFI01. Improvements over ST are shown.
Whether the generalization ability can be maintained in the slimmable representation? Inspired by the work 43 which replicates the success of vision foundation models on ImageNet-IK, we remove all the Pross-Entropy losses during training to alleviate the dataset bias issue and inherit the strong generalization ability of the teacher network DINOv2. Then we conduct linear probing on 12 fine-grat -ne-irained classification datasets following the setup in DINOv2. Tab. Jsshoss that Scala significantly outperforms DeiT variants on the average performance of fine ane-grained classification which suggests that Scala indeed inherits the fruitful knowledge from DINOv2 with remarkable improvement in its generalization ability. Moreover, the improvement over DeiT does not decrease when we scale down the width ratios during inference and it indicates that Scala maintains the flexible inference capability very well even though it contains more knowledge than before.
Table 5: Comparison of Scala and DeiT on 12 fine-grained classification datasets. Scala-B is distilled from DINOv2-B on ImageNet-IK for 300 epochs and then we conduct linear probing on fine-grained datasets. The average accuracy of 12 datasets shown in the last column.
Airezaft
Cal101
Flowers
Average
Car
C10
C100
DTD
Food
Pets
SUN
VOC
CUB
Method
Arch
DeiT-B
1.00
VIT-B
49.1
90.7
57.1
95.6
70.3
90.7
76.9
92.8
63.0
83.0
72.8
DeïT-S
1.00
VIT-S
44.5
90.3
94.1
69.5
88.8
72.1
92.1
60.5
83.3
71.9
74.5
DeiT-Ti
1.00
ViT-Ti
39.9
88.8
39.7
90.6
72.5
67.0
85.1
65.0
91.1
56.1
81.6
69.4
70.6
Scala-B
1.00
VTT-B
63.0
95.1
74.3
97.3
86.6
78.0
97.3
83.4
94.6
69.7
87.6
84.1
84,2
Scala-B
0.50
WT-S
55.2
93.9
64.4
95.5
81.9
74.9
96.0
78.6
94.0
65.9
86.1
81.1
80.6
Scala-B
0.25
VIT-i
49.6
92.1
55.4
93.6
78.1
72.1
93.4
72.8
92.9
61.4
84.2
76.7
76.8
5.5
Dense Prediction
In previous sections, we have validated the effectiveness Scala on classification tasks, we further examine whether the slimmable representation could be transferred for dense prediction task like

---

Table 6: Evaluation of slimmable representa-
Table 7: Ablation study of Scala over DeiT-S [29
tion on dense prediction task Semantic Seg-
on ImageNet-1K under various width ratios. IA,
mentation over ADE20K [45] dataset. We
PKT, SS, NC denote Isolated Activation, Progressive Knowledge Transfer, Stable Sampling, Noise Cali-
equipped the pre-trained Uniformer-S [20]
from Fig. with Semantic FPN [16] and com-
bration, respectively. The best results bold-faced.
pare the sub-networks extracted from Scala
Top-1 1 Acc. (%)
with Separate Training (ST).
Method
0.25
0.375
0.50
0.625
0.75
0.875
1.00
Backbone
mloU. (%)
Scala
58.7
63.4
68.3
71.3
73.3
74.4
76.1
0.25
0.50
0.75
1,00
w/o IA
57.3
61.5
66.4
69.8
72.0
73.4
75.8
PKT
53.0
60.1
65.6
68.8
71.7
73.7
76.2
Uniformer-S (ST)
33,9
40.4
42.6
45.3
w/ SS
58.7
62.7
68.1
71.3
73.1
74.2
75.9
Uniformer-S (Scala)
35.0
40.7
43.7
46.1
w/o NC
50.2
62.7
67.2
70.5
72.7
74.0
76.3
semantic segmentation. We utilize the pre-trained model Uniformer-S [20] drawn from Fig. 7 which has a hierarchical design and is obtained by 100-epoch training (our results lag behind official results where the backbone is trained for 300 epochs), and equip it with Semantic FPN [16]. To compare with Separate Training (ST), extract four subnets from Scala (Uniformer-S) and train them separately. Shown in Tab. 6 Scala outperforms ST at all widths which verifies the slimmable representation benefits the downstream tasks. Note that we do not scale the decoder as involves extra designs and is out of the scope of this work. However, we show that the slimmable representation can be generalized to semantic segmentation as feature extractors because the feature maps are spatially intact, promising its application as an end-to-end slimmable framework on dense prediction tasks.
5.6 Ablation Study
We conduct ablation to examine the effectiveness of our designs in Tab. 7 First, we build Scala without Isolated Activation so that the smallest sub-network will entangle with others and it shows an obvious performance drop at all width ratios. Then, remove Progressive Knowledge Transfer (PKT) and pass the knowledge from Fl (-) to smaller subnets through classification token following US-Net 33.. It shows much worse performance, especially small ratios, which proves the strength of PKT as it implicitly introduces some teacher assistants to simplify the optimization objective for small sub-networks. Further, we random sample the width ratios of m2 and m2 between (s, and compare it with Stable Sampling (SS). The results are slightly inferior to SS which suggests SS is helpful in securing the steady learning objective for each sub-network. Finally, we remove Noise Calibration (NC) from Scala and only use the predictions from larger networks to guide the small subnets. shows remarkable performance drops small width ratios, where the noise from the teacher network is most obvious, demonstrating the effectiveness of NC in calibrating the noise and providing accurate signals for sub-networks.
Conclusion and Limitations
In this paper, observed that smaller ViTs are intrinsically the sub-networks of a large VIT with different width ratios. However, slicing VIT is very challenging due to its poor interpolation ability. To address this issue, we proposed Scala to enable a single network to represent multiple smaller variants with flexible inference capability. Specifically, we proposed Isolated Activation to disentangle the representation of the smallest subnet from others and presented Scale Coordination to ensure the sub-network receives simplified, steady, and accurate learning objectives. Extensive experiments on different tasks prove that Scala, requiring only one-shot training, outperforms the of-he- art method under different computations and matches the performance of Separate Training with significantly fewer parameters, promising the potential as a new training paradigm.
One limitation of Scala is the longer training time compared to conventional supervised learning of single model, attributable to the activation of multiple subnets during training. Nevertheless. our training time is obviously less than separately training all the sub-networks. In the future, we aim to enhance the training efficiency of Scala.
10

---

References
[1] Achiam, Adler, S. Agarwal, L. Ahmad, Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint XXXxv:/908.000 2019.
[3] Y.-H. Cao, P. Sun, and Zhou. Three guidelines you should know for universally slimmable self-supervised learning. In CVPR, 2023.
[4] M. Chen, H. Peng, J. Fu, and H. Ling. Autoformer: Searching transformers for visual recognition. ICCV, 2021.
[5] X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman,
A. Gryener, B. Mustafa, L. Beyer, al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint 2rXiv:2299006994 2022.
[6] E. D. Cubuk, B. Zoph, J. Shlens, and Q. Le. Randaugment: Practical automated data augmentation with a reduced search space. In CVPRW, 2020.
[7] M. Dehghani, Djolonga, B. Mustafa, P. Padlewski. Heek, Gilmer, A. Steiner, M. Caron, R. Geirhos, L. Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In ICMI., 2023.
[8] Deng, W. Dong, Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
[9] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom,, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 11929, 2020.
[10] M. Grimaldi, Mocerino, Cipolletta, and A. Calimera. Dynamic convnets on tiny devices via nested sparsity. IEEE Internet of Things Journal, 2022.
[11] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
[12] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. CVPR, 2016.
[13] G. Hinton, O. Vinyals, J. Dean, et al. Distilling the knowledge in neural network. arXiv 2015.
preprint
[14] L. Hou, Z. Huang, L. Shang, X. Jiang, X. Chen, and Q Liu. Dynabert: Dynamic bert with adaptive width and depth. NeurIPS, 2020.
[15] Ioffe and Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[16] A. Kirillov, R. Girshick, K. He, and P. Dollár. Panoptic feature pyramid networks. In CVPR, 2019.
[17] A. Kinillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo. et al. Segment anything. arXiv preprint WrXi:2330.002433 2023.
[18] S. Kudugunta, A. Kusupati, T. Dettmers, K. Chen, Dhillon, Y. Tsvetkov, H. Hajishirzi, S. Kakade, A. Farhadi, P. Jain, et al. Matformer: Nested transformer for elastic inference, arXiv preprint arXiv:2310.07707, 2023.
[19] S. Kullback. Information theory and statistics. Courier Corporation, 1997.
[20] K. Li, Wang, J. Zhang, P. Gao, Song, Y. Liu, H. Li, and Y. Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. arXiv preprint 2201v02001.09450 2022.

---

[21] Z. Li and Q. Gu. I-vit: Integer-only quantization for efficient vision transformer inference. In ICCV, 2023.
[22] Z. Liu, Y. Wang, K. Han, W. Zhang, S. Ma, and W. Gao. Post-training quantization for vision transformer. NeurIPS, 2021.
[23] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[24] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini wrXiv:2304.07192 2023.
[25] Z. Pan, J. Cai, and B. Zhuang. Stitchable neural networks. In CVPR, 2023.
[26] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. ICML, 2021.
[27] I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. Dollár. Designing network design spaces. In CVPR, 2020.
[28] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
[29] H. Touvron, M. Cord, M. Douze, F, Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers distillation through attention. In ICML, 2021.
[30] H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou. Going deeper with image transformers. In ICCV, 2021.
[31] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[32] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaci, N. Bashlykov, S. Batra, Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09228, 2023.
[33] A. Vaswani, Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. NeurIPS, 2017.
[34] Vu, M. Eder, T. Price, and J.-M. Frahm. Any-width networks. In CVPRW, 2020.
[35] K. Wu, H. Peng. Z. Zhou, B. Xiao, M. Liu, L. Yuan, H. Xuan, M. Valenzuela, X. S. Chen,
X. Wang, et al. Tinyclip: Clip distillation via affinity mimicking and weight inheritance. In ICCV, 2023.
[36] K. Wu. J. Zhang, H. Peng. M. Liu, B. Xiao, J. Fu, and L. Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In ECCV, 2022.
[37] J. Yu and T. S. Huang. Universally slimmable networks and improved training techniques. In ICCV, 2019.
[38] J. Yu, P. Jin, H. Liu, G. Bender, P.-J. Kindermans, M. Tan, T. Huang, X. Song, R. Pang, and Q. Le. Bignas: Scaling up neural architecture search with big single-stage models. ECCV, 2020.
[39] Yu, L. Yang, N. Xu, J. Yang, and T. Huang. Slimmable neural networks. arXiv preprint arXiv:1812.0888 2018.
[40] Yun, D. Han, Oh, S. Chun, J. Choe, and Y. Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
12

---

[41] X. Zhai, A. Kolesnikov. N. Houlsby, and L. Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104-12113 2022.
[42] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. arXiv preprint arXiv:1110.09412, 2017.
[43] Y. Zhang, X. Ma, Bai, H. Wang, and Y. Fu, Accessing vision foundation models at imagenet- level costs. arXiv preprint 2rXiv:2407.10366 2024.
[44] Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang. Random erasing data augmentation. In AAAI, 2020.
[45] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Toraalb.. Scene parsing through ade20k dataset. CVPR, 2017.
[46] D. Zhou, B. Kang, X. Jin, L. Yang, X. Lian, Z. Jiang, Q. Hou, and J. Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv: 11886 2021.
13

---

A Appendix
A.1
Implementation Details
For Separate Training (ST). we follow the exact training strategy of the official DeiT Uniformer [20] setting. We use random horizontal flipping, random erasing 444 Mixup (2), CutMix [40], and RandAugment [6] for data augmentation. AdamW (3 is utilized as the optimizer with momentum of 0.9 and weight decay of 0.05. We set the learning rate 1e-3 and decay with a cosine shape. The models are trained on 4 V100 and A100 GPUs with a total batch size of 1024. We adopt Exponential Moving Average (EMA) following the official setting.
29 and
While we utilize the pre-trained ST model (r = 1.00) as teacher for F² (.) to facilitate training as mentioned in the Scale Coordination section in the main paper, we adopt larger learning rate (2e-3) and mild data augmentation (reduce the magnitude for RandAugment [6] to and turn off repeated augmentation) because Scale Coordination already regularizes the network training strongly. At every training iteration, we activate four sub-networks separately based on Stable Sampling and accumulate their gradients for backpropagation. The rest hyperparameters are set the same those in Separate Training.
A.2 Slimmable Ability of Vanilla Representation
Tab. T shows that ViT is not slimmable if we directly evaluate the vanilla pre-trained model at other widths. Here, further explore the slimmable ability of vanilla representation and fine-tune the vanilla pre-trained model with Scala. Tab. [ shows that fine-tuning obtains obviously worse performance small width ratios compared training from scratch, which denotes that the vanilla representation is not slimmable and is essentially different from the slimmable representation.
Table 8: Comparisons of training from scratch and fine-tuning ImageNet-I IK.
Top-1 1 Acc. (%)
Method
0.25
0.50
0.75
1.00
DeiT-S
29
75.0
Fine-tune
23.4
52.3
65.4
74.0
Scratch
58.7
68.3
73.3
76.1
A.3 Larger Slicing Bound
As discussed in the main text, the slicing bound has a huge impact on the performance Scala. Here further expand the slicing bound from [0.25, 1.00 to [0.125, 1.000). As shown in Fig. 12 Scala suffers from an obvious performance drop at r = 0.25 as it not constantly activated in the new setting. Nevertheless, our method still manages outperform ST all width ratios and shows significant advantage at the smallest ratio r = 0.125.
75
Acc (%)
55
Scala (s=0.125)
Scala (s=0.25)
Scala (s=0.50)
Scala (5=0.75)
25
1.0
1.5
2.0
2.5
3.0
3.5
4.0 4.5
GFLOPs
Figure 12: Increase the slicing bound to [0.125, 1.000) over DeiT-S
29 on ImageNet-IK
A.4 Longer Training Process
Previous experiments validate that Scala achieves achieves performance comparable to that of Separate Training (ST) on DeiT-B [29] in the 300-epoch training setting, even though the intermediate sub-
14

---

Table Comparison with Separate Training (ST) over DeiT-B 22 on ImageNet- IK with different training epochs under different r. & denotes the expected training epochs of each model.
0.25
0.50
0.73
1.00
Method
Epoch
Acc1.
Accl.
{
Accl.
AccI.
72.2%
79.9%
300
81.0%
300
81.8%
Scala (X=13)
300
75.3%
79.3%
55
81.2%
55
82.0%
Scala (X=7)
75.3%
79.7%
120
81.4%
120
82.0%
Scala (X=13)
400
75.7%
79.5%
73
81.5%
73
82.3%
Scala (X=7)
400
76.0%
80.1%
160
81.7%
82.4%
networks training time is much less. We further extend the training process to 400 epochs in this section and the results are shown Tab. 9 The overall performance at various width ratios is improved with longer training and Scala (X = 7) outperforms ST at all widths even though the expected training epochs for intermediate sub-networks are still much less than ST.
A.5 More Ablation Studies on Activation Method
F'()
Pud (+)
pm
PO
F'()
F ()
pmn (
F*())
F(()
(m
F(()
Figure 13: Illustration of different activation methods.
Table 10: Ablation study of various activation methods over DeiT-S 29
ImageNet-I IK under
various widths. Different activation methods are illustrated in Fig.
13
The best results are bold-faced.
Top-1 Acc. (%)
Method
0.25
0.375
0.50
0.625
0.75
0.875
1.00
Scala+(a)
58.7
63.4
68.3
71.3
73.3
74.4
76.1
Scala+(b)
57.6
61.3
64.9
72.4
74.1
74.7
76.1
Scala+(c)
58.3
63.3
67.2
69.4
72.0
72.9
74.8
We validated the effectiveness of Isolated Activation by removing this component and we further conduct more ablation studies on the activation methods where the designs are illustrated in Fig. [3 As shown in Tab. 10 choice (b) leads slightly better performance at F (.), but the performance drops at Fmi (.) significantly as is entangled with F (-) and the over-emphasize of F (.) adversely affect its performance. On the other hand, choice (c) results in similar performance at Fm
but the accuracy decreases significantly at = (-) which further verifies our hypothesis that F (-) should be isolated to reduce its negative impact other sub-networks.
A.6 Verification of Slimmable Ability
In the main text, we found that ViT has the minimal interpolation ability compared to the CNN structure. This suggests that optimizing larger sub-networks does not directly contribute to the performance improvement of smaller variants, even though their weights are shared a nested nature. A further question is, whether ViT can still maintain the slimmable ability for the unseen width ratios during training.
To verify this point, we respectively fix the width ratios of m2, m. to 0.8125 and 0.4375 during training, so that only one sub-network is optimized each range. Fig. shows that the accuracy of unseen sub netwokss is very low due to the lack of interpolation ability. Nevertheless, the performance
15

---

80
Scala
Scala [fi m2)
10
Scala (fix m;)
0.0
1.0
2.0
3.0
3.5
4.0
4.5
GFLOPs
Figure 14: Verification of Slimmable Ability over DeïT-S 129 on ImageNet-IK. We respectively fix the width ratio of m2, mi to 0.8125 and 0.4375, and observe the performance other ratios is affected.
other width ratios remains similar to the default setting even though their weights are shared with each other. This indicates that the correlation between sub-networks in VIT is weak and further highlights how challenging this problem
A.7 Comparisons with Distillation Baselines
Table 11: Comparison with baseline methods US-Net and Separate Training (ST) by adding external teacher on ImageNet- IK dataset under different width ratios r. All methods are built upon DeiT-S (29] which are trained for 100 epochs and denotes the expected training epochs of each model. The best results are bold-faced.
0.25
0.50
0.75
1.00
Method
Param
Accl.
5
GFLOPs
Acel.
GFLOPs
Acel.
{
GFLOPs
Accl.
GFLOPs
US Net
22M
52.7%
100
60.1%
1.3
66.9%
18
2.7
73.2%
100
4.6
ST
43M
45.8%
100
65.1%
100
1.3
70.7%
100
2.7
75.0%
100
4.6
US-Net-KD
51.6%
58.7%
73.8%
ST+KD
43M
48.8%
100
0.4
67.5%
100
1,3
73.3%
100
2.7
76.4%
100
4.6
Scala
58.7%
100
68.3%
1.3
73.3%
2.7
76.1%
While we have shown that Scala outperforms baseline methods US-Net [37] and Separate Training (ST), we further compare Scala with much stronger baselines, by adding an external teacher to their full network during training. Specifically, we adopt the pre-trained full model from ST as the teacher and conduct knowledge distillation for models with different widths separately. Tab. []] shows that ST+KD exhibits similar performance larger width ratios with Scala, despite that Scala clearly outperforms ST+KD at smaller widths, promising its application on edge devices. Although obtaining better full model, US-Net+ exhibits worse performance at smaller width ratios because utilizes the full network as the teacher for all subnets and this phenomenon verifies our motivation of proposing Progressive Knowledge Transfer.
A.8 Comparisons in Training Time
Assuming deliver 13 models in the end, we compare the training time (100 Epoch) of Scala with US-Net (37), Separate Training 8 A100 GPUs. The difference between US-Net and Scala is not large as the transformer architecture has been well-optimized on GPU and we do observe a significant time gap between Scala and Separate Training as they have to train 13 models iteratively. Moreover, Scala can configured to deliver models without an increase in training time as we sample networks at each iteration in all scenarios which further highlights strengths.
Table 12: Comparisons training time with baseline methods on A 100 GPUs.
Method
Training Hours
Separate Training
123
US-Net
20
Scala
21
16

---

A.9 Comparisons with MatForrer
MatFormer only slices the FFN block in the transformer architecture it offers minor computational adjustment space and we adapt their method on DeiT-S to compare with Scala. Fig. [5 shows that Scala achieves comparable performance with (better in most cases) when 0.5 with a larger adjustment scope.
75
70
Acc (%)
65
Acc (%)
60
ST
ST
MatFormer
MatFormer 25)
55
69
50
Scala (s-0 50)
67
Scala (s-0 50)
Scala (5=0 75)
Scala (s= 0.7)
45
65
0.0
0.5
1.0
1.5
2.5 GFLOPS
3.5
4.5
1.2
1.7
2.2
2.7
3,2
3.7
2
4.7
GFLOPs
Figure 15: Comparison Scala and MatFormer over DeiT-S. Scala offers significantly broader scope for computational adjustment compared to MatFormer as MatFormer only scales the FFN block in VIT. The right figure provides detailed magnification of the left figure.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our contribution is to enable VIT to become slimmable during inference which is described in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
We do not have theory included.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have included the implementation details in the main text and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
We have included the details and will release the code soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee 7/nips.com / ssiniiiiccypyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We conduct experiments following the standard protocol and include the
details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification: Repeating experiments on ImageNet requires lots of resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided details of our computing resources in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///aarrr.ccccoooaaa//aaaaaa
Answer: [Yes]
Justification:
We have read and follow the rules.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work has no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the papers that created the datasets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "2lgIFFBLKe", "reviewer": "Reviewer_rLHb", "summary": "This paper advances an approach for training Vision Transfomers (ViTs) such that at inference time they can be dynamically adjusted to fit different budget constraints with reduced drops of performance. To this end, the authors introduce Scala, a framework that allows a single network to encapsulate and train simultaneously multiple sub-networks of different capacities and widths. The methodological backbone of this work are the Universally slimmable networks (US-Net) [37], originally devised for CNNs. The authors identify and analyze a few flaws of US-Nets: difficulty to generalize to ViTs, small interpolation and extrapolation ability to sub-network size unseen during training, impact of sustained activation of the smallest sub-network that coupled with the sandwich rule for selecting sub-networks during training leads to an over-emphasis on it at the expense of the other sub-networks.\nThe authors propose two simple strategies towards such a method for ViTs: (i) Isolated activation that separates the smallest sub-network from the other sub-networks; (ii) scale coordination consisting of a set of heuristics to ensure that each sub-network gets simple, accurate and stable learning objectives: (a) progressive knowledge transfer from larger networks to smaller ones in gradual decrease of capacity, (b) stable sampling of intermediate width ratios to avoid large variations in capacities in the sandwich, (c) noise calibration, essentially a composite loss of supervised cross-entropy and distillation from the bigger sub-network.\nScala is evaluated on several settings on the ImageNet-1k dataset with ViT-Ti/S/B, hybrid CNN-ViT architectures, lightweight networks, but also for dense prediction on semantic segmentation and self-supervised pre-training with interesting results. The baselines used here were Separate Training,  Autoformer and US-Net.", "questions": "This paper takes an interesting direction of study: how to train ViTs such that a fine-grained elasticity in terms of sub-network sizes is possible at runtime.\nThe proposed Scala approach is well described, makes sense and achieves good results in several computer vision settings.\n\nI do have a few concerns related to the phrasing of this type of works (\"scalable representation learning\") which can be confusing, the absence of larger architectures and of recent relevant baselines.\nMy current rating is mildly positive (rather on the fence though) and I'm looking forward for the rebuttal.\n\nHere are a few questions and suggestions that could be potentially addressed in the rebuttal or in future versions of this work (please note that suggested experiments are not necessarily expected to be conducted for the short rebuttal period):\n\n1. Please clarify the points raised in the clarity section: use of teacher model for Scala and US-Net, implementation of transferability experiment. \n\n2. Comparison of training cost between Scala (including teacher training), US-Net and Separate Training baselines.\n\n3. Add a discussion of differences and when possible experimental comparison with Matformer and SortedNet baselines on image classification or semantic segmentation.\n\n4. Extension of experiments to NLP architectures and tasks in the style of SortedNed, Matformer", "limitations": "The authors addressed some of the limitations in the conclusion section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "### Significance\n- the paper deals with a challenging and useful task for deploying ViT models into different operational settings with different computational constraints without retraining or distilling specific architectures each time\n\n- although a computational overhead is expected for such methods, the main components of Scala are relatively simple and make sense \n\n- Scala achieves good performance with a higher boost in the low parameter regime\n\n### Originality\n- the proposed contributions are somehow incremental as they are improving the US-Net prior work, but do have some novelty and they are simple.\n\n### Clarity\n- in general this work is well argued and easy to follow. The authors construct well the arguments regarding the challenges when going from CNNs to ViT with US-Net and how to construct their Scala approach.\n\n### Quality\n- the paper offers several experiments and studies in the main paper and in the appendix (longer training, fast interpolation, ablation of components) that are well thought and improve the understanding of the method.\n\n- I appreciate the experiments beyond image classification, on semantic segmentation, as well as the self-supervised pretraining and subsequent linear probing on a downstream task.", "weaknesses": "### \"Scalable\" naming\n- I think that the framing of the method as _\"scalable representation learning\"_ is quite confusing as it's not representative for this task, it's not a name used by other related works. Importantly, it can be easily mistaken with most works that use \"scalable\" for depicting the ability/property of a system (method, architecture) to handle a growing amount of data, parameters, and the potential to accommodate this growth. In other words \"scalable\" is rather used for depicting scaling up, whereas this work depicts the property of the proposed approach to accommodate sub-networks of different lower sizes/scales from the original.\n\n- maybe other names us in related works would be more appropriate here: slimmable, elastic, modular, flexibile inference, etc.\n\n\n### Limited baselines and related work\n- some relevant related works dealing with tranformer networks are either just briefly mentioned, e.g., Matformer  [18], or not mentioned at all, e.g., SortedNet [a], Early exit [b]\n\n- One of the main baselines, US-Net is originally designed for CNNs and, as the authors mentioned, moving to ViTs is not straightforward. Matformer is criticized for the limited number of models produced, but can be considered in the several experiments with X=4 sub-networks. Matformer and SortedNet could be included in the experimental evaluation\n\n\n### Scope of experiments\n- While the authors considered several settings for computer vision tasks (image classification, segmentation, light architectures), transformer architectures are also encountered in NLP (as mentioned by the authors in L56). In such cases the original models can have much more parameters and elastic inference for lower computational budgets would be of high interest.\n\n- It would be useful to include an experiment from NLP in the style of those from Matformer or SortedNet.\n\n- The biggest architectures used here is a ViT-B (~86M params). Extending experiments to larger modern architectures would be definitely useful and interesting.\n\n### Clarity\n- it's not always clear in the text and cost estimations that Scala needs a pre-trained full network as teacher for the distillation. This add some cost in compute and time in the end. Besides it's not clear whether US-Net also needs and uses a pre-trained teacher in the reported results.\n\n- in the intro, the authors mention that they address the issue of minimal interpolation ability of ViTs. Results from Table 2 show that the interpolation abilities of ViTs with Scala are still very low. However the fast interpolation strategy from $\\S$A.2 is actually interesting for practical settings even though not fully solving this issue. It might be worth moving up in the main paper.\n\n- the idea of the transferability experiment ($\\S$5.4) with DINOv2 is nice. From the description it is not clear whether DINOv2 was used as teacher for the distillation or also as supervised pre-training on ImageNet-1k? Or the pre-training on ImageNet-1K was done in a supervised manner as in previous experiments?\n\n- the ablation experiment from Table 6 is nice. However the presentation with removing one component at once offers only a partial understanding of the contributions of each module. Different configurations with different modules in on/off mode should give a better global understanding.\n\n\n\n**References:**\n\n[a] Valipour et al., SortedNet: A Scalable and Generalized Framework for Training Modular Deep Neural Networks, arXiv 2023\n\n[b] Xin et al., Deebert: Dynamic early exiting for accelerating bert inference, ACL 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the Reviewer’s detailed comments and constructive suggestions for us to improve our work. We make the response as below.\n\n***\n\n**Weakness 1: Phrase.**  \n\nThanks for the great suggestion and we will modify the presentation in our final version. Our motivation for adopting 'scalable representation learning' is that different methods (DeiT, MAE, DINOv2) will learn different representations, which are not naturally slimmable (see Sec A.3). Scala is proposed to learn the original representations but with the slimmable ability, so we adopt 'scalable representation learning' to describe our method. However, we do agree that the term 'scalable' may result in misunderstanding and we will modify it in the final version.\n\n**Weakness 2: Limited baselines.**  \n\nThanks for the comment. \n\nSortedNet scales the network through multiple directions and results in irregularities in transformer architectures, compared to Scala which aligns with the inherent design of ViT to vary from widths. As their vision experiments are conducted on small-scale CIFAR datasets, we intended to reproduce their method on ImageNet but did not find released implementations. \n\nMatFormer only scales the FFN block in transformer so it offers a minor computational adjustment space and we adapt their method (written in JAX) on DeiT-S for a fair comparison. The results are shown in Fig.1 of the PDF file (reply to all reviewers) and Scala achieves comparable performance with it (better in most cases) when $s=0.5$ with a larger adjustment scope.\n\n\n**Weakness 3: Scope of experiments.**  \n\nWe thank the Reviewer for the valuable suggestion. Indeed, extending Scala to NLP and a larger network is part of our original plan but we are struggling with computation resources as both of the experiments demand much larger training costs. We will definitely consider it as our future plan.\n\n**Weakness 4: Teacher and training costs.**  \n\nAs explained in Weakness 1, Scala can be regarded as a tool to make existing methods become slimmable while trying to maintain the original representations. \nAlthough we can expect to learn similar representations of methods trained with supervised learning on ImageNet-1K (DeiT), some self-supervised learning methods (DINOv2) are trained with gigantic undisclosed data which is almost impossible to reproduce their performance without utilizing the pre-trained model. \nSo we simply use the pre-trained network (same size as the student) as the teacher to imitate the original model to inherit the original representations, instead of utilizing a larger network which usually leads to better performance.\n\nFor US-Net, we do not introduce teacher networks following their original design but we do compare with Separate Training with Distillation in Sec A.8 (Tab. 10), where Scala still outperforms the baseline at smaller widths. Here we further re-implement US-Net by adding the teacher and the results are shown in the table below. Although obtaining a better full model, US-Net exhibits worse performance at smaller width ratios because it utilizes the full network as the teacher for all subnets and this phenomenon verifies our motivation of proposing Progressive Knowledge Transfer.\n\n\n| Method  | $r=0.25$ | $r=0.50$ | $r=0.75$ | $r=1.00$ |\n| :-----: | ------------ | ------------ | ------------ | ------------ |\n| US-Net | 52.7% | 60.1% | 66.9% | 73.2% |\n| US-Net+KD | 51.6% | 58.7% | 66.6% | 73.8% |\n| ST | 45.8% | 65.1% | 70.7% | 75.0% |\n| Scala | 58.7% | 68.3% | 73.3% | 76.1% |\n\nAssuming to deliver 13 models in the end, we compare the training time (100 Epoch) of Scala with US-Net, Separate Training, and Separate Training with Distillation on 8 A100 GPUs and we do not include the teacher training time as Scala is aimed to scale down existing works where the same size pre-trained model can be easily downloaded. The difference between US-Net and Scala is not large as the transformer architecture has been well-optimized on GPU and we do observe a significant time gap between Scala and ST/STD as they have to train 13 models iteratively. Moreover, Scala can be configured to deliver 25 models without an increase in training time as we sample 4 networks at each iteration in all scenarios which further highlights our strengths.\n\n| Method  | Training Hours |\n| :-----: | ------------ |\n| ST | 123 |\n| STD | 128 |\n| Scala | 21 |\n| US-Net | 20 |\n\n**Weakness 5: Interpolation.**  \n\nWe did not fully address the interpolation problem as this issue is essentially related to the architecture design of transformer and we constrain ourselves from modifying the conventional structure as it has been well-optimized in various scenarios. On the other hand, it further emphasizes that the problem we target to solve is very challenging as the intermediate subnets ($X=13$) are only trained for around 18 epochs and they are expected to match the performance of networks that are trained for 100 epochs.\n\nThanks for carefully reading our paper and we will move Fast Interpolation into the main paper.\n\n**Weakness 6: DINOv2.**  \n\nWe follow the previous setting except for using DINOv2-B as the teacher which means we still train the model in a supervised manner. We have tried to adopt the self-supervised learning objective from DINOv2 but found the training cost too large, and it is almost impossible to reproduce the performance following the original learning objective with ImageNet-1K since DINOv2 is trained on the private dataset LVD-142M. However, we re-conduct the experiment by removing all the CE loss to alleviate dataset bias and find this choice results in surprising generalization ability by inheriting the fruitful knowledge of DINOv2 (please see the reply to all reviewers).\n\n**Weakness 7: Ablation.**  \n\nThanks for the advice and we will add more ablation in our final version.\n\n***\n\nWe hope the explanations could address your concerns and we would appreciate it a lot if you could recognize the contributions of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "smfMfLLH1q", "reviewer": "Reviewer_bSkR", "summary": "The paper introduces Scala, a novel framework designed to effectively scale down Vision Transformers (ViTs) for use in environments with fluctuating resource constraints. The key insight is that smaller ViTs can function as sub-networks within a larger ViT, differing mainly in width. Scala enables a singular network architecture that can emulate multiple smaller ViTs, thereby offering versatile inference capabilities while maintaining the structural principles of ViTs. The framework uniquely incorporates multiple sub-networks during its training phase, utilizes Isolated Activation to differentiate the smallest sub-network, and implements Scale Coordination to streamline the learning objectives for each sub-network, aiming for simplicity, stability, and accuracy. The empirical results across various tasks confirm that Scala can learn scalable representations efficiently with a single training iteration, maintaining the integrity of the original ViT architecture and achieving performance on par with networks trained separately.", "questions": "please refer the weakness.", "limitations": "This paper addresses the limitations in conclusion", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed Scala framework aims to enhance Vision Transformers (ViTs) by enabling them to learn scalable representations suitable for flexible inference. This is achieved through two key innovations: Isolated Activation, which effectively disentangles the representation of the smallest subnet to maintain clarity and specificity, and Scale Coordination, which ensures that each subnet within the larger network receives simplified, consistent, and accurate signals. These mechanisms are designed to optimize the performance and scalability of ViTs, addressing common challenges in adapting these architectures to varied and dynamic operational contexts.", "weaknesses": "1. Recent papers[1,2,3] with \"Scalable\" usually scale ViT to billion size with large scale datasets like DFN, JFT, and Datacomp. Therefore, I suggest authors should reconsider if the experiments can support \"Scalable\".\n\n\n[1] Zhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022). Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 12104-12113).\n\n[2] El-Nouby, A., Klein, M., Zhai, S., Bautista, M. A., Toshev, A., Shankar, V., ... & Joulin, A. (2024). Scalable pre-training of large autoregressive image models. arXiv preprint arXiv:2401.08541.\n\n[3] Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., ... & Houlsby, N. (2023, July). Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning (pp. 7480-7512). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the Reviewer's comments to point out the confusing description and we make the response as below.\n\n***\n\n**Weakness 1: Phrase.**  \n\nThank you for your insightful suggestion. We will revise our presentation in the final version to prevent any potential misunderstanding. Our motivation for adopting the term \"scalable representation learning\" stems from the observation that different methods (e.g., DeiT, MAE, DINOv2) learn distinct representations that are not inherently slimmable (see Appendix: A.3). Scala is proposed to learn the original representations while incorporating slimmability. Therefore, we initially chose \"scalable representation learning\" to describe our method. However, we acknowledge that the term \"scalable\" may lead to confusion, and we will revise it in the final version to enhance clarity.\n\n***\n\nWe hope the explanations could address your concerns and we would appreciate it a lot if you could recognize the contributions of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JB0FjHE479", "reviewer": "Reviewer_V4fU", "summary": "The paper presents Scala, a novel framework for scalable representation learning developed from US-Net. It identifies the issues of directly applying US-Net to ViTs and proposes solutions including Isolated Activation, Scale Coordination, and Stable Sampling. These innovations enable Scala to output several sub-networks in one-shot learning. Extensive experiments on various network architectures and datasets demonstrate that the sub-networks produced by Scala consistently outperform those generated by separate training, with significantly reduced training time.", "questions": "1. Regarding the issue shown in Fig. 4, is it always the smallest subnet that causes the issue, or does it occur with subnets having a scaling ratio near 0.25?\n\n2. Can the authors clarify any differences between Noise Calibration and standard knowledge distillation?\n\n3. What would be the impact if the distillation part were discarded and only Cross-Entropy (CE) loss were used for the initial epochs?", "limitations": "The novelty and significance of Scala are somewhat limited, as discussed in the weaknesses section. However, the extensive experimental results provide a robust foundation for the claims made in the paper. Overall, the work is well-executed and makes a valuable contribution to the field, justifying a recommendation for weak acceptance.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality: Scala addresses the limitations of US-Net and successfully applies the concept of scaling to ViT backbones. This is a significant step in the adaptation of scaling methods for more complex network architectures.\n\nQuality: The paper supports its claims with extensive experimental results, providing strong evidence for the effectiveness of Scala.\n\nClarity: The paper is clearly written and well-organized, making it accessible and easy to follow.\n\nSignificance: Scala has the potential to influence future research directions in scaling ViTs.", "weaknesses": "Originality: The novelty of Scala is somewhat constrained. For instance, Noise Calibration does not show a distinct difference from standard knowledge distillation. Essentially, Scala integrates US-Net with an alternative activation for the smallest subnet and fixed scaling ratios.\n\nQuality: The authors might consider emphasizing results from a more standard 300-epoch ViT training schedule to align with common practices in the field.\n\nClarity: No further issues.\n\nSignificance: The challenge of scaling ViTs with arbitrary ratios remains unresolved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the Reviewer's approval and valuable comments. We respond to the Reviewer's concerns as below.\n\n***\n\n**Weakness 1: Fixed scaling ratio.**  \n\nThank you for the comment. The hidden dimension of ViT has to be an integer multiple of the number of heads (e.g., 6/12) so ViT cannot support arbitrary width ratios like CNN. For instance, the theoretical maximum number of networks that ViT-B could represent is 25 if $s=0.25$. However, we do acknowledge that Scala still cannot generalize to unseen width ratios as this issue is essentially related to the architecture design of transformer and we constrain ourselves from modifying the conventional structure as it has been well-optimized in various scenarios. On the other hand, it further emphasizes that the problem we target to solve is very challenging as the intermediate subnets ($X=13$) are only trained for around 18 epochs and they are expected to match the performance of networks that are trained for 100 epochs.\n\n**Weakness 2: Results of 300 epochs.**  \n\nThanks for the advice and we will emphasize the 300-epoch results of Sec. 5.3 and 5.4 in the final version.\n\n**Question 1: Root cause.**  \n\nThanks for the great question. It is the smallest subnet that causes the issue no matter what the width is and we validate it by setting $s=0.5$ to be the smallest scaling ratio and removing Isolated Activation. The table shows that constantly activating the smallest subnet in the naive way still results in worse performance even if the width ratio is larger than 0.25.\n\n| Method  | $r=0.50$ | $r=0.75$ | $r=1.00$ |\n| :-----: | ------------ | ------------ | ------------ |\n| Scala ($s=0.5$) | 71.2% | 74.5% | 76.8% |\n| Scala ($s=0.5$) w/o IA | 70.2% | 73.1% | 75.9% |\n\n\n**Question 2: Noise Calibration.**  \n\n\nCompared to standard knowledge distillation, the teacher of those subnets in Scala is dynamically changing (different widths) and will be optimized during training like the students as their weights are shared. On the one hand, the intermediate teachers serve as teacher assistants which fill the gap between $F^{l}\\left (\\cdot  \\right )$ and $F^{s}\\left (\\cdot  \\right )$ and provide simplified optimization objective for small subnets, in contrast to US-Net which utilizes $F^{l}\\left (\\cdot  \\right )$ as the teacher for all subnets and results in inferior performance. On the other hand, it also indicates that those teachers cannot provide faithful predictions at the early stages and it is essential to add Cross-Entropy loss to those subnets during training which is overlooked in previous baseline methods.\n\n\n**Question 3: Impact of only CE at early epochs.**\n\nThanks for the valuable suggestion. We further conduct experiments by removing the distillation loss of subnets in the first 10 epochs (denoted by *) and there is only minor performance change in the subnets which suggests that Noise Calibration is indeed helpful at early stages.\n\n| Method  | $r=0.25$ | $r=0.50$ | $r=0.75$ | $r=1.00$ |\n| :-----: | ------------ | ------------ | ------------ | ------------ |\n| Scala | 58.7% | 68.3% | 73.3% | 76.1% |\n| Scala* | 58.8% | 68.1% | 73.6% | 76.1% |\n\n\n***\n\nWe hope the explanations could address your concerns and we would appreciate it a lot if you could recognize the contributions of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QhLxKgd6z1", "reviewer": "Reviewer_7ef6", "summary": "The paper targets scaling down Vision Transformers (ViT) to fit environments with dynamically changing resource constraints. The authors propose Scala, a framework enabling a single network to represent multiple smaller ViTs with flexible inference capability by activating various subnets during training. Scala introduces Isolated Activation to disentangle the smallest sub-network and uses Scale Coordination to provide stable and accurate learning objectives. Empirical validations on different tasks show that Scala achieves scalable representation with one-shot training, matching the performance of Separate Training without modifying the original ViT structure. Scala demonstrates an average improvement of 1.6% on ImageNet-1K compared to previous methods, using fewer parameters.", "questions": "Please refer to Weaknesses.", "limitations": "The authors have addressed the limitations and potential negative societal impacts of their work.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The problem is important in practice.\n2. The experimental results seem decent.", "weaknesses": "1. My major concern is that, the same aim of adapting ViTs to dynamically changing resource constraints, can also be achieved by multi-exit networks, e.g., [*1, *2, *3]. However, the paper does not discuss these highly relevant works or compare with them. Hence, I vote for rejection.\n2. The method seems to lack novelty. 'smaller ViTs are intrinsically the sub-networks of a larger ViT with different widths' is not a surprising observation. The key techniques (e.g., Isolated Activation and Knowledge Distillation) are not new (naive or have been widely adopted).\n\n\n[*1] Huang, Gao, et al. \"Multi-Scale Dense Networks for Resource Efficient Image Classification.\" International Conference on Learning Representations. 2018.\n\n[*2] Wang, Yulin, et al. \"Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition.\" Advances in neural information processing systems 34 (2021): 11960-11973.\n\n[*3] Han, Yizeng, et al. \"Dynamic perceiver for efficient visual recognition.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the Reviewer's feedback. We provide further explanations to clarify the Reviewer's concerns based on several key points as below.\n\n***\n\n**Weakness 1: Discussion with dynamic networks.**  \n\nWe thank the Reviewer for the valuable suggestion and we will add discussion with those dynamic networks in the final version, including MSDNet[1], RANets[2], GFNet[3], DVT[4], CF-ViT[5], Dyn-Perceiver[6], etc. \n\nHowever, the differences between our work and this line of research are: \n(1) Motivation: dynamic networks are designed to reduce inference costs with dynamic computational graphs; Scala is proposed to make ViT become slimmable without touching the specific architecture design.\n(2) Method: dynamic networks tailor their computational graphs to accommodate varying inputs, thereby optimizing resource allocation on a per-sample basis. In contrast, Scala treats every sample equally and modulates the computational costs by adjusting the width ratios which aligns with the inherit design of ViT and its inference costs can be explicitly controlled by a single hyperparameter $r$.\n\nEssentially, dynamic networks are agnostic with Scala and we will consider the integration of those two lines of research as our future plan.\n\n**Weakness 2: Lack of novelty.**  \n\nThank you for the comment. While we do agree that 'smaller ViTs are intrinsically the sub-networks of a larger ViT with different widths' is not a surprising observation, making ViTs become slimmable is indeed non-trivial.\n\nAccording to our analysis in Sec. 2, ViTs display minimal interpolation ability which emphasizes that the problem we target to solve is very challenging as the intermediate subnets ($X=13$) are only trained for around 18 epochs and they are expected to match the performance of networks that are trained for 100 epochs. \n\nTo the best of our knowledge, we are the first to propose Isolated Activation, which seems simple but is very effective as we have abundant analysis (Sec. 3) to support it. Moreover, the knowledge distillation in Scala is different from the conventional setting. Instead of choosing the full model as the teacher for all subnets, we enable the intermediate networks to be the teacher as well which fills the model gap between $F^{l}\\left (\\cdot  \\right )$ and $F^{s}\\left (\\cdot  \\right )$ and provide simplified optimization objective for small subnets. This is non-trivial as we find adding a teacher network to the baseline method US-Net will result in performance decreases in the smaller subnets (please see our reply to Reviewer rLHb) which further supports our motivation to propose Progressive Knowledge Transfer. Moreover, Noise Calibration is critical in our setting as well because those intermediate teachers cannot provide faithful predictions at the early stages and it is essential to add Cross-Entropy loss to those subnets during training which is overlooked in previous baseline methods. \n\n\nBased on these factors, Scala can solve this challenging problem in a simple but effective way.\n\n***\n\nWe hope the explanations could address your concerns and we would appreciate it a lot if you could recognize the contributions of our work.\n\n[1] Huang G, Chen D, Li T, et al. Multi-scale dense networks for resource efficient image classification[J]. arXiv preprint arXiv:1703.09844, 2017.  \n[2] Yang L, Han Y, Chen X, et al. Resolution adaptive networks for efficient inference[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 2369-2378.  \n[3] Wang Y, Lv K, Huang R, et al. Glance and focus: a dynamic approach to reducing spatial redundancy in image classification[J]. Advances in Neural Information Processing Systems, 2020, 33: 2432-2444.  \n[4] Wang Y, Huang R, Song S, et al. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition[J]. Advances in neural information processing systems, 2021, 34: 11960-11973.  \n[5] Chen M, Lin M, Li K, et al. Cf-vit: A general coarse-to-fine method for vision transformer[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(6): 7042-7052.  \n[6] Han Y, Han D, Liu Z, et al. Dynamic perceiver for efficient visual recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 5992-6002.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers:

Thanks for your valuable comments in the review process. We have an exciting experiment added during rebuttal which supports that Scala can effectively inherit the generalization ability from foundation models like DINOv2 while maintaining the flexible inference capability. This indicates that it may be possible to enable foundation models to become slimmable as well which would further enhance the application scenarios of our method.

Inspired by the comment of Reviewer rLHb, we build Scala over DeiT-B for ImageNet-1K training and utilize DINOv2-B as the teacher in order to inherit its strong generalization ability. However, we remove all the Cross-Entropy loss during training to alleviate the dataset bias issue as DINOv2 is trained on the gigantic private dataset. To examine the generalization ability of Scala, we conduct linear probing on 12 fine-grained classification datasets following DINOv2. Tab. 1 in the PDF file shows that Scala significantly outperforms DeiT variants on the average performance of fine-grained classification which suggests that Scala indeed inherits the fruitful knowledge from DINOv2 with remarkable improvement in its generalization ability. Moreover, the improvement over DeiT does not decrease when we scale down the width ratios during inference and it indicates that Scala maintains the flexible inference capability very well even though it contains more knowledge than before.

We thank the Reviewers for the constructive suggestions which help us to improve the work. We are actively available until the end of this rebuttal period and please let us know if you have any further questions. Thank you so much for being with us so far.

Sincerely,  
Authors

Meta-review: After rebuttal and discussions, reviewers generally lean towards acceptance. I do agree with the originally more conservative reviewers that the related work is not discussed enough. For two examples: while MatFormer is discussed, there are several more works from Aditya Kusupati that are highly relevant. Furthermore, FlexiViT tackles all the same issues but along sequence length.

Thus, for the general case of "flexible amount of computation at inference time", too few baselines are compared to. That being said, the presented observations and introduced ideas are interesting and novel enough that, compared to their obvious baselines/ablations, they are worth publishing. Hence, we still recommend acceptance.

Finally, the AC agrees with reviewer rLHb that the title is both much too broad, and a wrong use of the word "scalable" compared to its typical meaning within the community. Not only that, also "representation learning" is not a good fit for this as it usually has a different meaning/focus. Please change the title to something more fitting and commong for this scenario. Some examples could be "Elastic training of Vision Transformers" or "Vision Transformers with Flexible Width" or something to that effect.
Decision: Accept (poster)