Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation
Authors: Wangbo Zhao
Jiasheng Tang
Yizeng Han
Yibing Song
Kai Wang
Gao Huang
Fan Wang
Yang You
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision Transformers (ViTs)
Parameter Efficient Fine- tuning (PEFT)
Dynamic Tuning (DyT)
Primary area: machine_vision

Abstract: Existing parameter-efficient fine-tuning (PEFT) methods have achieved significant success on vision transformers (ViTs) adaptation by improving parameter efficiency. However, the exploration of enhancing inference efficiency during adaptation remains underexplored. This limits the broader application of pre-trained ViT models, especially when the model is computationally extensive. In this paper, we propose Dynamic Tuning (DyT), a novel approach to improve both parameter and inference efficiency for ViT adaptation. Specifically, besides using the lightweight adapter modules, we propose a token dispatcher to distinguish informative tokens from less important ones, allowing the latter to dynamically skip the original block, thereby reducing the redundant computation during inference. Additionally, we explore multiple design variants to find the best practice of DyT. Finally, inspired by the mixture-of-experts (MoE) mechanism, we introduce an enhanced adapter to further boost the adaptation performance. We validate DyT across various tasks, including image/video recognition and semantic segmentation. For instance, DyT achieves superior performance compared to existing PEFT methods while evoking only 71% of their FLOPs on the VTAB-1K benchmark.

Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation
Wangbo Zhao 1.2
Jiasheng Tang 2.31
Yizeng Han 2.4 Fan Wang?
Yibing Song
Kai Wang'
Gao Huang*
Yang You'
National University of Singapore
DAMO Academy, Alibaba Group
Hupan Laboratory
*Tsinghua University
Code: https:///tttiiiie com COw/NUSS-CC-AI-LLLBSA AICC-AI-abDDDyaamiiccccocc Tuning
Abstract
Existing parameter-efficient fine-tuning (PEFT) methods have achieved significant success vision transformers (ViTs) adaptation by improving parameter efficiency. However, the exploration of enhancing inference efficiency during adaptation remains underexplored. This limits the broader application of pre-trained VIT models, especially when the model is computationally extensive. In this paper, we propose Dynamic Tuning (DyT), novel approach to improve both parameter and inference efficiency for ViT adaptation. Specifically, besides using the lightweight adapter modules, propose a token dispatcher to distinguish informative tokens from less important ones, allowing the latter to dynamically skip the original block, thereby reducing the redundant computation during inference. Additionally, we explore multiple design variants to find the best practice of DyT. Finally, inspired by the mixture-of-expert (MoE) mechanism, we introduce an enhanced adapter further boost the adaptation performance. We validate DyT across various tasks, including image/video recognition and semantic segmentation. For instance, DyT achieves superior performance compared to existing PEFT methods while evoking only 71 % of their FLOPs on the VTAB-IK IK benchmark.
Introduction
With the remarkable success of Vision Transformers (ViTs) 20 53 26) fine-tuing pre-trained ViT on other data domains [90] or task applications [8 36 60
79 31 becoming common practice.
However, as model sizes increase [88 [6 52], the associated adaptation costs become prohibitive due the burden of fine-tuning and inference on the target task. Parameter-efficient fine-tuning (PEFT) methods (e.g. AdaptFormer [12] LoRA 34, and VPT (36)), are proposed to tackle the tuning problem by reducing tunable model parameters. They usually update small amount of parameters while keeping the original model fixed, which reduces learnable parameters effectively while maintaining fine-tuning accuracy.
Despite the extensive research in parameter efficiency, the inference efficiency on target tasks is less explored. We numerically demonstrate the inference efficiency of three representative PEFT methods in Figure. [a). revealing that none of them reduces the computation during inference compared full tuning. This limitation poses challenges for adapting pre-trained ViT downstream tasks, particularly when the model is computationally extensive. To this end, we aim to unify both parameter and inference perspectives for efficient ViT adaptations.
Work done during an internship at DAMO Academy, Alibaba Group, waagboozhao966 Shaao9@@gmall.com gmail.com Corresponding authors, jiasheng www.aaa-iiaaeccooom youy comp.mus.ccommmm wwoompuus.ed..sp..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

19
18.30
77.14
18
17.58
17.61
17.58
76
17
Accuracy (%)
74.80
FLOPs
16
+8.18
15
-29%
71.96
14
70
13
12.54
12
DyT
Full luning AdapiFormer
LoRA
VPT
DyT
tuning (b)
Figure 1: FLOPs and Accuracy ViT-B/16
20 on VTAB-1K 89
"Full tuning" denotes that all
parameters are fine-tuned. AdaptFormer 12 LoRA 34
and VPT 36 are typical PEFT methods.
Dynamic networks
demonstrate that token pruning helps reduce model inference costs. However, these designs primarily focus on pre-training from scratch or full tuning the same dataset, without considering data domain transfer. Furthermore, the token pruning operation constraints these methods visual recognition scenarios, limiting their further applications e.g. dense prediction [93] Our motivation thus arises from how to develop dynamic modules together with PEFT methods to enhance both parameter and inference efficiency during ViT adaptation, which should also suit wide range of visual tasks.
In this paper, we introduce Dynamic Tuning (DyT). efficient approach for ViTs adaptation that achieves both parameter and inference efficiency. Specifically, we design a token dispatcher for each transformer block learning to dynamically determine whether a token should be activated or deactivated. Only those activated tokens will traverse the entire block and an extra lightweight adapter module, while the remaining tokens skip the original block and will solely be processed by the adapter. DyT does not reduce the total number of tokens, making suitable for both visual recognition and dense prediction scenarios. Additionally, since the impact of token skipping during adaptation has not been explored, we propose four model variants to determine the best practice for DyT. Finally, as the adapter module is set to process all the visual tokens, propose mixture-of-ext mixture-of-expper(MMEE. Moased adapter design that further enhances token processing with negligible additional FLOPs. Through these designs, DyT fulfills both parameter and inference efficiency for ViT's adaptation.
Comprehensive evaluations for DyT are conducted from multiple perspectives. In the image domain, as shown in Figure [b). lb). DyT surpasses existing PEFT methods while consuming only 71% of the ViT-B FLOPs on the VTAB-IK benchmark [89]. When visual tokens are scaled up from images to videos, our DyT shows superior generalization ability on action recognition benchmarks, e.g. K400 10 and SSV2 (25,, with a reduction of 37GFLOPs. In the scenario where labels are scaled up from recognition to segmentation, reduction. These experimental results indicate that the proposed DyT is efficient in both parameter and inference across various visual tasks and directs promising field for efficient model adaptation.
DyT even outperforms full tuning ADE20K [93] with 21 21GFLOPs
2 Related Works
Parameter-efficien fine-tuning. Parameter-efficien fine-tuning (PEFT) is designed to adapt pre-trained model to downstream tasks by only tuning a small part of the parameters. Existing PEFT methods can broadly be categorized into three groups: adapter-based methods, re parppaarrriiztionn based methods, and prompt-based methods. Adapter-based methods [12
33
62
38
[4] insert
some tiny modules into the original model and only these inserted modules updated during fine-tuning. Re-parametrization approaches [87 34 2 usually cooperate with reparameteri-
zation techniques, directly modifying specific parameters in the pre-trained model. Prompt-based g involve appending a small number of learnable tokens to input
methods 36 95
94 40
sequences, thereby adapting intermediate features in frozen layers. Please refer to [84 46 to find more comprehensive reviews.
However, PEFT methods primarily focus on improving parameter efficiency during fine-tuning while overlooking inference cost reduction. In this paper, we propose to improve the parameter efficiency and address inference hosts simultancously for efficient VIT adaptation.

---

Dynamic neural nefworks. Dynamic neural networks [28
can dynamically adjust their architectures based on the ipput data, which enables them to control the computational redundancy based on input data. Existing methods have explored dynamic layer depth 771 4 81
63 92
30
27
86].
dynamic channel width [32 29, 82 and dynamic routing 445 in convolutional neural networks. 48
When it comes to the era of vision transformer, many works 77 76
67
74,
57
59 58
64
attempt to improve the inference efficiency by reducing the token redundancy. For instance, Liang et al. [48] identify and consolidate inattentive tokens into only one token in each transformer layer. Rao et al. [67] progressively discard less informative tokens between layers. Wang et al. [76) adjust the number of tokens represent images. A more detailed literature review can be found in [28 75]. Zhou et al. [96] propose assessing token significance with a ReLU function to effectively tackle point cloud analysis. Although these approaches have shown significant success in vision tasks, they require training a model from scratch or fine-tuning all parameters on the same dataset used in pre-training, making them unsuitable for efficient ViT adaptation.
In contrast to these methods, the proposed method can adapt pre- pre-trained knowledge to diverse downstream datasets and tasks, while reducing the computational cost during inference by introducing negligible additional parameters.
3 ViTs Adaptation with Dynamic Tuning
In the following, we first introduce the vision transformer and the adapter architecture in Section 3.1 Subsequently, we present the proposed dynamic tuning (DyT) and explore its four variants in Section a32 and Section (.33 respectively. Furthermore, we build an MoE-adapter, which effectively enhances the adaptation performance without introducing additional computational cost, as elaborated in Section 3.4 Finally, we introduce the loss functions of our method in Section 3.5
3.1 Preliminary
Vision Transformer (ViT) consists of stack of layers. The multi-head self-attention block denoted as Attn, and multi-layer perceptron block, termed as MLP are the main components in each layer. The
input of each layer can be denoted as X = [xale X1, X2, x..] E [NN+1jxC containing N image tokens and classification token X.ls-
The adapter architectures have been widely explored in previous PEFT methods 12
An adapter is generally designed as a bottleneck module, which consists of a project layer W doun R to squeeze the channel dimension and another project layer Wap E 1g for dimension
3
restoration. Here, d denotes the bottleneck dimension and d < C ensures the efficiency of the adapter. Given an input feature X, the adapter operation is formulated as:
Xadapt = AAdapter(X) = (XW down. W uap-
(1)
where a denotes a nonlinear function e.g. ReLU. In this study, we follow this general architecture to build the adapter within dynamic tuning. We place the adapter in parallel with Attn block, MLP, or an entire transformer layer, which will be presented in detail Section 3.3
3.2 Dynamic Tuning
In Vision Transformers (ViTs), the computational burden primarily resides in the transformer layers. Specifically, the operations in Attn and MLP account for approximately 35.8% and 63.5% of total FLOPs in ViT-B/16 [20]. Previous works [67] 48) have revealed that there exists a token- redundancy issue in ViTs and found that some tokens can be discarded without sacrificing the performance. Inspired by this, we propose an efficient ViT adaptation approach, named Dynamic Tuning (DyT), which not only maintains parameter efficiency during fine-tuning but also reduces redundant computation during inference. The core idea is dynamically selecting tokens for processing within transformer blocks. Given the input tokens X of a block, DyT can be formulated as:
X' Block(TD(X)) + Adapter(X)
where Block can represent an multi-head self-attention block Attn, multi-layer perceptron MLP, or an entire transformer layer. The proposed token dispatcher (TD) learns to selectively activate or deactivate tokens. Only the activated tokens are input into the Block, while all tokens are processed by the Adapter.
3

---

Forward propagation
Backward propagation
Sigmoid function
Element-wise product
Element-wiie add
Daquichh
Token dispatcch
based on M
M
X
0u.0.5
Gumbel Noss
Dosattch
Block
G. ...
Block
w
W
Adapter
Adapter
(a) Fine-tuning Stage
(b) Inference Stage
Figure 2: Overview of Dynamic Tuning, (a) In the fine-tuning stage, we adopt Gumbel Noise to enable and-to-end training. (b)In the inference stage, TD selects K activated tokens X, from X based on the mask M. which saves the computations on those deactivated tokens in Block. Block can represent a Attn block, MLP block, or an entire transformer layer.
Token dispatcher. The key point in DyT select partial tokens which will be passed through Attn and/or MLP. A straightforward approach is randomly selecting tokens with predefined probability and adapting the model conduct downstream tasks with these selected tokens. However, this simplistic strategy risks discarding informative tokens while retaining less important tokens, potentially hindering the adaptation performance. To tackle this problem, we propose a token dispatcher (TD), which learns to select tokens during the adaptation. Specifically, given the input tokens X E +X TD learns to selectively activate sequential of tokens X, E RR where K represents the number of activated tokens. To achieve this, should obtain mask M E RN+11 which indicates whether a token should be activated or deactivated.
To obtain M, we adopt projection layer Wa cCx1 followed by a sigmoid function to predict the activation probability S N+1 Then, we 0.5 the threshold to determine the activation of each token. This can be formulated as:
Sigmoid((WW) g),, M,, [1 if S, 0.5 E M.
(0 S,, 0.5
M, denotes that the n-th token is activated and will subsequently undergo the process of Block. Conversely, if M, 0, the token will be deactivated and skip the Block. In practice. we always set the mask value of the classification token Xels allowing it to traverse the entire network. Notably, the number of additional parameters introduced in TD is negligible, with only C parameters in the linear layer W
Fine-tuning stage. However, directly using threshold makes M a discrete decision, resulting in a non-differentiable problem during fine-tuning. To address this, incorporate the Gumbel Noise 32 into sigmoid to replace the original sigmoid function during fine-tuning. It can be formulated as:
XW + G GG
S Gumbel-Sigmoidl XW, Sigmoid
T
where G,, G2 G0, 1). T represents the temperature and is 5.0 by default. Further details of Gumbel-Sigmoid are provided in the Appendix A.7 Subsequently, the same operation in Equation 3 The Gumbel Noise makes the sampling of M stochastic during training and we adopt S as differentiable approximation of M. Both of these help TD to be trained an end-to-end manner. The calculation of forward and backward propagations during training can be formulated as:
obtain M using
Block(X] (X) - M
Forward Propagation
X.
(5)
Block - SS
Backward Propagation
The Equation 2 can be reformulated into:
X' - X X, + Adapter(X),
(6)
From Equation 5 only the block outputs, Block( X), of those activated tokens are retained, while others are masked out by M. As shown Figure (a), during the fine-tuning stage, all tokens within X still need to traverse the Block.

---

DyT
DyT MLP
DyT
DyT
DyT
Attn
MLP
Attn
Attn
MLP
Attn
MLP
Attn Dispatch
MLP Dispatch
Attn-MLP Dispatch
Layer Dispatch
Figure 3: Model variants. For brevity, we omit the LayerNorm in Attn and MLP blocks. "DyT"
denotes the dynamic tuning presented in Figure 2
Inference stage. During inference, we can directly adopt Equation 3 to generate the dispatch mask M and obtain activated tokens X, R Rxxx in TD. Then, processing K tokens effectively reduces the computational cost because K < N. In practice, we add padding the output from the Block maintain tensor shape. This results in Equation See Figure (( (b) for a detailed illustration the inference stage.
only feed them into Block. Only
3.3 Model Variants
The Block in Equation 2 can instantiated into any blocks the original ViT, such as a multi-head self-attention block Attn, a multilayer perceptron block MLP, in VTT. Since the impact of skipping tokens these blocks during the adaptation fine-tuning remains non-trivial to estimate and has not been explored in previous works, we propose four model variants and conduct experiments to determine the best practice.
complete transformer layer
Attention Dispatch: Considering the quadratic computational complexity of the Attn block with respect to the token numbers, skipping tokens before applying Attn can significantly reduce computation. In this design, multi-head self-attention is exclusively performed between activated tokens, while other tokens are bypassed, which may hurt the interaction between tokens.
MLP Dispatch: Based the analysis in Section 3.2 FLOPs in ViT-B/16 and propose to skip tokens only before MLP. It enables that the interaction between tokens in Attn is not affected.
observe that MLP takes ~63.5%
Attention-Men Dispatch: An alternative strategy is skipping tokens before both self-attention and MLP blocks. This design permits higher activation rate in both Attn and MLP while maintaining similar computational efficiency comparable to "Attention Dispatch" and "MLP Dispatch". However, it costs double the additional parameters in adapters.
Layer Dispatch: Inspired by AAttention-MLP Dispatch", transformer layer. Specifically, tokens are identified by one TD to be activated or deactivated in the subsequent entire layer. With the same activation rate, its computation is similar to "Attention-Mi Dispatch" while requiring fewer parameters to build adapters.
dispatch tokens before
We demonstrate the architecture variants in Figure 3 variants presented in Section 4.2
The experimental results and analyses of these
3.4 MoE-Adapter
In DyT, the adapter is responsible for processing all tokens, requiring it to have enough capability, particularly when the downstream tasks e.g. semantic segmentation are challenging to adapt. To tackle this problem, we propose a MoB-adapterr, inspired by mixture-of-expprts (68 80 It effectiveyyy enhances the capability of the adapter with introducing negligible computation cost.
A MoE-adapter comprises routing layer W, RCXN and N adapter experts, denoted as [Wi down E cxxd W up E RdxCJN - The routing layer generates a series of scalar as the weights for the experts based on input features. The features from different images may yield distinct expert weights. Specifically, we first conduct average pooling across all tokens to generate a feature as the global representation of them. Subsequently, this representation is fed into the routing layer to generate weight scalars (a' aa? ??....oNN N Finally, tokens are processed by each expert independently and combined with the corresponding weight. We demonstrate this in Figure
However, this increases the computational cost the adapter in proportion to the number of experts N. To address this problem, adopt its mathematical equivalence to process X in practice, which
5

---

can be formulated as:
Xddpt
Moee Wmote down
up
where Wmoee adapter with the same capacity as N individual adapters, while maintaining computational efficiency equivalent to that of single adapter (the computational cost in the routing layer is negligible).
down 1 W down and Wm Σi 1= W The design endows the MoE- up
up
3.5 Loss Functions
For an image I, we calculate the cross- eross-yntropy loss Lots -y Iog(}) between the predicted probability y and its corresponding one-hot label y, supervise the learning of classifi- cation. To control the average activation rate in the entire model, we add loss term constrain the number of activated tokens in the dynamic tuning, which be formulated as: Frtte LZN Σt= En=1
Elemankotis
X Adoott
Elermeiowisee
odd
Nonir
W
W
op
> M r)?, where denotes the number layers VIT. Mr represents the mask generated in TD from the 1-th layer. Additionally, we employ a loss f'ds yy log(y') ) to supervise the adaptation of the complete model, where y' is the output probability without employing the token dis-
Routing W
ax
ON
+
Wasan
W dress
Wilf'
X
Figure 4: The architecture of the MoE-adapter. is consist of N adapter experts.
patcher. Thus, this complete model can also act as a teacher to enhance the dynamic tuning during adaption by distillation loss Laistil gence loss. Therefore, the overall loss function is defined as C = Lets + C.ds + Edistau + aCCatee where serves the weight of the activation rate loss and is set to 2.0 by default. Note that, DyT can also achieve competitive performance without the distillation loss (see Appendix (.6)).
KL(y', y). where KL represents the Kullback-Leibler diver-
4 Experiments
4.1 Experiment Settings
Datasets. To evaluate the adaptation performance, conduct experiments VTAB-1K [89] benchmark. The training data in this benchmark is extremely limited, with only 1,000 training samples for each task. Different from existing PEFT works [12 38 39], which mainly focusing on VTAB-IK, we also conduct experiments on three image classification datasets with complete training sets, including CIFAR-100 41), SVHN (24). Food-101 6). Additionally,
adopt two V2 (SSv2) [25), to evaluate the performance when the number of tokens scaled up. All images frames are resize into 224 x 224. For the dense prediction task, we evaluate our method on two widely recognized semantic segmentation datasets, AED20K (93 and COCO-stuff (7]. The results of semantic segmentation presented in the Appendix A.4 We run each task three times and report the averaged results. The error bars are small (< 0.1) and omitted for simplification.
video datasets, Kinetic-400 (K400) [10] and Something-Somethhne
Implementation Details. We conduct all experiments based the VIT-B/16 (20), which is pre- trained on ImageNet2 IK 17 with full supervision, unless otherwise specified. Results based on ViT-L are presented in the Appendix A.10 The bottleneck dimension d is set to 64 by default. We adopt the same training schedule as [2]. Detailed hyperparameters for each experiment can be found in the Appendix A.8
The default setting in experiments is marked in color
4.2 Analysis
Model variants. In Table we compare the performance of four model variants across both image and video datasets. We set the activation rate in "Attention Dispatch" and "MLP Dispatch" variants to 0.5, and 0.7 for "Attention-MI.P Dispatch" and "Layer Dispatch" variants, and train each respectively. This results in similar average FLOPs for four variants. We observe that the default

---

introduce the MoE-adapter when the target dataset is Table 1: Comparison of model variants. "Params. (M)" indicates the additional parameters in
backbones. "FLOPs (G)" denotes the average FLOPs on CIFAR-100.
Image Accuracy (%)
f
Video Accuracy (%e
Model Variant
Params.(M) +
FLOPs (G)
CIFAR-100
SVHN
Food-101
K400
SSv2
Attention Dispatch
1.19
14.77
84.58
96.35
86.67
69.67
41.36
MLP Dispatch
1.19
12.21
91.37
90.32
75.28
45,43
Attention- MiatttionMMPPP Dispatch
2.38
12.93
90.03
96.66
88.61
74.62
41.83
Layer Dispatch
1.19
13.08
89.38
96.57
89.05
74.72
43,94
Table 2: Effectiveness of MoE-Adapter. DyTr denotes the DyT with MoE-adapters. Standard
adapter is enough handle image datasets while MoE-adapter is more suitable for challenging scenarios, such as video datasets. It theoretically does not increase extra computational cost, but
the FLOPs slightly vary in different models since the learned token dispatch strategy in the TD is
different. NV represents the number of experts.
Image Accuracy (%)
Video Accuracy (%)
Model
Params. (M) 4
FLOPs (G) +
CIFAR-100
SVHN
Food-101
K400
SSv2
DyT DyT+
1.19
12.21
91.37
97.08
90.32
75.28
45,43
2.40
12.58
91.07
96.09
89.98
74.55
45.78
DyTý N
4.80
12.29
91.01
96.90
89.77
75.43
46.56
DyTí N =
9.60
12.44
90.45
96.84
89.53
75.34
46.06
DyT+ N 12
14.40
12,43
90.31
96.72
89.32
75.17
44.97
setting "MLP Dispatch" achieves superior performance across five datasets while maintaining the lowest computational cost. Although "Attention-MLP Dispatch" and "Layer Dispatch" also exhibit good performance on K400, the former incurs double additional parameters while the latter lacks generalization capability on other datasets. The comparison between "MLP Dispatch" and other variants proves that only skipping tokens in MLP blocks is better design. More investigations on model variants can be found in our Appendix A.3
Effectiveness MoE-adapter. We conduct experiments to explore the effectiveness of the MoE- adapter and the results are illustrated in T The MoE-adapter design ensures that the FLOPs will theoretically remain the same as the ordinary adapter, with the computational cost from the routing function being negligible. However, in practical scenarios, the computational cost is also influenced by the learned token dispatch strategy within the Token Dispatcher (TD), leading to slightly varying FLOPs across different models in Table 2
We observe that the performance on image datasets drops when we increase the expert number in MoE-adapter. This phenomenon can be attributed to the simplicity of image datasets and the model does not require too many parameters to adapt. In contrast, for video datasets, such as K400 and SSv2, the best accuracy is achieved 4 experts. The reason behind this is that the domain gap between the pre-training dataset and video datasets is large and the model needs sufficient adapter capacity to learn the adaptation. This proves that difficult to adapt.
Visualization of token activation rate in each layer. In Figure 5 we visualize the token activation rates across different layers in ViT-B/16 (0). We observe that the model tends to activate more tokens in lower layers while deactivating tokens in higher layers. This phenomenon the model tends to take more general knowledge from the lower layers of the pre-trained model and learn more task-specific knowledge in higher levels. Additionally, the activation results vary across different datasets. For instance, SSv2 demonstrates increased token activation rate in Layer 5 and Layer 6 compared other datasets, whereas SVHN experiences substantial token deactivation in Layer 6, 7, and 8. This discrepancy arises from that the model needs different knowledge from the pre-trained weights address sppasit- challenges.
be attributed that
is noteworthy that nearly all tokens are deactivated in the final layer across five datasets, especially CIFAR-100, SVHN, and K400, where the activation rates exactly 0%. This indicates that these datasets, we can directly drop the original MLP block in Layerl without hurting the performance, which further reduces about 4.7M [ 5.4% of the ViT-B total parameters.
"There are two linear layers with weights of size 768 x (4 x 768) in MLP block, results in 2 x 768 x (4 768) z 4.7M parameters
7

---

Activation Rate (%)
CIFAR-100
80
SVHN
Food-101
60-
K400
SSv2
20
00140
Overall Layer0 Layer1 Layer2 Layer3
Layer4
Layer5 Layer6 Layer7 Layer8 Layerg Layer10Layer11
Figure 5: Token activation rate in different layers. We visualize the token activation rates in ViT-B/16. "Overall" denotes the mean activation rate in the whole model, which arrives at around 50% when is set to 0.5. "LayerO"" and "Layer11 1d denote the lowest and highest level, respectively. Notably, the activation rate in the last layer is exactly 0% on CIFAR-100 SVHN, and K400 datasets.
Visualization of activated tokens. In Figure 6 we visualize two representative samples from K400. We can observe that the model tends to deactivate those tokens that are less informative e.g. tokens of the sky in (a) and tokens of grass in (b). In higher layers, such as layer7 and layer10, only those tokens from the primary objects are activated. This further proves the the existence of token redundancy problems in VIT and provides validation for the rationality behind our approach. Additional visualizations are available Appendix(A.I. A.
Layer1
Layer4
Layer?
Layer10
Layer1
Layer4
Layer7
Layer10
(a) Deactivate less informative tokens from the sky
(b) Tokens from the primary object (boy) always activated
Figure 6:
Visualization of activated tokens. We present two representative samples from the K400 dataset. Blue patches represent the tokens activated in token dispatcher (Detailed in Section 3.2). Results verify that the token dispatcher has learned to identify informative tokens during fine-tuning
4.3 VTAB-1K Results
Comparison with PEFT methods. To evaluate the adaptation performance when the training data is limited, we adapt the VTAB-1K (89] benchmark, which widely employed to evaluate the performance of adaptation tasks. Following exiting works, we reduce the bottleneck dimension d to 8, resulting in only 0. 16M extra parameters. We vary the activation rate r in DyT in the range [0.5, 0.7, 0.9] and conduct fine-tuning, obtaining three models with different inference costs.
The results are presented in Table 3 Although previous methods, such ConvPass (38] and Res- Tuning (37, achieve satisfactory performance, they do not improve the efficiency during inference and even introduce additional FLOPs compared with full fine-tuning. GFLOPs, about 71 % of the cost in original ViT-B, our method has outperformed previous methods egg. LoRA 34 and VPT (3), by large margin. Remarkably. the DyT model with r 0.9 does not surpass the performance of the DyT model with 0.7. This observation suggests the presence of computational redundancy in the transformer after adaptation, which further validates the significance of dynmaic tuning. These experimental results validate that DyT effectively improves parameter efficiency and inference efficiency while maintaining the adaptation performance.
contrast, with only 12.54
Dynamic tuning achieves actual acceleration. As harowace-agnostic metric, FLOPs not the most suitable choice to evaluate the inference efficiency across diverse platforms. The real inference speed has usually been ignored in previous PEFT methods. Here, we adopt two GPUs (Tesla V100 and Tesla T4) and CPU Xeon(R) Platinum 8163 to comprehensively evaluate the efficiency of our methods and three representative PEFT methods, including LoRA 34 AdaptFormer [12), and VPT [36]. The batch size during inference is set to 512 and 32 for GPUs and the CPU, respectively. The results, as summarized Table 4 reveal that our method achieves better performance while effectively accelerating inference speed compared to existing PEFT methods different platforms.

---

Table 3: Performance and efficiency comparison on VTAB-IK. "Group Mean" indicates the averaged accuracy of three groups. *Params. (M)" denotes the number of trainable parameters in backbones. "FL.PSs (G)" is the average FLOPs across all datasets. Bold font and underline denote the best and the second-best performance respectively.
Natural
Specialized
Structured
CIFAR 100
Cattech011
DTD
Flowers
SVHN
Sun397
Camelyon
EuroSAT
Rosise45
Retinopatly
Clevr-Cone Count
Clev-Dist
DMI
KITTI-Dist
dSpr-Lcc
dSpr Oni
SNORB- Azim
SNORB-Eley
Group Mcan
Params. (M)
FLOPs (G)
Pots
Traditional methods
Full tuning Frozen
168.9 87.7 97.2
86.9
87.4 38.8 79.7 95.7 84.2 73.9| 58.6 65.5
57,5 46.7 25.7 29.1
19.1
85.80
17.58
63.4
63.2
97.0
86.3
36,6
51.0
78.5 87.5 34.3
34.3 30.6 33.2 55.4
12.5
20.0
9.6
19.2
0.00
17.58
Buruneter-efleiie luring methadds
Adapter
33
68.2 90.1 68.0
98.8
89.9
82.8 54.3 84.0 94.9 81.9 75.5 80.9 65.3 48.6 78.3 74.8
48.5
29.9
41.6
73.85
0.16
17.61
Bitit
72.8
97.5
85.3 59.9
51.4 78.7
61.5 55.6 32.4
15.7
25.1
65.21
17.58
LoRA 14
67.1
91.4 69.4
9%.8
90.4
85.3 54.0
84,
95.3
84.4 73.6
82.9
69.2 49.8 78.5
75.7
47.1
31.0
44.0
74.60
0.29
17.58
VPT [6
78.8
98.0
88.3 78. 49.6
96.
83
68.5
60.0 46.5
72.8 73.6
47.9
32,9
37.8
71,96
0.53
18,30
SSF 38
69.0 75.1 99.4
91.8
90.2
52.9
87.4
95.9
87.4
75.5
75.9
62.3 53.3 80.6
77.3
54.9
29.5
37.9
75.69
0.20
17.58
NOAH [0]]
69.6 92.7 70.2 99.1 90.4
86 53.7
84.4
95.4
83,9 75.8
82.8
68.9 49,9 81.7 81.8
48.3
44,8
75.48
0.36
17.58'
ConvPass [38
72.3
91. 72.2 99.2
90.9
91.3 54.9
84.2 9 85.3
82.3
67.9 51 80.0
53.1
36.4
44.4
76.56
0.33
17.64
AdaptFormer 12
70.8 91.2 70.5
19.1 90.9
86.6 54.8
83.0
95.8
84.4
76
81.9
64.3 49.1 80.3
76.3
45.7
31.7
41.
74.75
0.16
17.61
FacT-TT 39
71.3
89.6 70.7 98,9
91.0
87.8 54.6
85.2 95.5 83.4 75.7
82.0 69 89.8 80.0
79.2
48.4
34.2
41.4
75.30
0.04
17.58
Res-Tuning 37
75.2 92.7
91.9
86.7
58.5 86.7
74.6 80.2 63.6 50.6
85.4
55.7
31.9
42.0
76.32
0.51
17.67
The proposed Dynamic: Tuning
DyT r = 0.5
73.6 94.8 73.0 19.1 91 87.0 56,4
88.3 96.1 85.6 76.7 82.8 63.8 52.7 83.7 83.6 57.3 34.6
44.3
77.14
0.16
12.54
DyT 0.7
74.4 95.5 73.6 99.2 91.7 87.5 57.4 88.3 96.1 86.7 76.7 83.5 63.8 52.9 83.1 85.7
54.9
34.3
77.57
0.16
14.92
DyT 0.9
74.3 94.9 73.1 99.2 91.4 87.8 57.1 87.9 96,1 85.9 76.0 83.3 64.8 51.5 83.4 84.0 54.8
35.1
46.4
77.30
0.16
17.07
NOOAI cast hager than 17.58 FLOPS since combines PEFT methods via neural architeclure search.
Table 4: Comparison of throughput. "VTAB-1K Accuracy 1st denotes the averaged accuracy of
three dataset groups in VTAB-IK (89] benchmark.
VTAB-IK
V100
T4
Xcon(R) 8163
Method
Accuracy 1
FLOPs (G)
Throughput (img/k) f Throughpul (img/s)?
Throughput (img/s) f
Pall tuning
68.96
17.58
806.24
435.41
2.12
LoRA 34
74.60
17.58
806.24
435.41
2.12
AdaptFormer
12
74.75
17.61
767.30
400.42
1.97
VPT 36
71.96
18.30
762.55
392.13
1.95
DyTr = 0.5
77.14
12.54
912.30
524.93
3.89
Comparison and compatibility with methods for efficient transformers. We first investigate the domain adaptation performance of two representative methods DynamicVit ViT [67] and EVIT [48). These methods are designed for efficient vision transformers. We adopt the optimal configurations outlined in their original papers and conduct experiments on VTAB-IK IK [8) benchmark. The results, as summarized in Table 5 reveal that both methods achieve high throughput egg. > 1000 (img/s), while the performance is unsatisfying. Combining Dynamic ViT and EVIT with AdaptFormer [12] results in performance improvements, validating the importance of exploring both parameter and inference efficiency for vision transformers. Despite these gains, DyT obviously surpasses them, highlighting the superiority of our approach.
Then, we explore the compatibility of our method with token pruning methods. Specifically, we combine DyT with ToMe [5), a training-free technique that progressively prunes tokens through token merging. From the results in Table 5 we find that ToMe can further enhance the throughput of DyT while maintaining accuracy. This proves the potential of our methods to be combined with existing token pruning methods e.g. IT (8). Additionally, we apply ToMe to full tuning and AdaptFormer [12 in Table 3 and observe sub-optimal accuracy and throughput. These findings highlight that directly applying ToMe after fine-tuning or crrameter-efficienn fine-tuning is less effective compared to the proposed approach.
4.4 Further Exploration
Effectiveness on image datasets with sufficient training data. Although the results from VTAB- IK benchmark have proven the superiority of our approach, we extend our investigation to complete image datasets, to evaluate the adaptation performance with sufficient training data, We conduct
experiments on 6 datasets including CIFAR-100 [1]. SVHN (24, Food-101 (6), Air (56), Pet 661],

---

Table 5: Comparison with efficient transformers. The throughput measured on a Tesla V100 "Params. (M) + denotes the number of trainable parameters in backbones.
GPU.
VIĀB-IK Accuracy
Method
FLOPs (G)
Param. (M)
Throughput (img/s) f
Dynamic VTT 667
60.10
14.05
88.70
1010,40
75.48
14.23
3.10
954.82
EViT [88
67.42
11.62
85.80
1233,45
EViT+AdapPFomerll
74.63
11.77
0.16
1152.38
Full tuning ToMe 5
68.68
13.12
85,80
989.29
AdaptFormer [12] ToMe 5
74.30
13.29
0.16
941.70
DyTr 0.5
77.14
12.54
0.16
912.39
DyT 0.5 ToMe
76.60
9.85
0.16
1114.70
and Car [22]. The results are demonstrated in T We further explore a straightforward approach, "Dynamic-Full which has a token dispatcher as the same DyT and is fine-tuned with all parameters. We observe that its performance becomes unstable and drops significantly on some datasets e.g. CIFAR-100 and Food-101 This phenomenon may arise due to the potential adverse impact of dynamic dispatch on the pre-trained parameters during full-tuning adaptation, thereby validating the importance of DyT. In this data-sufficient scenario, although our method achieves performance slightly below that of full tuning and AdaptFormer, it brings significant reduction in FLOPs.
Scaling token counts from images to videos. We conduct experiments on video datasets to show the performance when the number of tokens scaled up. The number of input frames is set to 8. For video tasks, similar to [85] 13,, we employ cross- aotos-att layer and a query token to aggregate features from different frames. The video classification is conducted the query token. Additional implementation details are provided in the Appendix [A.8] We demonstrate the results in Table 6 Although DyT achieves slightly lower performance than AdaptFormer and LoRA, it costs obviously fewer FLOPs. DyTt containing four experts can achieve the best average accuracy with only cost 12.29 GFLOPs, which further verifies the superiority of our design.
Table 6: Results on image and video tasks. "Avg." denotes the average results from the correspond- ing task. The FLOPs are evaluated on CIFAR- 100 and K400.
Parmms.
Image Datasets
Datasets
Method
FLOP
Image Accuracy (96
FLOPs
Video Accumcy (%)
(M)
CIFAR-100
Fond-101
Avg.
K400
88v2
Avg.
Traditional methods
Fall maing Lincar
85,80
17.58
90.91
97.29
90.69
80.53
93.11
88.71
90.21
142.53
75.48
45.22
60.35
0
17.58
85.87
56.29
88.07
40.51
92.78
52.85
69.40
142.55
69.01
27.64
48.34
Dynamic-Fal Pall
$5.80
12.24
83.43
96.90
94.46
62,50
75.16
70.48
78.82
107 67
74.63
44.94
$9.79
Porumeter
tuning
AdapiFormer
P
1.19
17.81
92.03
97.23
90.84
78.23
94.46
87.33
90.02
144.39
75.53
45,36
60.45
LoRA 31
1.19
17.58
91.42
97.36
90.48
76.32
93.62
87.00
$9.36
142.53
75.18
45.62
60.55
VPT 36
0.07
18.32
91,46
95.72
90.41
68.91
93.92
80.72
86 88
73.46
38.17
55.82
The propoved Dynamic Timing
DyT
1.19
97.08
90
78.92
94.45
87. 80
108.31
75.28
45.43
60.36
DyTi =
4,80
12.29
91,01
96
89.77
78.27
93.85
87.61
89.56
105.45
75.43
46.56
60.98
5 Discussion and Conclusion
Previous methods for ViT adaptation primarily focus on improving the efficiency during the adap- lation, thus aiming to reduce additional parameters. However, with the increasing size and com- putational cost of ViT models, the inference cost after the adaptation is becoming a heavy burden. In this paper, unify both of these two problems into the efficiency problem for VIT adaptation and propose dynamic tuning (DyT) to tackle them simultancously. We validate its performance and generalization across various tasks and datasets.
Limitations and future works. DyT is currently designed for vision tasks. We believe it would be interesting to combine vision backbones with large language models e.g. (72) build efficient large multi-modal models e.g. 97). DyT could be further developed to compatible with multi-modal models, which would be our future direction.
10

---

Acknowledgements
This work was supported by Damo Academy through Damo Academy Research Intern Program. Yang You's research group is being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE Tier-l grant, ByteDance grant, ARCTIC grant, SMI grant (WBS number: A- 8001 1001004-))-)0) Alibaba grant, and Google grant for TPU usage.
References
[1] Jimmy Lei Ba, Jamic Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint urXiv:1607.064500 2016.
[2] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2233. 172774 2022.
[3] H Bao, L Dong, and F Wei. Beit: Bert pre-training of image transformers. arxiv 2021. arXiv preprint arXiv:2106. 08254.
[4] Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. ICML, pages 527-536 PMLR, 2017.
[5] Daniel Bolya, Cheng- Cheng-Yang Fu, Xiaoliang Dai. Peizhao Zhang, Christoph Feichtenhofer. and Judy Hoffman. Token merging: Your vit but faster. arXiv preprini arXiv22110.94611, 2022.
[6] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In ECCV, pages 446-461. Springer, 2014.
[7] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari, Coco-stuff: Thing and stuff classes context. In CVPR, pages 1209-1218, 2018.
Qinglong Cao, Zhengqin Xu, Yuantian Chen, Chao Ma, and Xiaokang Yang. Nomain-controlless prompt learning. arXiv preprint arXiv:2310.07730, 2023.
[9] Qinglong Cao, Zhengqin Xu, Yuntian Chen, Chao Ma, and Xiaokang Yang. Domain prompt learning with quaternion networks. arXiv preprint arXiv:2312.0887,, 2023.
[10] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, pages 2299-6305- 2017.
[11] Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate: Differentiable compression rate for efficient vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 17164-17174 17174, 2023.
[12] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. NeurIPS, 35:16664-166 16678, 2022.
[13] Xiaokang Chen, Mingyu Ding. Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. IJCV, 132(1):208-223, 2024.
[14] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.0855 2022.
[15] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quae V Le. Randaugment: Practical automated data augmentation with a reduced search space. CVPR workshops, pages 702-703, 2020.
[16] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heck, Justin Gilmer, An- dreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, al. Scaling vision transformers billion parameters. ICML,, pages 7480-7512, PMLR, 2023.
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. CVPR, pages 248 255. Icee, 2009.
[18] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lms. arXiv preprint ar3iv: 143/4, 2023.
[19] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Bootstrapped masked autoencoders for vision bert pretraining. In ECCV, pages 247 264. Springer, 2022.

---

[20] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai. Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010.1 2020.
[21] Ali Edalati, Marzich Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:22/2. 20050, 2022.
[22] Timmit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, and Fei-Fei. Fine-grained car detection for visual census estimation. In AAAI, volume 31, 2017.
[23] Xinwei Geng, Longyue Wang. Xing Wang, Bing Qin, Ting Liu, and Zhaopeng Tu. How does selective mechanism improve self-attention networks? arXiv preprint arXiv:2005. 2020.
[24] lan J Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Amoud, and Vinay Shet. Multi-digit number recog- nition from street view imagery using deep convolutional neural networks. arXiv preprint 131.:213,..6082, 2013.
[25] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim. Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freiter et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, pages 5842-5850, 2017.
[26] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jiamyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. TPAMI, 45(1)=87-110,, 2022.
[27] Yizeng Han. Dongchen Han, Zeyu Liu, Yulin Wang, Xuran Pan, Yifan Pu, Chao Deng. Junlan Feng. Shiji Song, and Gao Huang. Dynamic perceiver for efficient visual recognition. In ICCV, 2023.
[28] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang. and Yulin Wang. Dynamic neural networks: A survey. TPAMI, ():44333337 2021.
[29] Yizeng Han, Zeyu Liu, Zhihang Yuan, Yifan Pu, Chaofei Wang, Shiji Song, and Gao Huang. Latency-aware unified dynamic networks for efficient image recognition. TPAMI, 2024.
[30] Yizeng Han, Yifan Pu, Zihang Lai, Chaofei Wang, Shiji Song. Junfeng Cao, Wenhui Huang, Chao Deng, and Gao Huang. Learning to weight samples for dynamic early-exiting networks. In ECCV, pages 362-378. Springer, 2022.
[31] Chunming He, Kai Li, Yachao Zhang, Guoxia Xu, Longxiang Tang. Yulun Zhang, Zhenhua Guo, and Xiu Li. Weakly-supervised concealed object segmentation with sam-based pseudo labeling and multi-scale feature grouping. In NeurlPS, 2024.
[32] Charles Hermaann Richard Strong Bowen, and Ramin Zabih. Channel selection using gumbel softmax. In ECCV, pages 241-257. Springer, 2020.
[33] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nip. In JCML, pages 2790-2799. PMLR, 2019.
[34] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0966 2021.
[35] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzaiio with gumbel-softmax. arXiv preprint arXiv:1611.01 144, 2016.
[36] Menglin Jia, Luming Tang. Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, pages 709-727. Springer, 2022.
[37] Zeyinzi Jiang, Chaojie Mao, Ziyuan Huang, Ma. Yiliang Lv, Yujun Shen, Deli Zhao, and Jingren Zhou. Res-tuning: flexible and efficient tuning paradigm via unhinding tuner from backhone. arXiv preprint ar 19859, 2023.
[38] Shibo Jie and Zhi-Hong Deng. Convolutional bypasses are better vision transformer adapters. arXiv preprint 220::2207.070333 2022.
[39] Shibo Jie and Zhi Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer. In AAAI, volume 37, pages 1060 106800068,60000000001110 2023.
12

---

[40] Chen Ju. Tengda Han, Kunhao Zheng. Ya Zhang. and Weidi Xie. Prompting visual-language models for efficient video understanding. In ECCV, pages 105-124. Springer, 2022.
[41] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[42] Tao Lei. Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yangi Zhou, Nan Du, Vincent Zhao, Yuexin Wu, Bo Li, et al. Conditional adapters: Parameter-efficien transfer learning with fast inference. Advances in Neural Information Processing Systems, 36:8152-8177 2023.
[43] Changlin Li, Guangrun Wang, Bing Wang. Xiaodan Liang. Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. In CVPR, pages 8607-8617, 2021.
[44] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European conference on computer vision, pages 280-296. Springer, 2022.
[45] Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang, and Jian Sun. Learning dynamic routing for semantic segmentation. In CVPR, pages 8553 8562, 2020.
[46] Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down scale up: A guide to parameter- efficient fine-tuning, arXiv preprint arXiv:2303. 15647, 2023.
[47] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. NeurIPS, 35:109-12: 2022.
[48] Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xic, Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint 220iv:2202.07800 2022.
[49] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part 13. pages 740-755. Springer, 2014.
[50] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuming. NeurlPS, 36, 2024.
[51] Pengbo Liu. Hailong Cao, and Tiejun Zhao. Gumbel-attention for multi-modal machine translation. arXiv preprint arXiv:2103.0880 2021.
[52] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In CVPR, pages 12009-12019, 2022.
[53] 7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
[54] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint 2016.
[55] Ilya Loshchitov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.200 2017.
[56] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aireraft. arXiv preprint 1306v:1006.5151 2013.
[57] Lingehen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In CVPR, pages 12309-12318, 2022.
[58] Zanlin Ni, Yulin Wang, Renping Zhou, Yizeng Han, Jiayi Guo, Zhiyuan Liu, Yuan Yao, and Gao Huang. Enat: Rethinking spatial-temporal interactions token-based image synthesis. In NeurlPS, 2024.
[59] Zanlin Ni, Yulin Wang, Renping Zhou, Rui Lu, Jiayi Guo, Jinyi Hu, Zhiyuan Liu, Yuan Yao, and Gao Huang. Adanat: Exploring adaptive policy for based image generation. In ECCV, 2024.
[60] Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-efficient imagc-to-video transfer learning. NeurlPS, 35:26462 264777, 2022.
[61] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, pages IEEE, 2012.
13

---

[62] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych Adapterfusion Non-destructive task composition for transfer learning, arXiv preprint 0002:2000.000277, 2020.
[63]
Yifan Pu, Yiru Wang, Zhuofan Xia, Yizeng Han, Yulin Wang, Weihao Gan, Zidong Wang. Shiji Song, and Gao Huang. Adaptive rotated convolution for rotated object detection. ICCV, 2023.
[64]
Yifan Pu, Zhuofan Xia, Jiayi Guo, Dongchen Han, Qixiu Li, Duo Li, Yuhui Yuan, Ji Li, Yizeng Han, Shiji Song, et al. Efficient diffusion transformer with step-wise dynamic attention mediators. In ECCV, 2024.
[65]
Pytorch.
Gumbel-softmax. softmax.....oo html
httpa://pyytorch.orge http://t..........//./aaaaaaaaaa Pytorch.
functional gumbel_softmax.
[66]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21((40):1-61 2020.
[67] Yongming Rao. Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamievit: Efficient vision transformers with dynamic token sparsification. NeurlPS, 34:13937--3949, 2021.
[68] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experss layer. arXiv preprint 17Xiv:1701.06338, 2017.
[69] Lin Song. Songyang Zhang, Songtao Liu, Zeming Li. Xuming He, Hongbin Sun, Jian Sun, and Nanning Zheng. Dynamic grained encoder for vision transformers. NeurlPS, 34:5770-5783 2021.
[70] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 1911);1999-198,, 2014.
[71] Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsume Kung. Branchynet: Fast inference via carly exiting from deep neural networks. In ICPR, pages 166424699. IEEE, 2016.
[72] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[73] Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixture-of-Bit ofatutao---daataatiis for parameter-efficient model tuning. arXiv preprint arXiv:2205.124:1 2022.
[74] Yulin Wang, Zhaoxi Chen, Haojun Jiang, Shiji Song, Yizeng Han, and Gao Huang. Adaptive focus for efficient video recognition. ICCV, October 2021.
[75] Yulin Wang. Yizeng Han, Chaofei Wang, Shiji Song, Qi Tian, and Gao Huang. Computation--ffiiien deep learning for computer vision: A survey. Cybernetics and Intelligence, 2023.
[76] Yulin Wang, Rui Huang. Shiji Song, Zeyi Huang, and Gao Huang. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. NeurlPS, 34:1 1 1960-1 1973, 2021.
177] Yulin Wang, Kangehen Lv, Rui Huang, Shiji Song, Le Yang, and Gao Huang. Glance and focus: dynamic approach to reducing spatial redundancy in image classification. In NeurlPS, 2020.
[78] Tete Xiao, Yingcheng Liu, Bolei Zhou. Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding, In ECCV, pages 418 434, 2018
[79] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. NewrIPS, 14:12077- 2021.
[80] Brandon Yang, Gabriel Bender, Quoe Le, and Jiquan Ngiam. Condconv: Conditionally parameterized convolutions for efficient inference. NewrIPS, 32, 2019.
[81] Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang, Resolution adaptive networks for efficient inference. In CVPR, pages 2369-2378, 2020.
[82] Le Yang, Haojun Jiang, Ruojin Cai, Yulin Wang. Shiji Song, Gao Huang, and Qi Tian. Condensenet v2: Sparse feature reactivation for deep networks. In CVPR, pages 9569-3578, 2021.
14

---

[83] Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. Adavit: Adaptive tokens for efficient vision transformer. arXiv preprint arXiv:2112.07638, 2021.
[84] Bruce XB Yu, Jianlong Chang, Haixin Wang, Lingho Liu, Shijie Wang. Zhiyu Wang, Junfan Lin, Lingxi Xie, Haojie Li, Zhouchen Lin, et al. Visual tuning. arXiv preprint arXiv:2305.0660 2023.
[85] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[86] Yang Yue, Yulin Wang, Bingyi Kang, Yizeng Han, Shenzhi Wang. Shiji Song, Jiashi Feng, and Gao Huang. Dynamic inference of multimodal large language models for efficient robot execution. In NeurlPS, 2024.
[87] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for based masked language-models. arXiv preprint arXiv:2106. 10199, 2021.
[88] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Sealing vision transformers. In CVPR, pages 12104-12113, 2022.
[89] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910004807, 2019.
[90] Aston Zhang. Zachary Lipton, Mu Li, and Alexander Smola. Dive into deep learning. arXiv preprint arXiv:2 1142, 2023.
191] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv preprint arXiv:2206.04673, 2022.
[92] Ziwei Zheng, Le Yang, Yulin Wang, Miao Zhang, Lijun He, Gao Huang, and Fan Li. Dynamic spatial focus for efficient compressed video action recognition. TCSVT, 2024.
[93] Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scone parsing through ade20k dataset. CVPR, pages 633-641, 2017.
[94] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, pages 16816-16825, 2022.
[95] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. I/CV, 13009):2337-2348 2022.
[96] Xin Zhou, Dingkang Liang, Wei Xu, Xingkui Zhu, Yihan Xu, Zhikang Zou, and Xiang Bai. Dynamic adapter meets prompt tuning: Paramcter-efficient transfer learning for point cloud analysis. CVPR, pages 14707-14717, 2024.
[97] Deyao Zhu, Jun Chen, Xiaogian Shen, Xiang Li, and Mohamed Elhoseiny. Minignt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint ar3v 10592, 2023.
15

---

A Appendix
We organize our appendix as follows.
In Section
A.1
we present frequently asked questions along with their corresponding
answers.
In Section
we detail the difference between our method and other previous works.
In Section
A.3
we present more analysis on the proposed method.
In Section
A.4
we report the results on semantic segmentation datasets.
In Section
A.5
we report the performance on object detection and instance segmentation.
In Section
A.6
we verify the effectiveness of complete model and distillation during
adaption.
In Section
A.7
we provide the details and a formal proof related to the Gumbel-Sigmoid
mechanism.
In Section
A.8
presents the implementation details for each experiment.
In Section
A.9
demonstrate the generalization capability of method with Swin
Transformer
53
In Section A.10
we investigate the impact of scaling the model size to ViT-L (20].
In Section
A.
we provide additional visualizations of activated tokens in our appraoch.
A.1 Frequent Questions
Why the proposed method outperforms traditional adapters? We list the explanations below:
The dynamic architecture in DyT enhances generalization. It introduces a form of distur- bance in the input data, akin Dropout [70] This mechanism is particularly erucial when training data is limited e.g. VTAB-IK.
The distillation loss in DYT. We adopt the complete model as the teacher of the dynamic model, significantly enhancing performance. Such a self-distillation mechanism only available in the dynamic architecture.
Previous work 29 and Dynamic ViT also show dynamic architectures outperforming static models with fewer FLOPS.
Why Table 6 shows that using MoE-adapter results in fewer FLOPs? Mož-adapter results slight fewer FLOPs. We list the explanations below:
It is possible that
The FLOPs of DyT depend on learned token dispatcher during fine-tuning and may slightly fluctuate around the target FLOPs (controlled by Lrate).
The extra computational cost of the adapters and the MoE adapters is nearly equivalent.
Thus, a DyT model with the MoE-adapter may activate fewer tokens in the learned token dispatcher, resulting in slightly reduced FL.OPs.
Why the FLOPs of 12 are paradoxically lower than that of retically, the MoE-adapter with any number of experts should have similar FLOPS the standard adapter. Meanwhile, the actual activation rate of DiT during inference depends on the learned token dispatcher after fine-tuning, resulting in slight fluctuations in FLOPs between different models. These explain why DyT N = 12 may have slight lower FLOPs than DyT N = 8.
Table 2 Theo-
A.2 Difference with previous Works
compare DyT with more previous works and demonstrate the differences between our approach and these methods.
16

---

Difference with DynamieVin VIT and EVIT. Both DynamicViT 67 and EVIT 48 are token pruning methods, whereas DyT token skipping method. DynamicViT learns retain P% tokens at certain layers e.g. 3th,6th 9th layers in ViT-B. EViT only keeps top-K K attentive tokens and fuses the inattentive tokens certain layers e.g. 7th, and 10th layer in ViT-B. These methods primarily focuses on accelerating the model within the same dataset used for pre-training, whereas Dy'T aims to improve efficiency during cross-domain adaptation.
Difference with DiffRate. DiffRate pruning and merging simultaneously.
1
is a token compression method that performs token
In the DiffRate 11 token pruning and merging is inherently data-independent. After training, a transformer layer prunes or merges the same number of tokens across all input data. In contrast, DyT is data-dependent approach. The router in DyT learns skip varying numbers of tokens before each MLP block based on the input data.
The prune and merge operations in DiffRate do not preserve complete feature maps, pre- senting challenges for dense prediction tasks. Thus, DiftRate requires modifications to address these tasks. Conversely, with only performing token skipping, DyT maintains complete feature maps, allowing effectively handle dense prediction tasks without any modifications.
Difference with ToMe. ToMe [5] is training-frre free technique that enhances inference efficiency by merging tokens based on similarity at each layer. DyT employs token skipping instead of merging and can be seamlessly integrated with ToMe to further improve efficiency.
Difference with CoDA. CoDA
is a PEFT method that can also improve the inference efficiency.
The token selection strategy in the token dispatcher is different. CoDA selects top-K K tokens in each layer to pass through while DyT adopt learnable dispatchers to select an appropriate number of tokens for each input.
The target model is different. Although CoDA 42 also improve both the parameter and inference during adaptation, CoDA primarily focus on the language model e.g. T5 [66] while DyT is specifically designed for vision transformers.
The block to conduct token skipping is different. In CoDA, tokens directly skip the whole layer. In DyT, we propose four model variants, explore their effectiveness, and find that skipping the MLP block is the most suitable for vision transformer.
Difference with AdaMix. AdaMix [73 also leverage mixture of adapters, but it fuses all experts by weight averaging after training, resulting in an standard adapter akin to AdaptFormer [12]. In contrast, the proposed MoB-adapter employs learning- based router generate sealar weights for the experts based on input features, allowing features from different images to yield distinct expert weights.
A.3
More Analysis
A.3.1 Investigations of dispatch level and dispatch strategy
The proposed DyT performs dynamic dispatch at the token level using the token dispatcher (TD). In addition to token-level dispatch, we also investigate a sample-level dispatch, where all tokens within the selected samples are activated, while all tokens will be deactivated for other samples. To verify the importance of TD, we compare it against random dispatch, which randomly activates tokens or samples during both fine-tuning and inference. The experimental results are presented in Table
Our observations reveal that leven-level dispatch with TD consistently achieves superior performance across all datasets, except for a slight decrease compared sample-level dispatch on the SVHN dataset. Notably, TD can achieve much better performance than the random dispatch strategy on both teven and sample-level dispatch, particularly on video datasets K400 and SSv2, thereby validating the dispatch strategy learned in TD. Furthermore, the token-level dispatch surpasses the sample-level dispatch by a significant margin on most datasets, which demonstrates the importance of the finer-grained activation.
17

---

Table 7: Investigations of dispatch level and dispatch strategy. Combining the token-level dispatch and TD strategy results in the best performance. We consider two dispatch levels: "Token" (where dispatch is performed the token-level) and "Sample" (where dispatch is conducted the sample-level). "TD" and "Random" represent the learned dispatcher and random dispatch strategy, respectively.
Dispatch Level
Dispatch Strategy
Image Accuracy (%)
Video Accuracy (%)
Token
Sample
Random
CIFAR- 100
SVHN
Food-101
K400
SSv2
91.37
97.08
90.32
75.28
45.43
90.70
97.29
89.8
71.14
44.09
/
89.38
96.79
86.39
68.27
40.13
/ 87.15
97.07
85.26
68.18
39.79
A.3.2 FLOPs-Accuracy curves of model variants.
In Figure 7 we further visualize the FLOPs-Accuracy curves of four model variants. We control the FLOPss by changing the activation rate r for the fine-tuning stage. For "Attention Dispatch" and "MLP Dispatch", we explore the activation rate in the range [0.1, 0.3, 0.5, 0.7, 0.9]. To maintain similar Fl.OPs as "Attention Dispatch" and "MLP Dispatch", we adjust the activation rate for "Attention-ML.P Dispatch" and "Layer Dispatch" within the range [0.5, 0.6, 0.7, 0.8, 0.9]. Across all datasets, "MLP Dispatch" consistently outperforms other variants. The performance of "Attn Dispatch" experiences significant drop when the activation rate is lower than 0.9, which also indicates that skipping tokens for self-attention blocks ViT is not a suitable way, due to the importance function of self-attention. Remarkably, "MLP Dispatch" can surpass full tuning with obviously fewer FLOPs on both CIFAR-100 and Food- 101, further validating the effectiveness of our approach.
the token mixing
82
90
80
Accurslcy (9%)
89
Accuracy
Acnuracy
98.6
(Defsul)
Layer
98.4
AdtenSon-MILP
86.2
85
Atlention
Full turing
84
FLOPs (G3
(G)
FLOPs (G)
(a) CIFAR-100
(b) SVHN
(c) Food-101
Figure FLOPs-Accuracy curves of model variants on CIFAR100, SVHN, and Food-101 datasets. "MLP Dispatch" achieve the best FL.OP- trade-off. We explore the activation rate r within 10.1, 0.3, 0.5, 0.7, 0.9] for "Attention Dispatch" and "MLP Dispatch". For "Attention- MLP Dispatch" and "Layer Dispatch" models, r is adjusted within the range [0.5, 0.6, 0.7, 0.8, 0.9]. "Full tuning" denotes traditional full fine-tuning approach. conserve space, we use simplified names for the variants.
A.3.3 Different bottleneck dimensions
We investigate the impact of the bottleneck dimensions of the adapter in DyT and the results are summarized in Table S revealing that different datasets prefer different configurations. When the dataset is easy adapt, such as CIFAR-I00. bottleneck dimension of satisfactory performance. Conversely, video datasets require larger adapter dimensions e.g. d = 256 attain better performance. We set the default bottleneck dimension 64, which achieves balance between performance and cost, such as the number of additional parameters and FLOPs. is worth noting that the relationship between FLOPs and bottleneck dimension not strictly monotonic due to the existence of token dispatcher. For instance, when d 16, the FLOPss are lower than when d = 1, since the training may converge activating fewer tokens as the number of adapter's 's parameters increases.
is sufficient achieve
18

---

Table Different bottleneck dimensions. Different datasets prefer different bottleneck dimensions. d = 64 strikes balance between accuracy and cost, such as extra parameters and FLOPs. "FLOPs
GG) denotes the average FL.OSs on CIFAR-100.
Image Accuracy (%)
Video Accuracy (%)
Dimension
Params, (M)
FLOPs (G)
CIFAR-100
SVHN
Food-101
K400
SSv2
0.03
12.11
91.46
95.5
89.63
71.78
36.22
4
0.08
11.89
91.57
96.61
89.99
73.16
39.24
16
0.30
12.06
91.46
96.98
90.24
73.65
42.49
64
1.19
12.21
91.37
97.08
90.32
75.28
45.43
256
4.73
13.32
91.18
97.03
90.42
75.33
46.73
A.3.4
Investigation on the Temperature
We explore the temporal r the proposed dynamic tuning. The results are demonstrated Table 2 When the temperature smaller e.g. 0.1, the Gumbel Sigmoid tends produce binary outputs close to 0 or 1. Conversely, larger temperatures lead to more uniform outputs, approaching 0.5. Results demonstrate that the performance is not sensitive to the temperature and model can achieve reasonable performance with all temperatures in the table. We also observe that the model with 7 1 CIFAR-IO0 SVHN, and SSv2, while decaying the temperature
achieves the best performance with schedule leads to the best result on Food-101, which shows that adjusting the temperature can help the model to achieve better performance. Since identifying the optimal temperature the primary focus of this paper, we directly set the temperature to by default.
Table 9: Different temperature 7 in dynamic tuning, "Schedule" denotes that the temperature
gradually decays from 5 to 0.1 during fine-tuning. The default setting is marked in color
Image Accuracy (%) f
Video Accuracy (%) f
Temperature
CIFAR-100
SVHN
Food-10 101d-101
K400
SSv2
0.1
90.91
96.24
89.72
73.16
44.84
91.61
97.20
90.08
74.38
45.69
91.37
97.08
90.32
75.28
45.43
Schedule
91.58
97.13
90.39
74.57
45.51
A.4 Generalization in semantic segmentation
We also conduct experiments on two well-recognized semantic segmentation datasets ADE20K and COCO-stuff 7 to demonstrate the ability of DyT on dense prediction tasks. Results are presented Table TO Following previous works 19], we adopt the UperNet (78 as the segmentation head, and all images are resized into 512 512. We observe that the computational cost of semantic segmentation is much higher than the image and video discrimination tasks, primarily due to high-resolution feature maps. Both DyT and DyTf still can reduce the computational cost obviously and achieve better performance than other PEFT methods, only slightly lower than full tuning on COCO-Stur Stuff dataset.
A.5 Generalization in Object Detection and Instance Segmentation
We conduct experiments on COCO 49] to explore the generalization our method in object detection
and instance segmentation. We adopt ViTDet [444 as the detector and fine-tune it for 12 epochs the COCO dataset. The bottleneck dimension in adapters set to 128. As shown in Table DyT exhibits superior performance compared to AdapterFormer [2], with fewer FLOPs. Our MoE-
adapter further enhances DyT without additional computational cost, validating the effectiveness of design.
However, full tuning achieves the best performance, surpassing other methods significantly. This likely due to the gap between bounding box regression and the pre-training of the vision transformer, necessitating more parameter updates in the backbone. This challenge motivates us to design more powerful PEFT methods and integrate them with DyT to reduce the performance gap with full tuning.
19

---

Table 10: Results on semantic segmentation. "Avg." denotes the average results from the corre- sponding two datasets. The FLOPs is measured ADE20K.
Semantic Segmentation Datasets
Params.
Method
FLOPs
mIOU
(M)
(G)
ADE20K
COCO-stuff
Avg.
Traditional methods
Full tuning
85.80
605.36
47.66
45.89
46,78
Linear
605.36
43.86
44,46
44.16
Dynamic-Full
85.80
582.46
45.74
45.20
45.47
Porumeter Turumeter-eflecent tuning methodss
AdamtFormer
1.19
45.33
46,43
LoRA 36
1.19
605.36
47.34
45.53
46.44
0.07
606.36
46.38
44.87
45.63
The proposed Dynamic Tuning
DyT
1.19
584.57
46.90
45.63
46.27
DyTf N =
4.80
584.40
47.67
45.71
46.69
Table 11: Results object detection and instance segmentation. We only measure the FLOPs in backbone and feature pyramid module.
COCO Datasets
Params.
Method
FLOPs
BBox mAP
Seg mAP
(M)
(G)
Full tuning
85,80
554.86
44.67
39.78
AdaptFormer 12
2.56
564.53
38.71
35.27
DyT
2.56
468.40
39.78
36.11
DyTf N
10.24
466.32
40.97
37.11
A.6 Effectiveness of the Complete Model and Distillation
In the main paper, we adopt the complete model our teacher, which does not employ token dispatcher to skip tokens. Our proposed dynamic tuning allows model act as its own teacher during adaptation, capability not achievable by other PEFT methods. We consider it to be significant advantage of our approach. Joint training of the dynamic and complete models mitigates overfitting during adaptation, particularly with limited training data, such VTAB-IK. Furthermore, the complete model, acting as a teacher, enhances the dynamic model's learning. These factors contribute to DyT's superior performance. Experimental results in Table 12 demonstrate that both complete model loss and distillation loss are useful for improving the performance of DyT. We also notice that introducing the complete model during training results 1.8 longer training time. Given that our primary contribution focuses on improving parameter and inference efficiency, the additional training time introduced by the complete model would be acceptable.
In Table 13, we further present the results across datasets in detail. We can find that, DyT without flas + Latistiti can still outperform most previous PEFT methods, further validating the superiority of our method.
A.7 Details about Gumbel-Sigmoid
is proposed in 335 conduct the differentiable sampling from a distribution. Given can be formulated as:
an unnormalized log probability J the
p: exp (E, + /T)
(8)
-== exp (EE, GT,/T))
"The original definition in (35] assumes the input { J to be normalized log probability, while practical implementations (65) demonstrate its effectiveness even with unnormalized inputs. We also follow 665 to formulate and implement the Sunbel-Softmax.
20

---

Table 12: Effectiveness loss functions The default loss function in DyT is L Lets + f'dle Ldistiit + abrate- We gradually remove the complete model loss L' and distillation loss disttill from it, and find the performance drops.
V VIAB-IK
Training time
Method
Accuracy
DyT
71.14
1,8 x
DyT w/ Ldiasst DyT w/o L'd.
76.70
1,8 x
Cdistilt
75.73
1.0 x
Table 13: Performance and efficiency comparison VTAB-IK. "Group Mean" indicates the averaged accuracy of three groups. "Params. (M)" denotes the number of trainable parameters in backbones. "FLOPSs (G)" is the average FLOPs across all datasets. Bold font and underline denote the best and the second-best performance respectively.
* Natural
Specialized
Structured
sNORB-Azin Azim
CIFAR- 100
Caltech101
Flowers
SVHN
Sun397
Camel lyon
EaroSAT
Resise45
Retinopathy
Cleve-Count Count
Cleve-
Clevr-Dist Dist
DML
KITTI-Dist
dSpr-Loc
dSpr-
NNRBB Elev
Group Mean
Params. (M)
DTD
Pets
Traditional methods
Full tuning Frozen
68.99 87.7 64.3 97.2 86.9 4 38.8 79.7 0...7.900 95.7 84.2 53. 3 58.6 41.7 65.5 46.7
38.8
46.7
25.7
29.1
19..96
85.8
63.4
85.0
63.2
97.0
86.3
36.6 51.0
78.5
78.5 87.5 68.6 74.0 33.3 30.6 33.2
55.4
12.5
20.0
9.6
19.2 57.64
0
Purumeter-cffiiient tuning methods
Adupter
69 90.1 68.0 98.8
82.8
|8.3 94.9 81.9 75.5 80.9 65.3 78.3
74.8
48.5
29.9
41.6
73.85 0.16
BitFit [87
72.8 87.0 59.2 97.5 85.3
59.9
51.4
78.7 91.6 72.9 69.8 61.5 55,6
78.7
32.4 55.9
66.6
40.0
15.7
25.1
65.21 0.10
LoRA 34
67.1 91.4 69.4
98.8 90.4
85.3 54.0
84.9 95.3 84.4 73.6
82,9
69.2 49.8
78.5
75.7
47.1
31.0
44.0
74.60 0.29
VPT
78,8 90.8 65.8 98.0
88.3 78.1 49.6
81.8 96.1 83.4 68.4
60.0
46.5
72.8 73.6
47.9
32.9
37.8
71.96 0.53
SSF 38
69.0 92.6 75.1
90.2 52,9
87.4 95.9 87.4 75.5
75.9
53.3 80.6 77.3
54,9
37.9
75.69 0.20
NOAH [1
69.6 92.7 70.2 99.1 90.4
86. 53.7
84,4 95.4 83.9 75.8
82.8
68.9
49,9 81.7 81.8
48.3
32.8
44.2
75.48 0.36
ConvPass 38
72.3
91.2 72.2 99.2 90.9
91.3 54.9
84.2
96.1 85.3 75.6
82.3
67.9 51.3
80.0 85.9
53.1
36.4
44.4
76.56 0.33
AdaptFomner 12
70.8 91.2 70.5 99.1 90.9
86.6 54.8
83.0
95.8 84.4 76.3
81,9
64.3 49.3 80.3 76.3
45.7
31.7
41.1
74.75 0.16
FacT-TT
71.3
89.6 70.7 98.9
6
85.2
95.5 83.4 75.7
82.0
49.8
80.0
79.2
48.4
34.2
41.4
75.30 0.04
Res-Tuning 37
75.2 92.7
71.9 99.3 91.9 86.7 58.5 86.7 95.6 85.0 74.6 80.2 63.6 50.6
80.2
85,4
55.7
31.9
42.0
76.32
0.51
The proposed Dynamic Taning without C/2, + di
DyT r = 0.5
70.4 94,2
71.1 99.1 91.7 88.0 | 87.1 95.3 84.2 75.8 79.2 61.8 $1.0 82.4 79.7 52.3
35.3
75.5
44.5
0.16
DyT r 0.7
73.9 94.9
72.1 99.4 91.8 8 55.5 87.2 95.6 86.2 75.9 80.3 61.8 51.7 83.1 81.6
53.7
35.3
45.2
76.69
0.16
DyT 0.9
74.0 95.1
72.9
99.3 91.7 87.6 56.9 87.7 95.7 85.4 76. 81.6 63.2 50.1 83.0
83.3
52.0
34.5
44.5
76.74
0.16
The proposed Dynamic Tuning with distillation
DyT r 0.5
73.6 94.8 73.0 99.1 91.4 87.0 56.4 87.3 96.1 85.6 82.8 63.8 $2.7 83.7 83.6 57,3
34.6
44.
44.3
77.14
0.16
DyT 0.7
74.4 95.5 73.6 99.2 91.7 87.5 $7.4 88.3 96.1 86.7 76.7 83,5 63.8 52.9 83.1 85.7
54.9
34.3
45.9
77.57
0.16
DyT 0.9
74.3 94.9 73.1 99.2
87.8 57.1 87.9 96.1 85.9 76.0
64.8 51.5 83.4 84.0
54.8
35.1
46.4
77.30
where G, denotes the Gumbel Noise sampled from a Gumbel distribution (G; Gumbel 1)). We can consider the special case, where N = 2 and Eg = 0, then P1 can be defined as:
E, +
exp
P1
E, G
G2
exp
exp
(10)
+ G2
exp
Ej + G G2
Sigmoid
(11)
Gumbel-Sigmoi((E:)
(12)
obtaining the formulation of Gummel-Sigmoid. Researchers in previous works, such as [2,51] have also leveraged the Gumbel-Sigmoid formulation facilitate end-to-end training of neural networks.
A.8 Implementation Details for each Task
we fine-tune the model for 100 epochs on each dataset in VTAB-1K [89] We do not use any data augmentation strategy these experiments. We adopt the AdamW
Experimental settings on VTAB-IK.
Following previous works 38
55 optimizer. The learning rate is set to le-3
and gradually decays to 0 based on cosine schedule
54
21

---

Experimental settings on complete image datasets. We adopt the settings in Table 14 to fine-tune the ViT with the proposed dynamic tuning. Experiments on other parameter-efficinng methods such as AdaptFormer [12. LoRA 34 and VPT [36] also follow the settings in Table [4 When we train a model with full tuning, we adopt a 1/10 base learning rate to make the training stable, otherwise, the model can not achieve reasonable results.
Table 14: Experimental settings for complete image datasets. We present the hyperparameters in DyT. Following previous methods, we train the model with 1/10 base learning rate in the full tuning setting. Ir base ir batch seccc_s_ee//66
Optimizer
AdamW [55
Base learning rate
le-3
Weight decay
0.01
Batch size
1024
Training crop size
224
Learning rate schedule
Cosine decay [54
GPU numbers
Warmup epochs
20
Training epochs
100
Augmentation
RandomResizedCrop
Experimental settings video datasets. We adopt two video datasets, Kinetic-400 (K400) [10]
and Samething-SSmeetinnn V2 (SSv2) 25), to evaluate the performance as the token count scales up.
The experimental settings demonstrated in Table 15 Most of the settings The number of input frames set to 8. We adopt multi-view during the test, which a common practice in video action recognition. However, the original VIT lacks temporal modeling capabilities. To address this limitation, we draw inspiration from [3]. By introducing a cross-attention layer after the ViT, along with query token, we effectively aggregate temporal information across different frames. The final video action classification is performed based on this query token. Experiments on parameter-effciiett fine-tuning methods also follow these settings. We adopt 1/10 base learning rate for those experiments on full tuning.
borrowed from [60].
Table 15: Experimental settings for video datasets. We follow most of settings in [60]. The number of input frames is set in all experiments. Ir base Ir x batch_stt siccchssseee__ee /26
Configuration
K400 110
SSV2 (25
Optimizer
AdamW 55
Base learning
lc-3
Weight decay
0.01
Batch size
128
Training Epochs
12 ShortSideJitter
50
Training Resize
RandomResizedCrop
224 256
Training crop size Learning rate schedule
224
Cosine decay
54
dynamic, evenly covering the whole video
Frame sampling rate
16
Mirror
/
X
Randenggent [1
x
Num. testing views
spatial 3 temporal
3 spatial temporal
Experimental settings on semantic segmentation datasets. We conduct experiments on ADE20K [93 and COCO-stuff [7] to demonstrate the ability of DyT on dense prediction tasks. ADE20K contains 20,210 images from 150 fine-trained semantic concepts. COCO-Stuff consists of about 164k images with 172 semantic classes. The experimental settings demonstrated in Table I6 All
22

---

experiments for parameter-efficien fine-tuning methods also follow these settings. For the experiment on full tuning, we set adopt 1/10 learning rate for stable training and better performance.
Table 16: Experimental settings for semantic segmentation datasets. Following previous methods, train the model with 1/10 learning rate in the full tuning setting.
Configuration
ADE20K [93]
COCO-stuff [7]
Optimizer
AdamW [55]
Learning rate
le-3
Weight decay
0.05
Batch size
16
Training crop size
512
Learning rate schedule
cosine decay [54]
Training iterations
160K
80K
A.9 Generalization Capability of Transformer Architecture
To verify the generalization of the proposed dynamic tuning, conduct experiments based Swin-B [53]. The dynamic tuning can be directly applied to MLP blocks in the swin transformer without any modifications. The bottleneck dimension of the adapter in dynamic tuming also is set to 64. The results are demonstrated Table I7 both the tunable parameters and FLOPs while achieving comparable even better performance across three datasets. This verifies the generalization capability of the dynamic tuning.
We can observe that the dynamic tuning can reduce
Table 17: Generalization Capability on Swin Transformer 53
The experiments are conducted based on Swin-B. "Param. (M)" denotes the number of trainable parameters backbones. "FLOPs (G)" denotes the average FLOPs on CIFAR 100. The bottleneck dimension d is set to 64.
Image Accuracy (%)
Method
Params. (M) 4
FLOPs (G) J
CIFAR-100
SVHN
Foud-101
Full tuning
86.74
15.40
91.82
97.66
93.05
DyT r 0.1
1,55
10.72
90.55
97.43
89.84
DyT r 0.3
1,55
12.07
91.26
97.38
90.66
DyT r 0.5
1.55
13.25
91.62
97.40
91.30
DyT r 0.7
1.55
14.05
92.14
97.21
91.96
DyT r 0.9
1.55
15.23
92.31
97.37
92.21
A.10 Scaling-up Model Size
We verify the effectiveness of dynamic tuning when the model size is scaled up. We conduct experiments on VIT-L [20] and compare dynamic tuning with full tuning. The bottleneck dimension d of the adapter in dynamic tuning is also set to 64. The results are demonstrated in Table [8 We can observe that with the activation rate set to 0.3, DyT has outperformed "full tuning" obviously on both CIFAR- 100 and Food-101 while resulting in significantly lower computational cost.
We further compare the proposed dynamic tuning with full tuning on the VTAB-IK benchmark 89 The results are demonstrated in Table 19 With only 0.44M tunable parameters and 43.52
GFL.OPs, dynamic tuning surpasses full tuning across most datasets and achieves much better average performance.
23

---

Table 18: Scale up the model size VIT-L 20]
"Paaam. (M)" denotes the number of trainable parameters in backbones. "FLOPs (G)" denotes the average FLOPs on CIFAR-100, The default
setting is marked in color The bottleneck dimension d is set to 64.
Image Accuracy (%)
Method
Params. (M)
FLOPs (G)
CIFAR-100
SVHN
Food-101
Full tuning
303.3
61.60
92.05
97.44
90.62
DyT r 0.1
3.17
32.56
91.26
97.04
89.98
DyT r 0.3
3.17
36.77
92.66
97.27
90.85
DyT r 0.5
3.17
43.79
93,49
97.38
91.49
DyT r 0.7
3.17
51.11
93.28
97.25
91.60
DyT r 0.9
3.17
60.05
93.44
97.23
91.59
Table 19: Performance and efficiency comparison on VTAB-IK. "Group Mean" indicates the
averaged accuracy of three groups. "Full tuning" indicates fine-tuning all parameters. "Param. (M)" denotes the number of trainable parameters in backbones. "FLOPSs (G)" is the average FLOPs across all datasets. The bottleneck dimension set to d =.
Natural
Specialized
Structured
Structured
CIFAR-
Caltech101
Flowers102
SVHN
Sun397
Camelyon
EumSAT
Resise45
Retinopathy
Clevr-Count
Clev--Ditt
DMLab
KITTI-Dist
dSp-Loc
dSp--Ori
SNORB
SNORB-Elev Elev
Group Mean
Param. (M)
FLOPs (G)
DTD
Pots
tuning 169.5 96.2 73.8 98.8 90.7 91.6 44.8 85.8 96.2 87.8 18300 62.0 50.8 80.0 85.8 54.6 29.7 75.7
303.3
61.60
DyT
0.5
79.1 95.6 74.5 99.5 92.6 90.8
59.3
86.9 96.6
87.2 76.5
84.5
62.9
53.3
83.5 88.4
57.3 38.7
44.6 78.5 0.44 43.52
A.11 Additional Visualizations of Activated Tokens
We provide more visualizations activated tokens from samples in K400 [0] and SSv2 [25] in Figure88an and Figure respectively. Results demonstrate that most activated tokens in higher layers egg. Layer10 O come from the primary objects. This proves that the proposed token dispatcher learns to activate informative tokens.
24

---

Layert
Layer4
Layer7
Layer10
Layer1
Layer4
Layer7
Layer10
(a)
i
(m)
(n)
ff
Ser..
(p)
h
(q)
())
(n)
Figure 8: Visualization of activated tokens. We present representative samples from the K400 dataset. Blue patches represent the tokens activated in token dispatcher. Results verify that the token dispatcher has learned to identify informative tokens during fine-tuning. Zoom in for better view.
25

---

Layer1
Layer4
Layer7
Layer10
Layer1
Layer4
Layer7
Layer10
(a)
i
(b)
(K)
(m)
(n)
(0)
(p)
(h)
(q)
())
(n)
Figure 9: Visualization of activated tokens. We present representative samples from the SSv2 dataset. Blue patches represent the tokens activated in token dispatcher. Results verify that the token dispatcher has learned to identify informative tokens during fine-tuning. Zoom in for better view.
26

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Abstract
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Conclusion
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
See Method
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Experiments
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification:
We have released the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.ccppullc//uuuii wwwticccguie//eeieeeeeeii CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Experiments
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Experiments
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
29

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Experiments
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te:::////uis..eeeee http://ss.....b......hcccculeead
Answer: [Yes]
Justification: See the main paper
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
No negative societal impacts. Our model is only designed for fine-tuning.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
30

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not release of data models that have high risk
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: we cite all related paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
31

---

Answer: [NA]
Justification:
See the main paper.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: See the main paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not involve crowdsourcing research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "H0IYnfs1R4", "reviewer": "Reviewer_fmWb", "summary": "This paper proposes Dynamic Tuning, an efficient tuning method to enhance both parameter and inference efficiency for ViT adaptation. The method improves efficiency by dropping some tokens entering pre-trained layers through a token dispatcher while forwarding all tokens to the adapter, maintaining performance. Additionally, the method leverages a Mixture of Experts (MoE) adapter to enhance adaptation performance on certain datasets. The proposed approach achieves comparable or superior performance with significantly fewer FLOPs compared to traditional parameter-efficient transfer learning methods on the VTAB-1K dataset.", "questions": "- The authors should consider removing the MoE part from the paper, as it does not significantly contribute to the overall method.\n\n- Discuss the increased training time caused by the need for two forwards (with and without the complete model) during training as a limitation.\n\n- The paper should include an ablation study on the loss function. Given the significant performance differences observed in the distillation ablation study in the appendix, a comprehensive description of the loss function's impact is necessary.", "limitations": "The authors have adequately addressed some limitations of their work. However, they did not discuss the increased training time as a limitation. There are no negative societal impacts.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is the first to enhance inference efficiency and use fewer parameters for fine-tuning by drawing inspiration from dynamic neural networks, achieving higher efficiency and performance than existing methods.\n\n- The paper is well-written and easy to understand, with a clear and intuitive explanation of the proposed method.\n\n- The experiments are extensive, demonstrating results on various datasets and providing detailed analyses, such as model variants, the number of MoE experts, and the layers where token dropping occurs. This thorough experimentation helps in-depth understanding of the proposed method.", "weaknesses": "- The application of MoE in the paper detracts from the overall understanding and value of the proposed method. MoE improves performance only in video classification, not image classification, and is absent from most experiments. Furthermore, Table 6 shows that using MoE results in fewer FLOPs, which is theoretically implausible. The paper lacks an analysis explaining this and only provides experimental results, which hinders comprehension. MoE's inclusion in the main paper is questionable and might be better suited for the appendix.\n\n- The proposed method may appear as a simple combination of dynamic neural networks and adapters. While it differs in learning token dropping during fine-tuning, it essentially drops tokens from the original block while using all tokens in the adapter. The token dispatch mechanism is similar to existing dynamic neural network ideas. The paper should report the performance of DynamicViT and EViT's full fine-tuning and their combination with other PEFT methods in Table 5. If the proposed method performs better, an analysis explaining this phenomenon is necessary.\n\n- There are several instances where reported FLOPs differ significantly from theoretical values, requiring more detailed explanations. For instance, in Table 1, it is unclear why MLP Dispatch has lower FLOPs than Layer Dispatch, given that the attention layer uses all tokens in MLP Dispatch but fewer tokens in Layer Dispatch.\n\n- While the proposed method achieves higher performance despite added constraints compared to existing PEFT methods, the paper lacks explanations or analyses for this improvement. The authors should explain why their method outperforms traditional adapters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We will include all explanations and results in the revision.\n\n> #### W1:MoE improves performance only in video classification, not image classification, and is absent from most experiments\n\nWe appreciate this comment. We would like to clarify that the MoE-adapter is primarily designed to enhance the adapter’s capacity to **address challenging tasks** (Lines 180-183). In addition to video classification, we validate its effectiveness in semantic segmentation (Appendix Table 10) and object detection (In reponse to Reviewer uJS2). We do not employ it in the VTAB-1K experiments as it does not outperform the model without it, likely due to the extremely limited training data.\n\n> #### W2:Table 6 shows that using MoE results in fewer FLOPs, which is theoretically implausible\n\nThanks. We would like to clarify that it is reasonable for the model with the MoE adapter to result in fewer FLOPs:\n- The FLOPs of our models depend on **learned token dispather** during fine-tuning and may slightly **fluctuate around the target FLOPs** (controlled by $L_{rate}$). \n- The extra computational cost of the adapter and the MoE adapter is nearly equivalent (Lines 202-204). \n\n> #### W3 & Q1:MoE might be better suited for the appendix\n\nWe appreciate this valuable suggestion. In the revision, we will move the introduction of the MoE adapter and its corresponding experiments to the Appendix.\n\n> #### W4:The token dispatch mechanism is similar to existing dynamic neural network\n\nThanks. We present the differences below:\n|Method|Training|Efficient strategy|FLOPs for different samples| Classification|Dense Prediction|Target dataset\n|-|-|-|-|-|-|-\n|DynamicViT|Full-parameter|Learn to keep P% tokens at certain layers (e.g. 3th,6th 9th layers in ViT-B)|same|✅|❌|In-domain\n|EViT|Full-parameter|Only keep top-K attentive tokens and fuse other tokens at certain layers (e.g. 4th,7th, and 10th layer in ViT-B)|same|✅|❌|In-domain\n|DyT (Ours)|Parameter-efficient|Learn how many tokens to skip before which MLPs|data-dependent|✅|✅|Cross-domain\n\n**Analysis & Conclusion:**\n- Dynamic neural networks typically prune a **certain number or percentage** of tokens in fixed layers, while our token dispatcher **learns the number of tokens to skip** before each MLP block.\n- DynamicViT and EViT maintain the **same FLOPs for all samples**, while DyT achieves **data-dependent computation**, which is more flexible and reasonable.\n- Instead of token pruning, the token skipping in our token dispatcher allows the DyT to preserve **complete feature maps**, enabling it to perform dense prediction tasks.\n\n> #### W5:The paper should report the performance of DynamicViT and EViT's full fine-tuning and their combination with other PEFT methods\n\nSincerely thanks. We present the results below. \n\n**Setting:**\nWe combine DynamicViT and EViT with AdaptFormer. The bottleneck dimmension is set to 8.\n|Method|VTAB-1K accuracy|FLOPs(G)|Tunable Param.(M)| Throughput (img/s)\n|-|-|-|-|-\nDynamicViT|60.10|14.05|88.70|1010.40\nDynamicViT+AdaptFormer|75.48|14.23|3.10|954.82\nEViT|67.42|11.62|85.8|1233.45\nEViT+AdaptFormer|74.63|11.77|0.16|1152.38\nDyT r=0.5 |77.17|12.54|0.16|912.39\nDyT r=0.5 + ToMe|76.60|9.85|0.16|1114.70\n\n**Analysis and Conclusion:**\n- Combining DynamicViT and EViT with AdaptFormer enhances performance, validating the significance of exploring both parameter and inference efficiency for vision transformers\n- DyT outperforms “DynamicViT+AdaptFormer” and “EViT+AdaptFormer” by learning to skip an appropriate number of tokens at each MLP block, rather than pruning a fixed number (EViT) or percentage (DynamicViT) of tokens at specific layers. Figure 5 in the main paper illustrates that different datasets benefit from varying token-skipping strategies.\n\n> #### W6:It is unclear why MLP Dispatch has lower FLOPs than Layer Dispatch\n\nThanks. There are two reasons:\n- The actual token activation rate during inference depends on the **learned token dispatcher,** causing real FLOPs to fluctuate around the theoretical value. \n- We do not strictly control FLOPs across the four variants. Specifically, we set the activation rate $r$ to 0.5 for “Attention Dispatch” and “MLP Dispatch” variants, and to 0.7 for “Attention-MLP Dispatch” and “Layer Dispatch” variants, ensuring **similar** average FLOPs. Figure 6 (Appendix) shows that “MLP Dispatch” consistently achieves the best performance under similar FLOPs.\n\n> #### W7:The authors should explain why their method outperforms traditional adapters\n\nThanks. We list the explanations below:\n- The dynamic architecture in DyT enhances generalization. It introduces a form of disturbance in the input data, akin to Dropout. This mechanism is particularly crucial when training data is limited e.g. VTAB-1K.\n- The distillation loss in DyT. We adopt the complete model as the teacher of the dynamic model, significantly enhancing performance. Such a self-distillation mechanism is only available in the dynamic architecture.\n- Previous work [1] and DynamicViT also show dynamic architectures outperforming static models with fewer FLOPs.\n\n> #### Q2&L1:Discuss the increased training time caused by the need for two forwards\n\nThanks. We acknowledge that training with two forward passes takes 1.8 times longer than only one pass, but this significantly enhances performance without compromising parameter and inference efficiency. Given that our primary contribution focuses on improving inference efficiency, the additional training time would be acceptable.\n\n> #### Q3:The paper should include an ablation study on the loss function\n\nThanks. We present the results below.\n|Method|VTAB-1K accuracy \n|-|-\n|DyT|77.14 \n|DyT w/o $L_{distill}$|76.70| \n|DyT w/o $L_{distill}$ & $L^\\prime_{cls}$| 75.73\n\n**Analysis and Conclusion:**\n\nRemoving $L_{distill}$ or $L^\\prime_{cls}$ negatively impacts performance, validating the effectiveness of loss functions in DyT.\n\n> #### Reference:\n\n[1] Latency-aware Unified Dynamic Networks for\nEfficient Image Recognition. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HuMPFPslFV", "reviewer": "Reviewer_uJS2", "summary": "The paper introduces a method called Dynamic Tuning (DyT) designed to enhance both parameter and inference efficiency when adapting Vision Transformers (ViTs) for various visual tasks. DyT employs a token dispatcher to selectively process only the most informative tokens, reducing computational redundancy. Additionally, an enhanced adapter inspired by the mixture-of-experts mechanism is introduced to boost performance with minimal extra cost. The method is comprehensively evaluated across tasks like image and video recognition and semantic segmentation, demonstrating superior performance and significantly reduced FLOPs compared to existing parameter-efficient fine-tuning methods. The results underscore DyT's effectiveness in achieving state-of-the-art performance while being computationally efficient.", "questions": "1. I am concerned about the potential performance impact of token pruning on the detection task. Have the authors conducted any experiments in this regard?\n\n2. I came across a paper [1] that dynamically assesses token significance, and it would be beneficial to discuss and cite it in the article.\n\n   \n\n\n---\n   [1] Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysis[C]//CVPR. 2024.", "limitations": "The authors have discussed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and easy to follow, the figures are clear and easy to understand.\n2. Most previous PEFT methods often focus on decreasing tunable parameters in training to conserve memory. However, these approaches do not lessen computational load during inference (and may even increase it, as seen in Adapter, Prompt, etc.). This paper introduce token pruning to help to reduce model inference costs.", "weaknesses": "1. The authors introduced additional loss functions for DyT, does this affect the speed of convergence?\n\n1. Gumbel-Sigmoid is a prevalent technique for maintaining end-to-end training. However, its computational outcomes may vary compared to Sigmoid. It is valuable to explore the disparities in the calculated results of these two methods.\n\n1. Certain experimental phenomena may become more transparent with additional explanations provided within the text. For instance, **(a)** Image Accuracy in Table 2 demonstrates optimal performance when excluding MoE, and it appears that the performance does not exhibit an increase proportional to the number of experts. **(b)** Meanwhile, for N=12, the FLOPs are paradoxically lower than when N=8. **(c)** In Table 6, the FLOPs of DyT N = 4 is higher than Dynamic-Full, this seems counterintuitive. **(d)** In Table 10, the decrease in FLOPs for DyT is not as significant as in VTAB-1K, and the FLOPs for DyT N=4 is higher than the version not adapted to the MoE. *The reasons behind these observations warrant further exploration*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> #### W1: Does introduced additional loss functions affect the speed of convergence\n\nThanks. The introduced additional loss functions do not impact the speed of convergence. We plot the loss values throughout the fine-tuning process and record the test accuracy at every training epoch. Please find the figure in the **attachment PDF**.\nThe explanation and figures here will be included in the Appendix, with a new section discussing the convergence speed.\n\n> #### W2: It is valuable to explore the disparities in the calculated results of Sigmoid and Gumbel-Sigmoid.\n\nThanks. We would like to clarify that DyT **only employs the Gumbel-Sigmoid during training** and replaces it with Sigmoid in the inference. Based on the suggestion, we conduct an experiment using Sigmoid during both training and inference, as shown in the table below:\n\n**Setting:**\n\"DyT-\" denotes using Sigmoid during both training and inference. We report the average FLOPs on CIFAR-100.\n|Method|FLOPS|CIFAR-100|SVHN|Food-101\n|-|-|-|-|- \n|DyT|12.21|91.37|97.08|90.32\n|DyT-|12.52|88.48|96.63|85.41\n\n**Analysis & Conclusion**\n- Replacing the Gumbel-Sigmoid with Sigmoid negatively impacts performance, as it results in a non-differentiable problem and increases the difficulty of fine-tuning (Lines 137-149). We will include this experiment in the Appendix in the revision.\n\n\n> #### W3: Certain experimental phenomena may become more transparent with additional explanations\n\nThanks. These valuable questions and explanations will be added to the “Frequent Questions” section in the Appendix of the revised manuscript.\n\n|Question|Answer| \n|-|-|\n|(a) Image Accuracy in Table 2 demonstrates optimal performance when excluding MoE | The aim of the MoE-adapter is to **tackle challenging tasks** e.g. video classification and dense prediction, without introducing extra computational cost. For relatively easy image tasks, introducing more tunable parameters through MoE does not bring benefits.  \n|It appears that the performance does not exhibit an increase proportional to the number of experts | Performance improves from $N=2$ to $N=4$, achieving the best results on video tasks. This indicates that **an appropriate number** of experts can enhance performance. However, too many experts ($N=8$ and $N=12$) increase the difficulty of fine-tuning and degrade performance. \n|(b) Meanwhile, for N=12, the FLOPs are paradoxically lower than when N=8 | Theoretically, the MoE-adapter with any number of experts should have similar FLOPs to the ordinary adapter (Line 202-204). Meanwhile, the actual activation rate during inference depends on the **learned token dispatcher** after fine-tuning, resulting in **slight fluctuations in FLOPs** between different models. These exlain why DyT N=12 may have lower FLOPs than DyT N=8 |\n|(c) In Table 6, the FLOPs of DyT N=4 is higher than Dynamic-Full, this seems counterintuitive.| First, \"Dynamic-Full\" does not have additional cost of adapters compared to DyT and DyT $N=4$. Additionally, the **learned token dispatcher** varies between models, leading to slight differences in actual token activation rates during inference. This results in the posibility of Dynamic-Full (12.24 GFLOPs) being slightly more efficient than DyT $N=4$ (12.29 GFLOPs).\n| (d) In Table 10, the decrease in FLOPs for DyT is not as significant as in VTAB-1K | On one hand, in all experiments, DyT is applied **only to the Vision Transformer**. The vision transformer in Table 10 equips a segmentation head from UperNet [4] for semantic segmentation. On the other hand, the proposed \"MLP Dispatch\" in DyT does not reduce the FLOPs in self-attention, whose FLOPs increase with the increasing size of the feature map. This reuslts in a less significant FLOPs reduction ratio compared to VTAB-1K. \n| and the FLOPs for DyT N=4 is higher than the version not adapted to the MoE. |  In Table 10, the FLOPs of DyT $N=4$ (584.40) is slightly lower than DyT (584.57).  This slight difference is introduced by the **learned token dispatcher** when the fine-tuning convergence.\n\n> #### Q1: Potential performance impact of token pruning on the detection task.\n\nThanks for this valuable suggestion. We conduct experiments on the COCO. Unlike token pruning, DyT employs a **token skipping** method, enabling it to effectively address the object detection task.\n\n**Setting:**\nwe adopt ViTDet[2] as the detector. We fine-tune the model for 12 epochs on the COCO dataset. The bottleneck dimension $d$ in adapters is set to 128. Here, we only measure the **FLOPs in the vision transformer and feature pyramid**.\n|Method|FLOPs(G)|Tunable Param.(M) in backbone|BBox AP|Seg AP\n|-|-|-|-|-\n|Full tuning|554.86|85.80|44.57|39.78\n|AdaptFormer|564.531|2.56|38.71|35.27\n|DyT|468.40|2.56|39.78|36.11\n|DyT $N=4$|466.32|10.24|40.97|37.11\n\n**Analysis & Conclusion:**\n- The full-tuning method performs the best, likely due to the significant gap between bounding box regression and the pretraining of the vision transformer, requring more parameters in the backbone to be updated.\n- DyT demonstrates superior performance compared to the AdapterFormer[3] with fewer FLOPs, validating the effectiveness of our approach.\n- DyT with the proposed MoE-adapter (DyT $N=4$) surpasses the original DyT, highlighting the importance of the MoE-adapter.\n\n> #### Q2: Paper [1] dynamically assesses token significance, and it would be beneficial to discuss and cite it\n\nWe appreciate this valuable suggestion. We carefully read the paper and will cite it by \"Zhou et.al [1] propose assessing token significance with a ReLU function to effectively tackle point cloud analysis\" in Related Work.\n\n> #### Reference\n\n[1] Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysis, 2024\n\n[2] Exploring plain vision transformer backbones for object detection, 2022\n\n[3] Adaptformer: Adapting vision transformers for scalable visual recognition, 2022\n\n[4] Unified perceptual parsing for scene understanding, 2018", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tpctweWtXs", "reviewer": "Reviewer_Zpyf", "summary": "This paper proposes Dynamic Tuning (DyT) to improve both parameter and inference efficiency for ViT adaptation. DyT inserts a token dispatcher for each transformer block to learn to dynamically determine whether a token should be activated or deactivated.", "questions": "See Weakness", "limitations": "Yes, the authors discuss limitations in the Discussion and Conclusion section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper focuses on **inference efficiency**, which is an important problem for adapting large pre-trained models to downstream tasks using PEFT methods. \n- This paper proposes Dynamic Tuning to improve the inference efficiency for ViT adaptation. Extensive experiments and analyses validate the proposed method.", "weaknesses": "I don't see any major weakness in the paper, but I have a minor concern about its novelty. The core technique in Dynamic Tuning seems to be **token pruning** without explicitly reducing the number of tokens, but token pruning itself is not a novel technique. This paper does not well demonstrate the difference between existing dynamic token pruning methods and the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> #### W1: The core technique in Dynamic Tuning seems to be token pruning without explicitly reducing the number of tokens. This paper does not well demonstrate the difference between existing dynamic token pruning methods and the proposed method.\n\nThanks. We would like to kindly clarify that our method is **not a token pruning** method but rather **a token skipping method**. In our approach, some tokens skip the original blocks and only pass through the adapter, **preserving the feature map** completely. Below, we list the differences between our method and representative token pruning methods (DynamicViT[1], EViT[2], ToMe[3]) below. We will include these in Section 2 in the revision.\n\n\n| Method     | Training     |    Efficient Strategy  | FLOPs for different samples | Classification | Dense Prediction | Target dataset\n| -------- | -------- | -------- | -------- | -------- |-------- |-------- \n| DynamicViT | Full-parameter |  Learn to hierarchically keep P% tokens at certain layers (e.g. 3th,6th 9th layers in ViT-B) | same | ✅ | ❌ | In-domain\n| EViT |  Full-parameter |   Only top-K attentive tokens and fuse the inattentive tokens at certain layers (e.g. 4th,7th, and 10th layer in ViT-B) | same | ✅  | ❌ | In-domain\n| ToMe |  Training free | Prune through merging N tokens via similarity in each layer |  same | ✅| ❌ Need modification | In-domain\n| DyT (Ours) | Parameter-efficient | Learn how many tokens to skip before which MLP block | data-dependent | ✅ | ✅ |  Cross-domain\n\nNote that, our method can be seamlessly combined with ToMe (Table 5)\n\n**Analysis:**\n- Compared to these methods, our approach does **not rely on a manually set number or percentage** of tokens to prune. Instead, it **learns where and how many tokens to skip**, constrained by the average activation rate loss $L_{rate}$. \n- Our model allocates computation in a **data-dependent** manner, which varies between different samples. \n- Our method can also handle the dense prediction task, whereas  other methods fail due to the incomplete feature map. \n- Previous token pruning methods primarily focues on accelerating the model within the same dataset used for pretraining, while DyT aims to improve efficiency during **cross-domain adaption**.\n\n\n> ### Reference\n\n[1] Dynamicvit: Efficient vision transformers with dynamic token sparsification, 2021\n\n[2] Expediting Vision Transformers via Token Reorganizations, 2022\n\n[3] Token Merging: Your ViT But Faster, 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BiPeFdeVtA", "reviewer": "Reviewer_gv1w", "summary": "This paper proposes DyT, which is a parameter-efficient fine-tuning method that can also achieve inference efficiency. DyT contains token dispatchers, which let tokens dynamically skip the original block, and MoE-adapter which uses multiple adapters to compose a mixture-of-experts. The paper evaluates the proposed method on various vision benchmarks to demonstrate its effectiveness and efficiency.", "questions": "- Please clarify the novelty of the proposed methods over Conditional Adapters and AdaMix.\n- DyT employs a more complex loss function and distillation. Does this increase the training time?", "limitations": "The paper has discussed the limitations of DyT.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper provides sufficient experimental results on image and video tasks.\n- The problem is very meaningful. The mainstream PEFT methods achieve efficiency in training and storage, but cannot achieve efficiency in inference.", "weaknesses": "- My main concern is the novelty of the proposed method. DyT mainly includes two innovations: the token dispatcher and the MoE-Adapter. However, to my knowledge, both of these have been explored in earlier works. \n    - First, token dispatcher is very similar to [1], both of them use a linear router to determine which tokens are input into the transformer's block and use parallel adapters to process all tokens. The difference lies in that [1] uses a soft top-k function to generate the mask, whereas this paper employs the sigmoid function.\n    - Second, [2] also used MoE as the adapters. The difference is that [2] performs routing separately for the down projection and up projection, whereas in this paper, the two are bound together.\n    \n- The line spacing is too narrow.\n\n\n[1] Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference. In NeurIPS 2023.\n\n[2] AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning. In EMNLP 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> #### W1 & Q1: Clarify the novelty of the proposed methods over Conditional Adapters and AdaMix\n\nThanks. We would like to clarify the novelty of our method over Conditional Adapters and AdaMix as follows:\n\n**Conditional Adapters[1]:**\n- The token selection strategy in the token dispatcher is novel. The token dispather in conditional adapter[1] selects top-K tokens in each layer to pass through. However, the token dispatcher in DyT **learn to select** an approprate number of tokens in each layer based on inputs.\n- The target model is different. The conditional adapter[1] is primarily designed for the encoder-decoder langauge model T5 [3]. Our experiments below demonstrate that applying this approach to vision transformers is suboptimal.  DyT’s novelty lies in proposing a token dispatcher specifically beneficial for vision transformer.\n- The block to conduct token skipping is novel. In conditional adapter [1], tokens directly skip the whole layer. In DyT, we propose four model variants, explore their effectiveness, and find that skipping the MLP block is suitable for vision transformer.\n\nTo verify that the superiority of our token dispatcher in vision transformer, we conduct experiments by replacing the proposed token dispatcher with the soft top-K method from [1].\n\n**Setting:** DyT-top-K denotes using the soft top-K in [1] as the token dispatcher. We report the average FLOPs on CIFAR-100.\n\n| Method     | FLOPS | CIFAR-100     |   SVHN | Food-101\n| -------- | -------- | -------- | -------- | -------- \n| DyT      | 12.21 |  91.37 | 97.08  | 90.32\n| DyT-top-K | 12.26 | 81.4   | 93.57  | 79.47\n\n**Analysis:** DyT achieves obviously better performance than \"DyT-top-K\". There are two reasons:\n\n- The soft top-K operation in [1] always skips K tokens per block. In contrast, DyT’s **learned token dispatcher** skips **fewer tokens in lower layers and more in higher layers** (see Figure 5). This indicates that the number of skipped tokens should vary by layer, and skipping a fixed number of tokens, as in [1], is suboptimal.\n- The computational allocation for each test sample is constant in [1], whereas DyT **allocates different FLOPs for different samples**, allowing the model to handle hard and easy samples differently.\n\n**Conclusion:**\nThe token dispatcher in [1] is not suitable to vision transformer, verifing the importance of DyT. We will add this experiment into the Appendix in the revision.\n\n**AdaMix [2]:**\n- Our routing mechanism is novel. During training, AdaMix [2] lacks a **learning-based router**, instead randomly selecting a down projection layer and an up projection layer for each batch of data.\nConsequently, all images in a batch are processed by the same experts. In contrast, the MoE-adapter in DyT employs a learning-based router to generate scalar weights for the experts based on input features, allowing features from different images to yield distinct expert weights.\n- The adapter architecture is novel. During inference, AdaMix fuses all experts by weight averaging, resulting in an ordinary adapter like [4], whereas we retain all experts. Thus, the MoE-adapter in DyT has more capacity to tackle challenging tasks. Experiments below verify this.\n\nBelow, we conduct experiments by replacing our MoE-adapter with AdaMix.\n\n**Setting:**\n\"DyT-AdaMix\" denotes using the AdaMix [1] as the adapter. \"DyT $N=4$\" denotes the DyT with our MoE-Adapter.\nWe report the accuracy on three image datasets and two video datasets. The average FLOPs on CIFAR-100 and K400. The bottleneck dimension in the adapter is set to 64 for all models.\n| Method    | Params. (M) | Image FLOPs | CIFAR-100     |   SVHN | Food-101 | Video FLops|K400 | SSv2\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- \n| DyT      | 1.19 | 12.21  | 91.37 |  97.08 | 90.32 | 108.31  |75.28 | 45.43\n| DyT $N=4$ | 4.80 |12.29 |  91.01 | 96.90 | 89.77 | 105.45 | 75.43 | 46.56\n| DyT-AdaMix  | 1.19 |12.24 |  91.57 | 97.26 | 89.99 | 108.45 | 72.70 | 41.41\n\n**Analysis:**\n- \"DyT-AdaMix\" exhibits slightly better performance on image datasets due to its training strategy.\n- “DyT-AdaMix” does not enhance DyT’s performance on challenging tasks, such as video classification, and even significantly reduces performance.  This may be attributed to the random selection mechanism, which further diffuses learning on challenging tasks.\n- As an method to improve performance in challenging tasks without introducing extra FLOPs, our MoE-adapter achieves the best performance on K400 and SSv2.\n\n**Conclusion:**\n\nThe adapter in AdaMix [2] is **a training strategy** rather than an effective adapter architecture. It does not assist DyT in tackling challenging tasks. We will add this experiment to the Appendix in the revision.\n> #### W2: The line spacing is too narrow.\n\nWe sincerely appreciate this valuable suggestion. We find that author are permitted to add one page to the main paper in the final version. We will increase the line spacing as recommended.\n\n> #### Q2: DyT employs a more complex loss function and distillation. Does this increase the training time\n\nThanks. We employ the complete model loss $L^\\prime_{cls}$ and distillation loss $L_{distill}$ to improve the performance.  Although their use increases the duration of a training iteration by 1.8 times compared to not using them, this approach brings significant performance improvements (See Table 11). Given that our primary objective is to improve inference efficiency, this additional training cost is acceptable. Therefore, we believe it is valuable to include these losses during fine-tuning.\n> #### Reference\n\n[1] Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference. 2023.\n\n[2] AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning, 2022. \n\n[3] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. 2020.\n\n[4] Adaptformer: Adapting vision transformers for scalable visual recognition, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear ACs and reviewers,

We sincerely appreciate the time and effort provided by all reviewers and ACs in our work. In particular, we are encouraged to see that Reviewer fmWb finds that our method **"achieves higher efficiency and performance than existing methods"**. Reviewer uJS2 thinks the proposed method **”is comprehensively evaluated across tasks“** and demonstrates **"superior performance and significantly reduced FLOP“**. Reviewer Zpyf **“don't see any major weakness in the paper"**. Reviewer gv1w acknowledges that the problem proposed in this paper **"is very meaningful"**.

We will continue to refine our work based on the feedback. 


Thanks,

Authors of submission 3553

Meta-review: The reviewers recognized that the proposed approach is novel and achieves both efficiency and performance. They also appreciated the clarity of the manuscript and extensive experiments. However, at the same time, the reviewers raised concerns with incremental novelty, lack of detailed discussion on experimental results, convergence speed, and some presentation issues. These concerns are successfully addressed by the rebuttal and subsequent responses in the discussion. Consequently, the reviewers unanimously championed the paper after the discussion period. The AC agrees with the reviewers and recommends acceptance. The authors are strongly encouraged to polish the manuscript according to the valuable comments and to add new results and discussions brought up in the responses to the revision.
Decision: Accept (poster)