Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation

Authors: wangbo  zhao
jiasheng  tang
yizeng  han
yibing  song
kai  wang
gao  huang
fan  wang
yang  you
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 03:12:30

Abstract: Existing parameter-efficient fine-tuning (PEFT) methods have achieved significant success on vision transformers (ViTs) adaptation by improving parameter efficiency. However, the exploration of enhancing inference efficiency during adaptation remains underexplored. This limits the broader application of pre-trained ViT models, especially when the model is computationally extensive. In this paper, we propose Dynamic Tuning (DyT), a novel approach to improve both parameter and inference efficiency for ViT adaptation. Specifically, besides using the lightweight adapter modules, we propose a token dispatcher to distinguish informative tokens from less important ones, allowing the latter to dynamically skip the original block, thereby reducing the redundant computation during inference. Additionally, we explore multiple design variants to find the best practice of DyT. Finally, inspired by the mixture-of-experts (MoE) mechanism, we introduce an enhanced adapter to further boost the adaptation performance. We validate DyT across various tasks, including image/video recognition and semantic segmentation. For instance, DyT achieves superior performance compared to existing PEFT methods while evoking only 71% of their FLOPs on the VTAB-1K benchmark.

Main paper: 
 introduction: 
 With the remarkable success of Vision Transformers (ViTs) [20, 53, 26] , fine-tuing pre-trained ViT on other data domains [90] or task applications [89, 36, 60, 79, 31] is becoming a common practice. However, as model sizes increase [88, 16, 52] , the associated adaptation costs become prohibitive due to the burden of fine-tuning and inference on the target task. Parameter-efficient fine-tuning (PEFT) methods (e.g. AdaptFormer [12] , LoRA [34] , and VPT [36] ), are proposed to tackle the tuning problem by reducing tunable model parameters. They usually update a small amount of parameters while keeping the original model fixed, which reduces learnable parameters effectively while maintaining fine-tuning accuracy.Despite the extensive research in parameter efficiency, the inference efficiency on target tasks is less explored. We numerically demonstrate the inference efficiency of three representative PEFT methods in Figure. 1(a) , revealing that none of them reduces the computation during inference compared to full tuning. This limitation poses challenges for adapting pre-trained ViT to downstream tasks, particularly when the model is computationally extensive. To this end, we aim to unify both parameter and inference perspectives for efficient ViT adaptations. Figure 1 : FLOPs and Accuracy of ViT-B/16 [20] on VTAB-1K [89] . "Full tuning" denotes that all parameters are fine-tuned. AdaptFormer [12] , LoRA [34] and VPT [36] are typical PEFT methods.Dynamic networks [83, 67, 48] demonstrate that token pruning helps to reduce model inference costs. However, these designs primarily focus on pre-training from scratch or full tuning on the same dataset, without considering data domain transfer. Furthermore, the token pruning operation constraints these methods on visual recognition scenarios, limiting their further applications e.g. dense prediction [93] .Our motivation thus arises from how to develop dynamic modules together with PEFT methods to enhance both parameter and inference efficiency during ViT adaptation, which should also suit a wide range of visual tasks.In this paper, we introduce Dynamic Tuning (DyT), an efficient approach for ViTs adaptation that achieves both parameter and inference efficiency. Specifically, we design a token dispatcher for each transformer block learning to dynamically determine whether a token should be activated or deactivated. Only those activated tokens will traverse the entire block and an extra lightweight adapter module, while the remaining tokens skip the original block and will solely be processed by the adapter. DyT does not reduce the total number of tokens, making it suitable for both visual recognition and dense prediction scenarios. Additionally, since the impact of token skipping during adaptation has not been explored, we propose four model variants to determine the best practice for DyT. Finally, as the adapter module is set to process all the visual tokens, we propose a mixture-of-expert(MoE)-based adapter design that further enhances token processing with negligible additional FLOPs. Through these designs, our DyT fulfills both parameter and inference efficiency for ViTs adaptation.Comprehensive evaluations for DyT are conducted from multiple perspectives. In the image domain, as shown in Figure 1 (b). DyT surpasses existing PEFT methods while consuming only 71% of the ViT-B FLOPs on the VTAB-1K benchmark [89] . When visual tokens are scaled up from images to videos, our DyT shows superior generalization ability on action recognition benchmarks, e.g. K400 [10] and SSV2 [25] , with a reduction of 37GFLOPs. In the scenario where labels are scaled up from recognition to segmentation, our DyT even outperforms full tuning on ADE20K [93] with 21GFLOPs reduction. These experimental results indicate that the proposed DyT is efficient in both parameter and inference across various visual tasks and directs a promising field for efficient model adaptation. 
related works: 
 Parameter-efficient fine-tuning. Parameter-efficient fine-tuning (PEFT) is designed to adapt a pre-trained model to downstream tasks by only tuning a small part of the parameters. Existing PEFT methods can broadly be categorized into three groups: adapter-based methods, re-parametrizationbased methods, and prompt-based methods. Adapter-based methods [12, 33, 62, 38, 14] insert some tiny modules into the original model and only these inserted modules are updated during fine-tuning. Re-parametrization approaches [87, 34, 47, 21, 18] usually cooperate with reparameterization techniques, directly modifying specific parameters in the pre-trained model. Prompt-based methods [36, 2, 95, 94, 40, 9, 8] involve appending a small number of learnable tokens to input sequences, thereby adapting intermediate features in frozen layers. Please refer to [84, 46] to find more comprehensive reviews.However, PEFT methods primarily focus on improving parameter efficiency during fine-tuning while overlooking inference cost reduction. In this paper, we propose to improve the parameter efficiency and address inference hosts simultaneously for efficient ViT adaptation.Dynamic neural networks. Dynamic neural networks [28, 63, 92] can dynamically adjust their architectures based on the input data, which enables them to control the computational redundancy based on input data. Existing methods have explored dynamic layer depth [71, 4, 81, 30, 27, 86] , dynamic channel width [32, 43, 29, 82] and dynamic routing [45] in convolutional neural networks. When it comes to the era of vision transformer, many works [77, 76, 69, 67, 48, 74, 57, 59, 58, 64] attempt to improve the inference efficiency by reducing the token redundancy. For instance, Liang et al. [48] identify and consolidate inattentive tokens into only one token in each transformer layer. Rao et al. [67] progressively discard less informative tokens between layers. Wang et al. [76] adjust the number of tokens to represent images. A more detailed literature review can be found in [28, 75] . Zhou et al. [96] propose assessing token significance with a ReLU function to effectively tackle point cloud analysis. Although these approaches have shown significant success in vision tasks, they require training a model from scratch or fine-tuning all parameters on the same dataset used in pre-training, making them unsuitable for efficient ViT adaptation.In contrast to these methods, the proposed method can adapt pre-trained knowledge to diverse downstream datasets and tasks, while reducing the computational cost during inference by introducing negligible additional parameters.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See ConclusionGuidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
vits adaptation with dynamic tuning: 
 In the following, we first introduce the vision transformer and the adapter architecture in Section 3.1. Subsequently, we present the proposed dynamic tuning (DyT) and explore its four variants in Section 3.2 and Section 3.3, respectively. Furthermore, we build an MoE-adapter, which effectively enhances the adaptation performance without introducing additional computational cost, as elaborated in Section 3.4. Finally, we introduce the loss functions of our method in Section 3.5.Vision Transformer (ViT) consists of a stack of layers. The multi-head self-attention block denoted as Attn, and multi-layer perceptron block, termed as MLP are the main components in each layer. The input of each layer can be denoted as X = [x cls , x 1 , x 2 , ...x N ] ∈ R (N +1)×C , containing N image tokens and a classification token x cls .The adapter architectures have been widely explored in previous PEFT methods [12, 38, 33 ]. An adapter is generally designed as a bottleneck module, which consists of a project layer W down ∈ R C×d to squeeze the channel dimension and another project layer W up ∈ R d×C for dimension restoration. Here, d denotes the bottleneck dimension and d ≪ C ensures the efficiency of the adapter. Given an input feature X, the adapter operation is formulated as:EQUATIONwhere σ denotes a nonlinear function e.g. ReLU. In this study, we follow this general architecture to build the adapter within dynamic tuning. We place the adapter in parallel with an Attn block, MLP, or an entire transformer layer, which will be presented in detail in Section 3.3.In Vision Transformers (ViTs), the computational burden primarily resides in the transformer layers. Specifically, the operations in Attn and MLP account for approximately 35.8% and 63.5% of total FLOPs in ViT-B/16 [20] . Previous works [67, 48] have revealed that there exists a tokenredundancy issue in ViTs and found that some tokens can be discarded without sacrificing the performance. Inspired by this, we propose an efficient ViT adaptation approach, named Dynamic Tuning (DyT), which not only maintains parameter efficiency during fine-tuning but also reduces redundant computation during inference. The core idea is dynamically selecting tokens for processing within transformer blocks. Given the input tokens X of a block, DyT can be formulated as:EQUATIONwhere Block can represent an multi-head self-attention block Attn, a multi-layer perceptron MLP, or an entire transformer layer. The proposed token dispatcher (TD) learns to selectively activate or deactivate tokens. Only the activated tokens are input into the Block, while all tokens are processed by the Adapter. Token dispatcher. The key point in DyT is to select partial tokens which will be passed through Attn and/or MLP. A straightforward approach is randomly selecting tokens with a predefined probability and adapting the model to conduct downstream tasks with these selected tokens. However, this simplistic strategy risks discarding informative tokens while retaining less important tokens, potentially hindering the adaptation performance. To tackle this problem, we propose a token dispatcher (TD), which learns to select tokens during the adaptation. Specifically, given the input tokens X ∈ R (N +1)×C , TD learns to selectively activate a sequential of tokens X s ∈ R K×C , where K represents the number of activated tokens. To achieve this, it should obtain a mask M ∈ R N +1 , which indicates whether a token should be activated or deactivated.To obtain M, we adopt a projection layer W g ∈ R C×1 followed by a sigmoid function to predict the activation probability S ∈ R N +1 . Then, we set 0.5 as the threshold to determine the activation of each token. This can be formulated as:EQUATIONM n = 1 denotes that the n-th token is activated and will subsequently undergo the process of Block.Conversely, if M n = 0, the token will be deactivated and skip the Block. In practice, we always set the mask value of the classification token x cls to 1, allowing it to traverse the entire network. Notably, the number of additional parameters introduced in TD is negligible, with only C parameters in the linear layer W g .Fine-tuning stage. However, directly using threshold makes M a discrete decision, resulting in a non-differentiable problem during fine-tuning. To address this, we incorporate the Gumbel Noise [32] into sigmoid to replace the original sigmoid function during fine-tuning. It can be formulated as:EQUATIONwhere G 1 , G 2 ∼ Gumbel(0, 1). τ represents the temperature and is set to 5.0 by default. Further details of Gumbel-Sigmoid are provided in the Appendix A.7. Subsequently, we can obtain M using the same operation in Equation.3. The Gumbel Noise makes the sampling of M stochastic during training and we adopt S as a differentiable approximation of M. Both of these help TD to be trained in an end-to-end manner. The calculation of forward and backward propagations during training can be formulated as:EQUATIONThe Equation 2 can be reformulated into:EQUATIONFrom Equation 5, only the block outputs, Block(X), of those activated tokens are retained, while others are masked out by M. As shown in Figure 2 (a), during the fine-tuning stage, all tokens within X still need to traverse the Block. For brevity, we omit the LayerNorm [1] in Attn and MLP blocks. "DyT" denotes the dynamic tuning presented in Figure 2 .Inference stage. During inference, we can directly adopt Equation 3 to generate the dispatch mask M and obtain activated tokens X s ∈ R K×C in TD. Then, we can only feed them into Block. Only processing K tokens effectively reduces the computational cost because K < N . In practice, we add padding to the output from the Block to maintain tensor shape. This results in Equation 2. See Figure 2 (b) for a detailed illustration of the inference stage.The Block in Equation 2 can be instantiated into any blocks in the original ViT, such as a multi-head self-attention block Attn, a multilayer perceptron block MLP, or even a complete transformer layer in ViT. Since the impact of skipping tokens in these blocks during the adaptation fine-tuning remains non-trivial to estimate and has not been explored in previous works, we propose four model variants and conduct experiments to determine the best practice.• Attention Dispatch: Considering the quadratic computational complexity of the Attn block with respect to the token numbers, skipping tokens before applying Attn can significantly reduce computation. In this design, multi-head self-attention is exclusively performed between activated tokens, while other tokens are bypassed, which may hurt the interaction between tokens.• MLP Dispatch: Based on the analysis in Section 3.2, we observe that MLP takes ∼63.5% FLOPs in ViT-B/16 and propose to skip tokens only before MLP. It enables that the interaction between tokens in Attn is not affected.• Attention-MLP Dispatch: An alternative strategy is skipping tokens before both self-attention and MLP blocks. This design permits a higher activation rate in both Attn and MLP while maintaining similar computational efficiency comparable to "Attention Dispatch" and "MLP Dispatch". However, it costs double the additional parameters in adapters.• Layer Dispatch: Inspired by "Attention-MLP Dispatch", we can dispatch tokens before a transformer layer. Specifically, tokens are identified by one TD to be activated or deactivated in the subsequent entire layer. With the same activation rate, its computation is similar to "Attention-MLP Dispatch" while requiring fewer parameters to build adapters.We demonstrate the architecture variants in Figure 3 . The experimental results and analyses of these variants are presented in Section 4.2.In DyT, the adapter is responsible for processing all tokens, requiring it to have enough capability, particularly when the downstream tasks e.g. semantic segmentation are challenging to adapt. To tackle this problem, we propose a MoE-adapter, inspired by mixture-of-experts [68, 80] . It effectively enhances the capability of the adapter with introducing negligible computation cost.A MoE-adapter comprises a routing layer W r ∈ R C×N and N adapter experts, denoted as{W i down ∈ R C×d , W i up ∈ R d×C } N i=1 .The routing layer generates a series of scalar as the weights for the experts based on input features. The features from different images may yield distinct expert weights. Specifically, we first conduct average pooling across all tokens to generate a feature as the global representation of them. Subsequently, this representation is fed into the routing layer to generate weight scalars {α 1 , α 2 , ...α N }. Finally, tokens are processed by each expert independently and combined with the corresponding weight. We demonstrate this in Figure 4 .However, this increases the computational cost of the adapter in proportion to the number of experts N . To address this problem, we adopt its mathematical equivalence to process X in practice, which can be formulated as:EQUATIONwhereW moe down = N i=1 α i W i down and W moe up = N i=1 α i W i up .The design endows the MoEadapter with the same capacity as N individual adapters, while maintaining computational efficiency equivalent to that of a single adapter (the computational cost in the routing layer is negligible). For an image I, we calculate the cross-entropy loss L cls = -y log(ŷ) between the predicted probability ŷ and its corresponding one-hot label y, to supervise the learning of classification. To control the average activation rate r in the entire model, we add a loss term to constrain the number of activated tokens in the dynamic tuning, which can be formulated as: 2 , where L denotes the number of layers in ViT. M l n represents the mask generated in TD from the l-th layer. Additionally, we employ a loss L ′ cls = -y log(y ′ ) to supervise the adaptation of the complete model, where y ′ is the output probability without employing the token dispatcher. Thus, this complete model can also act as a teacher to enhance the dynamic tuning during adaption by a distillation loss L distill = KL(y ′ , y), where KL represents the Kullback-Leibler divergence loss. Therefore, the overall loss function is defined as L = L cls + L ′ cls + L distill + αL rate , where α serves as the weight of the activation rate loss and is set to 2.0 by default. Note that, DyT can also achieve competitive performance without the distillation loss (see Appendix A.6).L rate = ( 1 L×N L l=1 N n=1 M l n -r)Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiment settings: 
 Datasets. To evaluate the adaptation performance, we conduct experiments on VTAB-1K [89] benchmark. The training data in this benchmark is extremely limited, with only 1,000 training samples for each task. Different from existing PEFT works [12, 38, 39] , which mainly focusing on VTAB-1K, we also conduct experiments on three image classification datasets with complete training sets, including CIFAR-100 [41] , SVHN [24] , Food-101 [6] . Additionally, we adopt two video datasets, Kinetic-400 (K400) [10] and Something-Something V2 (SSv2) [25] , to evaluate the performance when the number of tokens scaled up. All images or frames are resize into 224 × 224. For the dense prediction task, we evaluate our method on two widely recognized semantic segmentation datasets, AED20K [93] and COCO-stuff [7] . The results of semantic segmentation are presented in the Appendix A.4. We run each task three times and report the averaged results. The error bars are small (< 0.1) and omitted for simplification.Implementation Details. We conduct all experiments based on the ViT-B/16 [20] , which is pretrained on ImageNet21K [17] with full supervision, unless otherwise specified. Results based on ViT-L are presented in the Appendix A.10. The bottleneck dimension d is set to 64 by default. We adopt the same training schedule as [12] . Detailed hyperparameters for each experiment can be found in the Appendix A.8. The default setting in experiments is marked in color .Model variants. In Table 1 , we compare the performance of four model variants across both image and video datasets. We set the activation rate r in "Attention Dispatch" and "MLP Dispatch" variants to 0.5, and to 0.7 for "Attention-MLP Dispatch" and "Layer Dispatch" variants, and train each respectively. This results in similar average FLOPs for four variants. We observe that the default setting "MLP Dispatch" achieves superior performance across five datasets while maintaining the lowest computational cost. Although "Attention-MLP Dispatch" and "Layer Dispatch" also exhibit good performance on K400, the former incurs double additional parameters while the latter lacks generalization capability on other datasets. The comparison between "MLP Dispatch" and other variants proves that only skipping tokens in MLP blocks is a better design. More investigations on model variants can be found in our Appendix A.3.We conduct experiments to explore the effectiveness of the MoEadapter and the results are illustrated in Table 2 . The MoE-adapter design ensures that the FLOPs will theoretically remain the same as the ordinary adapter, with the computational cost from the routing function being negligible. However, in practical scenarios, the computational cost is also influenced by the learned token dispatch strategy within the Token Dispatcher (TD), leading to slightly varying FLOPs across different models in Table 2 .We observe that the performance on image datasets drops when we increase the expert number in MoE-adapter. This phenomenon can be attributed to the simplicity of image datasets and the model does not require too many parameters to adapt. In contrast, for video datasets, such as K400 and SSv2, the best accuracy is achieved 4 experts. The reason behind this is that the domain gap between the pre-training dataset and video datasets is large and the model needs sufficient adapter capacity to learn the adaptation. This proves that we can introduce the MoE-adapter when the target dataset is difficult to adapt.Visualization of token activation rate in each layer. In Figure 5 , we visualize the token activation rates across different layers in ViT-B/16 [20] . We observe that the model tends to activate more tokens in lower layers while deactivating tokens in higher layers. This phenomenon can be attributed to that the model tends to take more general knowledge from the lower layers of the pre-trained model and learn more task-specific knowledge in higher levels. Additionally, the activation results vary across different datasets. For instance, SSv2 demonstrates increased token activation rate in Layer 5 and Layer 6 compared to other datasets, whereas SVHN experiences substantial token deactivation in Layer 6, 7, and 8. This discrepancy arises from that the model needs different knowledge from the pre-trained weights to address dataset-specific challenges.It is noteworthy that nearly all tokens are deactivated in the final layer across five datasets, especially CIFAR-100, SVHN, and K400, where the activation rates are exactly 0%. This indicates that on these datasets, we can directly drop the original MLP block in Layer11 without hurting the performance, which further reduces about 4.7M * , 5.4% of the ViT-B total parameters. We visualize the token activation rates in ViT-B/16. "Overall" denotes the mean activation rate in the whole model, which arrives at around 50% when r is set to 0.5. "Layer0" and "Layer11" denote the lowest and highest level, respectively. Notably, the activation rate in the last layer is exactly 0% on CIFAR-100, SVHN, and K400 datasets.Visualization of activated tokens. In Figure 6 , we visualize two representative samples from K400. We can observe that the model tends to deactivate those tokens that are less informative e.g. tokens of the sky in (a) and tokens of grass in (b). In higher layers, such as layer7 and layer10, only those tokens from the primary objects are activated. This further proves the the existence of token redundancy problems in ViT and provides validation for the rationality behind our approach.Additional visualizations are available in Appendix A.11.EQUATIONDeactivate less informative tokens from the sky Tokens from the primary object (boy) are always activated Results verify that the token dispatcher has learned to identify informative tokens during fine-tuning.Comparison with PEFT methods. To evaluate the adaptation performance when the training data is limited, we adapt the VTAB-1K [89] benchmark, which a widely employed to evaluate the performance of adaptation tasks. Following exiting works, we reduce the bottleneck dimension d to 8, resulting in only 0.16M extra parameters. We vary the activation rate r in DyT in the range [0.5, 0.7, 0.9] and conduct fine-tuning, obtaining three models with different inference costs.The results are presented in Table 3 . Although previous methods, such as ConvPass [38] and Res-Tuning [37] , achieve satisfactory performance, they do not improve the efficiency during inference and even introduce additional FLOPs compared with full fine-tuning. In contrast, with only 12.54 GFLOPs, about 71% of the cost in original ViT-B, our method has outperformed previous methods e.g. LoRA [34] and VPT [36] , by a large margin. Remarkably, the DyT model with r = 0.9 does not surpass the performance of the DyT model with r = 0.7. This observation suggests the presence of computational redundancy in the transformer after adaptation, which further validates the significance of dynmaic tuning. These experimental results validate that DyT effectively improves parameter efficiency and inference efficiency while maintaining the adaptation performance.Dynamic tuning achieves actual acceleration. As a hardware-agnostic metric, FLOPs is not the most suitable choice to evaluate the inference efficiency across diverse platforms. The real inference speed has usually been ignored in previous PEFT methods. Here, we adopt two GPUs (Tesla V100 and Tesla T4) and a CPU Xeon(R) Platinum 8163 to comprehensively evaluate the efficiency of our methods and three representative PEFT methods, including LoRA [34] , AdaptFormer [12] , and VPT [36] . The batch size during inference is set to 512 and 32 for GPUs and the CPU, respectively. The results, as summarized in Table 4 , reveal that our method achieves better performance while effectively accelerating inference speed compared to existing PEFT methods on different platforms. Comparison and compatibility with methods for efficient transformers. We first investigate the domain adaptation performance of two representative methods DynamicViT [67] and EViT [48] . These methods are designed for efficient vision transformers. We adopt the optimal configurations outlined in their original papers and conduct experiments on VTAB-1K [89] benchmark. The results, as summarized in Table 5 , reveal that both methods achieve high throughput e.g. > 1000 (img/s), while the performance is unsatisfying. Combining DynamicViT and EViT with AdaptFormer [12] results in performance improvements, validating the importance of exploring both parameter and inference efficiency for vision transformers. Despite these gains, DyT obviously surpasses them, highlighting the superiority of our approach.Then, we explore the compatibility of our method with token pruning methods. Specifically, we combine DyT with ToMe [5] , a training-free technique that progressively prunes tokens through token merging. From the results in Table 5 , we find that ToMe can further enhance the throughput of DyT while maintaining accuracy. This proves the potential of our methods to be combined with existing token pruning methods e.g. [5, 11, 48] . Additionally, we apply ToMe to full tuning and AdaptFormer [12] in Table 3 and observe sub-optimal accuracy and throughput. These findings highlight that directly applying ToMe after fine-tuning or parameter-efficient fine-tuning is less effective compared to the proposed approach.Effectiveness on image datasets with sufficient training data. Although the results from VTAB-1K benchmark have proven the superiority of our approach, we extend our investigation to complete image datasets, to evaluate the adaptation performance with sufficient training data. We conduct experiments on 6 datasets including CIFAR-100 [41] , SVHN [24] , Food-101 [6] , Air [56] , Pet [61] , and Car [22] . The results are demonstrated in Table 6 . We further explore a straightforward approach, "Dynamic-Full", which has a token dispatcher as the same in DyT and is fine-tuned with all parameters. We observe that its performance becomes unstable and drops significantly on some datasets e.g. CIFAR-100 and Food-101. This phenomenon may arise due to the potential adverse impact of dynamic dispatch on the pre-trained parameters during full-tuning adaptation, thereby validating the importance of DyT. In this data-sufficient scenario, although our method achieves performance slightly below that of full tuning and AdaptFormer, it brings a significant reduction in FLOPs.Scaling token counts from images to videos. We conduct experiments on video datasets to show the performance when the number of tokens scaled up. The number of input frames is set to 8. For video tasks, similar to [85, 13] , we employ a cross-attention layer and a query token to aggregate features from different frames. The video classification is conducted on the query token. Additional implementation details are provided in the Appendix A.8. We demonstrate the results in Table 6 .Although DyT achieves slightly lower performance than AdaptFormer and LoRA, it costs obviously fewer FLOPs. DyT † containing four experts can achieve the best average accuracy with only cost 12.29 GFLOPs, which further verifies the superiority of our design. Previous methods for ViT adaptation primarily focus on improving the efficiency during the adaptation, thus aiming to reduce additional parameters. However, with the increasing size and computational cost of ViT models, the inference cost after the adaptation is becoming a heavy burden.In this paper, we unify both of these two problems into the efficiency problem for ViT adaptation and propose dynamic tuning (DyT) to tackle them simultaneously. We validate its performance and generalization across various tasks and datasets.Limitations and future works. DyT is currently designed for vision tasks. We believe it would be interesting to combine vision backbones with large language models e.g. [72] to build efficient large multi-modal models e.g. [50, 97] . DyT could be further developed to be compatible with multi-modal models, which would be our future direction.Difference with DynamicViT and EViT. Both DynamicViT [67] and EViT [48] are token pruning methods, whereas DyT is a token skipping method. DynamicViT learns to retain P % tokens at certain layers e.g. 3th,6th 9th layers in ViT-B. EViT only keeps top-K attentive tokens and fuses the inattentive tokens at certain layers e.g. 4th,7th, and 10th layer in ViT-B. These methods primarily focuses on accelerating the model within the same dataset used for pre-training, whereas DyT aims to improve efficiency during cross-domain adaptation.Difference with DiffRate. DiffRate [11] is a token compression method that performs token pruning and merging simultaneously.• In the DiffRate [11] , token pruning and merging is inherently data-independent. After training, a transformer layer prunes or merges the same number of tokens across all input data. In contrast, DyT is a data-dependent approach. The router in DyT learns to skip varying numbers of tokens before each MLP block based on the input data. • The prune and merge operations in DiffRate do not preserve complete feature maps, presenting challenges for dense prediction tasks. Thus, DiffRate requires modifications to address these tasks. Conversely, with only performing token skipping, DyT maintains complete feature maps, allowing it to effectively handle dense prediction tasks without any modifications.Difference with ToMe. ToMe [5] is a training-free technique that enhances inference efficiency by merging tokens based on similarity at each layer. DyT employs token skipping instead of merging and can be seamlessly integrated with ToMe to further improve efficiency.Difference with CoDA. CoDA [42] is a PEFT method that can also improve the inference efficiency.• The token selection strategy in the token dispatcher is different. CoDA selects top-K tokens in each layer to pass through while DyT adopt learnable dispatchers to select an appropriate number of tokens for each input. • The target model is different. Although CoDA [42] also improve both the parameter and inference during adaptation, CoDA primarily focus on the language model e.g. T5 [66] , while DyT is specifically designed for vision transformers. • The block to conduct token skipping is different. In CoDA, tokens directly skip the whole layer. In DyT, we propose four model variants, explore their effectiveness, and find that skipping the MLP block is the most suitable for vision transformer.Difference with AdaMix. AdaMix [73] also leverage a mixture of adapters, but it fuses all experts by weight averaging after training, resulting in an standard adapter akin to AdaptFormer [12] . In contrast, the proposed MoE-adapter employs a learning-based router to generate scalar weights for the experts based on input features, allowing features from different images to yield distinct expert weights.The proposed DyT performs dynamic dispatch at the token level using the token dispatcher (TD). In addition to token-level dispatch, we also investigate a sample-level dispatch, where all tokens within the selected samples are activated, while all tokens will be deactivated for other samples. To verify the importance of TD, we compare it against random dispatch, which randomly activates tokens or samples during both fine-tuning and inference. The experimental results are presented in Table 7 .Our observations reveal that token-level dispatch with TD consistently achieves superior performance across all datasets, except for a slight decrease compared to sample-level dispatch on the SVHN dataset. Notably, TD can achieve much better performance than the random dispatch strategy on both token-level and sample-level dispatch, particularly on video datasets K400 and SSv2, thereby validating the dispatch strategy learned in TD. Furthermore, the token-level dispatch surpasses the sample-level dispatch by a significant margin on most datasets, which demonstrates the importance of the finer-grained activation.Table 7 : Investigations of dispatch level and dispatch strategy. Combining the token-level dispatch and TD strategy results in the best performance. We consider two dispatch levels: "Token" (where dispatch is performed at the token-level) and "Sample" (where dispatch is conducted at the sample-level). "TD" and "Random" represent the learned dispatcher and random dispatch strategy, respectively. In Figure 7 , we further visualize the FLOPs-Accuracy curves of four model variants. We control the FLOPs by changing the activation rate r for the fine-tuning stage. For "Attention Dispatch" and "MLP Dispatch", we explore the activation rate in the range [0.1, 0.3, 0.5, 0.7, 0.9]. To maintain similar FLOPs as "Attention Dispatch" and "MLP Dispatch", we adjust the activation rate for "Attention-MLP Dispatch" and "Layer Dispatch" within the range [0.5, 0.6, 0.7, 0.8, 0.9]. Across all datasets, "MLP Dispatch" consistently outperforms other variants. The performance of "Attn Dispatch" experiences a significant drop when the activation rate is lower than 0.9, which also indicates that skipping tokens for self-attention blocks in ViT is not a suitable way, due to the importance of the token mixing function of self-attention. Remarkably, "MLP Dispatch" can surpass full tuning with obviously fewer FLOPs on both CIFAR-100 and Food-101, further validating the effectiveness of our approach. (c) Food-101Figure 7 : FLOPs-Accuracy curves of model variants on CIFAR100, SVHN, and Food-101 datasets. "MLP Dispatch" achieve the best FLOPs-Accuracy trade-off. We explore the activation rate r within [0.1, 0.3, 0.5, 0.7, 0.9] for "Attention Dispatch" and "MLP Dispatch". For "Attention-MLP Dispatch" and "Layer Dispatch" models, r is adjusted within the range [0.5, 0.6, 0.7, 0.8, 0.9]. "Full tuning" denotes the traditional full fine-tuning approach. To conserve space, we use simplified names for the variants.We investigate the impact of the bottleneck dimensions of the adapter in DyT and the results are summarized in Table 8 , revealing that different datasets prefer different configurations. When the dataset is easy to adapt, such as CIFAR-100, a bottleneck dimension of d = 4 is sufficient to achieve satisfactory performance. Conversely, video datasets require larger adapter dimensions e.g. d = 256 to attain better performance. We set the default bottleneck dimension to 64, which achieves a balance between performance and cost, such as the number of additional parameters and FLOPs. It is worth noting that the relationship between FLOPs and bottleneck dimension is not strictly monotonic due to the existence of token dispatcher. For instance, when d = 16, the FLOPs are lower than when d = 1, since the training may converge at activating fewer tokens as the number of adapter's parameters increases. We explore the temporal τ of the proposed dynamic tuning. The results are demonstrated in Table 9 .When the temperature is smaller e.g. 0.1, the Gumbel Sigmoid tends to produce binary outputs close to 0 or 1. Conversely, larger temperatures lead to more uniform outputs, approaching 0.5. Results demonstrate that the performance is not too sensitive to the temperature and our model can achieve reasonable performance with all temperatures in the table. We also observe that the model with τ = 1 achieves the best performance on CIFAR-100, SVHN, and SSv2, while decaying the temperature with a schedule leads to the best result on Food-101, which shows that adjusting the temperature can help the model to achieve better performance. Since identifying the optimal temperature is not the primary focus of this paper, we directly set the temperature to 5 by default. We also conduct experiments on two well-recognized semantic segmentation datasets ADE20K [93] and COCO-stuff [7] to demonstrate the ability of DyT on dense prediction tasks. Results are presented in Table 10 . Following previous works [3, 19] , we adopt the UperNet [78] as the segmentation head, and all images are resized into 512 × 512. We observe that the computational cost of semantic segmentation is much higher than the image and video discrimination tasks, primarily due to the high-resolution feature maps. Both DyT and DyT † still can reduce the computational cost obviously and achieve better performance than other PEFT methods, only slightly lower than full tuning on COCO-Stuff dataset.We conduct experiments on COCO [49] to explore the generalization of our method in object detection and instance segmentation. We adopt ViTDet [44] as the detector and fine-tune it for 12 epochs on the COCO dataset. The bottleneck dimension d in adapters is set to 128. As shown in Table 11 , DyT exhibits superior performance compared to AdapterFormer [12] , with fewer FLOPs. Our MoEadapter further enhances DyT without additional computational cost, validating the effectiveness of our design.However, full tuning achieves the best performance, surpassing other methods significantly. This is likely due to the gap between bounding box regression and the pre-training of the vision transformer, necessitating more parameter updates in the backbone. This challenge motivates us to design more powerful PEFT methods and integrate them with DyT to reduce the performance gap with full tuning. In the main paper, we adopt the complete model as our teacher, which does not employ a token dispatcher to skip tokens. Our proposed dynamic tuning allows the model to act as its own teacher during adaptation, a capability not achievable by other PEFT methods. We consider it to be a significant advantage of our approach. Joint training of the dynamic and complete models mitigates overfitting during adaptation, particularly with limited training data, such as VTAB-1K. Furthermore, the complete model, acting as a teacher, enhances the dynamic model's learning. These factors contribute to DyT's superior performance. Experimental results in Table 12 demonstrate that both complete model loss and distillation loss are useful for improving the performance of DyT. We also notice that introducing the complete model during training results 1.8 × longer training time. Given that our primary contribution focuses on improving parameter and inference efficiency, the additional training time introduced by the complete model would be acceptable.In Table 13 , we further present the results across all datasets in detail. We can find that, DyT without L ′ cls + L distill can still outperform most previous PEFT methods, further validating the superiority of our method.Gumbel-Softmax is proposed in [35] to conduct the differentiable sampling from a distribution. Given an unnormalized log probability † {E i } N i=1 , the Gumbel-Softmax can be formulated as: where G i denotes the Gumbel Noise sampled from a Gumbel distribution (G i ∼ Gumbel(0, 1)). We can consider the special case, where N = 2 and E 2 = 0, then p 1 can be defined as:EQUATIONEQUATION= Gumbel-Sigmoid(E 1 ), (12) obtaining the formulation of Gumbel-Sigmoid. Researchers in previous works, such as [23, 51] , have also leveraged the Gumbel-Sigmoid formulation to facilitate end-to-end training of neural networks.Experimental settings on VTAB-1K. Following previous works [38, 39, 37] , we fine-tune the model for 100 epochs on each dataset in VTAB-1K [89] . We do not use any data augmentation strategy in these experiments. We adopt the AdamW [55] optimizer. The learning rate is set to 1e-3 and gradually decays to 0 based on a cosine schedule [54] .Experimental settings on complete image datasets. We adopt the settings in Table 14 to fine-tune the ViT with the proposed dynamic tuning. Experiments on other parameter-efficiency methods such as AdaptFormer [12] , LoRA [34] , and VPT [36] also follow the settings in Table 14 . When we train a model with full tuning, we adopt a 1/10 base learning rate to make the training stable, otherwise, the model can not achieve reasonable results. Experimental settings on video datasets. We adopt two video datasets, Kinetic-400 (K400) [10] and Something-Something V2 (SSv2) [25] , to evaluate the performance as the token count scales up. The experimental settings are demonstrated in Table 15 . Most of the settings are borrowed from [60] .The number of input frames is set to 8. We adopt multi-view during the test, which is a common practice in video action recognition. However, the original ViT lacks temporal modeling capabilities.To address this limitation, we draw inspiration from [85, 13] . By introducing a cross-attention layer after the ViT, along with a query token, we effectively aggregate temporal information across different frames. The final video action classification is performed based on this query token. Experiments on parameter-efficient fine-tuning methods also follow these settings. We adopt a 1/10 base learning rate for those experiments on full tuning. Experimental settings on semantic segmentation datasets. We conduct experiments on ADE20K [93] and COCO-stuff [7] to demonstrate the ability of DyT on dense prediction tasks. ADE20K contains 20,210 images from 150 fine-trained semantic concepts. COCO-Stuff consists of about 164k images with 172 semantic classes. The experimental settings are demonstrated in Table 16 . All experiments for parameter-efficient fine-tuning methods also follow these settings. For the experiment on full tuning, we set adopt 1/10 learning rate for stable training and better performance. To verify the generalization of the proposed dynamic tuning, we conduct experiments based on Swin-B [53] . The dynamic tuning can be directly applied to MLP blocks in the swin transformer without any modifications. The bottleneck dimension d of the adapter in dynamic tuning also is set to 64. The results are demonstrated in Table 17 . We can observe that the dynamic tuning can reduce both the tunable parameters and FLOPs while achieving comparable or even better performance across three datasets. This verifies the generalization capability of the dynamic tuning. We verify the effectiveness of dynamic tuning when the model size is scaled up. We conduct experiments on ViT-L [20] and compare dynamic tuning with full tuning. The bottleneck dimension d of the adapter in dynamic tuning is also set to 64. The results are demonstrated in Table 18 . We can observe that with the activation rate set to 0.3, DyT has outperformed "full tuning" obviously on both CIFAR-100 and Food-101, while resulting in significantly lower computational cost.We further compare the proposed dynamic tuning with full tuning on the VTAB-1K benchmark [89] . The results are demonstrated in Table 19 . With only 0.44M tunable parameters and 43.52 GFLOPs, dynamic tuning surpasses full tuning across most datasets and achieves much better average performance. We provide more visualizations of activated tokens from samples in K400 [10] and SSv2 [25] in Figure 8 and Figure 9 , respectively. Results demonstrate that most activated tokens in higher layers e.g. Layer10 come from the primary objects. This proves that the proposed token dispatcher learns to activate informative tokens. Justification: See Abstract Guidelines:Layer1 Layer4 Layer7 Layer10 (a) (b) (c) (d) (e) (f) (g) Layer1 Layer4 Layer7 Layer10 (i) (j) (k) (l) (m) (n) (h) (o) (p) (q) (r)(c) (d) (e) (f) (g) Layer1 Layer4 Layer7 Layer10 (i) (j) (k) (l) (m) (n) (h) (o) (p) (q) (r)• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Not release of data or models that have a high risk Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: we cite all related paper. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: See the main paper. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.* There are two linear layers with weights of size 768 × (4 × 768) in a MLP block, results in 2 × 768 × (4 × 768) ≈ 4.7M parameters† The original definition in[35] assumes the input {Ei} N i=1 to be normalized log probability, while practical implementations[65] demonstrate its effectiveness even with unnormalized inputs. We also follow[65] to formulate and implement the Gumbel-Softmax. 
 
 Appendix:  


Decision: The reviewers recognized that the proposed approach is novel and achieves both efficiency and performance. They also appreciated the clarity of the manuscript and extensive experiments. However, at the same time, the reviewers raised concerns with incremental novelty, lack of detailed discussion on experimental results, convergence speed, and some presentation issues. These concerns are successfully addressed by the rebuttal and subsequent responses in the discussion. Consequently, the reviewers unanimously championed the paper after the discussion period. The AC agrees with the reviewers and recommends acceptance. The authors are strongly encouraged to polish the manuscript according to the valuable comments and to add new results and discussions brought up in the responses to the revision.

Reviews: review: paper_summary: paper_summary: This paper proposes Dynamic Tuning, an efficient tuning method to enhance both parameter and inference efficiency for ViT adaptation. The method improves efficiency by dropping some tokens entering pre-trained layers through a token dispatcher while forwarding all tokens to the adapter, maintaining performance. Additionally, the method leverages a Mixture of Experts (MoE) adapter to enhance adaptation performance on certain datasets. The proposed approach achieves comparable or superior performance with significantly fewer FLOPs compared to traditional parameter-efficient transfer learning methods on the VTAB-1K dataset.
strength_weakness: strength_weakness: - The application of MoE in the paper detracts from the overall understanding and value of the proposed method. MoE improves performance only in video classification, not image classification, and is absent from most experiments. Furthermore, Table 6 shows that using MoE results in fewer FLOPs, which is theoretically implausible. The paper lacks an analysis explaining this and only provides experimental results, which hinders comprehension. MoE's inclusion in the main paper is questionable and might be better suited for the appendix.

- The proposed method may appear as a simple combination of dynamic neural networks and adapters. While it differs in learning token dropping during fine-tuning, it essentially drops tokens from the original block while using all tokens in the adapter. The token dispatch mechanism is similar to existing dynamic neural network ideas. The paper should report the performance of DynamicViT and EViT's full fine-tuning and their combination with other PEFT methods in Table 5. If the proposed method performs better, an analysis explaining this phenomenon is necessary.

- There are several instances where reported FLOPs differ significantly from theoretical values, requiring more detailed explanations. For instance, in Table 1, it is unclear why MLP Dispatch has lower FLOPs than Layer Dispatch, given that the attention layer uses all tokens in MLP Dispatch but fewer tokens in Layer Dispatch.

- While the proposed method achieves higher performance despite added constraints compared to existing PEFT methods, the paper lacks explanations or analyses for this improvement. The authors should explain why their method outperforms traditional adapters.
questions: questions: - The authors should consider removing the MoE part from the paper, as it does not significantly contribute to the overall method.

- Discuss the increased training time caused by the need for two forwards (with and without the complete model) during training as a limitation.

- The paper should include an ablation study on the loss function. Given the significant performance differences observed in the distillation ablation study in the appendix, a comprehensive description of the loss function's impact is necessary.
limitations: limitations: The authors have adequately addressed some limitations of their work. However, they did not discuss the increased training time as a limitation. There are no negative societal impacts.
paper_summary: paper_summary: The paper introduces a method called Dynamic Tuning (DyT) designed to enhance both parameter and inference efficiency when adapting Vision Transformers (ViTs) for various visual tasks. DyT employs a token dispatcher to selectively process only the most informative tokens, reducing computational redundancy. Additionally, an enhanced adapter inspired by the mixture-of-experts mechanism is introduced to boost performance with minimal extra cost. The method is comprehensively evaluated across tasks like image and video recognition and semantic segmentation, demonstrating superior performance and significantly reduced FLOPs compared to existing parameter-efficient fine-tuning methods. The results underscore DyT's effectiveness in achieving state-of-the-art performance while being computationally efficient.
strength_weakness: strength_weakness: 1. The authors introduced additional loss functions for DyT, does this affect the speed of convergence?

1. Gumbel-Sigmoid is a prevalent technique for maintaining end-to-end training. However, its computational outcomes may vary compared to Sigmoid. It is valuable to explore the disparities in the calculated results of these two methods.

1. Certain experimental phenomena may become more transparent with additional explanations provided within the text. For instance, **(a)** Image Accuracy in Table 2 demonstrates optimal performance when excluding MoE, and it appears that the performance does not exhibit an increase proportional to the number of experts. **(b)** Meanwhile, for N=12, the FLOPs are paradoxically lower than when N=8. **(c)** In Table 6, the FLOPs of DyT N = 4 is higher than Dynamic-Full, this seems counterintuitive. **(d)** In Table 10, the decrease in FLOPs for DyT is not as significant as in VTAB-1K, and the FLOPs for DyT N=4 is higher than the version not adapted to the MoE. *The reasons behind these observations warrant further exploration*.
questions: questions: 1. I am concerned about the potential performance impact of token pruning on the detection task. Have the authors conducted any experiments in this regard?

2. I came across a paper [1] that dynamically assesses token significance, and it would be beneficial to discuss and cite it in the article.

   


---
   [1] Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysis[C]//CVPR. 2024.
limitations: limitations: The authors have discussed the limitations.
paper_summary: paper_summary: This paper proposes Dynamic Tuning (DyT) to improve both parameter and inference efficiency for ViT adaptation. DyT inserts a token dispatcher for each transformer block to learn to dynamically determine whether a token should be activated or deactivated.
strength_weakness: strength_weakness: I don't see any major weakness in the paper, but I have a minor concern about its novelty. The core technique in Dynamic Tuning seems to be **token pruning** without explicitly reducing the number of tokens, but token pruning itself is not a novel technique. This paper does not well demonstrate the difference between existing dynamic token pruning methods and the proposed method.
questions: questions: See Weakness
limitations: limitations: Yes, the authors discuss limitations in the Discussion and Conclusion section.
paper_summary: paper_summary: This paper proposes DyT, which is a parameter-efficient fine-tuning method that can also achieve inference efficiency. DyT contains token dispatchers, which let tokens dynamically skip the original block, and MoE-adapter which uses multiple adapters to compose a mixture-of-experts. The paper evaluates the proposed method on various vision benchmarks to demonstrate its effectiveness and efficiency.
strength_weakness: strength_weakness: - My main concern is the novelty of the proposed method. DyT mainly includes two innovations: the token dispatcher and the MoE-Adapter. However, to my knowledge, both of these have been explored in earlier works. 
    - First, token dispatcher is very similar to [1], both of them use a linear router to determine which tokens are input into the transformer's block and use parallel adapters to process all tokens. The difference lies in that [1] uses a soft top-k function to generate the mask, whereas this paper employs the sigmoid function.
    - Second, [2] also used MoE as the adapters. The difference is that [2] performs routing separately for the down projection and up projection, whereas in this paper, the two are bound together.
    
- The line spacing is too narrow.


[1] Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference. In NeurIPS 2023.

[2] AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning. In EMNLP 2022.
questions: questions: - Please clarify the novelty of the proposed methods over Conditional Adapters and AdaMix.
- DyT employs a more complex loss function and distillation. Does this increase the training time?
limitations: limitations: The paper has discussed the limitations of DyT.

Comments: comment: Dear Reviewer gv1w,

We sincerely appreciate your insightful feedback and thoughtful suggestions, which have been instrumental in improving the quality of our paper. We will throughly discuss the aforementioned works，including but not limited to DynamicViT, EViT, ToMe, DiffRate, Conditional Adapters, AdaMix, and MoLoRA in the revision. 

Our approach, DyT, contributes a novel exploration of data-dependent computation in vision transformer using a PEFT manner. While our current work focuses on the vision transformer due to computational resource limitations,  we believe this work would be more valuable in this era, when combined with large language models or multi-modal models.


We are more than willing to address any additional questions you may have. Once again, thank you for your support.
 
Best Regards

Authors
Dear Reviewer fmWb,

We sincerely appreciate your support and suggestions! We will include all experimental results and follow the suggestions in the revision to improve our work. 

Our model, DyT, contributes a novel exploration of the data-dependent computation in vision transformer through a PEFT manner. As you have kindly pointed, it primarily focuses on improving the inference efficiency when the device has enough capacity to run the model. Adapting it to different resource scenarios requires training DyT with different FLOPs targets. We believe the comment of a resources-dependent model is quite insightful and promising. This valuable suggestion inspires us to explore a design that has both the advantages of data-dependent and resource-dependent models.

We are more than willing to answer any further questions you may have. Your support is appreciated and helps us improve our work. Thanks again.

Best Regards

Author
Taking into account the discussions between the author and the other reviewers, I have raised the score. However, I would like the author to include a discussion of the aforementioned work in the final version.
Thank you for your clear and concise response. I have reviewed all the reviews and rebuttals, and I am pleased to say that all of my concerns have been fully addressed. While the token pruning and PEFT involing the proposed method are not novel concepts on their own, I acknowledge that the combination of these techniques in the proposed approach is meaningful and has led to interesting and significant experimental results. As such, I will be raising my score.

However, there is one point of concern I would like to highlight. While data-dependent FLOPs can be an advantage in certain scenarios, they may often pose a drawback. In many cases, it is more practical for FLOPs to be adjusted based on available computational resources rather than varying with the data, especially when working within specific resource constraints.
Dear Reviewer fmWb,


Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we summarize our responses here.


> ### 1. [MoE improves performance only in video classification, not image classification, and is absent from most experiments]

**Response:**
- MoE-adapter is primarily designed to enhance the adapter’s capacity to address challenging tasks (Lines 180-183).
- We also validate its effectiveness in semantic segmentation (Appendix Table 10) and object detection (In reponse to Reviewer uJS2).
- We do not employ it in the VTAB-1K experiments as it does not outperform the model without it, likely due to the extremely limited training data


> ### 2. [Table 6 shows that using MoE results in fewer FLOPs, which is theoretically implausible]


**Response:**
- The FLOPs of our models depend on **learned token dispather** during fine-tuning and may slightly **fluctuate around the target FLOPs**. 
- The extra computational cost of the adapter and the MoE adapter is nearly equivalent (Lines 202-204). 


> ### 3. [MoE might be better suited for the appendix]

**Response:**
- In the revision, we will move the introduction of the MoE adapter and its corresponding experiments to the Appendix.


> ### 4.[The token dispatch mechanism is similar to existing dynamic neural network]


**Response:**
- DynamicViT and EViT prune a **certain number** or percentage of tokens in fixed layers, while our token dispatcher **learns the number of tokens to skip** before each MLP block.

- DynamicViT and EViT maintain the **same FLOPs for all samples**, while DyT achieves **data-dependent computation**.

- DyT to preserve **complete feature maps**


> ### 5.[The paper should report the performance of DynamicViT and EViT's full fine-tuning and their combination with other PEFT methods]

**Response:**
- Combining DynamicViT and EViT with AdaptFormer enhances performance.
- DyT outperforms “DynamicViT+AdaptFormer” and “EViT+AdaptFormer” by learning to skip an appropriate number of tokens at each MLP block.


> ### 6.[It is unclear why MLP Dispatch has lower FLOPs than Layer Dispatch]

**Response:**
- The actual token activation rate during inference depends on the **learned token dispatcher**.
-  We set the activation rate $r$ to 0.5 for “Attention Dispatch” and “MLP Dispatch” variants, and to 0.7 for “Attention-MLP Dispatch” and “Layer Dispatch” variants, ensuring **similar** average FLOPs.



> ### 7.[The authors should explain why their method outperforms traditional adapters]

**Response:**

- The dynamic architecture in DyT enhances generalization. 
- The distillation loss in DyT improves performance.
- Dynamic architectures outperforming static models is also observed in previous works.



> ### 8.[Discuss the increased training time caused by the need for two forwards]

**Response:**
- Two forward passes takes 1.8 times longer than only one pass
- Given that our primary contribution focuses on improving inference efficiency, the additional training time would be acceptable.



> ### 9.[The paper should include an ablation study on the loss function]


**Response:**

- Removing $L_{distill}$ or $L^\prime_{cls}$ negatively impacts performance.


Since the discussion stage is already halfway through, may we know if our rebuttal addresses the concerns? 
If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.


Best Regards


Authors
Dear Reviewer Zpyf,

Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we summarize our responses here.

1. **[Difference between existing dynamic token pruning methods and the proposed method]**

**Response:**

- Our model learns where and how many tokens to skip.

- Our model allocates computation in a data-dependent manner. 

- Our model preserves complete feature maps and can handle denste prediction tasks.

- Our model aims to improve efficiency during cross-domain adaption.


Since the discussion stage is already halfway through, may we know if our rebuttal addresses the concerns? 
If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.


Best Regards

Authors
Dear Reviewer uJS2,

Thanks so much for the support. We sincerely appreciate the valuable feedback.

We will include all experimental results and analysis in the revision to facilitate future work. We will clarify the differences with other studies highlighted by reviewers in the revised paper.

We hope our response resolves your concerns. We are more than willing to answer any further questions you may have. Your support is appreciated and helps us improve our work.

Best regards

Authors
I carefully read the author's rebuttal and the comments from other reviewers. The rebuttal addressed most of my concerns. While it’s evident that DyT shares similarities with token pruning methods, its dynamic token selection strategy for PEFT is valid. Most existing PEFT methods focus solely on reducing training costs, often overlooking inference overhead.

I also notice a performance gap between PEFT and FFT in the object detection task in the author's response. I encourage the authors to include relevant tables and analysis in the next version to guide future work. Additionally, the differences with other studies highlighted by reviewers could be addressed in the next version's appendix.

Based on these considerations, I am inclined to accept this paper.
Dear Reviewer gv1w,

Thanks for the reply and for considering raising the score for our work. We will discuss more with other reviewers in the next few days.  We will also sync the results with you. Thanks again for your time and effort in our work.


Best wishes,

Authors
Thank you for your response. I will carefully consider your clarifications, and if other reviewers also find the innovation of the paper to be sufficient during discussion, I will raise my score.
## Part2/2

In addition to distinguishing DyT from existing works, we summarize the contribution of our work below:
- **Verify the feasibility:** The proposed method DyT proposes to learn a data-dependent architecture in a parameter-efficient manner during the ViT adaption. It verifies the feasibility of achieving both parameter and inference in ViT adaptation, which may direct an avenue for efficient fine-tuning and adaptation.
- **Elaborate design:** To achieve better efficiency and performance, DyT explores the optimal placement of token skipping (Section 3.3 Model variants), through extensive analysis and experiments. We also design a MoE-adapter to improve the performane of DyT in challenging tasks.
- **DyT can handle various adaptation tasks:**. DyT achieves both parameter and inference efficiency in various tasks, including image classification (Table3 and Table6), video classification (Table 6), semantic segmentation (Table 10), and object detection & instance segmentation (Our response to Reviewer Reviewer uJS2).
Dear Reviewer gv1w:

We appreciate the reviewer’s feedback and would like to address the points raised below:

## Part1/2

**Compared with conditional adapter**
|Question|Answer
|-|-
|"DyT selects an appropriate number of tokens in each layer, while the conditional adapter selects a fixed number."It is indeed different in this regard.| We appreciate that the reviewer acknowledges this contribution. The soft-topK token dispatcher serves as a core component in 'conditional adapter', which is indeed different from our approach. This verifies the distinction between our approach and 'conditonal adapter'.
|"Conditional adapter is primarily designed for encoder-decoder models, while DyT is for ViT." Although experiments were conducted only on the encoder-decoder model, the conditional adapter is not limited to a specific model architecture, just like LoRA.| The 'conditional adapter' can be applied into ViT but does not work well. In our response, we conduct experiments and apply the 'conditional adapter' to ViT by replacing our token dispatcher with the soft-topK operation. However, DyT outperforms the 'conditional adapter' significantly, emphasizing the significance of our approach in ViT.
|"Conditional adapter skips the whole layer, while DyT skips the MLP blocks." Similarly, although the placement is different, this alone does not provide sufficient novelty unless a theoretical or intuitive motivation for doing so can be provided.|In Vision Transformers (ViT), the self-attention and MLP blocks serve distinct roles in token mixing and feature projection, respectively.However, the impact of token skipping at different blocks on adaptation performance remains unexplored in previous works. Its potential impacts encourage us to explore. In our main paper (Lines 155-178), we analyze four variants, carefully considering their advantages and disadvantages. Additionally, in Figure 6 (Appendix), we conduct experiments to explore this and demonstrate the superiority of the “MLP dispatch” method. These analyses and experiments underscore **the non-trivial nature of the placement of token skipping** and further validate the significance of the exploration in DyT.

**Compared with AdaMix**
|Question|Answer
|-|-
|"AdaMix lacks a learning-based router and fuses experts" There is still a lot of work being done on MoE adapters. For example, MoLoRA [1] uses a learnable router as an adapter and does not fuse experts.| The MoLoRA [1] is adopted to reduce the additional parameters when converting a dense model into a Mixture of Experts (MoE) model. Our MoE-adapter aims to improve the performance of DyT on challenging tasks, while avoiding introducing additional computational cost. Notably, our MoE-adapter adopts weighted summation at both down-projection $W_{down}$ and up-projection $W_{up}$ in adapters, in contrast to MoLoRA [1], which aggregates all LoRA blocks. Hence, our architecture is also different from MoLoRA [1]. While MoE is a widely used and effective technique in previous works, our MoE-adapter offers a unique design and is **tailored to improve the performance of DyT**. Finally, we would like to clarify that we pay more attention to achieving both parameter and inference efficiency for ViT adaptation and the MoE-adapter is **not the main contribution** in our work.

**Compared with DynamicViT and token compression**
|Question|Answer|
|-|-
|"DynamicViT and token compression methods prune a certain number or percentage of tokens in fixed layers, and cannot complete feature maps" This issue existed in early token compression work, but later efforts, such as DiffRate, have addressed it. Combining token compression methods with existing PEFT approaches, such as AdaptFormer and LoRA, could serve as a strong baseline.| In the Diffrate model, token pruning is **inherently data-independent**. After training, a transformer layer **prunes or merges the same number of tokens across all input data**. Additionally, the prune/merge operation **can not preserve complete feature maps**, which poses challenges for dense prediction tasks. In comparison, **DyT is a data-dependent approach**. The router in DyT learns to skip varying numbers of tokens before each MLP block based on the input data. The visualization in Figure 8 and Figure 9 (Appendix) demonstrate that the learned token skipping strategy in DyT **selects different numbers of tokens to skip based on different inputs**, which is more flexible and reasonable. By avoiding prune/merge operations and only conducting token skipping, DyT maintains complete feature maps, enabling it to **effectively handle dense prediction tasks**. Furthermore, in our response to Reviewer fmWb, we also explore combining token compression methods with AdaptFormer together, and find that our method can surpass them obviously, verifying the superiority of designs in DyT.
Thank you for your response.

My initial concerns were primarily about the novelty, as the designs of the token dispatcher and MoE adapter have overlap with existing work. After reading others' opinions, I realized that the mechanism of token dispatcher also bears a resemblance to  Dynamic ViT and token pruning methods. The authors’ response clarified the differences between these works, specifically:

Compared with conditional adapter
+ "DyT selects an appropriate number of tokens in each layer, while the conditional adapter selects a fixed number."
  + It is indeed different in this regard.
+ "Conditional adapter is primarily designed for encoder-decoder models, while DyT is for ViT."
  + Although experiments were conducted only on the encoder-decoder model, the conditional adapter is not limited to a specific model architecture, just like LoRA.
+ "Conditional adapter skips the whole layer, while DyT skips the MLP blocks."
  + Similarly, although the placement is different, this alone does not provide sufficient novelty unless a theoretical or intuitive motivation for doing so can be provided.

Compared with AdaMix
+ "AdaMix lacks a learning-based router and fuses experts"
  + There is still a lot of work being done on MoE adapters. For example, MoLoRA [1] uses a learnable router as an adapter and does not fuse experts.

Compared with DynamicViT and token compression
+ "DynamicViT and token compression methods prune a certain number or percentage of tokens in fixed layers, and cannot complete feature maps"
  + This issue existed in early token compression work, but later efforts, such as DiffRate, have addressed it. Combining token compression methods with existing PEFT approaches, such as AdaptFormer and LoRA, could serve as a strong baseline.

In summary, I appreciate the authors’ response and acknowledge the additional experiments that demonstrate the effectiveness of DyT. However, the issues mentioned above still lead me to believe that DyT lacks sufficient innovation. Therefore, I will keep my score unchanged.


[1] Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning. ICLR 2024

[2] DiffRate : Differentiable Compression Rate for Efficient Vision Transformers. ICCV 2023