Title: BayesTune: Bayesian Sparse Deep Model Fine-tuning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Parameter-efficient Foundation model fine-tuning

Bayesian methods

Stochastic-Gradient MCMC
Abstract: Deep learning practice is increasingly driven by powerful foundation models (FM), pre-trained at scale and then fine-tuned for specific tasks of interest. A key property of this workflow is the efficacy of performing sparse or parameter-efficient fine-tuning, meaning that by updating only a tiny fraction of the whole FM parameters on a downstream task can lead to surprisingly good performance, often even superior to a full model update. However, it is not clear what is the optimal and principled way to select which parameters to update. Although a growing number of sparse fine-tuning ideas have been proposed, they are mostly not  satisfactory, relying on hand-crafted heuristics or heavy approximation. In this paper we propose a novel Bayesian sparse fine-tuning algorithm: we place a (sparse) Laplace prior for each parameter of the FM, with the mean equal to the initial value and the scale parameter having a hyper-prior that encourages small scale. Roughly speaking, the posterior means of the scale parameters indicate how important it is to update the corresponding parameter away from its initial value when solving the downstream task. Given the sparse prior, most scale parameters are small a posteriori, and the few large-valued scale parameters identify those FM parameters that crucially need to be updated away from their initial values. Based on this, we can threshold the scale parameters to decide which parameters to update or freeze, leading to a principled sparse fine-tuning strategy. To efficiently infer the posterior distribution of the scale parameters, we adopt the Langevin MCMC sampler, requiring only two times the complexity of the vanilla SGD. Tested on popular NLP benchmarks as well as the VTAB vision tasks, our approach shows significant improvement over the state-of-the-arts (e.g., 1% point higher than the best SOTA when fine-tuning RoBERTa for GLUE and SuperGLUE benchmarks).

BayesTune: Bayesian Sparse Deep Model Fine-tuning

Abstract

Deep learning practice is increasingly driven by powerful foundation models (FM), pre-trained at scale and then fine-tuned for specific tasks of interest. A key property of this workflow is the efficacy of performing sparse or parameter-efficient finetuning, meaning that by updating only a tiny fraction of the whole FM parameters on a downstream task can lead to surprisingly good performance, often even superior to a full model update. However, it is not clear what is the optimal and principled way to select which parameters to update. Although a growing number of sparse fine-tuning ideas have been proposed, they are mostly not satisfactory, relying on hand-crafted heuristics or heavy approximation. In this paper we propose a novel Bayesian sparse fine-tuning algorithm: we place a (sparse) Laplace prior for each parameter of the FM, with the mean equal to the initial value and the scale parameter having a hyper-prior that encourages small scale. Roughly speaking, the posterior means of the scale parameters indicate how important it is to update the corresponding parameter away from its initial value when solving the downstream task. Given the sparse prior, most scale parameters are small a posteriori, and the few large-valued scale parameters identify those FM parameters that crucially need to be updated away from their initial values. Based on this, we can threshold the scale parameters to decide which parameters to update or freeze, leading to a principled sparse fine-tuning strategy. To efficiently infer the posterior distribution of the scale parameters, we adopt the Langevin MCMC sampler, requiring only two times the complexity of the vanilla SGD. Tested on popular NLP benchmarks as well as the VTAB vision tasks, our approach shows significant improvement over the state-of-the-arts (e.g., 1% point higher than the best SOTA when fine-tuning RoBERTa for GLUE and SuperGLUE benchmarks).

1 Introduction

Practical deep learning increasingly relies on a workflow of fine-tuning pre-trained foundation models for the specific task of interest to be solved. The foundation models (FM)s are often pre-trained at scale using large architectures and self-supervised objectives on large scale unlabelled data, allowing them to encode a great deal of general prior knowledge. Fine-tuning then specialises these general purpose FMs to the specific task of interest. The key challenge then becomes how to conduct fine-tuning so as to balance the conflicting goals of adapting to the task at hand; while retaining the knowledge obtained during foundational pre-training and avoiding overfitting on the (typically smaller) downstream training set. This is addressed from a variety of perspectives from careful learning rate scheduling to regularised fine-tuning [26, 12]. However, the mainstream approach in practice is largely based on some instantiation of sparse fine-tuning.

Sparse fine-tuning, also known as parameter efficient fine-tuning (PEFT) [23, 43, 14, 24, 22, 15, 16, 18, 45, 13, 10], manages the adaptation/forgetting trade-off by selectively updating a specific subset of parameters in the FM while keeping others frozen. These approaches also benefit from good parameter scalability with respect to the number of downstream tasks to solve since each downstream

task only requires storing a small number of parameters compared to the full FM. A large number of sparse/PEFT approaches have now been proposed, based on either additive or mask-based principles. Additive approaches [16, 15] inject a hand-chosen set of additional small modules to be learned, while keeping the original FM entirely frozen; while mask based approaches [43, 18] manually select a sparse subset of original FM parameters to consider as learnable - such as the prompt parameters in transformers [18]. Despite substantial engineering effort, prior sparse fine-tuning work mainly relies on human intuition and ad-hoc heuristics to specify additive modules and sparse fine-tuning masks. This raises the question how can we automatically select which parameters to insert or update in a principled and effective way?

In this paper, we address this issue by proposing an automated sparse fine-tuning method. Our BayesTune framework automatically discovers the parameters to update, with minimal human intervention. In contrast to some early concurrent attempts that rely on heuristics and/or heavy approximations [45, 13, 10], BayesTune is highly principled, underpinned theoretically by Bayesian marginalization of parameter uncertainty. Specifically, we place a sparse Laplace prior on the FM parameters, and the posterior inference of the sparse scale parameters reveals which parameters are most important to update from their initial values. BayesTune is efficiently implemented with Langevin MCMC, which only requires twice the cost of vanilla SGD. We demonstrate BayesTune with both RoBERTa transformer on (Super)GLUE benchmark suite and Vision Transformer on VTAB benchmark suite. BayesTune can achieve a specified sparsity level, and provides a reasonable way to identify a good sparsity/performance trade-off if a sparsity constraint is not pre-defined. Finally, we show that BayesTune can work with both mask-based search spaces to identify existing FM parameters to update by fine-tuning, as well as additive search spaces to identify which new modules to add where for additive adaptation.

2 Problem Setup and Motivation

First we introduce some key notations as follows:

θ = ( Variable ) Parameters (i.e., weights and biases) of the FM

θ = ( Constant ) Parameters (i.e., weights and biases) of the pre-trained FM

D = Training data for the downstream task

We denote by θ i ( ∈ R ) the i -th element of θ by considering θ as a whole vector of concatenated parameters, i.e., θ = [ θ 1 , . . . , θ d ] ⊤ where d is the number of parameters (so, θ i is scalar). Before we start, we clarify the goal of the sparse fine-tuning formally as follows.

̸

Desiderata. We aim to find θ which performs well on the downstream task, and at the same time remains close to θ sparsely in L 0 sense (i.e., || θ -θ || 0 = ∑ d i =1 I ( θ i = θ i ) is small). Roughly speaking, θ i = θ i for many i 's. Oftentimes we define and set the sparsity level p ∈ [0 , 1] , meaning that the proportion of the FM parameters updatable from the pre-trained values is no greater than (100 × p )% (i.e., || θ -θ || 0 ≤⌊ d · p ⌋ ).

About the sparsity level p . We remark here that the sparsity level p is often given by the users, for instance, in case we have a strict pre-specified memory constraint (e.g., embedded platforms), where the allowable extra space for saving the updated new parameters from θ for the downstream task is only (100 × p )% of the original FM storage. In other situations, it would be ideal if the sparse fine-tuning algorithm can estimate optimal p that trades off the downstream accuracy against memory overhead (e.g., find the smallest p that enables a certain level of accuracy guarantee). Unfortunately most existing sparse fine-tuning methods are not capable of this feature, and rather resort to time-consuming cross-validation-type search. Although this latter case opens up a new interesting research problem to pursue further, our proposed approach offers reasonable and principled criteria for estimating the optimal p (See Sec. 3 for details).

How about layer-wise selection? We also note that our approach is not tied to the layer-wise parameter treatment, namely selecting a few layers to be updated with the rest layers frozen, which is a popular strategy adopted in many sparse fine-tuning methods. The layer-wise treatment is based on the conjecture that whether the parameters important to the downstream task are deviated from the pre-trained ones or not, is determined in a layer-wise all-or-nothing manner. Although this is a reasonable assumption, our approach does not rely on the assumption, and is able to discover optimal sparse updates automatically by inspecting individual parameters over all layers.

Back to our notations, the downstream task must be associated with the relevant loss function l ( θ ; z ) with z ∈ D for fine-tuning. Here the task could be either supervised (i.e., z = ( x, y ) for a pair of input x and its target label y ) or unsupervised ( z = x with input x alone). In our model the loss function is turned into a likelihood model by the conventional trick, p ( z | θ ) ∝ exp( -l ( θ ; z )) . Although our approach can deal with both supervised and unsupervised cases seamlessly, we predominantly focus on the supervised classfication case, that is, we define the data likelihood p ( y | x, θ ) ∝ exp( -l ( θ ; x, y )) for ( x, y ) ∈ D , where l ( θ ; x, y ) is typically the cross-entropy loss.

3 (Proposed) Bayesian Sparse Fine-tuning Model

We propose a (hierarchical) Bayesian model to tackle the downstream prediction task of interest. Specifically, we treat the FM parameters θ as random variables and the downstream training data D as evidence . To encourage parameter sparsity, namely most parameters θ i remain at the pre-trained θ i , we impose the Laplace prior p ( θ | λ ) ,

where L ( x ; µ, b )= 1 2 b exp( -| x -µ | /b ) is the (univariate) Laplace distribution with mean µ and scale b . Thus we fix the prior means as the pre-trained values θ , and only model the scale parameters which are newly introduced random variates denoted by λ ( > 0 ). Note that every single parameter θ i (scalar) is associated with its own prior scale λ i (thus d scale variables in total). Another thing to note is that the final layer (also known as the readout head) of the FM is often completely replaced by a random one for the downstream task. Since this final layer needs to be learned from the scratch with the downstream data, we do not place the Laplace prior on the parameters of the readout head.

Through the scale variables λ of the Laplace distributions, we can express our (prior) preference to the degree of parameter deviation from the pre-trained values: small λ i leads to a peaky Laplace around its mean, penalizing even small deviation of θ i from θ i ; on the other hand, large λ i makes it flat, being less sensitive to deviation, allowing θ i to take values freely away from θ i . By carefully choosing λ i values, we can balance effectively between overfitting (too much deviation) and underfitting (too little deviation). Instead of choosing λ i 's manually, our idea is to learn them automatically from data, through the principled Bayesian inference: given the evidence (i.e., the downstream data D ) we infer the most probable values of λ i 's that best explain the evidence, while being small enough to lead to sparse updates overall.

To this end, we regard λ as random variables and impose hyper-prior on λ (hierarchical Bayes). As we prefer sparse updates in the end, the hyper-prior needs to put higher mass/density on small λ values. We adopt the Gamma distribution for this purpose, more specifically,

where G ( x ; α, β ) ∝ x α -1 exp( -βx ) for α, β > 0 . Note that although we can introduce individual λ i -specific parameters (i.e., ( α i , β i ) ), we instead use a single ( α, β ) shared over all λ i 's for simplicity. Since the mode of Gamma is 0 if α < 1 and the variance is α/β 2 , we use small α < 1 and large β to enforce a priori small λ values. In our experiments we choose 1 ( α =0 . 01 , β =100) for both the NLP and VTAB vision benchmarks.

Our full model can be written as the following joint distribution:

With this model, our ultimate goal is to infer the posterior distribution of the network weights and scales p ( θ, λ | D ) , namely

We adopt the stochastic-gradient MCMC approach [40, 7, 4] to obtain samples from the posterior, especially the Langevin dynamic method (SGLD) [40], which amounts to running the following recurrence to collect posterior samples (after some burn-in steps):

where B ( ⊂ D ) is a minibatch, η is small step size, and ϵ ∼N (0 , I ) . In standard SGLD, ˆ N = | D | (training data size) and γ = 1 . However, we allow them to take values different from these default ones: ˆ N is the effective training data size to account for data augmentation 2 (e.g., if there are 5 different augmentation strategies and each allows 10 different variations, the effective data size can be ˆ N =10 5 ×| D | ); the other hyperparameter γ can be used to discount the noise effect 3 , which is validated in the range 10 -4:0 in our experiments. See Sec. 5.1 for more details. Note that in the parentheses subject to derivatives, the first two terms admit closed-form gradients while the gradient of the last term can be computed by the conventional SGD backprop. Thus each step in (5) is efficient, requiring at most only two times the complexity of the vanilla SGD step.

After a burn-in period, we can maintain those samples ( θ, λ ) to approximate p ( θ, λ | D ) . For instance, the running average of the λ samples, denoted by ˆ λ , is a good estimate of the mean of the marginal posterior p ( λ | D ) . Since we imposed the hyper-prior that encourages small λ , ideally the majority of the scale parameters tend to remain small a posteriori , whereas there will be only a few largevalued ˆ λ i 's corresponding to those FM parameters that crucially need to be updated away from their pre-trained values to explain the data D . Roughly speaking, ˆ λ i s indicates how important it is for the corresponding FM parameters θ i s to be updated away from their pre-trained values θ i on the downstream data. Importantly, this ˆ λ can be used in the next stage (described in the next paragraph) to decide which network weights θ i need to be frozen (those with small ˆ λ i s) and which need to be updated from the pre-trained (those with large ˆ λ i s).

Thresholding scales and the second stage. Although we adopted the sparsity-inducing Laplace prior, it is not necessarily the case that the majority of posterior θ values are sharply staying at the pre-trained values θ . We may need some thresholding: If p is given, then we can directly take the top (100 × p )% of them. Otherwise we examine for cut-off point - Those i 's with small posterior means ˆ λ i are considered as frozen weights (i.e., θ i = θ i ), while those i 's with large ˆ λ i s can be treated as updatable parameters. In practice, we sort and plot the ˆ λ i s to eyeball and find a reasonable cut-off point. One can also use this ˆ λ -plot to decide a reasonable sparsity level p for good accuracy-memory trade-off (See Fig. 1 for illustration). Once we have decided which parameters are updated and frozen, we can run vanilla SGD to train the updatable parameters of the FM, which forms our second stage.

Our overall algorithmm dubbed BayesTune , is summarized as pseudocodes in Alg. 1.

4 Related Work

4.1 Comparison to Existing Sparse Fine-tuning Methods

The recent sparse/PEFT fine-tuning approaches broadly fall into the following three categories: 1) Heuristic search criteria, 2) Attaching small extra modules to the FM (e.g., adapter-based [15]), and 3) Directly formulating the sparse fine-tuning problem as an optimization problem.

1) Heuristic search criteria.

Random selection - Randomly select (100 × p )% parameters to update.

Sorted ; vs. proportion in FM

Testerrorvs.sparsity

18

1.6

BayesTune

Candidatecut-offpoints

1.4

Fullupdate

Test error (%)

16

Head only

1.2

Cand-1

Cand-1

Sorted

1.0

( = 0.3,p = 0.05)

14

( =0.3,p = 0.05)

0.8

Cand-2

Cand-2

0.6

( = 0.1,p = 0.35)

12

( = 0.1,p = 0.35)

0.4

10

0.2

0.0

8

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Proportionind (#-FM-parameters)

Proportionind (#-FM-parameters)

Algorithm 1 BayesTune (two-stage) sparse fine-tuning algorithm.

Input: Pre-trained model θ , downstream data D . Initialise: θ = θ , λ i = 0 . 0001 for all i . I. Stage-1 (Select (100 × p )% parameters to update) For t = 0 , 1 , 2 , . . . , T max: 0. Sample a minibatch B = { ( x, y ) } from D . 1. Run the SGLD step (5) to get a sample ( θ, λ ) . 2. If t > burn-in, accumulate ˆ λ ← (1 -ζ ) ˆ λ + ζλ where ζ = 1 / ( #-accumulated-samples+1). End-For (If p not given) Decide the cut-off point λ th and the corresponding sparsity level p using the ˆ λ -plot. Select those i 's with ˆ λ i > λ th as updatable parameters.

II. Stage-2 (Update the selected parameters)

Do vanilla SGD with data D while updating only those updatable parameters from Stage-1.

MixOut [23] - After training the FM model with full update, a random (100 × (1 -p ))% parameters are reset to the pre-trained values.

BitFit [43] - Update only the bias parameters with the rest frozen.

MagPruning [14, 24, 22] - Take the largest (100 × p )% in magnitude as updatable parameters.

Compared to our proposed 'BayesTune', these heuristic search methods are less principled, unable to explain why the resulting models should perform well on the downstream tasks

Approaches that attach small extra modules to FM. They do not search for a subset of FM parameters to update, but attach some small extra modules to the FM (e.g., insert a small MLP module after each layer). All of the FM parameters are frozen, and they only train those attached modules. So it can be seen as sparse fine-tuning considering that the size of the new modules is relatively small compared to that of the FM.

Adapter [15] - A small bottleneck MLP is inserted at the end of each layer of the FM.

LoRA [16] - A low-rank projection weight matrix is inserted before the query/key/value embeddings in each layer.

Prompt learning (VPT) [18] - A set of learnable tokens is inserted in the input of a Transformer attention block in each layer.

Neural Prompt Search [45] - A NAS-based approach (e.g., learning a supernet followed by evolutionary search for a downstream tasks specific subnet) that aims to search for the optimal combination of the above three methods.

Compared to our 'BayesTune', these adapter-based models involve the difficult problem of where to place the extra modules and what would be the optimal module dimensions/sizes, which is essentially a very difficult combinatorial search problem, often requiring heuristic search methods like the evolutionary search.

3) Directly optimizing the sparse optimization problem. One can formulate the sparse fine-tuning as an optimization problem, for instance,

Unfortunately this problem becomes a discrete optimization due to the L 0 cardinality constraints, thus difficult to solve in general. Some recent approaches aimed to relax the problem to continuous optimization, but relying on heavy approximation.

DiffPruning [13] - They relax the discrete problem into a continuous one by introducing Bernoulli variables indicating update or freeze, while the reparametrized sampling is further approximated by the Gumbel-softmax treatment.

ChildPruning [41, 31] - They iteratively train the full model parameters and calculate the projected mask to find the child network.

Second-order approximation method (SAM) [10] - They adopt several steps of secondorder Taylor approximation of the loss function, leading to a very succinct parameter selection strategy: after computing the gradients of θ on the data D , take the (100 × p )% parameters with the largest gradient magnitudes.

Compared to our 'BayesTune', these direct optimization strategies rely on strong assumptions and heavy approximation to relax the difficult discrete optimization.

4.2 Relation to Existing (Hierarchical) Bayesian Sparse Model Learning

(Hierarchical) Bayesian models are well known in statistical machine learning, and there were some prior works that adopt the Bayesian approaches for sparse deep learning [21, 32, 35, 29]. However, as far as we know, our approach has several key differences from these previous works.

First, most prior works are all about sparse training, instead of sparse fine-tuning . Thus they focus on zeroing out many parameters, instead of retaining the pre-trained weights. To the best of our knowledge, our BayesTune is the first to tackle the sparse fine-tuning at the Foundation Model scale using the hierarchical Bayesian framework.

More importantly, the neural networks used in those previous studies are rather small/toy scale (mostly focusing on MLPs and LeNet-sized architectures up to ResNet-18 at the largest) while our method can obtain the state-of-the-art results on large-scale foundation models (ViT, RoBERTa). For example the largest model considered in those prior works, ResNet-18 contains ∼ 11 Mparameters vs. RoBERTa's ∼ 123 Mparameters, a 10 × scale difference. The reason is that they adopt methods that entail extra memory cost like variational inference (VI), which impedes applicability to big networks. We have also done some experiments that compare the computational resources required by VI and SGLD: On ViT networks, the training time is increased by 1.7 times if we replace SGLD by VI; the GPU memory footprint is increased by 2.1 times.

5 Experiments

We test our BayesTune on two popular benchmark datasets from NLP and vision for the downstream fine-tuning tasks: ( language ) fine-tuning the pre-trained RoBERTa-base model [28] on the GLUE [37] and SuperGLUE [38] tasks; ( vision ) fine-tuning the ImageNet-22K [6] pre-trained ViT-B/16 model [9] on VTAB-1K [44] image classification/prediction tasks. The details of the experimental settings are discussed in the subsequent sections, Sec. 5.2 and Sec. 5.3.

5.1 Implementation of BayesTune

For the SGLD sampling in Stage-1 of our BayesTune algorithm (re: Alg. 1), we have three different training regimes: the warm-up phase that only runs vanilla SGD steps without considering the scale

variables λ and random noise; followed by the burn-in phase where the SGLD steps are performed as (5) but no ( θ , λ ) samples are collected; followed by the normal phase in which we do collect samples. Following the conventional practice, we also perform the thinning steps [27] (i.e., collecting samples at a certain frequency) to mitigate undesired temporal correlation effects.

Another important implementation tip is the use of reweighed cost terms in Langevin dynamics - As shown in (5), the cost function is involved with the training data size N = | D | . Even though N is known, the data augmentation would considerably increase the effective data size by several orders of magnitudes (e.g., if there are 5 different augmentation strategies and each allows 10 different variations, the effective data size can be 10 5 × N ). To account for it, we regard the effective data size as a hyperparameter denoted by ˆ N that can be chosen from validation 4 . Replacing | D | by ˆ N and taking it out of the parentheses in (5), we have the gradient (normalized by ˆ N ) multiplied by the step size ( η/ 2) ˆ N . First we choose η to match the learning rate in the neural net optimizer (e.g., Adam 5 [20]), that is, η = lr × 2 / ˆ N . Accordingly the noise term becomes ϵ · (2 / ( ˆ N × lr )) 1 / 2 . We also find it effective oftentimes to discount the noise effect: we introduce the noise discount factor γ (another hyperparameter) by which the noise term is multiplied. We validate γ in the range 10 -4:0 (e.g., γ = 1 corresponds to the default no-discount case).

Throughout all experiments we use the Gamma prior parameters ( α =0 . 01 , β =100) , the scale variables λ i 's are all initialized to 0 . 0001 , and the learning rate for λ is 0 . 01 without scheduling. Other task-specific implementation details can be found in the subsequent sections. The Python/PyTorch code to reproduce the results is available at https://github.com/SamsungLabs/BayesTune 6 .

5.2 NLP Benchmarks

We consider fine-tuning the pre-trained RoBERTa-base model [28], the large-scale foundation language model comprised of 125 million parameters, on several downstream tasks in GLUE [37] and SuperGLUE [38] benchmarks. We follow the experimental settings from [10], in which the original development sets serve as test sets, and the validation sets are formed by holding out random 10% of

0.28

0.300

0.30

cola

0.26

stsb

0.275

0.250

mrpc

2.0

2.5

rte

0.25

0.24

0.225

1.5

0.20

0.22

0.200

1.0

0.25

0.20

0.20

0.000.020.040.060.08

0.10

0.3

0.2

0.175

0.000.020.040.060.08

80.10

2.5

2.0

0.5

0.00

90000200

0.08

0.10

0.3

0.2

0.15

0.000.020.040.060.080.10

<0.15

“< 1.5

0.10

0.19

1.0

0.1

0.05

0.5

0.0

0.2

Sparsity level p

0.4

0.6

0.8

1.0

0.0

00

0.20.40.6

Sparsity level p

0.8

1.0

0'0

0.2

0.4

Sparsity level p

0.6

0.8

1.0

0.0

0°0

0.20.4

Sparsity level p

0.6

0.8

1.0

0.3000

cb

0.28

copa

0.295

WSC

0.2975

0.26

0.290

0.2950

0.24

0.285

0.2925

0.3

0.22

0.30

0.280

HE'O

0.2900

0.20

0.000.020.040.060.080.10

0.25

0.000.020.04 0.060.080.10

0.000.020.040.060.080.10

0.2

0.20

0.2

<0.15

0.1

0.10

0.19

0.0

0.05

00

0.0

0.2

0.40.6

Sparsity level p

80

1.0

0.0

0.2

0.4

Sparsity level p

0.6

0.8

1.0

0.0

0.2

Sparsity level p

0.4

0.8

1.0

the training sets. Instead of having a fixed number of training epochs, the development sets are used for an early stopping with the no-improvement tolerance of 40 epochs. We implement our BayesTune on top of the codebase of [10] which is based on the jiant framework ( https://jiant.info/ ). Similarly as [10], we consider 7 tasks: Corpus of Linguistic Acceptability ( CoLA ) [39], Semantic Textual Similarity Benchmark ( STS-B ) [2], Microsoft Research Paraphrase Corpus ( MRPC ) [8], Recognizing Textual Entailment ( RTE ) [5, 11, 1], Commitment Bank ( CB ) [30], Choice of Plausible Alternatives ( COPA ) [36], and Winograd Schema Challenge ( WSC ) [25].

Our BayesTune is compared against strong baselines and the state-of-the-arts, including: Random , Mixout [23], BitFit [43], MagPruning [24, 22], Adapter [15], LoRA [16], DiffPruning , ChildPruning [41, 31], and SAM [10], as discussed in Sec. 4. For all methods, we use the sparsity level p = 0 . 005 as suggested in [13]. For the competing methods, we follow the hyperparameter settings from [10]; In BayeTune, we use 10 K warm-up steps, 2 K burn-in steps, and thinning at every 100 steps for all tasks. The batch size is 16, and the learning rate for the model parameters is 10 -4 for Stage-1 and 10 -3 for Stage-2. For the (dataset-dependent) evaluation metrics, we adopt the protocols in [37, 38]. We run all models on a single NVIDIA V100 GPU with 32GB memory. The results are summarized in Table 1 where means and standard deviations over 10 random seeds are reported.

First, it is quite surprising that the random selection strategies ( Random and Mixout ) are as effective as many sparse fine-tuning methods. This may be attributed to the massive over-parametrization of the deep foundation model. However, more careful and sophisticated selection is important as those three methods, our BayesTune , SAM , and Adapter , outperform the random strategies by large margin. Overall our BayesTune performs the best among the competing methods, about 1% point higher on average than the runner-up SAM [10]. We achieve the first place on 5 out of 7 tasks, and also attain the smallest standard deviations, implying that the resulting sparse fine-tuned models are more robust and reliable. This result signifies the effectiveness of our sparse Laplace scale-based parameter selection strategy that is underpinned by the principled Bayesian posterior inference with efficient 7 stochastic gradient Langevin sampling.

In Fig. 2, we also show the posterior mean scale ˆ λ vs. sparsity level p . The cut-off threshold points corresponding to the sparsity level p =0 . 005 are around λ th ∈ [0 . 25 , 0 . 30] for all tasks except for MRPC . This cut-off scale range might be considered as reasonable choice as most of the parameters would have Laplace scales smaller than this range, exhibiting no pronounced indication of crucial deviation from the pre-trained values, although this is only conjecture based on our empirical results. Further rigorous theoretical analysis on the relation between the scale threshold and the generalization performance is beyond the scope of the current paper, and is left as future work.

5.3 Vision Benchmarks

Next we test our BayesTune on the VTAB-1K [44], an image classification/prediction task adaptation benchmark suite comprised of 19 different image datasets. It contains images/tasks that exhibit

highly diverse aspects/conditions such as: different image acquisition (by standard cameras or special-purpose ones for remote sensing or medical imaging), different objects/concepts (generic, fine-grained, or abstract), and tasks (object recognition, counting, or pose estimation). Each dataset in VTAB-1K consists of 1 K training examples, and we use the splits officially provided (train 80% and validation 20% ).

We aim to fine-tune the ImageNet-22K [6] pre-trained ViT-B/16 model [9] on each dataset. We implement our BayesTune using the codebase from [45] while employing most of their hyperparameters without changes (e.g., the number of training epochs 100 and batch size 64). In our BayesTune, we use 50 warm-up epochs, 10 burn-in epochs, and thinning at every 5 batch steps for all datasets. The learning rate for the model parameters is 10 -3 and weight decay 10 -3 for both stages.

We first attempted the proposed method with our original setting, i.e., place the λ variables to the backbone (ViT) parameters, and let the Bayesian inference figure out the most critical parameters to update, namely those with large λ i 's. Unfortunately the result was significantly worse than the adapter-based methods: VPT [18], Adapter [15], LoRA [16], and NOAH [45], for most datasets. As reported in several works in the literature, adapter-based models perform significantly better than the full backbone update for this benchmark (e.g., Table 1 in [45]). Along this line, we conjecture that this trend also holds for other sparse fine-tuning methods, namely the superiority of the adapter-based approaches to sparse backbone update strategies, particularly for this VTAB benchmark.

To fairly take the advantage of extra attachable modules as the adapter-based approaches, we attach extra modules to ViT for all three popular strategies: visual prompts in VPT [18], adapter modules in Adapter [15], and low-rank matrices in LoRA [16]. This design idea is similar to (and motivated from) the neural prompt search in NOAH [45], but they formed a grid search problem to determine which layers to attach the adapter modules and how many module dimensions to attach - solved by the neural architecture search (NAS) technique with the super-net training, specifically adopting Autoformer [3], followed by running Evolutionary algorithms for optimal architecture search.

On the other hand, we add 10-dim full parameters for each module/layer, and they are all initialized to 0. We let the posterior inference in our BayesTune automatically figure out which of these extra modules parameters to be updated away from 0 and which to be frozen at 0. We used the zeroinitialization because having all-0 extra modules can be considered to be identical to the original pre-trained ViT without extra modules , thus serving as pre-trained extra modules. We only place λ variables to the extra modules (ViT parameters frozen) 8 . In summary, our setup is similar to NOAH[45] in that we take all three adapter strategies into account, but the main difference is: NOAH does a difficult super-net training and Evolutionary search to determine the optimal module-wise dimensions. On the other hand we do sparse selection of the module parameters to be updated from 0 across all 10-dim full modules.

(a) Sparse update is better: Sun397 , CIFAR100 , DTD , and Retinopathy

sun397 (natural)

cifar100 (natural)

dtd (natural)

retinopathy (specialized)

Sparse group

50

698 7 6 65

95% confidence interval

(Normalized) Mean

蓝.6-

44

.05 .12.34.5.6.7.8.9 1.0

63

.05.123 .4.5.6.7.8.9 1.0

.05.1.2.3 .4.5.6.7.8.9 1.0

05 123 45.6.78.9 1.0

.05 .1.2±3.4.5.6.7.8.9 1.0

Sparsity p

Sparsity p

Sparsity p

Sparsity p

Sparsity p

dmlab (structured)

camelyon (specialized)

svhn (natural)

dsprite-loc (structured)

Dense group

86

80.0

0.6

75.0

80

72.5

(Normalized) Mean

78

84

70.0

95% confidence interval

.05.1.23.4.5.6.7.8.9 1.0

.05.1.2.3.4.5.6.7.8.9 1.0

.05.12.34.5.6.7.8.9 1.0

.05.1.2.3.4.5.6.7.8.9 1.0

.05.1.2.3.4.5.6.7.8.9 1.0

Sparsity p

Sparsity p

Sparsity p

Sparsity p

Sparsity p

clevr-dist (structured)

snorb-ele (structured)

kitti (structured)

dsprite-ori (structured)

In-between group

40

38

0.6

36

45

0.4

62

34

32

44

UON)

0.0

0.2

95% confidence interval

(Normalized) Mean

.05.12.3.4.5.6.7.8.9 1.0

.05.1.23.4.5.6.7.8.9 1.0

.05 1.2± 3 .4.5.6 .7.8.9 1.0

.05.1.2.34.5.6.7.8.9 1.0

.05.1234.5.6.7.8.9 1.0

Sparsity p

Sparsity p

Sparsity p

Sparsity p

Sparsity p

flower102 (natural)

eurosat (specialized)

6

pets (natural)

Independent group

(Normalized) Mean

Test accuracy (%)

95% confidence interval

.05 .1 2 3 4.5.6.7.8.9 1.0

.05 .1 23 4.5.6.7.8.9 1.0

87

.05.1.2.3.4.5.6.78.91.0

S8

05.123 4.5.6.78 9 1.0

.05 .1 2 3 45.6.7 8.9 1.0

Sparsity p

Sparsity p

Sparsity p

Sparsity p

Sparsity p

The results are summarized in Table 2. We report the performance of the best sparsity levels found from p ∈ { 0 . 05 , 0 . 1 , 0 . 2 , . . . , 0 . 9 , 1 . 0 } . Although the final accuracies of our BayesTune are not always the best, we achieve Rank 1 on 7 out of 19 datasets and the second place on average over all datasets. The performance of BayesTune is comparable to other adapter-based methods, within the same range as those. That is, our main message is that the adapter-based enhanced architecture finetuning can be done almost comparably well by the proposed Bayesian method, instead of relying on the heuristic search such as the Evolutionary Search or costly human expert design effort. Compared to NOAH, our BayesTune results in more sparse adapter updates on average (0.38M vs. 0.43M). In Fig. 3 we also visualize the clustering of VTAB-1K datasets according to the sparse/dense adapter update effects: we assign each dataset into either of sparse , dense , and in-between clusters according to the optimal sparse update level p : small, large, medium, respectively. Such a grouping of the VTAB datasets based on fine-tuning sparsity characteristics has not been explored in the community, and we can easily obtain this finding using our BayesTune.

6 Conclusion

We introduced BayesTune, a framework for automated fine-tuning that provides a principled yet efficient approach to identifying which parameters to update in sparse fine-tuning by posterior inference of parameter-wise scale in a hierarchical Bayesian model. We demonstrated BayesTune to be effective for parameter efficient fine-tuning in both NLP and vision tasks, using both mask-based and additive search spaces.

References

Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC , 2009.

Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv::1708.00055 , 2017.

Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In ICCV , 2021.

T. Chen, E. B. Fox, and C. Guestrin. Stochastic gradient Hamiltonian Monte Carlo. International Conference on Machine Learning , 2014.

Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop , 2005.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2009.

N. Ding, Y. Fang, R. Babbush, C. Chen, R. D. Skeel, and H. Neven. Bayesian sampling using stochastic gradient thermostats. In Advances in neural information processing systems , 2014.

Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005) , 2005.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR , 2021.

Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. On the Effectiveness of Parameter-Efficient Fine-Tuning. AAAI , 2023.

Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing , 2007.

Henry Gouk, Timothy M Hospedales, and Massimiliano Pontil. Distance-Based Regularisation of Deep Networks for Fine-Tuning. In ICLR , 2021.

Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In ACL , 2021.

Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. ICLR , 2016.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning , 2019.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. ICLR , 2022.

Pavel Izmailov, Sharad Vikram, Matthew D. Hoffman, and Andrew Gordon Wilson. What are Bayesian neural network posteriors really like? International Conference on Machine Learning , 2021.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV , 2022.

Sehwan Kim, Qifan Song, and Faming Liang. Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Drifts. arXiv preprint arXiv:2009.09535 , 2020.

D. P. Kingma and L. J. Ba. Adam: A Method for Stochastic Optimization. In ICLR , 2015.

Insung Kong, Dongyoon Yang, Jongjin Lee, Ilsang Ohn, Gyuseung Baek, and Yongdai Kim. Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference. International Conference on Machine Learning , 2023.

François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M. Rush. Block pruning for faster transformers. In Empirical Methods in Natural Language Processing , 2021.

Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In International Conference on Learning Representations , 2019.

Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-based pruning. In International Conference on Learning Representations , 2021.

Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning , 2012.

Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In ICML , 2018.

W. A. Link and M. J. Eaton. Methods in ecology and evolution. On thinning of chains in MCMC , 3(1): 112-115, 2012.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692 , 2019.

Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learning. In Advances in neural information processing systems , 2017.

Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23 , 2019.

Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. International Conference on Machine Learning , 2019.

Eric Nalisnick, Jose Miguel Hernandez-Lobato, and Padhraic Smyth. Dropout as a structured shrinkage prior. International Conference on Machine Learning , 2019.

Lorenzo Noci, Kevin Roth, Gregor Bachmann, Sebastian Nowozin, and Thomas Hofmann. Disentangling the roles of curation, data-augmentation and the prior in the cold posterior effect. In Advances in neural information processing systems , 2021.

Kazuki Osawa, Siddharth Swaroop, Anirudh Jain, Runa Eschenhagen, Richard E. Turner, Rio Yokota, and Mohammad Emtiyaz Khan. Practical deep learning with Bayesian principles. In Advances in neural information processing systems , 2019.

Nicholas Polson and Veronika Rockova. Posterior concentration for sparse deep learning. In Advances in neural information processing systems , 2018.

Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning , 2011.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , 2018.

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in neural information processing systems , 2019.

Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics , 7:625-641, 2019.

Max Welling and Yee Whye Teh. Bayesian Learning via Stochastic Gradient Langevin Dynamics. In International Conference on Machine Learning , 2011.

Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise a child in large language model: Towards effective and generalizable finetuning. Empirical Methods in Natural Language Processing , 2021.

L. I. Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. International Conference on Machine Learning , 2018.

Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. ACL , 2022.

Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867 , 2019.

Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural Prompt Search. arXiv preprint arXiv:2206.04673 , 2022.

A Chosen Hyperparameters

We grid-search hyperparameters on validation, where the two key hyperparameters are: the effective training data size ˆ N and the noise discount factor γ (re: Sec. 5.1). The candidate sets are formed as: ˆ N ∈ { 10 8 , 10 9 , · · · , 10 12 } , γ ∈ { 10 -4 , 10 -2 , 10 0 } for NLP, and ˆ N ∈ { 10 6 , 10 7 , · · · , 10 12 } , γ ∈ { 10 -4 , 10 -3 , · · · , 10 0 } for VTAB. The chosen hyperparameters are as follows ( ˆ N,γ ) : (NLP) cola = (11 , 10 -4 ) , stsb = (12 , 10 -4 ) , mrpc = (12 , 10 0 ) , rte = (8 , 10 -4 ) , cb = (10 , 10 -4 ) , copa = (8 , 10 -2 ) , wsc = (10 , 10 -4 ) ; (VTAB) cifar100 = (7 , 10 -1 ) , caltech101 = (9 , 10 -2 ) , dtd = (12 , 10 0 ) , flower102 = (12 , 10 -2 ) , pets = (12 , 10 0 ) , svhn = (10 , 10 0 ) , sun397 = (7 , 10 -1 ) , camelyon = (6 , 10 0 ) , eurosat = (7 , 10 -1 ) , resisc45 = (12 , 10 -2 ) , retinopathy = (7 , 10 -2 ) , clevr-count = (7 , 10 -3 ) , clevr-dist = (7 , 10 -3 ) , dmlab = (8 , 10 0 ) , kitti = (7 , 10 0 ) , dsprite-loc = (12 , 10 -4 ) , dsprite-ori = (12 , 10 -3 ) , snorbazim = (7 , 10 -2 ) , snorb-ele = (6 , 10 -1 ) .

B More Analysis

(NLP) Test accuracies at other sparsity levels. Although p = 0 . 005 is recognized as the optimal sparsity level overall for the GLUE and SuperGLUE tasks, we evaluate the test performance of our BayesTune for different sparsity levels: p ∈ { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 5 } . The average test accuracies are shown in Fig. 4. We see that overall there is less significant change in test performance so long as the sparsity level p is small enough, and the resulting sparse updates selected by our BayesTune lead to equally good performance as those with the default value. However, increasing p further (e.g., p = 0 . 5 ) considerably degrades the performance, which signifies the importance of sparse fine-tuning to avoid potential overfitting.

(VTAB) Scale posterior mean ˆ λ vs. sparsity level p . We visualize the plots that relate the sorted scale posterior means ˆ λ to the sparsity levels p in Fig. 5. The plots are aligned with the the test accuracy plots analyzed in the main paper. As the plots are grouped along the optimal sparsity values, we see certain trends: for the sparse group ( sun397 and cifar100 ), the scale ˆ λ values are overall small scaled (in the range of [0 , 0 . 2] ) with sharp drops at small ˆ λ ; for the dense group : ( camelyon and dmlab ), ˆ λ scale is even larger (in the range of [0 , 0 . 5] ) with relatively smooth decaying at small values; lastly for the in-between group ( clever-dist , dspr-ori , kitti , and snorb-ele ), we have much narrower ˆ λ ranges in between 0 and 0.1 except for kitti .

C Some Ablation Study

MAPvs. SGLD. The Maximum-A-Posterior (MAP) aims to find a mode of the posterior distribution p ( λ | D ) , and might be considered as an alternative solution to our SGLD posterior samples. In particular, MAP might be attained by turning off the noise term in (5), that is, by setting γ = 0 . However, MAP can be more sensitive to data noise than the mean of the posterior. We have some empirical comparison between MAP and SGLD on the NLP tasks in Table 3, and it shows that this distinction does lead to empirical benefit. Another benefit of SGLD

Test accuracy (%)

62

cola

Test accuracy (%)

90.4

stsb

Test accuracy (%)

90.50

mrpc

Test accuracy (%)

79

rte

90.3

90.25

6

90.2

90.00

76

59

90.1

89.75

75

0.005

100

0.05

0.005

100

0.05

0.5

0.005

T0'0

0.05

S00°0

10'0

S0'0

2

Sparsity level p

Sparsity level p

Sparsity level p

Sparsity level p

cb

copa

WSC

Test accuracy (%)

92

Test accuracy (%)

74

Test accuracy (%)

60

90

88

70

86

68

56

0.005

10'0

S0'0

2

3

0.005

T00

S0'0

2

0.005

10'0

$00

Sparsity level p

Sparsity level p

Sparsity level p

Appendix

(a) Sparse group ( cifar100 and sun397 )

(b) Dense group ( camelyon and dmlab )

cifar100(Sparsegroup)

sun397(Sparsegroup)

69

50

Test accuracy (%)

68

Test accuracy (%)

67

48

66

65

46

64

63

.05.1.2.3.4.5.6.7.8.91.0

44

.05.1.2.3.4.5.6.7.8.91.0

0.20

0.20

0.15

0.15

0.10

0.10

0.05

0.05

0.00

0.00

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Sparsity level p

Sparsity level p

camelyon(Densegroup)

dmlab (Dense group)

84

49

Test accuracy (%)

8

Test accuracy (%)

48

1828

47

46

80

45

79

44

78

.05.1.2

.3

.4

.5

.6

.7

.8

.91.0

43

.05.1.2.3.4.5.6.7.8.91.0

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1

0.0

0.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Sparsity level p

Sparsity level p

(c) In-between group ( clever-dist , dspr-ori , kitti , and snorb-ele )

clevr-dist(In-between)

kitti(In-between)

snorb-ele(In-between)

89

49

83

40

Test accuracy (%)

66

Test accuracy (%)

47

48

Test accuracy (%)

18

82

Test accuracy (

80

36

64

79

44

78

34

62

43

77

32

.05.1.2.3.4.5.6.7.8.91.0

.05.1.2.3.4.5.6.7.8.91.0

.05.1.2.3.4.5.6.7

8

.91.0

.05.1.2.3.4.5.6.7.8.9 1.0

0.100

0.100

0.4

0.100

0.075

0.075

0.3

0.075

0.050

→

0.050

0.2

0.050

0.025

0.025

0.1

0.025

0.000

0.000

0.0

0.000

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Sparsity level p

Sparsity level p

Sparsity level p

Sparsity level p

is that we can also exploit the variance of the posterior in parameter selection (e.g., for two parameters with similar posterior mean values, we prefer to select the one with smaller posterior variance). In future algorithms, we can also exploit this idea of variance-based weight pruning.

Comparison to L1 regulariser. The recent L1-regularised sparse model learning (L1-SP) [42] used the L1 regularisation term to encourage the updated weights to stay close to the original pre-trained weights. Thus this model might have a similar effect as our BayesTune. The L1-SP can provide an alternative regularisation-based approach to fine-tuning, however, is designed without the principled Bayesian modeling solution, which is the main difference from the proposed BayesTune. We have conducted some extra comparison experiments between the two approaches. Note that L1-SP does not necessarily lead to exactly sparse solutions. Therefore we offer a two-stage extension of L1-SP. in the first stage, we run the L1-SP training, and in the second stage those weights to be updated are selected based on the relative L1 distances from the pre-trained weights (i,e., taking those

(100 × p )% weights with the largest relative changes from the pre-trained weights). The results on the NLP tasks are shown in Table 4.

For the L1 penalty balancing constant hyperparameter, we choose optimal values by grid search from { 10 -3 , 10 -4 , 10 -5 , 10 -6 } . We can see that L1-SP considerably lags behind our BayesTune, which is mainly attributed to its failure to capture uncertainty in L1 regularisation, thus being potentially sensitive to noise in data (a similar reason as the MAP estimate). Moreover, only penalising the parameters deviation from the pre-trained weights as in Stage 1 significantly underperforms the sparse cut-off strategy in Stage 2, signifying that sparse update is critical.

D Layer-wise and Module-wise Sparsity Patterns of BayesTuned Networks

Sparsity patterns of RoBERTa-base on NLP tasks. We visualize the module-wise and layer-wise sparsity patterns of the BayesTuned RoBERTa-base networks on 7 NLP tasks in Fig. 6-19. First, for the layer-wise sparsity pattern: (Except for mrpc ) The proportions of the selected updatable parameters are more or less uniformly distributed across the 12 layers of the Transformer, while the first word embedding layer and the last classification layer are significantly less and more selected, respectively. This is intuitively appealing as the task-specific features may tend to be determined at the higher, more global levels in texts/sentences, to account for longer-range dependency. Next, looking at the module-wise sparsity patterns, the proportions are highly non-uniform, layer-specific, and also task/dataset-dependent. For instance, the bias modules in some layers are very densely selected, while they are very sparsely selected in other layers. This shows clear discrepancy to the heuristic strategies like BitFit [43] in which the bias modules are selected 100% for all layers.

Sparsity patterns of ViT-B/16 on VTAB vision tasks. The module-wise and layer-wise sparsity patterns of the BayesTuned ViT-B/16 networks on VTAB benchmark datasets are shown in Fig. 20-38. We also superimpose the optimal p values (dataset dependent). The resulting patterns are quite similar to the NLP case: Except for a few cases, the lowest level visual prompt layers are selected far less, sometimes ignored, compared to the later layers. The last linear classification head, although not shown here in the sparsity diagrams, is selected 100% . Overall the layer-wise selection patterns are nearly uniform while the module-wise selection patterns are highly non-uniform and dataset dependent.

Name

Module

cola ( Module-wise sparsity pattern)

embeddings.wordembeddings.weight

embeddings.position_embeddings.weight-

encoder.layer.6.attention.self.query.bias

embeddings.token_type_embeddings.weight

encoder.layer.6.attention.self.key.weight

embeddings.LayerNorm.weight

encoder.layer.6.attention.self.key.bias

embeddings.LayerNorm.bias

encoder.layer.0.attention.self.query.weight

encoder.layer.6.attention.self.value.bias

encoder.layer.0.attention.self.query.bias

encoder.layer.6.attention.output.dense.weight

encoder.layer.0.attention.self.key.weight

encoder.layer.6.attention.output.dense.bias

encoder.layer.0.attention.self.key.bias

encoder.layer.6.attention.output.LayerNorm.weight

encoder.layer..attention.self.value.weigh

encoder.layer.6.attention.output.LayerNorm.bias

encoder.layer.6.intermediate.dense.bias

encoder.layer.0.attention.output.dense.bias

encoder.layer.6.output.dense.weight

encoder.layer.0.attention.output.LayerNorm.weight

encoder.layer.6.output.dense.bias

encoder.layer.0.attention.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.bias

encoder.layer.7.attention.self.query.weight

encoder.layer.6.output.LayerNorm.bias

encoder.layer.0.output.dense.weight

encoder.layer.7.attention.self.query.bias

encoder.layer.0.output.dense.bias

encoder.layer.7.attention.self.key.weight

encoder.layer.0.output.LayerNorm.weight

encoder.layer.7.attention.self.key.bias

encoder.layer.0.output.LayerNorm.bias

encoder.layer.7.attention.self.value.weight

encoder.layer.7.attention.self.value.bias

encoder.layer.1attention.elfquerybias

encoder.layer.1.attention.self.key.weight

encoder.layer.7.attention.output.dense.bias

encoder.layer.1.attention.self.key.bias

encoder.layer.7.attention.output.LayerNorm.weight

encoder.layer.1.attention.self.value.weight

encoder.layer.7.attention.output.LayerNorm.bias

encoder.layer.1.attention.output.dense.weight

encoder.layer.1.attention.self.value.bias

encoder.layer.7.intermediate.dense.bias

encoder.layer.1.attention.output.dense.bias

encoder.layer.7.output.dense.weight

encoder.layer.1.attention.output.LayerNorm.weight

encoder.layer.7.output.dense.bias

encoder.layer.1.attention.output.LayerNorm.bias

encoder.layer.1.intermediate.dense.weight

encoder.layer.2.attention.self.query.weight

encoder.layer.1.output.LayerNorm.weight

encoder.layer.1.intermediate.dense.bias

encoder.layer.1.output.LayerNorm.bias

encoder.layer.1.output.dense.weight

encoder.layer.1.output.dense.bias

encoder.layer.8.attention.self.value.weight

encoder.layer.7.output.LayerNorm.weight

encoder.layer.8.attention.self.query.bias

encoder.layer.8.attention.self.key.weight

encoder.layer.8.attention.self.value.bias

encoder.layer.7.output.LayerNorm.bias

encoder.layer.8.attention.self.key.bias

encoder.layer.2.attention.self.query.bias

encoder.layer.8.attention.output.dense.weight

encoder.layer.2.attention.self.key.weight

encoder.layer.2.attention.output.LayerNorm.weight

encoder.layer.2.attention.output.dense.bias

encoder.layer.2.attention.self.value.weight

encoder.layer.2.attention.self.value.bias

encoder.layer.2.attention.self.key.bias

encoder.layer.8.attention.output.LayerNorm.bias

encoder.layer.8.intermediate.dense.weight

encoder.layer.8.output.LayerNorm.weight

encoder.layer.8.intermediate.dense.bias

encoder.layer.8.output.dense.weight

encoder.layer.8.output.dense.bias

encoder.layer.2.attention.output.LayerNorm.bias

encoder.layer.2.intermediate.dense.weight

encoder.layer.8.output.LayerNorm.bias

encoder.layer.2.output.LayerNorm.weight-

encoder.layer.2.intermediate.dense.bias

encoder.layer.3.attention.self.query.bias

encoder.layer.3.attention.self.key.weight

encoder.layer.2.output.LayerNorm.bias

encoder.layer.2.output.dense.weight

encoder.layer.2.output.dense.bias

encoder.layer.9.attention.output.dense.weight

encoder.layer.9.attention.output.dense.bias

encoder.layer.9.attention.self.value.weight

encoder.layer.9.attention.self.value.bias

encoder.layer.3.attention.output.LayerNorm.weight

encoder.layer.3.attention.output.dense.bias

encoder.layer.3.attention.self.value.weight

encoder.layer.3.attention.self.value.bias

encoder.layer.3.attention.self.key.bias

encoder.layer.9.attention.output.LayerNorm.bias

encoder.layer.9.intermediate.dense.bias

encoder.layer.9.output.dense.weight

encoder.layer.9.output.dense.bias

encoder.layer.3.attention.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.weight

encoder.layer.9.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.bias

encoder.layer.3.output.dense.weight

encoder.layer.10.attention.self.query.bias

encoder.layer.3.output.dense.bias

encoder.layer.10.attention.self.value.weight

encoder.layer.10.attention.self.key.bias

encoder.layer.3.output.LayerNorm.bias

encoder.layer.10.attention.self.value.bias

encoder.layer.4.attention.self.query.bias

encoder.layer.4.attention.self.key.bias

encoder.layer.10.attention.output.LayerNorm.weight

encoder.layer.10.attention.output.LayerNorm.bias

encoder.layer.10.attention.output.dense.weight

encoder.layer.10.attention.output.dense.bias

encoder.layer.4.attention.self.value.weight

encoder.layer.10.intermediate.dense.weight

encoder.layer.4.attention.self.value.bias

encoder.layer.10.intermediate.dense.bias

encoder.layer.4.attention.output.dense.bias

encoder.layer.10.output.dense.bias

encoder.layer.10.output.LayerNorm.weight

encoder.layer.4.attention.output.LayerNorm.bias

encoder.layer.10.output.LayerNorm.bias

encoder.layer.4.intermediate.dense.weight

encoder.layer.4.intermediate.dense.bias

encoder.layer.4.output.dense.weight

encoder.layer.11.attention.self.key.weight

encoder.layer.4.output.dense.bias

encoder.layer.11.attention.self.key.bias

encoder.layer.4.output.LayerNorm.weight

encoder.layer.11.attention.self.value.weight

encoder.layer.4.output.LayerNorm.bias

encoder.layer.11.attention.self.value.bias

encoder.layer.11.attention.output.dense.weight

encoder.layer.5.attention.self.query.bias

encoder.layer.11.attention.output.dense.bias

encoder.layer.5.attention.self.key.weight

encoder.layer.5.attention.self.key.bias

encoder.layer.11.attention.output.LayerNorm.bias

encoder.layer.11.intermediate.dense.weight

encoder.layer.5.attention.self.value.bias

encoder.layer.11.intermediate.dense.bias

encoder.layer.11.output.dense.weight

encoder.layer.5.attention.output.dense.bias

encoder.layer.11.output.dense.bias

encoder.layer.11.output.LayerNorm.weight

encoder.layer.5.intermediate.dense.weight

encoder.layer.11.output.LayerNorm.bias

pooler.dense.weight

encoder.layer.5.intermediate.dense.bias

pooler.dense.bias

encoder.layer.5.output.dense.weight

taskmodels_dict.cola.head.dense.weight

encoder.layer.5.output.dense.bias

taskmodels_dict.cola.head.dense.bias

encoder.layer.5.output.LayerNorm.weight

taskmodels_dict.cola.head.out_proj.weight

encoder.layer.5.output.LayerNorm.bias

taskmodels_dict.cola.head.out_proj.bias

0

2

3

4

0

2

Updatable Parameters (%) in Modul

Updatable Parameters (%) in Modu

Name

Module

cola ( Layer-wise sparsity pattern)

embeddings.wordembeddings.weight

embeddings.position_embeddings.weight

encoder.layer.6.attention.self.query.bias

embeddings.token_type_embeddings.weight

embeddings.LayerNorm.weight

encoder.layer.6.attention.self.key.bias

embeddings.LayerNorm.bias

encoder.layer.0.attention.self.query.weight

encoder.layer.6.attention.self.value.bias

encoder.layer.0.attention.selfquery.bias

encoder.layer.6.attention.output.dense.weight

encoder.layer.0.attention.self.key.weight

encoder.layer.6.attention.output.dense.bias

encoder.layer.0.attention.self.key.bias

encoder.layer.6.attention.output.LayerNorm.weight

encoder.layer..attention.selfvalue.weig

encoder.layer.6.attention.output.LayerNorm.bias

encoder.layer.6.intermediate.dense.weight

encoder.layer.6.intermediate.dense.bias

encoder.layer.6.output.dense.weight

encoder.layer.0.attention.output.LayerNorm.weight

encoder.layer.6.output.dense.bias

encoder.layer.0.attention.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.weight

encoder.layer.6.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.bias

encoder.layer.0.output.dense.weight

encoder.layer.7.attention.self.query.bias

encoder.layer.0.output.dense.bias

encoder.layer.7.attention.self.key.weight

encoder.layer.0.output.LayerNorm.weight

encoder.layer.7.attention.self.key.bias

encoder.layer.7.attention.self.value.weight

encoder.layer.7.attention.self.value.bias

encoder.layer.1.attention.self.query.bias

encoder.layer.7.attention.output.LayerNorm.weight

encoder.layer.7.attention.output.dense.bias

encoder.layer.1.attention.self.value.weight

encoder.layer.7.attention.output.LayerNorm.bias

encoder.layer.1.attention.output.dense.weight

encoder.layer.1.attention.self.value.bias

encoder.layer.7.intermediate.dense.bias

encoder.layer.1.attention.output.dense.bias

encoder.layer.7.output.dense.weight

encoder.layer.1.attention.output.LayerNorm.weight

encoder.layer.7.output.dense.bias

encoder.layer.1.attention.output.LayerNorm.bias

encoder.layer.1.intermediate.dense.weight

encoder.layer.1.output.LayerNorm.weight

encoder.layer.1.intermediate.dense.bias

encoder.layer.1.output.dense.weight

encoder.layer.1.output.dense.bias

encoder.layer.8.attention.self.key.weight

encoder.layer.8.attention.self.query.bias

encoder.layer.7.output.LayerNorm.bias

encoder.layer.8.attention.self.key.bias

encoder.layer.2.attention.self.query.bias

encoder.layer.1.output.LayerNorm.bias

encoder.layer.8.attention.output.dense.weight

encoder.layer.8.attention.self.value.weight

encoder.layer.8.attention.self.value.bias

encoder.layer.2.attention.self.key.weight

encoder.layer.2.attention.self.key.bias

encoder.layer.2.attention.output.LayerNorm.weight

encoder.layer.2.attention.output.dense.weight

encoder.layer.2.attention.output.dense.bias

encoder.layer.2.attention.self.value.weight

encoder.layer.2.attention.self.value.bias

encoder.layer.8.attention.output.LayerNorm.bias

encoder.layer.8.intermediate.dense.weight

encoder.layer.8.output.LayerNorm.weight

encoder.layer.8.intermediate.dense.bias

encoder.layer.8.output.dense.weight

encoder.layer.8.output.dense.bias

encoder.layer.2.attention.output.LayerNorm.bias

encoder.layer.2.intermediate.dense.weight

encoder.layer.2.output.LayerNorm.weight

encoder.layer.2.intermediate.dense.bias

encoder.layer.3.attention.self.query.bias

encoder.layer.2.output.LayerNorm.bias

encoder.layer.2.output.dense.weight

encoder.layer.2.output.dense.bias

encoder.layer.9.attention.output.LayerNorm.weight

encoder.layer.9.attention.output.dense.weight

encoder.layer.9.attention.output.dense.bias

encoder.layer.9.attention.self.value.weight

encoder.layer.9.attention.self.value.bias

encoder.layer.8.output.LayerNorm.bias

encoder.layer.9.attention.self.key.bias

encoder.layer.3.attention.output.LayerNorm.weight

encoder.layer.3.attention.output.dense.bias

encoder.layer.3.attention.self.value.weight

encoder.layer.3.attention.self.value.bias

encoder.layer.3.attention.self.key.bias

encoder.layer.9.attention.output.LayerNorm.bias

encoder.layer.9.intermediate.dense.weight

encoder.layer.9.intermediate.dense.bias

encoder.layer.9.output.dense.weight

encoder.layer.9.output.dense.bias

encoder.layer.3.attention.output.LayerNorm.bias

encoder.layer.9.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.bias

encoder.layer.10.attention.self.query.bias

encoder.layer.3.output.dense.weight

encoder.layer.3.output.dense.bias

encoder.layer.10.attention.self.value.weight

encoder.layer.10.attention.self.key.bias

encoder.layer.3.output.LayerNorm.bias

encoder.layer.10.attention.self.value.bias

encoder.layer.4.attention.self.value.weight

encoder.layer.4.attention.self.query.bias

encoder.layer.4.attention.self.key.bias

encoder.layer.10.attention.output.LayerNorm.weight

encoder.layer.10.attention.output.LayerNorm.bias

encoder.layer.10.attention.output.dense.weight

encoder.layer.10.attention.output.dense.bias

encoder.layer.10.intermediate.dense.weight

encoder.layer.4.attention.self.value.bias

encoder.layer.10.intermediate.dense.bias

encoder.layer.10.output.dense.bias

encoder.layer.10.output.LayerNorm.weight

encoder.layer.4.attention.output.LayerNorm.bias

encoder.layer.10.output.LayerNorm.bias

encoder.layer.4.intermediate.dense.weight

encoder.layer.4.intermediate.dense.bias

encoder.layer.11.attention.self.query.bias

encoder.layer.4.output.dense.weight

encoder.layer.11.attention.self.key.weight

encoder.layer.4.output.dense.bias

encoder.layer.11.attention.self.key.bias

encoder.layer.4.output.LayerNorm.weight

encoder.layer.11.attention.self.value.weight

encoder.layer.4.output.LayerNorm.bias

encoder.layer.11.attention.self.value.bias

encoder.layer.11.attention.output.dense.weight

encoder.layer.5.attention.self.query.bias

encoder.layer.11.attention.output.dense.bias

encoder.layer.5.attention.self.key.weight

encoder.layer.11.attention.output.LayerNorm.weight

encoder.layer.5.attention.self.key.bias

encoder.layer.11.attention.output.LayerNorm.bias

encoder.layer.5.attention.self.value.weight

encoder.layer.11.intermediate.dense.weight

encoder.layer.5.attention.self.value.bias

encoder.layer.11.intermediate.dense.bias

encoder.layer.5.attention.output.dense.weight

encoder.layer.11.output.dense.weight

encoder.layer.5.attention.output.dense.bias

encoder.layer.11.output.dense.bias

encoder.layer.11.output.LayerNorm.weight

encoder.layer.5.attention.output.LayerNorm.bias

encoder.layer.11.output.LayerNorm.bias

encoder.layer.5.intermediate.dense.weight

pooler.dense.weight

encoder.layer.5.intermediate.dense.bias

pooler.dense.bias

encoder.layer.5.output.dense.weight

taskmodels_dict.cola.head.dense.weight

encoder.layer.5.output.dense.bias

taskmodels_dict.cola.head.dense.bias

encoder.layer.5.output.LayerNorm.weight

taskmodels_dict.cola.head.out_proj.weight

encoder.layer.5.output.LayerNorm.bias

taskmodels_dict.cola.head.out_proj.bias

0.00.51

1.0

1.5 2.0 2.5

0

Updatable Parameters (%) in Layer

Updatable Parameters (%) in Laye

eName

Module

stsb ( Module-wise sparsity pattern)

embeddings.word_embeddings.weight -

embeddings.position_embeddings.weight

encoder.layer.6.attention.self.query.bias

embeddings.token_type_embeddings.weight

encoder.layer.6.attention.self.key.weight

embeddings.LayerNorm.weight

embeddings.LayerNorm.bias

encoder.layer.0.attention.self.query.weight

encoder.layer.6.attention.self.value.bias

encoder.layer.0.attention.self.query.bias

encoder.layer.6.attention.output.dense.weight

encoder.layer.0.attention.self.key.weight

encoder.layer.6.attention.output.dense.bias

encoder.layer.0.attention.self.key.bias

encoder.layer.6.attention.output.LayerNorm.weight

encoder.layer.0.attention.self.value.weight

encoder.layer.6.attention.output.LayerNorm.bias

encoder.layer.0.attention.output.dense.weight

encoder.layer.6.intermediate.dense.bias

encoder.layer.0.attention.output.dense.bias

encoder.layer.6.output.dense.weight

encoder.layer.0.attention.output.LayerNorm.weight

encoder.layer.6.output.dense.bias

encoder.layer.0.attention.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.bias

encoder.layer.6.output.LayerNorm.bias

encoder.layer.0.output.dense.weight

encoder.layer.7.attention.self.query.bias

encoder.layer.0.output.LayerNorm.weight

encoder.layer.0.output.dense.bias

encoder.layer.7.attention.self.key.weight

encoder.layer.0.output.LayerNorm.bias

encoder.layer.7.attention.self.value.weight

encoder.layer.7.attention.self.value.bias

encoder.layer.1.attention.self.query.bias

encoder.layer.1.attention.self.key.bias

encoder.layer.7.attention.output.LayerNorm.weight

encoder.layer.7.attention.output.dense.bias-

encoder.layer.1.attention.self.value.weight

encoder.layer.7.attention.output.LayerNorm.bias

encoder.layer.1.attention.output.dense.weight

encoder.layer.1.attention.self.value.bias

encoder.layer.7.intermediate.dense.bias

encoder.layer.1.attention.output.dense.bias

encoder.layer.7.output.dense.weight

encoder.layer.1.attention.output.LayerNorm.weight

encoder.layer.7.output.dense.bias

encoder.layer.1.attention.output.LayerNorm.bias

encoder.layer.1.intermediate.dense.weight

encoder.layer.1.output.LayerNorm.weight

encoder.layer.1.intermediate.dense.bias

encoder.layer.1.output.dense.weight

encoder.layer.1.output.dense.bias

encoder.layer.7.output.LayerNorm.weight

encoder.layer.8.attention.self.query.bias

encoder.layer.8.attention.self.key.weight

encoder.layer.7.output.LayerNorm.bias

encoder.layer.8.attention.self.key.bias

encoder.layer.2.attention.self.query.bias

encoder.layer.1.output.LayerNorm.bias

encoder.layer.8.attention.self.value.weight

encoder.layer.8.attention.self.value.bias

encoder.layer.2.attention.self.key.weight

encoder.layer.8.attention.output.dense.bias

encoder.layer.2.attention.output.LayerNorm.weight

encoder.layer.2.attention.output.dense.weight-

encoder.layer.2.attention.output.dense.bias

encoder.layer.2.attention.self.value.weight

encoder.layer.2.attention.self.value.bias

encoder.layer.2.attention.self.key.bias

encoder.layer.8.attention.output.LayerNorm.bias

encoder.layer.8.intermediate.dense.weight

encoder.layer.8.output.LayerNorm.weight

encoder.layer.8.intermediate.dense.bias

encoder.layer.8.output.dense.weight

encoder.layer.8.output.dense.bias

encoder.layer.2.attention.output.LayerNorm.bias

encoder.layer.2.intermediate.dense.weight

encoder.layer.8.output.LayerNorm.bias

encoder.layer.3.attention.output.LayerNorm.weight

encoder.layer.3.attention.output.dense.bias

encoder.layer.3.attention.self.value.weight

encoder.layer.2.output.LayerNorm.weight

encoder.layer.3.attention.self.value.bias

encoder.layer.3.attention.self.key.weight

encoder.layer.2.intermediate.dense.bias

encoder.layer.3.attention.self.query.bias

encoder.layer.2.output.LayerNorm.bias

encoder.layer.3.attention.self.key.bias

encoder.layer.2.output.dense.weight

encoder.layer.2.output.dense.bias

encoder.layer.9.attention.output.LayerNorm.weight

encoder.layer.9.attention.output.LayerNorm.bias

encoder.layer.9.attention.output.dense.weight

encoder.layer.9.attention.output.dense.bias

encoder.layer.9.intermediate.dense.weight

encoder.layer.9.attention.self.value.weight

encoder.layer.9.attention.self.value.bias

encoder.layer.9.intermediate.dense.bias

encoder.layer.9.attention.self.key.bias

encoder.layer.9.output.dense.weight

encoder.layer.9.output.dense.bias

encoder.layer.3.attention.output.LayerNorm.bias

encoder.layer.9.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.bias

encoder.layer.3.output.dense.weight

encoder.layer.10.attention.self.query.bias

encoder.layer.3.output.dense.bias

encoder.layer.10.attention.self.value.weight-

encoder.layer.10.attention.self.key.bias

encoder.layer.3.output.LayerNorm.bias

encoder.layer.10.attention.self.value.bias

encoder.layer.4.attention.self.query.weight

encoder.layer.4.attention.self.key.weight

encoder.layer.4.attention.self.query.bias

encoder.layer.10.attention.output.LayerNorm.weight

encoder.layer.10.attention.output.dense.weight-

encoder.layer.10.attention.output.dense.bias

encoder.layer.4.attention.self.key.bias

encoder.layer.10.attention.output.LayerNorm.bias-

encoder.layer.4.attention.self.value.weight

encoder.layer.10.intermediate.dense.weight

encoder.layer.4.attention.self.value.bias

encoder.layer.10.intermediate.dense.bias-

encoder.layer.4.attention.output.dense.bias

encoder.layer.10.output.dense.bias-

encoder.layer.10.output.LayerNorm.weight

encoder.layer.4.attention.output.LayerNorm.bias

encoder.layer.10.output.LayerNorm.bias

encoder.layer.4.intermediate.dense.weight

encoder.layer.4.intermediate.dense.bias

encoder.layer.11.attention.self.query.bias

encoder.layer.4.output.dense.weight

encoder.layer.11.attention.self.key.weight

encoder.layer.4.output.dense.bias

encoder.layer.11.attention.self.key.bias

encoder.layer.4.output.LayerNorm.weight

encoder.layer.11.attention.self.value.weight

encoder.layer.4.output.LayerNorm.bias

encoder.layer.11.attention.self.value.bias

encoder.layer.11.attention.output.dense.weight

encoder.layer.5.attention.self.query.bias

encoder.layer.11.attention.output.dense.bias

encoder.layer.5.attention.self.key.bias

encoder.layer.11.attention.output.LayerNorm.bias

encoder.layer.11.intermediate.dense.weight

encoder.layer.5.attention.self.value.bias

encoder.layer.11.intermediate.dense.bias

encoder.layer.5.attention.output.dense.weight

encoder.layer.11.output.dense.weight

encoder.layer.5.attention.output.dense.bias

encoder.layer.11.output.dense.bias

encoder.layer.11.output.LayerNorm.weight

encoder.layer.11.output.LayerNorm.bias

encoder.layer.5.intermediate.dense.weight

pooler.dense.weight

encoder.layer.5.intermediate.dense.bias

pooler.dense.bias

encoder.layer.5.output.dense.weight

taskmodels_dict.stsb.head.dense.weight

encoder.layer.5.output.dense.bias

taskmodels_dict.stsb.head.dense.bias

encoder.layer.5.output.LayerNorm.weight

taskmodels_dict.stsb.head.out_proj.weight

encoder.layer.5.output.LayerNorm.bias

taskmodels_dict.stsb.head.out_proj.bias

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

Updatable Parameters (%) in Modul

Updatable Parameters (%) in Modul

Name

Module

stsb ( Layer-wise sparsity pattern)

embeddings.wordembeddings.weight

embeddings.position_embeddings.weight

encoder.layer.6.attention.selfquery.bias

embeddings.token_type_embeddings.weight

embeddings.LayerNorm.weight

encoder.layer.6.attention.self.key.bias

embeddings.LayerNorm.bias

encoder.layer.0.attention.self.query.weight

encoder.layer.6.attention.self.value.bias

encoder.layer.0.attention.selfquery.bias

encoder.layer.6.attention.output.dense.weight

encoder.layer.0.attention.self.key.weight

encoder.layer.6.attention.output.dense.bias

encoder.layer.0.attention.self.key.bias

encoder.layer.6.attention.output.LayerNorm.weight

encoder.layer.0.attention.self.value.weight

encoder.layer.6.attention.output.LayerNorm.bias

encoder.layer.6.intermediate.dense.weight

encoder.layer.6.intermediate.dense.bias

encoder.layer.6.output.dense.weight

encoder.layer.0.attention.output.LayerNorm.weight

encoder.layer.6.output.dense.bias

encoder.layer.0.attention.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.weight

encoder.layer.6.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.bias

encoder.layer.0.output.dense.weight

encoder.layer.7.attention.self.query.bias

encoder.layer.0.output.dense.bias

encoder.layer.7.attention.self.key.weight

encoder.layer.0.output.LayerNorm.weight

encoder.layer.7.attention.self.key.bias

encoder.layer.7.attention.self.value.weight

encoder.layer.7.attention.self.value.bias

encoder.layer.1.attention.self.query.bias

encoder.layer.1.attention.self.key.weight

encoder.layer.1.attention.self.key.bias

encoder.layer.7.attention.output.LayerNorm.weight

encoder.layer.7.attention.output.dense.bias

encoder.layer.1.attention.self.value.weight

encoder.layer.7.attention.output.LayerNorm.bias

encoder.layer.1.attention.output.dense.weight

encoder.layer.1.attention.self.value.bias

encoder.layer.7.intermediate.dense.weight

encoder.layer.7.intermediate.dense.bias

encoder.layer.1.attention.output.dense.bias

encoder.layer.7.output.dense.weight

encoder.layer.1.attention.output.LayerNorm.weight

encoder.layer.7.output.dense.bias

encoder.layer.1.attention.output.LayerNorm.bias

encoder.layer.1.intermediate.dense.weight

encoder.layer.1.output.LayerNorm.weight

encoder.layer.1.intermediate.dense.bias

encoder.layer.1.output.dense.weight

encoder.layer.1.output.dense.bias

encoder.layer.8.attention.self.key.weight

encoder.layer.8.attention.self.query.bias

encoder.layer.7.output.LayerNorm.bias

encoder.layer.8.attention.self.key.bias

encoder.layer.2.attention.self.query.weight

encoder.layer.1.output.LayerNorm.bias

encoder.layer.8.attention.self.value.weight

encoder.layer.8.attention.self.value.bias

encoder.layer.2.attention.self.key.weight

encoder.layer.2.attention.self.query.bias

encoder.layer.8.attention.output.dense.weight

encoder.layer.2.attention.output.LayerNorm.weight

encoder.layer.2.attention.output.dense.bias

encoder.layer.2.attention.self.value.weight

encoder.layer.2.attention.self.value.bias

encoder.layer.2.attention.self.key.bias

encoder.layer.8.attention.output.LayerNorm.bias

encoder.layer.8.intermediate.dense.weight

encoder.layer.8.output.LayerNorm.weight

encoder.layer.8.intermediate.dense.bias

encoder.layer.8.output.dense.weight

encoder.layer.8.output.dense.bias

encoder.layer.2.attention.output.LayerNorm.bias

encoder.layer.2.intermediate.dense.weight

encoder.layer.2.output.LayerNorm.weight

encoder.layer.2.intermediate.dense.bias

encoder.layer.3.attention.self.query.bias

encoder.layer.2.output.LayerNorm.bias

encoder.layer.2.output.dense.weight

encoder.layer.2.output.dense.bias

encoder.layer.9.attention.output.dense.weight

encoder.layer.9.attention.output.dense.bias

encoder.layer.9.attention.self.value.weight

encoder.layer.9.attention.self.value.bias

encoder.layer.8.output.LayerNorm.bias

encoder.layer.9.attention.self.key.bias

encoder.layer.3.attention.output.LayerNorm.weight

encoder.layer.3.attention.output.dense.bias

encoder.layer.3.attention.self.value.weight

encoder.layer.3.attention.self.value.bias

encoder.layer.3.attention.self.key.bias

encoder.layer.9.attention.output.LayerNorm.bias

encoder.layer.9.intermediate.dense.bias

encoder.layer.9.output.dense.weight

encoder.layer.9.output.dense.bias

encoder.layer.3.attention.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.weight

encoder.layer.9.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.bias

encoder.layer.10.attention.self.query.bias

encoder.layer.3.output.dense.weight

encoder.layer.3.output.dense.bias

encoder.layer.10.attention.self.value.weight

encoder.layer.10.attention.self.key.bias

encoder.layer.3.output.LayerNorm.bias

encoder.layer.10.attention.self.value.bias

encoder.layer.4.attention.self.query.bias

encoder.layer.4.attention.self.key.bias

encoder.layer.10.attention.output.LayerNorm.weight

encoder.layer.10.attention.output.LayerNorm.bias

encoder.layer.4.attention.self.value.weight

encoder.layer.10.intermediate.dense.weight

encoder.layer.4.attention.self.value.bias

encoder.layer.10.intermediate.dense.bias

encoder.layer.10.output.dense.weight

encoder.layer.10.output.dense.bias-

encoder.layer.10.output.LayerNorm.weight

encoder.layer.4.attention.output.LayerNorm.bias

encoder.layer.10.output.LayerNorm.bias

encoder.layer.4.intermediate.dense.weight

encoder.layer.4.intermediate.dense.bias

encoder.layer.11.attention.self.query.bias

encoder.layer.4.output.dense.weight

encoder.layer.11.attention.selfkey.weight

encoder.layer.4.output.dense.bias

encoder.layer.11.attention.self.key.bias

encoder.layer.4.output.LayerNorm.weight

encoder.layer.11.attention.self.value.weight

encoder.layer.4.output.LayerNorm.bias

encoder.layer.11.attention.self.value.bias

encoder.layer.11.attention.output.dense.weight

encoder.layer.5.attention.self.query.bias

encoder.layer.11.attention.output.dense.bias

encoder.layer.5.attention.self.key.weight

encoder.layer.11.attention.output.LayerNorm.weight

encoder.layer.5.attention.self.key.bias

encoder.layer.11.attention.output.LayerNorm.bias

encoder.layer.5.attention.self.value.weight

encoder.layer.11.intermediate.dense.weight

encoder.layer.5.attention.self.value.bias

encoder.layer.11.intermediate.dense.bias

encoder.layer.11.output.dense.weight

encoder.layer.5.attention.output.dense.bias

encoder.layer.11.output.dense.bias

encoder.layer.5.attention.output.LayerNorm.bias

encoder.layer.11.output.LayerNorm.bias

encoder.layer.5.intermediate.dense.weight

pooler.dense.weight

encoder.layer.5.intermediate.dense.bias

pooler.dense.bias

encoder.layer.5.output.dense.weight

taskmodels_dict.stsb.head.dense.weight

encoder.layer.5.output.dense.bias

taskmodels_dict.stsb.head.dense.bias

encoder.layer.5.output.LayerNorm.weight

taskmodels_dict.stsb.head.out_proj.weight

encoder.layer.5.output.LayerNorm.bias

taskmodels_dict.stsb.head.out_proj.bias

0.0

0.5

1.0

1.5

0.0

0.5

1.0

1.5

Updatable Parameters (%) in Layer

Updatable Parameters (%) in Laye

Name

Module

mrpc ( Module-wise sparsity pattern)

embeddings.word_embeddings.weight -

embeddings.position_embeddings.weight

encoder.layer.6.attention.self.query.bias

embeddings.token_type_embeddings.weight

encoder.layer.6.attention.self.key.weight

embeddings.LayerNorm.weight

encoder.layer.6.attention.self.key.bias

embeddings.LayerNorm.bias

encoder.layer.6.attention.self.value.weight

encoder.layer.0.attention.self.query.weight

encoder.layer.6.attention.self.value.bias

encoder.layer.0.attention.self.query.bias

encoder.layer.6.attention.output.dense.weight

encoder.layer.0.attention.self.key.weight

encoder.layer.6.attention.output.dense.bias

encoder.layer.6.attention.output.LayerNorm.weight

encoder.layer.0.attention.self.value.weight

encoder.layer.6.attention.output.LayerNorm.bias

encoder.layer.6.intermediate.dense.bias

encoder.layer.0.attention.output.dense.bias

encoder.layer.6.output.dense.weight-

encoder.layer.0.attention.output.LayerNorm.weight

encoder.layer.6.output.dense.bias

encoder.layer.0.attention.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.bias

encoder.layer.6.output.LayerNorm.bias

encoder.layer.0.output.dense.weight

encoder.layer.7.attention.self.query.bias

encoder.layer.0.output.dense.bias

encoder.layer.7.attention.self.key.weight

encoder.layer.0.output.LayerNorm.weight

encoder.layer.7.attention.self.key.bias

encoder.layer.7.attention.self.value.weight

encoder.layer.1.attention.self.query.weight

encoder.layer.7.attention.self.value.bias

encoder.layer.1.attention.self.query.bias

encoder.layer.7.attention.output.dense.weight

encoder.layer.1.attention.self.key.bias

encoder.layer.7.attention.output.LayerNorm.weight

encoder.layer.7.attention.output.dense.bias

encoder.layer.7.attention.output.LayerNorm.bias

encoder.layer.1.attention.output.dense.weight

encoder.layer.1.attention.self.value.bias

encoder.layer.7.intermediate.dense.bias

encoder.layer.1.attention.output.dense.bias

encoder.layer.7.output.dense.weight

encoder.layer.1.attention.output.LayerNorm.weight

encoder.layer.7.output.dense.bias

encoder.layer.1.intermediate.dense.weight

encoder.layer.1.output.LayerNorm.weight

encoder.layer.2.attention.self.key.weight

encoder.layer.1.intermediate.dense.bias

encoder.layer.2.attention.self.query.bias

encoder.layer.1.output.LayerNorm.bias

encoder.layer.1.output.dense.weight

encoder.layer.1.output.dense.bias

encoder.layer.8.attention.output.dense.weight

encoder.layer.8.attention.self.value.weight

encoder.layer.8.attention.self.query.bias

encoder.layer.8.attention.self.key.weight

encoder.layer.8.attention.self.value.bias

encoder.layer.7.output.LayerNorm.bias

encoder.layer.8.attention.self.key.bias

encoder.layer.2.attention.output.LayerNorm.weight

encoder.layer.2.attention.output.dense.weight

encoder.layer.2.attention.output.dense.bias

encoder.layer.2.attention.self.value.weight

encoder.layer.2.attention.self.value.bias

encoder.layer.2.attention.self.key.bias

encoder.layer.8.attention.output.LayerNorm.bias

encoder.layer.8.intermediate.dense.weight

encoder.layer.8.output.LayerNorm.weight

encoder.layer.8.intermediate.dense.bias

encoder.layer.8.output.dense.weight-

encoder.layer.8.output.dense.bias

encoder.layer.3.attention.output.LayerNorm.weight

encoder.layer.2.attention.output.LayerNorm.bias

encoder.layer.3.attention.output.dense.weight

encoder.layer.3.attention.output.dense.bias

encoder.layer.2.intermediate.dense.weight

encoder.layer.3.attention.self.value.weight

encoder.layer.2.output.LayerNorm.weight

encoder.layer.2.intermediate.dense.bias

encoder.layer.3.attention.self.value.bias

encoder.layer.3.attention.self.key.weight

encoder.layer.3.attention.self.query.bias

encoder.layer.2.output.LayerNorm.bias

encoder.layer.3.attention.self.key.bias

encoder.layer.2.output.dense.weight

encoder.layer.2.output.dense.bias

encoder.layer.9.attention.output.LayerNorm.weight

encoder.layer.9.attention.output.LayerNorm.bias

encoder.layer.9.attention.output.dense.weight

encoder.layer.9.attention.output.dense.bias

encoder.layer.9.attention.self.value.weight

encoder.layer.9.attention.self.key.weight

encoder.layer.9.attention.self.query.bias

encoder.layer.9.attention.self.value.bias

encoder.layer.9.intermediate.dense.bias

encoder.layer.8.output.LayerNorm.bias

encoder.layer.9.attention.self.key.bias

encoder.layer.9.output.dense.weight

encoder.layer.9.output.dense.bias

encoder.layer.3.attention.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.bias

encoder.layer.3.output.dense.weight

encoder.layer.10.attention.self.query.weight

encoder.layer.10.attention.self.query.bias

encoder.layer.9.output.LayerNorm.bias

encoder.layer.3.output.dense.bias

encoder.layer.10.attention.self.key.bias

encoder.layer.10.attention.self.value.weight

encoder.layer.3.output.LayerNorm.bias

encoder.layer.10.attention.self.value.bias

encoder.layer.4.attention.self.query.weight

encoder.layer.4.attention.self.key.weight

encoder.layer.4.attention.self.query.bias

encoder.layer.10.attention.output.dense.bias

encoder.layer.4.attention.self.key.bias

encoder.layer.10.attention.output.LayerNorm.bias

encoder.layer.4.attention.self.value.weight

encoder.layer.10.intermediate.dense.weight

encoder.layer.4.attention.self.value.bias

encoder.layer.10.intermediate.dense.bias

encoder.layer.4.attention.output.dense.weight

encoder.layer.4.attention.output.dense.bias

encoder.layer.10.output.dense.bias-

encoder.layer.10.output.LayerNorm.weight

encoder.layer.4.attention.output.LayerNorm.bias

encoder.layer.10.output.LayerNorm.bias

encoder.layer.4.intermediate.dense.weight

encoder.layer.11.attention.self.query.weight

encoder.layer.4.intermediate.dense.bias

encoder.layer.11.attention.self.query.bias

encoder.layer.4.output.dense.weight

encoder.layer.11.attention.self.key.weight

encoder.layer.4.output.dense.bias

encoder.layer.11.attention.self.key.bias

encoder.layer.4.output.LayerNorm.weight

encoder.layer.11.attention.self.value.weight

encoder.layer.4.output.LayerNorm.bias

encoder.layer.11.attention.self.value.bias

encoder.layer.11.attention.output.dense.weight

encoder.layer.5.attention.self.query.bias

encoder.layer.11.attention.output.LayerNorm.weight

encoder.layer.11.attention.output.dense.bias

encoder.layer.5.attention.self.key.bias

encoder.layer.11.attention.output.LayerNorm.bias

encoder.layer.5.attention.self.value.weight

encoder.layer.11.intermediate.dense.weight

encoder.layer.5.attention.self.value.bias

encoder.layer.11.intermediate.dense.bias

encoder.layer.5.attention.output.dense.weight

encoder.layer.11.output.dense.weight

encoder.layer.5.attention.output.dense.bias

encoder.layer.11.output.dense.bias

encoder.layer.11.output.LayerNorm.bias

encoder.layer.5.intermediate.dense.weight

pooler.dense.weight

encoder.layer.5.intermediate.dense.bias

pooler.dense.bias

encoder.layer.5.output.dense.weight

taskmodels_dict.mrpc.head.dense.weight

encoder.layer.5.output.dense.bias

taskmodels_dict.mrpc.head.dense.bias

encoder.layer.5.output.LayerNorm.weight

taskmodels_dict.mrpc.head.out_proj.weight

encoder.layer.5.output.LayerNorm.bias

taskmodels_dict.mrpc.head.out_proj.bias

0.00

0.25

50.500.75

1.00

0.000.250.500.751.00

Updatable Parameters (%) in Modul

Updatable Parameters (%) in Modu

ModuleName

mrpc ( Layer-wise sparsity pattern)

embeddings.word_embeddings.weight -

embeddings.position_embeddings.weight

encoder.layer.6.attention.self.query.bias

embeddings.token_type_embeddings.weight

embeddings.LayerNorm.weight

encoder.layer.6.attention.self.key.bias

embeddings.LayerNorm.bias

encoder.layer.0.attention.self.query.weight

encoder.layer.6.attention.self.value.bias

encoder.layer.0.attention.self.query.bias

encoder.layer.6.attention.output.dense.weight

encoder.layer.0.attention.self.key.weight

encoder.layer.6.attention.output.dense.bias

encoder.layer.0.attention.self.key.bias

encoder.layer.0.attention.self.value.weight

encoder.layer.6.attention.output.LayerNorm.bias

encoder.layer.6.intermediate.dense.weight

encoder.layer.6.intermediate.dense.bias

encoder.layer.0.attention.output.dense.bias

encoder.layer.6.output.dense.weight

encoder.layer.6.output.dense.bias

encoder.layer.0.attention.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.bias

encoder.layer.7.attention.self.query.weight

encoder.layer.6.output.LayerNorm.bias

encoder.layer.0.output.dense.weight

encoder.layer.7.attention.self.query.bias

encoder.layer.0.output.dense.bias

encoder.layer.7.attention.self.key.weight

encoder.layr.7.attentionself.key.bias

encoder.layer.7.attention.self.value.weight

encoder.layer.7.attention.self.value.bias

encoder.layer.1.attention.self.query.bias

encoder.layer.1.attention.self.key.weight

encoder.layer.7.attention.output.LayerNorm.weight

encoder.layer.7.attention.output.dense.bias

encoder.layer.1.attention.self.value.weight

encoder.layer.7.attention.output.LayerNorm.bias

encoder.layer.1.attention.output.dense.weight

encoder.layer.1.attention.self.value.bias

encoder.layer.7.intermediate.dense.weight

encoder.layer.7.intermediate.dense.bias

encoder.layer.1.attention.output.dense.bias

encoder.layer.7.output.dense.weight

encoder.layer.1.attention.output.LayerNorm.weight

encoder.layer.7.output.dense.bias

encoder.layer.1.intermediate.dense.weight

encoder.layer.1.output.LayerNorm.weight

encoder.layer.1.intermediate.dense.bias

encoder.layer.1.output.dense.weight

encoder.layer.1.output.dense.bias

encoder.layer.8.attention.self.key.weight

encoder.layer.8.attention.self.query.bias

encoder.layer.7.output.LayerNorm.bias

encoder.layer.8.attention.self.key.bias

encoder.layer.2.attention.self.query.weight

encoder.layer.1.output.LayerNorm.bias

encoder.layer.8.attention.self.value.weight

encoder.layer.8.attention.self.value.bias

encoder.layer.2.attention.self.query.bias

encoder.layer.8.attention.output.dense.weight

encoder.layer.2.attention.self.key.bias

encoder.layer.2.attention.output.LayerNorm.weight

encoder.layer.2.attention.output.dense.bias

encoder.layer.2.attention.self.value.weight

encoder.layer.2.attention.self.value.bias

encoder.layer.8.attention.output.LayerNorm.bias

encoder.layer.8.intermediate.dense.weight

encoder.layer.8.output.LayerNorm.weight

encoder.layer.8.intermediate.dense.bias

encoder.layer.8.output.dense.weight

encoder.layer.8.output.dense.bias

encoder.layer.2.attention.output.LayerNorm.bias

encoder.layer.2.intermediate.dense.weight

encoder.layer.2.intermediate.dense.bias

encoder.layer.2.output.dense.weight

encoder.layer.2.output.dense.bias

encoder.layer.8.output.LayerNorm.bias

encoder.layer.9.attention.self.key.bias

encoder.layer.2.output.LayerNorm.weight

encoder.layer.3.attention.self.query.bias

encoder.layer.3.attention.self.key.weight

encoder.layer.2.output.LayerNorm.bias

encoder.layer.9.attention.output.dense.weight

encoder.layer.9.attention.output.dense.bias

encoder.layer.9.attention.self.value.weight

encoder.layer.9.attention.self.value.bias

encoder.layer.3.attention.output.LayerNorm.weight

encoder.layer.3.attention.output.LayerNorm.bias

encoder.layer.3.attention.output.dense.bias

encoder.layer.3.intermediate.dense.weight

encoder.layer.3.attention.self.value.weight

encoder.layer.3.attention.self.value.bias

encoder.layer.3.attention.self.key.bias

encoder.layer.9.attention.output.LayerNorm.bias

encoder.layer.9.intermediate.dense.bias

encoder.layer.9.output.LayerNorm.bias

encoder.layer.9.output.dense.weight

encoder.layer.9.output.dense.bias

encoder.layer.3.intermediate.dense.bias

encoder.layer.3.output.dense.weight

encoder.layer.10.attention.self.query.bias

encoder.layer.3.output.dense.bias

encoder.layer.10.attention.self.value.weight

encoder.layer.10.attention.self.key.bias

encoder.layer.4.attention.self.query.bias

encoder.layer.3.output.LayerNorm.bias

encoder.layer.4.attention.self.key.bias

encoder.layer.10.attention.output.LayerNorm.weight

encoder.layer.10.attention.output.LayerNorm.bias

encoder.layer.10.attention.output.dense.bias

encoder.layer.10.intermediate.dense.weight

encoder.layer.10.attention.self.value.bias

encoder.layer.4.attention.self.value.bias

encoder.layer.10.intermediate.dense.bias

encoder.layer.10.output.dense.bias

encoder.layer.10.output.LayerNorm.weight

encoder.layer.4.attention.output.LayerNorm.bias

encoder.layer.10.output.LayerNorm.bias

encoder.layer.4.intermediate.dense.weight

encoder.layer.4.intermediate.dense.bias

encoder.layer.4.output.dense.weight

encoder.layer.11.attention.self.key.weight

encoder.layer.4.output.dense.bias

encoder.layer.11.attention.self.key.bias

encoder.layer.4.output.LayerNorm.weight

encoder.layer.11.attention.self.value.weight

encoder.layer.4.output.LayerNorm.bias

encoder.layer.11.attention.self.value.bias

encoder.layer.11.attention.output.dense.weight

encoder.layer.5.attention.self.key.weight

encoder.layer.5.attention.self.query.bias

encoder.layer.11.attention.output.LayerNorm.weight

encoder.layer.11.attention.output.dense.bias

encoder.layer.5.attention.self.key.bias

encoder.layer.11.attention.output.LayerNorm.bias

encoder.layer.5.attention.self.value.weight

encoder.layer.11.intermediate.dense.weight

encoder.layer.5.attention.self.value.bias

encoder.layer.11.intermediate.dense.bias

encoder.layer.5.attention.output.dense.weight

encoder.layer.11.output.dense.weight

encoder.layer.5.attention.output.dense.bias

encoder.layer.11.output.dense.bias

encoder.layer.5.attention.output.LayerNorm.bias

encoder.layer.11.output.LayerNorm.bias

encoder.layer.5.intermediate.dense.weight

pooler.dense.weight

encoder.layer.5.intermediate.dense.bias

pooler.dense.bias

encoder.layer.5.output.dense.weight

taskmodels_dict.mrpc.head.dense.weight

encoder.layer.5.output.dense.bias

taskmodels_dict.mrpc.head.dense.bias

encoder.layer.5.output.LayerNorm.weight

taskmodels_dict.mrpc.head.out_proj.weight

encoder.layer.5.output.LayerNorm.bias

taskmodels_dict.mrpc.head.out_proj.bias

0.0

0.2

0.3

0.40.5

0.0

0.2

0.4

Updatable Parameters (%) in Layet

Updatable Parameters (%) in Laye

Name

Module

rte ( Module-wise sparsity pattern)

embeddings.word_embeddings.weight -

embeddings.position_embeddings.weight

encoder.layer.6.attention.self.query.bias

embeddings.token_type_embeddings.weight

encoder.layer.6.attention.self.key.weight

embeddings.LayerNorm.weight

encoder.layer.6.attention.self.key.bias

embeddings.LayerNorm.bias

encoder.layer.6.attention.self.value.weight

中

encoder.layer.0.attention.self.query.weight

encoder.layer.0.attention.self.query.bias

encoder.layer.6.attention.output.dense.weight

encoder.layer.6.attention.self.value.bias

encoder.layer.0.attention.self.key.weight-

encoder.layer.6.attention.output.dense.bias

encoder.layer.0.attention.self.key.bias

encoder.layer.6.attention.output.LayerNorm.weight

encoder.layer.0.attention.self.value.weight

encoder.layer.6.attention.output.LayerNorm.bias

encoder.layer.6.intermediate.dense.weight

encoder.layer.0.attention.output.dense.bias

encoder.layer.6.output.dense.weight

encoder.layer.0.attention.output.LayerNorm.weight

encoder.layer.6.output.dense.bias

encoder.layer.0.attention.output.LayerNorm.bias

encoder.layer.6.output.LayerNorm.weight

encoder.layer.0.intermediate.dense.weight-

encoder.layer.0.intermediate.dense.bias

口

encoder.layer.0.output.dense.weight

encoder.layer.7.attention.self.query.bias

encoder.layer.0.output.dense.bias

encoder.layer.7.attention.self.key.weight

encoder.layer.7.attention.self.key.bias

encoder.layer.0.output.LayerNorm.bias

encoder.layer.7.attention.self.value.weight

encoder.layer.7.attention.self.value.bias

encoder.layer.1.attention.self.query.bias

encoder.layer.7.attention.output.dense.weight

encoder.layer.1.attention.self.key.bias

encoder.layer.7.attention.output.LayerNorm.weight

encoder.layer.7.attention.output.dense.bias

encoder.layer.1.attention.output.dense.weight

encoder.layer.1.attention.self.value.weight

encoder.layer.1.attention.self.value.bias

encoder.layer.7.attention.output.LayerNorm.bias

encoder.layer.7.intermediate.dense.weight

encoder.layer.7.intermediate.dense.bias

encoder.layer.1.attention.output.dense.bias

encoder.layer.7.output.dense.weight

encoder.layer.1.attention.output.LayerNorm.weight

encoder.layer.7.output.dense.bias

encoder.layer.2.attention.output.LayerNorm.weight

encoder.layer.1.attention.output.LayerNorm.bias

encoder.layer.2.attention.output.LayerNorm.bias

encoder.layer.2.attention.output.dense.weight-

encoder.layer.2.attention.output.dense.bias

encoder.layer.1.intermediate.dense.weight

encoder.layer.2.intermediate.dense.weight

encoder.layer.2.attention.self.value.weight

encoder.layer.1.output.LayerNorm.weight

encoder.layer.2.output.LayerNorm.weight

encoder.layer.2.attention.self.key.weight

encoder.layer.2.attention.self.value.bias

encoder.layer.1.intermediate.dense.bias

encoder.layer.2.intermediate.dense.bias

encoder.layer.3.attention.self.key.weight

encoder.layer.2.attention.self.query.bias

encoder.layer.3.attention.self.query.bias

encoder.layer.2.output.LayerNorm.bias

encoder.layer.1.output.LayerNorm.bias

encoder.layer.2.attention.self.key.bias

encoder.layer.1.output.dense.weight

encoder.layer.2.output.dense.weight

encoder.layer.1.output.dense.bias

encoder.layer.2.output.dense.bias

encoder.layer.9.attention.output.LayerNorm.weight

encoder.layer.8.attention.output.dense.weight-

encoder.layer.8.attention.output.dense.bias

encoder.layer.9.attention.output.dense.bias

encoder.layer.8.intermediate.dense.weight-

encoder.layer.9.attention.self.value.weight

encoder.layer.7.output.LayerNorm.weight

encoder.layer.8.output.LayerNorm.weight

encoder.layer.8.attention.self.key.weight

encoder.layer.8.attention.self.value.bias

encoder.layer.8.intermediate.dense.bias

encoder.layer.9.attention.self.value.bias

encoder.layer.7.output.LayerNorm.bias

encoder.layer.8.output.LayerNorm.bias

encoder.layer.8.attention.self.key.bias

encoder.layer.8.output.dense.weight

encoder.layer.8.output.dense.bias

encoder.layer.3.attention.self.value.weight

encoder.layer.3.attention.self.key.bias

encoder.layer.9.attention.output.LayerNorm.bias

encoder.layer.3.attention.output.LayerNorm.weight

encoder.layer.3.attention.output.dense.bias

encoder.layer.3.attention.self.value.bias

encoder.layer.9.output.LayerNorm.weight

encoder.layer.9.intermediate.dense.bias

encoder.layer.9.output.dense.weight

encoder.layer.9.output.dense.bias

encoder.layer.3.attention.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.weight

encoder.layer.9.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.bias

encoder.layer.3.output.dense.weight

encoder.layer.10.attention.self.query.bias

encoder.layer.3.output.dense.bias

encoder.layer.10.attention.self.key.bias

encoder.layer.3.output.LayerNorm.bias

encoder.layer.10.attention.self.value.bias

encoder.layer.4.attention.self.query.weight

encoder.layer.4.attention.self.key.weight

encoder.layer.4.attention.self.query.bias

encoder.layer.10.attention.output.LayerNorm.weight-

encoder.layer.10.attention.output.dense.bias

encoder.layer.4.attention.self.key.bias

encoder.layer.10.attention.output.LayerNorm.bias

encoder.layer.4.attention.self.value.weight

encoder.layer.10.intermediate.dense.weight

encoder.layer.4.attention.self.value.bias

encoder.layer.10.intermediate.dense.bias

encoder.layer.4.attention.output.LayerNorm.weight

encoder.layer.4.attention.output.dense.bias

encoder.layer.10.output.LayerNorm.weight

encoder.layer.10.output.dense.bias

encoder.layer.4.attention.output.LayerNorm.bias

encoder.layer.10.output.LayerNorm.bias

encoder.layer.4.intermediate.dense.weight

encoder.layer.11.attention.self.query.weight

encoder.layer.4.intermediate.dense.bias

encoder.layer.11.attention.self.query.bias

encoder.layer.4.output.dense.weight

encoder.layer.11.attention.self.key.weight

encoder.layer.4.output.dense.bias

encoder.layer.11.attention.self.key.bias

encoder.layer.4.output.LayerNorm.weight

encoder.layer.11.attention.self.value.weight

encoder.layer.4.output.LayerNorm.bias

encoder.layer.11.attention.self.value.bias

encoder.layer.11.attention.output.dense.weight

encoder.layer.5.attention.self.query.bias

encoder.layer.11.attention.output.dense.bias

encoder.layer.5.attention.self.key.bias

encoder.layer.11.attention.output.LayerNorm.bias

encoder.layer.5.attention.self.value.weight

encoder.layer.11.intermediate.dense.weight

encoder.layer.5.attention.self.value.bias

encoder.layer.11.intermediate.dense.bias

encoder.layer.5.attention.output.dense.weight

encoder.layer.11.output.dense.weight

encoder.layer.5.attention.output.dense.bias

encoder.layer.11.output.dense.bias

encoder.layer.11.output.LayerNorm.weight

encoder.layer.5.attention.output.LayerNorm.bias

encoder.layer.5.intermediate.dense.weight

encoder.layer.11.output.LayerNorm.bias

pooler.dense.weight

encoder.layer.5.intermediate.dense.bias

pooler.dense.bias

encoder.layer.5.output.dense.weight

taskmodels_dict.rte.head.dense.weight

encoder.layer.5.output.dense.bias

taskmodels_dict.rte.head.dense.bias

encoder.layer.5.output.LayerNorm.weight

taskmodels_dict.rte.head.out_proj.weight

encoder.layer.5.output.LayerNorm.bias

taskmodels_dict.rte.head.out_proj.bias

0.02.55.0 7.5 10.0 12.5

0

5

10

Updatable Parameters (%) in Modul

Updatable Parameters (%) in Modu

Name

Module

rte ( Layer-wise sparsity pattern)

embeddings.word_embeddings.weight-

embeddings.position_embeddings.weight

encoder.layer.6.attention.selfquery.bias

embeddings.token_type_embeddings.weight

embeddings.LayerNorm.weight

encoder.layer.6.attention.self.key.bias

embeddings.LayerNorm.bias

encoder.layer.0.attention.self.query.weight

encoder.layer.6.attention.self.value.bias

encoder.layer.0.attention.selfquery.bias

encoder.layer.6.attention.output.dense.weight

encoder.layer.0.attention.self.key.weight

encoder.layer.6.attention.output.dense.bias

encoder.layer.0.attention.self.key.bias

encoder.layer.6.attention.output.LayerNorm.weight

encoder.layer.0.attention.self.value.weight

encoder.layer.6.attention.output.LayerNorm.bias

encoder.layer.6.intermediate.dense.weight

encoder.layer.6.intermediate.dense.bias

encoder.layer.6.output.dense.weight

encoder.layer.0.attention.output.LayerNorm.weight

encoder.layer.6.output.dense.bias-

encoder.layer.0.attention.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.weight

encoder.layer.6.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.bias

encoder.layer.0.output.dense.weight

encoder.layer.7.attention.self.query.bias

encoder.layer.0.output.dense.bias

encoder.layer.7.attention.self.key.weight

encoder.layer.0.output.LayerNorm.weight

encoder.layer.7.attention.self.key.bias

encoder.layer.7.attention.self.value.weight

encoder.layer.7.attention.self.value.bias

encoder.layer.1.attention.self.query.bias

encoder.layer.7.attention.output.LayerNorm.weight

encoder.layer.7.attention.output.dense.bias

encoder.layer.1.attention.self.value.weight

encoder.layer.7.attention.output.LayerNorm.bias

encoder.layer.1.attention.output.dense.weight

encoder.layer.1.attention.self.value.bias

encoder.layer.7.intermediate.dense.weight

encoder.layer.7.intermediate.dense.bias

encoder.layer.1.attention.output.dense.bias

encoder.layer.7.output.dense.weight

encoder.layer.1.attention.output.LayerNorm.weight

encoder.layer.7.output.dense.bias

encoder.layer.1.attention.output.LayerNorm.bias

encoder.layer.1.intermediate.dense.weight

encoder.layer.1.output.LayerNorm.weight

encoder.layer.1.intermediate.dense.bias

encoder.layer.1.output.dense.weight

encoder.layer.1.output.dense.bias

encoder.layer.7.output.LayerNorm.weight

encoder.layer.8.attention.self.key.weight

encoder.layer.7.output.LayerNorm.bias

encoder.layer.8.attention.self.key.bias

encoder.layer.2.attention.self.query.bias

encoder.layer.1.output.LayerNorm.bias

encoder.layer.8.attention.output.dense.weight

encoder.layer.8.attention.self.value.weight

encoder.layer.8.attention.self.value.bias

encoder.layer.2.attention.self.key.weight

encoder.layer.2.attention.self.key.bias

encoder.layer.2.attention.output.LayerNorm.weight

encoder.layer.2.attention.output.dense.weight

encoder.layer.2.attention.output.dense.bias

encoder.layer.2.attention.self.value.weight

encoder.layer.2.attention.self.value.bias

encoder.layer.8.attention.output.LayerNorm.bias

encoder.layer.8.intermediate.dense.weight

encoder.layer.8.output.LayerNorm.weight

encoder.layer.8.intermediate.dense.bias

encoder.layer.8.output.dense.weight

encoder.layer.8.output.dense.bias

encoder.layer.2.attention.output.LayerNorm.bias

encoder.layer.2.intermediate.dense.weight

encoder.layer.2.output.LayerNorm.weight

encoder.layer.2.intermediate.dense.bias

encoder.layer.3.attention.self.query.bias

encoder.layer.2.output.LayerNorm.bias

encoder.layer.2.output.dense.weight

encoder.layer.2.output.dense.bias

encoder.layer.9.attention.output.dense.weight

encoder.layer.9.attention.output.dense.bias

encoder.layer.9.attention.self.value.weight

encoder.layer.9.attention.self.value.bias

encoder.layer.8.output.LayerNorm.bias

encoder.layer.9.attention.self.key.bias

encoder.layer.3.attention.output.LayerNorm.weight

encoder.layer.3.attention.output.dense.bias

encoder.layer.3.attention.self.value.weight

encoder.layer.3.attention.self.value.bias

encoder.layer.3.attention.self.key.bias

encoder.layer.9.attention.output.LayerNorm.bias

encoder.layer.9.intermediate.dense.bias

encoder.layer.9.output.dense.weight

encoder.layer.9.output.dense.bias

encoder.layer.3.attention.output.LayerNorm.bias

encoder.layer.9.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.bias

encoder.layer.10.attention.self.query.bias

encoder.layer.3.output.dense.weight

encoder.layer.3.output.dense.bias

encoder.layer.10.attention.self.value.weight

encoder.layer.10.attention.self.key.bias

encoder.layer.3.output.LayerNorm.bias

encoder.layer.10.attention.self.value.bias

encoder.layer.4.attention.self.query.weight

encoder.layer.4.attention.self.query.bias

encoder.layer.4.attention.self.key.bias

encoder.layer.10.attention.output.LayerNorm.bias

encoder.layer.10.attention.output.dense.bias

encoder.layer.4.attention.self.value.weight

encoder.layer.10.intermediate.dense.weight

encoder.layer.4.attention.self.value.bias

encoder.layer.10.intermediate.dense.bias

encoder.layer.10.output.dense.weight

encoder.layer.10.output.dense.bias -

encoder.layer.10.output.LayerNorm.weight

encoder.layer.4.attention.output.LayerNorm.bias

encoder.layer.10.output.LayerNorm.bias

encoder.layer.4.intermediate.dense.weight

encoder.layer.4.intermediate.dense.bias

encoder.layer.11.attention.self.query.bias

encoder.layer.4.output.dense.weight

encoder.layer.11.attention.self.key.weight

encoder.layer.4.output.dense.bias

encoder.layer.11.attention.self.key.bias

encoder.layer.4.output.LayerNorm.weight

encoder.layer.11.attention.self.valu.weigh

encoder.layer.4.output.LayerNorm.bias

encoder.layer.11.attention.self.value.bias

encoder.layer.11.attention.output.dense.weight

encoder.layer.5.attention.self.query.bias

encoder.layer.11.attention.output.dense.bias

encoder.layer.5.attention.self.key.weight

encoder.layer.11.attention.output.LayerNorm.weight

encoder.layer.5.attention.self.key.bias

encoder.layer.11.attention.output.LayerNorm.bias

encoder.layer.5.attention.self.value.weight

encoder.layer.11.intermediate.dense.weight

encoder.layer.5.attention.self.value.bias

encoder.layer.11.intermediate.dense.bias

encoder.layer.5.attention.output.dense.weight

encoder.layer.11.output.dense.weight

encoder.layer.5.attention.output.dense.bias

encoder.layer.11.output.dense.bias

encoder.layer.11.output.LayerNorm.weight

encoder.layer.5.attention.output.LayerNorm.bias

encoder.layer.11.output.LayerNorm.bias

encoder.layer.5.intermediate.dense.weight

pooler.dense.weight

encoder.layer.5.intermediate.dense.bias

pooler.dense.bias

encoder.layer.5.output.dense.weight

taskmodels_dict.rte.head.dense.weight

encoder.layer.5.output.dense.bias

taskmodels_dict.rte.head.dense.bias

encoder.layer.5.output.LayerNorm.weight

taskmodels_dict.rte.head.out_proj.weight

encoder.layer.5.output.LayerNorm.bias

taskmodels_dict.rte.head.out_proj.bias

2

3

Updatable Parameters (%) in Layet

Updatable Parameters (%) in Laye

Name

Module

cb ( Module-wise sparsity pattern)

embeddings.word_embeddings.weight

embeddings.position_embeddings.weight-

encoder.layer.6.attention.selfquery.bias

embeddings.token_type_embeddings.weight

encoder.layer.6.attention.self.key.weight

embeddings.LayerNorm.weight

encoder.layer.6.attention.self.key.bias

embeddings.LayerNorm.bias

encoder.layer.0.attention.self.query.weight

encoder.layer.6.attention.self.value.bias

encoder.layer.0.attention.self.query.bias

encoder.layer.6.attention.output.dense.weight

encoder.layer.0.attention.self.key.weight

encoder.layer.6.attention.output.dense.bias

encoder.layer.0.attention.self.key.bias

encoder.layer.6.attention.output.LayerNorm.weight

encoder.layer..attention.selfvalue.weigh

encoder.layer.6.attention.output.LayerNorm.bias

encoder.layer.6.intermediate.dense.bias-

encoder.layer.0.attention.output.dense.bias

encoder.layer.6.output.dense.weight

encoder.layer.0.attention.output.LayerNorm.weight

encoder.layer.6.output.dense.bias

encoder.layer.0.attention.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.weight

encoder.layer.6.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.bias

encoder.layer.0.output.dense.weight

encoder.layer.7.attention.self.query.bias

encoder.layer.0.output.LayerNorm.weight

encoder.layer.0.output.dense.bias

encoder.layer.7.attention.self.key.weight

encoder.layer.7.attention.self.key.bias

encoder.layer.7.attention.self.value.weight

encoder.layer.7.attention.self.value.bias

encoder.layer.1attention.elfquerybias

encoder.layer.7.attention.output.dense.weight

encoder.layer.1.attention.self.key.weight

encoder.layer.7.attention.output.LayerNorm.weight

encoder.layer.7.attention.output.dense.bias

encoder.layer.1.attention.self.value.weight

encoder.layer.7.attention.output.LayerNorm.bias

encoder.layer.1.attention.output.dense.weight

encoder.layer.1.attention.self.value.bias

encoder.layer.7.intermediate.dense.bias

encoder.layer.1.attention.output.dense.bias

encoder.layer.7.output.dense.weight

encoder.layer.1.attention.output.LayerNorm.weight

encoder.layer.7.output.dense.bias

encoder.layer.1.intermediate.dense.weight

encoder.layer.1.output.LayerNorm.weight

encoder.layer.1.intermediate.dense.bias

encoder.layer.1.output.dense.weight

encoder.layer.1.output.dense.bias

encoder.layer.8.attention.self.query.bias

encoder.layer.8.attention.self.key.weight

encoder.layer.7.output.LayerNorm.bias

encoder.layer.8.attention.self.key.bias

encoder.layer.2.attention.self.key.weight

encoder.layer.2.attention.self.query.bias

encoder.layer.1.output.LayerNorm.bias

encoder.layer.8.attention.output.dense.weight

encoder.layer.8.attention.self.value.weight

encoder.layer.8.attention.self.value.bias

encoder.layer.2.attention.output.LayerNorm.weight

encoder.layer.2.attention.output.LayerNorm.bias

encoder.layer.2.attention.output.dense.bias

encoder.layer.2.intermediate.dense.weight

encoder.layer.2.attention.self.value.weight

encoder.layer.2.output.LayerNorm.weight

encoder.layer.2.attention.self.value.bias

encoder.layer.3.attention.self.query.bias

encoder.layer.2.intermediate.dense.bias

encoder.layer.3.attention.self.key.weight

encoder.layer.2.output.LayerNorm.bias

encoder.layer.2.attention.self.key.bias

encoder.layer.2.output.dense.weight

encoder.layer.2.output.dense.bias

encoder.layer.8.attention.output.LayerNorm.bias

encoder.layer.9.attention.output.dense.weight

encoder.layer.9.attention.output.dense.bias

encoder.layer.8.intermediate.dense.weight

encoder.layer.9.attention.self.value.weight

encoder.layer.8.output.LayerNorm.weight

encoder.layer.8.intermediate.dense.bias

encoder.layer.9.attention.self.value.bias

encoder.layer.8.output.LayerNorm.bias

encoder.layer.9.attention.self.key.bias

encoder.layer.8.output.dense.weight -

encoder.layer.8.output.dense.bias

encoder.layer.3.attention.output.LayerNorm.weight

encoder.layer.3.attention.output.dense.bias

encoder.layer.3.attention.self.value.weight

encoder.layer.3.attention.self.value.bias

encoder.layer.3.attention.self.key.bias

encoder.layer.9.attention.output.LayerNorm.bias

encoder.layer.9.intermediate.dense.weight-

encoder.layer.9.intermediate.dense.bias

encoder.layer.9.output.dense.weight

encoder.layer.9.output.dense.bias

encoder.layer.3.attention.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.weight

encoder.layer.9.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.bias

encoder.layer.3.output.dense.weight

encoder.layer.10.attention.self.key.weight

encoder.layer.10.attention.self.query.bias

encoder.layer.3.output.dense.bias

encoder.layer.10.attention.self.value.weight

encoder.layer.10.attention.self.key.bias

encoder.layer.3.output.LayerNorm.bias

encoder.layer.10.attention.self.value.bias

encoder.layer.4.attention.self.query.weight

encoder.layer.4.attention.self.key.weight

encoder.layer.4.attention.self.key.bias

encoder.layer.10.attention.output.LayerNorm.bias

encoder.layer.10.attention.output.dense.weight

encoder.layer.10.attention.output.dense.bias

encoder.layer.4.attention.self.value.weight

encoder.layer.10.intermediate.dense.weight

encoder.layer.4.attention.self.value.bias

encoder.layer.10.intermediate.dense.bias

encoder.layer.10.output.dense.weight

encoder.layer.4.attention.output.dense.bias

encoder.layer.10.output.dense.bias -

encoder.layer.10.output.LayerNorm.weight

encoder.layer.4.attention.output.LayerNorm.bias

encoder.layer.10.output.LayerNorm.bias

encoder.layer.4.intermediate.dense.weight

encoder.layer.4.intermediate.dense.bias

encoder.layer.4.output.dense.weight

encoder.layer.11.attention.self.key.weight

encoder.layer.4.output.dense.bias

encoder.layer.11.attention.self.key.bias

encoder.layer.4.output.LayerNorm.weight

encoder.layer.11.attention.self.value.weight

encoder.layer.4.output.LayerNorm.bias

encoder.layer.11.attention.self.value.bias

encoder.layer.11.attention.output.dense.weight

encoder.layer.5.attention.self.query.bias

encoder.layer.11.attention.output.dense.bias

encoder.layer.5.attention.self.key.weight

encoder.layer.11.attention.output.LayerNorm.weight

encoder.layer.5.attention.self.key.bias

encoder.layer.11.attention.output.LayerNorm.bias

encoder.layer.11.intermediate.dense.weight

encoder.layer.5.attention.self.value.bias

encoder.layer.11.intermediate.dense.bias

encoder.layer.11.output.dense.weight

encoder.layer.5.attention.output.dense.bias

encoder.layer.11.output.dense.bias

encoder.layer.11.output.LayerNorm.weight

encoder.layer.11.output.LayerNorm.bias

encoder.layer.5.intermediate.dense.weight

pooler.dense.weight

encoder.layer.5.intermediate.dense.bias

pooler.dense.bias

encoder.layer.5.output.dense.weight

taskmodels_dict.cb.head.dense.weight

encoder.layer.5.output.dense.bias

taskmodels_dict.cb.head.dense.bias

encoder.layer.5.output.LayerNorm.weight

taskmodels_dict.cb.head.out_proj.weight

encoder.layer.5.output.LayerNorm.bias

taskmodels_dict.cb.head.out_proj.bias

0.0

0.5

1.0

1.5

2.0

2.5

0

Updatable Parameters (%) in Modul

Updatable Parameters (%) in Modu

Name

Module

cb ( Layer-wise sparsity pattern)

embeddings.word_embeddings.weight

embeddings.position_embeddings.weight

encoder.layer.6.attention.selfquery.bias

embeddings.token_type_embeddings.weight

embeddings.LayerNorm.weight

encoder.layer.6.attention.self.key.bias

embeddings.LayerNorm.bias

encoder.layer.0.attention.self.query.weight

encoder.layer.6.attention.self.value.bias

encoder.layer.0.attention.selfquery.bias

encoder.layer.6.attention.output.dense.weight

encoder.layer.0.attention.self.key.weight

encoder.layer.6.attention.output.dense.bias

encoder.layer.0.attention.self.key.bias

encoder.layer.6.attention.output.LayerNorm.weight

encoder.layer..attention.self.value.weigh

encoder.layer.6.attention.output.LayerNorm.bias

encoder.layer.6.intermediate.dense.weight

encoder.layer.6.intermediate.dense.bias

encoder.layer.6.output.dense.weight

encoder.layer.0.attention.output.LayerNorm.weight

encoder.layer.6.output.dense.bias

encoder.layer.0.attention.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.weight

encoder.layer.6.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.bias

encoder.layer.0.output.dense.weight

encoder.layer.7.attention.self.query.bias

encoder.layer.0.output.dense.bias

encoder.layer.7.attention.self.key.weight

encoder.layer.0.output.LayerNorm.weight

encoder.layer.7.attention.self.key.bias

encoder.layer.0.output.LayerNorm.bias

encoder.layer.7.attention.self.value.weight

encoder.layer.7.attention.self.value.bias

encoder.layer.1.attention.self.query.bias

encoder.layer.7.attention.output.LayerNorm.weight

encoder.layer.7.attention.output.dense.bias

encoder.layer.1.attention.self.value.weight

encoder.layer.7.attention.output.LayerNorm.bias

encoder.layer.1.attention.output.dense.weight

encoder.layer.1.attention.self.value.bias

encoder.layer.7.intermediate.dense.bias

encoder.layer.1.attention.output.dense.bias

encoder.layer.7.output.dense.weight

encoder.layer.1.attention.output.LayerNorm.weight

encoder.layer.7.output.dense.bias

encoder.layer.1.attention.output.LayerNorm.bias

encoder.layer.1.intermediate.dense.weight

encoder.layer.1.output.LayerNorm.weight

encoder.layer.1.intermediate.dense.bias

encoder.layer.1.output.dense.weight

encoder.layer.1.output.dense.bias

encoder.layer.8.attention.self.key.weight

encoder.layer.8.attention.self.query.bias

encoder.layer.7.output.LayerNorm.bias

encoder.layer.8.attention.self.key.bias

encoder.layer.2.attention.self.query.bias

encoder.layer.1.output.LayerNorm.bias

encoder.layer.8.attention.output.dense.weight

encoder.layer.8.attention.self.value.weight

encoder.layer.8.attention.self.value.bias

encoder.layer.2.attention.self.key.weight

encoder.layer.8.attention.output.dense.bias

encoder.layer.2.attention.output.LayerNorm.weight

encoder.layer.2.attention.output.dense.weight

encoder.layer.2.attention.output.dense.bias

encoder.layer.2.attention.self.value.weight

encoder.layer.2.attention.self.value.bias

encoder.layer.2.attention.self.key.bias

encoder.layer.8.attention.output.LayerNorm.bias

encoder.layer.8.intermediate.dense.weight

encoder.layer.8.output.LayerNorm.weight

encoder.layer.8.intermediate.dense.bias

encoder.layer.8.output.dense.weight

encoder.layer.8.output.dense.bias

encoder.layer.2.attention.output.LayerNorm.bias

encoder.layer.2.intermediate.dense.weight

encoder.layer.2.output.LayerNorm.weight

encoder.layer.2.intermediate.dense.bias

encoder.layer.3.attention.self.key.weight

encoder.layer.3.attention.self.query.bias

encoder.layer.2.output.LayerNorm.bias

encoder.layer.2.output.dense.weight

encoder.layer.2.output.dense.bias

encoder.layer.9.attention.output.dense.weight

encoder.layer.9.attention.output.dense.bias

encoder.layer.9.attention.self.value.weight

encoder.layer.9.attention.self.value.bias

encoder.layer.8.output.LayerNorm.bias

encoder.layer.9.attention.sef.key.bias

encoder.layer.3.attention.output.LayerNorm.weight

encoder.layer.3.attention.output.dense.bias

encoder.layer.3.attention.self.value.weight

encoder.layer.3.attention.self.value.bias

encoder.layer.3.attention.self.key.bias

encoder.layer.9.attention.output.LayerNorm.bias

encoder.layer.9.intermediate.dense.bias

encoder.layer.9.output.dense.weight

encoder.layer.9.output.dense.bias

encoder.layer.3.attention.output.LayerNorm.bias

encoder.layer.9.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.bias

encoder.layer.10.attention.self.query.bias

encoder.layer.3.output.dense.weight

encoder.layer.3.output.dense.bias

encoder.layer.10.attention.self.value.weight

encoder.layer.10.attention.self.key.bias

encoder.layer.3.output.LayerNorm.bias

encoder.layer.10.attention.self.value.bias

encoder.layer.4.attention.self.query.weight

encoder.layer.4.attention.self.query.bias

encoder.layer.4.attention.self.key.bias

encoder.layer.10.attention.output.LayerNorm.weight

encoder.layer.10.attention.output.LayerNorm.bias

encoder.layer.10.attention.output.dense.bias

encoder.layer.4.attention.self.value.weight

encoder.layer.10.intermediate.dense.weight

encoder.layer.4.attention.self.value.bias

encoder.layer.10.intermediate.dense.bias

encoder.layer.10.output.dense.bias -

encoder.layer.10.output.LayerNorm.weight

encoder.layer.4.attention.output.LayerNorm.bias

encoder.layer.10.output.LayerNorm.bias

encoder.layer.4.intermediate.dense.weight

encoder.layer.4.intermediate.dense.bias

encoder.layer.11.attention.self.query.bias

encoder.layer.4.output.dense.weight

encoder.layer.11.attention.self.key.weight

encoder.layer.4.output.dense.bias

encoder.layer.11.attention.self.key.bias

encoder.layer.4.output.LayerNorm.weight

encoder.layer.11.attention.self.value.weight

encoder.layer.4.output.LayerNorm.bias

encoder.layer.11.attention.self.value.bias

encoder.layer.5.attention.self.query.bias

encoder.layer.11.attention.output.dense.bias

encoder.layer.5.attention.self.key.weight

encoder.layer.11.attention.output.LayerNorm.weight

encoder.layer.5.attention.self.key.bias

encoder.layer.11.attention.output.LayerNorm.bias

encoder.layer.5.attention.self.value.weight

encoder.layer.11.intermediate.dense.weight

encoder.layer.5.attention.self.value.bias

encoder.layer.11.intermediate.dense.bias

encoder.layer.5.attention.output.dense.weight

encoder.layer.11.output.dense.weight

encoder.layer.5.attention.output.dense.bias

encoder.layer.11.output.dense.bias

encoder.layer.11.output.LayerNorm.weight

encoder.layer.5.attention.output.LayerNorm.bias

encoder.layer.11.output.LayerNorm.bias

encoder.layer.5.intermediate.dense.weight

pooler.dense.weight

encoder.layer.5.intermediate.dense.bias

pooler.dense.bias

encoder.layer.5.output.dense.weight

taskmodels_dict.cb.head.dense.weight

encoder.layer.5.output.dense.bias

taskmodels_dict.cb.head.dense.bias

encoder.layer.5.output.LayerNorm.weight

taskmodels_dict.cb.head.out_proj.weight

encoder.layer.5.output.LayerNorm.bias

taskmodels_dict.cb.head.out_proj.bias

0.0

0.5

1.0

1.5

2.0

02.5

0

一

Updatable Parameters (%) in Layer

Updatable Parameters (%) in Laye

copa ( Module-wise sparsity pattern)

embeddings.word_embeddings.weight

embeddings.position_embeddings.weight

encoder.layer.6.attention.selfquery.bias

embeddings.token_type_embeddings.weight

embeddings.LayerNorm.weight

encoder.layer.6.attention.self.key.bias

embeddings.LayerNorm.bias

encoder.layer.6.attention.self.value.weight

encoder.layer.0.attention.self.query.weight

encoder.layer.6.attention.self.value.bias

encoder.layer.0.attention.self.query.bias

encoder.layer.6.attention.output.dense.weight

encoder.layer.0.attention.self.key.weight

encoder.layer.6.attention.output.dense.bias

encoder.layer.0.attention.self.key.bias

encoder.layer.6.attention.output.LayerNorm.weight

encoder.layer.0.attention.self.value.weight

encoder.layer.6.attention.output.LayerNorm.bias

encoder.layer.6.intermediate.dense.bias

encoder.layer.0.attention.output.dense.bias

encoder.layer.6.output.dense.weight

encoder.layer.0.attention.output.LayerNorm.weight

encoder.layer.6.output.dense.bias

encoder.layer.0.attention.output.LayerNorm.bias

encoder.layer.6.output.LayerNorm.weight

encoder.layer.0.intermediate.dense.weight

encoder.layer.6.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.bias

encoder.layer.0.output.dense.weight

encoder.layer.7.attention.self.query.bias

encoder.layer.0.output.dense.bias

encoder.layer.7.attention.self.key.weight

encoder.layer.0.output.LayerNorm.weight

encoder.layer.7.attentionself.keybias

encoder.layer.1.attention.self.query.weight

encoder.layer.7.attention.self.value.weight

encoder.layer.7.attention.self.value.bias

encoder.layer.1.attention.self.key.weight

encoder.layer.1.attention.self.query.bias

encoder.layer.1.attention.self.key.bias

encoder.layer.7.attention.output.LayerNorm.weight

encoder.layer.7.attention.output.dense.bias

encoder.layer.1.attention.self.value.weight

encoder.layer.7.attention.output.LayerNorm.bias

encoder.layer.1.attention.self.value.bias

encoder.layer.7.intermediate.dense.weight

encoder.layer.1.attention.output.dense.weight

encoder.layer.7.intermediate.dense.bias

encoder.layer.1.attention.output.dense.bias

encoder.layer.7.output.dense.weight

encoder.layer.1.attention.output.LayerNorm.weight

encoder.layer.7.output.dense.bias

encoder.layer.1.intermediate.dense.weight

encoder.layer.1.output.LayerNorm.weight

encoder.layer.2.attention.self.key.weight

encoder.layer.1.intermediate.dense.bias

encoder.layer.2.attention.self.query.bias

encoder.layer.1.output.LayerNorm.bias

encoder.layer.1.output.dense.weight

encoder.layer.1.output.dense.bias

encoder.layer.8.attention.output.dense.weight

encoder.layer.8.attention.output.dense.bias

encoder.layer.8.attention.self.value.weight

encoder.layer.7.output.LayerNorm.weight

encoder.layer.8.attention.self.query.bias

encoder.layer.8.attention.self.key.weight

encoder.layer.8.attention.self.value.bias

encoder.layer.7.output.LayerNorm.bias

encoder.layer.8.attention.self.key.bias

encoder.layer.2.attention.output.LayerNorm.weight

encoder.layer.2.attention.output.dense.bias

encoder.layer.2.attention.self.value.weight

encoder.layer.2.attention.self.value.bias

encoder.layer.2.attention.self.key.bias

encoder.layer.8.attention.output.LayerNorm.bias

encoder.layer.8.intermediate.dense.weight

encoder.layer.8.output.LayerNorm.weight

encoder.layer.8.intermediate.dense.bias

encoder.layer.8.output.dense.weight -

encoder.layer.8.output.dense.bias

encoder.layer.3.attention.output.LayerNorm.weight

encoder.layer.2.attention.output.LayerNorm.bias

encoder.layer.3.attention.output.dense.bias

encoder.layer.2.intermediate.dense.weight

encoder.layer.2.output.LayerNorm.weight

encoder.layer.3.attention.self.value.bias

encoder.layer.2.intermediate.dense.bias

encoder.layer.3.attention.self.query.bias

encoder.layer.2.output.LayerNorm.bias

encoder.layer.3.attention.self.key.bias

encoder.layer.2.output.dense.weight

encoder.layer.2.output.dense.bias

encoder.layer.9.attention.output.LayerNorm.weight

encoder.layer.9.attention.output.LayerNorm.bias

encoder.layer.9.attention.output.dense.weight

encoder.layer.9.attention.output.dense.bias

encoder.layer.9.attention.self.value.weight

encoder.layer.9.intermediate.dense.bias

encoder.layer.9.attention.self.value.bias

encoder.layer.8.output.LayerNorm.bias

encoder.layer.9.output.dense.weight

encoder.layer.9.output.dense.bias

encoder.layer.3.attention.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.weight

encoder.layer.9.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.bias

encoder.layer.3.output.dense.weight

encoder.layer.10.attention.self.query.bias

encoder.layer.3.output.dense.bias

encoder.layer.10.attention.self.value.weight

encoder.layer.10.attention.self.key.bias

encoder.layer.3.output.LayerNorm.bias

encoder.layer.10.attention.self.value.bias

encoder.layer.4.attention.self.query.weight

encoder.layer.10.attention.output.LayerNorm.weight

encoder.layer.10.attention.output.dense.weight

encoder.layer.10.attention.output.dense.bias

encoder.layer.4.attention.self.key.bias

encoder.layer.10.attention.output.LayerNorm.bias

encoder.layer.4.attention.self.value.weight

encoder.layer.10.intermediate.dense.weight

encoder.layer.4.attention.self.value.bias

encoder.layer.10.intermediate.dense.bias

encoder.layer.4.attention.output.dense.weight

encoder.layer.4.attention.output.dense.bias

encoder.layer.10.output.dense.bias -

encoder.layer.10.output.LayerNorm.weight

encoder.layer.4.attention.output.LayerNorm.bias

encoder.layer.10.output.LayerNorm.bias

encoder.layer.4.intermediate.dense.weight

encoder.layer.4.intermediate.dense.bias

encoder.layer.11.attention.self.query.bias

encoder.layer.4.output.dense.weight

encoder.layer.11.attention.self.key.weight

encoder.layer.4.output.dense.bias

encoder.layer.11.attention.self.key.bias

encoder.layer.4.output.LayerNorm.weight

encoder.layer.11.attention.self.value.weight

encoder.layer.4.output.LayerNorm.bias

encoder.layer.11.attention.self.value.bias

encoder.layer.11.attention.output.dense.weight

encoder.layer.5.attention.self.query.bias

encoder.layer.11.attention.output.dense.bias

encoder.layer.11.attention.output.LayerNorm.weight

encoder.layer.5.attention.self.key.bias

encoder.layer.11.attention.output.LayerNorm.bias

encoder.layer.11.intermediate.dense.weight

encoder.layer.5.attention.self.value.bias

encoder.layer.11.intermediate.dense.bias

encoder.layer.5.attention.output.dense.weight

encoder.layer.11.output.dense.weight

encoder.layer.5.attention.output.dense.bias

encoder.layer.11.output.dense.bias-

encoder.layer.5.attention.output.LayerNorm.weight-

encoder.layer.11.output.LayerNorm.weight

encoder.layer.11.output.LayerNorm.bias

encoder.layer.5.intermediate.dense.weight

encoder.layer.5.intermediate.dense.bias

pooler.dense.weight

pooler.dense.bias

encoder.layer.5.output.dense.weight

taskmodels_dict.copa.head.dense.weight

encoder.layer.5.output.dense.bias

taskmodels_dict.copa.head.dense.bias

encoder.layer.5.output.LayerNorm.weight

taskmodels_dict.copa.head.out_proj.weight

encoder.layer.5.output.LayerNorm.bias

taskmodels_dict.copa.head.out_proj.bias

0.0

0.5

1.0

1.!

0.0

0.5

1.0

1

Updatable Parameters (%) in Modul

Updatable Parameters (%) in Modu

Name

Module

copa ( Layer-wise sparsity pattern)

embeddings.word_embeddings.weight -

embeddings.position_embeddings.weight

encoder.layer.6.attention.selfquery.bias

embeddings.token_type_embeddings.weight

embeddings.LayerNorm.weight

encoder.layer.6.attention.self.key.bias

embeddings.LayerNorm.bias

encoder.layer.0.attention.self.query.weight

encoder.layer.6.attention.self.value.bias

encoder.layer.0.attention.self.query.bias

encoder.layer.6.attention.output.dense.weight

encoder.layer.0.attention.self.key.weight

encoder.layer.6.attention.output.dense.bias

encoder.layer.0.attention.self.key.bias

encoder.layer.0.attention.self.value.weight

encoder.layer.6.attention.output.LayerNorm.bias

encoder.layer.6.intermediate.dense.weight

encoder.layer.6.intermediate.dense.bias

encoder.layer.6.output.dense.weight

encoder.layer.0.attention.output.LayerNorm.weight

encoder.layer.6.output.dense.bias

encoder.layer.0.attention.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.bias

encoder.layer.7.attention.self.query.weight

encoder.layer.6.output.LayerNorm.bias

encoder.layer.0.output.dense.weight

encoder.layer.7.attention.self.query.bias

encoder.layer.0.output.dense.bias

encoder.layer.7.attention.self.key.weight

encoder.layer.0.output.LayerNorm.weight

encoder.layer.7.attention.self.key.bias

encoder.layer.0.output.LayerNorm.bias

encoder.layer.7.attention.self.value.weight

encoder.layer.7.attention.self.value.bias

encoder.layer.1.attention.self.query.bias

encoder.layer.1.attention.self.key.weight

encoder.layer.1.attention.self.key.bias

encoder.layer.7.attention.output.LayerNorm.weight

encoder.layer.7.attention.output.dense.bias

encoder.layer.7.attention.output.LayerNorm.bias

encoder.layer.1.attention.output.dense.weight

encoder.layer.1.attention.self.value.bias

encoder.layer.7.intermediate.dense.bias

encoder.layer.1.attention.output.dense.bias

encoder.layer.7.output.dense.weight

encoder.layer.1.attention.output.LayerNorm.weight

encoder.layer.7.output.dense.bias

encoder.layer.1.intermediate.dense.weight

encoder.layer.1.output.LayerNorm.weight

encoder.layer.1.intermediate.dense.bias

encoder.layer.1.output.dense.weight

encoder.layer.1.output.dense.bias

encoder.layer.8.attention.self.query.bias

encoder.layer.8.attention.self.key.weight

encoder.layer.7.output.LayerNorm.bias

encoder.layer.8.attention.self.key.bias

encoder.layer.2.attention.self.query.weight

encoder.layer.1.output.LayerNorm.bias

encoder.layer.8.attention.self.value.weight

encoder.layer.8.attention.self.value.bias

encoder.layer.2.attention.self.query.bias

encoder.layer.8.attention.output.dense.weight

encoder.layer.2.attention.output.LayerNorm.weight

encoder.layer.2.attention.output.dense.bias

encoder.layer.2.attention.self.value.weight

encoder.layer.2.attention.self.value.bias

encoder.layer.2.attention.self.key.bias

encoder.layer.8.attention.output.LayerNorm.bias

encoder.layer.8.intermediate.dense.weight

encoder.layer.8.output.LayerNorm.weight

encoder.layer.8.intermediate.dense.bias

encoder.layer.8.output.dense.weight

encoder.layer.8.output.dense.bias

encoder.layer.2.attention.output.LayerNorm.bias

encoder.layer.2.intermediate.dense.weight

encoder.layer.2.output.LayerNorm.weight

encoder.layer.2.intermediate.dense.bias

encoder.layer.3.attention.self.key.weight

encoder.layer.3.attention.self.query.bias

encoder.layer.2.output.LayerNorm.bias

encoder.layer.2.output.dense.weight

encoder.layer.2.output.dense.bias

encoder.layer.9.attention.output.dense.weight

encoder.layer.9.attention.output.dense.bias

encoder.layer.9.attention.self.value.weight

encoder.layer.9.attention.self.value.bias

encoder.layer.8.output.LayerNorm.bias

encoder.layer.9.attention.sef.key.bias

encoder.layer.3.attention.output.LayerNorm.weight

encoder.layer.3.attention.output.LayerNorm.bias

encoder.layer.3.attention.output.dense.bias

encoder.layer.3.intermediate.dense.weight

encoder.layer.3.attention.self.value.weight

encoder.layer.3.attention.self.value.bias

encoder.layer.3.attention.self.key.bias

encoder.layer.9.attention.output.LayerNorm.bias

encoder.layer.9.intermediate.dense.bias

encoder.layer.9.output.LayerNorm.bias

encoder.layer.9.output.dense.weight

encoder.layer.9.output.dense.bias

encoder.layer.3.intermediate.dense.bias

encoder.layer.10.attention.self.query.bias

encoder.layer.3.output.dense.weight

encoder.layer.3.output.dense.bias

encoder.layer.10.attention.self.value.weight

encoder.layer.10.attention.self.key.bias

encoder.layer.3.output.LayerNorm.bias

encoder.layer.10.attention.self.value.bias

encoder.layer.4.attention.self.query.bias

encoder.layer.4.attention.self.key.bias

encoder.layer.10.attention.output.LayerNorm.weight

encoder.layer.10.attention.output.LayerNorm.bias

encoder.layer.10.attention.output.dense.bias

encoder.layer.10.intermediate.dense.weight

encoder.layer.4.attention.self.value.bias

encoder.layer.10.intermediate.dense.bias

encoder.layer.10.output.dense.bias

encoder.layer.10.output.LayerNorm.weight

encoder.layer.4.attention.output.LayerNorm.bias

encoder.layer.10.output.LayerNorm.bias

encoder.layer.4.intermediate.dense.weight

encoder.layer.4.intermediate.dense.bias

encoder.layer.4.output.dense.weight

encoder.layer.11.attention.self.key.weight

encoder.layer.4.output.dense.bias

encoder.layer.11.attention.self.key.bias

encoder.layer.4.output.LayerNorm.weight

encoder.layer.11.attention.self.value.weight

encoder.layer.4.output.LayerNorm.bias

encoder.layer.11.attention.self.value.bias

encoder.layer.11.attention.output.dense.weight

encoder.layer.5.attention.self.query.bias

encoder.layer.11.attention.output.dense.bias

encoder.layer.5.attention.self.key.weight

encoder.layer.11.attention.output.LayerNorm.weight

encoder.layer.5.attention.self.key.bias

encoder.layer.11.attention.output.LayerNorm.bias

encoder.layer.5.attention.self.value.weight

encoder.layer.11.intermediate.dense.weight

encoder.layer.5.attention.self.value.bias

encoder.layer.11.intermediate.dense.bias

encoder.layer.11.output.dense.weight

encoder.layer.5.attention.output.dense.bias

encoder.layer.11.output.dense.bias

encoder.layer.5.attention.output.LayerNorm.bias

encoder.layer.11.output.LayerNorm.bias

encoder.layer.5.intermediate.dense.weight

pooler.dense.weight

encoder.layer.5.intermediate.dense.bias

pooler.dense.bias

encoder.layer.5.output.dense.weight

taskmodels_dict.copa.head.dense.weight

encoder.layer.5.output.dense.bias

taskmodels_dict.copa.head.dense.bias

encoder.layer.5.output.LayerNorm.weight

taskmodels_dict.copa.head.out_proj.weight

encoder.layer.5.output.LayerNorm.bias

taskmodels_dict.copa.head.out_proj.bias

0.0

0.2

0.4

0.6

80

0.0

0.2

0.4

0.6

0.8

Updatable Parameters (%) in Layer

Updatable Parameters (%) in Laye

wsc ( Module-wise sparsity pattern)

embeddings.word_embeddings.weight

embeddings.position_embeddings.weight

encoder.layer.6.attention.self.query.bias

embeddings.token_type_embeddings.weight

encoder.layer.6.attention.self.key.weight

embeddings.LayerNorm.weight

encoder.layer.6.attention.self.key.bias

embeddings.LayerNorm.bias

encoder.layer.0.attention.self.query.weight

encoder.layer.6.attention.self.value.bias

encoder.layer.0.attention.self.query.bias

encoder.layer.6.attention.output.dense.weight

encoder.layer.0.attention.self.key.weight

encoder.layer.6.attention.output.dense.bias

encoder.layer.0.attention.self.key.bias

encoder.layer.6.attention.output.LayerNorm.weight

encoder.layer.0.attention.self.value.weight

encoder.layer.6.attention.output.LayerNorm.bias

encoder.layer.6.intermediate.dense.weight

encoder.layer.0.attention.output.dense.weight

encoder.layer.6.intermediate.dense.bias

encoder.layer.0.attention.output.dense.bias

encoder.layer.6.output.dense.weight

encoder.layer.0.attention.output.LayerNorm.weight

encoder.layer.6.output.dense.bias

encoder.layer.0.attention.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.weight

encoder.layer.0.intermediate.dense.bias

encoder.layer.7.attention.self.query.weight

encoder.layer.6.output.LayerNorm.bias

encoder.layer.0.output.dense.weight

encoder.layer.7.attention.self.query.bias

encoder.layer.0.output.dense.bias

encoder.layer.7.attention.self.key.weight

encoder.layer.0.output.LayerNorm.weight

encoder.layer.7.attention.self.value.weight

encoder.layer.1.attention.self.query.weight

encoder.layer.7.attention.self.value.bias

encoder.layer.1.attention.self.query.bias

encoder.layer.1.attention.self.key.weight

encoder.layer.7.attention.output.LayerNorm.weight

encoder.layer.7.attention.output.dense.bias

encoder.layer.1.attention.self.value.weight

encoder.layer.7.attention.output.LayerNorm.bias

encoder.layer.1.attention.self.value.bias

encoder.layer.7.intermediate.dense.bias

encoder.layer.1.attention.output.dense.bias

encoder.layer.7.output.dense.weight

encoder.layer.1.attention.output.LayerNorm.weight

encoder.layer.7.output.dense.bias

encoder.layer.1.attention.output.LayerNorm.bias

encoder.layer.1.intermediate.dense.weight

encoder.layer.1.output.LayerNorm.weight

encoder.layer.1.intermediate.dense.bias

encoder.layer.1.output.dense.weight

encoder.layer.1.output.dense.bias

encoder.layer.7.output.LayerNorm.weight

encoder.layer.8.attention.self.key.weight

encoder.layer.8.attention.self.query.bias

encoder.layer.7.output.LayerNorm.bias

encoder.layer.8.attention.self.key.bias

encoder.layer.2.attention.self.query.weight

encoder.layer.1.output.LayerNorm.bias

encoder.layer.8.attention.self.value.weight

encoder.layer.8.attention.self.value.bias

encoder.layer.2.attention.self.query.bias

encoder.layer.8.attention.output.dense.weight

encoder.layer.2.attention.self.key.weight

encoder.layer.8.attention.output.dense.bias

encoder.layer.2.attention.output.LayerNorm.weight

encoder.layer.2.attention.output.dense.weight

encoder.layer.2.attention.output.dense.bias

encoder.layer.2.attention.self.value.weight

encoder.layer.2.attention.self.value.bias

encoder.layer.2.attention.self.key.bias

encoder.layer.8.attention.output.LayerNorm.bias

encoder.layer.8.intermediate.dense.weight

encoder.layer.8.output.LayerNorm.weight-

encoder.layer.8.intermediate.dense.bias

encoder.layer.8.output.dense.bias

encoder.layer.2.attention.output.LayerNorm.bias

encoder.layer.2.intermediate.dense.weight

encoder.layer.2.output.LayerNorm.weight

encoder.layer.2.intermediate.dense.bias

encoder.layer.3.attention.self.key.weight

encoder.layer.3.attention.self.query.bias

encoder.layer.2.output.LayerNorm.bias

encoder.layer.2.output.dense.weight

encoder.layer.2.output.dense.bias

Mh llu hululht.

encoder.layer.9.attention.output.dense.weight

encoder.layer.9.attention.output.dense.bias

encoder.layer.9.attention.self.value.weight

encoder.layer.9.attention.self.value.bias

encoder.layer.8.output.LayerNorm.bias

encoder.layer.9.attention.self.key.bias

encoder.layer.3.attention.output.LayerNorm.weight

encoder.layer.3.attention.output.dense.bias

encoder.layer.3.attention.self.value.bias

encoder.layer.3.attention.self.key.bias

encoder.layer.9.attention.output.LayerNorm.bias

encoder.layer.9.intermediate.dense.bias

encoder.layer.9.output.dense.weight

encoder.layer.9.output.dense.bias

encoder.layer.3.attention.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.weight

encoder.layer.9.output.LayerNorm.bias

encoder.layer.3.intermediate.dense.bias

encoder.layer.3.output.dense.weight

encoder.layer.10.attention.self.query.bias

encoder.layer.3.output.dense.bias

encoder.layer.10.attention.self.value.weight

encoder.layer.10.attention.self.key.bias

encoder.layer.3.output.LayerNorm.bias

encoder.layer.10.attention.self.value.bias

encoder.layer.4.attention.self.query.weight

encoder.layer.4.attention.self.query.bias

encoder.layer.10.attention.output.LayerNorm.weight

encoder.layer.10.attention.output.dense.weight

encoder.layer.10.attention.output.dense.bias

encoder.layer.4.attention.self.value.weight

encoder.layer.4.attention.self.key.bias

encoder.layer.10.intermediate.dense.weight

encoder.layer.4.attention.self.value.bias

encoder.layer.10.intermediate.dense.bias

encoder.layer.4.attention.output.dense.bias

encoder.layer.10.output.dense.bias

encoder.layer.10.output.LayerNorm.weight

encoder.layer.4.attention.output.LayerNorm.bias

encoder.layer.10.output.LayerNorm.bias

encoder.layer.4.intermediate.dense.weight

encoder.layer.4.intermediate.dense.bias

encoder.layer.11.attention.self.query.bias

encoder.layer.4.output.dense.weight

encoder.layer.11.attention.self.key.weight

encoder.layer.4.output.dense.bias

encoder.layer.11.attention.self.key.bias

encoder.layer.4.output.LayerNorm.weight

encoder.layer.11.attention.self.valu.weigh

encoder.layer.4.output.LayerNorm.bias

encoder.layer.11.attention.self.value.bias

encoder.layer.5.attention.self.query.bias

encoder.layer.11.attention.output.dense.bias

encoder.layer.5.attention.self.key.weight

encoder.layer.5.attention.self.key.bias

encoder.layer.11.attention.output.LayerNorm.bias

encoder.layer.11.intermediate.dense.weight

encoder.layer.5.attention.self.value.bias

encoder.layer.11.intermediate.dense.bias

encoder.layer.5.attention.output.dense.weight

encoder.layer.11.output.dense.weight

encoder.layer.5.attention.output.dense.bias

encoder.layer.11.output.dense.bias

encoder.layer.11.output.LayerNorm.weight

encoder.layer.5.attention.output.LayerNorm.bias

encoder.layer.11.output.LayerNorm.bias

encoder.layer.5.intermediate.dense.weight -

pooler.dense.weight

encoder.layer.5.intermediate.dense.bias

pooler.dense.bias

encoder.layer.5.output.dense.weight

taskmodels_dict.wsc.head.attention.weight

encoder.layer.5.output.dense.bias

taskmodels_dict.wsc.head.attention.bias

encoder.layer.5.output.LayerNorm.weight

taskmodels_dict.wsc.head.classifier.weight

encoder.layer.5.output.LayerNorm.bias

taskmodels_dict.wsc.head.classifier.bias

0.00

0.25

0.500.751.00

0.000.250.500.751.00

Updatable Parameters (%) in Modul

Updatable Parameters (%) in Modu

Name

Module

wsc ( Layer-wise sparsity pattern)

embeddings.word_embeddings.weight -

embeddings.position_embeddings.weight

encoder.layer.6.attention.self.query.bias

embeddings.token_type_embeddings.weight

embeddings.LayerNorm.weight

encoder.layer.6.attention.self.key.bias

embeddings.LayerNorm.bias

encoder.layer.0.attention.self.query.weight

encoder.layer.6.attention.self.value.bias

encoder.layer.0.attention.self.query.bias

encoder.layer.6.attention.output.dense.weight

encoder.layer.0.attention.self.key.weight

encoder.layer.6.attention.output.dense.bias

encoder.layer.0.attention.self.key.bias

encoder.layer.0.attention.self.value.weight

encoder.layer.6.attention.output.LayerNorm.bias

encoder.layer.6.intermediate.dense.weight

encoder.layer.6.intermediate.dense.bias

encoder.layer.6.output.dense.weight

encoder.layer.0.attention.output.LayerNorm.weight

encoder.layer.6.output.dense.bias

encoder.layer.0.attention.output.LayerNorm.bias

encoder.layer.0.intermediate.dense.bias

encoder.layer.7.attention.self.query.weight

encoder.layer.6.output.LayerNorm.bias

encoder.layer.0.output.dense.weight

encoder.layer.7.attention.self.query.bias

encoder.layer.0.output.dense.bias

encoder.layer.7.attention.self.key.weight

encoder.layer.0.output.LayerNorm.weight

encoder.layer.7.attention.self.key.bias

encoder.layer.0.output.LayerNorm.bias

encoder.layer.7.attention.self.value.weight

encoder.layer.7.attention.self.value.bias

encoder.layer.1.attention.self.query.bias

encoder.layer.1.attention.self.key.weight

encoder.layer.1.attention.self.key.bias

encoder.layer.7.attention.output.LayerNorm.weight

encoder.layer.7.attention.output.dense.bias

encoder.layer.7.attention.output.LayerNorm.bias

encoder.layer.1.attention.output.dense.weight

encoder.layer.1.attention.self.value.bias

encoder.layer.7.intermediate.dense.bias

encoder.layer.1.attention.output.dense.bias

encoder.layer.7.output.dense.weight

encoder.layer.1.attention.output.LayerNorm.weight

encoder.layer.7.output.dense.bias

encoder.layer.1.intermediate.dense.weight

encoder.layer.1.output.LayerNorm.weight

encoder.layer.1.intermediate.dense.bias

encoder.layer.1.output.dense.weight

encoder.layer.1.output.dense.bias

encoder.layer.8.attention.self.query.bias

encoder.layer.8.attention.self.key.weight

encoder.layer.7.output.LayerNorm.bias

encoder.layer.8.attention.self.key.bias

encoder.layer.2.attention.self.query.weight

encoder.layer.1.output.LayerNorm.bias

encoder.layer.8.attention.self.value.weight

encoder.layer.8.attention.self.value.bias

encoder.layer.2.attention.self.query.bias

encoder.layer.8.attention.output.dense.weight

encoder.layer.2.attention.output.LayerNorm.weight

encoder.layer.2.attention.output.dense.bias

encoder.layer.2.attention.self.value.weight

encoder.layer.2.attention.self.value.bias

encoder.layer.2.attention.self.key.bias

encoder.layer.8.attention.output.LayerNorm.bias

encoder.layer.8.intermediate.dense.weight

encoder.layer.8.output.LayerNorm.weight

encoder.layer.8.intermediate.dense.bias

encoder.layer.8.output.dense.weight

encoder.layer.8.output.dense.bias

encoder.layer.2.attention.output.LayerNorm.bias

encoder.layer.2.intermediate.dense.weight

encoder.layer.2.output.LayerNorm.weight

encoder.layer.2.intermediate.dense.bias

encoder.layer.3.attention.self.key.weight

encoder.layer.3.attention.self.query.bias

encoder.layer.2.output.LayerNorm.bias

encoder.layer.2.output.dense.weight

encoder.layer.2.output.dense.bias

encoder.layer.9.attention.output.dense.weight

encoder.layer.9.attention.output.dense.bias

encoder.layer.9.attention.self.value.weight

encoder.layer.9.attention.self.value.bias

encoder.layer.8.output.LayerNorm.bias

encoder.layer.9.attention.self.key.bias

encoder.layer.3.attention.output.LayerNorm.weight

encoder.layer.3.attention.output.LayerNorm.bias

encoder.layer.3.attention.output.dense.bias

encoder.layer.3.intermediate.dense.weight

encoder.layer.3.attention.self.value.weight

encoder.layer.3.attention.self.value.bias

encoder.layer.3.intermediate.dense.bias

encoder.layer.3.attention.self.key.bias

encoder.layer.9.attention.output.LayerNorm.bias

encoder.layer.10.attention.self.query.weight

encoder.layer.9.intermediate.dense.weight

encoder.layer.10.attention.self.query.bias

encoder.layer.9.intermediate.dense.bias

encoder.layer.9.output.LayerNorm.bias

encoder.layer.9.output.dense.weight

encoder.layer.9.output.dense.bias

encoder.layer.3.output.dense.weight

encoder.layer.3.output.dense.bias

encoder.layer.10.attention.self.value.weight

encoder.layer.10.attention.self.key.bias

encoder.layer.4.attention.self.query.bias

encoder.layer.3.output.LayerNorm.bias

encoder.layer.10.attention.output.LayerNorm.weight

encoder.layer.10.attention.output.dense.bias

encoder.layer.10.attention.self.value.bias

encoder.layer.4.attention.self.key.bias

encoder.layer.10.attention.output.LayerNorm.bias

encoder.layer.10.intermediate.dense.weight

encoder.layer.4.attention.self.value.bias

encoder.layer.10.intermediate.dense.bias

encoder.layer.10.output.dense.bias

encoder.layer.10.output.LayerNorm.weight

encoder.layer.4.attention.output.LayerNorm.bias

encoder.layer.10.output.LayerNorm.bias

encoder.layer.4.intermediate.dense.weight

encoder.layer.4.intermediate.dense.bias

encoder.layer.4.output.dense.weight

encoder.layer.11.attention.self.key.weight

encoder.layer.4.output.dense.bias

encoder.layer.11.attention.self.key.bias

encoder.layer.4.output.LayerNorm.weight

encoder.layer.11.attention.self.value.weight

encoder.layer.4.output.LayerNorm.bias

encoder.layer.11.attention.self.value.bias

encoder.layer.11.attention.output.dense.weight

encoder.layer.5.attention.self.query.bias

encoder.layer.11.attention.output.dense.bias

encoder.layer.5.attention.self.key.weight

encoder.layer.11.attention.output.LayerNorm.weight

encoder.layer.5.attention.self.key.bias

encoder.layer.11.attention.output.LayerNorm.bias

encoder.layer.5.attention.self.value.weight

encoder.layer.11.intermediate.dense.weight

encoder.layer.5.attention.self.value.bias

encoder.layer.11.intermediate.dense.bias

encoder.layer.5.attention.output.dense.weight

encoder.layer.11.output.dense.weight

encoder.layer.5.attention.output.dense.bias

encoder.layer.11.output.dense.bias

encoder.layer.5.attention.output.LayerNorm.bias

encoder.layer.11.output.LayerNorm.bias

encoder.layer.5.intermediate.dense.weight

pooler.dense.weight

encoder.layer.5.intermediate.dense.bias

pooler.dense.bias

encoder.layer.5.output.dense.weight

taskmodels_dict.wsc.head.attention.weight

encoder.layer.5.output.dense.bias

taskmodelsdict.wsc.head.attention.bias

encoder.layer.5.output.LayerNorm.weight

taskmodels_dict.wsc.head.classifier.weight

encoder.layer.5.output.LayerNorm.bias

taskmodels_dict.wsc.head.classifier.bias

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

80

Updatable Parameters (%) in Layer

Updatable Parameters (%) in Laye

Name

cifar100

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed

visual_prompt_token_pos_embed-

blocks.0.visual_prompt_token -

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias

blocks.1.visual_prompt_token

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight -

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.bias -

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.weight

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt.token

blocks.2.visual_prompt_token-

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_a.weight -

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight -

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.ln1.weight

blocks.4.adapter.In1.weight -

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias -

blocks.4.adapter.ln2.weight -

blocks.4.adapter.ln2.weight -

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight -

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight -

blocks.5.adapter.ln1.weight -

blocks.5.adapter.ln1.weight -

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias -

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_a.weight

blocks.6.attn.LoRA_b.weight-

blocks.6.attn.LoRA_b.weight -

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.bias

blocks.6.adapter.In2.bias -

blocks.7.visual_prompt_token -

blocks.7.visual_prompt_token -

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_b.weight

blocks.7.attn.LoRA_b.weight-

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.weight

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias -

blocks.7.adapter.ln2.weight -

blocks.7.adapter.In2.weight

blocks.7.adapter.ln2.bias

blocks.7.adapter.ln2.bias

blocks.8.visual_prompt_token -

blocks.8.visual_prompt_token

blocks.8.attn.LoRA_a.weight

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_b.weight-

blocks.8.attn.LoRA_b.weight-

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.weight -

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight -

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight

blocks.10.attn.LoRA_b.weight -

blocks.10.adapter.ln1.weight·

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.weight -

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token -

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

1015202530

5

4

6

8

10

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

Module

Name

Module

caltech101

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed

visual_prompt_token_pos_embed

blocks.0.visual_prompt_token -

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight

blocks.0.attn.LoRA_a.weight

blocks.0.attn.LoRA_b.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias

blocks.1.visual_prompt_token -

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight -

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.bias -

blocks.1.adapter.ln1.bias -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.weight

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt.token

blocks.2.visual_prompt.token

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight -

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.ln1.weight

blocks.4.adapter.In1.weight -

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias -

blocks.4.adapter.ln2.weight -

blocks.4.adapter.ln2.weight -

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token·

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight -

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight -

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight -

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token·

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_b.weight-

blocks.6.attn.LoRA_b.weight -

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.bias -

blocks.6.adapter.ln2.bias -

blocks.7.visual_prompt_token

blocks.7.visual_prompt_token

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_b.weight -

blocks.7.attn.LoRA_b.weight-

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias -

blocks.7.adapter.ln2.weight -

blocks.7.adapter.In2.weight

blocks.7.adapter.ln2.bias -

blocks.7.adapter.ln2.bias

blocks.8.visual_prompt_token

blocks.8.visual_prompt_token

blocks.8.attn.LoRA_a.weight-

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_b.weight -

blocks.8.attn.LoRA_b.weight-

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight -

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight -

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight

blocks.10.attn.LoRA_b.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln2.weight

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

5

10

15

2025

30

0.0

2.55.07.510.0 12.5 15.0 17.5 20.0

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

Name

Module

Updatable Parameters (%) in Module

dtd

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed

visual_prompt_token_pos_embed

blocks.0.visual_prompt_token -

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.bias ·

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias

blocks.1.visual_prompt_token -

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight -

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.weight

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln1.bias -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt_token

blocks.2.visual_prompt.token-

blocks.2.attn.LoRA_a.weight -

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight -

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias -

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token -

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight -

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.weight-

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln2.weight -

blocks.4.adapter.ln2.weight

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias

blocks.5.visual_prompt_token·

blocks.5.visual_prompt_token -

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight -

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight -

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token -

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_b.weight

blocks.6.attn.LoRA_b.weight-

blocks.6.adapter.ln1.weight -

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias -

blocks.6.adapter.ln1.bias -

blocks.6.adapter.lIn2.weight -

blocks.6.adapter.ln2.weight -

blocks.6.adapter.ln2.bias -

blocks.6.adapter.ln2.bias -

blocks.7.visual_prompt_token-

blocks.7.visual_prompt_token-

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_b.weight-

blocks.7.attn.LoRA_b.weight -

blocks.7.adapter.ln1.weight

blocks.7.adapter.ln1.weight

blocks.7.adapter.ln1.bias-

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln2.weight

blocks.7.adapter.ln2.weight

blocks.7.adapter.ln2.bias -

blocks.7.adapter.ln2.bias

blocks.8.visual_prompt_token

blocks.8.visual_prompt_token

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_a.weight

blocks.8.attn.LoRA_b.weight

blocks.8.attn.LoRA_b.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.In2.bias

blocks.9.visual_prompt_token·

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight-

blocks.9.attn.LoRA_b.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight

blocks.10.attn.LoRA_b.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.weight

blocks.10.adapter.ln2.bias

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

2.5

5.07.5 10.0 12.5 15.0 17.5 20.0

1

2

3

4

5

Updatable Parameters (%) in Layer

flower102

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed

visual_prompt_token_pos_embed

blocks.0.visual_prompt_token -

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight-

blocks.0.attn.LoRA_a.weight-

blocks.0.attn.LoRA_b.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias

blocks.1.visual_prompt_token -

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.bias -

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt.token

blocks.2.visual_prompt_token-

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token-

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.ln1.weight

blocks.4.adapter.In1.weight -

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln2.weight

blocks.4.adapter.ln2.weight -

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight -

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_b.weight

blocks.6.attn.LoRA_b.weight -

blocks.6.adapter.ln1.weight -

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln2.weight -

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.bias

blocks.6.adapter.ln2.bias -

blocks.7.visual_prompt_token

blocks.7.visual_prompt_token

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_b.weight

blocks.7.attn.LoRA_b.weight-

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln2.weight -

blocks.7.adapter.ln2.weight -

blocks.7.adapter.ln2.bias

blocks.7.adapter.ln2.bias

blocks.8.visual_prompt_token

blocks.8.visual_prompt_token

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_b.weight-

blocks.8.attn.LoRA_b.weight-

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight

blocks.9.adapter.n1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight

blocks.10.attn.LoRA_b.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight-

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

0

10

20

30

40506070

80

10

20

30

40

50

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

pets

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed

visual_prompt_token_pos_embed

blocks.0.visual_prompt_token

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight-

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.attn.LoRA_b.weight

blocks.0.adapter.ln1.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias

blocks.1.visual_prompt_token ·

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.weight

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln1.bias -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt_token

blocks.2.visual_prompt.token-

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.n1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token -

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight -

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias

blocks.4.adapter.n2.weight

blocks.4.adapter.ln2.weight

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token

blocks.5.visual_prompt_token -

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight -

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token -

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_a.weight

blocks.6.attn.LoRA_b.weight

blocks.6.attn.LoRA_b.weight-

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias -

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.weight -

blocks.6.adapter.ln2.bias -

blocks.6.adapter.ln2.bias -

blocks.7.visual_prompt_token-

blocks.7.visual_prompt_token-

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_b.weight -

blocks.7.attn.LoRA_b.weight

blocks.7.adapter.ln1.weight

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.bias-

blocks.7.adapter.ln1.bias

blocks.7.adapter.n2.weight

blocks.7.adapter.ln2.weight

blocks.7.adapter.ln2.bias -

blocks.7.adapter.ln2.bias

blocks.8.visual_prompt_token-

blocks.8.visual_prompt_token-

blocks.8.attn.LoRA_a.weight

blocks.8.attn.LoRA_a.weight

blocks.8.attn.LoRA_b.weight

blocks.8.attn.LoRA_b.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight

blocks.10.attn.LoRA_b.weight-

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.weight

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.weight

blocks.10.adapter.ln2.bias

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_.token

blocks.11.visual_prompt_token

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight -

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

10

2025303

35

40

0.0

2.5

5.07.5 10.0 12.5 15.0 17.5 20.0

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

Name

Module

Updatable Parameters (%) in Module

svhn

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed

visual_prompt_token_pos_embed-

blocks.0.visual_prompt_token

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.attn.LoRA_b.weight

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias -

blocks.1.visual_prompt_token -

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight-

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln1.bias -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt_token

blocks.2.visual_prompt.token

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight -

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt.token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.n1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token -

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight -

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln2.weight -

blocks.4.adapter.ln2.weight

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight-

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token -

blocks.6.visual_prompt_token

blocks.6.attn.LoRA_a.weight

blocks.6.attn.LoRA_a.weight

blocks.6.attn.LoRA_b.weight -

blocks.6.attn.LoRA_b.weight-

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias -

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.weight -

blocks.6.adapter.ln2.bias

blocks.6.adapter.ln2.bias -

blocks.7.visual_prompt.token-

blocks.7.visual_prompt_token

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_b.weight

blocks.7.attn.LoRA_b.weight -

blocks.7.adapter.ln1.weight

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln2.weight

blocks.7.adapter.ln2.weight

blocks.7.adapter.ln2.bias

blocks.7.adapter.ln2.bias

blocks.8.visual_prompt_token

blocks.8.visual_prompt_token

blocks.8.attn.LoRA_a.weight-

blocks.8.attn.LoRA_a.weight

blocks.8.attn.LoRA_b.weight

blocks.8.attn.LoRA_b.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.lIn2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight -

blocks.10.attn.LoRA_b.weight-

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.bias

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight -

blocks.11l.adapter.ln1.bias

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

0

20

40

60

80

100

10

30

40

50

60

70

80

Updatable Parameters (%) in Layer

Name

sun397

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed -

visual_prompt_token_pos_embed-

blocks.0.visual_prompt_token

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_a.weight-

blocks.0.attn.LoRA_b.weight

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias

blocks.1.visual_prompt_token -

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight -

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln1.bias -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.weight

blocks.1.adapter.ln2.bias -

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt.token-

blocks.2.visual_prompt_token-

blocks.2.attn.LoRA_a.weight-

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight -

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight -

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.ln1.weight

blocks.4.adapter.In1.weight -

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias -

blocks.4.adapter.ln2.weight -

blocks.4.adapter.ln2.weight -

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight -

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight -

blocks.5.adapter.ln1.weight -

blocks.5.adapter.ln1.weight -

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight -

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token·

blocks.6.visual_prompt_token·

blocks.6.attn.LoRA_a.weight

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_b.weight-

blocks.6.attn.LoRA_b.weight -

blocks.6.adapter.ln1.weight -

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln2.weight -

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.bias

blocks.6.adapter.ln2.bias

blocks.7.visual_prompt_token-

blocks.7.visual_prompt_token

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_b.weight

blocks.7.attn.LoRA_b.weight-

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.weight

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias -

blocks.7.adapter.ln2.weight -

blocks.7.adapter.In2.weight

blocks.7.adapter.ln2.bias

blocks.7.adapter.ln2.bias

blocks.8.visual_prompt_token

blocks.8.visual_prompt_token

blocks.8.attn.LoRA_a.weight-

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_b.weight-

blocks.8.attn.LoRA_b.weight-

blocks.8.adapter.ln1.weight -

blocks.8.adapter.ln1.weight -

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight -

blocks.9.adapter.n1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight-

blocks.10.attn.LoRA_b.weight

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.weight

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln2.weight-

blocks.10.adapter.ln2.weight -

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token

blocks.11.attn.LoRA_a.weight -

blocks.11.attn.LoRA_a.weight -

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

0.02.55.0 7.5 10.0 12.5 15.0 17.5 20.0

6

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

Module

Name

Module

camelyon

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed ·

visual_prompt_token_pos_embed

blocks.0.visual_prompt_token

blocks.0.visual_prompt_token-

blocks.0.attn.LoRA_a.weight

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_b.weight-

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.weight -

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias -

blocks.1.visual_prompt_token

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight -

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln1.bias -

blocks.1.adapter.ln2.weight

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt_token

blocks.2.visual_prompt_token

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_a.weight -

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight

blocks.2.adapter.ln1.weight -

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias -

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_.token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias -

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight -

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias -

blocks.4.adapter.In2.weight

blocks.4.adapter.In2.weight

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token-

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_b.weight-

blocks.5.attn.LoRA_b.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token

blocks.6.attn.LoRA_a.weight

blocks.6.attn.LoRA_a.weight

blocks.6.attn.LoRA_b.weight

blocks.6.attn.LoRA_b.weight -

blocks.6.adapter.ln1.weight -

blocks.6.adapter.In1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.n1.bias

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.weight -

blocks.6.adapter.ln2.bias

blocks.6.adapter.ln2.bias

blocks.7.visual_prompt_token

blocks.7.visual_prompt_token -

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_b.weight-

blocks.7.attn.LoRA_b.weight -

blocks.7.adapter.ln1.weight

blocks.7.adapter.ln1.weight

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias -

blocks.7.adapter.ln2.weight -

blocks.7.adapter.In2.weight -

blocks.7.adapter.ln2.bias -

blocks.7.adapter.ln2.bias -

blocks.8.visual_prompt_token

blocks.8.visual_prompt_token -

blocks.8.attn.LoRA_a.weight

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_b.weight-

blocks.8.attn.LoRA_b.weight -

blocks.8.adapter.ln1.weight

blocks.8.adapter.In1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight -

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight-

blocks.10.attn.LoRA_b.weight -

blocks.10.adapter.ln1.weight

blocks.10.adapter.ln1.weight-

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight -

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight -

blocks.11.adapter.ln1.bias -

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

0

20

40

60

80

10

OE

40

50

09

70

80

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

Name alnpo

eurosat

visual_prompt_token

visual_prompt_token-

visual_prompt_token_pos_embed

visual_prompt_.token_pos_embed -

blocks.0.visual_prompt_token

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight

blocks.0.attn.LoRA_a.weight

blocks.0.attn.LoRA_b.weight

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.weight -

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias -

blocks.1.visual_prompt_token

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight

blocks.1.attn.LoRA_a.weight

blocks.1.attn.LoRA_b.weight

blocks.1.attn.LoRA_b.weight

blocks.1.adapter.ln1.weight

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln2.weight

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt_token

blocks.2.visual_prompt_token

blocks.2.attn.LoRA_a.weight -

blocks.2.attn.LoRA_a.weight -

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight -

blocks.2.adapter.ln1.weight -

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.n2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias -

blocks.4.adapter.ln2.weight

blocks.4.adapter.In2.weight

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight -

blocks.5.attn.LoRA_b.weight-

blocks.5.attn.LoRA_b.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token

blocks.6.attn.LoRA_a.weight

blocks.6.attn.LoRA_a.weight

blocks.6.attn.LoRA_b.weight-

blocks.6.attn.LoRA_b.weight

blocks.6.adapter.ln1.weight -

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias -

blocks.6.adapter.ln2.weight -

blocks.6.adapter.ln2.weight -

blocks.6.adapter.ln2.bias

blocks.6.adapter.ln2.bias

blocks.7.visual_prompt_token

blocks.7.visual_prompt_token

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_b.weight

blocks.7.attn.LoRA_b.weight -

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias -

blocks.7.adapter.ln2.weight

blocks.7.adapter.ln2.weight -

blocks.7.adapter.ln2.bias -

blocks.7.adapter.ln2.bias -

blocks.8.visual_prompt_token-

blocks.8.visual_prompt_token

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_b.weight

blocks.8.attn.LoRA_b.weight-

blocks.8.adapter.ln1.weight -

blocks.8.adapter.ln1.weight -

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias -

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight

blocks.10.attn.LoRA_b.weight

blocks.10.adapter.ln1.weight-

blocks.10.adapter.ln1.weight-

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln2.weight

blocks.10.adapter.ln2.weight

blocks.10.adapter.ln2.bias

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token -

blocks.11.visual_prompt_token-

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight -

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight -

blocks.11.adapter.ln1.weight -

blocks.11.adapter.ln1.bias -

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

0

20

40

60

80

100

0

10

2030

405

506070

80

Updatable Parameters (%) in Layer

Updatable Parameters (%) in Module

Name

Module

resisc45

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed

visual_prompt_token_pos_embed-

blocks.0.visual_prompt_token -

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias -

blocks.1.visual_prompt_token

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight-

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.weight

blocks.1.adapter.ln1.bias -

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt.token

blocks.2.visual_prompt.token-

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token -

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight -

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.ln1.weight

blocks.4.adapter.In1.weight -

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias -

blocks.4.adapter.ln2.weight

blocks.4.adapter.In2.weight -

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight -

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_a.weight

blocks.6.attn.LoRA_b.weight-

blocks.6.attn.LoRA_b.weight -

blocks.6.adapter.ln1.weight -

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln2.weight -

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.bias

blocks.6.adapter.ln2.bias

blocks.7.visual_prompt_token

blocks.7.visual_prompt_token -

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_b.weight -

blocks.7.attn.LoRA_b.weight-

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias -

blocks.7.adapter.ln2.weight -

blocks.7.adapter.ln2.weight -

blocks.7.adapter.ln2.bias -

blocks.7.adapter.ln2.bias

blocks.8.visual_prompt_token

blocks.8.visual_prompt_token

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_b.weight-

blocks.8.attn.LoRA_b.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight

blocks.9.adapter.n1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias -

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight

blocks.10.attn.LoRA_b.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln2.weight

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight -

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

10

20

30

50

09

70

10

20

30

40

50

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

Name

Module

retinopathy

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed -

visual_prompt_token_pos_embed -

blocks.0.visual_prompt_token

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight

blocks.0.attn.LoRA_a.weight

blocks.0.attn.LoRA_b.weight

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight -

blocks.0.adapter.ln1.weight -

blocks.0.adapter.ln1.bias

blocks.0.adapter.In1.bias -

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias

blocks.1.visual_prompt_token -

blocks.1.visual_prompt_token

blocks.1.attn.LoRA_a.weight

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight -

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln1.bias -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt_token

blocks.2.visual_prompt_token -

blocks.2.attn.LoRA_a.weight-

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight

blocks.2.adapter.ln1.weight -

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias -

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_.token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight -

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.n1.weight

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias

blocks.4.adapter.n2.weight

blocks.4.adapter.In2.weight

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token-

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight -

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token-

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_b.weight-

blocks.6.attn.LoRA_b.weight -

blocks.6.adapter.lIn1.weight

blocks.6.adapter.In1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias -

blocks.6.adapter.ln2.weight -

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.bias

blocks.6.adapter.ln2.bias -

blocks.7.visual_prompt_token -

blocks.7.visual_prompt_token -

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_b.weight -

blocks.7.attn.LoRA_b.weight -

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.bias -

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln2.weight -

blocks.7.adapter.ln2.weight

blocks.7.adapter.ln2.bias

blocks.7.adapter.ln2.bias -

blocks.8.visual_prompt_token

blocks.8.visual_prompt_token -

blocks.8.attn.LoRA_a.weight

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_b.weight

blocks.8.attn.LoRA_b.weight-

blocks.8.adapter.ln1.weight -

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight -

blocks.9.adapter.ln1.weight

blocks.9.adapter.In1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight-

blocks.10.attn.LoRA_b.weight

blocks.10.adapter.ln1.weight

blocks.10.adapter.ln1.weight

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln2.weight

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.bias

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight-

blocks.11.adapter.ln2.weight -

blocks.11.adapter.ln2.weight -

0

5

10

15

20

25

30

2

4

6

8

10

12

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

Name

Module

clevr-count

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed

visual_prompt_token_pos_embed-

blocks.0.visual_prompt_token -

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_a.weight-

blocks.0.attn.LoRA_b.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias

blocks.1.visual_prompt_token -

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight

blocks.1.attn.LoRA_b.weight

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.weight

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt.token

blocks.2.visual_prompt_token-

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight -

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.ln1.weight

blocks.4.adapter.In1.weight -

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias -

blocks.4.adapter.ln2.weight -

blocks.4.adapter.ln2.weight -

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias

blocks.5.visual_prompt_token

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight -

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight -

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight -

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token·

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_b.weight-

blocks.6.attn.LoRA_b.weight -

blocks.6.adapter.ln1.weight -

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.bias

blocks.6.adapter.ln2.bias -

blocks.7.visual_prompt_token

blocks.7.visual_prompt_token -

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_b.weight -

blocks.7.attn.LoRA_b.weight-

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln2.weight -

blocks.7.adapter.ln2.weight -

blocks.7.adapter.ln2.bias -

blocks.7.adapter.ln2.bias

blocks.8.visual_prompt_token

blocks.8.visual_prompt_token

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_b.weight-

blocks.8.attn.LoRA_b.weight-

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight -

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias -

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight

blocks.10.attn.LoRA_b.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln2.weight

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

10

70

80

10

20

30

40

50

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

Name

clevr-dist

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed

visual_prompt_token_pos_embed-

blocks.0.visual_prompt_token ·

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight

blocks.0.attn.LoRA_a.weight

blocks.0.attn.LoRA_b.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias

blocks.1.visual_prompt_token

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight -

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.weight

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt.token

blocks.2.visual_prompt.token-

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight

blocks.4.adapter.ln1.weight

blocks.4.adapter.In1.weight -

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias -

blocks.4.adapter.ln2.weight

blocks.4.adapter.ln2.weight -

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight -

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight -

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token·

blocks.6.visual_prompt_token

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_a.weight

blocks.6.attn.LoRA_b.weight-

blocks.6.attn.LoRA_b.weight -

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln2.weight -

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.bias

blocks.6.adapter.ln2.bias -

blocks.7.visual_prompt_token-

blocks.7.visual_prompt_token

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_b.weight -

blocks.7.attn.LoRA_b.weight-

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln2.weight -

blocks.7.adapter.ln2.weight -

blocks.7.adapter.ln2.bias -

blocks.7.adapter.ln2.bias

blocks.8.visual_prompt_token

blocks.8.visual_prompt_token

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_b.weight-

blocks.8.attn.LoRA_b.weight-

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight

blocks.10.attn.LoRA_b.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln2.weight

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

0

10

20

OE

40

50

5

15

20

253

OE

35

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

Module

Name

Module

dmlab

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed

visual_prompt_token_pos_embed

blocks.0.visual_prompt_token -

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight-

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias -

blocks.1.visual_prompt_token -

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight-

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.weight

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln1.bias -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt_token

blocks.2.visual_prompt.token

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.n1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias -

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token -

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight -

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln2.weight

blocks.4.adapter.ln2.weight -

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight-

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token -

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_b.weight-

blocks.6.attn.LoRA_b.weight

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias -

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.weight -

blocks.6.adapter.ln2.bias

blocks.6.adapter.ln2.bias -

blocks.7.visual_prompt_token

blocks.7.visual_prompt_token

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_b.weight

blocks.7.attn.LoRA_b.weight

blocks.7.adapter.ln1.weight

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln2.weight

blocks.7.adapter.ln2.weight

blocks.7.adapter.ln2.bias

blocks.7.adapter.ln2.bias

blocks.8.visual_prompt_token

blocks.8.visual_prompt_token

blocks.8.attn.LoRA_a.weight

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_b.weight

blocks.8.attn.LoRA_b.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight

blocks.10.attn.LoRA_b.weight-

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.bias

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token -

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight -

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

0

20

40

60

80

100

10

OE

40

50

09

70

80

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

Name

Module

kitti

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed

visual_prompt_token_pos_embed

blocks.0.visual_prompt_token -

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias -

blocks.1.visual_prompt_token -

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight

blocks.1.attn.LoRA_b.weight

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln1.bias -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.weight

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt_token

blocks.2.visual_prompt_.token

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight -

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.n1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias -

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token -

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight -

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight-

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.weight-

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln2.weight

blocks.4.adapter.ln2.weight

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight -

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight -

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token -

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_b.weight

blocks.6.attn.LoRA_b.weight-

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias -

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.bias -

blocks.6.adapter.ln2.bias -

blocks.7.visual_prompt_token

blocks.7.visual_prompt_token

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_b.weight

blocks.7.attn.LoRA_b.weight

blocks.7.adapter.ln1.weight

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln2.weight

blocks.7.adapter.ln2.weight -

blocks.7.adapter.ln2.bias -

blocks.7.adapter.ln2.bias

blocks.8.visual_prompt_token-

blocks.8.visual_prompt_token

blocks.8.attn.LoRA_a.weight

blocks.8.attn.LoRA_a.weight

blocks.8.attn.LoRA_b.weight

blocks.8.attn.LoRA_b.weight -

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.lIn2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight-

blocks.9.attn.LoRA_b.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight

blocks.10.attn.LoRA_b.weight-

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.weight

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.weight

blocks.10.adapter.ln2.bias

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token -

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight -

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

40

60

0

80

100

10

OE

40

50

60

80

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

Name

Module

dsprite-loc

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed -

visual_prompt_token_pos_embed -

blocks.0.visual_prompt_token

blocks.0.visual_prompt_token-

blocks.0.attn.LoRA_a.weight

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_b.weight-

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.bias

blocks.0.adapter.In1.bias -

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias -

blocks.1.visual_prompt_token·

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight -

blocks.1.attn.LoRA_b.weight

blocks.1.adapter.ln1.weight

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln1.bias -

blocks.1.adapter.ln2.weight

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt_token

blocks.2.visual_prompt_token

blocks.2.attn.LoRA_a.weight -

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight -

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias -

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias -

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight -

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias

blocks.4.adapter.In2.weight

blocks.4.adapter.In2.weight

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token-

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_b.weight-

blocks.5.attn.LoRA_b.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token-

blocks.6.attn.LoRA_a.weight

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_b.weight

blocks.6.attn.LoRA_b.weight

blocks.6.adapter.ln1.weight -

blocks.6.adapter.In1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias -

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.weight -

blocks.6.adapter.ln2.bias

blocks.6.adapter.ln2.bias

blocks.7.visual_prompt_token-

blocks.7.visual_prompt_token -

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_b.weight

blocks.7.attn.LoRA_b.weight -

blocks.7.adapter.ln1.weight

blocks.7.adapter.ln1.weight

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias -

blocks.7.adapter.ln2.weight -

blocks.7.adapter.ln2.weight

blocks.7.adapter.ln2.bias -

blocks.7.adapter.ln2.bias -

blocks.8.visual_prompt_token

blocks.8.visual_prompt_token -

blocks.8.attn.LoRA_a.weight-

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_b.weight-

blocks.8.attn.LoRA_b.weight -

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight -

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight-

blocks.10.attn.LoRA_b.weight -

blocks.10.adapter.ln1.weight

blocks.10.adapter.ln1.weight

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight-

blocks.11.adapter.ln1.bias -

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

0

20

40

60

80

100

20

40

60

80

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

Name

Module

dsprite-ori

visual_prompt_token

visual_prompt_token -

visual_prompt_token_pos_embed -

visual_prompt_token_pos_embed -

blocks.0.visual_prompt_.token

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight

blocks.0.attn.LoRA_a.weight

blocks.0.attn.LoRA_b.weight

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias -

blocks.1.visual_prompt_token

blocks.1.visual_prompt_token

blocks.1.attn.LoRA_a.weight

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight

blocks.1.attn.LoRA_b.weight

blocks.1.adapter.ln1.weight

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln1.bias -

blocks.1.adapter.ln2.weight

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt_token

blocks.2.visual_prompt_token

blocks.2.attn.LoRA_a.weight -

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight -

blocks.2.adapter.ln1.weight -

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight -

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.weight

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln2.weight

blocks.4.adapter.In2.weight

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token-

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token-

blocks.6.attn.LoRA_a.weight

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_b.weight -

blocks.6.attn.LoRA_b.weight-

blocks.6.adapter.lIn1.weight

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.bias

blocks.6.adapter.ln2.bias -

blocks.7.visual_prompt_token-

blocks.7.visual_prompt_token -

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_b.weight-

blocks.7.attn.LoRA_b.weight-

blocks.7.adapter.ln1.weight

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln2.weight

blocks.7.adapter.ln2.weight

blocks.7.adapter.ln2.bias

blocks.7.adapter.ln2.bias -

blocks.8.visual_prompt_token

blocks.8.visual_prompt_token -

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_b.weight-

blocks.8.attn.LoRA_b.weight -

blocks.8.adapter.ln1.weight

blocks.8.adapter.In1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight-

blocks.9.attn.LoRA_b.weight -

blocks.9.adapter.ln1.weight

blocks.9.adapter.In1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight -

blocks.10.attn.LoRA_b.weight -

blocks.10.adapter.ln1.weight

blocks.10.adapter.ln1.weight-

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.bias

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight -

blocks.11.adapter.ln1.bias -

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

20

40

60

80

0

100

10

20

30

40

50

60

70

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

snorb-azim

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed

visual_prompt_token_pos_embed-

blocks.0.visual_prompt_token -

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_a.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight -

blocks.0.adapter.ln1.weight

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.weight -

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias

blocks.1.visual_prompt_token

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt.token

blocks.2.visual_prompt.token

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_a.weight -

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.ln1.weight

blocks.4.adapter.In1.weight -

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln2.weight -

blocks.4.adapter.In2.weight -

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight -

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight -

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_a.weight

blocks.6.attn.LoRA_b.weight

blocks.6.attn.LoRA_b.weight -

blocks.6.adapter.ln1.weight -

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln2.weight -

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.bias

blocks.6.adapter.ln2.bias

blocks.7.visual_prompt_token-

blocks.7.visual_prompt_token

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_b.weight -

blocks.7.attn.LoRA_b.weight -

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias -

blocks.7.adapter.ln2.weight -

blocks.7.adapter.ln2.weight -

blocks.7.adapter.ln2.bias -

blocks.7.adapter.ln2.bias

blocks.8.visual_prompt_token -

blocks.8.visual_prompt_token

blocks.8.attn.LoRA_a.weight-

blocks.8.attn.LoRA_a.weight

blocks.8.attn.LoRA_b.weight

blocks.8.attn.LoRA_b.weight-

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight -

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight

blocks.9.adapter.n1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight

blocks.10.attn.LoRA_b.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.bias-

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.weight-

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token -

blocks.11.attn.LoRA_a.weight -

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight -

blocks.11.adapter.ln1.weight-

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

20

40

60

80

100

10

OE

40

50

60

70

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

Name

Module

snorb-ele

visual_prompt_token

visual_prompt_token

visual_prompt_token_pos_embed

visual_prompt_.token_pos_embed

blocks.0.visual_prompt_token ·

blocks.0.visual_prompt_token

blocks.0.attn.LoRA_a.weight

blocks.0.attn.LoRA_a.weight-

blocks.0.attn.LoRA_b.weight -

blocks.0.attn.LoRA_b.weight -

blocks.0.adapter.ln1.weight -

blocks.0.adapter.ln1.weight -

blocks.0.adapter.ln1.bias

blocks.0.adapter.ln1.bias -

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.weight

blocks.0.adapter.ln2.bias

blocks.0.adapter.ln2.bias

blocks.1.visual_prompt_token

blocks.1.visual_prompt_token -

blocks.1.attn.LoRA_a.weight -

blocks.1.attn.LoRA_b.weight-

blocks.1.attn.LoRA_b.weight -

blocks.1.adapter.ln1.weight -

blocks.1.adapter.ln1.weight

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln1.bias

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.weight -

blocks.1.adapter.ln2.bias

blocks.1.adapter.ln2.bias -

blocks.2.visual_prompt.token

blocks.2.visual_prompt.token

blocks.2.attn.LoRA_a.weight

blocks.2.attn.LoRA_a.weight -

blocks.2.attn.LoRA_b.weight

blocks.2.attn.LoRA_b.weight

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln1.bias

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.weight

blocks.2.adapter.ln2.bias

blocks.2.adapter.ln2.bias

blocks.3.visual_prompt_token

blocks.3.visual_prompt_token

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_a.weight

blocks.3.attn.LoRA_b.weight

blocks.3.attn.LoRA_b.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.weight

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln1.bias

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.weight

blocks.3.adapter.ln2.bias

blocks.3.adapter.ln2.bias

blocks.4.visual_prompt_token

blocks.4.visual_prompt_token -

blocks.4.attn.LoRA_a.weight

blocks.4.attn.LoRA_a.weight -

blocks.4.attn.LoRA_b.weight

blocks.4.attn.LoRA_b.weight -

blocks.4.adapter.ln1.weight

blocks.4.adapter.In1.weight -

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln1.bias

blocks.4.adapter.ln2.weight -

blocks.4.adapter.In2.weight -

blocks.4.adapter.ln2.bias

blocks.4.adapter.ln2.bias -

blocks.5.visual_prompt_token

blocks.5.visual_prompt_token

blocks.5.attn.LoRA_a.weight

blocks.5.attn.LoRA_a.weight -

blocks.5.attn.LoRA_b.weight

blocks.5.attn.LoRA_b.weight -

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.weight

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln1.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.5.adapter.ln2.weight

blocks.5.adapter.ln2.bias

blocks.6.visual_prompt_token

blocks.6.visual_prompt_token-

blocks.6.attn.LoRA_a.weight -

blocks.6.attn.LoRA_a.weight

blocks.6.attn.LoRA_b.weight-

blocks.6.attn.LoRA_b.weight -

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.weight

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln1.bias

blocks.6.adapter.ln2.weight -

blocks.6.adapter.ln2.weight

blocks.6.adapter.ln2.bias

blocks.6.adapter.ln2.bias

blocks.7.visual_prompt_token -

blocks.7.visual_prompt_token

blocks.7.attn.LoRA_a.weight -

blocks.7.attn.LoRA_a.weight

blocks.7.attn.LoRA_b.weight -

blocks.7.attn.LoRA_b.weight-

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.weight -

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln1.bias

blocks.7.adapter.ln2.weight

blocks.7.adapter.ln2.weight -

blocks.7.adapter.ln2.bias

blocks.7.adapter.ln2.bias

blocks.8.visual_prompt_token -

blocks.8.visual_prompt_token

blocks.8.attn.LoRA_a.weight -

blocks.8.attn.LoRA_a.weight

blocks.8.attn.LoRA_b.weight

blocks.8.attn.LoRA_b.weight-

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.weight

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln1.bias

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.weight

blocks.8.adapter.ln2.bias

blocks.8.adapter.ln2.bias

blocks.9.visual_prompt_token

blocks.9.visual_prompt_token

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_a.weight

blocks.9.attn.LoRA_b.weight

blocks.9.attn.LoRA_b.weight

blocks.9.adapter.n1.weight

blocks.9.adapter.ln1.weight

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln1.bias

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.weight

blocks.9.adapter.ln2.bias

blocks.9.adapter.ln2.bias

blocks.10.visual_prompt_token

blocks.10.visual_prompt_token

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_a.weight

blocks.10.attn.LoRA_b.weight

blocks.10.attn.LoRA_b.weight

blocks.10.adapter.ln1.weight -

blocks.10.adapter.ln1.weight-

blocks.10.adapter.ln1.bias -

blocks.10.adapter.ln2.weight

blocks.10.adapter.ln2.weight -

blocks.10.adapter.ln2.bias

blocks.11.visual_prompt_token

blocks.11.visual_prompt_token -

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_a.weight

blocks.11.attn.LoRA_b.weight

blocks.11.attn.LoRA_b.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln1.weight

blocks.11.adapter.ln2.weight

blocks.11.adapter.ln2.weight

20

40

60

0

100

20

40

60

80

100

Updatable Parameters (%) in Module

Updatable Parameters (%) in Layer

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce BayesTune, a method for choosing which are the parameters to fine-tune in a pre-trained model. Their formulation is based on Bayesian inference where they use a Laplace prior over the parameters/network weights. The prior has two variables: mean, which is the value of the pre-trained parameters, and scale which specifies how important is to fine-tune it. These hyperparameters are also controlled by a hyperprior that is fixed among all the experiments. For inferring the posterior distribution of the weights and scales. they adopt Langevin dynamic method. After obtaining the scale value, they compute a cut-off value that determines the parameters to be updated. Experiments performed in Computer Vision and NLP tasks demonstrate that the method is competitive and outperform common techniques.\n\nSTRENGTHS:\n* The problem of efficient fine-tuning is very important nowadays given the availability of large pre-trained models.\n* The method is intuitive and interpretable, as it is based on the Bayesian interpretation of the network weights.\n* There is a large set of experiments demonstrating the utility of the method.\n\nWEAKNESSES:\n* This model is close to related work such as SP-regularization [1] which uses a regularization term to encourage the updated weights to stay close to the original pre-trained weights. Using an L1-SP regularizer might have a similar effect as the one introduced in this paper. I think it is important the authors mention this and compare BayesTune against it.\n* The efficiency of the method is not clear to me. For instance, in an attention layer, updating only some parameters (sparse updates) still demands computation of the matrix multiplications for query, key, and value matrices. The same logic applies when backpropagating. Thus, choosing only some parameters to update might only save memory demands, but not time. If my reasoning is correct, then the authors should specify this in the paper. Also, some measures of time and memory consumption might be useful. \n* There is a discrepancy between Algorithm 1 and the method described in the paragraph starting at line 206. Thus, I think Algorithm 1 is incomplete.\n* Their method seems competitive for NLP but not so for Computer Vision, based on Table 2.\n\n[1] Xuhong, L. I., Yves Grandvalet, and Franck Davoine. \"Explicit inductive bias for transfer learning with convolutional networks.\" International Conference on Machine Learning. PMLR, 2018.\n\nQUESTIONS:\n* Did you optimize the hyperparameters of the other methods?\n* Why table 2 does not have standard deviations?\n* Could you elaborate further on how you choose the cut-off point?\n* How did you choose the hyperprior values ($\\alpha, \\beta$)?\n\nLIMITATIONS:\nNo limitation was mentioned by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an approach for selecting a subset of weights in a foundation model to fine-tune on a downstream task. The method consists of a two-stage pipeline, where in the first stage a Laplace prior is placed on each weight with a Gamma hyper-prior on the scale. Samples are obtained via SGLD and only the weights with a mean posterior scale above some threshold are then trained via SGD in the second stage. The method is evaluated on GLUE and SuperGLUE tasks with RoBERTa and on VTAB-1k image prediction tasks with a vision transformer and compares overall favorably to a range of baselines from the literature.\n\nThere are quite a few design choices constituting the proposed method and, unfortunately, none of their added complexity is justified via ablation studies. Further, the paper in my view overstates how principled it is quite significantly, so that at this point I would lean towards rejection.\n\nSTRENGTHS:\n* The approach is new as far as I am aware.\n* The technical description of the method is clear.\n* Performance seems to be good and approaches for better fine-tuning are of high interest to the community.\n\nWEAKNESSES:\n* The core problem in my view is that the method consists of quite a few moving parts, but these aren’t justified via ablations. It is not clear at all where the performance improvements come from and whether all parts of the method are needed. E.g. it might be the case that the two-stage procedure with magnitude-based pruning would be enough (at least my understanding of MagPruning based on the description in the paper is that the smallest pre-trained values are pruned). Similarly I wonder if sampling in Stage 1 is necessary or if MAP estimates would be good enough for the scale parameters.\n* I don’t really see what makes the proposed method particularly principled as claimed at various points in the paper. I don’t think there is a probabilistic justification for the two-stage procedure and fudging the dataset size and noise scale for SGLD is just a hack.\n\nQUESTIONS:\n* How do the ablation baselines I mention in weakness 1 perform compared to the method as described in the paper?\n* Is adapting the noise scale and dataset size really needed for stage 1? I’m aware of the cold posterior effect when sampling for a Bayesian model average, however here we don’t seem to need the predictions but the actual parameter samples? How much does tuning these improve performance over the principled choice?\n\n**Typos/minor**:\n* l155: “algorithmm” -> “algorithm”, “pseudocodes” -> “pseudocode”\n* The abstract is extremely long without being particularly descriptive (e.g. the two-stage nature of the method isn’t even mentioned explicitly) and reads more like a small introduction. I’d suggest making it significantly more concise.\n* For me there is way too much going on in Tab 1. I’d suggest at least dropping the bold-facing and rank indicators for the standard deviations, and making the rank indicators gray rather than red (unless you consider them primary information, in which case they should be bigger. But at the moment they are secondary in terms of size and position, so overall the table is unnecessarily difficult to process).\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA principled approach for selecting a subset of parameters to fine-tune in large foundational models is proposed. The authors rely on Bayesian inference to identify this subset. They begin by placing a Laplace prior over the model weights and a gamma hyperprior over the weight scale. Next, they employ an MCMC method to obtain a posterior distribution. Finally, they rank the weights based on the posterior scale values and proceed to fine-tune the parameters with the highest inferred scale values. Across standard NLP and vision adaption tasks, they demonstrate a strong empirical performance compared to previous state-of-the-art approaches.\n\nSTRENGTHS:\n- I find the proposed methodology technically sound, novel, and appealing. Though it should be kept in mind that I am not particularly familiar with the related work on the finetuning of large models. Similarly, I like methods that are Bayesian, so this might add to my potentially biased evaluation here. \n- I like the simplicity of the proposed method. It does not happen often that I understand the method upon the first pass through the paper, but I think that was the case when reading this manuscript.\n- The empirical results are strong and I want to commend the authors on extensive evaluation (i.e., they do not stop at the language modality, but additionally consider a vision domain as well).\n\nWEAKNESSES:\nA presentation could be somewhat improved to further strengthen the manuscript. Some concrete suggestions:\n- I find it a bit strange to start introducing the notation already in the Introduction section. Hence, I would change your current section 1.1 into a separate section 2.\n- Similarly, I find it a bit weird to list out the related work as bullet-points they way you do in Section 3. I find it more natural to use separate paragraphs for that (see Section 8 in [1] for an example of that). This way you can also directly talk about how each related approach connects to your proposed model, instead of doing that in a separate paragraph as is currently done (lines 192-198).\n- Line 227: I assume you will include a GitHub link here, not put the code in the Supplementary material (whatever that means).\n- Grammar and style could be improved at some points to improve readbility. This is easily done these days via tools like Grammarly, ChatGPT...\n\nQUESTIONS:\n- Although the choice of Laplace prior is adequately discussed and justified in Section 2, the explanation of the mean-field assumption in equation (1) could benefit from further elaboration. While I understand that this simplification is made for computational tractability, it would be valuable to provide additional justification for this decision. Additionally, I am curious whether the authors anticipate any additional performance improvements by attempting to model correlations between different model parameters. Similarly, I am interested in their choice of the approximate inference scheme (SGLD). Did the authors consider any other approaches (e.g., variational inference)?\n\nLIMITATIONS:\nLimitations are currently not discussed. Perhaps authors could use the gained space from restructuring the related work section (see above) to add a paragraph or two on the limitations of their approach.\n\n\n\n[1] Daxberger, E., Nalisnick, E., Allingham, J.U., Antorán, J. and Hernández-Lobato, J.M., 2021, July. Bayesian deep learning via subnetwork inference. In International Conference on Machine Learning (pp. 2510-2521). PMLR.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new Bayes-based framework for sparse fine-tuning. The proposed method applies a Laplace prior (centered at the pre-trained weights) and a hyper Gamma prior over the scale parameter \\lambda of the Laplace prior. The method then performs posterior inference over \\lambda to determine whether or not a parameter is \"useful\": A large value of  \\lambda indicates a flat prior, i.e. an informative parameter, and vice versa. The paper then adopts SGLD to perform inference over \\lambda and then uses human assistance to determine the cut point for \\lambda given a budget. The method is evaluated on language and vision tasks and compared against a wide range of parameter efficient fine-tuning methods. The proposed method overall demonstrates performance superior and acquires a sparser model than baseline methods.\n\n## Post rebuttal update: I raised my score from 4 to 5 seeing the new results provided by the author.\n\nSTRENGTHS:\n- The proposed method is technically sound and well presented.\n\n- The proposed method is evaluated on a wide range of tasks.\n\n- The authors consider a thorough amount of baseline methods.\n\nWEAKNESSES:\nThe biggest issue of the proposed method, in my opinion, is the novelty. The use hierarchical prior, which induces sparsity, is a widely known idea since the last century, from Radford Neal and David MacKay's early works to recent works such as [1, 2, 3, 4]. Although these works do not consider the fine-tuning setting, they can technically be applied to the fine-tuning of large pre-trained models by simply letting the prior to be centered at the pre-trained weigh rather than zero. \n\nIn addition, the configuration of SGLD is not presented very clearly. I would suggest the author use formula to describe the modifications to SGLD described from line 212 to line 223, e.g. a modified version of Eq.5 where a few additional hyper-parameters are added to the likelihood and the injected noise term.\n\n*Minor: Some figures are not in vector format and the fonts are too small.\n\n*Minor: It would be interesting to see the performance of the proposed model on *Large* language model.\n\n\n[1] Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference\n[2] Dropout as a structured shrinkage prior.\n[3] Posterior concentration for sparse deep learning.\n[4] Bayesian compression for deep learning\n\nQUESTIONS:\n- What optimizers are used for BayesTune? Is it standard SGD through out the entire algorithm or it is a mix of SGD and Adam. \n\n- Does the baseline approach use same optimizer or they use different optimizers? The choice of optimizer can potentially affect the final performance.\n\n- What happen if you increase the rank of LoRA to let LoRA have the same the number of parameter used by BayesTune?\n\n- What's the advantage of posterior inference over MAP estimation in this setting, i.e. remove the last term in Eq.5?\n\nLIMITATIONS:\nThe proposed method introduces extra hyper-parameter for SGLD (besides step-size scheduling): Effective data size and noise discount factor.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

comments: SUMMARY:
The authors propose an automated sparse fine-tuning method for foundations model, that bypasses the need for human intuition-based heuristics. The neurons to update are revealed during the posterior inference of the sparse scale parameters of a Laplace prior, by thresholding the scale parameters.
The method is experimentally validated in both vision  and NLP tasks.

STRENGTHS:
The proposed method is principled, and relies on a hierarchical Bayesian model.

Posterior approximation is done with Langevin MCMC, which does not introduce a significant computational overhead. 

The experimental results show that the proposed method convincingly improves upon already existing heuristics.

WEAKNESSES:
I wonder what's the rationale behind using the elbow rule (figure 1) to select the proportion p of parameters to update.

QUESTIONS:
- Why was the Gamma distribution chosen as a hyperprior ?
- If the step after SGLD is to evaluate the mean of the scale parameters, aren't there cheaper methods that evaluate the mean, without stochastically approximating the posterior ?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-13