BayesTune: Bayesian Sparse Deep Model Fine-tuning
Authors: Minyoung Kim
Timothy Hospedales
Venue: NeurIPS 2023 poster (2023)
Keywords: Parameter-efficient Foundation model fine-tuning
Bayesian methods
Stochastic-Gradient MCMC
TL;DR: A novel Bayesian sparse fine-tuning algorithm for Foundation models for NLP and vision tasks, with hierarchical Laplace priors and SG-MCMC posterior inference to enable automatic estimation of the sparsity level for good accuracy-memory trade-off.

Abstract: Deep learning practice is increasingly driven by powerful foundation models (FM), pre-trained at scale and then fine-tuned for specific tasks of interest. A key property of this workflow is the efficacy of performing sparse or parameter-efficient fine-tuning, meaning that by updating only a tiny fraction of the whole FM parameters on a downstream task can lead to surprisingly good performance, often even superior to a full model update. However, it is not clear what is the optimal and principled way to select which parameters to update. Although a growing number of sparse fine-tuning ideas have been proposed, they are mostly not  satisfactory, relying on hand-crafted heuristics or heavy approximation. In this paper we propose a novel Bayesian sparse fine-tuning algorithm: we place a (sparse) Laplace prior for each parameter of the FM, with the mean equal to the initial value and the scale parameter having a hyper-prior that encourages small scale. Roughly speaking, the posterior means of the scale parameters indicate how important it is to update the corresponding parameter away from its initial value when solving the downstream task. Given the sparse prior, most scale parameters are small a posteriori, and the few large-valued scale parameters identify those FM parameters that crucially need to be updated away from their initial values. Based on this, we can threshold the scale parameters to decide which parameters to update or freeze, leading to a principled sparse fine-tuning strategy. To efficiently infer the posterior distribution of the scale parameters, we adopt the Langevin MCMC sampler, requiring only two times the complexity of the vanilla SGD. Tested on popular NLP benchmarks as well as the VTAB vision tasks, our approach shows significant improvement over the state-of-the-arts (e.g., 1% point higher than the best SOTA when fine-tuning RoBERTa for GLUE and SuperGLUE benchmarks).

Bayes Tune: Bayesian Sparse Deep Model Fine-tuning
Minyoung Kim! Samsung Al Center Cambridge, UK mikim21 @gmail com
Timothy Hospedales' 12 University of Edinburgh, UK t. hospedalles@ed ac
Abstract
Deep learning practice is increasingly driven by powerful foundation models (FM), pre-trained at scale and then fine-tuned for specific tasks of interest. A key property of this workflow is the efficacy of performing sparse parametter-efficient fine- tuning, meaning that by updating only tiny fraction of the whole FM parameters downstream task can lead surprisingly good performance, often superior to a full model update. However, it is not clear what is the optimal and principled way to select which parameters to update. Although a growing number of sparse fine-tuning ideas have been proposed, they are mostly not satisfactory, relying on hand-crafted heuristics or heavy approximation. novel Bayesian sparse fine-tuning algorithm: we place (sparse) Laplace prior for each parameter of the FM, with the mean equal to the initial value and the scale parameter having hyper-prion that encourages small scale. Roughly speaking, the posterior means of the scale parameters indicate how important is update the corresponding parameter away from its initial value when solving the downstream task. Given the sparse prior, most scale parameters are small a posteriori, and the few aargevaalued scale parameters identify those FM parameters that crucially need to be updated away from their initial values. Based on this, we can threshold the scale parameters decide which parameters to update or freeze, leading to principled sparse fine-tuning strategy. To efficiently infer the posterior distribution of the scale parameters, adopt the Langevin MCMC sampler, requiring only times the complexity the vanilla SGD. Tested on popular NLP benchmarks as well the VTAB vision tasks, approach shows significant improvement over the state-of-the-ard (e.g., % point higher than the best SOTA when fine-tuning RoBERTa for GLUE and SuperGL benchmarks).
this paper we propose
Introduction
Practical deep learning increasingly relies on a workflow of fine-tuning pre-trained foundation models for the specific task of interest to be solved. The foundation models (FM)s often pre-trained at scale using large architectures and self-supervised objectives on large scale unlabelled data, allowing them to encode great deal of general prior knowledge. Fine-tuning then specialises these general purpose FMs to the specific task of interest. The key challenge then becomes how to conduct fine-tuning so as balance the conflicting goals of adapting the task at hand; while retaining the knowledge obtained during foundational pre-training and avoiding overfitting on the (typically smaller) downstream training set. This is addressed from a variety of perspectives from careful learning rate scheduling to regularised fine-tuning [26] [12]. However, the mainstream approach in practice largely based on some instantiation of sparse fine-tuning.
Sparse fine-tuning, also known as parameter efficient fine-tuning (PEFT) 23
22 15
16
18 45 13 10), manages the adaptation/forgetting trade-off by selectively updating specific subset of parameters in the FM while keeping others frozen. These approaches also benefit from good parameter scalability with respect to the number of downstream tasks to solve since each downstream
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

task only requires storing small number of parameters compared to the full FM. A large number of sparse/PEFT approaches have now been proposed, based on either additive or mask-based principles. Additive approaches [16 15 inject a hand-chosen set of additional small modules to be learned, while keeping the original FM entirely frozen; while mask based approaches [3 18 manually select sparse subset of original FM parameters to consider as learnable such as the prompt parameters in transformers [8]. Despite substantial engineering effort, prior sparse fine-tuning work mainly relies on human intuition and ad-hoc heuristics to specify additive modules and sparse fine-tuning masks. This raises the question how can we automatically select which parameters to insert or update in a principled and effective way?
In this paper, we address this issue by proposing an automated sparse fine-tuning method. Our BayesTune framework automatically discovers the parameters to update, with minimal human intervention. In contrast to some early concurrent attempts that rely on heuristics and/or heavy approximations 45 13 10), Bayes'Tune is highly principled, underpinned theoretically by Bayesian marginalization of parameter uncertainty. Specifically, we place a sparse Laplace prior on the FM parameters, and the posterior inference of the sparse scale parameters reveals which parameters are most important to update from their initial values. BayesTune is efficiently implemented with Langevin MCMC, which only requires twice the cost of vanilla SGD, We demonstrate BayesTune with both RoBERTa transformer on (Super)GLUE benchmark suite and Vision Transformer on VTAB benchmark suite. BayesTune achieve specified sparsity level, and provides reasonable way to identify a good sparsity/performance trade-off if a sparsity constraint is not pre-defined. Finally, we show that BayesTune can work with both mask-based search spaces to identify existing FM parameters to update by fine-tuning, as well as additive search spaces to identify which new modules to add where for additive adaptation.
2 Problem Setup and Motivation
First we introduce some key notations follows:
= (Variable) Parameters (i.e., weights and biases) of the FM
0- (Constant) Parameters (i.e., weights and biases) of the pre-trained FM
D Training data for the downstream task
We denote by 0g (E IR) the i-th element of 0 by considering 0 as whole vector of concatenated parameters, i.e.,
[0,,... ......... _O,] where d the number of parameters (so, 0g is scalar). Before we start, we clarify the goal of the sparse fine-tuning formally as follows.
Desiderata. We aim to find 0 which performs well on the downstream task, and at the same time remains close 0 sparsely in Lo sense (i.e., ||0 O E +0,) is small). Roughly speaking, 8, k B. for many i's. Oftentimes we define and set the sparsity level p E [o. 1]. meaning that the proportion of the FM parameters updatable from the pre-trained values is no greater than (100 9)% (i.e.. ||0 0|o < [d p)).
About the sparsity level p. We remark here that the sparsity level p is often given by the users, for instance, in case we have a strict pre-specified memory constraint (e.g., embedded platforms), where the allowable extra space for saving the updated new parameters from for the downstream task is only (100 x 2)% of the original FM storage. In other situations, it would be ideal if the sparse fine-tuning algorithm can estimate optimal p that trades off the downstream accuracy against memory overhead (e.g., find the smallest p that enables a certain level of accuracy guarantee). Unfortunately most existing sparse fine-tuning methods are not capable of this feature, and rather resort to cinecconuming gross-validttion-yye search. Although this latter case opens up a new interesting research problem to pursue further, our proposed approach offers reasonable and principled criteria for estimating the optimal p (See Sec. 3 for details).
How about layer-wise selection? We also note that our approach is not tied to the layer-wise parameter treatment, namely selecting a few layers to be updated with the rest layers frozen, which is popular strategy adopted in many sparse fine-tuning methods. The layer-wise treatment is based on the conjecture that whether the parameters important to the downstream task are deviated from the pre-trained ones or not, is determined in a layer-wise all-or-nothing manner. Although this is reasonable assumption, our approach does not rely on the assumption, and is able to discover optimal sparse updates automatically by inspecting individual parameters over all layers.

---

Back to our notations, the downstream task must be associated with the relevant loss function [(0; 2) with E D for fine-tuning, Here the task could be either supervised (i.e.. 2 (z. y) for a pair of imput and its target label y) or unsupervised (= = 2 with input alone). In our model the loss function turned into a likelihood model by the conventional trick, p(= oc exp(-I(0; =)). Although our approach can deal with both supervised and unsupervised cases seamlessly, we predominantly focus on the supervised classfication case, that is, we define the data likelihood ply xz, 0) x exp(-1(0;x,, x, x))) for (I, 2) D, where 100; v) is typically the cross-entropy loss.
3 (Proposed) Bayesian Sparse Fine-tuning Model
We propose (hierarchical) Bayesian model tackle the downstream prediction task of interest. Specifically, we treat the FM parameters 0 as random variables and the downstream training data D as evidence. To encourage parameter sparsity, namely most parameters f3 remain the pre-trained d, we impose the Laplace prior p((IA).
p(011) = II d where ((z; u4, b)= 26 exp(-Ix= -11/b) is the (univariate) Laplace distribution with mean μ and scale Thus we fix the prior means as the pre-trained values 0, and only model the scale parameters which are newly introduced random variates denoted by À (> 0). Note that every single parameter 0, (scalar) is associated with its own prior scale A (thus scale variables in total). Another thing to note is that the final layer (also known as the readout head) of the FM is often completely replaced by a random one for the downstream task. Since this final layer needs to be learned from the scratch with the downstream data, we do not place the Laplace prior the parameters of the readout head.
i-1 24, ((O))A)) II. C(0,0,, Ar) II
d 1
exp( 1xp(-11: - 0,1/2,)
(I)
Through the scale variables of the Laplace distributions, we can express our (prior) preference to the degree of parameter deviation from the pre-trained values: small A; leads to peaky Laplace around its mean, penalizing even small deviation of 6g from 0g; on the other hand, large X; makes it flat, being less sensitive to deviation, allowing 0, to take values freely away from 8g.. By carefully choosing A values, we can balance effectively between overfitting (too much deviation) and underfitting (too little deviation). Instead of choosing A manually, our idea is to learn them automatically from data, through the principled Bayesian inference: given the evidence (i.e., the downstream data D) we infer the most probable values of X; 's that best explain the evidence, while being small enough to lead to sparse updates overall.
To this end, we regard À as random variables and impose hyper-prior on À (hierarchical Bayes). As prefer sparse updates in the end, the hyper-prior needs put higher mass/density on small X values. We adopt the Gamma distribution for this purpose, more specifically,
d p(lla, 3) III p(A.aa, 3) -II G()i;a,))
P(Ala,
(2)
where G(x; a; 3) x x0 exp( (BBr) for a, 3 O. Note that although we can introduce individual s--specific parameters (i.e., (a,, 9,)), we instead use a single (a, 3) shared over all A, 's for simplicity. Since the mode of Gamma is if a < 1 and the variance is a /B2, we use small a < 1 and large / to enforce priori small values. In experiments choose (a 0.01, B 100) for both the NLP and VTAB vision benchmarks.
Our full model can be written as the following joint distribution:
II. 9(A.a.9)
IewD p(yl=.0)
DDD,0,Na,B 3)= p(laa B)
X
((OA))
X
p(D(
II. ((0.0.A).)
With this model, our ultimate goal is to infer the posterior distribution of the network weights and scales p(0, X\D), namely
p(D, A.D.O.Ala.3 3)
(00 a)),,, = JS D(D,0, Xa, 3) d0dA
'We also tested with models with further hierarchy by placing priors on and 8, however, there was no significant advantage over the manually chosen ones.
3

---

We adopt the crcchsstc-gradiens MCMC approach 40 7 4
to obtain samples from the posterior,
especially the Langevin dynamic method (SGLD) 40 which amounts to running the following
recurrence to collect posterior samples (after some burn-in steps):
V log (og + 10 - 10 1 2
N [B] logr(B10)) + 7EV
(5)
where B (C D) is a minibatch, is small step size, and (0,, 1). In standard SGLD, N = D (training data size) and 7 1. However, we allow them take values different from these default ones: N is the effective training data size to account for data augmentation (c.g., if there are different augmentation strategies and each allows 10 different variations, the effective data size can be Ñ = 10 x [D); the other hyperparameter 7 can be used to discount the noise effect which is validated the range 10-1:0 in our experiments. See Sec. 5.1 for more details. Note that in the parentheses subject to derivatives, the first two terms admit closed-form gradients while the gradient of the last term can be computed by the conventional SGD backprop. Thus each step (5) is efficient, requiring at most only two times the complexity of the vanilla SGD step.
After burn-in period, we can maintain those samples (0, A) to approximate p(0, X D). For instance, the running average of the À samples, denoted by X, a good estimate of the mean of the marginal posterior p() Since imposed the hyper-prior that encourages small X, ideally the majority of the scale parameters tend to remain small a posteriori. whereas there will be only a few large- valued Å 's corresponding to those FM parameters that erucially need to be updated away from their pre-trained values to explain the data D. Roughly speaking, À₂S indicates how important it is for the corresponding FM parameters 0, to be updated away from their pre-trained values 0g on the downstream data. Importantly, this À can be used in the next stage (described in the next paragraph) to decide which network weights 0, need to be frozen (those with small A,s) and which need to be updated from the pre-trained (those with large Å.s)
Thresholding scales and the second stage. Although we adopted the sparsity-inducing Laplace prior, it is not necessarily the case that the majority of posterior 0 values are sharply staying at the pre-trained values B. We may need some thresholding: If p is given, then we can directly take the top (100 p)% of them. Otherwise we examine for cut-off point Those i's with small posterior means A are considered as frozen weights (i.e., 0, 0.), while those i's with large \ can be treated as updatable parameters. In practice, we sort and plot the λ;S to eyeball and find a reasonable cut-off memory
point. One can also use this À-plot to decide a reasonable sparsity level for good trade-off (See Fig. for illustration). Once we have decided which parameters updated and frozen, we can run vanilla SGD to train the updatable parameters of the FM, which forms our second stage.
Our overall algorithmm dubbed BayesTune, is summarized pseudocodes in Alg.
4
Related Work
4.1
Comparison to Existing Sparse Fine-tuning Methods
The recent sparse/PEFT fine-tuning approaches broadly fall into the following three categories: 1) Heuristic search criteria, 2) Attaching small extra modules to the FM (e.g.. adapter-based (151), and 3) Directly formulating the sparse fine-tuning problem as an optimization problem.
1) Heuristic search criteria.
Random selection Randomly select (100 x 7)% parameters update.
Similar discussions of how quantify dataset size when using Data Augmentation as well as data size inflation, were made in several prior Bayesian deep learning studies, for instance, 33
'The rationale for noise discounting is as follows. Purely finding the posterior mean (i.e., SGLD without noise discount) risks performing poorly if the posterior is truly multi-modal, because may converge low probability parameter. Also, purely searching for the posterior mode (i.e., MAP instead of SGLD) may be sensitive to data noise, because no stochasticity is taken into account properly. So for combining principle and practice it is reasonable to prefer discounted noise procedure that balances between identifying a particular mode, but gets mean estimate in the vicinity of that mode.

---

Sorted û₃ vs. proportion in FM
Test error vs. sparsity
15
BayesTune
Candidate cut-off points
Full update
Test error (%)
Head only
Sorted 7
12-
Cand-1
Cand-1
10-
(7 0. 0.05)
0. p 0.05)
6a-
Cand-2 0. p. 0.35)
Cand-2 1. p 35)
64-
60-
Propartion in d (H-FM-parameters)
Proportion in d ( (#-FM-parammeerr
Figure 1: (Left) The À-plot in our BayesTune after stage-1 training. This is with EfficientNet-BO on the Oxford-Pets dataset. After sorting the posterior mean ); values, we plot several sparsity levels and the corresponding À values. To decide reasonable sparsity level p for good accuracy- memory trade-off, we illustrate two candidates: Cand-1 I with more sparse updates (p = 0.05) but many parameters with large X, 's (e.g., 0.1 0.3) are forced frozen; Cand-2 with less sparse updates (p ==00 0.1) but ensures that those fixed parameters have indeed small A (e.g.. < 0.1). (Right) Test errors in stage-2 at different cutoff points in our BayesTune (red curve). Those two candidate points (Cand-1 and Cand-2) are highlighted and superimposed, consistent with the left figure. The more sparse Cand-1 has slightly higher error than the full-update, while less sparse Cand-2 achives the best test performance among all other choices. It exhibits nice accuracy-memory trade-off. Further increasing p (being far more dense) incurs overfitting.
Algorithm BayesTunee (two-stage) sparse fine-tuning algorithm.
Input: Pre-trained model 9, downstrcam data D. Initialise: 0 X2 0.0001 for all
I. Stage-1 (Select (100 x p)% parameters to update)
For = 2, m
0. Sample minibatch B {(xr,, v)} from D.
1. Run the SGLD step (5) get sample (0, A)-
2. I >
burn-in, accumulate À <- (1 <)Å + ÇA where ( a/fffaccumulated--ampeeeeee 1).
End-For
(If p not given) Decide cut-off point A.a and the corresponding sparsity level p using the X-plot.
Select those i'ss with d, > As updatable parameters.
II. Stage-2 (Update the selected parameters)
Do vanilla SGD with data D while updating only those updatable parameters from Stage-1.
After training the FM model with full update, a random (100 x (1 -p))% parameters are reset to the pre-trained values.
MixOut 23
BitFit 43
Update only the bias parameters with the rest frozen.
MagPruning 11
24 22
Take the largest (100 p)% in magnitude updatable parame-
ters.
Compared to our proposed "BayesTune", these heuristic search methods are less principled, unable to explain why the resulting models should perform well on the downstream tasks
2) Approaches that attach small extra modules to FM. They do not search for subset of FM parameters to update, but attach some small extra modules to the FM (e.g., insert a small MLP module after each layer). All of the FM parameters frozen, and they only train those attached modules. So can be seen as sparse fine-tuning considering that the size of the new modules is relatively small compared to that of the FM.
Adapter 15
A small bottleneck MLP is inserted at the end of each layer of the FM.
LoRA 16 A low-rank projection weight matrix is inserted before the query/key/value embeddings each layer.
set of learnable tokens inserted the input of Transformer attention block in each layer.
Prompt learning (VPT)
5

---

Neural Prompt Search [45| A NAS-basee approach (e.g., learning supernet followed by evolutionary search for a downstream tasks specific subnet) that aims search for the optimal combination of the above three methods.
Compared to our "BayesTund" these adapter- based models involve the difficult problem of where to place the extra modules and what would be the optimal module dimensions/sizes, which is essentially a very difficult combinatorial search problem, often requiring heuristic search methods like the evolutionary search.
3) Directly optimizing the sparse optimization problem. One can formulate the sparse fine-tuning as an optimization problem, for instance,
min ((a.)>>(((00,a x, s.t. |10 0llo - [d- pl.
Unfortunately this problem becomes discrete optimization due to the Lo cardinality constraints, thus difficult to solve in general. Some recent approaches aimed to relax the problem to continuous optimization, but relying on heavy approximation.
DiffPruning [13] They relax the discrete problem into a continuous one by introducing Bernoulli variables indicating update or freeze, while the reparametrized sampling is further approximated by the Gumbel-softmax treatment.
ChildPruning [1 31 They iteratively train the full model parameters and calculate the projected mask to find the child network.
Second-order approximation method (SAM) -0l-TTe [hey They adopt several steps of second- order Taylor approximation of the loss function, leading to a very succinct parameter selection strategy: after computing the gradients of on the data D, take the (100 x p)% parameters with the largest gradient magnitudes.
Compared to our "BayesTune" these direct optimization strategies rely strong assumptions and heavy approximation to relax the difficult discrete optimization.
4.2 Relation Existing (Hierarchical) Bayesian Sparse Model Learning
(Hierarchical) Bayesian models are well known in statistical machine learning, and there were some prior works that adopt the Bayesian approaches for sparse deep learning 32 35 29] However, as far as we know, our approach has several key differences from these previous works.
First, most prior works are all about sparse training, instead of sparse fine-tuning. Thus they focus on zeroing out many parameters, instcad of retaining the pre-trained weights. To the best of our knowledge, our BayesTune is the first to tackle the sparse fine-tuning at the Foundation Model scale using the hierarchical Bayesian framework.
More importantly, the neural networks used in those previous studies are rather small/toy scale (mostly focusing on MLPs and LeNet-sized architectures up to ResNet-18 at the largest) while our method can obtain the state-of-the-art results on large-scale foundation models (ViT, RoBERTa). For example the largest model considered those prior works, ResNet- 18 contains ~ 11M parameters vs. RoBERTa's 's 123M parameters, 10x scale difference. The reason is that they adopt methods that entail extra memory cost like variational inference (VI), which impedes applicability to big networks. We have also done some experiments that compare the computational resources required by VI and SGLD: On ViT networks, the training time is increased by 1.7 times if we replace SGLD by VI; the GPU memory footprint is increased by 2.1 times.
5 Experiments
We test our BayesTune on two popular benchmark datasets from NLP and vision for the downstream fine-tuning tasks: (language) fine-tuning the pre-trained GoBERTa-base model (28) on the GLUE (37
and SuperGLUE 38 tasks; (vision) fine-tuning the ImageNet-22K (6] pre-trained ViT-B/16 model 9
on VTAB-1K 1K 44 image clasification/prediitoon tasks. The details of the experimental settings are
discussed in the subsequent sections, Sec. [5.2andd and Sec. [5.3
5.1 Implementation of Bayes'Tune
For the SGLD sampling Stage-1 of our BayesTune algorithm (re: Alg. D. we have three different training regimes: the warm-up phase that only runs vanilla SGD steps without considering the scale

---

Method
CoLA
STS-B
MRPC
CB
COPA
WSC
AVG
Full update
58.36".77 1:74
89.80 5a
89.55 (2]
76.032-14
88.93 (3]
(3)
67.70%
53.106.15
74.789.00
Random
68.354.06(8)
89.81 1-11 89.81%.11p3
[x]
88.73" 18.730.80
72.719-28
90.34
68.802.64
52.885.97
74.652.40
MixOut
58.66"
90.
88.09 [4)4]
(7)
86.51 86.51
71.30* 9.84
52.98".75
75. 15.
Birfit
56.671.40
90.12" [19)
87.35 [3]
72.742-47
86.963.20 20
71.20 79
55.105.39
74.31
MagPruning
56.572.47
90.30
0. (11)
88 00.099 re
((4(4)
81, 81.25
71.50(4)
2 (-46
35.67" (P(a))
73.85 (90))
Adapter
62.11
1:22(0)
90.05
89.29
(4)
76.9374) 05
87.32 02
(9.502-64(44
37.02[s]
76.03
LoRA
60.88 1.
87.19 87.19
89.
62
(6.
84.613.76
69.70 as
36.8174
75. 2.24(4)
DifiPruning
58.53
80.590.34
78.796.00
69.937.97
86.25 (4[
72. 72.
53.37"
72.653.37
ChildPruning
60.00".29
80.971.51
87. 3. 86
75.764.35
86.61 3:22
69.404.00
55.599.51
74.93
SAM
60.8
p4 [2]
[]]
90.59
0.14
88
88.81 0.49
76.79"-7/p0
76.79"
72pa)
1,76 88.93
[3]
74.30
2.45 [1)
[a]
(.08(s) 59.52
[a]
77.12 [a)
(.51
0.47 (A)
0.36
0.84
[.1
2.02 (1)
78.12 1.17
BayesTune
60.86 kg
90.40
90.61
77.87
91.25
75.00 [1)
60.87
[1]
[3]
(X)
[x]
Table Results on NLP benchmarks. For each dataset/task (column), the average accuracy and the standard deviation (in superscript) shown over 10 runs with different random seeds are reported. The ranks (up to the fourth) among the competing methods are also shown in the brackets and in red. The figures of the competing methods are excerpted from [0]
variables À and random noise; followed by the burn-in phase where the SGLD steps are performed as
(5) but no (0.A) samples are collected; followed by the normal phase which we do collect samples. Following the conventional practice, we also perform the thinning steps [27] (i.e., collecting samples at a certain frequency) to mitigate undesired temporal correlation effects.
Another important implementation tip is the use of reweighed cost terms in Langevin dynamics As shown in ()) the cost function is involved with the training data size N N= |D] Even though N is known, the data augmentation would considerably increase the effective data size by several orders of magnitudes (c.g.. if there are different augmentation strategies and each allows 10 different variations, the effective data size can be 10 x N). To account for it, we regard the effective data size as hyperparameter denoted by Ñ that can be chosen from validation Replacing |D] by Ñ and taking it out of the parentheses in ( we have the gradient (normalized by Ñ) multiplied by the step size (n/2) First we choose 7) match the learning rate in the neural net optimizer (e.g.. Adam (20) that is, 17 = Ir 2/Ñ. Accordingly the noise term becomes e (2/(N x Ir)) We also find it effective oftentimes to discount the noise effect: we introduce the noise discount factor (another hyperparameter) by which the noise term multiplied. We validate y in the range 10-4:) 4:0 (e.g., 7 = corresponds to the default no-discount case).
Throughout all experiments we use the Gamma prior parameters (a 0.01, 100), the scale variables X₂'s are all initialized to 0.0001, and the learning rate for À is 0.01 without scheduling. Other task-specific implementation details can be found in the subsequent sections. The Python/PyTorch code to reproduce the results is available at https: // gā thub. com/ SamsungLabs/ Baaee/ayessuaaa Tune
5.2 NLP Benchmarks
We consider fine-tuning the pre-trained RoBERTa- model (28]. the large-scale foundation language model comprised of 125 million parameters, several downstream tasks in GLUE (37 and SuperGI.UE (38 benchmarks. We follow the experimental settings from 10,, in which the original development sets serve as test sets, and the validation sets formed by holding out random 10% of
"For instance, in the VTAB-IK, although N 1000 800, we let Ñ have range 107222
'We used the Adam optimiser for updating the model parameters, thus there may exist some internal gradient adaptation and momentum effect under the hood in our SGLD steps. This effect may be related to that of the adaptive drift and momentum in SGLD that were analysed in some previous works, e.g., [19]. However, believe that our SGLD update scheme with Adam would not result significantly different solution compared to the original SGLDD formulation at least in practice.
Alternatively, https: / /gggttu.bb..t/. cow///ttayyouugnnnnnll/yyyeltt
7

---

K275
mtpp
522
6330
SW
343
0A1
6M
050
082
-11
02
04 34
02
06
69
C3
53
1350
1.0
copa
C24
328
325 CA 0R
520
502
250
038
010
32 03 58
23 74
Figure 2: (NLP benchmarks) Posterior mean vs. sparsity level p. The default sparsity level p = 0.005 is depicted as red dotted vertical lines, For each dataset, we plot the original p E [0, range (lower-left), which is magnified for the smaller range p E 0.1] (upper-right).
the training sets. Instead of having a fixed number of training epochs, the development sets are used for an early stopping with the no-improvement tolerance of 40 epochs. We implement our BayesTune
on top of the codebase of (10) which based on the jiant framework (https://jiant httf/iinnnntoooo///.....o...o..t Similarly as [TO). we consider 7 tasks: Corpus of Linguistic Acceptability (CoLA) 39), Semantic Textual Similarity Benchmark (STS-B) (2), Microsoft Research Paraphrase Corpus (MRPC) (8)
Recognizing Textual Entailment (RTE) (5 5 Commitment Bank (CB) [30], Choice of Plausible
Alternatives (COPA) [36), and Winograd Schema Challenge (WSC) 25
Our BayesTune Tune is compared against strong baselines and the state-of-the -ftt-e--rts--,, including: Random, Mixout 23 BitFit 43). MagPruning (24 [2],, Adapter [S], LoRA (16), DiffPruning. Child-
4 For all methods, we use the sparsity level p = 0.005 as suggested in I3 For the competing methods, we follow the hyperparameter settings from 110: In Baye'Tune, we use 10K warm-up steps, 2K burn-in steps, and thinning at every 100 steps for all tasks. The batch size is 16, and the learning rate for the model parameters is 10 for Stage-1 and 10-3 for Stage-2. For the (dataset-dependeen) evaluation metrics, adopt the protocols in [37 38). We run all models on a single NVIDIA V100 GPU with 32GB memory. The results are summarized in Table where means and standard deviations over 10 random seeds are reported.
Pruming 41 31, and SAM [1O), as discussed in Sec.
First, it is quite surprising that the random selection strategies (Random and Mixout) are as effective as many sparse fine-tuning methods. This may be attributed to the massive over parametrizationn of the deep foundation model. However, more careful and sophisticated selection is important as those three methods, our BayesTTmee SAM, and Adapter, outperform the random strategies by large margin. Overall our TayesTTune performs the best among the competing methods, about 1% point higher on average than the runner-up SAM [10). We achieve the first place on 5 out of 7 tasks, and also attain the smallest standard deviations, implying that the resulting sparse fine-tuned models are more robust and reliable. This result signifies the effectiveness of our sparse Laplace scale-based parameter selection strategy that is underpinned by the principled Bayesian posterior inference with efficien"" stochastic gradient Langevin sampling.
In Fig. 2 we also show the posterior mean scale vs. sparsity level p. The cut-off threshold points corresponding to the sparsity level p 0.005 are around A [0.25, 0.30) for all tasks except for MRPC. This cut-off scale range might be considered reasonable choice as most of the parameters would have Laplace scales smaller than this range, exhibiting no pronounced indication of crucial deviation from the pre-trained values, although this is only conjecture based on our empirical results. Further rigorous theoretical analysis on the relation between the scale threshold and the generalization performance is beyond the scope of the current paper, and is left as future work.
5.3
Vision Benchmarks
Next we test our BayesTune on the VTAB-IK 44 an image classifecation/predittton task adaptation benchmark suite comprised of 19 different image datasets. It contains images/tasks that exhibit
TThe SGLD steps 5 require most only two times the complexity of the vanilla SGD steps.
8

---

#puram (M)
Cilar100
Caltech 101
Flower102
SVHN
Sun399
Camelyon
EurSSAT
Reshe44
Retinopathy
Clevr- Count
Clevr-Dist Dist
DMLab
KITTI
dSpr-Loc
dSpr-OOi
SNORB-Azin
SNORB-EIe
Arg Rank (L)
Rank ((7)
DTD
Pets
Method
Fall update
86.8
68.9
87.7
64.3
87.2
86.9
87.4
38.8
79.7
95.7
84.2
73.9
56.3
58.6
41.7
65.5
57.5
46.7
25.7
29.1
Linear
0.04
64.4
85.0
63.2
97.0
86.3
36.6
51.0
78.5
87.5
68.5
74.0
34.3
30.6
33.2
55.4
12.5
20.0
9.6
19.2
VPT
18
0.64
78.8
90.8
65.8
98.0
88.3
78.1 49.6
81.8
96,1
83,4
68.4
68.5
60.0
46.5
72.8
73.6
47.9
32.9
37.8
4.16 3
Adapter
0.16
90.1
89.9
84.0
94.9
75.5
48.6
48.5
41.6
3.68
LoRA
16
0.29
67.1
91.4
69.4
98.8
90.4
85.3
54.0
84,9
95.3
84.4
73.6
82.9
69.2
49.8
78.5
75.7
47.1
31.0
44.0
2.68
4
NOAH
45
0.43
69.6
92.7
70.2
99.1 90.4
86.1
53.7
84.4
95.4
83,9
75.8
82..8
68.9
49,9
81.7
81.8
48.3
32,8
44.2
1.95
5
BayesTune
Avg 68.9 92.6 69.5 99.1 90.8 88.1 50.0
84.6
95.8
82.8
76.0
82.6
67.4 49.6 82.3
81,9
49,9
22.6
39.3
2.37
7
0.37 (.07) (337) (.04) (.37) (.15) (.67) (044 (.60) (.60) (37)) (.07) (.30) (.2) (.52) (600) (.60) (330) (.52) (.67)
Table 2: VTAB-IK results. The accuracies at the optimal sparsity levels are reported for our BayesTune, For BayesTune, the optimal number of the updated parameters dataset-dependent, and these optimal numbers are depicted in the parentheses. The figures of the competing methods are exerpted from 18
16 45
highly diverse aspects/conditions such as: different image acquisition (by standard cameras or apecial-purpose ones for remote sensing or medical imaging), different objects/concepts (generic, fine-grained, or abstract), and tasks (object recognition, counting, or pose estimation). Each dataset in VTAB- IK consists of 1K training examples, and we use the splits officially provided (train 80% and validation 20%).
We aim to fine-tune the ImageNet-22KK 6 pre-trained ViT-B/16 model [9] on each dataset. We imple- ment our BayesTune using the codebase from [45] while employing most of their hyperparameters without changes (e.g., the number of training epochs 100 and batch size 64). In our Bayes we use 50 warm-up epochs, 10 burn-in epochs, and thinning every 5 batch steps for all datasets. The learning rate for the model parameters 10-3 and weight decay 10-3 for both stages.
We first attempted the proposed method with our original setting, i.c., place the À variables to the backbone (ViT) parameters, and let the Bayesian inference figure out the most critical parameters to update, namely those with large A Unfortunately the result was significantly worse than the adapter- methods: VPT [8], Adapter [15. LoRA [16,, and NOAH 455, for most datasets. As reported in several works in the literature, adapter- based models perform significantly better than the full backbone update for this benchmark (e.g.. Table in (45). Along this line, we conjecture that this trend also holds for other sparse fine-tuning methods, namely the superiority of the adapter-based approaches to sparse backbone update strategies, particularly for this VTAB benchmark.
To fairly take the advantage of extra attachable modules as the adapter-based approaches, we attach extra modules to ViT for all three popular strategies: visual prompts in VPT [18), adapter modules in Adapter [15), and low-rank matrices in LoRA [16]. This design idea is similar to (and motivated from) the neural prompt search in NOAH 45) but they formed a grid search problem determine which layers to attach the adapter modules and how many module dimensions to attach solved by the neural architecture search (NAS) technique with the super-net training, specifically adopting Autoformer 33, followed by rumning Evolutionary algorithms for optimal architecture search.
On the other hand, we add 10-dim full parameters for each module/layer, and they are all initialized to O. We let the posterior inference in our BayesTune automatically figure out which of these extra modules parameters be updated away from 0 and which to be frozen O. We used the zero- initialization because having all-O extra modules can be considered to be identical to the original pre-trained ViT without extra modules, thus serving as pre-trained extra modules. We only place A variables to the extra modules (VIT parameters frozen B In summary, our setup is similar to NOAH (45 in that we take all three adapter strategies into account, but the main difference is: NOAH does a difficult super-net training and Evolutionary search to determine the optimal module-wiss dimensions. On the other hand we do sparse selection of the module parameters to be updated from across all 10-dim full modules.
*We initially attempted to place À to ViT parameters as well, but unfortunately it did not perform well. The reason being probably is that most of the selected parameters for update come from the VIT backbone, essentially reducing the original sparse backbone update strategy that failed.

---

(a) Sparse update is better: Sun397, CIPAR100, DTD, and Ret-ioopathy
CMM202 (AMLIM)
reanspllyy Dunce and
Sparis STA
12
(b) Dense update is better: DMLab, Camelyon SVHN, and dSpr-Loc
crob om mnd
candlion Dowei 2011
Dense
(c) Optimal in-between: Clevr-Diste M.NOR--Eee KITTI, and -
Perpcine
(d) Sparsity Independent: 1lower-102, EuroSAT, Ca1tech101, and Pets
Figure 3: VTAB datasets grouped by the sparse/dense adapter update effects. E.g., the datasets in group-(a) exhibit higher accuracy when the adapter modules are updated more sparsely than densely. In groub-(c), neither extreme sparse nor dense updates are optimal, but the sparsity levels in between perform the best. In group-(d), we have no clear correlation between update sparsity and test accuracy. Individual plots show test accuracies vs. sparsity levels, where the rightmost plot in each group shows the mean and 95% confidence interval of the members within the group in normalized accuracy scale.
The results are summarized in Table 2 We report the performance of the best sparsity levels found Although the final accuracies of our BayesTune Tune are not always the best, we achieve Rank 1 on out of 19 datasets and the second place on average over all datasets. The performance of Bayes' Tune is comparable to other adapter-based methods, within the same range as those. That is, our main message is that the adapter-based enhanced architecture fine- tuning can be done almost comparably well by the proposed Bayesian method, instead of relying on the heuristic search such as the Evolutionary Search or costly human expert design effort. Compared to NOAH, our BayesTune results in more sparse adapter updates average (0.38M vs. 0.43M). In Fig. 3 we also visualize the clustering of VTAB-1K datasets according to the sparse/dense adapter assign each dataset into either of sparse, dense, and in-between clusters according to the optimal sparse update level p. small, large, medium, respectively. Such a grouping of the VTAB datasets based on fine-tuning sparsity characteristics has not been explored in the community, and we can casily obtain this finding using our BayesTune.
from p E (0.05, 0.1, 0.2, ...9 1.0
update effects:
Conclusion
We introduced BayesTune, a framework for automated fine-tuning that provides principled yet efficient approach to identifying which parameters to update in sparse fine-tuning by posterior inference of parameter-wise scale in hierarchical Bayesian model. We demonstrated BayesTTune to be effective for parameter efficient fine-tuning both NI.P and vision tasks, using both mask-based and additive scarch spaces.
10

---

References
[1] Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth paseal recognizing textual entailment challenge. In TAC, 2009.
[2] Daniel Cer, Mona Diab, Eneko Agirre, Inigo Gaziio, and Lucia Specia. Semeval2017 task Seman- tic textual similarity- multiiiiaa and cross-lingus focused evaluation. arXiv preprint arXiv: 2017.
[3] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In ICCV, 2021.
[4] Т. Chen, E. B. Fox, and C. Guestrin. Stochastic gradient Hamiltonian Monte Carlo. Conference on Machine Learning, 2014.
International
[5] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, 2005.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[7] N. Ding, Y. Fang, R. Babbush, C. Chen, R. D. Skeel, and H. Neven. Bayesian sampling using stochastic gradient thermostats. In Advances in neural information processing systems, 2014.
[8] Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005), 2005.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlshy. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[10] Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. On the Effectiveness of Enrfceter-ffficient Fine-Tuning, AAAI, 2023.
[11] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal recognizing textual entailment challenge. Proceedings of the ACI-PASCAL workshop on textual entailment and paraphrasing. 2007.
[12] Henry Gouk, Timothy M Hospedales, and Massimiliano Pontil. Distance-Based Regularisation of Deep Networks for Fine-Tuning. In /CLR, 2021.
[13] Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In ACL, 2021.
[14] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. ICLR, 2016.
[15] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, 2019.
[16] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. ICLR, 2022.
[17] Pavel Izmailov, Sharad Vikram, Matthew D. Hoffman, and Andrew Gordon Wilson. What are Bayesian neural network posteriors really like? International Conference Machine Learning, 2021.
[18] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
[19] Sehwan Kim, Qifan Song, and Faming Liang. Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Drifts, arXiv preprint arXiv:2009.05535 09535, 2020.
[20] D. P. Kingma and L. J. Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015.
[21] Insung Kong. Dongyoon Yang. Jongjin Lee, Ilsang Ohn, Gyuseung Back, and Yongdai Kim. Masked Bayesian Neural Networks: : Theoretical Guarantee and its Posterior Inference. International Conference on Machine Learning, 2023.

---

[22] François Lagunas. Ella Charlaix, Victor Sanh, and Alexander M. Rush. Block pruning for faster transform- ers. In Empirical Methods in Natural Language Processing, 2021.
[23] Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune
large-scale pretrained language models. In International Conference Learning Representations. 2019.
[24] Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-based pruning. In International Conference on Learning Representations, 2021.
[25] Hector I. Levesque, Ernest Davis, and I Leora Morgenstemn. The winegrad schema challenge. In Thirteenth international conference the principles knowledge representation and reasoning, 2012.
[26] Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In ICML, 2018.
[27] W. A. Link and M. J. Eaton. Methods in ecology and evolution. On thinning of chains in MCMC, 3(1): 112-115, 2012.
[28] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint (exxiv::9977 /692, 2019.
[29] Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learning. Advances in nearal information processing systems, 2017.
[30] Maric-Cathenin De Mamneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating
projection naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, 2019.
[31] Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. International Conference on Machine Learning, 2019.
[32] Eric Nalisnick, Jose Miguel Hormandez-Lobate and Padhraic Smyth. Dropout a structured shrinkage prior. International Conference on Machine Learning, 2019.
[33] Lorenzo Noci, Kevin Roth, Gregor Bachmann. Sebastian Nowozin, and Thomas Hofmann. Disentangling the roles of curation, aataettnmenn and the prior in the cold posterior effect. In Advances in neural information procesing systems, 2021.
[34] Kazuki Osawa, Siddharth Swaroop, Anirudh Jain, Runa Eschenhagen, Richard E. Turner, Rio Yokota, and Mohammad Emtiyaz Khan. Practical deep learning with Bayesian principles. In Advances in neural information processing systems, 2019.
[35] Nicholas Polson and Veronika Rockova. Posterior concentration for sparse deep learning. In Advances in neural information processing systems, 2018
[36] Melissa Roemmele. Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, 2011.
[37] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benehmark and analysis platform for natural language understanding. In In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018
[38] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanprect Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Supergluc: A stickier benchmark for purpose language understanding systems. In Advances in neural information processing systems, 2019.
[39] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019.
[40] Max Welling and Yee whye Teh. Bayesian Learning via Stochastic Gradient Langevin Dynamics. In International Conference on Machine Learning. 2011.
[41] Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang. Songfang Huang, and Fei Huang. Raise child in large language model: Towards effective and generalizable finetuning. Empirical Methods in Natural Language Processing, 2021.
[42] L. L. Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. International Conference on Machine Learning 2018
12

---

[43] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameler-efficint fine-tuning for transformer-based masked language-models. ACL, 2022.
[44] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lu- cic. Josip Djolonga, Andre Susano Pinto. Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski. Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 191.04990. 2019.
[45] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu, Neural Prompt Search. arXiv preprint XXiv:2206.0467. 2022.
13

---

Appendix
A Chosen Hyperparameters
We grid-search hyperparameters on validation, where two key hyperparameters are: the effective training data size N and the noise discount factor y (re: Sec. 5.1 The candidate sets formed E [100*,10** ", for NLP. and N E {10",10;,.. ..",..., ..0,,.,11)))) ,10%), for VTAB. The chosen hyperparametees are as follows (N, (NLP) cola (11, 10-)) stsb = (12, 10- mrpe = (12, 10") rte = (8, 10 10 cb = (10,100 (000100 copa (8, 10
as: N E {10*, 110",10".... 10.....,__.............. ,,0"2), 1.0')."....')... 7
y E (10-4 10-....0.....000 ".00,......."""""....
(10, 10 10=0: (VTAB) cifar100 (7, (0,10- caltech101 caltechh000
(9, 19.10-2, dtd (12, 10)). flower102 (7,10- camelyon
(12,
pets
(12, 10 svhn
100, sun397
(6, 10%), cu-
rosat = (7, 10100)) resise45 (12, 10 retinopathy = (7, 1,,10-)), clevr-count (7, 10 3) clevr-dist
= (7, 10 10 dmlab = (8, 10"), kitti = (7, 10"), dsprite-loc = (12, 10 10"")) dsprite-ori = (12, 10 10))), snorb-
azim = (7, 10 3), snorb-ele (6. 10 10-1).
B More Analysis
(NLP) Test accuracies at other sparsity levels. Although p 0.005 is recognized as the optimal sparsity level overall for the GLUE and SuperGLUE tasks. we evaluate the test performance of our Bayes Tune for different sparsity levels: p {0.01, 0.05, 0.1, 0.2, 0.3, 0.5). The average test accuracies are shown in Fig. 4 We see that overall there is less significant change in test performance so long as the sparsity level is small enough, and the resulting sparse updates selected by our BayesTune lead to equally good performance those with the default value. However, inereasing p further (c.g., p = 0.5) considerably degrades the performance, which signifies the importance sparse fine-tuning to avoid potential overfitting.
(VTAB) Scale posterior mean vs. sparsity level We visualize the plots that relate the sorted scale posterior means to the sparsity levels p in Fig.55 The plots are aligned with the the test accuracy plots analyzed in the main paper. As the plots are grouped along the optimal sparsity values, we see certain trends: for the sparse group (sun397 and cifar100), the scale À values are overall small scaled (in the range of [0, 0.2)) with sharp drops at small ); for the dense group: (camelyon and dmlab), Ả scale is even larger (in the range [0, 0.5)) with relatively smooth decaying at small values; lastly for the in-beiween group (clever-dist. dspr-ori, kitti, and snorb-ele). we have much narrower Ả ranges in between and 0.1 except kitti.
C Some Ablation Study
MAP vs. SGLD. The Maximum- A Posterior (MAP) aims to find a mode of the posterior distribution pt D)) and might be considered as an alternative solution to our SGLD posterior samples. In particular, MAP might be attained by turning off the noise term in (5), that is, by setting data noise than the
However, MAP be more sensitive of the posterior. We have some empirical comparison between MAP and SGLD the NLP tasks in Table and shows that this distinction does lead empirical benefit, Another benefit of SGLD
00.4
00.3
90.25
22
00.2
93.03
PUND
Sparsty
Sparsity
Sparsit level
Sparsty
Tolk eccumse
&
8
TwE
Figure 4: (NLP benchmarks) Test accuracies at sparsity levels other than the default p = 0.005. We evaluate the BayesTune sparse update models with p E {0.01, 0.05, 0.1, 0.2, 0.3, 0.5), where the default ones p 0.005 are shown as red square markers.
14

---

(a) Sparse group (cifar100 and sun397)
(b) Dense group (camelyon and dmlab)
(ppaee poup!
sunde7 (5parse group)
carneyyon (Dense group)
(Donse group)
42
0.70
0.4
0.3
3.50
02
0.66
0.03
0.1
0.03
0.0
02
9.4
6.6
88
3.0
0.
0.8
02
0.4
06
1.0
0.4
0.5
1.0
(c) In-between group (clever-dise dist, dspr ori, kitti, and snorb-ele
On betbenteenn)
depr-ont (Ir-bstween)
kottl (In-between)
(n-bbeweenn)
5
6.100
0.4
0.100
5015
0.015
5825
5.00
0.000
0.5
0.000
02
0.5
CA
Spanity
Figure 5: (VTAB benchmarks) The plots of the sorted scale posterior means Å vs. sparsity levels each of which is aligned with the the corresponding test accuracy plot. The plots are grouped along the optimal sparsity values where each group exhibits similar trends.
Method
CoLA
STS-B
MRPC
COPA
WSC
AVG
MAP
58.55
90.13
90.54
76.72
86.71
71.09
60.34
76.30
SGLD (Ours)
60.85
90.40
90.61
77.87
91.25
75.00
60.87
78.12
Table 3: MAP vs. SGLD results on NLP benchmarks. Averaged over 10 random runs.
we can also exploit the variance the posterior parameter selection (e.g., two parameters with similar posterior mean values, we prefer to select the one with smaller posterior variance). In future algorithms, we can also exploit this idea of variance-basce weight pruning.
Comparison to L1 regulariser. The recent L1 Ll-regularised sparse model lcarning (LI-SP) 42 used the LI regularisation term to encourage the updated weights to stay close to the original pre-trained weights. Thus this model might have similar effect approach to fine-tuning, however, is designed without main difference from the proposed BayesTune Tune. We have conducted some extra comparison experiments between the two approaches. Note that LI-SP does not necessarily lead to exactly sparse solutions. Therefore we offer two-stage extension of L1-SP. in the first stage, be updated are selected based on the relative LI distances from the pre-trained weights (i.e., taking those
BayesTune. The LI-SP can provide alternative regularisation-based
principled Bayesian modeling solution, which is
the LI LP training, and in the second stage those weights
Method
CoLA
STS-B
MRPC
RTE
CB
COPA
WSC
AVG
L1-SP (stago
50,50
88.07
84.54
50.00
62.44
60.40
52.88
64.12
LI-SP (stage p)%)
54.59
89.25
55.38
72.64
BayesTune (stage (100 9)%)
60.85
90.40
90.61
77.87
91.25
75.00
60.87
78.12
Table 4: LI-SP 42
vs. BayesTune results on NLP benchmarks, Averaged over 10 random runs.
15

---

(100 p) % weights with the largest relative changes from the pre-trained weights). The results on the NLP lasks shown T
For the L1 penalty balancing constant hyperparameter. choose optimal values by grid search from {10->, 10- 1005 10_"). We can see that LI-SP considerably lags behind our BayesTune, which mainly attributed to its failure to capture uncertainty in L1 regularisation, thus being potentially sensitive to noise in data (a similar reason as the MAP estimate). Morcover, only penalising the parameters deviation from the pre-trate pre-trained weights in Stage significantly underperforms the sparse cut-off' strategy in Stage 2, signifying that sparse update is critical.
D Layer-wise and Module-wise Sparsity Patterns of Tuned Networks
Sparsity patterns of RoBERTa-base NLP tasks. We visualize the module-wisc and layer-wise sparsity patterns of the BayesTuned RoBERTa-base networks on 7 NLP tasks in Fig. T9 First, for the layer- wise sparsity pattern: (Except for mxpc) The proportions of the selected updatable parameters are more or less uniformly distributed across the 12 layers of the Transformer, while the first word embedding layer and the last classification layer are significantly less and more selected, respectively. This is intuitively appealing as the task-specific features may tend to be determined at the higher, more global levels in texts/sentences, account for Longer-range dependency. Next, looking the module-wir non annuunffom, layer-spes speciffi,, and also task/dataset -eleaaaasse--dedeentttt For instance, the bias modules in some layers are very densely selected, while they are very sparsely selected in other layers. This shows clear discrepancy to the heuristic strategies like BitFit [3 in which the bias modules selected 100% for all layers.
sparsity patterns,
proportions are highly
Sparsity patterns of ViT-B/16 VTAB vision tasks. The module-wise and layer-wise sparsity patterns of the BayesTuned ViT-B/16 networks on VTAB henchmark datasets are shown in Fig. 20 We also superimpose the optimal p values (dataset dependent). The resulting patterns are quite similar to the NLP case: Except for a few cases, the lowest level visual prompt layers are selected far less, sometimes ignored, compared to the later layers. The last linear classification head, although not shown here in the sparsity diagrams, is selected 100%. Overall the layer-wise selection patterns are nearly uniform while the module-wise selection patterns highly non-uniform and datasct dependent.
16

---

cola (Module-wise sparsity pattern)
owhonrlegs and bnboooiigccssiiee enbesonsaatien ioiinn tops embedding insign
anender bigor aphortion query wnight
ortartion
lager atterton loy angg
atterdin
unteadiing Landhamm bas onc secodde leye ai diwiton sel quers bat secaden byerd shorton cop bry has secidel loyer attertion sal walle eecdde layee atrer
excode amtendion soi lut wegi
risver
encode bayor artemoon ouzgut dorse woort
oegode loyers zhtrben cumist Leyeniimm wrggtt
if
avander ayer intor
woight
reeimeditt, denon
denee
music layer' ougue derse
encoder bypedd
encoder apero attrthn ootase Eyeellom
enoader nyer cumnet unyerwom anggre
aecoller
secedaa leyur
be
encoee layer artoronn so8 suer wwoot
Layeriom
mnaii Wowget donuc bup
crundo
moddrr bygnt se mono in arage
auade loper
Loyo wagfe
meoder byer aetettion self
weggt
arceder attuncion
serayee atterhon sak
ywogft
ercode layer atteniion oaruue desse
arcedar attereinn nattrplun toggte
wecsow now attertan loyseggh
secoden athrben
oegade lyyrr aeberbon cumstt Lnyenioom wngtt
cvooter oyer momo edete doece wng
it Layellam
arcuderlayen attect aitout dense
onmaee tyyer's oumpue
encoder byer astenoon ouccut Loyonk moight
wocser layer corput Layinti
encodoriovel Rortedicts dendoweepp werehrbyrrr on wndt Syen oumpue don
encoeer tyyer ameetinn query weight
encoder byerd afteroon soi l0 kaget
evoue loyer out Loyent weggle
meseer byrra cemenin
encoder ayera aetrrbnn
wecddrr layst listion vellicse yoogle
wordee bayer atteetion curpue
whe all ar arllin will weight
excade loyers is curpat
veceder layer disetion well weogle
ensster
evotter ayen memm sore dnece gni
creodetloy loyer attert output donce
entarel byyea output doeze angee
denco
encoder byeer wetettonn euruue myyen weighe
recoder typor rungut toyintinn arigge
encodoring oe
denee
ModiA to staro
wocdde layer wetertion seld
query
monerr Syer eveu demon
encdde lypers aeeriton
encderlnyer::
ancedar laver attneion selloue ywoigte
urcodar layer cntinin sall wele osigle
croude
cacode layer atter bon sel duers
cacode byyer attorher est sersaaan
quou laye
lyye ac
wheer surputt derse
hoywesggt
Murm
buy
wegple
warade byara
winder attection if
mecdde byyer aetrenon corpue deete wragre
Scende attintion
weside ayen ineermeiio coigh
encoder afheebnn soe va weott
monter
encodel ayecd atterionn
entace byyer curpue deese weight
riseer
curpat
weighs
incodl lention
encocer nyeer rurest Lnverlo wnggtt
serederlape layer dhemreiet derse
mocodee oyer ommetonn cnll waggtt
dom
durce
wessee uyee whhetton
masdde 10
werde attertion
anceder
oacedel typer amhrndnn cot
evo les layer selfour
encadw aper attertinn surzet
attertion self
dende
mooder byee attent cutrut Loye sougee
odle leyar
will be poss
encsited
ancoder theme
escodei layer atterbicn sol velle
sccude
encodeting layer atturt outout dense bay
seenderlage
eecoder loyer corput lnyer
ancedle aapar allertann out aud Layel Nur
encodiring
mncoder nowr wheeton selt query
orcedarbyerr, Peemecictt dence
anoodie byerr currue dance Sar
oper weerthon
eecaeer attntton
ennnter myer suesul
wncaow laper
encoded loyer's antertion sol cvery Dass
ancaser ana alt
encoder kyee attent outpott
ayer abbutior bay
ancider
ircedar ditetton sel
en encoder lavor momm
uncndel layer
croodc allirt ou deruct
layet
arode
encode kayct innttinn cuistt
ercoedilyerr ougget
encodle
weenter pherradaate
oeigle
p0001
seraderlyeer ruttst denie werght
tasinsodess dict cola hold
wendaring inyve dence
ernderlayeed autpul Laverterm werght
taokmodele Cora kedd out Ling
cutSLE LayenNerm
L Paramctors (961 Mocul
Updotzbli Porometers 1961 in Modu
Figure 6: Sparsity pattern of the modules in RoBERTa-base on cola. Module-wise updatable parameters (%). (Left) The first half of the network and (Right) the second half. The default sparsity level p" =055 is shown as vertical line.
17

---

cola (Layer-wise sparsity pattern)
owhonnlege and woige embedonna tosinn tops embeddmit insign
enender bigori aphortion queny wright en lager attertion self anigi
atterdinn selfeey
unteadinng Landhamm bao
excoder liyer amtendion soif la wegg
onc secodde leye ai dilitinn sel quers Dat
encode byyor artemoon outgut dorse woott
secaden byerd shorton cop bey bat
oecodde loyers zhtrben cumist Leyeniomm wnggtt
if
secidel leyer attertion sal walte
avander ayer intor
woight
reeimeditt, denon
eecdee atter
dense Sas
music layera ougue derse weigt
encoder byeed
encoder apero attrthn ootate Eyeeloom
enoader nyer curnst unyenwom anggr
aecoller
secedal leyur rat danee
weight
encoee layer artoron so8 suer woopt
Layeniom
anaii dowgatt donec bup
crundo
moddrr byen se
auade loper
Loyo wagfe
meoder byer aetettio self
weggt
areedar attuncion
verayee atterhon sak
ywoift
ercode layer atteniio oaruee desse welptt
arcedar attereinn nottrplu oogghe
sacadellyy byyrr
secode aharhen
now attertan loyarggh
oegade lyyrr aeherbon cumstt Lnyenioo wngtt
cvooter oyer mome edete doece weigh
it Layellam
arcoderlayer sttecton aitout dense
onmaee tyyer's oumpue wrigh
encoder byer astenoon outcut Loyonk movigtt
Layehoom
wooser layer corput Layinti weight
encodoriaven Rortedicts dendoweeppe
werehrbyerr on wndd Syen oumpue don weihh
encoder tyyer ameetinn query wrighe
arterben
encoder byerd afteroon soit l0 kaage
evoue loyer out Loyent weggle
meseer byrra
encoder ayera aetreenn
wrigt
wecdder layst iiniion vellicse yoogle
wordder buyer atteetion curpue wright
orteeton end
whe all ar aellinsed weygt
excade loyers is curpat Layen weight
veceder layer disetion wel weogle
ensste
evotter ayen memm fore dnece soight
creedetloy loyer attert output donce woiple
entarel lyyea cuttut doeze angpt
dence
encoder byeer wetetttnn euruue syeem weighe
recoder typor rungut tyyintio arighe
encodoring over idorrctitt denee
ModiA stare
wocdde layer weeetton query weight
recoder
syertom
monerr Syer eveu demon
encdde lyper aeertton
seld
wright
quou laye Luyettanm wegll
encoder afheronn vt westtt
encodel apecd aetrrionn
encderinyer::
ancedar laver attneion selloue ywoige
urcodar ietiiion vellvelee osigle
cacode layer bon sel duers
cacode byye attorher est bas-
lyye ac hoywesght
wheer surput derse
attar
buy
marade byara
minder attection
mecder byyer aetrenon corpue deste wragt
Scende attintion
weside ayen
monter ineermeiio de
entace lyyer curpue deese weight
riseer
curpat tnyintiom weight
if
Geeae cvight
incodl lettion
weender oftertton dhemeeiet derse
encocer nyeer rumist Loverta anggte
Lyyellarm
serederlape layer
dom
dance
mecodie oyer oemetonn cnll waggtt
wessee uyee whhetton
massde 10
nelt
ancoder
evoider
dacedel uypra amhrnon cot bay
evo les laye seffower
encadwr aper attertinn surzet weotth
werdee attertion
attertion self
dende
mnooeer byer attent cutrut Laye souggt
odle layar al wnlin will toss
encsited
ancoder theme
deene wesgne
escodei leyer atterbicn sol velln
sccude to
encodeting layer atturt on output dense bay
seenderlage derse weighs
eecoder lnyer corput lnyer weighs
ancedle aapar alleriann coti aud Layel Nur
Nemm
encodiring
mncoder nowr wheethnn selt query westth
ercedarbyerr, Peemecictt dence
anoorie byer currue dance
oper weertton an
wecdd attert
ennater myer suesul
wncaow laper
encodel loyer's antertion sol Daas ancaser ana alt abuntior ss Bay
dence
encoder kyee allent on outpott Nom. wdggh
ircedar ditetton sel
en enceder lavor mome
sncndel layer
enceda
layet
arode
doruc
encode kayct innttinn cuistt
encodle
eccoedllyeraa ougget
wagg
weenter
p0001 wog
seraderlaeer rumtst denie
andder
tasinsodess dict cola hald weggt
wendaring 2noderingee dence
ernderlayeed autpul werght cutSLE LayerHerm
taokmodeee Tola kedd out Long
10 1/x
36
Jpdosoalc Po Porameters (9 Layor
Upsotable Poromotors (90) Laye
Figure 7: Sparsity pattern of the layers in RoBERTa-base on cola. Layer-wise updatable parameters (%). (Left) The first half of the network and (Right) the second half. The default sparsity level 0055% shown as vertical line.
18

---

stsb (Module-wise sparsity pattern)
owhonllegs and oniio weigt embesonsattinn hossnn tops emboddmi insign
anender bigor attortion query wnggt
ortartion
lager atcerton
atterdin sell
unteadinng Landhamm bas onc secodde leye ai diwiton sel
excode amtendion soi salut wdgi
risver
encode byorr artemoon ouzgut dorse woote
secaden shorhon bey has secidel loyer attertion sal
depodd layers zherben cumist Leyeniom
if
avader ayer mnto
eecdde atrer
dense
music layero ougue
encoder byeed
encoder apero attrrhnn ootsse
enoader nyer cumnst unyerwom
secoller
secedaa leyur
encoee layer arooron so8 quer woott byent
it Layeniam
auade lope
donec
meoder byer aeordan
arceder attuncion
seradee wtterhon sal
ercode layer atteniio oaruee
secade atarbon
now attertan
oeggde ayar? aeberbor cumist
arcedar
sacade dyyer attertor oogghe
cvooder oyer mom doece
arcuderlayen
mnmaee tyyer' oumuue
encoder byer astenoon outnit Loyonk magght
wooser corput
encodoriavel Rortedicss dendbwellp
werehrbyrrr on
encoee tyyer ameetinn query
arterben
encoder byerd afteroon sot
evoue loyer
meseer byrra
ayera aetreen
uecdder layst
wodde byyer
curpue
whe all arllin
eagdd layers i curpat
veceder layer disedion vellvelee
mnnter
evotter ayen memm dnece weight
creedetloye loyer attert output dones
entarel lyyra output doeze
rncomedio dence
encoder byeer
recuoee bypor cangut
wrigge
encodering ee
mocdde layer vetertion seld
query
encode lyper aeertton
encodel apecd attrdion
ancedar laver attneion selfoue ywagte
cacode layer atter sel
quo laye
aar mazade byara
mecdde byeed aetrenon curpue deate wnagte
Sernde attintion
encoder afhebonn soe ott weott
risver
encderinyer::
urcodar cntin call
cacode byye
lyye ac bay
autout
minder attectinn
wxside ayen
monter
entace' lyyer curpue deese
curpat
weighs
inccdl lettion
encocer nyer rumist Lover wegght
serederlape dhemeeiet
mocoeee oyer ommeton cnll waggtt
masdd 10
nelt
wesser ayee whhetton
werde attertoon
anceder
dacedel uypra amhrnonn cot
evo les laye seffour
encadw aper artertinn surzet weott
attertion self
dende
mocoer byer attent cutrut Loye sougge
encsiter
aecoder theme
deene wnigne
escodei leyee atterbenn sol
sccude
encodeting layer atturt output dense
seenderlage
weight
eecoder lnyer corput lnyer teom weighs
ancedle sapar altertan
encodiring
mocoder nowr wheethnn selt query westth
ercedarbyerr, Permecicto
anoorie byer
oper weerthon
ennater myer suesul
encodel loyer's antertion sol
encoder kyee allent outpott dence
laper
ancaser and alt
abuntior bay sncndel layer
ircedar
en enceder lavor mom
enceda
layet
andde
dorcc
encode kayct inntiinn
eccoedllyrra ougget
encodle
weenter
p000
serndelaaeer
mndder rumnst denie
tasénoodes dict cass Hold dosce tackmodek, det tos
mendaringee
weight ernderlayerd cutSLE LayerNerm
3.5
Updatabic Porometorr (P6) Modul
Updotablc Parometers (90) In Hodu
Figure& 8: Sparsity pattern of the modules in RoBERTa-base on stab. Module-wise updatable parameters (%). (Left) The first half of the network and (Right) the second half. The default sparsity level p" 0055 is shown as vertical line.
19

---

stsb (Layer-wise sparsity pattern)
owhonnlege and onioo embedonna iosinn tops embeddmit insign
anender bigori aptortion queny wright en lager attertion self aniget
atterdinn seileey
unteadinng Landhaam bas
excoder layer artendion soif salut wegg
onc secodde leye ai diwiton sel bat
encode byyor artenoon oucgut dorse woott
secaden byerd shorhon cop bey bar
oegodde loyers zhtrben cumist Leyenlomm anget
if
secidel leyer attertion sal wall
avander ayer intor
waggh
reemmeditt, denon
eecdde atter
dense tas
music layera ougue derse weggt
encoder byeed
encoder apero attrthn ootsse syerloom
enoader nyer curnst unyeraom adgre
secoller
secedaa leyur
weight
encoee layer arooron soi suer woppt
Layeriom
arceder attuncion
auade loper
veradde atterhon sak
anaii dowgat danec bup
crundo
uyo oo ywogft
das
ercode layer atteniio oaruue desse welpht
meoder byer geordan
moddrr byen se
self
in arage
wegst
arcedar
sacadellyy dyyer attertor
secade aharhen
now attertan loyarggh
nottrplu onggte
oegade lyyrr aeherbon cumstt Lnyenioo wogtt
ovodter oyer mome edete doece wnggh
it Layerlam
arcoderlayer sttecton aitout
dense
onmaee tyyer's oumpue
encoder byer astenoon outnit Loyon mavggt
wosser corput Layinti
encodoriaven Rortedicts dendoweepp
werehrbyrrr on wndd Syen oumpre don weigh
encoder tyyer ameetinn query weighe
arterben encoder byerd afteroon soi l0 kaage
evoue loyer out Loyert weggle
meseer byrra ayera retreenn
uecdder layst listiin vellouse yoogle
wordee buyer atteetion curpue wright
whe all an aellinned weight
excade loyers is curpat
weight
veceder layer disetion wel weegle
enssee
evodter ayen memm eoze dnece seigrt
creedetloy loyer attert output donce woiple
entare! lyyra cuttut doeze angge
dence
encoder byeer wetetttnn euruue syee weighe
recoder typor rungut
arigee
encodoriog over
denee
ModiA to stamo
wocdde layer weeetton query weight
recoder
moner Syer evmu
encdde lypers aeertton be wright
seld
ancedar laver attnnion selfoue ywoige
cruste
cacode layer atter sel duers
quou laye
wagle
mecder byyer aetrenon corpue deste wragte
Scende attintinn
encoder afhebonn vt westtt
encodel apecd aetrrionn
riseer
ercodarinyer byye attorher
urcodar layer ietinion wele osigle
lyyr ac buy
warade byara
minder attection if Geeae evight
wesdde ayen
menter ineermeii de
curpat tnyintiom weight
entace lyyer curpue deese wright
inccdle leetion
encacer nyee rumist Lnyertar anggtt
serederlape layer
dhemeeiet derse
mecodie oyer oemetonn cnll waggth
10 wessee ayee whhetton on wesggt
Lyyellarm
nelt
weedee attertion valuee loiggn
onceder
dacedel uypra amtrnonn cot
evo les laye seffower
encadwr aper artertinn surzet desse weotth
attertion self
dente
mooder byer attent cutrut Laye souggh
odle layar
encsited
ancoder theme ente deene wasgne
escodei layer atterbinn sol
sccude to
encodeting layer atturt on outout dense
seenderlage derse weight
eecoder lnyer corput lnyer weight
ancedle sapar allertann aud
encoderin
mncoder nowr wheethnn selt query westth
ercedarbyerr, Peemecictt
anoodie byer currue
oyer weertton an wasstt
ennater myer suesul
encadel loyer's antertion sol ancaser ana alt
encocer kyee allent on outpott dence
wncaow laper
ircedar ayer abuatior ss Bay
en enceder laver mome
sncndel layer
enceda
layet
ardde
dorcc
encode kayct inntiinn cuistt
ercoedilyerra ougget
encodee
weenter
p0001
seraderlaeer rumtst denie
tasinsodes dict cass wold dosse
wendaring in
weight ernderlayeed cutSLE LayerNerm
taskmodeke det tos bose out
Jpdosoal: Po pormmtters 2961 In Layor
Updatable Poromotors (440 Laye
Figure 9: Sparsity pattern of the layers in RoBERTa-base on stsb. Layer-wise updatable parameters (%). (Left) The first half of the network and (Right) the second half. The default sparsity level 0055% shown as vertical line.
20

---

mrpc (Module-wise sparsity pattern)
owhonnlegss and bnbbooiigggouuigg embesonsattinn hoiinn typs emboddmits insian unteadinng Landhaam bas onc secodde laye ai dietton sel quers baa secsden byerd shorton cop bey has secide leyer attertion sath walte eecdde layee atrer
anender tigore aptortion query wright coder lager atcerton soll anight
attrrdin selfee
excode layer amtendion soi salut wegi
encode byyrr artemoon ouztut dorse woott
oegode lyyers aetrben cumist Leyeniom wnggt
avader ayer mnmo
rteimedits wiight
dense
music layero ougue weggt
encoder byperd
encoder apero atterhin ootaue Eyeellom
enoader nyer cumnit unyenwom adgre
secoller
seceda layur nat be
encoee layer arornon soi quer woott
Layeniom
mnaii dowpat donec
moderr byent se mono
anade lopes
Loyoll
meoder byer aetettin self
weggt
arceder laye attuntinn sak
ywogft
ercode layer atteniion uaruee desse
arcedar baee atteriinn sattyplun coughe
wecsow now attertan boy
secaden
oxgode ayar7 aeterbor cumitt Lnyenioo wngtt
ovodter oyer mome edete doece gng
Layelomm
arcoderlaye allet
actout danse
mnmee tyyer' oumpue
encoder byer astenoon outnit Loyonk mavght
wosser layer corput
encoderiovel Rortedicts dendoweepl
werehblyrrr on wnde Syen oumpre don waihh
encoee tiyerd ameetin query
encoder byerd atteroon sot kaae
mvode layer oud Loyent weggle
meser byera cenenon
ayera attreenn
wecddr layst lisdiin vellicse woogle
wodde byyer attemton curpue wrigh
whe all an arllin will weyge
emgdl layers a aleetton carpat
weight
veceder layer disetion vell weogle
evotee ayen memm eore deece weight
creedetloy leyer attert output dones
entarel byyea cutput doeze angge
dence
encoder byeer wetertton exppee syyem
recoore typor rungut to wrigee
Modil SART
encodoring ee
denes
mocdde layer weeetton query weight
monerr oyer oumu
encdde typers aeeriton
well
aright
encoder apecd aetertonn
ancedar laver attnnion selloue ywogte
cacode layer atter set duers
quou laye
recdde byeed petrenon curpue deste wnagt
Sernde attintinn
encoder afheeonn soe valu weott
riseer
urcddar layer cntin sall wele osigle
cacode byye attorher est
encadal lyyr ac
monder
axsid ayen mem
byara attetttnn
monter reemmeeii ovight
curpat tnyiniiom weight
iecderlnyer:: wheer surput derse
entace byyer curpue deese weight
incodl lettion
anceder syeec aftertinn dhemeeiet derse
encocer nyer rumest Lnver wngght
Layellam
evoider layer dorce
peredeelaeer
dom
moooee oyer oemetton cnll quer waggth
wesser uyee whentton ln wesgtt
10
werde laper attertoon
ancoder
oaceder uypra amornonn cot
mvo les laye seffour
encadwr aper attertinn surzet weott
eacade attertinn selt
dende
mocoer byer attert cutput Laye soughe
odle loyar al
encsiter
aeceder thene ente deene snggee
excodel leyer atterbenn sol vele
sccude
encodeting layer atturt output dense
secnderlaye
weighs
eecode loyer corput lnyer teom weighs
ancedee sppar allertan cot aut Layel Nur
encodiring
mncoder nowr whenton selt query westth
orcedbrbyerr Peemecicte dence
encicer oyer wherthon
oncads th byer curpue dunce
eccoeet
ennnter myer suesul syeettom
wndee laper cerertion
wntertion
eacoded loyer's antertion sol qvery bass
en encoder kyel attent on outpott dence
ancaser ana alt
ayer abbatior bay
grcii en enceder lavor mome
ircedar laper ditetion sel
sncndel
onceda
laye
andde lye wo
encode kayct innttinn ouiset doroc
crcode doyeray ourget uoy wagg
encodle aetertion
wennter pherradaate
oeigte
p0001 wog
serndelayeer numns denee weggt mndder
tockmoodk fit mrpe Hold dosce
mendering inyve dence
eenderlayeer autpul Laverierm weight outSLE LayenNerm
lasemade dicpp hose eut weggee
0.50 825
1.00
Upsotzable Paromators (561 Mocul
Updotoblic Porometers 1961 in Modu
Figure 10: Sparsity pattern of the modules in RoBERTa-base on mrpc Module-wise updatable parameters (%). (Left) The first half of the network and (Right) the second half. The default sparsity level p" 0.55 is shown as vertical line.
21

---

mrpc (Layer-wise sparsity pattern)
owhonnlegss and subio embedoonaatin coisnn typs embeddmis insinn
anender bigore aptortion query wnight ortartion
laper attertoo self loy aniget
unteadinng Landhaam bas onc secodde leye ai diwiton sel quers baa
excoder layer amtendion soif salut wegg
encoder buyor artertio ouztut dorse woott
secsden byerd shorton cop bry bar
depodde loyers aetrrbn cumust Leyeniom adget
secide leyer attertion sat walle Sas
avader ayer mnto adicto waggh
denoe
eecdde layee atter curipe danse Sas
muse layero ougur derse weght
encoder byperd
encoder apero attrthnn ootase syerloom 203
enoader nyer cumnst unyenaom adgre
aecoller
secedal leysr rit be danee
wsight
encoee layer aroonon soi quer woopt
anaii wouast darec
weight
moderr byent se mono to arage
anade Joyes
uye ooc Nurm das
Loyol
mooder byer aetettin self
wegst
arceder attuniinn seffoue ywogft
ercode layer atteniion oaruue desse wegtt
arcedar attertion wottrplu wnggh
wecsow now attertan loyaeggh
aberbon
oetode: ayar? aeherbo cumst Lnyenioo angtt
ovodter oyer mome edite doece sdge
Layellam
arcoderlayer sttecton aitout danse
mnmaee tyyer' sumpue
encoder byer astenoon outnit Loyoni mavght
wosser layer corput Layinti weight
encodoriovel (Romiedicts dendowelple
werehbbyrrr on wnde Syen oumpue don weihh
encoeer tyyer ameedin query weige
encoder byerd atteroon sott kaar
mnoue layer out Loyent wegge
moseer byrra cenenon
encoder ayers attreenn
wecddr layst aistiion vellicse yoogle
eorode byyer attemtion curpue wrigh
whe all in bay weigt
escacl layers is aleetton curpat Laye tome weight
vecedrr layer diaetion wel weogle
evotter ayen memredoo eore dnece snight
creedetloy leyer attert on output donce wogple
entarel byyra cuttut doeze angee
dence
encoder byeer wetertion exppur syee weighe
recoore typor rungut toyintiom arige
encoderin ove rdorrcdttt deves wernt
Wood Mo stamoo
mocdde layer weeetinn query weight
monerr oyer eumu demon wnight
encdee lypers aeerton be wreght
sell
incderrnyeer: wheer sutput derse
ancedar laver attneion selloue ywogte
urcddar citiiinn well osigle
cacode layer atter set duers
cacode byye attorher sotuel bat-
quou laye Luyettumm me
encadal lyer ac ceriin buy
tin hoyweggh
Nur
warade byarr attrcttnn
winder
recode byeer petrenon curpue deste wnatt
Scends attintion
wxsid ayen reemmeein dense
encoder afheeonn soe vt weott
monter
encoder apecd atterionn
entace byyer curpue deese weight
riseer
curpet tnyiniiom weighs
Geeae ovigi
incodl lettion
anceder syper aftertton
peredeelaeer
laye dhemeeiet derse
layer darce
domos
dence
mocoeee oyer oemetonn cnll wagghh
wescer uyee whhetton bn wesgtt
encocer nyer rumist Lnverta anggt
masdde 10
Layellam
ealt
werdee laper attertoon vaice loigr
ancoder
oaceder uypra amornonn cot suery bay
mvo ler laye seffower Nur
incodwr byer attert cutrut Loye sought
encadwr aper 10 attertion sureet weoth
eacade
attertion self
dence
odle leyar al wrlen be pas
encsiter
anceder thene mnte deene snggee
excodellng leyer atterbern sol vele
sccude
encodeting layer atturt outout dense day
seenderlage derse weight
eecode loyer corpat lnyer teom weighs
ancedee appar altertann cot atd Layel Nurm
encodiring
mncoder nowr whenthon sell query westth
orcedbrbyerr, Peemecicto dence
anoorer
ensicer oyer whenthon an wasgtt
de byer currue dance Bar
encoder
wecdd laper athert
wnnner myer suesul syeetomm Say
wndte laper eerertion
wncaow
wntertion weggh
eacuded loyer's antertion sol cvery Das
dence
ancaser ana alt
ayer abuatior lo hogooiigt
encocler koye attent on outpott Laya Nomm wdgic
arciii en enceder lavor mome
ircedar ditetion sel noogle
sncndel loyer
layet
on deroc
arode lye wo
croodc layer allnt
encode kayct ianttin ouistt
rocoedlyyera ourget uoy wog
encodle altertion
weenter pherradaate
weigte
p0000 wool
serndelayeer
andder hqee numtst denie weigt mendering
tockmoodks fit mrpe Hold dosse wegg
dence
eenderlayer: autpul Laverierm weight outsuE LayenNerm
lademode deppp hose Jun wekge
0.5
Jpdosoblc Pa Porameters.ooo 2961 Layos
Updotable Poromotors (46 Laye
Figure 11: Sparsity pattern of the layers in RoBERTa-base on mrpc. Layer-wise updatable parameters (%). (Left) The first half of the network and (Right) the second half. The default sparsity level 0.5%% shown as vertical line.
22

---

rte (Module-wise sparsity pattern)
owhonrlegs and weigo embesonsaatinn cosinn tops emboddmit insign
anender bigori aptertion query wnight
en lager attertoon solf liy angget
atterdion
unteadinng Landhamm bao
excoder amtendion ka weggtt
risver
onc secodde leye ai dietton sel quers Dat
encode byyor artemoon outgut dorse woort
secaden byerd shorton cop bry hat
oecodde loyers zhtrben cumist Leyeniiom wrgght
if
secidel loyer attertion sal walle Sat
avaneer ayer intor
rteimeditt, woight
eecdee atrer
danee Saa
music layer' ougue derse weggt
encoder byped
encoder apero attrthn ootate Eyeeloom 2ay
enoader nyer cumnit unyenwom anggre
secoller
secedal leyur rat cheree
wsigle
encoee layer artoronn soi suery wwoot
Layeriom
mnaii dowgast daywe wiight
crundo
donuc
moddrr byen se mino in aragee
auade loper
uyoL oOC Nurm
Loyo wagfe
mooder byer
weight
areeder attuncion
veradee atterhon sak
ywoift
ercode layer atteniion oaruue desse wektt
self
arcedar attereinn nottrplu woughe
sacadellyy dyyrr mitout dense
secode athrhon
now attertan loyneggh
oegade lyyrr aeberbon cumstt Lnyenioom wogtt
cvooter oyer mom edete doece wei
it Layeliom
arcoderlayer
onmaee tyyer's sumpue
encoder byer astenoon ouccut Loyonk moiiht
wooser layer corput Layinii weight
encodoriaven Rortedicts denoowweppe werehrbyerr on wnddd Syen oumpue don waihh
encoee tyyer ameetinn query wiighe
encoder byerd afteroon soi t0
evoue loyer out Liyenti weggle
meseer byrra
encoder ayera aetreenn
wecddr layst liniion vellicse yoogle
wordee buyer orteeton
curpue
whe all ar arllinsed will weigh
excade loyers is curpat
veceder layer disetion welivelee weogle
ennste
evotter ayen memm sore deece goighe
creedetloy loyer attert output donce
entarel lyyea output doeze angee
dence
encoder byeer weterttnn euruue myeen
recooer typor rungut toyintinn wrige
encodoring ove idorrctitto denen
ModiA staro
wocdde layer vetertion seld
query weight
monerr Syer eveu demon
encdde typers aeertton be
weople
encoder afheebnn sot tut weott
encodel ayecd aetertonn bey
ercodarinyer
ancedar laver attneion selfoue yooigt
urcodar intinin sall wele osigle
cacode layer atter sel duers
cacode byye attorher est
quou laye
lyye ac
wheer surputt derse
tin buy
macade byara
winder attection
mecdde byyer aetrenon corpue deece wragee
Scende attintion
weside ayen ineereeiios
monter
entace byyer curpue deese wright
riseer
curpat inyiniiom weight
if
Caeae coight
incodl lettion
syper aftertton
ferm
encocer nyeer rurest Lnver angght
serederlape layer
laye dhemreiet derse
dom
mocodee oyer oemetonn cnll wnggt
10 wessee uyee whhetton
duric
werdee attertion
evo les laye seffour Nurm ancoder
attertion self
oacedel uypra amtrndnn cot suery day
encadwr aper artertinn sureet weotth
dende
mooder byee attent cutrut Laye souger
odle leyar al an will bos tuss
encsited
ancider theme
escodei leyer atterbicn sol velle encodeting layer atturt on output dense day
sccude
seenderlage
eecoder loyer corput lnyer weigh
ancedle aapar allertann cot aud Layse Nurm
encodiring
teepht-
mncoder nowr wheethn selt query
orcedarbyerr, Peemecictt dence
oper weertton
oncads byer currue dance Bar
eecoeet attntton
ennnder myer suesul tyeetbom slat
wncaow laper
encoded loyer's antertion sot cieryy iaas
encocer kyee allent outpott dence
ancaser and alt
ayer abuntiar bay
ircedar ditetion sel woogle
en enceder laver mome
sncnde layer
croodc allirt oo doruc
layet
arode
encode kayct inntiinn cuistt Loycil
eccoedllyer.a ourget
ercodle aitertion
Morm
weenter pherradaate dence
oeigte
p0000 wog
serndelaaeer
mndder ruttst denie weight
tackmodkk dituuta wood dosse
mendaring In dence
eenderlayer: autpul Laverterm werght cutSLE Layerlerm
tadiemder de hoad aut wegge
do 7/ 160 12.
Uppotzalle Foromctors (56) Mocul
Updotobli Porometers 190 in Modu
Figure 12: Sparsity pattern of the modules in RoBERTa-bar on rte. Module-wise updatable parameters (%). (Left) The first half of the network and (Right) the second half. The default sparsity level p" 0.5% is shown as vertical line.
23

---

rte (Layer-wise sparsity pattern)
owhonrlegs and oniio enbedoonaatien ioiinn tops embeddmit insign
enender bigor aphortion queny wnight en lager attertion self anigit
atterdion seifeey
unteadinng Landhaam bao onc secodde leye ai diwitin sel quers Dat
excoder liyer artendion soif la wegg
encode boyor artemoon outgut do15e woott
secaden byerd shorton cop bey bat
oepodde loyers aetrren cumist Leyeniiom wnggtt
if
secidel leyer attertion sal walle eecdde layee atter
avander ayer intor
woight
reeemeditt, denon
dense Das
musie layer' ougue derse weigtt
encoder bypedd
encoder apero attrthnn ootaue Lyeeloom
enoader nyer cumnet unyenwom anggre
secoller
secedal leyur rat
encoee layer artoronn soi suery wwdptt
Layeriom
mnaii dowtart donuc bup
crundo
moddrr bygnt se mono in arigge
auade loper
Loyo wagfe
meoder byer aetettio saker weigt
arceder attuncion
verayde wtterhon sak
ywoift
ercode layer atteniion oaruue desse welpht
arcedar attereinn nattyplun coughe
wecsow now attertan boyarggh
secode athrhon
oegade lyyrr aeherbon cumstt Lnyenioom wnght
cvooter oyer mome edete doece wei
it Layellam
arcuderlayen sttecton aitout dense
onmaee tyyer' oumpue doece wright
encoder byer astenoon ouccut Loyonk maviggt
wocser layer corput Layinii weight
encodoriavel Rortedicts dendowwepp
werehrbyrrr on wndd Syen oumpue don weihh
encoder tyyer ameetinn query wright
encoder byerd afteroon soit l0 waggee
evoue loyer out Liyenii weggle
meseer byrra cenenin end
encoder ayera attrrbnn
wrigts
wecddrr layst listiin vellicse yoogle
wordder layer atteetion curpue deese wright
whe all ar arriin will weigt
excade loyers is aleertion curpat Layen tame weight
vecedrr layer disetion well weogle
ensster
evotter ayen memm sore deece goi
creodetioy loyer attert output donce woiple
entarel lyyea outtut doeze aeggee
dence
encoder byeer wetetttnn euppue myyen weighe
recoder typor cungut tyyintio arigge
von
encodoring ov
denee werpte
Modil to staro
wocdde layer weeertin query weight
syertim
monerr Syer eveu demon
encdde typers aeeriton and bet wriign
seld
encderinyer::
ancedar laver attneion selloue ywoige
incodar layer intin call wele osigle
cacode layer atter sel duers tai
cacode byye attorher est bas-
quou laye Luyettumm wegple
lyye ac
wheer surputt derse
tin hoywesggt
Nurm
bay
warade byara
winder attection if syeetoom
mecode byyer aetrrnin corpue desce wragre
Scende attintion
weside agen ineereeiio
encoder afheebnn sot tut westtt
montee
encoder ayecd aetertoon trttlrytiigg
entace lyyer curpuet deese weight
riseer
curpat tnyiniiom weiggt
Ceeaue coiign
incodl lettion
seper oftertton dhemreiet derse
encoder nyeer rurist Lnyer arrght
Lyyellarm
serederlape layer
dom
dorce
mecodee oyer oemetoon cnll quer waggtt
wessee lyeer whhetton on wesatt
masdde 10
evoider
weedee laper attertton valioe corign
ancoder
oaceder uypra attrnonn cot suery bay
evo les layer seffower
encadwr aper artertinn surzet desse westt
eacade attertion self
dende
mnooeer byer attent cutrut Laye souight
odle layar al an will be toas
encsited
ancoder theme enter deene sosgge
escodei layer atterbicn sol velle
sccude to
encodeting layer atturt on output dense bay
seenderlage put derse weight
eecoder layer corput lnyer leome weight
ancedle aapar allertan coti aud Layel Nur
encodiring
mncoder nowr wheethon selt query wesgtt
orcedarbyerr, Permecictt dence
anoodie byer currue dance
encocer oyer wehntton an wasset
encoeet attntton
wecdd attert
ennader myer suesul
wncaow laper cemertiin
encoded loyer's antertion sol cvery Daas ancaser ana alt
dence
encocer kyee allent on outpott Nomm wnight
ayer abbatior ss bay
ircedar ditetton sel
en enceder lavor momn
sncndel layer
layet
arddet
dorcc
encode kayct innttinnn cuistt Norm
ercoeeilyerra ougget und woth wagg
encodle altertion
weenter pherradaate
p0000 wog
seraderlyeer
andder ruttst denie wergh
tackmodak dieuuta wood desse
wendaring inyve dence
ernderlayeed autpul werght
tadiemder de hoad jut wegge
cutSLE LayenNerm
Jpdosobl: Po prrmmttrrs 256 Layor
Updatable Poromotors (40 Laye
Figure 13: Sparsity pattern of the layers in RoBERTa-base on rte. Layer-wise updatable parameters (%). (Left) The first half of the network and (Right) the second half. The default sparsity level 0.5%% shown as vertical line.
24

---

cb (Module-wise sparsity pattern)
owhonrlegs and bn embescmsaatin coiinn typs emboddmit insign
anender bigore aptortion query wiight
ortartion
lager attertion soll loy angg
atterdinn
unteadiing Landhaam bas
excoder liyor artendion soi salut wegg
onc secodde laye ai diwiton sel quers bat
encode bayor artemton ouzgut dorse wogrt
secaden byerd shorton cop bey har
oegode loyer' aetrrbn cumnit Leyeniom anget
secidel loyer attertion sat valte Sat
avander ayer intor
reeimeditt, dense
eecdde layee attert curipe danse Sas
musie layero ougue derse wegt
encoder byperd
encoder apero attrthnn ootase Eyeeloome 2a3
enoader nyer curnis unyeraom adgre
secoller
secedaa leyur rot be danee
wsight
encoee layer arornon soi suer woott
Layeriam
mnaii worast daywe weigh
crunde
donec
moddrr bygnt se mono to anage
anade loper
uyoL coE Nurn
Loyoil wagfe
meoder byer aetettio self
wegst
arceder layer attuncion seffoue ywogft
ercode layer atteniion oaruue desse
wecsow now athertan boy
oegode byyrr aeberbor cumitt Lnyenioo wngtt
arcedar ba attereinn nattrplu ooggte
cvooter oyer mom edete doece wng
arcoderlayee sttec mitout weght
dense
onmaee tyyer's oumpue
encoder byer astenoon ouccut Loyonk maggtt
wooser layer corput Layinti
encoderionel mortcdicts dendowwepp
werehblyyrr on wndt oyen oumpue don weihh
to
encoeer tiyerd ameetinn query
arterben
encoder byerd atteroon sot t0
evoue layer oud out Liyenii weggle
meseer byrra cenenin
encoder ayera aetrrenn
wecdder layst lnniion cule weogle
wordee buyer atteetion curpue
whe all lian will bay weight
excade loyers is aleertion carpat
accede
veceder layer disedion well weegle
evotter ayen memm eore dnece
creedetloy loyer attert output donen woiple
entarel byyea cutput doese
encoder byeer aetettonn cuppue myyen weighe
recoore typor rungut to
encodoring ee do denee wentt
ModiA to stamn
wocdde layer weeettin sell
monerr oyer oumu demon ung
encdde typers aeerton be
ancedar laver aotunion selfoue wwagie
croute
cacode layer atter bon sol duers bai
quou layen
wagle
recode byyer petrenoo curpue deace wnagt
Scende attintion
encoder afheebnn soe satu weott
encoder apecd attrdion
riseer
encderlnyer. wheer sutput derse
uncodar layer cntinin sall wele osigle
cacode byye attorher est wahe bas-
encodal lyye ac
buy
warade byara athethen curpat
winder attection
wesdde ayen ineereeiio
monter
entace tyyer curpue deese
weigh
incodl lention
ancedel seper aftertton
encocer nyeer rurist Lnver weoghs
evoider layer dom
peredeelager
layer dhemredeeee derse
dence
durce
mocoeee oyer oemetton cnll wiigh
wesser uyee whhetton
masdde 10
werde laper attertion
anceder
oaceder typer attrndnn cot euery bay
evo les layer out seffo
encadwr aper attertinn surzet weatt
eacade attertion self
dende
mncoder byer attent cutput Laye sougee
odle loyar al an will be pass
encsited
ancoder theme ente deene snagne
escodei layer atterbinn sol velee
sccude
encodeting layer atturt on output dense bay
seenderlage
eecoder loyer corput lnyer teom weighs
ancedle aapar allertan cot aut Layel tur thes
encodiring
we
mncoder nowr whenton selt quer westth
orcedbrbyerr peomecactt dence
anoorir byerr currue dunce Bar
encicer oper wherthon
eecoer attnttn
ennater myer suesul tyeetomm Say
wncaow laper
encoded loyer's antertion sol cvery ias
ancaser ana alt
encoclr kyee attent outpett
lyer abbatior bay
ircedar ditetion sel noigle
en enceder laver mome
sncndel layer
layet
andde
encode kayct ianttinn cuiset doruc
crcode dooeeryyeaa curget
encodle
weenter pherradaate dence
oeigte
00000
serndellyeer rumtst denie weigt
mndder
tssimotic dictes Nold dosse
wendaring inyve dence
eenderlayeer autpul Laverierm weight
taskrode dio bost sut wegge
outSLE LayenNerm
1. 3x
Lp Poromctors (961 Mocul
Updotzblc Porometers 1961 in Modu
Figure 14: Sparsity pattern of the modules in RoBERTa-base on cb. Module-wise updatable parameters (%). (Left) The first half of the network and (Right) the second half. The default sparsity level p" 005% is shown as vertical line.
25

---

cb (Layer-wise sparsity pattern)
owhonnlege and bnboooiigccsuiie weide
enender bygor aptortion query wnight en lager attertinn sell anigit
enbesoonaatinn toiinn typs emboddmic insignt
atterdinn siileey
unteadinng Landhamm bas
excoder liyer artendion soif la wegi
onc secodde loye ai dieitin sel query Dats
encode bayor artemoon ouzgut dorse woort
secaden byerd shorton cop bey hat
oegode loyers ahtrben cumist Leyenlomm wnggtt
if
secidel leyer attertion sal walle Sas
avander ayer mnto
woight
reeimeditt, denon
eecdde layee atter curipe danse Das
musie layero ougue derse weggtt
encoder bypedd
encoder apero attrthnn ootate syerloom 20y
enoader nyer cumnet unyenwom adggr
aecoller
secedal leyur rat be daenee
wsight
encoee layer artoron so8 suer wwdot
Layeniom
anaii oorgst darue weiggh
crundo
donec
moddrr byen se mono in arrgg
auade loyer
uyeL ooC Nurm
Loyo wagfe
meoder byer aetettin self
weggt
arcedar attuncinn seffoue ywogft
ercode layer atteniio oaruee desse welptt
arcedar ba attereinn nottrplun coughe
wecsow now attertan boyneggh
secoden athrhon
oegade lyyrr aeherbon cumstt Lnyenioo wngtt
cvooter oyer mome edete doece weigh
it Layellam
arcuderlayen attecton aitout weght
dense
onmaee tyyer's oumpue wright
encoder byer astenoon ouccut Loyonk mavggge
wooser layer corput Layinti weight
encodoriovel Rortedicts dendoweepp
werehrbyrrr on wndd Syen oumpue don weihh
to
encoder tyyer ameetinn query weighe
encoder byerd afteroon soit l0 kaage
evoue loyer out out Liyenti wengle
meseer byrra cemenin
encoder ayera attreenn
wrigt
wecddr layst lisiion cell yoogle
wordee buyer atteetion curpue wright
end
whe all ar tiiinn will bay weight
excade loyers is aleertion curpat Laye weight
vecedrr layer disetion vell weogle
ennsee
evotter ayen memm oore dnece soigrt
creodetloy loyer attert on output donce woiphe
entarel lyyea outtut doeze anget
dence
encoder byeer wetetttnn curpue myyen im weighe
recooer typor rungut tyyintio arighe
encodoring ov idorrctitto denee werpte
ModiA staroo
wocdde layer weeetton query weight
syentom
monerr Syer eveu demon
encdde typers aeertton be wright
seld
encderlnyer::
ancedar laver aotneion selloue ywoige
uncodar layer intin call wele osigle
cacode layer atter bon sel duers tai
cacode byyer attorher est bas
quou laye
lyye ac hoywesght
wheer surputt derse
Murm
bay
weiple
warade byara
minder attection
mecdde byyer aetrenon corpue deste wragte
Scende attintion
wesdde ayen ineermeiio dense
encoder afheebnn vat westtt
monter
encodel apecd attrrionn
entace byyer curpue deese weight
riseer
curpat tnyintiom weight
if
Geeae cvight
incodle lention
seper aftertton dhemreiet derse
ferm
encocer nyeer rurist Lnyerlo anggtt
Lyyellamm
serederlape layer
dom
darce
mocodee oyer ommetonn cnll waggta
wessee uyee whhetton on weeggt
masdde 10
nelt
werde laper attertion valuee cotggn
ancoder
oacedel uypra attrnonn cot suery bay
evo les layer seffower
encadw aper attertinn surzet desse westth
attertion self
dende
mnoder byer attent cutrut Laye souggh
odle leyar al wn will be tlss
encsited
ancoder theme enter deene wnaghe
escodei layer atterbicn sol velle
sccude to
encodeting layer atturt output dense bay
seenderlage derse weighs
eecoder loyer corput lnyer weight
ancedle sapar alleriani coti aut Layel Nur
encodiring
we
mncoder nowr wheeton selt query westth
orcedarbyerr, Peemecictt dence
anoorir byer currue dance
oper weertton an wesgtt
wecdd attert
ennater myer suesul
wncaow laper eerertion
encoded loyer's antertion sol cvery iass
encocer kyee allent on outpott dence
ancaser ana alt
ayer abuntior bay
ircedar ditetion sel
en enceder laver mome
sncndel layer
layet
oo doruc
croodc layer allirt
encode kayct innttinn cuistt
eccoedllyeraa ougpet uod wogg
encodle aitertion
Worm
weenter pherradaate dence
oeigte
p0001 150 wo0
serndelaaeer
andder rumtst denie wergt
tosimotics dictes wold desse
mendaring inyve dence
ernderlayeed autpul Layerierm weight cutSLE LayerNerm
taskrode dioch boat snn wegge
100
30
3.5
Jpdosoblo Po Porameters 2n Layor
Updotable Poromotors (40 Laye
Figure 15: Sparsity pattern of the layers in RoBERTa-baal on cb. Layer- updatable parameters (%). (Left) The first half of the network and (Right) the second half. The default sparsity level 005%% shown as vertical line.
26

---

copa (Module-wise sparsity pattern)
owhonrlegs and wnige embesonsattinn hossnn tops embeddmig insignt
anender bigor aptortion query
ortartion
lager atcerton soll angge
atterdin
unteadinng Landhaam bas
excoder amtendion soi salut wegi
risverr
onc secodde leye ai dilitinn sel baa
encode byore artemoon outgut dorse woote
secaden byerd shorhon cop bey has secidel leyer attertion sat walte
depodd loyers zetrben cumist Leyeniom
if
avaner ayer mnto
eecdde layee atrer
denee
music layera ougue
encoder bypedd
encoder apero attrthn ootsse Lyeeloom
encader nyer cumne unyenwom
secoller
seceda leyur
encoee layer arooron so8 suer woott byent se in
it Layeriam
mnsis Wowget donec
auade lopes coc swogft
crund
meoder byer aeordan
arceder attuniinn
sersdee wtterhon sak
ercode layer atteniio oaruee desse
secode aharben
now attertan
oegade ayar? aeberbon cumstt
wrght
arcoderlayer dyyer mitout
arcedar attereon ooggt
ovodter oyer mome edete doece pog
mnmeee tyyer' sumuue
encoder byer astenoon outnit Loyon magght
wooser layer corput
encodoriavel Rortedicss dendowelpl
werehrbyrrr cumpre on
encoee tyyer ameetinn query encoder byerd afteroon sot 80
evoue loyer oud
moseer byrra
ayera aetreenn
wecdder layst iietiin vellouse
wrdde byyer
curpue
whe all an aellin
eagdl layers a curpat
veceder layer disetion vellvele
mnster
evotter ayen memm eore deece woigre rocomedio
creedetloy loyer attert output donee
entarel lyyra output doese
encoder byeer wetetttnn eupuue syeem
recooee bypor cangut ty arigge
encodoring oe
mocode layer vetertion seld
query
monerr oyer eemu
encdee lyper aeerttnn be
encodel apecd attrdion
ancedar laver attneion selfoue ywagte
cacode layer atter sel
quou laye
mecdee byeed petrenon corpue deste wnagte
Sernde attintion curpat
encoder afhebonn soe tut weott
riseer
encderinyerr: byye attorher
urcodar layer citinon call
lyyr ac buy
minder attectinn
wxside ayen
byara
monter
weight
ovigh
entace lyyer curpue deese weight
incodl lettion
encocer nyer rumest Lover weogtt
serederlape layer
mocoeee oyer oemeton cnll waggtt
mardde 10
nelt
wesser ayee whhetton
werde attertion
anceder
caceder uypra amtrnonn cot
evo les laye seffour
encadw aper artertinn surzet weott
eacade attertion self
dende
mooder byer attent cutrut Loye souggt
odee layar
encsited
anceder thene
escodei leyee atterbirn sol
sccude
encodeting layer atturt outout dense
seenderlage
eecoder loyer corput lnyer
ancedle sppar altertan cot aud
encodtring
mocoder nowr wheeton selt query
orcedarbyerr, Peemecicto
anoofir byer curpue dunce
oper wherthon
ennater myer suesul
encaded loyer's antertion sol
wnoaow laper
ancaser ane alt
encocer kyel allent outpott
ayer abunttor bay
ircedar ditetion
en enceder laver mome
sncndel layer
enceda
layet
andde
dorcc
encode kayct innttinn cuiset
ercoedllyerra ourget
ercodle
weenter pherradaate
p0001 wog
serndelayeer rumtst denie weight
mndder
tackodek ditt copo hood dosse
mendaring inyve dence
ernderlayeed autpul weight cutSLE LayerNerm
tollmodeis dicioape bost snn weggle
L Lpsottable Paromctors (961 Mocul
Updotobll Porometers 1961 in Modu
Figure 16: Sparsity pattern of the modules in RoBERTa-base on copa. Module-wise updatable parameters (%). (Left) The first half of the network and (Right) the second half. The default sparsity level p" 0.55 is shown as vertical line.
27

---

copa (Layer-wise sparsity pattern)
owhonnlegs and onigo enbesonsaatinn coiinn typs emboddmtt insign
anender bigore aptortion queny wright ortartion
en laper attertinn self loy aniget
unteadinng Landhaam bao
excoder liyer amtendion soif lal wegit
onc secodde loye ai dieitonn sel quers bat
encode buyor artemoon ouztut dorse woott
secsden byerd shorton cop bry bar
degoide loyers aetrrbn cumust Leyeniom anget
secide leyer attertion sat walte
avanee ayer intor aditto waigh
denon
eecdde layee atter
danse Sas
music layero ougue derse weggt
encoder byperd
encoder apero attrthn ootate syerloom 203
enoader nyer curnit unyenwom adgre
aecoller
secedal leyur nat be danee
weight
encoee layer arornon soi quer woott
Layeniom
mnaii woaast danwe weight
crund
donec
moddrr byent se mono in arage
anade Joyer
uye ooc Nurm
Loyol wagle
modder byer aetettin self
weggt
areeder attuncinn seffoue ywogft
ercode layer atteniio oaruue desse wegtt
arcedar attereinn cottrplu ooggte
wecsow now attertan loyaeggh
atarbon
oxgode lyyrr aeherbor cumstt Lnyenioo wngtt
ovooter oyer mome edete doece wng
Layellem
arcoderlayer sttecto actout waght
danse
onmaee tyyer' oumpue
encoder byer astenoon outnit Loyonk mavght
wosser layer corput Lay
encodoriovel (Romiedicts dendoweepp
werehbbyrrr on wnded Syen oumpue don weihh
encoder tyyer ameetinn query weige
encoder byerd atteroon soi l0 kaas
enoue layer oud Loyent weggle
meseer byrra cemenin
encoder ayera aetreenn
wecddr layst lisdiion velloue yoogle
eordder buyer attemtinn curpue wrigh
ancodw whe all an aellinnee wil bay weight
excade loyers is aleertin carpat Laye weight
veceder layer diaetion well weggle
evodter ayen memm fore dnece snight
creedetloye leyer attert on output donce wogple
entarel lyyea cutput doeze angee
dence
encoder byeer weteeton exppue syee weighe
recooer typor rungut tyyintio arige
encodoring ove idorrcottt denee wernt
ModE to stamoo
wocdde layer aeeettin query
monerr oyer evmu demon wright
encdde lypers aeeriton be aright
sell
ancedar laver attneion selloue ywogie
urcodar layer citiiion sellvelne osigle
cacode layer atter set duers Dai
cacode byye attorher sotue bat-
quou laye Luyettanm megle
encadal lyyr ac
tin hoyweggh
Nur
buy
warade byara athethen curpat tnyiniiom weighs
winder attectinn
recode byyer petrenoo curpue deste wnart
Scende attintion
wesdde ayen
encoder afheeonn vt weott
monter ineereeii ovigh
encoder apecd attrrionn
riseer
incderrnyeer: wheer surput derse
entace byyer curpue deese weight
incodl lention
anceder syper aftertton
evoider layer domos
peredeelager
laye dhemeeiet derse
ferm
dence
darce
mocoee oyer oemetton cnll waggta
encocer nyeer rurist Lnverlo anggt
wescer uyee whentton ln wesgtt
masdde 10
Layellam
ealt
werdee laper attertion vaice coiggn
anceder
oaceder uypra amtrnonn cot suery bay
evo les laye out seffower
encadwr aper attertinn surzet weott
eacade attertion self
dente
inzoder byee attert cutrut Loye sought
odle leyar al wrlinn be pass
encsiter
anceder thene enter deene snagee
excodellng leyer atterbern sol velee
sccude
encodeting layer atturt on output dense bay
seenderlage derse weight
eecoder loyer corput lnyer weighs
ancedee sppar altertani cot aut Layel Nurm
encodirigg
mncoder nowr whenton selt query westth
orcedbrbyerr, Peemecitto dence
encicer oper whenthon an wasgtt
oncads dh byer curpue dance Bar
eecoeet attntton
wecdd attert
ennner myer suesul syeettmm
wntertion
wncaow laper eerertion
eecuded loyer's antertion sol cvery Das
dence
ancaser ana alt
ayer abuatior le higoocigt
encodlrr koye attent on outpott Nomm wdgic
ircedar ditetion sel
en enceder lavor momn
sncndel layer
enceda
layet
ou doroc
anoded lye wo
croodc layer allnt
encode kayct ianttinn ouiset
eccoedllyeraa ourget uoy wog
encodee aitertion
weenter pherradaate
p0001 woo
serndelayeer numnst denie wergt
mndder
tackodek ditt copo hood dosse
mendering inyve dence
eenderlayer: autpul
weight
tollmodeis dicioape bost out snn wegge
cutSLE LayenNerm
0.8
0.
0/
Jpdosoblc Po pormmtters.. 2961 Layor
Updatable Poromotors (960 Laye
Figure 17: Sparsity pattern of the layers in RoBERTa-base on copa. Layer-wise updatable parameters (%). (Left) The first half of the network and (Right) the second half. The default sparsity level 0.5% shown as vertical line.
28

---

Wsc (Module-wise sparsity pattern)
owhonnlegss and bnb embesonsaatinn coiinn tops emboddmit insign
anender tigore aptortion query wnight en laper atterton soll loy angget
atterdin
unteadinng Landhaam bas
excoder liyer amtendinn soi salut wegit
onc secodde laye ai dieiton sel quers bat
encode byore artemton oucgut dorse woott
secsden byerd shorton cop bey has secide leyer attertion sat walte
oegode lyyers aetrren cumist Leyeniom wnget
avader ayer mnto
reeimedits waggh
eecdde layee atrer
danse
music layer' ougue
encoder bypedd
encoder apero attrthn ootsue Lyeeloom
encader nyer cumne unyeraom adgre
secoller
secedal leyur
encoee layer arornon so8 qur woott
it Layeniam
anade lopes
mnsii Wouget donec
crundo
mooder byer aeordan
moderr byent se
uyel ooc swogft
arceder attuniinn
seradee wtterhon sak
ercode layer atteniio oaruue desse
wecsow now attertan boyg
secode abarbon
oxgode ayar? aeterbor cumstt
arcedar attereinn noth weggt
ovodter oyer mom edete doece wng
arcuderlayen sttecto mitout
mnmaee tyyer's oumpue
encoder byer astenoon oucnut Loyon magght
wooser layer corput
encodoriovel Rortedicts dendowelpp
werehrbyrrr on wnde Syen oumpre
encoee tiyerd ameetinn query
encoder byerd afteroon sot t0 kaage ayera attreen meseen byrra cemenon
enoue layer oud
uecddr layst alntiion vellouse yoogle
wodde byyer atteetin curpue
whe all aeriinn
eggde layers a curpat
vecedr layer diaetion vel
evotee ayen memm eore deece gnight
creedetloy loyer attert output donee
entarel byyea output doeze angee
denco
encoder byeer wetetttnn exppue syee
rocooee typor cangut toyintinn wrige
encodoring oe
denes
mocode layer weeettnn query
monerr oyer oemu
encdde lypers aeerttnn
encodel apecd attrdion
ancedar laver attneion selloue ywagte
cacode layer atter set
quou laye
recdde byeer petrenoo curpue deste wnggt
sernde attintion curpat
encoder afheeonn soe va weott
riseer
incderinyerr: wheer
urcodar cntin callvelne
byyer attorher
lyyr ac buy
minder attettonn
axside ayee
byaar
monter
entace tyyer curpue deese
weighs
crigh
incodl lettion
anceder sypec aftertton
encocer nyer rumest Lover weoght
seredeelager
layer dhemeeiet derse
moooee oyer oemetton cnll waggtt
10
wecserr uye whhetton
werde laper attertoon
anceder
oaceder uypra amtrnonn cot
evo les laye seffour
encadwr aper 10 artertinn surzet weott
eacade attertion self
dende
incoder byer attert cutput Laye sougge
odee leyar
encsiter
ancoder thene ente deene wnggee
escodellng leyer atterbern sol wele
sccude
encodeting layer atturt output dense
secnderlage
eecode loyer corput lnyer teom
ancedee sppar altertan cot aud Layel Nur
encodirigg
mocoder nowr whenthn selt query
orcedarbyerr, teemecicte dence
anoofir byer curpue dunce
encocer oper wherthon
ennnter myer suesul
wncdow laper
encaded loyer's antertion sol cvery
ancaser ane alt
encoder kyel attent outpett
ayer abbatior bay
ircedar ditetion
en enceder laver mome
sncndel layer
croodcile allir oo dorcc
layet
andde lye
encode kayct innttinn cuiset
ecodedlyerra ourget uoy wagg
encodle
weenter pherradaate
p0001 wog
serndelayeer rumnst denee weigt andder
tadimoeess dictwss hard 2tbenton taskrooees dii wod wegget
mendering 2noder dence
eenderlayerd autpul Laverierm weight outsLE Layerlerm
hend dlssiieer
0.50 325 1.00
Upsotzabl Paromctors (561 Mocul
Updotoblic Porometers 1901 in Modu
Figure 18: Sparsity pattern of the modules in RoBERTa-bass on wsc Module-wise updatable parameters (%). (Left) The first half of the network and (Right) the second half. The default sparsity level p" 0.55 is shown as vertical line.
29

---

wsc (Layer-wise sparsity pattern)
owhonnlegss and bnboo enbedoonaatin coisnn typs emboddints insinn
anender bigore aphortion query wright en laper attertion self loy aniget
unteadinng Landhamm bao
excoder layer artendion soif salut wegit
secodee leye ai diwition sel quers baa
encode byyor artertio ouztut dorse woott
secsden byerd shorton cop bry har
decoide loyers aetrrbn cumist Leyeniom adget
secide leyer attertion sat walte Sas
avadee ayer into adicto waggh
denoe
eecdde layee atter
danse Sas
muse layero ougue derse weggt
encoder byperd
encoder apero attrthnn ootase syerloom 203
enoader nyer curnit unyenaom adgre
aecoller
secedal leysr nat be danee
wsight
encoee layer arornon soi quer woopt
anade Joyes
anaii wouast danue weight
crund
uyeL ooc Nurm
Loyol wagle
darec
mooder byer attettin self
moderr byen se mono to arage
mende
weggt
arceder attuncinn seffoue ywogft
ercode layer atteniion oaruue desse wegtt
arcedar attertion cottrplu ooggte
wecsow now attertan boyanggh
atarbon
oxgode: ayyrr aeherbo cumst Lnyenioo angtt
ovodter oyer mome ediee doece sdgee
Layellem
arcuderlaye sttecto aitout wagte
dense
onmaee tyyer' oumpue
encoder byer astenoon outnit Loyont mavght
wooser layer corput Layinti weight
Incoddrioner. (Romiedicts dendoweeple
encoeer tyyer ameedon query weige
on wnde Syen oumpue don weigh
encoder byerd atteroon sott lo kaar
mnoue layer oud Loyent weggee
meseer byrra cemenon
encoder ayers attrrenn
wecddr layst listiion vellicse yoogle
eorode byyer attemtion curpue wrigh
ancodw whe all an aeliineed wel bay weigt
escacl loyers is aleetton curpat Laye fome weight
wecedrr layer diaetion well weogle
evoiter ayen memredoo eore dnece snight
creedetloy leyer attert on output donce wogple
entarel byyea cuttut doeze angee
dence
encoder byeer wetertinn expper syyem weighe
recoore typor rungut toyintiom arige
encodorinn ove idorrcettt denes wernt
Moddulo starro
mocdde layer weeettnn query weight
monerr oyer evmu demon wnight
encdee lypers aeeritnn be wright
sell
incderrnyeer: wheer sutput derse
ancedar laver attnnion selloue ywogge
urcddar citiiin well osigle
cacode layer atter bon sel duers
cacode byyer attorher sotuel bat-
quou laye
encadal lyer ac ceriin buy
Luyettamm me
hoyweggh
Nur
warade byarr
winder attrcttnn
recode byeer petrenoo curpue deste wnatt
Scende attintion
wxsid ayen ieemmeeii dense
encoder afheeonn soe vt weott
monter
encoder apecd aeterionn
entace byyer curpue deese weeght
riseer
curpet tnyiniiom weighs
Geeae ovigi
incodl lettion
anceder syper aftertton
evoidee layer domos
peredeelaeer
laye dhemeeiete derse
dence
darce
mocodie oyer oemetonn cnll waggt
wesser uyee whentton bn wesgtt
encocer nyer rumist Lnverlo anggt
mazsde 10
Layellamm
eelt
weedee laper attertoon vaice coigrn
ancoder
daceder uypra amornonn cot suery bay
mvo les laye out seffower Nur
encadwr aper 10 attertion surzet weoth
eacade
attertion selt
dence
incoder byee attert cutrut Loye sought
odle loyar al wrlen will be pass
encsiter
ancedar thene enter deene snggee
excodellng leyer atterbern sol velee
sccude
encodeting layer atturt on outout dense bay
secnderlage derse weight
eeroder loyer corput lnyer teom weigh
ancedee appar altertan cot atd Layel Nurm
encodirigg
mncoder nowr whenton selt query westth
orcedbrbyerr, Peemecicto dence
encocer oyer wherthon an wangte
oncads dh byer currue dance Bar
encoeer
attntton
wecdd laper athert
ennner myer suesul syeetomm Say
wndd laper eerertiio
wncaow
wntertion
eecuded loyer's antertion sol cvery Das
dence
ancaser ana alt
encocler koye attent on outpot Nom. wdgic
ayer abuatior
ircedar ditetion sel noogle
en enceder laver mome
sncndel loyer
layet
arode lye wo
croodc layer allnt
doruc
encodo kayct ianttin ouiset
crcoeellyerra ourget uoy wag
encodle aitertion
weenter pherradaate
oeigte
p0001 wool
serndelayeer
andder hqee numtst denie weigt
tadimoteis dictwss haad Ponnttonn woget
mendering
dence
fa tackmoodes dii wod heod casso wugget
eenderlayer: autpul Laverierm weight
outsLE LayenNerm
hend dlssiifer
07 Updotable Poromocors (40) Layo
Jpdosoalc Po pormmtters. 2961 Layos
Figure 19: Sparsity pattern of the layers in RoBERTa-base on wsc. Layer-wise updatable parameters (%). (Left) The first half of the network and (Right) the second half. The default sparsity level 0.5% shown as vertical line.
30

---

cifar100
visul promust lokan pos amoud
Visual promet laken pos amouz
booke e cauut primet ioten
bookk y sual primrer tobes
mlecke
wiihh
laka weight
weight
bldis odapter weight
Morkc n petephe in hos
Moric n saapphe in foo
Uooka culllatto lnd weigl
looko tuollatii and wedgl
books adepter jies
Mocic adepler sie-
blods
blndas attn souggt
boeck art LaBA weight
brocta aithn Lolk to trwwgtt
breees aile Lolk weggt
Medes attephee waight
booker actipternt wiight
Modes edapter Liko
edapter Lio
bode, jadapter
Mockks 1 adapter lna time
Modis a adapter lnz tis
bocks visua promett Jaken
Bodks a susl prongr_toke Jokes
books
awaiggt
weigrt
Medoa 2 adapter weight
boda 2 audapterr weight
Morks adaptee in pass
Morie attepte in pans
Hindes adeptee weight
Hadls adaplee weight
bocks adapherass
sdapter
books sual, brack attn LoRA, a welggt
bocks, battn LoRk, a swaggtt
Blocka with Loik, booeegl
Blicka with LofA, weegl L
Mooko ataatto mayll
Mooko t.ttkaiii medy
bedes adupter waight
blade adupter werght-
Mocks 3 adapher in zin
Mocis 5 adapher ind zwa
bocks vesul prompt toben-
bocks i sual prompt taben
blocks
bracks
weiht
baodss e, adaptier weggrt
bocks 4, a.ddagtir weight
Mookks adapter In). tas bleda edapter weight
Mocks a adapter in). tas
blecks duddapter weight
bools couse
loben
blokss i sund
boben
blecke a aith tolk, a weight alto
trecki i ait Lokk, a weight
weegl gll
with io weight
Nobule Narra
bloks odapter woight Lodls
Nobule Name
block Soctpterr weight
Booko
books odapter lnd weight
books addgtte welght
Luddo h edep to hit bools, prompl
Blucs a edeppia hui bis
boocs sond prompp
blocke aatn bowptt
brocks attt LoRk weight
bldda adaatee and waight
blocks, achepter ind weigtt
Moo adapter lo llss
Maccks adopter lni lio-
Morks i adapter Flcc
Norite is pdaptor flac
books tubuull tolen
bodka to soul is elicks atti LoKA a woart
talen
blecss aatt LoRk awaatt
bowwght
brecks
weight
Mocis adtptee tue
Mocio it attepher ant tue-
Mode: T nelepher and weight
hoder nebetter and unigi
Mocks adapher tus-
Mociss adapter zoe-
niuus prompt sokenn
suol prompt
blokks
blocks
werptt
blocks alt Lonk, io boiigght
bricks in with Lank, weigtt
baadss i atlapto and wegght
bocas. it adlapte and wegrt
llocks
Bloks. adapterrit
bookks & edapter. and
blocks 3. addetterii bas
broks i uaii zmmmet toben
teeks u moal prrmrer tobes
blocke batti take, wiight
aime taka, a wright
weight
bloks.
weight
bocks s.odapter
Mocha adapter Ind time
Mooku adapter ind tim
(ookl 5 calasto liz woigl
looio huddacioo to ind woid
Moccss adepter lie
Mesic adepter biw-
Mocke with promptt
Mochs
weight
blocke wit Lakk,
blokk 10 atte LoRk a nngght
books 10 otne LofA neigtt
Nooke t0 adapper weght-
books 10 odapter seight
edoyter lie
blodls adapter inz tie
bookks adapter inz sie-
Mocck voull zromet saben
Mocka 12 vauval pronpe bobes
weigrt
LonA welgrt
bloca 21 udaptir in2 weight
blocaa udupter weight
bledis adaetee in zims
broets adaptee in zin-
blcts 11 shtpttrrhwwwepp adtpphr
bodks 11, odupter h2.welght
25
Uppeetable Parameters Misttile
Updatable Phrumettes (90)
Figure 20: Sparsity pattern of attached modules to ViT-B/16 on cifar100. (Left) Module-wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p* = 10% is shown as vertical line.
31

---

caltech101
Visil promust lokan pos amoud
Visual promet laken pos amouz
booke e cauut primet soben
blokks y sual primrer tobes
mlecke take wiight
laka weight
weight
bldis odapter weight
bldks Codict.ad
Morkc n pehepher in hos
Moric n saapphe in hoo
Uooka culllatto lnd weigl
looka tuollapti and wedgl
books adepter jie
Mocics adepler sie-
blods
blndas attn soeggt
bleck art LaBA weeght
brocta alte Lotk to trwwegt
breees ail alln Lolk weggt
Medes aetephee waggt-
booker actiptern wiight
Modes edapter Lim
edapter lio
bode jadapter
Mockks 1 adapter lnz time
Modis a adapter lnz tis
bocks visua promett Jaken
Bodks suol jrimer Jokes
books alt alwiggt
weigrt
Medoa 2 adapter waight
boda 2 audapterr weight
Morks adaptee in paes
Morie attepte in puns
Hindes adeptee weight
Harls adaplee weight
bocks adapherass
sdaptor
books sual, bonck 3 attn LoRA, a welggt
bocks. battn LoRk, a snaggtt
Blocka with Loik, booeegl
Blicka with LofA, weegl L
staatto mayil
Mooks Latllaio medy
bedes adupter warght
blnde adupter werght
Mocks 3 adapher in zis
Mocis 5 adapher ind zwa
bocks cesul prompt Joben-
bocks a sual prompt taben
blocks
brocks
weiht
baodss e, adaptier weggrt
books 4, a.adggtir weight
Mookks adapter In). tas
Mocks a adapter in). bas
bleda edapter weight
blecks duddapter weight
bools couse
loben
books i sund
boben
blecki a aith tolk, a weight alto woeght gll
Brecki i ait Lokk, a weight
with
weight
Notule Narr
odopter woight
Nodull Name
blokk Soctpterr weight
Lodls
Booko
books odapter lnd weight
books addgtte welght
Luddo h edepias to his
Blucs a edeppia bui lis
bools, prompl Johen
books sood prompp
blocke attn bowppt
blocks attt LoRk weight
bloda adaateec and waight
blocks, achepter ind weigtt
Moo adapter los llss
Maccks adopter lni lio-
Morks i adapter Flcc
Norite is adaptor flac
books tubuull token
booka to soul is eicccs atti LoKA a woart
talen
blecs. aatt LoRk awaatt
boowggt
brecks 7.ekkatn weight
ooclapte
Mocis adteteer til tue
Mocio it attepter ant tue-
Mode: T nelepher and weight
hoder nebetter an unigh
Mocks adapher rus-
Mocis adapter zos-
eisus prompt soken
suol prompt
blokks
blocks
wergtt
blocks alt Lonk, beiigght
bricks is with Lank, weigtt
baadss i atlapton and werght
bocas. it adlapte and wegrt
loock.
Books. adapterilt
bookk, & edapter. and
blocko 3. addetterii bas
broks i uaai zrmmet toben
toeks u moal prrmre tobes
blocke batti lake wiight
taka, a wright
weight
bloks Sdaatee weight
bocks Sodas...d
Mocha adapter Ind tiee
Mooku adapter ind tim
Uookl 5 calatto liz woigl
loois hudda to inz woid
Moccss adepter lie
Mecic adepler biw-
Mocke with promptt
Mocks
weight
bloie wit Lakk
blokk 10 atte LoRk a nngght
books 10 otn LofA neigtt
Nooke t0 adapper weigt-
books 10 odapter weigt
edatter lae
blodls adapter luz the
books adapter in sie-
Mocck voul zromet saben-
Mocka 12 vauell pronpe tobes
weigrt
LonA weirr
bloca 21 udapter in2 weigt
beoka udupter weight
bledis adaetee in zims
boots adaptee in zun-
blcts 11 odaptor htweegt adopphr
bodks 11 odupter h2.weight
30
15
25
5.8
1.8
300
12
150
17.5
30.0
Uppdetable Perametirns Mostulee
Updatable Parmmetees (80
Figure 21: Sparsity pattern of attached modules to ViT-B/16 on caltech101. (Left) Module-wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p² = 20% is shown as vertical line.
32

---

dtd
visul promutt lokan pos anoud booke e couut primet soten
Visual prompt laken pos amouz
bookk y sual primrer tobes
mlecke aatt take a wiigh
am laka a weight
bodks dodapterii weight
bldis odapter weight
Monkc n setiphe in hos
Moriic n sdappee in foo.
Uooka cuallatto ind weigl
looko.t tuoolatti ind wedgl
books adepter jue
Mocics adepter it? sie-
blods
bindas attn Labs, soeggtt
bieck artn LaBA neight
brocts alle Lotk to trweett
breees wiln Lolk a weegt
Meder atteptee waight
booker acteptecat wiight
(ouck edayter Liko
edapter lio
Modes
bode, jadapter
Mockks 1 adapter lna time
Modis a adapter lnz tie
bocks visua promet, Jaken
Books a susl pronerr Joles
books att awaight
sos aiti Lana weigrt
Medos 2 adapter waight
boda 2 audapttrr weight
Morie adaptee in ples
Maciee attepte in puns
Hinde adaptee weight
Hadls adaplee weight
adxpheriis
sdaptor
bocks.
books suat,
bocks battn LoRd, a swaggtt blocka with Loik, bowwegl K
brock 3 attn LoRA, a siiggtt
Biicka with LofA, i weell L
Mooks ataptt mayll
Mooks Laallaga mecyl
bedes adupter waight
blndee adapter werght
Mocks 3 adaphertnn in zis
Mocie 5 adapher in zwa
bocks vesull prompt toben-
bocks i sual prompr, baben
blocks
weight
bracks
weiht
baodss e, adaptier weggrt
bocks 4, a.adagtte weugtt
lookks adapter in). tas bleda edapter weight
Mocks a ada in). tas
blecks duddapter weight
adtpper
actephee
bodlo couse
loben
blokl i sand
boben
blecki a aith tolk, a weight altn
trecki i arte Lokk, a weight
woeght gll
with in weight
Notule Narra
blocs odopter woight Lodls
Nodule Name
blokk S.otopter weight
Bools
books odapter lnz weight
books addgter inz welght
Luddo h edep to hit Ere boods, cisuel prompl
Bluck a edhepios hui bis
blocss sood prompp
blocke aatt wowptt
brocks LoRk weight
bloces adaateecin inz waight
blocks, achppter and weigtt
Mos adapter lo llss
Macces adopter ln) lios
Morks i adapttr rlac
Norite is adapter flac
books tubuull tokan
booka to soul is eiccks attn LoKA a woart
talen
blocks aattn LoKA awarrt
trowght
bricks 7 autte Teckaatn Mocio attepter ant tun-
weight
Mocis adtepter iil tuee
Mode: T netephe an weight
hoder nebetter an wnigh
Mocks adspter tine
Mocis adapter zon-
eisus prompt
pwompt
blocks
blocks attn
werptt
blocks alte Lork, beiggght
fricks is with Lank, is wegglt
beadss i ataapter and werght
bocas. it adlapte and wegrt
Loocks
Bloks. adapterintt Ind
waight
boollo & edapter. and
blocks 3. addetterin and bus
beoks i unaie prmmet toben
teekss u scal prrmrer tobes
blocke batti take a weight
Is aite taka, a wright
S.da....d weight
bloks odaster woight
Mocha adapter Ind tine Uookl 5 calasto lid woigl L
Mooku adepter ind Sim
locko hudaapiio to ini wokk
Moo adepter tie
Mosic adepter biw
Mlocs
Mocke with prompit
Mocks
weight
borie with Lakk, weight
blokks 10 att LoRk a anight
books 10 otne LofA is neigtt
Nooke t0 adapper weight
books 10 odapter seight
tor
edayter lie-
blodss adapter luz tie
bookk 10 adapter inz siee
Mocks voul zromet saben
Mocka 12 vauval pronpe bobes
Docks Lonk weggtt
booka LonA welgrt
bloca 21 udapter weight bledss adaetee in zims
blecaa 21 uduptir weight
Broets adegtee in zin-
blctes 11 hiwwegtt adkphhr
bookss 11, odapter h2.weight
2.5
17.5 Updatatle Parameters Montle
5.0
7.5
16.0
12.5
15.0
20.0
Updatable Parmmettes (50)
Figure 22: Sparsity pattern of attached modules to ViT-B/16 on dtd. (Left) Module-wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p' 5% is shown as vertical line.
33

---

flower102
Visul promett lakan pos umoud
Visual promett laken pos amooz
booke e couut primet soten
blokks y sual prrmrer tobes
mlecke aatt take wiigh
aame laka, weight
Cadaptenn weigt
bldis odapter weight
Morkc n setiphor in hos
Moricc n sdapper in foo
(ooka cuallatto ind weigl
looks.t tuodlatto ind wedgl
books adepter jie
Mocic adepler sie-
books
blndas attn Labs, souggt
boeck art LaBA neight
boocta alln Lolk to trwwett
breees wiln Lolk weegt
Meder attephee waiht-
booker actepteeat wiight
(occk edayter Lim
edapter lio
Modes
bode, jadapter
Mockks 1 adapter lna time
Modis a adapter lnz tie
bocks visua promet, Jaken
Bodks susl pronerr Jokes
books alt awaigtt
sos aiti weigrt
Medos 2 adapter waight
boda 2 audapterr weight
Morks adaptee in ples
Maciee attepee in puns
Hidde adaptee weight
Hadls adaplee weight
bocks adapherans
sdaptor
books suat, brock 3 attn LoRA, a weiggtt
bocks battn LoRd, snnggtt
blocka with Loik, toweegl L
Biicka with LofA, i weell L
Mooko ataatt mayll
Mooks t.etkaiii mecyl
Medes adupter waight
blnde adapter werght
Mocks 3 adapherins in zwa
Mocie 5 adapher in zwa
bocks cesul prompt toben-
bocks i sual prompt taben
blocks
weight
brocks
weitt
baodss e, adaatier weggrt
bocks 4, a.adagtir weugtt
(oocks adapter In). tas
Mocks a ada in). tas
bleda edapter weight
blecks duddapter weight
adippee
actephee
bools couse
loben
bloks i sand
boben
blecki a aith tokk, a weigt- altn
trecki i aite Lokk, a weight
weeght gll
trinnss with in weight
Notule Narr
bldks odopter woigtt
Nodule Name
blokk Soctppter weight
Bools
books odapter lnz weight
books addgter inz welght
Luddo h edep to his
Bluck a edhepios hui lis
bools, prompl
blocss sood prompp
blocks aatt bowptt
brocks &f auttt LoRk weight
bloces adaapeecin and waight-
blocks achapter ind weagtt
Mockk adapter lo llss
Macces adopter ln) lios
Morks adapter Flcc
Norite is pdapter flac
bocks tubuull tolen
Bodka to soul is elicks attn LoKA a woart
talen
blocks atttn LoRk awaatt-
boowght
bricks autte
weight
Mocis adtpter iil tue
Mocio it attepter ant sue-
Mode: T nelephe and weight
hodee 7 nebetter and unigh
Mocks adspterns rus-
Mocis adapter zoe-
visus prompt soken-
pwwmpt
blokks
blocks attn
wergtt
blocks alt Lork, beigght
bricks is with Lank, is begglt
beadss i ataattor and weigtt
bocas. it adlapte and wegrt
Loock.
Bloks. adaptiriitt Ind
bookk, & edapter. and
blocks 3. addetterin and bus
beoks i unaie zrmmet toben
tooks u soal pirmrer tobes
blocke batti take a wiight
aite taka, a wright
weight
bloks. odaater waggt
bocks Sodac...d ind
Mocha adapter Ind tiee
Mooku adapter ind Sim
Uookt 5 calasto liz woigl
looio tuddapiis to inz wokk
Moo adepter tie
Mosic adepter biw-
Moccs
Mocke with prompii
Mocks
weight
bodie wite Lakk,
blokk 10 atte LoRk a nngght
books 10 otne LofA neigtt
Nooke t0 adapper weigt-
books 10 odapter seight
edopter lie-
blodss adapter luz the
books adapter inz siee
Mocks voul zromet saben-
Mocka 12 vauval pronpe tobes
Docks
weight
booka LonA welgrt
bloca 21 udapter we
blodaa 22 uduptir weight
bledis adaetee in zim-
broets adagtee in zin-
blcts 11 shtpterhh2wwgght adtpper
books 11 odapter h2.welght
10
20
Upddetable Parameters ES3)
Mostule
Updatable Parumettess (10) Layer
Figure 23: Sparsity pattern of attached modules to VIT-B/16 on flower102, (Left) Module-wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p* = 50% is shown as vertical line.
34

---

pets
siuuel vicusl poorup joknn pan bocks dliisa
viouel pronpe jokno prompt
blokl Sual, bincka with LanA swight
block o eite Lank, awsigt Lioko duttin LinA lonsgg
liock duitn Line lonagl
Mos atlepte weight
laell adlaple weight
Medde adletter weight
bldda it adaetee weigh
Mooks o adapter 1n tos
bocis D adapter in. ras-
hodes pronpt
bodke eial prampe
Line, aowilt
brocks alth tunk, weljtt
bocks arm Lokd
ladee y atlagt lis wiihht
lockl 3 atlagtoo litt weight
Mocka adapter ln bua boois adapter woight
Mocia edapter ind awo
pocis Luddaptar woight
Morke
bode. Jueuus
taken
block pual
taken
miekss ain toick a wright
traeks wine inka, a neight
bookss zattn LoRA bowggt
brocks atth Loka wogght
stspher
sttppho
weight
Mocks
das-
Moces
bodes 2 adapter 1n2 weight
blokd 2 2odaptering in weight-
Mocck 2 adapter lid lhe
Locol a edepter lit haw
bodo coust prompt ioten
blocs prompt laten
bonkee atto Laks boogght
blacke atto Laka a weigh-
Medea 3 adaater ind weight
bodka 3 aclapter and wegght
Mocks adapter in] tas-
Noces adspter in tos-
Mocks odagtio weight
Mocks adagttor weight
bocks cisuall promet takan
bocks i sual prompt tobon-
blocks aattn Lokd, wight
Blecks atti LokA, a wight
biocko.4.att 4.ttt bowight
bricks atte Latt bweight
actepier
Mooks adapter int Fiee
Moolo adapte in! sie-
Modes adspter an weight-
Nodds adheter to weight
bocks Mocks a adapter ni, tap
bocis adapter n2, tac-
promot
suol prompt
brocks a at LiNe, bowggtt
blocks arn LonA, i wegggt
Modile rane
laades 5 ad epee lns weigl
Modile stav
blods 5 edlaptee lis weigtt
eodee degpter and
Bocks adapterint bas
waggt
wwggt
bodke 5 aduster ind
booke 5. aduater ind
bodks Commul promet soban
bocks d eual prompt soben
blocks eutth Lokd, weight
bicks durt toka, a weight
satth
att
Modee atayter wiiht
Modil artagtor weigtt
Mocla adapter lvd Lup
Mooia edaptor ind lup
blodss adaatte in2 woigtt
bocks Gadaglir in2 weight
Mocks adepter tie
Mocicu adepter hin-
biects y ailn Lates soeptt
toents witn loti, twegt
hoekl aath loka i nngght
trneke aime ioka a weigh
bledes aduptee weight
blocle adaeptentn weight
Mocks adopterin tas-
bloces adspter ras
Mades
Modes
Mockk it adapter Iiz Lio
Macko in adapter liz lio
bocks & couaal promet loban
pooks a sual promes laban
blocks
alti LotA bwight
birdle adupter weigh
bicdlo adspter weigt
tirdss a adocter in has-
Hronks a adoaterrns int his-
Noddes odagtee wnight
Nodle adaattor weight
bocks
books sual,
blocks du alte Lark, aweiigt
tiicka with Latk, seight
booko bulti LiM, lonsial
liock uith Lukk, lonsig
tirdec adaepte weight
tndds adaplee weight
adapperinn
adspher
Medes adleteer weight
bedds adaapted weight
Wooks 9 adapter ii tiss
Mocks 3 ada in. ras-
Morke
prompt
Morke weill smarpe
alto Link, weeglt
alte Lona, weight
bookk
bockk
blocs 10 adlaterr 1n neight
books 30 atlapter neight
Blecka edapter ind dup
brecks 20 edaster ind cwe
bocks t0 adaptar weigt
books odaptar weight
Morke 13 sinuo
saben
Morks 13 sneuod Miniic 11 win ioka, A teight
tobes
Morie 13 aite Tokk A Weight
blocks i1 otte Lokk pongght
books 11, otn Lokk woiggt
odtpper weght
weiht
books,
bookk
Mlccs 11 uelapte oeggle
backs 21 aelepte weight
shodo edepter bnd Shee
Books 22 edeptor hnd bote
7/
1o's
Updatable Paromotoors
Module
Updatablc Parameters (50 Layor
Figure 24: Sparsity pattern of attached modules to ViT-B/16 on pets. (Left) Module-wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune, optimal sparsity level p- = -00 is shown as vertical line.
35

---

svhn
visul promust lakan amoud
Visual promet laken pos amouz
booke e cauu prrmet soben
bookks y sual prrmre tobes
mlecke
wiigh
taka weight
weiht
bldis odapter weight
bldks Cadapttrn
Monkc petipher inn hos
Moric n salpphe in for
Uooka cullaatto lnd wesgl
Wodls tuellati and wedgl
books atepter jie
Mocis adepler sue-
blods
blnda attn nought
bieck art Labk neight
brocts aiti Lotk browegt
breees all Lold weggt
Modes aetephee waggt
booker actipter wiigh
Modes edapter
edapter
bode aadaperr
Mockss adapter lnz Lime
Modis a adaptor lnz tis
bocks visua promett Jaken
Bloks suol promerr Jokes
books
alaiggt
Weight
Medos 2 adapter weight
boda 2 addapter weight
Moris adaptee in pae
Morie attepee in puns
adaptee weight
Hards adaplee weight
bocks
books sual, brack attn LoRA, welggt-
bocks. battn LoRk, snndgtt
Blocka with Lotk, boweegl
Blacka with LofA, weegl
staato mayil
Mooks Letllaii medy
bedes adupter waight
blade adapter wergrt
Mocks 3 adapher in zia
Mocis 5 adapter ind zws
bocks cesul prompt Joben
bocks a sual prompt taben
blocks
bracks
wiitt
baodss 4, adaptier weagrt
bocks 4, a.ddggtir weight
Mookks adapter In). tas
Mocks a adapter in). bas
bleda edapter woight
blecks duddapte weight
bools couse
loben
blokss i sund
boben
tlecki a aith tolk, a weight alto
trecki i ait Lokk, a weight
gll
weight
Nobule Narr
odopter woight
Nodill Name
Soctpteer weight
Booko
books odapter lnz weight
bloks addgtte welght
Luddo h edep to his
Blucs a edeppie bui bis
bools, prompl
books sond prompp
blocke attn bowptt
brocks attt LoRk weigtt
bldds adaptee and waight
blckks achepter and weigtt
adapter los Llss
adopter lni lla-
Morks i
flcc
Norite is
flac
books tubuull tolen
Booka to soul is clicks atti LoKA a woart
talen
blods. aatt LoRk awadtt
bowweht
weight
Mocis adaeteer tue
Mocio it attepter ant sun-
Mode: 7 nelepher and weight
heder nebepter and unigi
Mocks adapher ius-
adapter zoe-
nisus prompt sokenn
blocks prompt
blokks
werptt
blocks alt Lonk, beiiight
brickls in with Lank, biiglt
baadss atlatton and wesght
bocas. it adlapte and wegrt
looks.
Bloks. adapteriitt
bookl & edapter. and
tlocks 3. addetterii trs
broks i uaii zmmmet toben
teeks u moal prrmrr tobes
blocke batti lake weight
lake, wrigh
weight
bloks. adaattr weight
books Sodact..
Mocha adapter Ind time
Moolu adapber ind tim
Ulodl calasto liz woigl
bodio tuddac to inz wold
Moccss actepeer lie
Mosics adeple biw-
Mocke with prompii
Mocks
weight
blocke wit Lakk
bookk 10 att LoRk a nngght
books 10 otne LofA neirtt
Nooke t0 odapper weigt-
books 10 odapter weigt
edayter lie
blodss adapter lnz the
books adapter in tie-
Mockk voul zromet saben-
Mocka 12 vauval pronpe bobes
weight
welgrt
bloca 21 udapter weiott
bleka uduptir weight
bledis adtetee in zims
boeess adaetee in zue-
blcts 11 odaptor htweeght adopher
bodks 11 odupter h2.welggtt
100
30
60
52
Uppdetable Perametern
Mostuse
Updaaaable Phrnemetees (90)
Figure 25: Sparsity pattern of attached modules to ViT-B/16 on svhn. (Left) Module- updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p = 80% is shown as vertical line.
36

---

sun397
visul promett lokan pos amoud
Visual promett laken pos amouz
booke e cauut prrmet soten
biokks y sual prrmre tobes
mlecke aatt take wiihh
aa laka weight
weiht
bldis odapter weight
Monkc n pehephe in hos-
Moric n sdappee in foo.
(ooka cuallatto ind weigl
looks.t toollatti ind wedgl
books adepter jies
Mocics adepler sie-
blods
bindas attn Labs, souggt
boeck art LaBA neight
broctal alte Lolk to trwwett
breees wiln Lolk weggt
Meder attephee waiht
booker acteptecnn wiight
(ouck edayter Lik
edapter lio
Modes
bode, jadapter
Mockks 1 adapter lna time
Modis a adapter lnz tie
bocks visua promet, Jaken
Bloks susl pronerr Jokes
books att awaight
weigrt
Medoa 2 adapter weight
boda 2 audapttrr weight
Morks adaptee in Has
Maciee attepte in puns
Hinde adaptee weight
Harls adaplee weight
bocks adapheriss
sdapter
books suat, trock attn LoRA, a seiggtt
books battn LoRd, a swaggtt
blocka with Loik, toweeg! t
Blacka with LofA, i weegl L
Mooko staptt mayll
Mooko Latlliii mecyl
bedes adupter waight
blndee adapter werght
Mocks 3 adaphertnn in zis
Mocis 5 adapher in zws
bocks vesull prompt toben-
bocks a sual prompt taben
blocks
brocks
weitt
baodss e, adaatier waggtt
bocks 4, a.adagtirn weugtt
(ookks adapter in). tas
Mocks a ada in). tas
bleda edapter weight
blecks duddapter weight
adipper
actephee
bools couse
loben
blodls i sand
boben
blecki a aith tolk, a weight alto
trecki i arte Lokk, a weight
weegl gll
with in weight
Nobule Narra
bloks odopter woight Lodls
Nodule Name
blokk S.otopter weight
Bools
books odapter lnd weight
books addgter inz welght
Luddo h edep to hit brs bools, cisuel prompl
Bluck a edhepios hui bis
blocss sood prompp
blocks catt wowptt
brocks &f auttt LoRk weight
bloces adaateecin and waight
blocks, achepter ind weigtt
Mockk adapter lo llss
Macces adopter ln) lios
Morks adapter rlac
Norite is adapter flac
books tubooul tolen
booka to soul is eicces atti LoKA a woart
talen
blocks atttn LoKA awaart
toowght
bricks autte
weight
Mocis adtptee iil tue
Mocio it attepter ani tue-
Mode: T nelepher and weight
hoder nebetter an wnigh
Mocks adspterns tus-
Mocis adapter zon-
eisus prompt sokenn
suul pwwmpt
blocks
blocks
wergtt
blocks altn Lork, boiigg
bricks in with Lank, is begglt
beadss i atlaptor and weight
bocas. it adlapte and wegrt
Llocks
Bloks. adapteriitt Ind
boolls & edapter. and
blocks 3. addetterin and bus
broks i unaie prmmet toben
teoks u soal prrmrer tobes
blocke batti take, wiight
aite taka, a wright
weight
bloks Sdasttr waigtt
bocks s.odapterih
Mocha adapter Ind tine
Mooku adapter ind Sim
Uookt 5 ealasto liz woigl L
looko hudaapiio to inz wokk
Moo adepter lie
Mesic adepter biw-
Moccs
prompii
Mocks
Mocke with Lake weight
bodie wite Lakk, weight
blokks 10 atte LoRk a nnggtt
books 10 otne LoRA neigtt
Nooke t0 adapper weigt-
books 10 odapter seight
tor
edopter lie-
weegttt
blods adapter inz tiee
bookks adapter inz sie-
Mocks vaue zromet saben
Mocka 12 vauval pronpe tobes
Docks Lonk weggtt
books LonA weigrt
bloca 21 udapter weight
blecaa 22 udupter weight
bledss adaeter in zim-
broets adagtee in zue-
blcte 11 hewegtt adtpper
bodks 11, odapter h2.weight
25
15.0 17.5 30.0 Updatatll Parametens US) Monule
5.0 7.5
30.0
12.3
3
Updatable Phrmmettess (10) Layer
Figure 26: Sparsity pattern of attached modules to VIT-B/16 on sun397. (Left) Module-wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p² = 5% is shown as vertical line.
37

---

camelyon
viuuel vocus poorup joknn bockc dliisaa block ou eite Lank, aweigtt Lioko duttin LinA lonagl L
vicuel prompe jokno prompt
blockc Sual, biicka with LanA, swight
liook duinn Line is lonagl
Mox atlepte weight
laedl adaplee weight
Medee adaeter weight
bedi adaatee weight
Mooks o adapter in top
bodis D adapter in. ras-
hode: pronet
bodke eial pranpe
Line, awwigt
brocks arm tuna, weljtt
bocks Lokd
ladel y atlite lis wiiht
lookl 3 atlagto litt seight
Mocks adapter ho bua
Mocia edapter ind awo
moois adapte woight
pocis Ladapiar woight
bode. Jueuut
taken
block pual
taken
nieets ain toik a weight
treeks wite inka, a neight
bookk 2attn LoRA bowght
seck ortn Loka bogght
sttppor
weight
Mocks
das-
Moces
bodes 2 adapter 1n2 weight
books 2 2ddapterigg in weight-
Mocck adapter lnd law
Loocl a edepter lit baw
book coust prompt ioten
bloks prompt loben
bonkee attn Loks bonggt-
blacke atto Laka weight
Medea 3 adaater and waight
bldca 3 aclatter and wegght
Mocks adapter in] tas-
Noces adapter mn tos-
Mocks odagiir weight
Mocks adagtior weight
bocks visuall promet takan
books i sual prompt tobon
blocks aattn Lokd, wighh
blecks att Loki, a wight
biocko.4.ath 4.ttt bouight
bricks attn Lotk bosight
adaepie
Morks a adapter int Fixe
Moolo adapfe in! sue-
Modes adspter an weight-
Nodo adseter to weight
Mocks adapter nn, tac
bocis adapter n2, rac-
brooko prompt
brocks a am LiNe, bowggtt
Brickd amn LonA, b weggtt
Modile nanne
laades 5 ad epte lns weigt
Modile stave
bldds 5 edleptee lis weigtt
eedee deppter ind
Bocks adapterint
waggt
bodke 5 aduster ind
books 5. aduater ind bas
bodks Commal promet sokan
bocks d eual prompt soben
blocks eurth Lokd, weight
bickss durt toka, weight
att
wight
Modil
weight
Mocla adaptor Lup
Mooia edaptoo lup
blocss adaapter in2 woigtt
bocks Gadaptor in2 woight
Mooks adepter tim
Mocia adepter hin-
biects y ailn Late, seeptt
toacks witn loti, twigtt
hoekl aath loka & nngght
teeei aime inka a weigh
bledes aduetee weight
blotee adaeptentt weight
Mocks adoptering
Mods adspter
Mades
Mockk i adapter Iiz Liz
Maclo in adapter lnz lio
bocks & couall promot loban
pooks a sual promes laban
blocks
alti LokA bwight
bixdes adspter weigh
bicll adspter weigt
terds a adoater in bas-
fronks a adooterins int has.
Nodde adletter wsight
booke adlettor weight
bocks
books sual,
block du alta Lark, aweigtt
biecla with Lank, swight
biokl bulti LiM, loniigl
liocl uith lukk, is lonsil
tirdes ataepte weight
tncls adaplee weight
adapherini
Medes adletee weight
bede adaapeed weight
Wooks 9 adapter ni tis-
Mocks 5 adapter in. ras
Morke prompt
Morks weall smorpe
bookk L
blckk Lonk, weigit
blocs t0 adapter neiglt
books 30 atlapter neight
Blecka edapter ind zup
brecko edester ind cwe
bocks adapter weigt
books oduptir weight
Morke 13 sinuo
saben
Morks uneuod
tobes
Morie aite Tokk, weigh
Miniic 11 win ioka, teight
blokks i1 otte Lokk wonght
bloks otn Lokk wiight
odtpper weght
weigt
books.
bookk
Mlacs 11 uelaptic oeggle
backs 11 aeleste weight
shocl edapter bnd Sie
Books 22 edeptor hnd boee
35
Updatable Paromotors
Module
Updatabic Parameters
Layor
Figure 27: Sparsity pattern of attached modules ViT-B/16 camelyon. (Left) Module-wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p" = 80% is shown as vertical line.
38

---

eurosat
Liken
Jiken
Visuel prompr joken emrrd
Visudl prompt joken, embed
bocks i. iiual promol
Books a sual promro joben
blecka awargtt
aveigtt
hivdes adepte weigh
horls e adepte weigh
Mooik adapter in tin
bloks adapter in con
Nirdss adaple weight
Nodds adaple weigh
adopter
adopter
horke woase
Worke wnol
toben
blocko suttn Lank, ameial
tinkk a Lunk, sedyl
bocks Lottn LokA, bo awegrt-
brcck Loke bwcght
atlagte weight
adlapiee weight
taddo stteple weugt
sacds audallla wagght
Mocks I adapber in ros-
Noces I adveter in ros-
bode promet
bode sual prompt
blocks Loka, wright
brocks Losa, weight
Madde odaiter in weight
blocee 2 adlitter in weight
loccs adopter na das
beoces adapter nu fas
odapter weight
bldk: Ladagter welght
Mocca
booio couse ioben
loben
bleeks eitn Tolkk Gregtt
Beens wite lolk neight
wonght
wnggh
weight
adapiee veight
Mocks adapter in)
bloks
Medd anlapti lnd seigh
Modo acllpiie Inct weight
(ou adapter hiz Lio
Lauilo a edaptes lnc lio
cisue prompt
blocks promps
blocks
booda attn Lonk bowrght
brocks attn Lonk weaght
Modes adapter in weight
bieioo at adspter in weight
Morks adapher rus-
Mocie adapter zon
adspter weight
adagter weight
bocks S.viual iitin
books 5 suall promolt Jollin-
blocko s-tttn Lonk, awiiigt
beokk Latk, weiats
bwagft
bwagft
prodell ranne
prodle hame
hodkes adayter int hase
Bocks 5 atigter ind bise
taados actapte and wesglt
saads aclapiee and wegnt
actes. adbator id
bocks. adbotterin bize
blocks
Blocks
blocks a alt Lokk, bowiglt
Blocks aita Lokk, weiglt
tivdes alaete lis, weight
bocks, atlagtte ini weight
Mocka adapter tun
Mocia B adapter tun
Mooks a adapter tux
Mocke it adapter tux-
brokks saust prompt sabeo
Biocks u sced prompe sobes
alto Geegt
Geight
wrigh
Modes adugter waight
hoddo aduitee inl
Mocks 7 adopter in das
bocis adaptor nnl tac
Udddo ad tee lndt weight
blddo adepple intt weight
Mooks adapter Liz
Mooiia adupter lit
blocks au malln Latt, teiplt
biicks wiln Luti twigtt
blecke atth Lokk, & weggt
trecks atth LoRk, a weght
blodes odsprer in weight
blocks, adlpper weight
bockke
Modils
Modis adugtor
Lucko, a adayter fer liz
Boock a adeiter tha
bocks &. Goual promot Joban
Books sual prompt Jobsn
blocks aattn awanght
aweight
Nirdes adlepter weiph
bevds a adepte weigh
Morkc adtppherin in rise
Moriie o selyphe in roc.
Nlodec adeptee weight
bodds adaptee weight
Mooks adopter
bocks adopter
Morke
soben
Morke wecl
tobes
Lookka 20ulta Lank, teiglt
bookk lunk, weight
Idocib Lolk, o weeglt
bastice 10 wilh tolk, is wesght
selapter oeeglt
ueleste weight
Mlucss antapiir 1x2
bucks 30 untaptir weegt
Medee adapper in roe-
booke adurter in cac
Mocks slsuo, prompr, Jaken
Macks visuel prange, prmmprooke bokes
Mocks Losa weight
bodka Losa,
wight
Mocit 11 odapter in weigtt
biokls 31 odlapter in weight
plods adapter in
books adastor whapte weight
wiapher neight
103
40
Updatable Paramotars 1982 Module
Updatable Parameters (90 Layer
Figure 28: Sparsity pattern of attached modules to ViT-B/16 on eurosat. (Left) Module-wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p² 80% is shown as vertical line.
39

---

resisc45
visil promust lokan pos amoud
Visual promet laken pos amouz
booke e cauut primet soben
bookk y sual primrer tobes
mlecke
wiigh
laka weight
weight
bldks odapter weight
bldks Cadagttnn
Monkc n petepher in hos
Moric n saapphe in hoo
Uooka culllatto lnd weigl
looka tuellapte and wedgl
books atepter jie
Mocic adepter sie-
blods
blndas attn soeght
bleck art LabA weight
brocts Lutti Lotk trowegt
breees ail alln Lolk weggt
Medes aetephee waggt-
booker acteptern wiight
Modes edapter Lim
edapter lio
bode jadapter
Mockss 1 adapter lna Lime
Modis a adapter lnz tis
bocks visua promet, Jaken
Bloks susu promer Jokes
books alt awaigtt
Weight
Medoa 2 adapter weight
boda 2 audapterr weight
Morks adaptee in Haes
Moriee attepee in pans
adaptee weigh
Harls adaplee weight
sdaptor
bocks
books sual, brack 3 attn LoRA, weiggt
bocks battn LoRk, snaggtt
Blocka with Loik, boweegl
Blocka with LofA, weegl L
stlatto mayil
Mooko Latllaii medy
Medes adupter warght
blade adupter werght
Mocks 3 adapher in zis
Mocis 5 adapher ind zwa
bocks cesul prompt Joben
bocks a sual prompt taben
blocks
brocks
wiiht
baods, e, adaptier weagrt
bocks 4, a.ddggtir weight
Mookks adapter In). tas
Mocks a adapter in). bas
bleda edapter woight
blecks duddapte weight
bools couse
loben
blokks i sund
boben
tlecke a aith tokk, a weigt-
trecki i ait Lokk, a weight
alto weeght gll
weight
Notule Narr
odopter woight
Nodule Name
block Soctpteer weight
Booko
booes odapter lnd weight
books addgtee welght
Luddo h edep to hit
Blocs a edeppios bui bis
bools prompl
boocs sond prompp
blocke aatn bowptt
brocks & atttt LoRk weight
bldda adaatee and waight
blocks, achepter and weight
Mocks adapter lo llss
Maccks adopter lni llo-
Morks i adapter Flcc
Norile is
flac
books tubuull tolen
Booka to soul is blicks atti LoKA a woart
talen
blocs. aatt LoRk awadtt
bowwggt
brecks
weight
Mocis adapter tue
Mocio it attepter ant sue-
Mode: T nelepher and weight
hedee nebepter and unigi
Mocks adopher tus-
Mociss adapter zos-
visus prompt sokenn
suol prompt
blokks
blocks
werptt
blocks alt Lonk, boiiggh
briekks in with Lank, beiglt
baadss i ataatto and wegght
bocas. it aclapte and wegrt
loock.
Bloks. adapteriitt
books & edapter. and
tlocks 3. addetterii bus
broks i unii zmmme toben
teoks u moal prrmrr tobes
blocke batti lake wiight
lake, weigh
weight
bloks. Sdaatee weight
bocks S.dast..d ind
Moohs adapter Ind tiee
Moolu adapter ind tim
(ookl 5. calatto liz woigl
bodio tuddac to inz woid
Moccs aceeter lie
Mesics adepler biw-
Mocke with prompiiisee
Mocks
weight
bloike wit Lank,
blokks 10 att LoRk a nngght
books 10 otne LofA neigtt
Nooke t0 adapper weiht-
books 10 odapter weigh
edoyter lie
blodss adapter luz the
books adapter inz tie-
Mocks voul zromet saben-
Mocka 12 vauval pronpe babes
weigrt
LonA welgrt
bloca 21 udapter we
bloca uduptir weight
bledss adaetee in zims
broess adaptee in zue-
blcts 11 odapter htwegght adopher
bodks 11 odapter h2.welght
70
20
80
Lipdetable
Updatable Phrumetters (10) Layer
Figure 29: Sparsity pattern of attached modules to ViT-B/16 on resisc45. (Left) Module-wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p² 50% is shown as vertical line.
40

---

retinopathy
viuuel vocus poorup joknn pann bocks dxiisaa
viouil prompe joknn prompg
bookl Sual, bincla with LanA, swight
block o eite Lank, awsigt Liokk duttin LinA lonngl
liocks duitn Lune is londal
Mox atlepte weight
laedl adlalle weight
Medde adleteer weight
bldd adaetier weight
Mooks o adapter in tip
bocis D adapter in. ras-
hodk: vaus pronpt
bocke eial pranne
bocks arm Line, aowilt
brocks alth tunk, weljlt
ladee y atligt lis wiight
lockl 3 atlagtoo litt seight
Mocks adapter ln bua moois adapter woight
Mecia edapter ind awo
pocis Ladaait woight
Morke
bode. Jueuut nieets ain toick a wright bookss 2attn LoRA bowggt
taken
bloda pual
taken
treeks wite inka, a neight
brocks atth Loka to wogght
wiight
sttpphor
Mocks
Moces
bodes 2 adapter in2 weight
bookd 2 2odapterigg in welght-
Mocck adepter lid lhe
Loool a edepter lit baw
booo coust prompt ioten
blocs prompt loben
bonkee attn Loks boogght
blacke atto Laka a weigh-
Medea 3 adaatee and weight
bedka 3 aclapter and wegght
Mocks adapter in] tas
Noces adapter in tos-
Mocks odagtio weight
Mocks adagtior weight
bocks cisuall promet taken
bocks i sual prompt tobon-
blocks aattn Lokd, wighh
Blecks atti LokA, a wight
biocko.4.ath 4.ttt bouight
bricks atte Latt bowight
actepie
bo a adapter int Fixe
Moolo adapfe in! sie-
Modes adspter an weight
Node adheter to weight
Mocks adapter ni, tap
bocis adapter n2, rac-
bocks
promot
tiooko prompt
brocks a am Line, bowggtt
pocck am LonA, b weggtt
Modile ranne
laades ad epee lns weigl
Modile stave
blods 5 edleptee lis weigtt
eodee degpter ind
Bocks adupterind bas
weggt
bodke 5 aduster ind
blokk 5. aduater ind
bodks Commal promet soban
bocks d eual prompt soben
blocks eurth Lokd, weight
bocks durt toka, weight
att
Modee atayter wiiht
weight
Mocla adapter Lup
Mooia edaptor lup
blocss adaatte in2 woigtt
bocks Gaddcpior in2 woight
Mooks adepter tim
Mocibu adepter hin-
biects y ailn Lates steptt
toenks with loti, twegtt
hoekle aath loka & nnggtt
treeie aime ioka a weigh
bledes aduetee weight
blocte adaepteett weight
Mocks adoptering tos-
blcces adspter ras
Mades
Modes
Mockk it adapter liz Lio
Macko in adapter lnz lio
bocks & coual promet laban-
pooks d sual promes laban
blocks
alti LokA twight
bixdls adspter weigh
binclo adspter weigt
tords a adoater in bas-
tronks a adoaterins int his
Nodde adaetee wnight
Nodle adlegtor weight
bocks
blokk sual,
blocks du alte Lark, awsiigt bioko bulti LuM, lonsial
tiieka with LatA swight
liock uith Lukk, is lonsil
Mirdec adaepee weight
tncls adaplee weight
adapperini
Medes adleteer weight
bede adaapted weight
Wooks 9 adapter n2 tiss
Mocks 5 adapter in. ras-
Morke
prompt
Morke weall smarpe babes
alto L weight
aite Lonk, weight
bookk
blckk
blocs t0 adlaterr 1n neiglt
books 30 atlapter neight
Blocka edapter ind zlep
brecka 20 edaster ind cwe
bocks t0 adupter weigt
books oduptar weight
Morke 13 sinuo
saben
Morks uneuod
tobes
Morie wite Tokk, weight
Miniic 11 wine ioka, l teight
blocks i1 otte Lokk wongght
bloks otn Lokk wiight
odtpper weght
weiht
books,
bookk
Mlccs 11 uelapte lxz oniglt
backs 21 aelepte weight
shodos edapter bnd Sie
Books edeptor hn boe
Updatable Paromotors 1983 Module
Updatabic Parameters (50) Layor
Figure 30: Sparsity pattern of attached modules to VIT-B/16 retinopathy. (Left) Module-wiss updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p" 10% is shown as vertical line.
41

---

clevr-count
Visul promutt lakan pos umoud
visual promett laken pos amouz
booke e couut primet soten
bookk y sual prrmrer tobes
mlecke aatt take wiigh
aame laka weight
weight
bldis odapter weight
Morkc n petipher in hos
Moric n sdappee in foo
(ooka culllatto ind weigl
looks.t tuodlatto and wedgl
books adepter jies
Mocic adepter sie-
boods
blndls attn soeggtt
boekk art LaBA neight
boocta alle Lolk to trweett
breess wiln Lolk weggt
Meder attephee waigt-
booker actepternt wiight
Modes edayter Liko
edapter lio
bode jadapter
Mockks 1 adapter lna time
Modis a adapter lnz tis
bocks visua promet, Jaken
Bodks susl jniner_ooee Joles
books alt awaigtt
weight
Medos 2 adapter waight
boda 2 audapterr weight
Morks adaptee in ples
Maciee attepee in pues
Hende adeptee weight
Hadls adaplee weight
bocks adapherans
sdapter
books suat, brock attn LoRA, a weiggtt
bocks battn LoRk, a swaggtt
blocka with Loik, toweegl &
Biicka with LofA, is weegl L
Mooko atapto mayll
Mooko taallaia mecyl
bedes adupter waight
blndee adapter werght
Mocks 3 adaphertng in zin
bocis 5 adapher ind zws
bocks cesul prompt toben-
bookks a sual prompt taben
blocks
brocks
weitt
baodss e, adaptier weggrt
bocks 4, a.adagtir weugtt
(ookks adapter In). tas
Mocks a ada in). tas
bleda edapter weight
blecks duddapter weight
adippeer
bools couse
loben
blodl i sand
boben
blecki a aith tokk, a weigh- altn
trecki i aite Lokk, a weight
weeght gll
with in weight
Notule Narra
blods odopter woight
Nodule Name
blocks S.otppter weight
Lodls
Books
books odapter lnd weight
books addgter inz welght
Luddo h edepias to hit bools, prompl
Bluck a edhepias hui lis
blocs sood prompp
blocke aatt wowptt
brocks LoRk weight
bloce adaateec and waight-
blocks achepter ind weight
Mos adapter lo llss
Maccks adopter lni lios
Morks adapter Flac
Norite is pdapter flac
bocks tubuull tolen
booka to soul is bicces attn LoKA a woart
talen
blocks aattn LoRk awadtt-
toowggt
brecks autte
weight
Mocis adtepter iil tue
Mocio it attepher ani tue-
Mode: T nelepher and weight
hoder nebetter and unigh
Mocks adapter rus-
Mociss adapter zoe-
eisus prompt
suol prompt
blokks
blocks
wergtt
blocks alte Lork, beiigght
fricks is with Lank, wegglt
beadss i atlaptor and weight
booas. it aclapte and wegrt
Loock.
Bloks. adapteriitt Ind
bookks & edapter. and
blocks 3. addetterin and bus
beoks i unai zrmmet toben
toeks u soal prrmrer tobes
blocke batti take wiight
aite taka, a wright
weight
bloks. odaater waigt
bocks, S.dk....d
Mocha adapter Ind tiee
Mooiu a adapter ind Sim
Uookt 5 calasto liz woigl L
looio huddapiio to inz wokk
Mos adepter tie
Mesic adepter biw-
Moccs
Mocke with prompii
Mocks
weight
bodie wite Lafk,
blokk 10 atte LoRk a nnight
books 10 otne LofA neigtt
Nooke t0 adapper weight
books 10 odapter seight
edopter lie-
weegett
blodss adapter lnz tie
books adapter inz sie-
Mocks voul zromet saben-
Mocka 12 vasual pronpe tobes
weight
booka LonA welgrt
bloca 21 udapter weigh
blecaa udupter weight
bledis adaetee in zim-
broets adaptee in zie-
blcts 11 hhhpeeehh2wwighh adtpphr
bookss 11, odapter h2.welght
22
50
10
20
40
Updaetable Parameters
Mostule
Updatable Parumettes (10)
Figure 31: Sparsity pattern of attached modules to ViT-B/16 on clevr-count, (Left) Module-wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p² 40% is shown as vertical line.
42

---

clevr-dist
visul promust lokan pos umoud
Visual promet laken pos amouz
booke e cauut primet ioten
blokk y sual primrer tobes
mlecke take wiigh
laka weight
weight
bldis odapter weight
bodks Cadapttnn
Morkc n pehepher in hos
Moris n saapphe in foo
Uooka culllatto lnd weigl
Wodks tuollatt and wedgl
books adepter jies
Mocis adepler sie-
blods
blndas attn soeggt
bleck art LaBA weight
brocta alte Lolk to trwwett
Breess ail Lolk weggt
Medes attephee waigh-
bloker attepternt wiigh
Modes edapter Limo
adapter lio
bode jadapter
Mockks 1 adapter lnz time
Modis a adapter lnz tis
bocks visua promet, Jaken
Bodks suul jniner_okee Jokes
books alt awwiggt
bocks
Weigrt
Medoa 2 adapter waight
boda 2 audapterr weight
Morks adaptee in paes
Morie attepte in pans
Hindes adeptee weight
Hadls adaplee weight
bocks adapherans
sdapter
books sual, brock 3 attn LoRA, a weiggt
bocks, battn LoRk, a snnggtt-
Blocka with Loik, toweegl
Blicka with LofA, weegl L
stapto mayil
Mooks t.ttkaiii medy
Medes adupter warght
blnde adapter werght-
Mocks 3 adapher ind zia
Mocis 5 adapher ind zwa
bocks cesul prompt toben-
bocks a sual prompr taben
blocks
brocks
weiht
baodss e, adaptie weggrt
bocks 4, a.adagtir weight
Mookks adapter In). tas
Mocks a adapter in). bas
bleda edapter weight
blecks duddapter weight
bools couse
loben
blokls i sund
boben
blecki a aith tokk, a weight altn weeght gll
trecki i ait Lokk, a weight
with in weight
Notule Narr
odopter woight
Nodule Name
blocks Soctpteer weight
Lodls
Booko
books odapter lnd weight
books addgter wedght
Luddo h edepias to his
Blucs a edeppia hui lis
bools, prompl
blocs sood prompp
blocke aatn bowptt
brocks & auttt LoRk weight
bldda adaateec and waight
blocks, achepter ind weight
Moo adapter lo llss
Maccks adopter lni lio-
Morks i adapter Flcc
Norite is pdapter flac
books tubuull tolen
Booka to soul is bicce. atti LoKA a woalt
talen
blecs. aatt LoRk awaatt
boowggt
brecks
weight
Mocis adteteer tue
Mocio it attepter ant sun-
Mode: T nelepher and weight
hodee nebepter and unight
Mocks adapher tus-
Mociss adapter zos-
eiuus prompt soken
suol prompt
blokks
blocks
wergtt
blocks alt Lonk, boiigght
briekss is with Lank, weigtt
baadss i atlapton and wegght
bocas. it adlapte and wegrt
loock.
Bloks. adapteriit
bookko & edapter. and
blocks 3. addetterii bus
broks i unai zmmmet toben
teeks u moal prrmrr tobes
blocke batti lake wiight
aime taka, a wright
weight
bloks odaatete weight
bocks Sodast.. in
Mocha adapter Ind time
Moolu adapter ind Sim
(ookl 5 calasto liz woigl
bodio tuddapioo to ind woid
Moccss adepter lie
Mocic adeple biw-
Mocke with prompii
Mochs
weight
bodie wit Lakk,
blokk 10 atte LoRk a nngght
books 10 otne LofA neagtt
Nooke t0 adapper weigt-
books 10 odapter weight
edoyter lie
blodls adapter lnz the
books adapter in tie-
Mocck voul zromet saben-
Mocka 12 vasual pronpe tobes
weigrt
LonA welgrt
bloca 21 udapter be
bloca udupter weight
bledis adaetee in zims
broets adaptee in zie-
blcts 11 odapter hiweeght adophhr
bodks 11 odapter h2.weight
10
Lipdetable Perameters US) Mostule
Updatable
(10)
Layar
Figure 32: Sparsity pattern of attached modules to ViT-B/16 on clevr-dist. (Left) Module-wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p² = 30% is shown as vertical line.
43

---

dmlab
visul promust lakan amoud
Visull promest laken pos amouz
booke e caua prrmet soben
bookks y sual primre tobes
mlecke
wiigh
taka, weight
weihtt
bldis odapter weight
bloks Locks...dagterhii
Moric putipher inn hos
Moric n salpphe in hoo
Uooka cullaatto lnd weigl
Wooko. tuellatie and wedgl
books atepter tie
Mocics adepler tie-
blnda attn nought
bieck art Labk neight
brocts aiti Lotk to browegt
breces all LoRd, weggt
Modes aetephee waght
booker actipter wiigh
Modes edapter
edapter
bode jadapter
Mockss adapter laz Lime
Modis a adaptor lnz tis
bocks visua promet, jaken
Blcks anou promer Jokes
books
awaigtt
Weight
Medos 2 adapter weight
beda 2 addapter weight
Moris adaptee in Hae-
Morie attepte in pues
adaptee weigh
Hards adaplee weight
bocks
books suat, brnck attn LoRA, welggt-
bocks. bottn LoRk, swdggtt
Blocka with Lotk, boweegl
Blicka with LofA, weegl
staatto mayil
Mooks teollaio medy
Medes adupter waight
blade adapter wergrt
Mocks 3 adapher in zia
Mocis 5 adapter ind zws
bocks cesul prompt Joben
bocks a sual prompt taben
blocks
bracks
wiitt
boodss adaptier weagrt
bocks 4, a.ddgtiir weugtt
Mookks adapter hn) tas
Mocks a adapter in). bas
bleda edapter woight
blecks Auddapte weight
bools couse
loben
blokls i sund
boben
tlecke a aith tolk, a weight
breckie i ait Lokk, a weight
gll
weight
Nobule Narr
odopter woihtt
Nodule Name
Soctpteer weight
Booko
books odapter lnz weight
bloke addgtte welght
Luddo h edepias ta his
Blucs edeppie hui bis
bools prompl
boocs sessell promppp
blocke
bowptt
brocks LoRk weigtt
bldds adaptee and waight
blocks, achepter and weigtt
adapter los llas
Maccks adopter lni lls-
Morks i
flc
Norile is
flac
books tubuull tolen
Booka to is blicks atti LoKA a woaft
talen
blods. aatt LoRk awadtt
boweght
neight
Mocis adapter tue
Mocio it attepter ant sue-
Mode: 7 nelepher an weight
heder nebepter an wnigh
Mocks adopher tus-
adapter zoe
nusus prompt sokenn
blocks prompt
blokks
werptt
blocks witt Lonk, beiigght
triekls in with Lank, biiglt
taadss atlapton and wesght
bocas. it aclaptis and wegrt
looks.
Bloks. adapterilnt
books & edapter. an
tlocko 3. addetterii tue
broks i unii zmmme toben
teeks u moal prrmrr tobes
blocke
weight
take, wrigh
weight
bloks. adaattr in weight
bocks S.dast..d
Mocha adapter Ind tiee
Moolu adapter ind tim
(oodl calasto liz woigl
bodio tuodacioo li inz wold
Moccss actepeer lie
Mosics adeple biw
Mocke with prompiiisee
Mochs
weight
blocke wit Lokk,
books 10 attt LoRk a nngght
books 10 orne LofA neirtt
Nooke to odapper weght-
books 10 odapter weigt
edayter lie
blodss adapter luz the
bookks adapter ln tie-
Mockk voull zromet saben
Mocka 12 vasval pronpe bobes
bocks
weight
welgrt
bloca 21 udapter in2 weiott
blecaa uduptir seight
bledis adtepee in zims
breess adaptee in zue-
bects 11 odaptor hiwwght adopher
bodks 11 odupter h22welggtt
100
20
30
Uipdatable Perametenn
Mostute
Upriatable
Phrnimatees (10)
Figure 33: Sparsity pattern of attached modules to ViT-B/16 on dmlab. (Left) Module-wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p = 80% is shown as vertical line.
44

---

kitti
visul promust lokan pos amoud
Visual promet laken pos amouz
booke e cauut prrmet soben
books y sual primrer tobes
mlecke take wiigh
laka weight
weiht
bldis odapter weight
bodks Lodks..adacterhii
Morkc n petepher in hos
Moric n sdapper in foo
Uooka culllatto lnd weigl
(ooko.tued tuellapti and wedgl
books adepter jie
Mocics adepler sie-
bloks
blndas attn Labs, sought
blecks art LaBA weight
brocta alte Lotk trowegt
breces ail Lolk weggt
Medes attephee waggt-
booker actiptern wiigh
Modes edapter Lim
edapter lio
bode jadapter
Mockss 1 adapter laz time
Modis a adapter lnz tis
bocks visua promet, Jaken
Bloks suol jrimer Jokes
books alt awaigtt
weigrt
Medoa 2 adapter waight
boda 2 addapterr weight
Morks adaptee in Hles
Morie attepte in pans
Hindes adeptee weight
Hadls adaplee weight
bocks adaphernt
sdapter
books sual, brock 3 attn LoRA, a wilggt
bocks, battn LoRk, snaggtt
Blocka with Loik, toweegl L
Blacka with LofA, weegl L
staatt mayil
t.akkaii medy
bedes adupter warght
blnde adapter werght-
Mocks 3 adapher in zis
Mocis 5 adapher ind zwa
bocks cesul prompt toben-e
bocks a sual prompt taben
blocks
brocks
weiht
baodss e, adaptier weggrt
books 4, a.adagtir weight
Mookks adapter In). tas
Mocks a adapter in). bas
bleda edapter woight
blecks duddapter weight
adipper
bools couse
loben
blokss i sund
boben
blecki a aith tokk, a weigh- altn
treckie i ait Lokk, a weight
weeght gll
with
weight
Notule Narr
bloks odopter woigtt
Nodiul Name
blocks Soctpteer weight
Lodls
Booko
books odapter lnd weight
books addgtte welght
Luddo h edepias to his
Blucs a edeppia bui lis
bools, prompl
books sond prompp
blocke aatt bowptt
blocks & atttt LoRk weight
bloda adaatee and waight
blokks achepter ind weigtt
Mockk adapter lo llas
Maccks adopter lni lio-
Morks i adapter rlcc
Norite is pdaptor flac
books tubuull token
Bodka to soul is elecks attn LoKA a woart
talen
blecs. aatt LoRk awaatt
bowwght
brecks
weight
Mocis adapter tue
Mocio it attepter ant tue-
Mode: T nelepher and weight
heder nebepter and unigi
Mocks adapher zus-
Mocis adapter zos-
eiuus prompt soken
suol prompt
blokks
blocks
wergtt
blocks alt Lork, boiiggh
brickss in with Lank, weigtt
baadss atlapto and wegght
bocas. it adlapte and wegrt
loock.
Bloks. adapteriit Ind
bookk, & edapter. and
blocks 3. addetterii bus
broks i unai zmmmet toben
tooks u soal prrmrer tobes
blocke batti take weight
taka, wright
weight
bloks addastr weight
bocks S.da....d
Mocha adapter Ind tiee
Moolu adapter ind tim
(ookl 5 calatto liz woigl
booio tudda to ind woid
Moccss adepter lie
Mosics adepler biw-
Mocke with prompii
Mocks
weight
blocke wite Lakk,
blokk 10 atte LoRk a nngght
books 10 otne LofA neigtt
Nooke t0 adapper weght-
books 10 odapter weight
edoyter lie
blodss adapter luz the
bookks adapter inz tie-
Mocck voul zromet saben-
Mocka 12 vauval pronpe bobes
weight
LonA welrrt
bloca 21 udapter in2 weight
bleca uduptir weight
bledis adaetee in zim-
broets adaptee in zie-
blcts 11 odapter hiweght adopphr
bodks 11, odupter h2.weight
100
20
30
Lipdetable Perameters
Mostute
Updaaaable
Phrnematees (10)
Layaer
Figure 34: Sparsity pattern of attached modules to ViT-B/16 on kitti. (Left) Module-wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune optimal sparsity level p = 80% is shown as vertical line.
45

---

dsprite-loc
viuuel vocus poorup joknn pann bocks dxiisaa
vicuel prompe jokno prompt
blocke Sual,
block o eit Lank, awsigt
bincka with LanA, swight
Liokk duttin LinA lonngl L
livcks duinn Lune is lonagl
Mox atlepte weight
laedl adlaple weight
Medde adleteer weight
bldda adaepte weight
Mooks o adapter 1n tos
bodis D adapter in. ras-
hodk: pronpt
bodke eial pranpe
bocks arm Line, aewigt
brocks arm tunk, weljtt
ladel y atlige lns wight
looke 3 atlagtoo litt seight
Mocks adapter io bua
Mocia edapter ind awo
moois adapter woight
pocis L.daapt woight
bode. Jueu
taken
block pual
taken
nieets ain toick a origt
treeks a wite inka, a neight
books 2attn LoRA bowght
seck atth Loka to wogght
sttppho
weight
Mocks
das-
Moces
bodes 2 adapter 1n2 weight
books 2 2ddapterig in welght-
Mocck 2 adapter hid law
Locol a edepter lit baw
bodo coust prompt ioten
blocs prompt loben
bocke attn Laks bonggtt
blacke atto Laka weigh-
Medea 3 adapter and waight
bedaa 3 aclapter and weggt
Mocks adapter in] tas-
Noces adapter mn tos-
Mocks odagtoo weight
Mocks adagtior weight
bocks cisuall promet taken
books i sual prompt tobon
blocks aattn LokA, wighh
blecks atti Loki, a wight
biocko.4.att 4.ttt bowigh
bricks atte Latk bweight
actepier
Morks a adapter int Fixe
Moolo adapte in! sue-
Modes adspter an weight-
Node adspter to weight
bocks Mocks a adapter n1, tac
bocis adapter n2 rac-
trook prompt
brocks a am LiNe, bowggtt
Blockd arn LonA, b weggtt
Modile nanne
laades 5 ad epee lns weigt
Modile stave
bldds 5 edleptee lis weigtt
bodee deppter ind
Bocks adapterint
wargt
weggt
bodke 5 aduster ind
booke 5. aduater ind
bodks Commal promet sokan
bocks d eual prompt soben
blocks eutth Lokd, weight
blecks durt toka, weight
Model att
Modee otkyter wiiht
weight
Mocla adapter lvd Lup
Mooia edaptoo ind iup
blocss adaaptei in2 woigtt
bocks Gadaclii in2 woight
Mooks adepter tim
Mociau adepter hin-
biects y ailn Late, seeptt
toenks with loti, twegtt
hoekle aath loka & nngght
teeei aime ioka a weigh
bledes aduptee weight
blodee adieptentt weight
Mocks adoptering
blccss adspter ras
Mades
Mods
Mockk i adapter Iiz Lio
Macko in adapter lnz lio
bocks & coual promet loban
pooks a sual promes laban
blocks
alti LoMA bwight
bixdes adspter weigh
bicdls adspter weigt
tords a adoater in Eas-
tronks a adoaterins int his
Noddes adaetee wnight
Nodle adaettor weight
bocks
books sual,
block du alte Lark, aweigtt
biicks with LatA, swight
bioko bulti LiM, lonsigl
liock uith lukk, is lonsil
tirdes adaepte weight
tncls adaplee weight
adapherini
Medes adletee weight
bedi adaapee weight
Wooks 9 adapter ni tis-
Mocks 5 ada in. ras-
Morke
prompt
Mocks weall smarpe
alto L
wite Lonk, weigtt
block
blocka
blcks 10 adapter 1n neiglt
books 30 atlapter neight
Blecia edapter ind dup
brecks 20 edester ind cwe
bocks t0 adupter weigt
books oduptar weight
Morke 13 sinuo
saben
Morks uneuod
tobes
Morie aite Tokk, A weigh
Misiic 11 wine ioka, l teight
blocks i1 otte Lokk tonghtt
bloks otn Lokk wiight
odtpper weght
weiht
books.
boock
Mlccs 11 uelapte oeggle
backs 21 aelepte weight
shodo edapter bnd Siee
Books edeptor hnd boe
Updatable Paromotorss
Module
Updatabic Parameters (80) Layor
Figure 35: Sparsity pattern of attached modules to VIT-B/16 dsprite-loc. (Left) Module wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's s optimal sparsity level p" 90% is shown as vertical line.
46

---

dsprite-ori
vourl poorup jokon bockc duiisaa
viouil prompe jokno prompt
bookl Sual, bineka with LanA, swight
blocks du eite Lank, awsiatt Bioolo duttin LinA londgl
linck duitn Lune, londgl
Mod atlepee weight
baell adaplle weight
Medee adleteer weight
bldd adaatee for weight
Mooks o adapter in top
Mocis D adapter in) rag
hode: pronet
bodke eial pranne
bocks aewilt
tunk,
brocks arm
weigh
ladee atlgge lis wiight
lookl atlagto list seight
Mocka adapter io bua
Mocia edapter in awo
moois adapter woight
pocis Ladapier woight
bode. Juuu
taken
booda pual
taben
nietts ain toick a wright
trenks wite inka, a neight
bookks zattn LoRA bowggt
brocks Loka bogghtt
wiight
weigh
Mocks
Moces
bodes 2 adapter 1n2 weight
books 2 Lodapter ind weight
Mocck adepter hnd law
Moocl a adepter lit baw
book coust prompt
books prompt loben
bonkee attn Loks bonggtt
blacke atto Loka, weigt-
Medea 3 edaptee and waight
bedia 3 aclapter and weggh
Mocks adapter in) tas
Noces adapter inl tos-
odagtio weight
Mocks adaeiior weight
Mocks
bocks visull promet taken
bocks a sual promes toben
blocks aattn Lokd, wighh
Blecks LokA, wiigh
biocko.4.att 4.ttt bouigh
bricks attn Lotk bosight
adaepie
bo adapter int Fixe
Moolos adaptee in? sue-
Modes adspter an weight-
Nods adtepeer to weight-
Mocks adopter ni tac
bocis adapter n2, Fac
biooko prompt
brocks a LiNe, bowgght
bricks Lone, wegggt
Wodilt mtnne
Maades ad epte lis weigl
Modile stane
bldds 5 ecleptee hit weigtt
eedee degpter and
Bocks adupterand
waggt
bodke 5 aduster ind
books 5. aduater ind bas
bodks d Cammal promet soban
bocks du sual prompt soben
blocks eurth tokd, weight
sicks duam toka, weight
Modle
wight
Model arfigtor weight
Mocla adupter Lup
Mooka edaptes lup
boocs adlapter 1n2 woigtt
bocks Gadactor in2 woight
Mooks adepter tie
Mocie adepter hin-
biects y all Late, seeptt
toacks witn loti, twigt
hoeke aath toka, & nnnght
teeei aime inka a weigh
bledes aduptee weight
blote adtepteett weight
Mocks adapterin
Mods adspter
Mades
Mockk adapter Iiz Liz
Moclo in adapter inz lio
bocks & coual promot loban
pooks d sual promes laban
bwight
bixdes adspter weight
boxlo adappe weigt
terls a adoater in baw-
Hrokks a adboteerin int has
Nodie odletter wsight
hodle adleptor weight
bocks
books sual,
block du Lark, aweiiat
biecla with Latk, swight
bioklo bulti LuM, lonsigl
liokl uith luke, boncal
Mindes atlepte weight
tacls adaplee weight
adapperini
Medes adleteer weight
bede adaapted weight
Wooks 9 adapter nn tis-
Mocks 5 ada in. ras
Morke prompt
Morke weal smarpe
bookk Lona, we
Mocka tonk, weight
blocs t0 adapter neiglt
books 30 atlapter neight
Blocka edapter ind zup
breck edester iew
adaptir weigt
oduptar weight
Morks sinue
saben
Morks uneued tobes
Morke wite Tokk, Weight
Minice 11 win ioka, teight
blocks ottr pwwght
bloks orn Lokk wiight
bookss weght
weigt
bookk
Mlacs unlapte lngglt
backs 11 atlett weight
shodo edapter Ind
Books edeptor boe
Updatable Paromoterss Module
Updatabic Parameters
Layor
Figure 36: Sparsity pattern of attached modules to VIT-B/16 dsprite-ori. (Left) Module-wiss updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p" 70% is shown vertical line.
47

---

snorb-azim
visul promust lakan anoud
visual promest Jaken pos umooz
booke e caua prrmet soben
blokks y sual primre tobes
mlecke
wiihh
laka weight
bldks E.dapterii weiht
bldis odapter weight
Moric putipher inn hos
Moric n salpphe in for
Uooka cullaatto lnd weigl
Wodls. tuedlapto and wedgl
books atepter tie
Mocic adepler sue-
bloks
blnda attn nought
boeck art LabA neight
brocts aiti Lotk to trowegt
Breees ail LoRd weggt
Modes aetephee waigh
booker actipttr wiggh
Modes edapter
edapter
bode jadapter
Mockss adapter laz Lime
Modis a adaptor lnz tis
bocks visua promett Jaken
Bocks a promer Jokes
books
alaiggt
Weight
Medos 2 adapter waight
beda 2 addapter weight
Moris adaptee in Hae
Morie attepte in pums
adaptee weigh
Hards adaplee weight
bocks.
books suat, boack aatt LoRA, welggt-
bocks. battn LoRk, s.idgtt
Blocka with Lotk, boweegl
Blicka with LofA, weegl
atapto mayll
Mooko Letllaii medy
bedes adupter waight
blade adapter wergrt
Mocks 3 adapher in zis
Mocis 5 adapter ind zws
bocks cesul prompt Joben
bocks a sual prompt taben
blocks
brocks
wiitt
boodss adaptier weagrt
books 4, a.ddggtir weugtt
Mookks adapter In). tas
Mocks a adapter in). bas
bleda edapter woight
blecks duddapter weight
bools couse
loben
blokls i sand
boben
tlecke a aith tolk, a weight
trecki i ait Lokk, a weight
weeght
weight
Notule Narr
odopter woight
Nodule Name
S.otppter weight
Booko
books odapter lnz weight
bloke addgtte welght
Luddo h edep to his
Blocs edeppie hui bis
bools prompl
boocs promppp
blocke
bowept
brocks LoRk weigt
bldds adaptee and waight
blckks, achepter and weight
adapter los llss
Maccks adopter lni lla-
Morks
flcc
Norile is
flac
books tubuull tolen
Booka to is sicces atti LoKA a woaft
talen
blods. aatt LoRk awadtt
boweght
brecks
neight
Mocis adaptee tue
Mocio it attepter ani tun-
Mode: 7 nelepher an weight
heder oebepter and wnigh
Mocks adopher tus-
adapter zoe
visus prompt sokenn
blocks prompt
blokks
werptt
blocks sutti Lonk, boiiight
brickss in Lank, biiglt
taadss atlapton and wesght
bocas. it aclapte and wegrt
looks.
Books adapterilt
books & edepter. and
tlocko 3. addetterii
broks i uaii zmmmet toben
teeks u moll prrmrr tobes
blocke take, weight
take, wright
weight
bloks. adaater weight
bocks S.dast..
Mocha adapter Ind tiee
Mooku adapter ind tim
Ulodl calasto liz woigl
bodio tudda to inz wold
Moccss actepeer lie
Mocics adeple biw
Mocke with prompii
Mocks
weight
blocke win Lakk
books 10 attt Lolk a nngght
books 10 orne LofA neirtt
Nooke to odapper weigt-
books 10 odapter weigt
edayter lie
blodss adapter lnz the
books adapter in tie-
Mockk voull zromet saben-
Mocka 12 vauval pronpe bobes
weight
welgrt
bloca 21 udapter weiott
blecaa udupter weight
bledis adtetee in zims
boeess adaetee in zue-
blcts 11 odapter hiwweght adopher
bodks 11 odupter h2.weight
100
20
60
Uppdetable Perametern Mostute
Updabable
Figure 37: Sparsity pattern of attached modules to ViT-B/16 on snorb-azim. (Left) Module-wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p² 70% is shown as vertical line.
48

---

snorb-ele
Visul promett lokan pos umoud
Visual promet taken pos amouz
booke e cauut prrmet soten
bookk y sual primrer tobes
mlecke aatt take wiigh
aa laka weight
weigt
bldis odapter weight
bodks Codict..dccterii
Morkc n setepherin in hos
Moric n sdapper in foc
(ooka cuallatto ind weigl
tooko tuollatto and wedgl
books adepter jies
Mocic adepler sie-
blods
blndls attn souggt
bieck art LaBA neight
broctal alln Lolk to trwwett
breees wiln Lolk weegt
Meder aetephee waight
booker actepteeat wiight
Modes adapter Lim
edapter lio
bode jadapter
Mockks 1 adapter lnz time
Modis a adapter lnz tie
bocks visua promet, Jaken
Bloks suol jriinrr Joles
books alt alwiggt
weigrt
Medoa 2 adapter waight
boda 2 audapterr weight
Morks adaptee in ples
Maciee attepee in puns
Hende adeptee weight
Hadls adaptee weight
bocks adapherins
sdapter
books sual,
bocks, battn LoRk, a anaggtt
brock 3 attn LoRA, a seiggtt
Blocka with Loik, toweegl L
Biicka with LofA, weell L
Mooko staatt mayll
Mooks Laallaga mecyl
bedes adupter waight
blnde adapter werght-
Mocks 3 adaphertnn ind zia
Mocis 5 adapter ind zws
bocks cesul prompt Joben-
bookk i sual prompt taben
blocks
brocks
weiht
baodss e, adaptier weggrt
books 4, a.adagtir weugtt
(oocks adapter In). tas
Mocks a adayter in). tas
bleda edapter weight
blecks duddapter weight
adipperr
bools couse
ioben
bloks i sand
boben
blecki a aith tokk, a weight altn
trecki i ait Lokk, a weight
weeght gll
with in weight
Notule Narra
blods odopter woight
Nodule Name
block Sootppter weight
Booko
books odapter lnd weight
books addgter welght
Luddo h edep to his
Blucs a edeppie hui lis
bools, prompl
Boocs sood prompp
blocks aatt bowptt
blocks atttt LoRk weight
bloce adaateec and waight-
blocks achppter ind weagtt
Moo adapter lo llss
Maccks adopter lni lio-
Morks i
Flcc
Norite is adaptor flac
bocks tubuoll tolen
Booka to soul is eicces atti LoKA a woart
talen
blocks atttn LoRk awadtt-
boowggt
brecks auttn
weight
Mocis adteteer iil tue
Mocio it attepter ant sue-
Mode: T nelephe and weight
hoder nebetter and unigh
Mocks adapter rus-
Mociss adapter zoe-
eisus prompt sokenn
suol pwwmpt
blokks
blocks
wergtt
blocks alte Lork, boiiggh
brickss is with Lank, weigtt
beadss i ataapto and wegght
booas. it adlapte and wegrt
loock.
Bloks. adaptiri Ind
bookk, & edapter. and
blocks 3. addetterin and bus
broks i unai zrmmet toben
teeks u soal prrmre tobes
blocke batti lake wiight
aime taka, a wrigh
weight
bloks odaater weight
Mocha adapter Ind time
Mooku a adapter ind Sim
Uookt 5 calatto liz woigl L
bodio tuddapiis to inz wokk
Mos adepter lie
Mosic adepter biw-
Moccs
Mocke with prompiiisse
Mocks
weight
blocke wite Lakk,
blokk 10 otte LoRk a nngght
books 10 otne LofA neigtt
Nooke t0 adapper weiht-
books 10 odapter seight
edopter lie-
blodls adapter lnz the
books adapter lnz siee
Mocks voul zromet saben-
Mocka 12 vasual pronpe babes
weight
books LonA welgrt
bloca 21 udapter weight
blecaa 21 uduptir weight
bledis adaetee in zim-
broets adaptee in zie-
blcte 11 odapter hiwwgtt adtpper
bodks 11, odapter h2.welght
100
100
Uipdetable Perameters US) Mostute
Updataable Phrnmattess (10) Layer
Figure 38: Sparsity pattern of attached modules to VIT-B/16 on snorb-ele. (Left) Module-wise updatable parameters (%) and (Right) Layer-wise updatable parameters (%). The BayesTune's optimal sparsity level p* = 90% is shown as vertical line.
49


Reviews:
[{"review_id": "ndev69LQ1c", "reviewer": "Reviewer_2Yw3", "summary": "The authors propose an automated sparse fine-tuning method for foundations model, that bypasses the need for human intuition-based heuristics. The neurons to update are revealed during the posterior inference of the sparse scale parameters of a Laplace prior, by thresholding the scale parameters.\nThe method is experimentally validated in both vision  and NLP tasks.", "questions": "- Why was the Gamma distribution chosen as a hyperprior ?\n- If the step after SGLD is to evaluate the mean of the scale parameters, aren't there cheaper methods that evaluate the mean, without stochastically approximating the posterior ?", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The proposed method is principled, and relies on a hierarchical Bayesian model.\n\nPosterior approximation is done with Langevin MCMC, which does not introduce a significant computational overhead. \n\nThe experimental results show that the proposed method convincingly improves upon already existing heuristics.", "weaknesses": "I wonder what's the rationale behind using the elbow rule (figure 1) to select the proportion p of parameters to update.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**1. I wonder what's the rationale behind using the elbow rule (figure 1) to select the proportion** $p$ **of parameters to update.**\n\nThis is just an illustration of one potential heuristic method to select the sparsity level $p$. Other criteria could also be used. \n\n>**2. Why was the Gamma distribution chosen as a hyperprior ?\nIf the step after SGLD is to evaluate the mean of the scale parameters, aren't there cheaper methods that evaluate the mean, without stochastically approximating the posterior ?**\n\nWe adopt Gamma because with $\\alpha<1$ it has mode at $0$ and it is decreasing, thus allowing us to express our preference of small $\\lambda$ a priori. But any other distribution with this property could also be used as a hyperprior.\n\n*Regarding SGLD posterior mean*: Yes, one can use other methods to evaluate the posterior mean or any surrogate for it, eg, SWAG or EMA could potentially be used to estimate the posterior mean. However SGLD is also as efficient as these other alternative methods, and benefits from the fact that SGLD’s stochastic dynamics guarantee (in theory) to lead to the posterior mean exactly. "}, {"review_id": "3Ur8iAtdAs", "reviewer": "Reviewer_F3ir", "summary": "The paper proposes a new Bayes-based framework for sparse fine-tuning. The proposed method applies a Laplace prior (centered at the pre-trained weights) and a hyper Gamma prior over the scale parameter \\lambda of the Laplace prior. The method then performs posterior inference over \\lambda to determine whether or not a parameter is \"useful\": A large value of  \\lambda indicates a flat prior, i.e. an informative parameter, and vice versa. The paper then adopts SGLD to perform inference over \\lambda and then uses human assistance to determine the cut point for \\lambda given a budget. The method is evaluated on language and vision tasks and compared against a wide range of parameter efficient fine-tuning methods. The proposed method overall demonstrates performance superior and acquires a sparser model than baseline methods.\n\n## Post rebuttal update: I raised my score from 4 to 5 seeing the new results provided by the author.", "questions": "- What optimizers are used for BayesTune? Is it standard SGD through out the entire algorithm or it is a mix of SGD and Adam. \n\n- Does the baseline approach use same optimizer or they use different optimizers? The choice of optimizer can potentially affect the final performance.\n\n- What happen if you increase the rank of LoRA to let LoRA have the same the number of parameter used by BayesTune?\n\n- What's the advantage of posterior inference over MAP estimation in this setting, i.e. remove the last term in Eq.5?\n", "limitations": "The proposed method introduces extra hyper-parameter for SGLD (besides step-size scheduling): Effective data size and noise discount factor.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is technically sound and well presented.\n\n- The proposed method is evaluated on a wide range of tasks.\n\n- The authors consider a thorough amount of baseline methods.", "weaknesses": "The biggest issue of the proposed method, in my opinion, is the novelty. The use hierarchical prior, which induces sparsity, is a widely known idea since the last century, from Radford Neal and David MacKay's early works to recent works such as [1, 2, 3, 4]. Although these works do not consider the fine-tuning setting, they can technically be applied to the fine-tuning of large pre-trained models by simply letting the prior to be centered at the pre-trained weigh rather than zero. \n\nIn addition, the configuration of SGLD is not presented very clearly. I would suggest the author use formula to describe the modifications to SGLD described from line 212 to line 223, e.g. a modified version of Eq.5 where a few additional hyper-parameters are added to the likelihood and the injected noise term.\n\n*Minor: Some figures are not in vector format and the fonts are too small.\n\n*Minor: It would be interesting to see the performance of the proposed model on *Large* language model.\n\n\n[1] Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference\n[2] Dropout as a structured shrinkage prior.\n[3] Posterior concentration for sparse deep learning.\n[4] Bayesian compression for deep learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**1. The biggest issue is the novelty. They use hierarchical prior, which induces sparsity, is a widely known idea since the last century, from Radford Neal and David MacKay's early works to recent works [1,2,3,4]. Although these works do not consider the fine-tuning setting, they can technically be applied to the fine-tuning of large pre-trained models ...**\n\nHierarchical Bayes (HB) is indeed well known, and there were some works that adopt HB for sparse deep learning as the reviewer listed. But as far as we know, our approach has three main differences from these previous works: \n\n  - i) Prior works are all about sparse *training*, instead of sparse *fine-tuning*. Thus they focus on zeroing out many parameters, instead of retaining pre-trained weights. The reviewer said “These works can technically be applied to fine-tuning …”, but to the best of our knowledge, no one has done it before. \n\n  - ii) Importantly, the neural networks used in those previous studies are rather small/toy scale (mostly focusing on MLPs and LeNet sized architectures up to RN18 at largest)  while our method can obtain state of the art results on large-scale foundation models (ViT, RoBERTa). For example the largest model considered by the cited papers, RN18 in [1] is ~11M parameters vs RoBERTa’s ~123M parameters, a 10X scale difference. \nThe reason is that the Bayesian learning methods the reviewer referred to used inefficient MCMC approaches like Metropolis-Hastings, or methods that entail extra memory cost like variational inference (VI). All of this impedes applicability to big networks. We have just done some experiments that compare the computational resources required by VI and SGLD: On ViT networks, the training time is increased by 1.7 times if we replace SGLD by VI; the GPU memory footprint is increased by 2.1 times. \n\n  - iii) As far as we know, no one has used SGLD in sparse deep learning at these large network scales. Although [1] (Masked BNN) used MCMC, it adopted Metropolis-Hastings, which is less sample efficient due to the rejection probability. \n\n>**2. I would suggest the author use formula to describe the modifications to SGLD described from line 212 to line 223.**\n\nAs the reviewer suggested, and also for the purpose of better clarification, we will re-write Eq.5 to reflect the modification as described in Line212-223.\n\n>**3. What optimizers are used for BayesTune? Does the baseline approach use same optimizer?**\n\nThe Adam optimizer is used for all competing methods on Tab 1, following the common practice in [Lee19, Jiant20, Xu21]. For the vision tasks (VTAB), all methods use the AdamW optimizer following standard practice in the VTAB benchmark suite.\n\n  - [Lee19] Mixout: Effective regularization to finetune large-scale pretrained language models, C. Lee et al., ICLR 2019.\n  - [Jiant20] Jiant2.0: A software toolkit for research on general-purpose text understanding models, J. Phang et al., http://jiant.info/, 2020.\n  - [Xu21] Raise a child in large language model: Towards effective and generalizable finetuning, R. Xu et al., EMNLP 2021.\n\n>**4. What happens if LoRA have the same the number of parameter used by BayesTune?**\n\nFor NLP tasks, we have the same sparsity level ($p=0.005$) for all competing methods, so it is already a completely controlled comparison. For vision (VTAB) tasks, we think that the reported LoRA with dim=8 was the optimal hyperparameter choice. However, as the reviewer suggested, we can match the sparsity levels of LoRA and our BayesTune for a more controlled comparison. Please recall that LoRA-dim8 amounts to updating 0.29M parameters while BayesTune updated 0.38M parameters. \n\nInstead of increasing the dimension of LoRA, which might require corresponding re-tuning of other LoRA learner hyperparameters, thus potentially being unfair to LoRA, we instead decrease the sparsity level for our  BayesTune so that we have the same 0.29M parameters updated. The results are: \n\nModel (# params updated) | Average Rank | # times Rank=1\n:------: | :---: | :----:\nLoRA (0.29M)       |  2.68        | 4 \nBayesTune (0.38M)        |  2.37            | 7 \nBayesTune (0.29M)          |  2.58            | 6 \n\nSo, even when the number of parameters are made equal, BayesTune outperforms LoRA. \n(Note here that we used the latest version Table 3 in Appendix, instead of Table 2). \nPlease also note that the other existing competitors in the VTAB comparison used to compute ranks above are not parameter count controlled like this, as parameter count controlling was not standard practice in prior work.\n\n>**5. What's the advantage of posterior inference over MAP estimation in this setting, i.e. remove the last term in Eq.5?**\n\nMAP vs. SGLD: MAP aims to find a mode of the posterior distribution, which might be more sensitive to data noise than the mean of the posterior. \n\nThe following is some empirical comparison between MAP and SGLD on NLP tasks, showing that this distinction does lead to empirical benefit: \n\nAvg 10 runs | CoLA | STS-B | MRPC | RTE | CB | COPA | WSC | **AVG**\n:--: | :--: | :--: |  :--: |  :--: |  :--: |  :--: | :--: | :--: \nMAP   | 58.55 | 90.13 | 90.54 | 76.72 | 86.71 | 71.09 | 60.34 | 76.30\nSGLD  | 60.85 | 90.40 | 90.61 | 77.87 | 91.25 | 75.00 | 60.87 | 78.12\n\nAnother benefit of SGLD is that we can also exploit the variance of the $\\lambda$ posterior in parameter selection (e.g., for two parameters with similar posterior mean $\\lambda$ values, we prefer to select the one with smaller posterior variance).   In future algorithms, we can also exploit this idea of variance-based weight pruning (but we didn’t do this yet). \n\n>**6. (Minor) Some figures are not in vector format and the fonts are too small. \nIt would be interesting to see the performance of the proposed model on Large language model.**\n\nWe apologize for this. After the review/rebuttal phase, we will replace them with vector format figures. We plan to do it on LLMs even larger than RoBERTa, eg, LLAMA, in our future work. "}, {"review_id": "DwVnJyMZwq", "reviewer": "Reviewer_GHms", "summary": "A principled approach for selecting a subset of parameters to fine-tune in large foundational models is proposed. The authors rely on Bayesian inference to identify this subset. They begin by placing a Laplace prior over the model weights and a gamma hyperprior over the weight scale. Next, they employ an MCMC method to obtain a posterior distribution. Finally, they rank the weights based on the posterior scale values and proceed to fine-tune the parameters with the highest inferred scale values. Across standard NLP and vision adaption tasks, they demonstrate a strong empirical performance compared to previous state-of-the-art approaches. ", "questions": "- Although the choice of Laplace prior is adequately discussed and justified in Section 2, the explanation of the mean-field assumption in equation (1) could benefit from further elaboration. While I understand that this simplification is made for computational tractability, it would be valuable to provide additional justification for this decision. Additionally, I am curious whether the authors anticipate any additional performance improvements by attempting to model correlations between different model parameters. Similarly, I am interested in their choice of the approximate inference scheme (SGLD). Did the authors consider any other approaches (e.g., variational inference)?", "limitations": "Limitations are currently not discussed. Perhaps authors could use the gained space from restructuring the related work section (see above) to add a paragraph or two on the limitations of their approach.\n\n\n\n[1] Daxberger, E., Nalisnick, E., Allingham, J.U., Antorán, J. and Hernández-Lobato, J.M., 2021, July. Bayesian deep learning via subnetwork inference. In International Conference on Machine Learning (pp. 2510-2521). PMLR.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- I find the proposed methodology technically sound, novel, and appealing. Though it should be kept in mind that I am not particularly familiar with the related work on the finetuning of large models. Similarly, I like methods that are Bayesian, so this might add to my potentially biased evaluation here. \n- I like the simplicity of the proposed method. It does not happen often that I understand the method upon the first pass through the paper, but I think that was the case when reading this manuscript.\n- The empirical results are strong and I want to commend the authors on extensive evaluation (i.e., they do not stop at the language modality, but additionally consider a vision domain as well).", "weaknesses": "A presentation could be somewhat improved to further strengthen the manuscript. Some concrete suggestions:\n- I find it a bit strange to start introducing the notation already in the Introduction section. Hence, I would change your current section 1.1 into a separate section 2.\n- Similarly, I find it a bit weird to list out the related work as bullet-points they way you do in Section 3. I find it more natural to use separate paragraphs for that (see Section 8 in [1] for an example of that). This way you can also directly talk about how each related approach connects to your proposed model, instead of doing that in a separate paragraph as is currently done (lines 192-198).\n- Line 227: I assume you will include a GitHub link here, not put the code in the Supplementary material (whatever that means).\n- Grammar and style could be improved at some points to improve readbility. This is easily done these days via tools like Grammarly, ChatGPT...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**1. (The reviewer provided various detailed comments on paper layout.) \nLine 227: I assume you will include a GitHub link here, not put the code in the Supplementary material (whatever that means). \nGrammar and style could be improved.**\n\nThank you very much for careful reading and all detailed suggestions that can improve the paper. We will refine the paper as the reviewer suggested. Regarding GitHub link: Because of the blind reviewing, we were not able to include the GitHub link at this point, but we will do it after the review phase. \n\n>**2. Although the choice of Laplace prior is adequately discussed and justified in Section 2, the explanation of the mean-field assumption in equation (1) could benefit from further elaboration. While I understand that this simplification is made for computational tractability, it would be valuable to provide additional justification for this decision. \nAny additional performance improvements by attempting to model correlations between different model parameters. \nAbout their choice of the approximate inference scheme (SGLD). Did the authors consider any other approaches (e.g., variational inference)?**\n\nThe main reason for mean-field assumption (we guess that the reviewer meant *the prior being factorised over individual parameters*), is for simplicity. There might be benefits to correlation modeling/block sparsity, (which is easy to implement in our framework by using a common $\\lambda$ shared across a block), but we did not do this initially as it raises the question of how to choose the block structure. Therefore we leave this to future work for now. \n\nVariational inference can be used in principle, but as it incurs additional complexity (both memory and FLOPS) for handling doubled posterior parameters (means and scales), we opted for SGLD. It may also suffer from lower accuracy due to needing an additional assumption on the posterior form (eg., Gaussian), which may introduce an additional source of approximation inaccuracy. "}, {"review_id": "wPuOBKmvLJ", "reviewer": "Reviewer_ZJiD", "summary": "The paper proposes an approach for selecting a subset of weights in a foundation model to fine-tune on a downstream task. The method consists of a two-stage pipeline, where in the first stage a Laplace prior is placed on each weight with a Gamma hyper-prior on the scale. Samples are obtained via SGLD and only the weights with a mean posterior scale above some threshold are then trained via SGD in the second stage. The method is evaluated on GLUE and SuperGLUE tasks with RoBERTa and on VTAB-1k image prediction tasks with a vision transformer and compares overall favorably to a range of baselines from the literature.\n\nThere are quite a few design choices constituting the proposed method and, unfortunately, none of their added complexity is justified via ablation studies. Further, the paper in my view overstates how principled it is quite significantly, so that at this point I would lean towards rejection.", "questions": "* How do the ablation baselines I mention in weakness 1 perform compared to the method as described in the paper?\n* Is adapting the noise scale and dataset size really needed for stage 1? I’m aware of the cold posterior effect when sampling for a Bayesian model average, however here we don’t seem to need the predictions but the actual parameter samples? How much does tuning these improve performance over the principled choice?\n\n**Typos/minor**:\n* l155: “algorithmm” -> “algorithm”, “pseudocodes” -> “pseudocode”\n* The abstract is extremely long without being particularly descriptive (e.g. the two-stage nature of the method isn’t even mentioned explicitly) and reads more like a small introduction. I’d suggest making it significantly more concise.\n* For me there is way too much going on in Tab 1. I’d suggest at least dropping the bold-facing and rank indicators for the standard deviations, and making the rank indicators gray rather than red (unless you consider them primary information, in which case they should be bigger. But at the moment they are secondary in terms of size and position, so overall the table is unnecessarily difficult to process).", "limitations": "n/a", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The approach is new as far as I am aware.\n* The technical description of the method is clear.\n* Performance seems to be good and approaches for better fine-tuning are of high interest to the community.", "weaknesses": "* The core problem in my view is that the method consists of quite a few moving parts, but these aren’t justified via ablations. It is not clear at all where the performance improvements come from and whether all parts of the method are needed. E.g. it might be the case that the two-stage procedure with magnitude-based pruning would be enough (at least my understanding of MagPruning based on the description in the paper is that the smallest pre-trained values are pruned). Similarly I wonder if sampling in Stage 1 is necessary or if MAP estimates would be good enough for the scale parameters.\n* I don’t really see what makes the proposed method particularly principled as claimed at various points in the paper. I don’t think there is a probabilistic justification for the two-stage procedure and fudging the dataset size and noise scale for SGLD is just a hack.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and suggestions. Our responses are as follows.\n\n>**1. Regarding ablation studies:** \n\nIn response to the reviewer’s request, we have done the ablation studies: 1) Mag-Pruning with two-stage, and 2) MAP instead of SGLD. Please see the table below:\n\nAvg 10 runs | CoLA | STS-B | MRPC | RTE | CB | COPA | WSC | **AVG**\n:---: | :---: | :---: |  :---: |  :---: |  :---: |  :---: | :---: | :---: \n2-stg MagPrune | 53.51 | 88.33 | 88.96 | 70.83 | 80.36 | 67.60 | 58.07 | 72.52\nMAP   | 58.55 | 90.13 | 90.54 | 76.72 | 86.71 | 71.09 | 60.34 | 76.30\nSGLD (Ours) | 60.85 | 90.40 | 90.61 | 77.87 | 91.25 | 75.00 | 60.87 | 78.12\n\nThe results show that two stage mag pruning is not sufficient, and that posterior mean estimation via SGLD provides empirical benefit over MAP optimisation. We will add this ablation study in the revised version. We are grateful to the reviewer for helping improve the paper.\n\n>**2. Doubts about principled method:** \n\nThe reviewer raised three components of the algorithm which s/he felt undermined the claims of a principled solution. We explain these as follows:\n\n  * **Two-stage procedure**: \n  Sparse Bayesian learners express a prior that prefers unmodified weights, but to enforce this as a hard constraint that can practically be used for memory saving, it is standard practice to  threshold after posterior probability inference (e.g. in the seminal Bayesian Compression for Deep Learning, NeurIPS’17; and the recent “Masked Bayesian Neural Networks” mentioned by reviewer **F3ir**). Since there is solid precedent for this standard step to bridge Bayesian models with practical implementation, we do not see it as compromising the principle of our method.\n\n  * **Dataset size inflation**: \nIn Bayesian deep learning which depends on an explicit measure of dataset size, there has been discussion about how to quantify dataset size when using Data Augmentation. While this is not yet a completely solved question, several prior Bayesian deep learning studies have made similar suggestions on inflating the original training data size to account for augmentation (References below). Thus we believe this is entirely reasonable, and disagree that it is a hack: \n\n    - Disentangling the roles of curation, data-augmentation and the prior in the cold posterior effect, L. Noci et al., NeurIPS 2021.\n    - Practical deep learning with Bayesian principles, K.Osawa, et al., NeurIPS 2019. \n    - What are Bayesian neural network posteriors really like?, P. Izmailov et al., ICML 2021. \n\n  * **Noise scale**: \nPurely finding the posterior mean (ie, SGLD without noise discount) risks performing poorly if the posterior is truly multi-modal, because it may converge to a low probability parameter. \nAlso, purely searching for the posterior mode (ie, MAP instead of SGLD) may be sensitive to data noise, because no stochasticity is taken into account properly. So for combining principle and practice it’s reasonable to prefer a discounted noise procedure that balances between identifying a particular mode, but gets a mean estimate in the vicinity of that mode. \n"}, {"review_id": "NM7xSUIRUr", "reviewer": "Reviewer_JDrk", "summary": "The authors introduce BayesTune, a method for choosing which are the parameters to fine-tune in a pre-trained model. Their formulation is based on Bayesian inference where they use a Laplace prior over the parameters/network weights. The prior has two variables: mean, which is the value of the pre-trained parameters, and scale which specifies how important is to fine-tune it. These hyperparameters are also controlled by a hyperprior that is fixed among all the experiments. For inferring the posterior distribution of the weights and scales. they adopt Langevin dynamic method. After obtaining the scale value, they compute a cut-off value that determines the parameters to be updated. Experiments performed in Computer Vision and NLP tasks demonstrate that the method is competitive and outperform common techniques.", "questions": "* Did you optimize the hyperparameters of the other methods?\n* Why table 2 does not have standard deviations?\n* Could you elaborate further on how you choose the cut-off point?\n* How did you choose the hyperprior values ($\\alpha, \\beta$)?\n\n", "limitations": "No limitation was mentioned by the authors.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* The problem of efficient fine-tuning is very important nowadays given the availability of large pre-trained models.\n* The method is intuitive and interpretable, as it is based on the Bayesian interpretation of the network weights.\n* There is a large set of experiments demonstrating the utility of the method.", "weaknesses": "* This model is close to related work such as SP-regularization [1] which uses a regularization term to encourage the updated weights to stay close to the original pre-trained weights. Using an L1-SP regularizer might have a similar effect as the one introduced in this paper. I think it is important the authors mention this and compare BayesTune against it.\n* The efficiency of the method is not clear to me. For instance, in an attention layer, updating only some parameters (sparse updates) still demands computation of the matrix multiplications for query, key, and value matrices. The same logic applies when backpropagating. Thus, choosing only some parameters to update might only save memory demands, but not time. If my reasoning is correct, then the authors should specify this in the paper. Also, some measures of time and memory consumption might be useful. \n* There is a discrepancy between Algorithm 1 and the method described in the paragraph starting at line 206. Thus, I think Algorithm 1 is incomplete.\n* Their method seems competitive for NLP but not so for Computer Vision, based on Table 2.\n\n[1] Xuhong, L. I., Yves Grandvalet, and Franck Davoine. \"Explicit inductive bias for transfer learning with convolutional networks.\" International Conference on Machine Learning. PMLR, 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**1. This model is close to related work such as SP-regularization [1] which uses a regularization term to encourage the updated weights to stay close to the original pre-trained weights. Using an L1-SP regularizer might have a similar effect as the one introduced in this paper. I think it is important the authors mention this and compare BayesTune against it.**\n\nThanks for the citation and good question. We will discuss this paper in the revision. We agree that SP [1] provides an alternative regularisation-based approach to fine-tuning (but without the principled Bayesian modeling solution), and that if SP [1] is extended to a L1 regulariser it might provide an alternative approach to sparse fine-tuning. Thus we have conducted some extra comparison experiments.\n\nNote that despite using an L1 regulariser, SP does not necessarily lead to exactly sparse solutions. Therefore we offer a two-stage extension of L1-SP. in the first stage, we run the L1-SP training, and in the second stage those weights to be updated are selected based on the relative L1 distances from the pretrained weights (ie, taking those $p$% weights with the largest relative changes from the pretrained weights). \n\nThe results on NLP tasks are as follows:\n\nAvg 10 runs | CoLA | STS-B | MRPC | RTE | CB | COPA | WSC | **AVG**\n:---: | :---: | :---: |  :---: |  :---: |  :---: |  :---: | :---: | :---: \nL1-SP (stg 1) | 50.50 | 88.07 | 84.54 | 50.00 | 62.44 | 60.40 | 52.88 | 64.12\nL1-SP (stg-2, $p$%) | 54.59 | 88.11 | 89.25 | 68.85 | 81.55 | 70.75 | 55.38 | 72.64\nOur SGLD ($p$%) | 60.85 | 90.40 | 90.61 | 77.87 | 91.25 | 75.00 | 60.87 | 78.12\n\nFor the L1 penalty balancing constant hyperparameter, we choose optimal values by grid search from {$10^{-3},10^{-4},10^{-5},10^{-6}$}. We can see that L1-SP considerably lags behind our SGLD, which is mainly attributed to its failure to capture uncertainty in L1 regularisation, thus being potentially sensitive to noise in data (a similar reason as the MAP estimate). Moreover, only penalising the parameters deviation from the pre-trained weights as in Stage 1 significantly underperforms the sparse cut-off strategy in Stage 2, signifying that sparse update is critical. \n\n\n>**2. The efficiency of the method is not clear to me. For instance, in an attention layer, updating only some parameters (sparse updates) still demands computation of the matrix multiplications for query, key, and value matrices. The same logic applies when backpropagating. Thus, choosing only some parameters to update might only save memory demands, but not time. If my reasoning is correct, then the authors should specify this in the paper. Also, some measures of time and memory consumption might be useful.**\n\nThank you for the insightful comment. Yes, we agree on the overall backpropagation timing overhead in attention layers. We emphasise that our selective sparse fine-tuning competitors are also edge-wise (eg. SAM, DiffPrune, MagPrune), and thus our method is the same as theirs in this regard. We will clarify this in our revised paper. \n\nIf saving latency/computation is important, our algorithm can easily be modified to provide this by sharing $\\lambda$ across blocks or layers. EG: A layer-wise sparse $\\lambda$ could avoid the need for attention computation in a certain layer. We actually explored this in our preliminary studies, but didn’t ultimately go down this path in the paper. Obtaining improved computation efficiency in this way would typically lead to slightly worse accuracy than the current edge-wise sparsity assumption. But we emphasise that this would be the same for all competitors, if they were correspondingly modified for layer/block-wise sparsity. \n\n>**3. There is a discrepancy between Algorithm 1 and the method described in the paragraph starting at line 206. Thus, I think Algorithm 1 is incomplete.**\n\nWe left out the modifications from Alg.1 to make it concise, but we will add them to Alg.1. \n\n>**4. The method seems competitive for NLP but not so for Computer Vision, based on Table 2.**\n\nYes. As we commented in the text Line 308-310, the main benefit/message is to avoid complex heuristic (evolutionary) search used by the state of the art competitor NOAH.\n\n>**5. Did you optimize the hyperparameters of the other methods?**\n\nWe did not do it for ourselves as we excerpted the results from the respective previous papers. But each of the competing methods reports its best result after some hyperparameter tuning.\n\n>**6. Why Table 2 does not have standard deviations?**\n\nBecause of the high cost of running the VTAB tasks, it is difficult to have many runs. We rather follow the official train/val/test split. Reporting the point estimates is also common practice for this benchmark. \n\n>**7. Could you elaborate further on how you choose the cut-off point?**\n\nFor the NLP, we have a strict user-specified cut-off point $p=0.005$, being fair with other competing methods. For the VTAB, we have different cut-off points across tasks, and the optimal one is chosen as per the performance on the validation sets. \n\n>**8. How did you choose the hyperprior values ($\\alpha$, $\\beta$)?**\n\nIt is just a heuristic choice."}]

Author rebuttal:

Meta-review: This paper introduces an interesting algorithm for fine-tuning foundation models, namely a sparse Bayesian fine-tuning method employing hierarchical Laplace priors. The proposed method has received positive feedback from most reviewers, who acknowledge its novelty and potential practical impact. However, a notable concern revolves around the implementation details of the SGLD procedure. Specifically, the authors have implemented SGLD as Adam with noise, which raises questions about whether this truly corresponds to a valid adaptive version of SGLD. This aspect, in turn, prompts doubts about the paper's claim that the proposed algorithm constitutes a "Bayes" Tuning method. Reviewers encourage the authors to address this issue in the final version of the manuscript.
Decision: Accept (poster)