SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4.2 Relation to Existing (Hierarchical) Bayesian Sparse Model Learning
(Hierarchical) Bayesian models are well known in statistical machine learning, and there were some prior works that adopt the Bayesian approaches for sparse deep learning [21, 32, 35, 29]. However, as far as we know, our approach has several key differences from these previous works.
First, most prior works are all about sparse training, instead of sparse fine-tuning. Thus they focus on zeroing out many parameters, instead of retaining the pre-trained weights. To the best of our knowledge, our BayesTune is the first to tackle the sparse fine-tuning at the Foundation Model scale using the hierarchical Bayesian framework.
More importantly, the neural networks used in those previous studies are rather small/toy scale (mostly focusing on MLPs and LeNet-sized architectures up to ResNet-18 at the largest) while our method can obtain the state-of-the-art results on large-scale foundation models (ViT, RoBERTa). For example the largest model considered in those prior works, ResNet-18 contains  $\sim 11\mathrm{M}$  parameters vs. RoBERTa's  $\sim 123\mathrm{M}$  parameters, a  $10\times$  scale difference. The reason is that they adopt methods that entail extra memory cost like variational inference (VI), which impedes applicability to big networks. We have also done some experiments that compare the computational resources required by VI and SGLD: On ViT networks, the training time is increased by 1.7 times if we replace SGLD by VI; the GPU memory footprint is increased by 2.1 times.
We test our BayesTune on two popular benchmark datasets from NLP and vision for the downstream fine-tuning tasks: (language) fine-tuning the pre-trained RoBERTa-base model [28] on the GLUE [37] and SuperGLUE [38] tasks; (vision) fine-tuning the ImageNet-22K [6] pre-trained ViT-B/16 model [9] on VTAB-1K [44] image classification/prediction tasks. The details of the experimental settings are discussed in the subsequent sections, Sec. 5.2 and Sec. 5.3.
5.1 Implementation of BayesTune
For the SGLD sampling in Stage-1 of our BayesTune algorithm (re: Alg. 1), we have three different training regimes: the warm-up phase that only runs vanilla SGD steps without considering the scale
Method	CoLA	STS-B	MRPC	RTE	CB	COPA	WSC	AVG
Full update	58.361.74	89.800.52	89.550.81[2]	76.032.14	88.932.37[3]	67.704.41	53.106.18	74.782.60
Random	58.351.05[3]	89.810.11[1]	88.730.80	72.713.23	90.543.39[2]	68.802.64	52.885.97	74.552.46
MixOut	58.661.96	90.150.17[4]	88.690.60[4]	77.551.64[2]	86.514.13	71.304.84	52.986.78	75.122.88[4]
Bitfit	56.671.45	90.120.14[3]	87.350.58[3]	72.742.47	86.963.20	71.203.79	55.105.39	74.312.43
MagPruning	56.572.47	90.300.14[3]	88.090.79	73.531.84[4]	81.253.50	71.502.46[2]	55.672.73[2]	73.851.99[3]
Adapter	62.111.22[4]	90.050.13[2]	89.290.60[4]	76.932.05[4]	87.324.62	69.502.54[4]	57.025.27[3]	76.032.35[3]
LoRA	60.881.48[3]	87.190.51	89.530.62[3]	76.971.92[3]	84.643.76	69.702.83	56.844.52[4]	75.112.24[4]
DiffPruning	58.531.49	89.590.34	78.796.09	69.937.87	86.252.65[4]	72.102.91[3]	53.373.60[4]	72.653.57
ChildPruning	60.001.29	89.971.51	87.193.86	75.764.38	86.613.22	69.404.00	55.593.81	74.933.15
SAM	60.890.96[2]	90.590.14[3]	88.840.49[1]	76.791.72[3]	88.931.75[2]	74.302.45[1]	59.523.08[3]	77.121.51[2]
BayesTune	60.850.47[1]	90.400.14[3]	90.610.56[2]	77.870.64[1]	91.251.25[1]	75.002.49[3]	60.872.62[1]	78.121.17[1]
Table 1: Results on NLP benchmarks. For each dataset/task (column), the average accuracy and the standard deviation (in superscript) shown over 10 runs with different random seeds are reported. The ranks (up to the fourth) among the competing methods are also shown in the brackets and in red. The figures of the competing methods are excerpted from [10].
variables  $\lambda$  and random noise; followed by the burn-in phase where the SGLD steps are performed as (5) but no  $(\theta, \lambda)$  samples are collected; followed by the normal phase in which we do collect samples. Following the conventional practice, we also perform the thinning steps [27] (i.e., collecting samples at a certain frequency) to mitigate undesired temporal correlation effects.
Another important implementation tip is the use of reweighed cost terms in Langevin dynamics - As shown in (5), the cost function is involved with the training data size  $N = |D|$ . Even though  $N$  is known, the data augmentation would considerably increase the effective data size by several orders of magnitudes (e.g., if there are 5 different augmentation strategies and each allows 10 different variations, the effective data size can be  $10^{5} \times N$ ). To account for it, we regard the effective data size as a hyperparameter denoted by  $\hat{N}$  that can be chosen from validation<sup>4</sup>. Replacing  $|D|$  by  $\hat{N}$  and taking it out of the parentheses in (5), we have the gradient (normalized by  $\hat{N}$ ) multiplied by the step size  $(\eta / 2)\hat{N}$ . First we choose  $\eta$  to match the learning rate in the neural net optimizer (e.g., Adam<sup>5</sup> [20]), that is,  $\eta = lr \times 2 / \hat{N}$ . Accordingly the noise term becomes  $\epsilon \cdot (2 / (\hat{N} \times lr))^{1/2}$ . We also find it effective oftentimes to discount the noise effect: we introduce the noise discount factor  $\gamma$  (another hyperparameter) by which the noise term is multiplied. We validate  $\gamma$  in the range  $10^{-4:0}$  (e.g.,  $\gamma = 1$  corresponds to the default no-discount case).
Throughout all experiments we use the Gamma prior parameters  $(\alpha = 0.01, \beta = 100)$ , the scale variables  $\lambda_i$ 's are all initialized to 0.0001, and the learning rate for  $\lambda$  is 0.01 without scheduling. Other task-specific implementation details can be found in the subsequent sections. The Python/PyTorch code to reproduce the results is available at https://github.com/SamsungLabs/BayesTune<sup>6</sup>.
5.2 NLP Benchmarks
We consider fine-tuning the pre-trained RoBERTa-base model [28], the large-scale foundation language model comprised of 125 million parameters, on several downstream tasks in GLUE [37] and SuperGLUE [38] benchmarks. We follow the experimental settings from [10], in which the original development sets serve as test sets, and the validation sets are formed by holding out random  $10\%$  of
4For instance, in the VTAB-1K, although  $N = 1000$  or 800, we let  $\hat{N}$  have a range  $10^{7:12}$
We used the Adam optimiser for updating the model parameters, thus there may exist some internal gradient adaptation and momentum effect under the hood in our SGLD steps. This effect may be related to that of the adaptive drift and momentum in SGLD that were analysed in some previous works, e.g., [19]. However, we believe that our SGLD update scheme with Adam would not result in significantly different solution compared to the original SGLD formulation at least in practice.
6Alternatively, https://github.com/minyoungkim21/BayesTune
Figure 2: (NLP benchmarks) Posterior mean  $\hat{\lambda}$  vs. sparsity level  $p$ . The default sparsity level  $p = 0.005$  is depicted as red dotted vertical lines. For each dataset, we plot the original  $p \in [0,1]$  range (lower-left), which is magnified for the smaller range  $p \in [0,0.1]$  (upper-right).
the training sets. Instead of having a fixed number of training epochs, the development sets are used for an early stopping with the no-improvement tolerance of 40 epochs. We implement our BayesTune on top of the codebase of [10] which is based on the jiant framework (https://jiant.info/). Similarly as [10], we consider 7 tasks: Corpus of Linguistic Acceptability (CoLA) [39], Semantic Textual Similarity Benchmark (STS-B) [2], Microsoft Research Paraphrase Corpus (MRPC) [8], Recognizing Textual Entailment (RTE) [5, 11, 1], Commitment Bank (CB) [30], Choice of Plausible Alternatives (COPA) [36], and Winograd Schema Challenge (WSC) [25].
Our BayesTune is compared against strong baselines and the state-of-the-arts, including: Random, Mixout [23], BitFit [43], MagPruning [24, 22], Adapter [15], LoRA [16], DiffPruning, ChildPruning [41, 31], and SAM [10], as discussed in Sec. 4. For all methods, we use the sparsity level  $p = 0.005$  as suggested in [13]. For the competing methods, we follow the hyperparameter settings from [10]; In BayeTune, we use  $10K$  warm-up steps,  $2K$  burn-in steps, and thinning at every 100 steps for all tasks. The batch size is 16, and the learning rate for the model parameters is  $10^{-4}$  for Stage-1 and  $10^{-3}$  for Stage-2. For the (dataset-dependent) evaluation metrics, we adopt the protocols in [37, 38]. We run all models on a single NVIDIA V100 GPU with 32GB memory. The results are summarized in Table 1 where means and standard deviations over 10 random seeds are reported.
First, it is quite surprising that the random selection strategies (Random and Mixout) are as effective as many sparse fine-tuning methods. This may be attributed to the massive over-parametrization of the deep foundation model. However, more careful and sophisticated selection is important as those three methods, our BayesTune, SAM, and Adapter, outperform the random strategies by large margin. Overall our BayesTune performs the best among the competing methods, about  $1\%$  point higher on average than the runner-up SAM [10]. We achieve the first place on 5 out of 7 tasks, and also attain the smallest standard deviations, implying that the resulting sparse fine-tuned models are more robust and reliable. This result signifies the effectiveness of our sparse Laplace scale-based parameter selection strategy that is underpinned by the principled Bayesian posterior inference with efficient<sup>7</sup> stochastic gradient Langevin sampling.
In Fig. 2, we also show the posterior mean scale  $\hat{\lambda}$  vs. sparsity level  $p$ . The cut-off threshold points corresponding to the sparsity level  $p = 0.005$  are around  $\lambda_{th} \in [0.25, 0.30]$  for all tasks except for MRPC. This cut-off scale range might be considered as reasonable choice as most of the parameters would have Laplace scales smaller than this range, exhibiting no pronounced indication of crucial deviation from the pre-trained values, although this is only conjecture based on our empirical results. Further rigorous theoretical analysis on the relation between the scale threshold and the generalization performance is beyond the scope of the current paper, and is left as future work.
5.3 Vision Benchmarks
Next we test our BayesTune on the VTAB-1K [44], an image classification/prediction task adaptation benchmark suite comprised of 19 different image datasets. It contains images/tasks that exhibit
The SGLD steps (5) require at most only two times the complexity of the vanilla SGD steps.
Method	#param (M)	Cifar100	Caltech101	DTD	Flower102	Pets	SVHN	Sun397	Camelyon	EuroSAT	Resisc45	Retinopathy	Clevy-Count	Clevy-Dist	DMLab	KITTI	dSpr-Loc	dSpr-Ori	sNORB-Azim	sNORB-Ele	Avg Rank (-) # Rank (+)
Full update	85.8	68.9	87.7	64.3	87.2	86.9	87.4	38.8	79.7	95.7	84.2	73.9	56.3	58.6	41.7	65.5	57.5	46.7	25.7	29.1
Linear	0.04	64.4	85.0	63.2	97.0	86.3	36.6	51.0	78.5	87.5	68.5	74.0	34.3	30.6	33.2	55.4	12.5	20.0	9.6	19.2
VPT [18]	0.64	78.8	90.8	65.8	98.0	88.3	78.1	49.6	81.8	96.1	83.4	68.4	68.5	60.0	46.5	72.8	73.6	47.9	32.9	37.8	4.16	3
Adapter [15]	0.16	69.2	90.1	68.0	98.8	89.9	82.8	54.3	84.0	94.9	81.9	75.5	80.9	65.3	48.6	78.3	74.8	48.5	29.9	41.6	3.68	1
LoRA [16]	0.29	67.1	91.4	69.4	98.8	90.4	85.3	54.0	84.9	95.3	84.4	73.6	82.9	69.2	49.8	78.5	75.7	47.1	31.0	44.0	2.68	4
NOAH [45]	0.43	69.6	92.7	70.2	99.1	90.4	86.1	53.7	84.4	95.4	83.9	75.8	82.8	68.9	49.9	81.7	81.8	48.3	32.8	44.2	1.95	5
BayesTune	Avg	68.9	92.6	69.5	99.1	90.8	88.1	50.0	84.6	95.8	82.8	76.0	82.6	67.4	49.6	82.3	81.9	49.9	22.6	39.3	2.37	7
0.37	(.07)	(.37)	(.04)	(.37)	(.15)	(.67)	(.04)	(.60)	(.60)	(.37)	(.07)	(.30)	(.22)	(.52)	(.60)	(.60)	(.30)	(.52)	(.67)
Table 2: VTAB-1K results. The accuracies at the optimal sparsity levels are reported for our BayesTune. For BayesTune, the optimal number of the updated parameters is dataset-dependent, and these optimal numbers are depicted in the parentheses. The figures of the competing methods are excerpted from [18, 15, 16, 45].
highly diverse aspects/conditions such as: different image acquisition (by standard cameras or special-purpose ones for remote sensing or medical imaging), different objects/concepts (generic, fine-grained, or abstract), and tasks (object recognition, counting, or pose estimation). Each dataset in VTAB-1K consists of  $1K$  training examples, and we use the splits officially provided (train  $80\%$  and validation  $20\%$ ).
We aim to fine-tune the ImageNet-22K [6] pre-trained ViT-B/16 model [9] on each dataset. We implement our BayesTune using the codebase from [45] while employing most of their hyperparameters without changes (e.g., the number of training epochs 100 and batch size 64). In our BayesTune, we use 50 warm-up epochs, 10 burn-in epochs, and thinning at every 5 batch steps for all datasets. The learning rate for the model parameters is  $10^{-3}$  and weight decay  $10^{-3}$  for both stages.
We first attempted the proposed method with our original setting, i.e., place the  $\lambda$  variables to the backbone (ViT) parameters, and let the Bayesian inference figure out the most critical parameters to update, namely those with large  $\lambda_{i}$ 's. Unfortunately the result was significantly worse than the adapter-based methods: VPT [18], Adapter [15], LoRA [16], and NOAA [45], for most datasets. As reported in several works in the literature, adapter-based models perform significantly better than the full backbone update for this benchmark (e.g., Table 1 in [45]). Along this line, we conjecture that this trend also holds for other sparse fine-tuning methods, namely the superiority of the adapter-based approaches to sparse backbone update strategies, particularly for this VTAB benchmark.
To fairly take the advantage of extra attachable modules as the adapter-based approaches, we attach extra modules to ViT for all three popular strategies: visual prompts in VPT [18], adapter modules in Adapter [15], and low-rank matrices in LoRA [16]. This design idea is similar to (and motivated from) the neural prompt search in NOAA [45], but they formed a grid search problem to determine which layers to attach the adapter modules and how many module dimensions to attach - solved by the neural architecture search (NAS) technique with the super-net training, specifically adopting Autoformer [3], followed by running Evolutionary algorithms for optimal architecture search.
On the other hand, we add 10-dim full parameters for each module/layer, and they are all initialized to 0. We let the posterior inference in our BayesTune automatically figure out which of these extra modules parameters to be updated away from 0 and which to be frozen at 0. We used the zero-initialization because having all-0 extra modules can be considered to be identical to the original pre-trained ViT without extra modules, thus serving as pre-trained extra modules. We only place  $\lambda$  variables to the extra modules (ViT parameters frozen)<sup>8</sup>. In summary, our setup is similar to NOAA [45] in that we take all three adapter strategies into account, but the main difference is: NOAA does a difficult super-net training and Evolutionary search to determine the optimal module-wise dimensions. On the other hand we do sparse selection of the module parameters to be updated from 0 across all 10-dim full modules.
We initially attempted to place  $\lambda$  to ViT parameters as well, but unfortunately it did not perform well. The reason being probably is that most of the selected parameters for update come from the ViT backbone, essentially reducing to the original sparse backbone update strategy that failed.
(a) Sparse update is better: Sun397, CIFAR100, DTD, and Retinopathy
(b) Dense update is better: DMLab, Camelyon, SVHN, and dSpr-Loc
(c) Optimal in-between: Clevr-Dist, sNORB-Ele, KITTI, and dSpr-Ori
(d) Sparsity Independent: Flower-102, EuroSAT, Caltech101, and Pets
Figure 3: VTAB datasets grouped by the sparse/dense adapter update effects. E.g., the datasets in group-(a) exhibit higher accuracy when the adapter modules are updated more sparsely than densely. In group-(c), neither extreme sparse nor dense updates are optimal, but the sparsity levels in between perform the best. In group-(d), we have no clear correlation between update sparsity and test accuracy. Individual plots show test accuracies vs. sparsity levels, where the rightmost plot in each group shows the mean and  $95\%$  confidence interval of the members within the group in normalized accuracy scale.
The results are summarized in Table 2. We report the performance of the best sparsity levels found from  $p \in \{0.05, 0.1, 0.2, \dots, 0.9, 1.0\}$ . Although the final accuracies of our BayesTune are not always the best, we achieve Rank 1 on 7 out of 19 datasets and the second place on average over all datasets. The performance of BayesTune is comparable to other adapter-based methods, within the same range as those. That is, our main message is that the adapter-based enhanced architecture fine-tuning can be done almost comparably well by the proposed Bayesian method, instead of relying on the heuristic search such as the Evolutionary Search or costly human expert design effort. Compared to NOAA, our BayesTune results in more sparse adapter updates on average (0.38M vs. 0.43M). In Fig. 3 we also visualize the clustering of VTAB-1K datasets according to the sparse/dense adapter update effects: we assign each dataset into either of sparse, dense, and in-between clusters according to the optimal sparse update level  $p$ : small, large, medium, respectively. Such a grouping of the VTAB datasets based on fine-tuning sparsity characteristics has not been explored in the community, and we can easily obtain this finding using our BayesTune.
We introduced BayesTune, a framework for automated fine-tuning that provides a principled yet efficient approach to identifying which parameters to update in sparse fine-tuning by posterior inference of parameter-wise scale in a hierarchical Bayesian model. We demonstrated BayesTune to be effective for parameter efficient fine-tuning in both NLP and vision tasks, using both mask-based and additive search spaces.
[43] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. ACL, 2022.
[44] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867, 2019.
[45] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural Prompt Search. arXiv preprint arXiv:2206.04673, 2022.
A Chosen Hyperparameters
We grid-search hyperparameters on validation, where the two key hyperparameters are: the effective training data size  $\hat{N}$  and the noise discount factor  $\gamma$  (re: Sec. 5.1). The candidate sets are formed as:  $\hat{N} \in \{10^8, 10^9, \dots, 10^{12}\}$ ,  $\gamma \in \{10^{-4}, 10^{-2}, 10^0\}$  for NLP, and  $\hat{N} \in \{10^6, 10^7, \dots, 10^{12}\}$ ,  $\gamma \in \{10^{-4}, 10^{-3}, \dots, 10^0\}$  for VTAB. The chosen hyperparameters are as follows ( $\hat{N}, \gamma$ ): (NLP) cola  $= (11, 10^{-4})$ , stsb  $= (12, 10^{-4})$ , mrpc  $= (12, 10^0)$ , rte  $= (8, 10^{-4})$ , cb  $= (10, 10^{-4})$ , copa  $= (8, 10^{-2})$ , wsc  $= (10, 10^{-4})$ ; (VTAB) cifar  $100 = (7, 10^{-1})$ , caltech  $101 = (9, 10^{-2})$ , dtd  $= (12, 10^0)$ , flower  $102 = (12, 10^{-2})$ , pets  $= (12, 10^0)$ , svhn  $= (10, 10^0)$ , sun397  $= (7, 10^{-1})$ , camelyon  $= (6, 10^0)$ , eurosat  $= (7, 10^{-1})$ , resisc  $45 = (12, 10^{-2})$ , retinopathy  $= (7, 10^{-2})$ , clevr-count  $= (7, 10^{-3})$ , clevr-dist  $= (7, 10^{-3})$ , dmlab  $= (8, 10^0)$ , kitti  $= (7, 10^0)$ , dsprite-loc  $= (12, 10^{-4})$ , dsprite-ori  $= (12, 10^{-3})$ , snorb-azim  $= (7, 10^{-2})$ , snorb-ele  $= (6, 10^{-1})$ .
B More Analysis
(NLP) Test accuracies at other sparsity levels. Although  $p = 0.005$  is recognized as the optimal sparsity level overall for the GLUE and SuperGLUE tasks, we evaluate the test performance of our BayesTune for different sparsity levels:  $p \in \{0.01, 0.05, 0.1, 0.2, 0.3, 0.5\}$ . The average test accuracies are shown in Fig. 4. We see that overall there is less significant change in test performance so long as the sparsity level  $p$  is small enough, and the resulting sparse updates selected by our BayesTune lead to equally good performance as those with the default value. However, increasing  $p$  further (e.g.,  $p = 0.5$ ) considerably degrades the performance, which signifies the importance of sparse fine-tuning to avoid potential overfitting.
(VTAB) Scale posterior mean  $\hat{\lambda}$  vs. sparsity level  $p$ . We visualize the plots that relate the sorted scale posterior means  $\hat{\lambda}$  to the sparsity levels  $p$  in Fig. 5. The plots are aligned with the test accuracy plots analyzed in the main paper. As the plots are grouped along the optimal sparsity values, we see certain trends: for the sparse group (sun397 and cifar100), the scale  $\hat{\lambda}$  values are overall small scaled (in the range of [0, 0.2]) with sharp drops at small  $\hat{\lambda}$ ; for the dense group: (camelyon and dmlab),  $\hat{\lambda}$  scale is even larger (in the range of [0, 0.5]) with relatively smooth decaying at small values; lastly for the in-between group (clever-dist, dspr-ori, kitti, and snorb-ele), we have much narrower  $\hat{\lambda}$  ranges in between 0 and 0.1 except for kitti.
C Some Ablation Study
MAP vs. SGLD. The Maximum-A-Posterior (MAP) aims to find a mode of the posterior distribution  $p(\lambda | D)$ , and might be considered as an alternative solution to our SGLD posterior samples. In particular, MAP might be attained by turning off the noise term in (5), that is, by setting  $\gamma = 0$ . However, MAP can be more sensitive to data noise than the mean of the posterior. We have some empirical comparison between MAP and SGLD on the NLP tasks in Table 3, and it shows that this distinction does lead to empirical benefit. Another benefit of SGLD
Figure 4: (NLP benchmarks) Test accuracies at sparsity levels other than the default  $p = 0.005$ . We evaluate the BayesTune sparse update models with  $p \in \{0.01, 0.05, 0.1, 0.2, 0.3, 0.5\}$ , where the default ones  $p = 0.005$  are shown as red square markers.
(a) Sparse group (cifar100 and sun397)
(b) Dense group (camelyon and dmlab)
(c) In-between group (clever-dist, dspr-ori, kitti, and snorb-ele)
Figure 5: (VTAB benchmarks) The plots of the sorted scale posterior means  $\hat{\lambda}$  vs. sparsity levels  $p$ , each of which is aligned with the corresponding test accuracy plot. The plots are grouped along the optimal sparsity values where each group exhibits similar trends.
Method	CoLA	STS-B	MRPC	RTE	CB	COPA	WSC	AVG
MAP	58.55	90.13	90.54	76.72	86.71	71.09	60.34	76.30
SGLD (Ours)	60.85	90.40	90.61	77.87	91.25	75.00	60.87	78.12
is that we can also exploit the variance of the posterior in parameter selection (e.g., for two parameters with similar posterior mean values, we prefer to select the one with smaller posterior variance). In future algorithms, we can also exploit this idea of variance-based weight pruning.