SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison to L1 regulariser. The recent L1-regularised sparse model learning (L1-SP) [42] used the L1 regularisation term to encourage the updated weights to stay close to the original pre-trained weights. Thus this model might have a similar effect as our BayesTune. The L1-SP can provide an alternative regularisation-based approach to fine-tuning, however, is designed without the principled Bayesian modeling solution, which is the main difference from the proposed BayesTune. We have conducted some extra comparison experiments between the two approaches. Note that L1-SP does not necessarily lead to exactly sparse solutions. Therefore we offer a two-stage extension of L1-SP in the first stage, we run the L1-SP training, and in the second stage those weights to be updated are selected based on the relative L1 distances from the pre-trained weights (i.e., taking those
Table 3: MAP vs. SGLD results on NLP benchmarks. Averaged over 10 random runs.
Method	CoLA	STS-B	MRPC	RTE	CB	COPA	WSC	AVG
L1-SP (stage I)	50.50	88.07	84.54	50.00	62.44	60.40	52.88	64.12
L1-SP (stage II, (100 × p)%):	54.59	88.11	89.25	68.85	81.55	70.75	55.38	72.64
BayesTune (stage II, (100 × p)%):	60.85	90.40	90.61	77.87	91.25	75.00	60.87	78.12
Table 4: L1-SP [42] vs. BayesTune results on NLP benchmarks. Averaged over 10 random runs.
$(100 \times p)\%$  weights with the largest relative changes from the pre-trained weights). The results on the NLP tasks are shown in Table 4.
For the L1 penalty balancing constant hyperparameter, we choose optimal values by grid search from  $\{10^{-3}, 10^{-4}, 10^{-5}, 10^{-6}\}$ . We can see that L1-SP considerably lags behind our BayesTune, which is mainly attributed to its failure to capture uncertainty in L1 regularisation, thus being potentially sensitive to noise in data (a similar reason as the MAP estimate). Moreover, only penalising the parameters deviation from the pre-trained weights as in Stage 1 significantly underperforms the sparse cut-off strategy in Stage 2, signifying that sparse update is critical.
D Layer-wise and Module-wise Sparsity Patterns of BayesTuned Networks
Sparsity patterns of RoBERTa-base on NLP tasks. We visualize the module-wise and layer-wise sparsity patterns of the BayesTuned RoBERTa-base networks on 7 NLP tasks in Fig. 6-19. First, for the layer-wise sparsity pattern: (Except for mrpc) The proportions of the selected updatable parameters are more or less uniformly distributed across the 12 layers of the Transformer, while the first word embedding layer and the last classification layer are significantly less and more selected, respectively. This is intuitively appealing as the task-specific features may tend to be determined at the higher, more global levels in texts/sentences, to account for longer-range dependency. Next, looking at the module-wise sparsity patterns, the proportions are highly non-uniform, layer-specific, and also task/dataset-dependent. For instance, the bias modules in some layers are very densely selected, while they are very sparsely selected in other layers. This shows clear discrepancy to the heuristic strategies like BitFit [43] in which the bias modules are selected  $100\%$  for all layers.
Sparsity patterns of ViT-B/16 on VTAB vision tasks. The module-wise and layer-wise sparsity patterns of the BayesTuned ViT-B/16 networks on VTAB benchmark datasets are shown in Fig. 20-38. We also superimpose the optimal  $p$  values (dataset dependent). The resulting patterns are quite similar to the NLP case: Except for a few cases, the lowest level visual prompt layers are selected far less, sometimes ignored, compared to the later layers. The last linear classification head, although not shown here in the sparsity diagrams, is selected  $100\%$ . Overall the layer-wise selection patterns are nearly uniform while the module-wise selection patterns are highly non-uniform and dataset dependent.
cola (Module-wise sparsity pattern)
Figure 6: Sparsity pattern of the modules in RoBERTa-base on cola. Module-wise updatable parameters (\%). (Left) The first half of the network and (Right) the second half. The default sparsity level  $p^* = 0.5\%$  is shown as vertical line.
cola (Layer-wise sparsity pattern)
Figure 7: Sparsity pattern of the layers in RoBERTa-base on cola. Layer-wise updatable parameters  $(\%)$ . (Left) The first half of the network and (Right) the second half. The default sparsity level  $p^* = 0.5\%$  is shown as vertical line.
stsb (Module-wise sparsity pattern)
Figure 8: Sparsity pattern of the modules in RoBERTa-base on stsb. Module-wise updatable parameters (\%). (Left) The first half of the network and (Right) the second half. The default sparsity level  $p^* = 0.5\%$  is shown as vertical line.
stsb (Layer-wise sparsity pattern)
Figure 9: Sparsity pattern of the layers in RoBERTa-base on stsb. Layer-wise updatable parameters  $(\%)$ . (Left) The first half of the network and (Right) the second half. The default sparsity level  $p^* = 0.5\%$  is shown as vertical line.
mrpc (Module-wise sparsity pattern)
Figure 10: Sparsity pattern of the modules in RoBERTa-base on mrpc. Module-wise updatable parameters (\%). (Left) The first half of the network and (Right) the second half. The default sparsity level  $p^* = 0.5\%$  is shown as vertical line.
mrpc (Layer-wise sparsity pattern)
Figure 11: Sparsity pattern of the layers in RoBERTa-base on mrpc. Layer-wise updatable parameters  $(\%)$ . (Left) The first half of the network and (Right) the second half. The default sparsity level  $p^* = 0.5\%$  is shown as vertical line.
rte (Module-wise sparsity pattern)
Figure 12: Sparsity pattern of the modules in RoBERTa-base on rte. Module-wise updatable parameters (\%). (Left) The first half of the network and (Right) the second half. The default sparsity level  $p^* = 0.5\%$  is shown as vertical line.
rte (Layer-wise sparsity pattern)
Figure 13: Sparsity pattern of the layers in RoBERTa-base on rte. Layer-wise updatable parameters  $(\%)$ . (Left) The first half of the network and (Right) the second half. The default sparsity level  $p^* = 0.5\%$  is shown as vertical line.
cb (Module-wise sparsity pattern)
Figure 14: Sparsity pattern of the modules in RoBERTa-base on cb. Module-wise updatable parameters (\%). (Left) The first half of the network and (Right) the second half. The default sparsity level  $p^* = 0.5\%$  is shown as vertical line.
cb (Layer-wise sparsity pattern)
Figure 15: Sparsity pattern of the layers in RoBERTa-base on cb. Layer-wise updatable parameters  $(\%)$ . (Left) The first half of the network and (Right) the second half. The default sparsity level  $p^* = 0.5\%$  is shown as vertical line.
copa (Module-wise sparsity pattern)
Figure 16: Sparsity pattern of the modules in RoBERTa-base on copa. Module-wise updatable parameters (\%). (Left) The first half of the network and (Right) the second half. The default sparsity level  $p^* = 0.5\%$  is shown as vertical line.
copa (Layer-wise sparsity pattern)
Figure 17: Sparsity pattern of the layers in RoBERTa-base on copa. Layer-wise updatable parameters  $(\%)$ . (Left) The first half of the network and (Right) the second half. The default sparsity level  $p^* = 0.5\%$  is shown as vertical line.
wsc (Module-wise sparsity pattern)
Figure 18: Sparsity pattern of the modules in RoBERTa-base on wsc. Module-wise updatable parameters  $(\%)$ . (Left) The first half of the network and (Right) the second half. The default sparsity level  $p^* = 0.5\%$  is shown as vertical line.
wsc (Layer-wise sparsity pattern)
Figure 19: Sparsity pattern of the layers in RoBERTa-base on wsc. Layer-wise updatable parameters  $(\%)$ . (Left) The first half of the network and (Right) the second half. The default sparsity level  $p^* = 0.5\%$  is shown as vertical line.
Figure 20: Sparsity pattern of attached modules to ViT-B/16 on CIFar100. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 10\%$  is shown as vertical line.
Figure 21: Sparsity pattern of attached modules to ViT-B/16 on caltech101. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 20\%$  is shown as vertical line.
caltech101
Figure 22: Sparsity pattern of attached modules to ViT-B/16 on dtd. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 5\%$  is shown as vertical line.
flower102
Figure 23: Sparsity pattern of attached modules to ViT-B/16 on flower102. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 50\%$  is shown as vertical line.
Figure 24: Sparsity pattern of attached modules to ViT-B/16 on pets. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 20\%$  is shown as vertical line.
Figure 25: Sparsity pattern of attached modules to ViT-B/16 on svhn. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 80\%$  is shown as vertical line.
Figure 26: Sparsity pattern of attached modules to ViT-B/16 on sun397. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 5\%$  is shown as vertical line.
sun397
Figure 27: Sparsity pattern of attached modules to ViT-B/16 on camelyon. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 80\%$  is shown as vertical line.
camelyon
eurosat
Figure 28: Sparsity pattern of attached modules to ViT-B/16 on eurosat. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 80\%$  is shown as vertical line.
Figure 29: Sparsity pattern of attached modules to ViT-B/16 on resisc45. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 50\%$  is shown as vertical line.
resisc45
retinopathy
Figure 30: Sparsity pattern of attached modules to ViT-B/16 on retinopathy. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 10\%$  is shown as vertical line.
clevr-count
Figure 31: Sparsity pattern of attached modules to ViT-B/16 on clevr-count. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 40\%$  is shown as vertical line.
clevr-dist
Figure 32: Sparsity pattern of attached modules to ViT-B/16 on clevr-dist. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 30\%$  is shown as vertical line.
Figure 33: Sparsity pattern of attached modules to ViT-B/16 on dm1ab. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 80\%$  is shown as vertical line.
dmlab
Figure 34: Sparsity pattern of attached modules to ViT-B/16 on kitti. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 80\%$  is shown as vertical line.
dsprite-loc
Figure 35: Sparsity pattern of attached modules to ViT-B/16 on dsprit-loc. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 90\%$  is shown as vertical line.
dsprite-ori
Figure 36: Sparsity pattern of attached modules to ViT-B/16 on dsprit-ori. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 70\%$  is shown as vertical line.
snorb-azim
Figure 37: Sparsity pattern of attached modules to ViT-B/16 on snorb-azim. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 70\%$  is shown as vertical line.
snorb-ele
Figure 38: Sparsity pattern of attached modules to ViT-B/16 on snorb-ele. (Left) Module-wise updatable parameters (\%) and (Right) Layer-wise updatable parameters (\%). The BayesTune's optimal sparsity level  $p^* = 90\%$  is shown as vertical line.