Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models
Authors: Junjiao Tian
Chengyue Huang
Zsolt Kira
Venue: NeurIPS 2024 poster (2024)
Keywords: Fine-tuning
foundation models
language models
robustness
Primary area: optimization_for_deep_networks
TL;DR: We present a novel regularization technique for fine-tuning large pre-trained models.

Abstract: Modern optimizers such as AdamW, equipped with momentum and adaptive learning rate, are designed to escape local minima and explore the vast parameter space. This exploration is beneficial for finding good loss basins when training from scratch. It is not necessarily ideal when resuming from a powerful foundation model because it can lead to large deviations from the pre-trained initialization and, consequently, worse robustness and generalization. At the same time, strong regularization on all parameters can lead to under-fitting. We hypothesize that selectively regularizing the parameter space is the key to fitting and retraining the pre-trained knowledge. This paper proposes a new weight decay technique, Selective Projection Decay (SPD), that selectively imposes a strong penalty on certain layers while allowing others to change freely. Intuitively, SPD expands and contracts the parameter search space for layers with consistent and inconsistent loss reduction, respectively. Experimentally, when equipped with SPD, Adam consistently provides better in-distribution generalization and out-of-distribution robustness performance on multiple popular vision and language benchmarks.

Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models
Junjiao Tian
Chengyue Huang Georgia Institute of Technology chuang475@gatech. edu
Georgia Institute of Technology j jtien73@gatech.c edu
Zsolt Kira
Georgia Institute of Technology
zkirafgatech. edu
Abstract
Modern optimizers such as AdamW, equipped with momentum and adaptive learn- ing rale, are designed escape local minima and explore the vast parameter space. This exploration is beneficial for finding good loss basins when training from scratch. It is not necessarily ideal when resuming from a powerful foundation model because it can lead to large deviations from the pre-trained initialization and, consequently, worse robustness and generalization. At the same time, strong regularization on all parameters can lead to under-fitting. We hypothesize that se- lectively regularizing the parameter space is the key to fitting and retraining the pre- trained knowledge. This paper proposes new weight decay technique, Selective Projection Decay (SPD), that selectively imposes strong penalty on certain layers while allowing others to change freely. Intuitively, SPD expands and contracts the parameter search space for layers with consistent and inconsistent loss reduc- tion, respectively. Experimentally. when equipped with SPD, Adam consistently provides better in-distribution generalization and out-of-distribution robustness performance multiple popular vision and language benchmarks. Code available at /https://gitteei http:///w.....aaaeaaaaaaaaeeaaar
git
1 Introduction
Modern optimizers, such Adam M. LARS 21, and LAMB 3 usually include momentum and adaptive learning rates. They help optimizers avoid local minima and accelerate learning [4]
[3] to
explore wider parameter spaces. However, hypothesize that this behavior is not always beneficial for fine-tuning from well pre-trained foundation model, especially when fine-tuning a few layers is already sufficient for fitting the target data 66] exploration will lead to large deviation from the initialization and worse robustness [T0] [1] and constraining the deviation can improve model's generalization
g 19. Several prior works have found that unnecessary
in-distribution (ID) data and
robustness to out-of-distribution (OOD) data 2 regularization term the distance between the current and pre-trained models. More recently, TPGM 10] and FTP I propose to learn different hard constraints for each layer. These new works have demonstrated impressive results on benchmarks. However, they are either difficult to tune, specialized to specific settings, or require significant computation and storage overhead. This motivates us ask whether simple few-liner solution exists for this fundamental problem.
13 | For example, L2-SP [3 imposes
In this paper, ID generalization and OOD robustness refer test accuracy on the fine-tuning distribution and robust accuracy on other shifted distributions, respectively.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

We propose re-examining the existing methods and summarizing their findings to find this solution. Starting from the simplest: L2-SP [3]. Specifically. L2-SP adds an L2 regularization term to the original objective function. Formally,
> 2010 00112 2
[()) [(0) +
(1)
where 0 denotes the model parameters, 0g the initialization, _(0) the original objective function, and the hyper-parameter for regularization strength. When 0g = 0, L2-SP reduces to an ordinary weight decay. This simple method should be effective enough to constrain the model, as our experiments show it can reduce the deviation between the fine-tuned and pre-trained models (Sec. 4.I). However, it is held back by an important design choice: the penalty is always applied to all model parameters. Our empirical results identify that a large prevents every layer from deviating too much and leads to poor fitting, while small À cannot provide enough regularization. This significantly limits the otherwise effective design (Sec. 4.1). So, what is missing in this algorithm? ?
Recent works in robust fine-tuning and parameter-effiieen fine-tuning (PEFT) have shown that customizing constraints for each layer and selectively choosing layers for fine-tunning can improve robustness [10 I 6]. Inspired by these findings, we hypothesize that selectively imposing the regularization to different layers is the key. Therefore, we propose a simple selective version of L2-SP weight decay: selective projection decay (SPD). This new algorithm innovates in two aspects: selection condition and a regularization strength ratio. The former determines when to apply regularization to layer, and the latter determines the strength of regularization for intuitive hyper- parameter tning. Specifically, we derive the selection condition from hyper-optimization [5] 16 17
by treating the condition as an optimizable parameter (Sec. 3.3 and the regularization strength ratio by re-writing I.2-SP as a projection operation (Sec. 3.4). Intuitively, when the condition met, the algorithm imposes large regularizations on selected layers. This allows the algorithm to avoid unnecessary deviation and simultancously fit into the fine-tuning data, We test SPD on large-scale computer vision, and NLP benchmarks with popular foundation models and test ID and OOD performance on various distribution and domain shifts. SPD achieves SOTA performance while being much simpler than other competing methods. Our contributions are:
We propose a selective projection decay, a selective variant of the popular L2-SP/weight decay regularization methods, for robust fine-tuning of large foundation models. We show that selectivity is important to make regularization effective.
We conduct detailed study of ID and OOD performance image classification and semantic segmentation with natural distribution and domain shifts. SPD improves ID and OOD performance these benchmarks.
We show that SPD consistently improves the performance of PEFT methods (e.g. LoRA 7 and adapters g on common sense reasoning language tasks with LLaMA-7B (-13B).
2 Related Works
Robust Fine- TuninT with Distance Regularization. Constraining the distance or deviation between the tnnettneed and pre-trained models has been studied in several prior works. L2-SP [13] explicitly adds an L. norm penalty on the deviation and shows improved ID generalization for fine-tuning. MARS-SP [12] studies different forms of norms the penalty. It shows that the Matrix Row Sum (MARS) norm can be a superior alternative to the L.2 norm. These two methods impose "soft" penalties and can be less effective [18] Instead, LCC [T proposes constraining the deviation through direct projection on the parameters, which also enforces a hard constraint on the Lipschitz continuity of the fine-tunce model. However, LCC is hard tune because the projection radius is not an intuitive hyper- parameerre.rrr.. Furthermore, using a single projection constraint for all layers is not an ideal strategy [TO]. More recently, TPGM [T0] proposes automatically learn the constraints in LCC during fine-tuning, customizing a different projection radius for each layer through bi- level optimization scheme. FTP further improves the computation efficiency of TPGM by in its computation. Nevertheless, FTP is still difficult to control because hyper-optimizaation requires a secondary optimizer with additional optimization hyper-parameters, and the learned regularization can be too strong with no intuitive way to adjust. In contrast, SPD is a much simpler and more intuitive method, which can be implemented with just a few lines of code. The superior controllability makes SPD potentially applicable to more applications.
adopting hyper-optimization [[5]
16 [7

---

Parameter Efficient Fine-Tuning (PEFT), PEFT methods such as adapters 9 g and LoRA 7 have been proposed reduce training memory usage and computation complexity. Recent works have found that PEFT methods also provide good robustness because they modify fewer parameters and retain more knowledge of the pre-trained models [1]. Surgical fine-tuning [6] concludes that fine-tuning selective few layers can improve ID) generalization. These new works motivate us to re-evaluate L2-SP and weight decay, often uniformly applied to all layers. We identify that the inferior performance of the simple methods is because of this uniformity, which exhibits strong trade-off between fitting and regularization. Other robust fine-tuning methods, such as LP-FT [9] and FLYP (2)], focus feature distortion. We will review them in the Appendix 8.1
Other Robust Fine-Tuning Methods. WiSE-FT [14] discovers that linearly interpolating between the fine-tuned and pre-trained models after fine-tuning can improve out-of- -itto--distriuttion robustness. This demonstrates that closer distance the pre-trained model can improve robustness. However, only applies to models with zero-shot capabilities. Another orthogonal line of research for robust fine- tuning focuses on feature distortion. LP-FT 19] shows that fine-tuning with randomly initialized head layer distorts learned features. It proposes simple two-stage method to train the head layer first and then fine-tune the entire model. FLYP (20) shows that fine-tuning a foundation model using the same objective as pre-training can better preserve the learned features. Our contribution an optimization method to penalize the derivation between the fine-tuned and pre-trained models explicitly during fine-tuning, which is orthogonal to them.
Methods
In this section, we first provide an overview the Selective Projection Decay (SPD) method and then describe the intuition behind SPD with a numerical example. Finally, we provide a concrete mathematical motivation for our method's s algorithmic design.
3.1 Selective Projection Decay (SPD)
Formulation. SPD is regularization technique that penalizes significant deviation from the pre- trained model. We motivate the formulation from an existing method: L2-SP [33] (Eq. D. L2-SP adds distance penalty on the deviation between the fine-tuned and pre-trained models. The penalty is applied to all model parameters at all times. A large À prevents every layer from deviating too much and empirically leads to poor fitting, while small À cannot provide enough regularization. This significantly limits the otherwise effective design. We propose selective version of this simple technique: selective projection decay (SPD). We will examine L2-SP and SPD Alg. | and Alg. []
Notations. We follow the notations in prior works I 21
Let m U2 denote the moving average of the gradient and squared gradient, 91, 82 their hyper-paramettrss, and a the learning rate.
Alg. D shows the Adam optimizer with the L2-SP regularization in Eq. [ The effects of the
regularization are highlighted in blue, also shown in Eq. 2 Intuitively, the regularization leads to an interpolation-like equatior If the product λa 1, then 0g +- 0g and if Xa 0, then O2 <- O,. where 9g and O, denote the initialization and the updated model without regularization.
0. 0-- -0 00)
(2)
Alg. shows the proposed SPD. There are two changes compared Alg.
condition,
C (0,-1 0g).
new interpolation equation with multiplier, rt, replacing the learning rate a,
8, - 0 Xr, 6)).
"Mathematically, this is not the precise formulation of L2-SP written in Eq. See the Appendix 8.3 for further discussion.
3

---

Compared to L2-SP, SPD only imposes penalty when the condition is met (c, < 0), and the strength of the penalty is controlled by hyyee--araameter À and an analytical quantity deviation ratio Tt- which we will introduce later.
Algorithm 2: Adam with Selective L2- -2-Reg.
Algorithm 1: Adam with L2- Regularization
Initialize my f O, vo t- 0, t + 0,0u + 0
Initialize mo < O, % + 0, E <- 0
While 0, not converged
While 0g not converged
t + t ++1
t 2t=t t +
9t. - VoL(0,-1)
gt. 6 VaL(0,-1)
m4 - 3pm-- + (1 B11gt
1 + Bime-1 + (1 81 31
V2 B20 + (1 B2)9tt
Ut. 6 8214-1 + (1 B2)
Bias Correction
Bias Correction
mi 1-37 Ug 1- B
1-3 , U2 x I- $1 my
my
12
U Update
Update
0, x- 0/-
am,
Voi+e
0r- 2- 01 Vor+e
C ol (0t-1 On)
0g + 0.- - A0x 00)
If 4 < 0
0, + 0g >rr 00)
3.2 Intuition Behind SPD
SPD prioritizes layers with consistent improvement. SPD adds regularization on layers that meet the condition e < 0 to slow their growth. The condition is determined by the sign of the inner product between two vectors. One vector is the negative gradient direction (-9t), i.e., the descent direction, and the other is the current progress direction (0,-1 0o)- The inner product between them measures the alignment between the munilddJupdate direction and the progress so far. When the inner product is positive, the current progress direction generally points to a low loss region, and following it will lead to consistent loss reduction. Conversely, if the inner product is negative, the current progress direction will likely lead to a higher loss region, indicating inconsistent improvement. In this case, SPD will impose penalty to slow down updates for those layers. Recall that modern optimizers use momentum to escape local minima and explore wider regions. Without this penalty, the model will likely head towards the higher loss region to overcome it. SPD chooses to slow down these layers and prioritizes layers with more consistent loss reduction. We will motivate this strategy in principled manner and validate our experiments.
3.3 Deriving o, from Hyper-Optimization
Previously, we explained the intuition behind SPD. Specifically, we interpreted the condition e as a measure of alignment and test of update consistency. Nevertheless, there is more profound reason why the quantity e is a natural choice for selective regularization. In this section, we motivate SPD from a more mathematical perspective.
#yptr--ptimizzttion Setup. Hyper-optimization a technique to optimize hyper-parameters inside an optimizer [1 16 [7]. They treat the hyper-paramcters trainable parameters and optimize them using another gradient- based optimizer. Let's start from the vanilla Adam with L2-SP algorithm (Alg.)) [ and treat the regularization strength dyyer-parametee À trainable parameter. To update X, we need to obtain its gradient by taking a derivative w.r.t. X after applying it.
aC(O.) aC(),) T 00: = a - (6. 60)
VA=
(5)
0x
a0, ax
Selection Condition c.. Intuitively, if the quantity VA is negative, applying the update in gradient
descent will increase the value of X, thus increasing the regularization strength of L2-SP. Conversely,
a positive quantity will decrease the regularization strength. Therefore, the sign(-9441 determines the change of regularization strength in the hyper-optimizationn of X. Formally, we detine the condition et as,
0o))
== - - - 0o)
(6)
the direction w/o momentum.

---

For memory efficiency, we use (0, - 80) instead of (θ, Pb) because both vectors point in the same direction and won't ton't affect the sign of et. This allows to discard 0,. Otherwise, we need to keep an additional copy in memory. In summary, when Gr < O, we apply a regularization for that layer as shown in Alg. 2 This calculation is done for each layer, and the regularization is selectively applied.
Alternative Interpretation: We just interpreted the selection condition e in SPD measure of consistency between the current heading direction and the gradient direction. This perspective is more valid when the algorithm has accumulated some updates, i... ||0 Boll2 >> and less justified when a heading has not been established at the beginning of training. To analyze this, we discuss the behavior SPD from the perspective of stochastic optimization when ||0, 0ol12 is small the beginning of training in the Appendix 8.
3.4 Deriving from Projection
The selection condition e determines when to apply regularization to which layers. However, one remaining question is the strength of regularization, which is not intuitive tune. To overcome this, we introduced an analytical quantity, the deviation ratio in Eq. 2 and Alg. I In this section, we will motivate it from the perspective of projection.
L2-SP is projection. Projection onto norm ball is common in constrained optimization. While 1.2-SP is not constrained optimization problem, its operation bears similarity to projection. Suppose project model Og L2-norm ball with radius y centered around its initialization 0g. The equation of projection is the following.
0- 6u maxfy, |10, 02ll2}
(0. 6g)
(7)
Equivalently, we can rewrite the equation as,
0p = 0, - -(1-max/
7 max 110 (1=max(7,
(0. - 6g)-
1000-01112] 0u112)
Now, we can equate this equation the highlighted L2-SP equation Eq. and Alg. we can see that if Aa maax/>_1l0: 8ol
the regularization equivalent to projection with radius 7. Deviation Ratio rt. This equivalence inspires us to define a deviation ratio ry:
max (0, 7t -
rt
7
where 7/ = 110, - Gnll2 and " == ||0-- 0ol12 denote the current deviation (before regularization) and the previous deviation from the initialization Og. respectively. We use Tr in SPD (Alg 2 to replace the learning rate in L2-SP (Alg. D to make hyper-parameter (A) tuning more intuitive. Specifically, suppose the hyper-parameter À then the regularization in SPD
0g < 0
max (0, (0.7 -- (0. 0o) 0o
7-1
(0e Ou).
(10)
max (/-1, 7e
Intuitively, with À = 1. the regularization in SPD is equivalent to projection with radius equal to the previous deviation if the current deviation is larger. In summary:
No regularization (A = 0): the projection radius is ||0, d0112. meaning no projection.
Weak regularization (1 X 0): the projection radius lies between ||0 0 2 and |0:-1 - 0oll2- Within this range, all layers will expand or remain unchanged.
Strong regularization (A 1): the projection radius lies between and |0.-1 1 - 0 00 In this range, it's possible that regularized layers can contract.
We recommend starting with À and adjusting the strength according to the specific needs.
5

---

Full Tune-Tuning
PEFT Fine- Fine-Tuning
(Wo) Pre- pre-trained model
HaAa l
(I%) Fine-tuned model
IIW; Wall2 *WMIINMMll!
(AW;) PEFT model
"Zero' model
Figure Selective Projection Decay (SPD) imposes regularization on layers selectively during
fine-tuning. It regularizes |[; - Woll2 for full fine-tuning and ||W, |2 for PEFT fine-tuning.
3.5 Compatibility with PEFT methods.
As shown in Alg. 2 SPD retains copy of the pre-trained model in memory. This adds additional memory requirements to the overhead of vanilla optimizers. While this is practical for moderate- sized models, as fine-tuning focuses more and more on large models, additional memory requirements become undesirable. Fortunately, in extremely large models, the prevalent fine-tuning strategy is parameter-efficien fine-tuning (PEFT), such as LoRA 77. series adapters 19, and parallel adapters [8]. SPD naturally compatible with these methods without the additional memory. Intuitively. SPD selectively projects the current model towards the pre-trained initialization. PEFT methods generally initialize new parameters to add to the original model weights. To recover the behavior of SPD, we can instead project the new parameters towards the origin, equivalent to a selective version of regular weight decay, i.e., replacing On with 0 in Alg. 2 Consequently, this does not require memory copy of the pre-trained model. It consistently improves PEFT fine-tune fine-tuning for large language models on common sense reasoning benchmarks in Sec. 4.4
For example, LoRA decomposes a linear layer h = Wa into two components, where h E Rmx1 We IR m and x E Rax! are the output, weights, and input of this layer.
= Wi = (Wo + AW,)r Wox + Wup W Wdown
(11)
where Wo R W up Rmxr and W down RR the pre-trained model, up-projection and down-projection matrices. If < min(m, (m, nJ. (WapWdown is a low-rank approximation of AW, To regularize the overall deviation |[W, Woll2. suffices to regularize ||Wug down ||2 to be close to zero. In this case, SPD acts as selective weight decay on Wap and W Wdown individually.
In summary, we propose selective projection decay (SPD) to impose strong regularization layers during munituning selectively. As shown in Fig. [ SPD regularizes the deviation of the fine-tuned model from the pre-trained model I[W, W for full fine-tuning and the deviation from the origin ||WW for PEFT fine-tuning.
4 Experiments
We test Selective Projection Decay diverse of benchmarks, architectures, and tasks to demonstrate its effectiveness. We will test both ID generalization and OOD robustness across various domain and distribution shifts.
Image Classification. We first analyze the behavior SPD on conventional image classification datasets DomainNet (22 and ImageNet (23) We use a CLIP ViT-Base model for both experiments as the pre-trained initialization [24]. Specifically, DomainNet consists of images from several domains with 345 classes. We fine-tune
domain and test on all domains. ImageNet is large-scale dataset with 1000 classes. We fine-tune on ImageNet and test on ImageNet and four variants, namely ImageNet-V V2 25 ImageNet-A (26), ImageNet-R (27). and ImageNet-S 28
Semantic Segmentation. We further test SPD on the PASCAL-Context semantic segmentation dataset (29. Following prior works [30 11, we use Swin ViT-Tiny [1], pre-trained on ImageNet- 22K, and Segformer 32 segmentation architecture. To construct OOD datasets, we follow the popular natural robustness literature 33 and apply four representative image corruptions (fog, defocus blur, Gaussian noise, and brightness) with 5 severity each. We fine-tune on the clean segmentation data and test on clean and corrupted data.

---

Common Sense Reasoning, Moreover, we show that SPD can benefit PEFT
on large
language models (LLMs). We use the Commonsense- 170K dataset [4,, which consists of training
data from eight common sense reasoning benchmarks. Following the prior work [34] we fine-tune
LLaMa-7B (-13B)
35
using LoRA
77, series adapters 9 and parallel adapters g
Visual Question Answering. Finally, we demonstrate SPD's superiority on multi-modal task. We use Google's recently released PaliGemma [36] pretrained on broad mixture of large-scale vision-
language tasks. We fine-tune on VQAv2 37 and test on nine OOD datasets using LoRA [7]
For the near OODs. we evaluate on VQAv2's six variants, namely IV- VV- [38,, CV-VQA [38. VQA-Rephrasinns [39]. VQA-CP v2 40), VQA-CE [4]] and AdVQA 42), which cover uni-modal, multi-modal and adversarial distribution shifts from VQAv2. We also include TextVQA [4]]
VizWiz 44 and OK OK-VQAv2 45
which are constructed from different sources than VQAv2, as the
far OOD datasets.
4.1 NemainNet Experiments
Table I: Comparisons between AdamW and Adam-SPD on DomainNet. A pre-trained CLIP ViT- Base model fine-tuned on each of the five domains in DomainNet and tested on all domains. Each row represents the evaluation of model fine-tuned blue. The last column shows the deviation of the final model from its initialization. Adam-SPD shows much better OOD performance with significantly less Deviation (1O2 2) than vanilla Adam W.
domain. II) performance is highlighted in
Test Domains Sketch
Statistics
Optimizer
Fine-tune Domain
Real
Clipart
Painting
Quickdraw
Infograph
1
OOD Avg.
Deviation
AdamnW
53.13
44.11
8.44
84.83
1.2.5P
Real
82.33
53.35
51.82
42.04
8.21
30.84
82.33
37.25
0.70
Adam-SPD
87.10
63.45
60.34
54.12
11.73
39.99
87.10
45.93
AdamW
54.50
79.88
40.97
46.87
13.14
26.31
79.88
36.36
0.83
L.2SPP
Clipart
55.73
79.67
41.61
47.12
11.51
26.51
79.67
36.50
0.70
Adam-SPD
61.44
81.43
48.31
52.06
13.73
31.62
81.43
41.43
0.40
AdamW
55.62
46.64
74.90
40.56
8.55
26. 18
74.90
35.51
0.81
L2-SP
Painting
54.73
45,15
73.45
38,
4.3
24.87
73.45
33.56
0.67
Adam-SPD
60.66
52,43
77.77
47.81
6.38
34.84
77.77
36.92
0.38
AdamW
45.02
52,97
39.70
72.26
15.16
18.79
72.26
34.33
0.95
L2-SP
Sketch
47.45
52.7
40.74
71.05
14.96
23.36
71.05
35.84
0.67
Adam-SPD
52.81
$7.39
46.90
74.00
15.77
24.35
74.00
39.44
0.40
AdamW
3.08
10.12
1666
9.61
68.68
1.04
68.68
1.72
1.2.5P
Quickdraw
4.03
11.06
2.11
9.13
62.21
1.61
62.21
5.59
0.77
Adam-SPD
18.72
24.36
12.77
20.61
66.81
66.81
16.70
0.58
AdamW
$1.49
42.46
37.20
35.46
6.02
52.71
52,71
34,53
0.85
1.2.SP
Infograph
51.46
41.99
38.39
35.75
6.8
$3.33
53.33
34.88
0.70
Adam-SPD
58.29
48.25
46.00
43.38
7.88
56.36
56.36
40.76
0.36
In this section, we utilize the DomainNet benchmark to test our claims. Specifically, we will show that Adam-SPD consistently outperforms AdamW in OOD robustness across multiple domains, and this is due to a much smaller deviation from the pre-trained model. Furthermore, by sweeping across range of hyper-parametery adequate constraints, while Adam-SPD shows robust performance.
show that uniform regularization, such as L2-SP fails to provide
Small deviation correlates with better OOD performance. Earlier, we hypothesized that significant deviation can lead to worse OOD performance. Theoretically, prior works 112 [T] have shown that large deviations from the initialization result in large Liptschtz constant and, hence, worse robustness. In Tab. [ we present a comprehensive study by fine-tuning a pre-trained CLIP model on different domains from DomainNet separately and reporting test results all domains. Across all domains, Adam-SPD consistently achieves higher OOD performance and shows noticeably less deviation. This empirical result corroborates with prior works' findings and our hypothesis.
Selective regularization exhibits stronger deviation-robustness correlation. compare the behavior of L2-SP and SPD using DomainNet. Specifically, we fine-tune CLIP model on the Clipart domain (ID domain) and report performance on Clipart and other domains (OOD domains). In Tab. 2a we observe that while I.2-SP successfully restrains the model's 's deviation from its initialization, it does not effectively improve OOD performance. With very large regularization, the ID performance deteriorates well. On the contrary, SPD effectively restrains the model's deviation
Tab. we

---

Table 2: Comparisons between L2-SP and Adam-SPD. ID dataset: (clipart), OOD datasets: (real, sketch, quickdraw, painting)- Selective regularization can effectively restrain model's deviation (I],, 2) and improve OOD robustness without significantly impacting ID robustness.
Hyper- Prper-Parameter
lc-1
le-2
6e-3
3e-3
1e-3
6c-4
3e-4
lc-4
1e-5
lc-6
le-7
0.0
Deviation
0.03
0.14
0.18
0.24
0.34
0.39
0.46
0,53
0.58
0.58
0.58
0.59
OOD
14.90
37.20
39.43
40.52
41.13
41,76
40.52
41.26
41.35
41.73
40.62
41.34
ID
27.25
69.74
73.76
76.62
78.90
79.30
79.30
79.84
79.80
79.95
79.80
79.91
(a) L.2SPP hyper-paramess (A) sweep. Stronger regularizations (larger values) decrease deviation; however, they do not improve OOD performance and even deteriorate IDD performance.
Hyper-Parameler Purameter
2.1
1.9
1.7
1.5
1.3
0.9
0.7
0.5
0.3
0.1
0.0
Deviation
0.31
0.32
0.33
0.34
0.36
0.36
0.42
0.44
0.48
0.51
0.54
0.59
OOD
45.67
45.23
45.27
44.81
44.18
42.73
11.84
41.20
41.34
81.21
80.76
81.25
80.67
81.11
79.89
79.57
80.00
79.92
80.26
80.00
79.91
(b) Adam-SPD hyper-parameter (A) sweep. Stronger regularizations (larger values) decrease deviation, simulta- neously improving OOD performance. The ID performance not impacted significantly.
and significantly improves OOD performance while matching the best ID performance. Under SPD, the correlation coefficient between OOD performance and deviation is -0.96, which indicates a strong negative correlation between the two quantities, i.e., smaller deviation and higher OOD
accuracy. This experiment shows that selective regularization is superior uniform regularization.
Training Details. use the vision transformer public repository for DEIT (46 to fine-tune all methods. We use À 1 for all Adam-SPD results in Tab. [ More details are in Appendix 8.4
4.2 ImageNet Experiments
Table 3: ImageNet Fine- Tine-Tunigg Result using CLIP ViT-Base. SPD outperforms more complicated algorithms and beats L2-SP 8.8% by selectively imposing regularization.
ID Im
OOD
Statistics OOD Avg.
Im-V2
Im-Adversarial
Im-Rendition
Im-Sketch
Avg.
Zero-Shot
67.68
61.41
30.60
56.77
45.53
48.58
52.40
Vanilla FT
83.66
73.82
21.40
43.06
45.52
46.98
54.29
Linear Prob.
78.25
67.68
26.54
52.57
48.26
48.76
54.66
LP-FT
82.99
72.96
21.08
44.65
47.56
46.56
53.85
L2-SP 83.44
73.2
20.55
43.89
46.60
46.06
53.54
FTP
84.19
74.64
26.50
47.23
50.23
49.65
56.56
SAM
83.67
73.66
20.48
42.98
45.70
45.71
53.30
Adam-SPD
84.21
74.83
25.42
49.09
51.18
50.13
56,95
WISE-FT 80.94
72.47
33.18
63.33
54.20
55.58
60.82
WISE-SPD
81.70
73.29
34.37
63.69
54.55
56.48
61.52
SPD outperforms more complicated works on image classification. Following the training recipe from the prior work [T], we fine-tune a CLIP ViT-Base model on ImageNet using Adam-SPD We use the same hyper-paree dyper-parameters parameter SPD. Tab. 3 we observe that Adam-SPD provides the best ID performance (strong ID generalization) and best average OOD performance (strong OOD robustness) on four ImageNet variants. SPD achieves a level of competitive performance with just a few lines of code. SPD's simplicity and strong performance show that selective regularization is fundamental improvement for robust fine-tuning.
the prior work and only adjust the regularization hyper-
Training Details. For Adam-SPD, we fine-tune the model with learning rate of 3e - 5 and X = 1.4. The regularization hyper-paramest is found through vals-valii,tion, and the model with the best ID validation accuracy is taken. More details are in Appendix 8.4

---

robust 4.3 PASACAL Dense Semantic Segmentation
Table 4: Pascal Semantic Segmentation Results with SWIN-Tiny transformers (ImageNet2 K pre-
trained). Performance is measured by mloUt. SPD improves OOD robustness compared to vanilla fine-tuning without regularization and L2-SP by 36.5% and 5.8%, respectively.
ID
OOD
Statistics ID A (%)
Clean
Fog
Defocus
Gaussian
Brightness
OOD Avg.
OOD A (%)
Vanilla FT
66.03
56.72
38.04
23.21
58.03
44.00
0.00
0.00
Adapter 9
71.85
69.36
50.94
37.43
68.26
56.50
8.82
28.40
BitFit 47
70.31
67.00
46.39
30.61
66.22
$2.56
6.49
19.44
L2-SP [3
73.47
69.87
49.20
39.10
68.61
56.70
11.27
28.85
MARS-SP [22
66.24
56.97
37.29
21.82
58.27
43.59
0.32
-0.94
LLRD 48
72.09
68.13
46.18
37.28
66.30
$4.47
9.18
23.79
TPGM 10
72.56
69.51
50.88
38.62
68.82
56.96
9.89
29.44
FTP 11
73.79
71.10
52.63
40.25
69.81
58.45
11.76
32.83
Adam-SPD
74.27
71.74
53.41
44.17
70.92
60.06
12.47
36,50
SPD outperforms more complicated works on semantic segmentation. The same trend is observed on semantic segmentation in Tab. 4 Again, SPD achieves the best ID generalization and OOD robustness across four different corruptions. This shows that proper regularization is not only important for achieving strong ID generalization (performance the test set) but also for strong OOD robustness (performance on distribution shifted test sets) to domains shift (Tab, 3) and distribution shift such natural corruptions (Tab. 4 The model fine-tuned with SPD is consistently across different levels of corruption and severity.
Training Details. For Adam-SPD, we fine-tune the model with learning rate of le 4 and X = 2.2. The regularization hyper-parameter parameter is found through cross vooss-vaiiattion, and the model with the best ID validation accuracy is taken. More details are in Appendix 8.4
4.4 LLaMA PEFT Fine-Tuning Experiments
Table 5: Accuracy comparison of LLaMA-7B (-13B) with different adapters and optimizers eight commonsense reasoning datasets. SPD consistently improves fine-tuning performance on multiple PEFT methods across all datasets. Note that Adam W employs uniform weight decay by default.
PEPT
LLM
Optimizer
BeolQ
PIQA
SIQA
HellaSwag
WingGrande
ARC
ARC
OBQA
Avg.
Series
LLaMAra
Adam-SPD (1.0)
Adama
63.0
79.2
77.4
76.3
67.9
79.7
75.7
74,5
57.1
63.5
78.4
72.4
76.1
70.8
Parallel
IJaMAAB
AdamW
67.9
76.4
78.8
69.8
78.9
73.7
$7.3
75.2
72.3
Adam-SPD (1.0)
80.9
78.3
LoRA
LLaMAra
AdamW
68.9
80.7
77.4
78.1
78.8
77.8
61.3
74.8
74.7
Adam SPD (0.7)
69.1
82.8
78.9
84.8
80.7
80.9
65.8
79.2
77.8
LoRA
LLaMAsso
AdamW
83.5
80.5
80.5
83.7
82.4
80.5
Adam-SPD (1.2)
72.9
85.6
80.7
92.0
83.7
85.6
71.6
85.6
82.2
SPD compatible and consistently improves PEFT methods. Previous experiments have shown that SPD imposes effective regularization for full fine-tuning, Furthermore, SPD also improve the performance of PEFT methods. We fine-tune LLaMa-7B (-13B) models on the Commonsense-117 dataset B.]. As shown in Tab. 5 SPD consistently improves regular fine-tuning with AdamW, which uses a uniform weight decay for all tested PEFT methods. This demonstrates that selective and PEFT fine-tume fine-tuning- Combined with its simplicity, SPD can potentially improve generalization and robustness for more tasks in deep learning.
regularization benefits full
Training Details. We follow the training code released by prior work 34 best performance from the original paper and compare them with Adam-SPI More details are in Appendix 8.4
We report the

---

4.5 Visual Question Answering (VQA) Experiments
Table Visual Question Answering Result using PaliGemma-33 SPD outperforms baselines across ID, near OOD and far OOD datasets using LoRA. Note that L2-SP reduces to Vinilla FT with Adam under LoRA.
Vision IV-VQA
Question VQA-Rephrasings
Answer VQA-CP v2
Maltimodal VQA-CE
Advensrial AdVQA
VQAv2
CV-VQA
ToxVQA
VizWiz
OK-VQA
Zero-Shot
54.42
63.95
44.72
50.10
$4.29
30.68
30.46
14.86
16:84
28.60
Vanilla FTLLRR))
78:90
71.73
49.82
42.08
22.92
48.30
LP.FT(LORA)
85.97
93.30
65.93
76.49
$6.16
72:73
45.68
31.41
19.01
43.27
WISE FT(LeRA)
71.36
85.06
64.55
66.42
70.89
48.74
43.95
36.98
22.41
42.35
Adam SPD(LoRA)
87.39
95.25
68.85
79.48
87.27
73.52
50.90
43.56
23,05
50.11
SPD shows competitiveness across ID, near OOD, and far OOD datasets multimodal tasks. Apart from uni-modal tasks, SPD outperforms other baselines on multi-modal tasks. We fine-tune PaliGemma-3B model VQAv2 [37] dataset with LoRA. In Tab. SPD improves vanilla fine- tuning and other robust fine-tuning methods, achieving best ID and average OOD performance w.r.t. distribution shifts across single modalities such vision, question, answer and combinations of multiple modalities. We also show the performance evaluation for both near and far OOD datasets. SPD is consistently more robust under different types and degrees of distribution shifts.
Training Details. For Adam-SPP SPD, we fine-tune the model with a learning rate of 1e - 3 and À = 0.5. The regularization Syper-paramedd is found through cross-validation, and the model with the best ID validation accuracy is taken. More details are in Appendix 8.4
Limitations
SPD is selective regularization technique explicitly designed for fine-tuning. While can be theoretically used for pre-training, it will likely lead to poor performance because it will hinder the training of some layers. For fine-tuning, works well because the pre-trained foundation model is assumed to be good initialization, and only small changes in selected few layers can lead to good local minimum. Furthermore, the level of performance gain depends on how well the foundation models are exposed to the fine-tuning and OOD data distributions during pre-training. For example, in the DomainNet experiment (Tab. []. fine-tuning CLIP VIT model on any other domain does not have reasonably good OOD robustness on the Quickdraw domain. One can deduce that Quickdraw is well represented in the pre-training data of CLIP VIT.
Conclusion
Fine-tuning differs from training from scratch because it starts from good initialization. Therefore, effective regularization is critical to retaining the knowledge the pre-trained foundation model while fitting a model to the target distribution. We identified that 1) regularization is necessary to keep the tinettuned model close to its initialization and maintain robustness; 2) uniform regularization can hurt model fitting if regularization is too strong. In this paper, we proposed selective projection decay (SPD), a selective version of the popular weight decay/L22SP regularization method. With an additional few lines of code, SPD can be integrated into existing optimizers and performs selective regularization. It demonstrates superior regularization performance on different tasks and modalities in our experiments.
10

---

7 Acknowledgement
This work was supported by ONR grant N80014-1-112829
References
[1] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[2] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprini arXiv: 1708.0388, 2017.
[3] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv: voXvv:994.00062,, 2019.
[4] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning,
pages 1139-1222 1147. PMLR, 2013.
[5] Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04777, 2016.
[6] Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. arXiv preprint arXiv:2210. 1466, 2022.
[7] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
[8] Shwai He, Liang Ding, Daize Dong, Miao Zhang, and Dacheng Tao. Sparseadapter: An easy approach for improving the parameter-efficiency of adapters. arXiv preprint arXiv:2210.04284, 2022.
[9] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nip. In International Conference Machine Learning, pages 2790-2799. PMLR, 2019.
[10] Junjiao Tian, Xiaoliang Dai, Chih-Ya Yao Ma, Zecheng He, Yen-Cheng Liu, and Zsolt Kira. Trainable projected gradient method for robust fine-tuning, arXiv preprint 2303. 2023.
[11] Junjiao Tian, Yen-Cheng Liu, James S Smith, and Zsolt Kira. Fast trainable projection for robust fine-tuning. Advances in Neural Information Processing Systems, 36, 2024.
[12] Henry Gouk, Timothy M Hospedales, and Massimiliano Pontil. Distance-based regularisation of deep networks for fine-tuning. ICLR, 2021.
[13] LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International Conference on Machine Learning, pages 2825- 2834. PMLR, 2018
[14] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongseok Namkoong, al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9959-7977, 2022.
[15] Kartik Chandra, Audrey Xie, Jonathan Ragan- Kelley, and Erik Meijer. Gradient descent: The ultimate optimizer. Advances in Neural Information Processing Systems, 35:8214-8225, 2022.
[16] Xiang Wang, Shuai Yuan, Chenwei Wu, and Rong Ge. Guarantees for tuning the step size using a Marning-to-llane approach. International Conference on Machine Learning, pages 10981-10990. PMLR, 2021.

---

[17] Matthias Feurer and Frank Hutter. Hyperparameter optimization. Automated machine learning: Methods, systems, challenges, pages 3-33, 2019.
[18] Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 110:393-416, 2021.
[19] Ananya Kumar et al. Fine-tuning can distort pretrained features and underperform out-of- distribution. ICLR, 2022.
[20] Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetume like you pretrain: Improved finetuning of zero-shot vision models. arXiv preprint 1rXiv:2212.00638, 2022.
[21] Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795-1888 18806, 2020.
[22] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406-1415, 2019.
[23] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8763. PMLR, 2021.
[25] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR, 2019.
[26] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[27] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340,8349, 2021.
(28) Haohan Wang, Songwei Ge, Zachary Lipton, and Fric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[29] Mark Everingham, Luc Van Gool, Christopher KI Williams. John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJV,, 2010.
[30] Yen-Cheng Liu, Yao Ma, Junjiao Tian, Zijian He, and Zsolt Kira. Polyhistor: Parameter- efficient multi-task adaptation for dense vision tasks. arXiv preprint arXiv:22/0.03225,, 2022.
[31] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[32] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34:1 -2090, 2021.
[33] Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common 2019.
corruptions and perturbations. arXiv preprint
12

---

[34] Zhiqiang Hu, Lei Wang. Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka- Wei Lee. Alm-adapters: An adapter family for parameter-efficien fine-tuning of large language models. arXiv preprint arXiv:2304. 0tti::3344009333,, 2023.
[35] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amn Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 3971, 2023.
[36] Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang. Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. PaliGemma: A versatile 3B VLM for transfer, July 2024. urXiv:2407.07726 [cs] version: 1.
[37] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering, May 2017.
[cs].
[38] Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards Causal VQA: Revealing and Re- ducing Spurious Correlations by Invariant and Covariant Semantic Editing. In 2020 JEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9687-9695, Seattle, WA, USA, June 2020. IEEE
[39] Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. Cycle-Consisteng for Robust Visual Question Answering, February 2019. arXiv: 1902.05660 [cs].
[40] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't Just As- sume; Look and Answer: Overcoming Priors for Visual Question Answering, June 2018. arXiv:1712.00377 [cs].
[41] Corentin Dancette, Remi Cadene, Damien Teney, and Matthieu Cord. Beyond Question-Bass Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering, September 2021. arXiv:2104.03149 [es].
[42] Sasha Sheng, Amanpreet Singh, Vedanuj Goswami, Jose Alberto Lopez Magana, Wojeiech Galuba, Devi Parikh, and Douwe Kiela. Hluman-Adversari Visual Question Answering, June 2021. arXiv:2106.0222 [cs].
[43] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA Models That Can Read, May 2019. 1XXv:1904.00920 [es].
[44] Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. VizWiz: nearly real-time answers to visual questions.
[45] Benjamin Z. Reichman, Anirudh Sundar, Christopher Richardson, Tamara Zubatiy, Prithwijit Chowdhury, Aaryan Shah, Jack Truxal, Micah Grimes, Dristi Shah, Woo Ju Chee, Saif Punjwani, Atishay Jain, and Larry Heck. Outside Knowledge Visual Question Answering Version 2.0. In ICASSP 2023- 2023 IÉEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, June 2023. ISSN: 290X-
[46] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers and distillation through attention. In International Conference on Machine Learning, volume 139, pages 10347-10357, July 2021.
[47] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-base masked language-models, arXiv preprint arXiv:2 106. 10199, 2021.
13

---

[48] Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting few-sample bert fine-tuning. arXiv preprint arXiv:2006.05595 05987, 2020.
[49] Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In International Conference Machine Learning, pages 404-413. PMLR, 2018.
[50] Guillaume Garrigos and Robert M Gower. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint a:Xiv:2301.11235, 2023.
[51] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60(2):223-311, 2018.
[52] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:
2016.
[53] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 18-2626, 2016.
[54] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint
2017.
[55] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023 6023-6032. 2019.
[56) Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven C. H. Hoi. Lavis: library for language-vision intelligence, 2022.
14

---

8 Appendix
8.1 Extended Related Works
Other Robust Fine- Tuning Methods. WiSE-FT [14] discovers that linearly interpolating between the fine-tuned and pre-trained models after fine-tuning can improve out-of-distribution robustness. This demonstrates that a closer distance to the pre-trained model can improve robustness. However, it only applies models with zero-shot capabilities. Another orthogonal line of research for robust fine- tuning focuses on feature distortion. LP-FT [9] shows that fine-tuning with randomly initialized head layer distorts learned features. It proposes simple two-stage method to train the head layer first and then fine-tune the entire model. FLYP 20 shows that fine-tuning a foundation model using the same objective as pre-training can better preserve the learned features. Our contribution is an optimization method to penalize the derivation between the fine-tuned and pre-trained models explicitly during fine-tuning, which is orthogonal to them.
8.2 Interpreting o as an Early Layer Selection Criterion
In previous sections, we interpreted the selection condition in SPD as measure of consistency between the current heading direction and the gradient direction. This perspective more valid when the algorithm has accumulated some updates, i.e., ||0, 8u Poll2- > 0. and less justified when a heading has not been established at the beginning of training. This section discusses SPD from the perspective of stochastic optimization when |0, oall2 is small at the beginning of training.
Inner product of gradients captures gradient variance. Modern deep learning models trained by stochastic optimization techniques, e.g., mini-batch SGD, leading to stochasticity due to sampling. We first show that the inner product of gradients captures the variance of a sampling process. We invoke a common assumption in the convergence analysis of stochastic gradient descent [| 21). Assuming that the stochastic gradient gr is stationary process G over short period, with small step size, successive gradients, e.g., 9t- It+1, can be seen as samples drawn from the same distribution G. Given two successive draws Ø1 and 92, we can approximate the first and second moment of g.
1 (or + 92)ll
||aal = 2 (11 1192113),
||E [g]
11 = ||
(12)
50 51 where ÿ Elg] can show Define the variation of gradients as Var(g) :- E [llg ø| that
g = [11la1]] 1| + "4 |112211 2 + 2.
1
1
(119 112 + (1a2113)
(13)
2
~ 1| - (E [[|=|3] - E [gl I1²) = | Var(g)
Remarks. Eq. 13 shows that the inner product of two consecutive stochastic gradients, under certain assumptions, can be seen as the estimator for the difference between the gradient norm and the variance of gradients. When the inner product is negative, this indicates that the variance outweighs the magnitude of the gradient.
SPD prioritizes layers with higher expected gain. At the beginning of training, the heading direction (0: 0o) is dominated by early gradients. For example, at the direction of (0, 0o) is the same as 91 in Adam. The sign of -9.] (01 0o) is the same as the sign of 9. This shows that the condition e captures the difference between gradient norm and gradient variance. With this interpretation, we show that reflects expected performance gain in stochastic optimization. To see it, we can invoke the descent lemma for SGD. For an L-smooth function f(W) (50), the descent lemma for SGD states that,
L 2 +
m L
Lemma 1. [f f(Ox) mh
|
Var(gh),
2
Expected Performance Gain
50
20
where ?/ f is the learning rate. Remarks. The term on the left hand side [[[(0<+1)] (0x) is the expected performance improve- ment for each step. Idcally, this should be negative quantity. On the right-hand side, observe
15

---

that improvement depends on two quantities | 2 and Var(g)). To lower the upper bound, we want large g 2 and a small Var Var(g)). According to the decoupling Eq. [3 the inner product between successive gradients approximates this proportionality. Consequently, negative likely indicates higher upper bound on the expected gain, meaning smaller improvement. Therefore, SPD will prioritize layers with potentially larger expected gains.
8.3
Approximation in L2-SP
Algorithm 3: (Ours) Adam with L2-
Algorithm 4: (Original) Adam with L2- Regularization
Regularization
Initialize mo t O, vo + 0, + While not converged
Initialize my f 0, v + 0, t + 0
While 02 not converged
t - t ++1
t + t ++1
9r VoL(O;-1)
gt Vo((O:-1)
m4 < Bime- + (1 31 31)
my 3pm
B1 19t
Vt. 8204-1 (1 B2)
02 820 + (1 $2)91
Bias Correction
Bias Correction
my + I-3 Ut x I- $
mi
my t--
I- Ug t 1-
m.
Update
Update
0₂ <- 81-1
0, t 0
amm
V
Voite
0. t- 0g
(0: 0o)
0. + 0: Aa 00)
The Adam with L2-SP Regularization algorithm in the main paper is not the precise mathematical implementation of the original formulation written Eq. [ To see the difference, we compare ours and the accurate implementation here in Alg. 3 and Alg. 4 In our implementation, we replaced
f--1 0p (Alg. 4 with 0, On (Alg- 3). This is done intentionally to improve memory efficiency when transitioning to the selective version (see Adam-SPD in Sec.3.3). We can make the following substitution see how this modification changes computation. Starting from our implementation,
0, = 0 Aa(6, 0o)
(14)
am
= 0 - Aa(00-11
6o)
Vo, +
= 0. - Ac - 0o) + Xa²
mn
Vor- +
We can further combine the Xa
me Ve
into the update of 0. The new 0 is
8, 0
at774
+ Xa
(15)
U1 +
V +
am
0,1 (1 - Aa
Vūr +
Therefore, our implementation of L2-SP adds minor additional dampening of the learning rate by a factor of (1 - xa).
What if we followed the original implementation of L.2SPP in Alg. 4 This would change the condition e in the main paper (Eq. 3 to
C = - (0:-2 0o).
(16)
At the current time step f. we only have access to the parameters 0,-1 from the previous step t 1. To calculate the e, in Eq. 16 we would have to store the weights from two steps back in memory. This increases memory consumption of the algorithm. As we have shown in Eq. [5] our implementation only differs from the original implementation slightly but reduces memory consumption. Therefore, we decided to make this approximation.
16

---

8.4 Training Details
DomainNet. We use the vision transformer public repository for DEIT [46 to fine-tune all methods.
Standard augmentations are used for all: weight-decas (0.1), drop-path (0.2) [52]. label-smoothing
(0.1) [53], Mixup (0.8) [54] and Cutmix (1.0) [55]. The learning rate is 2e 5 and trained for 60 epochs for Tab. and 30 epochs for Tab. We use À 1 for all Adam-SPD results in Tab. [ We use A40 GPU for each experiment.
ImageNet. The same procedure as the DomainNet experiment is used for training the ImageNet models. Standard augmentations are used for all: weight-decay (0.1), drop-path (0.2) [52], label- smoothing (0.1) [53], Mixup (0.8) [54] and Cutmix (1.0) [55]. We fine-tune all methods for 30 epochs and use the best hyper-parameters reported by the prior work MТ.. For Adam-SPD, fine-tune the model with learning rate of 3e and À = 1.4. The regularization pyper-parameter found through cross- vooss-valdation,, and the model with the best ID validation accuracy taken. We use 2 A40 GPUs for each experiment.
Pascal Segmentation. We follow the training code released by prior work 30 methods for 60 epochs and use the best hyper-parameter reported by the prior work. For Adam-SPD, fine-tune the model with learning rate of 1e - 4 and À = 2.2. The regularization hyper-paramett is found through cross-validation, and the model with the best ID validation accuracy is taken. We use 4 2080Ti GPUs for each experiment.
We fine-tune all
We report the best performance from the original paper and compare them with Adam-SPD. For Adam-SPD fine-tune the model with an identical hyper-ypramee regularization strength A. The regularization typer-parameter is found through cross- cosss-vlidation, and the model with the best ID validation loss is taken. We use A40 GPU for each experiment.
Commonsense-1700 We follow the training code released by a prior work 34
setup as the released code and only adjust the
Visual Question Answering. We follow the LAVIS [56] public repository to fine-tune all methods. We use the PaliGemma (36] model pretrained with 224 * 224 input images and 128 token input/output text sequences and fine-tune with the precision of bfloat Standard hyper-parameters are used for all: learning rate (le 3), weight-decay (le - 4), optimizer (AdamW), scheduler (Linear Warmup With Cosine Annealing), warm-up learning rate (le - 4), minimum learning rate (le 4), accumulation steps (2), beam size (5). The model is trained for 10 epochs with batch size of 16 for Tab. 6 For LoRA 77 we limit our study to only adapting the attention weights and freeze the MLP modules for srrrametr--ffccieneyy specifically apply LoRA to Wg. Ws, W., W, with r = in Tab. 6 We use x 0.5 for SPD. The regularization hyper-parameter is found through cross-validation, and the model with the best ID validation accuracy is taken. We use 8 A40 GPU for each experiment.
17

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpipp/.ccppullc//uuuii wwwticccguie//eeeeeeeeeii CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /aaps::///uripss/ http://urrs.cccupuippe/e http://...cc.u.cccttttccccceeeea
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The work does present obvious negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
21

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.ree.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
22

---

Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "GVafIfd0Ov", "reviewer": "Reviewer_dDi9", "summary": "This work proposes a new regularization scheme called Selective Projection Decay (SPD), which selectively imposes penalties on layers where the current progress direction does not align with the vanilla update direction. This method is compatible with PEFT methods, including LoRA-type algorithms, making it practical for end users. Experiments show that ADAM equipped with SPD consistently outperforms baselines, including L2 regularization, in terms of in-distribution generalization and out-of-distribution robustness on multiple popular vision and language benchmarks.", "questions": "#1. How does the performance compare when using EWC regularization for fine-tuning, since EWC is generally better than L2 regularization?\n\n#2. Comparing the performance of L2-SP and Adam-SPD in hyper-parameter sweeping experiments seems challenging since these are reported in tables. Perhaps heatmaps could be a clearer alternative to visualize and highlight the results.\n\n#3. The authors argue that SPD selectively imposes penalties on large deviations in a layer-wise manner. However, in Algorithm 2, there is no layer-wise information. It appears that all parameters are updated based on the alignment score denoted by c_t. From my understanding of Algorithm 2, if the alignment is high (i.e., c_t is smaller than zero), the updated parameters are further adjusted by the interpolation-like equation. If c_t is larger than zero, do we use the vanilla ADAM update, since \\theta_t has already been updated before calculating c_t? Am I correctly understanding Algorithm 2? In such cases, it seems that the parameters shouldn’t be updated, since SPD will impose a penalty to slow down updates for those layers with negative c_t.\n\n#4. In the LLaMA PEFT experiments, there is no baseline with L2 regularization or its advanced versions (such as EWC). While I expect that ADAM with SPD is better than AdamW, I would like to see if there are also improvements compared to ADAM with L2 regularization in PEFT settings.", "limitations": "The authors state that the proposed approach only works well if the pretrained model is good enough to provide good initial parameters. No clear negative societal impacts have been observed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The simplicity of the proposed method is its major strength, making it easy for many users to adopt. Compatibility with parameter-efficient fine-tuning is a significant advantage. Additionally, its simplicity ensures good reproducibility. Moreover, the experiment details are well described, further supporting reproducibility.\n\nThe presentation is very clear. Mostly, it’s easy to follow the main idea, interpretations, and experiment results. Specifically, the difference between L2-SP and SPD is clearly explained by comparing the pseudo-codes and providing easy-to-understand interpretations.", "weaknesses": "No obvious weaknesses are observed. One minor issue is that the performance improvement, while consistently observed, is not significantly high. Additionally, I have noted some unclear justifications in the manuscript, as described in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1nSvAAOlWK", "reviewer": "Reviewer_Ex5N", "summary": "This paper proposes a novel weight decay strategy called Selective Projection Decay (SPD).\nSPD selectively imposes a stronger penalty on certain layers, and is designed to improve both in-distribution (ID) and out-of-distribution (OOD) performance.\nThe paper demonstrates the effectiveness of SPD through experiments on several benchmarks.", "questions": "Please see the weaknesses section.", "limitations": "Please see the weaknesses section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- SPD is well-motivated and simple to implement.\n- The paper is well-written and easy to follow. It clearly lays out intuition and motivation. I especially liked sections 3.3 and 3.4, which relate the condition to online hyperparameter optimization and the deviation ratio to a re-interpretation of L2-SP as a projection. I thought the line of logic here was quite clear.\n- The paper demonstrates strong performance on several standard benchmarks for robust fine-tuning.", "weaknesses": "- Overall, I think this is a strong paper, and its strengths outweigh its weaknesses.\nI think the paper could benefit from an ablation experiment, for example, trying Adam-SPD without the condition or benchmarking SPD's performance with optimizers other than Adam.\n- [1] reports that ensembling with the initial weights (whether in model space or weight space) is a simple strategy that improves OOD robustness, and a few recent works [2, 3] report that ensembling continues to improve performance after their proposed fine-tuning strategies. Could the authors validate how the benefits from SPD extend to the ensembling setting?\n\n[1] Robust fine-tuning of zero-shot models\n[2] Finetune like you pretrain: Improved finetuning of zero-shot vision models\n[3] AutoFT: Learning an Objective for Robust Fine-Tuning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hSR7Riy7q5", "reviewer": "Reviewer_CVio", "summary": "The authors propose Selective Projection Decay, a modification on the AdamW optimizer where a regularization penalty is applied that controls for an overall drift from the prior weights such that as long as the overall shift is controlled for but individual layers can potentially adapt differently. The authors provide experiments on OOD generalization tasks and PEFT language modelling fine-tuning.", "questions": "- I think a more interesting way to use the proposed method would be to have $\\lambda$ to be a parameter or layer-specific hyper-parameter rather than a global hyper-parameter. This way, having individual $c_t$ for different components might make the method more robust and in fact help the model adapt relevant weights while keeping specific weights close to the pre-trained model. In many cases, different layers can begin to exhibit different functionalities and therefore there might be more use in having some layers change more than others based on the features that might be captured there.\n- Although not a deal-breaker, I would appreciate a perhaps more in-depth analysis of the language modelling experiments, in particular without PEFT. While I understand the relationship with PEFT and the proposed optimizer, I believe that it would be quite valuable to show how AdamW-SPD works even under naive fine-tuning scenarios.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The explanation of the method is clear and detailed. The method itself is straightforward and intuitive to understand.\n- There are experiments on a number of different tasks and domains that show some improvement upon prior methods in terms of raw performance metrics.", "weaknesses": "- The way the authors frame it, there appears to be limited novelty with the proposed method, with it appearing to simply involve an additional conditional check before applying a potential regularization penalty that has been explored quite a bit in existing literature. I do think that there is something of merit here, however it's hard to pinpoint it under the current framing of the work.\n- The statement \"We recommend starting with  $\\lambda= 1$ and adjusting the strength according to the specific needs\" is rather unsatisfying and doesn't provide the reader with a good idea of whether or not $\\lambda$ is easy to tune. This can probably be resolved by adding additional analysis along this end compare do just Table 2.\n- Experimentally, the results feel somewhat lacking and somewhat misleading. For example, in the Domain-Net experiments, the only comparison is with AdamW, which inherently should lack the exploration capabilities the authors are attempting to account for. There are quite a few baselines that are missing as a result in my opinion, such as sharpness aware minimization (SAM) and/or other exploration based optimizers.\n- For language modelling, it would be more useful to show performance over a multitude of models and sizes, as the results appear to suggest for example that the gap between AdamW and Adam-SPD narrows quite significantly between sizes 7B and 13B, as well as when the PEFT method changes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate the thoughtful comments, especially on suggesting a layer-specific hyper-parameter configuration. This was what we thought would be interesting as well. In the response, we provided clarifications to your concerns and a new experiment. If anything is not clear, we are happy to discuss it during the rebuttal period. \n\n**I think a more interesting way to use the proposed method would be to have to be a parameter or layer-specific hyper-parameter rather than a global hyper-parameter. This way, having individual for different components might make the method more robust and in fact help the model adapt relevant weights while keeping specific weights close to the pre-trained model. In many cases, different layers can begin to exhibit different functionalities and therefore there might be more use in having some layers change more than others based on the features that might be captured there.**\n\nSorry for the confusion! SPD is a layer-specific method, as the reviewer suggested. SPD has a different selecting condition $c_t$ for each layer. In other words, algorithm 2 is displayed for each layer. SPD behaves precisely like the reviewer suggests, and we will update the writing to clarify this. \n \nThe layer-specific $c_t$ allows specific layers to have small regularization and other layers to have large regularization. This gives SPD the edge over other methods by fitting the new dataset (better ID performance) and retraining knowledge from the pre-trained model (better OOD performance). We will update the description to make this point more clear.\n \n**The statement \"We recommend starting with and adjusting the strength according to the specific needs\" is rather unsatisfying and doesn't provide the reader with a good idea of whether or not is easy to tune. This can probably be resolved by adding additional analysis along this end compare do just Table 2.**\n \nThank you for mentioning the ease of tuning this method. This is a contribution of this work. Practically, the method is just as easy to tune as regular weight decay and L2 regularization because the hyper-parameter $\\lambda$ in SPD has the same effect as in those methods, i.e., adjusting regularization strength. In some cases, it can be much easier to tune as shown in the sensitivity analysis in Table 2 of main paper. SPD is much less sensitive to the hyperparameter, consistently yielding good performance on both ID and OOD datasets, whereas L2-SP is more sensitive to the choice of hyperparameter. \n\nMoreover, SPD is even simpler to understand as described in Section 3.4. We re-parametrized L2 regularization (weight decay) as a projection. This gives an intuitive interpretation of the regularization hyper-parameter. For example, in a conventional weight decay setup, we can set the hyper-parameter as $\\lambda=0.01$. We do not necessarily know what $0.01$ means as a hyper-parameter. However, with the re-parametrization in Section 3.4, we can set it to be $\\lambda=1$; this intuitively means that the regularization projects the current model weights back to the deviation of the last update. \n \n\n**Experimentally, the results feel somewhat lacking and somewhat misleading. For example, in the Domain-Net experiments, the only comparison is with AdamW, which inherently should lack the exploration capabilities the authors are attempting to account for. There are quite a few baselines that are missing as a result, in my opinion, such as sharpness aware minimization (SAM) and/or other exploration-based optimizers.**\n\nThank you for bringing up this work. We have added a comparison to SAM on ImageNet (Table 2 in the rebuttal PDF). SPD outperforms SAM in all experiments. SPD is designed explicitly for robust fine-tuning of pre-trained foundation models, whereas SAM does not consider pre-trained initialization. SAM focuses on finding a good local minimum with uniformly low loss. This property could lead to better generalization for regular training but not necessarily a more robust fine-tuned model. Fine-tuning requires careful balancing between fitting the new data and retaining knowledge in the pre-trained model. SPD explicitly considers this problem.\n\nThere may be other exploration-based optimizers to achieve better generalization. However, they are not designed for fine-tuning and generally do not consider the pre-trained initialization in their formulation. Instead, our experiments focus on comparing baselines for robust fine-tuning (Tables 3 and 4 in the main paper). In our setup, AdamW is the best example of an exploration-enhanced optimizer. It is equipped with an adaptive learning rate and momentum. Both mechanisms encourage AdamW to escape local minima and explore further. However, our understanding of exploration may differ from that of the reviewer. We welcome further discussion. \n\n\n**Although not a deal-breaker, I would appreciate a perhaps more in-depth analysis of the language modelling experiments, in particular without PEFT. While I understand the relationship with PEFT and the proposed optimizer, I believe that it would be quite valuable to show how AdamW-SPD works even under naive fine-tuning scenarios.**\n\nWe provide a new experiment for VQA tasks. Specifically, we fine-tune the PaliGemma-3B model on the VQAv2 dataset with LoRA and test it with nine additional VQA datasets consisting of different types of distribution shifts across vision, question, and answer (IV-VQA, CV-VQA, VQA-Rephrasings, VQA-CE, AdVQA, TextVQA, VizWiz and OK-VQA). In Tab.~\\ref{tab:vqa} of the rebuttal PDF, SPD achieves the best ID and average OOD performance. We also show the performance evaluation for both near and far OOD datasets. SPD is consistently more robust under different types and degrees of distribution shifts.\n\nWe acknowledge that full fine-tuning of large language models using SPD would be very interesting. However, our experiments are limited by computation resources and time. It will be an interesting next step in our exploration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4rp3FXOy5r", "reviewer": "Reviewer_Jrpf", "summary": "This paper proposes a new weight decay technique to adapt foundation models to target tasks, focusing on fitting the target data while maintaining the pre-trained knowledge. Specifically, the method, Selective Projection Decay (SPD), selectively imposes a strong penalty on certain layers while allowing others to change freely. Experimentally, the method consistently provides better in-distribution generalization and out-of-distribution robustness across multiple popular vision and language benchmarks.", "questions": "- Why is L2-SP not compared in the experiments shown in Tables 1 and 5? The authors are requested to provide the results for this method as well.\n- Tables 2a and 2b should be converted into figures to enhance readability.\n- Minor Typo: On line 8, \"retraining\" should be corrected to \"retaining\".\n- Several of the results are wrongly highlighted in Table 3.\n- The appendix sections referred in the paper are missing.", "limitations": "The limitation of this work is clearly highlighted: the method's performance depends on the quality of the foundation model.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- This is a well-motivated approach for fine-tuning foundation models on target tasks. By selectively imposing regularization on certain layers, it better fits the target data while retaining the pre-trained knowledge.\n- The method has also proven to be effective for parameter-efficient fine-tuning, which is particularly beneficial. \n- Extensive experiments conducted on both vision and language benchmarks demonstrate the method's strong performance.", "weaknesses": "There are no major weaknesses, please respond to my questions and issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ENQEGtzsg", "reviewer": "Reviewer_33UR", "summary": "The paper proposes a method for robust fine-tuning of deep networks. The proposed method is a variant of L2-SP. The original formulation of L2-SP applied a uniform L2 penalty on the difference between the weights of the pre-trained model and the fine-tuned model. The proposed method (L2-SPD) aims to make this penalty non-uniform for different layers. This is achieved by only applying weight decay to layers where the direction of descent and weight decay are sufficiently similar. The efficacy of the method is demonstrated over various image classification, segmentation and reasoning tasks, showing better ID and OOD performance than baselines.", "questions": "* Is there a way to disentangle the gains from selective projection and adaptive regularization? Related to this, how often does the selective projection kick in in practice, and how does it change as training progresses (this is not crucial to the paper, but is a good-to-have analysis)? \n* I am unable to understand some of the trends in the results on ImageNet. It is not immediately clear to me how regularizing the model to minimize distance from the Zero-Shot model can outperform both the Zero-Shot model and the vanilla FT model on one domain (Im-Sketch), but is in between the two on another domain (Im-Rendition). I had the intuition that L2-SPD will pull the weights to be closer to the Zero-shot model, hence inheriting its robustness and biases, but I could be mistaken. \n* Can this technique of selective regularization be done at a more granular level beyond layers? \n* In section 4.1, why are all domainnet domains not used, omitting the infograph domain? The paper will be helped by a justification of this.\n* For table 2a, the range of deviation is quite different for L2-SP and L2-SPD. While L2-SPD does dominate performance for similar deviations, I wonder if it is possible to use even larger $\\lambda$ and make the deviation very low? Will the same correlation with OOD accuracy still hold?\n* Hyper-parameter selection is not clear for ImageNet. Was k-fold cross validation used? Is the ID val used to select hyper-parameters?\n* On quickdraw, will more aggressive regularization yield better OOD accuracy? Since the pretrained model is supposed to be closer to other domains? \n* For ImageNet, what is the linear layer for final classification?\n* How does the method compare against a simpler baseline like Wise-FT?", "limitations": "The authors have addressed limitaions of their method, pointing out worse performance if the base model is not aligned with the OOD distribution.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The projection interpretation is very interesting. \n* The exposition of the method and its insights is very nice.\n* The insight behind the method is simple, but the method is widely applicable for an important problem.\n* The method is well compatible with LoRA style approaches, increasing its practical applicability. \n* The method is stable across a wide range of hyper-parameters.\n* Wide range of benchmarking is performed, and the empirical results are strong.", "weaknesses": "There are some minor issues with the presentation - \n* It would be nice to have a short preliminaries section to define all notations used in the paper, along with their dimensions. \n* It is not clear from Alg 2 how SPD is selective to different layers. It appears that $\\theta_t$ represents the entire model’s parameters, and $c_t$ is a single scalar for the entire model. Hence, it seems like the “weight decay” only happens for the entire model if the total gradient is in the opposite direction of the weight decay. I might be incorrect here, but I am unable to see where the group structure comes in (possibly $c_t$ is separate for each layer?)\n\nApart from this, some of the experiments have the following weaknesses -  \n* The Himmelbau example is not fully clear to me either. It would be interesting to compare the trajectories against L2-SP to actually understand the “selective penalization”, since regular L2-SP will also penalise deviations similarly. I tried running the code that the authors provided, and with some tuning of the hyper-parameters I was able to get similar trajectories with L2-SP as well. \n* Lack of ablation studies on the two components of the method. It would be good to understand the gains from selective projection and adaptive regularization separately as well.  \n* I am unable to understand some of the trends in the results on ImageNet. It is not immediately clear to me how regularizing the model to minimize distance from the Zero-Shot model can outperform both the Zero-Shot model and the vanilla FT model on one domain (Im-Sketch), but is in between the two on another domain (Im-Rendition). I had the intuition that L2-SPD will pull the weights to be closer to the Zero-shot model, hence inheriting its robustness and biases, but I could be mistaken. \n* Minor - The caption for Table 3 states that \"SPD... beats L2-SP by 8.8%\", however, I cannot understand how this was computed.\n* Minor - While the paper compares against L2-SP, it misses Elastic Weight Consolidation, a similar method using Fisher information, which also adaptively sets the regularization parameters for different layers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all of the reviewers for their positive comments on this work's adaptability (dDi9), performance (Ex5N, CVio, Jrpf, 33UR), and insights (Ex5N, 33UR). We aim to provide concrete responses to your questions and clarify your confusion.

The rebuttal PDF includes new experiments and studies requested by the reviewers. The new experiments are summarized below.

*  (Jrpf,33UR) We added L2-SP and Infograph to Table 1 in the main paper. Our method remains the best-performing method. 
* (CVio) We added a comparison to Sharpness-Aware-Minimization (SAM) on ImageNet . SPD outperforms SAM on ImageNet because SAM is not explicitly designed for robust fine-tuning. 
* (CVio) We added a new VQA fine-tuning experiment with 9 additional VAQ OOD datasets with distribution shifts across vision and language. SPD is consistently more robust under different types and degrees
of distribution shifts.  

We hope our response answers your questions and are open to further discussion during the rebuttal period.

Meta-review: The paper introduces a regularisation technique for finetuning large pretrained models by imposing penalties on specific layers to enhance both ID and OOD generalisation. 

The reviewer consensus is an acceptance. They appreciate the simplicity of the method, clear presentation and strong empirical performance. There were some remaining reservations:
- CVio: I still believe that increased analysis of the proposed method (from a more theoretical perspective rather than completely empirical) would be more meaningful given that the nature of the work.
- Jrpf: I agree with the other reviewer that the theoretical perspective is lacking in this work, though the empirical results are strong.

The merits greatly outweigh the concerns in this case. We recommend an acceptance.
Decision: Accept (poster)