Title: Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Fine-tuning

foundation models

language models

robustness
Abstract: Modern optimizers such as AdamW, equipped with momentum and adaptive learning rate, are designed to escape local minima and explore the vast parameter space. This exploration is beneficial for finding good loss basins when training from scratch. It is not necessarily ideal when resuming from a powerful foundation model because it can lead to large deviations from the pre-trained initialization and, consequently, worse robustness and generalization. At the same time, strong regularization on all parameters can lead to under-fitting. We hypothesize that selectively regularizing the parameter space is the key to fitting and retraining the pre-trained knowledge. This paper proposes a new weight decay technique, Selective Projection Decay (SPD), that selectively imposes a strong penalty on certain layers while allowing others to change freely. Intuitively, SPD expands and contracts the parameter search space for layers with consistent and inconsistent loss reduction, respectively. Experimentally, when equipped with SPD, Adam consistently provides better in-distribution generalization and out-of-distribution robustness performance on multiple popular vision and language benchmarks.

Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models

Abstract

Modern optimizers such as AdamW, equipped with momentum and adaptive learning rate, are designed to escape local minima and explore the vast parameter space. This exploration is beneficial for finding good loss basins when training from scratch. It is not necessarily ideal when resuming from a powerful foundation model because it can lead to large deviations from the pre-trained initialization and, consequently, worse robustness and generalization. At the same time, strong regularization on all parameters can lead to under-fitting. We hypothesize that selectively regularizing the parameter space is the key to fitting and retraining the pretrained knowledge. This paper proposes a new weight decay technique, Selective Projection Decay (SPD), that selectively imposes a strong penalty on certain layers while allowing others to change freely. Intuitively, SPD expands and contracts the parameter search space for layers with consistent and inconsistent loss reduction, respectively. Experimentally, when equipped with SPD, Adam consistently provides better in-distribution generalization and out-of-distribution robustness performance on multiple popular vision and language benchmarks. Code available at https://github.com/GT-RIPL/Selective-Projection-Decay.git .

1 Introduction

Modern optimizers, such as Adam [1], LARS [2], and LAMB [3] usually include momentum and adaptive learning rates. They help optimizers avoid local minima and accelerate learning [4, 5] to explore wider parameter spaces. However, we hypothesize that this behavior is not always beneficial for fine-tuning from a well pre-trained foundation model, especially when fine-tuning a few layers is already sufficient for fitting the target data [6, 7, 8, 9]. Several prior works have found that unnecessary exploration will lead to large deviation from the initialization and worse robustness [10, 11], and constraining the deviation can improve a model's generalization on in-distribution (ID) data and robustness to out-of-distribution (OOD) data [12, 13, 14] 1 . For example, L2-SP [13] imposes a regularization term on the distance between the current and pre-trained models. More recently, TPGM [10] and FTP [11] propose to learn different hard constraints for each layer. These new works have demonstrated impressive results on benchmarks. However, they are either difficult to tune, specialized to specific settings, or require significant computation and storage overhead. This motivates us to ask whether a simple few-liner solution exists for this fundamental problem.

We propose re-examining the existing methods and summarizing their findings to find this solution. Starting from the simplest: L2-SP [13]. Specifically, L2-SP adds an L2 regularization term to the original objective function. Formally,

where θ denotes the model parameters, θ 0 the initialization, ˜ L ( θ ) the original objective function, and λ the hyper-parameter for regularization strength. When θ 0 = 0 , L2-SP reduces to an ordinary weight decay. This simple method should be effective enough to constrain the model, as our experiments show it can reduce the deviation between the fine-tuned and pre-trained models (Sec. 4.1). However, it is held back by an important design choice: the penalty is always applied to all model parameters. Our empirical results identify that a large λ prevents every layer from deviating too much and leads to poor fitting, while a small λ cannot provide enough regularization. This significantly limits the otherwise effective design (Sec. 4.1). So, what is missing in this algorithm?

Recent works in robust fine-tuning and parameter-efficient fine-tuning (PEFT) have shown that customizing constraints for each layer and selectively choosing layers for fine-tunning can improve robustness [10, 11, 6]. Inspired by these findings, we hypothesize that selectively imposing the regularization to different layers is the key. Therefore, we propose a simple selective version of L2-SP weight decay: selective projection decay (SPD). This new algorithm innovates in two aspects: a selection condition and a regularization strength ratio . The former determines when to apply regularization to a layer, and the latter determines the strength of regularization for intuitive hyperparameter tuning. Specifically, we derive the selection condition from hyper-optimization [15, 16, 17] by treating the condition as an optimizable parameter (Sec. 3.3), and the regularization strength ratio by re-writing L2-SP as a projection operation (Sec. 3.4). Intuitively, when the condition is met, the algorithm imposes large regularizations on selected layers. This allows the algorithm to avoid unnecessary deviation and simultaneously fit into the fine-tuning data. We test SPD on large-scale computer vision, and NLP benchmarks with popular foundation models and test ID and OOD performance on various distribution and domain shifts. SPD achieves SOTA performance while being much simpler than other competing methods. Our contributions are:

We propose a selective projection decay, a selective variant of the popular L2-SP/weight decay regularization methods, for robust fine-tuning of large foundation models. We show that selectivity is important to make regularization effective.

We conduct a detailed study of ID and OOD performance on image classification and semantic segmentation with natural distribution and domain shifts. SPD improves ID and OOD performance on these benchmarks.

We show that SPD consistently improves the performance of PEFT methods (e.g. LoRA [7] and adapters [9]) on 8 common sense reasoning language tasks with LLaMA-7B (-13B).

2 Related Works

Robust Fine-Tuning with Distance Regularization. Constraining the distance or deviation between the fine-tuned and pre-trained models has been studied in several prior works. L2-SP [13] explicitly adds an L2 norm penalty on the deviation and shows improved ID generalization for fine-tuning. MARS-SP [12] studies different forms of norms as the penalty. It shows that the Matrix Row Sum (MARS) norm can be a superior alternative to the L2 norm. These two methods impose 'soft' penalties and can be less effective [18]. Instead, LCC [18] proposes constraining the deviation through direct projection on the parameters, which also enforces a hard constraint on the Lipschitz continuity of the fine-tuned model. However, LCC is hard to tune because the projection radius is not an intuitive hyper-parameter. Furthermore, using a single projection constraint for all layers is not an ideal strategy [10]. More recently, TPGM [10] proposes to automatically learn the constraints in LCC during fine-tuning, customizing a different projection radius for each layer through a bilevel optimization scheme. FTP [11] further improves the computation efficiency of TPGM by adopting hyper-optimization [15, 16, 17] in its computation. Nevertheless, FTP is still difficult to control because hyper-optimization requires a secondary optimizer with additional optimization hyper-parameters, and the learned regularization can be too strong with no intuitive way to adjust. In contrast, SPD is a much simpler and more intuitive method, which can be implemented with just a few lines of code. The superior controllability makes SPD potentially applicable to more applications.

Parameter Efficient Fine-Tuning (PEFT). PEFT methods such as adapters [9, 8] and LoRA [7] have been proposed to reduce training memory usage and computation complexity. Recent works have found that PEFT methods also provide good robustness because they modify fewer parameters and retain more knowledge of the pre-trained models [11]. Surgical fine-tuning [6] concludes that fine-tuning a selective few layers can improve ID generalization. These new works motivate us to re-evaluate L2-SP and weight decay, often uniformly applied to all layers. We identify that the inferior performance of the simple methods is because of this uniformity, which exhibits a strong trade-off between fitting and regularization. Other robust fine-tuning methods, such as LP-FT [19] and FLYP [20], focus on feature distortion. We will review them in the Appendix 8.1.

Other Robust Fine-Tuning Methods. WiSE-FT [14] discovers that linearly interpolating between the fine-tuned and pre-trained models after fine-tuning can improve out-of-distribution robustness. This demonstrates that a closer distance to the pre-trained model can improve robustness. However, it only applies to models with zero-shot capabilities. Another orthogonal line of research for robust finetuning focuses on feature distortion. LP-FT [19] shows that fine-tuning with a randomly initialized head layer distorts learned features. It proposes a simple two-stage method to train the head layer first and then fine-tune the entire model. FLYP [20] shows that fine-tuning a foundation model using the same objective as pre-training can better preserve the learned features. Our contribution is an optimization method to penalize the derivation between the fine-tuned and pre-trained models explicitly during fine-tuning, which is orthogonal to them.

3 Methods

In this section, we first provide an overview of the Selective Projection Decay (SPD) method and then describe the intuition behind SPD with a numerical example. Finally, we provide a concrete mathematical motivation for our method's algorithmic design.

3.1 Selective Projection Decay (SPD)

Formulation. SPD is a regularization technique that penalizes significant deviation from the pretrained model. We motivate the formulation from an existing method: L2-SP [13] (Eq. 1). L2-SP adds a distance penalty on the deviation between the fine-tuned and pre-trained models. The penalty is applied to all model parameters at all times. A large λ prevents every layer from deviating too much and empirically leads to poor fitting, while a small λ cannot provide enough regularization. This significantly limits the otherwise effective design. We propose a selective version of this simple technique: selective projection decay (SPD). We will examine L2-SP and SPD in Alg. 1 and Alg. 2.

Notations. We follow the notations in prior works [1, 21]. Let m t , v t denote the moving average of the gradient and squared gradient, β 1 , β 2 their hyper-parameters, and α the learning rate.

Alg. 1 shows the Adam optimizer with the L2-SP regularization in Eq. 1. The effects of the regularization are highlighted in blue, also shown in Eq. 2. Intuitively, the regularization leads to an interpolation-like equation 2 . If the product λα = 1 , then θ t ← θ 0 and if λα = 0 , then θ t ← ˜ θ t , where θ 0 and ˜ θ t denote the initialization and the updated model without regularization.

Alg. 2 shows the proposed SPD. There are two changes compared to Alg. 1.

a condition,

a new interpolation-like equation with a multiplier, r t , replacing the learning rate α ,

Compared to L2-SP, SPD only imposes a penalty when the condition is met ( c t < 0 ), and the strength of the penalty is controlled by a hyper-parameter λ and an analytical quantity deviation ratio r t , which we will introduce later.

3.2 Intuition Behind SPD

SPD prioritizes layers with consistent improvement. SPD adds regularization on layers that meet the condition c t < 0 to slow their growth. The condition is determined by the sign of the inner product between two vectors. One vector is the negative gradient direction ( -g t ) , i.e., the descent direction, and the other is the current progress direction ( θ t -1 -θ 0 ) . The inner product between them measures the alignment between the vanilla 3 update direction and the progress so far. When the inner product is positive, the current progress direction generally points to a low loss region, and following it will lead to consistent loss reduction. Conversely, if the inner product is negative, the current progress direction will likely lead to a higher loss region, indicating inconsistent improvement. In this case, SPD will impose a penalty to slow down updates for those layers. Recall that modern optimizers use momentum to escape local minima and explore wider regions. Without this penalty, the model will likely head towards the higher loss region to overcome it. SPD chooses to slow down these layers and prioritizes layers with more consistent loss reduction. We will motivate this strategy in a principled manner and validate it in our experiments.

3.3 Deriving c t from Hyper-Optimization

Previously, we explained the intuition behind SPD. Specifically, we interpreted the condition c t as a measure of alignment and a test of update consistency. Nevertheless, there is a more profound reason why the quantity c t is a natural choice for selective regularization. In this section, we motivate SPD from a more mathematical perspective.

Hyper-Optimization Setup. Hyper-optimization is a technique to optimize hyper-parameters inside an optimizer [15, 16, 17]. They treat the hyper-parameters as trainable parameters and optimize them using another gradient-based optimizer. Let's start from the vanilla Adam with L2-SP algorithm (Alg. 1) and treat the regularization strength hyper-parameter λ as a trainable parameter. To update λ , we need to obtain its gradient by taking a derivative w.r.t. λ after applying it.

Selection Condition c t . Intuitively, if the quantity ∇ λ is negative, applying the update in gradient descent will increase the value of λ , thus increasing the regularization strength of L2-SP. Conversely, a positive quantity will decrease the regularization strength. Therefore, the sign ( -g ⊺ t +1 ( ˜ θ t -θ 0 )) determines the change of regularization strength in the hyper-optimization of λ . Formally, we define the condition c t as,

For memory efficiency, we use ( θ t -θ 0 ) instead of ( ˜ θ t -θ 0 ) because both vectors point in the same direction and won't affect the sign of c t . This allows us to discard ˜ θ t . Otherwise, we need to keep an additional copy in memory. In summary, when c t < 0 , we apply a regularization for that layer as shown in Alg. 2. This calculation is done for each layer, and the regularization is selectively applied.

Alternative Interpretation: We just interpreted the selection condition c t in SPD as a measure of consistency between the current heading direction and the gradient direction. This perspective is more valid when the algorithm has accumulated some updates, i.e., ∥ θ t -θ 0 ∥ 2 ≫ 0 , and less justified when a heading has not been established at the beginning of training. To analyze this, we discuss the behavior SPD from the perspective of stochastic optimization when ∥ θ t -θ 0 ∥ 2 is small at the beginning of training in the Appendix 8.1.

3.4 Deriving r t from Projection

The selection condition c t determines when to apply regularization to which layers. However, one remaining question is the strength of regularization, which is not intuitive to tune. To overcome this, we introduced an analytical quantity, the deviation ratio r t , in Eq. 2 and Alg. 1. In this section, we will motivate it from the perspective of projection.

L2-SP is projection. Projection onto a norm ball is common in constrained optimization. While L2-SP is not a constrained optimization problem, its operation bears similarity to projection. Suppose we project a model ˜ θ t to an L 2 -norm ball with radius γ centered around its initialization θ 0 . The equation of projection is the following,

Equivalently, we can rewrite the equation as,

Now, we can equate this equation to the highlighted L2-SP equation in Eq. 2 and Alg. 1, we can see that if λα = ( 1 -γ max { γ, ∥ θ t -θ 0 ∥ 2 } ) , the regularization is equivalent to projection with radius γ .

Deviation Ratio r t . This equivalence inspires us to define a deviation ratio r t :

where γ t := ∥ ˜ θ t -θ 0 ∥ 2 and γ t := ∥ θ t -1 -θ 0 ∥ 2 denote the current deviation (before regularization) and the previous deviation from the initialization θ 0 , respectively. We use r t in SPD (Alg. 2) to replace the learning rate α in L2-SP (Alg. 1) to make hyper-parameter ( λ ) tuning more intuitive. Specifically, suppose the hyper-parameter λ = 1 , then the regularization in SPD is:

Intuitively, with λ = 1 , the regularization in SPD is equivalent to projection with a radius equal to the previous deviation if the current deviation is larger. In summary:

No regularization ( λ = 0 ): the projection radius is ∥ ˜ θ t -θ 0 ∥ 2 , meaning no projection.

Weak regularization ( 1 ≥ λ > 0 ): the projection radius lies between ∥ ˜ θ t -θ 0 ∥ 2 and ∥ θ t -1 -θ 0 ∥ 2 . Within this range, all layers will expand or remain unchanged.

Strong regularization ( λ > 1 ): the projection radius lies between 0 and ∥ θ t -1 -θ 0 ∥ 2 . In this range, it's possible that regularized layers can contract.

We recommend starting with λ = 1 and adjusting the strength according to the specific needs.

Full Fine-Tuning

PEFT Fine-Tuning

𝑊

!

Pre-trained model

𝑊

"

Fine-tuned model

'Zero' model

(Δ𝑊

"

)

PEFT model

𝑊

"

-𝑊

!

#

Δ𝑊

"

#

3.5 Compatibility with PEFT methods.

As shown in Alg. 2, SPD retains a copy of the pre-trained model in memory. This adds additional memory requirements to the overhead of vanilla optimizers. While this is practical for moderate-sized models, as fine-tuning focuses more and more on large models, additional memory requirements become undesirable. Fortunately, in extremely large models, the prevalent fine-tuning strategy is parameter-efficient fine-tuning (PEFT), such as LoRA [7], series adapters [9], and parallel adapters [8]. SPD is naturally compatible with these methods without the additional memory. Intuitively, SPD selectively projects the current model towards the pre-trained initialization. PEFT methods generally initialize new parameters to add to the original model weights. To recover the behavior of SPD, we can instead project the new parameters towards the origin , equivalent to a selective version of regular weight decay, i.e., replacing θ 0 with 0 in Alg. 2. Consequently, this does not require a memory copy of the pre-trained model. It consistently improves PEFT fine-tuning for large language models on common sense reasoning benchmarks in Sec. 4.4.

For example, LoRA decomposes a linear layer h = W t x into two components, where h ∈ R m × 1 , W t ∈ R m × n and x ∈ R n × 1 are the output, weights, and input of this layer.

where W 0 ∈ R m × n , W up ∈ R m × r and W down ∈ R r × n are the pre-trained model, up-projection and down-projection matrices. If r ≪ min { m,n } , ( W up W down ) is a low-rank approximation of ∆ W t . To regularize the overall deviation ∥ W t -W 0 ∥ 2 , it suffices to regularize ∥ W up W down ∥ 2 to be close to zero. In this case, SPD acts as selective weight decay on W up and W down individually.

In summary, we propose selective projection decay (SPD) to impose strong regularization on layers during fine-tuning selectively. As shown in Fig. 1, SPD regularizes the deviation of the fine-tuned model from the pre-trained model ∥ W t -W 0 ∥ 2 for full fine-tuning and the deviation from the origin ∥ ∆ W t ∥ 2 for PEFT fine-tuning.

4 Experiments

We test Selective Projection Decay on a diverse set of benchmarks, architectures, and tasks to demonstrate its effectiveness. We will test both ID generalization and OOD robustness across various domain and distribution shifts.

Image Classification. We first analyze the behavior of SPD on conventional image classification datasets DomainNet [22] and ImageNet [23]. We use a CLIP ViT-Base model for both experiments as the pre-trained initialization [24]. Specifically, DomainNet consists of images from several domains with 345 classes. We fine-tune on one domain and test on all domains. ImageNet is a large-scale dataset with 1000 classes. We fine-tune on ImageNet and test on ImageNet and four variants, namely ImageNet-V2 [25], ImageNet-A [26], ImageNet-R [27], and ImageNet-S [28].

Semantic Segmentation. We further test SPD on the PASCAL-Context semantic segmentation dataset [29]. Following prior works [30, 11], we use a Swin ViT-Tiny [31], pre-trained on ImageNet22K, and Segformer [32] segmentation architecture. To construct the OOD datasets, we follow the popular natural robustness literature [33] and apply four representative image corruptions (fog, defocus blur, Gaussian noise, and brightness) with 5 severity each. We fine-tune on the clean segmentation data and test on clean and corrupted data.

Common Sense Reasoning. Moreover, we show that SPD can benefit PEFT fine-tuning on large language models (LLMs). We use the Commonsense-170K dataset [34], which consists of training data from eight common sense reasoning benchmarks. Following the prior work [34], we fine-tune LLaMa-7B (-13B) [35] using LoRA [7], series adapters [9], and parallel adapters [8].

Visual Question Answering. Finally, we demonstrate SPD's superiority on multi-modal task. We use Google's recently released PaliGemma [36] pretrained on a broad mixture of large-scale visionlanguage tasks. We fine-tune on VQAv2 [37] and test on nine OOD datasets using LoRA [7]. For the near OODs, we evaluate on VQAv2's six variants, namely IV-VQA [38], CV-VQA [38], VQA-Rephrasings [39], VQA-CP v2 [40], VQA-CE [41] and AdVQA [42], which cover uni-modal, multi-modal and adversarial distribution shifts from VQAv2. We also include TextVQA [43], VizWiz [44] and OK-VQAv2 [45], which are constructed from different sources than VQAv2, as the far OOD datasets.

4.1 DomainNet Experiments

In this section, we utilize the DomainNet benchmark to test our claims. Specifically, we will show that Adam-SPD consistently outperforms AdamW in OOD robustness across multiple domains, and this is due to a much smaller deviation from the pre-trained model. Furthermore, by sweeping across a range of hyper-parameters, we show that uniform regularization, such as L2-SP fails to provide adequate constraints, while Adam-SPD shows robust performance.

Small deviation correlates with better OOD performance. Earlier, we hypothesized that a significant deviation can lead to worse OOD performance. Theoretically, prior works [12, 11] have shown that large deviations from the initialization result in a large Liptschtz constant and, hence, worse robustness. In Tab. 1, we present a comprehensive study by fine-tuning a pre-trained CLIP model on different domains from DomainNet separately and reporting test results on all domains. Across all domains, Adam-SPD consistently achieves higher OOD performance and shows noticeably less deviation. This empirical result corroborates with prior works' findings and our hypothesis.

Selective regularization exhibits stronger deviation-robustness correlation. In Tab. 2, we compare the behavior of L2-SP and SPD using DomainNet. Specifically, we fine-tune a CLIP model on the Clipart domain (ID domain) and report performance on Clipart and other domains (OOD domains). In Tab. 2a, we observe that while L2-SP successfully restrains the model's deviation from its initialization, it does not effectively improve OOD performance. With a very large regularization, the ID performance deteriorates as well. On the contrary, SPD effectively restrains the model's deviation

(a) L2-SP hyper-parameter ( λ ) sweep. Stronger regularizations (larger values) decrease deviation; however, they do not improve OOD performance and even deteriorate ID performance.

(b) Adam-SPD hyper-parameter ( λ ) sweep. Stronger regularizations (larger values) decrease deviation, simultaneously improving OOD performance. The ID performance is not impacted significantly.

and significantly improves OOD performance while matching the best ID performance. Under SPD, the correlation coefficient between OOD performance and deviation is -0 . 96 , which indicates a strong negative correlation between the two quantities, i.e., smaller deviation and higher OOD accuracy. This experiment shows that selective regularization is superior to uniform regularization.

Training Details. We use the vision transformer public repository for DEIT [46] to fine-tune all methods. We use λ = 1 for all Adam-SPD results in Tab. 1. More details are in Appendix 8.4.

4.2 ImageNet Experiments

SPD outperforms more complicated works on image classification. Following the training recipe from the prior work [11], we fine-tune a CLIP ViT-Base model on ImageNet using Adam-SPD. We use the same hyper-parameters as the prior work and only adjust the regularization hyperparameter in SPD. In Tab. 3, we observe that Adam-SPD provides the best ID performance (strong ID generalization) and best average OOD performance (strong OOD robustness) on four ImageNet variants. SPD achieves a level of competitive performance with just a few lines of code. SPD's simplicity and strong performance show that selective regularization is a fundamental improvement for robust fine-tuning.

Training Details. For Adam-SPD, we fine-tune the model with a learning rate of 3 e -5 and λ = 1 . 4 . The regularization hyper-parameter is found through cross-validation, and the model with the best ID validation accuracy is taken. More details are in Appendix 8.4.

4.3 PASACAL Dense Semantic Segmentation

SPD outperforms more complicated works on semantic segmentation. The same trend is observed on semantic segmentation in Tab. 4. Again, SPD achieves the best ID generalization and OOD robustness across four different corruptions. This shows that proper regularization is not only important for achieving strong ID generalization (performance on the test set) but also for strong OOD robustness (performance on distribution shifted test sets) to domains shift (Tab. 3) and distribution shift such as natural corruptions (Tab. 4). The model fine-tuned with SPD is consistently more robust across different levels of corruption and severity.

Training Details. For Adam-SPD, we fine-tune the model with a learning rate of 1 e -4 and λ = 2 . 2 . The regularization hyper-parameter is found through cross-validation, and the model with the best ID validation accuracy is taken. More details are in Appendix 8.4.

4.4 LLaMA PEFT Fine-Tuning Experiments

SPD is compatible and consistently improves PEFT methods. Previous experiments have shown that SPD imposes effective regularization for full fine-tuning. Furthermore, SPD can also improve the performance of PEFT methods. We fine-tune LLaMa-7B (-13B) models on the Commonsense-170k dataset [34]. As shown in Tab. 5, SPD consistently improves regular fine-tuning with AdamW, which uses a uniform weight decay for all tested PEFT methods. This demonstrates that selective regularization benefits full fine-tuning and PEFT fine-tuning. Combined with its simplicity, SPD can potentially improve generalization and robustness for more tasks in deep learning.

Training Details. We follow the training code released by a prior work [34]. We report the best performance from the original paper and compare them with Adam-SPD. More details are in Appendix 8.4.

4.5 Visual Question Answering (VQA) Experiments

Table 6: Visual Question Answering Result using PaliGemma-3B. SPD outperforms baselines across ID, near OOD and far OOD datasets using LoRA. Note that L2-SP reduces to Vinilla FT with AdamW under LoRA.

SPD shows competitiveness across ID, near OOD, and far OOD datasets on multimodal tasks. Apart from uni-modal tasks, SPD outperforms other baselines on multi-modal tasks. We fine-tune PaliGemma-3B model on VQAv2 [37] dataset with LoRA. In Tab. 6, SPD improves vanilla finetuning and other robust fine-tuning methods, achieving best ID and average OOD performance w.r.t. distribution shifts across single modalities such as vision, question, answer and combinations of multiple modalities. We also show the performance evaluation for both near and far OOD datasets. SPD is consistently more robust under different types and degrees of distribution shifts.

Training Details. For Adam-SPD, we fine-tune the model with a learning rate of 1 e -3 and λ = 0 . 5 . The regularization hyper-parameter is found through cross-validation, and the model with the best ID validation accuracy is taken. More details are in Appendix 8.4.

5 Limitations

SPD is a selective regularization technique explicitly designed for fine-tuning. While it can be theoretically used for pre-training, it will likely lead to poor performance because it will hinder the training of some layers. For fine-tuning, it works well because the pre-trained foundation model is assumed to be a good initialization, and only small changes in a selected few layers can lead to a good local minimum. Furthermore, the level of performance gain depends on how well the foundation models are exposed to the fine-tuning and OOD data distributions during pre-training. For example, in the DomainNet experiment (Tab. 1), fine-tuning a CLIP ViT model on any other domain does not have reasonably good OOD robustness on the Quickdraw domain. One can deduce that Quickdraw is not well represented in the pre-training data of CLIP ViT.

6 Conclusion

Fine-tuning differs from training from scratch because it starts from a good initialization. Therefore, effective regularization is critical to retaining the knowledge of the pre-trained foundation model while fitting a model to the target distribution. We identified that 1) regularization is necessary to keep the fine-tuned model close to its initialization and maintain robustness; 2) uniform regularization can hurt model fitting if regularization is too strong. In this paper, we proposed selective projection decay (SPD), a selective version of the popular weight decay/L2-SP regularization method. With an additional few lines of code, SPD can be integrated into existing optimizers and performs selective regularization. It demonstrates superior regularization performance on different tasks and modalities in our experiments.

References

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888 , 2017.

Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962 , 2019.

Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning , pages 1139-1147. PMLR, 2013.

Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 , 2016.

Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. arXiv preprint arXiv:2210.11466 , 2022.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Shwai He, Liang Ding, Daize Dong, Miao Zhang, and Dacheng Tao. Sparseadapter: An easy approach for improving the parameter-efficiency of adapters. arXiv preprint arXiv:2210.04284 , 2022.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning , pages 2790-2799. PMLR, 2019.

Junjiao Tian, Xiaoliang Dai, Chih-Yao Ma, Zecheng He, Yen-Cheng Liu, and Zsolt Kira. Trainable projected gradient method for robust fine-tuning. arXiv preprint arXiv:2303.10720 , 2023.

Junjiao Tian, Yen-Cheng Liu, James S Smith, and Zsolt Kira. Fast trainable projection for robust fine-tuning. Advances in Neural Information Processing Systems , 36, 2024.

Henry Gouk, Timothy M Hospedales, and Massimiliano Pontil. Distance-based regularisation of deep networks for fine-tuning. ICLR , 2021.

LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International Conference on Machine Learning , pages 28252834. PMLR, 2018.

Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7959-7971, 2022.

Kartik Chandra, Audrey Xie, Jonathan Ragan-Kelley, and Erik Meijer. Gradient descent: The ultimate optimizer. Advances in Neural Information Processing Systems , 35:8214-8225, 2022.

Xiang Wang, Shuai Yuan, Chenwei Wu, and Rong Ge. Guarantees for tuning the step size using a learning-to-learn approach. In International Conference on Machine Learning , pages 10981-10990. PMLR, 2021.

Matthias Feurer and Frank Hutter. Hyperparameter optimization. Automated machine learning: Methods, systems, challenges , pages 3-33, 2019.

Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning , 110:393-416, 2021.

Ananya Kumar et al. Fine-tuning can distort pretrained features and underperform out-ofdistribution. ICLR , 2022.

Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. arXiv preprint arXiv:2212.00638 , 2022.

Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems , 33:18795-18806, 2020.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1406-1415, 2019.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning , pages 5389-5400. PMLR, 2019.

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15262-15271, 2021.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8340-8349, 2021.

Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems , 32, 2019.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV , 2010.

Yen-Cheng Liu, Chih-Yao Ma, Junjiao Tian, Zijian He, and Zsolt Kira. Polyhistor: Parameterefficient multi-task adaptation for dense vision tasks. arXiv preprint arXiv:2210.03265 , 2022.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems , 34:12077-12090, 2021.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 , 2019.

Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933 , 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. PaliGemma: A versatile 3B VLM for transfer, July 2024. arXiv:2407.07726 [cs] version: 1.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering, May 2017. arXiv:1612.00837 [cs].

Vedika Agarwal, Rakshith Shetty, and Mario Fritz. Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editing. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9687-9695, Seattle, WA, USA, June 2020. IEEE.

Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. Cycle-Consistency for Robust Visual Question Answering, February 2019. arXiv:1902.05660 [cs].

Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering, June 2018. arXiv:1712.00377 [cs].

Corentin Dancette, Remi Cadene, Damien Teney, and Matthieu Cord. Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering, September 2021. arXiv:2104.03149 [cs].

Sasha Sheng, Amanpreet Singh, Vedanuj Goswami, Jose Alberto Lopez Magana, Wojciech Galuba, Devi Parikh, and Douwe Kiela. Human-Adversarial Visual Question Answering, June 2021. arXiv:2106.02280 [cs].

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA Models That Can Read, May 2019. arXiv:1904.08920 [cs].

Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. VizWiz: nearly real-time answers to visual questions.

Benjamin Z. Reichman, Anirudh Sundar, Christopher Richardson, Tamara Zubatiy, Prithwijit Chowdhury, Aaryan Shah, Jack Truxal, Micah Grimes, Dristi Shah, Woo Ju Chee, Saif Punjwani, Atishay Jain, and Larry Heck. Outside Knowledge Visual Question Answering Version 2.0. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5, June 2023. ISSN: 2379-190X.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers and distillation through attention. In International Conference on Machine Learning , volume 139, pages 10347-10357, July 2021.

Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199 , 2021.

Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting few-sample bert fine-tuning. arXiv preprint arXiv:2006.05987 , 2020.

Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In International Conference on Machine Learning , pages 404-413. PMLR, 2018.

Guillaume Garrigos and Robert M Gower. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint arXiv:2301.11235 , 2023.

Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review , 60(2):223-311, 2018.

Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648 , 2016.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6023-6032, 2019.

Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven C. H. Hoi. Lavis: A library for language-vision intelligence, 2022.

8 Appendix

8.1 Extended Related Works

Other Robust Fine-Tuning Methods. WiSE-FT [14] discovers that linearly interpolating between the fine-tuned and pre-trained models after fine-tuning can improve out-of-distribution robustness. This demonstrates that a closer distance to the pre-trained model can improve robustness. However, it only applies to models with zero-shot capabilities. Another orthogonal line of research for robust finetuning focuses on feature distortion. LP-FT [19] shows that fine-tuning with a randomly initialized head layer distorts learned features. It proposes a simple two-stage method to train the head layer first and then fine-tune the entire model. FLYP [20] shows that fine-tuning a foundation model using the same objective as pre-training can better preserve the learned features. Our contribution is an optimization method to penalize the derivation between the fine-tuned and pre-trained models explicitly during fine-tuning, which is orthogonal to them.

8.2 Interpreting c t as an Early Layer Selection Criterion

In previous sections, we interpreted the selection condition c t in SPD as a measure of consistency between the current heading direction and the gradient direction. This perspective is more valid when the algorithm has accumulated some updates, i.e., ∥ θ t -θ 0 ∥ 2 ≫ 0 , and less justified when a heading has not been established at the beginning of training. This section discusses SPD from the perspective of stochastic optimization when ∥ θ t -θ 0 ∥ 2 is small at the beginning of training.

Inner product of gradients captures gradient variance. Modern deep learning models are trained by stochastic optimization techniques, e.g., mini-batch SGD, leading to stochasticity due to sampling. We first show that the inner product of gradients captures the variance of a sampling process. We invoke a common assumption in the convergence analysis of stochastic gradient descent [1, 49, 21]. Assuming that the stochastic gradient g t is a stationary process G over a short period, with a small step size, successive gradients, e.g., g t , g t +1 , can be seen as samples drawn from the same distribution G . Given two successive draws g 1 and g 2 , we can approximate the first and second moment of G .

Define the variation of gradients as V ar ( g ) := E [ ∥ g -¯ g ∥ 2 ] [50, 51], where ¯ g := E [ g ] ,we can show that

Remarks. Eq. 13 shows that the inner product of two consecutive stochastic gradients, under certain assumptions, can be seen as the estimator for the difference between the gradient norm and the variance of gradients. When the inner product is negative, this indicates that the variance outweighs the magnitude of the gradient.

SPD prioritizes layers with higher expected gain. At the beginning of training, the heading direction ( θ 1 -θ 0 ) is dominated by early gradients. For example, at t = 2 the direction of ( θ 1 -θ 0 ) is the same as -g 1 in Adam. The sign of -g ⊺ 2 ( θ 1 -θ 0 ) is the same as the sign of g ⊺ 2 g 1 . This shows that the condition c t captures the difference between gradient norm and gradient variance. With this interpretation, we show that c t reflects expected performance gain in stochastic optimization. To see it, we can invoke the descent lemma for SGD. For an L-smooth function f ( W ) [50], the descent lemma for SGD states that,

where η k ≤ 2 L is the learning rate.

Remarks. The term on the left hand side E [ f ( θ k +1 )] -f ( θ k ) is the expected performance improvement for each step. Ideally, this should be a negative quantity. On the right-hand side, we observe

that improvement depends on two quantities ∥ ¯ g k ∥ 2 and V ar ( g k ) . To lower the upper bound, we want a large ∥ ¯ g k ∥ 2 and a small V ar ( g k ) . According to the decoupling Eq. 13, the inner product between successive gradients approximates this proportionality. Consequently, a negative c t likely indicates a higher upper bound on the expected gain, meaning a smaller improvement. Therefore, SPD will prioritize layers with potentially larger expected gains.

8.3 Approximation in L2-SP

Algorithm 4: (Original) Adam with L2Regularization

The Adam with L2-SP Regularization algorithm in the main paper is not the precise mathematical implementation of the original formulation written in Eq. 1. To see the difference, we compare ours and the accurate implementation here in Alg. 3 and Alg. 4. In our implementation, we replaced θ t -1 -θ 0 (Alg. 4) with ˜ θ t -θ 0 (Alg. 3). This is done intentionally to improve memory efficiency when transitioning to the selective version (see Adam-SPD in Sec. 3.3). We can make the following substitution to see how this modification changes computation. Starting from our implementation,

We can further combine the λα 2 ̂ m t √ ̂ v t + ϵ into the update of ˜ θ t . The new ˜ θ t is

Therefore, our implementation of L2-SP adds a minor additional dampening of the learning rate α by a factor of (1 -λα ) .

What if we followed the original implementation of L2-SP as in Alg. 4 ? This would change the condition c t in the main paper (Eq. 3) to

At the current time step t , we only have access to the parameters θ t -1 from the previous step t -1 . To calculate the c t in Eq. 16, we would have to store the weights from two steps back in memory. This increases memory consumption of the algorithm. As we have shown in Eq. 15, our implementation only differs from the original implementation slightly but reduces memory consumption. Therefore, we decided to make this approximation.

8.4 Training Details

DomainNet. We use the vision transformer public repository for DEIT [46] to fine-tune all methods. Standard augmentations are used for all: weight-decay ( 0 . 1 ), drop-path ( 0 . 2 ) [52], label-smoothing ( 0 . 1 ) [53], Mixup ( 0 . 8 ) [54] and Cutmix ( 1 . 0 ) [55]. The learning rate is 2 e -5 and trained for 60 epochs for Tab. 1 and 30 epochs for Tab. 2. We use λ = 1 for all Adam-SPD results in Tab. 1. We use 1 A40 GPU for each experiment.

ImageNet. The same procedure as the DomainNet experiment is used for training the ImageNet models. Standard augmentations are used for all: weight-decay ( 0 . 1 ), drop-path ( 0 . 2 ) [52], labelsmoothing ( 0 . 1 ) [53], Mixup ( 0 . 8 ) [54] and Cutmix ( 1 . 0 ) [55]. We fine-tune all methods for 30 epochs and use the best hyper-parameters reported by the prior work [11]. For Adam-SPD, we fine-tune the model with a learning rate of 3 e -5 and λ = 1 . 4 . The regularization hyper-parameter is found through cross-validation, and the model with the best ID validation accuracy is taken. We use 2 A40 GPUs for each experiment.

Pascal Segmentation. We follow the training code released by a prior work [30]. We fine-tune all methods for 60 epochs and use the best hyper-parameters reported by the prior work. For Adam-SPD, we fine-tune the model with a learning rate of 1 e -4 and λ = 2 . 2 . The regularization hyper-parameter is found through cross-validation, and the model with the best ID validation accuracy is taken. We use 4 2080Ti GPUs for each experiment.

Commonsense-170K. We follow the training code released by a prior work [34]. We report the best performance from the original paper and compare them with Adam-SPD. For Adam-SPD, we fine-tune the model with an identical hyper-parameter setup as the released code and only adjust the regularization strength λ . The regularization hyper-parameter is found through cross-validation, and the model with the best ID validation loss is taken. We use 1 A40 GPU for each experiment.

Visual Question Answering. We follow the LAVIS [56] public repository to fine-tune all methods. We use the PaliGemma [36] model pretrained with 224 ∗ 224 input images and 128 token input/output text sequences and fine-tune with the precision of bfloat16. Standard hyper-parameters are used for all: learning rate ( 1 e -3 ), weight-decay ( 1 e -4 ), optimizer (AdamW), scheduler (Linear Warmup With Cosine Annealing), warm-up learning rate ( 1 e -4 ), minimum learning rate ( 1 e -4 ), accumulation steps ( 2 ), beam size (5). The model is trained for 10 epochs with a batch size of 16 for Tab. 6. For LoRA [7], we limit our study to only adapting the attention weights and freeze the MLP modules for parameter-efficiency, specifically apply LoRA to W q , W k , W v , W o with r = 8 in Tab. 6. We use λ = 0 . 5 for SPD. The regularization hyper-parameter is found through cross-validation, and the model with the best ID validation accuracy is taken. We use 8 A40 GPU for each experiment.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: The work does not present an obvious negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for robust fine-tuning of deep networks. The proposed method is a variant of L2-SP. The original formulation of L2-SP applied a uniform L2 penalty on the difference between the weights of the pre-trained model and the fine-tuned model. The proposed method (L2-SPD) aims to make this penalty non-uniform for different layers. This is achieved by only applying weight decay to layers where the direction of descent and weight decay are sufficiently similar. The efficacy of the method is demonstrated over various image classification, segmentation and reasoning tasks, showing better ID and OOD performance than baselines.\n\nSTRENGTHS:\n* The projection interpretation is very interesting. \n* The exposition of the method and its insights is very nice.\n* The insight behind the method is simple, but the method is widely applicable for an important problem.\n* The method is well compatible with LoRA style approaches, increasing its practical applicability. \n* The method is stable across a wide range of hyper-parameters.\n* Wide range of benchmarking is performed, and the empirical results are strong.\n\nWEAKNESSES:\nThere are some minor issues with the presentation - \n* It would be nice to have a short preliminaries section to define all notations used in the paper, along with their dimensions. \n* It is not clear from Alg 2 how SPD is selective to different layers. It appears that $\\theta_t$ represents the entire model’s parameters, and $c_t$ is a single scalar for the entire model. Hence, it seems like the “weight decay” only happens for the entire model if the total gradient is in the opposite direction of the weight decay. I might be incorrect here, but I am unable to see where the group structure comes in (possibly $c_t$ is separate for each layer?)\n\nApart from this, some of the experiments have the following weaknesses -  \n* The Himmelbau example is not fully clear to me either. It would be interesting to compare the trajectories against L2-SP to actually understand the “selective penalization”, since regular L2-SP will also penalise deviations similarly. I tried running the code that the authors provided, and with some tuning of the hyper-parameters I was able to get similar trajectories with L2-SP as well. \n* Lack of ablation studies on the two components of the method. It would be good to understand the gains from selective projection and adaptive regularization separately as well.  \n* I am unable to understand some of the trends in the results on ImageNet. It is not immediately clear to me how regularizing the model to minimize distance from the Zero-Shot model can outperform both the Zero-Shot model and the vanilla FT model on one domain (Im-Sketch), but is in between the two on another domain (Im-Rendition). I had the intuition that L2-SPD will pull the weights to be closer to the Zero-shot model, hence inheriting its robustness and biases, but I could be mistaken. \n* Minor - The caption for Table 3 states that \"SPD... beats L2-SP by 8.8%\", however, I cannot understand how this was computed.\n* Minor - While the paper compares against L2-SP, it misses Elastic Weight Consolidation, a similar method using Fisher information, which also adaptively sets the regularization parameters for different layers.\n\nQUESTIONS:\n* Is there a way to disentangle the gains from selective projection and adaptive regularization? Related to this, how often does the selective projection kick in in practice, and how does it change as training progresses (this is not crucial to the paper, but is a good-to-have analysis)? \n* I am unable to understand some of the trends in the results on ImageNet. It is not immediately clear to me how regularizing the model to minimize distance from the Zero-Shot model can outperform both the Zero-Shot model and the vanilla FT model on one domain (Im-Sketch), but is in between the two on another domain (Im-Rendition). I had the intuition that L2-SPD will pull the weights to be closer to the Zero-shot model, hence inheriting its robustness and biases, but I could be mistaken. \n* Can this technique of selective regularization be done at a more granular level beyond layers? \n* In section 4.1, why are all domainnet domains not used, omitting the infograph domain? The paper will be helped by a justification of this.\n* For table 2a, the range of deviation is quite different for L2-SP and L2-SPD. While L2-SPD does dominate performance for similar deviations, I wonder if it is possible to use even larger $\\lambda$ and make the deviation very low? Will the same correlation with OOD accuracy still hold?\n* Hyper-parameter selection is not clear for ImageNet. Was k-fold cross validation used? Is the ID val used to select hyper-parameters?\n* On quickdraw, will more aggressive regularization yield better OOD accuracy? Since the pretrained model is supposed to be closer to other domains? \n* For ImageNet, what is the linear layer for final classification?\n* How does the method compare against a simpler baseline like Wise-FT?\n\nLIMITATIONS:\nThe authors have addressed limitaions of their method, pointing out worse performance if the base model is not aligned with the OOD distribution.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper proposes a new weight decay technique to adapt foundation models to target tasks, focusing on fitting the target data while maintaining the pre-trained knowledge. Specifically, the method, Selective Projection Decay (SPD), selectively imposes a strong penalty on certain layers while allowing others to change freely. Experimentally, the method consistently provides better in-distribution generalization and out-of-distribution robustness across multiple popular vision and language benchmarks.

STRENGTHS:
- This is a well-motivated approach for fine-tuning foundation models on target tasks. By selectively imposing regularization on certain layers, it better fits the target data while retaining the pre-trained knowledge.
- The method has also proven to be effective for parameter-efficient fine-tuning, which is particularly beneficial. 
- Extensive experiments conducted on both vision and language benchmarks demonstrate the method's strong performance.

WEAKNESSES:
There are no major weaknesses, please respond to my questions and issues.

QUESTIONS:
- Why is L2-SP not compared in the experiments shown in Tables 1 and 5? The authors are requested to provide the results for this method as well.
- Tables 2a and 2b should be converted into figures to enhance readability.
- Minor Typo: On line 8, "retraining" should be corrected to "retaining".
- Several of the results are wrongly highlighted in Table 3.
- The appendix sections referred in the paper are missing.

LIMITATIONS:
The limitation of this work is clearly highlighted: the method's performance depends on the quality of the foundation model.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Selective Projection Decay, a modification on the AdamW optimizer where a regularization penalty is applied that controls for an overall drift from the prior weights such that as long as the overall shift is controlled for but individual layers can potentially adapt differently. The authors provide experiments on OOD generalization tasks and PEFT language modelling fine-tuning.\n\nSTRENGTHS:\n- The explanation of the method is clear and detailed. The method itself is straightforward and intuitive to understand.\n- There are experiments on a number of different tasks and domains that show some improvement upon prior methods in terms of raw performance metrics.\n\nWEAKNESSES:\n- The way the authors frame it, there appears to be limited novelty with the proposed method, with it appearing to simply involve an additional conditional check before applying a potential regularization penalty that has been explored quite a bit in existing literature. I do think that there is something of merit here, however it's hard to pinpoint it under the current framing of the work.\n- The statement \"We recommend starting with  $\\lambda= 1$ and adjusting the strength according to the specific needs\" is rather unsatisfying and doesn't provide the reader with a good idea of whether or not $\\lambda$ is easy to tune. This can probably be resolved by adding additional analysis along this end compare do just Table 2.\n- Experimentally, the results feel somewhat lacking and somewhat misleading. For example, in the Domain-Net experiments, the only comparison is with AdamW, which inherently should lack the exploration capabilities the authors are attempting to account for. There are quite a few baselines that are missing as a result in my opinion, such as sharpness aware minimization (SAM) and/or other exploration based optimizers.\n- For language modelling, it would be more useful to show performance over a multitude of models and sizes, as the results appear to suggest for example that the gap between AdamW and Adam-SPD narrows quite significantly between sizes 7B and 13B, as well as when the PEFT method changes.\n\nQUESTIONS:\n- I think a more interesting way to use the proposed method would be to have $\\lambda$ to be a parameter or layer-specific hyper-parameter rather than a global hyper-parameter. This way, having individual $c_t$ for different components might make the method more robust and in fact help the model adapt relevant weights while keeping specific weights close to the pre-trained model. In many cases, different layers can begin to exhibit different functionalities and therefore there might be more use in having some layers change more than others based on the features that might be captured there.\n- Although not a deal-breaker, I would appreciate a perhaps more in-depth analysis of the language modelling experiments, in particular without PEFT. While I understand the relationship with PEFT and the proposed optimizer, I believe that it would be quite valuable to show how AdamW-SPD works even under naive fine-tuning scenarios.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper proposes a novel weight decay strategy called Selective Projection Decay (SPD).
SPD selectively imposes a stronger penalty on certain layers, and is designed to improve both in-distribution (ID) and out-of-distribution (OOD) performance.
The paper demonstrates the effectiveness of SPD through experiments on several benchmarks.

STRENGTHS:
- SPD is well-motivated and simple to implement.
- The paper is well-written and easy to follow. It clearly lays out intuition and motivation. I especially liked sections 3.3 and 3.4, which relate the condition to online hyperparameter optimization and the deviation ratio to a re-interpretation of L2-SP as a projection. I thought the line of logic here was quite clear.
- The paper demonstrates strong performance on several standard benchmarks for robust fine-tuning.

WEAKNESSES:
- Overall, I think this is a strong paper, and its strengths outweigh its weaknesses.
I think the paper could benefit from an ablation experiment, for example, trying Adam-SPD without the condition or benchmarking SPD's performance with optimizers other than Adam.
- [1] reports that ensembling with the initial weights (whether in model space or weight space) is a simple strategy that improves OOD robustness, and a few recent works [2, 3] report that ensembling continues to improve performance after their proposed fine-tuning strategies. Could the authors validate how the benefits from SPD extend to the ensembling setting?

[1] Robust fine-tuning of zero-shot models
[2] Finetune like you pretrain: Improved finetuning of zero-shot vision models
[3] AutoFT: Learning an Objective for Robust Fine-Tuning

QUESTIONS:
Please see the weaknesses section.

LIMITATIONS:
Please see the weaknesses section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new regularization scheme called Selective Projection Decay (SPD), which selectively imposes penalties on layers where the current progress direction does not align with the vanilla update direction. This method is compatible with PEFT methods, including LoRA-type algorithms, making it practical for end users. Experiments show that ADAM equipped with SPD consistently outperforms baselines, including L2 regularization, in terms of in-distribution generalization and out-of-distribution robustness on multiple popular vision and language benchmarks.\n\nSTRENGTHS:\nThe simplicity of the proposed method is its major strength, making it easy for many users to adopt. Compatibility with parameter-efficient fine-tuning is a significant advantage. Additionally, its simplicity ensures good reproducibility. Moreover, the experiment details are well described, further supporting reproducibility.\n\nThe presentation is very clear. Mostly, it’s easy to follow the main idea, interpretations, and experiment results. Specifically, the difference between L2-SP and SPD is clearly explained by comparing the pseudo-codes and providing easy-to-understand interpretations.\n\nWEAKNESSES:\nNo obvious weaknesses are observed. One minor issue is that the performance improvement, while consistently observed, is not significantly high. Additionally, I have noted some unclear justifications in the manuscript, as described in the questions section.\n\nQUESTIONS:\n#1. How does the performance compare when using EWC regularization for fine-tuning, since EWC is generally better than L2 regularization?\n\n#2. Comparing the performance of L2-SP and Adam-SPD in hyper-parameter sweeping experiments seems challenging since these are reported in tables. Perhaps heatmaps could be a clearer alternative to visualize and highlight the results.\n\n#3. The authors argue that SPD selectively imposes penalties on large deviations in a layer-wise manner. However, in Algorithm 2, there is no layer-wise information. It appears that all parameters are updated based on the alignment score denoted by c_t. From my understanding of Algorithm 2, if the alignment is high (i.e., c_t is smaller than zero), the updated parameters are further adjusted by the interpolation-like equation. If c_t is larger than zero, do we use the vanilla ADAM update, since \\theta_t has already been updated before calculating c_t? Am I correctly understanding Algorithm 2? In such cases, it seems that the parameters shouldn’t be updated, since SPD will impose a penalty to slow down updates for those layers with negative c_t.\n\n#4. In the LLaMA PEFT experiments, there is no baseline with L2 regularization or its advanced versions (such as EWC). While I expect that ADAM with SPD is better than AdamW, I would like to see if there are also improvements compared to ADAM with L2 regularization in PEFT settings.\n\nLIMITATIONS:\nThe authors state that the proposed approach only works well if the pretrained model is good enough to provide good initial parameters. No clear negative societal impacts have been observed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}