Title: PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Generalization

Regularization

Transfer Learning

Parameter-Efficient Fine-Tuning
Abstract: Parameter-Efficient Fine-Tuning (PEFT) effectively adapts pre-trained transformers to downstream tasks. However, the optimization of tasks performance often comes at the cost of generalizability in fine-tuned models. To address this issue, we theoretically connect smaller weight gradient norms during training and larger datasets to the improvements in model generalization. Motivated by this connection, we propose reducing gradient norms for enhanced generalization and aligning fine-tuned model with the pre-trained counterpart to retain knowledge from large-scale pre-training data. Yet, naive alignment does not guarantee gradient reduction and can potentially cause gradient explosion, complicating efforts to manage gradients. To address such an issue, we propose PACE, marrying generalization of PArameter-efficient fine-tuning with Consistency rEgularization. We perturb features learned from the adapter with the multiplicative noise and ensure the fine-tuned model remains consistent for same sample under different perturbations. Theoretical analysis shows that PACE not only implicitly regularizes gradients for enhanced generalization, but also implicitly aligns the fine-tuned and pre-trained models to retain knowledge. Experimental evidence supports our theories. PACE surpasses existing PEFT methods in visual adaptation tasks (VTAB-1k, FGVC, few-shot learning, domain adaptation) showcasing its potential for resource-efficient fine-tuning. It also improves LoRA in text classification (GLUE) and mathematical reasoning (GSM-8K).

PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization

Abstract

Parameter-Efficient Fine-Tuning (PEFT) effectively adapts pre-trained transformers to downstream tasks. However, the optimization of tasks performance often comes at the cost of generalizability in fine-tuned models. To address this issue, we theoretically connect smaller weight gradient norms during training and larger datasets to the improvements in model generalization. Motivated by this connection, we propose reducing gradient norms for enhanced generalization and aligning finetuned model with the pre-trained counterpart to retain knowledge from large-scale pre-training data. Yet, naive alignment does not guarantee gradient reduction and can potentially cause gradient explosion, complicating efforts to manage gradients. To address such an issue, we propose PACE, marrying generalization of PArameterefficient fine-tuning with Consistency rEgularization. We perturb features learned from the adapter with the multiplicative noise and ensure the fine-tuned model remains consistent for same sample under different perturbations. Theoretical analysis shows that PACE not only implicitly regularizes gradients for enhanced generalization, but also implicitly aligns the fine-tuned and pre-trained models to retain knowledge. Experimental evidence supports our theories. PACE surpasses existing PEFT methods in visual adaptation tasks (VTAB-1k, FGVC, few-shot learning, domain adaptation) showcasing its potential for resource-efficient finetuning. It also improves LoRA in text classification (GLUE) and mathematical reasoning (GSM-8K). The code is available at github.com/MaxwellYaoNi/PACE.

1 Introduction

Transformers [68], with the self-attention mechanism [3] capturing long-range dependencies in data, succeed in various deep learning tasks, including image classification (ViT [16]), multimodal learning (CLIP [55]), image synthesis (StableDiffusion [57]), semantic segmentation (SAM [33]) and text generation (LLaMA [65]). The success of transformers can be largely attributed to the availability of abundant data, such as ImageNet [11] and Laion5B [60], which empower researchers to scale up these models by training them under an enormous number of parameters.

Such huge models, with knowledge from large-scale pre-training [63], constitute on foundation models that can be easily adapted to various downstream tasks through full fine-tuning or linear probing [20], eliminating the need for task-specific model design [8]. However, full fine-tuning is storage-intensive and infeasible for maintaining separate model weights as the number of tasks grows, while linear probing, which only trains the last head layer, yields inferior adaptation performance.

To overcome these limitations, Parameter-Efficient Fine-Tuning (PEFT) [24] fine-tunes only a small subset of parameters, thereby reducing storage requirements while surpassing the performance of

full fine-tuning and linear probing. These advantages have popularized PEFT and inspired the development of various PEFT methods for deep learning tasks, which can be categorized into two groups: those increasing inference cost and cost-efficient ones. The first group introduces additional learning branches, such as non-linear adapters [25, 8], or concatenates learnable parameters with input tokens, e.g ., visual prompts [28, 82, 52], increasing inference cost. The second group, focuses on cost-efficiency by lower-rank adaptation in linear layers [7, 26], or affine transformations such as SSF [41] and RepAdapters [45], which can be reparameterized during inference for efficiency.

Despite the superiority and efficiency of PEFT, prioritizing optimization for downstream tasks compromises the generalizability of fine-tuned models, yielding suboptimal performance. Although some analyses have been conducted on PEFT [63, 27, 18, 72, 39], they fail to fully explain the generalization of PEFT, leading to ineffective strategies for improving generalization.

To address this gap in understanding generalization in PEFT, we establish a theoretical connection from generalization theory: smaller weight gradient norms and larger data volumes contribute to better generalization. Motivated by this, we propose reducing weight gradient norms and aligning output space of the fine-tuned model with the pre-trained one to retain knowledge captured from large pre-training data. Yet, theoretical analyses reveal this naive alignment does not guarantee gradient regularization and can even cause gradient explosion, complicating efforts for gradient management. To address this issue, we propose perturbing features learned from the adapter with multiplicative noise and constraining the network output to be consistent across different perturbations.

Our method, called PACE, marries generalization of PArameter-efficient fine-tuning with Consistency rEgularization. Its name, PACE, reflects our goal of keeping the output behavior of the fine-tuned model in pace with the pre-trained one. Despite its simplicity, theoretical analysis confirms that PACE not only implicitly regularizes weight gradients for better generalization but also implicitly aligns the fine-tuned model with the pre-trained counterpart to retain knowledge from large-scale pre-training data. Experimental evidence supports our theories. PACE improves existing PEFT methods, achieving superior results across six adaptation benchmarks. Our key contributions are:

We establish a theory connecting smaller weight gradient norms and larger datasets with enhanced generalization, motivating gradient reduction and model alignment for fine-tuning.

We propose PACE, a simple yet effective method perturbing features from adapters with multiplicative noise and constraining output of fine-tuned model to be consistent across perturbations.

Our theoretical and empirical evidence confirms that PACE implicitly regularizes gradients and aligns the fine-tuned model with the pre-trained one. PACE excels on 4 visual adaptation tasks.

We provide novel theoretical explanations of how gradient penalization and consistency regularization benefit generalization, offering fundamental insights applicable across deep learning.

2 Related work

Parameter-Efficient Fine-Tuning (PEFT). LoRA [26] uses low-rank decomposition to reduce parameters and treats adapters as side paths. SSF [41] proposes affine transformations on latent features. FacT [30] decomposes and reassembles parameter matrices in ViT. Surgical fine-tuning [36] of different network parts improves adaptation to distribution shifts. FLoRA [74] performs a batched low-rank adaptation. GLoRA [7] unifies cost-efficient PEFT methods. NOAH [82] uses parameter search on neural prompts. ARC [14] leverages cross-layer ViT similarity, parameter-sharing adapter and scaling factors for lower fine-tuning cost. RLRR [15] incorporates a residual term for flexibility while preserving pre-trained representation. RepAdapter [45] reparameterizes adapters for efficient inference. Res-tuning [29] unbinds tuners from the backbone for memory efficiency. Zhao et al . [84] show impressive fine-tuning results by tuning layernorm in attention. OFT [54] and BOFT [42] propose orthogonal fine-tuning to preserve hypersphere energy between neurons.

Consistency Regularization. Fixmatch [61] applies consistency regularization over augmented images for semi-supervised learning. Openmatch [59] utilizes it on outlier predictions for open-set semi-supervised learning. R-Drop [76] applies it to transformers [68] with dropout for NLP tasks. CR [79] applies it over augmented real and fake images for GAN training. CAGAN [50] enforces consistency on discriminators with dropout for GAN training. Despite the empirical success of consistency regularization demonstrated by previous works, theoretical analysis is lacking. While NICE [47] demonstrates that consistency regularization lowers latent feature gradients for stable

GAN training, it fails to reveal reduced weight gradient for enhanced generalization. Our study goes beyond prior works by providing a theoretical link between smaller weight gradients and improved generalization, effectively marrying generalization of PEFT with consistency regularization.

Generalization of Fine-Tuning. Li et al . [38] constrain the fine-tuned model's closeness to the pretrained model in weight space. Fu et al . [18] induce sparsity on PEFT for better generalization. Wang et al . [72] studies generalization of PEFT fine-tuning graph neural network. Zhang et al . [83] employ rank-1 gradient boosting (GB) updates supported by the GB theoretical framework. VioLET [73], PromptSRC [31] and CoPrompt [58] naively align the fine-tuned model with the pre-trained one for enhanced generalization or avoiding forgetting. Additionally, L2SP [77], DELTA [40], and FTP [64] aim to retain pre-trained knowledge by aligning fine-tuned models with pre-trained ones, reducing distance in weight space, feature space and using projected gradient descent, respectively. However, they fail to provide a theoretical analysis for this alignment. Our study goes beyond understanding generalization of PEFT by discovering the benefits of gradient regularization and model alignment. We propose PACE to match both requirements, paving a comprehensive understanding for PEFT.

Gradient regularization. Previous studies have empirically shown that gradient regularization improves performance [67, 85, 48, 49] and adversarially robust accuracy [13]. However, they lack theoretical connection between smaller gradient norms and better generalization [17, 81, 6]. We bridge this gap by establishing a fundamental theory between reduced gradient norms and improved generalization, providing a solid foundation for future research on enhancing generalization.

3 Approach

We begin with a unified perspective on cost-efficient PEFT based on GLoRA [7], linking generalization with gradients and large-scale data, and motivating the alignment of the fine-tuned model with the pre-trained model to leverage its knowledge. We identify limitations of naive alignment in gradient regularization and introduce PACE, which implicitly enhances gradient regularization and model alignment. We conclude with theoretical justification and efficient implementations.

3.1 A unified perspective on cost-efficient PEFT methods

The transformer architectures [68, 16] have excelled in natural language processing and computer vision tasks through their powerful sequential modeling capabilities. This success stems from their ability to process text/image tokens through L transformer blocks, where each block contains selfattention and MLP modules primarily composed of linear layers. These linear layers enable the self-attention mechanism to capture long-range dependencies, allowing transformers to achieve superior performance when scaled to a huge number of parameters and trained on extensive datasets.

With massive parameters, pre-trained on large-scale data, transformers serve as foundation models that can be fine-tuned for downstream tasks using limited data. However, fully fine-tuning all parameters for various downstream tasks requires substantial memory and can lead the forgetting of pre-trained knowledge. To alleviate this without increasing inference cost, adapters with lightweight parameters are often preferred for fine-tuning. Let ¯ h 0 ( · ) be a transformation within the pre-trained transformer. Current adapters can be unified as introducing a residual branch ∆¯ h to form a new transformation ¯ h :

Here, a is the input and ¯ h 0 ( · ) can represent MLP modules, as in Adapter [25] and AdaptFormer [8], or linear layers in self-attention and MLP modules, as in [26, 7, 12, 34]. In SSF [41], ¯ h 0 ( · ) is the identity mapping and ∆¯ h ( a ) = a ⊙ ( γ -1 ) + β with γ and β as affine transformation parameters.

Given that linear layers are key components in transformer, tuning them offers a flexible and effective way to adapt models to downstream tasks. This work focuses on methods that tune the linear layer without increasing inference cost. Let ( W 0 , b 0 ) , (∆ W , ∆ b ) , and ( W , b ) be the parameters of pre-trained model, adapter and fine-tuned model, respectively, where W 0 , ∆ W , W ∈ R d out × d in and b 0 , ∆ b , b ∈ R d out . Fine-tuning a linear layer in self-attention or MLP module can be formed as:

Based on GLoRA [7], cost-efficient PEFT methods for linear layers vary in the form of ∆ W , ∆ b :

LoRAadd : ∆ W = W d W u , ∆ b = b lora where W d ∈ R d out × r , W u ∈ R r × d in , and r is the rank.

VPTadd : ∆ W is zero, ∆ b = W 0 P , with learnable P ∈ R d in × 1 as layer-wise visual prompt. We use VPTadd to differentiate from VPT [28], which concatenates P with tokens, increasing inference cost.

LoRAmul : ∆ W = W 0 ⊙ ( W d W u ) , ∆ b = b 0 ⊙ b lora, including RepAdapter [45] via reparameterization.

3.2 Generalization of deep neural networks

Having established a unified perspective on cost-efficient PEFT, we now motivate our method from a perspective on improving generalization of neural networks to enhance performance on unseen data. Consider a network f := ϕ ( g ( x )) with l layers, where g is feature extractor and ϕ is the classification head. Let θ := { ( W ( i ) , b ( i ) ) } l i =1 be the parameter set with dimension d and D n := { ( x i , y i ) } n i =1 be the training set of size n drawn i.i.d. from distribution D , which contains infinite data. The following lemma from [17] explains the relationship between the empirical and population loss.

Lemma 1 (Theorem 1 from [17]) Let L D n ( θ ) be the empirical loss function over f on training set D n and L D ( θ ) be the population loss. For any ρ > 0 , with high probability over D n ∼ D , we have

where R : ( R + , R + ) → R + is an increasing function (under conditions on L D ( θ ) and n as in §B.5).

Lemma 1 bounds the population loss by the empirical loss with perturbed weights, indicating that a minimal empirical loss increase from small weight perturbations implies low population loss.

By observing that the maximum of L D n is achieved at ϵ = ρ ∇ θ ∥ ∇ θ ∥ 2 , where ∇ θ is the gradient of L D n at θ , and performing a Taylor expansion of L D n around θ , we formulate the following theorem.

Theorem 1 Denote ∇ θ as the gradient and λ H max as the largest eigenvalue of the Hessian matrix H θ of L D n at θ . For any ρ > 0 , with high probability over training set D n ∼ D , we have

Here, higher-order terms from the Taylor expansion are incorporated into R ( ∥ θ ∥ 2 2 ρ 2 , 1 n ) , which is related to weights norm and inversely related to the training data size n .

Theorem 1 (proof in §B.1) outlines strategies for enhancing generalization. They involve regularizing weight norms and the largest Hessian eigenvalues, and crucially, increasing data size n and reducing the weight gradient norms (illustrated in Figure 1). However, excessive reduction should be avoided as it could impair network's representation capacity, yielding higher empirical and population loss.

3.3 Motivation and limitation of aligning the fine-tuned model with the pre-trained model

Theorem 1 emphasizes that large-scale data and smaller gradient magnitudes are essential for better generalization in neural network training. Therefore, aligning the fine-tuned model with the pretrained one is crucial, as it ensures retention of knowledge obtained from large-scale data, preserving generalization. PEFT methods, often outperforming full fine-tuning, achieve this alignment by limiting the number of trainable parameters, restricting the model's capacity to deviate from the pretrained one. However, the training objective prioritizes downstream task performance, compromising alignment with pre-trained knowledge. While sparsity regularization [18] and weight decay on adapter weights help, they do not ensure alignment, as even small weight changes can lead to significant divergence in output space [75, 21, 17]. Therefore, we propose to achieve the alignment by reducing the FP-distance (output distance between fine-tuned and pre-trained models on training samples):

where θ , θ 0 , ∆ θ ∈ R d are parameters for the fine-tuned model, pre-trained model and the adapter.

While reducing FP-distance keeps the fine-tuned model close to the pre-trained model, thus preserving its knowledge, it does not ensure reduced gradient magnitudes, leading to suboptimal generalization.

minimum

Sharper

minimum

𝑓(𝜽

!

)

𝑓(𝜽

!

+Δ𝜽)

𝑓(𝜽

!

+𝒛

"

⊙Δ𝜽)

𝑓(𝜽

!

+𝒛

#

⊙Δ𝜽)

Δ𝜽

Large gradient &

Hessian norms

Small gradient &

Hessian norms

Large

difference

2𝒛

2𝒛

Large grad norm

Small grad norm

𝑓(𝜽)

𝑓(𝜽)

PACE

Small

difference

𝑓(𝜽

!

)

𝑓(𝜽

!

+Δ𝜽)

Large FP-distance

Small FP-distance

PACE

Theorem 1

Theorem 2

Theorem 3

To understand the gradient-related limitations in this alignment, we assume ∆ θ is small enough for a Taylor expansion approximation. Following standard practices [17, 80, 2], we perform the expansion up to the second-order terms. Given the independence between elements in squared L 2 distances (§B.4) and to simplify our theories, we analyze a one-dimensional output for a single i.i.d. sample, which leads us to the following proposition.

Proposition 1 Assuming ∆ θ is small, denote f ( θ ) ∈ R as the one-dimensional output for x , with ∇ and H as its gradient and Hessian at θ . FP-distance over x can be decomposed as follows:

Prop. 1 establishes the relationship between weight gradients, adapter weights, and FP-distance. However, it remains unclear if it regulates gradients. Our experiments show that minimizing FPdistance can sometimes increase gradient magnitude, complicating efforts for managing gradient.

3.4 Consistency regularization

To achieve better generalization by both regularizing gradients and aligning the fine-tuned model with the pre-trined model, we propose a consistency regularization loss for f , encouraging invariance of f to the same input under varying multiplicative noise perturbations on the adapter weights, as follows:

where z 1 , z 2 ∼ N ( 1 , σ 2 I ) is the multiplicative noise applied on adapter weight. To understand the generalization benefits in this consistency regularization, we simplify the analysis by focusing on one-dimensional output for a single sample, resulting in the following theorem.

Theorem 2 Using notations from Prop. 1, let f ( θ 0 + z ⊙ ∆ θ ) ∈ R be the one-dimensional output for x . Define ∆ θ j as j -th element in ∆ θ , ∇ j as the j -th element in ∇ and H jk as the ( j, k ) -entry in H . With z 1 , z 2 ∼ N ( 1 , σ 2 I ) , the consistency loss over x can be approximated as:

Theorem 2 (proof in §B.2) shows that the consistency regularization essentially penalizes the first- and second-order gradients of f at θ (illustrated in Figure 1), with the regularization strength controlled by the noise variance σ 2 and adaptively influenced by the magnitude of elements in adapter weight ∆ θ . Thus, minimizing the consistency loss implicitly regularizes the gradients, improving generalization.

With the FP-distance in Prop. 1 and consistency loss in Theorem 2, we establish their relationship as:

Theorem 3 With d as the dimension of θ , Eq. 6 can be upper-bounded as:

||𝑓

!

𝒙 -𝑓

"

(𝒙)||

"

𝒙

Transformer

Block

loss

𝑓

!

(𝒙)

𝑓

"

(𝒙)

share weights

non-shared noises

Transformer block with adapter perturbed by noise

Consistency regularization between two outputs of x

head

Multi-Head

Attention

Norm

MLP

Norm

𝑾

#

𝒃

#

Δ𝑾

∆𝒃

ℎ

#

(⋅ )

∆ℎ(⋅)

ℎ(⋅) = ℎ

#

(⋅ ) + 𝒛 ⊙ Δℎ(⋅)

where

𝒛 ∼ 𝒩(𝟏, 𝜎

"

𝑰)

×𝐿

Transformer

Block

head

×𝐿

Transformer

Adapter

∆ℎ

and

pre-trained

ℎ

#

in linear layer

ℎ

Block

Theorem 3 (proof in B.3) establishes the relationship between Eq. 6 and Eq. 8, showing Eq. 6 is upperbounded by terms involving ∥ ∆ θ ⊙ ∇ ∥ 2 2 and ∥ (∆ θ ∆ θ T ) ⊙ H ∥ 2 F which appear in Eq. 8. Reducing these terms results in a decrease in Eq. 6. Thus minimizing the consistency loss implicitly aligns the fine-tuned and pre-trained models (illustrated in Figure 1), preserving pre-trained knowledge.

3.5 Efficient implementation of PACE

Providing different weight perturbations for each input in a mini-batch increases memory and computational demands. To avoid this, we perturb feature outputs from the adapter ∆ h ( · ) , effectively simulating perturbation that shares noise across each row in the weight matrix ∆ W . Our simple pipeline is shown in Figure 2. Consider X ∈ R B × T × d in as a batch of data where B and T are the batch and token sizes. The calculation for the linear layer of the fine-tuned model, which utilizes pre-trained weights W 0 , b 0 and adapter weights ∆ W , ∆ b , processes an output size of d out as:

Operator ⊙ is the element-wise multiplication after expanding the left matrix Z ∈ R B × d out ∼ N ( 1 , σ 2 I ) into B × T × d out where tokens within the same example share the same noise. Motivated by [37], the σ decreases linearly as block depth increases. Let f 1 ( · ) and f 2 ( · ) be two networks share same weights but do not share the noise patterns. The loss function for PACE is:

where ℓ is the classification loss and λ is a hyperparameter controlling regularization strength. During inference, noise and regularization are ommitted, ∆ W , ∆ b are integrated with W 0 , b 0 for efficiency:

Efficient PACE variants. In §C, we present two variants that match the computational/memory costs of the baseline while achieving superior performance with substantially reduced resources.

4 Experiments

We combine LoRAmul and VPTadd to form a strong baseline LoRAmul+VPTadd, outperforming other combinations in most cases. We evaluate our method across four visual classification adaptation tasks: VTAB-1K [78], few-shot learning [30], FGVC [28] and domain adaptation [82]. We demonstrate PACE improves LoRA on GLUE [70] for text classification and GSM-8K [9] for text generation.

Datasets and evluations. VTAB-1K comprises 19 datasets organized into (i) Natural images, (ii) Specialized datasets (remote sensing, medical) and (iii) Structured datasets (scene structure) domains. Each dataset has 1K training examples. Following [78, 28], we use the provided 800-200 train split for hyperparameter selection, evaluate using the full training set and report average accuracy across three trails. Few-shot learning involves 5 fine-grained datasets: FGVC-Aircraft [46], Food101 [4], OxfordFlowers102 [51], OxfordPets [53] and StanfordCars [35]. Following [30], we evaluate 1,

2, 4, 8 and 16 shots, train on the provided training set, tune hyperparameters using validation and report average test accuracy over three random seeds. FGVC includes 5 fine-grained datasets: CUB200-2011 [69], NABirds [66], OxfordFlowers [51], StanfordDogs [10] and StanfordCars [35]. We follow [28] to use validation set for hyperparameter and report test results. For domain adaptation , following [82, 7], we train on ImageNet [11] with a 16-shot setting, use the validation split by [82] for hyperparameter selection and report the results on the official validation set and 4 out-of-domain datasets: ImageNet-Sketch [71], ImageNet-V2 [56], ImageNet-A [23] and ImageNet-R [22]. We evaluate on GLUE [70] for text classification and GSM-8K [9] for mathematical reasoning .

Pre-trained backbones . We experiment with two vision transformers, Vision Transforms (ViT-B/16) [16] and Swin Transformer (Swin-B) [44]. These two are pre-trained on ImageNet-21K [11]. We test a ViT-B-Laion-IN12K model, pre-trained on Laion-2B [60] and fine-tuned on ImageNet-12K [11]. We use RoBERTabase [43] and Phi-3-mini-4k-instruct [1] for text classification and generation.

Implementation details . We follow [28] for image processing: 224 × 224 resizing for VTAB-1K; random flips and crops to 224 × 224 for FGVC and few-shot learning; stronger augmentation for domain adaptation task, following [16, 82, 41]. We use the Adam optimizer [32] with cosine learning rate decay and linear warm-up (first 10 epochs). Models are fine-tuned for 300 epochs on VTAB-1K and 100 epochs on other vision adaptation tasks, with batch size 64. For text classification we follow [26]. See §G for mathematical reasoning details. All experiments used an NVIDIA H100 GPU.

Baseline. For each dataset, we identified the better method (LoRAmul+VPTadd or LoRAadd) and tuned the rank, learning rate, and weight decay to form a strong baseline. The detailed baseline settings for each task and the number of trainable parameters are provided in §F, where LoRAmul+VPTadd generally outperformed other variants. Building on the strong LoRAmul+VPTadd, we use the grid search for our λ and σ , following strategies from previous studies [28, 41, 26]. Beyond LoRAmul+VPTadd, PACE also enhances PEFT methods such as AdaptFormer, GLoRA, COFT, and BOFT (§D.4).

4.1 Comparison with the State of the Arts

Results on VTAB-1K. Table 1 presents the results comparing PACE with recent state-of-the-art PEFT methods. PACE improves the strong baseline by 2.6% accuracy, surpassing the previous SOTA GLoRA [7] by 1%, which uses two stages for parameter search. In §D.1, we show that reducing training epochs to 50 or 100 has minimal impact on PACE performance.

Results on Few-shot Learning . Table 2 compares performance w/ and w/o our PACE. PACE improves LoRAadd, VPTadd, LoRAmul+VPTadd, with LoRAmul+VPTadd +PACE performing best in most cases. PACE yields notable improvement, especially when the number of shot is small.

Results on FGVC. Table 3 shows that PACE improves the strong LoRAmul+VPTadd by 0.7%, outperforming SSF [41], ARC [14] and RLRR [15] that use strongly pre-trained ViT with augmentations. In §D.2, PACE achieves larger improvements on smaller datasets.

Results on domain adaptation. Table 4 compares PACE with others. LoRAmul+VPTadd outperforms GLoRA [7] which relies on parameter search. Meanwhile, PACE improves LoRAmul+VPTadd by 1.5%, outperforming other PEFT methods, demonstrating superior performance on domain adaptation.

Results on text classification and mathematical reasoning. Table 5 shows that PACE outperforms LoRA by 1% on GLUE text classification and by 3.11% on GSM-8K mathematical reasoning.

Generalization on other backbones. We evaluate PACE on CIFAR-100 (VTAB-1K) and domain adaptation using Swin-B [44] pre-trained on ImageNet-21K and ViT-B (pre-trained on Laion 2B, then fine-tuned on ImageNet-12K). Table 7 shows PACE outperforms LoRAmul+VPTadd and other PEFT methods across all backbones, demonstrating its strong generalizability. Further experiments in §D.3 show PACE works effectively with self-supervised models such as MAE [19] and DINO [5].

4.2 Analyses

To verify our theories, we conduct experiments on CIFAR-100 (VTAB-1K) using ViT-B/16 and Camelyon (VTAB-1K) on Swin-B. Figures 3 & 4 show the gradient norm (summed across all layers) and FP-distance (Eq. 5) and the train & validation accuracy during training for baseline LoRAmul+VPTadd and PACE on validation set. Figures 3a & 4a show that PACE has a smaller gradient norm than baseline, verifying Theorem 2 that PACE can implicitly lower the weight gradient norm for better generalization. Figures 3b & 4b demonstrate that PACE maintains a lower FP-distance than the baseline, verifying Theorem 3 that PACE can implicitly align the fine-tuned model with pre-trained model, retaining knowledge from large-scale pre-training. Owing to the advantages of the gradient regularization and model alignment, PACE shortens the performance gap between seen and unseen data, yielding higher accuracy on the unseen validation set, as shown in Figures 3c & 4c.

100

200

300

epoch=

3e3

6e3

9e3

12e3

‖

∂f

∂θ

‖

2

Baseline

+PACE

100

200

300

epoch=

20

60

100

140

D fp

Baseline

+PACE

100

200

300

epoch=

70

80

90

100

Acc

train acc

val acc

Baseline

+PACE

(a) Gradient Norm.

(b) FP-Distance

(c) Train and validation accuracy.

100

200

300

epoch=

2e3

4e3

6e3

8e3

‖

∂f

∂θ

‖

2

Baseline

+PACE

100

200

300

epoch=

10

40

70

100

D fp

Baseline

+PACE

100

200

300

epoch=

0.85

0.90

0.95

1.00

Acc

train acc

val acc

Baseline

+PACE

(a) Gradient Norm.

(b) FP-distance

(c) Train and validation accuracy.

To clarify why naive alignment is problematic, we vary the regularization strength λ over a wide range (1e-3 to 5e4) for both Fine-tuned Pre-trained model Alignment (FPA) by minimizing D fp in Eq. 5 and PACE. Figure 5 shows the averaged gradient norm over training (see also Figures 8 & 9 for more visualizations). PACE robustly lowers gradient norms with larger λ , while FPA exhibits unpredictable behavior, even causing gradient explosion. This verifies Prop. 1 that minimizing D fp is problematic for gradient regularization, complicating gradient management.

1e-3 5e-3 0.01 0.05

0.1

0.5

1

5

10

50

100 500

1e3

5e3

1e4

5e4

λ

=

1e4

2e4

‖

∂f

∂θ

‖

2

Baseline

+FPA

+PACE

2

3

4

5

6

7

8

M

=

78.0

78.5

79.0

1

2

4

6

8

10

12

N

=

78.0

78.5

79.0

4.3 Ablation studies

We ablate PACE based on the baseline LoRAmul+VPTadd on CIFAR-100 (VTAB-1K) and ImageNet1K in domain adaption as shown in Table 8. The ablations include Noise (baseline w/ noise perturbing

adapter), PACEadd (replacing the multiplicative noise with the additive noise), PACE h (perturbing h ( · ) instead of ∆ h ( · ) in Eq. 11), PACEdrop (replacing the Gaussian noise with the dropout noise), PACE σ = (all transformer blocks share the same σ ), PACE σ ↑ ( σ increases linearly with depth), FPA (fine-tuned and pre-trined alignment by minimizing Eq. 5), SAM (sharpness-aware minimization [17]), GP (gradient penalization), ℓ 1 (sparsity regularization), and transfer learning methods L2SP [77], DELTA [40] and FTP [64]. We grid-search hyperparameters and report the best results.

Table 8 presents the results for all variants. PACE improves over Noise, which itself is better than baseline, justifying our adapter perturbation and consistency regularization. PACEadd performs worse than PACE, showing the superiority of the multiplicative noise. Although PACE h can implicitly regularize gradients, it performs worse than PACE, verifying the advantages of perturbing adapter to implicitly align models. PACEdrop is worse than PACE, indicating the dropout noise is suboptimal. PACE σ = and PACE σ ↑ perform worse, justifying our design of linearly decreasing σ . FPA, SAM and GP, which either only align models or only regularize gradients, are outperformed by PACE. Despite combining FPA+GP, it still performs worse than ours, suggesting ineffective combination. ℓ 1 , L2SP, DELTA, and FTP obtain worse results than PACE, showing their limitations in improving generalization. PACE regularizes gradients for better generalization and aligns models to retain knowledge, surpassing all other variants.

Method

CIFAR

ImageNet-1K

-100

Source -Sketch -V2

-A

-R

LoRAmul+VPTadd

74.9

78.3

30.6

68.5 14.1 32.5

+Noise

77.4

78.3

31.3

68.6 14.3 33.0

+PACE

79.0

79.0

31.8

69.4

16.3

35.2

+PACEadd

75.7

78.3

31.2

68.7 13.7 32.7

+PACE

h

75.9

78.4

31.2

68.1 13.8 32.6

+PACEdrop

78.3

78.9

31.2

68.9 16.0 34.6

+PACE

σ

=

77.9

78.8

31.6

68.3

16.6

34.7

+PACE

σ

↑

77.3

78.7

31.3

68.9 14.0 33.6

+FPA

76.6

78.8

31.2

68.6 14.7 33.5

+SAM [17]

75.4

78.4

31.4

68.5 13.8 32.9

+GP

75.8

78.3

31.7

68.4 14.2 32.1

+FPA+GP

74.9

78.1

31.5

68.1 13.5 32.6

+

ℓ

1

75.2

78.2

30.6

68.6 13.7 32.8

+L2SP [77]

75.9

78.5

30.4

68.7 14.9 33.5

+DELTA [40]

76.4

78.4

30.8

68.7 14.6 33.7

+FTP [64]

76.2

78.6

30.8

68.6 15.8 33.6

0.1

0.2

0.5

1.0

1.5

σ

=

0.02

0.05

0.1

0.2

0.5

λ

1-Shot

0.1

0.2

0.5

1.0

1.5

σ

=

0.02

0.05

0.1

0.2

0.5

λ

2-Shot

0.1

0.2

0.5

1.0

1.5

σ

=

0.02

0.05

0.1

0.2

0.5

λ

4-Shot

0.1

0.2

0.5

1.0

1.5

σ

=

0.02

0.05

0.1

0.2

0.5

λ

8-Shot

66

69

72

75

84

85

86

87

89

90

90

91

We further evaluate applying PACE across multiple M networks during training or applying it lazily with half-batch size at every N steps (PACE half lazy in §C). Figure 6 presents the results, showing that applying PACE among two networks at every training step performs best. However, lazy regularization applied every few steps can still provide reasonable results while saving computational/memory costs.

We test the sensitivity of hyperparameters λ and σ introduced in our PACE on OxfordPets for few-shot learning across 1, 2, 4, 8 shots. The results presented in Figure 7 demonstrate that with less data, larger λ and σ are favored, verifying the effectiveness of PACE in improving generalization.

5 Conclusions

We have introduced PACE, a novel and effective method that combines generalization of PArameterefficient fine-tuning with Consistency rEgularization. Through rigorous theoretical analyses, we have shown PACE reduces weight gradient for improved generalization and it aligns the fine-tuned model with the pre-trained model for retaining pre-training knowledge. Our experimental results support the theoretical analyses, justifying the generalization advantages of PACE over other PEFT methods. With its dual advantages, PACE consistently outperforms other variants across different backbones, firmly establishing PACE as a powerful solution for enhancing generalization for PEFT methods. Limitations and border impacts are discussed in §A.

Acknowledgments. We thank Moyang Liu, Melody Ip, Chenyi Du, and Yinuo Xu for their valuable discussions and support. PK is funded by CSIRO's Science Digital.

References

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 , 2024. 7, 23

Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating distribution. JMLR , 15(110):3743-3773, 2014. 5

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 , 2014. 1

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13 , pages 446-461. Springer, 2014. 6

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021. 8

Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems , 34:22405-22418, 2021. 3

Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. One-for-all: Generalized lora for parameter-efficient fine-tuning. arXiv preprint arXiv:2306.07967 , 2023. 2, 3, 7, 8, 22

Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems , 35:16664-16678, 2022. 1, 2, 3, 22

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021. 6, 7, 23

E Dataset. Novel datasets for fine-grained image categorization. In First Workshop on Fine Grained Visual Categorization, CVPR. Citeseer. Citeseer . Citeseer, 2011. 7

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009. 1, 7, 21

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems , 36, 2024. 3

Junhao Dong, Piotr Koniusz, Xinghua Qu, and Yew-Soon Ong. Stabilizing modality gap & lowering gradient norms improves zero-shot adversarial robustness of vlms. 31st SIGKDD Conference on Knowledge Discovery and Data Mining , 2025. 3

Wei Dong, Dawei Yan, Zhijun Lin, and Peng Wang. Efficient adaptation of large vision transformer via adapter re-composing. Advances in Neural Information Processing Systems , 36, 2024. 2, 8

Wei Dong, Xing Zhang, Bihui Chen, Dawei Yan, Zhijun Lin, Qingsen Yan, Peng Wang, and Yang Yang. Low-rank rescaled vision transformer fine-tuning: A residual design approach. arXiv preprint arXiv:2403.19067 , 2024. 2, 8

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021. 1, 3, 7

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021. 3, 4, 5, 10, 18

Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. On the effectiveness of parameter-efficient fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 12799-12807, 2023. 2, 3, 4

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022. 8, 21

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020. 1

Zhezhi He, Adnan Siraj Rakin, Jingtao Li, Chaitali Chakrabarti, and Deliang Fan. Defending and harnessing the bit-flip based adversarial weight attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14095-14103, 2020. 4

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision , pages 8340-8349, 2021. 7

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15262-15271, 2021. 7

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning , pages 2790-2799. PMLR, 2019. 1

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning , pages 2790-2799. PMLR, 2019. 2, 3

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022. 2, 3, 7

Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang, Yasheng Wang, Zhiyuan Liu, and Maosong Sun. Sparse structure search for delta tuning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. 2

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision , pages 709-727. Springer, 2022. 2, 4, 6, 7, 22

Zeyinzi Jiang, Chaojie Mao, Ziyuan Huang, Ao Ma, Yiliang Lv, Yujun Shen, Deli Zhao, and Jingren Zhou. Res-tuning: A flexible and efficient tuning paradigm via unbinding tuner from backbone. Advances in Neural Information Processing Systems , 36, 2024. 2

Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 1060-1068, 2023. 2, 6

Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15190-15200, 2023. 3

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. 7

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015-4026, 2023. 1

Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations , 2024. 3

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops , pages 554-561, 2013. 6, 7

Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. In The Eleventh International Conference on Learning Representations , 2023. 2

Bonan Li, Yinhan Hu, Xuecheng Nie, Congying Han, Xiangjian Jiang, Tiande Guo, and Luoqi Liu. Dropkey for vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22700-22709, 2023. 6

Dongyue Li and Hongyang Zhang. Improved regularization and robustness for fine-tuning in neural networks. Advances in Neural Information Processing Systems , 34:27249-27262, 2021. 3

Shengrui Li, Xueting Han, and Jing Bai. Adaptergnn: Parameter-efficient fine-tuning improves generalization in gnns. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 13600-13608, 2024. 2

Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, Zeyu Chen, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. arXiv preprint arXiv:1901.09229 , 2019. 3, 10

Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. In Advances in Neural Information Processing Systems (NeurIPS) , 2022. 2, 3, 7, 8, 22

Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael J. Black, Adrian Weller, and Bernhard Schölkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. In ICLR , 2024. 2, 22

Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019. 7

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021. 7, 8

Gen Luo, Minglang Huang, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, and Rongrong Ji. Towards efficient visual adaption via structural re-parameterization. arXiv preprint arXiv:2302.08106 , 2023. 2, 4, 22

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013. 6

Yao Ni and Piotr Koniusz. Nice: Noise-modulated consistency regularization for data-efficient gans. Advances in Neural Information Processing Systems , 36:13773-13801, 2023. 2, 16

Yao Ni and Piotr Koniusz. Chain: Enhancing generalization in data-efficient gans via lipschitz continuity constrained normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6763-6774, June 2024. 3

Yao Ni, Piotr Koniusz, Richard Hartley, and Richard Nock. Manifold learning benefits gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11265-11274, 2022. 3

Yao Ni, Dandan Song, Xi Zhang, Hao Wu, and Lejian Liao. Cagan: Consistent adversarial training enhanced gans. In IJCAI , pages 2588-2594, 2018. 2

Maria-Elena Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In IEEE Conference on Computer Vision and Pattern Recognition , volume 2, pages 1447-1454, 2006. 6, 7

Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, and Kyungwoo Song. Blackvip: Black-box visual prompting for robust transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 24224-24235, 2023. 2

Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition , 2012. 6

Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In NeurIPS , 2023. 2, 22

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021. 1

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning , pages 5389-5400. PMLR, 2019. 7

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022. 1

Shuvendu Roy and Ali Etemad. Consistency-guided prompt learning for vision-language models. In The Twelfth International Conference on Learning Representations , 2024. 3

Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set semi-supervised learning with open-set consistency regularization. Advances in Neural Information Processing Systems , 34:25956-25967, 2021. 2

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022. 1, 7

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020. 2

Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research , 2022. 8

Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, et al. On transferability of prompt tuning for natural language processing. arXiv preprint arXiv:2111.06719 , 2021. 1, 2

Junjiao Tian, Yen-Cheng Liu, James S Smith, and Zsolt Kira. Fast trainable projection for robust fine-tuning. Advances in Neural Information Processing Systems , 36, 2024. 3, 10

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023. 1

Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 595-604, 2015. 7

Dániel Varga, Adrián Csiszárik, and Zsolt Zombori. Gradient regularization improves accuracy of discriminative models. arXiv preprint arXiv:1712.09936 , 2017. 3

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. 1, 2, 3

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Thecaltech-ucsdbirds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. 7

Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018. 6, 7

Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems , 32, 2019. 7

Yihan Wang, Jatin Chauhan, Wei Wang, and Cho-Jui Hsieh. Universality and limitations of prompt tuning. Advances in Neural Information Processing Systems , 36, 2024. 2, 3

Yaoming Wang, Yuchen Liu, Xiaopeng Zhang, Jin Li, Bowen Shi, Chenglin Li, Wenrui Dai, Hongkai Xiong, and Qi Tian. Violet: Vision-language efficient tuning with collaborative multi-modal gradients. In Proceedings of the 31st ACM International Conference on Multimedia , pages 4595-4605, 2023. 3

Yeming Wen and Swarat Chaudhuri. Batched low-rank adaptation of foundation models. arXiv preprint arXiv:2312.05677 , 2023. 2

Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. Advances in neural information processing systems , 33:2958-2969, 2020. 4

Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, Tie-Yan Liu, et al. Rdrop: Regularized dropout for neural networks. Advances in Neural Information Processing Systems , 34:10890-10905, 2021. 2

LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International Conference on Machine Learning , pages 2825-2834. PMLR, 2018. 3, 10

Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867 , 2019. 6

Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. arXiv preprint arXiv:1910.12027 , 2019. 2

Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In ICLR , 2021. 5

Shan Zhang, Yao Ni, Jinhao Du, Yanxia Liu, and Piotr Koniusz. Semantic transfer from head to tail: Enlarging tail margin for long-tailed visual recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1350-1360, 2024. 3

Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv preprint arXiv:2206.04673 , 2022. 2, 6, 7

Yifei Zhang, Hao Zhu, Aiwei Liu, Han Yu, Piotr Koniusz, and Irwin King. Less is more: Extreme gradient boost rank-1 adaption for efficient finetuning of llms. arXiv preprint arXiv:2410.19694 , 2024. 3

Bingchen Zhao, Haoqin Tu, Chen Wei, Jieru Mei, and Cihang Xie. Tuning layernorm in attention: Towards efficient multi-modal LLM finetuning. In The Twelfth International Conference on Learning Representations , 2024. 2

Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference on Machine Learning , pages 26982-26992. PMLR, 2022. 3

PACE: Marrying generalization of PArameter-efficient fine-tuning with Consistency rEgularization (Supplementary Material)

A Broader impacts and limitations

A.1 Broader impacts

Our work provides a powerful solution for improving generalization in Parameter Efficient FineTuning (PEFT), allowing for effective fine-tuning of pre-trained models while reducing the heavily reliance on pre-training from scratch using large scale data. Our advancements in PEFT, supported by Theorems 1, 2 and 3, offer novel insights into gradient regularization and model alignment. These insights extend beyond PEFT and can be applied to other areas such as continual learning and transfer learning, potentially enhancing the performance and efficiency of models in various domains. By leveraging our findings, practitioners can develop more robust and adaptable models that generalize well to new tasks and environments, leading to more intelligent and versatile AI systems. In terms of negative impacts, the robustness of our fine-tuning method could potentially be misused to create more convincing deepfakes, raising concerns about the spread of misinformation, manipulation of public opinion, and malicious activities such as fraud, blackmail, or harassment. However, potential misuse is a downside with any improvements that have universal nature.

A.2 Limitations

While our work effectively improves generalization ability, it introduces additional computational costs by requiring input samples to be passed through the network twice for regularization. However, this can be mitigated by using two efficient variants, PACEfast and PACE half lazy , proposed in §C, where we demonstrate the potential for resource-efficient fine-tuning. Additionally, our method introduces extra hyperparameters λ and σ , which require caution during hyperparameter search. Nonetheless, Figure 7 suggests that fewer training data requires larger λ and σ values, providing insight for hyperparameter tuning.

B Proofs

B.1 Proof of Theorem 1

Settting ϵ = ρ ∇ θ ∥ ∇ θ ∥ 2 , we perform a second-order Taylor expansion of L D n around θ . By incorporating the higher-order terms from the Taylor expansion into R ( ∥ θ ∥ 2 2 ρ 2 , 1 n ) , we derive:

Assuming that the approximation does not alter the inequality relationship, i.e ., it preserves the ≤ relation on both sides and considering the largest eigenvalue of H θ as λ H max , implying v T H θ v ≤ λ H max ∥ v ∥ 2 2 for any v , we further bound Eq. 14 as follows and arrive at:

B.2 Proof of Theorem 2

The proof is motivated by Ni and Koniusz [47]. We include the proof process for completeness. Denote m 1 = z 1 -1 , m 2 = z 2 -1 thus m 1 , m 2 ∼ N ( 0 , σ 2 )

Defining v := m 1 ⊙ ∆ θ and u := m 2 ⊙ ∆ θ , where v , u ∼ N ( 0 , σ 2 diag (∆ θ ⊙ ∆ θ )) , we can rewrite Eq. 15 as follows:

Next, we derive the four terms, Eq. 16, 17, 18, and 19, respectively as follows:

Eq. 16. Using E z 1 ,z 2 [( z 1 -z 2 ) 2 ] = 2 σ 2 for z 1 , z 2 ∼ N (0 , σ 2 ) , we can simplify (Eq. 16) as follows, noting that terms related to different dimensions are canceled due to zero-mean independent Gaussian noise:

Eq. 17. Utilizing E [ z 3 ] = µ 3 +3 µσ 2 for z ∼ N ( µ, σ 2 ) , and noting that E [ z 3 ] = 0 for µ = 0 , Eq. 17 is derived as:

Eq. 18. We first decompose Eq. 18, then discuss each case and obtain the final result:

Given the independence of elements in v , only terms with an element repeated two or four times contribute non-zero results, leading to four distinct, non-overlapping cases. Using E [ z 2 ] = σ 2 + µ 2 and E [ z 4 ] = µ 4 +6 µ 2 σ 2 +3 σ 4 for z ∼ N ( µ, σ 2 ) , and simplifying to E [ z 2 ] = σ 2 and E [ z 4 ] = 3 σ 4 when µ = 0 , we have:

Case 1: j = k = p = q , given the independence of v j and v p , we have:

̸

̸

̸

Case 2: For j = p = k = q , the independence of v j and v k simplifies our calculation, leading to:

̸

̸

̸

̸

Case 3: For j = q = k = p , utilizing the independence of v j and v k as well as the symmetry H jk = H kj , we obtain:

̸

̸

̸

Case 4: For j = q = k = p , using E [ z 4 ] = 3 σ 4 where z (0 , σ 2 ) , we have:

Combining above four cases together, we have the result for Eq. 18:

̸

Eq. 19:

̸

With results of Eq. 20, 21, 27, 28, we have the final results:

̸

̸

̸

̸

̸

B.3 Proof of Theorem 3

The Cauchy-Schwarz inequality states that for u , v ∈ R d , we have ( ∑ j u j v j ) 2 ≤ ( ∑ j u 2 j )( ∑ j v 2 j ) . Let u = 1 , it follows that ( ∑ j v j ) 2 ≤ d ∥ v ∥ 2 2 . Using this inequality, we then prove the following:

Here, the inequality is obtained by treating ∆ θ j ∆ θ k H jk as an element of a vector with size of d 2 . This leads to the final results.

B.4 Rationale for one-dimensional output analysis

We use the squared L 2 distance for multi-dimensional outputs for D fp and D pace , which allows our one-dimensional analysis to naturally generalize to multiple dimensions. For example, for a vector-valued function in the naive alignment, f ( θ ) = [ f 1 ( θ ) , ..., f m ( θ )] , where m is the output dimension, we have:

This equality shows that the squared L 2 distance in multiple dimensions is simply the sum of nonnegative squared differences in each dimension. Consequently, this additive nature enables our one-dimensional analysis to extend seamlessly to multiple dimensions in practice, aligning with our empirical observations.

B.5 R increases with 1 n

According to [17], the function R ( ∥ θ ∥ 2 2 ρ 2 , 1 n ) in Eq. 3 is defined as:

Here k is the number of parameters, n is the number of training samples, δ ∈ (0 , 1] is the confidence level and ρ is the max norm of the Gaussian perturbation noise.

To ensure R is valid, we require n > 1 . To analyze how R changes with n , we fix ∥ θ ∥ 2 2 ρ 2 and break the expression under the square root of R into three terms:

We analyze each term separately to determine whether it decreases with increasing n .

Analysis for R 1 : The derivative for R 1 w.r.t. n is:

Since ∥ θ ∥ 2 2 ρ 2 is generally large, the smallest n is 2 and log ( 1 + √ log n k ) 2 > 0 . Therefore, for n > 1 , R ′ 1 < 0 , meaning R 1 decreases as n increase.

Analysis of R 2 : The derivative for R 2 w.r.t. n is

Analysis of R 3 : The derivative for R 3 w.r.t. n is

Since δ ≤ 1 , for n > 1 , R ′ 2 < 0 , indicating that R 2 decreases with increasing n .

For n > 1 , log(6 n +3 k ) > 1 , implying that R ′ 3 < 0 and R 3 decrease as n increases.

Conclusion. For n > 1 , all terms R 1 , R 2 and R 3 decreases as n increases. Thus R ( ∥ θ ∥ 2 2 ρ 2 , 1 n ) is a decreasing function of n .

C Efficient PACE variants

Building upon strong theoretical foundation of PACE for generalization, we demonstrate that simple modifications can reduce memory and training time requirements of PACE. In this section, we explore two efficient variants, PACEfast and PACE half lazy , both maintaining similar computational and memory requirements as the baseline while improving performance. We then provide empirical results which show that PACEfast slightly outperforms PACE half lazy while requiring no additional hyperparameters and using fewer computational resources. Given its superior efficiency, we further explore the potential of PACEfast for resource-efficient fine-tuning. By simply reducing the batch size and epochs, PACEfast outperforms the baseline while using significantly less GPU memory and training time.

PACEfast : Building on the observation that only small datasets are typically available for fine-tuning, we assume that the model behavior changes gradually across epochs. Under this assumption, we store the model outputs from the previous epoch ( f e -1 ( x ) ), which contain inherent noise due to the adapter perturbation, and compute the consistency regularization loss between these stored outputs and the current epoch's noised outputs:

Here the output vector o ∈ R C , where C is the number of classes. Since f ( · ) applies noise perturbation to the adapter and changes gradually between epochs, f e -1 ( x ) and f ( x ) can be seen as applying different i.i.d. noises to similar model states. This approach preserves the theoretical foundation of PACE while incurring minimal storage and computation costs. With typically few classes C and a limited number of samples in fine-tuning, storing o e -1 within GPU or CPU memory is manageable.

PACE half lazy : During training, the network always applies noise perturbations. Every N -th iteration uses a half batch size and consistency regularization, while all other iterations use the full batch size.

Memory and computational efficiency of two variants. Both variants maintain similar computational and memory requirements as the baseline. To demonstrate this, we conduct experiments on CIFAR-100 (VTAB-1K) using ViT-B/16, Camelyon (VTAB-1K) with Swin-B, and ImageNet (domain adaptation) with ViT-B/16. Table 9 compares maximum GPU memory usage, total training time, and accuracy for each task, showing that PACEfast and PACE half lazy significantly improve upon the baseline while maintaining similar computational demands.

We find that PACEfast slightly outperforms PACE half lazy without requiring additional hyperparameters, yet it needs to store outputs from the previous epoch. We therefore analyze its memory requirements.

Memory efficiency of PACEfast. We compare the additional memory requirement of PACEfast with the baseline GPU memory consumption. Table 10 shows that the memory overhead of PACEfast is negligible compared to the baseline GPU memory requirements and can be easily stored in GPU. Moreover, even in the rare scenario of fine-tuning on the full ImageNet 1K dataset (1.2 million samples), PACEfast requires only 4.8GB of additional memory for storing the output of the model's classification head. This is significantly smaller than the dataset itself (>100GB) and can be easily accommodated in the CPU/GPU memory.

Resource-Efficient training with PACEfast. Given the superior performance, minimal memory overhead, and no need for additional hyperparameters of PACEfast, we explore its potential for resource-efficient training by maintaining the same number of updates with reduced batch size and proportionally reduced epochs. Table 11 shows that even with 1/8 batch size and epochs, PACEfast still outperforms the baseline by 1.7% while only using ∼ 1/3 GPU memory and ∼ 1/4 training time. This demonstrates the robustness and generalization benefits that PACEfast brings to models, enabling them to excel under constrained training configurations. Such an efficiency is particularly valuable for fine-tuning large foundation models, where resource constraints necessitate small batch sizes and typically lead to sharp loss landscapes, yet the theoretical guarantee of PACE for smooth loss landscapes provides a promising solution for these challenges.

D Additional Experiments

In this section, we provide additional experiments of PACE on VTAB-1K with different epochs, varying training data sizes on FGVC benchmarks, self-supervised pre-trained backbones and combinations with other PEFT methods.

D.1 Experiments of VTAB-1K with different epochs

In Table 1, We use 300 epochs for VTAB-1K tasks as we observed slight improvements over 100 epochs. However, this does not mean PACE requires longer training to converge. Since the optimizer uses the cosine learning rate decay, reducing the number of training epochs to 100 has a minimal impact on performance, as shown in Table 12.

To ensure fair memory and computational budgets, we also tested PACE with half the batch size and 50 epochs. Table 12 shows that under these conditions, PACE still improves baseline accuracy by 2.10%, and outperforms the previous SOTA GLoRA, which uses 500 epochs for training and 30 for parameter search. These results demonstrate PACE's efficiency and effectiveness across various training configurations.

D.2 Experiments on FGVC with limited training data

To validate generalization benefits of PACE on limited data settings, we conduct experiments on FGVC using 50%, 20%, and 10% of the original training samples. Table 13 shows that PACE achieves larger improvements with smaller data sizes, aligning with our theoretical analyses.

D.3 Experiments on self-supervised pre-trained backbones

To further verify the effectiveness of PACE on a self-supervised pre-trained backbone, we conduct VTAB-1K experiments on SVHN, Camelyon, and Clevr-Count using MAE [19] and DINO [19], with ViT-B/16 pre-trained on ImageNet-1K [11]. Table 14 shows that PACE improves the baseline on these self-supervised backbones, confirming its applicability to fine-tuning self-supervised models.

D.4 Experiments of Combining PACE with Other PEFT

We conducted experiments combining PACE with several PEFT methods, including AdaptFormer [8], GLoRA [7], COFT [54], and BOFT [42], on CIFAR-100 (VTAB-1K) and ImageNet (domain adaptation) using ViT-B/16. Table 15 shows that integrating PACE improves the baseline performance.

E Additional Plots

Figures 8 and 9 show the gradient issues in FPA and the gradient regularization effects of PACE.

100

200

300

epoch=

1e4

2e4

3e4

‖

∂f

/

∂

Θ

‖

2

100

200

300

epoch=

1e4

2e4

3e4

Baseline

0.001

0.005

0.01

0.05

0.1

0.5

(a) FPA

(b) PACE

F Hyperparameter settings

For each dataset, we follow strategies from previous works [41, 28, 7, 45] to apply grid search on the rank, learning rate and weight decay to establish strong baselines. Table 16, 17, 18 and 19 present the hyperparameters and number of trainable parameters used in our strong baseline for VTAB-1K, few-shot learning, FGVC and domain adaptation tasks.

With these strong baselines, we apply grid search on λ ∈ { 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 , 1 } and σ ∈ { 0 . 1 , 0 . 5 , 1 , 1 . 5 , 2 } for PACE to optimize its performance.

1e-3 5e-3 0.01 0.05

0.1

0.5

1

5

10

50

100 500

1e3

5e3

1e4

5e4

λ

=

3e3

6e3

‖

∂f

∂θ

‖

2

Baseline

+FPA

+PACE

G Experiment details for GSM-8K

We conduct experiments on text generation tasks by fine-tuning Phi-3-mini-4k-instruct [1] on the GSM-8K [9] dataset using causal language modeling. We use learning rate of 2e-6, batch size of 4, LoRA rank of 16, prompt 'Answer below question. First think step-by-step and then answer the final number:\n\n<Question>' as instruction and fine-tune models on the training set and evaluated the performance on the test set.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We theoretically and empirically verify the claims and contributions made in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of our work are discussed in §A

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Complete proofs for each theorem are provided in §B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Training details and hyperparameter selection are presented in Sec. 4 and §F, respectively.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release our code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental settings and details are presented in Sec. 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All reported results are averaged over three random seeds.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments were conducted on a single NVIDIA H100 GPU with 96 GB memory, with each experiment completing within 8 hours.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] ,

Justification: We have carefully reviewed and adhered to the code of ethics throughout our research and writing process.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Potential impacts are discussed in §A.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: Our work poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All publicly available assets (models, code, and data) used in this work have been properly credited, and their respective licenses and terms of use have been explicitly mentioned and adhered to.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets in the submission.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a consistency regularization for improving the generalization performance of PEFT methods. Specifically, the regularization constructs two predictions perturbed by different noises and penalizes the L2 distance between them. The paper theoretically studies the benefits and shows that the regularization reduces gradient norms. Empirically, the paper benchmarks the method on multiple tasks and datasets and achieves state-of-the-art performance.\n\nSTRENGTHS:\n* Theoretical results are clearly presented and easy to follow. \n*  The paper presents an extensive evaluation of multiple datasets and tasks.\n\nWEAKNESSES:\n* **Missing related works and comparisons.** The major weakness of the paper is the lack of discussion and comparisons with related works. For example, L2-SP [1] directly penalizes deviation in the weight space and ensures alignment. DELTA [2] penalizes feature differences similar to this paper. FTP [3] shows that feature deviation can be reduced to weight deviation in linear layers. While the experiment settings are diverse, the paper only compares to vanilla PEFT methods. The method should be compared to other regularization methods for a fair evaluation. \n\n* **Increased computation.** To construct the two predictions. The method requires two forward passes through the model. This can significantly increase the computation and memory requirement of the algorithm. \n\n* **How does $D^{PACE}$ ensure alignment?**  It is clear that $D^{fp}$ (eq.5) encourages alignment. However, the proposed PACE regularization does not seem to encourage alignment explicitly.\n\n[1] Xuhong, L. I., Yves Grandvalet, and Franck Davoine. \"Explicit inductive bias for transfer learning with convolutional networks.\" International Conference on Machine Learning. PMLR, 2018.\n\n[2] Li, Xingjian, et al. \"Delta: Deep learning transfer using feature map with attention for convolutional networks.\" arXiv preprint arXiv:1901.09229 (2019).\n\n[3] Tian, Junjiao, et al. \"Fast trainable projection for robust fine-tuning.\" Advances in Neural Information Processing Systems 36 (2024).\n\nQUESTIONS:\n* Could the authors provide discussion and comparisons to other robust regularization techniques?  \n\n* Could the authors comment on how $D^{PACE}$ ensures alignment.\n\nLIMITATIONS:\nThe paper does not have a potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PACE, an extension to PEFT methods for ViTs that includes consistency regularization. The paper shows that consistency regularization encourages smaller gradient norms and better alignment between pre-trained and fine-tuned models, resulting in better fine-tuning performance than existing PEFT techniques.\n\nSTRENGTHS:\n**(S1)**: The technique introduced is well-presented and easy to implement, borrowing ideas from parameter-efficient fine-tuning (PEFT) and consistency regularization (CR) to present a combined technique that is empirically effective.\n\n**(S2)**: The paper rigorously grounds their approach via theoretical analysis, demonstrating how reduced gradient norms lead to better generalization. This is well presented and the analysis itself will be relevant to future work and related research in this field, as other techniques that emulate the same properties (eg: reduced gradient norms) can benefit from PACE.\n\n**(S3)**: The ablation study is well conducted and systematically demonstrates the usefulness of the authors’ design. Throughout the experimental section, the authors also include experiments that probe properties of the model relevant to their method (eg: Fig 2, 3, 4), demonstrating that their theoretical hypotheses are well justified in empirical results.\n\nOverall, this paper is well-presented, technically sound, and is a pleasure to read. I think it will have a good impact on future research in this area.\n\nWEAKNESSES:\n**(W1)**: The experimental baseline is not really clarified. Only a ViT-B/16 pre-trained on ImageNet is used. Was the pre-training supervised or unsupervised? Do the analysis and results differ for supervised vs self-supervised backbones? Most common and performant ViTs nowadays (eg: MAE [1], Dino [2]) are self-supervised, so it would be good to clarify if there are differences in the analysis that depend on the method of pre-training. \n\n**(W2)**: It would be interesting to see this technique extended to larger domain shifts (eg: WILDS [3]). Also, it is not clear how the amount of fine-tuning data affects the performance (beyond few-shot learning). I.e. can PACE make better use of large datasets from new domains more effectively than prior methods.\n\n**(W3)**: Minor quibble. It would be useful to include comparisons with orthogonal fine-tuning techniques (eg: BOFT [4], OFT [5]).\n\n---\nReferences:  \n\n[1] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[2] Oquab, Maxime, et al. \"Dinov2: Learning robust visual features without supervision.\" arXiv preprint arXiv:2304.07193 (2023).\n\n[3] Koh, Pang Wei, et al. \"Wilds: A benchmark of in-the-wild distribution shifts.\" International conference on machine learning. PMLR, 2021.\n\n[4] Liu, Weiyang, et al. \"Parameter-efficient orthogonal finetuning via butterfly factorization.\" arXiv preprint arXiv:2311.06243 (2023).\n\n[5] Qiu, Zeju, et al. \"Controlling text-to-image diffusion by orthogonal finetuning.\" Advances in Neural Information Processing Systems 36 (2023): 79320-79362.\n\nQUESTIONS:\nL220-221: Which pre-training method was used for these backbones? Was it supervised or self-supervised?\n\nI don’t see PACE_h in table 6.\n\nIt would be great to see this extended to language applications!\n\nLIMITATIONS:\nThe authors adequately discuss this section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to regularize the model consistency by optimizing the fine-tuned model to remain consistent for the same sample under different perturbations.\n\nSTRENGTHS:\n1. The paper is well-written.\n2. Several experiments are conducted on four visual adaptation tasks: VTAB-1k, FGVC, few-shot learning, and domain adaptation.\n\nWEAKNESSES:\n1. **Novelty Concerns**: Incorporating consistency regularization with PEFT approaches is already commonplace in multimodal parameter-efficient fine-tuning. For example, VioLET [1],  PromptSRC [2], and CoPrompt [3] propose to use additional encoders with shared weights to keep the consistency between the updated model and the original one. Considering that these papers focus on the consistency constraints on each modality itself, rather than cross-modal constraints. Therefore, the approach proposed in this paper to copy the original model parameters to enforce consistency lacks innovation and is simply a repetition of existing ideas.\n \n\n2. **Efficiency Concerns**: The proposed method, as referenced in Eq. 12, necessitates multiple forward passes during fine-tuning, whereas most compared methods require only a single pass. This discrepancy significantly impacts efficiency. The authors should provide detailed efficiency metrics alongside performance results to enable fair comparisons.\n\n3. **Experimental Settings**: Several experimental settings appear unconventional. The proposed method is based on a new baseline, LoRAmul+VPTadd. However, PACE does not seem to depend on specific designs of the base PET methods. Therefore, it would be more compelling to incorporate PACE into more general baseline methods such as AdapterFormer or existing SOTA methods like GLoRA. Demonstrating PACE's ability to enhance various PET methods would strengthen its contribution. Additionally, PACE requires 300 epochs for fine-tuning on the VTAB-1K dataset, whereas other methods typically require only 100 epochs. The necessity of such prolonged fine-tuning should be justified. Furthermore, in some benchmarks (e.g., Tables 2 and 3), PACE shows only limited improvement, which undermines its robustness.\n\n4. **Typos and Notational Errors**:  For instance, L505 misses a square in the second row. Additionally, the equal sign should be replaced with an approximate equal sign.\n\n[1] Wang Y, Liu Y, Zhang X, et al. VioLET: Vision-Language Efficient Tuning with Collaborative Multi-modal Gradients[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 4595-4605.\n\n[2] Khattak M U, Wasim S T, Naseer M, et al. Self-regulating prompts: Foundational model adaptation without forgetting[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 15190-15200.\n\n[3] Roy, Shuvendu, and Ali Etemad. \"Consistency-guided prompt learning for vision-language models.\" arXiv preprint arXiv:2306.01195 (2023).\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a consistency regularization that minimizes the squared L2 distance between two outputs of a model obtained using the same parameters, but multiplying the activations by 2 different noise samples. It proves that the population loss is bounded by the gradient norm, indicating that smaller gradients are better for generalization. It proposes the consistency regularization for aligning the fine-tuned model with the pretrained model for better generalization, and proves that the consistency regularization penalizes the first and second order gradients of the parameters of the model. This results in the training process preserving the knowledge learned by the pretrained model. It also proposes an efficient method for implementing the consistency regularization where the regularization is applied every N iterations.\n\nSTRENGTHS:\n- The paper provieds a theoretical analysis and intuition for the consistency regularization and links generalization to small gradient norm.\n- The empiracal results show that using the consistency regularization along with existing finetuning methods improves their perfromance.\n- The consistency regularization can be made somewhat efficient by adding the noise to the features instead of parameters, and by using the consistency regularization in intervals rather than at every iteration.\n\nWEAKNESSES:\n**Gap between theoretical analysis and practical implementation**\n- The theoretical analysis is done for functions that map from R^d R. However, the in practice, the outputs of the heads of the classification models are multi-dimensional. It's not clear from the proofs that the theoretical results hold for multi-dimensional outputs. While the experimental results do suggest this, a theoretical proof for this would strengthen the paper.\n2. **Increased compute and memory requirements**\n- Since the consistency regularization needs to run the model twice (with shared parameters but with two different noise samples), and also backpropagate through both, the compute and the memory required is doubled.\n- The authors use H100 GPUs (96GB) for their experiments, hence the experiments are feasible. However, with lower end GPUs, this might not be feasible. Moreover, the requirements for large models (in the scale of billion parameters) might be prohibitive.\n- With this increased compute and memory requirements, the method's use as a PEFT method is limited.\n- While the paper proposes a method for efficient implementation by applying the regularization every N iterations, it is only tested on one dataset from VTAB-1k (CIFAR100). This should be tested on more datasets for the claim to hold.\n3. **Lack of evaluation beyond vision tasks**\n- The paper does not study natural language tasks (language understanding and generation). This would be necessary to show the generality of the method to other domains besides vision tasks.\n4. **Large training time**\n- The method is trained on VTAB for a large number of epochs (300). This means that the method can take a long time to converge. Are the comparisons with the other methods (other than the chosen baselines) done with a similar compute budget?\n- The method is trained for 100 epochs for the FGVC tasks. Why does the method require fewer epochs for FGVC tasks compared to VTAB?\n\nQUESTIONS:\n- Line 155-156 \"as even smaller weight changes can lead to significant divergence in the output space\": Can the authors provide references/experimental results to backup this statement?\n- In Figure 2 (a) and Figure 3 (b), how is the plot shown for the gradient norm of parameters in multiple layers?\n- See Point 4 in Weaknesses.\n\nLIMITATIONS:\nAuthors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}