PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization
Authors: Yao Ni
Shan Zhang
Piotr Koniusz
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Generalization
Regularization
Transfer Learning
Parameter-Efficient Fine-Tuning
Primary area: optimization_for_deep_networks

Abstract: Parameter-Efficient Fine-Tuning (PEFT) effectively adapts pre-trained transformers to downstream tasks. However, the optimization of tasks performance often comes at the cost of generalizability in fine-tuned models. To address this issue, we theoretically connect smaller weight gradient norms during training and larger datasets to the improvements in model generalization. Motivated by this connection, we propose reducing gradient norms for enhanced generalization and aligning fine-tuned model with the pre-trained counterpart to retain knowledge from large-scale pre-training data. Yet, naive alignment does not guarantee gradient reduction and can potentially cause gradient explosion, complicating efforts to manage gradients. To address such an issue, we propose PACE, marrying generalization of PArameter-efficient fine-tuning with Consistency rEgularization. We perturb features learned from the adapter with the multiplicative noise and ensure the fine-tuned model remains consistent for same sample under different perturbations. Theoretical analysis shows that PACE not only implicitly regularizes gradients for enhanced generalization, but also implicitly aligns the fine-tuned and pre-trained models to retain knowledge. Experimental evidence supports our theories. PACE surpasses existing PEFT methods in visual adaptation tasks (VTAB-1k, FGVC, few-shot learning, domain adaptation) showcasing its potential for resource-efficient fine-tuning. It also improves LoRA in text classification (GLUE) and mathematical reasoning (GSM-8K).

PACE: Marrying generalization in PArameter-effiiiet fine-tuning with Consistency rEgularization
Yao Ni Shan Zhanghi Piotr Koniusz*$1 S8.
The Australian National University $Data61 Australian Institute for Machine Learning, The University of Adelaide
yao minann. edu au shan zhangoadelladde edu. au 'piotr. Koodrkkknuszzdtta..... cetroi.cssi..aa. au
Abstract
Parameter-Efficient Fine-Tuning (PEFT) effectively adapts pre-trained transformers to downstream tasks. However, the optimization of tasks performance often comes at the cost of generalizability in fine-tuned models. To address this issue, we theoretically connect smaller weight gradient norms during training and larger datasets to the improvements in model generalization. Motivated by this connection, propose reducing gradient norms for enhanced generalization and aligning fine- tuned model with the pre-trained counterpart to retain knowledge from large-scale pre-training data. Yet, naive alignment does guarantee gradient reduction and can potentially cause gradient explosion, complicating efforts to manage gradients. To address such an issue, we propose PACE, marrying generalization of PArameter- efficient fine-tuning with Consistency rEgularization. We perturb features learned from the adapter with the multiplicative noise and ensure the fine-tuned model remains consistent for same sample under different perturbations. Theoretical analysis shows that PACE not only implicitly regularizes gradients for enhanced generalization, but also implicitly aligns the fine-tuned and pre-trained models to retain knowledge. Experimental evidence supports our theories. PACE surpasses existing PEFT methods in visual adaptation tasks 1kk, FGVC, few-shot learning, domain adaptation) showcasing its potential for resource-effie -ffricceefcc fine- tuning. It also improves LoRA in text classification (GLUE) and mathematical reasoning (GSM-8K). The code available at github.com/ www./aww//llloooooooowooooowwwww Maw.ee/aoelll//AACCEEE.
Introduction
Transformers [68], with the self-attention mechanism [3] capturing long-range dependencies in data, succeed in various deep learning tasks, including image classification (ViT [16]), multimodal learning (CLIP [55]). image synthesis (StableDiffusion [57]). semantic segmentation (SAM [33]) and text generation (LLaMM [65]). The success of transformers of abundant data, such as ImageNet [11] and Laion5B [60], which empower researchers to scale up these models by training them under enormous number of parameters.
be largely attributed to the availability
Such huge models, with knowledge from large-scale pre-training [63]. constitute on foundation models that can be easily adapted to various downstream tasks through full fine-tuning linear probing [20], eliminating the need for task-specific model design [8]. However, full fine-tuning is storage-intensing and infeasible for maintaining separate model weights the number of tasks grows, while linear probing, which only trains the last head layer, yields inferior adaptation performance.
To overcome these limitations, Barameter-Efficien Fine-Tune Tuning (PEFT) [24] fine-tunes only small subset of parameters, thereby reducing storage requirements while surpassing the performance of
'The corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

full fine-tuning and linear probing. These advantages have popularized PEFT and inspired the development various PEFT methods for deep learning tasks, which can be categorized into two groups: those increasing inference cost and cost-efficient ones. The first group introduces additional learning branches, such as non-linear adapters [25, 8], or concatenates learnable parameters with input tokens, e-g., visual prompts [28, 82, 52], increasing inference cost. The second group, focuses on cost-efficiency by lower-rank adaptation in linear layers [7, 26]. or affine transformations such as SSF [41] and RepAdapters [45], which can be reparameterized during inference for efficiency.
Despite the superiority and efficiency of PEFT, prioritizing optimization for downstream tasks compromises the generalizability of fine-tuned models, yielding suboptimal performance. Although some analyses have been conducted on PEFT [63, 27. 18, 72. 39], they fail to fully explain the generalization of PEFT. leading to ineffective strategies for improving generalization.
To address this gap in understanding generalization in PEFT, establish a theoretical connection from generalization theory: smaller weight gradient norms and larger data volumes contribute to better generalization. Motivated by this, we propose reducing weight gradient norms and aligning output space of the fine-tuned model with the pre-trained one to retain knowledge captured from large pre-training data. Yet, theoretical analyses reveal this naive alignment does not guarantee gradient regularization and can even cause gradient explosion, complicating efforts for gradient management. To address this issue, we propose perturbing features learned from the adapter with multiplicative noise and constraining the network output to be consistent across different perturbations.
Our method, called PACE, marries generalization of PArameter-efficien fine-tuning with Consistency rFgularization. Its name, PACE, reflects our goal of keeping the output behavior of the fine-tuned model in pace with the pre-trained one. Despite its simplicity, theoretical analysis confirms that PACE not only implicitly regularizes weight gradients for better generalization but also implicitly aligns the fine-tuned model with the pre-trained counterpart retain knowledge from large-scale pre-training data. Experimental evidence supports our theories. PACE improves existing PEFT methods, achieving superior results across six adaptation benchmarks. Our key contributions are:
i. We establish a theory connecting smaller weight gradient norms and larger datasets with en- hanced generalization, motivating gradient reduction and model alignment for fine-tuning. ii We propose PACE, a simple yet effective method perturbing features from adapters with multi- plicative noise and constraining output of fine-tuned model to be consistent across perturbations. iii. Our theoretical and empirical evidence confirms that PACE implicitly regularizes gradients and aligns the fine-tuned model with the pre-trained one. PACE excels on visual adaptation tasks. iv. We provide novel theoretical explanations of how gradient penalization and consistency regu- larization benefit generalization, offering fundamental insights applicable across deep learning.
2 Related work
Parameter-Efticien Fine-Tuning (PEFT). LoRA [26] uses low-rank decomposition to reduce parameters and treats adapters as side paths. SSF [41] proposes affine transformations on latent features. FacT [30] decomposes and reassembles parameter matrices in ViT. Surgical fine-tuning [36] of different network parts improves adaptation to distribution shifts. FLoRA [74] performs a batched
low-rank adaptation. GLoRA [7] unifies cost-efficient PEFT methods. NOAH [82] uses parameter search on neural prompts. ARC [14] leverages cross-layer ViT similarity, parameter-sharing adapter and scaling factors for lower fine-tuning cost. RLRR [15] incorporates a residual term for flexibility while preserving pre-trained representation. RepAdapter [45] reparameterizes adapters for efficient inference. Res-tuning [29] unbinds tuners from the backbone for memory efficiency. Zhao et al. [84] show impressive fine-tuning results by tuning layernorm in attention. OFT [54] and BOFT [42] propose orthogonal fine-tuning to preserve hypersphere energy between neurons.
Consistency Regularization. Fixmatch [61] applies consistency regularization over augmented images for sumi-super learning. Openmatch [59] utilizes on outlier predictions for open-set semi-supervised learning. R-Drop [76] applies it to transformers
[68] with dropout for NLP tasks.
CR [79] applies it over augmented real and fake images for GAN training. CAGAN [50] enforces consistency on discriminators with dropout for GAN training. Despite the empirical success of consistency regularization demonstrated by previous works, theoretical analysis is lacking. While
NICE [47] demonstrates that consistency regularization lowers latent feature gradients for stable
2

---

GAN training, it fails to reveal reduced weight gradient for enhanced generalization. Our study goes beyond prior works by providing a theoretical link between smaller weight gradients and improved generalization, effectively marrying generalization of PEFT with consistency regularization.
Generalization of Fine-Tuning. Li al. [38] constrain the fine-tuned model's closeness to the pre- trained model in weight space. Fu et al. [18] induce sparsity on PEFT for better generalization. Wang al. [72] studies generalization of PEFT fine-tuning graph neural network. Zhang et al. [83] employ rank-1 gradient boosting (GB) updates supported by the GB theoretical framework. VioLET [73], PromptSRC [31] and CoPrompt [58] naively align the fine-tuned model with the pre-trained one for enhanced generalization or avoiding forgetting. Additionally, L2SP [77]. DELTA [40], and FTP [64] aim retain pre-trained knowledge by aligning fine-tuned models with pre- pre-trained ones, reducing distance in weight space, feature space and using projected gradient descent, respectively. However, they fail to provide a theoretical analysis for this alignment. Our study goes beyond understanding generalization of PEFT by discovering the benefits of gradient regularization and model alignment. We propose PACE to match both requirements, paving a comprehensive understanding for PEFT.
Gradient regularization. Previous studies have empirically shown that gradient regularization improves performance [67, 85, 48, 49] and adversarially robust accuracy [13]. However, they lack theoretical connection between smaller gradient norms and better generalization [17, 81, 6]. We bridge this gap by establishing& a fundamental theory between reduced gradient norms and improved generalization, providing solid foundation for future research on enhancing generalization.
3 Approach
We begin with a unified perspective on cost-efficient PEFT based on GLoRA [7], linking general- ization with gradients and large-scale data, and motivating the alignment of the fine-tuned model with the pre-trained model to leverage its knowledge. We identify limitations of naive alignment in gradient regularization and introduce PACE, which implicitly enhances gradient regularization and model alignment. We conclude with thcoretical justification and efficient implementations.
3.1 A unified perspective on cost-efficient PEFT methods
The transformer architectures [68, 16] have excelled in natural language processing and computer vision tasks through their powerful sequential modeling capabilities. This success stems from their ability to process text/image tokens through T. transformer blocks, where each block contains self- attention and MLP modules primarily composed of linear layers. These linear layers enable the self-attention mechanism to capture long-range dependencies, allowing transformers to achieve superior performance when scaled to a huge number of parameters and trained on extensive datasets.
With massive parameters, pre-trained on large-scale data, transformers serve as foundation models that can be fine-tuned for downstream tasks using limited data. However, fully fine-tuning all parameters for various downstream tasks requires substantial memory and can lead the forgetting of pre-trained knowledge. To alleviate this without increasing inference cost, adapters with lightweight parameters are often preferred for tine-tuning Let (-))) be a transformation within the pre-trained transformer. Current adapters can be unified as introducing a residual branch Ah to form a new transformation h:
Ta)))= ho(a) + Ah(a).
(1)
Here, a is the input and ho(-) can represent MLP modules, as in Adapter [25] and AdaptFormer [8], or linear layers in self-attention and MLP modules. in [26, 7, 12, 34]. In SSF [41], ho(-) is the identity mapping and Ah(a) = a @ (7 - 1) + 33 with 7 and ,3 as affine transformation parameters.
Given that linear layers are key components in transformer, tuning them offers flexible and effective way to adapt models to downstream tasks. This work focuses on methods that tune the linear layer without increasing inference cost. Let (Wo, (AW, Ab), and (W. b) be the parameters of pre-trained model, adapter and fine-tuned model, respectively, where W AW, W E Rduxdd, Xdo and bu, Ab, b Rd Fine-tuning a linear layer in self- attenttention or MLP module can be formed as:
h(a)) Wa b (Wo + AW)a ja. + (be + Ab) ha Ah(a) (W oa + by) + (AWa Ab).
(2)
Based on GLoRA [7], cost-efficient PEFT methods for lincar layers vary in the form of AW, Ab:
3

---

LoRAadd: AW W,W, Ab biora where Wa E R W. E R' and r is the rank.
LoRA mul: AW- Wo W W.), Ab= including RepAdapter [45] via reparameterizzation VPT AW is zero, Ab = WoP. with learnable P E [ddax1 as layer-wise visual prompt. We use VPT wad to differentiate from VPT [28], which concatenates P with tokens, increasing inference cost.
3.2
Generalization of deep neural networks
Having established a unified perspective on cost-efficient PEFT, we now motivate our method from a perspective improving generalization of neural networks to enhance performance unseen data. Consider a network f ;= (((())) with / layers, where is feature extractor and ø is the classification head. Let 0 == {( b H be the parameter set with dimension d and D := {(x,, v;) 1 be the training set of size n drawn i.i.d. from distribution 2, which contains infinite data. The following lemma from [17] explains the relationship between the empirical and population loss.
Lemma (Theorem from [17]) Let LD= (0) be the empirical loss function over I on training D and Lo (0) be the population loss. For any p > O, with high probability over D ~ D, we have
(g( S max LD- (0 E) + R |
P2
where R: (R.,, Rx is increasing function (under conditions on L (0) and as in $B.5).
Lemma 1 bounds the population loss by the empirical loss with perturbed weights, indicating that minimal empirical loss increase from small weight perturbations implies low population loss.
By observing that the maximum of LDn is achieved at e . where Vo is the gradient of LDn ['
at 0, and performing Taylor expansion of LD= around 0, we formulate the following theorem.
Theorem Denote Va as the gradient and X as the largest eigenvalue of the Hessian matrix He of Lon at O. For any p > O, with high probubility over training set D ~ D, we have
Ø²
11013 P²
C ())= S LD= (0) + pl Vella
A max
+
Here, higher-order terms from the Taylor expansion are incorporated into R related to weights norm and inversely related to the training data size n.
1011 1
which is
Theorem (proof in $B.1) outlines strategies for enhancing generalization. They involve regularizing weight norms and the largest Hessian cigenvalues, and crucially, increasing data size n and reducing the weight gradient norms (illustrated in Figure 1). However, excessive reduction should be avoided as it could impair network's representation capacity, yielding higher empirical and population loss.
3.3 Motivation and limitation of aligning the fine-tuned model with the pre-trained model
Theorem emphasizes that large-scale data and smaller gradient magnitudes are essential for better generalization in neural network training. Therefore, aligning the fine-tuned model with the pre- trained one crucial, as it ensures retention of knowledge obtained from large-scale data, preserving generalization. PEFT methods, often outperforming full fine-tuning, achieve this alignment by limiting the number of trainable parameters, restricting the model's ss capacity to deviate from the pre- trained one. However, the training objective prioritizes downstream task performance, compromising alignment with pre-trained knowledge. While sparsity regularization [18] and weight decay on adapter weights help, they do not ensure alignment, as even small weight changes can lead to significant divergence in output space [75, 21, 17]. Therefore, we propose achieve the alignment by reducing the FP- distanee (output distance between fine-tuned and pre- pre-trained models on training samples):
D
Σ [f(=: f(z;; 00)1 0 = 0, A0,
(5)
where 0, 0o, AO Rdd parameters for the fine-tuned model, pre-trained model and the adapter.
While reducing FP-distance keeps the fine-tuned model close to the pre-trained model, thus preserving its knowledge, it does not ensure reduced gradient magnitudes, leading to suboptimal generalization.

---

Theorem
Theorem 2
Theorem 3
Sharper
Flatter
2x
minimumy
minimum
1(0 + 50)
Large difference
[(0,)
40
f 2, DOY
110
10 agy
Smhlll diBerence
(
Lange gradient& Hessiun
Small gradient Hession
Lange grad norm
PACE
Small grad norm
I Lange FP-distance
PACE
Small FP-distance
Figure 1: Thm. 1: A flatter minimum has smaller gradient and Hessian norms, yielding better generalization. Thm. 2: Large gradient norms indicate large differences among perturbations. PACE minimizes these differences, reducing gradient norms. Thm. 3: Minimizing all pairs of distances between f(0o + O A0) and f(Oo- =2 AO) where 21, Z2 ~1,, o² I² also reduces FP-distance (between fine-tuned f(Ou+ A0) and pre-trained f(O.)), especially when ž =1=1, 22 =0 0 or vice versa.
To understand the gradient-related limitations in this alignment, we assume Ao is small enough for Taylor expansion approximation. Following standard practices [17, 80, 2]. we perform the expansion up to the second-order terms. Given the independence between elements in squared L2 distances ($B.4) and to simplify our theories, we analyze dieddmenniooall output for a single i.i.d. sample, which leads us to the following proposition.
Proposition 1 Assuming 20 small, denote s(0) E I as the one-dimensional output for æ. with V and H as its gradient and Hessian O. FP-distance over can be decomposed as follows:
[f(0) f(o))) [f(0) f(e 20)] = [f(0) [f(0) - 00" v " ~0"
1 AO HAO?? |AO
Prop. 1 establishes the relationship between weight gradients, adapter weights, and FP-distance. However, it remains unclear if it regulates gradients. Our experiments show that minimizing FP- distance can sometimes increase gradient magnitude, complicating efforts for managing gradient.
3.4 Consistency regularization
To achieve better generalization by both regularizing gradients and aligning the fine-tuned model with the pre-trined model, we propose a consistency regularization loss for f. encouraging invariance of f to the same input under varying multiplicative noise perturbations the adapter weights, follows:
D 0
Ex,,, we (x;; On + =1 @ A0) f(x.; Oo + = 3 A0)
(7)
where 21, Z2
(1, the multiplicative noise applied adapter weight. To understand the generalization benefits in this consistency regularization, we simplify the analysis by focusing on one-dimensional output for a single sample, resulting in the following theorem.
Theorem Using notations from Prop. 1, let f(0, = O AO) E I. be the one-dimensional output for æ. Define Ad, as j-th element 20, V, as the j-th element V and H₃ as the (j, k) -e)-entry in H. With 21, 22 _N11
(2, I), the consistency loss over be approximated as:
E, = ④ 20) f + C 20)]²
~ 20 E,
A0 = 20
1 1 c H
(8)
Theorem 2 (proof in $B.2) shows that the consistency regularization essentially penalizes the first- and second-order gradients of ff at 0 (illustrated in Figure 1). with the regularization strength controlled by the noise variance o² and adaptively influenced by the magnitude of elements in adapter weight A6. Thus, minimizing the consistency loss implicitly regularizes the gradients, improving generalization.
With the FP-distance in Prop. and consistency loss in Theorem 2, we establish their relationship as:
Theorem With d the dimension of 0, Eq. 6 can be upper-bounded as:
1 T " < 2d] AO v + & (10000")) H
A
2
5

---

Transformer block with adapter perturbed by noise
|Consistency regularization between two outputs of
+Gp
ARO
Transformer Block
MLP
head
loss
xl.
Norm
haO
(xx)
*
Wo bog
10=10++2 + 4h
share weights
[1/(() - 11(1())
Multi-Head
where
x1)
non-shared noises
Attention
f2(x)
Norm
Adapter Åh and pre-trained Ro linear layer
Transformer Block
head
Transformer Block
Figure 2: Our pipeline. Adapter Ah(-) and ho(-) from pre-trained model form the linear layer l of Multi-Head Attention and MLP in fine-tuned model. We perturb An(+) with multiplicative noise and ensure the network remains consistent to same inputs under varying perturbations.
Theorem 3 (proof B.3) establishes the relationship between Eq. and Eq. 8, showing Eq. is upper- bounded by terms involving || V and (A0S0")) @ l| which appear in Eq. Reducing these terms results in a decrease in Eq. 6. Thus minimizing the consistency loss implicitly aligns the fine-tuned and pre-trained models (illustrated in Figure 1), preserving mr-ttrineed knowledge.
3.5 Efficient implementation of PACE
Providing different weight perturbations for each input in a mini-batch increases memory and computational demands. To avoid this, we perturb feature outputs from the adapter Ah(.), effectively simulating perturbation that shares noise across each row in the weight matrix A AW. Our simple pipeline is shown in Figure 2. Consider X R xx as a batch of data where B and T the batch and token sizes. The calculation for the linear layer of the fine-tuned model, which utilizes pre-trained weights Wo, by and adapter weights A AW, Åb, processes an output size of dout as:
(x(X): = W Wo X + by:
Ah (X))= = AWX + Ab.
(10)
((X)= ( X Z O (
(1I)
Operator @ is the element-wise multiplication after expanding the left matrix Z E BB (2) into x dou where tokens within the same example share the same noise. Motivated by [37], the decreases lincarly block depth increases. Let fi(-) and f2(.) be two networks share same weights but do not share the noise patterns. The loss function for PACE is:
PACE Σ ((f((z;), v.) + (1) - f2(2;; i=1
(12)
where is the classification loss and is a hyperparameter controlling regularization strength. During inference, noise and regularization are ommitted, AW, Ab are integrated with Wo. by for efficiency:
W Wo + AW; b bo + Ab; A(X) W +
(13)
Efficient PACE variants. $C, we present two variants that match the computatiional/memory costs of the baseline while achieving superior performance with substantially reduced resources.
4 Experiments
We combine LoRAmul and VPT to form a strong baseline LoRAmu++VTTad outperforming other combinations in most cases. We evaluate method across four visual classification adaptation tasks:
VTAB-IK [78], few-shot learning [30], FGVC [28] and domain adaptation [82]. We demonstrate PACE improves LoRA on GLUE [70] for text classification and GSM-8K [9] for text generation.
Datasets and evluations. VTAB-1K comprises 19 datasets organized into (i) Natural images, (ii) Specialized datasets (remote sensing, medical) and (iii) Structured datasets (scene structure) domains. Each dataset has 1K training examples. Following [78, 28], we use the provided 800-200 train split for hyperparameter selection, evaluate using the full training set and report average accuracy across three trails. Few-shot learning involves 5 fine- fine-grained datasets: FGVC-Aireraft [46], Food101 [4], OxfordFlowers 102 [51], OxfordPets [53] and StanfordCars [35]. Following [30], evaluate 1,

---

Table Results on VTAB-IK with ViT-B/16. Mean Ace. is the average of group mean values.
Natural
Specialized
Structured
Method
SNORB- Axim
Cifar100
Caltech101
Flowers102
SVHN
Sun397
Camelyon
EuroSAr AT
Resise45
Retinopathy
Clevr- Count
Clevr-Dist
DML.ab
KITTI-Dist
dSpr-Loe
dSpr-Ori
N-ORB-EIe Ele
DTD
Pets
Mean
Full
68.9 87.7 64.3 97.3 86.9 87.4
38.8
79.7
95.7
84.2
73.9
56.3
58.6
41.7
65.5
57.5
46.7
25.7
29.1
68.9
Linear
64.4 85.0 63.2 97.0 86.3 36.6 51.0
78.5
87.5
68.5
74.0
34.3
30.6
33.2
55.4
12.5
20.0
9.6
19.2
57.6
VeT-Deep
78.8 90.8 65.8 98.0 88.3 78.1 49.6
81.8 96.1 83.4
68.4
68.5
60.0 46.5
72.8
73.6
47,9
32,9
37.8
72.0
Adapter
69.2. 90.1 68.0 98.8 89.9 82.8 54.3
84.0
94,9 81.9
75.5
80.9
65.3 48.6
78.3
74.8
48.5
29.9
41.6
73.9
AdaptFormer
70.8 91.2 70.5 99.1 90.9
86.6 54.8
83.0
95.8 84.4 76.3
81,9
64.3 49.3
80.3
76.3
45.7
31.7
41.1
74.7
LoRA
67.1 91.4 69.4 98.8 90.4
85.3 54.0
84,9
95.3 84,4 73.6
$2.9
69.2 49.8
78.5
75.7
47.1
31.0
44.0
74.5
NOAH
69.6 92.7 70.2 99.1 90.4
86.1 53.7
84.4
95.4 83.9 75.8
82.8
68.9 49.9
81.7
81.8
32.8
44.2
74.2
RepAdapter
69.0 92.6 75.1 99.4
91.8 90.2 52.9
87.4 95.9 87.4 75.5
75.9
62.3 53.3
80.6
77.3 54.9
29.5
37,9
76.1
RLRR
75.6 92.4 72,9 99.3 91.5 89.8 57.0
86.8
95.2 85.3 75.9
79.7
64.2 53,9
82.1
83,9
53.7
33.4
43.6
76.7
GLoRA
76.4 92.9 74.6 99.6 92.5 91.5 57.8
87.3
96.8 88.0 76.0
83.1
67.3 54.5
86.2
83.8
52.9
37.0
41.4
78.0
Baseline
74.9 93.3 72.0 99.4
91.0
91.5
54.8
83.2
95.7
86.9 74.2
83.0
70.5
51.9
81.4
77.9
51.7
33.6
44.4
76.4
+PACE
79.0 94.2 73.6 99.4
92.4 93.7 58.0
87.4
96.4
89.3 77.1
84.9
70.9
54,9 84.3 84.7
57.3
39.3
44.8
79.0
Table 2: Classification accuracy on Few-shot learning with ViT 1/16 pre-trained on ImageNet-2113
Shot
FGVCAireratt
Food101
Flowers 102
Method
2
4
8
16
1
2
4
8
16
8
16
LoRA,M
10.4
15.2
27.2
41.7
59.2
33.9
51.9
59.3
66.0
71.3
93.3
96.4
98.0
98.6
98.7
+PACE
10.7
16.3
28.2
42.1
61.0
40.6
55.9
63.8
70.3
75.2
95.0
98.0
98.9
99.5
99.6
VPT
11.2
15.1
23.7
36.3
51.5
34.3
56.6
64.8
71.7
75.4
94.3
97.6
98.2
99.3
99.6
+PACE
11.6
16.2
24.0
37.0
52.4
39.9
57.2
66.7
72.4
76.1
95.3
97.8
98.6
99.4
99.6
+VPT
10.5
15.6
28.4
44.8
61.8
35.4
54.3
64.8
72.1
76.4
90.4
97.3
98.4
99.4
99.5
+PACE
12.3
16.8
29.9
45.7
62.5
39.3
57.2
66.7
73.4
77.8
93.4
98.1
99.1
99.5
99.7
OxfordPets
StanfordCars
Average
LoRA,s
73.2
83.1
87.5 89.2
91.1
8.7
15.3
30.2
55.3
74.5
43.9
52.3
60.4
70.1
78.9
+PACE
75.3
85.0
90.7 90.8
92.4
9.4
16.0
30.9
56.1
75.9
46.2
54.2
62.5
71.7
80.8
VPT
75.9
85.6
90.3
90.6
92.3
9.3
15.0
27.8
46.6
65.1
45.0
53.9
60.9
68.9
76.7
+PACE
78.2
87.4
90.3
91.1
92.3
9.9
15.4
27.9
47.0
65.9
46.9
54.8
61.5
69.3
77.2
LoRA ndd
69.9
84.1
89.1
91.3
91.9
9.0
16.3
32.7
59.0
76.4
43.0
53.5
62.6
73.2
81.2
+PACE
76.5
88.0
90.3
91.4
92.4
9.7
16.4
33.7
59.8
77.3
46.2
55.3
63,9
73.9
81.9
2, 4, 8 and 16 shots, train on the provided training set, tune hyperparameters using validation and report average test accuracy over three random seeds. FGVC includes 5 fine-grained datasets: CUB- 200-2011 1 [69]. NABirds [66]. OxfordFlowers [51], StanfordDogs [10] and StanfordCars [35]. We follow [28] to use validation
for hyperparameter and report test results. For domain adaptation, following [82, 7], we train on ImageNet (1 with a 16-shot setting, use the validation split by [82] for hyperparameter selection and report the results the official validation
and 4 out-of-domain datasets: ImageNet-Sketch [71]. V2 [56], ImageNet-A [23] and ImageNet-R R [22]. We evaluate on GLUE [70] for text classification and GSM-8K [9] for mathematical reasoning.
Pre-trained backbones. We experiment with two vision transformers, Vision Transforms (ViT-B/16) [16] and Swin Transformer (Swin-B) [44]. These two are pre-trained on ImageNet-21K [11]. We test a ViT-B-Laion LNiK model, pre-trained on Laion-2B 2B [60] and fine-tuned on ImageNet- 12K [11]. We use RoBERTapese [43] and 1iii---min-44--iie
[1] for text classification and generation.
Implementation details. We follow [28] for image processing: 224 x 224 resizing for VTAB-IK: random flips and crops to 224 x 224 for FGVC and few-shot learning; stronger augmentation for domain adaptation task, following [16, 82, 41]. We use the Adam optimizer [32] with cosine learning rate decay and linear warm-up (first 10 epochs). Models are fine-tuned for 300 epochs on VTAB-IK and 100 epochs on other vision adaptation tasks, with batch size 64. For text classification we follow [26]. See $G for mathematical reasoning details. All experiments used an NVIDIA H100 GPU. Baseline.
For each dataset, we identified the better method (LoRAmuu+VPT..u add or LoRAnas) and tuned the rank, learning rate, and weight decay to form a strong baseline. The detailed baseline settings for each task and the number of trainable parameters are provided in $F, where LoRAmut VPT ndd gener- ally outperformed other variants. Building on the strong LoR V we use the grid search for our À and a, following strategies from previous studies [28, 41, 26]. Beyond LoR A V add. PACE also enhances PEFT methods such as AdaptFormer, GLoRA, COFT, and BOFT ($D.4).
7

---

Table 3: Results FGVC with ViT-B/16.
Table 4: Results domain adaptation with ViT-
* denotes using augmented ViT by AugReg [62].
B/16 pre-trained on ImageNet-21K
Method
CUB
NA-
Oxford Stan. Stan. Mcan
Method
Source
Target
Mcan
-2011
Birds
Flowers
Dogs
Cars Acc.
ImageeNe
-Sketch
VV2
-A
-R
Acc.
Full
87.3
82.7
98.8
89.4
84.5
85.9
Full
63.9
18.5
52.5
3.2
21.2
31.8
Linear
85.3
75.9
97.9
86.2
51.3
79.3
Lincar
67.9
14.4
60.8
9.4
25.6
35.6
VPT
88.5
84.2
99.0
90.2
83.6
89.1
Adapter
70.5
16.4
59.1
5.5
22.1
34.7
LoRA
88.3
85.6
99.2
91.0
83.2
89.5
VPT
70.5
18.3
58.0
4.6
23.2
34.7
SSF*
89,5
85.7
99.6
89.6
89.2
90.7
LoRA
70.8
20.0
59.3
6.9
23.3
36.0
ARC
89.3
85.7
99.7
89.1
89.5
90.7
NOAH
71.5
24.8
66.1
28.5
40.5
RLRR
89.8
85.3
90.0
90.4
91.0
GLoRA
78.3
30.6
67.5
13.3
31.0
44.1
LoRAmo+
88.9
87.1
99.4
91.2
87.5
90.8
LoRAma+
78.3
30.6
68.5
14.1
32.5
44.8
+PACE
89.8
87.3
99.5
92.2
88.8
91.5
+PACE
79.0
31.8
69.4
16.3
35.2
46.3
Table 5: Results for GLUE w/ RoBERTatsse Matthew's correlation for COLA, Pearson correlation for STSB, and accuracy for others.
Table 6: Results for GSM-8K
using
Mcthod
COLA
STSB
MRPC
RTE
QNLI
SST2
Avg.
Method
Accuracy
Full
63.6
91.2
90.2
78.7
92.8
94.8
85.2
BitFit
62.0
90.8
92.7
81.5
91.8
93.7
85.4
Pre-trained
62.01
Adapt
62.6
90.3
88.4
75.9
93.0
94.7
84.2
Full
73.16
VeRA
65.6
90.7
89.5
78.7
91.8
94.6
85.2
LoRA
75.66
LoRA
63.4
91.5
89.7
86.6
93.3
95.1
86.6
+PACE
66.2
92.0
91.4
86.9
93.6
95.6
87.6
+PACE
78.77
Table 7: Classification results on domain adaptation and CIFAR-100 in VTAB-IK based different pre- rre-trained models. Src. is short for 'source' in Table 4.
VTT-B (imageNet-2IK)
VTT-B (Laion2B Baannl---mageee----- 12K)
Swin-B (ImageNet-2]1)
Method
CIFAR
Iet
CIFAR
ImageNel-IK
CIFAR
ImageNet-11 -S
-100
Src.
-S
-V
-A
-R
-100
Src.
-S
-V
-A
-100
Src.
-A
Full
51.6
63.9
18.5 52.5
3.2
21.2
51.2
66.0
29.0
56.1
8.1
27.9
65.6
71.7
27.0
61.1
10.8
24.4
Linear
63.4
67.9
14.4
60.8
9.4
25.6
61.9
79.2
43.2 69.5
23.4 40.9
65.0
78.8
36.7 68.8
23.2 35.9
LoRAzdd
71.2
73.8
27.1 64.8
13.6
25.0
71.3
77.5
39.8
67.8
20.4
35.6
74.3
76.3 30.7 65.7
16.8 28.9
VPT.a
73.6
74.3
27.1 65.9
11.5
26.7
71.8
78.4
40.4
68.7
22
38.4
72.7
76.2
30.6 66.2
17.6 29.1
LoRAmal
73.4
78.1
31.2 68.3
13.4
32.7
73.2
78.6
41.9
68.8
22.6
37.8
73.9
76.1
30.8 65.7
18.1 28.9
LoRAgia + +VP add
70.3
76.8
28.7 66.6
13.7
29,9
71.8
78.0
41.4
68.3
20.6
36.9
74.5
76.3
30.7 65.7
16.8 28.9
LoRAmui VVT add
74.9
78.3
30.6
68.5
14.1
32.5
73.8
78.3
41.5
68.6
21.6
38.2
74.6
76.6
31.2
66.5
18.5 29.4
+PACE
79.0
79.0
31.8
69.4
16.3
35.2
78.0
80.1
45.8
71,2
24.6
43.6
78.9
79.6
39.2
70.1
25.2 38.0
4.1
Comparison with the State of the Arts
Results VTAB-1K. Table 1 presents the results comparing PACE with recent state-of-the-art PEFT methods. PACE improves the strong baseline by 2.6% accuracy, surpassing the previous SOTA GLoRA [7] by 1 1%, which uses two stages for parameter search. In SD.I, show that reducing training epochs to 50 or 100 has minimal impact on PACE performance.
Results Few-shot Learning. Table 2 compares performance w/ and w/o our PACE. PACE improves LoRAgau VPT LoRAm add. with LoRAmuI+ +PACE performing best in most cases. PACE yields notable improvement, especially when the number of shot is small.
Results on FGVC. Table shows that PACE improves the strong LoRAmu+ VVPT sas by 0.7%, outper- forming SSF [41], ARC [14] and RLRR [15] that use strongly pre-trained ViT with augmentations. In $D.2, PACE achieves larger improvements on smaller datasets.
Results on domain adaptation. Table 4 compares PACE with others. LoRAmat VPT add outperforms GLoRA [7] which relies on parameter search. Mcanwhile, PACE improves AoRAmut+VPT by 1.5%, outperforming other PEFT methods, demonstrating superior performance on domain adaptation.
Results on text classification and mathematical reasoning. Table 5 shows that PACE outperforms LoRA by 1% on GLUE text classification and by 3.11% 3.11% on GSM-8K mathematical reasoning.
Generalization on other backbones. We evaluate PACE on CIFAR- 100 (VTABB 1K) and domain adaptation using Swin-B [44] pre-trained on ImageNet-21 K and ViT-B (pre-trained on Laion 2B, then fine-tuned ImageNet- 12K). Table 7 shows PACE outperforms AoRAmu++PP VPT and other PEFT methods across all backbones, demonstrating strong generalizability. Further experiments in $D.3 show PACE works effectively with self-supervised models such as MAE [19] and DINO [5].
8

---

4.2 Analyses
To verify our theories, we conduct experiments on CIFAR-100 (VTAB-IK) using V/T- and Camelyon (VTAB-1K) on Swin-B. Figures & 4 show the gradient norm (summed across all layers) and FP-distance (Eq. 5) and the train & validation accuracy during training for baseline LoRAm add and PACE validation set. Figures 3a & 4a show that PACE has a smaller gradient norm than baseline, verifying Theorem that PACE can implicitly lower the weight gradient norm for better generalization. Figures 3b & 4b demonstrate that PACE maintains lower FP-distance than the baseline, verifying Theorem 3 that PACE can implicitly align the fine-tuned model with pre-trained model, retaining knowledge from large-scale pre-training. Owing to the advantages of the gradient regularization and model alignment, PACE shortens the performance gap between seen and unseen data, yielding higher accuracy on the unseen validation set, shown in Figures 3c & 4c.
09 l
D
Acc
12e3
140
100
Baseline +PACE
Baseline +PACE
train acc
Bascline PACE
9e3
100
90
val acc
6e3
60
80
3c3
20
70
epoch=
100
200
300
epoch=
100
200
300
epoch=
100
200
300
(a) Gradient Norm.
(b) FP-Distance
(c) Train and validation accuracy.
Figure 3: Analysis for PACE. (a) gradient norm, (b) FP-Distance and (c) train & val. accuracy are
evaluated on validation set of CIFAR- 100 (VTAB- IK) with baseline LoRAmutVPPT on ViT-B/16.
D
Acc
70
8e3
100
1.00
6e3
70
Baseline +PACE
0.95
train acc
Baseline PACE
val
4e3
Baseline +PACE
40
0.90
2c3
10
0.85
epoch=
100
200
300
epoch=
100
200
300
epoch=
100
200
300
(a) Gradient Norm.
(b) FP-distance
(c) Train and validation accuracy.
Figure 4: Analysis for PACE. (a) gradient norm, (b) FP-Distance and (c) train & val. accuracy are evaluated on the validation set of Camelyon (VTAB-IK) with baseline
Swin-B.
To clarify why naive alignment is problematic, we vary the regularization strength À over a wide range (Ie-3 5e4) for both Fine-tuned Pre-trained model Alignment (FPA) by minimizing D in Eq. 5 and PACE. Figure shows the averaged gradient norm over training (see also Figures 8 & for more visualizations). PACE robustly lowers gradient norms with larger X, while FPA exhibits unpredictable behavior, even causing gradient explosion. This verifies Prop. that minimizing D is problematic for gradient regularization, complicating gradient management.
0s
79.0
20
Baseline
78.5
+FPA
78.0
204
+PACE
M
3
4
6
7
79.0
le4
78.5
78.0
A- 1e-3 0.05
500
564
N=
2
4
8
10
12
Figure 5: Gradient norms of models across wide range of regu-
Figure 6: Ablation results for ap- plying PACE among M nets and lazily at every N steps.
larization strengths À CIFAR-100 (VTAB-IK) w/ ViT-B/16.
Line and shadow represent mean and across training epochs.
4.3 Ablation studies
We ablate PACE based on the baseline LoRAmut VPT nid on CIFAR-100 (VTAB-IK) and ImageNet- IK in domain adaption as shown in Table 8. The ablations include Noise (baseline w/ noise perturbing

---

adapter), PACEa (replacing the multiplicative noise with the additive noise), PACE₁ (perturbing h(-) instead of Ah(-) Eq. 11), PACEdmp (replacing the Gaussian noise with the dropout noise), PACE,= (all transformer blocks share the same d), PACEat (a increases linearly with depth), FPA (fine-tuned and pre-trined alignment by minimizing Eq. 5), SAM (sharpness-awe) awwpre minimization [17]) GP (gradient penalization), & (sparsity regularization), and transfer learning methods L.SPP [77], DELTA [40] and FTP [64]. We grid-search hyperparameters and report the best results.
Table 8 presents the results for all variants, PACE improves over Noise, which itself is better than baseline, justifying our adapter perturbation and consisteney regularization. PACE performs worse than PACE, showing the superiority of the multiplicative noise. Although PACE, can implicitly regularize gradients, it performs worse than PACE, verifying the advantages of perturbing adapter to implicitly align models. PACEdmp is worse than PACE, indicating the dropout noise is suboptimal. PACE,= and PACEat perform worse, justifying our design of linearly decreasing a. FPA, SAM and GP, which either only align models or only regularize gradients, are outperformed by PACE. Despite combining FPA+GP, it still performs worse than ours, suggesting ineffective combination. L, L2SP, DELTA, and FTP obtain worse results than PACE, showing their limitations in improving generalization. PACE regularizes gradients for better generalization and aligns models to retain knowledge, surpassing all other variants.
CIFAR
ImageNer-IK
1-Shot
2-Shot
Method
-100
Source
Skktthh -V2 -A
-R
0.02
75
0.02
87
LoRAmut VVT Add
74.9
78.3
30.6
68.5
14.1 32.5
+Noise
+PACE
79.0
77.4
79.0
78.3
31.8
31.3
68.6
69.4
14.3 33.0
16.3 35.2
0.1
72
0.1
86
PACE,,
75.7
78.3
31.2
68.7
13.7 32.7
0.2
69
0.2
PACE,,
75.9
78.4
31.2
68.1
13.8 32.6
0.5
0.5
PACEEpop
78.3
78.9
31.2
68.9 16.0 34.6
66
84
+PACE
77.9
78.8
31.6
68.3
16.6 34.7
0.5
1.0 1.5
0.5
1.0
1.5
+PACE,
77.3
78.7
31.3
68.9
14.0 33.6
A
4-Shot
A
8-Shot
+FPA
76.6
78.8
31.2
68.6
14.7 33.5
0.02
190
0.02
(17M
75.4
78.4
31.4
68.5
13.8 32.9
+GP
75.8
78.3
31.7
68.4
14.2 32.1
0.05
0.05
+FPA+GP
78.1
31,5
68.1
13.5 32.6
0.1
89
0.1
+f1
75.2
78.2
30.6
68.6
13.7 32.8
+L2SP [77]
75.9
78.5
30.4
68.7
14,9 33.5
0.2
0.2
+DELTA [40]
76.4
78.4
30.8
68.7
14.6
33.7
0.5
0.5
+FTP [64]
76.2
78.6
30.8
68.6
15.8 33.6
a-
0.1 0.2
0.5
1.0
1.5
0-0.1
0.1
0.2
0.5
1.0
1.5
90
Table 8: Accuracy results on domain adaptation and VTAB-IK based different pre-trained models.
Figure 7: Results for varied À and a well as shot on OxfordPets in few-shot learning.
We further evaluate applying PACE across multiple M networks during training applying it lazily with half-batch size at every N steps (PACElal) in $C). Figure 6 presents the results, showing that applying PACE among two networks at every training step performs best. However, lazy regularization applied every few steps can still provide reasonable results while saving computationna/mmemory costs.
We test the sensitivity of hyperparameters and a introduced in our PACE on OxfordPets for few-shot learning across 1, 2, 4, 8 shots. The results presented in Figure 7 demonstrate that with less data, larger and a are favored, verifying the effectiveness of PACE in improving generalization.
Conclusions
We have introduced PACE, novel and effective method that combines generalization of PArameter- efficient fine-tuning with Consistency rEgularization. Through rigorous theoretical analyses, we have shown PACE reduces weight gradient for improved generalization and it aligns the fine-tuned model with the pre-trained model for retaining pre-training knowledge. Our experimental results support the theoretical analyses. justifying the generalization advantages of PACE over other PEFT methods. With its dual advantages, PACE consistently outperforms other variants across different backbones, firmly establishing PACE as a powerful solution for enhancing generalization for PEFT methods. Limitations and border impacts discussed in $A.
Acknowledgments. We thank Moyang Liu, Melody Ip, Chenyi Du, and Yinuo Xu for their valuable discussions and support. PK is funded by CSIRO's Science Digital.
10

---

References
[1] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Iyoti Aneja, Ahmed Awadallah, Hany Awadalla,
Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.142) 2024. 7.
[2] Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the lata-generating
distribution. JMLR, 15(110)374333711 2014, 5
[3 Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint XXx::1409.0473 2014.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-10l-minin discriminative components
with random forests. In Computer Vision-ECC EitionECCV// 2014; 13tth European Conference, Zurich, Switzerland,
September 1-, 2014, 4, Proceedings, Parr VI 13, pages 446-461. Springer, 2014, 6
[5] Mathilde Caron. Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660, 2021. 8
[6] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lec, and Sungrac Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing
Systems, 34:22405-2241 18, 2021. 3
[7] Arnv Chavan, Zhuang Liu, Deepak Gupta, Eric Xing. and Zhiqiang Shen. One-for-all: Generalized lora
for parameter-efficient fine-tuning. arXiv preprint arXiv:2306.0790 2023. 2, 3, 7, 8, 22
[8] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. Advances in Neural Information
Processing Systems, 66:16666166678, 2022. 1, 2, 3, 22
[9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek. Jacob Hilton, Reicchrr Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.14666. 14168, 2021. 6,7, 23
[10] E Dataset. Novel datasets for fine-grained image categorization. In First Workshop on Fine Grained Visual
Categorization, CVPR. Citeseer. Citeseer. Citeseer, 2011. 7
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. pages 248 255.
lece, 2009. 1, 21
[12] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of
quantized lls.. Advances Neural Information Processing Systems, 36, 2024.
[13] Junhao Dong, Piotr Koniusz, Xinghua Qu, and Yew-Soon Ong. Stabilizing modality gap & lowering gradient norms improves sero-shot adversarial robustness of vlms. 31 SIGKDD Conference on Knowledge Discovery and Data Mining, 2025. 3
[14] Wei Dong, Dawei Yan, Zhijun Lin, and Peng Wang. Efficient adaptation of large vision transformer via adapter re-composing. Advances in Neural Information Processing Systems, 36, 2024. 2, 8
[15] Wei Dong. Xing Zhang, Bihui Chen, Dawei Yan. Zhijun Lin, Qingsen Yan, Peng Wang. and Yang Yang. Low-rank rescaled vision transformer fine-tuning: A residual design approach. arXiv preprint
arXiv:2403.1906 2024. 2, 8
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at seale. In International Conference on Learning Representations, 2021. 1,
[17] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving gencralization. In International Conference on Learning Representations, 2021. 3, 4,
5, 10, 18
[18] Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. On the effectiveness of parameter-efficient fine-tuning. In Proceedings the AAAI Conference on Artificial Intelligence, volume 37, pages 12799-12807, 2023. 2, 3,
[19] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16000-16009, 2022. 8, 21.
[20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and puttern recognition, pages 9729 9738, 2020.
[21] Zhezhi Hc, Adnan Siraj Rakin, Jingtao Li, Chaitali Chakrabarti, and Deliang Fan. Defending and harnessing the bit-flip based adversarial weight attack. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14095-14103, 2020. 4
[22] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of ont-of-diste distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8340 8349,8399,4000 2021. 7
11

---

[23] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings the IEEE/CVF conference on computer vision and pattem recognition, pages 15262-15271, 2021.
[24] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Auariyan, and Sylvain Gelly. Parameter-efficiee transfer learning for alp. In International conference on machine learning, pages 2790-2799. PMLR. 2019. 1
[25] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzchski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-effictinn transfer learning for nlp. In International conference on machine learning, pages 2790-2799. PMLR, 2019. 2.3
[26] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representationss 2022. 2, 3, 7
[27] Shengding Hu, Zhen Zhang. Ning Ding, Yadao Wang, Yasheng Wang, Zhiyuan Liu, and Maosong Sun. Sparse structure search for delta tuning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022. 2
[28] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727.
Springer, 2022. 2, 4, 6, 7, 22
[29] Zeyinzi Jiang, Chaojie Mao, Ziyuan Huang, Ao Ma, Yiliang Lv, Yujun Shen, Deli Zhao, and Jingren Zhou. Res-tuning A flexible and efficient tuning paradigm via unbinding tuner from backbone. Advances in Neural Information Processing Systems, 36, 2024. 2
[30] Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 1060-1068, 2023. 2, 6
[31] Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 15190-15200 2023.
[32] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6908 42.69800. 2014.7
[33] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023. 1
[34] Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: fector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024. 3
[35] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013. 6.7
[36] Yoonho Lee. Annie S Chen, Fahim Tajwar, Ananya Kumar. Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. In The Eleventh International Conference on Learning Representationss 2023. 2
[37] Bonan Li. Yinhan Hu, Xuecheng Nie. Congying Han, Xiangjian Jiang, Tiande Guo, and Luoqi Liu. Dropkey for vision transformer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 22709, 2023.
[38] Dongyue Li and Hongyang Zhang. Improved regularization and robustness for mne-tuning in neural
networks. Advances in Neural Information Processing Systems, 01-22299-22666 2021. 3
[39] Shengrui Li, Xueting Han, and Jing Bai. Adaptergnn: Parameter-effiiint fine-tuning improves gener- alization in gnns. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13600- 1 13608, 2024. 2
[40] Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, Zeyu Chen, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. arXiv preprint wwXiv:9911.09299 2019. 3, 10
[41] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features; A new baseline for efficient model tuning. In Advances in Neural Information Processing Systems (NeurIPS),
2022. 2, 3, 7, 8,22
[42] Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juycon Hco, Songyou Peng, Yandong Wen, Michael J. Black, Adrian Weller, and Bernhard Schölkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. In ICLR, 2024. 2, 22
[43] Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1
2019. 7
[44] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021. 7,8
12

---

[45] Gen Luo, Minglang Huang. Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang. Zhiyu Wang, and Rongrong Ji. Towards efficient visual adaption via structural re-parameeterization. arXiv preprint arXiv:2302.08//6 2023. 2, 4, 22
[46] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual
classification of aireraft. arXiv preprint arXiv:1306.515// 2013. 6
[47] Yao Ni and Piotr Koniusz. Nice: Noise-modulated consistency regularization for data-efficient gans.
Advances in Neural Information Processing Systems, 1111133777180.11 2023. 2, 16
[48] Yao Ni and Piotr Koniusz. Chain: Enhancing generalization in data-efficient gans via lipschitz continuity constrained normalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern
Recognition (CVPR), pages 6763-6774, June 2024, 3
[49] Yao Ni, Piotr Koniusz, Richard Hartley, and Richard Nock. Manifold learning benefits gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,, pages 1 1265-11277, 2022. [50] Yao Ni, Dandan Song, Xi Zhang, Hao Wu, and Lejian Liao. Cagan: Consistent adversarial training
enhanced gans. In IJCAI, pages 2588 2594, 2018. 2
[51] Maria-Elewe Nilsback and Andrew Zisserman. A visual vocabulary for flower classification, In IEEE Conference on Computer Vision and Pattern Recognition, volume 2. pages 1447-1454, 2006. 6,
[52] Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung. Hosik Choi, and Kyungwoo Song. Blackvip: Black-box visual prompting for robust transfer learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 24224-2444444 24235, 2023.
[53] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE
Conference on Computer Vision and Pattern Recognition, 2012. 6
[54] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue. Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling texl-to-image diffusion by orthogonal finetuning. In NeurlPS, 2023. 2,
22
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 3748-8763. PMLR, 2021.
[56] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on muchine learning, pages 5389-5400. PMLR, 2019.
[57] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolttion image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[58] Shuvendu Roy and Ali Etemad. Consistency-guides prompt learning for vision-language models. In The Twelfth International Conference on Learning Representations, 2024. 3
[59] Kuniaki Saito. Donghyun Kim, and Kate Saenko. Openmatch: Open-set sumi-super learning with open-set consistency regularization. Advances in Neural Information Processing Systems, 34:25956-25961 2021. 2
[60] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski. Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-SU An open large-scale dataset for training next generation image-text models. In Thitty Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. 1,7
[61] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Fkin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
[62] Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research, 2022. 8
[63] Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi L.i, et al. On transfcrability of prompt tuning for natural language processing. Xiviv preprint arx:: 1eee22111.07119 2021. 1,2
[64] Junjiao Tian, Yen- Cheng Liu, James S Smith, and Zsolt Kira. Fast trainable projection for robust fine-tuning.
Advances in Neural Information Processing Systems, 36, 2024. 3. 10
[65] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Ilama: Open and efficient foundation
language models. arXiv preprint 2302v: / 13971, 2023. 1
[66] Grant Van Horn, Steve Branson, Ryan Farrell. Scott Haber, Jessie Barry, Panos Ipcirotis. Pietro Perona, and Serge Belongie. Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine- mne-grrined dataset collection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 595-604. 2015. 7
13

---

[67] Dániel Varga, Adrián Csiszárik, and Zsolt Zombori. Gradient regularization improves accuracy of discriminative models. arXiv preprini arXi: 029929.999..... 2017.
[68] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. 2,
[69] C. Wah, S. Branson. P. Welinder. P. Perona, and S. Belongie. Shecaltech-ucsddirds--00
dataset.
Technical Report ENS-TR-200 1-001, California Institute of Technology. 2011. 7
[70] Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding.
arXiv preprint (0Xx::8804007461 2018. 6.7
[71] Haohan Wang. Songwei Ge. Zachary Lipton, and Eric P Xing. Learning robust global representations by
penalizing local predictive power. Advances in Newral Information Processing Systems, 32, 2019. 7
[72] Yihan Wang, Jatin Chauhan, Wei Wang, and Cho-Jui Hsieh. Universality and limitations of prompt tuning.
Advances in Neural Information Processing Systems, 36, 2024. 2.3
[73] Yaoming Wang. Yuchen Liu, Xiaopeng Zhang, Jin Li, Bowen Shi, Chenglin Li, Wenrui Dai, Hongkai Xiong, and Qi Tian. Violet: Vision-language efficient tuning with collaborative multi-modal gradients. In Proceedings of the 31 ACM International Conference on Multimedia, pages 4595-4605, 2023. 3
[74] Yeming Wen and Swarat Chaudhuri. Batched low-rank adaptation of foundation models. arXiv preprint
arXiv:2312.00 12.067777777... 2023. 2
[75] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. Advances in neural information processing systems, 33:2958-2966 2020. 4
[76] Lijun Wu, Juntao Li, Yue Wang. Qi Meng. Tao Qin, Wei Chen, Min Zhang, Tie- Liu, et al. R- drop: Regularized dropout for neural networks. Advances in Neural Information Procexsing Systems, 14:19000-9900 2021. 2
[77] Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International Conference on Muchine Learning, pages 2825-2834. PMLR,
2018. 3, 10
[78] Xiaohua Zhai, Joan Puigcerver. Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme. Mario Lucic. Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 1910,0467, 2019.
[79] Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lec. Consistency regularization for generative adversarial networks. arXiv preprint arXiv; 1910. /2027, 2019. 2
[80] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In /CLR, 2021.
[81] Shan Zhang, Yao Ni, Jinhao Du, Yanxia Liu, and Piotr Koniusz. Semantic transfer from head to tail: Enlarging tail margin for long-tailed visual recognition. In Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1350-13600 1360, 2024. 3
[82] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt scarch. arXiv preprint arX1v:2206004673,
2022. 6,7
[83] Yifci Zhang, Hao Zhu, Aiwei Liu, Han Yu, Piotr Koniusz, and Irwin King. Less is more: Extreme gradient boost 1 adaption for efficient finetuning of llms. arXiv preprint arXiv:2410. 19694, 2024. 3
[84] Bingchen Zhao, Haoqin Tu, Chen Wei, Jieru Mei, and Cihang Xie. Tuning layemorr in attention: Towards efficient multi-modal LLM finetuning. In The Twelfth International Conference on Learning Representations, 2024. 2
[85] Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference on Machine Learning, pages 26982-26992. PMLR, 2022. 3
14

---

PACE: Marrying generalization of PArameter-efficiet fine-tuning with Consistency r'gularization (Supplementary Material)
Yao Ni Shan Zhang :-1 Piotr Koniusz +:5
The Australian National University $Cata6
Australian Institute for Machine Learning, The University of Adelaide
yao ni ninanu. edu au shan, shaan. zhangdadelside. edu au spietr moniusz@datail.ccirr
A Broader impacts and limitations
A.1 Broader impacts
Our work provides a powerful solution for improving generalization in Parameter Efficient Fine- Tuning (PEFT), allowing for effective fine-tuning of pre-trained models while reducing the heavily reliance on pre- pre-training from scratch using large scale data. Our advancements PEFT, supported by Theorems 1, and 3, offer novel insights into gradient regularization and model alignment. These insights extend beyond PEFT and can be applied to other areas such as continual learning and transfer learning, potentially enhancing the performance and efficiency of models in various domains. By leveraging our findings, practitioners can develop more robust and adaptable models that generalize well to tasks and environments, leading more intelligent and versatile AI systems. In terms of negative impacts, the robustness of our fine-tuning method could potentially be misused to create more convineing deepfakes, raising concerns about the spread of misinformation. manipulation of public opinion, and malicious activities such as fraud, blackmail, or harassment. However, potential misuse is a downside with any improvements that have universal nature.
A.2 Limitations
While work effectively improves generalization ability, introduces additional computational costs by requiring input samples to be passed through the network twice for regularization. However, this can be mitigated by using two efficient variants, PACEfst and PACEbaaí proposed in $C, where method introduces
demonstrate the potential for resource-efficiemn fine-tuning. Additionally, extra hyperparameters and σ, which require caution during hyperparameter search. Nonetheless, Figure 7 suggests that fewer training data requires larger À and a values, providing insight for hyperparameter tuning.
'The corresponding author.
15

---

B Proofs
B.1 Proof of Theorem 1
Settting e = Vall
Ve
we perform a second-order Taylor expansion of Lpn around 0. By incorporating
0
the higher-order terms from the Taylor expansion into It
we derive:
p
n2
p
0
12
1
L
L
0
R
Vol2
7
p
11013 O² 7
LDn 0 + p
Voll2 +
HoVV R
(14)
V
Assuming that the approximation does not alter the inequality relationship, Le., it preserves the S relation on both sides and considering the largest eigenvalue of H a A implying v' Hov S A 2l1 for any v. we further bound Eq. 14 as follows and arrive at:
p²
1 13
1 1
(0) < LD" ( + pl Vella
+ R
max
B.2 Proof of Theorem 2
The proof is motivated by Ni and Koniusz [47]. We include the proof process for completeness. Denote m 2 1, m2 22 thus m. m2 NN o²)
Juce = -E =2 [f(0 + @ A0) [(0 + 22 ① A0)]²
-1E,, #2 [f(0o A0 + (=1 1) @ A0) f(0. + AO (z2 1) 3 A0))
E Em ma [f m A0) f(@ + m2 @ A0)
(15)
Defining v = m. @ A0 and u := m2 @ AO, where ~ NO diag A0 @ AO)), we can
rewrite Eq. 15 as follows:
Ef,ulf(0 + v) - f(O + =
1 0"
R. [5()) + 4² V +
HU [(0) n² V
u² Hu
1 8"
1 u Hu]
-E -E, [2" V
+
Hu u
2
2
-Ev,u [vV - u) V
T Ho u Hu 2
2
EE (u - (uo-u)7 T
(16)
+ Ev,u ((v u) v) (v Hv - u Hu)
(17)
+ Ev(e" " + 1
7 Eulu² Hul
(18)
4
B..u [(0" Hv) (u" Hu)).
(19)
Next, we derive the four terms, Eq. 16, 17, 18, and 19, respectively as follows:
Eq. 16. Using (,, 22 ²² 20 for 21, 22 (O o²), we can simplify (Eq. 16) as follows, noting that terms related to different dimensions are canceled due to zero-mean independent Gaussian noise:
E,, [vv u) ' Ev,u EC - -(),-,)) 20t Σ A
(20)
3
Eq. 17. Utilizing E[ = Sي² + 3uo² for N (W,, o²), and noting that E| = for μ = 0, Eq.
17 is derived as:
Ev,u ((v u) 3)TT v (v H xt Hu)
=E, (v V ( Ho) Hb (u" Hu)] E, (v" -E,, ".o..((u' (("T))(' Ho)) Hv)
2E (v ( (8 Hv) 0.
(21)
16

---

Eq. 18. We first decompose Eq. 18, then discuss each case and obtain the final result:
1 E.p [V' Ho 2
= Bulv" Ho]² + Eu Hu)²
Exp
vyH vp Hp j.k.p.g
Σ
(22)
Given the independence of elements v. only terms with an element repeated two or four times
contribute non-zero results, leading to four distinct, non-overlapping cases. Using B[z²] = 0² + 11²
and [=[="] = ja" 64t20 + 30 for 2 ~ N(u, and simplifying to B[2²] a² and B[e*] 30
when μ = 0, we have:
Case I: j k + p= q, given the independence of Uy and Up. have:
E ΣΣ of o
Σ j.Pj
H4s p
a² Σ
Ho.HHADAA00.. (23)
pfj
Case 2: For j = k= q, the independence of oj and up simplifies our calculation, leading to:
E., ΣΣΣn U H yu [zvuR]
Σ FA
o² Σ H3. (24)
J.KF
Case 3: For = A k = utilizing the independence of Uy and as well as the symmetry
5 = Hkj, we obtain:
Ev Σ>
by Hajj
= Σ
Hog B = a Σ
(25)
kgj
j.pj
jktj
Case 4: For j =q= k = p. using B[2*]
30 where 2 NO0 o?), we have:
Ev
uj ب
Σ
U3 30 Σ E13,003
(26)
3
Combining above four cases together, we have the result for Eq. 18:
σ² 2
3H A0 +
HH + 207
(27)
1,kz
Eq. 19:
Eu [u" Hv)(u Hu)
2
Ey [To" Hei] Eu [u Hu)]
Ev Σ H 2ز Eu Σ HaLu
Σ
H Blij) Σ HnLE(UR)
Σ
H2 + Σ
(28)
2
3
With results of Eq. 20, 21, 27, 28, we have the final results:
pace =20
V
+
i
0
3H AB
I (H HAS + A0
-HH -HH
j.ktj
jktj
22
Σ
A0VY + Σ A0 + Σ H2A
1
-202
Σ
10 + Σ
E.AA00AA0
20 20 AA = H (29)
j.k
17

---

B.3 Proof of Theorem 3
The Mauchy-Schward inequality states that for u, R we have (E, ugu,) ",, vz).
Let follows that (E, 22)² I dlle| Using this inequality, we then prove the following:
1 _ ~0 HAOR |00" V
HDO
|AQ" (Σ 5dIA0 vI
(30)
A HAO
Σ (0,00HH11,) S <'l(<)))")) o HIE
(31)
J.k
Here, the inequality is obtained by treating A0 Hfk an element of a vector with size of d? This leads the final results.
B.4 Rationale for one-dimensional output analysis
We use the squared L2 distance for multi-dimensional outputs for D and Drace, which allows our one-dimensional analysis to naturally generalize to multiple dimensions. For example, for vector-valued function in the naive alignment, f(0) fm where is the output dimension, we have:
ll((0)) - f(e. + A0)1 = > (f((0)) + 2
This equality shows that the squared L2 distance in multiple dimensions simply the sum of non- negative squared differences in each dimension. Consequently, this additive nature enables our one-dimensional analysis to extend seamlessly to multiple dimensions practice, aligning with our empirical observations.
B.5
R increases with
0012 According to [17], the function R( p 1 Eq. 3 defined as:
0
1
A: log
[ 1+ 00
log n
+ 4 loog Y + 8 log(6n + 3k)
R
Here is the number parameters, n is the number of training samples, & (0, 1] is the confidence level and p is the max norm of the Gaussian perturbation noise.
| and break the
To ensure R valid, we require 1. To analyze how R changes with n, we fix expression under the square root At into three terms:
log (1 + I
2
4 log n log
8 llog (6n + 3k)
R1 We analyze each term separately to determine whether it decreases with increasing n. Analysis for Rú: The derivative for Rí w.r.t. is:
R
R3
n-1
18

---

1
loga
(n
1)
log
1+
10
(1+
log
km
R4
(n 1)3
Ing
n-1
t: log
1 + I1011
n
1)2
Ing
log
[1011 (1 + V
log n b
1+v 1sg
(n 1)2
log
0 1
+ log (1
log k
19
lown
nn - 1)2
k log 1011
klog (1+ V
Iog b
(n 1)2
k
|0 p²
log (1 V log n )
log
(n 1)2
log n
Since 1012 generally large, the smallest n 2 and log R2 0, meaning Ri decreases increase.
log )² 0. Therefore, for n > 1,
Analysis of R2: The derivative for R2 w.r.t. ni is
4
R2
log + log 8).
1)3
Since 8 < for n >
R O. indicating that R2 decreases with increasing n.
Analysis of Rg: The derivative for R3 w.r.t. n is
8 6(n-1)
log (6n + 3k)) (n 1)2
8(1 - log(6n + 3k))
R3
6n + 36
(n 1)2
For 1, log( 6n + 3k) 1, implying that R'g 0 and R3 decrease increases.
Conclusion. For n > I, all terms R1, R2 and R3 decreases as increases. Thus R 1013 1) is decreasing function of n.
C Efficient PACE variants
Building upon strong theoretical foundation of PACE for generalization, demonstrate that simple modifications can reduce memory and training time requirements PACE. In this section, we explore two efficient variants, PACEra and PACEbal 'lay" both maintaining similar computational and memory requirements the baseline while improving performance. We then provide empirical results which show that PACEaa slightly outperforms PACH using fewer computational resources. Given its superior efficiency, we further explore the potential of PACEast for resource-efficien fine-tuning. By simply reducing the batch size and epochs, PACEast outperforms the baseline while using significantly less GPU memory and training time.
Plazy while requiring no additional hyperparameters and
PACE
Building on the observation that only small datasets are typically available for fine-tuning, we assume that the model behavior changes gradually across epochs. Under this assumption, we store the model outputs from the previous epoch (fa-1 (z))), which contain inherent noise due to the adapter perturbation, and compute the consistency regularization loss between these stored outputs and the current epoch's noised outputs:
dace (x) [[(()) 0
where Oc-1 fx)
(32)
19

---

Here the output vector E R where C' is the number of classes. Since f() applies noise perturbation the adapter and changes gradually between epochs. f and f()) can be seen as applying different i.id. noises to similar model states. This approach preserves the theoretical foundation of PACE while incurring minimal storage and computation costs. With typically few classes C' and limited number of samples in fine-tuning, storing 0 within GPU or CPU memory is manageable.
lazy' PACE During training, the network always applies noise perturbations. Every N-th iteration uses a half batch size and consistency regularization, while all other iterations use the full batch size.
Memory and computational efficiency of two variants. Both variants maintain similar compu- tational and memory requirements as the baseline. To demonstrate this, we conduct experiments on CIFAR- 100 (VTAB-IK) K)) using ViT-B/16, Camelyon (VTAB-IK) I with Swin-B, and ImageNet (domain adaptation) with ViT-B/16. Table 9 compares maximum GPU memory usage, total training time, and accuracy for each task, showing that PACE and PAC significantly improve upon the baseline while maintaining similar computational demands.
We find that PACE slightly outperforms PAC without requiring additional hyperparameters, yet it needs to store outputs from the previous epoch. We therefore analyze its memory requirements.
Table 9: GPU memory usage, training time, and accuracy for PACE and PACE here, 'm' denotes minutes, Both variants outperform the baseline while maintaining similar computational demands.
CIFAR-100 (ViT/16-B)
Camelyon (Swin-B) B)
ImageNet (VTT/16-B)
Method GPU Memory
Time Accuracy
GPU Memory
Time Accuracy
GPU Memory
Time
Mcan Acc.
8.9GB
29m
74.6
15.7GB
33m
86.7
8.9GB
161m
44,8
+PACE
17.7GB
53m
79.0
29.4GB
89.3
17.7GB
278m
46.3
+PACEf2st
9.0GB
29m
78.3
15.7GB
34m
88.8
9.0GB
162m
46.1
+PACEbalt
(N 2)
9,3GB
29m
78.7
15.7GB
36m
89.2
9,0GB
165m
46.0
+ +aCC
(N = 4)
9,3GB
29m
78.4
15.7GB
35m
88.9
9,0GB
163m
45.6
+PAC
(N 6)
9,3GB
29m
78.4
15.7GB
35m
89.0
9,0GB
163m
45.7
Tazy
+PACEball
(N= 10)
9.3GB
29m
78.2
15.7GB
35m
88.9
9.0GB
162m
45.6
Tuzy
Memory efficiency of PACErsst We compare the additional memory requirement of PACEfast with the baseline GPU memory consumption. Table 10 shows that the memory overhead of PACE is negligible compared to the baseline GPU memory requirements and can be easily stored in GPU. Moreover, even in the rare scenario of fine-tuning on the full ImageNet 1K dataset (1.2 million samples). PACE requires only 4.8GB of additional memory for storing the output of the model's classification head. This is significantly smaller than the dataset itself (> (>100GB) and can be easily accommodated in the CPU/GPU memory.
Table 10: Comparison of fast memory overhead and the baseline GPU memory requirements.
Dataset
Memory of PACEnst
Bascline GPU Memory
Ratio
CIFAR-IO0 (VTAB-IK VIT/I6-B)
390KB
8.9GB
0.0042%
Camelyon I w/ Swin-B)
7.81KB
15.7GB
0.000047%
ImageNet (Domain adaptation w/ VIT/16-B)
61MB
8,9GB
0.67%
Resource-Efficienn training with PACEpst Given the superior performance, minimal memory overhead, and no need for additional hyperparameters of PACE we explore its potential for resource-efficient training by maintaining the same number of updates with reduced batch size and proportionally reduced epochs. Table 11 shows that even with 1/8 batch size and epochs, PACE still outperforms the baseline by 1.7% while only using 1// GPU memory and ~ 1/4 training time. This demonstrates the robustness and generalization benefits that PACE brings to models, enabling them to excel under constrained training configurations. Such an efficiency is particularly valuable for fine-tuning large foundation models, where resource constraints necessitate small batch sizes and typically lead to sharp loss landscapes, yet the theoretical guarantee of PACE for smooth loss landscapes provides promising solution for these challenges.
20

---

Table 11: Results PaCEfast with a reduced batch size and epochs on CIFAR-100 (VTAB-IK w/ VIT ViT-B/16), Camelyon (VTAB-IK IK w/ Swin-B), ImageNet (Domain adaptaion w/ ViT-B/16), PACEfsst outperforms baseline while using less GPU memory and training time.
CIFAR-100 100
Camelyon
ImageNet
Average
Method
Mem. Time Acc.
Mem.
Time Acc.
Mem.
Time MeanAcc.
Mem.
Time
Acc.
LoRA VPT
8.9GB 29m 74.6
15.7GB 33m 86.7
8.9GB 161m
44.8
11.1GB
74m
68.7
+pACE batch size, epochs)
5.4GB
17m 78.1
8.6GB
21m 88.9
5.4GB 85m
45.8
6.5GB
4lm
70.9
+PACEn
batch size, epochs)
3.5GB
10m
6.0GB
14m
88.7
3.5GB
50m
45.6
4.3GB
25m
70.7
+PACEas
batch size,
epochs)
2.9GB
5.2GB
10m
88.6
2.9GB
32m
45.5
3.7GB
16m
70.4
Table 12: Classification results for different methods VTAB-IK with different training epochs.
#Epoch
Method
Natural
Specialized
Structured
Avg.
530
GLoRA
83,61
87.02
63.27
77.97
100
Baseline
81.94
85.40
61.40
76.24
100
+PACE
83.94
87.44
64.62
78.67
50
+PACE (half batch size)
83.77
87.32
63.92
78.34
200
Bascline
82,28
85,30
61.64
76.40
200
+PACE
84.13
87.57
64.85
78.85
300
Baseline
82.41
85.00
61.80
76.40
300
+PACE
84.32
87.55
65.13
79.00
D
Additional Experiments
In this section, we provide additional experiments of PACE on VTAB-IK with different epochs, vary- ing training data sizes on FGVC benchmarks, self-suuprrvsed pre-trained backbones and combinations with other PEFT methods.
D.1 Experiments of VTAB-1K with different epochs
In Table 1, We use 300 epochs for VTAB-1K tasks as we observed slight improvements over 100 epochs. However, this does not mean PACE requires longer training converge. Since the optimizer uses the cosine learning rate decay, reducing the number of training epochs to 100 has a minimal impact performance, shown in Table 12.
To ensure fair memory and computational budgets, we also tested PACE with half the batch size and 50 epochs. Table 12 shows that under these conditions, PACE still improves baseline accuracy by 2. 2.10 and outperforms the previous SOTA GLoRA, which uses 500 epochs for training and 30 for parameter search. These results demonstrate PACE's efficiency and effectiveness across various training configurations.
D.2 Experiments on FGVC with limited training data
To validate generalization benefits of PACE on limited data settings, we conduct experiments on FGVC using 50%, 20% and 10% of the original training samples. Table shows that PACE achieves larger improvements with smaller data sizes, aligning with our theoretical analyses.
Table 13: Classification results on FGVC using varying percentages of data based ViT-B/16.
CUB
NAB
Flowers
Stanford Dogs
Stanford Cars
Method
50%
20%
10%
50%
20%
10%
50%
20%
10%
50%
20%
10%
50%
20%
10%
baseline
87.1
83.9
79.1
80.7
75.0
70.2
98.5
96.5
93.1
90.6
88.7
86.9
78.7
54.9
30.1
+PACE
88.4
85,5
81.4
82.9
77.5
73.8
99.2
97.9
96.1
91.8
90.9
89.8
80.5
57.3
33.2
D.3
Experiments on self-supervised pre-trained backbones
To further verify the effectiveness PACE on a self-supervised pre-trained backbone, we conduct VTAB-1K experiments SVHN, Camelyon, and Clevr-Count using MAE [19] and DINO [19], with ViT-B/16 pre-trained on ImageNet- [11]. Table shows that PACE improves the baseline on these self-supervised backbones, confirming its applicability to fine-tuning self-supervised models.
21

---

Table 14: Classification results on VTAB-IK using sulf-supervised DINO and MAE, with ViT-B/16
pre- tre-trained on the ImageNet- IK dataset.
MAE
Method
SVHN
Camelyon
Clevr-Count
SVHN
Camelyon
Clevr-Count
Full
90.1
74.6
52.5
89.7
73.1
34.5
Lincar
79.9
50.7
82.5
44.2
LoRAmut
82.7
82.1
85.4
55.7
+PACE
93.5
85.8
86.4
91.7
88.1
61.0
D.4
Experiments of Combining PACE with Other PEFT
We conducted experiments combining PACE with several PEFT methods, including AdaptFormer [8]. GLoRA [7], COFT [54], and BOFT [42], on CIFAR-IO0 (VTAB-IK) and ImageNet (domain adaptation) using ViT-B/16. Table 15 shows that integrating PACE improves the baseline performance.
Table 15: Classification results of different PEFT methods based on ViT-B/16.
ImageNet (Domain Adaptation)
Method
CIFAR-100 (VTAB-IK)
Source
Sketch
-V2
Avg.
AdapiForner
70.6
77.4
26.5
67.4
12.4
28.7
42.4
+PACE
74.8
78.2
27.4
67.9
13,9
31.7
43,8
GLoRA
75.9
78.2
30.3
68.1
13.5
31.6
44.3
+PACE
78.6
78.8
31.7
69.0
15.9
34.4
45.9
COFT
71.8
76.9
26.4
66.7
13.1
30.7
42.7
+PACE
75.3
77.8
27.9
68.2
14.9
32.9
44.3
BOFT
72.3
77.1
27.0
66.8
12.8
31.1
42.9
+PACE
75.7
77.9
28.3
68.2
14.7
33,4
44.5
E Additional Plots
Figures 8 and show the gradient issues in FPA and the gradient regularization effects of PACE.
Baseline
0.001
0.005
0.01
0.05
0.1
0.5
3e4
3e4
|
2c4
2e4
1c4
1e4
epoch=
100
200
300
epoch=
100
200
300
(a) FPA
(b) PACE
Figure 8: Gradient norms of (a) FPA and (b) PACE with different regularization strengths À during training on CIFAR-100 (VTAB-1K) w/ ViT-B/16. Figure illustrates the average gradient norm over training epochs.
F Hyperparameter settings
For each dataset, we follow strategies from previous works [41, 28, 45) to apply grid search on the rank, learning rate and weight decay to establish strong baselines. Table 16, 17, 18 and 19 present the hyperparameters and number of trainable parameters used our strong baseline for VTAB-1K, few-shot learning, FGVC and domain adaptation tasks.
With these strong baselines, we apply grid search on À (0.02,0.05, 0.1, 1) and (0.1, 0.5, 1, 1.5, 2} for PACE to optimize its performance.
22

---

0f
1
Baseline
+FPA
+PACE
6e3
3e3
x= 1c-3 5c-3 0.01 0.05 0.1 0.5
10
50
100
500
1e3
5c3 1e4 5e4
Figure 9: Gradient norms of models across wide range of regularization strengths À on Camelyon (VTAB-1K) w/ Swin-B. Line and shadow represent mean and std over training epochs. While gradient explosion is less frequent for FPA in this setting, it exhibits unpredictable gradient norm with varied regularization strengths. In contrast, PACE reliably lowers gradient norms regularization strength increases, demonstrating its robustness for effective gradient control.
Table 16: Hyperparameters for bascline on VTAB-IK with ViT-B/16. A: LoRAmul VPT add- B:
LoRA Ir: learning rate. WD: weight decay. Specialized
Natural
Structured
Average parameter (M)
Hyperparaeeto
Cifar100
Catech111
Flowers102
SVHN
Sun397
Camelyon
EuroSAT
Resise45
Retinopathy
Clevr- Count
Clevr-Dist
DMLab
KITTI-Dist
dSpr-Loc
dSpr- Oni
INORB-Azin
NsORB-EE Ele
DTD
Pets
Method
A
A
4
4
4
A
B
B
B
A
4
A
B
Rank
10
14
12
18
18
14
10
8
10
2
2
8
18
10
10
22
4
1.81
Ir
lc-3
1c-3
10-3
10.3
le-3
le-2
le-3
5e-3
So-3
5o-3
5c-4
5e-4
1c-4
56-3
56-3
56-3
5o-3
1c-2
2c-4
WD
le44
le-4
le-3
le-2
le-3
l6-3
le-2
le-2
le-2
1e-2
le-4
le-3
le-4
1e-3
le-3 1e.4
le-2
le-2
le-2
Table 17: Ranks for baselines in Few-shot learning. Weight decay is fixed le-4.
learning rate
FGVCAueaaf
Food101
Flowers
OxfordPets
StanfordCars
Mcan
Baseline
36-3
3e-3
Se-3
20-3
2e-3
Parameter (M)
LoRAgdd
4
4
10
0.93
VPT
1
1
1
0.14
LoRA LoRAmu VPT
14
10
18
18
24
2.70
Table 18: Hyperparameters for the bascline LoRAmul VPTadd add in FGVC.
Hyperparameter
CUB-200-2011
NABirds
OxfordFlowers
StanfordDogs
StanfordCars
Mcan Parameter (M)
learning
5c-3
5c-4
5e-3
5c-3
2c-4
weight decuy
le-2
lc-3
lc-3
lc-2
lc-3
2.80
rank
14
18
18
24
14
Table 19: Hyperparameters for baseline LoR L domain adaptation.
VPT weight decay Parameter (M)
Baseline
rank
learning
LoRA +VPT
10
5c-4
lc-2
2.39
G Experiment details for GSM-8K
We conduct experiments text generation tasks by fine-tuning Phi-3-mini-4k-intreee [1] the GSM-SK [9] dataset using causal language modeling. We use learning rate of 2e-6, batch size of LoRA rank of 16, prompt "Answer below question. First think step-by-step and then answer the final mumber:\nin<<Qustioon
as instruction and fine-tune models on the training set and evaluated the performance on the test set.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We theoretically and empirically verify the claims and contributions made in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of work are discussed in $AA
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
Complete proofs for each theorem are provided in $B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Training details and hyperparameter selection are presented in Sec. and $F, respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
We will release our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
onnioicy))))) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.cooc wwtppppplcggcpiieeceeeeeoooceppe gwtc///ggee///eee CCodeSubmiieeeeece
Stt.l///ggiiieee/eeeuubbbiiiiiii iaatemmsssiooooooli (eionnolccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental settings and details are presented in Sec. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All reported results are averaged over three random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments were conducted on single NVIDIA H100 GPU with 96 GB memory, with each experiment completing within hours.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the wttpurip..cpppulll.i. i ines?
NeurlPS Code of Ethics
Answer: [Yes],
Justification: We have carefully reviewed and adhered to the code of ethics throughout research and writing process.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., . if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Potential impacts are discussed in $A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: Our work poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All publicly available assets (models, code, and data) used in this work have been properly credited, and their respective licenses and terms of have been explicitly mentioned and adhered to.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.tcod...c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets in the submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "loyO5M4vKG", "reviewer": "Reviewer_sUXG", "summary": "The paper proposes a consistency regularization that minimizes the squared L2 distance between two outputs of a model obtained using the same parameters, but multiplying the activations by 2 different noise samples. It proves that the population loss is bounded by the gradient norm, indicating that smaller gradients are better for generalization. It proposes the consistency regularization for aligning the fine-tuned model with the pretrained model for better generalization, and proves that the consistency regularization penalizes the first and second order gradients of the parameters of the model. This results in the training process preserving the knowledge learned by the pretrained model. It also proposes an efficient method for implementing the consistency regularization where the regularization is applied every N iterations.", "questions": "- Line 155-156 \"as even smaller weight changes can lead to significant divergence in the output space\": Can the authors provide references/experimental results to backup this statement?\n- In Figure 2 (a) and Figure 3 (b), how is the plot shown for the gradient norm of parameters in multiple layers?\n- See Point 4 in Weaknesses.", "limitations": "Authors have addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provieds a theoretical analysis and intuition for the consistency regularization and links generalization to small gradient norm.\n- The empiracal results show that using the consistency regularization along with existing finetuning methods improves their perfromance.\n- The consistency regularization can be made somewhat efficient by adding the noise to the features instead of parameters, and by using the consistency regularization in intervals rather than at every iteration.", "weaknesses": "**Gap between theoretical analysis and practical implementation**\n- The theoretical analysis is done for functions that map from R^d R. However, the in practice, the outputs of the heads of the classification models are multi-dimensional. It's not clear from the proofs that the theoretical results hold for multi-dimensional outputs. While the experimental results do suggest this, a theoretical proof for this would strengthen the paper.\n2. **Increased compute and memory requirements**\n- Since the consistency regularization needs to run the model twice (with shared parameters but with two different noise samples), and also backpropagate through both, the compute and the memory required is doubled.\n- The authors use H100 GPUs (96GB) for their experiments, hence the experiments are feasible. However, with lower end GPUs, this might not be feasible. Moreover, the requirements for large models (in the scale of billion parameters) might be prohibitive.\n- With this increased compute and memory requirements, the method's use as a PEFT method is limited.\n- While the paper proposes a method for efficient implementation by applying the regularization every N iterations, it is only tested on one dataset from VTAB-1k (CIFAR100). This should be tested on more datasets for the claim to hold.\n3. **Lack of evaluation beyond vision tasks**\n- The paper does not study natural language tasks (language understanding and generation). This would be necessary to show the generality of the method to other domains besides vision tasks.\n4. **Large training time**\n- The method is trained on VTAB for a large number of epochs (300). This means that the method can take a long time to converge. Are the comparisons with the other methods (other than the chosen baselines) done with a similar compute budget?\n- The method is trained for 100 epochs for the FGVC tasks. Why does the method require fewer epochs for FGVC tasks compared to VTAB?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## We thank the reviewer for insightful questions that help refine our work further.\n\n\n# 1. Theoretical analysis done for functions from $R^d\\rightarrow R$. \nThank you. In practice, we use the squared L2 distance for multi-dimensional outputs for $D^{fp}$ and $D^{pace}$, which allows our one-dimensional analysis to naturally generalize to multiple dimensions. For example, for a vector-valued function in the naive alignment, $f({\\bf \\theta}) = [f_1({\\bf \\theta}), ..., f_m({\\bf \\theta})]$, where $m$ is the output dimension, we have:\n\\\n\\\n$\n||f({\\bf \\theta}_0) - f({\\bf \\theta}_0 + \\Delta {\\bf \\theta})||\\_2^2 = \\sum\\_{i=1}^m [f_i({\\bf \\theta}_0) - f_i({\\bf \\theta}_0 + \\Delta {\\bf \\theta})]^2.\n$\n\\\n\\\nThis equality shows that the squared L2 distance in multiple dimensions is simply the sum of non-negative squared differences in each dimension. Consequently, this additive nature enables our one-dimensional analysis to extend seamlessly to multiple dimensions in practice, aligning with our empirical observations.\n\n# 2. Increased computation and memory requirements.\nThank you. Motivated by your questions, we have explored two variants to maintain similar computational and memory requirements as the baseline:\n\n* **1), PACE$\\_{fast}$:** We store model outputs for each sample from the previous epoch (${\\bf o}\\_{e-1}=f_{e-1}({\\bf x})$) in the CPU memory (or disk if preferred). During training, we feed these into the consistency regularization w.r.t. the current outputs ($||f\\_{e}({\\bf x})-{\\bf o}_{e-1}||\\_2^2$) as two versions used different i.i.d. noises (meeting our theoretical need).\n\n* **2), PACE$_{lazy}^{half}$:** At every $N^{th}$ iteration during training, we apply consistency regularization and halve the batch size.\n\nThe table below compares the maximum GPU memory usage, total training time, and accuracy for each task, demonstrating that **PACE$\\_{fast}$ and PACE$\\_{lazy}^{half}$ significantly improve over the baseline while maintaining similar computational demands as baselines**. Moreover, techniques such as gradient accumulation can be applied for smaller GPU memory if needed.\n\n|Method|CIFAR-100|VTAB|(ViT/16-B)|Camelyon|VTAB|(Swin-B)|ImageNet|DomAda|(ViT/16-B)|\n|:-|-:|:-:|:-|-:|:-:|:-|-:|:-:|:-|\n||Mem|Time|Acc|Mem|Time|Acc|Mem|Time|MeanAcc|\n|Baseline|8.9g|29m|74.6|15.7g|33m|86.7|8.9g|161m|44.8|\n|PACE|17.7g|53m|79.0| 29.4g|60m|89.3|17.7g|278m|46.3|\n|PACE$_{fast}$|**9.0g**|**29m**|**78.3**|15.7g|34m|88.8|**9.0g**|**162m**|**46.1**|\n|PACE$_{lazy}^{half}$ (N=2)|9.3g|29m|78.7|**15.7g**|**36m**|**89.2**|9.0g|165m|46.0|\n|PACE$_{lazy}^{half}$ (N=4)|9.3g|29m|78.4|15.7g|35m|88.9|9.0g|163m|45.6|\n|PACE$_{lazy}^{half}$ (N=6)|9.3g|29m|78.4|15.7g|35m|89.0|9.0g|163m|45.7|\n|PACE$_{lazy}^{half}$ (N=10)|9.3g|29m|78.2|15.7g|35m|88.9|9.0g|162m|45.6|\n\n# 3. Experiments on language tasks.\nFollowing VeRA (Kopiczko et al., ICLR 2024), we conducted GLUE benchmark experiments using RoBERTa-base. We report Matthew's correlation for COLA, Pearson correlation for STSB, and accuracy for other tasks. The table below compares PACE+LoRA with other methods, demonstrating PACE's effectiveness in language tasks.\n\n|Method|COLA|STSB|MRPC|RTE|QNLI|SST2|Avg.|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Full|63.6|91.2|90.2|78.7|92.8|94.8|85.2|\n|BitFit|62.0|90.8|**92.7**|81.5|91.8|93.7|85.4|\n|Adapt$^D$|62.6|90.3|88.4|75.9|93.0|94.7|84.2|\n|VeRA|**65.6**|90.7|89.5|78.7|91.8|94.6|85.2|\n|LoRA|63.4|91.5|89.7|86.6|93.3|95.1|86.6|\n|+PACE|65.0|**92.4**|92.0|**86.9**|**93.8**|**95.8**|**87.6**|\n\n# 4. Results of 100 epochs on VTAB.\nWe use 300 epochs for VTAB tasks as we observed slight extra improvements over 100 epochs. However, it does not mean PACE needs more training time to converge. **Since the optimizer uses cosine learning rate decay, reducing the number of training epochs to 100 has minimal impact on performance.** For FGVC tasks, we maintained 100 epochs as their larger datasets (1k-20k samples vs. VTAB's 1k) ensure convergence with fewer epochs.\n\nThe table below shows results for different training epochs on VTAB.\n\nTo ensure fair memory and computational budgets, **we tested PACE with half batch size and 50 epochs.** Under these conditions, **PACE still improves baseline accuracy by 2.10% and outperforms the previous SOTA GLoRA, which uses 500 epochs for training and 30 for parameter search.** This demonstrates PACE's efficiency and effectiveness across various training configurations.\n\n|#Epoch|Method|Natural|Specialized|Structured|Avg|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|530|GLoRA|83.61|87.02|63.27|77.97|\n|100|Baseline|81.94|85.40|61.40|76.24|\n|100|+PACE|83.94|87.44|64.62|78.67|\n|50|+PACE (half batch size)|83.77|87.32|63.92|78.34|\n|200|Baseline|82.28|85.30|61.64|76.40|\n|200|+PACE|84.13|**87.57**|64.85|78.85|\n|300|Baseline|82.41|85.00|61.80|76.40|\n|300|+PACE|**84.32**|87.55|**65.13**|**79.00**|\n\n\n# 5. Why smaller weight changes lead to significant divergence in the output space.\nSmaller weight changes can lead to significant divergence in output space, particularly when a network is sensitive to weight changes or has entered a sharp local minimum. In such cases, even minor weight adjustments can cause substantial output changes or even incorrect predictions. This behavior contrasts with robust networks or those in flatter minima. Consequently, constraining changes in weight space alone is suboptimal, as it disregards the loss landscape, which is also influenced by gradients. For further details on this phenomenon, kindly refer to the papers listed below.\n\n> 1. Wu et al, Adversarial Weight Perturbation Helps Robust Generalization, NeurIPS 2020.\n> 2. Foret et al, Sharpness-aware minimization for efficiently improving generalization, ICLR 2021.\n> 3. He et al, Defending and Harnessing the Bit-Flip based Adversarial Weight Attack, CVPR 2020.\n\n# 6. How is the plot shown for the gradient norm of parameters in multiple layers?\nWe plotted the sum of gradient norms from all layers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jpxksqJsEj", "reviewer": "Reviewer_neYP", "summary": "This paper proposes to regularize the model consistency by optimizing the fine-tuned model to remain consistent for the same sample under different perturbations.", "questions": "Please see the weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written.\n2. Several experiments are conducted on four visual adaptation tasks: VTAB-1k, FGVC, few-shot learning, and domain adaptation.", "weaknesses": "1. **Novelty Concerns**: Incorporating consistency regularization with PEFT approaches is already commonplace in multimodal parameter-efficient fine-tuning. For example, VioLET [1],  PromptSRC [2], and CoPrompt [3] propose to use additional encoders with shared weights to keep the consistency between the updated model and the original one. Considering that these papers focus on the consistency constraints on each modality itself, rather than cross-modal constraints. Therefore, the approach proposed in this paper to copy the original model parameters to enforce consistency lacks innovation and is simply a repetition of existing ideas.\n \n\n2. **Efficiency Concerns**: The proposed method, as referenced in Eq. 12, necessitates multiple forward passes during fine-tuning, whereas most compared methods require only a single pass. This discrepancy significantly impacts efficiency. The authors should provide detailed efficiency metrics alongside performance results to enable fair comparisons.\n\n3. **Experimental Settings**: Several experimental settings appear unconventional. The proposed method is based on a new baseline, LoRAmul+VPTadd. However, PACE does not seem to depend on specific designs of the base PET methods. Therefore, it would be more compelling to incorporate PACE into more general baseline methods such as AdapterFormer or existing SOTA methods like GLoRA. Demonstrating PACE's ability to enhance various PET methods would strengthen its contribution. Additionally, PACE requires 300 epochs for fine-tuning on the VTAB-1K dataset, whereas other methods typically require only 100 epochs. The necessity of such prolonged fine-tuning should be justified. Furthermore, in some benchmarks (e.g., Tables 2 and 3), PACE shows only limited improvement, which undermines its robustness.\n\n4. **Typos and Notational Errors**:  For instance, L505 misses a square in the second row. Additionally, the equal sign should be replaced with an approximate equal sign.\n\n[1] Wang Y, Liu Y, Zhang X, et al. VioLET: Vision-Language Efficient Tuning with Collaborative Multi-modal Gradients[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 4595-4605.\n\n[2] Khattak M U, Wasim S T, Naseer M, et al. Self-regulating prompts: Foundational model adaptation without forgetting[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 15190-15200.\n\n[3] Roy, Shuvendu, and Ali Etemad. \"Consistency-guided prompt learning for vision-language models.\" arXiv preprint arXiv:2306.01195 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## We thank the reviewer for helpful comments.\n\n# 1. Copy the original model for consistency regularization not novel.\n\nWe believe there is misunderstanding.\n\\\n\\\nExisting \"consistency\" models (we will cite/compare these interesting works in paper) align features of fine-tuned model with the pretrained model. They provide no guarantees for generalization error. Our PACE guarantees control of generalization by interplay of noise modulator in network branches. **Our \"consistency\" is thus completely different.**\n\\\n\\\nWe introduce 3 key innovations:\n\n**1) Novel consistency mechanism:** Unlike previous methods that apply consistency regularization between finetuned and pretrained models, PACE applies consistency between differently perturbed versions of the same finetuned model while learning adapter parameters. This crucial difference addresses gradient limitations of traditional alignment approaches.\n\n**2) Gradient reduction:** We prove that **naive finetuned-pretrained alignment can not guarantee gradient reduction** and even cause gradient explosion (**Prop. 1, Figs 4, 7, 8**). **PACE overcomes this issue**, ensuring gradient reduction and better generalization.\n\n**3) Theoretical guarantees:** We provide rigorous analysis (Thm. 2 & 3) showing how **PACE implicitly achieves gradient regularization** and model alignment even though PACE does not explicitly do so, a key advancement over previous works. **Thm. 1 explains how controlling gradient helps generalization.**\n\n**4) Superior performance:** Our experiments show **PACE outperforms both FPA** (explicitly align finetuned and pretrained model in output space) **and DELTA** (Li et al, ICLR 2019) (explicitly align finetuned and pretrained model in feature space with supervised attention)  across various datasets (table below).\n\nMethod|CIFAR-100 (VTAB)|ImageNet |(Dom|Ada)||||\n:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:\n||Source|-Sketch|-V2|-A|-R|Avg\nLoRA$\\_{mul}$+VPT$\\_{add}$|74.9|78.3|30.6|68.5|14.1|32.5|44.8\n$\\ $+DELTA|76.4|78.4|30.8|68.7|14.6|33.7|45.2\n$\\ $+FPA|76.6|78.8|31.2|68.6|14.7|33.5|45.3\n$\\ $+PACE|**79.0**|**79.0**|**31.8**|**69.4**|**16.3**|**35.2**|**46.3**\n\nPACE is a significant step forward in consistency regularization, with a theoretically grounded/empirically superior approach.\n\n# 2. Efficiency concern. \n\nWe have explored two variants with similar computational/memory needs as baseline:\n\n**1) PACE$\\_{fast}$:** We store model outputs for each sample from the previous epoch (${\\bf o}\\_{e-1}=f_{e-1}({\\bf x})$) in CPU memory (or disk). During training, we feed these into the consistency regularization w.r.t. the current outputs ($||f\\_{e}({\\bf x})-{\\bf o}_{e-1}||\\_2^2$) as two versions used different i.i.d. noises (our theoretical need).\n\n**2), PACE$_{lazy}^{half}$:** At every $N^{th}$ iteration during training, we apply consistency regularization and halve the batch size.\n\nTable below compares the max GPU memory usage, total training time, and accuracy for each task, showing that PACE$\\_{fast}$ and PACE$\\_{lazy}^{half}$ significantly improve over the baseline under similar compute demands.\n\nMethod|CIFAR-100|VTAB|(ViT/16-B)|Camelyon|VTAB|(Swin-B)|ImageNet|DomAda|(ViT/16-B)|\n:-|-:|:-:|:-|-:|:-:|:-|-:|:-:|:-\n||Mem|Time|Acc|Mem|Time|Acc|Mem|Time|MeanAcc\nBaseline|8.9g|29m|74.6|15.7g|33m|86.7|8.9g|161m|44.8\nPACE|17.7g|53m|79.0|29.4g|60m|89.3|17.7g|278m|46.3\nPACE$_{fast}$|9.0g|29m|78.3|15.7g|34m|88.8|**9.0g**|**162m**|**46.1**\nPACE$_{lazy}^{half}$ (N=2)|**9.3g**|**29m**|**78.7**|**15.7g**|**36m**|**89.2**|9.0g|165m|46.0\nPACE$_{lazy}^{half}$ (N=4)|9.3g|29m|78.4|15.7g|35m|88.9|9.0g|163m|45.6\nPACE$_{lazy}^{half}$ (N=6)|9.3g|29m|78.4|15.7g|35m|89.0|9.0g|163m|45.7\nPACE$_{lazy}^{half}$ (N=10)|9.3g|29m|78.2|15.7g|35m|88.9|9.0g|162m|45.6\n\n# 3. PACE+AdaptFormer & GLoRA.\n\nMethod|CIFAR-100 (VTAB)|ImageNet |(Dom|Ada)||||\n:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|||Source|-Sketch|-V2|-A|-R|Avg\nAdaptFormer|70.6|77.4|26.5|67.4|12.4|28.7|42.4\n+PACE|**74.8**|**78.2**|**27.4**|**67.9**|**13.9**|**31.7**|**43.8**\nGLoRA|75.9|78.2|30.3|68.1|13.5|31.6|44.3\n+PACE|**78.6**|**78.8**|**31.7**|**69.0**|**15.9**|**34.4**|**45.9**\n\n# 4. PACE with 100 epochs on VTAB.\nWe use 300 epochs for VTAB tasks (small gain over 100 ep.) but PACE does not needs more time than others to converge. **As the optimizer uses cosine learning rate decay, reducing epochs to 100 has minimal impact.**\n\nTo ensure fair memory/compute budgets, **we tested PACE with 1/2 batch size, 50 epochs** where **PACE still improves baseline accuracy by 2.1% and outperforms the previous SOTA GLoRA (500 train epochs and 30 for parameter search).** Thus, PACE is efficient/effective.\n\n#Epoch|Method|Natural|Specialized|Structured|Avg\n:-:|:-:|:-:|:-:|:-:|:-:\n530|GLoRA|83.61|87.02|63.27|77.97\n100|Baseline|81.94|85.40|61.40|76.24\n100|+PACE|83.94|87.44|64.62|78.67\n50|+PACE (1/2 batch size)|83.77|87.32|63.92|78.34\n200|Baseline|82.28|85.30|61.64|76.40\n200|+PACE|84.13|**87.57**|64.85|78.85\n300|Baseline|82.41|85.00|61.80|76.40\n300|+PACE|**84.32**|87.55|**65.13**|**79.00**\n\n# 5. Tables 2/3: results not that good. \n\n**1) In Table 2, PACE shows gains on all shot settings, especially in low-data scenarios** crucial for fine-tuning large models on downstream tasks with limited data, and consistently with our motivation to improve the generalization error (Thm. 1-3).\n\nTable below shows average results across datasets and shots. **PACE consistently outperforms baseline by 0.9 to 2.9.**\n\nMethod|AvgAcc|Improvement\n:-|-:|:-\nLoRA|61.1\n$\\ $+PACE|63.0|+2.9\nVPT|61.0\n$\\ $+PACE|61.9|+0.9\nLoRA$\\_{mul}$+VPT$\\_{add}$|62.7\n$\\ $+PACE|64.2|+1.5\n\n**2) In Table 3, we observe a 0.7 gain (expected on large data size).** When reducing the data size, we see more gains.\n\nBelow is averaged accuracy over different data percentages on FGVC tasks (average gain: 1.8 points).\n|Method|100% data|50% data|20% data|10% data|Avg|\n:-|:-:|:-:|:-:|:-:|:-:\nLoRA$\\_{mul}$+VPT$\\_{add}$|90.8|87.1|79.8|71.8|82.3\n$\\ $+PACE|91.5|88.5|81.8|74.8|84.1\n\n# 6. Typos.\nDuly noted. We will fix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2INtVVc7LD", "reviewer": "Reviewer_bhst", "summary": "This paper introduces PACE, an extension to PEFT methods for ViTs that includes consistency regularization. The paper shows that consistency regularization encourages smaller gradient norms and better alignment between pre-trained and fine-tuned models, resulting in better fine-tuning performance than existing PEFT techniques.", "questions": "L220-221: Which pre-training method was used for these backbones? Was it supervised or self-supervised?\n\nI don’t see PACE_h in table 6.\n\nIt would be great to see this extended to language applications!", "limitations": "The authors adequately discuss this section.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**(S1)**: The technique introduced is well-presented and easy to implement, borrowing ideas from parameter-efficient fine-tuning (PEFT) and consistency regularization (CR) to present a combined technique that is empirically effective.\n\n**(S2)**: The paper rigorously grounds their approach via theoretical analysis, demonstrating how reduced gradient norms lead to better generalization. This is well presented and the analysis itself will be relevant to future work and related research in this field, as other techniques that emulate the same properties (eg: reduced gradient norms) can benefit from PACE.\n\n**(S3)**: The ablation study is well conducted and systematically demonstrates the usefulness of the authors’ design. Throughout the experimental section, the authors also include experiments that probe properties of the model relevant to their method (eg: Fig 2, 3, 4), demonstrating that their theoretical hypotheses are well justified in empirical results.\n\nOverall, this paper is well-presented, technically sound, and is a pleasure to read. I think it will have a good impact on future research in this area.", "weaknesses": "**(W1)**: The experimental baseline is not really clarified. Only a ViT-B/16 pre-trained on ImageNet is used. Was the pre-training supervised or unsupervised? Do the analysis and results differ for supervised vs self-supervised backbones? Most common and performant ViTs nowadays (eg: MAE [1], Dino [2]) are self-supervised, so it would be good to clarify if there are differences in the analysis that depend on the method of pre-training. \n\n**(W2)**: It would be interesting to see this technique extended to larger domain shifts (eg: WILDS [3]). Also, it is not clear how the amount of fine-tuning data affects the performance (beyond few-shot learning). I.e. can PACE make better use of large datasets from new domains more effectively than prior methods.\n\n**(W3)**: Minor quibble. It would be useful to include comparisons with orthogonal fine-tuning techniques (eg: BOFT [4], OFT [5]).\n\n---\nReferences:  \n\n[1] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[2] Oquab, Maxime, et al. \"Dinov2: Learning robust visual features without supervision.\" arXiv preprint arXiv:2304.07193 (2023).\n\n[3] Koh, Pang Wei, et al. \"Wilds: A benchmark of in-the-wild distribution shifts.\" International conference on machine learning. PMLR, 2021.\n\n[4] Liu, Weiyang, et al. \"Parameter-efficient orthogonal finetuning via butterfly factorization.\" arXiv preprint arXiv:2311.06243 (2023).\n\n[5] Qiu, Zeju, et al. \"Controlling text-to-image diffusion by orthogonal finetuning.\" Advances in Neural Information Processing Systems 36 (2023): 79320-79362.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## We thank the reviewer for insightful questions that help refine our work further.\n\n\n# 1. Experiment settings.\nViT/16-B and Swin-B were pre-trained on ImageNet-21K using supervised learning. Our analysis extends to self-supervised pre-trained models as well, since our goal is to retain knowledge from large pre-training datasets, regardless of the specific supervision/pre-training method used.\n\nTo demonstrate the effectiveness of our approach across different pre-training methods, we conducted experiments on three VTAB datasets (CIFAR-100, Camelyon and Clevr-Count) using ViT/16-B models pre-trained by Masked Autoencoder (MAE) and DINO, both self-supervised methods trained on ImageNet-1K. The results are shown in the table below.\n\n|Method|MAE|||Dino|||\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|\n||CIFAR-100|Camelyon|Clevr-Count|CIFAR-100|Camelyon|Clevr-Count|\n|Full|10.3|74.6|52.5|37.2|73.1|34.5|\n|Linear|18.7|79.9|57.1|49.1|82.5|44.2|\n|$LoRA_{mul}+VPT_{add}$|25.1|82.7|82.1|58.1|85.4|55.7\n|$\\quad$+PACE|**44.8**|**85.8**|**86.4**|**60.8**|**88.1**|**61.0**|\n\n# 2. Large domain shift.\nIndeed, our experiments on the VTAB benchmark already demonstrate PACE's effectiveness in handling large domain shifts. VTAB consists of 19 diverse tasks drawn from various domains, including several that exhibit significant domain shifts from the ImageNet pre-training data, especially the specialized and structured datasets.\n\nSpecifically, **we have tested PACE on Camelyon (which is also part of WILDS) and Diabetic Retinopathy**. These **medical imaging tasks represent a substantial domain shift** from natural images in ImageNet.\n\nOur results show consistent improvements over the baseline across these diverse task with large domain shifts. This performance demonstrates PACE's ability to effectively adapt to new domains while leveraging pre-trained knowledge.\n\n# 3. Vary number of training samples on FGVC.\n\nAs per request, we varied the data percent (10% to 100%) of training samples on FGVC. Table below shows that PACE gain more improvement when data size is small, which is motivated by our generalization error analysis of PACE (Theorem 1-3).\n\n|Method|CUB||||NAB||||Flowers||||StanDogs||||StanCars||||\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|-:|:-:|:-:|:-:|-:|:-:|:-:|:-:|\n||100%|50%|20%|10%|100%|50%|20%|10%|100%|50%|20%|10%|100%|50%|20%|10%|100%|50%|20%|10%|\n|$LoRA_{mul}+VPT_{add}$|88.9|87.1|83.9|79.1|87.5|80.7|75.0|70.2|99.4|98.5|96.5|93.1|91.2|90.6|88.7|86.9|87.5|78.7|54.9|30.1|\n|$\\ $+PACE|**89.8**|**88.4**|**85.5**|**81.4**|**88.8**|**82.9**|**77.5**|**73.8**|**99.5**|**99.2**|**97.9**|**96.1**|**92.2**|**91.8**|**90.9**|**89.8**|**88.8**|**80.5**|**57.3**|**33.2**|\n\n# 4. Experiments on large domain shift and large dataset.\n\nIn line with current trends, our focus is on finetuning pre-trained models for downstream tasks, where small/medium size datasets are common. In these scenarios, PACE shows significant improvements. For larger datasets, our improvements are smaller, **which aligns with Lemma 1 and Theorem 1** - large datasets already allow models to achieve good generalization, diminishing the need for additional generalization improvement techniques.\n\nHowever, the theoretical findings from PACE, particularly the implicit gradient reduction and alignment, could motivate future work in areas dealing with large domain shifts in big datasets. While we have not extensively tested PACE in such scenarios, **these insights could inspire new approaches for handling significant domain shifts, even when data is abundant.**\n\n# 5. Combine with OFT and BOFT.\n\nWe re-implemented OFT and BOFT following the code and experimental settings provided by the authors. For OFT, we found that the constrained version (COFT) yields better results than the unconstrained version.\n\nThe table below compares results with/without PACE on CIFAR-100 (VTAB) and ImageNet (Domain Adaptation), demonstrating that incorporating PACE leads to improved performance.\n|Method|CIFAR-100 (VTAB)|ImageNet|(Dom|Ada)||||\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|||Source|-Sketch|-V2|-A|-R|Avg|\n|COFT|71.8|76.9|26.4|66.7|13.1|30.7|42.7|\n|$\\ $+PACE|**75.3**|**77.8**|**27.9**|**68.2**|**14.9**|**32.9**|**44.3**|\n|BOFT|72.3|77.1|27.0|66.8|12.8|31.1|42.9|\n|$\\ $+PACE|**75.7**|**77.9**|**28.3**|**68.2**|**14.7**|**33.4**|**44.5**|\n\n# 6. PACE$_h$.\nPACE$_\\text{merge}$ refers to PACE$_h$, where perturbation is applied after merging the adapter feature $\\Delta h(\\cdot)$ with the pretrained layer feature $h_0(\\cdot)$, namely, PACE$_h$ perturbs $h(\\cdot)=\\Delta h(\\cdot)+h_0(\\cdot)$. This typo has been corrected.\n\n# 7. Experiments on language tasks.\nFollowing VeRA (Kopiczko et al., ICLR 2024), we conducted GLUE benchmark experiments using RoBERTa-base. We report Matthew's correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks. The table below compares PACE+LoRA with other methods, demonstrating PACE's effectiveness in language tasks.\n\n|Method|COLA|STSB|MRPC|RTE|QNLI|SST2|Avg.|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Full|63.6|91.2|90.2|78.7|92.8|94.8|85.2|\n|BitFit|62.0|90.8|**92.7**|81.5|91.8|93.7|85.4|\n|Adapt$^D$|62.6|90.3|88.4|75.9|93.0|94.7|84.2|\n|VeRA|**65.6**|90.7|89.5|78.7|91.8|94.6|85.2|\n|LoRA|63.4|91.5|89.7|86.6|93.3|95.1|86.6|\n|+PACE|65.0|**92.3**|92.0|**86.9**|**93.6**|**95.8**|**87.6**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NmO5j6AiLs", "reviewer": "Reviewer_JSoa", "summary": "The paper proposes a consistency regularization for improving the generalization performance of PEFT methods. Specifically, the regularization constructs two predictions perturbed by different noises and penalizes the L2 distance between them. The paper theoretically studies the benefits and shows that the regularization reduces gradient norms. Empirically, the paper benchmarks the method on multiple tasks and datasets and achieves state-of-the-art performance.", "questions": "* Could the authors provide discussion and comparisons to other robust regularization techniques?  \n\n* Could the authors comment on how $D^{PACE}$ ensures alignment.", "limitations": "The paper does not have a potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Theoretical results are clearly presented and easy to follow. \n*  The paper presents an extensive evaluation of multiple datasets and tasks.", "weaknesses": "* **Missing related works and comparisons.** The major weakness of the paper is the lack of discussion and comparisons with related works. For example, L2-SP [1] directly penalizes deviation in the weight space and ensures alignment. DELTA [2] penalizes feature differences similar to this paper. FTP [3] shows that feature deviation can be reduced to weight deviation in linear layers. While the experiment settings are diverse, the paper only compares to vanilla PEFT methods. The method should be compared to other regularization methods for a fair evaluation. \n\n* **Increased computation.** To construct the two predictions. The method requires two forward passes through the model. This can significantly increase the computation and memory requirement of the algorithm. \n\n* **How does $D^{PACE}$ ensure alignment?**  It is clear that $D^{fp}$ (eq.5) encourages alignment. However, the proposed PACE regularization does not seem to encourage alignment explicitly.\n\n[1] Xuhong, L. I., Yves Grandvalet, and Franck Davoine. \"Explicit inductive bias for transfer learning with convolutional networks.\" International Conference on Machine Learning. PMLR, 2018.\n\n[2] Li, Xingjian, et al. \"Delta: Deep learning transfer using feature map with attention for convolutional networks.\" arXiv preprint arXiv:1901.09229 (2019).\n\n[3] Tian, Junjiao, et al. \"Fast trainable projection for robust fine-tuning.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## We thank the reviewer for insightful questions that help refine our work further.\n\n\n# 1. Comparison with L2SP, DELTA & FTP.\nL2SP, DELTA, and FTP aim to retain pre-trained knowledge by aligning finetuned models with pre-trained ones, reducing distance in weight space, feature space and using projected gradient descent, respectively. They provide **no guarantees for generalization error nor study it.** But **we will of course cite/compare with these interesting works in paper.**\n\n**Our PACE is very different. We leverage the fact that small gradient norms contribute to flat loss landscapes, which enhances generalization** (as per our Th. 2 & 3). PACE introduces a novel consistency regularization on output space over different adapter perturbations, implicitly reducing gradient norms (which improves generalization as per Th. 1) and aligning models.\n\nThus, **Our alignment is very different** than the above methods.\n\nPACE offers a key advantage over other alignment methods: \n* guaranteed gradient reduction, which is crucial for model convergence, \n* robustness and generalization (Th. 1-2).\n\nTo validate PACE's effectiveness, we compared it with L2SP, DELTA, and FTP on CIFAR-100 (VTAB) and ImageNet (Domain Adaptation) datasets. \n\nThe results, presented in the table below, clearly demonstrate PACE's superior performance.\n\nMethod|CIFAR-100 (VTAB)|ImageNet |(Domain |Adapt)||||\n:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:\n||Source|-Sketch|-V2|-A|-R|Avg\n$LoRA_{mul}+VPT_{add}$|74.9|78.3|30.6|68.5|14.1|32.5|44.8\n$\\ $+L2SP|75.9|78.5|30.4|68.7|14.9|33.5|45.2\n$\\ $+DELTA|76.4|78.4|30.8|68.7|14.6|33.7|45.2\n$\\ $+FTP|76.2|78.6|30.8|68.6|15.8|33.6|45.4\n$\\ $+PACE|**79.0**|**79.0**|**31.8**|**69.4**|**16.3**|**35.2**|**46.3**\n\n# 2. Increased computation. \nTo achieve computation and memory efficiency on par with compared baselines, we have explored two variants to maintain similar computational and memory requirements:\n\n* **1), PACE$\\_{fast}$:** We store model outputs for each sample from the previous epoch (${\\bf o}\\_{e-1}=f_{e-1}({\\bf x})$) in the CPU memory (or disk if preferred). During training, we feed these into the consistency regularization w.r.t. the current outputs ($||f\\_{e}({\\bf x})-{\\bf o}_{e-1}||\\_2^2$) as two versions used different i.i.d. noises (meeting our theoretical need).\n\n* **2), PACE$_{lazy}^{half}$:** At every $N^{th}$ iteration during training, we apply consistency regularization and halve the batch size.\n\nThe table below compares the maximum GPU memory usage, total training time, and accuracy for each task, demonstrating that **PACE$\\_{fast}$ and PACE$\\_{lazy}^{half}$ significantly improve over the baseline while maintaining similar computational demands as baselines**. \n\n|Method|CIFAR-100|VTAB|(ViT/16-B)|Camelyon|VTAB|(Swin-B)|ImageNet|DomAda|(ViT/16-B)|\n|:-|-:|:-:|:-|-:|:-:|:-|-:|:-:|:-|\n||Mem|Time|Acc|Mem|Time|Acc|Mem|Time|MeanAcc|\n|Baseline|8.9g|29m|74.6|15.7g|33m|86.7|8.9g|161m|44.8|\n|PACE|17.7g|53m|79.0| 29.4g|60m|89.3|17.7g|278m|46.3|\n|PACE$_{fast}$|**9.0g**|**29m**|**78.3**|15.7g|34m|88.8|**9.0g**|**162m**|**46.1**|\n|PACE$_{lazy}^{half}$ (N=2)|9.3g|29m|78.7|**15.7g**|**36m**|**89.2**|9.0g|165m|46.0|\n|PACE$_{lazy}^{half}$ (N=4)|9.3g|29m|78.4|15.7g|35m|88.9|9.0g|163m|45.6|\n|PACE$_{lazy}^{half}$ (N=6)|9.3g|29m|78.4|15.7g|35m|89.0|9.0g|163m|45.7|\n|PACE$_{lazy}^{half}$ (N=10)|9.3g|29m|78.2|15.7g|35m|88.9|9.0g|162m|45.6|\n\n\n# 3. How $D^{pace}$ align models.\nRecall that $D^{pace}$ aims to reduce $||f({\\bf \\theta}_0+{\\bf z}_1\\odot \\Delta{\\bf \\theta}) - f({\\bf \\theta}_0+{\\bf z}_2\\odot \\Delta{\\bf \\theta})||_2^2$ where ${\\bf \\theta}_0, \\Delta{\\bf \\theta}$ are pretrained and adapter weights and $f$ is the network, and ensure it is small for all noise ${\\bf z}_1, {\\bf z}_2\\sim\\mathcal{N}({\\bf 1}, \\sigma^2)$.\n\n* Intuitively, imagine we drew  $({\\bf z}_1={\\bf 0}, {\\bf z}_2={\\bf 1})$ or $({\\bf z}_1={\\bf 1}, {\\bf z}_2={\\bf 0})$. Then it becomes clear \nwe indeed reduce the distance between finetuned and pretrained model $D^{fp}=||f({\\bf \\theta}_0+\\Delta{\\bf \\theta})-f({\\bf \\theta}_0)||_2^2$.\n\n\\\nTo understand general case with any $({\\bf z}_1,{\\bf z}_2)$, follow these steps:\n\n* Theoretically, with Prop. 1 and Theorem 3, the distance between finetuned and pretrained model can be approximated and upper bounded:\n$$\nD^{fp}=[f({\\bf \\theta}_0+\\Delta{\\bf \\theta}) - f({\\bf \\theta}_0)]^2\\approx[\\Delta{\\theta}^T{\\bf \\nabla}-\\frac{1}{2}\\Delta{\\theta}^T{\\bf H}\\Delta{\\bf \\theta}]^2 \\leq 2d{\\color{red}{||\\Delta{\\bf \\theta}\\odot{\\bf \\nabla}||_2^2}}+d^2{\\color{red}{||(\\Delta{\\bf \\theta}\\Delta{\\bf \\theta}^T)\\odot{\\bf H}||_F^2}}\n$$\nwhere symbols ${\\bf \\nabla, H}, d$ are the gradient, hessian matrix and dimension of weights.\n\n* Through Theorem 2, $D^{pace}$ can be approximated as:\n$$\nD^{pace}\\approx2\\sigma^2{\\color{red}||\\Delta{\\bf \\theta}\\odot{\\bf \\nabla}||_2^2}+\\sigma^4{\\color{red}||(\\Delta{\\bf \\theta}\\Delta{\\bf \\theta}^T)\\odot{\\bf H}||_F^2}\n$$\nwhere $\\sigma^2$ is the noise variance.\n\n* Since $\\sigma$ and $d$ are constants during training, **minimizing** $D^{pace}$ **leads to small** ${\\color{red}||\\Delta{\\bf \\theta}\\odot{\\bf \\nabla}||_2^2}$ and ${\\color{red}||(\\Delta{\\bf \\theta}\\Delta{\\bf \\theta}^T)\\odot{\\bf H}||_F^2}$. **This results in a small upper bound for** $D^{fp}$, **effectively reducing** $D^{fp}$ and **ensuring alignment of the finetuned model with the pretrained model.**\n  \\\n  \\\n  **Thus, PACE effectively aligns the finetuned model with the pretrained model even though PACE does not explicitly do that**. \n  \\\n  \\\n  Figures 2(b) & 3(b) illustrate that distance from finetuned model to pretrained model become small after applying PACE, **verifying that PACE ensures alignment.**\n\n\n# **Importantly**:\n* Beside alignment, $D^{pace}$ ensures gradient reduction, as proven in Theorem 2 & shown empirically in Fig. 2(a) and 3(a). As per our motivation, **this reduction is crucial for improving generalization, as established in Theorem 1.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# We thank all the reviewers for constructive feedback and questions shaping our revised paper.
\
\
We have addressed all comments in individual responses to each reviewer.
\
\
\
Below we just provide few highlights:
* 1. Increased computation and memory requirements: we have provided now **PACE$\_{fast}$ and PACE$\_{lazy}^{half}$** variants which **enjoy almost identical memory and compute time footprint as baselines** while still bringing **2-3% gains.**
* 2.  Experiments on language tasks. Following VeRA (Kopiczko et al., ICLR 2024), **we conducted GLUE benchmark** experiments using RoBERTa-base, and showed gains.
* 3. We provided results on 50, 100 epochs on VTAB.
* 4. Consistency novelty. We have explained that **our "consistency" model is very different to existing works.** 
  * We do not explicitly align per se with a frozen pretrained model.
  * We use noise modulators to align two branches. These noise modulators implicitly **regularize gradient (Theorems 2 & 3)** and **improve generalization error (Theorem 1)**. Standard works on "consistency" with frozen pretrained model for alignment do not enjoy these properties. 
* 5. We provided results on PACE+AdaptFormer & GLoRA.
* 6. We have provided experiments on finetuning MAE and DINO (self-supervised models).
* 7. We provided experiments on OFT+PACE and BOFT+PACE.
* 8. We provided comparisons with L2SP, DELTA and FTP. We will of course cite these works and discuss accordingly.
* 9. We detailed how $D^{pace}$ align models while improving generalization error.

\
\
**We truly hope this rebuttal, theoretical arguments and empirical arguments will convince reviewers about novelty of our work:**
* Current alignment methods do not study or lower generalisation error (Theorems 1-3)
* Our "consistency" is not what other works do. We use noise modulation to achieve regularized gradients and hence improved generalization error.
* Our experiments as predicted by theory consistently outperform other 'consistency" approaches.
* Our compute and memory cost remains in line with baselines thanks to **PACE$\_{fast}$ and PACE$\_{lazy}^{half}$** while enjoying 2-3% gains.

\
Kind regards,
\
Authors

Meta-review: This paper presents a consistency-regularization method for the knowledge/generalizability retention of pretrained (large) backbones in parameter-efficient fine-tuning (PEFT). It was reviewed by several knowledgable reviewers. After the rebuttal, the main consensus from all the reviewers was that the proposed method provides a new consistency regularization analysis with effective practical tricks in implementation.  This meta-review concurs and recommends **acceptance**.

The authors are encouraged to carefully revise the paper due to **the significant updates provided in the rebuttal and discussions**. In addition to the experimental results, the authors should also add more in-depth analyses for the two new implementation tricks provided in the rebuttal to address the reviewers' concerns remained after the reviewer-author discussion.
Decision: Accept (spotlight)