Fast Trainable Projection for Robust Fine-tuning
Authors: Junjiao Tian
Yen-Cheng Liu
James Smith
Zsolt Kira
Venue: NeurIPS 2023 poster (2023)
Keywords: fine-tuning
transfer learning
regularization
TL;DR: We propose a computationally efficient algorithm for fine-tuning pre-trained models, which uses projection to enforce a hard constraint.

Abstract: Robust fine-tuning aims to achieve competitive in-distribution (ID) performance while maintaining the out-of-distribution (OOD) robustness of a pre-trained model when transferring it to a downstream task. Recently, projected gradient descent has been successfully used in robust fine-tuning by constraining the deviation from the initialization of the fine-tuned model explicitly through projection. However, algorithmically, two limitations prevent this method from being adopted more widely, scalability and efficiency.  In this paper, we propose a new projection-based fine-tuning algorithm, Fast Trainable Projection (FTP) for computationally efficient learning of per-layer projection constraints, resulting in an average 35% speedup on our benchmarks compared to prior works. FTP can be combined with existing optimizers such as AdamW, and be used in a plug-and-play fashion. Finally, we show that FTP is a special instance of hyper-optimizers that tune the hyper-parameters of optimizers in a learnable manner through nested differentiation. Empirically, we show superior robustness on OOD datasets, including domain shifts and natural corruptions, across four different vision tasks with five different pre-trained models. Additionally, we demonstrate that FTP is broadly applicable and beneficial to other learning scenarios such as low-label and continual learning settings thanks to its easy adaptability. The code will be available at https://github.com/GT-RIPL/FTP.git.

Fast Trainable Projection for Robust Fine-Tuning
Junjiao Tian Georgia Institute of Technology j stian73dgatech. edu
Yen-Cheng Liu Georgia Institute Technology ycliu@gatech. edu
James Seale Smith Georgia Institute of Technology Jamessaalesa th@gatech. edu
Zsolt Kira
Georgia Institute Technology zkira@gatech.c
Abstract
Robust fine-tuning aims to achieve competitive in-distribution (ID) performance while maintaining the out-of-distribution (OOD) robustness of pre-trained model when transferring it to a downstream task. Recently, projected gradient descent has been successfully used in robust fine-tuning by constraining the deviation from the initialization of the fine-tuned model explicitly through projection. However, algorithmically, two limitations prevent this method from being adopted more widely, scalability and efficiency. In this paper, we propose new projection-bass based tine-tuning algorithm, Fast Trainable Projection (FTP) for computationally efficient learning of per-layer projection constraints, resulting in an average 35% speedup on our benchmarks compared to prior works. FTP can be combined with existing optimizers such as AdamW, and be used in plug-and-play fashion. Finally, we show that FTP special instance hyper-optimizers that tune the hyper-parameters. of optimizers in learnable manner through nested diffcrentiation. Empirically, we show superior robustness on OOD datasets, including domain shifts and natural corruptions, across four difffrrent vision tasks with five different pre-trained models. Additionally, we demonstrate that FTP is broadly applicable and beneficial other learning scenarios such as low-label and continual learning settings thanks to its easy adaptability. The code will be available /tttps://giii com/ com/com CT-RIPL/FTP. git
Introduction
With new progress being made in pre-training of foundation models every year, such as self- language-supervised training 4 their potential has gone far beyond merely speeding up convergence [5). They have demonstrated superior transferability to other tasks, reducing the need for data and improving robustness and generalization capabilities 6 g The problem of how to fine-tume tune (transfer) a foundation model such that we maintain its robustness and generalization capabilities acquired during pre- pre-training on large datasets has therefore become an essential research topic. This problem is hard because the conventional machine learning paradigm of validating on held-out training data does not impose any constraints on robustness and generalization w.r.t. the foundation models. For example, fine-tuning with a slightly large learning rate can easily destroy capabilitics that reside in the foundation models 8), while performing well on the target task.
supervised M
To maintain the robustness and generalization capability of the pre-trained model when fine-tuning, recent projection- methods explicitly constrain the distance between the fine-tuned and the pre-traine rre-traneed models through projection. For example, MARS-SP [9] specifies distance constraint shared by all layers in a neural network. However, it practically intractable tune a constraint for each layer (poor scalabiliry). TPGM (10] proposes to automatically learn different constraints
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Time (itss
3. Projection
W
1.75
W
1.0
Real
Brightndis
(00
1.0
Yt
1. UGD
ProjUpdate W
Infograph-
Sketch
Go Gouss
Fog
TPGM
TPGM
FTP
FTP
(a) One Step FTP Diagram
(b) Image Classification
(c) Semantic Segmentation
Figure I: (a): FTP updates the model using (unconstrained) gradient descent (UGD) to calculate W then updates the projection constraint "/e (ProjUpdate), and finally projects W, to W, (Projection), all in a single forward pass. (b),(c): Visualizations of in-distribution (Real/Clean, labeled as ID), out-of-distribution (Sketch/Fog. etc.) accuracy and computation time (iterations/sec) as a percentage of vanilla fine-tuning (FT) for classification on DomainNet (Tab. 2) and semantic segmentation on PASCAL-Context (Tab. 4 respectively. FTP improves the OOD robustmess of FT and is much more computationally efficient than prior work TPGM.
each layer, solving the issue scalability in MARS-SR however, with increased computational overhead (poor efficiency)). These limitations prevent the method from being adopted more widely,
To achieve scalability and efficiency simultaneously, we propose Fast Trainable Projection (FTP). for learning both the projection constraints and the main model in single forward pass (Fig. Tap. significantly reducing computation overhead in prior works while achieving competitive performance. Specifically, FTP removes the algorithmic redundancy of extra training procedures required in TPGM [10], which requires sampling separate batch of data and running nested training loop. FTP achieves this by 1) utilizing different batches of training data sampled at consecutive steps and 2) re-using gradients calculated for the main model update (Sec. 3.2). This leads to a 35% speedup with comparable performance in fine-tuning (Fig. Tb Icp. The efficiency improvement and easy adaptability as drop-in replacement with existing optimizers are essential to making projection-based methods applicable to more fine-tuning problems. For example, implement SGDP, an SGD variant with built-in FTP. SGDP can be used as a drop-in replacement for SGD (details in Appendix 8.7 as:
optimiser SGDP (param_group wtt.:rrrouuppp.pt........ **optinizer _arrams)_paamm #See Appendix 8.7.
To demonstrate this, we test FTP on four different vision tasks, image classification, semantic segmentation, human parts segmentation, and surface normal estimation. FTP shows superior OOD performance under domain shift or natural corruptions on benchmarks. Moreover, we apply FTP to continual learning (CL) benchmark and achieve state of the art performance when combined with a simple CL technique.
Finally, we show that FTP is special instance of hyper-optimizers [T] 13 that aims to reduce the manual tuning of optimization syper-parameters such as learning rate by learning them automatically through automatic differentiation and nested optimization. Theoretically. understand why FTP and other projection methods can maintain the robustness of the pre-trained models, we propose to establish a theoretical connection between robustness and projection through
16
18
the lens of Liptschitz continuity, a widely adopted measure of robustness [99 20 21 our contributions are:
In summary,
We present a new fine-tuning algorithm, Fast Trainable Projection, to efficiently learn the projection constraints and fine-tune the model simultaneously, bringing significantly improved computation efficiency w.r.t. prior works [10] in Sec. 3.2
We show that FTP is a special instance of hyper-optimizers that aims to reduce manual tuning of haper-parameters through nested optimization in Sec. 3.3
We discuss dual perspective of the fine-tuning robustness in the feature space and the weight space of model to mathematically understand why projection can maintain the robustness of the pre-trained models in Sec. 3.4

---

We show superior robustness on OOD datasets on four vision tasks with five pre-trained models and SOTA performance on a continual learning benchmark, all with a 35% speedup in Sec.
2 Related Works
We summarize related works in (general) robust fine-tuning into three categories: when, where, and how much fine-tune, depending on their underlying strategy. Moreover, we diseuss recent advances in fine-tuning language-imas pre- tre-trained models, which have inspired specialized fine-tuning strategies. When to fine-tune: LP-FT 77 discovers that fine-tuning the entire network can distort features in the pre-trained models and proposes to first only fine-tune the last linear layer followed by training the entire network with a small learning rate. We will include LP-FT in our experiments. Where to fine-tune: Instead of fine-tuning the entire network, some methods investigate the choice of weights to fine-tune. SpotTune [22 learns where to fine-tune through an additional policy network. However, Spot needs to retain the policy network, the pre-trained model, and the fine-tuned model in memory for inference, adding sigmificant computation at inference time. Recently, SurgicalFT [23 proposes to use the GradientNorm heuristic, the ratio of the gradient norm to the parameter norm, to determine which layer to fine-tune. parameter-efficiee this category. While they aim to minimize the parameters tuned, they have been shown improve
fine-tuning methods are another example of
OOD generalization performance as well in NLP applications [24] 25 26 27 28 (2). We specifically
compare to two recent parameter-efficient methods that only tune the bias terms: Bitfit (30) for
Transformers [3 and Partial Fusion 32 for ResNets 33 How much to fine-tune: Our work
belongs to this category where the entire neural network fine-tuned simultaneously. Specifically, we can split works into two sub- cub-catgoo regularization and projections. Regularization: DELTA (34] proposes to regularize the output (feature maps) of a neural network to that of its pre-trained model. This requires two separate passes through the pre-trained model and the fine-tuned model increasing both memory and computation overhead. L2-SPP 335 instead regularizes the L2 distance between the fine-tuned model and the pre-trained model, serving as strong baseline. Projection: Utilizing projection to enforce a close distance to the pre-trained model has been studied in prior works: MARS-SP [9] and TPGM [O). We dedicate section to revisit them later in the method section Pre-trained Models. Several recent works have proposed special fine-tuning strategies for language-image pre-trained models with zero-shot capability. WISE-FT g achieves SOTA performance by linearly interpolating a fine-tuned model and its initialization at the end of fine-tuning. However, it only applies to subset of pre-trained models with linear connectivity such as CLIP 4 Hi-Like-Pretraii (36) proposes to use a contrastive fine-tuning strategy, the same strategy used in pre-training for those models, instead of the conventional cross-entropy loss for many vision tasks. The method has demonstrated superior results when combined with WISE-FT, where WISE-FT contributes the most to the improvement. Similarly, will also combine method with WISE-FT to show improved OOD performance using CLIP.
(Sec. 3.1 Language-Iman
3 Method
3.1
Review: Enforcing Projection and Learning Constraints
In this work, we focus on fine-tuning pre-trained model, where Wo Raxm is the weights of linear layer in the pre-trained model, to a downstream task. We denote W, as the fine-tuned model at training iteration and w' the ith row of matrix W E Raxm Several prior works TO] have attempted to use projection to improve fine-tuning robustness. The most vanilla formulation in MARS-SP g has two steps: unconstrained gradient descent and projection.
Unconstrained Gradient Descent (Abbrev. UGD), the projection- based methods first compute the updated model weights W. without projection. For example, at iteration f, given a batch of training data D we first obtain W, as the following,
ge TDpp-(WWL1 Raxm
W. <- Opt (..(:W...):).
(I)
where g: the derivative of the loss function Per (-) calculated on D' w.r.t. W and Opt(.) is an existing optimization algorithm, such SGD, AdamW (37.
3

---

Projection. MARS-SP 21 projects the updated model We towards its initialization Wo with pre-defined projection constraint y for all layers using the MARS matrix norm (see Appendix 8.2) as shown below in Eq. 2
[iwi-wáll (w) w/) wi)
W. n(W, Wo.n)=
[W7 (W? w + w
However, MARS-SA SP has poor scalability because is practically intractable to hand-tune different constraints for each layer, which results in sub- aub-optimal performance as reported by TPGM [IO]. Instead of a pre-defined y for all layers, TPGM proposes to learn a different constraint "/ for each layer] and updates them iteratively during training. This enables TPGM customize different regularization strength for each layer and to have superior performance for both ID and OOD data.
ProjUpdate. Given as input the frozen unconstrained model W. from UGD (Eq. TPGM adds an intermediate ProjUpdate function before projection, which samples separate set of data from the validation dataset Dj and uses a standalone training loop to update the projection constraints Tt while keeping the model W, frozen. Specifically, ProjUpdate creates a temporary projected p by projecting We towards Wo based on the previous constraint "/-1 using Eq. i.e.
model Wp
W, p = II( Wr
Therefore, Wp(nt- can be viewed as a function of 7r-1- Then FTP calculates the gradient Vn by taking derivative of the loss function L (W (Wp(n-1)) w.r.t. 7/11
V* + De (Wp(1--)),
7t = Opt VTs)
where
W p = (w wpr... WIlT and
w
wolli (W/ w + wo.
With the calculated gradient, TPGM uses an existing optimizer Opt(:) :) to update 7 This procedure, sampling Diaa and calculating the derivative VD (Wp(1-1)), is the key to learning projection constraints because the unconstrained model W, (highlighted above and calculated in D.. was updated on the training data D and ?t is now updated on separate data Di The discrepancy between D and Di allows TPGM to find a better projected model W, p (projected between W; and Wo) by updating ?/. which balances between fitting the training data D.r and generalizing to Deal Finally, with an updated "/. TPGM again projects W, towards Wo to obtain the final model W, using Eq. [ replacing the pre-defined y with a learned 7t- A flow chart of TPGM is in Fig. 2
The algorithm demonstrated the capability to automatically learn different constraints for each layer, solving the scalability issue in MARS-SP. However, TPGM introduces extra computation in the additional training loop. the next section, we propose a scalable and efficient projection algorithm that learns the projection constraints for each layer without separate validation data and loops.
3.2 FTP: Fast Trainable Projection
To inherit the scalability of TPGM while reducing the computational overhead, we propose Fast Trainable Projection (FTP) (Algorithm D Similar to TPGM, the algorithm has three components: UGD, ProjUpdate, Projection. The ProjUpdate component the major contributor to efficient computation. It builds on a key insight: Instead of sampling separate data D each time, we use two training data batches sampled independently at consecutive steps, e.g., D and D Specifically, we use D to update 7t instead of Dooot As a result, the optimization of 7t re-uses most of the computation used for the optimization of the main model.
ProjUpdate. Specifically, instead of taking derivative of L (W,) w.n.t. Tt-1 as TPGM, FTP calculates the gradient of 7 by the derivative of the loss function on the current training
data D (Wr-1) w.r.t. 7t-1. Note that W II( W.-1 Wo, 7t-1) is also function of the
We omit the index for different layers to avoid notation clutter and the subscript t indicates training iterations. We use this functional form W Wy(x-1) to highlight the dependency 7--1.

---

Algorithm FTP: Fast Trainable Projection.
Require: Wo the pre-trained model
Require: x positive gradient annealing rate
Require: μ f 1e 2, 81, $2 + (0.9, 0.999) fixed parameters for AdamUpdane
for t 1... I do
g: + VLD. (W,-1)
b Unconstrained Gradient Descent (Eq.
W + Opt(Wr-1.8)
1 then
7 le -
Initialize
else
Vi + Ei gi AT (wi w wi-1 wall
if V 0: Vi ~VV
ProjUpdate (Eq. 4. 5 Alg.
Ttt AdamUpdatee76-1: V)::
W. (IW,, Wo
b Projection (Eq.
TPGM
was updated on IDf", Y: updated on 10pe!
W1-1
Lg Lg:::We.. e-.))
W
W II Wo Wo-
(Wp (ri-z)
Yt
W. (1 Wo.rt)
FTP
W was updated on Df is updated on DD?"
W1-1
Cyg (Wg-(We-1)
W.
n(W
oY
We (W.-1 (2-13)
Yi
(w..).
Unconstrained GD
ProjUpdate
Projection
Figure 2: Computation Flow Chart of TPGM (top) and FTP (bottom) at iteration The main difference between TPGM and FTP is the PorjUpdate step. FTP uses the previous model W,-1 and cached gradients from Cp; (W(-1) to update the projection constraints t
constraint 7t-11 as result projection from the previous step. Hence, by virtue of the chain rule, the gradient of Dip (W, (7-1(71-1) w.r.t. 7t-1 is,
Ow
1
V
Di (w (wi-1 (76-1))
Σ 1T [w/
w)) [w
(4)
θn-1
wolli
where the summation loops over each row in the matrix W, because the same constraint 7t- is enforced for all rows (see MARS norm in Appendix Eq. 15 and Eq. 2 so the final gradient the summation of all gradients for each row. Similar to TPGM, because the starting point of projection Wi-1 (highlighted above) was updated using the previous training batch D and the gradient V calculated using the current batch D the discrepancy between D and D enables FTP to learn meaningful projection constraints. Crucially, proposed novel formulation that allows for re-using the gradient g, used for calculating the unconstrained model W, in the UGD step (Eq. 1).
Gradient Annealing, Prior work [10] noticed that learning projection constraints for each layer can suffer from underfitting because the learned constraints can be too conservative, and used an additional regularization to help reduce this negative effect. For FTP, we introduce simple technique that uses a single gradient annealing factor for all layers, к E [0, 1] to modulate the magnitude of the positive gradient Vyr > which contributes the shrinkage of the constraints. When V > O,
V = KV
(5)
For example, when 0, the projection constraint will not receive any positive gradient and is therefore non-decreasing. With the annealed gradients Vir, we update the constraint using the Adam update rule (38 because Adam is suitable for non-stationary optimization, where the optimal values change over time. Please see Appendix 8.3 for detailed algorithmic description of AdamUpdate and additional discussion how FTP saves computation.

---

Finally, after obtaining the updated "/ from Adam FTP applies the learned constraints to project the current unconstrained model W, towards the pre-trained model Wo using Eq. with different constraint for each layer. The complete algorithm summarized in Alg. [ For quick comparison with TPGM, we provide a side-by-side computation flow chart of FTP in Fig. 2
Implicit Assumption. The algorithmic difference between TPGM and FTP makes an implicit assumption. Specifically, after obtaining the updated constraints 7t after AdamUpdate if the algorithm were to follow TPGM, the next step would be applying the updated constraints to re- calculate the previous model W since is updated based on W (Eq. 4.. However, instead of rolling back. FTP applies the updated constraints directly to the current unconstrained model Wr to calculate Wt- This step assumes smoothness in the update of "t. i.e., the " does not change drastically in consecutive steps. The assumption is valid since is updated by Adamp in Appendix (..3) which uses moving average update with momentum of 0.9. So the change of y is very smooth because of the high discount factor of 0.9. Importantly, it enables us to re-use the same gradient g: available for computing the current unconstrained model W, to update "/-. This is the key to saving computation because the separate training loop as a result of "rolling back" the main computation bottleneck in TPGM.
(Alg. 2
3.3 FTP as a Hyper-Optimizer for Fine-Tuie Fine-Tuning
The FTP algorithm Alg. T bears motivational and algorithmic similarity to recent resurrection
of hyper-optimizers [11 12 13 14
17 18] Specifically, hyper-optimizers aim to learn the hyper-parameters such as the learning rate in an optimizer by treating them as learnable parameters through nested differentiation and optimization because manual tuning of those hyper-parameterd can be time-consuming and can lead to sub-optimal performance. FTP stems from the same motivation as the manual specification of projection constraints can be computationally infeasible [TO].
15
16
To understand the algorithmic similarity better, let's use SGD as an example. Suppose iteration t 1, we have updated the model parameters W through SGD with a learning rate at-1-
W W - 04 (L(
At the current step I, hyper-optimizers first calculate the gradient w.r.t to the learning rate a; and update it using another SGD optimizer with new learning rate parameter K.
OC(We-1) a.- (.
02 04--1
da
Finally, using the updated og hyper-optimizers update the main model parameters.
= W
(8)
It's not hard spot the algorithmic similarity between the FTP algorithm and hyper-optimizers. Both algorithms first update the typer-parameters (projection constraints " in 4q. the learning rate o in Eq. 7 using the cached information from the previous iteration and the gradient from the current iteration (known as grprr-ggadinnts.. Then, they apply the updated parer-parameters to calculate the current model. Finally, hyper-optimizers make the same assumption of smoothness in the update of the hyyer-parameed such that the update of the hyper-pane paaeer--rameterss and the model parameters can be performed consecutively in a single forward pass. In this regard, FTP can be seen as a special instance of opper-opt for fine-tuning.
3.4 Dual Perspective: Fine-tuning Robustness in Feature Space and Weight Space
It is not immediately clear why FTP's and other methods' projections the weight space maintain the robustness of the pre- pre-trained model in the feature space besides the intuition that the closer to the pre-train model the more likely the fine-tuned model will behave like it. To fully understand this, we study the mathematical connection between projection and robustness. Let x E R. denote an input vector and h(x)) I + R² function mapping to a feature space. Given two input vectors x, x' E Rmm we denote the distance between them in the original space by their vector norms ||x - x'||x and in the feature space by [[^(x) h(xx)]),. Let hy(:) and h- denote fine-tuned (-) - ho(-) denotes the difference function. model and its pre-trained initialization, and Ah(-) = h-(())

---

To capture the robustness of a tine-tuned model, we apply the notion of Lipschitz continuity on the difference function because its Lipschitz constant captures the maximum rate of change differences between the fine-tuned model and the pre-trained model in the feature space. Formally,
|| - Ah(x'Iln S LaI|x x'Ixx V(x, x') E R
(9)
where La 0 is the Lipschitz constant of the difference function Ah(x). If the inequality is satisfied, in this paper, we call (-) La-LLipschitz-robus w.r.t. the pre-trained initialization ho(-).
The definition has natural intuition stemming from Lipschitz continuity, measure of robustness [19 20 21]. A Lipschitz function is limited by how fast it can change, governed by the Lipschitz constant. Traditionally. a small Lipschitz constant is associated with better robustness, because a small constant means less sensitivity to changes in the input. We provide the following lemma (proof in Appendix to illustrate the connection between the difference function and the robustness of the fine-tuned model.
Lemma 1. If a fine-tuned model h (-) is trained initialization ho(-). i..., v(X, x') E IR""
peapsshhtt-rrobuss with respect to its Lo Lo-Lipschitz pre-
[|a((x) Allx')ll S Lal|x - xlIx and [1x(x)) h(x')olaa S Lollx - xxlxxxxllx
then, h is (Ia + -iipscict i.e., [}b{(x) - hx(x')ll S (La+ Lo)l|x - x'Ixx V(x, x') E E
Feature Space. From Lemma| [ we can see that minimizing La can improve the robustness of the fine-tuned model, defined by its Lipschitz constant (Tug + Too), which equals To when T.a O. Therefore, the fine-tuned model can achieve similar level of robustness as the pre-trained model if La is minimized. Colloquially, given two inputs (x, x'), where x' is a perturbed version of x, hy will be just as sensitive/robust to the perturbation as (-) is if La is small.
Weight Space. The definition of fine-tuning robustness (Eq. 9 not only leads to interpretation of robustness in the feature space (Lemma I but also conveniently a projection operation in the weight space. Specifically, we investigate a single linear layer in neural network and show that enforcing the inequality in Eq. leads to a projection operation by virtue of linear operators and matrix norms. We illustrate this in the following lemma with full discussion and proof in Appendix 8.2
(..2
Lemma 2. Assuming linear models h(()) = Wx + b. W E Raxm b E R", and both the input space vector norm || lIx and the feature space vector norm [| I/ are defined by loo. norm. w p
satisfies
the inequality in Eq. if
Ls
min 1,
(w) w)) + wo. Vi (L,..,,
(10)
(w'y
willa
where w denotes the i-th row of the matrix W and w/ is the new projected fine-tuned model.
This is an equation of projection between W and We defined by the MARS norm in the weight space for a single linear layer and is the projection operation used in FTP and prior works [9] 10 (Eq. 2 It indicates that we can choose an arbitrarily small L and enforce it through Eq. 10 potentially trading off fitting the downstream task and preserving robustness. In summary, this section demonstrates the connection between robustness and projection. Specifically, we have shown that to achieve good fine-tuning robustness, we can enforce small Lipschitz constant La the difference function A (x(x) in the feature space (Lemma D. which can be physically enforced through the projection of the fine-tuned model towards the pre-trained model in the weight space (Lemma.
4 Experiments
Overview. To validate the effectiveness of FTP in fine-tuning pre-trained models, we benchmark FTP on both image classification [.1c and dense vision tasks (Sec. (42) with different network architectures and pre- pre-trained models. For each benchmark, we report both in-distribution (ID) performance as well as out-tf-distribution (OOD) performance. We show that FTP not only achieves competitive ID performance and superior OOD performance but is also much more computationally efficient than prior works. We further test FTP's regularization capability on a continual learning benchmark and show state of art performance against recent SOTA methods (Sec.
7

---

Table 1: DomainNet Results using MOCO-V V-3 pre-trained ResNet50 with 100% Real Data. FTP achieves the best OOD performance and is much faster than prior work TPGM 10] by 36%.
ID Real
OOD
Staristies
Sketch
Painting
Infograph
Cligurt
0OD Avg.
ID A (%)
0OD A (%)
Time (s/i))
Vanilla FT
81.99 (0.03)
31.52 (0.33)
42.89 (0.53)
18.51 (0.28)
44,98 (0.24)
34.17
0.00
0.00
0.35
73.01
(0.23)
(0.15)
12:27 (0.02)
-10.96
Partial Fusson
32
78.27 (0.03)
27.72 (0.07)
39.74 (0.12)
15.56 (0.08)
38.18 (0.12)
30.30
.4.55
-12.11
0.21
L2-SP [15
$1.51 (0.02)
34,91 (0.22)
45.76 (0.16)
18.97 (0.11)
45.29 (0.18)
36.23
-0.59
5.09
0.46
MARS- SP 9
81.89 (0.01)
34,44 (2.54)
45.05 (1.91)
19.97 (1.48)
46.36 (1.29)
36.45
-0.13
5.74
0.43
20.12 (0.43)
(0.27)
36.79
TPGM 10
82.66 (0.13)
35.35 (0.33)
46.20 (0.20)
20.13 (0.12)
45.75 (0.12)
36.86
0.82
6.91
0.80
FTP 82.17 (0.02)
36.26 (0.06)
46.58 (0.10)
20.67 (0.03)
46,97 (0.06)
37.62
0.22
9.13
0.51
Table 2: DomainNet Results using CLIP pre-trained ResNet50 with 100% Real Data. FTP achieves competitive OOD performance and is much faster than prior work TPGM [T0] by 36%.
Staristies
Real
Sketch
Painting
Infograph
Clipurt
OOD Avg.
ID (%)
OOD A (%%
Time (shi))
Vanilla FT
80.95 (0.08)
31.81 (0.06)
41.02 (0.10)
20.29 (0.08)
43.59 (0.15)
34.18
0.:00
0.00
0.58
Linear Proh.
52
(0.09)
20.05 (0.21)
24.92 (2.49)
19.18
21.15 (0.18)
21:33
-35.05
-37.60
0.14
(0.32)
24.71 (0.18)
36.73
1.2-SP
82.07 (0.09)
36.67 (0.11)
45.62 (0.35)
22.97 (0.42)
47.78 (0.30)
38.26
1.40
11.94
0.62
MARS-SP SPS-SPP0 9
77.19 (0.63)
25.33 (1.07)
33.43 (2.06)
14.81 (0.43)
39.20 (0.74)
28. 19
-4.62
-17.53
0.61
LP-FT
80.82 (0.95)
34.85 (1.93)
44.03 (0.05)
22.23
46.13
36.81
-0.14
7.69
TPGM 10
83.64 (0.01)
(0.42)
(0.31)
(0.25)
39.65
FTP
84.22 (0.11) 37.66(0.45)
(6.11
28.33 (0.33)
47.67 (0.18)
39.94
4.05
16,87
0.68
4.1
Image Classification Experiments
4.1.1
DomainNet
For the DomainNet experiment (image classification), which consists of five domains, Real, Sketch, Painting, Infographics, and Clipart, we follow the setup of the prior work [10] and use its released code to train FTP. Specifically, ResNet50 [3] and a CLIP pre-trained ResNet50 [4. For FTP, we only tuned the learning rate while keeping the other hyper-parameteri fixed in the prior work. We use the Real domain the II) training dataset and the rest as OOD testing datasets. Please refer to Appendix 8.4 for more details.
two pre-trained models, an ImageNet pre-trained MOCO-V3
FTP achieves the best OOD accuracy and is much more efficient. In Tab. T and Tab. we show results training on 100% DomainNet-Real data using CLIP and MOCO-V3 pre-trained initialization respectively. Compared to the previous SOTA methods TPGM [IO, FTP achieves competitive ID accuracy and better OOD generalization performance. More importantly, in addition to favorable results. FTP is faster on average both benchmarks compared to TPGM. Following TPGM [10), we also report results training only on 10% DomainNet- -eal data in Appendix Tab.
4.1.2 ImageNet
Recently, zero-shot language-vision pre-trained models such as CLIP 4 have demonstrated strong generalization capability to other tasks. Notably, WISE [ showed that linear interpolation between a fine-tuned model and its initialization achieves significant improvement in OOD generalization.
0.00 perfommacce
Table 3: ImageNet Fine-tuning Result using CLIP ViT-Base.
54
ID
OOD
Statisties OOD Ave.
8
Tm
ImV2
Im-A
Im. R
Im-S
Ave.
27
zero zero-shot shot
67.68
61.41
30.60
56.77
45.53
48.58
52.40
WISE-FT FTS
vanilla FT
83.66
73.82
21.40
43.06
45.52
46.98
54.29
Linear Prob.
78.25
67.68
26.54
52.57
48.26
48.76
54.66
67
80
72
10
73
performance
75
77
72
L.--T
82.99
72.96
21.08
44.65
47.56
46.56
53.85
1.2.PP
35
83.44
73.2
20.55
43,89
46.60
46.06
53.54
Figure 3: ImageNet WISE Interpola- tion 8 Result using CLIP ViT-Base Fine-tuned models.
FTP
84.19
74.64
26.50
47.23
50.23
49.65
56.56
WISE.FT
8
80.94
72.47
33.18
63.33
$4.20
55.58
60.82
WISE
82.61
74.09
61.18
55.06
56.22
61.50
8

---

However, there are two limitations: 1) not all pre-trained models have this property of linear connectivity and 2) a zero-shot classifier head is needed to initialize the linear classifier head. Our contribution is orthogonal to WISE because FTP is general optimization algorithm whereas WISE is a post-training algorithm for specific zero-shot models. Therefore, we first compare FTP to vanilla fine-tuning and then apply WISE to both models. We follow the public code base of DEIT [39 to train our CLIP pre-trained ViT-Base. Specifically, we use weight-decay (0.1), drop-path (0.2) 40), labelssmothinn (0.1) [41) Mixup (0.8) (42 and Cutmix (1.0) 43] We train our model on ImageNet and report OOD performance on ImageNet-V2 44], ImageNet-A 45 ImageNet-R [46]. and ImageNet-S 477. Please refer Appendix 8.4 for more details on implementation.
FTP outperforms vanilla fine-tuning and improves WISE performance. In Tab. 3 we report performance for competing methods. Even with various regularizations and augmentations in place, FTP can further improve ID performance on ImageNet. Furthermore, FTP brings better OOD) performance on all four OOD datasets. This shows that FTP successfully maintains the robustness of the pre-trained CLIP model while existing regularization such as weight decay and drop-path do not. We also report the interpolation results using WISE 8 for the vanilla fine-tuned and FTP fine-tuned models. We sweep a range of interpolation ratios E {0.1, ...,....... and show the trajectory of ID vs. OOD performance plot in Fig. 3 The models with the best average performance are reported in the lower portion of Tab. 3 As expected. WISE interpolation significantly improves OOD generalization for both methods. However, WISE-FTP has significantly better ID performance while still having better OOD performance. This shows that improvement to the base fine-tuning strategy can further benefit pose- training methods such WISE.
4.2 PASCAL Dense Vision Task Experiments
Table 4: Pascal Semantic Segmentation Results using SWIN-Tiny transformers pre-trained on ImageNer21 K. Performance is measured by mloUf. FTP achieves the best OOD performance and is much faster than prior work TPGM [10 by 34%.
ID
OOD
Statistics
Clean
Fog
Defocus
Gaussian
Brightness
OOD Avg.
I1) (56
0OD (%)
Time (s/1t)
Vanilla FT
66.03 (0.37)
56.72 (0.83)
38.04 (0.83)
23.21 (0.96)
58.03 (0.66)
44.00
0.00
0.00
0.288
Adapter 24
71,85 (0.06)
69. (0.07)
50.94 (0.25)
37.43 (0.64)
68.26 (0.08)
56.50
8.82
28.40
0.233
BitFa 30
70.31 (0.11)
67.00 (0:24)
46.39 (0.35)
30.61 (0.51)
66.22 (0.16)
52.56
6.49
19.44
0.248
L2-SP 335
73.47 (0.06)
69.87 (0.04)
49.20 (0:43)
39.10 (0.54)
68.61 (0.24)
56.70
11.27
28.85
0.347
MARS-SP [
66.24 (0.23)
56.97 (0.79)
37.29 (1.20)
21.82 (2.05)
58.27 (0.33)
43.59
0.32
-0.94
0.318
LLRD 4%
72,09 (0:06)
68. 13 (0:25)
46. (1:30)
37.28 (2.54)
66.30 (0.29)
54.47
9.18
23.79
0.289
TPGM [b]
72.56 (0.06)
69.51 (0.57)
50.88 (0.97)
38.62 04)
68.82 (0.25)
56.96
9.89
29.44
0.611
FTP
73.79 (0.10)
71.10 (0.23)
52.63 (0.75)
40.25 (0:21) 69.81 (0.49)
58.45
11.76
32,83
0,401
To further demonstrate the effectiveness of FTP in more diverse scenarios, we test it on PASCAL- Context [49]. Specifically, following the prior work [50). we use the PASCAL-Contex datasets (49) which consist of labels for semantic segmentation, human parts segmentation, and surface normal estimation. For OOD performance, following the popular natural robustness literature [5I]. we report results on various degradations including fog, defocus blur, Gaussian noise, and brightness Corruption, with 5 severity each. We use a combination of Swin ViT-Tiny [52] (pre-trained on ImageNet-222) and Segformer [53]. In this architecture, Swin Transformer serves as the feature extraction backbone and Segformer is the task-specific decoder. While the feature backbone is initialized with pre-trained weights, a significant part of the entire model (the Segformer decoder) is randomly initialized; In contrast, in simple classification (Sec. 1.1.), only the last linear classification layer is randomly initialized. Please refer to Appendix 8.5 for details.
FTP achieves the best ID performance and OOD generalization. We report results for semantic segmentation, human parts segmentation, and surface normal estimation in Tab. [ Appendix Tab. 7 and Appendix Tab. 8 respectively. We additionally add Layer-Wiss Learning Rate decay [48] LLRD) as a strong baseline. Notably, in all three tasks, FTP outperforms vanilla fine-tuning on ID performance by 11.71% 4.48%, and 18.30% respectively. This demonstrates the effectiveness of projection as a regularization technique for transfer learning. More importantly, the OOD performance improves as large as 33.02% in semantic segmentation. This shows that 1) FTP can effectively maintain the robustness of the original pre-trained model; 2) even though the entire decoder component is randomly initialized, it is worthwhile to put regularizations on the pre- pre-trained feature backbone.

---

0.14
0.12
Table CL Results on ImageNct-R
Magnitude
0.1
Method
A.x (t)
(J)
0.06
FT++ [54]
18.93 = 1.15
9.81 = 0.31
LWFMC
66.73 +
3.52
0.02
L2P++ 56
71.66 = 0.61
1.78 0.16
DualPrompe
71.32
1.71 0.24
Taski
Tatk2
Tasld
Tasid
TaskS
Taske
Task?
Tasks
Tadl
Task10
CODA-P 54
75.45 0.56
1.61
EWC FS
64.66 2: 2.04
1.35 0.25
L2 64
76.06 = 0.63
1.68 0.16
Figure 4: Average Learned Constraints for each task using FTP.
FTP
76.06 # 0.35
2.27 0.18
0.40
0.15
4.3 Continual Learning (CL) Experiments
Recently, pre-trained models have been shown greatly improve the performance of CL algo- rithms [59]. We follow the settings in this work (59] to partition ImageNet-R (200 classes) into 10 sequential tasks with 20 non-overlapping classes in each task. A model is trained each task only once sequentially. To use FTP for CL. tasks, unlike supervised vision tasks (Sec. 4.
we
re-initialize FTP after each task and use the current model the "pre-trained model" for the next task.
Moreover, inspired by the prior work [59], we use FTP to only fine-tune the attention blocks. We report both the final task accuracy across all tasks AINN and the global forgetting F'v in Tab. 5 analyze plasticity and forgetting. Please refer to Appendix 8.6 for more on the metrics and experimental setup. In T we benchmark against the popular and recent rehearsal-free continual learning methods. FTP alone achieves state of art accuracy against all methods and relatively good forgetting compared vanilla FT, sign of superior plasticity and balanced forgetting. We visualize the learned constraints for each task in Fig. We observe that while each task is independent and FTP is re- re-initialized each time, FTP learns stronger regularization for later tasks. This contributes to lower forgetting compared to FT. We found that FTP combined with a simple continual learning method, EWC (58), achieves state-of- tate-of-theaare the-art in this setting. Compared the prompting methods L2P, DualPrompt, and the recent CODA-Prompe FTP has clear and significant improvements. Our intuition is that the combination of the superior plasticity of FTP and the low forgetting of EWC is the key to improvement.
Limitations
Like any regularization method, FTP has arper-parameter to adjust its regularization strength. In this case, the positive gradient annealing factor 0 < к < 1 (default 1) (Sec. 3.22) controls the strength of projection with smaller values indicating weaker regularization. Note that that the projection constraints are non-decreasing during training. this case, FTP still provides regularization. For example, found that 0 is necessary obtain the best performance for some dense vision tasks Appendix Generally, we recommend starting with the default k and only tuning it if underfitting observed.
means
Conclusion
In this paper, we proposed Fast Trainable Projection, fine-tuning algorithm to maintain the robustness and the generalization capability of the pre-trained model. FTP learns projection constraints for each layer in neural network efficiently by carefully re-using past information to save computation. To understand the connection between robustness and projection, we provided holistic discussion of fine-tuning robustness from its feature space definition to the weight space dual. The new perspective lends mathematical foundation to the idea of using projection in fine-tuning. Across four vision tasks with different pre- pre-trained models, FTP demonstrated superior ID and OOD generalization capability and significantly better computation efficiency. Furthermore, the continual learning experiments demonstrated FTP's potential in other deep learning paradigms beyond simple fine-tuning. Combined with its compatibility with popular optimization algorithms, we believe FTP can broadly beneficial in improving the performance of learning tasks using pre-trained initialization.
10

---

7 Acknowledgements
This work was supported by ONR grant N00014-1--1-229.
References
[1] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[2] Mathilde Caron, Hugo Touvron, Ishan Mísra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. Proceedings of the JEEE/CVF international conference computer vision, pages 9650-9660, 2021.
[3] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self- vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
[4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 8748-8763. PMLR, 2021.
[5] Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4927,, 2019.
[6] Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning, pages 2712- 2721. PMLR, 2019.
[7] Ananya Kumar et al. Fine-tuning can distort pretrained features and underperform out-of- distribution. ICLR, 2022.
[8] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959-7971, 2022.
[9] Henry Gouk, Timothy M Hospedales, and Massimiliano Pontil. Distance-based regularisation of deep networks for fine-tuning. ICLR, 2021.
[10] Junjiao Tian, Xiaoliang Dai, Chih-Yao Ma, Zecheng He, Yen-Cheng Liu, and Zsolt Kira. Trainable projected gradient method for robust fine-tuning. arXiv preprint arXiv:2303. 10720, 2023.
[11] Kartik Chandra, Audrey Xie, Jonathan Bagan-Kelley and Erik Meijer. Gradient descent: The ultimate optimizer. Advances in Neural Information Processing Systems, 35:8214-8225, 2022.
[12] Xiang Wang, Shuai Yuan, Chenwei Wu, and Rong Ge. Guarantees for tuning the step size using a soarning-to-leare approach. In International Conference on Machine Learning, pages 10991- PMLR, 2021.
[13] Matthias Feurer and Frank Hutter. Hyperparameter optimization. Automated machine learning: Methods, systems, challenges, pages 3-33, 2019.
[14] Atilim Gunes Baydin, Robert Comish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. arXiv preprint arXiv: 2017.
[15] Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In International conference on machine learning, pages 737-746. PMLR, 2016.
[16] Justin Domke. Generic methods for optimization-based modeling. In Artificial Intelligence and Statistics, pages 318-326. PMLR, 2012.

---

[17] Luís B Almeida, Thibault Langlois, José D Amaral, and Alexander Plakhov. Parameter adaptation in stochastic optimization. In On-line learning in neural networks, pages 111-134. 1999.
[18] Yoshua Bengio. Gradient-based optimization of hyperparameters. Neural computation, 12(8):1889-1900) 2000.
[19] Patricia Pauli, Anne Koch, Julian Berberich, Paul Kohler, and Frank Allgöwer. Training robust neural networks using lipschitz bounds. IEEE Control Systems Letters, 6:121-126, 2021.
[20] Yujia Huang, Huan Zhang, Yuanyuan Shi, J Zico Kolter, and Anima Anandkumar. Training certifiably robust neural networks with efficient local lipschitz bounds. Advances in Neural Information Processing Systems, 34:22745-22757, 2021.
[21] Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 110:393-41 416, 2021.
[22] Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4805 4814, 2019.
[23] Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. arXiv preprint arXiv:2200111666 2022.
[24] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference Machine Learning, pages 1790-2799, PMLR, 2019.
[25] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint MxXiv:211.00/990 2021.
[26] Sang Michael Xie, Tengyu Ma. and Percy Liang. Composed fine-tuning: Freezing pre-trained denoising autoencoders for improved generalization. In International Conference on Machine Learning, pages 11424-1 1135.. PMLR, 2021.
[27] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, arXiv preprint arXiv:2104.080 04.69991..86.... 2021.
[28] Prasetya Ajie Utama, Nafise Sadat Moosavi, Victor Sanh, and Iryna Gurevych. Avoiding inference heuristics in few-shot prompt-based finetuning. arXiv preprint Sitiv:2009.0144: 2021.
[29] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 13009)23377-2388, 2022.
[30] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efffcient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2 10199, 2021.
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[32] Fahdi Kanavati and Masayuki Tsuneki. Partial transfusion: on the expressive influence of trainable batch norm parameters for transfer learning. In Medical Imaging with Deep Learning, pages 333-353. PMLR, 2021.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[34] Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, Zeyu Chen, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. arXiv preprint 10X1:09900.002220 2019.
12

---

[35] LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International Conference on Machine Learning, pages 2825- 2834. PMLR, 2018.
[36] Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. arXiv preprint (xXiv:2222.006388, 2022.
[37] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[38] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
[39] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers amp; distillation through attention. In International Conference on Machine Learning, volume 139, pages 10347-10357, July 2021.
[40] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07645 2016.
[41] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[42] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond 2017.
empirical risk minimization. arXiv preprint
[43] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
[44] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR, 2019.
[45] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[46] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli. Mike Guo, et al. The many faces of robustness: A critical analysis of out distrribbtt generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8349, 2021.
[47] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[48] Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting few-sample bert fine-tuning. arXiv preprint arXiv:2006 200:2000.05977, 2020.
[49] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.
[50] Yen- Yen-Cheng Liu, Chih- Yao Ma, Junjiao Tian, Zijian He, and Zsolt Kira. Polyhistor: Parameter- efficient multi-task adaptation for dense vision tasks. arXiv preprint arXiv:2210.03265, 2022.
[51] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint
12261, 2019.
[52] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
13

---

[53] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 34:12077-, 2021.
[54] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-promph, Continual decomposed iitention-based prompting for rehearsal-fit -free continual learning. arXiv preprint arXiv:2211.132188 2022.
[55] Zhizhong Li and Derek Hoiem. Learning without forgetting. JEEE transactions on pattern analysis and machine intelligence, 40(12):2955-2947, 2017.
[56] Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149 2022.
[57] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning, arXiv preprint arXiv:2204.044 2022.
[58] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Ba Barkkk,ka et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
[59] James Seale Smith, Junjiao Tian, Yen-Chang Hsu, and Zsolt Kira. A closer look at rehearsal- free continual learning. arXiv preprint 1rXiv:2203.17266 2022.
[60] Sungyoon Lee, Jaewook Lee, and Saerom Park. Lipschitz-ceertiffblle training with a tight outer bound. Advances in Neural Information Processing Systems, 13:16891-16902, 2020.
[61] Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 312-321, 2019.
14

---

8 Appendix
8.1 Proof of Lemma
Prior works p 10 have used the high-level notion that staying "close" to the pre-trained model can help maintain robustness capability to justify using projection for fine-tuning. However, there is more than one way to encourage this, for example, regularization [35], a small learning rate [7], and projection [T]. It not immediately clear why projection is a principled approach. To understand FTP's capabiliiy to maintain the pre-trained mode's s robustness, we first propose to establish connection between Lipschitz continuity, a commonly used measure of robustness [9 20 21], fine-tuning through a new definition of difference function in the Lemma
and
Proof. We first expand the difference functions in Eq. 9 i.c. plugging in Ah(-) = hT(() ho(-),
||A(()) -1((x))ln S La] x' ||x V(x, x') E R
(11)
+| (1g((x) - -xo(x)) - (1((x))->xx)))) ) La || x'11= 1|x
|| (hy(x) -x)(x²)) (ho(x)) hxxx)) ||. La |x x x
Then we apply the reverse triangular inequality to the hent- side of Eq. 11
(x] h (x'))l [[ho(x) - hu(x')lll - (h)(x) hy()))) (ho(x) ho(x))) )) lla
Therefore, we have,
[1,{(x) ,,(('))(lll) [|ho(x) - ho('')) In S Lallx |X x'1lx 11))(x)) (x'))) l1a Lal[x- - x'1lx + [l/o(x)) ho(x'))l
(12)
Assuming that the pre-trained model ho is Lo ha-Liischitz, we know that [[ho(x) - ho(x')lln S
Tollx x/|x,, v(x, x') E Rm Plug this into Eq. 12
[x) (x)))ln S (La + Lo)l x'llx
(13)
8.2 Proof of Lemma 2
In the previous section, we established connection between the robustness of fine-tuned model h.(.) and its difference function Ah(.). Naturally, if we can limit the Lipschitz constant La of the difference function, we can maintain the robustness the pre-trained model. In this section, we show projection as an effective method to enforce the La-Lipschitz condition in Eq
Proof. Linear Operators. A neural network is composed of linear operators with connecting non-linear activations. Following prior works [9] 10). we analyze the linear operators B] h(x)
Wx + b, W E Raxm R", Let's define f = W by and ho(x) = Wox + bo and
plug them in Eq.
[[ (x) x'Ill & Lall x - x'1 llx
V(x,x') E R
Rearranging the above equation gives us an upper bound on Lds
[[(W,
Wo) x'IIn
La = sup
v(x,x) E Rm)
(14)
x x' ||x
= Matrix Norms. Eq. 14 matches the definition of a matrix norm for a matrix W E Raxm; ||
sup [ Walll. Ya Ra with x + #0} Therefore, to minimize La in Eq. we just need to |=]|x
minimize the matrix norm ||W, Woll Note that different vector norm combinations (|) |/I. and || +1xx) will lead to a different matrix norm || !lb.x. Certain vector norm combinations have a closed-form matrix norm while the majority do not. Following prior works
p 10], we use Maximum
Absolute Row Sum (MARS) matrix norm, which is characterized by loo vector norms in both domains.
"Convoltiooal layers be also written in the matrix multiplication form using Toeplitz matrix.
15

---

Specifically, given a desired constraint La, we want |W, Wolloc.oo S Ld. Per the definition of the MARS matrix norm, which is the largest I norm of each row of a matrix, the inequality can be equivalently enforced for each row independently, i.e.
[W f Wolloc,0 S La
ww woll I La, Vi fl.....n.
(15)
where w' denotes the i-th row of the matrix W.
Projection. To ensure the inequality in Eq. 15 we can project W towards Wo using the following projection equation. For each row w in matrix W, the projected weight w is calculated by
w min
1, [wrj walla (w3 w/) + wó-
?
It is easy to check that w satisfies Eq. 15 i.e., l!w woll S La if S V Lo.
Lipschitz Bound. Since a neural network is composition of linear operators and non-linear activations, by the composition rule of the Lipschitz functions, an upper bound of the entire network just the product of the Lipschitz constant for each linear operator and non- non-linear activations, where most non-linear activations are I-Lipschitz [21] However, the Lipschitz bound obtained by using the composition rule not a tight bound on the entire network. While it an active research area to find tighter bounds for neural networks without relying on the layer-wise wise composition rule [60] (0). the layer-wise approach is particularly suitable for connecting the fine-tuning process and Lipschitz continuity because it leads to layer-wise regularization techniques we demonstrated above.
8.3 FTP: Additional Discussion
In the main paper Sec. Sec.3.2 3.2 we described the algorithmic difference hetween TPGM and FTP. However, there is an implicit assumption made as result of the difference. We now discuss the implications of it. After obtaining the updated constraints "/r in Eq. 5 if the algorithm were to follow TPGM, the next step would be applying the updated constraints to re-calculate the previous model Wr-1- However, instead of rolling back, FTP applies the updated constraints directly to the current unconstrained
model W. This step assumes smoothness in the update of nt i.e., the "/t does not change drastically in consecutive steps. The assumption is valid since V/e updated by AdampUpdate (Alg. beloow which uses moving average update with a momentum of 0.9. So the change of % is very smooth because of the high discount factor of 0.9. Importantly, we have re-used the same gradient g: available for computing the current unconstrained model W.. This is the key to saving computation because calculating the forward and backward pass through the model is the main computation bottleneck in TPGM because requires a separate training loop as a result of "rolling back".
Algorithm 2 AdampUpdate: AdamUpdate implements one step update of Adam 38
Require: 1- Vic,
> Input
Require: / 4 le - 2, (81, 82) + 0.999) + le - 8 > Fixed parameters for AdamUpdate
Require: mı 5-
Initialize 1st moment vector Initialize 2nd moment vector
Require: +0
my + B1m4-1 + (1 31 7
Dt +- 1201
+ (1 (2) Vn?
my + m4/(1 (3))() 3()
6, + 11/(1-13
no + 71-1 - - (vē, +c)
8.4 Image Classification Experiments Details and Additional Results
In Sec. 1.1.1 we presented image classification results on DomainNer- 100% data (1 (111,307 images). Now we further present results using only 10% (11,031 images) of the training data in Tab. In this case, projection-baxee methods, TPGM and FTP achieved the best performance, demonstrating their regularization capability under low-label conditions. Similar to findings in the main paper, FTP is
16

---

Table 6: DomainNet Results using CLIP pre-trained ResNet50 with 10% Real Data. FFTP
achieves competitive OOD performance and much faster than prior work TPGM 10] by 37%.
ID Real
OOD
Statistics
Sketch
Painting
Infograph
Clipprt
(OD Avg.
ID (%5
0OD A (%)
Time (shit)
Vanills
L.
47.19 (0.93)
17.81 (0:25)
22.71 (2.08)
17.13 (0.75)
17.39 (0.69)
18.81
-17.71
-1.52
0.13
PV 13
71.04 (0.91)
27.87 (1.04)
38.31(1.05)
19.85 (0.70)
33,92 (1.59)
29.99
23,86
57.01
0.31
30.48
MARS-SP
9
52.53 (0.84)
15.34 (0.54)
21.57 (0:45)
8.49 (0.60)
19.96 (0.01)
16.34
-8.41
-14.4
0.60
LP.PT
64,11 (0.78)
20.54 (0.27)
30.89 (0.41)
13.58 (0.63)
29.55 (0.82)
23.64
11:78
23.77
TPGM
73.16 (1.27)
29.88
36,80
19.72 (0.12)
35.28 (0.74)
30.42
27.56
59.27
1,10
FTP
72.89 (0.34)
27.44 (0.13)
38.11 (0.26) 20.20 (0.26)
33.58 (0.49)
29.83
27.10
56.19
0.69
FTP Constraints for Each Layer a ResNec50
wellarcann Los ant
Figure 5: Visualization of learned FTP constraints. Settings: We fine-tune a pre-trained ResNet50 on DomainNet-Recd Real for 150 epochs. There are in total 174 constraints imposed on the model excluding the last linear layer. Observations: 1) Early layers (dark colors) generally have smaller constraints than the latter layers (light colors) throughout training. 2) Constraints grow from small to large and converge in the end.
up 37% faster than TPGM during training. Next, describe the hyper-parameters for all image classification experiments in Sec. 4.1 and above.
DomainNet. We the released code from the prior work, TPGM [10 to train FTP model. Therefore, we directly use the reported results from TPGM for competing methods. For FTP, we apply constraints all trainable layers except for the last linear classification layers. For all experiments, we use SGD) as the base optimizer with a weight decay of 5e 4. For DomainNet- 100% and DomainNet- 10% experiments, of 256. We sweep a range of learning rates and use the validation split to determine the best learning rate for FTP for each experiment. Here is the list of best-validated learning rates for all DomainNet experiments. We also provide a visualization of the learned constraints in Fig. lb
train models for 50 and 150 epochs respectively with batch size
DomainNet-100% MOCO-V3 V3 ResNet50 (Tab. 1 1c - 2
1omainNet-100% CLIP ResNet50 (Tab. 2 : 1e
DomainNet-10% CLIP ResNet50 (Tab. 6 le
Note that we use the default x = 1 for all these experiments. Every DomainNet experiment was conducted using 4 RTX 2080 GPUs.
17

---

ImageNet. For ImageNet experiments (Tab. Fig. we use a CLIP pre-trained ViT-Base [4. Unlike the DomainNet experiments, we also mitiialze the last linear layer with zero-shot weights extracted from CLIP text encoder, following the prior work WISE [8] Therefore, FTP is applied to all trainable layers including the last linear layer. Training Transformers have been well-studied with abundant regularization and augmentation techniques. To obtain the best fine-tuning performance, we follow the public code base of DEIT [39 to fine-tune all methods. Specifically, we use weight-decay (0.1), drop-path (0.2) [40], label-smoothing (0.1) [41], Mixup (0.8) [42] and Cutmix (1.0) 43 One exception is Sinear-Probing (LP), where we do not use any of the above augmentations because they have been shown to degrade linear probing performance B I. We train all methods using AdamW [37] as the base optimizer with a weight decay of 0.1, cosine learning rate schedule, and batch size of 256 for 30 epochs. We also sweep relevant document them below.
for each method and
FT: learning rate 2e 5
LP: learning rate 5e 33
LP-FT: learning rate 2e - 5. We take the best LP model (trained for 30 epochs) and then fine-tune it for another 15 epochs with the learning rate specified above.
L2-SP: learning rate 2e 5, regularization hyper-parameter le 5.
FTP: learning rate 3e 5, regularization hyper-parameter default = 1.
Every ImageNet classification experiment was conducted on 2 A40 GPUs.
8.5 PASCAL Dense Vision Task Experiments Details and Additional Results
In Sec. 4.2 we presented results semantic segmentation. In this section, we provide the additional results on semantic segmentation and surface normal estimation in Tab.7ann Tab. [ FTP achieves the best ID and OOD performance with significantly improved computation efficiency over TPGM (10). Next, we will give more details on implementation.
Table 7: Pascal Human Parts Segmentation Results using SWIN-Tiny transformers pre-trained on ImageNet211 K. Performance is measured by mloUf. FTP achieves the best OOD performance and much faster than prior work TPGM 10 by 34%.
ID
OOD
Statisties
Fog
Defocus
Gaussian
Brightness
Avg.
(%)
OOD A (%)
(s/1)..
Vanilla FT
62.61 (0.31) $7.50 (0.73)
40.76 (0.19)
30.64 (0.88)
$7.47 (0.33)
46.59
0.00
0,00
0.280
Adapter
60.84
(1.27)
57.11 (0.39)
45.03 (3.96)
33.12 (1.92)
57.25 (0.68)
48.13
-2.81
3.30
0,221
BidFit
59.06 (0:97)
55.66 (1.36)
45.81 (1.27)
32.18 (2.59)
$5,89 (0.97)
47.39
-5.67
1.70
0.235
12-SP
62.26 (3.17)
58.46 (2.83)
45.35 (1.30)
34.36 (2.79)
58.40 (2.52)
49.14
-0.56
5.47
0.336
MARS-SP
(0.94)
(1.75)
42.51
32.66
58.33
0.50
0,308
LLRD
64.37 (1.80)
60. 10 (2.58)
44.61 (1.95)
36.90 (4.84)
59.84 (2.06)
50.36
2.81
8.09
0.278
TPGM
63.29 (1.72)
60.16 (L.44)
46.91 (1.78)
37,30 (2.60)
59.81 (1.00)
51.04
1,10
9.55
0.602
FTP
65.50 (0.17)
61.73 (0:36)
44.97 (0.70)
40.55 (1.71)
61.23 (0.12)
52.12
4.63
11.86
0.397
Table 8: Pascal surface normal Results using SWIN-Tin transformers pre-trained on ImageNet2 K. Performance is measured by RMSEJ. FTP achieves the best OOD performance and is much faster than prior work TPGM 10 by 35%.
Statistics
Clean
Fog
Defocus
Gaussian
Brightness
0OD Avg.
ID A (%)
OOD A (%)
Time (s/i),
Vanilla
(0.05)
27.33 (0.20)
(0.06)
0.00
Adapter
18.19 (0.05)
20.15 (0.04)
21.45 (0.02)
23.90 (0.14)
19.23 (0.06)
21.19
-4.15
.9.77
0.229
BitFit
20:01 (0.05)
21.93 (0:03)
23.95 (0.
26.92 (0.18)
21.28 (0.05)
23.52
5.43
0.17
0,240
(0.04)
19.26 (0.13)
(0:29)
(0.04)
MARS-SP
19.01 (0.04)
22.15 (0.13)
23.69 (0:11)
27.53 (0.29)
20.86 (0.04)
23.56
0.18
0.32
0.313
LLRD
15.54 (0.08)
18.31 (0.03)
20.01 (0.20)
26.47 (1.45)
17.36 (0.07)
20.54
-18.11
-12.54
0,279
TPGM
18.17 (0.02)
19.74 (0.04)
21.00 (0.15)
23.53
(0.27)
19.02 (0.03)
20.82
-4.24
-11.32
0.616
FTP
15.51 (0.10)
18.19 (0.09) 20.01 (0.21) 26.39(0.788
17.32 (0.10)
20.48
-18.30
-12.79
0.403
Following prior works
50
we use a combination of Swin-Tiny Transformer 52 encoder and
Segformer [33 decoder. The decoder is customized to allow different output formats. Only the Swin
18

---

segme sigmertacoo.fog
segine rtation defocod
2ogmontatinn.ooo gowusiannn
sogmencatto tenss
30
TRE
human perts.log
hummn perbbcefooos
human parts-gaaaa gusssguosssr
Juman partib Strees
nommW Ang
nomm defirru
25.0
24
275
25.0
15.0
Figure 6: Performance Breakdown for each Level of Corruption on PASCAL-Context Vision Tasks.
encoder is initialized with pre-trained weights (pre-trained on ImageNet-22k). Therefore, we only apply FTP to the encoder. For all methods, we Adam as the base optimizer with weight decay of le 4 and learning rate of for 60 epochs. For methods with regularization hyper- pperr-parameters sweep range of values and report the best one. We provide Tab. for reference.
Table 9: Hyper-parameter for PASCAL Dense Vision Tasks Experiments.
Semseg
Human Parts
Surface Normal
L2-SP
5e-4
lc-4
lc-4
LLRD
0.65
0.45
0.65
MARS-SP
4
FTP
1.0
0.0
0.0
To test OOD robustness on the PASCAL-Context benchmark, we apply natural corruptions to the orig- select four types of corruptions from the popular benchmark [51]. each of which is sampled from a main category: noise, blur, weather, and digital. Each corruption has five levels of severity. We report the average values over the five severity in our paper. Here, also provide a detailed breakdown for each level of corruption in Fig. Every PASCAL experiment was conducted single RTX 2080 GPU.
inal clean images. Specifically,
8.6 Continual Learning Experiments Details and Additional Results
In this section, we provide a brief overview of the settings in continual learning (CL). In CL, model trained on sequence task n E (1,... N}. Each task has a non-overlapping set of class labels T,, and we denote the number of classes [Tal For ImageNet-R, we split the 200 classes into 10 tasks with labels each, i.e., 10, [T, = 20. Our experiments belong to the class-increment category in CL. With each new task, the final linear classifier layer is expanded
19

---

with randomly initialized weights. We denote 8,n as the model that has been trained on tasks and the classifier has all classes up and including the n-th task (i 12).
To measure global performance, we first define the global task accuracy A1:N as,
AI.N
Σ (Z.
[(((,,N,INN) = =y).
D
where Diest is the test dataset which has data from all N tasks and ((z, 0) denotes the predicted class from the model with weights Then we define the global forgetting Fv 66 as,
Tyl T Tim
FN
(Rn, R
where,
R1,n
[(j(x, 01,1:)), 1)-
D
(
Following the prior work [59]. all experiments in Tab. 3 use a ViT-Base pre-trained on ImageNet. We tune FTP with the code provided by the authors and directly compare it to the results from the prior work. Specifically, all methods use Adam as the base optimizer with weight decay and batch size of 128. All results are averaged over random seed trials where the class allocation to each task is shuffled. For FTP, we train the model for 25 epochs with an initial learning rate of 5e 4 and a cosine learning rate schedule. For all methods, we freeze the majority of the backbones and only fine-tune the QKV attention layers in the VIT. Please refer to the prior work for more detailed description of the compared methods. Every CL experiment was conducted on RTX2080 GPUs.
8.7 Pytorch Code Example of FTP
Here is example of using SGDP (SGD+FTP) in Pytorch format. SGDP requires the common arguments for initializing an SGD optimizer class Pytorch with two additional inputs: & and exclude set. k is the hyper-parameter for positive gradient annealing (Sec. (.2) and exclude set contains the set of the names of parameters to excluded from the projection operation. A complete demonstration of image classification is provided in the supplementary. You should be able to reproduce FTP results in Tab. and Tab.
from FTP import SGDP
Parameters to be optimized
carams_to_opp [x[1] for x model. named.rreeesss ())
Names of parameters to be optimized
= [x[0] for x in model. named_paraeeterss ()]
Copy the initial parameters as the anchor
params_anchor copy deepcopy (params_to_opt)
Set up the parameter groups param_group = "pre": pachor_
"name": params_to_opt_name]]
Set up optimization ayper-paramett antimizer_parame = "Ir": 1e-2, wveight_decay"" 5.0e-4,
"momentum"; 0.9,
"nesterov"; True, "k":1.0,
"exclude_sed set :f "module. head. weight", "module head. bias")
optimizer SGDP (param groupp oxptiiieer (araiiieer_ppmmm)
20


Reviews:
[{"review_id": "pKPlw7DN2j", "reviewer": "Reviewer_FZ4t", "summary": "This paper presents Fast Trainable Projection (FTP), an efficient fine-tuning algorithm. FTP learns a projection radius based on the current training batch. Through both analysis and experiments, the paper shows that FTP improves out-of-distribution robustness while maintaining in-distribution performance. Experiments also show that FTP can accelerate learning by up to 35% compared to previous methods.", "questions": "- There seem to be similarities between FTP and the general hyperparameter tuning method proposed in https://arxiv.org/abs/1909.13371, in that both methods delay updates with the current batch to optimize hyperparameters in an online fashion (gamma here corresponds to alpha there). Could the authors comment on how these two methods relate?\n- see other questions in \"Weaknesses\" above", "limitations": "Yes.\nNo particular negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea seems to be a natural improvement following the lines of MARS-SP (project after unconstrained GD step) and TPGM (learn a different radius for each layer).\n- Section 3.3 was clear and interesting; it establishes a connection between weight-space projection and \"robustness\" in terms of the Lipschitz constant.\n- The experimental results are strong, across DomainNet, ImageNet, and PASCAL fine-tuning experiments.", "weaknesses": "- A main claimed benefit of FTP (in the abstract and intro) is its computational efficiency. To what extent is computational cost a bottleneck in the experimental settings you consider? I think fine-tuning is generally considered to be pretty computationally light, especially compared to the training of the foundation model itself. Given the experimental results, maybe the more relevant benefits are (1) better OOD acc (2) no need for held-out val data like TPGM.\n- While section 3.3 was interesting, I'm not sure if it contributes the to the main point of the paper (benefits of FTP over e.g TPGM), since this simplified analysis really applies to all projection-based methods. Does this analysis motivate FTP over other ways of using projection for fine-tuning?\nminor typo: Fig 2 caption PorjUpdate -> ProjUpdate", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive and constructive feedback!! We have added additional results/visualization according to other reviewers' comments. We hope they would further strengthen your confidence in our method. \n\n* **Regarding Benefits of FTP**\n\nIndeed, fine-tuning is much less computationally heavy than pre-training. In our experiments, fine-tuning on ImageNet takes about 2 days using TPGM but roughly 1 day using FTP. The speed-up of FTP vs. TPGM is roughly 2x. Nevertheless, as the reviewer mentioned, FTP brings other benefits such as no need for held-out validation data. This is a major improvement because it allows us to integrate FTP into existing optimizers for much better adaptability. For example, we demonstrated the regularization strength of FTP on continual learning experiments (sec.4.3). This is not possible with TPGM due to the lack of validation data under a continual learning setting. Thank you for the suggestions; we will better emphasize all of the benefits in the revised version!\n\n* **Regarding Sec 3.3**\n\nThank you for pointing out the typo. No, the theory does not indicate that FTP is better than TPGM. The improvement of FTP over TPGM is algorithmic in terms of efficiency and applicability, not theoretical. \nNevertheless, section 3.3 introduces a general theory on why projection is useful for fine-tuning, a theoretical question that hasn’t been answered in prior works. While you are right that it generally applies to all projection-based methods including FTP and TPGM, we believe it was important to include to progress this area empirically and theoretically. We will add text and organization to explicitly discuss this. \n\n* **Regarding https://arxiv.org/abs/1909.13371**\n\nThank you for bringing up this work, which is a very valuable reference for FTP. We will cite and discuss this work. The work introduces a smart backpropagation modification for computing “hyper gradients” to optimize internal hyper-parameters in an existing optimizer. From this perspective, FTP can be seen as an extension of this idea of projection-based optimization, where the projection constraints are the hyper-parameters. \nNevertheless, our novelty stands out in two aspects. First, FTP introduces the projection operation as an integral component of an optimizer. The projection has always been treated as a separate operation outside the update of optimizers [1,2]. It is not clear how to apply the idea of “hyper-gradient” unless we formalize projection as part of the gradient update as in this paper. Second, the update of “gamma” uses the Adam update rule and is not simply gradient descent as in the referenced work. The Adam update rule introduces smoothness in the updates of the projection parameters and enables us to apply the updated projection parameters to the current updated model without rolling back to the previous state, greatly saving computation (please see Appendix 7.3 for detailed discussion). \n\n\n[1] Gouk, Henry, Timothy M. Hospedales, and Massimiliano Pontil. \"Distance-based regularisation of deep networks for fine-tuning.\" arXiv preprint arXiv:2002.08253 (2020).\n\n[2] Tian, Junjiao, et al. \"Trainable Projected Gradient Method for Robust Fine-tuning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n"}, {"review_id": "xQiQAFs3dF", "reviewer": "Reviewer_HDd6", "summary": "The paper presents a new algorithm for robust fine-tuning of pre-trained models, specifically focusing on maintaining out-of-distribution (OOD) robustness while achieving competitive in-distribution (ID) performance. The algorithm, dubbed Fast Trainable Projection (FTP), is designed to overcome the scalability and efficiency limitations of current methods, offering a 35% speedup compared to prior work (to be specific, TPGM). FTP achieves this by efficiently learning per-layer projection constraints during the fine-tuning process.\n\nThe claimed contributions of this paper are:\n\n1. The introduction of the FTP algorithm that significantly improves computational efficiency while learning projection constraints and fine-tuning the model simultaneously. This algorithm can be integrated with existing optimizers like SGD and AdamW and can be adopted as a new drop-in fine-tuning optimizer.\n\n2. Empirical validation of the FTP algorithm's robustness on OOD datasets. They tested the algorithm across four vision tasks with five different pre-trained models, demonstrating robustness especially in scenarios with domain shifts and natural corruptions. The FTP algorithm also achieves state-of-the-art performance on a continual learning benchmark.\n\n3. Theoretical explanation of FTP's robustness maintenance capabilities. The authors provide a mathematical perspective that explains why FTP is effective at preserving the robustness of pre-trained models. They explore this through the lens of Lipschitz continuity, taking into account both the feature space and weight space of a model.\n", "questions": "What is the memory footprint and runtime overhead of the proposed method?\n\nHow does the proposed method affect generalization of neural networks?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The introduction of the Fast Trainable Projection (FTP) algorithm is an interesting contribution. It addresses the real-world limitations of scalability and efficiency associated with TPGM, making it a practical solution for various tasks.\n\nThe robustness of the FTP algorithm has been validated with experiments across four vision tasks and five pre-trained models. Superior results on OOD datasets and state-of-the-art performance on a continual learning benchmark provide empirical support for the authors' claims.", "weaknesses": "1. **Limited Scope:** the proposed method focuses on a specific Trainable projected gradient method (TPGM) and aims to improve its efficiency. However, the extent of its practical application could be questioned as it's not entirely clear how well it would generalize to other types of models or algorithms. Its significance might be seen as limited if it only optimizes a specific kind of model, especially when there are many alternative and possibly more efficient methods available.\n\n2. **Lack of Efficiency:** Despite the fact that the FTP improves on the TPGM's efficiency, it remains slower than many other methods, which raises questions about its practicality in real-world applications where computational resources and processing times are often crucial factors.\n\n3. **Theoretical Contributions:** The theoretical contribution of this paper may be seen as somewhat trivial, as the authors basically just formalize an intuitive understanding of Lipschitz continuity in relation to model robustness. The assumption that robustness equates to Lipschitz continuity may oversimplify the complex nature of robustness in deep learning models. Real-world robustness often depends on various other factors such as data variance, architecture, and loss landscape, which aren't addressed in this paper. Furthermore, they fail to delve deeper into the specifics of how the generalization relates to pre-trained weights, which might have offered more novel insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback! \n\n* **Regarding limited scope**\n\nFlexibility and extended applicability are advantages of the proposed method FTP because FTP is a model-agnostic optimizer.  Specifically, FTP is a generic projection-based optimization technique that can be integrated into existing optimizers such as Adam and SGD to formulate new generic optimizers such as AdamP and SGDP (Sec.5 and Appendix 7.7). In this regard, theoretically, FTP can be applied to any fine-tuning problems in deep learning regardless of the underlying model architecture. In our experiments, we have successfully applied it to the below models. **Note that we have added an entirely new task/model result for masked language learning, demonstrating the significant flexibility of our method.**\n* Different deep-learning models\n  * CNNs (Tab.1, 2, 6 )\n  * Transformers (Tab.3, 4, 5,  7, 8)\n* Different Tasks\n  * Classification (Tab.1, 2, 6)\n  * Dense classification \n    * Segmentation (Tab. 4)\n    * human parts segmentation (Tab.7)\n  * Dense regression \n    * Surface normal estimation (Tab.8)\n   * Masked language learning \n     * See discussion with Reviewer WZxc.\n* Different learning paradigms \n  * Supervised learning(Tab. 1, 2, 3, 4, 6, 7, 8)\n  * Continual learning (Tab.5)\n  * Unsupervised learning(See discussion with Reviewer WZxc)\n\n[1] Sanh, Victor, et al. \"DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.\" (2019).\n\n[2] https://huggingface.co/learn/nlp-course/chapter7/3?fw=pt \n\n* **Regarding efficiency** \n\nAs mentioned by Reviewer Fz4t, the benefits of FTP are many-fold including better OOD generalization and lack of need of held-out validation data. Further, in terms of efficiency as mentioned computation tends to be smaller than the large-scale pre-training, so some slowdown can be tolerated. More specifically, FTP’s efficiency can be optimized to suit different practical fine-tuning applications that may vary what is finetuned. To see this, we need to understand that FTP is an optimizer that can be applied to any gradient descent-based fine-tuning problems. Just like we can fine-tune an entire model using SGD or just fine-tune the bias terms for faster speed (Partial Fusion in Tab.1 and Tab.2), FTP can be used to just fine-tune the bias terms as well, which will lead to much better efficiency. For example, in the continual learning experiments (Sec.4.3), FTP is used only to fine-tune the QKV components in a Transformer model. To further demonstrate the point, for the rebuttal here we show the efficiency of applying FTP to different parts of a model.  Specifically, we adopt the same setting as in the segmentation experiment (Tab.4), which uses a Swin-Transfomer as the backbone.  Here we only profile the time used by the optimization process, excluding data loading and the model forward pass, which are not part of FTP,  to more directly demonstrate the efficiency of FTP. \n\n| s/it | Full Model | Bias Only | QKV Only | Decoder Only |\n|------|------------|-----------|----------|--------------|\n| Adam | 0.204      | 0.187     | 0.169    | 0.168        |\n| FTP  | 0.314      | 0.280     | 0.188    | 0.186        |\n\nThere are two important observations. 1) on average FTP is only 30% slower than the vanilla optimizer, which is not a major bottleneck in most cases. 2) as the number of tuning parameters decreases, the speed difference between FTP and the vanilla optimizer further diminishes (only 10% slower when only tuning the decoder). **This means that in extreme cases where computation resources and processing times are critical, FTP is virtually just as fast as vanilla optimizers for example when only finetuning a single last linear layer as in linear probing.** \n\n* **Regarding Theory**\n\n Lipschitz continuity is a standard and popular mathematical measurement of robustness in deep learning literature [1,2]. While it does not encompass all the aspects of deep learning robustness such as data variance, architectures, etc., it is a valuable mathematical tool to start analyzing the robustness of deep learning models. The generalization capability of a pre-trained model in its relationship to the pre-trained weights is explored in the prior work [3] through linear systems.  Further, we believe our extremely thorough empirical results of a practical algorithm bolster the practicality of the analysis, despite limitations.  In short, the better the pre-training datasets cover the downstream datasets, the more robust the fine-tuned model will be.\n\n[1] Weng, Tsui-Wei, et al. \"Evaluating the robustness of neural networks: An extreme value theory approach.\" (2018).\n\n[2] Zhang, Bohang, et al. \"Rethinking Lipschitz neural networks and certified robustness: A boolean function perspective.\"Neurips (2022): 19398-19413.\n\n[3] Tian, Junjiao, et al. \"Trainable Projected Gradient Method for Robust Fine-tuning.\" CVPR. 2023.\n\n* **Regarding memory**\n\nThe only major memory requirement is caching the previous gradient. This is rather a common requirement. For example, the Adam optimizer keeps running copies of aggregated first and second-moment information from previous iterations.\nThe memory consumption consumed by storing the projection constraints is negligible since they are scalars, the size of which are orders of magnitude smaller than the size of gradients.\n\n* **Regarding Generalization**\n\nFTP has a positive effect on the generalization of neural networks as seen in our OOD generalization experiments in Tab.1 and Tab.2, where FTP achieves the best performance. The goal of the proposed optimizer is to improve the generalization ability of the fine-tuned model by maintaining the knowledge acquired during pre-training. In this regard, the effects on generalization should scale positively with the generalization ability of the pre-trained model. In other words, if we use a stronger pre-trained model, the positive effect on generalization from FTP will be even bigger. \n\n\n"}, {"review_id": "X3wcAme83C", "reviewer": "Reviewer_2kDG", "summary": "In this paper, the authors propose an efficient fine-tuning method for deep learning models that learns the constraint for each layer more efficiently.\nDuring neural network training, the proposed method optimizes the layer-wise constraint using the last batch of data, unlike a prior method (TPGM) that updates layer-wise constraints on a \"validation\" dataset. This new method is less computationally burdensome since the gradient of the constraint can be computed using the chain rule, which reuses the gradient stored for the last batch.\nThe authors also propose a notation to describe fine-tuning robustness and provide theoretical justification for the proposed method under this notation. To demonstrate the effectiveness of the new method, the authors conduct a comprehensive list of experiments and show that the proposed method is not only more efficient but also has better performance.", "questions": "What’s the pattern of learned constraint for each layer, with regard to training epochs? Do they converge during the training?\n\nI am really curious about the statement of discrepancies here. The author mentioned multiple times that the discrepancy between the validation data and training data or between the different batches of data enables the learning of meaningful projection constraints but without further elaborations. Typically, the training and validation data should be subject to the same distribution.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-The writing in this paper is of high quality, as the authors have presented the methodology in a clear manner.\n\n-The related work section is well-discussed. The authors have categorized the existing fine-tuning studies into three categories: when, where, and how much to fine-tune. This is an insightful way to approach the current studies.\n\n-The authors have provided theoretical justification for the proposed method.\n\n-The experimental results are comprehensive, as the authors have included a list of different datasets and model architectures. The proposed method has been shown to outperform all baseline methods.", "weaknesses": "-My first concern is that the proposed method's novelty might be overshadowed due to its similarity to a previous method called TPGM. The major difference between the two seems to be the selection of batch data for updating the constraint parameter. The previous work used validation data, which may not be a common choice in the supervised learning setting, while this work uses the previous batch data. Although using the stored gradient indeed improves computational efficiency, the authors did not clearly state why this method has better performance than TPGM. The validation data and the other training data should have the same distribution.\n\n-The proposed difference function's meaning requires in-depth justification. While (Lipschitz) smoothness is usually associated with better performance and (adversarial) robustness for deep neural networks, it is not directly implied that the proposed difference function will improve the downstream performance of a fine-tuned model. Additionally, this notation only describes the difference in feature space; how does it describe the linear-probing fine-tuning method, which only changes the linear last layer? Can it be shown that this proposed notation indicates fine-tuning downstream performance?\n\n-The robustness notation here seems to be related to the OOD generalization capability. Are they the same thing here? Usually, the robustness is used to describe the model’s performance under some unwanted perturbation (adversarial samples or corruption)\n\n-The Low-rank fine-tuning methods have been really popular recently. Is it possible to compare the proposed method with them?\n\nMinor typo:\n\n Line 52, Liptschitz → Lipschitz\n\nLine 127, “Then FTP calculates the gradients”, is here FTP meant to be TPGM?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback! \n\n* **Regarding Novelty**\n\nThe novelty of our method lies in its algorithmic improvement to the prior work TPGM in terms of efficiency, flexibility, and robustness. To successfully use the previous information to update internal parameters in FTP requires rigorous analysis. It requires careful caching of the correct model components from previous steps. For example, in Eq.4, we store previous “unconstrained” update $\\tilde{W}_{t-1}$ instead of naively the previous update $W_{t-1}$ for mathematical correctness that’s not immediately clear without analysis. While the end result is a simple algorithm (a strength), our contribution is to come up with this method and rigorously evaluate it, showing significant benefits as other reviewers have mentioned. \n\nThe differences in performance can be partially attributed to the algorithmic differences between the two and also see discussion in Appendix 7.3, but we agree that future work could have potential to understanding and designing even better methods. \nIn addition to this, the improvement of FTP is also in its efficiency and flexibility.  For example, we demonstrated the regularization strength of FTP in continual learning experiments (sec.4.3). This is not possible with TPGM due to the lack of validation data under a continual learning setting. Hence such differences have practical import. \n\n* **Regarding the difference function**\n\n Optimizing the difference function improves the robustness of the fine-tuned model. To understand the logic, we need to combine both Lemma 1 and Lemma 2. \n\n* Lemma 1 says that if we can minimize the right-hand side of the difference function, we can maximally maintain the robustness of the pre-trained model. \n* Lemma 2 says that minimizing the difference function, in fact, leads to a projection operation in the weight space. Therefore, the conclusion can be drawn that projection can lead to better fine-tuning robustness. \n\nThe main contribution of the difference function is that by virtue of Lemma 2, it establishes an equivalence between feature space and weight space, i.e., projection in weight space is equivalent to minimizing the difference function in feature space. This applies to any weight layer including the last linear layer in a neural network. \n\n* **Regarding low-rank methods**.\n\nThis is a great point, and for this rebuttal, we add experiments to compare to low-rank finetuning methods, Polyhistor [1] and LORA [2], using  Adapter and low-rank factorization.  Specifically, we compare them on the PASCAL dense vision benchmarks in Sec.4.2.  In the following table, we directly compare them to the reported results from Polyhistor since we used its open-sourced code to benchmark FTP in our paper. \n\n|            | Segmentation $\\uparrow$ | Human Parts $\\uparrow$ | Surface Normal $\\downarrow$|\n|------------|--------------|-------------|----------------|\n| Vanilla    | 66.03        | 62.21       | 18.98          |\n| LORA       | 70.12        | 57.73       | 18.96          |\n| Polyhistor | 70.87        | 59.54       | 17.47          |\n| FTP        | **73.79**        | **65.50**       | **15.51**          |\n\nWe observe that on all fine-tuning tasks, FTP achieves better performance than LORA and Polyhistor. \n\n[1] Liu, Yen-Cheng, et al. \"Polyhistor: Parameter-efficient multi-task adaptation for dense vision tasks.\" Neurips (2022)\n\n[2] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\"  (2021).\n\n* **Regarding FTP learned pattern**\n\nPlease see the attached PDF for a visualization. \nThe learned constraints grow slowly from a small value, e.g., 1e-8, at different paces for each layer and eventually converge. The pattern is highly similar to that of TPGM as reported in its paper [1]. We observed that layers closer to the output tend to have larger constraints (less constrained and more changes allowed) whereas the layers closer to the input tend to have smaller constraints (more constrained). This is similar to our intuition that early layers learn more general information and later layers learn task-specific information. The constraints do converge because they are calculated based on the current model gradient g_t which decreases to zero as the learning rate drops (Eq.4). \nTo qualitatively show this, we record the history of FTP constraints for the classification experiment in Appendix Tab.6. In this setting, we fine-tune a pre-trained ResNet50 on DomainNet real images. We visualize the learned constraints for each layer through time in the attached pdf file. There are two observations. 1) Early layers (dark colors) generally have smaller constraints than the latter layers (light colors) throughout training. 2) Constraints grow from small to large and converge in the end. \n\n[1] Tian, Junjiao, et al. \"Trainable Projected Gradient Method for Robust Fine-tuning.\" CVPR 2023.\n\n* **Regarding \"discrepancies\"**\n\nThis is a great question. Here the discrepancies refer to the variances between sampled batches. Even though training data and validation data are sampled from the same distribution, each mini-batch can lead to very different directions of updates on the current model. Intuitively, we utilize these variances to “control” how much the model should deviate from the pre-trained weights. If two batches disagree more on where the gradients should go in a certain direction, then the projection will be stronger towards the pre-trained model. \nMathematically, this is reflected by the dot product in the updated equation 4 between \n$\\mathbf{g}_t^{i,\\intercal}$  \n\nand $-({\\tilde{\\mathbf{w}}_{t-1}^i}-\\mathbf{w}_0^i)$ where the first quantity represents the gradients of the current batch and the second quantity represents the gradients of the previous batch. If the two batches disagree, then the product will be positive, which leads to smaller projection constraints (i.e., the gradient $\\nabla\\gamma_t$ is positive), i.e., stronger projection.  \n"}, {"review_id": "R1cuTHzopo", "reviewer": "Reviewer_wzxc", "summary": "This paper introduces Fast Trainable Projection (FTP), a new algorithm aimed at improving the robustness and efficiency of fine-tuning pre-trained models. FTP achieves visible improvement in terms of computational speed and adaptability, demonstrated across various vision tasks and models, contributing to an average 35% speedup on benchmarks compared to prior works. The authors also provide a theoretical explanation for FTP's ability to maintain the robustness of pre-trained models through the lens of Lipschitz continuity.", "questions": "1. Does the proposed algorithm work on low-rank adaptation (LoRA)? Would be interesting to see related experiments for that.", "limitations": "The authors have addressed the limitations", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. papers are well-written, figures are well-made and very easy to read\n2. problem being studied is important -- the efficient finetuning of models are particularly relevant given the popularity of foundation models today.", "weaknesses": "1. Limited to Vision Tasks: The experiments are largely focused on vision tasks. Additional experiments in other application areas such as natural language processing could further demonstrate the algorithm's effectiveness.\n2. Does the method work for foundation-level models? For example, in vision, there's stable diffusion, segment anything etc. There are already very mature finetuning pipeline and benchmarks for these models that can be experimented with. If the proposed method works for these models, the impact will be significantly increased.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive and constructive feedback! \n\n* **Regarding limited-to-vision tasks**\n\nTheoretically, our method applies to any fine-tuning problems with pre-trained weights. To further showcase the adaptability of our method, we conducted a quick experiment on NLP for the rebuttal. Specifically, we use the DistilBert model [1], pre-trained on Wikipedia and Bookcorpus, and fine-tune it to 1% of the IMDB dataset. We finetune the DistilBert model in an unsupervised masked fashion and measure its performance on the fine-tuned IMDB dataset and the original Wikipedia dataset using perplexity (lower the better). We use the implementation open-sourced by Hugginface [2] for this experiment (15% token masking rate, 50 epochs, and learning rate 5-e5). \n\n|         | IMDB $\\downarrow$ | Wikipedia  $\\downarrow$ |\n|---------|-------|-----------|\n| Vanilla | 11.05 | 9.94      |\n| FTP     | **10.88** | **9.69**     |\n\nWe use 1% data to simulate the situation of overfitting due to a small amount of data. Compared to vanilla fine-tuning, FTP can better avoid overfitting (lower perplexity on IMDB) and maintain more information from the original pre-training dataset (lower perplexity on Wikipedia). These observations corroborate our findings in other experiments. \n\n[1] Sanh, Victor, et al. \"DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.\" arXiv preprint arXiv:1910.01108 (2019).\n\n[2] https://huggingface.co/learn/nlp-course/chapter7/3?fw=pt \n\n* **Regarding other foundation models**\n\nTheoretically yes. FTP is a generic projection-based optimization technique that can be integrated into existing optimizers to formulate new generic optimizers such as SGDP (sec.5). In this regard, theoretically, FTP can be applied to any fine-tuning problems (with pre-trained weights) in deep learning regardless of the underlying model architecture. In our experiments, we have successfully applied it to different deep learning models (CNNs and Transformers), tasks (classification and dense vision tasks), and even different learning paradigms (supervised, unsupervised and continual learning). While the rebuttal did not leave sufficient time to try additional foundation models, we agree this is a great area of future work to maximize impact, and we plan on doing so. \n\n* **Regarding LoRA.**\n\nNo, the method does not apply to LoRA because LoRA adds additional weights that are not pre-trained and randomly initialized. Our method relies on the projection toward pre-trained weights. If a method introduces new model parameters, then our method does not apply. \n\nNevertheless, we add experiments to compare to low-rank finetuning methods (as suggested by Reviewer 2kDG), Polyhistor [1], and LORA [2], using Adapter and low-rank factorization.  Specifically, we compare them on the PASCAL dense vision benchmarks in Sec.4.2.  In the following table, we directly compare them to the reported results from Polyhistr since we used its open-sourced code to benchmark FTP in our paper. \n\n|            | Segmentation $\\uparrow$ | Human Parts $\\uparrow$ | Surface Normal $\\downarrow$|\n|------------|--------------|-------------|----------------|\n| Vanilla    | 66.03        | 62.21       | 18.98          |\n| LORA       | 70.12        | 57.73       | 18.96          |\n| Polyhistor | 70.87        | 59.54       | 17.47          |\n| FTP        | **73.79**        | **65.50**       | **15.51**          |\n\nWe observe that on all fine-tuning tasks, FTP achieves better performance than LORA and Polyhistor.\n\n[1] Liu, Yen-Cheng, et al. \"Polyhistor: Parameter-efficient multi-task adaptation for dense vision tasks.\" Advances in Neural Information Processing Systems 35 (2022): 36889-36901.\n\n[2] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" arXiv preprint arXiv:2106.09685 (2021).\n\n\n\n\n"}, {"review_id": "3X7LL6leyi", "reviewer": "Reviewer_yjP3", "summary": "This paper proposes a robust fine-tuning technique that has the decent scalability and efficiency to achieve higher ID and OOD performance when transferring a pre-trained model to downstream tasks. A new projection-based fine-tuning algorithm, Fast Trainable Projection (FTP) is proposed for computationally efficient learning of per-layer projection constraints with the theoretically analyses in the lens of Lipschitz continuity. Extensive experiments show its effectiveness.", "questions": "I understand FTP is more efficient than TPGM. But why FTP achieves better performance? Could the author give an intuitive explanation?\n\nFor the continual learning experiment, the comparison is somewhat not fair, since the proposed method needs to know the task id. (Page 9 line 288 we re-initialize FTP after each task and use the current model as the “pre-trained model” for the next task.)\n", "limitations": "Well addressed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to read.\n\nThe proposed FTP is reasonable and easy to implement with the provided pseudo code. \n\nThe theoretical analysis is also appreciated.\n\nExtensive experiments on several downstream tasks and datasets are conducted to show the strong performances of the proposed method\n", "weaknesses": "While the method is simple with strong performances, the novelty is somewhat limited. It seems like a direct extension of TPGM. Using the previous model and cached gradients to update the constraints is trivial to me and makes limited technical contributions. Gradient annealing is also like a trick.\n\nCan the theoretical analyses prove that the proposed FTP is more powerful than TPGM? Since the experimental results show the superior performance of FTP.\n\nThe ablation study is insufficient. The ablation should be done on ProjUpdate and Gradient Annealing to show the effectiveness of each component.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive and constructive feedback!\n\n* **Regarding novelty**\n\nThe novelty of our method lies in its algorithmic improvement to the prior work TPGM in terms of efficiency, flexibility, and robustness. To successfully use the previous information to update internal parameters in FTP requires rigorous analysis. It requires careful caching of the correct model components from previous steps. For example, in Eq.4, we store previous “unconstrained” update $\\tilde{W_{t-1}}$ instead of naively the previous update $W_{t-1}$ for mathematical correctness that’s not immediately clear without analysis. While the end result is a simple algorithm (a strength), our contribution is to come up with this method and rigorously evaluate it, showing significant benefits as other reviewers have mentioned.  \n\nThe differences in performance can be partially attributed to the algorithmic differences between the two and also see discussion in Appendix 7.3, but we agree that future work could have potential to understanding and designing even better methods. \nIn addition to this, the improvement of FTP is also in its efficiency and flexibility.  For example, we demonstrated the regularization strength of FTP in continual learning experiments (sec.4.3). This is not possible with TPGM due to the lack of validation data under a continual learning setting. Hence such differences have practical import.\nGradient annealing is the most direct way to adjust the regularization strength of FTP much like hyper-parameters in other works. \n\n* **Regarding the theory**\n\n No, the theory does not indicate that FTP is better than TPGM. The improvement of FTP over TPGM is algorithmic in terms of efficiency and applicability, not theoretical. \nNevertheless, section 3.3 introduces a general theory on why projection is useful for fine-tuning, a theoretical question that hasn’t been answered in prior works. While you are right that it generally applies to all projection-based methods including FTP and TPGM, we believe it was important to include to progress this area empirically and theoretically. We will add text and organization to explicitly discuss this. \n\n* **Regarding ablation study**\n\nThe ProjUpdate component introduces the core update equation in the FTP optimization algorithm. It serves as the mathematical foundation for FTP and cannot be easily replaced by other alternatives without breaking the core mechanism. Therefore, it is not clear to us how to ablate this component immediately. Nevertheless, We are more than happy to hear further discussion from the reviewer.\n\nOn the choice of the Gradient annealing hyper-parameters,  we treat it as a hyper-parameter that regularizes the strength of projection. In our experiments, we performed a hyper-parameter search for different annealing values and use the best one on validation data. Please refer to Appendix 7.4 and 7.5 to see the exact value used for each experiment. \n\n* **Regarding intuition on why FTP is better than TPGM.**\n\nFTP improves TPGM algorithmically in terms of efficiency, in terms of flexibility by removing limiting factors such as the need for validation data and nested update loops, and in some cases in terms of OOD robustness.  The differences in performance fluctuations can be partially attributed to the algorithmic differences between the two and also see discussion in Appendix 7.3, as well as simplified tuning, but we agree that future work could have the potential to understand and design even better methods.\n\n* **Regarding the continual learning experiments**\n\nOur comparison is absolutely fair because our method does not require knowledge of the task id during inference. It is privileged only to the knowledge of task boundary, which is also required in LwF.MC [45], L2P [46], DualPrompt [47], CODA-P [44], EWC [48], and L2 [44]. Specifically,\n* LwF changes the reference regularization model at the task boundary and treats new and old logit heads differently.\n* DualPrompt and Coda-p initialize new model parameters at the task boundary\n* L2P/DualPrompt/Coda-P treat old and new logits differently\n* EWC and L2 use task boundaries to determine the regularization parameters. EWC actually does significant regularization calculations at each task boundary.\n\nIn other words, we do not use any additional information beyond what other compared methods do. \n"}]

Author rebuttal:
We thank the reviewers for all the constructive feedback and positive comments on the real-world relevance (wzxc), strong performance (Fz4t), and extensive experiments (HDd6, 2kDG, yjp3). We have provided a detailed discussion for each of your questions below. 
For this rebuttal, we clarified some misunderstandings and introduced new experiments to address specific questions.

To recap, we proposed FTP to improve the generalization and robustness of fine-tuned models. FTP is 
* An optimization technique that can be integrated into existing optimizers. 
* **Easy to use** in a plug-and-play fashion.
* **Broadly applicable** to most fine-tuning problems regardless of the underlying tasks and models. 
* **More efficient** than prior works, achieving ~2x speedup. 

We have demonstrated its effectiveness across different deep learning models (CNNs and Transformers), tasks (classification and dense vision tasks), and even different learning paradigms (supervised, unsupervised and continual learning).



New experiments: 

* Masked Langage modeling experiments (suggested by Reviewer WZxc)
* The efficiency of FTP when fine-tuning to different parts of a model (inspired by Reviewer HDd6)
* Comparison to low-rank fine-tuning methods e.g., LoRA (suggested by Reviewer 2kDG)
* Visualization of FTP’s learned per-layer constraints with respect to training time (suggested by Reviewer 2kDG)

All these experiments are included in the attached PDF file. 

Meta-review: All of the reviewers are positive about the paper, and the rebuttal helped clarify key points. The reviewers noted the strong empirical results.
Decision: Accept (poster)