SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junjiao Tian
jtian73@gatech.edu
Yen-Cheng Liu
ycliu@gatech.edu
James Seale Smith
jamessealesmith@gatech.edu
zkira@gatech.edu
Robust fine-tuning aims to achieve competitive in-distribution (ID) performance while maintaining the out-of-distribution (OOD) robustness of a pre-trained model when transferring it to a downstream task. Recently, projected gradient descent has been successfully used in robust fine-tuning by constraining the deviation from the initialization of the fine-tuned model explicitly through projection. However, algorithmically, two limitations prevent this method from being adopted more widely, scalability and efficiency. In this paper, we propose a new projection-based fine-tuning algorithm, Fast Trainable Projection (FTP) for computationally efficient learning of per-layer projection constraints, resulting in an average  $35\%$  speedup on our benchmarks compared to prior works. FTP can be combined with existing optimizers such as AdamW, and be used in a plug-and-play fashion. Finally, we show that FTP is a special instance of hyper-optimizers that tune the hyper-parameters of optimizers in a learnable manner through nested differentiation. Empirically, we show superior robustness on OOD datasets, including domain shifts and natural corruptions, across four different vision tasks with five different pre-trained models. Additionally, we demonstrate that FTP is broadly applicable and beneficial to other learning scenarios such as low-label and continual learning settings thanks to its easy adaptability. The code will be available at https://github.com/GT-RIPL/FTP.git.
With new progress being made in pre-training of foundation models every year, such as self-supervised [1, 2, 3] or language-supervised training [4], their potential has gone far beyond merely speeding up convergence [5]. They have demonstrated superior transferability to other tasks, reducing the need for data and improving robustness and generalization capabilities [6, 7, 8]. The problem of how to fine-tune (transfer) a foundation model such that we maintain its robustness and generalization capabilities acquired during pre-training on large datasets has therefore become an essential research topic. This problem is hard because the conventional machine learning paradigm of validating on held-out training data does not impose any constraints on robustness and generalization w.r.t. the foundation models. For example, fine-tuning with a slightly large learning rate can easily destroy capabilities that reside in the foundation models [8], while performing well on the target task.
To maintain the robustness and generalization capability of the pre-trained model when fine-tuning, recent projection-based methods explicitly constrain the distance between the fine-tuned and the pre-trained models through projection. For example, MARS-SP [9] specifies a distance constraint shared by all layers in a neural network. However, it is practically intractable to tune a constraint for each layer (poor scalability). TPGM [10] proposes to automatically learn different constraints
(a) One Step FTP Diagram
(b) Image Classification
Figure 1: (a): FTP updates the model using (unconstrained) gradient descent (UGD) to calculate  $\tilde{\mathbf{W}}_t$ , then updates the projection constraint  $\gamma_t$  (ProjUpdate), and finally projects  $\tilde{\mathbf{W}}_t$  to  $\mathbf{W}_t$  (Projection), all in a single forward pass. (b),(c): Visualizations of in-distribution (Real/Clean, labeled as ID), out-of-distribution (Sketch/Fog, etc.) accuracy and computation time (iterations/sec) as a percentage of vanilla fine-tuning (FT) for classification on DomainNet (Tab. 2) and semantic segmentation on PASCAL-Context (Tab. 4) respectively. FTP improves the OOD robustness of FT and is much more computationally efficient than prior work TPGM.
(c) Semantic Segmentation
for each layer, solving the issue of scalability in MARS-SP, however, with increased computational overhead (poor efficiency). These limitations prevent the method from being adopted more widely.
To achieve scalability and efficiency simultaneously, we propose Fast Trainable Projection (FTP), for learning both the projection constraints and the main model in a single forward pass (Fig. 1a), significantly reducing computation overhead in prior works while achieving competitive performance. Specifically, FTP removes the algorithmic redundancy of extra training procedures required in TPGM [10], which requires sampling a separate batch of data and running a nested training loop. FTP achieves this by 1) utilizing different batches of training data sampled at consecutive steps and 2) re-using gradients calculated for the main model update (Sec. 3.2). This leads to a  $35\%$  speedup with comparable performance in fine-tuning (Fig. 1b, 1c). The efficiency improvement and easy adaptability as a drop-in replacement with existing optimizers are essential to making projection-based methods applicable to more fine-tuning problems. For example, we implement SGDP, an SGD variant with built-in FTP. SGDP can be used as a drop-in replacement for SGD (details in Appendix 8.7) as:
optimiser  $=$  SGDP(param_group,\*\*optimizer_parameters）#See Appendix 8.7.
To demonstrate this, we test FTP on four different vision tasks, image classification, semantic segmentation, human parts segmentation, and surface normal estimation. FTP shows superior OOD performance under domain shift or natural corruptions on all benchmarks. Moreover, we apply FTP to a continual learning (CL) benchmark and achieve state of the art performance when combined with a simple CL technique.
Finally, we show that FTP is a special instance of hyper-optimizers [11, 12, 13, 14, 15, 16, 17, 18] that aims to reduce the manual tuning of optimization hyper-parameters such as learning rate by learning them automatically through automatic differentiation and nested optimization. Theoretically, to understand why FTP and other projection methods can maintain the robustness of the pre-trained models, we propose to establish a theoretical connection between robustness and projection through the lens of Lipschitz continuity, a widely adopted measure of robustness [19, 20, 21]. In summary, our contributions are:
- We present a new fine-tuning algorithm, Fast Trainable Projection, to efficiently learn the projection constraints and fine-tune the model simultaneously, bringing significantly improved computation efficiency w.r.t. prior works [10] in Sec. 3.2.
- We show that FTP is a special instance of hyper-optimizers that aims to reduce manual tuning of hyper-parameters through nested optimization in Sec. 3.3.
- We discuss a dual perspective of the fine-tuning robustness in the feature space and the weight space of a model to mathematically understand why projection can maintain the robustness of the pre-trained models in Sec. 3.4.
- We show superior robustness on OOD datasets on four vision tasks with five pre-trained models and SOTA performance on a continual learning benchmark, all with a  $35\%$  speedup in Sec. 4.
3.1 Review: Enforcing Projection and Learning Constraints
In this work, we focus on fine-tuning a pre-trained model, where  $\mathbf{W}_0 \in \mathbb{R}^{n \times m}$  is the weights of a linear layer in the pre-trained model, to a downstream task. We denote  $\mathbf{W}_t$  as the fine-tuned model at training iteration  $t$  and  $\mathbf{w}^i$  as the  $i$ th row of a matrix  $\mathbf{W} \in \mathbb{R}^{n \times m}$ . Several prior works [9, 10] have attempted to use projection to improve fine-tuning robustness. The most vanilla formulation in MARS-SP [9] has two steps: unconstrained gradient descent and projection.
Unconstrained Gradient Descent (Abbrev. UGD), the projection-based methods first compute the updated model weights  $\tilde{\mathbf{W}}_t$  without projection. For example, at iteration  $t$ , given a batch of training data  $\mathcal{D}_t^{tr}$ , we first obtain  $\tilde{\mathbf{W}}_t$  as the following,
$$
\mathbf {g} _ {t} \leftarrow \nabla \mathcal {L} _ {\mathcal {D} _ {t} ^ {t r}} \left(\mathbf {W} _ {t - 1}\right) \in \mathbb {R} ^ {n \times m}, \quad \tilde {\mathbf {W}} _ {t} \leftarrow \mathbf {O p t} \left(\mathbf {W} _ {t - 1}, \mathbf {g} _ {t}\right). \tag {1}
$$
where  $\mathbf{g}_t$  is the derivative of the loss function  $\mathcal{L}_{\mathcal{D}^t r}(\mathbf{W}_{t-1})$  calculated on  $\mathcal{D}_t^{tr}$  w.r.t.  $\mathbf{W}_{t-1}$  and  $\mathbf{Opt}(\cdot)$  is an existing optimization algorithm, such as SGD, AdamW [37].
Projection. MARS-SP [21] projects the updated model  $\tilde{\mathbf{W}}_t$  towards its initialization  $\mathbf{W}_0$  with a pre-defined projection constraint  $\gamma$  for all layers using the MARS matrix norm (see Appendix . 8.2) as shown below in Eq. 2.
$$
\mathbf {W} _ {t} = \Pi \left(\tilde {\mathbf {W}} _ {t}, \mathbf {W} _ {0}, \gamma_ {t}\right) = \left[ \begin{array}{c} \mathbf {w} _ {t} ^ {i \top} \\ \vdots \\ \mathbf {w} _ {t} ^ {i \top} \end{array} \right] = \left[ \begin{array}{c} \left(\frac {\gamma}{\| \tilde {\mathbf {w}} _ {t} ^ {i} - \mathbf {w} _ {0} ^ {i} \| _ {1}} \left(\tilde {\mathbf {w}} _ {t} ^ {i} - \mathbf {w} _ {0} ^ {i}\right) + \mathbf {w} _ {0} ^ {i}\right) ^ {\top} \\ \vdots \\ \left(\frac {\gamma}{\| \tilde {\mathbf {w}} _ {t} ^ {n} - \mathbf {w} _ {0} ^ {n} \| _ {1}} \left(\tilde {\mathbf {w}} _ {t} ^ {n} - \mathbf {w} _ {0} ^ {n}\right) + \mathbf {w} _ {0} ^ {n}\right) ^ {\top} \end{array} \right] \tag {2}
$$
However, MARS-SP has poor scalability because it is practically intractable to hand-tune different constraints for each layer, which results in sub-optimal performance as reported by TPGM [10]. Instead of a pre-defined  $\gamma$  for all layers, TPGM proposes to learn a different constraint  $\gamma_{t}$  for each layer<sup>1</sup> and updates them iteratively during training. This enables TPGM to customize a different regularization strength for each layer and to have superior performance for both ID and OOD data.
ProjUpdate. Given as input the frozen unconstrained model  $\tilde{\mathbf{W}}_t$  from UGD (Eq. 1), TPGM adds an intermediate ProjUpdate function before projection, which samples a separate set of data from the validation dataset  $\mathcal{D}_t^{val}$  and uses a standalone training loop to update the projection constraints  $\gamma_t$  while keeping the model  $\tilde{\mathbf{W}}_t$  frozen. Specifically, ProjUpdate creates a temporary projected model  $\mathbf{W}_p$  by projecting  $\tilde{\mathbf{W}}_t$  towards  $\mathbf{W}_0$  based on the previous constraint  $\gamma_{t-1}$  using Eq. 2, i.e.,  $\mathbf{W}_p = \Pi(\tilde{\mathbf{W}}_t, \mathbf{W}_0, \gamma_{t-1})$ . Therefore,  $\mathbf{W}_p(\gamma_{t-1})^2$  can be viewed as a function of  $\gamma_{t-1}$ . Then FTP calculates the gradient  $\nabla \gamma_t$  by taking a derivative of the loss function  $\mathcal{L}_{\mathcal{D}_t^{val}}(\mathbf{W}_p(\gamma_{t-1}))$  w.r.t.  $\gamma_{t-1}$ :
$$
\nabla \gamma_ {t} \leftarrow \nabla \mathcal {L} _ {\mathcal {D} _ {t} ^ {v a l}} \left(\mathbf {W} _ {p} \left(\gamma_ {t - 1}\right)\right), \quad \gamma_ {t} = \mathbf {O p t} \left(\gamma_ {t - 1}, \nabla \gamma_ {t}\right) \tag {3}
$$
$$
\mathrm {w h e r e} \quad \mathbf {W} _ {p} = [ \mathbf {w} _ {p} ^ {i}, \ldots , \mathbf {w} _ {p} ^ {n} ] ^ {\intercal} \quad \mathrm {a n d} \quad \mathbf {w} _ {p} ^ {i} = \frac {\gamma_ {t - 1}}{\| \tilde {\mathbf {w}} _ {t} ^ {i} - \mathbf {w} _ {0} ^ {i} \| _ {1}} (\tilde {\mathbf {w}} _ {t} ^ {i} - \mathbf {w} _ {0} ^ {i}) + \mathbf {w} _ {0} ^ {i}.
$$
With the calculated gradient, TPGM uses an existing optimizer  $\mathbf{Opt}(\cdot)$  to update  $\gamma_{t}$ . This procedure, sampling  $\mathcal{D}_t^{val}$  and calculating the derivative  $\nabla \mathcal{L}_{\mathcal{D}_t^{val}}(\mathbf{W}_p(\gamma_{t-1}))$ , is the key to learning projection constraints because the unconstrained model  $\tilde{\mathbf{W}}_t$  (highlighted above and calculated in Eq. 1), was updated on the training data  $\mathcal{D}_t^{tr}$  and  $\gamma_{t}$  is now updated on separate data  $\mathcal{D}_t^{val}$ . The discrepancy between  $\mathcal{D}_t^{tr}$  and  $\mathcal{D}_t^{val}$  allows TPGM to find a better projected model  $\mathbf{W}_p$  (projected between  $\tilde{\mathbf{W}}_t$  and  $\mathbf{W}_0$ ) by updating  $\gamma_{t}$ , which balances between fitting the training data  $\mathcal{D}_t^{tr}$  and generalizing to  $\mathcal{D}^{val}$ . Finally, with an updated  $\gamma_{t}$ , TPGM again projects  $\tilde{\mathbf{W}}_t$  towards  $\mathbf{W}_0$  to obtain the final model  $\mathbf{W}_t$  using Eq. 2, replacing the pre-defined  $\gamma$  with a learned  $\gamma_{t}$ . A flow chart of TPGM is in Fig. 2.
The algorithm demonstrated the capability to automatically learn different constraints for each layer, solving the scalability issue in MARS-SP. However, TPGM introduces extra computation in the additional training loop. In the next section, we propose a scalable and efficient projection algorithm that learns the projection constraints for each layer without separate validation data and loops.
3.2 FTP: Fast Trainable Projection
To inherit the scalability of TPGM while reducing the computational overhead, we propose Fast Trainable Projection (FTP) (Algorithm 1). Similar to TPGM, the algorithm has three components: UGD, ProjUpdate, Projection. The ProjUpdate component is the major contributor to efficient computation. It builds on a key insight: Instead of sampling separate data  $\mathcal{D}_t^{val}$  each time, we use two training data batches sampled independently at consecutive steps, e.g.,  $\mathcal{D}_{t - 1}^{tr}$  and  $\mathcal{D}_t^{tr}$ . Specifically, we use  $\mathcal{D}_t^{tr}$  to update  $\gamma_{t}$  instead of  $\mathcal{D}_t^{val}$ . As a result, the optimization of  $\gamma_{t}$  re-uses most of the computation used for the optimization of the main model.
ProjUpdate. Specifically, instead of taking a derivative of  $\mathcal{L}_{\mathcal{D}_t^{val}}(\mathbf{W}_p)$  w.r.t.  $\gamma_{t - 1}$  as in TPGM, FTP calculates the gradient of  $\gamma_{t - 1}$  by the derivative of the loss function on the current training data  $\mathcal{L}_{\mathcal{D}_t^{tr}}(\mathbf{W}_{t - 1})$  w.r.t.  $\gamma_{t - 1}$ . Note that  $\mathbf{W}_{t - 1} = \Pi (\tilde{\bar{\mathbf{W}}}_{t - 1},\mathbf{W}_0,\gamma_{t - 1})$  is also a function of the
<sup>1</sup>We omit the index for different layers to avoid notation clutter and the subscript  $t$  indicates training iterations.
2We use this functional form  $\mathbf{W}_p(\gamma_{t - 1})$  to highlight the dependency on  $\gamma_{t - 1}$
Algorithm 1 FTP: Fast Trainable Projection.
Require:  $\mathbf{W}_0$  the pre-trained model   
Require:  $\kappa$  positive gradient annealing rate   
Require:  $\mu \leftarrow 1e - 2,\beta_{1},\beta_{2}\leftarrow (0.9,0.999)$  fixed parameters for AdamUpdate for  $t = 1\dots T$  do  $\begin{array}{r}\left\{\mathbf{g}_t\gets \nabla \mathcal{L}_{\mathcal{D}^{tr}}(\mathbf{W}_{t - 1})\right.\\ \left.\tilde{\mathbf{W}}_t\gets \mathbf{Opt}(\mathbf{W}_{t - 1},\mathbf{g}_t)\right.\\ \end{array}$  Unconstrained Gradient Descent (Eq. 1) if  $t = 1$  then  $\gamma_{t} = 1e - 8$ $\triangleright$  Initialize  $\gamma$  else  $\begin{array}{rl} & {\left\{\nabla \gamma_t\leftarrow \sum_i\left(\mathbf{g}_t^{i,\top}(\tilde{\mathbf{w}}_{t - 1}^i -\mathbf{w}_0^i)\frac{1}{\|\tilde{\mathbf{w}}_{t - 1}^i - \mathbf{w}_0^i\|_1}\right)}\\ & {\left.\text{if}\nabla \gamma_t > 0:\nabla \gamma_t = \kappa \nabla \gamma_t\right.}\\ & {\left.\gamma_t\leftarrow \mathbf{AdamUpdate}(\gamma_{t - 1},\nabla \gamma_t,t)\right.}\\ & {\mathbf{W}_t = \Pi (\tilde{\mathbf{W}}_t,\mathbf{W}_0,\gamma_t)} \end{array}$  ProjUpdate (Eq. 4,Eq. 5, Alg. 2)
Figure 2: Computation Flow Chart of TPGM (top) and FTP (bottom) at iteration  $t$ . The main difference between TPGM and FTP is in the PorjUpdate step. FTP uses the previous model  $\mathbf{W}_{t-1}$  and cached gradients from  $\mathcal{L}_{\mathcal{D}_{tr}^t}(\mathbf{W}_{t-1})$  to update the projection constraints  $\gamma_t$ .
constraint  $\gamma_{t - 1}$  as a result of projection from the previous step. Hence, by virtue of the chain rule, the gradient of  $\mathcal{L}_{\mathcal{D}_t^{tr}}(\mathbf{W}_{t - 1}(\gamma_{t - 1}))$  w.r.t.  $\gamma_{t - 1}$  is,
$$
\nabla \gamma_ {t} = \sum_ {i = 1} ^ {n} \nabla \underbrace {\mathcal {L} _ {\mathcal {D} _ {t} ^ {t r}} \left(\mathbf {w} _ {t - 1} ^ {i} \left(\gamma_ {t - 1}\right)\right) ^ {\intercal}} _ {\mathbf {g} _ {t} ^ {i}} \frac {\partial \mathbf {w} _ {t - 1} ^ {i}}{\partial \gamma_ {t - 1}} = \sum_ {i = 1} ^ {n} \mathbf {g} _ {t} ^ {i, \intercal} \left(\tilde {\mathbf {w}} _ {t - 1} ^ {i} - \mathbf {w} _ {0} ^ {i}\right) \frac {1}{\left\| \tilde {\mathbf {w}} _ {t - 1} ^ {i} - \mathbf {w} _ {0} ^ {i} \right\| _ {1}} \tag {4}
$$
where the summation loops over each row in the matrix  $\mathbf{W}_{t-1}$  because the same constraint  $\gamma_{t-1}$  is enforced for all rows (see MARS norm in Appendix Eq. 15 and Eq. 2) so the final gradient is the summation of all gradients for each row. Similar to TPGM, because the starting point of projection  $\tilde{\mathbf{W}}_{t-1}$  (highlighted above) was updated using the previous training batch  $\mathcal{D}_{t-1}^{tr}$  and the gradient  $\nabla \gamma_t$  is calculated using the current batch  $\mathcal{D}_t^{tr}$ , the discrepancy between  $\mathcal{D}_{t-1}^{tr}$  and  $\mathcal{D}_t^{tr}$  enables FTP to learn meaningful projection constraints. Crucially, we proposed a novel formulation that allows for re-using the gradient  $\mathbf{g}_t$  used for calculating the unconstrained model  $\tilde{\mathbf{W}}_t$  in the UGD step (Eq. 1).
Gradient Annealing. Prior work [10] noticed that learning projection constraints for each layer can suffer from underfitting because the learned constraints can be too conservative, and used an additional regularization to help reduce this negative effect. For FTP, we introduce a simple technique that uses a single gradient annealing factor for all layers,  $\kappa \in [0,1]$  to modulate the magnitude of the positive gradient  $\nabla \gamma_{t} > 0$ , which contributes to the shrinkage of the constraints. When  $\nabla \gamma_{t} > 0$
$$
\nabla \gamma_ {t} = \kappa \nabla \gamma_ {t}. \tag {5}
$$
For example, when  $\kappa = 0$ , the projection constraint  $\gamma$  will not receive any positive gradient and is therefore non-decreasing. With the annealed gradients  $\nabla \gamma_{t}$ , we update the constraint using the Adam update rule [38] because Adam is suitable for non-stationary optimization, where the optimal values change over time. Please see Appendix 8.3 for a detailed algorithmic description of AdamUpdate and additional discussion on how FTP saves computation.
Finally, after obtaining the updated  $\gamma_{t}$  from AdamUpdate, FTP applies the learned constraints to project the current unconstrained model  $\tilde{\mathbf{W}}_t$  towards the pre-trained model  $\mathbf{W}_0$  using Eq. 2 with a different constraint for each layer. The complete algorithm is summarized in Alg. 1. For a quick comparison with TPGM, we provide a side-by-side computation flow chart of FTP in Fig. 2.
Implicit Assumption. The algorithmic difference between TPGM and FTP makes an implicit assumption. Specifically, after obtaining the updated constraints  $\gamma_{t}$  after AdamUpdate, if the algorithm were to follow TPGM, the next step would be applying the updated constraints to recalculate the previous model  $\mathbf{W}_{t-1}$  since  $\gamma_{t}$  is updated based on  $\mathbf{W}_{t-1}$  (Eq. 4). However, instead of rolling back, FTP applies the updated constraints directly to the current unconstrained model  $\tilde{\mathbf{W}}_t$  to calculate  $\mathbf{W}_t$ . This step assumes smoothness in the update of  $\gamma_{t}$ , i.e., the  $\gamma_{t}$  does not change drastically in consecutive steps. The assumption is valid since  $\gamma_{t}$  is updated by AdampUpdate (Alg. 2 in Appendix 8.3) which uses a moving average update with a momentum of 0.9. So the change of  $\gamma_{t}$  is very smooth because of the high discount factor of 0.9. Importantly, it enables us to re-use the same gradient  $\mathbf{g}_t$  available for computing the current unconstrained model  $\tilde{\mathbf{W}}_t$  to update  $\gamma_{t}$ . This is the key to saving computation because the separate training loop as a result of "rolling back" is the main computation bottleneck in TPGM.
3.3 FTP as a Hyper-Optimizer for Fine-Tuning
The FTP algorithm in Alg. 1 bears motivational and algorithmic similarity to a recent resurrection of hyper-optimizers [11, 12, 13, 14, 15, 16, 17, 18]. Specifically, hyper-optimizers aim to learn the hyper-parameters such as the learning rate in an optimizer by treating them as learnable parameters through nested differentiation and optimization because manual tuning of those hyper-parameters can be time-consuming and can lead to sub-optimal performance. FTP stems from the same motivation as the manual specification of projection constraints can be computationally infeasible [10].
To understand the algorithmic similarity better, let's use SGD as an example. Suppose at iteration  $t - 1$ , we have updated the model parameters  $\mathbf{W}_{t - 2}$  through SGD with a learning rate  $\alpha_{t - 1}$ .
$$
\mathbf {W} _ {t - 1} = \mathbf {W} _ {t - 2} - \alpha_ {t - 1} \nabla \mathcal {L} \left(\mathbf {W} _ {t - 2}\right) \tag {6}
$$
At the current step  $t$ , hyper-optimizers first calculate the gradient w.r.t to the learning rate  $\alpha_{t - 1}$  and update it using another SGD optimizer with a new learning rate parameter  $\kappa$ .
$$
\alpha_ {t} = \alpha_ {t - 1} - \kappa \frac {\partial \mathcal {L} \left(\mathbf {W} _ {t - 1}\right)}{\partial \alpha} = \alpha_ {t - 1} + \kappa \nabla \mathcal {L} \left(\mathbf {W} _ {t - 1}\right) ^ {T} \nabla \mathcal {L} \left(\mathbf {W} _ {t - 2}\right) \tag {7}
$$
Finally, using the updated  $\alpha_{t}$ , hyper-optimizers update the main model parameters.
$$
\mathbf {W} _ {t} = \mathbf {W} _ {t - 1} - \alpha_ {t} \nabla \mathcal {L} \left(\mathbf {W} _ {t - 1}\right) \tag {8}
$$
It's not hard to spot the algorithmic similarity between the FTP algorithm and hyper-optimizers. Both algorithms first update the hyper-parameters (projection constraints  $\gamma_{t}$  in Eq. 4 vs. the learning rate  $\alpha_{t}$  in Eq. 7) using the cached information from the previous iteration and the gradient from the current iteration (known as hyper-gradients). Then, they apply the updated hyper-parameters to calculate the current model. Finally, hyper-optimizers make the same assumption of smoothness in the update of the hyper-parameters such that the update of the hyper-parameters and the model parameters can be performed consecutively in a single forward pass. In this regard, FTP can be seen as a special instance of hyper-optimizer for fine-tuning.
3.4 Dual Perspective: Fine-tuning Robustness in Feature Space and Weight Space
It is not immediately clear why FTP's and other methods' projections in the weight space maintain the robustness of the pre-trained model in the feature space besides the intuition that the closer to the pre-train model the more likely the fine-tuned model will behave like it. To fully understand this, we study the mathematical connection between projection and robustness. Let  $\mathbf{x} \in \mathbb{R}^m$  denote an input vector and  $h(\mathbf{x}) : \mathbb{R}^m \to \mathbb{R}^n$  a function mapping it to a feature space. Given two input vectors  $\mathbf{x}, \mathbf{x}' \in \mathbb{R}^m$ , we denote the distance between them in the original space by their vector norms  $\| \mathbf{x} - \mathbf{x}' \|_{\mathbf{x}}$  and in the feature space by  $\| h(\mathbf{x}) - h(\mathbf{x}') \|_{\mathrm{h}}$ . Let  $h_f(\cdot)$  and  $h_0(\cdot)$  denote a fine-tuned model and its pre-trained initialization, and  $\Delta h(\cdot) \equiv h_f(\cdot) - h_0(\cdot)$  denotes the difference function.
To capture the robustness of a fine-tuned model, we apply the notion of Lipschitz continuity on the difference function because its Lipschitz constant captures the maximum rate of change of differences between the fine-tuned model and the pre-trained model in the feature space. Formally,