SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \Delta h (\mathbf {x}) - \Delta h \left(\mathbf {x} ^ {\prime}\right) \right\| _ {\mathrm {h}} \leq L _ {d} \| \mathbf {x} - \mathbf {x} ^ {\prime} \| _ {\mathrm {x}} \quad \forall \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) \in \mathbb {R} ^ {m}. \tag {9}
$$
where  $L_{d}\geq 0$  is the Lipschitz constant of the difference function  $\Delta h(\mathbf{x})$ . If the inequality is satisfied, in this paper, we call  $h_f(\cdot)L_d$ -Lipschitz-robust w.r.t. the pre-trained initialization  $h_0(\cdot)$
The definition has a natural intuition stemming from Lipschitz continuity, a measure of robustness [19, 20, 21]. A Lipschitz function is limited by how fast it can change, governed by the Lipschitz constant. Traditionally, a small Lipschitz constant is associated with better robustness, because a small constant means less sensitivity to changes in the input. We provide the following lemma (proof in Appendix 8.1) to illustrate the connection between the difference function and the robustness of the fine-tuned model.
Lemma 1. If a fine-tuned model  $h_f(\cdot)$  is  $L_{d}$ -Lipschitz-robust with respect to its  $L_{0}$ -Lipschitz pretrained initialization  $h_0(\cdot)$ , i.e.,  $\forall (\mathbf{x},\mathbf{x}^{\prime})\in \mathbb{R}^{m}$ ,
$$
\| \Delta h (\mathbf {x}) - \Delta h \left(\mathbf {x} ^ {\prime}\right) \| _ {\mathrm {h}} \leq L _ {d} \| \mathbf {x} - \mathbf {x} ^ {\prime} \| _ {\mathrm {x}} \quad a n d \quad \| h (\mathbf {x}) _ {0} - h \left(\mathbf {x} ^ {\prime}\right) _ {0} \| _ {\mathrm {h}} \leq L _ {0} \| \mathbf {x} - \mathbf {x} ^ {\prime} \| _ {\mathrm {x}}
$$
then,  $h_f(\cdot)$  is  $(L_d + L_0)$ -Lipschitz, i.e.,
$$
\left\| h _ {f} (\mathbf {x}) - h _ {f} \left(\mathbf {x} ^ {\prime}\right) \right\| _ {\mathrm {h}} \leq \left(L _ {d} + L _ {0}\right) \left\| \mathbf {x} - \mathbf {x} ^ {\prime} \right\| _ {\mathrm {x}} \quad \forall (\mathbf {x}, \mathbf {x} ^ {\prime}) \in \mathbb {R} ^ {m}.
$$
Feature Space. From Lemma 1, we can see that minimizing  $L_{d}$  can improve the robustness of the fine-tuned model, defined by its Lipschitz constant  $(L_{d} + L_{0})$ , which equals  $L_{0}$  when  $L_{d} = 0$ . Therefore, the fine-tuned model can achieve a similar level of robustness as the pre-trained model if  $L_{d}$  is minimized. Colloquially, given two inputs  $(\mathbf{x},\mathbf{x}^{\prime})$ , where  $\mathbf{x}^{\prime}$  is a perturbed version of  $\mathbf{x}$ ,  $h_f(\cdot)$  will be just as sensitive/robust to the perturbation as  $h_0(\cdot)$  is if  $L_{d}$  is small.
Weight Space. The definition of fine-tuning robustness (Eq. 9) not only leads to an interpretation of robustness in the feature space (Lemma 1) but also conveniently a projection operation in the weight space. Specifically, we investigate a single linear layer in a neural network and show that enforcing the inequality in Eq. 9 leads to a projection operation by virtue of linear operators and matrix norms. We illustrate this in the following lemma with a full discussion and proof in Appendix 8.2.
Lemma 2. Assuming linear models  $h(\mathbf{x}) = \mathbf{W}\mathbf{x} + \mathbf{b}, \mathbf{W} \in \mathbb{R}^{n \times m}, \mathbf{b} \in \mathbb{R}^n$ , and both the input space vector norm  $\| \cdot \|_{\mathbf{x}}$  and the feature space vector norm  $\| \cdot \|_{\mathrm{h}}$  are defined by  $l_{\infty}$  norm.  $\mathbf{w}_p^i$  satisfies the inequality in Eq. 9 if
$$
\mathbf {w} _ {p} ^ {i} = \min  \left(1, \frac {L _ {d}}{\| \mathbf {w} _ {f} ^ {i} - \mathbf {w} _ {0} ^ {i} \| _ {1}}\right) \left(\mathbf {w} _ {f} ^ {i} - \mathbf {w} _ {0} ^ {i}\right) + \mathbf {w} _ {0} ^ {i}, \quad \forall i \in \{1, \dots , n \}. \tag {10}
$$
where  $\mathbf{w}^i$  denotes the  $i$ -th row of the matrix  $\mathbf{W}$  and  $\mathbf{w}_p^i$  is the new projected fine-tuned model.
This is an equation of projection between  $\mathbf{W}_f$  and  $\mathbf{W}_0$  defined by the MARS norm in the weight space for a single linear layer and is the projection operation used in FTP and prior works [9, 10] (Eq. 2). It indicates that we can choose an arbitrarily small  $L_{d}$  and enforce it through Eq. 10, potentially trading off fitting the downstream task and preserving robustness. In summary, this section demonstrates the connection between robustness and projection. Specifically, we have shown that to achieve good fine-tuning robustness, we can enforce a small Lipschitz constant  $L_{d}$  on the difference function  $\Delta h(\mathbf{x})$  in the feature space (Lemma 1), which can be physically enforced through the projection of the fine-tuned model towards the pre-trained model in the weight space (Lemma 2).
Overview. To validate the effectiveness of FTP in fine-tuning pre-trained models, we benchmark FTP on both image classification (Sec. 4.1) and dense vision tasks (Sec. 4.2) with different network architectures and pre-trained models. For each benchmark, we report both in-distribution (ID) performance as well as out-of-distribution (OOD) performance. We show that FTP not only achieves competitive ID performance and superior OOD performance but is also much more computationally efficient than prior works. We further test FTP's regularization capability on a continual learning benchmark and show state of art performance against recent SOTA methods (Sec. 4.3).
Table 1: DomainNet Results using MOCO-V3 pre-trained ResNet50 with  $100\%$  Real Data. FTP achieves the best OOD performance and is much faster than prior work TPGM [10] by  $36\%$ .
ID Real	Sketch	OOD	Statistics
Painting	Infograph	Clipart	OOD Avg.	ID Δ (%)	OOD Δ (%)	Time (s/it)↓
Vanilla FT	81.99 (0.03)	31.52 (0.33)	42.89 (0.53)	18.51 (0.28)	44.98 (0.24)	34.47	0.00	0.00	0.35
Linear Prob.	73.01 (0.03)	24.10 (0.23)	39.56 (0.15)	12.27 (0.02)	30.38 (0.08)	26.58	-10.96	-22.90	0.10
Partial Fusion [32]	78.27 (0.03)	27.72 (0.07)	39.74 (0.12)	15.56 (0.08)	38.18 (0.12)	30.30	-4.55	-12.11	0.21
L2-SP [35]	81.51 (0.02)	34.91 (0.22)	45.76 (0.16)	18.97 (0.11)	45.29 (0.18)	36.23	-0.59	5.09	0.46
MARS-SP [9]	81.89 (0.01)	34.44 (2.54)	45.05 (1.91)	19.97 (1.48)	46.36 (1.29)	36.45	-0.13	5.74	0.43
LP-FT [7]	82.92 (0.01)	34.50 (0.22)	45.42 (0.31)	20.12 (0.43)	47.11 (0.27)	36.79	1.13	6.72	-
TPGM [10]	82.66 (0.13)	35.35 (0.33)	46.20 (0.20)	20.13 (0.12)	45.75 (0.12)	36.86	0.82	6.91	0.80
FTP	82.17 (0.02)	36.26 (0.06)	46.58 (0.10)	20.67 (0.03)	46.97 (0.06)	37.62	0.22	9.13	0.51
Table 2: DomainNet Results using CLIP pre-trained ResNet50 with  $100\%$  Real Data. FTP achieves competitive OOD performance and is much faster than prior work TPGM [10] by  $36\%$ .
ID Real	OOD	Statistics
Sketch	Painting	Infograph	Clipart	OOD Avg.	ID Δ (%)	OOD Δ (%)	Time (s/fit)↓
Vanilla FT	80.93 (0.08)	31.81 (0.06)	41.02 (0.10)	20.29 (0.08)	43.59 (0.15)	34.18	0.00	0.00	0.58
Linear Prob.	52.56 (0.09)	20.05 (0.21)	24.92 (2.49)	19.18 (0.46)	21.15 (0.18)	21.33	-35.05	-37.60	0.14
Partial Fusion [32]	78.27 (0.11)	36.77 (0.32)	42.13 (0.35)	24.71 (0.18)	43.31 (0.53)	36.73	-3.29	7.46	0.33
L2-SP [35]	82.07 (0.09)	36.67 (0.11)	45.62 (0.35)	22.97 (0.42)	47.78 (0.30)	38.26	1.40	11.94	0.62
MARS-SP [9]	77.19 (0.63)	25.33 (1.07)	33.43 (2.06)	14.81 (0.43)	39.20 (0.74)	28.19	-4.62	-17.53	0.61
LP-FT [7]	80.82 (0.95)	34.85 (1.93)	44.03 (0.05)	22.23 (2.01)	46.13 (2.34)	36.81	-0.14	7.69	-
TPGM [10]	83.64 (0.01)	38.78 (0.42)	43.11 (0.25)	28.70 (0.31)	48.01 (0.25)	39.65	3.34	16.01	1.07
FTP	84.22 (0.11)	37.66(0.45)	46.11(0.29)	28.33 (0.33)	47.67 (0.18)	39.94	4.05	16.87	0.68
4.1 Image Classification Experiments
4.1.1 DomainNet
For the DomainNet experiment (image classification), which consists of five domains, Real, Sketch, Painting, Infographics, and Clipart, we follow the setup of the prior work [10] and use its released code to train FTP. Specifically, we use two pre-trained models, an ImageNet pre-trained MOCO-V3 ResNet50 [3] and a CLIP pre-trained ResNet50 [4]. For FTP, we only tuned the learning rate while keeping the other hyper-parameters fixed as in the prior work. We use the Real domain as the ID training dataset and the rest as OOD testing datasets. Please refer to Appendix 8.4 for more details.
FTP achieves the best OOD accuracy and is much more efficient. In Tab. 1 and Tab. 2, we show results training on  $100\%$  DomainNet-Real data using CLIP and MOCO-V3 pre-trained initialization respectively. Compared to the previous SOTA methods TPGM [10], FTP achieves competitive ID accuracy and better OOD generalization performance. More importantly, in addition to favorable results, FTP is  $36\%$  faster on average on both benchmarks compared to TPGM. Following TPGM [10], we also report results training only on  $10\%$  DomainNet-Real data in Appendix Tab. 6.
4.1.2 ImageNet
Recently, zero-shot language-vision pre-trained models such as CLIP [4] have demonstrated strong generalization capability to other tasks. Notably, WISE [8] showed that linear interpolation between a fine-tuned model and its initialization achieves significant improvement in OOD generalization.
Figure 3: ImageNet WISE Interpolation [8] Result using CLIP ViT-Base Fine-tuned models.
Table 3: ImageNet Fine-tuning Result using CLIP ViT-Base.
ID Im	ImV2	OOD	Statistics
Im-A	Im-R	Im-S	OOD Ave.	Ave.
zero-shot	67.68	61.41	30.60	56.77	45.53	48.58	52.40
vanilla FT	83.66	73.82	21.40	43.06	45.52	46.98	54.29
Linear Prob.	78.25	67.68	26.54	52.57	48.26	48.76	54.66
LP-FT [7]	82.99	72.96	21.08	44.65	47.56	46.56	53.85
L2-SP [35]	83.44	73.2	20.55	43.89	46.60	46.06	53.54
FTP	84.19	74.64	26.50	47.23	50.23	49.65	56.56
WISE-FT [8]	80.94	72.47	33.18	63.33	54.20	55.58	60.82
WISE-FTP	82.61	74.09	34.56	61.18	55.06	56.22	61.50
However, there are two limitations: 1) not all pre-trained models have this property of linear connectivity and 2) a zero-shot classifier head is needed to initialize the linear classifier head. Our contribution is orthogonal to WISE because FTP is a general optimization algorithm whereas WISE is a post-training algorithm for specific zero-shot models. Therefore, we first compare FTP to vanilla fine-tuning and then apply WISE to both models. We follow the public code base of DEIT [39] to train our CLIP pre-trained ViT-Base. Specifically, we use weight-decay (0.1), drop-path (0.2) [40], label-smoothing (0.1) [41], Mixup (0.8) [42] and Cutmix (1.0) [43]. We train our model on ImageNet and report OOD performance on ImageNet-V2 [44], ImageNet-A [45], ImageNet-R [46], and ImageNet-S [47]. Please refer to Appendix 8.4 for more details on implementation.
FTP outperforms vanilla fine-tuning and improves WISE performance. In Tab. 3, we report performance for competing methods. Even with various regularizations and augmentations in place, FTP can further improve ID performance on ImageNet. Furthermore, FTP brings better OOD performance on all four OOD datasets. This shows that FTP successfully maintains the robustness of the pre-trained CLIP model while existing regularization such as weight decay and drop-path do not. We also report the interpolation results using WISE [8] for the vanilla fine-tuned and FTP fine-tuned models. We sweep a range of interpolation ratios  $\in$ $\{0.1, 0.2, \dots, 0.9\}$  and show the trajectory of ID vs. OOD performance plot in Fig. 3. The models with the best average performance are reported in the lower portion of Tab. 3. As expected, WISE interpolation significantly improves OOD generalization for both methods. However, WISE-FTP has significantly better ID performance while still having better OOD performance. This shows that improvement to the base fine-tuning strategy can further benefit pose-training methods such as WISE.
4.2 PASCAL Dense Vision Task Experiments
Table 4: Pascal Semantic Segmentation Results using SWIN-Tiny transformers pre-trained on ImageNet21K. Performance is measured by mIoU↑. FTP achieves the best OOD performance and is much faster than prior work TPGM [10] by  $34\%$ .
ID Clean	OOD	Statistics
Fog	Defocus	Gaussian	Brightness	OOD Avg.	ID Δ (%)	OOD Δ (%)	Time (s/it)↓
Vanilla FT	66.03 (0.37)	56.72 (0.83)	38.04 (0.83)	23.21 (0.96)	58.03 (0.66)	44.00	0.00	0.00	0.288
Adapter [24]	71.85 (0.06)	69.36 (0.07)	50.94 (0.25)	37.43 (0.64)	68.26 (0.08)	56.50	8.82	28.40	0.233
BitFit [30]	70.31 (0.11)	67.00 (0.24)	46.39 (0.35)	30.61 (0.51)	66.22 (0.16)	52.56	6.49	19.44	0.248
L2-SP [35]	73.47 (0.06)	69.87 (0.04)	49.20 (0.43)	39.10 (0.84)	68.61 (0.24)	56.70	11.27	28.85	0.347
MARS-SP [9]	66.24 (0.23)	56.97 (0.79)	37.29 (1.20)	21.82 (2.06)	58.27 (0.33)	43.59	0.32	-0.94	0.318
LLRD [48]	72.09 (0.06)	68.13 (0.25)	46.18 (1.30)	37.28 (2.54)	66.30 (0.29)	54.47	9.18	23.79	0.289
TPGM [10]	72.56 (0.06)	69.51 (0.57)	50.88 (0.97)	38.62 (1.04)	68.82 (0.25)	56.96	9.89	29.44	0.611
FTP	73.79 (0.10)	71.10 (0.23)	52.63 (0.75)	40.25 (0.21)	69.81 (0.49)	58.45	11.76	32.83	0.401
To further demonstrate the effectiveness of FTP in more diverse scenarios, we test it on PASCAL-Context [49]. Specifically, following the prior work [50], we use the PASCAL-Context datasets [49], which consist of labels for semantic segmentation, human parts segmentation, and surface normal estimation. For OOD performance, following the popular natural robustness literature [51], we report results on various degradations including fog, defocus blur, Gaussian noise, and brightness corruption, with 5 severity each. We use a combination of Swin ViT-Tiny [52] (pre-trained on ImageNet-22K) and Segformer [53]. In this architecture, Swin Transformer serves as the feature extraction backbone and Segformer is the task-specific decoder. While the feature backbone is initialized with pre-trained weights, a significant part of the entire model (the Segformer decoder) is randomly initialized; In contrast, in simple classification (Sec. 4.1.1), only the last linear classification layer is randomly initialized. Please refer to Appendix 8.5 for details.
FTP achieves the best ID performance and OOD generalization. We report results for semantic segmentation, human parts segmentation, and surface normal estimation in Tab. 4, Appendix Tab. 7, and Appendix Tab. 8 respectively. We additionally add Layer-Wise Learning Rate decay [48] (LLRD) as a strong baseline. Notably, in all three tasks, FTP outperforms vanilla fine-tuning on ID performance by  $11.71\%$ ,  $4.48\%$ , and  $18.30\%$  respectively. This demonstrates the effectiveness of projection as a regularization technique for transfer learning. More importantly, the OOD performance improves as large as  $33.02\%$  in semantic segmentation. This shows that 1) FTP can effectively maintain the robustness of the original pre-trained model; 2) even though the entire decoder component is randomly initialized, it is worthwhile to put regularizations on the pre-trained feature backbone.
Figure 4: Average Learned Constraints for each task using FTP.
Table 5: CL Results on ImageNet-R
Method	A1:N(↑)	FN(↓)
FT++ [54]	48.93 ± 1.15	9.81 ± 0.31
LwF.MC [55]	66.73 ± 1.25	3.52 ± 0.39
L2P++ [56]	71.66 ± 0.64	1.78 ± 0.16
DualPrompt [57]	71.32 ± 0.62	1.71 ± 0.24
CODA-P [54]	75.45 ± 0.56	1.64 ± 0.10
EWC [58]	64.66 ± 2.04	1.55 ± 0.25
L2 [54]	76.06 ± 0.65	1.68 ± 0.16
FTP	76.06 ± 0.35	2.27 ± 0.18
FTP + EWC	77.26 ± 0.40	1.48 ± 0.15
4.3 Continual Learning (CL) Experiments
Recently, pre-trained models have been shown to greatly improve the performance of CL algorithms [59]. We follow the settings in this work [59] to partition ImageNet-R (200 classes) into 10 sequential tasks with 20 non-overlapping classes in each task. A model is trained on each task only once sequentially. To use FTP for CL tasks, unlike supervised vision tasks (Sec. 4.1, 4.2), we re-initialize FTP after each task and use the current model as the "pre-trained model" for the next task. Moreover, inspired by the prior work [59], we use FTP to only fine-tune the attention blocks. We report both the final task accuracy across all tasks  $A_{1:N}\uparrow$  and the global forgetting  $F_{N}\downarrow$  in Tab. 5 to analyze plasticity and forgetting. Please refer to Appendix 8.6 for more on the metrics and experimental setup. In Table 5, we benchmark against the popular and recent rehearsal-free continual learning methods. FTP alone achieves state of art accuracy against all methods and relatively good forgetting compared to vanilla FT, a sign of superior plasticity and balanced forgetting. We visualize the learned constraints for each task in Fig. 4. We observe that while each task is independent and FTP is re-initialized each time, FTP learns stronger regularization for later tasks. This contributes to lower forgetting compared to FT. We found that FTP combined with a simple continual learning method, EWC [58], achieves state-of-the-art in this setting. Compared to the prompting methods L2P, DualPrompt, and the recent CODA-Prompt, FTP has clear and significant improvements. Our intuition is that the combination of the superior plasticity of FTP and the low forgetting of EWC is the key to the improvement.
Like any regularization method, FTP has a hyper-parameter to adjust its regularization strength. In this case, the positive gradient annealing factor  $0 \leq \kappa \leq 1$  (default 1) (Sec. 3.2) controls the strength of projection with smaller values indicating weaker regularization. Note that  $\kappa = 0$  means that the projection constraints are non-decreasing during training. In this case, FTP still provides regularization. For example, we found that a  $\kappa = 0$  is necessary to obtain the best performance for some dense vision tasks in Appendix 8.5. Generally, we recommend starting with the default  $\kappa$  and only tuning it if underfitting is observed.
In this paper, we proposed Fast Trainable Projection, a fine-tuning algorithm to maintain the robustness and the generalization capability of the pre-trained model. FTP learns projection constraints for each layer in a neural network efficiently by carefully re-using past information to save computation. To understand the connection between robustness and projection, we provided a holistic discussion of fine-tuning robustness from its feature space definition to the weight space dual. The new perspective lends a mathematical foundation to the idea of using projection in fine-tuning. Across four vision tasks with different pre-trained models, FTP demonstrated superior ID and OOD generalization capability and significantly better computation efficiency. Furthermore, the continual learning experiments demonstrated FTP's potential in other deep learning paradigms beyond simple fine-tuning. Combined with its compatibility with popular optimization algorithms, we believe FTP can be broadly beneficial in improving the performance of learning tasks using pre-trained initialization.
This work was supported by ONR grant N00014-18-1-2829.
[53] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34:12077-12090, 2021.
[54] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. arXiv preprint arXiv:2211.13218, 2022.
[55] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
[56] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
[57] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. arXiv preprint arXiv:2204.04799, 2022.
[58] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
[59] James Seale Smith, Junjiao Tian, Yen-Chang Hsu, and Zsolt Kira. A closer look at rehearsal-free continual learning. arXiv preprint arXiv:2203.17269, 2022.
[60] Sungyoon Lee, Jaewook Lee, and Saerom Park. Lipschitz-certifiable training with a tight outer bound. Advances in Neural Information Processing Systems, 33:16891-16902, 2020.
[61] Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 312-321, 2019.
8 Appendix
8.1 Proof of Lemma 1
Prior works [9, 10] have used the high-level notion that staying "close" to the pre-trained model can help maintain its robustness capability to justify using projection for fine-tuning. However, there is more than one way to encourage this, for example, regularization [35], a small learning rate [7], and projection [10]. It is not immediately clear why projection is a principled approach. To understand FTP's capability to maintain the pre-trained mode's robustness, we first propose to establish a connection between Lipschitz continuity, a commonly used measure of robustness [19, 20, 21], and fine-tuning through a new definition of difference function in the Lemma 1.
Proof. We first expand the difference functions in Eq. 9, i.e. plugging in  $\Delta h(\cdot) = h_f(\cdot) - h_0(\cdot)$ ,
$$
\begin{array}{l} \left\| \Delta h (\mathbf {x}) - \Delta h \left(\mathbf {x} ^ {\prime}\right) \right\| _ {\mathrm {h}} \leq L _ {d} \| \mathbf {x} - \mathbf {x} ^ {\prime} \| _ {\mathrm {x}} \quad \forall (\mathbf {x}, \mathbf {x} ^ {\prime}) \in \mathbb {R} ^ {m} \tag {11} \\ \rightarrow \| \left(h _ {f} (\mathbf {x}) - h _ {0} (\mathbf {x})\right) - \left(h _ {f} \left(\mathbf {x} ^ {\prime}\right) - h _ {0} \left(\mathbf {x} ^ {\prime}\right)\right) \| _ {\mathrm {h}} \leq L _ {d} \| \mathbf {x} - \mathbf {x} ^ {\prime} \| _ {\mathrm {x}} \\ \rightarrow \| \left(h _ {f} (\mathbf {x}) - h _ {f} \left(\mathbf {x} ^ {\prime}\right)\right) - \left(h _ {0} (\mathbf {x}) - h _ {0} \left(\mathbf {x} ^ {\prime}\right)\right) \| _ {\mathrm {h}} \leq L _ {d} \| \mathbf {x} - \mathbf {x} ^ {\prime} \| _ {\mathrm {x}} \\ \end{array}
$$
Then we apply the reverse triangular inequality to the left-hand side of Eq. 11.
$$
\left|\left|\left| h _ {f} (\mathbf {x}) - h _ {f} \left(\mathbf {x} ^ {\prime}\right)\right|\right|\right| _ {\mathrm {h}} - \left|\left| h _ {0} (\mathbf {x}) - h _ {0} \left(\mathbf {x} ^ {\prime}\right)\right|\right| _ {\mathrm {h}} | \leq \left|\left|\left| h _ {f} (\mathbf {x}) - h _ {f} \left(\mathbf {x} ^ {\prime}\right)\right|\right| - \left(h _ {0} (\mathbf {x}) - h _ {0} \left(\mathbf {x} ^ {\prime}\right)\right)\right|\left. \right| _ {\mathrm {h}}
$$
Therefore, we have,
$$
\begin{array}{l} \left\| h _ {f} (\mathbf {x}) - h _ {f} \left(\mathbf {x} ^ {\prime}\right) \right\| _ {\mathrm {h}} - \left\| h _ {0} (\mathbf {x}) - h _ {0} \left(\mathbf {x} ^ {\prime}\right) \right\| _ {\mathrm {h}} \leq L _ {d} \left\| \mathbf {x} - \mathbf {x} ^ {\prime} \right\| _ {\mathrm {x}} \tag {12} \\ \rightarrow \| h _ {f} (\mathbf {x}) - h _ {f} \left(\mathbf {x} ^ {\prime}\right) \| _ {\mathrm {h}} \leq L _ {d} \| \mathbf {x} - \mathbf {x} ^ {\prime} \| _ {\mathrm {x}} + \| h _ {0} (\mathbf {x}) - h _ {0} \left(\mathbf {x} ^ {\prime}\right) \| _ {\mathrm {h}} \\ \end{array}
$$
Assuming that the pre-trained model  $h_0$  is  $L_0$ -Lipschitz, we know that  $\| h_0(\mathbf{x}) - h_0(\mathbf{x}') \|_{\mathrm{h}} \leq L_0 \| \mathbf{x} - \mathbf{x}' \|_{\mathrm{x}}$ ,  $\forall (\mathbf{x}, \mathbf{x}') \in \mathbb{R}^m$ . Plug this into Eq. 12,
$$
\left\| h _ {f} (\mathbf {x}) - h _ {f} \left(\mathbf {x} ^ {\prime}\right) \right\| _ {\mathrm {h}} \leq \left(L _ {d} + L _ {0}\right) \| \mathbf {x} - \mathbf {x} ^ {\prime} \| _ {\mathrm {x}} \tag {13}
$$
8.2 Proof of Lemma 2