Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting
Authors: Chen Huang
Skyler Seto
Hadi Pouransari
Mehrdad Farajtabar
Raviteja Vemulapalli
Fartash Faghri
Oncel Tuzel
Barry-John Theobald
Joshua M. Susskind
Venue: ICML 2025 poster (2025)
Keywords: Proxy-FDA
robust fine-tuning
concept forgetting
vision foundation model
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: Forgetting-free fine-tuning of vision foundation models via Proxy-based Feature Distribution Alignment (Proxy-FDA)

Abstract: Vision foundation models pre-trained on massive data encode rich representations of real-world concepts, which can be adapted to downstream tasks by fine-tuning. However, fine-tuning foundation models on one task often leads to the issue of *concept forgetting* on other tasks. Recent methods of robust fine-tuning aim to mitigate forgetting of prior knowledge without affecting the fine-tuning performance. Knowledge is often preserved by matching the original and fine-tuned model weights or feature pairs. However, such point-wise matching can be too strong, without explicit awareness of the feature neighborhood structures that encode rich knowledge as well. We propose a novel regularization method **Proxy-FDA** that explicitly preserves the structural knowledge in feature space. Proxy-FDA performs Feature Distribution Alignment (using nearest neighbor graphs) between the pre-trained and fine-tuned feature spaces, and the alignment is further improved by informative proxies that are generated dynamically to increase data diversity. Experiments show that Proxy-FDA significantly reduces concept forgetting during fine-tuning, and we find a strong correlation between forgetting and a distributional distance metric (in comparison to L2 distance). We further demonstrate Proxy-FDA's benefits in various fine-tuning settings (end-to-end, few-shot and continual tuning) and across different tasks like image classification, captioning and VQA.

ProXy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting
Chen Huang' Skyler Seto
Hadi Pouransari'
Mehrdad Farajtabar' Raviteja Vemulapalli Fartash Faghri
Oncel Tuzel Barry-John Theobald' Josh Susskind
Abstract
1. Introduction
Vision foundation models like CLIP (Radford al., 2021) and DINOv2 (Oquab et al., 2024) pre-trained on large amounts of data demonstrate remarkable performance across various tasks and data distributions. Such foun- dation models are known to have learned vast knowledge on real-world concepts that can serve as a useful prior for downstream task adaptation via fine-tuning. Existing fine- tuning methods include end-to-end finetuning. linear prob- ing, prompt tuning (Zhou et al., 2022a;b), and adapter learn- ing (Gao et al., 2021). While these methods prove effective, empirical evidence shows that they frequently suffer from an undesirable effect called concept forgetting (Mukhoti et al., 2024). Forgetting occurs when fine-tuned model overfits on the downstream task, and unlike its pre-trained counter- part, significantly loses the ability to recognize concepts on other tasks.
Vision foundation models pre-trained on massive data encode rich representations of real-world concepts, which can be adapted downstream tasks by fine-tuning, However, fine-tuning foun- dation models on one task often leads to the is- sue of concept forgetting on other tasks. Recent methods of robust fine-tuning aim to mitigate for- getting of prior knowledge without affecting the fine-tuning performance. Knowledge is often pre- served by matching the original and fine-tuned model weights feature pairs. However, such point-wise matching can be too strong, without ex- plicit awareness of the feature neighborhood struc- tures that encode rich knowledge as well. We pro- pose novel regularization method Proxy-FDA that explicitly preserves the structural knowledge in feature space. Proxy-FDA performs Feature Distribution Alignment (using nearest neighbor graphs) between the pre-trained and fine-tuned feature spaces, and the alignment is further im- proved by informative proxies that are generated dynamically to increase data diversity. Experi- ments show that Proxy-FDA significantly reduces concept forgetting during fine-tuning, and we find strong correlation between forgetting and dis- tributional distance metric (in comparison to L2 distance). We further demonstrate Proxy-FDA's benefits various fine-tuning settings (end-to- end, few-shot and continual tuning) and across different tasks like image classification, caption- ing and VQA.
Concept forgetting has driven recent research on robust fine-tuning. The goal is to preserve the pre-trained knowl- edge and perform well on downstream tasks. One sim- ple approach is to ensemble models before and after fine- tuning (Wortsman al., 2022b). Alternative methods use regularization techniques constrain the fine-tuned model to remain close to the original foundation model in either weight space (Li et al., 2018) or feature space (Mukhoti et al., 2024). Feature-space regularization by matching the pre-trained and fine-tuned features across samples shows more promising effect in reducing forgetting, since di- rectly minimizes the change in input-ouiput behaviour of the model. One key assumption behind such regularization is that the L2 feature space distance is good indicator of the similarity of encoded concepts in different models.
We argue that aligning individual feature points imposes too strong of a constraint. Without an explicit insight feature neighborhoods, the concepts preserved point-wise are found to be limited, resulting in sub-optimal performance of for- getting reduction. Here suggest that it is desirable to plicitly inform the fine-tuning process of the local structure of feature neighborhoods. By preserving this neighborhood structure with a structure-wise regularization term, the rich knowledge encoded in the local structure of the original
'Apple, Cupertino. United States. Correspondence to: Chen Huang <chen- <ehen-huanger @ @applc.copp..........
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Proxy-FDA
Poa/neg @
Real-world concepts
Classes
proxies
task on dataset Dt
Pre- Pre-trained model fa
Pre-trat tra-trahndd feature space
Local nelghborhood
Fine-tuning
Proxy-FDA aligns neighborhood structures
???
Concept forgetting on unseen task or dataset D # Dtt
Structure-wise feature regularization
Fine tured featura space
Fine-tuned model fg
LPr
Preserving concepts and generalizing on unseen task or dataset D # Dtt
(a)
(b)
Figure 1: (a) Motivation: alleviating concept forgetting during model fine-tuning by novel feature regularization method- Proxy-FDA (Proxy-based Feature Distribution Alignment). (b) Proxy-FDA aligns the pre-trained and fine-tuned feature distributions by their local neighborhood structures, which is further aided by proxies (i.e., synthetic features). We show Proxy-FDA indeed preserves the rich knowledge in local feature neighborhood (example visualization in Fig. 5).
feature space will be transferred to the fine-tuned one. As a result, the fine-tuned model can forget significantly less while still maintaining its downstream performance (Fig. I).
thereby further reducing concept forgetting.
We have extensive experiments of fine-tuning vision founda- tion models end-to-end ten classification tasks. Results show that Proxy-FDA significantly outperforms other fine- tuning baselines in preventing concept forgetting, without hurting the downstream accuracy. We also find strong correlation between concept forgetting and a distance met- ric OTDD - Optimal Transport Dataset Distance (Alvarez- Melis & Fusi, 2020) which is ideal to measure the alignment quality for feature distributions with local structures. Cru- cially, the correlation between concept forgetting and the OTDD metric indicates the need for some form of structure- wise FDA to better mitigate forgetting. Results confirm that our structure-wise Proxy-FDA forgets much less than point-wise feature regularization (Mukhoti et al., 2024).
The above idea motivates us to propose new feature reg- ularization term called Feature Distribution Alignment (FDA). Specifically, we first model the structural relations of pre-trained features using a nearest neighbor graph. Then we transfer the graph to the fine-tuned feature space, where fea- ture neighbors are pulled together while non-neighbors are pushed away regardless of their labels. Such FDA process enables sharing knowledge beyond class concepts (e.g., vi- sual attributes) in local feature neighborhoods, Fig. 5 pro- vides an example of the white color attribute of two dog breeds mined from a local neighborhood on ImageNet. This example represents the common-sens, prior knowledge em- bedded in vision foundation model that often richer than the class labels on downstream datasets. Preserving such knowledge (e.g., about color) during fine-tuning is im- portant to maintain the generalizability of the foundation model, which can facilitate recognizing unfamiliar classes from different tasks. What is harmful is to just specialize on the task at hand, since all information (e.g., color sensitivity) but its class label will be discarded.
We further show Proxy-FDA can be plugged into various prompt tuning methods for few-shot fine-tuning. In all cases, Proxy-FDA shows superior performance and data efficiency for lowering forgetting. Proxy-FDA also proves effective on continual fine-tuning tasks, outperforming specialized continual learning baselines. Lastly, we show the benefits of Proxy-FDa when fine-tuning for tasks beyond classification, like image captioning and VQA. Proxy-FDA also demon- strates its utility in the domain of knowledge distillation.
Another key contribution of this paper an improvement to FDA, with the introduction of new regularization called Proxy-FDA, which uses proxies as synthetic features. This full method is particularly useful on data-deficient fine- tuning tasks (such as few-shot ones), where the limited task data do not allow sufficient alignment of complex feature distributions. To further increase data diversity, Proxy-FDA learns to generate set of instance-wise proxies both within and outside target feature's local neighborhood. Fig. 5 ex- emplifies some proxies that synthesize informative unseen data or unseen class concepts. We empirically show that the generated proxies improves FDA with richer data/concepts,
In summary, our main contributions include:
A novel regularization method, Proxy-FDA, that aligns the local structures of feature distributions with learned proxies, aiming to preserve concepts when fine-tuning vision foundation models;
Correlation analysis between concept forgetting and a structure- distributional distance metric, OTDD, which implicitly explains the success of our structure- wise FDA method;
2

---

ProXy-FDA
State-of-the-art performance on reducing forgetting in various fine-tuning settings and across different tasks.
Proxy learning. This approach is widely adopted in deep metric learning (Movshovitz-Atiias al., 2017; Kim et al., 2020; Roth et al., 2022) to reduce the sampling complexity of pure sample- baseee methods. Proxies are learned as class prototypes to optimize sample-proxy distances in place of sample-sample distances, resulting in faster convergence. By contrast, our proxy learning is different both imple- mentation and motivation: we learn instance- -wie proxies via adaptive pooling of true samples; we also do not use the proxies as sample stand-ins, but as rich augmentations for improving FDA. This makes our approach more related to those feature augmentation methods, such as by random linear interpolation (Verma et al., 2019) and outlier feature synthesis (Du et al., 2022; Tao et al., 2023). Empirically, our method is more effective than these feature augmentation methods by generating diverse augmented features from the entire feature neighborhood.
2. Related Work
Robust fine-tuning. End-to-end fine-tuning often suffers from concept forgetting and degraded out-of-distribution (OOD) performance. In the foundation model era, linear probing or that followed by end-to-end tuning (Kumar et al., 2022) are common remedies to maintain the OOD robust- ness of a pre-trained model. Alternative methods either ensemble the original and fine-tuned models (Wortsman et al., 2022b;a) or use the contrastive pre-training loss di- rectly for fine-tuning (Goyal et al., 2023). More recently, Song et al. (2023) propose a method called FD-Align, which trains a spurious feature classifier and maintains its output consistency during tine-tuning. As result, FD-Align sig- nificantly improves the OOD accuracy. To prevent forget- ting. regularization methods are often used to minimize the model distance before and after fine-tuning in either weight space (Li et al., 2018) or feature space (Mukhoti et al., 2024). In few-shot settings, regularization is even more important. For example, the prompt learning method CLIPood (Shu et al., 2023) regularizes via temporal model ensembling, while PromptSRC (Khattak et al., 2023b) directly regular- izes the output features and logits between pre- pre-trained and prompt-tuned models. Nevertheless, all existing methods do not explicitly account for feature neighborhood structures, which we show is key for robust fine-tuning.
3. Method
We aim for forgetting- free fine-tuning of vision foundation models (e.g., CLIP and DINOv2), using feature-space regu- larization based on Feature Distribution Alignment (FDA), Specifically, given pre-trained model fĄ. we use the down- stream dataset Di to fine-tune the model into fo. Our goal is to specialize the fine-tuned model on Dn with low task loss Lusk (e.g., cross-entropy loss for classification), whilst preventing concept forgetting on any target dataset D & Da. To prevent forgetting, we introduce an FDA-based regular- ization term the downstream task loss, which gives:
Feature and data distribution alignment. These tech- niques have been explored in different contexts. At the core of measuring distributional distances, Optimal Transport (OT) (Villani, 2008) provides a principled approach to com- pare data distributions in geometrically meaningful way. Given the similar nature of our FDA method that aligns the "clustering" structures of distributions, we use an OT-based distance metric OTDD (Alvarez-Melis & Fusi, 2020) to measure FDA quality. Feature alignment is also key to Do- main Adaptation (DA) (Wang & Deng, 2018). However, most DA methods learn a separate domain-invariant feature subspace to align domains implicitly, which differs from our explicit FDA during fine-tuning. More related to our method is the Knowledge Distillation (KD) field (Wang & Yoon, 2021), where traditional KD methods match features or probability distributions between teacher and student models. Relation-based KD methods are particularly sim- ilar to our high-level idea by distilling feature relations in form of kNNs (Zhu et al., 2022), feature similarities (Park
B
L ==
(Cask + ACPDA)
(1)
B
where LIDA is the FDA loss for each sample i in a mini- batch {i]e of size B, and À is weighting parameter.
3.1. Feature Distribution Aligument (FDA)
Having defined the learning problem and its general loss function, we now present our FDA method in detail. During fine-tuning on Dn. we first use the pre-trained model fg and fine-tuned fo to extract batch features X E RdxB and X E Rdx F , respectively. Note X = [#1...., &g) are the pre-trained batch features with t; E Rd, while X = [æ]..... L3} are the features currently being fine- tuned with æ; E Rdd. To transfer the structural knowledge in X into X, we align the structural relations of X and X based their nearest neighbor graphs.
et al., 2019; Passalis & Tefas, 2018; Tung & Mori, 2019;
Concretely, for each pre-trained feature point á;, we main- tain its k-nearest neighbor set R, = Liا, E kNN():)} within the batch. Note |R = K, and we will detail later how to construct batches to facilitate the kNN search. This way, we obtain an instance- wise batch partition from the
Peng et al., 2019) and relative ranks (Chen et al., 2018). Our Proxy-FDA can be seen as an alternative relational KD method that distills both kNNs and similarities, and further improves with proxy learning.
3

---

ProXy-FDA
Neighbor indices by fa
Neighbor indices by. Sá
(a)
(b)
Similarity estimates
Proxies
Neighbar W similarities
Neighbor similarites byfa W
W
Proxy generator
by
Wi
(P (
X
X;
Batch
Batch
P
X P7l
X
FDA loss
ProXy-FDA loss
Figure 2: Batch construction and nearest neighbor graph transfer for our (a) FDA and (b) ProXy-FDA methods, both regularizing the fine-tuned features (X x, using the pre-trained model fa- (Proxy-) FDA loss penalizes local distribution overlap between the positives X, and negatives X," (weighted by the associated similarities w, and uे ) (with) and without using the generated proxies {P. } and their similarity estimates [w² u.,"" } Fig. 6 shows the network architecture of our proxy generator (details in Appendix B).
pre-trained model's 's perspective, leading to the positive set of neighhors X (R,) E R and negative set of non-neighbors x E Rdx(B-K-1) To form the com- plete nearest neighbor graph, we further compute the cosine similarities between pre-trained features vigj cos( ât)) for j E {1..... .). B} and j + í. Accordingly, we organize them into similarity vectors for neighbors w R-K and non-neighbors w;;- E B-K-1
way as in (Zhai et al., 2023). The above FDA loss helps to preserve local neighborhood structures in the fine-tuned feature space, without involving class labels.
Batch construction and neighborhood size K. To have a meaningful characterization and alignment of local neigh- borhood structures, we need to ensure that each mini- batch has diverse class distributions that may overlap locally in the feature space, and that a sufficient number of neighbors RR K are identified for each sample in batch.
For efficient graph matching between X and X, we choose to simply transfer the neighbor indices R, and similarities {wott , from X to X. This means neighbors in the pre-trained feature space should remain neighbors in the fine-tuned feature space. Hence among X, we similarly have positive set X X (R,) Ra where the identified neighbors are pulled together in the fine-tuned fea- ture space, and negative set X, d K-1) where non-neighbors are pushed away. On the other hand, we as- sociate the pre-trained feature similarities { u } with (X,,, "x,} to preserve fine-ge seeeggained feature neighborhood structures. We will show that transferring both the neighbor indices and similarities works better than only transferring neighbor indices. Fig. 2(a) visualizes the high-level idea.
To meet the above requirements, we sumple butch data in a class-balanced manner, with samples for each of the m classes. For a fixed batch size = m n that best fits in the available GPU memory, we choose a high value of m to increase the diversity of class concepts in batch, but at the cost of reducing the number of examples per class n. By default, m 16 and n = 4. More critically, we perform hard class mining to construct batches where samples from different classes are similar (details in Appendix A). This enables meaningful kNN search within a batch.
For the neighborhood size K, we choose > n to guaran- tee that there more than one class in any identified local feature neighborhood R, This way, each neighborhood in- cludes an adaptive selection of "small clusters" from related classes. FDA between such neighborhoods will encourage transferring high-level knowledge beyond class concepts. Fig. 5 exemplifies a common color attribute mined for two similar dog classes. Preserving this knowledge that is em- bedded in foundation models is important to prevent forget- ting during fine-tuning. Note is possible that the inter-class similarity is not high enough in RG (thus relatively low thij for inter-class samples and there are no shared properties between neighboring classes). In this case, FDA reverts back to aligning class semantics.
To capture the desired structures, we use the noise-resistant Sigmoid loss (Zhai et al., 2023) to handle a variable number of positives and negatives per batch:
CiDA ((X+t X, J: [wt wū })
log ( wis confor
X Ejvii
where Wij is weighting parameter. wy equals Wis if j E R3 (i.e., weighting by v for neighbors), and -w. j & R² (i.e,, weighting by - for non noi-neigbb.rs). 7 and are learnable parameters which are initialized similar
4

---

ProXy-FDA
3.2. ProXy-FDA
architecture of the proxy generator. At high level, con- ditioned on X and X our proxy generator is trained to predict the instance-wise proxies {P: P.P.. and their sim- ilarity estimates {aift } all at once. Finally, we use all the predictions to augment the true features (X;+, } and similarities {w,,", ~े, }. arriving at our ProXy-FDA loss for feature-space regularization (see also Fig. 2(b)):
One challenge with FDA is that the downstream dataset Da can be limited in both data size and diversity. In this case Da does not allow adequate FDA, thereby preserving only limited concepts from those learned during pre-training. To address the data challenge, one could retrieve external data. However, using external data will inevitably suffer from higher compute/memory cost as well as various levels of distributional shift. Here we propose a compute- and data- efficient approach to improve downstream data diversity and eventually improve FDA quality. Our approach involves generating synthetic features or proxies on-the-fly fhy-fly from ob- served fine-tuning data. Such generated proxies have no distributional shift since they adapt to the considered feature distribution. We leave sampling suitable external data for FDA to future work.
Lirosy-FDA = CiDDA (1XX,, P, x P. }.
f100,t 2oft 1,000 1)),
B
L =
(Cinsk + AL B
Σ
(5)
4. Experiments
learn to generate two-sets of proxies P.+ = (pi................. Pttt+ E Rdxn'* and P. [PT PI....., PPo- E [dxnn out of X;* E Rdxk and X, E Rd II respectively. n and n²- are made proportional to the size of X and x; using a scalar s, see details in Appendix D. The proxies are learned to be as diverse as possible but still lie in the corresponding true feature manifold. Fig. 5 shows that both P. and P. can synthesize unseen data/concepts. Such unseen information will provide fine- -re-irr regularization of the neighborhood boundary, and will improve FDA with richer concepts.
For proxy synthesis,
In this section, we benchmark concept forgetting and differ- ent methods in 3 settings: end-to-end, few-shot and contin- ual fine-tuning for image classification. We then move on to fine-tuning tasks of image captioning and VQA, and lastly to the application to knowledge distillation. Appendix D studies the hyper-parameters of our Proxy-FDA method, and Appendix ablates the key components of ProXy-FDA.
Compute cost. Our ProXy-FDA mainly involves FDA and proxy generation. The proxy generator is lightweight with only one attention and two convolutional layers (totalling 23 23.6k parameters), which is negligible in comparison to the foundation model size. Here we show our feature regulariza- tion process only incurs decent compute cost (on Nvidia A100 GPU). For end-to-end and few-shot fine-tuning tasks, averaged across the corresponding datasets, Proxy-FDA in- creases the fine-tuning time by 17% and 21% respectively, while FDA increases by 7% and 9%. Note Proxy-FDA does not impact the inference stage, hence we maintain the same FPS at the test time.
Following the above intuitions, we define our proxy learning loss Carnay Cp. pp where:
cou(p) an)
+b
C pp
log
lte
(3)
np+
X
jml
X
+ Lovar (P:).
contp 12))
1
4.1. End-to-End Fine-tuning
C P.
log
l+e
(4)
nd
1=1
xIEX
Datasets. We follow (Mukhoti et al., 2024) to use 10 im- age classification datasets: Stanford Cars (Krause et al., 2013), CIFAR- (Krizhevsky, 2009), DTD (Cimpoi et al., 2014), EuroS (Helber et al., 2019), GTSRB (Stal- Ikamp et al., 2012), MNIST (LeCun et al., 2010), RE- SISC45 (Cheng et al., 2017). SVHN (Netzer et al., 201 1) and ImageNet (Deng et al., 2009). These datasets include various semantic concepts, thus are perfect to benchmark forgetting of the rich pre-trained concepts.
+a- Lvar P.))
The first loss term constrains proxies P. and P. towards the feature manifolds X;+ and X,- This is achieved us- ing the binary label wi which, in case of Cp. equals if an E X and 1 if aq E X;; while in case of fp, is the opposite. The variance loss Lvar (P) maximizes proxy diversity in form of 1/d E/ max(0, 1 /Va((( + c) with being small scalar. a is weighting parameter.
Setting and baselines. The image encoder of CLIP model (ViT-B/32) is fine-tuned end-to-end on the 10 datasets. We compare with popular end-to-end fine-tuning methods all using the cross-entropy loss as Cusk The baselines include naive fine-tuning and L.--TT methods (Kumar et al., 2022).
In practice, we use Eq. (3-4) to train our proxy generator online during the model fine-tuning process. This ensures the generated proxies always adapt to the current feature distribution. Fig. 6 and Appendix B detail the network
5

---

ProXy-FDA
Table 1: Test accuracy ALP of end-to-end fine-tuned model on each dataset and its average ALP computed over other datasets. The image encoder of CLIP ViT-B/32 used here. ALP denotes the change in ALP between pre-trained and fine-tuned models on target dataset D, quantifying the level of concept forgetting. Higher ÅLP shows lower forgetting or positive forward transfer (App² 0).
Dataset
Naive End-to-End End
LP-FT
L2SP
LDIFS
FDA (ours)
Proxy-FDA (ours)
Aup
ALp 1
Aup
ALr
ALP
ALP 1
ALP
ALP f
Aur
ALP
ALP
ALp
Cars
83.48
-1.56
84,95
-0.63
83.87
0.47
85.26
-0.18
85,36
1.02
84.69
1.26
CIFAR1O
97.73
-1.60
97.71
-0.81
97.66
1.16
97.24
1.18
97.53
1.55
97.61
1.63
CIFAR100
88.60
-0.96
88.41
-0.11
86,94
1.03
88,99
0.86
88.21
1.44
88.33
1.51
DTD
77.18
-3.01
72.18
-1.76
74.63
0.01
75.27
0.53
77.22
1.04
77.28
1.19
EuroSAT
98,76
-5.72
98.87
-3.75
98.
-0.85
98.
1.32
98.53
1.61
98.63
1.74
GTSRB
98,52
-5.90
98,53
-0.94
95.00
1.18
97.81
1.27
98.16
1.58
97.79
1.69
MNIST
99.67
-8.76
99.68
-6.02
99.18
1.49
99.52
2.64
99.43
2.76
99.49
2.81
RESISC45
95.76
-3.79
95.56
-2.27
94.13
0.66
95.13
0.90
95.31
1.18
95.63
1.43
SVHN
97.30
-11.12
97,50
-8.73
96.54
-2.11
96.95
-0.29
96,96
0.67
96.65
0.92
ImageNet
82,02
-1.26
82.12
-0.87
80,78
-0.10
82.21
0.35
81.93
1.05
82.16
1.22
Mean actoss
91.90
4.37
91.55
-2.59
90.69
0.29
91.66
0.86
91.86
1.39
91.82
1.54
10 datasets
FDA
FDA
LDIFS
L2SP
LP-FT
Naive End-to-End
CIFAR10
CIFAR100
6.0
3.5
3.0
25
-2.9
-3.
-3.0
-5.0
.7.
-000
-50
-1.0
20
40
60
80
100
23
60
60
100
20
20
102
20
45
50
100
010
100
Training completion %
Training completion %
Training completion 96
Training completion %
Training completion %6
Cars
CIFAR10
CIFAR100
GTSRB
EuroSAT
12 festure distannc x109
9 feelure distance x10
L2 fostture distance x10*
focture distonce x10
1.77
1:50
1.50
feature dietance
1.50
120
0.25
1.00
1.0
1.00
0.0
0.25
0.75
0.75
0.0
0.50
0-
6.00
0.50
0.25
0.5
022
0.00
40
0.00
100
20
0.00
40
30ng
Training completion
Training completion
Training completion
Training completion
Training completion
Cars
CIFAR10
CIFAR100
GTSRB
EuroSAT
0.29
0.29
0:40
0.25
0.20
0.30
0.90
0.30
0.24
0.25
0.25
0.30
0.25
0.00
0T10
0.20
0.20
0.20
0:16
0.20
0.16
0.15
0.10
0.06
0.00
0.00
Training completion 96
Training completion
Training completion 36
Training completion %
Training completion
Figure 3: Three metrics computed over the course of model fine-tuning (CLIP ViT-B/32) on EuroSAT: ALP (Top row), L2 feature-space distance (Middle row) and distributional distance metric OTDD (Bottom row), all between pre-trained and fine-tuned models. Our (Proxy-)FDA achieves the best results in preventing concept forgetting on other datasets (highest positive ALP) without hurting the downstream performance on EuroSAT. We also observe that concept forgetting measured by ALP p is more correlated to OTDD than L2 feature distance (see text for details).
They differ in the linear head initialization, with zero-shot weights (text encodings of class name) and Linear Probe (LP) weights respectively. While L2SP (Li et al., 2018) and LDIFS (Mukhoti et al., 2024) add a point-wise regu- larization between the original and fine-tuned models in weight- and feature-space respectively. By contrast, our (Proxy-)FDA imposes a structure-wise regularization in fea-
ture space. Note except for the naive fine-tuning baseline, LP initialization is used for all methods including ours for a fair comparison of different regularization techniques.
Metrics. When fine-tuning on dataset D we report two evaluation metrics: LP accuracy ALP on the test set of Dr (i.e., the fine-tuning performance itself), and the change
6

---

ProXy-FDA
AuP in ALP between pre-trained and fine-tuned models on different dataset DD # Dn- Negative ALP indicates concept forgetting on D. while positive value indicates positive forward transfer. Clearly, the higher ALP the better. When D = Dn. ALP on 'D simply denotes the change of down- stream performance, and we expect ÅLP to increase over the course of fine-tuinng
OTDD is a better indicator of concept forgetting com- pared to the point-wise 1.2 distance. More crucially, the fact that OTDD is more correlated to forgetting than L2 distance reaffirms that having some form of structure-wise FDA can mitigate forgetting better. Finally, we note our (Proxy-)FDA is only applied on EuroSAT samples, but the mitigation of forgetting extends to all other datasets. This indicates the generalizing effect of our feature regularization method, which can preserve pre-trained knowledge without requiring third party datasets during fine-tuning.
To gain insights on what impacts the concept forgetting performance, we further monitor two distance metrics for distribution alignment during fine-tuning: point-wise L.2 distance between pre-trained and fine-tuned feature pairs, and Optimal Transport Dataset Distance (OTDD) (A (Alvarez- Melis & Fusi, 2020) that takes feature distribution structures into consideration (details in Appendix C). Between the two distance metries, OTDD is generally more suited measure the alignment quality for feature distributions with local structures as in our case.
Table 5 in Appendix shows that our benefits still hold when end-to-end fine-tuning happens with different architectures of CLIP (Radford et al., 2021), FLAVA (Singh et al., 2022), DINOv2 (Oquab et al., 2024) and MAE (He et al., 2022). ProXy-FDA consistently provides the highest ÂLP values across foundation models and architectures, achieving posi- tive forward transfer in all cases. Proxy-FDA also achieves the best ALP in many cases, which is encouraging.
Results. Table compares Alp on each fine-tuning dataset and the Arp averaged over other datasets. We observe that FDA obtains a positive average ALP for all fine-tuning tasks, thereby achieving a positive forward transfer. ProXy-FDA further improves the average ÅLP consistently. This is not the case for naive fine-tuning and LP-FT where the average ALP all negative indicating concept forgetting, Point-wise regularization methods L2SP and LDIFS obtain mostly pos- itive ALP but significantly lower than our results, highlight- ing the benefits of our structure-wis feature regularization and proxy feature generation.
4.2. Few-shot Prompt Tuning
Datasets. We follow (Zhou et al., 2022b) to use 11 datasets, consisting of a wide range of visual concepts again: ImageNet (Deng et al., 2009), Caltech101 (Fei-Fei et al., 2004), OxfordPets (Parkhi et al., 2012), Stanford- Cars (Krause et al., 2013), Flowers 102 (Nilsback & Zis- serman, 2008), Food101 (Bossard al., 2014), FGVC- Aircraft (Maji et al., 2013), SUN397 (Xiao et al., 2010), DTD (Cimpoi al., 2014), EuroSAT (Helber et al., 2019) and UCF101 (Soomro et al., 2012).
We also observe that our good performance on forgetting pre- vention does not compromise (much) the downstream fine- tuning accuracy AIp. The mean ALp (across 10 datasets) of (Proxy-)FDA is (91.82) 91.86, which is only slightly lower than that of naive fine-tuning 91.90 but outperforms all other results. Overall, our structure-wist regulariza- tion method achieves the best trade-off between concept forgetting and downstream performance. Fig. 3 (top row) exemplifies the fine-tuning task on EuroSAT, where (Proxy- FFA consistently outperforms other baselines in forgetting prevention during fine-tuning (higher ALP), but has similar performance on AT in the meantime.
Settings and metrics. Prompt tuning is adopted for parameter-effiient fine-tuning in the few-shot scenario. We consider the two settings introduced in (Zhou et al., 2022b): 1) Base-to-new class generalization within each dataset, i.e., prompt tuning on the base class split as Dn. and evaluating on the disjoint base and new class splits to obtain Anase and ANew- To quantify concept forgetting on the unseen new class split, we further report ANew as the change in ANe between pre-trained and prompt-tuned mod- els - the higher ANew the lower forgetting. 2) Cross-dataset generalization with ImageNet for prompt tuning and other 10 datasets for evaluation. Similarly, we report both the test accuracy A and accuracy change A on each dataset to quantify forgetting. For all experiments, we report results an average over three random seeds.
Fig. 3 (middle and bottom rows) shows how L2 feature dis- tance and OTDD change during EuroSAT fine-tuning. Over- all, both the distance metrics are correlated to concept forget- ting -ag-ffi fine-tuning methods with smaller L2 distance/OTDD forget less with higher ALP. while methods with a larger distance suffer more from forgetting with lower ALP The only exception to the overall trend is when we use 1.2 feature distance to compare (PrAxy-)FDA with LDIFS. We see that (Proxy-) FDA, while having larger L2 distance than LDIFS, still forgets less. On the contrary, (Proxy-) FDDA consistently gets lower OTDD. This suggests that the structure-aware
Implementation.
We apply our ProXy-FDA regularization to different prompt tuning baselines. For fair comparisons, we use the same implementation details of each baseline, including the prompt length, learning rate schedule and tuning epochs for each dataset. By default, all methods use 16 shots per class to prompt tune the CLIP model (Radford et al., 2021) with ViT-B/16.
7

---

ProXy-FDA
Table 2: Few-shot prompt tuning in the base-to-new class generalization setting (16 shots per class), Ah denotes the Harmonic mean of ABuse and AN New denotes the change in ANew between pre-trained and prompt-tuned CLIP models. Higher New shows lower level of concept forgetting on the new class split. On average, our Proxy-FDA consistently improves New for all prompt tuning methods, with competitive ABase at the same time. Full results in Table 6.
Prompt tuning without regularization
Regularization-base
CoOp
CoCoOp
VPT
MaPL
CLiPood
PromptSRC
+ PPoXX-FFD
X
/
Anss
82.69
83.16
80,47
80.36
81.61
81.55
82.28
82.74
83,91
84.33
84.26
84.47
Avg across
ANer
63.22
73.67
71.69
76.44
69.61
73.89
75.14
77.13
74.50
76.54
76,10
77.45
11 datasets
ANco
-10.99
-0.55
-2.53
2.22
4.61
-0.33
0.92
2.91
0.28
2.33
1.88
3.23
An
71.66
78.13
75.83
78.35
75.14
77.53
78.55
79.84
78.93
80.25
79.97
80.81
Anow (avg over datasets)
Brow (avg over 11 datasets)
Decw (svg over datasets)
-50% fower
2
>50% fewer
-10
.2
-3
75% fewer
PromptSRG-Praxy-F
PromptSRC+FDA
-16
CoOp=Proxy- FDA
-6
PrommptSC-~Prxxy
-7
FD-Allgn
CDDp+FDA
E2E fine-turing
CoOp
PrompiSRC
WISE-FT
-10
0
2
4
8
16
2
16
0
2
8
16
(a)
(b)
(c)
shots per class
shots per class
# shots per class
Figure 4: (a-b) The average ANew with varying number of shots per class for prompt tuning in the base-to aase-to-new new setting, FDA achieves higher gains over the baselines in low-data regime, and our proxy learning further improves data efficiency. (c) TrompuSRC+Proxy-FD.. scales better with data than end-to-end fine-tuning and its improved variants (FD-Align and WiSE-FT) in the few-shot setting.
Results. In Table 2, report results in the base-to-oe nownew setting. Proxy-FDA is applied to two categories of methods: 1) regularization-free free prompt tuning baselines, which learn text prompts (CoOp (Zhou et al., 2022a), CoCoOp (Zhou et al., 2022b)), image prompts (VPT (Jia et al., 2022)) or both (MaPLe (Khattak et al., 2023a)). 2) regularization- based prompt learners. CLiPood (Shu et al., 2023) main- tains a weighted ensemble of the pre-trained and fine-tuned models. offte-of-thh-art PromptSRC (Khattak et al., 2023b) combines the ensembling strategy with both feature- and logit- level regularization between the original and fine- tuned models (but in a point-wise manner).
to the two regularization methods and can further improve them in all metrics.
Fig. 4 shows the superior data efficiency of ProXy-FDA when lowering forgetting for few-shot prompt tuning. In the base-to-new setting, we vary the amount of tuning data and find that the ANew gain of FDA increases with less data. Meanwhile, our proxy learning component further improves data efficiency, often matching the FDA performance on half the data. Fig. 4(c) also shows the benefits of (Proxy- )FA over end-to-end fine-tuning and its improved variants FD-Align (Song et al., 2023) and WiSE-FT (Wortsman et al., 2022b) - in lama-eimited regimes.
We can see from Table 2 that, averaged across 11 datasets, Proxy-FDA consistently improves the ANew of all regularization-free baselines, sometimes by a large mar- gin (10.45 for CoOp), with competitive ABase at the same time. The gains in ANEW translate to gains in ANew. indi- cating the utility of ProXy-FDA in lowering forgetting for few-shot settings. The per-dataset results in Table 6 (in Ap- pendix) show that ANew sees particularly large gains on 3 semantically distant datasets (DTD, EuroSAT and UCF101), thanks our strong capability of preserving pre-trained knowledge. Overall, Proxy-FDA boosts the Au of MaPLe to 79.84, being already better than or on par with that of the regularization methods CLIPood (78.93) and Prompt- SRC (79.97). Encouragingly, Proxy-FDA is complementary
In the Appendix, Table further shows results under the cross-dataset generalization setting. Proxy-FDA is shown to prevent concept forgetting consistently, with uniformly increased A on target datasets and good trade-off with A on the source dataset ImageNet. Table 8 shows our advan- tage over more recent prompt tuning methods that benefit from either LLM or advanced regularization techniques.
4.3. Continual Fine-tuning
Finally, we perform continual fine-tuning and see whether we can learn a sequence of downstream tasks without for- getting concepts. We follow (Mukhoti et al., 2024) train on three task sequences:
10 RESSSS5
8

---

ProXy-FDA
Table 3: Continual fine-tuning: test accuracy ALP and ALP for models fine-tuned on three task sequences. The first 3
TOWs show performance on fine-tuned tasks and the 4th row shows performance averaged on 6 other datasets.
Fine-tune
Evaluation
Naive End-to- Bad-to-End
LP-FT
L2SP
LDIFS
FDA (ours)
Proxy- -DA (ours)
dataset
dataset
Air
Air f
Aup
ALr
Aur
Air f
ALp
ALP f
Aup
Aur
Air
AuP
SVHN
90.29
-7.13
90.97
-6.46
91.93
-4.3
96.68
-0.41
96.77
0.61
96.72
0.93
SVHN-
1O- RESISC45
CIFARIO
95.25
-2.31
96.31
-1.57
97.26
-0.25
97.41
-0.21
97.13
0.57
97.29
1.02
RESISC45
95.30
4.00
94.29
2.98
93.44
2.16
95.00
3.70
95.22
4.14
95.38
4.22
Others
80.91
-5.08
82.13
-4.24
86.89
-0.01
87.08
0.10
87.21
0.76
86.95
1.08
SVHN
90.05
-7.28
94,42
-2.73
90.42
-6.12
96.32
-0.65
96.18
0.63
96.43
0.71
SVHN- + CIFAR100-> RESISC45
CIFAR100
81.08
-7.18
82.63
-3.04
85.72
-0.88
86.54
-0.30
86.33
0.72
14
0.85
RESISC45
95.40
4.13
93.81
2.51
93.21
1.90
95.11
3.83
95.32
3.95
95.46
4.01
Others
83.76
-4665
85.14
-4.02
89.04
-0.37
89.12
-0.23
89.02
0.68
89.09
0.96
SVHN
95.93
-1.45
96.58
-0.76
95,98
-0.44
96.90
-0.17
96.74
0.79
96.91
0.94
SVHN-> Cars - RESISC45
Cars
76.96
-4.18
71.60
-8.36
81.82
-0.40
84.23
0.47
84.38
1.14
84.32
1.36
RESISC45
95.17
3.89
94.35
3.00
93.43
2.13
95.27
3.73
95.12
3.92
95.23
4.07
Others
83.38
-4.93
84.39
-4.51
87.15
-1.67
89.39
0.23
89.54
0.96
89.67
1.17
and SVHN RE- SISC45. Table 3 shows our FDA and Proxy-FDA methods progressively improve the ALP for each task sequence, both achieving positive forward transfer with all positive Aur values. Proxy-FDA always attains the highest ALP (except on RESISC45 in the second sequence), while still remaining competitive in ALP. Table 9 and 10 in Appendix F show our benefits over popular continual learning baselines for both the 3-task setup and the class-incremental setting on Split ImageNet-R (Wang et al., 2022a).
is shown to effectively preserve the concepts in pre-trained vision foundation models. One potential societal impact is that, when the pre-trained concepts reflect (unintentional) biases, our regularization method could inherit or amplify those biases in fine-tuned features. As result, one may ob- serve perpetual unfair or discriminative outcomes in down- stream tasks and more critical applications such as Al-driven planning and decision-making
References
4.4. Applications Beyond Classification
Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., John- son, M., Batra, D., Parikh, D., Lee, S., and Anderson, P. NoCaps: novel object captioning at scale. In ICCV, 2019.
Appendix G shows that the benefits of Proxy-FDA hold for fine-tuning tasks beyond classification. We consider the vision-language tasks of image captioning and VQA, where Proxy-FDA outperforms baselines in mitigating forgetting. We further show ProXy-FDA is applicable knowledge distillation and achieves quite promising performance.
Melass D. and Fusi, N. Geometric dataset distances via optimal transport. In NeurlPS, 2020.
Bossard, L., Guillaumin, M., and Gool, L. V. Food-101- mining discriminative components with random forests. In ECCV, 2014.
5. Conclusion
Chen, Y., Wang, N., and Zhang, Z. Darkrank: Accelerating deep metric learning via cross sample similarities transfer. In AAAI, 2018
This paper introduces Proxy-FDA, novel feature-space regularization method that preserves concepts during fine- tuning. The core idea is align the local structures of pre-trained and fine-tuned feature distributions with learned proxies. A structure-awan distributional distance metric is used to assess the feature alignment quality, demonstrating a strong correlation with concept forgetting. Our approach achieves sftte-ofthhe-a results in mitigating forgetting in various fine-tuning menitunin settings and across different tasks.
Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceed- ings of the IEEE, 105:1865-111 2017.
Cimpoi, M.. Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild, In CVPR, 2014.
Impact Statement
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
The main contribution of this work is a new feature-space regularization method for robust fine-tuning. Our method

---

ProXy-FDA
Du, X., Wang, Z., Cai, M., and Li, Y. Vos: Learning what you don't know by virtual outlier synthesis. In ICLR, 2022.
Huang, T., You, S., Wang, F., Qian, C., and Xu, C. Knowl- edge distillation from stronger teacher. In NeurIPS, 2022.
Fei-Fei, L.,, Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In CVPR workshop, 2004.
Jia, M., Tang, L.., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In ECCV, 2022.
Jung, H., Ju, J., Jung, M., and Kim, J. Less-forgetting learning in deep neural networks. arXiv preprint WrXiv:667.00012, 2016.
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., and Qiao, Y. CLIP-Adapter: Better vision- language models with feature adapters. arXiv preprint arXiv:2110.0444..22 2021.
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. MaPLe: Multi-modal prompt learning. In CVPR, 2023a.
Goyal, S., Kumar, A., Garg, S., Kolter, Z., and Raghunathan, A. Finetune like you pretrain: Improved finetuning of shoo vision models. In CVPR, 2023.
Khattak. M. U., Wasim, S. T., Naseer, M., Khan, S., Yang, M.-H., and Khan, F. S. Self-regulating prompts: Foun- dational model adaptation without forgetting. In ICCV, 2023b.
Goyal, Y., Khot, T., Summers-Stay. D., Batra, D., and Parikh, D. Making the V VQA matter: Elevating the role of image understanding Visual Question An- swering, In CVPR, 2017.
khattak, M. U., Ferjad, M., Muzzamal, N., Gool, L. V., and Tombari, F. Learning to prompt with text only su- pervision for vision-language models. arXiv preprint arXiv:2401.02418. 2024.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.
Kim, S., Kim, D.,, Cho, M., and Kwak, Proxy anchor loss for deep metric learning. In CVPR, 2020.
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., and Girshick, R. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Krause, J., Stark, M., Deng. J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In ICCV workshops, 2013.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 2019.
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Kumar, A., Raghunathan, A., Jones, R. M., Ma, T., and Liang, P. FFin-ttning can distort pretrained features and underperform -it-o---iitribttion.. In ICLR, 2022.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of out-of-distributie generalization. In ICCV, 2021a.
Lavoie. S., Kirichenko, P., Ibrahim, M., Assran, M., Wildon, A. G., Courville, A., and Ballas, N. Modeling caption diversity in contrastive lisionllanguage pretraining, arXiv preprint arXiv:2405.00 00044000, 2024.
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In CVPR, 2021b.
Hessel, J.. Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y. CLIPScore: A reference-free evaluation metric for image captioning. In EMNLP, 2021.
LeCun, Y., Cortes, C., and Burges, C. Mnist hand- written digit database. ATT Labs [Online]. Available: http://yam.leccn. http://haanleec...cm//xddnnnttt 2, 2010.
Li, X., bias for transfer learning with convolutional networks. In ICML, 2018
Grandvalet, Y., and Davoine, F. Explicit inductive
Hou, S., Pan, X.. Change Loy, C., Wang, Z., and Lin, D. Lifelong learning via progressive distillation and retro- spection. In ECCV, 2018.
Li, Z. and Hoiem, D. Learning without forgetting. TPAMI, 2017.
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. Mo- bileNets: Efficient convolutional neural networks for mo- bile vision applications. arXiv preprint arXiv:1704.0486. 2017.
Lin, T., Maire, M., Belongie, S. J., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft COCO: common objects in context, In ECCV, 2014.
10

---

ProXy-FDA
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft. arXiv preprint 1iXvv:330651555 2013.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? In JCML, 2019.
Merullo, J., Castricato, L.,, Eickhoff, C., and Pavlick, E. Linearly mapping from image to text space. In ICLR, 2023.
Roth, K., Vinyals, O., and Akata, Z. Non-isotropy regular- ization for proxy-based deep metric learning. CVPR, 2022.
Miles, R., formation theoretic representation distillation. In BMVC, 2022.
A., and Mikolajezyk, K. In-
Shu, Y., Guo, X., Wu, J., Wang. X., Wang. J., and Long, M. CLIPood: Generalizing clip to out- -itribsbbiuuions. In ICML, 2023.
Movshovitz-Attian Y., Toshev, A., Leung. T., loffe, S., and Singh, S. No fuss distance metric learning using proxies. In ICCV, 2017.
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W. Rohrbach, M., and Kiela, D. FLAVA: A foundational language and vision alignment model. In CVPR, 2022.
Mukhoti, J., Gal, Y., Torr, P., and Dokania, P. K. Fine-tuning can cripple your foundation model; preserving features may the solution. TMLR, 2024. ISSN 2835-8856.
Smith, J. S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P.,
Kim, D., Arbelle, A., Panda, R., Feris, R., and Kira, Z. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In CVPR, 2023.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop, 201 1.
Song, K., Ma, H., Zou, B., Zhang, H., and Huang, W. FD- align: Feature discrimination alignment for fine-tuning pre-trained models in few-shot learning. In NeurIPS, 2023.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over large number of classes. In ICVGIP, 2008.
Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 12Xiv:122..04000 2012.
Oquab, M. Darcet, T., Moutakanni, T. Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rab- bat, M., Sharma, V., Synnaeve, G., Xu, H. Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. DINOv2: Learning robust visual features without super- vision. TMLR, 2024. ISSN 2835-8856.
Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. Man vs. computer: Benchmarking machine learning algo- rithms for traffic sign recognition. Neural Networks, 32: 323-332, 2012.
Tao, L., Du, X., Zhu, J., and Li, Y. Non-parametric outlier synthesis. In ICLR, 2023.
Park, W., Kim, D., Lu, Y., and Cho, M. Relational knowl- edge distillation. CVPR, 2019.
Thengane, V., Khan, S., Hayat, M., and Khan, F. Clip model is an efficient continual learner. arXiv preprint arXiv:2210Aa
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In CVPR, 2012.
2022.
Passalis, N. and Tefas, A. Learning deep representations with probabilistic knowledge transfer. In ECCV, 2018
Tian, X., Zou, S., Yang, Z., and Zhang, J. ArGue: Attribute- Guided Prompt Tuning for Vision-Languaae Models. In CVPR, 2024.
Peng, B., Jin, X.. li, D., Zhou, S., Wu, Y., Liu, J.. Zhang, Z., and Liu, Y. Correlation congruence for knowledge distillation. In ICCV, 2019.
Tung, F. and Mori, G. Similarity-preservinng knowledge distillation. In ICCV, 2019.
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Vedantam, R., Zitnick, C. L.,, and Parikh, D. CIDEr: Consensus- based image description evaluation. In CVPR. 2015.
Verma, V., Lamb, A., Beckham, C., Najafi, A., Mitliagkas, I., Lopez-Paz, D., and Bengio, Y. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019.
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In CVPR, 2017.

---

ProXy-FDA
Villani, C. Optimal Transport: Old and New. Springer Berlin Heidelberg, 2008. ISBN 9783540710509.
Zheng, Z., Ma. M., Wang, K., Qin, Z.,, Yue, X., and You, Y. Preventing zero-shot transfer degradation in continual learning of vision-language models. In ICCV, 2023.
Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning ro- bust global representations by penalizing local predictive power. NeurIPS, 2019.
Zhou, K. Yang. J., Loy, C. C., and Liu, Learning to prompt for vision-language models. IJCV, 2022a.
Wang, L. and Yoon, K.-J. Knowledge distillation and student-teacher learning for visual intelligenne:: A review and new outlooks. TPAMI, 44:3048-300 3068, 2021.
Zhou, K., Yang, J., Loy. C. C., and Liu, Z. Conditional prompt leannng for vision- -ainn-lan models. In CVPR, 2022b.
Wang, M. and Deng, W. Deep visual domain adaptation: A survey. Neurocomput, 312(C):135- 2018.
Zhu, P., Cai, Z.,, Xiong, Y., Tu, Z., Goncalves, L.,, Ma- hadevan, V., and Soatto, S. Contrastive neighborhood alignment. arXiv preprint arXi::200.01922, 2022.
Wang, Y., Cheng, L., Duan, M., Wang, Y., Feng, Z., and Kong, S. Improving knowledge distillation via reg- ularizing feature norm and direction. arXiv preprint arXiv:2305. 2023.
Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y., Ren, X., Su, G., Perot, V., Dy, J., al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, 2022a.
Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. Learning to prompt for continual learning. In CVPR, 2022b.
Wortsman, M., Ilbarco, G., Gadre, S. Y.. Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Korbblith, S., and Schmidt, L. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022a.
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Korbblith, S., Roelofs, R., Gontijo-Lopes, R., Hajishirzi, H., Farhadi, A., Namkoong, H., and Schmidt, L. Robust fine-tuning of zero-shot models. CVPR, 2022b.
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
Zang, Y., Goh, H., Susskind, M., and Huang, C. Over- coming the pitfalls of vision-language model finetuning for OOD generalization. In ICLR, 2024.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In ICCV, 2023.
Zhang, G., Wang, L., Kang, G., Chen, L., and Wei, Y. SLCA: Slow learner with classifier alignment for continual leamn ing on pre-trained model. In ICCV, 2023.
Zheng, K. and Yang, E.-H. Knowledge distillation based on transformed teacher matching. ICI.R, 2024.
12

---

ProXy-FDA
French buildog: black
Miniature poodle: brown
Unseen class
NNs of negative proxy
NNs of positive proxy
French buildog: white
Miniature poodle: white
Figure 5: t-SNE visualization of the local feature neighborhood (circled) on ImageNet for the pre-trained CLIP ViT-B/16 model. In this neighborhood, observe the same white color from two dog breeds "French bulldog" and "Miniature poodle". Preserving CLIP's common-sense knowledge (in this case the color attribute shared across different classes) using FDA maintains the generalizability of foundation models. On the other hand, the generated proxies include diverse information from both seen and unseen (e.g.. "Malamute") classes that can regularize the neighborhood boundary and further improve FDA. The synthesized seen/unseen class data are illustrated by kNN retrieval from the base/new class splits of ImageNet when fine-tuning on the base only.
P+
Pooling
Pooling
Pooling
Pooling
Conv layer
Conv layer
Conv layer
Conv layer
Attention
Attention layer
mask
Figure 6: Efficient architecture of our proxy generator that generates dynamic proxies synthetic features.
A. Hard Class Mining
As mentioned in the main text (Section 3.1), we perform hard class mining in the mini-batch to facilitate the modeling and alignment of local neighborhood structures. The high-level idea of hard class mining is greedily select class distributions that are close one another. More specifically, we construct mini-batch in the following way:
1. Randomly choose a large number of classes C' >> m; for each class, randomly sample n examples to extract their feature embeddings using both fa and fo-
2. Sample seed class randomly from the C' classes. Then greedily add a new class that has the largest class-wiss wise loss Σド LiDA (Eq. (2)) w.r.t. the selected classes till reach m classes. Note in this greedy process, set the neighborhood size K when computing CEDA
3. Construct batch with the selected m classes, each with examples.
13

---

ProXy-FDA
B. Efficient Architecture of Instance-wise Proxy Generator
Fig. 6 shows the network architecture of our proxy generator that is trained online using Eq. (3-4). The input X; and X; first go through an attention layer to model the global context within each set and fuse features thoroughly. Attention mask used to ensure the independence between the two sets, Next, we dynamically pool the intermediate features X k K and T E Rdx(B-K-I) via learned pooling functions, as summarized below. Through such pooling, we can predict proxies {P." P. and their similarity estimates { x
all once.
Predict pooling weights:
S = -+((((X)) E R xxxn*
ST = 1 (x,(1,) E (RB---I)xx
(6)
Pooling in matrix form:
P X Ss Rd
P. =x;- S E Bdxxn
(7)
iº S T wb B w. E ER
wo = =s," w; E R
(8)
where
w, E ER B-B--- K
Note both (-) and h h-(-) are implemented by two convolutional layers, but with different output channel sizes (n* and n²- respectively). The output pooling weights S and S. are softmax-nnomalizedd leading to convex combinations of features and feature similarities during the pooling stage. This cases training of pooling functions and makes sure the pooled results are valid (especially the pooled similarity estimates).
C. Distributional Distance Metric: OTDD
To measure FDA quality, there are many distance metries for distribution alignment. Here we choose the distributional distance metric based on Optimal Transport Dataset Distance (OTDD) (Alvarez-Meli, & Fusi, 2020). OTDD is especially suited to measure the alignment quality of feature distributions with local structures, because this distance metric takes both the label distribution and clustering structure of the feature distributions into consideration.
Specifically, OTDD uses the feature and label distributions (x, y) x> compute the distance between two datasets. Given that the source and target datasets may have different label sets, the high-level idea of OTDD is represent each class label as a distribution over the in-class features. This transforms the source and target label sets into the shared space of feature distributions over X. In our context of model fine-tuning, we have pre-trained features á and fine-tuned features æ that are likely shifted from æ. They form the source and target feature distributions respectively, and have different labels 9 and y (details later). Then we can define the label distance Dy (Ū. y) using the p- Wasserstein distance associated with the 1.2 distance || - 2112 in X. This enables one to measure the distributional difference in X' )::
Dxxy ((t. 1), ()) = (Dx(# - x)²² + 9))
(9)
Please refer to (Alvarez-Melis & Fusi, 2020) for the exact formulation. To capture the clustering structure of both the pre-trained and fine-tuned feature distributions, we perform K-Means clustering per class on each feature distribution. This results in pscudolabels & and y that are more fine-grained than class labels for OTDD computation.
D. Analysis of Hyper-parameterrs
Hyper- paramerssss Fig. 7(a) shows Proxy-FDA approach benefits from relatively large batch size B to preserve meaningful structures of feature neighborhoods. Performance decreases when B < 64; when B grows larger than 64, performance seems quite robust to varying batch size. By default, we set B = 64 that best fits in our GPU memory.
Based on the hard class mining strategy (Section A), we construct a mini-batch with m = 16 hard-mined classes, each with n class samples. Note in few-shot settings, each class may not have enough data (< 4) for sampling, e.g., only or shots are available per class. In this case, we perform random data augmentation guarantee n = 4 samples per class. On the other hand, relatively large m ensures diverse class distributions in batch, which allows better characterization of local feature neighborhoods. Diverse classes also allow pooling rich proxies from them, resulting in unseen data variations or new class concepts to further improve FDA.
Our Proxy-FDA method has two key hyper- parameters: the neighborhood size K > n and a scalar s. The latter makes the number of positive proxies nº = s- K and negative proxies n² = 5. (B - K - -1) proportional to the set size of the true positives X E R and true negatives X,- E Rdx
14

---

ProXy-FDA
(a) AH vs. batch size B
(b) AH vs. fixed across datasets
AH (PromptSRC+Piox (FD)Y-DDD)
AH (PromplSRC+ ProcC--PP------- FDA))
AH (CoOp+Proxy-FDe (DN)
79.0
81.1
AH AH (CoOO--rrxyy-FA)
79.0
81.1
78.0
80.9
78.0
80.9
77.0
80.7
77.0
80.7
76.0
80.5
76.0
80.5
75.0
80.3
75.0
80.3
32
40
48
56
84
72
80
0.5n
2n
3n
4n
B
(c) per dataset
PromptSRC-Proxy FDA -DA
4n
4n
CoOp-Proxy-FDD FDA
3n
2n
2n
ImageNet
Caltech101
OxfordPets
StenfordCars
Flowers 102
Food101
FGVCAiraaat
SUN397
DTD
EuroSAT
UCF101
(d) AH vs. proxy percent
AH (PromptSRC+Prox -DAA)
78.5
81.0
AH (CoOp+Praxy-FDD
80.9
77.5
80.8
77.0
80.7
76.0
80.5
10%
20%
30%
40%
50%
80%
Figure 7: Sensitivity analysis for hyper-parameters: (a) batch size B, (b) neighborhood size K that is fixed across datasets, (c) optimal K per dataset, and (d) scalar that decides the percent number of generated proxies compared that of real samples. Analysis is performed for few-shot prompt tuning in the base-to-new setting (16 shots per class). We report the AH averaged across 11 datasets, when applying Proxy-FDA to two representative baselines CoOp and PromptSRC. Note Au the Harmonic mean ABuse (representing prompt-tuning accuracy itself) and ANew (representing generalization and can derive Aeew). Hence Ay is ideal for hyper-paramede sweeping since Ah denotes a trade-off between downstream accuracy and concept forgetting (ANew).
The intuition of setting K > n is to identify sufficient neighbors from more than one class, for meaningful FDA between similar clusters of related classes. Nevertheless, the exact value of K is varied as function of dataset distribution, as each dataset has different levels intra- and inter-class variation. In practice, we pick the best K per dataset from (n. 2n, 3n, 4n). Fig. 7(b) shows how performance generally varies with K when K fixed across 11 datasets. We see that 2n works best, while noticeably hurts performance when K < n, confirming our intuition above. Hence stick to the constraint of K for per-dataset K selection (Fig. 7(c)).
On the other hand, the scalar is set to 0.4 by default. This leads to virtual batch size of around 90 (increased from 64). The virtual batch now consists of true and synthetic features for FDA. Fig. 7(d) shows the sensitivity analysis for
Lastly, the weighting parameter for Lyar (Eq. (3-4)) is fixed at a = 5 for experiments. We observe no meaningful improvements via more careful tuning a. The weighting parameter used to balance the task loss against our regularization loss (Eq. (1) and (5)). We tune on held-out validation set of each dataset.
15

---

ProXy-FDA
Batch sampling strategy
PProx generatori architecture
Proxy generation algorithm
More batch data ivs. using proxies
strategy
Ah (Promp:SRC+ Prox-ss-----P----PI -FDA)
AH (CoOp+Proxx-FF)) (DDYy-DDA)
79.0
81.1
78.0
80.9
77.0
80.7
76.0
80.5
75.0
80.3
Proxy-FDA
Entropy
No neighbar similarities
Attention free
Random interpolation
VOS outlier NPOS outlier,
Larger batch w/o proxies
Random
(Default)
based
synthesis
synthesis
Figure 8: Ablating the Proxy-FDA components based on few-shot prompt tuning the base-to-new setting (16 shots per class). We report AH averaged across 11 datasets, when applying Proxy-FDA two representative baselines CoOp and PromptSRC. Note Au is the Harmonic mean of ABase (representing prompt-tuning accuracy itself) and ANew (representing generalization and can derive ANew). Hence Au is ideal for ablation studies since An denotes a trade-off between downstream accuracy and concept forgetting (ANew). For the proxy generation strategy, we compare with random linear interpolation (Verma et al., 2019) and outlier feature synthesis methods VOS (Du et al., 2022) and NPOS (Tao et al., 2023).
Table 4: Quantifying proxy diversity using the variance loss in Eq. (3-4). Particularly, we report the diversity metric as the average standard deviation term 1/d Ej= / (Var(P); in the variance loss: higher value indicates larger proxy diversity. To further aggregate the metrics of the positive and negative proxies (P, P. ). we take their mean and compute its moving average till fine-tuning is completed. We compare the aggregated diversity metric of all the proxy generation methods as ablated in Fig. 8.
Proxy generation (ours)
Random interpolation
VOS
NPOS
Diversity metric x 10
3.14
2.89
1.53
1.72
E. Ablating ProXy-FDA Components
Fig. 8 includes ablation studies on the key components of Proxy-FDA, in the few-shot prompt tuning setting.
Batch sampling strategy. We start with comparing the default hard class mining method with random class sampling, Their considerable performance difference shows that hard class mining is crucial. Indeed, one can better model the nearest neighbor graphs from close class samples, which facilitates the following graph matching for FDA purpose. We further compare with an entropy-based batch sampling strategy that prioritizes similar class samples simply by low entropy. This sampling strategy found less performant, likely because entropy cannot characterize sample similarity adaptively as a function of current feature distribution structure. As result, such batch sampling criterion is decoupled with the structural FDA within sampled batch. While our default strategy samples similar classes directly using FDA loss, which could adapt to the feature distribution structure, and is coupled with FDA in batch.
FDA strategy. One may wonder what if only align the neighbor indices during FDA, without considering the neighbor similarities (i.e., keeping ü. 1)) We see that this baseline leads to large performance drop, demonstrating that both neighbor indices and similarities are indispensable for effective FDA.
Proxy generator architecture. We first note that our proxy generator is learned to produce unseen data out of diverse feature combinations within the positive set X negative set X;- The attention layer helps to achieve this goal by modeling the global context among all input features with pairwise attention. Convolutional layers, however, only have local receptive fields and have to rely on pooling operations to capture long-range dependencies. Here we compare with an attention- free architecture that has the attention layer replaced with convolutional plus pooling layers - the resulting proxy generator maintains a similar parameter count. The attention freee architecture is observed to achieve consistently lower performance, likely due to the lower quality of generated proxies.
16

---

ProXy-FDA
Table 5: Test accuracy ALP of end-to-end fine-tuned model on ImageNet and its average ALP computed over 5 datasets (DTD, EuroSAT, GTSRB, RESISC45 and SVHN). We study different architectures of CLIP (Radford et al., 2021), 1), FLAVA (Singh et al., 2022), DINOv2 (Oquab et al., 2024) and MAF (He et al., 2022). ALP denotes the change in ALP between pre-trained and fine-tuned models on target dataset, quantifying the level of concept forgetting. Higher ALP shows lower forgetting or even positive forward transfer (ALP 0). Note we initialize the model's linear head with zero-shot weights for naive fine-tuning, and with Linear Probe (LP) weights for all other methods including ours. The initialized zero-shot weights are the text encodings of class name for CLIP and FLAVA, and random weights for DINOv2 and MAE.
Model
Architocture Naive End-to-End
LP-FT
L25P
LDIFS
FDA (ours)
Proxy-FDA (ours)
Aip
Aip f
Aip
ALP
Aip
Aup 1
Aup
Arp
A
ALP
Aip
A
ResNet-50
78.39
-4.01
78.45
-3.40
76.13
-1.54
78.16
-0.11
78.43
0.62
78.58
0.89
ViT-B/32
82.02
-3.02
82.12
-2.17
80.78
-0.88
82.21
0.10
81.93
0.81
82. 16
1.15
CLIP
VIT-B/16
85.21
-2.92
85.36
-1.73
82.19
-0.74
85,31
0.16
85.41
0.92
85.40
1.03
VIT-I/14
87.88
-2.33
87.91
-1.52
86.87
-0.43
87.85
0.22
87.99
1.02
87.96
1,28
FLAVA
VIT-B/16
81.18
-3.94
81.36
-3.04
80.11
-1.10
81.61
0.04
81.47
0.61
81.59
0.96
VIT-B/14
85.32
-2.71
85.48
-1.86
84.50
-0.66
86.02
0.06
86.23
0.68
86.34
0.85
DINOv2
VIT-L/14
87.60
-1.92
87,90
-1.40
87.02
-0.19
87.91
0.13
87.87
0.77
87.71
0.94
ViT-B/16
83.57
-5.10
83.81
-4.36
82.84
-3.03
83.76
-0.94
83.73
-0.08
83.94
0.39
MAB
ViT-L/16
85,86
.4.26
86.04
-3.59
85.10
-1.82
85,90
-0.12
85.86
0.79
85.67
0.94
Proxy generation algorithm. We compare with three baselines. One simple method ís based on linear interpolation between random feature pairs from both X and X, Feature similarity estimates are interpolated in the same way. We see random interpolation obtains inferior performance than our learning-based approach. This is because our approach can learn to synthesize informative proxies that best help FDA: the diverse proxies can not only enrich data but also refine the decision boundary between positive and negative feature manifolds. This not possible with random interpolation. On the other hand, the parametric VOS and non-parametric NPOS methods learn to synthesize outlier features only in likelihoood regions (often around decision boundaries between classes). The two methods are observed to achieve even worse results than random interpolation. We conjecture that this is because outliers are not able to encode diverse unseen data/concepts that are crucial for improving FDA.
Lastly, we quantify the proxy feature diversity in Table 4 using our variance loss. Interestingly, is observed that the diversity metric of a proxy generation method highly correlates with its performance: our proxy generation method outperforms random interpolation in both proxy diversity and final accuracy. This trend also holds when comparing our method with VOS/NPOS.
Using more batch data or proxies. To further quantify the effect of proxy learning that virtually increases the batch size B from 64 to around 90, we compare with FDA simply on larger batch with a similar number of true feature points. Specifically, we construct the batch with m = 22 hard-mined classes, each with n = 4 examples. Hence the batch size is comparable to that of Proxy-FDA, but without proxies. We observe from Fig. 8 that simply using a larger batch size does not perform as well. Instead, it is worth using our proxy generator to increase data diversity with only a small overhead.
F. More Results
End-to-end fine-tuning. Table shows ImageNet fine-tuning results with different foundation models and architectures. We see that both FDA and Proxy-FDA consistently improve the ALP over other baselines, with Proxy-FDA offering the highest ALP values. This comes with competitive downstream accuracy Aur ImageNet. Notably, our obtained ALp values are mostly positive, with sole exception of MAE model (ViT-B/16 architecture) when fine-tuned using FDA. This indicates that we can achieve positive forward transfer in most cases and otherwise minimized concept forgetting.
Few-shot prompt-tuning. Table 6 lists the full results of prompt tuning on each of the 11 datasets under the base-to- new class generalization setting. Table 7 shows results under the cross-dataset generalization setting, i.e., quantifying generalization from ImageNet to 10 target datasets. In both settings, Proxy-FDA is plugged into different prompt tuning baselines. Proxy-FDA is observed to reduce concept forgetting consistently on unseen data with comparable performance on seen data.
17

---

ProXy-FDA
We further compare with more recent prompt tuning methods in Table 8. Comparisons are conducted under the base-to- new class generalization setting, and additional domain generalization setting. In the latter setting, we prompt tune on ImageNet (16 shots per class) and evaluate OOD generalization on ImageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang et al.. 2019), ImageNet-A (Hendrycks et al., 2021b) and ImageNet-R (Hendrycks et al., 2021a) with different types of domain shift. The compared methods include ProTexx (khattak et al., 2024) and ArGue-N (Tian et al., 2024) that use L.MSs to distill language priors into the learned prompts, as well as more related regularization methods OGEN (Zang et al., 2024) and CLAP (Lavoie et al., 2024). OGEN regularizes the prediction probabilities with an improved Mean Teacher, while CLAP regularizes the class prototypes (i.e., class-wise feature means) for linear probing.
Table 8 shows that structure-wise feature regularization method Proxy-FDA outperforms OGEN and CLAP in all metrics under the considered settings. Proxy-FDA achieves particularly large gains in generalization performance on the new classes or new domains, maximizing the positive forward transfer with higher A When compared to ProText and ArGue-N using external LLMs, our approach is L..M-free but achieves on-par or even better performance for both prompt-tuning and OOD generalization.
Continual fine-tuning, Table in the main paper compares our method with robust fine-tuning methods in the 3-task setting. In the same setting, Table 9 compares our method with 5 classic continual learning methods: LwF (Li & Hoiem, 2017), LFL (Jung et al., 2016), iCaRL (Rebuffi et al., 2017), Distillation + Retrospeccion (D+R) (Hou et al., 2018) and ZSCL (Zheng et al., 2023).
Table 10 compares our method with recent continual learning methods on the class-incremental learning benchmark Split ImageNet-R. This benchmark divides the 200 classes from ImageNet-R into 10 tasks with 20 classes per task. The compared methods include LDIFS as well as L2P (Wang et al., 2022b), DualPrompt (Wang et al., 2022a), CODA- Prompt (Smith et al., 2023), Continual-CLIP (Thengane et al., 2022) and SLCA (Zhang et al., 2023). All methods use the same training (24,000) and testing (6,000) images. To further ensure fair comparisons, we follow the widely-adopted implementation: fine-tuning for 50 epochs using the Adam optimizer with B1 = 0.9 and B. 0.999. The initial learning rate is Le 4, and we use a cosine learning rate scheduler as in (Mukhoti et al., 2024).
In both Table 9 and 10, our (Proxy-)FDA method outperforms all other methods in preventing forgetting. At the same time, (Proxy-)FDA is able to achieve the best fine-tuning performance.
G. Applications Beyond Classification
Fine-tuning for image captioning& & VQA. Here we test our (Proxy-)FDA method can address the forgetting issue for fine-tuning tasks beyond classification. Specifically, we consider the foundation model CLIP and fine-tune for two vision-language tasks: image captioning (COCO (Lin et al., 2014) and NoCaps (Agrawal et al., 2019) datasets) and Visual Question-Answering (VQA2 dataset (Goyal et al., 2017)). The baseline approach that enables CLIP to perform such vision-language understanding and generation tasks is LiMBeR (Merullo et al., 2023). LiMBeR maps the CLIP image features to the text space of generative language model, using only a linear projection that aligns the image and text spaces. As result, although the image encoder and language model are both frozen, LiMBeR allows CLIP to flexibly caption an image or perform some task relating to it.
For the ease of comparisons, we follow LiMBeR use the same language model and image encoder (RN50x16) of CLIP. Starting with LiMBeR, we perform fine-tuning on COCO captions, and then benchmark the fine-tuning performance of COCO captioning as well as concept forgetting. We choose to measure forgetting in terms of the performance change between fine-tuned model and LiMBeR on two types of tasks: image captioning on a different dataset NoCaps, and VQA on VQA2. For efficient fine-tuning with only 5 captions per COCO image, we use the method of Visual Prompt Tuning (VPT) (Jia et al., 2022) with CLIP and the language model kept frozen.
To evaluate image captioning performance, we report results of CIDEr-D (Vedantam et al., 2015), CLIPScore, and Ref- CLIPScore (Hessel al., 2021). While for VQA, the model is prompted using the "[image] Q: [q] A:" format. The generated answer is truncated to the length of the longest ground truth answer. As the evaluation metric of VQA under the few-shot setting, accuracy is reported for every K-shot.
Table 11 shows that VPT-based prompt tuning on COCO leads to forgetting on other tasks, e.g., ARESS is negative on NoCaps captioning. On the other hand, LDIFS and our (Froxy-)FDA methods prove effective in regularizing the tuning process, all achieving positive forward transfer in all metrics. Encouragingly, our (Proxy-)FDA is better than LDIIS at
18

---

ProXy-FDA
promoting positive forward transfer, while maintaining competitive prompt tuning performance COCO the time.
Knowledge distillation. As metioned in the Related Work section, the high-level idea of method resembles Knowledge Distillation (KD), epseically those relational KD methods that distill feature relations between models.
Table 12 shows our method is directly applicable to KD and quite performant. We follow the standard KD settings in (Zheng & Yang, 2024), and test teacher-student pairs using the same or different architectures of ResNet (He et al., 2016) and MobileNet (Howard et al., 2017) on ImageNet. We compare with state-of-the-ar logits matching methods KD++ (Wang et al., 2023), DIST (Huang et al., 2022) and WTTM (Zheng & Yang, 2024). Note DIST can viewed as a relational KD method at the logit level. We further compare with KD methods that match feature relations form of kNNs (CNA (Zhu al., 2022)) and feature similarities (ITRD (Miles al., 2022)). CNA and ITRD more related distilled in the feature space. We see from the table that FDA
FDA method, but
FDA differs in that both neighbor indices and similarities consistently outperforms CNA and ITRD, and is competitive or better than logits-based DIST. Our proxy learning further improves performance, and Proxy-FDA is on par with the best prior work WTTM.
19

---

ProXy-FDA
Table 6: Few-shot prompt tuning in the base-to-new class generalization setting (16 shots per class). Ah denotes the Harmonic mean of ABase and ANew New denotes the change in ANew between pre-trained and prompt-tuned CLIP models. Higher New shows lower level of concept forgetting on the new class split of the considered dataset.
Prompt tuning without regularization
Regularization
CoOp
CoCoOp
VPT
MaPLe
CLIPood
PromptSRC
PProxy-FDA
X
V
X
ABue
82.69
83.16
80.47
80.36
81.61
81.55
82.28
82.74
83.91
84.33
84.26
84.47
Avg across 11 datasets
ANew
63.22
73.67
71.69
76.44
69.61
73.89
75.14
77.13
74.50
76.54
76.10
77.45
New 1
-10.99
-0.55
-2.53
2.22
-4.61
-0.33
0.92
2.91
0.28
2.33
1.88
3.23
Au
71.66
78.13
75.83
78.35
75.14
77.53
78.55
79.84
78.93
80.25
79.97
80.81
ABse
76.47
76.22
75.98
76.95
75.96
75.26
76.66
77.35
77.50
78.47
77.60
77.81
ANew
67.88
72.97
70.43
73.48
67.32
71.25
70.54
71.51
70.30
72.07
70.73
71.55
ImageNet
New f
-0.26
4.83
2.29
5.34
-0.82
3.11
2.40
3.37
2.16
3.93
2.59
3.41
An
71.92
74.56
73.10
75.17
71.38
73.20
73.47
74.32
73.72
75.13
74.01
74.55
ABuse
98.00
96.84
97.96
97.21
97.50
96.14
97.74
98.71
98.70
99.08
98.10
98.49
ANew
89.81
97.45
93.81
97.15
94.10
95.93
94,36
95.42
94.60
95.01
94.03
95.34
Caltech101
NNe f
4.19
3.45
-0.19
3.15
0.10
1.93
0.36
1.42
0.60
1.01
0.03
1.34
A
93.73
97.14
95.84
97.18
95.77
96.03
96.02
97.04
96.61
97.00
96.02
96.89
ARme
93.67
95.01
95.20
96.96
96.05
95.32
95.43
95.42
95.70
97.63
95.33
96.31
Ax
95.29
98.97
97.69
98.64
95.84
98.42
97.76
98.09
96.40
98.21
97.30
98.09
OxfordPets
New f
-1.97
1.71
0.43
-1.42
1.16
0.50
0.83
-0.86
0.95
0.04
AA
94.47
96.95
96.43
97.79
95.94
96.85
96.58
96.74
96.05
97.92
96.30
97.19
Aguse
78.12
78.33
70.49
69.53
75.00
74.16
72.94
74.01
78.60
78.07
78.27
77.95
Stanford Cars
ANew
60.40
69.87
73.59
78.95
63.45
72.17
74.00
75.15
73.50
76.12
74.97
75.75
New f
-14.49
-5.02
-1.30
4.06
-11.44
-2.72
-0.89
0.26
-1.39
1.23
0.08
Au
68.13
73.86
72.01
73.94
68.74
73.15
73.47
74.58
75.96
77.08
76.58
76.83
Agus
97.60
97.21
94.87
94.52
96.89
97.11
95.92
96.85
93.50
97.91
98.07
97.69
ANew
59.67
72.36
71.75
77.54
70.02
73.49
72.46
75.59
74.50
76.59
76.50
78.49
Flowers 102
New f
-18.13
-5.44
-6.05
-0.26
-7.78
-4.31
-5.34
-2.21
-3.30
-1.21
-1.30
0.69
Au
74.06
82.96
81.71
85.19
81.29
83.66
82.56
84.91
82.93
85.95
85.95
87.04
ABoso
88.33
88.59
90.70
91.33
88.88
90.35
90.71
91.40
90.70
92.94
90.67
91.07
Asew
82.26
90.12
91.29
94.79
88.95
92.27
92.05
93.12
91.70
92.76
91.53
92.25
Food101
New f
-8.96
-1.10
0.07
3.57
-2.27
1.05
0.83
1.90
0.48
1.54
0.31
1.03
An
85.19
89.35
90.99
93.03
88.91
91.30
91.38
92.25
91.20
92.85
91.10
91.66
Auwe
40)44
41.24
33.41
35.12
38.33
38.75
37.44
37.41
43.30
42.26
42.73
41.63
FGVC Aireraft
Asew
22.30
33.83
23.71
36.36
25.27
31.36
35.61
37.79
37.20
37.54
37.87
40.61
Agew f
-13.99
-2,46
-12,58
0.07
-11.02
-4.93
-0.68
1.50
0.91
1.25
1.58
4.32
Ah
28.75
37.17
27.74
35.73
30.46
34.67
36.50
37.60
40.02
39.76
40.15
41.11
Auuse
80.60
80.63
79.74
80.36
80.27
79.54
80.82
81.24
81.00
83.04
82.67
82.71
Asew
65.89
72.11
76.86
78.97
74.36
76.11
78.70
82.15
79.30
79.92
78.47
79.73
SUN397
ANew f
-9.46
-3.24
1.51
3.62
-0.99
0.76
3.35
6.80
3.95
4.57
3.12
4.38
An
72.51
76.13
78.27
79.66
77.20
77.79
79.75
80.14
81.45
80.52
81.19
Apue
79.44
79.51
77.01
75.92
77.08
76.68
80.36
80.05
80.80
80.14
83.37
84.04
Asew
41.18
54.24
56.00
59.84
53.62
59.97
59.18
63.13
58.60
63.32
62.97
63.06
DTD
New f
-18.72
-5.66
-3.90
-0.06
-6.28
0.07
-0.72
3.23
-1.3
3.42
3.07
3.16
Au
54.24
64.49
64.85
66.93
63.24
67.30
68.
70.59
67.93
70.74
71.75
72.05
Ague
92.19
91.98
87.49
81.24
91.67
90.42
94.07
94.27
97.50
92.18
92.90
93.66
Asew
54.74
78.29
60.04
66.87
58.31
67.02
73.23
75.
64.10
71.01
73.90
77.12
FuroSAT
ANew f
-9.31
14.24
-4.01
2.82
-5.74
2.97
9.18
11.06
0.05
6.96
9.85
13.07
Au
68.69
84.58
71.21
73.36
71.28
76.98
82.35
83.61
77.35
80.22
82,32
84.59
Apze
84.69
89.15
82.33
84.86
80.07
83.37
83.00
83.43
85.70
85.95
87.10
87.79
ANew
56.05
70.16
73.45
78.23
74.50
74.77
78.66
81.40
79.30
79.44
78.80
79.95
UCF101
ANew t
-21.45
-7.34
-4.05
0.73
-3.00
-2.73
1.16
1.94
1.30
2.45
An
67.46
78.52
77.64
81.41
77.18
78.84
80.77
82.40
82.38
82.57
82.74
83.69
20

---

ProXy-FDA
Table 7: Few-shot cross-dataset generalization where CLIP prompt-tuned on the source dataset ImageNet (16 shots per class) and tested both ImageNet and 10 target datasets. We compare the test set accuracy A and the accuracy change A (higher is better) between pre-trained and prompt-tuned models to quantify generalization and concept forgetting on each target dataset.
CoOp
CoCoOp
PromptSRC
+Proxy-FDA
X
X
/
X
71.51
71.36
71.02
71.24
71.27
71.32
Source
ImageNet
4.78
4.63
4.29
4.51
4,54
4.59
Avg across
A
63.88
66.09
65.74
66.48
65.81
66.86
datasets
A
-1.20
1.01
0.66
1.40
0.72
1.78
A
93.70
94.35
94.43
94.51
93.60
94.42
Caltech101
A
0.76
1.41
1.49
1.57
0,66
1.48
89.14
90.53
90.14
90.62
90.25
90.78
OxfordPets
AAt
-0.07
1.32
0.93
1.41
1.04
1.57
Stanford
A
64.51
66.18
65.32
66.22
65.70
66.55
Cars
AA
-0.81
0.86
0.00
0.90
0.38
1.23
Target
Flowers102
68.71
71.54
71.88
72.32
70.25
72.04
A
-2.63
0.20
0.54
0.98
-1.09
0.70
85.30
86.06
86.91
$6.15
Food101
A
40.76
0.80
0.00
0.85
0.09
1.32
FGVC
18.47
22.09
22.94
23.49
23,90
24.79
Aircraft
A f
-6.25
-2.63
-1.78
-1.23
-0.82
0.07
64.15
66.12
67.36
67.62
67.10
67.53
SUN397
A
1,65
4.86
5.12
4.60
5.03
A
41.92
45.13
45.73
46.15
46.87
47.31
DTD
-2.47
1.34
1.76
2.48
A
46.39
49.08
45.37
47.89
45.50
48.37
EuroSAT
A
-1.21
-2.23
0.29
-2.10
A
66.55
69.01
68.21
69.10
68.75
69.42
UCFI01
A
-0.20
2.26
1.46
2.35
2,00
2.67
Table 8: Few-shot prompt tuning in both base-to-new class generalization and domain generalization settings. Here we compare with more recent prompt tning methods. Note both OGEN and our Proxy-FDA plugged into the PromptSRC baseline. For fair comparison with CLAP, with the ViT-B/16 backbone used by all other methods. The domain generalization results of CLAP are directly extracted from the CLAP paper. Ay denotes the Harmonic mean of ABase and ANow-
obtain its base-to-new generalization results by re-running its official codes
Base-to-New Class Generalization
Domain Generalization
Avg across 11 datasets
Ascurce ImageNet
Anager
ABase
ANew
New
Au
-V2
-Sketch
-A
-R
Text Knowledge from L.LM
ProText
72.95
76.98
2.76
74.91
70.22
63.54
49.45
51.47
77.35
ArGuee-N
83.77
78.74
4.52
81.18
71.84
65.02
49.25
51.47
76.96
OGEN
84.17
76.86
2.64
80.34
73.13
65.37
48.96
50.75
77.12
Regularization method
CLAP
84.34
76.62
2.40
80.29
73.38
65.00
48.35
49.53
77.26
ProXy-FDA
84.47
77.45
3.23
80.81
73.44
65.79
49.83
51.54
77.45
21

---

ProXy-FDA
Table 9: Continual fine-tuning; test accuracy ALP and ALp for models fine-tuned on three task sequences. The first
rows show performance on fine-tuned tasks and the 4th row shows performance averaged on other datasets, comparing our method with classic continual learning methods.
Fine-tune
Evaluation
LwF
LFL
iCaRL
D+R
ZSCL
FDA (ours)
Proxy-FDA (ours)
dataset
dataset
ALp
Air
ALP
Aip
ALp
ALp
ALP
Aur
ALP
ALp
Aup
Aip
Aue
ALP f
90.48
-3.81
91.90
-3.21
91.62
-3.67
93.30
-2.78
92.70
-3.23
96.77
96.72
SVHN-
CIFARIO- RESISC45
CIFARIO
93.90
-2.90
94.88
-2.32
95.17
-2.10
95.41
-1.0
95.82
-1.60
97.13
0.57
97.29
RESISC45
94.22
3.10
93.90
2.98
93.72
2.83
94.94
3.68
94.89
3.62
95.22
4.14
95.38
4.22
Others
80.73
-4.20
81.31
-3.76
80.78
-4,11
81.86
-3.20
83.10
-2.80
87.21
0.76
86.95
1.08
SVHN
89.48
-4.34
90.29
-4.08
90.97
-4.31
92.30
-3.23
91.81
-3.92
96.18
0.63
96.43
0.71
SVHN-
CIFAR100
83.24
-3.25
83.95
-3.01
84.06
-3.13
84.82
-2.60
85.07
-2.13
86.33
0.72
86. 14
0.85
CIFARI00
RESISC45
RESISC45
93.80
3.21
94.91
3.62
94.87
3.54
95.08
3.71
94.96
3.65
95.32
3.95
95.46
4.01
Others
81.73
-4.11
82.04
-3.80
81.62
-4.02
82.17
-3.43
82.86
-3.11
89.02
0.68
89.09
SVHN
91.43
-3.64
92.74
-2.92
91.75
-3.13
92.86
-2.84
92.98
-2.72
96.74
0.79
96.91
0.94
SVHN-
Cars
81.69
-2.79
81.82
-2.64
81,70
-2.80
82.11
-2.12
82.68
-1.84
84.38
1.14
84.32
1.36
Cars
RESISC45
RESISC45
93.92
3.34
94.96
3.55
94.97
3.58
95.19
3.72
95.04
3.63
95.12
3.92
95.23
4.07
Others
81.63
-4.07
82.24
-3.60
81.88
-3,89
82.73
-3.12
83.10
-2.80
89.54
0.96
89.67
1.17
Table 10: Continual fine-tuning: comparing the average accuracy on Split ImageNet-R.
L2P
DualPrompt CODA-Prompt
Continual-CLIP
SLCA
LDIFS
FDA (ours)
ProXy-FDA (ours)
74.60
77.2411.27
18.1311.18 76.2311.18
81.2211.23
83.62.1.116
85.9711.05
86.71.11.24
Table 11: Prompt tuning for image captioning and VQA. The CLIP model with LiMBeR projection is prompt-tuned on COCO dataset, and the fine-tuning performance for COCO captioning is reported in three metrics: CIDEr-D, CLIPScore, and Ref-CLIPScore. While forgetting is benchmarked in terms of the performance change between prompt-tume and original models on two different tasks: captioning on NoCaps (in ACIDEDD. AcuP AREts), and VQA on VQA2 (in accuracy change A). Higher performance change indicates lower forgetting or even positive forward transfer (A 0).
Image Captioning
VQA2 K-shots
COCO
NoCaps
2
CIDEr-D
CLIP-S
Ref-S
ACIDELD
ACLIPS
ARCSS 1
A f
A t
A f
A
f
VPT (Jia et al., 2022)
57.1
79.6
82.8
0.6
1.2
-0.3
1.1
0.7
0.8
1.9
VPT+LDIFS
56.8
80.3
82.4
1.5
1.8
0.6
2.1
1.3
1.6
3.2
VPT+FDA (ours)
56.2
80.7
83.4
2.2
2.3
1.4
2.6
1.5
2.1
3.9
VPT+Proxy-FDA (ours)
56.6
81.1
83.2
2.6
2.5
1.7
2.7
1.9
2.4
4.4
Table 12: Knowledge distillation: comparing the top-1 accuracy on ImageNet.
Lagits-based
Feature-based
Teacher
Student
KD++
DIST
WTTM
CNA
ITRD
FDA (ours)
Proxy- -DA (ours)
ResNet-34 (73.31)
ResNet-18 (69.76)
71.98
72.07
72.19
71.38
71.68
72.02
72.17
ResNet-50 (76.16)
MobileNet (68.87)
72.77
73.24
73.09
72.39
73.31
73.45
22


Reviews:
[{"review_id": "Bui6Hh3crx", "reviewer": "Reviewer_mrBn", "summary": "This paper introduces a novel approach to mitigate concept forgetting during model fine-tuning by extending existing feature-matching methods. The authors propose to preserve feature structure by regularizing the feature space using k-nearest neighbors (KNN) within each batch. Additionally, they develop a method for generating proxies from the data to maintain diversity across datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive feedback on our work. Below we include the results of requested experiments, and respond to your specific comments.\n\n**Q1: Justify motivation of using kNN feature-based FDA, and differences from knowledge distillation.**\n\nIn response, our introduction section (paragraphs 3 & 4) motivates that we aim to improve over existing feature regularization methods that are often point-wise and preserve limited concepts since they lack explicit awareness of feature neighborhood structures. We propose Feature Distribution Alignment (FDA), a structure-wise regularization method that aligns the structural relations between pre-trained and fine-tuned feature distributions. The structural relations are modeled by feature graphs, and we choose kNN feature graphs because they not only enable efficient graph matching, but also are effective enough to capture the rich knowledge in local feature neighborhoods (e.g. shared visual attribute).\n\nThe related work section (L145-152) mentions that Proxy-FDA is indeed similar to Knowledge Distillation (KD), especially to those relational KD methods. Our main difference is that we distill knowledge from both neighbor indices and similarities, with an additional proxy learning component. Appendix G shows Proxy-FDA is directly applicable to KD and is quite performant compared to related KD baselines.\n\n**Q2: Improve FDA by kNN clustering with label constraints**\n\nDuring our FDA-based fine-tuning, class labels are mainly used in the task loss $\\mathcal L_{\\text{task}}$, while $\\mathcal L_{\\text{FDA}}$ is only treated as a feature regularization term without involving labels. The intuition behind the label-free $\\mathcal L_{\\text{FDA}}$ is that we aim to preserve a foundation model's general knowledge, which can be much richer than class labels on downstream datasets. More specifically, $\\mathcal L_{\\text{FDA}}$ matches kNN feature graphs to align their structural relations only based on feature (not label) similarities. Fig. 4 shows this can go beyond class concepts in a feature neighborhood with different classes (e.g., cross-class attributes), which is key to maintain the generalizability of foundation models. On the other hand, matching kNN graphs with label constraints may end up aligning class semantics on the downstream task, thus may risk forgetting the general knowledge embedded in foundation models.\n\nIn the table below, we empirically compare with an FDA variant that models and matches kNN feature graphs using both feature similarities $\\hat w_{ij}$ and label similarities $w_{ij}^t$. Note we use the text encoder of CLIP to compute $w_{ij}^t$ as the text-text similarity between the class templates \"a photo of a {class}\". Comparisons are performed under the base-to-new setting for few-shot prompt tuning (average across 11 datasets). Results show that the use of $w_{ij}^t$ may produce comparable or better $\\mathcal A_{\\text{Base}}$ (i.e., fine-tuning accuracy on seen classes), but always leads to much lower $\\mathcal A_{\\text{New}}$ (i.e., worse generalization on unseen classes - more concept forgetting).\n|                                           | $\\mathcal A_{\\text{Base}}$ | $\\mathcal A_{\\text{New}}$ | $\\mathcal A_{\\text{H}}$ |\n|-------------------------------------------|----------------------------|---------------------------|-------------------------|\n| CoOp                                      |                      82.69 |                     63.22 |                   71.66 |\n| +Proxy-FDA ($\\hat w_{ij}$ - default)      |                  **83.16** |                 **73.67** |               **78.13** |\n| +Proxy-FDA ($\\hat w_{ij} \\cdot w_{ij}^t$) |                      83.02 |                     70.91 |                   76.49 |\n| PromptSRC                                 |                      84.26 |                      76.10 |                   79.97 |\n| +Proxy-FDA ($\\hat w_{ij}$ - default)      |                      84.47 |                 **77.45** |               **80.81** |\n| +Proxy-FDA ($\\hat w_{ij} \\cdot w_{ij}^t$) |                  **84.55** |                     77.12 |                   80.66 |", "claims_and_evidence": "Yes, the claim has been supported by proposed experimental results", "methods_and_evaluation_criteria": "Yes. It make sense", "theoretical_claims": "No new theory proposed in this paper", "experimental_designs_or_analyses": "Some improvement of their experimental results looks marginal. The experimental designs are valid.", "relation_to_broader_scientific_literature": "The solution is interesting but the broader impact of this work is limited.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n- The research addresses robust fine-tuning, which is a highly active and valuable area with significant practical relevance.\n- The approach is well-motivated, with the preservation of data structure during feature matching being both reasonable and innovative.\n- The paper is clearly written, making the technical approach accessible and easy to follow.\n\nWeakness:\n\n- Distribution Alignment Method: The motivation for using distribution alignment through feature matching with KNN regularization lacks sufficient justification. The approach bears similarities to knowledge distillation, but the paper doesn't adequately explain how this method differs from distillation between original and fine-tuned models.\n\n- KNN Clustering Limitations: The method clusters features in each batch using KNN, which may not effectively group samples with similar labels, particularly during fine-tuning. The clustering approach could be enhanced with label-based constraints that either include or exclude samples based on label proximity to improve feature alignment.", "other_comments_or_suggestions": "NA", "questions_for_authors": "## Update After rebuttal\n\nI appreciate the authors' thorough response, which has successfully addressed most of my initial concerns. I will maintain my current score.\n\nFor the final version of the paper, I strongly recommend that the authors clearly articulate the fundamental differences between traditional KL based Knowledge Distillation and the proposed FDA method. This distinction should be presented prominently in the main paper to help readers immediately understand the novel contribution.\n\nThank you for your attention to the review comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6oQuImQCeo", "reviewer": "Reviewer_3vR2", "summary": "This paper presents a new approach to mitigate concept forgetting in model fine-tuning (robust fine-tuning) by building on existing feature-matching methods. Specifically, this work aims to align the feature structure by regularizing the feature space using k-nearest neighbors (KNN) within each batch. They also propose generating proxies from the data to preserve diversity across datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback on our work. Below is our point-by-point response to your questions.\n\n**Q1: Motivation discussion: why Proxy-FDA improves performance.**\n\n(Proxy-)FDA is essentially a feature-space regularization term added to the task loss during model fine-tuning. The goal is to keep the fine-tuned model in the desired vicinity of the pre-trained one, so that the tuned model can preserve pre-trained knowledge while still learning the task at the same time. Extensive experiments show that (Proxy-)FDA can significantly reduce forgetting while achieving strong fine-tuning performance (sometimes better).\n\nFor better forgetting mitigation, our high-level idea is to extend existing point-wise feature regularization methods that lack explicit awareness of feature neighborhood structures. Proxy-FDA is proposed to fill this gap - it aligns the structural relations between kNN feature graphs, which is further improved by a proxy feature generator that increases feature diversity. **Two empirical observations** confirm our benefits, and hence reaffirm the motivation behind Proxy-FDA: 1) FDA can transfer the structural knowledge in kNN feature graphs, e.g. visual attribute shared between class concepts (Fig. 4). Preserving such common-sense knowledge is useful to maintain the generalizability of foundation model. 2) There's a strong correlation between forgetting and a structure-aware distributional distance metric OTDD (Fig. 3). Such correlation suggests the need of structure-wise feature regularization in some form to effectively mitigate forgetting, and our structural method Proxy-FDA is one such instantiation. In other words, this observation explains our advantage from an optimization perspective, i.e., optimizing our Proxy-FDA objective is close to optimizing a metric directly related to forgetting.\n\n**Q2: Theoretical explanations of the benefits of Proxy-FDA in reducing forgetting**\n\nGood suggestion. Note the primary focus of this paper is on empirical evaluation of our new method, along with **two empirical observations** (refer to our response to Q1) that shed light on the benefits of Proxy-FDA. However, we argue that it's promising to derive theoretical guarantees based on the 2nd observation that forgetting is strongly correlated with OTDD metric. Interestingly, OTDD is computed in an extremely similar way to our FDA loss -- they both use clustering techniques to account for the clustering structure of the underlying space, and hence both can compare feature distributions with rich geometry awareness. In other words, unlike L2 loss, our FDA loss is a good proxy of a metric (OTDD) that itself is directly related to forgetting. Such finding has two key implications: 1) There exists clear advantage of optimizing FDA loss over L2 loss-based optimization for direct forgetting prevention. 2) More importantly, the generalization error (or forgetting effect) could be bounded by some function of our own distance metric FDA (akin to OTDD). We leave such theoretical analysis for future work.", "claims_and_evidence": "Yes. Section 4 is about empirical results.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There is no theoretical results.", "experimental_designs_or_analyses": "Yes. Section 4 is about empirical results. And there are also some empirical details in appendix.", "relation_to_broader_scientific_literature": "This paper is mainly related to model generalization performance, which mainly focusing on the controlling the upper bound of test error, using the information on training error and function class.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n1. This paper is well-written, which provide a clear statement of the results.\n2. There is a strong performance on the new prosed method, which seems reasonable and significant.\n\nWeaknesses\n1. There is a lack of discussion about the motivation of the new method. Could you provide more discussion on why such method can improve model performance?\n2. There is a lack of theoretical guarantee. Is there some theoretical explanations about the benefits of such method in reducing forgetting?", "other_comments_or_suggestions": "See strengths and weaknesses.", "questions_for_authors": "See strengths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yAQXlY4mwn", "reviewer": "Reviewer_saQq", "summary": "The paper proposes Proxy-FDA, a regularization method for fine-tuning vision foundation models that mitigates concept forgetting by aligning local structural relationships in feature spaces. The core innovation lies in preserving neighborhood structures via nearest neighbor (kNN) graphs derived from pre-trained and fine-tuned features, augmented by dynamically generated proxies to enhance data diversity. By regularizing both neighbor indices and similarity scores, Proxy-FDA transfers rich semantic attributes (e.g., color, texture) encoded in foundation models while adapting to downstream tasks. Extensive experiments on classification, captioning, VQA, and continual learning demonstrate significant reductions in forgetting (quantified via LP and OTDD metrics) compared to point-wise regularization baselines (e.g., LDIFS, L2SP). The method excels in data-scarce settings (e.g., 16-shot tuning) and integrates seamlessly with prompt-tuning frameworks, achieving state-of-the-art performance without compromising downstream accuracy.\n\n## update after rebuttal\n\nDear authors, I've reviewed your rebuttal. While your responses to some questions, like proxy diversity and hard class mining, show thought, there are gaps. You didn't fully answer on training time and theoretical aspects in - line, just referring elsewhere, which makes it hard for readers. And for 1 - 2 shot tuning results, a summary was lacking. I appreciate your efforts, yet considering these, I'm keeping the “Weak Accept” rating. Revise to make your answers more complete and straightforward.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed feedback! Below is our response to the main questions and \"weaknesses\".\n\n**Q1: Proxy diversity and novelty lack quantitative metrics. Could FID evaluate synthetic feature quality?**\n\nAs suggested, many metrics are available to quantify our proxy feature diversity. Also, high proxy diversity often implies a high probability of proxy novelty, especially when the feature distribution is sparsely sampled and has a vast space of unseen data. Hence we focus on quantifying proxy diversity, and the novelty is simply examined through qualitative analysis — for example, one can perform some visual validation by image retrieval (Fig. 4) or even training a decoder on feature representations.\n\nTo quantify proxy diversity, we choose the variance loss in Eqs. (3-4), which is widely used in many domains like self-supervised learning to measure feature diversity. Note FID can be used to measure how well our generated proxies maintain the diversity of true features. Our adopted variance loss actually serves a similar purpose, since it is computed in an embedding space that's forced to align with the true one.\n\nHere we report the averaged standard deviation of proxy features in the variance loss: higher value indicates larger diversity. To further aggregate the standard deviation values of the positive and negative proxies, we take their mean and compute its moving average till fine-tuning is completed. The table below compares the aggregated diversity metric of all the proxy generation baselines ablated in Fig. 7.\n\n|                            | Diversity metric $\\times 10^{-2}$ |\n|----------------------------|------------------------------------|\n| Proxy generation (default) | **3.14**                           |\n| random interpolation       |                               2.89 |\n| VOS                        |                               1.53 |\n| NOPS                       |                               1.72 |\n\nOur method clearly achieves higher proxy diversity than VOS/NPOS. The latter two methods focus on outlier synthesis in low-likelihood regions, thus miss the chance to encode diverse unseen data that are crucial for improving FDA. Our method also obtains marginally higher proxy diversity than random interpolation. More importantly, our learning-based method improves diversity in a way that best helps FDA: the diverse proxies not only enrich data but also refine the decision boundary between positive and negative feature manifolds. This is not possible with random interpolation, which explains its lower performance in Fig. 7.\n\n**Q2: Compare Proxy-FDA’s training time with that of retrieval-augmented methods.**\n\nPlease refer to our response to Q2 of Reviewer bxDC.\n\n**Q3: Lack theoretical guarantees for FDA. Also, the correlation between OTDD and forgetting is observational, not causal.**\n\nPlease refer to our response to Q2 of Reviewer 3vR2.\n\n**Q4: Hard class mining: does random batch sampling degrade performance? Comparing to entropy-based sampling.**\n\nAs detailed in Appendix A, our batch sampling is performed by hard class mining plus random data sampling within class. Our ablation studies (Fig. 7) already compare with random batch sampling—the \"No hard class mining\" baseline—where classes are randomly sampled too. The average $\\mathcal A_{\\text{H}}$ is compared for few-shot prompt tuning, when we apply Proxy-FDA to CoOp/PromptSRC baselines. We see Proxy-FDA obtains $\\mathcal A_{\\text{H}}$ of 78.13/80.81 with hard class mining, and 75.23/80.35 with random batch sampling. As mentioned in Appendix E (L800-804), the big performance difference shows the hard class mining is crucial — it samples *close* class distributions, among which we can have meaningful modeling and matching of kNN graphs.\n\nWe now compare with the entropy-based batch sampling strategy that is also implemented under our greedy framework in Appendix A (for fair comparison). We simply change the inter-class similarity metric in step 2: our default hard class mining uses FDA loss to select similar class samples, while the entropy-based strategy prioritizes them by low entropy. The entropy-based strategy is found to have moderate decrease in $\\mathcal A_{\\text{H}}$ (77.46/80.75 vs. 78.13/80.81). This is because entropy cannot characterize similarity adaptively as a function of current *feature distribution structure*. As a result, the batch sampling criterion is decoupled with the structural FDA within sampled batch. While FDA loss-based sampling adapts to the feature distribution structure, and is coupled with FDA-based regularization in batch. Will add the results in paper.\n\n**Q5: Clarifications needed.**\n\nResults for 1–2 shot tuning are shown in Fig. 8 (more details in L860-865).\n\nTable 4 (and L853-857) contains results across different foundation models and architectures (including the large-scale one ViT-L/14), where proxy generation (Proxy-FDA vs. FDA) improves $\\Delta_{\\text{LP}}$ consistently.", "claims_and_evidence": "Supported Claims:  \n1.\tProxy-FDA reduces concept forgetting: Empirical validation across 10+ datasets (Tables 1, 4) and correlation analysis with OTDD (Fig. 3) robustly support this claim. The method consistently achieves higher LP (up to +1.54) than baselines, indicating superior retention of pre-trained knowledge.  \n2.\tProxies enhance feature distribution alignment: Ablation studies (Fig. 7) confirm that proxy generation improves FDA by synthesizing features from underrepresented regions of the data manifold. Comparisons with interpolation-based augmentation (e.g., VOS, NPOS) further validate its effectiveness.  \n3.\tStructure-wise alignment outperforms point-wise methods: Proxy-FDA’s use of kNN graphs and similarity transfer (Eq. 2) yields statistically significant gains over LDIFS and L2SP (Tables 1–3), with OTDD analysis (Fig. 3) demonstrating stronger correlation to forgetting than L2 distance.  \nUnsubstantiated Claims:  \n1.\tProxy diversity and novelty: While proxies are qualitatively shown to represent unseen concepts (Fig. 4), their diversity and novelty lack quantitative metrics (e.g., entropy, coverage).  \n2.\tComputational efficiency: Though Proxy-FDA incurs a 17–21% training time overhead (Appendix D), comparisons with retrieval-augmented methods or gradient-based alternatives are absent.", "methods_and_evaluation_criteria": "Methods: The integration of kNN graph alignment and proxy generation is novel. The use of OTDD, a structure-aware distribution distance, appropriately evaluates alignment quality.  \nEvaluation:  \n1.\tStrengths: Broad validation across tasks (classification, VQA), architectures (CLIP, DINOv2), and settings (end-to-end, continual).  \n2.\tWeaknesses: The selection of \"other datasets\" for ΔLP computation (Table 1) lacks explicit justification (e.g., domain overlap or task relevance).", "theoretical_claims": "The paper does not present formal theoretical guarantees. Claims about the superiority of structure-wise alignment are supported empirically but lack proofs (e.g., convergence analysis or generalization bounds). The correlation between OTDD and forgetting is observational, not causal.", "experimental_designs_or_analyses": "Strengths:  \n1.\tComprehensive ablations (Fig. 7) isolate contributions of hard class mining, proxy architecture, and similarity transfer.  \n2.\tHyperparameter sensitivity analysis (Fig. 6) validates robustness to batch size and neighborhood size.  \nWeaknesses:  \n1.\tHard class mining: The heuristic batch construction (Appendix A) is not rigorously compared to alternative strategies (e.g., entropy-based sampling).  \n2.\tExtreme low-data regimes: Results for 1–2 shot tuning are omitted, limiting insights into Proxy-FDA’s applicability to ultra-scarce data.", "relation_to_broader_scientific_literature": "Proxy-FDA bridges gaps in robust fine-tuning and relational knowledge distillation:  \n1.\tRobust Fine-Tuning: Extends LDIFS by replacing point-wise feature matching with structural alignment, akin to graph-based knowledge transfer (Park et al., 2019).  \n2.\tProxy Learning: Differs from metric learning proxies by synthesizing instance-wise features rather than class prototypes.  \n3.\tContinual Learning: Outperforms rehearsal-free methods (e.g., DualPrompt) by preserving structural knowledge without task-specific prompts (Table 9).", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:  \n1.\tPractical Versatility: Demonstrated efficacy across vision-language tasks (captioning, VQA) and architectures (ViT, ResNet).  \n2.\tNovel Regularization: Combines structural preservation with proxy-augmented diversity, advancing beyond point-wise or logit-based methods.  \nWeaknesses:  \n1.\tInterpretability: The t-SNE visualization (Fig. 4) lacks statistical rigor (e.g., clustering metrics).  \n2.\tScalability: Proxy generation for large-scale models (e.g., ViT-L/14) is not benchmarked.", "other_comments_or_suggestions": "Clarity: Eq. 3–4 could be simplified by merging redundant terms.", "questions_for_authors": "1.\tProxy Diversity: How is the diversity of generated proxies quantified (e.g., feature entropy, pairwise distance)? Could metrics like Fréchet Inception Distance (FID) evaluate synthetic feature quality?  \n2.\tHard Class Mining: Does random batch sampling degrade performance compared to the proposed greedy strategy? If so, by what margin?  \n3.\tComputational Trade-offs: How does Proxy-FDA’s training time compare to methods using external data augmentation (e.g., Mixup or retrieval-augmented tuning)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "48vZtJTjg3", "reviewer": "Reviewer_bxDC", "summary": "The paper introduces Proxy-FDA, a novel feature-space regularization method designed to prevent concept forgetting during the fine-tuning of vision foundation models. The key idea is to align the local structures of pre-trained and fine-tuned feature distributions using nearest neighbor graphs, which is further enhanced by generating synthetic features (proxies) to increase data diversity. Experiments demonstrate that Proxy-FDA significantly reduces concept forgetting across various fine-tuning settings and tasks, including end-to-end fine-tuning, few-shot prompt tuning, continual fine-tuning, and applications beyond classification like image captioning and visual question answering. The method achieves state-of-the-art results in mitigating forgetting, showing a strong correlation between a structure-aware distributional distance metric (OTDD) and concept forgetting.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the recognition of our work and constructive feedback. Below is our response to each concern, as well as new comparisons on compute cost.\n\n**Q1: Sensitivity to hyperparameters like the neighborhood size K and the scalar s for proxy generation**\n\nGiven the held-out validation set of each downstream dataset (a reasonable assumption for model fine-tuning), we adjust Proxy-FDA's hyperparameters on that validation set. Fig. 6 shows that results are mostly insensitive to each key hyperparameter in a wide range. We acknowledge there are methods that automatically learn hyperparameters instead of manual tuning, in order to further reduce hyperparameter sensitivity. Examples include meta-learning methods, or recent \"validation data-free\" methods (e.g., CLAP paper cited in Table 7) that optimize hyperparameters online without even requiring a validation set. Integrating such ideas into Proxy-FDA will be a promising future plan.\n\n**Q2: Computational overhead by the proxy generator could be a limitation in resource-constrained settings**\n\nThis concern is related to our hypothetical claims in the 1st paragraph of Section 3.2: despite incurring some computational overhead, proxy feature generation is still compute- and data-efficient (comparatively) to improve data diversity. Other viable strategies like retrieving external data should have higher cost and suffer from distribution shift. After paper submission, we implemented one type of Retrieval Augmented Fine-Tuning (RAFT) method using external data, and the comparison results support the above claims.\n\nConcretely, to enrich both the positive feature set $\\mathbf X_{i}^{+}$ and negative set $\\mathbf X_{i}^{-}$ (by the amount controlled by scalar $s$), we retrieve top similar samples from the large-scale LAION-400M dataset, instead of synthesizing proxy features \\{$\\mathbf P_{i}^{+}, \\mathbf P_{i}^{-}$\\} online. Both methods involve the augmented features (retrieved or synthesized) with FDA loss computation, and we call the methods as RAFT-FDA and Proxy-FDA, respectively.\n\nNote for RAFT-FDA to effectively retrieve external data, the same model is needed to extract features for the sampled batch in $\\mathcal D_{\\text{ft}}$ and external $\\mathcal D_{\\text{LAION}}$ to measure their feature similarity. It'd be obviously expensive to use the fine-tuned model to repeatedly refresh all the feature representations of $\\mathcal D_{\\text{LAION}}$. Hence we use the pre-trained model to pre-compute similarities between the entire $\\mathcal D_{\\text{ft}}$ and $\\mathcal D_{\\text{LAION}}$, so each sample in $\\mathcal D_{\\text{ft}}$ has its external neighbor indices that remain fixed. We treat this as a \"pre-processing step\" for RAFT-FDA whose training time excludes this step. In each fine-tuning iteration of RAFT-FDA, what requires additional feature extraction using the fine-tuned model is now only a limited number of external data (i.e., indexed neighbors from $\\mathcal D_{\\text{LAION}}$). Overall, the computational overhead of RAFT-FDA is mainly affected by FDA and external feature extraction/augmentation.\n\nThe table below compares RAFT-FDA against Proxy-FDA for few-shot prompt tuning based on CoOp.\n\n|                                                | Proxy-FDA |       | RAFT |  -FDA |       |\n|------------------------------------------------|-----------|:-----:|:----:|:-----:|:-----:|\n| Percent (%) of augmented features              | s         | s     | 2s   | 4s    | 6s    |\n| Avg $\\mathcal A_{\\text{H}}$ across 11 datasets |     78.13 | 76.02 | 77.1 | 78.29 | **78.84** |\n| Training time overhead                         |       **21%** |   58% |  83% |  127% |  171% |\n\nObservations: 1) Proxy generation is data-efficient in the low-data regime. Our Proxy-FDA outperforms RAFT-FDA when the latter retrieves the same or a double amount of external data. Generated proxies achieve higher data efficiency because they adapt to the fine-tuned feature distributions, while using external data will inevitably suffer from distribution shift and provide less effective feature regularization. However, our performance benefits diminish as the size of external data increases (over 4$s$). 2) On the other hand, external data augmentation is costly and the cost increases drastically with the retrieved data size. This is due to the need of extra feature extraction for each external sample using the large vision model. While our proxy generator is lightweight, and can generate proxy features all at once (not individually).\n\nMore comments: 1) We may potentially improve RAFT-FDA by feature fusion strategies (e.g., Mixup) to address the distribution shift issue, but the training cost remains high. 2) It's possible to speedup Proxy-FDA through a more efficient architecture design of our proxy generator, which we leave as future work.", "claims_and_evidence": "The claims made in the submission are well-supported by clear and convincing evidence. The authors provide extensive experimental results across multiple datasets and tasks, demonstrating the effectiveness of Proxy-FDA in reducing concept forgetting. They also conduct ablation studies to analyze the impact of different components of their method, such as the feature distribution alignment and proxy generation. The correlation analysis between OTDD and concept forgetting further strengthens the validity of their approach. However, the claim that Proxy-FDA consistently outperforms all other methods across all settings might be slightly overstated, as there could be specific scenarios where other methods perform comparably or better, though the provided evidence strongly supports its general effectiveness.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem of mitigating concept forgetting during fine-tuning. The use of feature distribution alignment with nearest neighbor graphs captures the local structure of the feature space effectively, and the introduction of synthetic proxies addresses data scarcity issues in few-shot scenarios. The evaluation criteria, including the change in performance on unseen tasks (∆LP) and the distributional distance metric OTDD, are appropriate for assessing the extent of concept forgetting and the quality of feature alignment.", "theoretical_claims": "I did not check the correctness of any proofs for theoretical claims in this paper, as the focus is primarily on empirical evaluation and methodological innovation rather than theoretical analysis.", "experimental_designs_or_analyses": "The experimental designs and analyses appear sound and valid. The authors conduct experiments on a diverse range of datasets and tasks, ensuring the robustness of their findings. They compare against relevant baselines, including naive fine-tuning, LP-FT, L2SP, and LDIFS, providing a comprehensive evaluation of their method's performance. The ablation studies and sensitivity analyses further validate the effectiveness of individual components and design choices in Proxy-FDA.", "relation_to_broader_scientific_literature": "The key contributions of this paper are well situated within the broader scientific literature on robust fine-tuning of foundation models. The work builds upon and advances prior research in regularization techniques for fine-tuning, such as L2SP and LDIFS, by introducing a structure-aware feature distribution alignment method. It also relates to knowledge distillation and domain adaptation literature, where preserving and transferring knowledge across different data distributions is a central concern. The use of optimal transport for measuring distributional distances connects to a broader body of work on optimal transport in machine learning.", "essential_references_not_discussed": "There are no critical references missing from the discussion that would significantly impact the understanding of the paper's contributions. The authors adequately cite and discuss relevant prior work in the areas of fine-tuning, concept forgetting, and related regularization methods.", "other_strengths_and_weaknesses": "Strengths:\nThe method is conceptually simple yet effective, leveraging feature space regularization with nearest neighbor graphs and proxy generation.\nExtensive experiments across diverse datasets and tasks demonstrate consistent improvements over existing methods.\nThe analysis of the correlation between OTDD and concept forgetting provides valuable insights into the effectiveness of structure-aware regularization.\n\n\nWeaknesses:\nThe computational overhead introduced by the proxy generator could be a limitation in resource-constrained settings.\nThe method's effectiveness might be sensitive to the choice of hyperparameters, such as the neighborhood size K and the scalar s for proxy generation, though the authors provide some analysis on their sensitivity.", "other_comments_or_suggestions": "The paper is well-written and clearly presents the methodology and experimental results.", "questions_for_authors": "no", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel approach to mitigating concept forgetting during model fine-tuning by extending existing feature-matching methods. It demonstrates strengths such as practical versatility across vision-language tasks, innovative regularization techniques. The paper is well-written. By employing feature space regularization with nearest neighbor graphs and proxy generation, the paper achieves significant improvements across diverse datasets. After the author-reviewer discussion, all reviewers recommended acceptance.
Decision: Accept (poster)