Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization
Authors: Yuhang Zang
Hanlin Goh
Joshua M. Susskind
Chen Huang
Venue: ICLR 2024 poster (2024)
Keywords: Vision-Language Model
Finetuning
Prompt Learning
OOD Generalization
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Existing vision-language models exhibit strong generalization on a variety of visual domains and tasks. However, such models mainly perform zero-shot recognition in a closed-set manner, and thus struggle to handle open-domain visual concepts by design. There are recent finetuning methods, such as prompt learning, that not only study the discrimination between in-distribution (ID) and out-of-distribution (OOD) samples, but also show some improvements in both ID and OOD accuracies. In this paper, we first demonstrate that vision-language models, after long enough finetuning but without proper regularization, tend to overfit the known classes in the given dataset, with degraded performance on unknown classes. Then we propose a novel approach OGEN to address this pitfall, with the main focus on improving the OOD GENeralization of finetuned models. Specifically, a class-conditional feature generator is introduced to synthesize OOD features using just the class name of any unknown class. Such synthesized features will provide useful knowledge about unknowns and help regularize the decision boundary between ID and OOD data when optimized jointly. Equally important is our adaptive self-distillation mechanism to regularize our feature generation model during joint optimization, i.e., adaptively transferring knowledge between model states to further prevent overfitting. Experiments validate that our method yields convincing gains in OOD generalization performance in different settings. Code: https://github.com/apple/ml-ogen.

Published as a conference paper at ICLR 2024
OVERCOMING THE PITFALLS OF VISION-LANGUAGE MODEL FINETUNING FOR OOD GENERALIZATION
Yuhang Zang" Hanlin Goh", Josh Susskind Chen Huang?
Nanyang Technological University ?Apple Inc.
zang00 www.:o001nnnn...mm edu.
{hanlin, jsusski nd, chen-huang 2 Aapple. com
ABSTRACT
Existing vision-language models exhibit strong generalization variety of vi- sual domains and tasks. However, such models mainly perform zero-shot recogni- tion in a closed-set manner, and thus struggle to handle open-domain visual con- cepts by design. There are recent finetuning methods, such as prompt learning, that not only study the discrimination between in-distribution (ID) and out-of- distribution (OOD) samples, but also show some improvements in both ID and this paper, we first demonstrate that vision-language mod-
OOD accuracies. els, after long enough finetuning but without proper regularization, tend to overfit the known classes in the given dataset, with degraded performance on unknown classes. Then we propose novel approach OGEN to address this pitfall, with the main focus on improving the OOD GENeralization of finetuned models. Specif- ically, class clas-cooniiional feature generator is introduced synthesize OOD fea- tures using just the class name of any unknown class. Such synthesized features will provide useful knowledge about unknowns and help regularize the decision boundary between ID and OOD data when optimized jointly. Equally important is our adaptive self-distillation mechanism to regularize our feature generation model during joint optimization, i.e., adaptively transferring knowledge between model states to further prevent overfitting. Experiments validate that our method yields convincing gains in OOD generalization performance in different settings. Code:
INTRODUCTION
Large-scale pre-trained vision-language models like CLIP (Radford et al., 2021) demonstrate promising generalizability various visual domains and tasks in the real world. However, their zero-shot in-distribution (ID) performance can be limited for some downstream datasets. Also due their zero-shot evaluation in closed-set manner (i.e., match input image predefined set of classes), vision-language models often struggle to handle the out-of-distribution (OOD) samples from novel classes. Such shortcomings create major safety risks in the open domain that often require capabilities of OOD detection and/or accurate identification of both novel and seen classes.
Some recent works attempt improve the zero-shot OOD detection performance of existing vision- language models, either by simple softmax scaling (Ming et al., 2022) or training an extra text generator (Esmaeilpour al., 2022). Alternatively, Fort et al. (2021) first show the promise of CLIP models finetuned on an ID dataset. Encouragingly both ID and OOD accuracies are improved after finetuning. Parameter-efficient finetuning methods, such as prompt learning (Zhou et al., 2022a;b) or adaptor tuning (Zhang et al., 2022), illustrate similar benefits without heavy training.
Despite the success of prior finetuning methods, we found from our extensive benchmarking that finetuning on ID datasets is prone to overfitting (Fig. 1(b)). More specifically, we observed that models after long enough finetuning but without proper regularization, tend to overfit the known classes in the given dataset, with inferior generalization unknown classes. Unfortunately, plicit regularization mechanism has not been explored in literature to address this pitfall, and simple Work done while interning at Apple.

---

Published as a conference paper at ICLR 2024
(a)
Vision-language model
ImageNet (vithin-datsae))
ImageNet (ID)
EuroSAT (OOD)
ID
cross- dataset
Finetuning
(known)
Dowrstrcam tasks
OOD
ImageNet
EuroSAT
(unknowmi)
(b)
UCF101
EuroSAT
DTD
100
80
unknoown closs
known class
Acturmy
80
Aco.ray (98)
2
(00
80
a CoOpp
LO00N--COp CoCp
103
150
800
100
203
100
200
Epochs
Eppcete
Figure 1: (a) We study OOD generalization when finetuning the vision-language model CLIP on various downstream tasks. We consider both within-dataset generalization where one dataset has ID vs. OOD (or known vs. unknown) class splits for finetuning and evaluation respectively, and the more challenging cross-dataset generalization setting. More clarifications on the problem definition in Appendix A. (b) Examples of within-dataset generalization: we show learning curves of the prompt learning method CoOp (Zhou et al., 2022a) that finetunes CLIP for long enough (200 epochs) on three datasets (more Appendix B). Apparently, CoOp overfits the known classes of each dataset with notable accuracy drop on the unknowns. Our proposed method OGEN largely reduces such overfitting through effective regularization.
regularization strategies like early stopping seem insufficient. E.g. in Fig. 1(b), it is difficult to find an early model checkpoint with good trade-off between the known and unknown class performance.
One main challenge of effective model regularization is the missing knowledge about unknowns. Such knowledge could actually offer useful supervision signals to avoid overconfident predictions on OOD data. In this paper, we propose novel method that features 1) image feature synthesis for unknown classes and 2) an unknown-aware finetuning algorithm with effective model regularization. The goal is to improve OOD generalization without hurting the ID) performance of finetuned models. To synthesize unknown features, we introduce class-conditional feature generator: i.e., generating image features just given the name of an unknown class. This is made possible by CLIP's well- aligned image-text feature spaces. Our feature generator implemented by a lightweight attention module, with an "extrapolating bias" on the unknown classes. It generalizes well to "unknown knowns" and hence can model the complex distributions of visual classes the open domain. Then both the ID and synthesized OOD data for joint optimization, leading better regularized decision boundary. Another contribution is an adaptive self-distillation mechanism that regularizes our feature generator to further reduce overfitting during joint optimization. The idea is to find an adaptive teacher model of the feature generator from historical training epochs (with less overfitting) guide optimization at the current epoch (student model, often with more overfitting).
Our overall approach OGEN is applicable to different finetuning methods egg., (Zhou et al., 2022a;b; Jia et al., 2022) for CLIP-like models. OGEN is shown to consistently improve their OOD gener- alization performance (by up to absolute 18.77%) under two settings: within dataset (base-to-nen) class) generalization and crass-dataset generalization. Summarizing, our main contributions are:
Provide the first comprehensive study on OOD generalization that unveils the pitfalls of finetuning methods (based on prompt learning) for vision-language models.
A class-conditional feature generator to synthesize OOD data for effective regularization.
Adaptive self-distillation on our feature generator to further reduce overfitting.
2 RELATED WORK
Vision-Language Models. Recent large-scale vision-language models like ViLT (Kim et al., 2021) and PaLI (Chen et al., 2023) simply consume image-and-tedt features for multimodal downstream tasks with remarkable performance. Another popular paradigm used in CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) contrastively aligns image and text encoders. These contrastive mod- els trained on massive web-scale image-text pairs, also showing strong adaptability range of
2

---

Published as a conference paper at ICLR 2024
downstream tasks, such as semantic segmentation (Zang al., 2022a; Ghiasi et al., 2021) and video classification (Qian et al., 2022). Numerous follow-up works (Li et al., 2022; Zhou et al., 2022a) aim to improve CLIP-like like models in data efficiency generalization. However, the zero-shot per- formance on some tasks can still be limited for existing vision-language models. Hu et al. (2023) found that they make different kinds of errors, egg., PaI.I is erroneous at tail visual concepts while CLIP may fail for common ones. This paper mainly studies and improves the generalization of fine- tuned CLIP models, but our approach is model-agnostic and thus applicable other vision-language models as well.
Finetuning methods have been studied to improve the downstream performance of vision-language models over their zero-shot counterparts. Fort et al. (2021) showed that after finetuning the CLIP model on datasets of interest, both the ID and OOD generalization performance will be improved. More parameter-efficient finetuning methods are popularized in recent years. In particular, prompt learning focuses on learning visual (Jia et al., 2022), textual (Zhou et al., 2022a;b; Yao et al., 2023; Wang et al., 2023; Shu et al., 2023; Khattak et al., 2023b) or multi-modal Zang al. (2022b); Khattak et al. (2023a) prompts, while adaptor tuning (Zhang et al., 2022) optimizes feature repre- sentations with the model backbone kept frozen. In this paper, we first unveil the overfitting issue of recent finetuning methods, and then propose new regularization method to prevent overfitting. Our approach is orthogonal the finetuning research, and shows consistent gains over various finetuning baselines.
Outlier synthesis proves effective for model regularization in the absence of OOD data. Previous methods rely on GANs (Lee et al., 2018) to synthesize outlier images. More recent methods like VOS (Du et al., 2022) directly synthesize virtual features which allows greater flexibility. Tao al. (2023) propose non-parametric outlier synthesis, without the restrictive Gaussian assumption on feature distributions in VOS. Here we present a new feature synthesis method that has the same format as the CLIP framework and hence facilitates multimodal regularization. Specifically, given the name of an unknown class, we synthesize its example features a generalizable way.
Model distillation techniques transfer knowledge from teacher model to student models, e.g., from a large model to its efficient counterparts (Hinton et al., 2015) or from weakly augmented model to the strongly augmented (Sohn et al., 2020). Here we aim to reduce overfitting for unseen classes and propose to distill knowledge from early to current epochs (i.e., self-distillation). Specifically, we extend Mean teacher (Tarvainen & Valpola, 2017) to adaptive localized one with suitable teacher curriculum. In the vision-language domain, our approach differs from distillation into smaller mod- els (Li et al., 2023) towards various downstream tasks (Gu al., 2022; Dai al., 2022; Mal et al., 2022). Our approach is also orthogonal (and applicable) to recent distillation frameworks for improved multimodal pretraining (Dong al., 2023; Li et al., 2021; Zhong al., 2022).
METHODOLOGY
3.1 PRELIMINARIES
CLIP (Radford et al., 2021) is the vision-language model that we mainly study in this paper, al- though our study is applicable other popular models. CLIP consists of image encoder Φ and text encoder v/, which map the image and text inputs into a joint feature space. The CLIP training aims aligning the image and text modalities by maximizing their feature similarity. Given input image x that belongs to one of the classes Y {yi, y/, .... }c}. J. the image encoder & first extracts image features z = (a)) E Rd To obtain the corresponding text features W (I....CC) class names deseriptions A which are further encoded by vb into the text embeddings W fo (A) E R (hence w.- = W..). The image-text alignment optimized based on the cosine feature similarity:
. all the given be fed into fixed prompt template (a photo of [CLASS]], leading text
exp (cos (we, 2) /T)
plu = c Σ exp (cos (w;, 2) /T)
where 7 is the temperature. A larger cosine score often indicates stronger image-text alignment in their underlying semantics.
Prompt Learning. For efficient model finetuning on downstream tasks, recent prompt learning approaches like CoOp (Zhou al., 2022a) replace the aforementioned fixed prompts with learnalee
3

---

Published as a conference paper at ICLR 2024
unknown [CLASS]
No training image
Learned prompts
unknown
known known unknown
KNN
KNN
[CLASS]
Text Encoder y
image
R
W
Retrieval
Value
Key
Query]
Image Teature space
Synthessizig
unknown text features
Multi-Head
Cross-Altention
FFN
W
Class conditiona feeture generator 0
Synthesized unknowns
Matching
Known concepts
KNN known (text, image) features
W
Layer Normalization
unknown
unknown
Regularized decision boundary
image features
text features
Image
(a)
(c)
Figure 2: (a) To improve OOD generalization, we propose to gain knowledge of unknown classes by directly synthesizing their image features. This helps to learn more reliable decision boundary between known and unknown classes in the feature space. (b) Prompt learning based on discriminat- ing both the known and synthesized unknown features (from our condiioniii feature generator 0, see details in text). (c) Implementation of / using a lightweight attention module.
ones V = [1, ......, 8n] E d where L is the prompt length. Then the text encoder a of CLIP will be able to convert the learned prompts V (together with Y) into adapted text embeddings W = (IV, YI) Bdx Note V is learned on each downstream task using the task task-specific loss. The image encoder Φ and text encoder vb of CLIP are kept frozen during prompt learning.
3.2 CLASS-CONDITIONAL FEATURE GENERATOR
As shown in Fig. 1(b), the "prompt-tunee tunded" CLIP model tends to overfit the known classes (aka base classes Y {,,, y/2 ... yc, }) from the downstream task, while OOD generalization on unknown classes (aka new classes Y with Y = C,)) will deteriorate. To reduce overfitting, one might choose model regularization strategies, which will inevitably suffer from the missing knowledge about unknowns. Moreover, the potential number of unknown classes C₁ is huge and C, >> C-- Hence is very challenging model their complex distributions for effective regularization.
Here we make one step towards gaining knowledge of unknowns in a class-conditional manner, in order provide supervision signals for the vast space of unknown data. Given a textual description or simply the class name of any unknown class, we aim to synthesize the class example features without seeing labeled instances (Fig. 2(a)), leveraging the well-aligned image-text feature spaces of CLIP. Such synthesized image features will then facilitate learning a regularized decision boundary between known and unknown classes, leading improved OOD generalization capabilities.
In early experiments, we found that directly generating OOD image features out of class names hard due the highly non-linear and high-dimensional nature of the former. This is similarly observed in those strong cases of OOD generalization in (Abbe et al., 2023), where the manifold embeddings are typically nonlinear and, more critically, part of the distribution domain entirely unseen at training. It is proved that successful learning under such extreme distribution shift leads to extrapolating solutions since memorization is voided on the unseen domain. Following the "ex- trapolating bias" on the unknown, we reframe our feature synthesis problem as an easier one extrapolating from the most similar classes of the seen data, e.g., to generate features of the un- known class raccoon by extrapolating features of the similar training classes like cat and bear.
More specifically, for prompt learning, given the learned prompts and one unknown [ [CLASS] from the open set Y". we first obtain the corresponding text features th" E Rd through the text encoder z of CLIP. Then we find for uh its KNN classes from the entire set of text features of known classes W E RdxX's resulting in W E dR where R is the neighbor set with R = K. From each of the kNN classes, we randomly sample only one class example and obtain its text-aligned image features from the image encoder é, leading to the same number of K image feature vectors Z d Our goal is to train class-conditional feature generator fe(w=" W R: Z%) that can synthesize unknown image features conditioned on the text features ú" of an unknown class and auxiliary text/image features (Wh- it: %%²) of kNN known classes, see Fig. 2 (b).
Remarks. To retrieve semantically similar kNN classes W R from Wh, we choose to use the cosine similarity score between the text features (not image features) of class pairs. Then the kNN retrieval
4

---

Published as a conference paper at ICLR 2024
process can be formally defined as:
Σ cos (wo", w.)), where uo W EER
arg max RC (CC]......))II
On another note, our empirical study shows that the one random example sampled from each kNN class is enough for assisting new feature generation. Such randomness encourages the diversity of the synthesized features for new classes.
Extrapolating per class. Recall the tuple (Wh- R: Z'ze) consists of K' text and image feature vectors respectively (one for each similar known class). One straightforward feature synthesis method for an unknown class (with text features w²" is to extrapolate each image feature vector in Z based on some notion of similarity with wo" leading to total of K extrapolated image features from K known classes (e.g., caf->raccoon, bear-traccoon...). The similarity notion can be well learned by Multi-Head Head Cross-Attention (MHCA) that operates on the triplets of queries, keys and values (sd", W 2. This way, we can effectively take into account the similarity between the unknown class and each known class in R as well all other between-class similarities. Summarizing, the matrix form of our cestrapolating-pe--laee
scheme is given as:
Z LN( + Z") R 2n = MHCA(W" 1 W Z
Rd*K
(3)
where 2n are the learned feature residuals when extrapolating each of the K known classes. LN denotes layer normalization. Obviously, our feature generator # lightweight with only one MHCA layer and one LN layer. The simplicity benefits from the "extrapolating bias" in our generator design.
Finally, we use the synthesized features Z" to regularize prompt learning and perform joint discrim- ination of C. known and Cn unknown class features. The objective of maximizing the image-text alignment in Eq. (1) now becomes:
exp (cos (w.,, 2T)/T) Ve (l,...,C, CnJ: Ch+CCn exp (cos (ro,, xg) /T)
pΦy - c 1n)
(4)
K
Di=1
where u. = [W W"]. and x = Z Note under the "extrapolating scheme, we have synthesized K image features for the same unknown class. We simply aggregate them the score level when computing the cosine feature similarity score in Eq. (4).
Extrapolating jointly is more collaborative approach new feature synthesis. As the name hints, we extrapolate a single image feature vector xn from all the kNN known class features (W), hh), based on the cross attention against rh"
LN(FFN(WO) + 2n) E Rdd, 2" = MHC W Z E IRd
(5)
where 2th is the residual image feature vector, while text features vh"h are projected into the image feature space via a two-layer fully connected feed-forward network FFN. Note FFN(x"") could be replaced by some anchor point directly searched in the image feature space, e.g., weighted average of KNN image features from Z However, searching is hard problem itself and learning an explicit text-to-image feature mapping works consistently better in experiments. Fig. 2 (c) summarizes the overall network architecture, and the objective function in Eq. (4) could be updated as:
exp (cos (w.r /T)
plu c x²)
Ve E (1....,Co+C,,
(6)
C.+CC. exp (cos (w, x) /T) mml
Remarks. Our ablation study (Table 4) shows that "extrapolating jointly" (our default approach) is better than "extrapolating per class" at synthesizing useful unknown features for joint optimization. We train our class-conditional feature generator using the "known" and "unknown" class splits from the training set of downstream tasks. Fig. demonstrates the ability of our feature generator to generalize to "unknown unknowns" during testing, with faithful image feature synthesis.
3.3 ADAPTIVE SELF-DISTILLATION
Optimizing both known and synthesized unknown features generally improves OOD generalization and oftentimes the ID performance too. However, that does not take into account the optimization
5

---

Published as a conference paper at ICLR 2024
Image feature space
KNN retrieval
unknown cless Japanese anemone
unknowwm class Japanese anteronee
unseen
KNN class
Hippeastrum
KNNN Hippeastrum
KNN class Desert-rose
Desert-rose
KNN class
Columbine
Columbine
class unlnoww class
synthesicoed
Figure 3: Visualizing image feature synthesis based on the joint extrapolation scheme (Eq. (5)) on Flowers 102 dataset. Note our feature generator is not trained on the unknown classes, but can still synthesize faithful image features (red triangle) lying close to the real ones (gray cross). This is achieved by extrapolating an unseen instance from the kNN class examples (only a random one per kNN class is used), effectively combining their related patterns like the shape and texture of flowers.
dynamics that could also impact the ID-OOD performance tradeoff, especially with long finetuning runs. Take Fig. ((b) for example. Without proper regularization, the CoOp baseline achieves either suboptimal ID performance early epochs, saturated ID performance but decreasing OOD per- formance (i.e., overfitting) later on. To address this issue, we introduce an adaptive self-distillation method that regularizes optimization dynamics to further reduce overfitting.
More specifically, we use the model checkpoints from earlier epochs (i.e., teacher model often with less overfitting) guide optimization at the current epoch (i.e., student model often with more overfitting). Since the CLIP model is frozen during prompt learning, the "model" we consider here is our feature generator & whose synthesized OOD features will impact the joint ID-OOD optimization. Hence we enforce the consistency between the final prediction probabilities (Eq. (4) or (6)) induced by the teacher model y" and student model p using the mean squared error MSE(p" p²)) Ideally, this will help to avoid OOD performance drop while preserving the ID performance.
The key to our self-distillation method is the choice of teacher model 0", Obviously, selecting O as one single model checkpoint at a historical epoch time is unlikely to strike a good trade-off between the ID and OOD performance. Mean Teacher (MT) (Tarvainen & Valpola, 2017) is better alternative, which calculates Exponential Moving Average (EMA) over the past checkpoints up until the current time (Eq. (7)). Here we propose Adaptive Local Mean Teacher (ALMT) that extends MT in two ways: 1) calculating EMA only within local time window [t my, f) using the last my checkpoints. This avoids the negative impact on the teacher's ID performance from those underfit early checkpoints. 2) the window size me is time-adaptive such that m, small in the early stage of finetuning (for the same purpose of ruling out underfit checkpoints), and then my gradually increases order to cover older checkpoints with improved ID performance but less overfitting. Such curriculum is summarized in Eq. (8) as below:
(..) =T=a0T- (1-oa)os for i= -1,...,;.}
ALMT, MT () my [1+com cos
(tmax +
1
π
. (Mmax mmin) Mmin
(8)
tmax
where Mmmx = Miin = 2, mmxx is the maximum number of finetuning epochs, and the window size m, is increased following cosine schedule. Note our ALMT method requires maintaining a queue of past m, checkpoints and re-calculating EMA for each time t, both of which are cheap thanks to our compact model size of 0, and the small window size m, E 9].
EXPERIMENTS
We evaluate OOD generalization under the two settings introduced in (Zhou et al., 2022b) (more details in Appendix A): 1) generalization from ID (base) to OOD (new) classes within dataset. The base and new class splits are used for finetuning and evaluation respectively. 2) cross-dataset generalization with one IDD dataset for finetuning and other datasets for OOD evaluation. The cross-

---

Published as a conference paper at ICLR 2024
Table 1: Base-to-new class generalization. Our OGEN approach consistently improves the new class gener- alization for all prompt learning baselines on average (across 11 datasets). OGEN also maintains or improves the average performance on base classes. H: Harmonic mean of base and new accuracies (%)
CoOp
CoCoOp
VPT
SHIP
KgCoOp X
MaPLe X
PrompiSRC
+OGEN
X
x
/
Avg across
Base
82.69
83.47
80.47
79.86
82.51
82.52
80.03
80.79
80.73
81.34
82 75.14
82.40
84.26 76.10
84.
New
63.22
69.54
71.69
73.35
69.01
70.61
73.69
76.14
73.60
75.68
76.37
76.86
11 datasets
A
+6.32
+1.66
+1.60
+2.08
+1.23
+0.76
H
71.66
75.87
75.83
76.47
75.16
76.
76.73
78.40
77.00
78.40
78.55
79.27
79.97
80.34
Base
76.47
76.40
15,98
76.50
75.96
75.09
75.87
76.14
75.83
75.88
76.66
77.02
77.60
77.50
ImageNet
New
67.88
68.80
70.43
70.23
67.32
67.66
69.95
71.18
69.96
70.93
70.54
70.73
70.73
70.97
H
71.92
72.40
73,10
73.23
71.38
71.18
72.79
73.58
72. 72
73.32
73.47
73.74
74.01
74.09
Base
98,00
96.67
97.96
96.67
97.50
96.33
97.55
98.
97.72
98.52
97.74
98. 98.37
98. 10
98.32
1a
New
89.81
92.61
93.81
94.79
94.10
92.36
95.20
95 26
94
94.12
94.36
94.54
94.03
94.76
H
93.73
94.59
95.84
95.72
95.77
94.30
96.36
96.65
96.03
96.27
96.02
96
96. 02
96.50
Base
93.67
95.
95.20
96.49
96.05
96.05
95.37
96.95
94.65
95.91
95.43
95. 11
95.33 33
95.96
OxfordPets
New
95.29
96.45
97.69
97. 86
95.84
96.84 84
97.87
97.33
97.
97.65
97.76
97. 89
97.30
97.48
H
94.47
95.81
96.43
97.17
95. 94
96.45
96.61
97.14
96.
96.77
96.58
96.47
96.33 36.30
96.71
Stanford
Base
78.12
78.65
70.49
68.96
75.00
74,223
68.57
68
71.76
71.86
72.944
73.63
78.27
77.59
Cars
New
H
60.40
68.13
65.28
71.35
73.59
72.01
71.50
74.23
68.74
70.96
67.97
71.14
73.90
75.45
71.88
73.36
75.0%
75.95
73.84
74.00
73.47
73.96
74.30
74.97
76.58
75.17
76.38
Base
97.60
97.38
94.87
93.95
96.89
98.03
94.02
95.00
95.83
95.92
96.52
07
97.34
Flowers102
New
59.67
67.70
71.75
72.08
70.02
69.15
74.40
76 76.49
74.73
74.75
72.46
74.46
76.50
77.67
H
74,06
79.87
81.71
81.57
81.29
81.09
83,06
84.61
83.65
83,98
82,56
84.06
85,95
86.39
Base
88.33
89.21
90.70
91.17
88.88
91.50
90.54
91.07
90.50
90,80
90,71
91. 02
90.67
90.69
Food101
New
82.26
87.22
91.29
91.67
88 9.5
88.53
91.03
92.79
91.
92.01
92.05
92.02
91.53
91.68
H
85.19
88.21
90.99
88.9
89,99
90.78
91.92
91.09
91.40
91.38
91.52
91.10
91.19
FGVC
Base
40.44
41.67
33.41
35.33
38.33
39.33
34.27
35.47
36.21
37.08
37.44
37.07
42.73
41.26
Aireraft
New
22.30
29.14
23.771
34.41
25.27
26.55
34. 32.33
33.55
37.19
35.61
37.41
37.87
40.26
H
28.75
34.29
27.74
34.86
30.46
31.70
33.28
34.89
34.83
37.14
36.50
37.24
40.15
40.75
Base
80.60
80.86
79.74
80.27
80.27
79.06
79.54
81.14
80.
81.91
80.82
81.06
82.67
82.57
SUN397
New
67.49
76.86
69
74.36
75.27
75
76.53
78.83
78.70
01
78.47
78.83
H
72,51
73.57
78.27
77.91
77.20
76.71
77.35
78 45
78.36
80.34
79.75
81.06
80.52
80.65
Base
79.16
77.01
75.00
77.08
77.43
74.88
02
77.55
78.01
80.36
79.73
83.37
83.75
DTD
New
41.18
50.96
56.00
56.44
53.62
$5.79
56.88
64-62
99.99
62.56
59.18
62. 68
62.97
62.54
H
54.24
62.01
64.85
64.41
63.24
64.85
64.65
69
64.35
69.43
68.16
70.18
71.75
71.60
Base
92,19
91.67
87.49
78.33
91.67
90. 00
88,
89.17 17
85 64
86.05
94.07
93.83
92.90
93.40
EuroAAT
New
54.74
73.51
60.04
64.69
58.31
66.75
66.87
74.28
64_.4
70.18
73.23
74.30
73.90
76.74
H
68.69
81.59
71.21
70.86
71.28
76.65
76.22
81.05
73.48
77.30
82.35
62.93
82.32
84.25
Base
84,69
91.33
82,33
85,78
80.07
90.68
81.08
81 33
82.89
82.84
83.00
82,99
87. 10
87 44
UCF101
New
56.05
65.81
73.45
74.78
74.50
70.54
76.85
79.83
76.67
78.28
78.66
80.68
80
79.28
H 67.46
76.50
77.64
79.90
77.18
79.35
78.91
80.57
79.65
80.49
80.77
81.82
82.74
83.16
dataset setting is more challenging since there will be both domain- and iluss-inrremenal distribu- tion shift, e.g., from generic object classification on ImageNet (Deng et al., 2009) to satellite imagery recognition on EuroSAT (Helber et al., 2019).
Datasets. For both settings we use 11 datasets: ImageNet (Deng et al., 2009), Caltech (Fei-Fei et al., 2004), OxfordPets (Parkhi et al., 2012), StanfordCars (Krause et al., 2013), Flowers102 (Nils- back & Zisserman, 2008), Food101 (Bossard et al., 2014), FGVC-Aircraft (Maji et al., 2013), SUN397 (Xiao et al., 2010), UCF101 (Soomro et al., 2012), DTD (Cimpoi et al., 2014) and Eu- roSAT (Helber et al., 2019).
Baselines. For finetuning, we consider prompt learning approaches CoOp (Zhou et al., 2022a), Co- CoOp (Zhou et al., 2022b), VPT (Jia et al., 2022), and the state-of-the-an methods SHIP (Wang et al., 2023), KgCoOp (Yao al., 2023), MaPLe (Khattak al., 2023a) and PromptSRC (Khattak et al., 2023b). For each baseline, we apply our method (dubbed OGEN) to obtain an OOD GENer- alization improved version. For fairesss, we use the same implementation details of each baseline, including the prompt length, vision backbone in CLIP (Radford et al., 2021) (i.e., ViT-B/16) and train/test data splitting. The reported results are an average over three random seeds.
4.1 GENERALIZATION FROM BASE TO NEW CLASSES
The base-to-new generalization setting creates strictly class-incremental distribution shift since the base and new class splits in one dataset are disjoint. All prompt learners are trained on the base classes, and tested on the base and new classes separately to evaluate the trade-off between ID and OOD performance. Here we follow (Xian et al., 2017) to report the harmonic mean of base and new class accuracies to quantify such trade-off.
7

---

Published as a conference paper at ICLR 2024
Table 2: Cross-dataset generalization: CLIP finetuning (prompt learning) on the source dataset ImageNet, followed by testing on 10 target datasets. Our method OGEN improves the generalization performance CoOp and CoCoOp all the target datasets.
perfforrc of both
Source
Target
StanfordCars
ImageNat
Caltech101
OxfordPPats
Flowers102
Fond101
FGVCAiR
SUN397
EuroSAT
UCF101
DTD
CoOp
71.51
93.70
89.14
64,51
68.71
85.30
18.47
64.15
41.92
46,39
66.55
63.88
OGEN-CCOPp
71.52
94,60
90.73
65.07
70.55
87.26
19.84
65.77
44,90
49,53
69.36
65.76
CoCoOp
71.02
94.43
90.14
65.32
71.88
86.06
22.94
67.36
45.73
45.37
68.21
65.74
OGEN-CoCoOp
71.28
95.12
91.37
66.04
72.90
86.54
22.95
68.42
46.38
45.82
69.74
66.53
Table summarizes the results on 11 datasets. On average, our OGEN method consistently im- proves the new class accuracy for all the prompt learning baselines. CoOp is particularly interesting since its default learning schedule (200 epochs) is much longer than that of CoCoOp and VPT (10 epochs). Without proper regularization, CoOp inevitably shows more serious overfitting to the base classes 6%.69% on average) with low performance on new classes (63.22%) after long training runs. Our OGEN is especially useful in this case, significantly improving the average new class accuracy of CoOp from 63.22 63.22%% to 69.54%. As also visualized in Appendix C 6(a), the new class gen- eralization notable gains on 3 datasets DTD for texture classification, EuroSAT for satellite image recognition and UCF101 for action recognition, which all demonstrate large inter-class varia- tions. This validates the superior generalizability of OGEN, thanks to its capability of OOD feature synthesis and regularization. OGEN also improves the average base class accuracy of CoOp from 82.69% 83.47%. Specifically, OGEN improves 6 datasets with negligible performance drop on other 5, see Fig. 6(b). The gains on base classes can be attributed to 1) the joint discrimination of known and unknown classes and 2) our adaptive self-distillation method that strikes good ID-OOD performance tradeoff.
For CoCoOp and VPT with a significantly shorter training schedule, they suffer from much less overfitting with higher new but lower base accuracies than CoOp. This makes our OGEN unable to unleash full potential to address overfitting. That said, we find both OGEN-CoCoOp and OGEN- VPT can still improve the average new class accuracy while achieving a similar base class accuracy. We are likely to further improve the base accuracy when given longer optimization schedule that allows more ID-OOD performance balancing.
Among the state-of-the-art methods, SHIP (+CoOp) and PromptSRC are related to our OGEN ap- proach in the use of similar techniques of feature synthesis and self-regularization respectively. Ta- ble 1 shows that OGEN can improve the new class generalization of both SHIP and PromptSRC by exploring the synergy between regularization and OOD feature synthesis. OGEN also consistently improves the average base and new class accuracies for KgCoOp and MaPLe. Fig. 5 uses KgCoOp to exemplify how these methods still suffer from overfitting (although reduced to some extent by various techniques), and how our OGEN improves the learning curves of both base and new classes. It is worth noting that different methods are trained for different numbers of epochs, thus again, they have different levels of overfitting. OGEN improves generalization more over SHIP (200 epochs) and KgCoOp (100 epochs) with long learning schedules (more serious overfitting). Our gains are smaller over MaPLe (5 epochs) and PromptSRC (20 epochs) with short training runs, but larger gains are expected when trained for longer runs.
4.2 CROSS-DATASET GENERALIZATION
Table shows the generalization performance from ImageNet to 10 target datasets. We consider the representative CoOp and CoCoOp baselines with long and short training runs respectively. As shown in the table, our OGEN uniformly improves the generalization performance (across baselines and target datasets) with competitive source dataset performance. The improvements are especially large on those low performing datasets DTD, EuroSAT, UCF101 with large distribution shift from ImageNet. This highlights the benefits of our OOD feature generation module. OGEN also ob- tains reasonable gains on the high performing datasets like OxfordPets that contains similar classes (e.g., different dog breeds) with ImageNet, demonstrating the universality of our approach.
8

---

Published as a conference paper at ICLR 2024
Table 3: Ablating class-conditonnal feature generator 8 (Fq. (5)) and self-distillation method ALMT (Eq. (8)). H: Harmonic mean of the base and new class accuracies averaged on 11 datasets.
Table 4: Class-conditional feature generator: different design choices of no extrapolation from kNN classes, extrapolating per class (Eq. (3)) and extrapolating jointly (Eq. (5))
8
ALMT
Base
New
Basc
New
X
12.69=1008
63.22=0.51 71.66:0.54
No Extrap
03.34=0.266
64.08=0.95 72.46:+0.68
/
02.99
75.15+1.12 69.54-0.34 75.88++0.11
69.02-0.64
Extrap per class
02.9
66.04 06.8
73.52-40.63
83.47=0.300
Extrap jointly
02.49=00955
69.02=1.22 1.25 75.15=1.12
Table 5:
Class-conditional feature generator: kNN retrieval vs. Random sampling of known classes with varying K.
Table 6: Self-distillation: Mean Teacher (MT)
vs. Adaptive Local Mean Teacher (ALMT) with fixed window size or adaptive my (default).
K
Base
New
H
Base
New
1
82.76::0.49
67.01=1.35
74.06=0.63
No distillation
82.49:0.665
69.02:40.64
15.1521.12
2
82.35+00.76
67.79+2.37
74.36-1.19
MT
83.34+0.15
68.30+0.77
0.44
KNN
3
82,49:10.95
69.02::.22
75.15=1.12
ALMT (m 2)
81.70:0.,9
68.47:0.599
04.50=0..55
82.37::0.46
68.85:0..2
75.00=0.13
ALMT (m 9)
82.2140.80
68.57:0.855
74.77=0.29
ALMT (m,)
83,47+0.30
69,54:10.34
0.11
Rand
3
81.69:00.35
68.30:10.38
74.40=0.36
4.3 ABLATION STUDIES
Our ablation studies are conducted using OGEN-CoOp with a meaningfully long learning schedule. We start with ablating the two main components of OGEN: dlass-conditional feature generator & and adaptive self-distillation method ALM.. Table confirms both components are useful (more visuals in Appendix B). We see the feature generator improves the new class accuracy by large margin without hurting the base class accuracy. This suggests the high quality of its generated OOD features and the need joint ID-OOD feature optimization. ALMT shown further improve on both base and new classes, leading to a higher Harmonic mean and a much lower performance variance. This highlights the need of regularizing joint optimization for good performance tradeoff.
Table 4 compares different design choices of our caas-coonitional feature generator. Recall that we adopt an extrapolation scheme that extrapolates new image features from the kNN base class fea- tures. What if we use extrapolation at all, and directly learn a mapping from the new class name to new image features? As shown in the table, this only slightly helps the new class generalization probably because the generated features are not faithful enough from an unconstrained text-to-image mapping. Then between the "Extrapolating per class" and "Extrapolating jointly" schemes, the latter improves on new classes much more significantly, showing the benefits of collaborative class rela- tion modeling for extrapolation. Table further ablates on the number of kNN base classes needed for extrapolation, arriving the best K 3. By comparison, randomly selecting 3 base classes does not perform as well. Finally, Appendix D illustrates the advantage of our feature synthesis approach over replay-based methods using real data.
Table compares various self-distillation baselines applied on top of our feature generator. No- tably, the simple Mean Teacher (MT) is not helping, which inspires us use a local version to completely rule out the early-stage underfit model checkpoints. We further propose Adaptive Local Mean Teacher (ALMT) that calculates EMA within local time window of increasing size m, (from Mmin = 2 to max 9). As shown in the table. ALMT achieves the best performance due to the adaptiveness of m,, which effectively avoids both the underfitting (from early epochs) and overfit- ting (from recent epochs) effects in the teacher model. Apparently, this is not possible with fixed window size (m mmin or mmxx) which hurts performance.
CONCLUSIONS AND FUTURE WORK
In this paper, we study the OOD generalization of recent CLIP finetuners and propose an effective approach to reduce their overfitting to seen classes. For that, a class-conditional feature generator is used to synthesize OOD features for joint optimization, and the optimization dynamics are fur- ther regularized by an adaptive distillation scheme. The superior generalization capability of our approach is demonstrated under different OOD settings. In the future, we plan to go beyond prompt learning and evaluate how much our benefits hold for other finetuning methods like adaptor tuning. Moreover, it would be interesting to figure out how well our "unknown-awarc" approach can model uncertainties unseen data, which can be evaluated on existing OOD detection benchmarks.

---

Published as a conference paper at ICLR 2024
REFERENCES
Emmanuel Abbe, Samy Bengio, Aryo Lotfi, and Kevin Rizk. Generalization on the unseen, logic reasoning and degree curriculum. In ICML, 2023.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101-mining discriminative compo- nents with random forests. In ECCV, 2014.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebas- tian Alexander Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding. Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. PaLI: A jointly-scaled multilingual language-image model. In ICLR, 2023.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. Enabling multimodal generation on CLIP via vision-language knowledge distillation. In ACL, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng. Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. MaskCLIP: Masked self- distillation advances contrastive language-image pretraining. In CVPR, 2023.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. In ICLR, 2022.
Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distributiin detec- tion based on the pretrained model clip. In AAAI, 2022.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: incremental bayesian approach tested on 101 object categories. CVPR workshop, 2004.
Stanislav Fort, Jie Jessie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of- distribution detection. In NeurIPS, 2021.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Yi Lin. Open-vocabulary image segmentation. arXiv preprint 122iv:2112.12143 2021.
Xiuye Gu, Tsung-Y Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2022.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs Remote Sens., 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0255: 2015.
Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-We Wei Chang. Open-domain visual entity recognition: Towards recogniz- ing millions of wikipedia entities. arXiv preprint arXiv:2302.11154, 2023.
Chao Jia, Yinfei Yang, Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and lasion-language representation learning with noisy text supervision. ICMI., 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Nam Lim. Visual prompt tuning. In ECCV, 2022.
10

---

Published as a conference paper at ICLR 2024
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. MaPLe: Multi-modal prompt learning. In CVPR, 2023a.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming- Huuga- Yang. and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In ICCV, 2023b.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and -an-gaag transformer without convo- lution or region supervision. In JCML, 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshops, 2013.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-callbrated classifiers for detecting outt----iistiittiion samples. In ICLR, 2018.
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurlPS, 2021.
Xuanlin Li, Yunhao Fang. Minghua Liu, Zhan Ling, Zhuowen Tu, and Hao Su. Distilling large vision-language model with out -isto--disttitution generalizability. In ICCV, 2023.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre- training paradigm. In ICLR, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
Z. Mal, G. Luo, Gao, L. Li, Y. Chen, Wang. C. Zhang, and W. Hu. Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation. In CVPR, 2022.
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with vision-language representations. In NeurlPS, 2022.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
Rui Qian, Yeqing Li, Zheng Xu, Ming-Hsuan Yang. Serge Belongie, and Yin Cui. Multimodal open-vocabulary video classification via pre-trained vision and language models. arXiv preprint 0rXiv:2207.076644 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
Yang Shu, Xingzhuo Guo, Jialong Wu. Ximei Wang. Jiammin Wang, and Mingsheng Long. CLIPood: Generalizing clip to out-of-distribution In ICML,, 2023.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervise learning with consistency and confidence. In NewrIPS, 2020.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset 101 human actions classes from videos in the wild. arXiv preprint arXiv: /2/ 2012.

---

Published as a conference paper at ICLR 2024
Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. Non-parametric outlier synthesis. In ICLR, 2023.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In NeurIPS, 2017.
Zhengbo Wang, Jian Liang. Ran He, Nan Xu, Zilei Wang, and Tieniu Tan. Improving zero-shot generalization for clip with synthesized prompts. In ICCV, 2023.
Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In CVPR, 2017.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. In CVPR, 2010.
Hantao Yao, Rui Zhang, and Changsheng Xu. guided context optimization. In CVPR, 2023.
Visual-language prompt tuning with knowledge-
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In ECCV, 2022a.
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXi::2110.02235 2022b.
Renrui Zhang, Zhang Wei, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Jiao Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In ECCV, 2022.
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-baset baaeed Languae-immaee pretraining. In CVPR, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. IJCV, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022b.
12

---

Published as a conference paper at ICLR 2024
UCF101
EuroSAT
DTD
100
Accurary (Wg
80
class
Accuracy (ho)
unknoww class
70
8
60
8
50
7*"ar Oan
50
40
30
100
150
200
100 Epachs
230
0
10U Epochs
200
Epochs
Flowers102
StanfordCars
OxfordPets
100
98
Acsuracy (86)
80
ACco Acouracy mayy (96)
75
98
Acouracy
75
82
86
90
88
100 Epochs
100
200
100
200
Epochs
Epochs
Figure 4: More example learning curves the long finetuning runs (200 epochs) with CoOp (Zhou et al., 2022a) method. Under the within-datase generalization setting, CoOp typically overfits the known classes and achieves decreasing accuracy for the unknown classes. The class-conditional feature generator plays key role in our full method OGEN, which reduces overfitting by generating OOD features for the unknown-aware optimization. Our adaptive self-distillation method further reduces overfitting via regularizing the optimization dynamics.
UCF101
EuroSAT
DTD
05.0
80
75
62.5
known class
Accursoy (9G)
unlerown
80
Acouracy
70
77.5
70
85
75.0
70.0
DkpCoOp
80
20
40
50
100
20
30
100
55
0
20
40
30
00
100
Epochs
Epochs
Epochs
Figure 5: Learning curves of the long finetuning runs (100 epochs) with KgCoOp (Yao et al., 2023) vs. OGEN-KgCoOp methods (within-dataset generalization setting). Despite the overfitting reduc- ing technique used in KgCoOp, it still suffers from some extent of overfitting. See how OGEN often improves the learning curves of both base and new classes
A REMARKS ON PROBLEM DEFINITION
The focus of this paper is to understand the behaviors of and improve OOD generalization of CLIP finetuning. As shown in Fig. ((a) of the main paper, the CLIP model be finetuned various downstream tasks before evaluating OOD generalization. Two settings are considered: 1) within- dataset generalization where one dataset has (base) vs. OOD (new) class splits for finetuning and evaluation respectively: 2) cross-dataset generalization with one ID dataset for finetuning and other OOD datasets for evaluation.
Note CLIP is pretrained enormous volumes of data, which inevitably could have class overlap with some "OOD" data for evaluation. Also, there could be potential class overlap between the curated ID and OOD data themselves, e,g. under the cross-dataset generalization setting where the dataset pair may include similar categories in their class taxonomies. Therefore, we consider more of generalized OOD generalization test for the large-scale pretrained CLIP model. That is, whenever the class overlap happens, the intremntnaaemmttalll distribution shift is considered in our evaluation, otherwise evaluate under the strictly class-incremental distribution shift.
13

---

Published as a conference paper at ICLR 2024
(e) OGEN-CoOp va. CoOp on new clesses
(b) OGEN-CoOp vs. CoOp on base clesses
18.77
Nosolute charohargga (94)
884
16
12
9.78
1.88
4.95
2.80
1.18
0.07
-0.2
-0.20
Imegertee
Chtaroiity
Octoraied
RantorCro
Poowre
Foontot
FGNCA
SUNS37
DTD
Eunsay
UCFtoT
imagetier
OxfordPadd
Sannbrdraadd
Plowrrs
Foort01
FavCAiconf
SUNd9T
oro
Eueose
uanoo
Figure Base-to-new class generalization when our OGEN approach is applied to the CoOp baseline that suffers from overfitting due to long learning schedule (200 epochs). OGEN largely overcomes overfitting and (a) improves OOD generalization on new classes for all datasets, some- times by large margin. (b) At the same time, OGEN is able to improve the base class accuracies on most datasets, with only minor accuracy drop on few others.
B OVERFITTING WITH CLIP FINETUNING: MORE EXAMPLES
Fig. 4 shows more learning curves of the prompt learning method CoOp (Zhou et al., 2022a) that finetunes CLIP long schedule. Clearly, CoOp overfits all the considered datasets with de- creasing generalization performance on the unknown classes. While both components of our pro- posed OGEN method class-conditional feature generator and adaptive self-distillation. are found useful to address overfitting. Our feature generator is observed to play a key role by generating OOD features for the following unknown-award optimization and regularization.
Fig. 5 shows example learning curves of one of the state-of-the-ar methods KgCoOp (Yao et al., 2023). This method has relatively long finetuning runs by default (100 epochs). We can that KgCoOp still suffers from some extent of overfitting, although there is an overfitting reducing com- ponent in it. Our OGEN can further alleviate overfitting with KgCoOp, improving learning curves of both base and new classes.
C VISUALIZING PER-DATASET RESULTS
Fig. breaks down the performance gap between CoOp and our OGEN-CoOp on both the base and new classes for 11 datasets. The base-to-new class generalization setting is considered on each dataset.
D FEATURE SYNTHESIS VS. REPLAY-BASED METHOD
Recall the goal of our class-conditional feature generator is to model the vast space and complex distribution of unknown class data in OOD domains. Here we investigate how well our synthesized OOD features can represent the real world of unknown data. Specifically, we explore the use of replay methods by sampling real OOD data from the large-scale LAION-400M dataset (Schuhmann et al., 2021) and using the sampled data for replay. We compare synthesized OOD features against those real OOD data in terms of their contribution to training regularization on downstream tasks.
Experimental details. We experiment under the base-to-new class generalization setting where CoOp (Zhou et al., 2022a) is the CLIP finetuning baseline each of the 11 downstream datasets. For fair comparison between the OGEN- and replay-boostar CoOp variants, we always use ALMT with the only difference in the OOD data source (synthesis replay).
Sampling of replay data.
Class filtering: to ensure the replay data serve as a good proxy of OOD data for each downstream task. need to perform class filtering when sampling image-text-paiie from LAION-400M, i.e., we filter image-text-pain that are semantically close to those "known" classes the considered dataset. We do so by using CLIP calculate the cosine similarity
14

---

Published as a conference paper at ICLR 2024
(a) Average across 11 datasets
(b) Caltech101
(c) DTD
70
00.1
82
78
bi efficienty
Sst efficencc
04.7
91.
efficiency
-5x afficiency
81
82
-Bx efficlenay
75
94.6
80
efficennyy
Hammoric
74
Homonic
94.8
Harmonic
68
23
a111
29
21
50.7
14
2X
4x
7x
5x
25x
2k
44
26
1x
2x
3x
Oo
de De. 10x 1ix 12x
Repley data scale
Replay data
Replay data scale
CoOp+OGEN
CoOpereplay (hard)
CoOp-replay (random))
Figure 7: OOD feature synthesis is much more data efficient than replaying real OOD data. Real OOD data are sampled from LAION-400M dataset with varying data scale, i.e., multiple times more than the synthesized data. Here use the CoOp baseline for basc-to- and measure the Harmonic mean of base and new accuracies.
class generalization, (a) We can see that OGEN with OOD feature synthesis creates about 3.5x and 7.5x gain in data efficiency on average across 11 datasets, when compared to the replay method with hard sample mining and random sampling strategy re- spectively (see text for details). (b-c) The replayed OOD samples, despite being data inefficient, are more helpful when they have a closer semantic or data distribution with the downstream dataset semi-hard negatives. They are far less useful for distributionally dif- ferent DTD, the texture database. On the other hand, OGEN benefits from dataset-specifet OOD feature synthesis. which often offers hard negatives boost data efficiency for training.
(e.g., Caltech so
between the text features of the query text and known class names, and then dropping query texts with the maximum similarity score higher than 0.82. Note our synthesized features are guaranteed to be OOD since they are generated in a class-conditional manner with the query classes disjoint from known classes.
Sampling strategy: with class filtering in place, we consider both random data sampling and hard sample mining to retrieve replay data from LAION-400M For hard sample mining, we first rank the query texts from sufficiently large pool (randomly sampled) in terms of the maximum similarity score mentioned above. Then we simply select the top similar query texts and use the corresponding images as hard replay data (similar to hard negatives from hard negative sampling).
Sampling data size: given the large amount of image-text-pairs in LAION-400M, we in- crease the data size of the replay data (either random or hard) by 1-to- more than the size of our synthesized OOD data on each downstream dataset. Note our feature generator synthesizes the same amount of OOD features as the "unknown" class splits of the training set of considered dataset.
O Observations from Fig. 7.
Better performance is generally obtained with the use of more real OOD data for replay, and performance grows faster with hard mined replay data. When sufficient replay data are used, the performance can surpass that of our synthesized OOD features. This demonstrates the benefits of sampling big and diverse OOD data for training regularization purposes.
However, our feature synthesis approach is much more data efficient than replay-based methods. The key reason behind such advantage is that our feature generator is more likely to generate hard OOD samples to better regularize decision boundaries, in comparison to real-world samples. our case, 1) the feature generator itself trained on the downstream dataset, thus can synthesize dataset-specific OOD features that adapt better to the task at hand. 2) Recall that we extrapolate OOD features from kNN known class features. This suggests there is inevitable shared information between the known class and synthesized features, further increasing the hardness of the latter. On the other hand, both of the afore- mentioned factors are missing for the real OOD data sampled from separate domain, which contributes to their data inefficiency in training regularization. Real OOD data are most useful when they are distributionally similar to the downstream dataset (e.g., Cal- (ech101) and hence can act as semi-hane hard negatives. Otherwise, data efficiency will see a significant drop when replaying OOD data on distributionally distant dataset like DTD. On
15

---

Published as a conference paper at ICLR 2024
average (across 11 datasets), feature synthesis method is about 3.5x and 7.5x more data efficient than the replay method with hard mining and random sampling respectively.
Summary. Replay-based methods perform well with large data size, but suffer from low data effi ciency as well as large memory cost (to maintain replay data). Our class-conditional feature gener- ator avoids these issues by synthesizing hard OOD features on the fly. Note our feature generator lightweight and only incurs small computational cost. Its runtime on GPU is 0.019 seconds, which significantly smaller than that the feature extraction step of CLIP (text encoder: 0.046 seconds, image encoder: 1.016 seconds). One promising future direction is the combination of our feature synthesis method and replay methods, aiming to take advantage of their respective benefits of data efficiency and diversity.
16


Reviews:
[{"review_id": "MwCINaRT0K", "reviewer": "Reviewer_sauS", "summary": "The paper improves the out-of-distribution (OOD) generalization of vision-language models, especially CLIP, when they are finetuned on downstream tasks. The paper makes the following contributions:\n\n- It reveals the overfitting problem of existing finetuning methods, such as prompt learning, that degrade the OOD performance of CLIP models.\n\n- It proposes a novel method called OGEN, which consists of two components: a class-conditional feature generator and an adaptive self-distillation mechanism.\n\n- The paper evaluates OGEN on various downstream tasks and datasets and shows that it consistently improves the OOD generalization of different finetuning methods for CLIP models.", "questions": "My main concern is the baselines selected to compare in this paper are too old (methods published in 2022). MaPle (CVPR'23) and  PromptSRC (ICCV'23) need to be discussed in this paper.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method addresses a novel and important problem of improving the OOD generalization of vision-language models, especially CLIP when they are finetuned on downstream tasks.\n\n- The paper proposes a novel method called OGEN, which consists of two components: a class-conditional feature generator and an adaptive self-distillation mechanism. The feature generator synthesizes OOD image features given the name of an unknown class, by extrapolating from the most similar known classes. The self-distillation mechanism uses an adaptive teacher model that is an exponential moving average of past model checkpoints within a local time window. The teacher model guides the student model to avoid overfitting and maintain a good trade-off between in-distribution and OOD performance.\n\n- The paper evaluates OGEN on various downstream tasks and datasets and shows that it consistently improves the OOD generalization of different finetuning methods for CLIP models. It also provides comprehensive ablation studies and analysis to validate the effectiveness of each component of OGEN.", "weaknesses": "(1) The proposed method mainly compared with CoOp (IJCV'22), Co-CoOp (CVPR'22), and VPT (ECCV'22).\nHowever, before the deadline of ICLR, the state-of-the-art methods are released here: https://github.com/muzairkhattak/PromptSRC\nMaPle (CVPR'23) and  PromptSRC (ICCV'23) need to be discussed in this paper.\n\n(2) In Tab 3 and Tab 4, the proposed class-conditional feature generator slightly decreases the performance of the base classes.\nIn the appendix Fig 5, there are some explanations regarding the performance increase in the new classes and performance variation in the base classes. These discussions need to move to the main script.\n\n(3)\tThe first step of OGEN, novel class extrapolation, is problematic. It is unreasonable to utilize base features to extrapolate novel features, since there are usually large conceptual gaps between base and novel classes. The authors provide a special case, that is “cat, bear->raccoon”. But in CIFAR-10, for example, I think the “ship” class is not conceptual close to any other classes.\n\n(4)\tAnother contribution of this paper, claimed by authors, is Adaptive Local Mean Teacher (ALMT), which I think is just a trivial trick of hyperparameter tuning. The difference between ALMT and conventional MT is just modifying the sliding window size. The novelty is quite low, and the performance improvement brought by ALMT over “No distillation” is insignificant (less than 1%), as shown in Table 6.\n\n(5)\tThe performance of OGEN is too low and outdated. In existing prompt learning papers in CVPR’23 (such as [1,2]) and ICCV’23 (such as [3,4]), the “New” accuracy in base-to-novel generalization setting is already about 75%, but OGEN can only achieve around 70%. \n\n(6)\tThe paper writing is quite poor and hard to follow. The objective function is missing in Sec 3.3.\n\n[1] CVPR 2023. MaPLe: Multi-modal Prompt Learning\n[2] CVPR 2023. LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Models\n[3] ICCV 2023. Self-regulating Prompts: Foundational Model Adaptation without Forgetting\n[4] ICCV 2023. Read-only Prompt Optimization for Vision-Language Few-shot Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UvMt5WbYwu", "reviewer": "Reviewer_Y9rR", "summary": "This paper addresses the limited generalization capabilities of existing vision-language models, which struggle to handle open-domain visual concepts. The authors propose a novel approach called OGEN to improve the out-of-distribution (OOD) generalization of finetuned models. OGEN introduces a class-conditional feature generator that synthesizes OOD features using only the class name of any unknown class, helping to regularize the decision boundary between in-distribution (ID) and OOD data. Additionally, an adaptive self-distillation mechanism is employed to prevent overfitting. Experimental results demonstrate that OGEN achieves considerable improvements in OOD generalization performance across different settings.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The forgetting problem is important in foundation models during fine-tuning.\n* The over-fitting observation can support the paper's main claim.\n* The proposed method is reasonable.", "weaknesses": "I believe it would be beneficial for this paper to include a comparison with relay-based methods, such as sampling a subset from Lioan-5B and using it for replay. The \"class-conditional feature generator\" seems to serve as a proxy for the replay data, so it would be valuable to directly explore the use of replay methods. As a result, I find the novelty of the proposed approach to be somewhat limited considering this concern.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ogTCr4fcvC", "reviewer": "Reviewer_Rp4n", "summary": "This paper proposes a method to regularize ERM for OOD generalization with CLIP models. The method uses a feature prediction network to hallucinate image features corresponding to unknown texts at training time. The resulting training procedure is more robust, since it takes into account synthesized features from unseen classes. The authors also propose a self-distillation mechanism to complement the method.", "questions": "None.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I found the method interesting and novel.\n- I found the paper easy to follow. Section 3.2 and Figure 2 are especially informative and organized very intuitively.\n- The method seems like it could be useful.", "weaknesses": "- the self-distillation mechanism is easy-to-think-of, but this is okay, since it is not the main innovation.\n- My main concern with this paper would be the results. In particular, they are not state-of-the-art (see [Maple] and [Clipood]). Furthermore, the reported CoOp performance seems low. With some tuning, CoOp can be much better, e.g. [KgCoOp] reported a harmonic mean of 74.6 for CoOp on average, compared to 71.7 reported by the authors.  From personal experiments, I know that simply finetuning both encoders along with the prompt with cross-entropy can achieve much better results on these benchmarks, (80.3 % HM on the base-to-novel benchmark, average of 11 datasets). However, the authors seem to focus on just prompt tuning, so it might be ok.\n\nI'm incline to think that that the method is interesting enough for acceptance, even though, in my opinion, the results are not state-of-the-art.\n\n[Maple] Muhammad Uzair Khattak et al. \"MaPLe: Multi-modal Prompt Learning\"\n\n[Clipood] Yang Shu, Xingzhuo Guo et al. \"CLIPood: Generalizing CLIP to Out-of-Distributions\"\n\n[KgCoOp] Hantao Yao et al. \"Visual-Language Prompt Tuning with Knowledge-guided Context Optimization\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C5Rpz9OUSJ", "reviewer": "Reviewer_v4h7", "summary": "The paper works on improving the OOD generalization of finetuned vision-language models. Specifically, a class-conditional feature generator and adaptive self-distillation mechanism are proposed to serve the goal.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[$\\textbf{Interesting Idea}$] The idea of generating unknown-class features is interesting.\n\n[$\\textbf{Presentation Quality}$] The presentation is clear and easy to follow.", "weaknesses": "[$\\textbf{Unconvincing Statement}$] The work mentions that it is the first to unveil the pitfalls of finetuning VLMs by prompt learning can cause overfitting on base classes, resulting in poor performance on novel classes. However, CoCoOp “Conditional Prompt Learning for Vision-Language Models, CVPR 2022” has already observed this and proposed conditional prompt learning to address it.\n\n[$\\textbf{Unclear Model Design}$] This work uses known image and text features as K and V, while unknown text features as Q to generate unknown image features. The rationale behind this is not clear. It would be nice to explain this in more details.\n\n[$\\textbf{Missing Related Works}$] For finetuning methods, there are many works that need to be discussed, e.g., “CLIP-Adapter: Better Vision-Language Models with Feature Adapters”, “Task Residual for Tuning Vision-Language Models”, “Improving Zero-Shot Generalization for CLIP with Synthesized Prompts”, “MaPLe: Multi-modal Prompt Learning” and “Self-regulating Prompts: Foundational Model Adaptation without Forgetting”.\n\n[$\\textbf{Small Performance Gains}$] The results in Table 1 show the improvements from adding the proposed method are rather limited. Moreover, the performance is much worse than some SOTA methods, e.g., “MaPLe: Multi-modal Prompt Learning, CVPR 2023”, “Improving Zero-Shot Generalization for CLIP with Synthesized Prompts, ICCV 2023” and “Self-regulating Prompts: Foundational Model Adaptation without Forgetting, ICCV 2023”. It would be nice to see the performance of these SOTA methods by adding the proposed components.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)