Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models
Authors: Kaican Li
Weiyan Xie
Yongxiang Huang
Didan Deng
Lanqing HONG
Zhenguo Li
Ricardo Silva
Nevin L. Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: robustness
fine-tuning zero-shot models
CLIP
concept descriptions
Primary area: optimization_for_deep_networks
TL;DR: Combining empirical and worst-case risk minimization enhances the robustness of fine-tuned CLIP models.

Abstract: Fine-tuning foundation models often compromises their robustness to distribution shifts. To remedy this, most robust fine-tuning methods aim to preserve the pre-trained features. However, not all pre-trained features are robust and those methods are largely indifferent to which ones to preserve. We propose dual risk minimization (DRM), which combines empirical risk minimization with worst-case risk minimization, to better preserve the core features of downstream tasks. In particular, we utilize core-feature descriptions generated by LLMs to induce core-based zero-shot predictions which then serve as proxies to estimate the worst-case risk. DRM balances two crucial aspects of model robustness: expected performance and worst-case performance, establishing a new state of the art on various real-world benchmarks. DRM significantly improves the out-of-distribution performance of CLIP ViT-L/14@336 on ImageNet (75.9$\to$77.1), WILDS-iWildCam (47.1$\to$51.8), and WILDS-FMoW (50.7$\to$53.1); opening up new avenues for robust fine-tuning. Our code is available at https://github.com/vaynexie/DRM.

Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models
Kaican Li
Weiyan Xie
Yongxiang Huang
Didan Deng²
Lanqing Hong?
Zhenguo Li 2
Ricardo Silva
Nevin L. Zhang"
'The Hong Kong University of Science and Technology Huawei
University College London
Abstract
Fine-tuiing foundation models often compromises their robustness to distribution shifts. To remedy this, most robust fine-tuning methods aim to preserve the pre- trained features. However, not all pre-trained features are robust and those methods are largely indifferent to which ones to preserve. We propose dual risk minimiza- tion (DRM), which combines empirical risk minimization with worst-case risk minimization, to better preserve the core features of downstream tasks. In particu- lar, utilize core-feature descriptions generated by LLMs to induce core-based zero-shot predictions which then serve as proxies to estimate the worst-case risk. DRM balances two crucial aspects of model robustness: expected performance and worst-case performance, establishing a new state of the art on various real-world benchmarks. DRM significantly improves the out-of-distribution performance of CLIP ViT-L/14 336 on ImageNet (75.9-77.11 I). WILDSSiWillCant (47.1 11111111.)) and WILDS-FMoW (50.7->5311) opening up new avenues for robust fine-tuning. Our code is available at https: /gi thub. com/vaynexie/De
Introduction
Foundation models such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) have revolu- tionized machine learning with their remarkable zero-shot and adaptive capabilities. Research has shown that such capabilities are mainly due to robust feature representations gained from large-scale training data (Fang et al., 2022: Xu et al., 2024). The models have been proven useful various downstream tasks (Shen et al., 2022; Zhang et al., 2022; Betker et al., 2023; Pi et al., 2024) and are the cornerstones of large multimodal models (Alayrac et al., 2022; Liu et al., 2023; Zhu et al., 2024).
Fine-tuning is one of the most common approaches to the downstream adaptation of foundation models (Bommasani al., 2021: Shen et al., 2022). However, such adaptation often comes at the cost of robustness (Radford et al., 2021; Pham et al., 2023), resulting in larger gaps between downstream in-distribution (ID) and disti--dittiitution (OOD) performance (Wortsman et al., 2022).
Kumar et al. (2022) showed that fine-tuning tends to distort pre-trained features, and the distortion is exacerbated by randomly initialized heads which would significantly alter the pre-trained features fit ID examples. The proposed remedy, LP-FT, first learns a linear probe (LP) on frozen features, and then followed by regular fine-tuning (FT). Goyal et al. (2023) took this idea further by reusing the pre-trained text encoder of CLIP as the classification head for fine-tuning This method improves LP-FT and is colloquially known as "fine-tune like you pre-train' (FLYP). WiSE-FT (Wortsman et al., 2022) investigated combining pre-trained models with their fine-tuned versions by weight averaging, which can be seen as yet another approach to recover robust features lost during fine-tuning.
Equal contribution, listed in alphabetical order. Correspondence to Izhang @ us..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

clear
unclear
puo skila)
robust
p(x):: high
ERM
WRM
DRM
p(x): low
ERM
WRM
DRM
x
spurious
x
p(x)):
p(x): very
Figure 1: Dual risk minimization (DRM) combines empirical risk minimization (ERM) and worst-case risk minimization (WRM) to complement their weaknesses. this simple binary classification task predicting if there are skis in given image, (i) ERM underperforms when the core features (the appearance of ski) are clear but the non-core features such as background/contex are spurious (i.e. negatively correlated with ski), and (ii) WRM underperforms when the features unclear but the non-core features are robust (i.e. positively correlated with ski). DRM outperforms ERM and WRM under mild conditions such that the core features not always clear and the non-core features are more often robust than not.
While the existing approaches aim to preserve pre-trained features, the fine-tuning process still guided by empirical risk minimization (ERM; Vapnik. 1998), which favors the most predictive but not necessarily the most robust features. In general, there are two kinds of robust features: core features which essentially define the target classes, and non-core features that may aid prediction when the core features are not clear (Gao et al., 2023). ERM models tend to exploit the non-core features even when the core features are clear (Geirhos et al., 2020; Shah et al., 2020). This often harms OOD performance as non-core features are generally less reliable out-of-distributiin.
To better preserve the core features, we propose new principle called dual risk minimization (DRM) which combines ERM with worst-case risk minimization (WRM; Wald, 1945), common principle for domain generalization (Arjovsky et al., 2019; Sagawa et al., 2020; Cha et al., 2021; Kirichenko al., 2023). This combination rests on our view that robustness involves two main aspects: the expected (or average) performance and the worst-case performance over all domains. While there often a trade-off between these two aspects (Tsipras et al., 2019; Teney et al., 2023), Figure illustrates how DRM balances the trade-off to improve overall robustness.
The main challenge of applying DRM real- tasks is to assess of worst-case risk. To this end, we use concept descriptions (Pratt et al., 2023) short texts that describe the core features of each class-obtaine with GPT-4 (Achiam et al., 2023). The description for cougar, for instance, is "a large, tawny cat with muscular build and a small head." We feed these descriptions to a pre-trained CLIP text encoder (Radford et al., 2021) for the text embeddings, which are then used construct soft class labels for each training image according to the similarity scores between the image and text embeddings. The risk w.r.t. the soft labels can seen as a proxy of the worst-case risk and is thus minimized instead. Empirically, DRM significantly outperforms the state of the art on challenging benchmarks such as ImageNet (Deng al., 2009) and WILDS (Koh al., 2021).
In summary, we make the following key contributions in this paper:
We propose dual risk minimization (DRM), a novel approach that combines ERM and WRM to improve downstream robustness of zero-shot foundation models while addressing the intractability of WRM through innovative use of concept descriptions.
We highlight that robustness for many real-world problems concerns both expected and worst-case performance while most previous works focus on only one. We then show that DRM offers simple and effective way to balance these two important aspects of robustness.
We establish strong new state of the art on multiple real-world benchmarks, promising next-level robustness in fine-tuning zero-shot models. On CLIP ViT-I/14@333 DRM achieves significant, over 5% relative improvement in OOD performance over the best bascline method.

---

2
Related work
Robust fine-tuning of pre-trained models. Prior to the work of Kumar et al. (2022); Wortsman et al. (2022); Goyal et al. (2023) which we have introduced, Li et al. (2018) proposed to restrict the I distance between the parameters of pre-trained and fine-tuned models via regularization. Some other work explored updating only small number of ppre-trainedaadd on) parameters (Guo et al., 2019; Zhang et al., 2020; Gao et al., 2024b). Similar ideas (Kirkpatrick et al., 2017: Zenke et al., 2017) were also discussed in continual learning to mitigate catastrophic forgetting (McCloskey and Cohen, 1989). Apart from explicit constraints on model parameters, Ge and Yu (2017) turned to the source of robust features and proposed to incorporate subset of pre-trained data for fine-tuning, while Cha et al. (2022) aimed to enhance the mutual information between pre-trained and fine-tuned features. Jiang et al. (2019); Zhu et al. (2020) added smoothness constraints model predictions for adversarial examples (Szegedy et al., 2013) to help retain robust features. Andreassen et al. (2021) showed that OOD accuracy tends to improve initially but then plateaus the fine-tuning proceeds. For more discussion on related work including concurrent ones, see Appendix B.
Worst-case risk minimization. The study of worst-case risk minimization (WRM) dates back to the work of Wald (1945), which has gradually evolved into what we know as robust optimization today (Ben-Tal et al., 2009). More recently, WRM has been considered (by many) a basic principle for domain generalization (DG; Blanchard et al., 2011; Muandet et al., 2013). A notable example is invariant risk minimization (IRM; Arjovsky et al., 2019), which aims learn core-feature represen- tations from multi-domain data. Such representations, under mild causal assumptions, give rise to classifiers that minimize the worst (Peters et al., 2016). Another key method, GroupDRO (Sagawa et al., 2020), imposes higher penalties on domains with higher empirical risks. Unlike DRM, neither IRM nor GroupDRO formulates WRM as an explicit optimization constraint for ERM. Eastwood et al. (2022) pointed out that sacrificing too much average performance for worst-case performance is not ideal for DG. Hence, they proposed to minimize the risk among the most likely domains. Lastly, our setup is partly similar to Alabdulmohsin et al. (2023) which also relies on external information.
Prompt design for zero-shot classification. To better leverage the capability of zero-shot models, various prompt designs have been proposed. Menon and Vondrick (2022); Pratt et al. (2023); Maniparambil al. (2023) mainly explored prompts for zero-shot classification. Their prompts were generated by LLMs (Radford et al., 2019) with slightly different instructions than ours, not explicitly focusing on core features. For example, Pratt al. (2023) used "Describe an image from the internet of a(n) which may inadvertently introduce descriptions of non-core features in the resulting prompts. The prompts considered by Yang et al. (2023); Yan al. (2023) are closer to ours this respect, where they used LLM -egeerrtd concept descriptions to build concept bottleneck models for interpretable image classification. More recently, Mao et al. (2024) proposed to use context-awant prompts such as "a [context] of [class name]." while Cheng et al. (2024) used both -omain-invariiat and domain-specific prompts generated by LLMs. However, both methods require either image context or specific domain information to generate the prompts.
3 Dual risk minimization
Data model. Let X and Y be the input and ground-truth target variables for which we adopt the following data generation model:
X + (x(xxxxxx Y hxx(;
(1)
where (Xe, are latent variables and E is exogenous noise. We call Xe core features and Xn non-core features of (X, Y). X,, and Y may be correlated due to hidden confounders of (Xe, X,) and direct causal mechanisms between (Xe, X,). Following Peters et al. (2016), we assume the causal mechanisms and the distribution of E are invariant across domains. There are no other hidden variables or mechanisms. Similar models were widely adopted in the literature (Tenenbaum and Freeman, 1996; Mahajan et al., 2021; Mitrovic et al., 2021; Ahuja et al., 2021; Liu et al., 2021; Lv et al., 2022; Ye et al., 2022; Zhang et al., 2023a; Gao et al., 2024a) where X. and X,, are sometimes referred to as 'content" and 'style". We use calligraphic letters such as X' and y to denote the set of possible outcomes of the random variables.
3

---

Ideal objective for robustness. Let D be all possible domains of task, and 9 be some natural distribution over D. By definition, P (d) > 0 for all d E D. Every domain is associated with a data
distribution pa( y; Re, Fn) consistent with (1). Let po(y) be prediction model parameterized by 0 E O. Its risk in terms of negative ooeliikl,hood,
(0) = (aaa)~rll-- log Po (v/[x)]
(2)
can be seen as a measure of its performance in domain d. Let d, E DD be the training domain. For simplicity, we will omit d when it is clear from the context, e.g., Ro, (0) will be written as R,
For real-world applications, we argue that robust model should optimize its expected performance over S while maintaining acceptable worst-case performance across D. The expected performance implies how well the model would perform at the most general population level, while the worst-case performance tells us the model's ' performance in the worst scenario one may encounter. We note that Eastwood et al. (2022) and Zhang et al. (2023b) share a similar view with us on robustness.
We formalize the above intuition as the following constrained optimization problem. namely idealized dual risk minimization (IDRM), which aims to minimize the empirical risk of po(y(2) while ensuring its worst-case risk is below some threshold value a::
min (0) subject to max Ra((0) a. dED
(IDRM)
EEB
IDRM generalizes ERM (Vapnik, 1998) and WRM (Wald, 1945) it reduces to ERM when is large and to WRM when a is small. IDRM also bears some resemblance to IRM (Arjovsky et al., 2019), which involves an implicit WRM constraint. The constraint, however, requires the classification head to be optimal in all training domains and thus may be too demanding in practice, Another closely related work, GroupDRO (Sagawa et al., 2020), proposes to minimize the worst training-domen risk- a more empirical flavor of WRM. Both IRM and GroupDRO rely on ideally grouped training data to capture invariance across domains. In Section 4, we will show that this largely unnecessary for zero-shot models and provide practical solution for IDRM with just single-domain data.
From IDRM to DRM. IDRM can be solved as the following unconstrained optimization problem due to strong duality (proof in Appendix A).
Theorem 1. Strong duality holds between IDRM and the following dual problem:
(R) + X max Ra(()] -aa.
max min X BEE
dED
Let X* be any solution of X' to (3). By the strong duality, IDRM then reduces to mingee (Ra X' max,eD Ra(O)]. The worst-case risk, i.c., maxaęD (0), is still intractable in itself, but it is closely related to the degree to which the model ps (y(x) relies on core features to predict y. This is because for diverse set of domains D, leveraging non-core features would always lead to worse performance in certain domains (Arjovsky et al., 2019; Geirhos et al., 2020). To minimize the worst-case risk, therefore, the model must only utilize the core features to make prediction.
Suppose there is an oracle feature extractor fe that returns faithful representation of the core features of any input As the core features may not always be clear, [a: can be viewed as some distribution over the core features for each x. Let pe (y]xx) be the optimal model that can be built upon fe())). The risk of pa (ylar w.r.t. pe(y)ar) on the training domain dy is given by
R (0) Paw E (ylpp. [- log (y)))))
Assuming Ps (1) is fairly diverse, the risk (0) measures the degree to which the model's prediction is based on the core features and thus can be viewed as proxy for the worst-case risk. Hence, we can replace maxseD R&) with R (0) while still achieving similar optimization effect.
In summary, we relax IDRM to the following DRM formulation:
min (0) + (R) AE
(DRM)
with some properly chosen À 2 O. Now the risk (0) can also be interpreted as a regularization term for ERM to help preserve the core features. In the following section, we demonstrate how to utilize zero-shot models like CLIP models (Radford al., 2021) to estimate pe(y)ar), and subsequently how to apply DRM to robustly fine-tune the same CLIP models.

---

Original
w/o
w/o FG
Original
w/o BG
w/o
image of balance beam.
An image gymmastic horizontal bar. Long metal or wood bar held up hy upright supports.
long, thin piece wood metal elevated the ground.
df cd
0.367
0.262 28.6%%
0.395 +7.6%)
0,333
0.267 ( 19.8%)
0.367 (+10.2%)
0.286
0.281 (-1.%%
0.123 ( 57.0%)
0.258
0.273 +5.8%)
0.131 (-49.2%)
Figure 2: Concept descriptions better capture core features than default prompts. The affinities between images and default prompts (df) are not stable w.r.t. changes in image background (BG) containing non-core features and are insensitive changes image foreground (FG) containing core features, as indicated by the relative changes (gray numbers in parentheses) w.r.t. the affinities of the original images. In contrast, the affinities between images and concept descriptions (cd) are stable w.r.t to changes in BG while being highly responsive to changes in FG, making them good detector for core features. See Appendix D.1 1 for more examples and a full quantitative study on this.
4 Fine-tuning zero-shot models with DRM
Zero-shot models like CLIP typically consist of an image encoder fo and text encoder ge with parameters 0 = (o, 1²). Image classification with such models is usually done by first creating a text prompt by for each class label y E V, and then assigning a probability for each y to an image by
(xp
p0(y) Ever exp(AA (z, by )/T)
(5)
where Ag(x, (.y) = (fo()), Io (f,))) and r is the temperature. The inner product (fo Io. (t,()) can be intuitively understood the affinity between and ly, and we thus denote it As( by.)
The classifier (5) was originally introduced by Radford et al. (2021) for zero-shot classification. We follow Goyal et al. (2023) to directly fine-tune this classifier, viewing the text embeddings ge (ty) as the weights of a standard linear classification head for the image embeddings fa(z). Unlike standard classifiers, however, (5) additionally depends the text prompt ty of which the design is important.
4.1 Dual prompts for fine-tuning zero-shot models
Let T [ly } be the of text prompts used construct the classifier po(y)æ) according to Eq. (5). For such zero-shot classifiers, the general DRM objective (DRM) becomes
min Rs (0; T) + (O)(0: VEB
(6)
where the risks Rg and RS not only depend on the model parameters 0 but also on the prompts T. For Rg. usually set of default prompts, T [ y E }}. like "an image of [class name]" is used (Goyal et al., 2023; Oh et al., 2023). However, such prompts may not be suitable for R as they not specifically designed to bind with core features. In fact, elicit representation that biased towards non-core features. is also know that non-visual and spurious descriptions contribute significantly to CLIP's representation (Esfandiarpoor al., 2024).
will show, default prompts
To generate better prompts for R,, we ask GPT-4 (Achiam et al., 2023) describe the core visual features of each class, producing set of concept descriptions, T = {19d y E VJ. For example, to generate a concept description for cougar, we prompt GPT-4 with
"Q: Generate short sentence that describes the visual features of Cougar. Do not include its function, its surroundings, or the environment it usually inhabits. The sentence should be concise. For example, [goldfish: a long, golden body with back fins]."
and the concept deseription returned is
"a large, tawny cat with a muscular build and a small head."
MMore details about concept description generation can be found in Appendix C.
5

---

Figuratively, the text embedding Iu. (ty") of the concept description represents the core features of the class from the text side. We use it to "pull out" the core features from the image embedding (a) via inner-product. As illustrated in Figure 2, the affinity between an image and its concept description is indeed much better measure of the significance of core visual features. Regarding this point, full quantitative study can be found in Appendix D.1.
Together, the two sets of prompts naturally give rise to the following objective:
min R2(0; T + (0; AE
where we use default prompts T for ERM and concept descriptions T for WRM. We do not use concept descriptions for ERM because it would erode the text-side core feature representation elicited by concept descriptions. This is supported by empirical evidence from our ablation study (Section 5.3) showing that (7) works best among various alternatives.
The dual prompts elicit separate predictions from the same model for the two sub-objectives of DRM. The ERM part, Rg (6; Tra),, is supervised by regular one-hot labels. The WRM part, R, TT*), is supervised by Pe(y] (x) as in (4), Next, we show how the same set of concept descriptions 7 can be used to obtain a good estimate of pe(w(z).
4.2 Estimating (ylz) with concept descriptions
Recall that the oracle model (y(lx) is based on a faithful representation of core features. Since have demonstrated that concept descriptions bind well with core features on pre-trained CLIP models, direct estimate for (y(y))r can be obtained via (5) with ty <- ty and <- On where 0g (øu, 3/b) denote the pre-trained CLIP parameters.
However, there is a crucial caveat. To illustrate, consider image of class For another class y' whose core features are nol present æ, the affinity An, 199) should ideally be very small. In practice, however, we find this is seldom the case. These extraneous affinity values, which we call artifact terms, often vary among classes and lead to poor estimates po(y(z) with high entropy.
To mitigate the impact of artifact terms, we perform simple min-max normalization on E(x, y) = exp(A8, (Ae (z, //T) w.r.t. all training images X, X' labeled the same class y, as follows:
E(x, v) min E(x', .9)
<(x, y) max E(x', y) ming'e, EX, {(x',y)
(8)
This effectively adjusts the affinity range of each class, reducing the difference the artifact terms of different classes. Based on the normalization, the final estimation we propose for pe Po(y/a) (r) is
(2,,))) y),
Yni
(y(xx) [1 11, M=)] Eyeu, (x,y)) V J Yoi
7(E
where Yr is the ground-truth label of x. This ensures that for every class y. there exists at least one x E Xy for which pe (y)la) = 1, promoting balanced learning. When Pe (r) < for y = Yzs the remaining probability is distributed to other classes y # yz according to the relative scale of the respective affinities.
We pre-compute the estimate (y(a)) with the pre-trained CLIP model 0g before fine-tuning, which now the learning target of R2 To)). Since the computation requires the class labels, (v(la= can only be used for training (not for inference). Intuitively. learning from (y(2)) can be seen as form of oif--istiilatton targeted at core features. In comparison, standard oif--istillation methods (Furlanello et al., 2018; Zhang et al., 2019; Ji et al., 2021; Zhang et al., 2021) are largely indifferent to what specific information should be distilled.
4.3 Inference
The fine-tuning objective (7) involves two classifiers: the ERM classifier p (y|=) induced by T and the WRM classifier pad (y|z) induced by Tod. While either alone can be used for inference, find that their mixture,
iduel (g|zz 3 p (y|z) + (1 - B) p5 (v/z),
(10)
where 3 E (0, 1), performs the best. This is expected as (10) essentially combines ERM with WRM as depicted in Figure 1. By default, we set 3 1/(1 X) so to be consistent with (7) possible.

---

Experiments
In this section, we evaluate DRM on multiple real-world benchmarks and conduct ablation studies to assess the impacts of various design choices. We conduct our experiments on three varying sizes of pre-trained CLIP models: ViT-B/16, ViT-L/14 and ViT-L/14@336 (Radford et al., 2021). Finally, we analyze the reliability of LLM--generaedd concept descriptions and the impact of on performance.
5.1 Setup
Datasets.
IMAGENET (Deng et al., 2009) comprises a million natural images across 1,000 classes We use the training set for fine-tuning and the validation set for assessing ID accuracy. For consider ImageNet variants: IMAGENET-V2 (Recht al., 2019), IMAGENET-R (Hendrycks et al., 2021a), IMAGENET-SKETCH (Wang et al., 2019), IMAGENET-A (Hendrycks et al., 2021b), and OBJECTNET (Barbu et al., 2019). We report accuracy for both ID/OOD performance.
OOD evaluation,
WILDS-IWILDCA IWMMDCAM (IWILDCAM) (Koh et al., 2021) contains camera- trap images for wildlife classification, with training images from 200 locations and OOD images from different locations. Both ID and OOD performances are measured using macro F1 scores.
HILDS-FMOW (FMoW) (Koh et al., 2021) is dataset of satellite images from different years and continents for land use prediction. The dataset is split into training, validation, and testing domains based on the year of collection. There is also notable shift between different continents. We report the ID testing accuracy and the worst-region OOD testing accuracy.
DOLLAR STREET-DE DA and GEOYFCC-DA (Prabhu et al., 2022) are datasets for testing model generalization from images specific countries to new ones. For Dollar Street-DA, training images are from North America and Europe, with testing images from other continents. GeoYFCC-DA has similar setup. Model effectiveness measured by accuracy in seen and unseen countries.
Baseline methods. The key baseline we compare our DRM method with is FLYP (Goyal et al., 2023). Following the FLYP paper, we include several baselines that do not utilize the text encoder. These methods are LP (linear probing), FT (fine-tuning), L2-SP (Li et al., 2018), and LP-FT (Kumar al., 2022). In addition, we incorporate some more recent fine-tuning methods for zero-shot vision models. We also consider combining the weight- spacte averaging method, WiSE-FT (Wortsman al., 2022), with DRM and the baselines. For more introduction to these methods, see Appendix
Implementation details. We update both the image encoder and text encoder during tne-ttnnigg following FLYP (Goyal al., 2023). Furthermore, FLYP uses the CLIP contrastive loss (Radford et al., 2021) instead of the standard cross-entropy loss. We adopt this approach for the ERM part of DRM to facilitate comparison. short, when the hyperparameter the WRM loss term vanishes and DRM reduces to exactly FLYP.
the DRM objective (7),
We choose all hyperparameters of DRM and baseline methods based on the performance on the ID validation set, i.e., training-domain validation (Gulrajani and Lopez-Paz, 2021). The hyperparameter of DRM is picked from 21, 4, 5). More implementation details are presented in Appendix F.
5.2 Main results
We report the main results on IMAGENET, IWILDCAM, and FMOW in Table 1 and 2. The results on DOLLAR STREET-DA and GEOYFCC-DA can be found in Appendix E.2. We also compare DRM with some concurrent methods in Appendix E.3. All performance statistics, except some reported by previous papers (which we simply reuse), are averaged over 5 runs with different random seeds. The 95% confidence intervals over the 5 runs are reported.
Table 1 shows the results on CLIP ViT-B/16. the smallest of the three CLIP models. DRM achieves consistently better OOD performance than the baselines across all datasets, with and without WiSE- FT. Without WiSE-FT, DRM attains 5.0%, 12.4%, and 11.1% relative improvements over the best baseline method, FLYP, on the three benchmarks respectively. With WiSE-FT, the improvements remain significant 1.9%, 11.6%, and 9.8% respectively. terms of ID performance, DRM roughly on par with FLYP, with notable advantage on IWILDCAM.
7

---

Table 1: ID and OOD performances of DRM and baselines methods on CLIP ViT-B/16, with and without WiSE-FT, FT. Best performances are highlighted in bold. For IMAGENET, we report the average performance over its OOD test sets. Results on individual test sets are provided in Appendix I.
IMAGENET
IWILDCAM
FMoW
w/o WiSE-FT
W
w/o WiSE-FT
WISE-FT
w/o WISE-IT
WISE-FT
Method
ID
OOD
ID
00D>
ID
OOD
ID
OOD
ID
OOD
ID
OOD
0-shot
68.3
58.7200
8,7.00
11.0:a0
20.4:0m
18.7-00
LP
79.9
57.2+00
80.0a0o
58.3+00
44.5=06
31.1=04
45.5a06
31.7
48.2+a.1
30.5=0a
48.7
31,5
FT
81.4ia 54.810.
82.5
61.3101
18.5
35.0ins
18.1
35.0
68.5au1
39.2-07
68.51a1
41.5
1.2SP
81.6*a1 57.9*0.
82,2+6.1
58.9
48.6=04
35.3+a3
48.6+04
35.3203
68.62a1
39.4=06
68.4
40.3+06
LP-FT
81.8ga1
60.52..
82.1
61.8 11.8
49.7.05
34.7.04
50,2a05
35.7.294
68 42n2
40.4g10
68. 53.5
42, 4207
FLYP
82.6=a0
60.2+a.1
82.9:aa
63.2aa1
52.2=06
35.6212
52.5+06
37.1
68 68.6202
41.3-05
68.9+a.
42.0+09
DRM
82.0aas
63.2
82.4
64.0
54.1
40.0
55.3
41.4:
68.7.a03
45.9
68. 7897 46.1208
Table 2: ID and OOD performances of DRM and FLYP on two larger CLIP models.
IMAGENET
IWILDCAM
FMOW
Pre-trained model
Method
ID
0OD
TD
OOD
TD
OOD
FLYP
84.6=03
73.4+a.
56.0= 11 41,9007
71.2+0.5 48.2+05
WISE-FT
85.1
19.1
57.2207
42.
72.0s04
49.
CLIP VIT-L/14
DRM
85,0102
75.5
61.8105
49.210.
70.910s
51.310.7
DRM-WISE-FT
86,2=a1
76.2
61.6203
49.8
71.440.
51.3+07
FLYP
85,4s02
75,0=a3
58.7+06
45,4+1.
72.5+mu
50.5
WISE-FT
86.1
75.9
60.5+0.
47.1+1.2
72.6+n.3
50.7+06
CLIP VTT-L/140366
DRM
85.9
76.0soo
62.8206
51 51.4
73.820.5
52.5209
DRM-WISE-FT
87.4100
77.1100
62.5104
51.810.
73.8103 53.1
Table 2 compares the performance of DRM and FLYP on two larger CLIP models. DRM again consistently outperforms FLYP in all cases. The previous state-of-the-art OOD performance for IWILDCAM and FMOW are 47.1 and 50.6 respectively. both achieved by FLYP+WISE-FF with CLIP ViT-L/144 336. DRM improves those scores by 10.0% and 5.0% to 51.8 and 53.1 respectively.
Although DRM incurs more computational costs compared to FLYP due to an additional pass through the text encoder, the training and inference costs for DRM only increase by about 20% from FLYP (see Appendix F.3). This cost is insignificant relative to the performance gain.
Overall, our experiments suggest that DRM is highly effective in the robust fine-tuning of zero-shot models, outperforming previous methods by a large margin while maintaining scalability. As a bonus, we show in Appendix E.6 that DRM is also effective in fine-tuning ImageNet pre-trained CNNs.
5.3 Ablation study
We conduct our ablation study with CLIP ViT-L/14 on IWILDCAM. The main results are reported in Table 3 and discussed below. For additional results and details, see Appendix E.4 and E.5.
Impact of dual risks. The DRM objective (7) consists of an ERM term, (0; and a WRM term, 12500 Td). From Table 3, we can see that, in terms of OOD performance, models fine-tuned with only the ERM term (Rows 4 & 5) significantly underperform models fine-tuned with both terms (Rows 1-3). Conversely, models fine-tuned with only the WRM term (Rows 6 & 7) have much worse ID performance. Although the OOD performance is improved. there is still a large gap from the DRM model (Row 1). Note that these hold regardless of the type of text prompts used by ERM/WRM.
Impact of dual prompts for fine-tuning. With both ERM and WRM in effect, we further investi- gate the impact of their prompts during fine-tuning. DRM uses two sets of prompts: default prompts Th for ERM, and concept descriptions add for WRM. Rows 8 & 9 of Table 3 show the performance of models fine-tuned using the same set of prompts for ERM and WRM. either case, the model underperforms DRM (Row 1), validating the use of tailored prompts for specific learning targets.
8

---

Table Results of ablation studies on DRM with CLIP ViT-L/14 performance and WWILDCAM. We use "df" and "cd" to denote the type of text prompts used to produce model predictions. "dual"
refers to the mixture model (10) for inference. "_" means the corresponding loss term not use.
Performance
ERM Re(0: T)
WRM (0: T)
Affinity norm.
Inference w/ model
Row
ID
OOD
dual df
61.8
49.2
df
cd
60.4
cd
54.8
47.2
df
df 56.0
41.9
5
cd
cd 56.9
43.4
df
df 52.4
45.3
cd
cd
51.7
46.3
df
df
df
54.4
45.1
cd
cd
cd
54.0
46.0
10 df cd x
dual
32.1
24.2
Intuitively. 7dd more aligned with WRM than T as the learning targets (9) for WRM are based on the predictions of the pre- pre-trained model prompted by T So, reducing the WRM term R (0; T in some sense limits the divergence of the predictions between the pre-trained models and the fine-tuned models, hence helping better preserve pre-trained (core) features. This explains why using 7 for both ERM and WRM would lead to poorer outcomes (Row vs. 9). ERM targets, typically one-hot class labels, do not capture subtle core visual differences. As a result, ERM would weaken the bond between T and core visual features as perceived by the model during fine-tuning, and therefore reduce the power of WRM in preserving those features.
Interestingly, comparing Row 1 & 8, we find that T not only improves OOD performance but also ID performance. We hypothesize that this is because the core features are better preserved with T and thus the fine-tuned model relies on more diverse set of features make predictions. reducing overfitting and improving ID generalization as well.
Impact of dual prompts for inference. After DRM fine-tuning, we obtain a new CLIP model with updated parameters 0. Since the model is fine-tuned with dual prompts, it is natural to use the same dual prompts for inference, as (10). Indeed, we find that pa (Row 1) outperforms p (Row 2) and p (Row 3) in Table 3. In particular, while 1 is better than ōō² in-distribution and the latter is better out-of out-off-istribbl of-distribution, they underperform pa on both ID and OOD fronts. This is similar to the phenomenon have observed in the fine-tuning scenario. Dual prompts generally reduce overfitting and the effect carries over from fine-tuning to inference.
Impact of aflinity normalization. In Section 4.2, we pointed out problematic issue regarding the direct estimate for Po (o(y)) obtained via (5) with f, + ged and 0 < 0g. To address the issue, we proposed another estimate (x(wx) in (9) based on normalized affinities. Comparing the two approaches, Row 10 of Table shows that the direct estimate leads severe degradation both ID and OOD performance, demonstrating the importance of affinity normalization.
5.4 Reliability of LLM-ggenrated concept descriptions
Consistency across repeated generations. The concept descriptions used in our experiments are generated by GPT-4. Since the generation process is stochastic, it might impact the performance of DRM. To evaluate this impact, we repeatedly ask GPT-4 to generate concept description for each IWILDCAM class for three times and then find the standard deviation of the resulting image-text affinities. The average standard deviation over 20,000 randomly sampled images of IWILDCAM is 0.0061, which is very small compared to the mean affinity, 0.2659. This suggests that the affinities are robust to the randomness in the generation process, which is therefore unlikely have any noticeable

---

Table Performance of fine-tuned CLIP ViT- L/14 on IWILDCAM with concept descriptions generated by different LLMs of various sizes.
Table Performance of DRM under different À on IWILDCAM and IMAGENET with CLIP ViT-L/14 and CLIP ViT-B/16 respectively.
Method
LLM (#params)
ID
OOD
IWILDCAM
IMAGENET
FLYP
N/A
52.2
35.6
A
ID
OOD
ID
OOD
DRM
GPT-3.5 (20B?)
53.4
38.7
0.0
56.0
41.9
82.6
60.2
DRM
GPT-4 (>1T?)
54.1
40.0
1.0
59.1
47.3
81.5
62.5
2.0
60.0
48.1
81.8
63.1
DRM
Llama-3 (8B)
53.8
39.2
3.0
61.8
49.2
82.0
63.2
DRM
Llama-3 (70B)
54.0
39.9
4.0
60.9
48.6
81.9
63,4
DRM
Llama-3 (405B)
53.9
40.5
5.0
60.1
48.5
81.7
63.3
impact on the performance of DRM. Examples of the generated descriptions in Appendix D.2 show that the same core visual features are described quite consistently across generations.
Generation across different LLMs. We also experiment with different LLMs of various sizes (from 8B to over 400B parameters) to generate concept descriptions. The results are reported in Table 4. While larger and more advanced models lead to better performance, DRM is not sensitive to the specific choice of L.M for generating the concept descriptions. Even with a relatively small LLM, Llama-3 (8B), DRM still maintains significant edge over FLYP on IWILDCAM. Examples of the concept descriptions generated by the LLMs are provided Appendix D.3.
5.5 Study on the effect of DRM
As in (7), DRM involves a hyperparameter À that balances the weight of the empirical risk and the worst-case risk. When 0, only the empirical risk is involved during fine-tuning, resulting in an ERM model. As À increases, the influence of the empirical risk reduces, and the resulting model becomes closer to a WRM model. practice, we choose the value of which is often positively correlated with OOD performance (Taori et al., 2020; Miller et al., 2021). In Table 5, we show the ID and OOD performance CLIP ViT-L/I4 fine-tuned on IWILDCAM and CLIP VIT-B/16 fine-tuned IMAGENET under different choices of X.
based on ID validation,
Compared to FLYP (A 0), the results indicate that DRM maintains high-level OOD performance across A between 1 and 5, suggesting that DRM is fairly insensitive to the choice of X. Furthermore, it interesting note that ID and OOD performance improve together on 1WILDCAM as À increases to 3.0. Our further analysis reveals that this occurs because DRM assists in reducing overfitting there a decrease training accuracy (from 88.29 to 87.41) and an increase validation accuracy (from 81.6 6.64 to 82.43) as À increases (from to 3) thereby enhancing both ID and OOD performance. While DRM does not always improve ID performance (e.g., on IMAGENET), the result suggests that it still achieves good trade-off between ID and OOD performance (-0.6 ID performance for +3.2 OOD performance). which in turn could lead better Pareto front.
Conclusion
In conclusion, this paper introduces dual risk minimization (DRM), novel method that enhances the robustness of models fine-tuned from zero-shot foundation models against distribution shifts. DRM combines ERM with WRM, focusing on the preservation of core features which essentially define the target classes. To guide the fine-tuning process, DRM utilizes concept descriptions generated by LLMs like GPT-4. By balancing expected and worst-case performance, DRM overcomes the traditional limitations of ERM and achieves significant OOD performance improvements on multiple real-world benchmarks, establishing new state of the art. Potential future directions include deeper theoretical investigation into DRM as a general principle, using DRM to improve the general robustness of zero-shot models across broad range of tasks, and better understanding the role of concept descriptions in vision- language modeling.
10

---

Acknowledgements
This research was supported in part by Hong Kong Research Grants Council under grant 16204920. Kaican Li and Weiyan Xie were supported in part by the Huawei PhD Fellowship Scheme. Ricardo Silva acknowledges support of the UKRI AI programme, and the Engineering and Physical Sciences Research Council, for CHAI - EPSRC AI Hub for Causality in Healthcare Al with Real Data (grant number EP/Y028856/1).
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint 0rX1v:23303.08774, 2023.
Kartik Ahuja, Jun Wang, Amit Dhurandhar, Karthikeyan Shanmugam, and Kush Varshney. Empirical or invariant risk minimization? sample complexity perspective. In ICLR, 2021.
Ibrahim Alabdulmohsin, Nicole Chiou, Alexander D' A'mour, Arthur Gretton, Sanmi Koyejo, Matt Kusner, Stephen R Pfohl. Olawale Salaudeen, Jessica Schrouff, and Katherine Tsai. Adapting to latent subgroup shifts via concepts and proxies. In AISTATS, pages 9637-9661. PMLR, 2023.
Jcan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurlPS, volume 35, pages 23716-23736 2022.
Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs. The evolution of out-of-distribution robustness throughout fine-tuning. arXiv preprint 1xXiv:2106.188:1 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. /517:29907.0899 2019.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. NeurIPS, 32, 2019.
Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization, volume 28.
Princeton university press, 2009.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e3.. pdf. 2(3):8, 2023.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In NeurlPS, 2011.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2 108.00288 2021.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. NeurIPS, volume 34, pages 22405-2241 2021.
Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual- information regularization with pre-trained models. In ECCV, pages 440-457, Springer, 2022.
De Cheng, Zhipeng Xu, Xinyang Jiang, Nannan Wang, Dongsheng Li, and Xinbo Gao. Disentangled prompt representation for domain generalization. In CVPR, pages 23595-23604 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, 2009.

---

Cian Eastwood, Alexander Robey, Shashank Singh, Julius Von Kügelgen, Hamed Hassani, George J Pappas, and Bernhard Schölkopf. Probable domain generalization via quantile risk minimization. In NeurlPS, volume 35, pages 17340-17358, 2022.
Reza Esfandiarpoor, Cristina Menghini, and Stephen H Bach. If clip could talk: Understanding vision-language model representations through their preferred concept descriptions. arXiv preprint arXiv:2403.164422 2024.
Alex Fang. Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre- training (clip). In ICML, pages 6216-6234. PMLR, 2022.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In ICML, pages 1607-1616. PMLR, 2018.
Han Gao, Kaican Li, Weiyan Xie, Lin Zhi, Yongxiang Huang, Luning Wang, Caleb Chen Cao, and Nevin L. Zhang. Consistency regularization for domain generalization with logit attribution matching. In UAI, 2024a.
Irena Gao, Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang, Out-of-domain robustness via targeted augmentations. In ICML, 2023.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2):581-59) 2024b.
Weifeng Ge and Yizhou Yu. Borrowing treasures from the wealthy: Deep transfer learning through selective joint fine-tuning. In CVPR, pages 1086-1095, 2017.
Robert Geirhos, Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020.
Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In CVPR, pages 19338-19347, 2023.
Ishaan Guirajani and David Lopez-Paz. search of lost domain generalization. In ICLR, 2021.
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning, In CVPR, pages 4805-4814, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical 2021a.
analysis of duitof-disttibution generalization. In ICCV, pages
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, pages 15262-15271, 2021b.
Mingi Ji, Seungjae Shin, Seunghyun Hwang, Gibeom Park, and I1-Chul Moon. Refine myself by teaching myself: Feature refinement via self-knowledge distillation. In CVPR, pages 10664-10673, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-languse representation learning with noisy text supervision. In JCML, pages 4904-4916, PMLR, 2021.
Haoming Jiang, Pengcheng He. Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regu- larized optimization. arXiv preprint arXiv: /9/ 1. 1.03437, 2019.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training sufficient for robustness to spurious correlations. In ICLR, 2023,
12

---

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho. Agnieszka Grabska- BarokkkkBBaikkaaa..aa et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In ICMI., pages 5637-5664. PMLR, 2021.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distrriutiin. In ICLR, 2022.
Xuhong Li, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In ICML, pages 2825 2834.. PMLR, 2018.
Chang Liu, Xinwei Sun, Jindong Wang. Haoyue Tang, Tao Li, Tao Qin, Wei Chen, and Tie-Ya Yan Liu. Learning causal semantic representation for out-of-distribution prediction. In NeurlPS, 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, volume 36, 2023.
Fangrui Lv, Jian Liang, Shuang Li, Bin Zang, Chi Harold Liu, Ziteng Wang, and Di Liu. Causality inspired representation learning for domain generalization. In CVPR, pages 8046-8055 2022.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching, In ICMI., pages 7313-7324. PMLR, 2021.
Mayug Maniparambil, Chris Vorster, Derek Molloy, Noel Murphy, Kevin McGuinness, and Noel O'Connor. Enhancing clip with gpt-4: Harnessing visual descriptions as prompts. In ICCV, pages 262-271, 2023.
Xiaofeng Mao, Yufeng Chen, Xiaojun Jia, Rong Zhang, Hui Xue, and Zhao Li. Context-aware robust fine-tuning. IJCV, 13855))6855-170, 2024.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2022.
John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: the strong correlation between out-of-distribution and in-distribution generalization. In International conference on machine learning, pages 7721-7735. PMLR, 2021.
Jovana Mitrovic, Brian McWilliams, Jacob C Walker, Lars Holger Buesing, and Charles Blundell. Representation learning via invariant causal mechanisms. In ICLR, 2021.
Mazda Moayeri, Sahil Singla, and Soheil Feizi. Hard imagenet: Segmentations for objects with strong spurious cues. NewrlPS, 35:10068-10097 2022.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In ICML, 2013.
Giung Nam, Byeongho Heo, and Juho Lee. Lipsum-ft: Robust fine-tuning of zero-shot models using random text guidance. In ICLR, 2024.
Changdae Oh, Mijoo Kim, Hyesu Lim, Junhyeok Park, Euiseog Jeong, Zhi-Qi Cheng, and Kyung- woo Song. Towards calibrated robust fine-tuning of vasion-language models. arXiv preprint arXiv:2311.01723, 2023.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5):447-1012. 2016.
13

---

Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong. Yonghui Wu, et al. Combined scaling for zero-shot transfer learning. Neurocomputing. 555:126658 2023.
Renjie Pi, Lewei Yao, Jianhua Han, Xiaodan Liang. Wei Zhang, and Hang Xu. Ins-detclip: Aligning detection model to follow human-language instruction. In ICLR, 2024.
Viraj Prabhu, Ramprasaath R Selvaraju. Judy Hoffman, and Nikhil Naik. Can domain adaptation make object recognition work for everyone? In CVPR, pages 3981-3988 2022.
Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does platypus look like? generating customized prompts for zero-shot image classification. In ICCV, pages 15691-15701, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI., pages 8748-8763. PMLR, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, pages 5389-5400. PMLR, 2019.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In ICLR, 2020.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In NeurlPS, volume 33, pages 9573-9588 2020.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and-language tasks? In ICLR, 2022.
Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long. Clipood: Generalizing clip to dittii-distiibuiions. In ICML, pages 31716-31731. PMLR, 2023.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6111 2013.
2013.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. NeurIPS, 33:18583- 18599, 2020.
Joshua Tenenbaum and William Freeman. Separating style and content. In NeurlPS, volume 9, 1996.
Damien Teney, Yong Lin, Seong Joon Oh, and Ehsan Abbasnejad. Id and ood performance are sometimes inversely correlated on real-world datasets. In NeurIPS, volume 36, 2023.
Junjiao Tian, Zecheng He, Xiaoliang Dai, Chih-Yao Ma. Yen-Cheng Liu, and Zsolt Kira. Trainable projected gradient method for robust fine-tuning, In CVPR, pages 7836-7845, 2023a.
Junjiao Tian, Yen-Cheng Liu, James S Smith, and Zsolt Kira. Fast trainable projection for robust fine-tuning. NeurlPS, 36, 2023b.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In ICLR, 2019.
Vladimir Vapnik. Statistical Learning Theory. Wiley, 1998.
Abraham Wald. Statistical decision functions which minimize the maximum risk. Annals of Mathematics, pages 265-280, 1945.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric Xing. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019.
14

---

Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, al. Robust fine-tuning of zero-shot models. In CVPR, pages 7959-7971, 2022.
Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In ICLR, 2024.
An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In ICCV, pages 3090-3100, 2023.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison- and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In CVPR. pages 19187-19197, 2023.
Nanyang Ye, Kaican Li, Haoyue Bai, Runpeng Yu, Lanqing Hong, Fengwei Zhou, Zhenguo Li, and Jun Zhu. Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization. In CVPR, pages 7947-7958, 2022.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, pages 3987-3995. PMLR, 2017.
Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: baseline for network adaptation via additive side networks. In ECCV, pages 698-714. Springer, 2020.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In ICCV, pages 3713-3722, 2019.
Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self-distillatio:: Towards efficient and compact neural networks. TPAMI, 44(8):4388 4403, 2021.
Nevin L. Zhang, Kaican Li, Han Gao, Weiyan Xie, Zhi Lin, Zhenguo Li, Luning Wang, and Yongxiang Huang. A causal framework to unify common domain generalization approaches. arXiv preprint w00iv:2307.06825. 2023a.
Renrui Zhang, Ziyu Guo. Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In CVPR, pages 8552-8562, 2022.
Yang Zhang, Yue Shen, Dong Wang, Jinjie Gu, and Guannan Zhang. Connecting unseen domains: Cross-domain invariant learning recommendation. Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1894-1898. 2023b.
Chen Zhu, Yu Cheng, Zhe Gan, Sigi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. In ICLR, 2020.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024.
15

---

A Proofs
Lemma 1. Let p and q be two probability distributions over X V. The cross-entropy between plyla) and q(ylz) over p(z), i.e., Hp(g) - ((y/2)) is convex w.r.t 9.
Proof. It suffices to show that for any pair of (q1, 92) and ar E [0, 1] we have Hp(aqq + - ox)gg)) S (Hp(1) + a:) H,(22))
Hp(agi + (1 a) = B(a,))p log(ag) + (1 a1 [)]]
I (x.)))p( aa (u/z) - (1 a) log (2(y)z))
(11)
= =allp(91) + +(1 H
Theorem 1. Strong duality holds between IDRM and the following dual problem:
max min X AEG
(2(() + X max Ra(()] - -aa.
(3)
dED
Proof. Recall that IDRM aims to solve for
min Rg subject to max (0) S a. EEB
(IDRM)
dED
Here, Ra(0)- Hy, (v)()))) = E ((.,)>>pal- log Po (3((a)
is the cross-entropy between Pa(u] 2)) and (y(æz) over (z). It follows from Lemma that Ro) is convex w.r.t. (0lxx) for all d E D.
Since the point-wise maximum of multiple convex functions is also convex, maxaeD Ra(0) is convex and therefore IDRM is a convex optimization problem w.r.t. fo(y)a:) By Slater's condition, strong duality holds between IDRM and the Lagrangian dual of IDRM:
max max (f) + X X22 OEB
max
(a -a
(12)
dED
for any a > mingge maxded (0), i.., when a strictly feasible solution to (IDRM) exists.
B Introduction to baseline methods
The key baseline compare our DRM method with is "fine-tune like you pre-train" (FLYP) (Goyal et al., 2023). While traditional fine-tuning of CLIP models adds a randomly initialized classification head on top of the image encoder, Goyal et al. (2023) demonstrated that it is more effective to simply reuse the text encoder. We therefore follow FLYP to update both the image encoder and text encoder in fine-tuning. They now differ only in the loss functions used in fine-tuning. FLYP uses the CLIP contrastive loss (Radford et al., 2021) for the ERM and showed that this is better than using the standard cross-entropy loss. We adopt this approach for the ERM component (the first term) of DRM to facilitate comparison. Thus, the only difference between DRM and FLYP is the additional regularization term in the DRM loss function.
Before the introduction of FLYP, conventional tie-ttnnng of CLIP models involved adding a linear classification head to the image encoder. The linear probing method (LP) only fine-tunes this new classification head, keeping the image encoder fixed, whereas the full fine-tuning method (FT) trains both the head and the encoder. The LP-FT approach (Kumar al., 2022) begins with I.P and then transitions to full fine-tuning.
Besides, L2-SP (Li et al., 2018) and WiSE-FT (Wortsman et al., 2022) are two established fine-tuning variants that restrict the divergence from the pre-trained model. L2-SP specifically integrates an L regularization term into the loss function to constrain the parameter shifts of the fine-tuned model relative to the pre-teai model. WiSE-FT (Wortsman et al., 2022) interpolates parameters of a pre-trained zero-shot model Oz and that of fine-tuned model On using dwise-n p 8_ + (1 p) On.
Alongside L2-SP and WiSE-FT, several works concurrent with our own have also introduced robust fine-tuning methods via reducing the difference between pre-trained and fine-tuned models. For example, CAR-FT (Mao et al., 2024) and the method proposed by Cheng et al. (2024) seek to minimize the distance between the context distributions generated by pre-trained and fine-tuned CLIP
16

---

models. However, significant limitation of them is that they require prior knowledge of image contexts, such as background and viewpoint, which restricts their practical applicability.
Alternatively, Lipsum-ft (Nam et al., 2024) implements a regularization strategy without requiring prior context information, focusing minimizing the 1.2 distance between the affinities of images and random texts from both pre-trained and fine-tuned models. CLIPood (Shu et al., 2023) utilizes a beta moving average for updating parameters during training, and CaRot (Oh al., 2023) focuses on regularizing singular value distributions while incorporating an exponential moving average for parameter updates. TPGM (Tian et al., 2023a) and FTP (Tian et al., 2023b) take a more fine -en-grai approach by autonomously determining the most effective regularization for each layer's s parameters and enabling more efficient learning of layer-specific projection regularization, respectively.
While these methods aim to prevent the fine-tuned model from deviating too far from the pre-trained one, they do not specify which pre-trained features should be preserved. In contrast, our proposed DRM focuses on preserving the dataset-related core visual features, making it a more targeted approach compared to the aforementioned methods.
C Details on concept description creation
We utilized GPT-4 (Achiam al., 2023), leading-edge language model developed by OpenAI, to create concept descriptions. We supplied GPT-4 with the class name and tailored prompts designed to guide the model to focus the essential visual features of the class while omitting unrelated contextual information.
Specifically, for the iWildCam dataset, our prompt to GPT-4 was:
Q: Generate short sentence that describes the visual features of a given animal. Do not include its function, its surroundings, or the environment it usually inhabits. The sentence should be concise. For example, [goldfish: long, golden body with back fins].
After that, we asked it to generate concept description for each wildlife class. For example:
Q: Now the given animal is Cougur.
A: [Cougar: large, tawny cat with a muscular build and a small head.]
Another example is the FMoW dataset. We found it is challenging to encapsulate the core visual features of land used for specific purposes in just one short sentence. Consequently, we employed GPT-4 to generate six sentences for each type of land use. When we implemented DRM on this dataset, we slightly modified the image classifier to use the average of the six text embeddings for each class as the classification head.
For this dataset, our initial prompt GPT-4 was:
Q: The Functional Map of the World (fMoW) dataset is designed identify the functions of buildings and land use by examining satellite images over time.
Your task is to generate six sentences, each describing the visual features likely to be seen in these satellite images, given the functional purpose of buildings and land use. The features you describe should be visible from aerial perspective and consistent across various countries. Each sentence should clearly and com- prehensively describe a visual feature, focusing solely on its appearance without including details of the surrounding area or the purpose of the features, such as "Helipad for urgent aerial access".
After that, we asked it to generate concept description for each land use. For example:
Q: Now the given land use is Road Bridge.
A: [Road Bridge: Visibly characterized by a straight or slightly curved linear structure that spans across natural or man-made obstacles.
17

---

Constructed from materials that appear robust in satellite images, often showing as grey or dark tones against the landscape.
The bridge deck is rypically marked with parallel lines, indicating the lanes for vehicular traffic visible from above.
Elevation above the surrounding area is noticeable, with shadows often highlighting the height and structure of the bridge.
Support features like piers or columns are regularly spaced, visible as distinct vertical elements that support the span.
End points of the bridge integrate with road networks, appearing as seamless transitions from elevated to ground-level roads.]
The full list of concept descriptions generated by GPT-4 for each class of different datasets considered in this work will be publicly available online later.
By utilizing GPT-4, we were able to automatically produce precise concept descriptions of various classes. The use of LLMs to generate class description from the class name is not new, which have been explored in Menon and Vondrick (2022); Pratt et al. (2023); Maniparambil et (2023) and Yang et al. (2023); Yan et al. (2023). Among them, the first three works mainly explored the use of class descriptions for zero-shot classification, while this work focuses on fine-tuning. The class descriptions generated in these studies often include contextual information due to prompts like "Describe an image from the internet of a(n) ...", which may not be suitable for our application of estimating worst-case risk. The descriptions in Yang et al. (2023); Yan et al. (2023) are more aligned with our approach, while they focus on using LLM- --geerrtte concept descriptions to develop concept bottleneck models for interpretable image classification.
D Empirical study on concept descriptions
D.1 Qualitative and quantitative study on the reliability of concept descriptions
Complementing Figure 2, Figure 3 provides additional examples indicating that with the pre-tra pre-trained CLIP models, concept descriptions have the ability to extract core features. To further support this claim, we present a full quantitative study. The study is conducted with Hard ImageNet (Moayeri al., 2022) and consists of two parts. First, we remove image background (BG), and observing how the image-text affinities change for default prompts (df) and concept descriptions (ed) respectively. In the second part, do the same but with foreground (FG) removed.
Table 6: Quantitative study on the reliability of concept descriptions verse default prompts. The affinities to concept descriptions sensitive to changes in the core features of image foregrounds (FG) and remain relatively stable against changes in the non-core features backgrounds (BG).
FG & BG
w/o BG
w/o FG
df
0.3473
0.2393 (-31.1%) 11) 0.2387 (-10.3%)
0.3407 (-1.9%) 0.1 (-55.6%)
cd
0.2660
Table 6 shows the average affinities over all 19,097 images across all 15 classes of Hard ImageNet. The percentages in the table indicate the relative changes w.r.t. the affinities of the original images (FG & BG). The result shows that the affinities of concept descriptions are much more invariant to changes in non-core features than default prompts (-10.3% vs. -)1.1%) Moreover, the affinities of concept descriptions are quite responsive (-55.6%) to changes in core features. In contrast, the affinities of default prompts barely change (-1.9%) in response to the absence of core features. These results indicate that the affinities associated with concept descriptions are reliable indicators of core visual features in the images. contrast, the affinities from default prompts are more sensitive to the changes in non-core visual features.
18

---

of affinity: cd affinity:
0.367
0.262
0.267 An image of gymmastic horizontal bar. Long wood bar held up upright supports.
0.333
0.286
0.281
0.258
0.273
df prompt:
An image of balance bean. long, piece of wood metal that is elevated off the ground.
cd prompt:
of affinity: cd affinity:
0.395
0.265 0.250
0.344 0.254
0.264
0.261
0.272
of prompt:
An image of howler monkey. Four-limbed silhowette with a long tiil and large throat.
An image of hockey puck. small, hard, round black rubber disc.
cd prompt:
af affinity:
0.341
0.248 0.245
0.365
0.297 0.290 An image of seat belt.
cd affinity:
0.240
0.284
df prompt:
An image of volleyball. round, inflated ball made of synthetic leather or rubber.
cd prompt:
Flut, strap metal buckle.
Concept description prompts (cd) yield affinities which are more robust to the change of context information than the affinities yielded by the default text prompts (df).
Figure 3:
D.2 Examples of repeated generations of concept descriptions
To evaluate the stochasticity of LLM in generating concept descriptions, we repeatedly ask GPT-4 to generate concept descriptions for each iWildCam class. Below are some examples:
White White-lipped Peccary:
Output 1: Compact, dark grey body with distinctive while markings around the mouth.
Output 2: Compact, dark gray body with distinctive white markings around the lips.
Output 3: A stocky body with coarse, dark hair and distinct white markings around the mouth.
Waterbuck:
Output 1: Stocky body with long fur, white ring on the rump. and shaggy brown coat.
Output 2: Thick, shaggy brown coat with white ring around the rump and long horns.
Output 3: robust antelope with a shaggy brown coat and white ring around the rump.
These examples show that while GPT-4's outputs for the same class vary slightly, they consistently highlight key visual features. The variations are primarily in the language details, such as the order of features or the terminology used. For example, for the waterbuck, the consistent visual features across all three outputs are the "stocky (thick, robust) body". "white ring on (around) the rump", and "shaggy brown coat".
19

---

D.3 Examples to generate concept descriptions with different LLMs
The examples and experiment results reported our paper are based on concept descriptions generated by GPT-4. We now consider different LLMs to generate concept descriptions for the classes in the iWildCam dataset. present some examples and our findings below.
Examples:
White-lipped Peccary:
GPTT4: Compact, dark grey body with distinctive white markings around the mouth.
GPT-3.5-Turbo: Grayish body with white stripe across the lips.
Llama-3-8b: baldish-brown, fur with distinctive white stripe.
Llama-3-70b: Dark brown pig- like animal with distinctive white stripe on the lips.
Llama-3-405b: Stout, brown body with distinctive white stripe on the lips and sharp tusks.
Waterbuck:
GPT-4: Stocky body with long fur, a white ring on the rump, and shaggy brown coat.
GPT-3.5-Turbo: Antelope species with shaggy coat and a white ring on its rump.
Llama-3-8h: Reddish-brows coat with a distinctive horn shape and long, slender body.
Llama-3-70b: Large, brown antelope with a distinctive white ring on rump and a long, shaggy coat.
Llama-3-405b: Large, brown body with a distinctive pair of curved horns and a white ring on the rump.
Zebra:
GPT-4: Distinctive black and white striped coat with an upright mane and slender legs.
GPT-3.5-Turbo: Medium-sized equid with black and white stripes.
Llama-3-8h: Black and white striped coat with a distinctive mane and a long, slender body.
Llama-3-70b: Black and white striped equine with distinctive body shape and long mane.
Llama-3-405b: Medium-sized, black and white body with a distinctive striped pattern and long, flowing mane.
Findings:
The concept descriptions generated by different LLMs capture common key visual features for the same class, such as "white stripe" for white-lipped peccary, "white ring on the rump" for waterbuck, and "black and white stripes" for zebra.
2. Comparing outputs of GPT-3.5- Turboo and GPT-4, the outputs generated by GPT-4 cover more visual details, while those by G are generally shorter.
3. As model size increases, from Llama-3-88 Llama-3-405b, the generated outputs become more detailed. For instance, Llama-3-8h mentions "white stripe" but does not specify its location the lips as 1lama-3-70b/4055 do. Additionally, Llama-3-8b sometimes makes factual errors, such as describing the incorrect color for white-lipped peccary and waterbuck.
4. GPT-3.5-Turbo sometimes deviated from the prompt instructions, generating outputs that include non-visual features, such the "distinctive call" of the Great Tinamou, which not visible.
Despite variations in the quality of concept descriptions generated by different LLMs, as discussed in Section 5.4, DRM-trained models using descriptions from any of these tested LLMS consistently show significant improvements in OOD performance compared FLaPnnrd models. This demonstrates that the effectiveness of DRM-trained models is not sensitive to the choice of the LLM.
20

---

E
Additional experiment results
E.1
Defailed performance on ImageNet OOD test sets
The average aceuracy across the five ImageNet OOD test sets has been presented in Table We report the detailed results for each OOD test set in Table 7. Without WiSE-FT, DRM substantially outperforms the previous best fine-tuning results by FLYP on ImageNet-R and ImageNet-A, with increases from 71.4 to 77.8 and from 48.1 to 53.3, respectively. Meanwhile, the ID performance is at a comparable level. With WiSE-FT, improvements remain significant, rising from 76.0 to 79.5 on ImageNet-R and from 53.0 to 54.2 ImageNet-A.
Table 7: Performance on ImageNet OOD variants with CLIP VIT-B/16. "OOD" stands for the average performance over the OOD datasets.
w/o WiSE-FT
WiSE-FT
Method
ID
Im-V2
Im-R
Im-A
Sketch
ONet
0OD
ID
Im-V2 V2
Im-R
Im-A
Sketch
ONet
OOD
0-shot
68.3
61.9
77.7
50.0
48.3
55.4
58.7
68.3
61.9
77.7
50.0
48.3
55.4
58.7
LP
79.9
69.8
70.8
46.4
46.9
52.1
57.2
80.0
70.3
72.4
47.8
48.1
52.8
58.3
FT
81.3
71.2
66.1
37.8
46.1
53.3
54.9
82.5
72.8
74.9
48.1
51.9
59.0
61.3
L2-SP
81.7
71.8
70.0
42.5
48.5
56.2
57.8
82.2
72.9
75.1
48.6
51.4
58.9
61.4
LP-FT
81.7
72.1
73.5
47.6
50.3
58.2
60.3
82.1
72.8
75.3
50.1
51.7
59.2
61.8
FLYP
82.6
73.0
71.4
48.1
49.6
58.7
60.2
82.9
73.5
76.0
53.0
52.3
60.8
63.1
DRM
82.0
73.4
77.8
53,3
$2,5
58.6
63.2
82.4
73.9
79.5
54.2
52.8
59.7
64.0
E,2
Performance on Dollar Street-DA and GeoYFCC-DA
We followed the train-test split outlined by Prabhu et al. (2022). As there was no dedicated validation set, split 20% of the training set for validation purposes. The ID and OOD performance results are reported based on ID performance on the validation set and the OOD performance on the test set, which consists of images from countries not included in the training and validation sets.
The results presented in Table demonstrate that, compared FLYP FLYPetrained models. DRM-trained trained models exhibit improved performance on images from new countries.
Table 8: ID and OOD performance on Dollar Street-DA and GeoYFCC- with CLIP ViT-B/16.
Dollar Street-DA
GeoYFCC-DA
Method
ID
OOD
ID
OOD
0-shot
64.0
53.74o.
56.2-1o0
52.3400
FLYP
82.4
71.810.
71.0.103
58.040.
FLYP+WiSE-FT
82.4.0.
72.710.2
71.2:10.3
58.7102
DRM
81,4+02
73.9+0.3
71.8=0.4
62,5+04
DRM-WiSE-FT
82.0+01
74.7=0.2
71.8=0.3
63.0+0.
E.3 Comparison to some more recent methods
As discussed Appendix B, there are some more recent robust fine-tuning methods. We include comparison to some of those methods based on the results of fine-tuning CLIP ViT-B/16 on iWildCam and FMoW datasets. The results are reported Table 9. The results clearly show that, the more recent methods still significantly lag behind DRM in term of OOD performance.
21

---

Table Performance results for iWildCam and FMoW with CLIP ViT-B/16 including some more recent methods.
iWildCam
FMoW
Method
w/o WiSE-FT
WiSE-FT
w/o WiSE-FT
WiSE-FT
OOD
ID
OOD
ID
OOD
ID
OOD
O-shot
8.74000
11.0+00
20.4
18.7+00
LP
44.5106
31.11004
45.5
31.7.004
48.2
30.5
48.7.0.
31.510.3
FT
48.1+
35.0++05
48.1
35.0=0.5
68.5
39.2+0.7
68.5+0.1
41.5+0..
L2-SP
48.6110.4
35.310.3
48.6.104
35.3.0.3
68.61.0.1
39.41006
68.4.00.1
40.3..0.6
LP-FT
49.7
34.7+20.4
50.2+005
35.7=04
68.4
40,4
68.5+002
42.4+00.7
FLYP
52.2.006
35.6.11.2
12.5
37.1
68.61002
41.3
68.9100
42.010.9
CLIPood
18.4
36.1
48.3
36.5.10
68.2
40.8 1009
68.31003
41.2.10.7
TPGM
47.5+003
35.9+0.4
46.8+0.
36,2#003
68 +8.3
39,6=088
67 67.8+02
39. 39.9+0.7
LipSum-FT
50.710.
36.610.7
48.4
36.910.6
68.4100
41.3
68.1103
42.0.0..
CaRot
49.7+t0.4
34.3++0.3
48.3+0.3
34.7=0.3
68.8=0.2
39.8+0.6
68.3+02
40.7
DRM
54.1++0.5
40.0
55.3+04
41.4=1077
68.7=
45.9
68.7+02
46.1++0.8
E.4
Full Ablation Study
Setup. Given a labeled dataset {x;; 1)) sampled from the training domain dg, the final DRM objective (7), i.e. (0; T + AR (0; Taa), for fine-tuning zero-shot models can be expanded as
n
log po (x)
Σ E'EY
p (3'(x,) [x,) log ( (v/ lz.).
(13)
where pg (y)z)) and p (y|z) are the classifiers (5) induced by the default prompts T {(yy ly
}/} and the concept descriptions T (tga y E V), respectively; and pe (ylx) = Pe (v (r) which defined by (9) to estimate pe (v/zz) Here, we use p% (y/xx) (where pr stands for 'proxy') instead of (y(x)) to ease the discussion of possible variations of DRM.
Consider the following generalized form of (13) with three varying options, t1 and t2 indicating the classifier types defined with different sets of text prompts, and type indicating the type of model used as the proxy for (y]rr):
log PO (uA)z
type pttps(y Σ y'EY
Poo ( (2/))) log 2 [
(14)
As stated in (13), our final DRM training objective (7) uses t1 = df in the ERM term, with t2 = cd and type = pr in the regularization term. We denote this standard setting, (S) in short. We conduct the following ablation study with the pre-trained CLIP ViT-L/14 and fine-tune the model on the iWildCam dataset, with results presented in Table 10.
(a) Inference options after dual classifier training: Two classifiers are involved in our DRM training: f8t (ylæ) and p5d (y/z), zz). As outlined in (10). we combine both classifiers for inference. An alternative is to only use one of the two classifiers for inference. We denote inference with only P (y]x)) as (al), and with only fp (y/x) as (a2). The comparison between (al), (a2), and (S) in Table 10 shows combining both classifiers for inference enhances both ID) and OOD performance compared to using either alone. This reveals that the two classifiers have complementary effect as illustrated in Figure 1, and corroborates our view that ERM and WRM are both vital to OOD robustness.
(b) Vanilla DRM using a single set of text prompts: In our standard DRM setting (S), t1 = and t2 = cd. The vanilla DRM we discussed in Section 4 uses t1 = t2 df. A Alternatively, one can also consider t1 = t2 cd. We experiment with these two alternative settings denoted by (bl) and (b2) in Table 10. The contrast between (bl) and (S) confirms our intuition: using the concept
22

---

Table 10: Ablation study on DRM with CLIP ViT-L/14 (w/o WiSE-FT) on iWildCam.
Specification
Performance
General setting
t1
t2
type
Classifier comb.
Infer w/
ID
OOD
Standard DRM
cd
pr
joint training
(10)
61.8
49.2
(a)
after dual classifier training
Infer with one classifier
(al)
(a2) df
df
cd
cd
pr
pr
joint training
joint training
df
cd
60.4
54.8
47.2
45.1
Vanilla DRM using
(bl)
df
df
pr
joint training
df
54.4
45.1
one set of text prompts
(b2)
cd
cd
pr
joint training
cd
54.0
46.1
(cl)
df
cd
cd
joint training
(10)
32.1
24.2
(c) Use different proxy models
(c2) df
cd
pr-df
joint training
(10)
54.4
45.1
(c3) df
cd one-hot
joint truining
(10)
57.3
45.1
Use only one
(dl)
df
/
56.0
41.9
(d)
(d2)
cd /
56.9
43.4
risk training
(d3)
/ cd
pr
51.7
46.3
(c)
Combine independently
(el)
df
cd
pr
model ensemble
59.7
45.7
trained classifiers
(e2)
df
cd
pr
weight average
(10)
57.5
44.7
descriptions Toad for p (v) i.e., t2 = cd, enhances robust feature preservation and leads to better OOD performance. The other alternative (b2), which employs ' for both i.e., t1 = t2 = cd, slightly improves (bl). Intriguingly, (b2) is still much worse than (S) despite they both use cd for t2.
f% (y/ar) and piz (y)
(c) Proxy model design: In standard setting, type pr. As discussed in Section 4, the proxy term Po (3a) is based on the affinity Ao, (x, 1g = (foo (z), Iwo (fg))) according to the pre-trained CLIP model 6g = (øu, 2o and the set of concept descriptions T In Section 4, we also mentioned the following direct estimation of the oracle model pe (g)lar):
exp(AA, (I, (2, //7)
po (g)lr) EvEy exp (Ao, (x, tgt)
(15)
However, discussed in Section 4, Por (y(æ) ) is susceptible artifact terms. Consequently, made technical adjustment mitigate the influence of these terms, resulting in the refined (e(y))) which denoted as re, (y (r) here. As shown in Table 10, the importance of this adjustment is empirically verified by the much lower performance of (c1) compared to (S).
One can also define p.r (y/ar) by replacing Tea with T in the formulation of 1 (y(zz) As shown by the result of (c2), this alternative still underperforms i (yyl (z) used in the standard setting. This discrepancy can be explained by the fact that the affinities between default text prompts and images easily affected by changes in the non-core visual features instead of focusing on the core visual features, which has been discussed in Appendix D.
Another simple alternative, denoted by (c3), is to employ the ground-truth one-hot labels as the proxy. Perhaps unsurprisingly, the OOD performance of (c3) is notably inferior to (S) based on the affinities between the images and the concept descriptions.
(d) Training with either ERM or WRM: Training with only the first term in (7) results in ERM models (dl) and (d2), whereas training with only the second term leads to a WRM model (d3). Comparing them with DRM models (al) and (a2), it is clear that models trained to minimize single risk underperform those trained to minimize both risks, highlighting the importance of dual risk minimization.
(e) Classifier combination strategy: Our standard DRM training jointly minimizes the two risks, one can also train an ERM model pet (y/x) and a WRM model psd (y/z:) separately. These models can be combined for inference using techniques like model ensembling or weight-space averaging. The two rows of Table show that combining (dI) and (d3) via model ensembling or weight-space averaging generally underperforms joint training (S).
23

---

E,5 Results of applying DRM with FT and LP-FT
Our experiment results reported in Section are based on combining DRM with FLYP. Specifically, we follow FLYP to update both the image encoder and text encoder, and utilize the FLYP loss for the ERM component of (7). The results showed in Table indicate that even without FLYP, where DRM is applied with standard full fine-tuning (Row 2), our method still outperforms FLYP (Row 1). which itself has been shown to surpass standard full fine-tuning in the ERM setting (Goyal et al., 2023). This performance advantage is demonstrated in the results of fine-tuning CLIP ViT-L/14 on iWilCCam:
Table 11: Results of applying DRM under different settings.
Row
FLYP
LP-FT
DRM
ID
OOD
1
Yes
No
No
56.0
41.9
No
No
Yes
54.4
43.9
3
No
Yes
Yes
56.5
46.3
Yes
No
Yes
61.8
49.2
As the table above demonstrates, combining DRM with LP-FT (Row 3) enhances performance over just DRM with standard full fine-tuming (Row 2). Furthermore, integrating DRM with FLYP (Row 4) yields even more significant improvements. Consequently, we adopt the combination of DRM and FLYP default setting when fine-tuning CLIP models.
E.6 Results of applying DRM on ImageNet pre-trained ResNet50
While this work focus on the fine-tuning of zero-shot models that pre-trained large-scale image-text pairs, we also explore the possibility of applying DRM on fine-tuning the ImageNet pre-trained CNN models.
When applying DRM to CNN models, we add two randomly initialized classification heads on top of the model. Analogous the application of DRM zero-shot models, one classification head trained using cross-entropy loss with respect to the ground-truth labels, while the other trained using cross-entropy loss relative to the soft labels generated the pre-trained zero-shot model. We employ this approach to fine-tune an ImageNet pre-trained ResNet50 model on the iWildCam dataset, utilizing soft labels generated by the CLIP ViT-L/14 model. During inference, the outputs from the two classification heads are combined in a manner similar to that described in (10). The results presented in Table 12. It is evident from the results that DRM significantly enhances the OOD performance of ResNet50 compared to the ERM.
Table 12: Results of applying DRM on fine-tuning ImageNet pre-trained ResNer50 on iWildCam.
Method
ID
0OD
ERM+FT
51.6
33.7
ERM+LP-FT
50.5
36.4
DRM+LP-FT
51.0
39.1
F Training details
F. Hyperparameter settings
We primarily adopted the hyperparameter settings from the code released by FLYP (Goyal et al., 2023).
Specifically, for iWildCam, the settings were as follows: training 2pochs=20, learning rate=1e-5, batch-size=256, and optimizer=Adami with weight decay=0.22
For FMoW, the settings were as follows: training epochs=20, learning rate=1e-5, batch-size=255 and optimizer=Adamv with weight decay=0.2.
24

---

For ImageNet, the settings were follows: training epochs=10, learning rate=1e-5. batch-size=255 and Addimi with weight decay=0.
In all experiments, for the images, we applied the standard CLIP image pre-processing, which included resizing, center cropping, and normalization. For the texts, applied the standard CLIP text pre-processing, which tokenized the texts into a series of integers, each representing a unique series of characters.
The value used in our DRM training was picked from {1, 3,,,,,,, 4,5} based on the performance on the ID validation set. Following Goyal et al. (2023), also implemented early stopping based on the ID validation performance.
The hyperparameter in WiSE-FT, w p Oz5 + p) On, is chosen from the range 0.1 to 0.9 via II) validation.
F.2 Machines
All experiments were conducted on a high-performance computing cluster equipped with NVIDIA DGX H800 nodes. Two H800 GPUs with 80 GB VRAM were utilized for all trainings involving CLIP ViT-B/16 and CLIP ViT-L/14, while four H800 GPUs were employed for the training of CLIP ViT-L/14@336.
F.3
Analysis of computational costs
In our experiments, we implemented the ERM part of DRM with FLYP (Goyal et al., 2023). The additional computational cost of DRM, compared FLYP, primarily arises from the preparation of soft labels for the targets of the second risk in DRM and the minimization of this second risk. In short, the training and inference cost for DRM increased by about 20% from FLYP. This additional cost is insignificant compared to the attained performance gain. In Table 13, we detail the computational costs and timing for fine-tuning CLIP ViT-L/14 models on ImageNet.
The computation time reported below is based on the setting that training batch size=256 and inference batch size=1024. There 1, 1,281,167 training images in ImageNet, and thus there 5005 training batches.
Table 13: Analysis of computational costs for DRM.
Generating Concept Description by LLM
Label Generation (9)
Training
Inference
FLYP
N/A
N/A
In average: S8s/100 batches. mins
Inference on 100 batches of images takes ~1.5.
training epoch
DRM
We utilized the GPT-4-turbo API generate concept descriptions 1.000 ImageNet classes, inputting classes
The primary computational cost from using pro-trained CLIP models to generate image and text 1,281,166 training images and I 1,000 concept descrip- tions. Soft labels are created using inner products between these embed- dings, with some technical adjust- ments. The entire process takes less than minutes.
In avernee: 7ls/100 batches, mins per training epech
Inference on 100 batches of images takes ~1.8.
ensure qual- ity. The generation cost is under US Dollar (the price of the API is US Dollar'l million prompt lokens). We are unaware putational the model details of GPT-4 are unknown.
embeddlings
com-
25

---

G
Limitations
Our research utilized GPT-4 to generate concept descriptions for the core visual features of various classes across different domains. It is important to note that the scope of GPT-4's knowledge in certain domains might be limited, and as result, the model may not always generate useful concept descriptions, For instance, we found that GPT-4 generated inaccurate concept descriptions in medical imaging fields like ocular disease and breast histology.
Additionally, due to the vast number of concept descriptions generated, we have not been able to verify the accuracy of each generated concept description. To enhance the quality of these descriptions, potential improvements could involve engaging domain experts to review and correct errors, or generating descriptions manually. Another approach could be to gather visual prototypes and use advanced multimodal LLMs such as GPT-4V (Achiam et al., 2023), LLaVA (Liu et al., 2023), or MiniGPT-4 (Zhu al., 2024), which might yield more precise descriptions of the core visual features.
further limitation concerns the CLIP models used in our experiments. These models may not perform optimally across all domains, particularly in less common areas, where they may lack requisite knowledge in both images and text. The effectiveness of our DRM method therefore contingent upon the breadth and depth of the pre-training data of CLIP models. Unfortunately, the specifics of the CLIP pre-training dataset have not been disclosed by OpenAl, adding an element of uncertainty to the performance of method in niche domains.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in Appendix G.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
The complete proof to proposed theorem has been provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The specifies of DRM training objective and its implementation are detailed in Section 4.2 and 5.1. Information regarding the generation of concept descriptions can be found in Section 4.2 and Appendix C. with sample concept descriptions included in the supplemental material. Additional training details, including the choices of hyperparameters, are provided in Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
28

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The datasets employed in experiments described in Section 5.1 and all publicly accessible. The code of our work will be made available the given GitHub repository.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines fattps://nips.co cc/ Btapli/gguiide//deeeuubbbn wttps/uumsssssoooooooooooooeeeee onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //n http:/uullcggguuuceee /tteg/guuddeeeeeediiiiieeeeeeeee (CodeSubmissseeeeeee ssisnsoliccyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details of data split have been provided Section 5. All other training details, including the selection of hyperparameters, optimizers and others provided in Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification; Yes. The 95% confidence intervals over five training runs are reported in Table and 2.
Guidelines:
The answer NA means that the paper does not include experiments.
29

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information is provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: h/tp:://////.u///ee/eee wwtpuuripss.pppplieeeeepepeeeeee httpppic/tthissuuiiiieee
Answer:[Yes)
Justification: We have reviewed and ensured the research conducted in this paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We do not particular negative socictal impacts of the work performed.
30

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not see any data or models in this work that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of assets (e.g., code, data, models), used in the paper, have been properly credited.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
31

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcae thcode. com/ comddataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The specifics of our DRM model training objective and its implementation are detailed in Section 4.2 and 5.1. detailed documentation for running our codes will be provided in the GitHub respository.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
32

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "IY5lkTIvF8", "reviewer": "Reviewer_D5j1", "summary": "This paper presents Dual Risk Minimization (DRM), a novel approach that combines empirical risk minimization (ERM) and worst-case risk minimization (WRM) for fine-tuning the CLIP model while maintaining its out-of-distribution robustness. The idea is to create a classifier that utilizes concept descriptions of each class generated by large language models (LLMs). This classifier helps “pull out” core features from the image embeddings, serving as a regularizer to enhance the robustness of CLIP fine-tuning. Moreover, the paper introduces a min-max normalization technique to address a caveat in the regularization. Experimental results demonstrate the effectiveness of the proposed DRM, yielding promising in-distribution and out-of-distribution accuracy.", "questions": "Besides the weakness shown in the above section, please also see the following questions: \n\n- Given that the paper suggests that the classifiers built with LLM-generated concept descriptions have the ability to extract core features, could we simply fine-tune the CLIP model utilizing these concept descriptions instead of using the one with default descriptions? If we do so, could standard fine-tuning with ERM already maintain its robustness? \n\n- What would be the in-distribution and out-of-distribution accuracy for the concept description classifiers? Given that these classifiers aim to extract more core features, would they be more robust to distribution changes, even without fine-tuning? \n\n- How does the min-max normalization affect the training on long-tailed datasets, like iWILDCam? Specifically, since the min-max normalization computes $x$ with all images in a class $y$, could tailed classes result in less stable normalization because they have much fewer samples?", "limitations": "The paper acknowledges that a potential limitation of the proposed method lies in the potential limited domain knowledge of LLMs in specific domains. To better illustrate this limitation, it would be beneficial for the paper to provide concrete examples, such as failure cases, that show how the method behaves when it reaches its limits.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and generally easy to follow. \n- The paper brings an interesting viewpoint for CLIP fine-tuning that separates the image features into core (content) and non-core (style) features. \n- The experimental results look promising, especially on the two specialized datasets, iWILDCam and FMoW.", "weaknesses": "- At Line 199, the paper claims that the LLM-generated concept descriptions can “pull out” the core features from the image embeddings. However, this claim is only demonstrated by the few examples shown in Figure 2, which is insufficient. Moreover, the stochastic nature of LLM generation can further complicate the matter. The paper lacks a thorough analysis of the reliability and robustness of the core features obtained through this proposed method. \n\n- At Line 217, the paper mentions that the affinity between $x$ and any other class $y'$ should ideally be zero, thereby motivating the modification of the second term in Eq. 10. However, this might ignore the fact that the ground-truth class $y$ exhibits varying degrees of similarity with different $y’$. It remains unclear how the proposed method can achieve better performance, as shown in Table 3, while disregarding the learning of such inter-class similarities.\n\n- The second term in Eq. 10. is similar to knowledge distillation, as acknowledged by the paper at Line 207. However, the paper neglects to discuss or compare this term with other robust fine-tuning methods, such as [1], which also incorporate knowledge distillation. \n\n[1] DELTA: Deep Learning Transfer using Feature Map with Attention for Convolutional Networks, ICLR 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's time and insightful feedback. Regarding the concerns and questions the reviewer raised, we provide detailed clarifications below. If not specified otherwise, all experiments below are conducted with CLIP ViT-B/16. \n\n---\n### 1. Reliability of the core features obtained with concept descriptions\n\nPlease see **A. Quality and Robustness of LLM-Generated Concept Descriptions** in the global response. \n\n---\n\n### 2. “At Line 217, the paper mentions that the affinity between $x$ and any other class $y'$ should ideally be zero ... this might ignore the fact that the ground-truth class $y$ exhibits varying degrees of similarity with different $y'$ ...”\n\nThis is not what we stated in the paper. At lines 217-218, please note that “$x$ is *completely void* of core visual features for $y'$” and only in such case the affinities should ideally be 0. \n\nIt is generally true that, as you mentioned, “the ground-truth class $y$ exhibits varying degrees of similarity with different $y’$”. Our method does *not* disregard the learning of such inter-class similarities; on the contrary, our method specifically captures the similarities with Eq. (11) where the probabilities of non-ground-truth classes are weighted by their affinities w.r.t. the image. \n\nTo illustrate this point, for each class in ImageNet, we identify the classes with the highest average probabilities in the soft labels generated by Eq. (11). Below are some examples: \n\n- horizontal bar: horizontal bar, balance beam, parallel bars, pole, swing; \n- whiskey jug: whiskey jug, water jug, pitcher, beer bottle, beaker; \n- goldfish: goldfish, rock beauty, tench, barracouta, anemone fish.\n\nWe can see that the top-1 class is consistently the class itself as intended (in fact, this holds for all 1k ImageNet classes), and the other top classes are indeed visually related to the top-1 class. \n\n---\n\n### 3. Comparison with knowledge distillation methods \n\nYes, the term is related to knowledge distillation, or more precisely, self-distillation. There is a self-distillation method, L2-SP, in Table 1 of our paper. Here, we compare our method with two more state-of-the-art self-distillation methods, CaRot [1] and MIRO [2]. Their ID/OOD performances are shown in the table below.\n\n||ImageNet |iWildCam |FMoW | \n|--|--|--|--| \n|FLYP|82.6/60.2|52.2/35.6|68.6/41.3| \n|CaRot|83.1/62.5|49.8/34.3|68.8/39.8| \n|MIRO|----/----|51.6/37.2|66.1/42.2| \n|DRM|82.0/63.2|54.1/40.0|68.7/45.9| \n\nWe will further discuss related methods in the next version of the paper, including CaRot, MIRO, and DELTA [3] which you kindly suggested. Please note that DELTA has been surpassed by MIRO [2] on various benchmarks so we only included MIRO here. \n\n---\n\n### 4. “... could we simply fine-tune the CLIP model utilizing these concept descriptions? ...” \n\nPlease see **B. Why not directly fine-tune a concept-description classifier?** in the global response. \n\n---\n\n### 5. “What would be the ID and OOD accuracy for the concept description classifiers? ... would they be more robust to distribution changes, even without fine-tuning?” \n\nAs shown below, they are slightly better than the default-prompt classifiers, but much worse than the fine-tuned DRM classifiers. In short, concept descriptions are helpful, but fine-tuning is still important, especially when the domain gap is large, e.g., in the cases of iWildCam and FMoW. \n\n|Model|ImageNet |iWildCam |FMoW | \n|--|--|--|--| \n|Zero-shot w/ default prompts|68.3/58.7|8.7/11.0|20.4/18.7| \n|Zero-shot w/ concept descriptions|68.6/59.0|11.49/12.84|20.6/19.8| \n|DRM|82.0/63.2|54.1/40.0|68.7/45.8| \n\n---\n\n### 6. “How does the min-max normalization affect the training on long-tailed datasets ...?” \n\nThis is a great question. We concur that using min-max normalization on long-tailed classes can lead to less stable normalization. That said, its impact on overall performance should be small.  \n\nThe reason is two-fold. First, even if there are only a few samples for a class, most samples of the class would still have higher affinities with the class compared to samples from other classes. This means that $\\gamma(x, y)$ would be small for $y \\neq y_x$, and thus any impact on the estimation of $p_c(y|x)$ (via Eq. (11)) would likely also be small. Second, for the estimation of $p_c(y_x|x)$, although with fewer samples the difference in $p_c(y_x|x)$ between $x$ of the same class $y_x$ would likely be magnified, their relative ordering would still be intact, and thus the classifier would still learn to preserve the core features of the class. \n\nSurprisingly, we find that DRM can even enhance the learning of long-tailed classes on iWildCam. This is reflected by the table below where we can see the training F1 score increases while the training accuracy decreases as $\\lambda$ increases. \n\n| $\\lambda$ | training acc | training F1 score | ID val acc | ID val F1 score | \n|--|--|--|--|--| \n| 0 | 92.37 | 67.72 | 79.44 | 46.64 | \n| 1 | 88.29 | 78.96 | 81.64 | 51.36 | \n| 3 | 87.41 | 79.44 | 82.43 | 52.68 | \n  \nIn particular, for classes with fewer than 50 training examples, we find that FLYP achieves an accuracy of 59.61% on the ID validation set, whereas DRM achieves a significantly higher accuracy of 69.38%. \n\n---\n\n### 7. About limitations \n\nThank you for suggesting the use of specific examples to illustrate DRM limitations due to LLMs' restricted domain knowledge. In our next version, we will showcase GPT-4's inaccuracies in medical imaging fields like ocular disease and breast histology. \n\n---\n\n**References:**\n\n[1] Towards calibrated robust fine-tuning of vision-language models. arXiv, 2023.  \n[2] Domain generalization by mutual-information regularization with pre-trained models. ECCV, 2022.  \n[3] DELTA: Deep Learning Transfer using Feature Map with Attention for Convolutional Networks, ICLR 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nCRhtymvse", "reviewer": "Reviewer_eddC", "summary": "This paper introduces dual risk minimization (DRM), a novel approach that combines empirical risk minimization (ERM) with worst-case risk minimization (WRM) to enhance the robustness of fine-tuning zero-shot foundation models. The authors address the limitations of existing methods that fail to effectively preserve robust features during fine-tuning by utilizing concept descriptions from LLMs to create soft labels for estimating worst-case risk, focusing on core features that define target classes. Empirical results demonstrate that DRM significantly improves out-of-distribution performance on benchmarks such as ImageNet and WILDS, establishing new state-of-the-art results.", "questions": "Please refer to the weaknesses.", "limitations": "Yes. They address the limitations of their work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The main idea of dual risk minimization (DRM), which combines ERM and WRM, is novel and interesting. Also, DRM effectively balances expected and worst-case performance.\n- Experimental results show that DRM achieves state-of-the-art results on various benchmarks.", "weaknesses": "- The dependence on the lambda seems to be quite significant, and there is no ablation study addressing this in the main paper. It would be good to see results regarding the impact of lambda.\n- It would be helpful if the paper included details on the computational time and cost for each experiment to understand the efficiency and scalability of the proposed methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's time and insightful feedback. It is encouraging to know that the reviewer found our DRM “novel and interesting” and “achieves state-of-the-art results on various benchmarks”. We appreciate that the reviewer has thoroughly engaged with our work and acknowledged its merits.\n\nRegarding the concerns of the reviewer, we provide detailed clarifications below. \n\n---\n\n### 1. Impact of hyperparameter $\\lambda$ in DRM\n\n$\\lambda$ is a balancing hyperparameter for the two risks in DRM. We have discussed its impact on the performance of DRM in Appendix D.3 of our paper. Here, we copy Table 6 from Appendix D.3 below. Notably, for $\\lambda$ within the wide range from 2 to 5, DRM maintains high-level OOD performance (above 48.1, compared to FLYP’s 42.1) on iWildCam. \n\n**Copy of Table 6** \n|$\\lambda$|ID|OOD| \n|--|--|--| \n|0 (FLYP)|56.0|41.9| \n|0.1|56.4|42.6| \n|0.5|57.2|43.9| \n|1|59.1|47.3| \n|2|60.0|48.1| \n|3|61.8|49.2| \n|4|60.9|48.6| \n|5|60.1|48.5| \n|10|55.4|47.7| \n|50|52.5|46.6| \n\nTo further study the impact of $\\lambda$, we have also conducted the same experiments on ImageNet with CLIP ViT-B/16: \n\n|$\\lambda$|ID|OOD| \n|--|--|--| \n|0 (FLYP)|82.6|60.2| \n|1|81.5|62.5| \n|2|81.8|63.1| \n|3|82.0|63.2| \n|4|81.9|63.4| \n|5|81.7|63.3| \n|6|81.8|63.2| \n|20|81.5|62.3| \n|50|81.2|61.9| \n\nSimilar to the results on iWildCam, DRM achieves great performance across a wide range of $\\lambda$ between 2 and 6. All in all, these results suggest that DRM is fairly insensitive to the choice of $\\lambda$ as it is easy to find appropriate values of $\\lambda$ for DRM to significantly outperform the state of the art. \n\n---\n\n### 2. Computation time and cost\n\nIn our experiments, we implemented the ERM part of DRM with FLYP [1]. The additional computational cost of DRM, compared to FLYP, primarily arises from the preparation of soft labels for the targets of the second risk in DRM and the minimization of this second risk. **In short, the training and inference cost for DRM increased by about 20% from FLYP. This additional cost is insignificant compared to the attained performance gain.** Below, we detail the computational costs and timing for fine-tuning CLIP ViT-L/14 models on ImageNet. \n \nWe use two Nvidia H800 GPUs with 80GB VRAM (a modified version of the Nvidia H100 with a reduced chip-to-chip data transfer rate) and 24 CPU cores from Intel Xeon Scalable processors. The computation time reported below is based on the setting that training batch size=256 and inference batch size=1024. There are 1,281,167 training images in ImageNet, and thus there are 5005 training batches.  \n\n| Model | Generating Concept Description by LLM | Soft Label Generation - Eq. (11) | Training | Inference | \n|---|-------|----------------------------------|----------|-----------| \n| FLYP  | N/A                                   | N/A                              | In average: **58s/100 batches**, ~48 mins per training epoch | Inference on 100 batches of images takes **~1.5s**. | \n| DRM   | We utilized the GPT-4-turbo API to generate concept descriptions for 1,000 ImageNet classes, inputting 10 classes at a time to ensure quality. The generation cost is **under 10 US Dollar** (the price of the API is 10 US Dollar/1 million prompt tokens). We are unaware of the computation cost as the model details of GPT-4 are unknown. | The primary computational cost arises from using pre-trained CLIP models to generate image and text embeddings from 1,281,167 training images and 1,000 concept descriptions. Soft labels are created using inner products between these embeddings, with some technical adjustments. The entire process takes **less than 3 minutes**. | In average: **71s/100 batches**, ~58 mins per training epoch | Inference on 100 batches of images takes **~1.8s**. | \n\nThis detailed analysis of the computation cost and time will be included in the next version of this paper. \n\n---\n\n**References:**\n\n[1] Finetune like you pretrain: Improved finetuning of zero-shot vision models. CVPR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ciNwRAxa0d", "reviewer": "Reviewer_zSx8", "summary": "This paper proposes a method for robust fine-tuning by combining empirical risk minimization with worst-case risk minimization to better preserve core features. The approach uses descriptions of core features obtained from large language models (LLMs) like GPT-4 and employs these descriptions to estimate worst-case risk. This method aims to enhance robustness by minimizing empirical risk on the training set of downstream tasks while improving the understanding of core features. As a result, the proposed method demonstrates significant performance improvements over existing methods on datasets such as ImageNet, iWildCam, and FMoW.", "questions": "- The proposed method has only been tested with full fine-tuning. I think this method also can be combined with methods like LP-FT or FLYP. I am curious about the results when they are combined with the proposed method.\n- Why is there no observed trade-off between ID and OOD performance in Table 6? An explanation for this phenomenon is needed.\n- How robust is the method to the quality of class concept descriptions? What happens when a different LLM is used?", "limitations": "The authors have addressed some limitations of their work but have not sufficiently discussed the critical issue of hyperparameter sensitivity and its impact on performance. There are no identified negative societal impacts.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper introduces a novel approach by combining empirical risk minimization with worst-case risk minimization, particularly when no other domain is provided. The use of LLMs to obtain core-feature descriptions is innovative and allows for practical implementation without human annotations. Additionally, integrating this method with techniques like weight ensemble can lead to synergistic performance improvements.\n- The method is well-supported by theoretical foundations and empirical results. The significant performance improvements on challenging datasets like iWildCam and FMoW highlight the robustness and effectiveness of the approach.\n- The paper clearly explains the motivation behind combining empirical and worst-case risk minimization, and the experimental results support the claimed performance improvements. Table 3 effectively highlights the necessity of redefining the new proxy oracle due to artifact terms.", "weaknesses": "- The main weakness of the proposed method lies in the sensitivity to the hyperparameter $\\lambda$ . As shown in Table 6, slight deviations from the optimal $\\lambda$  value can lead to significant performance drops. Finding the optimal $\\lambda$  for different datasets could be impractical, and this issue needs to be addressed by reporting ablation studies on additional datasets beyond iWildCam.\n- The results in Table 6 show no trade-off between in-distribution (ID) and out-of-distribution (OOD) performance, which is counterintuitive. The authors should provide an explanation for this observation.\n- While the overall quality of writing is high, certain sections and figures need improvement. For example:\n\t- Line 159 and Eq. 3 need a clearer explanation of the DRM formulation relaxation process.\n\t- Figure 1 should explicitly state the meaning of the first and second bars.\n\t- Figure 2 could be misleading as it shows the original image having higher affinity with the class concept description than the background-free image without showing affinities with other images.\n- The method relies heavily on the quality of the class concept descriptions generated by GPT-4. An analysis and ablation study on the quality of these descriptions are necessary. Additionally, the impact of using different LLMs should be explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's time and valuable insights. It is encouraging that the reviewer found our DRM approach \"novel\", the use of LLMs to obtain core-feature descriptions is \"innovative\", and the method is \"well-supported by theoretical foundations and empirical results\". We appreciate that the reviewer has thoroughly engaged with our work and acknowledged its merits. \n\nRegarding the concerns of the reviewer, we provide detailed clarifications below. \n\n---\n\n### 1. Impact of hyperparameter $\\lambda$ in DRM\n\n$\\lambda$ is a balancing hyperparameter for the two risks in DRM.  While it may seem that DRM is sensitive to $\\lambda$, we would like to draw your attention to the fact that the range of $\\lambda$ with decent performance is still wide. As shown in Table 6 (which we copy below), for $\\lambda$ between 2 and 5, DRM maintains high-level OOD performance (above 48.1, compared to FLYP’s 42.1) on iWildCam. In other words, the performance drop (from 49.2 to 48.1) is relatively small compared to the gain. \n\n**Copy of Table 6** \n\n|$\\lambda$|ID|OOD| \n|--|--|--| \n|0 (FLYP)|56.0|41.9| \n|0.1|56.4|42.6| \n|0.5|57.2|43.9| \n|1|59.1|47.3| \n|2|60.0|48.1| \n|3|61.8|49.2| \n|4|60.9|48.6| \n|5|60.1|48.5| \n|10|55.4|47.7| \n|50|52.5|46.6| \n\nFollowing your great suggestion, we have conducted the same experiments on ImageNet with CLIP ViT-B/16 to further study the impact of $\\lambda$: \n\n|$\\lambda$|ID|OOD| \n|--|--|--| \n|0 (FLYP)|82.6|60.2| \n|1|81.5|62.5| \n|2|81.8|63.1| \n|3|82.0|63.2| \n|4|81.9|63.4| \n|5|81.7|63.3| \n|6|81.8|63.2| \n|20|81.5|62.3| \n|50|81.2|61.9| \n\nSimilar to the results on iWildCam, DRM achieves great performance across a wide range of $\\lambda$ between 2 and 6. All in all, these results suggest that DRM is fairly insensitive to the choice of $\\lambda$ as it is easy to find appropriate values of $\\lambda$ for DRM to significantly outperform the state of the art. \n\n---\n\n### 2. “The results in Table 6 show no trade-off between ID and OOD performance, which is counterintuitive ...” \n\nIndeed, this was also counterintuitive to us at first, but then a closer inspection revealed a rather intuitive explanation. The ID performance is determined by two factors: (i) how well the model fits the training data, and (ii) how severe the overfitting (if any) is. The results in Table 6 show no trade-off because for relatively small $\\lambda$, (i) the model can still fit the training data reasonably well, and (ii) the WRM objective can in fact reduce overfitting, as shown by the training/validation performance of DRM on iWildCam below: \n\n|$\\lambda$|Training Acc|ID Val Acc| \n|--|--|--| \n|0|92.37|79.44| \n|1|88.29|81.64| \n|3|87.41|82.43| \n \nThis explains why ID performance improves as $\\lambda$ increases (for $\\lambda \\leq 3$), and consequently why no trade-off is observed in Table 6. \n\n---\n\n### 3. Writing\n\nThank you for your great suggestions. \n\n- We will make the explanation of the DRM formulation relaxation process clearer by showing the key intermediate steps from Eq. (3) to line 159. \n- We will specify the meaning of the bars in Figure 1: the 1st bar – the predicted probability of skis being present in the image; and the 2nd bar - no skis being present in the image. \n- To avoid potential misunderstanding, we will add the affinities w.r.t. other images for Figure 2. We provide several such examples in Figure 2 of the attached PDF in the global response. \n\n---\n\n### 4. Quality of concept descriptions and its impact on the results\n \nPlease see **A. Quality and Robustness of LLM-Generated Concept Descriptions** in the global response. There, we have included the results of a quantitative study on the reliability of concept descriptions, an analysis of the stochastic variability in LLM generations, and an investigation into the robustness of DRM with respect to concept descriptions generated by various LLMs. \n\nIn summary, the results demonstrate that GPT-4's concept descriptions effectively capture the core features, with the resulting affinities proving stable and unaffected by randomness in the LLM generation process, and DRM is robust to the concept descriptions generated by different LLMs. \n\n---\n\n### 5. Combining DRM with LP-FT or FLYP \n\nYes, our DRM can indeed be integrated with other methods such as LP-FT and FLYP. In fact, we have combined DRM with FLYP in our experiments, as detailed in lines 276-278 of our paper. Specifically, we utilized FLYP for the ERM component of Eq. (12). It is crucial to highlight that even without FLYP, where DRM is applied with standard full fine-tuning (row 2), our method still outperforms FLYP (row 1), which itself has been shown to surpass standard full fine-tuning in the ERM setting [1]. This performance advantage is demonstrated in the results of fine-tuning CLIP ViT-L/14 on iWildCam: \n\n|Row|FLYP|LP-FT|DRM|ID|OOD| \n|--|--|--|--|--|--| \n|1|Yes|No|No|56.0|41.9| \n|2|No|No|Yes|54.4|43.9| \n|3|No|Yes|Yes|56.5|46.3| \n|4|Yes|No|Yes|61.8|49.2| \n \nAs the table above demonstrates, combining DRM with LP-FT (row 3) enhances performance over just DRM with standard full fine-tuning (row 2). Furthermore, integrating DRM with FLYP (row 4), as we have explored in our paper, yields even more significant improvements. \n\nIn the next version of our paper, we will further clarify that our primary experimental configurations involve DRM combined with FLYP, and will include the above discussions you kindly suggested. \n\n---\n\n**References:**\n\n[1] Finetune like you pretrain: Improved finetuning of zero-shot vision models. CVPR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P6yyhnrr8l", "reviewer": "Reviewer_gi7f", "summary": "To address the robustness of foundational models under distribution shift conditions, this paper proposes a dual risk minimization approach. Specifically, the authors combine Empirical Risk Minimization (ERM) and Worst-Case Risk Minimization (WCRM) to optimize the model fine-tuning process. To achieve accurate distribution estimation, the authors use class descriptions generated by GPT-4 as prompts inputted into the model and apply max-min normalization to the classification probabilities to address certain items that should be zero. Experiments demonstrate that the dual risk minimization method significantly enhances robustness in zero-shot fine-tuning.", "questions": "See Weakness.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors propose dual risk minimization through rigorous mathematical proofs.\n2. Experiments show that dual risk minimization can effectively enhance the robustness of the model.", "weaknesses": "1. To accurately estimate $p_c(y|c) $, the authors use prompts generated by GPT-4 to initialize the classifier parameters. Nevertheless, this classifier may still not be the optimal choice.\n2. Given that better prompts have been used to initialize the classifier cd, why not directly fine-tune cd instead of using cd to assist in fine-tuning df? Moreover, why is the performance of cd inferior to df when comparing row 2 and row 3 in Table 3?\n3. In Table 7, if the models trained with df and cd (row d1 and d2) are directly used for inference with (13), how will the performance be?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's time and valuable insights. It is encouraging that the reviewer found our DRM approach is supported by \"rigorous mathematical proofs\" and it “effectively enhances the robustness of the model”. We appreciate that the reviewer has thoroughly engaged with our work and acknowledged its merits. \n\nRegarding the concerns of the reviewer, we provide detailed clarifications below. \n\n---\n\n### 1. Accuracy of the estimate of $p_c(y|x)$ \n\nThere is certainly still a gap between our estimate and the real $p_c(y|x)$. However, we would like to highlight that the estimate is already substantially better than one-hot labels, as demonstrated by the comparison between (S) and (c3) in Table 7: OOD performance of 49.2 vs. 45.1. Moreover, our approach requires minimal human input, making it highly scalable. In principle, with more human input, we should be able to further improve the estimate, and this would be an interesting future direction. \n\n---\n\n### 2. “... why not directly fine-tune cd instead of using cd to assist in fine-tuning df?” \n\nPlease see **B. Why not directly fine-tune a concept-description classifier?** in the global response. There, we show experiment results and discuss why solely fine-tuning the concept-description classifier is not better than our DRM fine-tuning. \n\n---\n\n### 3. “... why is the performance of cd inferior to df when comparing row 2 and row 3 in Table 3?” \n\nPlease note that only the ID performance of cd is inferior to df. This is expected since cd is not (while df is) trained with ERM which aims to maximize ID performance. \n\n---\n\n### 4. “In Table 7, if the models trained with df and cd (rows d1 and d2) are directly used for inference with (13), how will the performance be?” \n\nThis is an interesting question. Upon further experiments, we find that: 1. Using a mixture model to make predictions for (d1), which is trained with default prompts (df), results in slightly improved OOD performance, though it still significantly lags behind that of the full DRM approach; 2. Conversely, using a mixture model for predictions in (d2), which is trained with concept descriptions (cd), actually leads to a decrease in OOD performance. \n\n|Model|t1|t2|Inference|ID|OOD| \n|--|--|--|--|--|--| \n|DRM|df|cd|mixture|61.8|49.2| \n|(d1)|df|/|df|56.0|41.9| \n||||cd|10.1|8.8| \n||||mixture|56.4|42.4| \n|(d2)|cd|/|cd|56.9|43.4| \n||||df|12.3|9.4| \n||||mixture|56.3|42.7| \n\nRecall that, (d1) refers to the ERM-trained model that uses a concentration of text embeddings from default prompts as the classification head, while (d2) represents the ERM-trained model utilizing a concentration of text embeddings from concept descriptions as the classification head. For both (d1) and (d2), the mixture hyperparameter $\\beta$ is determined based on achieving the best ID validation performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their meticulous reviews of our work. In this global response, we address two main concerns shared by some reviewers. 

## A. Quality and Robustness of LLM-Generated Concept Descriptions

Reviewers D5j1 and zSx8 stressed the importance of thoroughly analyzing concept descriptions generated by LLMs. D5j1 focused on the reliability and robustness of these descriptions, while zSx8 recommended an ablation study to assess their quality and the effects of using various LLMs. Following their kind suggestions, we conducted further empirical study. 

### A.1. Quantitative study on the reliability of concept descriptions

In Figure 2 of our paper, we have showcased two examples that “LLM-generated concept descriptions have the ability to extract core features”. To further support this claim, we present a full quantitative study. The study is conducted with Hard ImageNet [1] and consists of two parts. First, we follow the setting described in Appendix C, i.e., removing image background (BG), and observing how the image-text affinities change for default prompts (df) and concept descriptions (cd) respectively. In the second part, we do the same but with foreground (FG) removed (see Figure 1 of the attached PDF for examples).

The following table shows the average affinities over all 19,097 images across all 15 classes of Hard ImageNet. The percentages in the table indicate the relative changes w.r.t. the affinities of the original images (FG & BG). 

||FG & BG|w/o BG|w/o FG| 
|-|-|-|-| 
|df|0.3473|0.2393 (-31.1%)|0.3407 (-1.9%)|  
|cd|0.2660|0.2387 (-10.3%)|0.1180 (-55.6%)| 

The result shows that **the affinities of concept descriptions are much more invariant to changes in non-core features than default prompts (-10.3% vs. -31.1%)**. This is consistent with Figure 2 and other examples in Appendix C. Moreover, **the affinities are quite responsive (-55.6%) to changes in core features**. In contrast, the affinities of default prompts barely change (-1.9%) in response to the absence of core features. **These results suggest that concept descriptions are indeed pulling out the core features.** For detailed results of each class, please see Table 1 & 2 in the PDF. 
  
### A.2. On the stochasticity of LLM generations

To evaluate the impact of the stochasticity of LLM generations, we repeatedly ask GPT-4 to generate concept descriptions for each class three times and compute the standard deviation of the resulting image-text affinities. As examples, for class “white-lipped peccary”, the generated descriptions are as follows: 
   - Compact, dark grey body with distinctive white markings around the mouth.  
   - Compact, dark gray body with distinctive white markings around the lips.  
   - A stocky body with coarse, dark hair and distinct white markings around the mouth. 

The average standard deviation of the corresponding affinities over 20k randomly sampled images of iWildCam is 0.0061, which is surprisingly small compared to the mean, 0.2659. This shows that **the affinities are quite stable and insensitive to randomness in the generation process**. 

### A.3. On the robustness of DRM w.r.t. concept descriptions by different LLMs

We have also experimented with different LLMs of various sizes (from 8B to over 405B parameters) to generate concept descriptions. 

|Method|LLM (#params)|ID|OOD| 
|-|-|-|-|
|FLYP|-|52.2|35.6| 
|DRM|GPT-3.5 (~20B?)|53.4|38.7| 
|DRM|GPT-4 (>1T?)|54.1|40.0| 
|DRM|Llama-3 (8B)|53.8|39.2| 
|DRM|Llama-3 (70B)|54.0|39.9| 
|DRM|Llama-3 (405B)|53.9|40.5| 

All the LLMs greatly improve baseline performance on iWildCam. **This shows our method is not sensitive to the quality of the concept descriptions either.**
 
## B. Why not directly fine-tune a concept-description classifier?
 
Reviewer D5j1 raised a question about the feasibility of directly fine-tuning the CLIP model using concept descriptions (cd). Meanwhile, reviewer gi7f questioned why not directly fine-tune the cd classifier but using it to assist in the default-prompt (df) classifier. These are both great questions. 
 
In fact, we have tried to directly fine-tune the cd classifier. The results on iWildCam have been reported in (b2), (d2) and (d3) of Table 7 in the paper. In (b2), the cd classifier is fine-tuned with both ERM and WRM. In (d2), it is fine-tuned with only ERM; whereas in (d3), it is fine-tuned with only WRM. We copy the results here and provide more explanations. 

|||ID|OOD| 
|-|-|-|-| 
|(b2)|cd classifier (ERM+WRM)|54.0|46.1| 
|(d2)|cd classifier (ERM)|56.9|43.4| 
|(d3)|cd classifier (WRM)|51.7|46.3| 
|DRM|df classifier (ERM) + cd classifier (WRM)|61.8|49.2| 

**Both (b2) and (d2) involve ERM in fine-tuning the cd classifier. Notably, they both underperform DRM.** This is because the good performance of DRM does not only come from the cd classifier, but also the soft labels constructed from the concept descriptions (via Eq.(11)). These soft labels are the targets for WRM, a crucial component of DRM besides ERM. 

**The problem with ERM in (b2) and (d2) is that it uses one-hot labels which do not capture subtle differences of core visual features among images.** While pre-trained cd classifiers do have the ability to extract core features, fine-tuning them with ERM would certainly strip some of this ability away. In comparison, the df classifier is more aligned with the ERM objective. **In DRM, we therefore separate ERM and WRM for the two classifiers, reducing the interference of ERM in fine-tuning the cd classifier which aims to extract the core features and achieve WRM.**

Finally, comparing (d2) and (d3), we observe that the (d3) cd classifier, fine-tuned with our proposed core-feature-aware soft labels, shows much better OOD performance than the (d2) classifier, which is fine-tuned with one-hot labels. This result clearly demonstrates the advantage of using the soft labels. 

**Reference:**

[1] Hard imagenet: Segmentations for objects with strong spurious cues. NeurIPS, 2022.

Meta-review: This paper presents a new technique based on a combination of empirical risk minimization and worst-case risk minimization for robust finetuning of zero-shot models. The reviewers appreciated the novel and interesting motivation, theoretical foundation, and strong results of the proposed method as well as the clarity of the manuscript. They also raised concerns with lack of in-depth analysis (e.g., reliability and robustness of the core features obtained by the method, complexity analysis, impact of the quality of generated descriptions and that of the LLM used) (D5j1, eddC, zSx8), sensitivity to the hyperparameter $\lambda$ (zSx8), potential risk of using imperfect estimate of the oracle model (gi7f) and that of ignoring inter-class similarities (D5j1), lack of discussion on counterintuitive results (zSx8), and missing comparisons with closely related work (D5j1). Most of these concerns were successfully addressed by the rebuttal and subsequent responses in the discussion. Consequently, the reviewers unanimously championed the paper after the discussion period. The AC agrees with the reviewers and recommends acceptance. The authors are strongly encouraged to carefully revise the paper to reflect the valuable comments by the reviewers (in particular those on the sensitivity to $\lambda$), to provide a tighter connection between the theorem and the actual DRM objective and its implementation, and to add new results and discussions brought up in the rebuttal and discussion.
Decision: Accept (poster)