Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models

Authors: kaican  li
weiyan  xie
yongxiang  huang
didan  deng
lanqing  hong
zhenguo  li
ricardo  silva
nevin l zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-22 22:00:58

Abstract: Fine-tuning foundation models often compromises their robustness to distribution shifts. To remedy this, most robust fine-tuning methods aim to preserve the pre-trained features. However, not all pre-trained features are robust and those methods are largely indifferent to which ones to preserve. We propose dual risk minimization (DRM), which combines empirical risk minimization with worst-case risk minimization, to better preserve the core features of downstream tasks. In particular, we utilize core-feature descriptions generated by LLMs to induce core-based zero-shot predictions which then serve as proxies to estimate the worst-case risk. DRM balances two crucial aspects of model robustness: expected performance and worst-case performance, establishing a new state of the art on various real-world benchmarks. DRM significantly improves the out-of-distribution performance of CLIP ViT-L/14@336 on ImageNet (75.9$\to$77.1), WILDS-iWildCam (47.1$\to$51.8), and WILDS-FMoW (50.7$\to$53.1); opening up new avenues for robust fine-tuning. Our code is available at https://github.com/vaynexie/DRM.

Main paper: 
 introduction: 
 Foundation models such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) have revolutionized machine learning with their remarkable zero-shot and adaptive capabilities. Research has shown that such capabilities are mainly due to robust feature representations gained from large-scale training data (Fang et al., 2022; Xu et al., 2024) . The models have been proven useful in various downstream tasks (Shen et al., 2022; Zhang et al., 2022; Betker et al., 2023; Pi et al., 2024) and are the cornerstones of large multimodal models (Alayrac et al., 2022; Liu et al., 2023; Zhu et al., 2024) .Fine-tuning is one of the most common approaches to the downstream adaptation of foundation models (Bommasani et al., 2021; Shen et al., 2022) . However, such adaptation often comes at the cost of robustness (Radford et al., 2021; Pham et al., 2023) , resulting in larger gaps between downstream in-distribution (ID) and out-of-distribution (OOD) performance (Wortsman et al., 2022) . Kumar et al. (2022) showed that fine-tuning tends to distort pre-trained features, and the distortion is exacerbated by randomly initialized heads which would significantly alter the pre-trained features to fit ID examples. The proposed remedy, LP-FT, first learns a linear probe (LP) on frozen features, and then followed by regular fine-tuning (FT). Goyal et al. (2023) took this idea further by reusing the pre-trained text encoder of CLIP as the classification head for fine-tuning. This method improves LP-FT and is colloquially known as "fine-tune like you pre-train" (FLYP). WiSE-FT (Wortsman et al., 2022) investigated combining pre-trained models with their fine-tuned versions by weight averaging, which can be seen as yet another approach to recover robust features lost during fine-tuning. In this simple binary classification task predicting if there are skis in a given image, (i) ERM underperforms when the core features (the appearance of ski) are clear but the non-core features such as background/context are spurious (i.e. negatively correlated with ski), and (ii) WRM underperforms when the core features are unclear but the non-core features are robust (i.e. positively correlated with ski). DRM outperforms ERM and WRM under mild conditions such that the core features are not always clear and the non-core features are more often robust than not.While the existing approaches aim to preserve pre-trained features, the fine-tuning process is still guided by empirical risk minimization (ERM; Vapnik, 1998) , which favors the most predictive but not necessarily the most robust features. In general, there are two kinds of robust features: core features which essentially define the target classes, and non-core features that may aid prediction when the core features are not clear (Gao et al., 2023) . ERM models tend to exploit the non-core features even when the core features are clear (Geirhos et al., 2020; Shah et al., 2020) . This often harms OOD performance as non-core features are generally less reliable out-of-distribution.To better preserve the core features, we propose a new principle called dual risk minimization (DRM) which combines ERM with worst-case risk minimization (WRM; Wald, 1945) , a common principle for domain generalization (Arjovsky et al., 2019; Sagawa et al., 2020; Cha et al., 2021; Kirichenko et al., 2023) . This combination rests on our view that robustness involves two main aspects: the expected (or average) performance and the worst-case performance over all domains. While there is often a trade-off between these two aspects (Tsipras et al., 2019; Teney et al., 2023) , Figure 1 illustrates how DRM balances the trade-off to improve overall robustness.The main challenge of applying DRM to real-world tasks is to assess of worst-case risk. To this end, we use concept descriptions (Pratt et al., 2023 )-short texts that describe the core features of each class-obtained with GPT-4 (Achiam et al., 2023) . The description for cougar, for instance, is "a large, tawny cat with a muscular build and a small head." We feed these descriptions to a pre-trained CLIP text encoder (Radford et al., 2021) for the text embeddings, which are then used to construct soft class labels for each training image according to the similarity scores between the image and text embeddings. The risk w.r.t. the soft labels can be seen as a proxy of the worst-case risk and is thus minimized instead. Empirically, DRM significantly outperforms the state of the art on challenging benchmarks such as ImageNet (Deng et al., 2009) and WILDS (Koh et al., 2021) .In summary, we make the following key contributions in this paper:• We propose dual risk minimization (DRM), a novel approach that combines ERM and WRM to improve downstream robustness of zero-shot foundation models while addressing the intractability of WRM through innovative use of concept descriptions.• We highlight that robustness for many real-world problems concerns both expected and worst-case performance while most previous works focus on only one. We then show that DRM offers a simple and effective way to balance these two important aspects of robustness.• We establish a strong new state of the art on multiple real-world benchmarks, promising next-level robustness in fine-tuning zero-shot models. On CLIP ViT-L/14@336, DRM achieves a significant, over 5% relative improvement in OOD performance over the best baseline method.Worst-case risk minimization. The study of worst-case risk minimization (WRM) dates back to the work of Wald (1945) , which has gradually evolved into what we know as robust optimization today (Ben-Tal et al., 2009) . More recently, WRM has been considered (by many) a basic principle for domain generalization (DG; Blanchard et al., 2011; Muandet et al., 2013) . A notable example is invariant risk minimization (IRM; Arjovsky et al., 2019) , which aims to learn core-feature representations from multi-domain data. Such representations, under mild causal assumptions, give rise to classifiers that minimize the worst risk (Peters et al., 2016) . Another key method, GroupDRO (Sagawa et al., 2020) , imposes higher penalties on domains with higher empirical risks. Unlike DRM, neither IRM nor GroupDRO formulates WRM as an explicit optimization constraint for ERM. Eastwood et al. (2022) pointed out that sacrificing too much average performance for worst-case performance is not ideal for DG. Hence, they proposed to minimize the risk among the most likely domains. Lastly, our setup is partly similar to Alabdulmohsin et al. (2023) which also relies on external information.Prompt design for zero-shot classification. To better leverage the capability of zero-shot models, various prompt designs have been proposed. Menon and Vondrick (2022) ; Pratt et al. (2023) ; Maniparambil et al. (2023) mainly explored prompts for zero-shot classification. Their prompts were generated by LLMs (Radford et al., 2019) with slightly different instructions than ours, not explicitly focusing on core features. For example, Pratt et al. (2023) used "Describe an image from the internet of a(n) ...", which may inadvertently introduce descriptions of non-core features in the resulting prompts. The prompts considered by Yang et al. (2023) ; Yan et al. (2023) 3 Dual risk minimization Data model. Let X and Y be the input and ground-truth target variables for which we adopt the following data generation model:EQUATIONwhere (X c , X n ) are latent variables and ε is exogenous noise. We call X c core features and X n non-core features of (X, Y ). X n and Y may be correlated due to hidden confounders of (X c , X n ) and direct causal mechanisms between (X c , X n ). Following Peters et al. (2016) , we assume the causal mechanisms and the distribution of ε are invariant across domains. There are no other hidden variables or mechanisms. Similar models were widely adopted in the literature (Tenenbaum and Freeman, 1996; Mahajan et al., 2021; Mitrovic et al., 2021; Ahuja et al., 2021; Liu et al., 2021; Lv et al., 2022; Ye et al., 2022; Zhang et al., 2023a; Gao et al., 2024a) where X c and X n are sometimes referred to as 'content' and 'style'. We use calligraphic letters such as X and Y to denote the set of possible outcomes of the random variables.Ideal objective for robustness. Let D be all possible domains of a task, and P be some natural distribution over D. By definition, P(d) > 0 for all d ∈ D. Every domain d is associated with a data distribution p d (x, y, x c , x n ) consistent with (1). Let pθ (y|x) be a prediction model parameterized by θ ∈ Θ. Its risk in terms of negative log-likelihood,EQUATIONcan be seen as a measure of its performance in domain d. Let d s ∈ D be the training domain. For simplicity, we will omit d when it is clear from the context, e.g., R ds (θ) will be written as R s (θ).For real-world applications, we argue that a robust model should optimize its expected performance over P while maintaining acceptable worst-case performance across D. The expected performance implies how well the model would perform at the most general population level, while the worst-case performance tells us the model's performance in the worst scenario one may encounter. We note that Eastwood et al. (2022) and Zhang et al. (2023b) share a similar view with us on robustness.We formalize the above intuition as the following constrained optimization problem, namely idealized dual risk minimization (IDRM), which aims to minimize the empirical risk of pθ (y|x) while ensuring its worst-case risk is below some threshold value α:min θ∈Θ R s (θ) subject to max d∈D R d (θ) ≤ α. (IDRM)IDRM generalizes ERM (Vapnik, 1998) and WRM (Wald, 1945) as it reduces to ERM when α is large and to WRM when α is small. IDRM also bears some resemblance to IRM (Arjovsky et al., 2019) , which involves an implicit WRM constraint. The constraint, however, requires the classification head to be optimal in all training domains and thus may be too demanding in practice. Another closely related work, GroupDRO (Sagawa et al., 2020) , proposes to minimize the worst training-domain risk-a more empirical flavor of WRM. Both IRM and GroupDRO rely on ideally grouped training data to capture invariance across domains. In Section 4, we will show that this is largely unnecessary for zero-shot models and provide a practical solution for IDRM with just single-domain data.From IDRM to DRM. IDRM can be solved as the following unconstrained optimization problem due to strong duality (proof in Appendix A). Theorem 1. Strong duality holds between IDRM and the following dual problem:EQUATIONLet λ ⋆ be any solution of λ ′ to (3). By the strong duality, IDRM then reduces tomin θ∈Θ [R s (θ) + λ ⋆ max d∈D R d (θ)].The worst-case risk, i.e., max d∈D R d (θ), is still intractable in itself, but it is closely related to the degree to which the model pθ (y|x) relies on core features to predict y. This is because for a diverse set of domains D, leveraging non-core features would always lead to worse performance in certain domains (Arjovsky et al., 2019; Geirhos et al., 2020) . To minimize the worst-case risk, therefore, the model must only utilize the core features to make prediction.Suppose there is an oracle feature extractor f c that returns a faithful representation of the core features of any input x. As the core features may not always be clear, f c (x) can be viewed as some distribution over the core features for each x. Let p c (y|x) be the optimal model that can be built upon f c (x). The risk of pθ (y|x) w.r.t. p c (y|x) on the training domain d s is given byEQUATIONAssuming p s (x) is fairly diverse, the risk R c s (θ) measures the degree to which the model's prediction is based on the core features and thus can be viewed as a proxy for the worst-case risk. Hence, we can replace max d∈D R d (θ) with R c s (θ) while still achieving a similar optimization effect. In summary, we relax IDRM to the following DRM formulation:min θ∈Θ R s (θ) + λR c s (θ) (DRM)with some properly chosen λ ≥ 0. Now the risk R c s (θ) can also be interpreted as a regularization term for ERM to help preserve the core features. In the following section, we demonstrate how to utilize zero-shot models like CLIP models (Radford et al., 2021) to estimate p c (y|x), and subsequently how to apply DRM to robustly fine-tune the same CLIP models. 4 Fine-tuning zero-shot models with DRM Zero-shot models like CLIP typically consist of an image encoder f ϕ and a text encoder g ψ with parameters θ = (ϕ, ψ). Image classification with such models is usually done by first creating a text prompt t y for each class label y ∈ Y, and then assigning a probability for each y to an image x byEQUATIONwhere A θ (x, t y ) = ⟨f ϕ (x), g ψ (t y )⟩ and τ is the temperature. The inner product ⟨f ϕ (x), g ψ (t y )⟩ can be intuitively understood as the affinity between x and t y , and we thus denote it as A θ (x, t y ).The classifier (5) was originally introduced by Radford et al. (2021) for zero-shot classification. We follow Goyal et al. (2023) to directly fine-tune this classifier, viewing the text embeddings g ψ (t y ) as the weights of a standard linear classification head for the image embeddings f ϕ (x). Unlike standard classifiers, however, (5) additionally depends on the text prompt t y of which the design is important. 
dual prompts for fine-tuning zero-shot models: 
 Let T = {t y | y ∈ Y} be the set of text prompts used to construct the classifier p θ (y|x) according to Eq. ( 5). For such zero-shot classifiers, the general DRM objective (DRM) becomesEQUATIONwhere the risks R s and R c s not only depend on the model parameters θ but also on the prompts T . For R s , usually a set of default prompts, T df = {t df y | y ∈ Y}, like "an image of [class name] " is used (Goyal et al., 2023; Oh et al., 2023) . However, such prompts may not be suitable for R c s as they are not specifically designed to bind with core features. In fact, as we will show, default prompts elicit representation that is biased towards non-core features. It is also know that non-visual and spurious descriptions contribute significantly to CLIP's representation (Esfandiarpoor et al., 2024) .To generate better prompts for R c s , we ask GPT-4 (Achiam et al., 2023) to describe the core visual features of each class, producing a set of concept descriptions, T cd = {t cd y | y ∈ Y}. For example, to generate a concept description for cougar, we prompt GPT-4 with "Q: Generate a short sentence that describes the visual features of Cougar. Do not include its function, its surroundings, or the environment it usually inhabits. The sentence should be concise. For example, [goldfish: a long, golden body with back fins]." and the concept description returned is "a large, tawny cat with a muscular build and a small head." 3 Figuratively, the text embedding g ψ (t cd y ) of the concept description represents the core features of the class from the text side. We use it to "pull out" the core features from the image embedding f ϕ (x) via inner-product. As illustrated in Figure 2 , the affinity between an image and its concept description is indeed a much better measure of the significance of core visual features. Regarding this point, a full quantitative study can be found in Appendix D.1.Together, the two sets of prompts naturally give rise to the following objective:EQUATIONwhere we use default prompts T df for ERM and concept descriptions T cd for WRM. We do not use concept descriptions for ERM because it would erode the text-side core feature representation elicited by concept descriptions. This is supported by empirical evidence from our ablation study (Section 5.3) showing that (7) works best among various alternatives.The dual prompts elicit separate predictions from the same model for the two sub-objectives of DRM.The ERM part, R s (θ; T df ), is supervised by regular one-hot labels. The WRM part, R c s (θ; T cd ), is supervised by p c (y|x) as in (4). Next, we show how the same set of concept descriptions T cd can be used to obtain a good estimate of p c (y|x).Recall that the oracle model p c (y|x) is based on a faithful representation of core features. Since we have demonstrated that concept descriptions bind well with core features on pre-trained CLIP models, a direct estimate for p c (y|x) can be obtained via (5) with t y ← t cd y and θ ← θ 0 where θ 0 = (ϕ 0 , ψ 0 ) denote the pre-trained CLIP parameters.However, there is a crucial caveat. To illustrate, consider an image x of class y. For another class y ′ whose core features are not present in x, the affinity A θ0 (x, t cd y ′ ) should ideally be very small. In practice, however, we find this is seldom the case. These extraneous affinity values, which we call artifact terms, often vary among classes and lead to poor estimates of p c (y|x) with high entropy.To mitigate the impact of artifact terms, we perform a simple min-max normalization on ξ(x, y) = exp(A θ0 (x, t cd y )/τ ) w.r.t. all training images X y ⊆ X labeled the same class y, as follows:EQUATION)This effectively adjusts the affinity range of each class, reducing the difference in the artifact terms of different classes. Based on the normalization, the final estimation we propose for p c (y|x) ispc (y|x) = γ(x, y), y = y x ; [1 -γ(x, y x )] • γ(x,y) y ′ ̸ =yx γ(x,y ′ ) , y ̸ = y x ; (9)where y x is the ground-truth label of x. This ensures that for every class y, there exists at least one x ∈ X y for which pc (y|x) = 1, promoting balanced learning. When pc (y|x) < 1 for y = y x , the remaining probability is distributed to other classes y ̸ = y x according to the relative scale of the respective affinities.We pre-compute the estimate pc (y|x) with the pre-trained CLIP model θ 0 before fine-tuning, which is now the learning target of R c s (θ; T cd ). Since the computation requires the class labels, pc (y|x) can only be used for training (not for inference). Intuitively, learning from pc (y|x) can be seen as a form of self-distillation targeted at core features. In comparison, standard self-distillation methods (Furlanello et al., 2018; Zhang et al., 2019; Ji et al., 2021; Zhang et al., 2021) are largely indifferent to what specific information should be distilled.The fine-tuning objective (7) involves two classifiers: the ERM classifier pdf θ (y|x) induced by T df , and the WRM classifier pcd θ (y|x) induced by T cd . While either alone can be used for inference, we find that their mixture, pdualEQUATION) where β ∈ (0, 1), performs the best. This is expected as (10) essentially combines ERM with WRM as depicted in Figure 1 . By default, we set β = 1/(1 + λ) so to be as consistent with (7) as possible. 
experiments: 
 In this section, we evaluate DRM on multiple real-world benchmarks and conduct ablation studies to assess the impacts of various design choices. We conduct our experiments on three varying sizes of pre-trained CLIP models: ViT-B/16, ViT-L/14 and ViT-L/14@336 (Radford et al., 2021) . Finally, we analyze the reliability of LLM-generated concept descriptions and the impact of λ on performance.Datasets. IMAGENET (Deng et al., 2009) comprises over a million natural images across 1,000 classes. We use the training set for fine-tuning and the validation set for assessing ID accuracy. For OOD evaluation, we consider ImageNet variants: IMAGENET-V2 (Recht et al., 2019) , IMAGENET-R (Hendrycks et al., 2021a) , IMAGENET-SKETCH (Wang et al., 2019) , IMAGENET-A (Hendrycks et al., 2021b) , and OBJECTNET (Barbu et al., 2019) . We report accuracy for both ID/OOD performance.WILDS-IWILDCAM (IWILDCAM) (Koh et al., 2021) contains camera-trap images for wildlife classification, with training images from 200 locations and OOD images from different locations. Both ID and OOD performances are measured using macro F1 scores.WILDS-FMOW (FMOW) (Koh et al., 2021) is a dataset of satellite images from different years and continents for land use prediction. The dataset is split into training, validation, and testing domains based on the year of collection. There is also a notable shift between different continents. We report the ID testing accuracy and the worst-region OOD testing accuracy.DOLLAR STREET-DA and GEOYFCC-DA (Prabhu et al., 2022) are datasets for testing model generalization from images in specific countries to new ones. For Dollar Street-DA, training images are from North America and Europe, with testing images from other continents. GeoYFCC-DA has a similar setup. Model effectiveness is measured by accuracy in seen and unseen countries.Baseline methods. The key baseline we compare our DRM method with is FLYP (Goyal et al., 2023) . Following the FLYP paper, we include several baselines that do not utilize the text encoder. These methods are LP (linear probing), FT (fine-tuning), L2-SP (Li et al., 2018) , and LP-FT (Kumar et al., 2022) . In addition, we incorporate some more recent fine-tuning methods for zero-shot vision models. We also consider combining the weight-space averaging method, WiSE-FT (Wortsman et al., 2022) , with DRM and the baselines. For more introduction to these methods, see Appendix B.Implementation details. We update both the image encoder and text encoder during fine-tuning, following FLYP (Goyal et al., 2023) . Furthermore, FLYP uses the CLIP contrastive loss (Radford et al., 2021) instead of the standard cross-entropy loss. We adopt this approach for the ERM part of DRM to facilitate comparison. In short, when the hyperparameter λ = 0 in the DRM objective (7), the WRM loss term vanishes and DRM reduces to exactly FLYP.We choose all hyperparameters of DRM and baseline methods based on the performance on the ID validation set, i.e., training-domain validation (Gulrajani and Lopez-Paz, 2021) . The hyperparameter λ of DRM is picked from {1, 2, 3, 4, 5}. More implementation details are presented in Appendix F.We report the main results on IMAGENET, IWILDCAM, and FMOW in Table 1 and 2 . The results on DOLLAR STREET-DA and GEOYFCC-DA can be found in Appendix E.2. We also compare DRM with some concurrent methods in Appendix E.3. All performance statistics, except some reported by previous papers (which we simply reuse), are averaged over 5 runs with different random seeds. The 95% confidence intervals over the 5 runs are reported.Table 1 shows the results on CLIP ViT-B/16, the smallest of the three CLIP models. DRM achieves consistently better OOD performance than the baselines across all datasets, with and without WiSE-FT. Without WiSE-FT, DRM attains 5.0%, 12.4%, and 11.1% relative improvements over the best baseline method, FLYP, on the three benchmarks respectively. With WiSE-FT, the improvements remain significant at 1.9%, 11.6%, and 9.8% respectively. In terms of ID performance, DRM is roughly on par with FLYP, with a notable advantage on IWILDCAM. Table 2 compares the performance of DRM and FLYP on two larger CLIP models. DRM again consistently outperforms FLYP in all cases. The previous state-of-the-art OOD performance for IWILDCAM and FMOW are 47.1 and 50.6 respectively, both achieved by FLYP+WiSE-FT with CLIP ViT-L/14@336. DRM improves those scores by 10.0% and 5.0% to 51.8 and 53.1 respectively.Although DRM incurs more computational costs compared to FLYP due to an additional pass through the text encoder, the training and inference costs for DRM only increase by about 20% from FLYP (see Appendix F.3). This cost is insignificant relative to the performance gain.Overall, our experiments suggest that DRM is highly effective in the robust fine-tuning of zero-shot models, outperforming previous methods by a large margin while maintaining scalability. As a bonus, we show in Appendix E.6 that DRM is also effective in fine-tuning ImageNet pre-trained CNNs.We conduct our ablation study with CLIP ViT-L/14 on IWILDCAM. The main results are reported in Table 3 and discussed below. For additional results and details, see Appendix E.4 and E.5.Impact of dual risks. The DRM objective (7) consists of an ERM term, R s (θ; T df ), and a WRM term, R c s (θ; T cd ). From Table 3 , we can see that, in terms of OOD performance, models fine-tuned with only the ERM term (Rows 4 & 5) significantly underperform models fine-tuned with both terms (Rows 1-3). Conversely, models fine-tuned with only the WRM term (Rows 6 & 7) have much worse ID performance. Although the OOD performance is improved, there is still a large gap from the DRM model (Row 1). Note that these hold regardless of the type of text prompts used by ERM/WRM. Impact of dual prompts for fine-tuning. With both ERM and WRM in effect, we further investigate the impact of their prompts during fine-tuning. DRM uses two sets of prompts: default prompts T df for ERM, and concept descriptions T cd for WRM. Rows 8 & 9 of Table 3 show the performance of models fine-tuned using the same set of prompts for ERM and WRM. In either case, the model underperforms DRM (Row 1), validating the use of tailored prompts for specific learning targets.Table 3 : Results of ablation studies on DRM with CLIP ViT-L/14 performance and IWILDCAM. We use "df" and "cd" to denote the type of text prompts used to produce model predictions. "dual" refers to the mixture model (10) for inference. "-" means the corresponding loss term is not in use. Intuitively, T cd is more aligned with WRM than T df as the learning targets (9) for WRM are based on the predictions of the pre-trained model prompted by T cd . So, reducing the WRM term R c s (θ; T cd ) in some sense limits the divergence of the predictions between the pre-trained models and the fine-tuned models, hence helping better preserve pre-trained (core) features. This explains why using T cd for both ERM and WRM would lead to poorer outcomes (Row 1 vs. 9). ERM targets, typically one-hot class labels, do not capture subtle core visual differences. As a result, ERM would weaken the bond between T cd and core visual features as perceived by the model during fine-tuning, and therefore reduce the power of WRM in preserving those features.Interestingly, comparing Row 1 & 8, we find that T cd not only improves OOD performance but also ID performance. We hypothesize that this is because the core features are better preserved with T cd and thus the fine-tuned model relies on a more diverse set of features to make predictions, reducing overfitting and improving ID generalization as well.Impact of dual prompts for inference. After DRM fine-tuning, we obtain a new CLIP model with updated parameters θ. Since the model is fine-tuned with dual prompts, it is natural to use the same dual prompts for inference, as in (10). Indeed, we find that pdual θ (Row 1) outperforms pdf θ (Row 2) and pcd θ (Row 3) in Table 3 . In particular, while pdf θ is better than pcd θ in-distribution and the latter is better out-of-distribution, they underperform pdual θ on both ID and OOD fronts. This is similar to the phenomenon we have observed in the fine-tuning scenario. Dual prompts generally reduce overfitting and the effect carries over from fine-tuning to inference.Impact of affinity normalization. In Section 4.2, we pointed out a problematic issue regarding the direct estimate for p c (y|x) obtained via (5) with t y ← t cd y and θ ← θ 0 . To address the issue, we proposed another estimate pc (y|x) in ( 9) based on normalized affinities. Comparing the two approaches, Row 10 of Table 3 shows that the direct estimate leads to severe degradation in both ID and OOD performance, demonstrating the importance of affinity normalization.Consistency across repeated generations. The concept descriptions used in our experiments are generated by GPT-4. Since the generation process is stochastic, it might impact the performance of DRM. To evaluate this impact, we repeatedly ask GPT-4 to generate a concept description for each IWILDCAM class for three times and then find the standard deviation of the resulting image-text affinities. The average standard deviation over 20,000 randomly sampled images of IWILDCAM is 0.0061, which is very small compared to the mean affinity, 0.2659. This suggests that the affinities are robust to the randomness in the generation process, which is therefore unlikely to have any noticeable Generation across different LLMs. We also experiment with different LLMs of various sizes (from 8B to over 400B parameters) to generate concept descriptions. The results are reported in Table 4 . While larger and more advanced models lead to better performance, DRM is not sensitive to the specific choice of LLM for generating the concept descriptions. Even with a relatively small LLM, Llama-3 (8B), DRM still maintains a significant edge over FLYP on IWILDCAM. Examples of the concept descriptions generated by the LLMs are provided in Appendix D.3.As in ( 7), DRM involves a hyperparameter λ that balances the weight of the empirical risk and the worst-case risk. When λ = 0, only the empirical risk is involved during fine-tuning, resulting in an ERM model. As λ increases, the influence of the empirical risk reduces, and the resulting model becomes closer to a WRM model. In practice, we choose the value of λ > 0 based on ID validation, which is often positively correlated with OOD performance (Taori et al., 2020; Miller et al., 2021) . In Table 5 , we show the ID and OOD performance of CLIP ViT-L/14 fine-tuned on IWILDCAM and CLIP ViT-B/16 fine-tuned on IMAGENET under different choices of λ.Compared to FLYP (λ = 0), the results indicate that DRM maintains high-level OOD performance across λ between 1 and 5, suggesting that DRM is fairly insensitive to the choice of λ. Furthermore, it is interesting to note that ID and OOD performance improve together on IWILDCAM as λ increases to 3.0. Our further analysis reveals that this occurs because DRM assists in reducing overfitting-there is a decrease in training accuracy (from 88.29 to 87.41) and an increase in validation accuracy (from 81.64 to 82.43) as λ increases (from 1 to 3)-thereby enhancing both ID and OOD performance. While DRM does not always improve ID performance (e.g., on IMAGENET), the result suggests that it still achieves a good trade-off between ID and OOD performance (-0.6 ID performance for +3.2 OOD performance), which in turn could lead to a better Pareto front. 
conclusion: 
 In conclusion, this paper introduces dual risk minimization (DRM), a novel method that enhances the robustness of models fine-tuned from zero-shot foundation models against distribution shifts. DRM combines ERM with WRM, focusing on the preservation of core features which essentially define the target classes. To guide the fine-tuning process, DRM utilizes concept descriptions generated by LLMs like GPT-4. By balancing expected and worst-case performance, DRM overcomes the traditional limitations of ERM and achieves significant OOD performance improvements on multiple real-world benchmarks, establishing a new state of the art. Potential future directions include a deeper theoretical investigation into DRM as a general principle, using DRM to improve the general robustness of zero-shot models across a broad range of tasks, and better understanding the role of concept descriptions in vision-language modeling.A Proofs Lemma 1. Let p and q be two probability distributions over X × Y. The cross-entropy between p(y|x) and q(y|x) over p(x), i.e., H p (q) = E (x,y)∼p [-log q(y|x)], is convex w.r.t. q.Proof. It suffices to show that for any pair of (q 1 , q 2 ) and α ∈ [0, 1] we have H p (αq 1 + (1 -α)q 2 ) ≤ αH p (q 1 ) + (1 -α)H p (q 2 ).H p (αq 1 + (1 -α)q 2 ) = E (x,y)∼p [-log(αq 1 (y|x) + (1 -α)q 2 (y|x))] ≤ E (x,y)∼p [-α log q 1 (y|x) -(1 -α) log q 2 (y|x)] (11) = αH p (q 1 ) + (1 -α)H p (q 2 ).Theorem 1. Strong duality holds between IDRM and the following dual problem:max λ ′ ≥0 min θ∈Θ R s (θ) + λ ′ max d∈D R d (θ) -λ ′ α.(3)Proof. Recall that IDRM aims to solve for min θ∈Θ R s (θ) subject to max d∈D R d (θ) ≤ α. (IDRM) Here, R d (θ) = H p d (p θ (y|x)) = E (x,EQUATIONfor any α > min θ∈Θ max d∈D R d (θ), i.e., when a strictly feasible solution to (IDRM) exists. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: We do not see particular negative societal impacts of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We do not see any data or models in this work that have a high risk for misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, have been properly credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The specifics of our DRM model training objective and its implementation are detailed in Section 4.2 and 5.1. A detailed documentation for running our codes will be provided in the GitHub respository. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The key baseline we compare our DRM method with is "fine-tune like you pre-train" (FLYP) (Goyal et al., 2023) . While traditional fine-tuning of CLIP models adds a randomly initialized classification head on top of the image encoder, Goyal et al. (2023) demonstrated that it is more effective to simply reuse the text encoder. We therefore follow FLYP to update both the image encoder and text encoder in fine-tuning. They now differ only in the loss functions used in fine-tuning. FLYP uses the CLIP contrastive loss (Radford et al., 2021) for the ERM and showed that this is better than using the standard cross-entropy loss. We adopt this approach for the ERM component (the first term) of DRM to facilitate comparison. Thus, the only difference between DRM and FLYP is the additional regularization term in the DRM loss function.Before the introduction of FLYP, conventional fine-tuning of CLIP models involved adding a linear classification head to the image encoder. The linear probing method (LP) only fine-tunes this new classification head, keeping the image encoder fixed, whereas the full fine-tuning method (FT) trains both the head and the encoder. The LP-FT approach (Kumar et al., 2022) begins with LP and then transitions to full fine-tuning.Besides, L2-SP (Li et al., 2018) and WiSE-FT (Wortsman et al., 2022) are two established fine-tuning variants that restrict the divergence from the pre-trained model. L2-SP specifically integrates an L 2 regularization term into the loss function to constrain the parameter shifts of the fine-tuned model relative to the pre-trained model. WiSE-FT (Wortsman et al., 2022) interpolates parameters of a pre-trained zero-shot model θ zs and that of a fine-tuned model θ ft using θ wise-ft = ρ • θ zs + (1 -ρ) • θ ft .Alongside L2-SP and WiSE-FT, several works concurrent with our own have also introduced robust fine-tuning methods via reducing the difference between pre-trained and fine-tuned models. For example, CAR-FT (Mao et al., 2024) and the method proposed by Cheng et al. (2024) seek to minimize the distance between the context distributions generated by pre-trained and fine-tuned CLIP models. However, a significant limitation of them is that they require prior knowledge of image contexts, such as background and viewpoint, which restricts their practical applicability.Alternatively, Lipsum-ft (Nam et al., 2024) implements a regularization strategy without requiring prior context information, focusing on minimizing the L 2 distance between the affinities of images and random texts from both pre-trained and fine-tuned models. CLIPood (Shu et al., 2023) utilizes a beta moving average for updating parameters during training, and CaRot (Oh et al., 2023) focuses on regularizing singular value distributions while incorporating an exponential moving average for parameter updates. TPGM (Tian et al., 2023a) and FTP (Tian et al., 2023b ) take a more fine-grained approach by autonomously determining the most effective regularization for each layer's parameters and enabling more efficient learning of layer-specific projection regularization, respectively.While these methods aim to prevent the fine-tuned model from deviating too far from the pre-trained one, they do not specify which pre-trained features should be preserved. In contrast, our proposed DRM focuses on preserving the dataset-related core visual features, making it a more targeted approach compared to the aforementioned methods.We utilized GPT-4 (Achiam et al., 2023) , a leading-edge language model developed by OpenAI, to create concept descriptions. We supplied GPT-4 with the class name and tailored prompts designed to guide the model to focus on the essential visual features of the class while omitting unrelated contextual information.Specifically, for the iWildCam dataset, our prompt to GPT-4 was:Q: Generate a short sentence that describes the visual features of a given animal. Do not include its function, its surroundings, or the environment it usually inhabits. The sentence should be concise. For example, [goldfish: a long, golden body with back fins].After that, we asked it to generate concept description for each wildlife class. For example:Q: Now the given animal is Cougar.A: [Cougar: a large, tawny cat with a muscular build and a small head.]Another example is the FMoW dataset. We found it is challenging to encapsulate the core visual features of land used for specific purposes in just one short sentence. Consequently, we employed GPT-4 to generate six sentences for each type of land use. When we implemented DRM on this dataset, we slightly modified the image classifier to use the average of the six text embeddings for each class as the classification head.For this dataset, our initial prompt to GPT-4 was:Q: The Functional Map of the World (fMoW) dataset is designed to identify the functions of buildings and land use by examining satellite images over time.Your task is to generate six sentences, each describing the visual features likely to be seen in these satellite images, given the functional purpose of buildings and land use. The features you describe should be visible from an aerial perspective and consistent across various countries. Each sentence should clearly and comprehensively describe a visual feature, focusing solely on its appearance without including details of the surrounding area or the purpose of the features, such as "Helipad for urgent aerial access".After that, we asked it to generate concept description for each land use. For example:Q: Now the given land use is Road Bridge.A: [Road Bridge: Visibly characterized by a straight or slightly curved linear structure that spans across natural or man-made obstacles.Constructed from materials that appear robust in satellite images, often showing as grey or dark tones against the landscape.The bridge deck is typically marked with parallel lines, indicating the lanes for vehicular traffic visible from above.Elevation above the surrounding area is noticeable, with shadows often highlighting the height and structure of the bridge.Support features like piers or columns are regularly spaced, visible as distinct vertical elements that support the span.End points of the bridge integrate with road networks, appearing as seamless transitions from elevated to ground-level roads.]The full list of concept descriptions generated by GPT-4 for each class of different datasets considered in this work will be publicly available online later. 2023). Among them, the first three works mainly explored the use of class descriptions for zero-shot classification, while this work focuses on fine-tuning. The class descriptions generated in these studies often include contextual information due to prompts like "Describe an image from the internet of a(n) ...", which may not be suitable for our application of estimating worst-case risk. The descriptions in Yang et al. (2023) ; Yan et al. (2023) are more aligned with our approach, while they focus on using LLM-generated concept descriptions to develop concept bottleneck models for interpretable image classification.Complementing Figure 2 , Figure 3 provides additional examples indicating that with the pre-trained CLIP models, concept descriptions have the ability to extract core features. To further support this claim, we present a full quantitative study. The study is conducted with Hard ImageNet (Moayeri et al., 2022) and consists of two parts. First, we remove image background (BG), and observing how the image-text affinities change for default prompts (df) and concept descriptions (cd) respectively. In the second part, we do the same but with foreground (FG) removed. An image of volleyball.An image of seat belt.cd prompt:A round, inflated ball made of Flat, narrow strap with a metal buckle.synthetic leather or rubber. To evaluate the stochasticity of LLM in generating concept descriptions, we repeatedly ask GPT-4 to generate concept descriptions for each iWildCam class. Below are some examples:White-lipped Peccary:• Output 1: Compact, dark grey body with distinctive white markings around the mouth.• Output 2: Compact, dark gray body with distinctive white markings around the lips.• Output 3: A stocky body with coarse, dark hair and distinct white markings around the mouth.Waterbuck:• Output 1: Stocky body with long fur, a white ring on the rump, and shaggy brown coat.• Output 2: Thick, shaggy brown coat with a white ring around the rump and long horns.• Output 3: A robust antelope with a shaggy brown coat and a white ring around the rump.These examples show that while GPT-4's outputs for the same class vary slightly, they consistently highlight key visual features. The variations are primarily in the language details, such as the order of features or the terminology used. For example, for the waterbuck, the consistent visual features across all three outputs are the "stocky (thick, robust) body", "white ring on (around) the rump", and "shaggy brown coat".The examples and experiment results reported in our paper are based on concept descriptions generated by GPT-4. We now consider different LLMs to generate concept descriptions for the classes in the iWildCam dataset. We present some examples and our findings below.White-lipped Peccary:• GPT-4: Compact, dark grey body with distinctive white markings around the mouth.• GPT-3.5-Turbo: Grayish body with a white stripe across the lips.• Llama-3-8b: Reddish-brown fur with a distinctive white stripe.• Llama-3-70b: Dark brown pig-like animal with a distinctive white stripe on the lips.• Llama-3-405b: Stout, brown body with a distinctive white stripe on the lips and sharp tusks.Waterbuck:• GPT-4: Stocky body with long fur, a white ring on the rump, and shaggy brown coat.• GPT-3.5-Turbo: Antelope species with a shaggy coat and a white ring on its rump.• Llama-3-8b: Reddish-brown coat with a distinctive horn shape and a long, slender body.• Llama-3-70b: Large, brown antelope with a distinctive white ring on its rump and a long, shaggy coat.• Llama-3-405b: Large, brown body with a distinctive pair of curved horns and a white ring on the rump.Zebra:• GPT-4: Distinctive black and white striped coat with an upright mane and slender legs.• GPT-3.5-Turbo: Medium-sized equid with black and white stripes.• Llama-3-8b: Black and white striped coat with a distinctive mane and a long, slender body.• Llama-3-70b: Black and white striped equine with a distinctive body shape and a long mane.• Llama-3-405b: Medium-sized, black and white body with a distinctive striped pattern and a long, flowing mane.1. The concept descriptions generated by different LLMs capture common key visual features for the same class, such as "white stripe" for white-lipped peccary, "white ring on the rump" for waterbuck, and "black and white stripes" for zebra.2. Comparing outputs of GPT-3.5-Turbo and GPT-4, the outputs generated by GPT-4 cover more visual details, while those by GPT-3.5-Turbo are generally shorter.3. As model size increases, from Llama-3-8b to Llama-3-405b, the generated outputs become more detailed. For instance, Llama-3-8b mentions "white stripe" but does not specify its location on the lips as Llama-3-70b/405b do. Additionally, Llama-3-8b sometimes makes factual errors, such as describing the incorrect color for white-lipped peccary and waterbuck.4. GPT-3.5-Turbo sometimes deviated from the prompt instructions, generating outputs that include non-visual features, such as the "distinctive call" of the Great Tinamou, which are not visible.Despite variations in the quality of concept descriptions generated by different LLMs, as discussed in Section 5.4, DRM-trained models using descriptions from any of these tested LLMs consistently show significant improvements in OOD performance compared to FLYP-trained models. This demonstrates that the effectiveness of DRM-trained models is not sensitive to the choice of the LLM.The average accuracy across the five ImageNet OOD test sets has been presented in Table 1 . We report the detailed results for each OOD test set in Table 7 . Without WiSE-FT, DRM substantially outperforms the previous best fine-tuning results by FLYP on ImageNet-R and ImageNet-A, with increases from 71.4 to 77.8 and from 48.1 to 53.3, respectively. Meanwhile, the ID performance is at a comparable level. With WiSE-FT, the improvements remain significant, rising from 76.0 to 79.5 on ImageNet-R and from 53.0 to 54.2 on ImageNet-A. We followed the train-test split outlined by Prabhu et al. (2022) . As there was no dedicated validation set, we split 20% of the training set for validation purposes. The ID and OOD performance results are reported based on the ID performance on the validation set and the OOD performance on the test set, which consists of images from countries not included in the training and validation sets.The results presented in Table 8 demonstrate that, compared to FLYP-trained models, DRM-trained models exhibit improved performance on images from new countries. As discussed in Appendix B, there are some more recent robust fine-tuning methods. We include a comparison to some of those methods based on the results of fine-tuning CLIP ViT-B/16 on iWildCam and FMoW datasets. The results are reported in Table 9 . The results clearly show that, the more recent methods still significantly lag behind DRM in term of OOD performance. Setup. Given a labeled dataset {(x i , y i )} N i=1 sampled from the training domain d s , the final DRM objective (7), i.e., R s (θ; T df ) + λR c s (θ; T cd ), for fine-tuning zero-shot models can be expanded asEQUATIONwhere pdf θ (y|x) and pcd θ (y|x) are the classifiers (5) induced by the default prompts T df = {t df y | y ∈ Y} and the concept descriptions T cd = {t cd y | y ∈ Y}, respectively; and ppr θ0 (y|x) = pc (y|x) which is defined by (9) to estimate p c (y|x). Here, we use ppr θ0 (y|x) (where pr stands for 'proxy') instead of pc (y|x) to ease the discussion of possible variations of DRM.Consider the following generalized form of (13) with three varying options, t1 and t2 indicating the classifier types defined with different sets of text prompts, and type indicating the type of model used as the proxy for p c (y|x):EQUATIONAs stated in (13), our final DRM training objective (7) uses t1 = df in the ERM term, with t2 = cd and type = pr in the regularization term. We denote this as our standard setting, (S) in short. We conduct the following ablation study with the pre-trained CLIP ViT-L/14 and fine-tune the model on the iWildCam dataset, with results presented in Table 10 .(a) Inference options after dual classifier training: Two classifiers are involved in our DRM training: pdf θ (y|x) and pcd θ (y|x). As outlined in (10), we combine both classifiers for inference. An alternative is to only use one of the two classifiers for inference. We denote inference with only pdf θ (y|x) as (a1), and with only pcd θ (y|x) as (a2). The comparison between (a1), (a2), and (S) in Table 10 shows combining both classifiers for inference enhances both ID and OOD performance compared to using either alone. This reveals that the two classifiers have a complementary effect as illustrated in Figure 1 , and corroborates our view that ERM and WRM are both vital to OOD robustness.(b) Vanilla DRM using a single set of text prompts: In our standard DRM setting (S), t1 = df and t2 = cd. The vanilla DRM we discussed in Section 4 uses t1 = t2 = df. Alternatively, one can also consider t1 = t2 = cd. We experiment with these two alternative settings denoted by (b1) and (b2) in Table 10 . The contrast between (b1) and (S) confirms our intuition: using the concept descriptions T cd for pt2 θ (y|x), i.e., t2 = cd, enhances robust feature preservation and leads to better OOD performance. The other alternative (b2), which employs T cd for both pt1 θ (y|x) and pt2 θ (y|x), i.e., t1 = t2 = cd, slightly improves (b1). Intriguingly, (b2) is still much worse than (S) despite they both use cd for t2.(c) Proxy model design: In our standard setting, type = pr. As discussed in Section 4, the proxy term ppr θ0 (y|x i ) is based on the affinity A θ0 (x, t cd y ) = ⟨f ϕ0 (x), g ψ0 (t cd y )⟩ according to the pre-trained CLIP model θ 0 = (ϕ 0 , ψ 0 ) and the set of concept descriptions T cd . In Section 4, we also mentioned the following direct estimation of the oracle model p c (y|x):EQUATION)However, as discussed in Section 4, pcd θ0 (y|x) is susceptible to artifact terms. Consequently, we made a technical adjustment to mitigate the influence of these terms, resulting in the refined pc (y|x), which is denoted as ppr θ0 (y|x) here. As shown in Table 10 , the importance of this adjustment is empirically verified by the much lower performance of (c1) compared to (S).One can also define ppr-df θ0 (y|x) by replacing T cd with T df in the formulation of ppr θ0 (y|x). As shown by the result of (c2), this alternative still underperforms ppr θ0 (y|x) used in the standard setting. This discrepancy can be explained by the fact that the affinities between default text prompts and images are easily affected by changes in the non-core visual features instead of focusing on the core visual features, which has been discussed in Appendix D.Another simple alternative, denoted by (c3), is to employ the ground-truth one-hot labels as the proxy. Perhaps unsurprisingly, the OOD performance of (c3) is notably inferior to (S) based on the affinities between the images and the concept descriptions. Comparing them with DRM models (a1) and (a2), it is clear that models trained to minimize a single risk underperform those trained to minimize both risks, highlighting the importance of dual risk minimization.(e) Classifier combination strategy: Our standard DRM training jointly minimizes the two risks, but one can also train an ERM model pdf θERM (y|x) and a WRM model pcd θWRM (y|x) separately. These models can be combined for inference using techniques like model ensembling or weight-space averaging. The last two rows of Table 10 show that combining (d1) and (d3) via model ensembling or weight-space averaging generally underperforms joint training (S).For ImageNet, the settings were as follows: training epochs=10, learning rate=1e-5, batch-size=256, and optimizer=AdamW with weight decay=0.1.In all experiments, for the images, we applied the standard CLIP image pre-processing, which included resizing, center cropping, and normalization. For the texts, we applied the standard CLIP text pre-processing, which tokenized the texts into a series of integers, each representing a unique series of characters.The value of λ used in our DRM training was picked from {1, 2, 3, 4, 5} based on the performance on the ID validation set. Following Goyal et al. (2023) , we also implemented early stopping based on the ID validation performance.The hyperparameter ρ in WiSE-FT, θ wise-ft = ρ • θ zs + (1 -ρ) • θ ft , is chosen from the range 0.1 to 0.9 via ID validation.All experiments were conducted on a high-performance computing cluster equipped with NVIDIA DGX H800 nodes. Two H800 GPUs with 80 GB VRAM were utilized for all trainings involving CLIP ViT-B/16 and CLIP ViT-L/14, while four H800 GPUs were employed for the training of CLIP ViT-L/14@336.In our experiments, we implemented the ERM part of DRM with FLYP (Goyal et al., 2023) . The additional computational cost of DRM, compared to FLYP, primarily arises from the preparation of soft labels for the targets of the second risk in DRM and the minimization of this second risk. In short, the training and inference cost for DRM increased by about 20% from FLYP. This additional cost is insignificant compared to the attained performance gain. In Table 13 , we detail the computational costs and timing for fine-tuning CLIP ViT-L/14 models on ImageNet. Our research utilized GPT-4 to generate concept descriptions for the core visual features of various classes across different domains. It is important to note that the scope of GPT-4's knowledge in certain domains might be limited, and as a result, the model may not always generate useful concept descriptions. For instance, we found that GPT-4 generated inaccurate concept descriptions in medical imaging fields like ocular disease and breast histology.Additionally, due to the vast number of concept descriptions generated, we have not been able to verify the accuracy of each generated concept description. To enhance the quality of these descriptions, potential improvements could involve engaging domain experts to review and correct errors, or generating descriptions manually. Another approach could be to gather visual prototypes and use advanced multimodal LLMs such as GPT-4V (Achiam et al., 2023) , LLaVA (Liu et al., 2023) , or MiniGPT-4 (Zhu et al., 2024) , which might yield more precise descriptions of the core visual features.A further limitation concerns the CLIP models used in our experiments. These models may not perform optimally across all domains, particularly in less common areas, where they may lack requisite knowledge in both images and text. The effectiveness of our DRM method is therefore contingent upon the breadth and depth of the pre-training data of CLIP models. Unfortunately, the specifics of the CLIP pre-training dataset have not been disclosed by OpenAI, adding an element of uncertainty to the performance of our method in niche domains.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).More details about concept description generation can be found in Appendix C. 


Decision: This paper presents a new technique based on a combination of empirical risk minimization and worst-case risk minimization for robust finetuning of zero-shot models. The reviewers appreciated the novel and interesting motivation, theoretical foundation, and strong results of the proposed method as well as the clarity of the manuscript. They also raised concerns with lack of in-depth analysis (e.g., reliability and robustness of the core features obtained by the method, complexity analysis, impact of the quality of generated descriptions and that of the LLM used) (D5j1, eddC, zSx8), sensitivity to the hyperparameter $\lambda$ (zSx8), potential risk of using imperfect estimate of the oracle model (gi7f) and that of ignoring inter-class similarities (D5j1), lack of discussion on counterintuitive results (zSx8), and missing comparisons with closely related work (D5j1). Most of these concerns were successfully addressed by the rebuttal and subsequent responses in the discussion. Consequently, the reviewers unanimously championed the paper after the discussion period. The AC agrees with the reviewers and recommends acceptance. The authors are strongly encouraged to carefully revise the paper to reflect the valuable comments by the reviewers (in particular those on the sensitivity to $\lambda$), to provide a tighter connection between the theorem and the actual DRM objective and its implementation, and to add new results and discussions brought up in the rebuttal and discussion.

Reviews: review: paper_summary: paper_summary: This paper presents Dual Risk Minimization (DRM), a novel approach that combines empirical risk minimization (ERM) and worst-case risk minimization (WRM) for fine-tuning the CLIP model while maintaining its out-of-distribution robustness. The idea is to create a classifier that utilizes concept descriptions of each class generated by large language models (LLMs). This classifier helps “pull out” core features from the image embeddings, serving as a regularizer to enhance the robustness of CLIP fine-tuning. Moreover, the paper introduces a min-max normalization technique to address a caveat in the regularization. Experimental results demonstrate the effectiveness of the proposed DRM, yielding promising in-distribution and out-of-distribution accuracy.
strength_weakness: strength_weakness: - At Line 199, the paper claims that the LLM-generated concept descriptions can “pull out” the core features from the image embeddings. However, this claim is only demonstrated by the few examples shown in Figure 2, which is insufficient. Moreover, the stochastic nature of LLM generation can further complicate the matter. The paper lacks a thorough analysis of the reliability and robustness of the core features obtained through this proposed method. 

- At Line 217, the paper mentions that the affinity between $x$ and any other class $y'$ should ideally be zero, thereby motivating the modification of the second term in Eq. 10. However, this might ignore the fact that the ground-truth class $y$ exhibits varying degrees of similarity with different $y’$. It remains unclear how the proposed method can achieve better performance, as shown in Table 3, while disregarding the learning of such inter-class similarities.

- The second term in Eq. 10. is similar to knowledge distillation, as acknowledged by the paper at Line 207. However, the paper neglects to discuss or compare this term with other robust fine-tuning methods, such as [1], which also incorporate knowledge distillation. 

[1] DELTA: Deep Learning Transfer using Feature Map with Attention for Convolutional Networks, ICLR 2019.
questions: questions: Besides the weakness shown in the above section, please also see the following questions: 

- Given that the paper suggests that the classifiers built with LLM-generated concept descriptions have the ability to extract core features, could we simply fine-tune the CLIP model utilizing these concept descriptions instead of using the one with default descriptions? If we do so, could standard fine-tuning with ERM already maintain its robustness? 

- What would be the in-distribution and out-of-distribution accuracy for the concept description classifiers? Given that these classifiers aim to extract more core features, would they be more robust to distribution changes, even without fine-tuning? 

- How does the min-max normalization affect the training on long-tailed datasets, like iWILDCam? Specifically, since the min-max normalization computes $x$ with all images in a class $y$, could tailed classes result in less stable normalization because they have much fewer samples?
limitations: limitations: The paper acknowledges that a potential limitation of the proposed method lies in the potential limited domain knowledge of LLMs in specific domains. To better illustrate this limitation, it would be beneficial for the paper to provide concrete examples, such as failure cases, that show how the method behaves when it reaches its limits.
paper_summary: paper_summary: This paper introduces dual risk minimization (DRM), a novel approach that combines empirical risk minimization (ERM) with worst-case risk minimization (WRM) to enhance the robustness of fine-tuning zero-shot foundation models. The authors address the limitations of existing methods that fail to effectively preserve robust features during fine-tuning by utilizing concept descriptions from LLMs to create soft labels for estimating worst-case risk, focusing on core features that define target classes. Empirical results demonstrate that DRM significantly improves out-of-distribution performance on benchmarks such as ImageNet and WILDS, establishing new state-of-the-art results.
strength_weakness: strength_weakness: - The dependence on the lambda seems to be quite significant, and there is no ablation study addressing this in the main paper. It would be good to see results regarding the impact of lambda.
- It would be helpful if the paper included details on the computational time and cost for each experiment to understand the efficiency and scalability of the proposed methods.
questions: questions: Please refer to the weaknesses.
limitations: limitations: Yes. They address the limitations of their work.
paper_summary: paper_summary: This paper proposes a method for robust fine-tuning by combining empirical risk minimization with worst-case risk minimization to better preserve core features. The approach uses descriptions of core features obtained from large language models (LLMs) like GPT-4 and employs these descriptions to estimate worst-case risk. This method aims to enhance robustness by minimizing empirical risk on the training set of downstream tasks while improving the understanding of core features. As a result, the proposed method demonstrates significant performance improvements over existing methods on datasets such as ImageNet, iWildCam, and FMoW.
strength_weakness: strength_weakness: - The main weakness of the proposed method lies in the sensitivity to the hyperparameter $\lambda$ . As shown in Table 6, slight deviations from the optimal $\lambda$  value can lead to significant performance drops. Finding the optimal $\lambda$  for different datasets could be impractical, and this issue needs to be addressed by reporting ablation studies on additional datasets beyond iWildCam.
- The results in Table 6 show no trade-off between in-distribution (ID) and out-of-distribution (OOD) performance, which is counterintuitive. The authors should provide an explanation for this observation.
- While the overall quality of writing is high, certain sections and figures need improvement. For example:
	- Line 159 and Eq. 3 need a clearer explanation of the DRM formulation relaxation process.
	- Figure 1 should explicitly state the meaning of the first and second bars.
	- Figure 2 could be misleading as it shows the original image having higher affinity with the class concept description than the background-free image without showing affinities with other images.
- The method relies heavily on the quality of the class concept descriptions generated by GPT-4. An analysis and ablation study on the quality of these descriptions are necessary. Additionally, the impact of using different LLMs should be explored.
questions: questions: - The proposed method has only been tested with full fine-tuning. I think this method also can be combined with methods like LP-FT or FLYP. I am curious about the results when they are combined with the proposed method.
- Why is there no observed trade-off between ID and OOD performance in Table 6? An explanation for this phenomenon is needed.
- How robust is the method to the quality of class concept descriptions? What happens when a different LLM is used?
limitations: limitations: The authors have addressed some limitations of their work but have not sufficiently discussed the critical issue of hyperparameter sensitivity and its impact on performance. There are no identified negative societal impacts.
paper_summary: paper_summary: To address the robustness of foundational models under distribution shift conditions, this paper proposes a dual risk minimization approach. Specifically, the authors combine Empirical Risk Minimization (ERM) and Worst-Case Risk Minimization (WCRM) to optimize the model fine-tuning process. To achieve accurate distribution estimation, the authors use class descriptions generated by GPT-4 as prompts inputted into the model and apply max-min normalization to the classification probabilities to address certain items that should be zero. Experiments demonstrate that the dual risk minimization method significantly enhances robustness in zero-shot fine-tuning.
strength_weakness: strength_weakness: 1. To accurately estimate $p_c(y|c) $, the authors use prompts generated by GPT-4 to initialize the classifier parameters. Nevertheless, this classifier may still not be the optimal choice.
2. Given that better prompts have been used to initialize the classifier cd, why not directly fine-tune cd instead of using cd to assist in fine-tuning df? Moreover, why is the performance of cd inferior to df when comparing row 2 and row 3 in Table 3?
3. In Table 7, if the models trained with df and cd (row d1 and d2) are directly used for inference with (13), how will the performance be?
questions: questions: See Weakness.
limitations: limitations: Yes

Comments: title: Message to Reviewers: We look forward to hearing from you
comment: Thank you for your thorough response. I have carefully reviewed all the rebuttals and your clarifications, which have addressed many of my concerns. However, some issues still remain.

First, the observation that increasing $\lambda$ leads to improvements in both ID and OOD performance appears to be specific to the iWildCam dataset. The results on ImageNet show a different trend. Although it is true that $\lambda$ values between 2 and 5 generally result in good ID and OOD performance, the consistent improvement in OOD performance as ID performance increases does not seem to hold universally. 

Moreover, all ablation studies have been conducted solely on iWildCam, whereas the qualitative examples are shown on different datasets. This narrow focus on a single dataset for ablation and analysis makes it difficult to fully understand and validate the proposed method. I believe it is necessary to provide experimental results and analyses across a wider variety of datasets to ensure a comprehensive understanding of the proposed approach.

The concept of combining ERM and WRM is intriguing, and the approach is interesting. However, the need to generate concept descriptions for all classes, along with the fact that the effectiveness of this approach varies depending on the quality of the generated descriptions, the $\lambda$ hyperparameter reduces the practical utility of this method. For these reasons, I intend to maintain my current score.
Dear Reviewers,

Thank you again for reviewing our paper. We are more than happy to address any further questions or concerns you may have before the conclusion of the author-reviewer discussion period (in 30 hours). Please feel free to reach out with any further feedback. We look forward to hearing from you.

Best regards,

Authors
Thank you for taking the time to read our response and increasing your score! We are glad to hear that the response addressed your concerns.
The author have addressed my concern, and I will change my score from 5 to 6.