Contrastive Adapters for Foundation Model Group Robustness
Authors: Michael Zhang
Christopher Re
Venue: NeurIPS 2022 Accept (2022)
Keywords: foundation models
robustness
hidden stratification
subpopulation shift

Abstract: While large pretrained foundation models (FMs) have shown remarkable zero-shot classification robustness to dataset-level distribution shifts, their robustness to subpopulation or group shifts is relatively underexplored. We study this problem, and find that foundation models such as CLIP may not be robust to various group shifts. Across 9 robustness benchmarks, zero-shot classification with their embeddings results in gaps of up to 80.7 percentage points (pp) between average and worst-group accuracy. Unfortunately, existing methods to improve robustness require retraining, which can be prohibitively expensive on large foundation models. We also find that efficient ways to improve model inference (e.g. via adapters, lightweight networks that transform FM embeddings) do not consistently improve and can sometimes *hurt* group robustness compared to zero-shot. We therefore develop an adapter training strategy to effectively and efficiently improve FM group robustness. Our motivating observation is that while poor robustness results from groups in the same class being embedded far apart in the foundation model "embedding space," standard adapter training may not actually bring these points closer together. We thus propose contrastive adapting, which contrastively trains adapters to bring sample embeddings close to both their ground-truth class embeddings and same-class sample embeddings. Across the 9 robustness benchmarks, contrastive adapting consistently improves group robustness, raising worst-group accuracy by 8.5 to 56.0 pp over zero-shot. Our approach is also efficient, doing so without any FM finetuning and only a fixed set of FM embeddings. On popular benchmarks such as Waterbirds and CelebA, this leads to worst-group accuracy comparable to state-of-the-art methods, while only training <1% of the model parameters.

Contrastive Adapters for Foundation Model Group Robustness
Michael Zhang and Christopher Ré Department of Computer Science
Stanford University wtw.seeeeeeststsoorrr.oooo
{mzhang, chrismre &acs. www.ssrrd...... edu
Abstract
While large pretrained foundation models (FMs) have shown remarkable zero-shot classification robustness to dataset-level distribution shifts, their robustness to sub- population or group shifts is relatively underexplored. We study this problem, and find that foundation models such CLIP may not be robust to various group shifts. Across robustness benchmarks, zero-shot classification with their embeddings re- sults in gaps of up to 80.7 percentage points (pp) between average and worst-group gropp accuracy. Unfortunately, existing methods to improve robustness require retraining, which can be prohibitively expensive large foundation models. We also find that efficient ways to improve model inference (e.g.. via adapters, lightweight networks that transform FM embeddings) do not consistently improve and can sometimes hurt group robustness compared to zero-shot. We therefore develop an adapter training strategy effectively and efficiently improve FM group robustness. Our motivating observation is that while poor robustmess results from groups in the same class being embedded far apart in the foundation model "embedding space," standard adapter training may not actually bring these points closer to- gether. We thus propose contrastive adapting, which contrastively trains adapters to bring sample embeddings close to both their ground-truth class embeddings and same-class sample embeddings. Across the robustness benchmarks, contrastive adapting consistently improves group robustness, raising worst-group accuracy by 8.5 to 56.0 pp over zero-shot. Our approach is also efficient, doing without any FM finetuning and only fixed set of FM embeddings. On popular benchmarks such as Waterbirds and CelebA, this leads to worst-group accuracy comparable state-of-the the-hartt art methods, while only training <1% % of the model parameters.
Introduction
Foundation models (FMs)-large pretrained models trained on massive datasets-offer an exciting new paradigm for deep learning. Recent works have shown that without any finetuning, foundation models can generalize well to various datasets [11, 36, 59, 69] and exhibit impressive robustness to certain distribution shifts [42, 76]. Under this zero-shot paradigm, practitioners can avoid training task-specific models, and instead use FM embeddings for efficient and effective inference.
However, underexplored question is how robust this zero-shot inference is to "group shifts," distribution shifts between subpopulations or meaningful groups in data. Prior works have established that group robustness-i.e, performing well on all groups-is fundamental and real-world challenge for modern deep learning [5, 12, 40, 51, 55, 66, 71J. Yet most prior foundation model evaluations focus overall average performance [42, 59, 76]; few works consider FM accuracy across groups.
In this work, we thus study foundation model group robustness. We motivate this problem by first showing that foundation models can have poor zero-shot group robustness. Evaluating 11 foundation
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Cross- entropy Adapler
Adapter Classification
Clso
Enooder
Contrastive Adapter
Adapter
Frooder
[(u)
f(u)-V;
f(u)
Vo
(w)
Pb
P2
==1
(a)
(b)
(c)
Figure 1: (a) Adapter classification with FM embeddings. Adapters learn transformations to align sample embeddings to ground-truth class embeddings. (b) Cross-entropy loss encourages alignment between class embeddings [22]. (c) Contrastive adapting adds alignment between sample embeddings.
models across 9 robustness benchmarks, we find they achieve up to an 80.7 percentage point (pp) gap between average and worst group accuracy, only classifying 6.0% of worst-group samples correctly.
We therefore aim to improve FM group robustness. This poses several challenges and open questions. First, while improving group robustness in machine learning is well-studied, existing robustness methods require retraining one (and often more than one) entire models [1, 16, 39, 47, 51, 65, 71, 72, 79]. This be prohibitively expensive for foundation models due to their size and scale, raising the question of whether we can make these models more robust without any retraining or finetuning. Second, for zero-shot classification, many practitioners may also only access foundation model outputs or embeddings (e.g., via APIs To improve robustness, ideal solutions should only require pretrained FM embeddings. However, these same embeddings lead to poor zero-shot robustness, raising the question of if they even encode the information needed to classify all groups correctly.
Motivated by these challenges and questions, study effective and efficient solutions for better FM group robustness. As a baseline, we first find that while efficient methods to improve FM inference- such training linear probes [42, 59] and adapters [22, 33] top of FM embeddings- improve group robustness over zero-shot (reducing the gap by up to 50.2 pp on representative benchmarks), they fail to do consistently, and can hurt robustness. They reduce worst-group accuracy by up to 37.9 pp. and increase the accuracy gap by up to 74.9 pp. To reason about this inconsistency, we note that poor zero-shot robustness results when FMs embed same-class samples in different groups "far apart" in embedding space. While adapter training achieves higher robustness than linear probing, we find settings where it still fails to close this distance, e.g., if training data is group- imbalanced.
To then handle these scenarios and consistently improve group robustness over zero-shot, we propose contrastive adapting, simple adapter training method that places greater emphasis bringing these initially "far apart" points together. For each task, we first use foundation models to compute embeddings for each training sample and class. We then train adapters- small bottleneck MLPs on these embeddings. Like prior work [22], these adapters take sample embeddings as inputs, and output transformed embeddings with greater cosine similarity their ground-truth class embeddings. However, the key difference is that contrastive adapting also applies a supervised contrastive loss over other sample embeddings. Specifically, we provide way "pull together" far apart sample embeddings in the same class, and "push apart" nearby sample embeddings in different classes
In our experiments, we validate that contrastive adapting effectively and efficiently improves FM group robustness. First, across all robustness benchmarks, we find contrastive adapting consistently improves worst-group accuracy over zero-shot (by 8.5 to 56.0 pp), using training group labels and only training MLPs with 0.1 % to 0.3% of the original FM parameters. Then, on a representative set of benchmarks with various group shifts and training data group sizes, we find contrastive adapting can substantially outperform prior adapter training strategies, and outperforms other approaches that only use fixed FM embeddings (achieving up to 12.4 pp higher worst-group accuracy than the next best method on average). Finally, beyond just improving FM robustness, we find contrastive adapting also achieves effective and efficient group robust classification in general. We achieve near state-of-the-art (SoTA) or SoTA vorst-group accuracy on popular robustness benchmarks with only 1.0% of the trainable parameters (e.g., improving 0.2 pp over the prior SoTA [52] on CelebA [48]).
http::///tttooooomm..ccoooooocct http://toooo.ttttsssssssuutoooot http:://ssuuto.aa...omooo////// https///docs.chhrraail

---

In summary, we find that while FM zero-shot classification may not be group-robust, we can significantly improve robustness without any finetuning. This suggests the information to classify groups is frequently in their pretrained embeddings; we may just need proper methods to extract it.
2 Related Work
Our work builds on (i) methods to improve group robustness, and (ii) methods to improve foundation model inference without accessing or finetuning their original weights. We briefty describe these works here, and include an expanded discussion in Appendix D.
Improving group robustness. Many works aim improve group robustness. If training group labels are known, prior methods often balance group sizes during training, via sample balancing [17, 28, 34, 39], importance weighting [13, 68]. or robust optimization [2, 65]. We do not assume training group labels. With these assumptions, common approach first trains a model with empirical risk minimization (ERM), before using this model's predictions to infer groups. Methods then train a second robust model with sample balancing [47, 51] or robust optimization [16, 52, 71J using inferred group labels, or representation learning to learn similar representations for groups in the same class [79]. While effective at improving group robustness, these solutions require training one (and often more than one) models. This can make applying them foundation models prohibitively expensive.
Improving foundation model inference efficiently. Other prior works improve foundation model downstream performance, without having finetune or update original model weights. Prompt tuning optimizes the inputs of a FM while keeping the original model weights frozen. Optimizing either text [43, 45, 83, 84) image [3. 77) inputs can improve frozen foundation model's 's downstream task accuracy. However, doing so can require multiple passes through the foundation model, which may become expensive in certain situations (e.g., interacting with the model via a commercial API), Another paradigm adds small trainable parameters to the original model, either within its layers or on top of its embeddings. These include linear probes (linear classifiers) [59] and adapters (small bottleneck MLPs) [33, 57, 58, 60]. Recently, Kumar et al. [42], Wortsman et al. [76] propose methods with linear probes to improve robustness after finetuning
-iittrriittiittoi (OOD) shifts [30, 32, 62, 74]. Gao et al. [22] train adapters on pretrained embeddings to improve average downstream aceuracy. We focus on group shifts within a dataset. We also show standard adapter training can hurt group robustness, and propose alternatives to consistently improve group robustness.
3 Problem
In Section 3.1, we first describe the group robustness problem setting. In Section 3.2, we illustrate this problem with foundation models. We show that zero-shot classification with foundation models, and existing baseline approaches to improve downstream inference, can result in poor group robustness.
3.1 Preliminaries: group robustness and task setup
We emphasize robustness to distribution shifts between groups in this work. For setup, we follow prior works [40, 47, 65, 71] that alternatively describe the phenomenon as hidden stratification [71] or subpopulation shift [40]. For some task, we have N samples {xx, Vs: J. with sample features or inputs 2; X. class labels y/4 E Y. and group labels g, E G. Let C = | be the number of classes. We use 9i to indicate the group that each sample belongs in, but do not observe group labels during training. Distribution shifts may occur between samples in different groups but the same class.
Every sample (I,, Vi- 9:) is drawn from some unknown joint distribution P. Let P, be the specific distribution conditioned on g for any g E G. For classification loss l y x y + R and classifier fu X + D, we want Sa to be accurate, i.e. achieving low average error:
Lave (fo) == ((( [((fo()) v)]
(1)
and group robust, i.e., achieving small gap between its average error and its worst-group error:
Lwy (So) == max gEg
((..,.))~ (((o(()))) v)
Different from domain generalization or OOD evaluation settings [31, 32, 44, 64, 82], we observe each data group in training, validation, and test splits. However, standard training via empirical risk minimization (ERM) can still lead to poor test set group robustness because training groups may be imbalanced [65, 71, 79]. Here, foundation models are not trained on the training data, but we show that zero-shot classification with foundation models can still result in poor group robustness.
3

---

Waterbirds
BREEDS Living-17
CIFAR-10.02
Figure 2: Samples of different group shifts for robust evaluation (2 classes, 2 groups per class shown).
3.2 Empirical findings of poor foundation model group robustness
To motivate the rest of this work, we now demonstrate the group robustness problem with foundation models. We first describe different natural group shifts for evaluation. We next detail primary baseline approaches. We finally summarize our findings after evaluating these baselines on 11 popular foundation models across 9 standard group robustness benchmarks used in prior work (40, 49, 61, 65, 67]. We present four representative scenarios based on training data assumptions and group robustness outcome. Critically, we find that zero-shot classification with foundation models may result poor group robustness. We also find that baseline methods to improve downstream transfer do not consistently improve group robustness, and can make group robustness worse. Dataset group shifts. We benchmark methods on the following sources of group shift (Figure 2):
Spurious confounders. We evaluate across groups which may or may not carry spurious confounders -input features predictive for some, but not all groups in a class. For example, Waterbirds [65, 75J, a water background a confounder for the waterbirds class.
Subelass variance. We evaluate across groups which are different fine-grained subclasses. For example, in BREEDS Living-17 [67], the ape class includes images of gibbons and gorillas.
Data source variance. We evaluate across groups which are the same class but sourced from different datasets. For example, set up the CIFAR-10.01 dataset by combining CIFAR-10 [41] and CIFAR-10.2 [49]. The airplanes class contains samples from both datasets.
Baseline methods. To evaluate foundation model group robustness, we consider the following baseline methods. Following prior work [20. 36, 46, 50, 59], for all approaches we first compute N sample embeddings and C' class embeddings using foundation model. With foundation model embedding dimension D, let un E RDD be a sample embedding and 6, E TRD be a class embedding. Zero-shot classification [59]: We classify each sample via the nearest class embedding to its sample embedding un. Specifically, we compute the class-wise logits for each sample V', as
fo(xa; T On//
where ún = /n////u is the (f-2) normalized sample embedding of xn, W E DxC is a matrix whose columns are the normalized class embeddings {6.) and The highest class logit corresponds to the nearest neighbor and largest dot product. As standard, for class embeddings
temperature parameter.
convert each class name natural language prompt, e.g., "photo of [class namel", and feed the tokenized prompt to a foundation model's text encoder. As in prior work [59], we engineer class prompts by trying several templates. We defer details to Appendix A.2, such as optimal templates (Table 11) and a list of all templates tried (Table 20).
Linear Probe [59, 76]: We train linear classifier on top of training data sample embeddings. Specifically, with classifier fa(u)= = W u, we update the weights W E IR DxC with a cross-entropy
1N loss applied over training data sample embeddings { and labels {ym, J
Adapter [22, 60]: We train single 2-layer bottleneck multilayer perception (MLP) output transformed sample embeddings, which we use instead of the original sample embeddings to classify with in the zero-shot procedure above. Specifically, with adapter hidden-layer dimension H, ReLU activation function a. and adapter weights ó [W1. W2]-where where W1 E Dx is linear down down-projection and W2 e BR/x 1xD a linear up- up-projection we compute "adapted" embeddings
fa(u) W₂ (W]Tw)
We classify samples with the zero-shot class matrix W, temperature т, and normalized adapted embeddings (=( = (u(w//l/o(u [[)])]] The final outputs are given by fo(w; W,+) = WT fo(u)/T. Like with linear probes, we update & with cross-entropy loss using training data labels and softmax over the dot product-computed logits class wise probabilities.
For evaluation, we train both linear probes and adapters with standard empirical risk minimization (ERM), which aims to minimize the empirical risk: ((fe) 1 E ((fo(un), yn).

---

Table 1: Baseline gorst-group (WG) and average (Avg) accuracies with zero-shot classification, linear probes. and adapters. Best metric in bold. While training linear probes and adapters can improve group robustness (reducing the worst-group versus average accuracy gap by 57.4 pp on BREEDS Living-17), can also result in poorer robustness (in red), increasing the gap by 74.9 pp on CelebA.
Method
Waterbirds
CelebA
BREEDS Living-17 17
CIFAR-10.00
Accuracy (%)
WG
Avg
Gap
WG
Avg
Gap
WG
Avg
Gap
WG
Avg
Gap
Zero-shot
36.6
92.2
55.6
74.0
81.9
7.9
6.0
86.7
80.7
39.1
69.9
30.8
Linear Probe
7.9
93.5
85.6
11.9
94.7
82.8
53.3
90.8
37.5
51.3
77.7
26.4
Adapter
60.8
96.0
35.2
36.1
94.2
58.1
70.7
94.0
23.3
68.8
86.0
17.2
Table 2: Representative outcomes for improving group robustness.
Class-wise Group Size
Improved Group Robustness?
Example Dataset
Group Shift
Largest
Smallest
Balanced?
Linear Probe
Adapter
Waterbirds
Confounder
1057
56
CelebA
Confounder
22880
1387
X
X
BREEDS Living-17 17 DEAAR-10022
Subclass 1076
1009
V
Data source
4039
431
X
Discussion and representative outcomes. In Table 1, we report worst-group and average accuracies along with their corresponding gaps four representative group robustness datasets, using zero-shot classification, linear probes, and adapters on CLIP ResNet-50 embeddings. We select datasets to report based on training data setup and group robustness outcome, where we find that (i) the relative group size ratios, (ii) the type of group shift, and (iii) the choice of adapter or linear probe influences group robustness improvements. We note descriptive characteristics and outcomes in Table and summarize three main takeaways below. Appendix A contains results for all datasets and models.
1 Foundation model zero-shot classification may not be group robust: Across datasets, we find that zero-shot classification with CLIP ResNet-50 embeddings can achieve 7.9 to 80.7 pP gaps between worst-group and average accuracy. Worryingly, poor group robustness is accompanied by high average error (from 69.9% to 92.9%), the usual metric for evaluating zero-shot classification. This further supports the importance of improving group robustness.
2 Efficient baselines do not consistently improve robustness: We find that while previously proposed linear probes and adapters are efficient ways to improve accuracy on downstream tasks, these benefits do not consistently carry over to improving group robustness.
When training data is balanced, both linear probes and adapters can substantially improve group robustness and worst- group accuracy (reducing the robustness gap by 43.2 and 54.7 pp respectively on BREEDS Living-17), However, when minority groups are rare, in some instances, approaches can hurt group robustness. On CelebA, adapters and linear probes increase the gap by 50.2 and 74.9 pp, and reduce worst-group accuracy by 37.9 and 62.1 pp.
3 We can improve group robustness with only foundation model embeddings: Our positive results suggest that poor zero-shot classification may not be because sample embeddings lack the information required to classify groups correctly. Rather, we may just require the right training strategies to learn how to better classify by this information.
Altogether, takeaways 1 and 2 motivate the need for methods to effectively improve robustness in the foundation model setting. Takeaway 3 suggests we can make progress on this problem.
4 Method
Having established the group robustness problem in Section 3, we now propose a simple contrastive adapter training strategy to improve group robustness. In Section 4.1, we setup our approach by identifying possible sources of limitation with standard adapter training. In Section 4.2, we then use these insights to propose simple yet effective approach that counteracts these limitations.
4.1 Understanding prior limitations via embedding metrics
To guide a first-step strategy for improving robustness, we first outline high-level reasoning for why zero-shot and ERM-trained adapters fail to classify groups correctly. Recall that a key property of group robust classification is that all sample embeddings belonging to the same class should embed closer to their ground-truth class embedding than any other class embedding. If zero-shot classification for specific class is accurate average but not group robust, then in the pretrained
5

---

foundation model embedding space there exists groups that embed "close" to their ground ground-truth class embedding, and groups in the same class that embed "far away" (measured via cosine similarity). One way to interpret standard adapter training with FM embeddings via ERM is that it aims to bring these initially far apart sample embeddings closer to their ground-truth class embedding. Restating the standard sample cross- eross-entropy loss with adapters makes this clear as an InfoNCE loss [14, 56]:
exp(fo(u)TO/T)
v) log Ee=1 ( Ūe/T))
(5)
with sample embedding u as an anchor, class embedding v of ground-truth y as single positive, and the other C - 1 class embeddings as negatives. Via ERM of the sample cross-entropy loss, adapters thus bring
anchors closer to their class embedding positives (minimizing Eq. 5).
However, in Section 3 we found this loss works in some scenarios but not others. Intuitively, Eq. 5 can fail to bring samples closer to their correct class embedding (e.g., on CelebA). To find additional ways to bring points together, we hypothesize that poor robustness also accompanies poor similarity between sample embeddings from different groups but the same class. We verify this Figure by empirically measuring the average pairwise cosine similarity and group alignment loss Lalign [79]- which measures the pairwise Euclidean distance-between sample embeddings in the same class but different groups. We compare these metries with embeddings computed with trained adapters and the initial foundation model embeddings, and find that higher worst-group accuracy corresponds to higher cosine similarity and lower alignment loss between groups in the same class.
Watnrhirds
CelenA
EnEeDs Living-17
CIFAR-10.02
Act, Sm0m
fot S6/m
80
grap
8
80
60
10
g
20
Adapter
10
80
70
Zaro-shot
04
Culign (t-)
Gargn (-)
Caigr (-)
Laign (+)
Water
CelebA
living-17
CIFAR-10.02
10
Warst group foc: %e [+)
16
Warst Warst-gruuu poost-grruuppoo acc. S [-)
18
13
-2
8
8
8
1
88
8
88
d
8
5
*
Zarc-shat
Adapter
5uB 03
022
03
0.4 0.5
0.6
00.5
0.7
0.85
0.20
0.20
0.30
0:30
Cooase Simmlaetty [-]
Coeere Sier Simdenntyy (-4]
Codine Similarity (-)
Costie Siillarity (-+)
Figure 3: Across CLIP model architectures, cosine similarity and alignment loss between groups of the same
class tracks worst-group error. Notably, training ERM adapters may fail to move these metrics in the desired direction, which corresponds with poorer robustness (e.g.. on CelebA).
4.2 Approach: Contrastive Adapting
To improve robustness, we therefore propose to more effectively bring far away samples together by introducing greater training signal via other sample embeddings. Instead of limiting ourselves to single class embedding positive and a limited set of C' - 1 negatives, we expand our positives by including sample embeddings for points in the same class far away from the anchors among pretrained embeddings (e.g., likely in different groups). We expand our negatives with sample embeddings from different classes. Following prior work [23, 79] that finds sampling hard negatives beneficial for robust contrastive learning, we also use the computed foundation model sample embeddings to sample negatives from points nearest to the anchors but in different classes. As the number of training data points N is often much larger than the number of classes C. these choices are further supported by prior work suggesting more positives and negatives are beneficial for contrastive learning [38, 63). In practice, contrastive adapting is simple to implement with three components:
Foundation model embedding and prediction: We compute FM embeddings over labeled training data. To guide sampling, we collect zero-shot predictions over this data.
Contrastive sampling: For each class, we identify an "anchor" sample embedding u E l4 that zero-shot predicts incorrectly, and P "positive" sample embeddings P(u) that zero-shot classifies correctly. We do this as a heuristic for finding samples "far apart" the FM embedding space, so pushing them together improves robustness over zero-shot. We also identify M hard "negative" sample embeddings M(u) by computing the nearest neighbors to the anchors in different classes, using cosine similarity between the sample embeddings.

---

Training objective: We use a supervised contrastive loss [38] with the sample embeddings, i.e.
(oo()(())) -1
Σ log
exp(fa(u) fo(p)/T)
(6)
(p((fol)TTool)//)) + EmEM(w) exp(fo(u) fo(m))/T)
pEP(M)
We also use a standard cross-entropy loss over minibatches of sample embeddings and their class embeddings. This aims to keep adapted embeddings close to their ground-truth class embeddings. To avoid undoing Eq. 6 and push "far away" points closer to their ground-truth class embeddings, we upsample the number of zero-shot incorrect samples equal the number of zero-shot correct samples in each minibatch. We thus use contrastive supervision from class and sample embeddings.
Robust generalization with adapters. The contrastive loss in Equation 6 is also supported by recent results suggesting that minimizing the class-wise alignment loss Lalign helps bound the worst-group versus average error gap for that class (cf. Thm 3.1, Zhang al. [79]). The bound however scales with the Lipschitz constant of the neural network, and upper bounds for estimating this constant can grow with the size of the network [19, 73]. However, as our adapters are small 2-layer MLPs, estimates of this constant suggest we can obtain better generalization with fewer training samples [25, 37, 53, 78]. In Section 5.3, we later show this corresponds to better data efficiency.
5 Experiments
We now validate that contrastive adapting enables effective and efficient group robustness, First, in Section 5.1. we evaluate the effectiveness of contrastive adapting against efficient methods to improve FM inference. We study whether the approach consistently improves worst-group accuracy and group robustness over zero-shot classification, how contrastive adapting compares against other efficient methods that only require pretrained model embeddings, and whether contrastive adapting scales to a variety of pretrained model architectures. Next, in Section 5.2, we shed further light on contrastive adapting's 's performance by studying the importance of its individual components, ablating the contrastive objective and sampling strategy. Finally, in Section 5.3, we study the efficiency of contrastive adapting against effective group robustness approaches. We find that the prior robustness gains are not only relative to other efficient FM training methods; contrastive adapting also enables state-of-the thettt robustness on popular benchmarks, but with greater parameter and data efficiency.
5.1 Robustness comparison for efficient foundation model methods
To first judge the effectiveness of contrastive adapting, we evaluate the method across the same set of initial robustness benchmarks and foundation model architectures discussed in Section 3. As in prior group robustness evaluation, we do not assume training groups labels, but do assume group labels in validation data for hyperparameter tuning and model selection [40]. We include experimental details for all models and hyperparameters in Appendix C.
As baselines, we compare against zero-shot shot classification [59], ERM linear probing [42, 59], and ERM adapter training [22]. We also compare against recent methods designed to improve downstream transfer in related settings, while similarly only requiring pretrained model embeddings:
Weight space ensembling (WiSE-FT) [76], which first trains linear classifier with standard ERM, and then ensembles the classifier outputs with the initial zero-shot predictions. While proposed for both training linear classifiers and finetuning the original weights of foundation model, we focus on the linear classifier version for fair comparison in our setting.
Deep feature reweighting (DFR) [39], which first trains a linear probe on embeddings computed from a pretrained model over group- balanced data. As we do not assume training group labels, we first infer groups using zero-shot classification with foundation model embeddings. As in prior work [47, 79], we treat the incorrect and correctly classified samples proxies for different groups.
Finally, if we have validation group labels, we plausibly know what groups are in the test data. We thus also compare against group-informed prompting (Group Prompt ZS), which performs zero-shot classification using prompts with group information (e.g., "a waterbird on a land background").
Consistent robustness improvements over zero-shot. In Figure 4 we report contrastive adapting's relative gains in worst-group accuracy over zero-shot classification on all robustness benchmarks. Unlike prior adapter training approaches, contrastive adapting consistently improves group robustness over zero-shot classification, achieving 8.5 56.0 pp higher worst-group accuracy.
7

---

Table 3: Evaluation of methods for improving group robustness of CLIP models. Across representative benchmarks and CLIP models, contrastive adapters consistently improve gorst-group accuracy over shot classification (by 10.2 to 76.0 pp). 1st 2nd best gorst-group (WG) and robustness gaps bolded/ / underlined.
Waterbirds
CelebA
BREEDS Living- 17
CIFAR- -0.02
Method / Acc. (%)
WG
Avg
Gap
WG
Avg
Cap
WG
Avg
Cap
WG
Avg
Cap
Zero-shot (ZS)
36.6
92.2
55.6
74,0
81.9
7.9
6.0
86.7
80.7
39.1
69.9
30.8
CLIP ResNet-S50
Group Prompt
55.9
87.8
31.9
70.8
82.6
11.8
30.0
90.6
60.6
N/A
N/A
N/A
ERM Linear Probe
7.9
93.5
85.6
11.9
94.7
82.8
53.3
90.8
37.5
51.3
77.7
26.4
ERM Adapter
60.8
96.0
35.2
36.1
94.2
58.1
70.7
94.0
23.3
68.8
86.0
17.2
WiSE-FT
49.8
91.0
41.2
85.6
88.6
3.0
53.3
90.8
37.5
58.2
79.1
20.9
DFR (Subsample)
63.9
91.8
27.9
76.9
92.5
15.6
46.7
89.4
42.7
45.0
75.0
30.0
DFR (Upsample)
51.3
92.4
41.1
89.6
91.8
2.2
44.0
86.4
42.4
38.5
77.9
39.4
Contrastive Adapter
83.7
89.4
5.7
90.0
90.7
0.7
62.0
90.9
28.9
60.7
80.9
20.2
Zero-shot (ZS)
25.7
87.3
61.6
62.1
71.9
9.8
4.0
86.6
82.6
72.0
93.2
21.2
CLIP VIT-1/14
Group Prompt ZS
27.4
85.5
58.1
72.4
81.8
9.4
48.0
96.6
48.6
N/A
N/A
N/A
ERM Linear Probe
65.9
97.6
31.7
28.3
94.7
66.4
84.0
98.6
14.6
87.5
96.1
8.6
ERM Adapter
78.4
97.8
19.4
36.7
94.2
57.5
82.8
98.2
15.5
87.0
96,9
9.9
WiSE-FT
65.9
97.6
31.7
80.0
87.4
7.4
84.0
98.6
14.6
87.5
97.0
9.5
DFR (Subsample)
51.9
95.7
43.8
76.3
92.1
15.8
84.0
98.5
14.5
85.5
96.6
11.1
DFR (Upsample)
65.9
96.1
30.2
83.7
91.2
7.5
78.7
97.3
18.6
72.5
93.8
21.3
Contrastive Adapter
86.9
96.2
9.3
84.6
90.4
5,8
80.0
97.5
17.5
82.2
96.1
13.9
Table 4: On the Waterbirds dataset, contrastive adapters consistently improve group robustness across various
Ussion-languaee large pretrained models (CLIP [59], CLOOB (20J) and backbones (ResNets and ViTs).
CLOOB RN-50 CLOOB RN-50x4
CLIP RN-101 CLIP ViT-B/32 CLIP ViT-B/16
Accuracy (%)
WG
Avg Gap
WG
Avg
Gap
WG
Avg Gap
WG
Avg
Gap
WG
Avg
Gap
Zero-shot
41.6 60.4
18.8
24.1
51.1
27
33.6
90.0
56.4
47.0
88.8
41.8
34.0
88.1
54.1
Contrastive Adapter
83.0
86.8
3.8
85.8
88.5
2.7
82.0
86.0 4.0
80.7 84.2
3.5 83.1
90.9
7.8
Representative dataset evaluation. In Table 3 we compare contrastive adapting to other lightweight methods for improving robustness. We evaluate with gralaeimbalanned and balanced training data across spurious confounder, subclass, and data source group shifts, using CLIP ResNet-50 (RN-50) and CLIP ViT- L/14 models. On average, contrastive adapters raise worst-group accuracy by 12.4 and 4.1 pp over the next best methods on CLIP RN-50 and ViT-L/14 models.
Contrastive Adapter Improvement over 7ero-shot
BREEDS Living 17
+56.0
DREEDS Nor
+49.3
civ Icomments-UILDS
+:4.1
Waterbirds
FMOW-WILDS
+28.7
CIFAR-10
+28.3
CIFAR 02
121.6
Celeok
+16.0
GPT Neo 1.30
Amazon-WILDS
+8.5
CLIP ResNet-50
Transfer across architectures. We also study how the prior contrastive adapting improvements transfer to other pretrained models. Table 4 shows contrastive adapters substantially improve group robustness for models such as CLOOB [20]. The method also scales across model sizes, raising worst-group accuracy by 33.7 to 61.7 pp via training adapters with only 0.52% to 1.03% of the model parameters [20, 59].
20
60
D0
100
Worst-group Accuracy (%) Figure 4: Across 9 group robustness benchmarks. contrastive adapting consistently improves worst- group acc. over pretrained zero-shot classification.
5.2 Ablation on sampling strategy and contrastive training objective
To next better understand how contrastive adapting's individual components affect group robustness, we ablate the proposed contrastive objective (Eq. 6) and "hard" sampling strategy, and report worst- group and average accuracies on CLIP RN-50 adapters (Table 5). On three datasets, we find the contrastive loss alone improves robustness more than hard sampling alone. However, on Waterbirds and whhre ERM adapters perform having both components substantially improves robustness (+5.5 to 8.5 pp). Meanwhile, on BREEDS Living- 17 and adapters perform best across all weethods-removing hard sampling improves contrastive adapting performance. On these datasets, the random sampling in ERM may be beneficial (discussed further in App. E.6). Contrastive adapting may thus also benefit from random sampling in these settings.
10.02-10..0--111. ERM
5.3 Measuring efficiency among effective group robustness solutions
While in Section 5. we found contrastive adapters could significantly improve group robustness for foundation models, we now expand on contrastive adapting's ss efficiency. We find that for group robust classification in general, contrastive adapting can achieve state-of-the-art performance despite only training 51% of the usual model parameters. The lightweight nature of contrastive adapting also leads to better data efficiency than existing state-of-the-art approaches.
8

---

Table 5: Contrastive adapter ablation over contrastive loss and sampling. When ERM obtains poor worst-grouu group (WG) accuracy. both contrastive loss and "hard" sampling lead to best robustness. When ERM improves WG acc, over shro (c.f. Table 3, L.e. random sampling helps), no hard sampling also helps contrastive adapters.
Adapter Ablation
Contrastive Loss
Hard
Waterhirds
CalehA
BREEDS
Living-17
CIFAR
10.02
Sampling
WG
Avg.
WG
Avg.
WG
Avg.
WG
Avg.
ERM
60.8 0.9
96.0
36.1 1.4
70.7 942
68.8 9.1
86.0 i 0.5
Hard sample only
<
56.3 4 1.5
0.
84.5
92
58.7
89
80.4 280
0,7
Contrastive only
75.2 1.0
0.
5.9
93.2226
67
012
66.9
82.9 +
Defoult proposed
83.7 0.7
09
9..0
9..7
62.0
1.6
9.3
80.9 6..7
Table 6: On popular Waterbirds and CelebA benchmarks, contrastive adapters achieve near state-of tftte-of-the-ar art worst-group accuracy (WG Acc.) with <1% of the trainable parameters. AAcc. is percentage point gap with prior SoTA. 1st / 2nd best metries bolded / underlined. We report numbers from original works.
Waterbirds
CelebA
Model
Trained Params % Params
Method
WG Acc. (%)
AAcc
WG Acc. (%)
AAcc.
EIIL [16]
78.7
-10.3
83.3
-6.5
CIM [72]
83.6
-5.4
83.6
-6.2
JTT [47]
86.7
-2.3
81.1
-8.7
ResNet-50
25557032
100
RWY [34]
86.1
-2.9
82.9
-6.9
CNC [79]
88.5
-0.5
88.8
-1.0
SSA [52]
89.0
0.0
89.8
0.0
Adapter + CLIP RN-50
263424
1.03
Ours
83.7
-5.3
90.0
0.2
Adapter + CLIP ViT-L/14
197632
0.77
Ours
86.9
-2.1
84.6
-5.2
Robustness comparison to state-of-the-aat methods. Table 6, we evaluate how contrastive adapting with CLIP RN-50 and ViT-L/14 embeddings compares to current state-of-the-ar robustness techniques. We use the popular Waterbirds and CelebA datasets. Existing group robustness methods train ImageNet-pretrainee ResNet-50s. On both datasets, contrastive adapting achieves comparable worst-group accuracy to these methods, despite only training S 11% of their parameters. Notably, contrastive adapting outperforms some methods by up to 5.0 and 10.1 pp for Waterbirds and CelebA, and only falls short of the state-of-the- the- Spread Spurious Attribute (SSA) method by 2.1 pp on Waterbirds. These results suggest contrastive adapters not only effectively improve group robustness for pretrained models, but also enable competitive robust classification in general at a fraction of prior approaches' trainable parameter counts.
Data Efficiency on Waterbirds
Data efficiency evaluation. Beyond model parame- ter count, we also study if the lightweight nature of contrastive adapting transfers to better data efficiency.
Worst-group Acc. (%)
40
70
We compare contrastive adapting to the best perform-
ing SSA on subsampled versions of Waterbirds. To evaluate how well methods maintain group robust-
Contrastive Adapter (CLIP RN-50)
Spread Spurious Attribute (S5A)
Zero shot (CLIP RN-50)
ness with less training data available, we keep group ratios preserved (i.e., 1% of all training samples be- longs to the smallest Waterbirds group [65]).
250
500
730
1900
1258 1500 1750
2000
Training Samples
Figure 5: With less data, contrastive adapters maintain higher group robustness than zero-shot and significantly outperform standard models trained with the state-of-the- SSA method.
Figure 5 shows that contrastive adapting substantially outperforms SSA in lower data regimes. With only 100 and 250 training samples, contrastive adapting outperforms SSA-trained ResNet-50s by 42.6 and 25.9 pp. With 100 training samples, contrastive adapting still achieves 20.8 pp higher worst-group accuracy than zero-shot classification with only 1 training sample in the smallest Waterbirds group. In contrast, SSA's accuracy drops significantly, resulting in 17.6 pp lower worst-group accuracy than zero-shot classification. To connect this fewer data result to our generalization discussion in Section 4.2, with prior methods [19] we estimate the trained adapter Lipschitz constant. We estimate the constant to be 29.3, much lower than that reported for larger networks (e.g., ResNet-50s) [19, 73].
Conclusion
We study the group robustness of popular foundation models. We find their zero-shot classification may not be robust to various group shifts, establish that baseline linear probe and adapter strategies do not reliably improve robustness, and propose a simple adapter strategy to significantly and consistently improve FM robustness without finetuning. This suggests FM embeddings do contain group-relevant information, and we show that we can use FM embeddings to efficiently achieve state-of-the-art robust classification. We recognize the limitations of computational solutions to subgroup performance disparities, and the need to understand FMs in broader socio-technical systems [9].

---

7 Acknowledgements and Funding
We thank Simran Arora, Megan Leszezynski, Kush Bhatia, Maya Varma, Gautam Machiraju, and Laurel Orr for helpful discussions and feedback.
We gratefully acknowledge the funding support of NIH under No. U54EB020405 (Mobilize). NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); ARL under No. 1I-N--1--22025 (Interactive Human-AI AI Teaming); ONR under No. N000141712266 (Unifying Weak Supervision); ONR N00014-20-1 1-2400: Understanding and Apply- ing Non-Euclidean Geometry Machine Learning:
(NEPTUNE); NXP, Xilinx, LETI-EE, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Eric- sson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Facebook, Google, and VMWare. The U.S. Government is authorized to reproduce and distribute reprints for Govermmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
We have no other additional revenues to disclose related to this work.
References
[1] Faruk Ahmed, Yoshua Bengio, Harm van Seijen, and Aaron C. Courville. Systematic generali- sation with group invariant predictions. In ICLR, 2021.
[2] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXiv: 2019.
[3] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Visual prompting: Modifying pixel space to adapt pre-trained models. arXiv preprint 2rXiv:2203.17274, 2022.
[4] Andrei Barbu, David Mayo, Julian Alverio, William Luo,, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for push- ing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Bue Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: pproceedgg//oo wew.eeerinnnnnn.riss..... neurips. Papere pdf.
cc cc/paper/2019/// /tte1eeee9//// aataaa/a44cccaaaeeeccccooaa
[5] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV), pages 456-473, 2018.
[6] Hugo Berg, Siobhan Mackenzie Hall, Yash Bhalgat, Wonsuk Yang, Hannah Rose Kirk, Aleksan- dar Shtedritski, and Max Bain. A prompt array keeps the bias away: Debiasing vasion-language models with adversarial learning. arXiv preprint arXiv: 11933, 2022.
[7] Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tene Tensorflow, March 2021. URL https: org/10. 5281/zenodo 5297715. If you use this software, please cite it using these metadata.
[8] Su Lin Blodgett, Lisa Green, and Brendan O'Connor. Demographic dialectal variation in social media: A case study of american english. arXiv preprint arXiv:1608.0888 2016.
[9] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx. Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint Siti:2208.00258.. 2021.
[10] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nu- anced metries for measuring unintended bias with real data for text classification. CORR, abs/1903.04561. 2019. URL http:/ / /ttp///axvvvv/////////eee//aae.. org/ abs/1903 04561.
10

---

[11] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1 33:1877-1901, 2020.
[12] Joy Buolamwini and Timnit Gebru, Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountubility and Iransparency, pages 77-91. PMLR, 2018.
[13] Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In International Conference on Machine Learning, pages 872-881. PMLR, 2019.
[14] Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[15] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 6172-6180, 2018.
[16] Elliot Creager, Jörn-Honon Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference on Machine Learning, pages 2189-2200. PMLR, 2021,
[17] Yin Cui, Menglin Jia, Tsung-Yr Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2268-9277, 2019.
[18] Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annota- tions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11162-11 111331 2021.
[19] Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George Pappas. Efficient and accurate estimation of lipschitz constants for deep neural networks, Advances in Neural Information Processing Systems, 32, 2019.
[20] Andreas Fürst, Elisabeth Rumetshofer, Viet Tran, Hubert Ramsauer, Fei Tang, Johannes Lehner, David Kreil, Michael Kopp. Günter Klambauer, Angela Bitto-Nemling, et al. Cloob: Modern hopfield networks with infoloob outperform clip. arXiv preprint arXiv:2110.11316. 2021.
[21] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000: 2020.
[22] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:21 10.00544, 2021.
[23] Songwei Ge, Shlok Mishra, Chun-Liang Li, Haohan Wang, and David Jacobs. Robust contrastive learning using negative samples with diminished semantics. Advances in Neural Information Processing Systems, 34, 2021.
[24] Giorgio Giannone, Serhii Havrylov, Jordan Massiah, Emine Yilmaz, and Yunlong Jiao. Just mix once: Mixing samples with implicit group distribution. In NeurIPS 2021 Work- shop on Distribution Shifts, 2021. URL https:/ //wbw..//www../ amazon. ccciencc/publicateee just-mix- mwmmmmixxooemmiie miw.eee--iiinnnaaapee.e. with-withhhe weshh-imlliiittiiie wtttogorroooooooottiieeeeee
[25] Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 110(2)393-416, 2021.
[26] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint urXiv:2007.01434 2020.
[27] Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographies in repeated loss minimization. In International Conference on Machine Learning, pages 1929-1938. PMLR, 2018.

---

[28] Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering. 2169):1263-1284 2009.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[30] D. Hendrycks, Basart, N. Mu, S. Kadavath, Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, J. Steinhardt, and J. Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8320-8329, Los Alamitos, CA, USA, oct 2021. IEEE Computer Society, doi: 10. 10.1109/CCCC49222 2021 1.00233.
[31] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
URL https: ptttps://openeee wtw.:arrrrieeeeeeeoouuuuumaaaaae
[32] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 15271, June 2021.
[33] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruma Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR, 09-15 Jun 2019. URL https: ptttp://prrceeddngg mmlr. hassss/v7//hollle httml.mmlll html.
[34] Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data halancing achieves competitive worst-group-accuragy arXiv preprint arXiv:2 10. 10.15033, 2021.
[35] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. PMLR, 2015.
[36] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904 4916. PMLR, 2021.
[37] Matt Jordan and Alexandros G Dimakis. Exactly computing the local lipschitz constant of relu networks. Advances in Neural Information Processing Systems, 33:7344-7933 7753,,7333330 2020.
[38] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pages 18661-18673, 2020.
[39] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re- re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.00 022937. 2022.
[40] Pang Wei Koh, Shiori Sagawa, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, et al. Wilds: A benchmark of in-the-wild wild distribution shifts. In International Conference on Machine Leurning, pages 5637-5664. PMLR, 2021.
[41] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[42] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202. 10054, 2022.
[43] Yoav Levine, Itay Dalmedigos, Ori Ram, Yoel Zeldes, Daniel Jannai, Dor Muhlgay, Yoni Osin, Opher Lieber, Barak Lenz, Shai Shalev-Shwartz, et al. Standing on the shoulders of giant frozen language models. arXiv preprint rXXv:2204.10019 2022.
12

---

[44] D. Li. Y. Yang. Y. Song, and T. M. Hospedales. Deeper, broader and artier domain generalization. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5543-5551, 2017.
[45] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2
2021.
[46] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208, 2021.
[47] Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pages 6781-6792. PMLR, 2021.
[48] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEFE international conference on computer vision, pages 3730-3738, 2015.
[49] Shangyun Lu, Bradley Nott, Aaron Olson, Alberto Todeschini, Hossein Vahabi, Yair Car- mon, and Ludwig Schmidt. Harder or different? a closer look at distribution shift in dataset reproduction. In ICMI. Workshop on Uncertainty and Robustness in Deep Leurning, 2020.
[50] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-traine pre-training. arXiv preprint 1r 2021.
[51] Junhyun Nam. Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. In Advances in Neural Information Processing Systems, volume 33, pages 20673-20684, 2020.
[52] Junhyun Nam, Jachyung Kim, Jacho Lee, and Jinwoo Shin. Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. arXiv preprint arXiv:2204.0207 2022.
[53] Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally-normalizee margin bounds for neural networks. arXiv preprint urXiv:/707.095:: 2017.
[54] Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly- labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP),, pages 188-197, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 1111655//11/11--011 URL https: / laaclannhology.orrge DD181088.
[55] Luke Oakden-Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré. Hidden strat- ification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of the ACM conference on health, inference, and learning, pages 151-159, 2020.
[56) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.0377 07.00778.... 2018
[57] Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 46-54, Online, October 2020. Association for Computational Linguisties. doi: 10. 10.18655/111200.emmmlp/ 2020. emnlp-demos.7.
URL https::/accaatthlllll
org/
[58] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguis- lics: Main Volume, pages 487-503, Online, April 2021. Association for Computational Lin- guistics. doi: 10. 18653/v1/2021 wwwp://0011eeamaaaaaa eacl-main.39, eacl-main. 39.
URL. https: / //////ac //aaa/aaatlloyoo.ooro hww.hoologg..og/22000 org/2021.
13

---

[59] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 9748-8763 PMIR, 2021.
[60] S-A Rebuffi, H. Bilen, and A. Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, 2017.
[61] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar- 10 2018
classifiers generalize to cifar-10? 10? arXiv preprint /rXiv:/806.000
[62] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Kamalika Chaudhuri and Ruslan Salakhutdinov, edi- tors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5389-5400. PMLR, 09-15 Jun 2019. URL https://proceeddingg wtt.:ea......pp..pppepeeere
mlr wwwppe 1tt99//eechteee /rr//htechtttaeee html.
[63] J. Robinson, Ching- Yao Chuang. S. Sra, and S. Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2021.
[64] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Kostas Danilidiss, Petros Maragos, and Nikos Paragios, editors, Computer Vision - ECCV 2010, pages 213-226, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg. ISBN 978-3-642-15561-11
[65] Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations, 2019.
[66] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Perey Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning, pages 8346-8356. PMLR, 2020.
[67] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry, Breeds: Benchmarks for subpopu- lation shift. arXiv preprint htt::20008048555 2020.
[68] Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
[69] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. arXiv preprint arXi::2112.04482, 2021.
[70] Sahil Singla, Mazda Moayeri, and Soheil Feizi, Core risk minimization using salient imagenet. arXiv preprint arXiv:2203. 15566, 2022.
[71] Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse -arsengrained classification problems. In Advances in Neural Information Processing Systems, volume 33, pages 19339-19352, 2020.
[72] Sacid Asgari Taghanaki, Kristy Choi, Amir Khasahmadi, and Anirudh Goyal. Robust represen- tation learning via perceptual similarity metrics. arXiv preprint arXiv: (00:2/00606620 2021.
[73] Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances Neural Information Processing Systems, 31, 2018
[74] Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global represen- tations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506-10518, 2019.
[75] Welinder, Branson, T. Mita, C. Wah, Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report TN2-10-000,-00,, California Institute of Technology, 2010.
14

---

[76] Mitchell Wortsman, Gabriel Ilharco, Mike Li. Jong Wook Kim, Hannanch Hajishirzi, Ali Farhadi, Hongseok Namkoong. and Ludwig Schmidt. Robust fine-tuning of zero-shot models. arXiv preprini arXiv:2109.01903. 2021.
[77] Yuan Yao, Ao Zhang. Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:2199.1197, 2021.
[78] Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generaliz- ability of deep learning. arXiv preprint arXiv:
2017.
[79] Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Ré. Conrect-n-contrast: contrastive approach for improving robustness to spurious correlations. arXiv preprint 2rx0v:2203.01517, 2022.
[80] Renrui Zhang, Rongyao Fang. Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-fres clip-adapter for better laigunna modeling. arXiv preprint arXiv:2111.03933 2021.
[81] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747, 2020.
[82] Bingehen Zhao, Shaozuo Yu, Wufei Ma, Mingxin Yu, Shenxiao Mei, Angtian Wang, Ju He, Alan L.. Yuille, and Adam Kortylewski. Robin : A benchmark for robustness to individual nuisancesin real-world out-of-distribution shifts. CoRR, abs/2111.14311, 2021. URL https: //arxiv. htwpxiv.orrgbb//111111 ab1/ 14341.
[83] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. arXiv preprint arXiv:2109.01 134, 2021.
[84] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for Aision-language models. arXiv preprint arXiv:2203.055577 2022.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] We do not include theoretical results
(b) Did you include complete proofs of all theoretical results? [N/A] Not Applicable
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See attached
zip in supplementary.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] For space we defer these to appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] For space defer to the appendix.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Appendix.
(b) Did you mention the license of the assets? [Yes] See Appendix.
(c) Did you include any assets either in the supplemental material URL? [N/A]
We only use existing assets, which we discuss the appendix.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No] We use existing popular robustness benchmarks for all our datasets. While some involve people (CelebA, CivilComments-WIL
Amazon-
WILDS),, we were not able to locate information on how this data was collected regarding subject consent. However, we do discuss the licenses and agreements set by the original curators of these popular benchmarks.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Appendix.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] We do not crowdsource or conduct research with human subjects. (b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] We do not crowdsource conduct research with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] We do not crowdsource conduct research with human subjects.
16


Reviews:
[{"review_id": "YqfErGFRRJb", "reviewer": "Reviewer_TAXS", "summary": "This paper studies the group robustness of foundation models, and shows that large scale pretrained foundation models can still have a bad robustness on some group shifts. Then the paper proposed a contrastive adapting method that take advantage of supervised contrastive learning to train the adapter to improve the alignment and cosine similarity metrics. Experimental results show that the proposed contrastive adapter is successful in improving the group robustness of the foundation models.\n\n", "questions": "\nQ1: In fig 3, we can see that only on CelebA that the learned adapter has a worse group robustness, but as in tab 2, the group shift in CelebA and Waterbirds are the same (Confounder), why would the robustness of learned adapter on waterbirds be better than zero shot while on CelebA it's the other way around? Is something other than the group shift influencing the robustness of the model?\n\nQ2: It is known that supervised contrastive learning (SupCon) outperforms regular supervised training with CE loss. It would be good if the paper could provide an analysis of how does SupCon helps the group robustness of foundation models, could it be that the performance gain is because SupCon improves the performance of the model overall (including improvement in IID performance), and the improvement in group robustness is a side effect? \n\nQ3: If I understood correctly, the proposed solution only tunes the adapter on the feature from the image encoder of the CLIP model, I am wondering how this method compare to the visual-language adapters like in [R1] or [R2].\n\nQ4: The proposed method is quite similar to the ones proposed in [57], both uses supervised contrastive to help the group robustness, I would like to see a comparison between the proposed method in this paper and the one in [57].\n\n\n[R1] Learning to Prompt for Vision-Language Models, arXiv 2021.\n\n[R2] Conditional Prompt Learning for Vision-Language Models, CVPR 2022.\n", "limitations": "\n1. This work proposed a way to improve the foundation model's performance wrt to group robustness, however, the method still needs to run inference on the entire foundation model which can be computational expensive.\n", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nS1: The problem this paper tries to tackle is practically important, as the foundation model is used in more applications, the robustness of these models is important to study.\n\nS2: The performance of proposed method seems to be improving over the baseline by a large margin.\n\nW1: Some of the experimental results are not very clearly explained, see Questions.\n\nW2: The supplementary material is not properly anonymized, the `.git` folder in the code submission contains a commit message that have the author's name on it.\n\n\nNote that most of this review is completed before I unwillingly find out the identity of the author.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SQUPxsd2F0M", "reviewer": "Reviewer_2hnT", "summary": "The paper studies the classification group robustness of large-scale image-text pre-trained models. The authors identified that, under the prompt-based zero-shot classification protocol, the worse-group classification accuracy is considerably lower than the average group classification accuracy. The authors proposed to use a metric-learning loss (i.e., contrastive loss), on top of a MLP adapter from previous works, for finetuning on target datasets. Experimental results on a few benchmarks show improvement over baselines on worse-category classification accuracy.", "questions": "In the author response phase I encourage the authors to respond to the suggested baselines in my comments, with evidence of why they might be inferior or unnecessary. ", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper studies a few interesting questions regarding these large-scale pre-trained models: 1) how to adapt these models for a target dataset, and 2) how do these models actually transfer across datasets and groups. These questions are important and they need to be addressed because the practice, and the hope of the community, is to rely more and more on the pre-trained models. The paper shows that neither a naive version of the zero-shot, nor linear probing on the target dataset, is the optimal choice. Benchmarked on standard datasets, The proposed method is effective across model architectures.\n\nOn the other hand, I am curious about a few baselines that I feel essential in evaluating the pre-trained models but seem to be missing from the draft. The first one is more prompt-tuning for zero-shot classification. Although impressed by the CLIP zero-shot classification accuracy, one must recognize the necessity of using properly tuned prompts or hints, as pointed out by the authors of the CLIP paper. It can be argued for the downstream evaluation/benchmarks in this paper such as WaterBirds, we can use prompts like \"A land bird, on water background\" to give more hints to the base model. I didn't find a discussion on how the zero-shot baseline is constructed or if meticulous tuning was conducted, and I believe this procedure is necessary for reliable conclusions.\n\nSecondly, the metric-learning term (i.e., the contrastive loss) that is at the center of the proposed method looks heavily inspired by the hard-negative sampling strategy (see Line 235), acknowledged by the authors as well in Line 227. A potential ablation study on decoupling this hard-negative sampling procedure from the more complicated contrastive loss, in my opinion, is essential for understanding if the sampling procedure and/or the contrastive term is (more) important in the proposed method. If it is the sampling procedure that works, we don't necessarily need the embedding-based loss and it would be simple.\n\nThirdly, I can imagine a simple two-stage baseline that has the same essence of negative-sampling procedure. In the first stage, one can train a classifier using the naive sampling strategy. In the second stage, a new classifier can focus more on the misclassified samples, like a cascade classification framework. With a large-scale pre-trained model such as the CLIP model, the first stage can be its native zero-shot classifier. If interpreted this way, the sampling strategy proposed by the authors is is akin to the two-stage framework. Upon searching, a paper named *Just Train Twice: Improving Group Robustness without Training Group Information* does this for group robustness and it seems to be a valid baseline that should be studied in the paper.\n\nLastly, I'd like to raise my concern for using the term \"foundation model\" in an official and scientific research conference, despite it didn't play any role in my judgement for reviewing the submission. The practice of grouping a set of large-scale pre-trained models, large-scale being the amount of data, and/or the number of parameters in the model, and terming them \"foundation\" is subjective and without scientific background. Personally, I work on scaling models as well, and I'm a firm believer in the potential of these big models for the future of AI and ML research. It is precisely that the first-hand experience that makes me realize the flaws in this term. These models aren't foundational yet. They make silly mistakes all the time. They don't do grounding well. And they are far from reaching the ideal performance that we hope to get eventually, as evident in the experimental results of the submission as well. \"Foundation model\" is a hope, an inspiration, for now, not a reality. And a research paper is, by its nature, about the presence. Therefore in my humble opinion discarding the term will make the paper a stronger submission.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BhPSVcV72lY", "reviewer": "Reviewer_8jHt", "summary": "This work proposes a new method to improve group (subpopulation) robustness in foundational models like transformers. The problem of group robustness (in terms of worst-group performance) is particularly challenging when some groups are underrepresented in the training data. Classical approaches use under/over-sampling, reweighting, and two-phase training to deal with this issue. The authors propose to use contrastive adapters without retraining from scratch. The idea is to contrast sample embeddings from the same class to improve the embedding structure. The paper provides experimental evidence that such an approach is effective when dealing with different kinds of shifts in the data, obtaining good performance.", "questions": "* Regarding the contrastive adapter name: isn't the (standard) adapter contrastive too if we follow your argument in 4.1 and Eq. 5? \n\n* In Table 2) and Table 3) is the (standard) adapter trained by ERM or using the InfoNCE loss in Eq.5?\n    * If you use ERM, I would like to see the adapter performance trained explicitly with Eq 5.\n    * Additionally, it would be interesting to see the adapter performance training with Eq 5 only, Eq 6 only, and both equations.\n\n* How large is P for Eq.6? Did you try to use different values for P? \n\n* How do you mix CIFAR10 and CIFAR02?", "limitations": "* The method seems to work well only with a few classes (binary classification), and performance deteriorates for different shifts and more complex datasets like CIFAR10.02. In particular, increasing the number of classes, the standard adapter and the linear probe work better than the contrastive adapter (BREEDS and CIFAR in Table 3). This is not the behavior I would expect if the argument in 4.1 is correct.\n\n* Eq 5 and Eq 6 are both contrastive losses. You link the ERM adapter with the InfoNCE but did not use Eq 5 to train the standard adapter (as far as I understand). So this connection is not backed by experiments.\n\n* Looking at Table 2, I agree that all the baselines fail to improve robustness consistently. But so your method in Table 3.\n\n* I would compare your approach with JM1 (https://openreview.net/forum?id=HI2ilxFli0W): both methods have access to fine-grained label structure. The \"retraining\" in JM1 starts from a pre-trained foundational model.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths\n\n* The paper is well written and easy to follow with a clear goal (improving group robustness in foundational models).\n* The problem tackled is relevant for the use and deployment of foundational models.\n* The idea of \"contrastive adapter\" is novel.\n* Method easy to implement and to apply to a large variety of scenarios.\n\nweaknesses\n\n* The idea of increasing the adapter contrasting not only class and sample but also samples from the same class is incremental.\n* Results do not clearly show that contrastive adapters work better (in terms of robustness) with all the considered distribution shifts in Table 3. In particular, standard adapters seem to work better when increasing the dataset complexity (number of classes in CIFAR10).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept