Zero-Shot Robustification of Zero-Shot Models
Authors: Dyah Adila
Changho Shin
Linrong Cai
Frederic Sala
Venue: ICLR 2024 poster (2024)
Keywords: zero-shot classification
spurious correlations
invariant embedding
foundation model
language model
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We make zero-shot models more robust with zero-shot insights from language models---no fine-tuning or labeled data required.

Abstract: Zero-shot inference is a powerful paradigm that enables the use of large pretrained models for downstream classification tasks without further training. However, these models are vulnerable to inherited biases that can impact their performance. The traditional solution is fine-tuning, but this undermines the key advantage of pretrained models, which is their ability to be used out-of-the-box. We propose RoboShot, a method that improves the robustness of pretrained model embeddings in a fully zero-shot fashion. First, we use language models (LMs) to obtain useful insights from task descriptions. These insights are embedded and used to remove harmful and boost useful components in embeddings---without any supervision. Theoretically, we provide a simple and tractable model for biases in zero-shot embeddings and give a result characterizing under what conditions our approach can boost performance. Empirically, we evaluate RoboShot on nine image and NLP classification tasks and show an average improvement of 15.98% over several zero-shot baselines. Additionally, we demonstrate that RoboShot is compatible with a variety of pretrained and language models and propose a way to further boost performance with a zero-shot adaptation variant.

Published as a conference paper at ICLR 2024
ZERO-SHOT ROBUSTIFICATION OF ZERO-SHOT MOD- ELS
Dyah Adila", Changho Shin', Linrong Cai, Frederic Sala Department of Computer Science University of Wisconsin-Mudison
fadila, cshin23, 1 1cai54 54, fredsala @wwiewwwsc.......
ABSTRACT
Zero-shot inference is powerful paradigm that enables the use of large pretrained models for downstream classification tasks without further training. However, these models are vulnerable to inherited biases that can impact their performance. The traditional solution is fine-tuning, but this undermines the key advantage of pretrained models, which is their ability to be used out-of-the-box. We propose ROBOSHOT,, a method that improves the robustness of pretrained model embed- dings in a fully zero-shot fashion. First, we use language models (LMs) to obtain useful insights from task descriptions. These insights are embedded and used to remove harmful and boost useful components in embeddings wwthout any su- pervision. Theoretically, we provide a simple and tractable model for biases zero-shot embeddings and give a result characterizing under what conditions approach can boost performance. Empirically, we evaluate ROBOSHOT on nine image and NLP classification tasks and show an average improvement of 15.98% on worst group accuracy, with trivial decrease in overall accuracy over several zero-shot baselines. Additionally, we demonstrate that ROBOSHOT is compatible with variety of pretrained and language models and propose a way to further boost performance with a zero-shot adaptation variant
INTRODUCTION
Zero-shot prediction is among the most exciting paradigms in machine learning. Zero-shot models obviate the need for data collection and training loops by simply asking for prediction on any set of classes. Unfortunately, such models inherit biases or undesirable correlations from their large-scale training data (Dixon et al. 2018; Torralba & Efros 2011). In a now-canonical example (Koh al. 2021) they often associate waterbirds with water background, This behavior leads to decreased performance, often exacerbated on rare data slices that break in-distribution correlations.
A growing body of literature (Yang et al. 2023 Goyal et al., 2022 Zhang & Ré 2022 seeks to improve robustness in zero-shot models. While promising, these works require labeled data to train or fine-tune models, and so do not tackle the zero-shot setting. A parallel line of research seeking to debias word embeddings (Aboagye et al.; Bolukbasi et al. [0166] Dev & Phillips 2019 Lauscher eet al., (2200) often sidesteps the need for labeled data. Unfortunately. these works often require domain expertise and painstaking manual specification in order to identify particular concepts that embeddings must be invariant to. As a result, out-of-the-bol word embedding debiasing methods also cannot be applied to zero-shot robustification.
Can we robustify zero-shot models without (i) labeled data, (ii) training or fine-tuning, or (iii) manual identification? Surprisingly, despite this seemingly impoverished setting, often possible to do so. Our key observation is that language models contain actionable insights that can be exploited to improve themselves or other models. These insights are noisy but cheaply available scale and can be easily translated into means of refinement for zero-shot representations. These refinements improve performance, particularly on underperforming slices, at nearly no cost.
"These authors contributed cqually to this work
"Code can be found in https://gituh..oo//ppoeeeoe haw.rrooooceealaaaoooohoooo ab/roboshot

---

Published as a conference paper at ICLR 2024
"aadiddee" embedalin
descrition: waaorro
Auell any
giggtts
foundation mode
boak shaps Spurtous bachgrouutd
"Landbize""
Image emblddaiing
Inigghts embodding ng
zeroohat e-
Landbrdwwh mizK
budggound
(NNatoroooo" "landbtre")
Labels
tocaground
"naceebizs"
("aatertiirs", "landbire"))
projectd enteiciing
Labes
Origiaal embedding
"vetuzbiadd embedding
Figure I:
Left: vanilla zero-shot classification. Right: ROBOSHOT projects original embeddings to space with reduced spurious components and increased useful components
We propose ROBOSHOT, a system that robustifies zero-shot models via language model-based in- sights without labels, training, or manual specification. Using just the task description, ROBOSHOT obtains positive and negative insights from a language model (potentially the model to be improved itself). uses embeddings of these noisy insights to recover harmful, beneficial, and benign sub- spaces of zero-shot latent representation spaces. Representations are then modified neutralize and emphasize their harmful and beneficial components, respectively.
Theoretically, introduce simple and tractable model capture and quantify failures in zero- shot models. We provide result that characterizes the quantity and quality of insights that must be obtained function of the severity of harmful correlations. Empirically, ROBOSHOT achieves 15. 55.98% improvement across nine image and NLP datasets while offering sufficient versatility to apply to diverse variety of base models. Most excitingly, in certain cases, it reaches comparable or greater improvements even when compared to fine-tuned models that rely on labeled data. In summary, our contributions include:
1. A simple theoretical model describing zero-shot failures along with theoretical analysis of our approach that characterizes the amount of information required for obtaining im- provements as a function of the most harmful unwanted correlation.
2. ROBOSHOT, algorithm that implements tion models and uses them to improve zero-shot representations,
core idea. It extracts insights from founda-
3. Extensive experimental evidence on zero-shot language and multimodal models, showing improved worst-group accuracy of 15.98% across nine image and NLP datasets,
4. A technique to add further robustness by training an adapter without any labels requiring only minimal amounts of validation data.
2 RELATED WORK
We describe related work in zero-shot model robustness and debiasing embeddings. We provide more exhaustive list of related work in Appendix B including papers studying guiding multi-modal models using language and using LMs as prior information.
Zero-shot inference robustness. Improving model robustness to unwanted correlations is heavily studied area (Sagawa et al., 2019; Arjovsky et al.) 2019) Krueger et al.) [02T] Kirichenko et al.) 2022 Liu et al. 2021 Lee al.) 2022). Some methods require training from scratch and are less practical when applied to large pretrained architectures. Existing approaches to improve robustness poxt-pretraining predominantly focus on fine-tuning. (Yang et al. 2023) detects spurious attribute descriptions and fine-tunes using these descriptions. A specialized contrastive loss is used to fine- tune a pretrained architecture in (Goyal et al. 2022) and to train an adapter on the frozen embed- dings in (Zhang & Ré 2022). While promising, fine-tuning recreates traditional machine learning pipelines (c.g., labeling, training, etc.), which sacrifices some of the promise of zero-shot models. In contrast, our goal is to avoid any training and any use of labeled data. Concurrent work seeks to
2

---

Published as a conference paper at ICLR 2024
original embedding
reject w
increase
both
0.2
X
X
X
X
0.3
0.5
-0.3
-0.3
x cless c²
cless c² YO X
Y1
Figure 2: Visualization on CelebA (200 random samples). L-R: (i) original embedding (ii) harmful concept removal (iii) helpful concept addition (iv) full ROBOSHOT. YO and Y1 are class labels
robustify CLIP zero-shot predictions against spurious features by debiasing the classifier (i.e., the labels embedding) against harmful concepts (Chuang et al. 2023) but does so via manual spec- ification. In contrast, our work amplifies helpful concepts and automates the process of obtaining debiasing vectors.
Debiasing embeddings. A parallel line of work seeks debias embeddings (Aboagye et al.) (Bolukbasi et al., 2016) (Dev & Phillips 2019) (Lauscher et al. 2020) and multimodal embeddings (Wang et al.J [022] Berg et al.J 2022; Wang al., 2021) by removing subspaces that contain un- wanted concepts. We use similar procedure
building block. However, these methods either target specific fixed concepts (such as, for example, gender in fairness contexts) or rely on concept annotations, which limits their applicability across wide range of tasks. In contrast, method au- tomates getting both beneficial and unwanted concepts solely from the task descriptions. Moreover, goal is simply to add robustness low zero-cost; do not seek produce fully-invariant representations as is often desired for word embeddings.
ROBOSHOT: ROBUSTIFYING ZERO-SHOT MODELS
We are ready to provide our setup and describe the ROBOSHOT algorithm. As mentioned before, we use embedding debiasing principles as building blocks. For our purpose, we utilize concepts ob- tained from language models and get their embeddings build the beneficial and unwanted concept subspaces to work with. We call these embeddings the insight representations.
3.1 MODELING AND SETUP
Suppose that the zero-shot model's s latent space contains an (unknown) concept ser; similar notions have been studied frequently in the literature (Dalvi et al. 2022). For simplicity, assume that this concept set is given by the orthonormal vectors {- }. The model's encoder produces, for particular input, representation x that is a mixture of concepts Σ, 7/24, where 7 O are weights.
We work with the following theoretical model for zero-shot classification. For simplicity, we assume that there two classes. It is straightforward extend the analysis below to multi-class. We take Σ; 04 to be the embedding of a datapoint, while e Σ, 8i,02 is the embedding of the first class and c² = E, o; that the second. We assume that we have access to m answers v v'..... n from set of queries to the language model; we describe how these queries used practically further on. These are given by od =, Ti for j & m. We call these insight representations.
In the standard approach, the prediction is made by 9 aE,,) ((,,))) so that we predict the class that has the higher inner product with datapoint's is embedding. Next, assume that each input representation can be represented by partitioning the mixture components into three groups.
1((2, a, 31,0%))
S-R
S+R-B Σ
harmful
+
helpful
benign a
(1)
5+
b R+1
In other words, representations comprise of mixture of embeddings pertaining to harmful, helpful, and benign or neutral concepts- this holds for class and insight representations. In Appendix F5 empirically show that this assumption holds in real scenarios.
3

---

Published as a conference paper at ICLR 2024
We illustrate how harmful correlations produce errors slices of data through standard task setting, Waterbirds (Koh et al. 202T). Here the goal is to classify landbir rand versus waterbirds rds, and the background (land water) is spurious. Suppose that we have these terms relate to concepts such that Satter = - annd and Eusterbird -aaddiird
Example.
Consider datapoint coming from a data slice rarely encountered in the training set, for instance, an image of landbird over wa- ter. Its embedding might be = 0.72water + 0.321enabird- We may also have that coaterbied
Algorithm 1: ROBOSHOT
Parameters: Input embedding x. class embeddings c², ct, harmful insight representations vt helpful insight representations u u'.... utt
.... U
2: for j {1,2..... ...., S) do
0.4 + 0.62uaterbire and gaandbred_.... = 0.42 and a0 +0.62amddbidd- Then, d''caaterbir = 0.1 > x² clandbird = -0.1, which gives us waterbird prediction, and is incorrect. This is caused by the presence of harmful components in both the class embedding (caused by seeing too many images with water described as waterbirds) and the datapoint embedding (where the water background appears). Our goal to remove harmful components (the 2, and boost helpful components (the 2,'s) without labels or training. Our approach follows.
3: Remove harmful insight u²: set
x (I, v /²)/((² , 4: Renormalize
x ||=|||
5: end for
6: for k E {1, 2........ R) do
7: Amplify helpful insight up: set
I x + (Ir, w² > // (u u* )))u
8: end for
-= 1(=Tc" x² 10: Returns: Robustified zero-shot prediction 9
3.2 ROBOSHOT: ROBUSTIFYING ZERO-SHOT INFERENCE
We describe ROBOSHOT in Algorithm It uses representations of insights from language models to shape input and class embeddings to remove harmful components and boost helpful ones. Figure 2 is helpful in understanding the intuition behind these procedures. Note how unhelpful directions are neutralized while perpendicular directions are boosted.
Obtaining insight representations from LMs. The first question is how to obtain insight repre- sentations a zero-shot way- we use textual descriptions of harmful and helpful concepts by query- ing language models using only the task description. For example, in the Waterbirds dataset, we use the prompt "What are the biased/spurious differences between waterbirds and landbirds?". We list the details of the prompts used in Appendix D.2 Let s s² be the text insights obtained from the answer (e.g., { ('water background, 'land background")> We obtain a spurious insight representation by taking the difference of their embedding
(g(s') g(s²)) () g(s²))] where y is the text encoder of our model. In addition to attempting to discover harmful correlations. we seek to discover helpful components in order to boost their magnitudes past the harmful ones. We obtain insight representations using language models. For example, ask "What are the true characteristics of waterbirds and landbirds?" and obtain c.g., ('short beak,' 'long beak'), The remainder of the procedure is identical to the case of harmful components.
Prompting a language model is typically inexpensive, which will enable obtaining multiple insight vectors 0 ..... "m From these, obtain an orthogonal basis v! ...., n separately for harmful and helpful components using standard matrix decomposition methods. Thus we have access to recovered subspaces spanned by such components.
Removing and boosting components. ROBOSHOT applies simple vector rejection to mitigate harmful components (lines 2-5 of Algorithm [1 and boosts helpful ones (lines 6-9). To see the impact of doing so, we return to our earlier example. Suppose that we have a single harmful insight harrmful 0.9 + 0.121andbied and single helpful insight o helpfal 0.1 + 0.9 andbirdd Note that even these insights can be imperfect: they do not uniquely identify what are harmful or helpful concepts, they have non-zero weights other components.
4

---

Published as a conference paper at ICLR 2024
From removing the harmful component (ignoring normalization for ease of calculation), obtain (I, 2 hhrmffll )
a <- (phurmful, yharmfuly y harmful -0.02442mater + 0.219551andbirrd Then, we already we have
-0.1415 x² clendbird 0.1415, thus the correct class is obtained. From a single insight we have neutralized a harmful correlation and corrected what had been an error. Adding in the helpful component further helps. Using vector addition equation in Algorithm || [line This further increases our margin. Note that it is not necessary to fully neutralize (i.e., to be fully invariant to) spurious or harmful components in our embeddings. The only goal is to ensure, as much as possible, that their magnitudes are reduced when compared to helpful components (and to benign components). In Section 4 model for the magnitudes of such components and characterize the conditions under which it will be possible to correct zero-shot errors.
that x.' catterbiid
7, we obtain -0.0006 + 0.4337
we provide a theoretical
3.3 LABEL-FREE ADAPTATION (LFA)
Additionally, we explore the limit of neutralizing harmful and boost- ing helpful insights in the embed- ding space via an alternative adap- tation approach when users seek to maximize robustness and have access to unlabeled training set and small labeled validation set (with as 100 samples). We learn a fea- ture space parameterized by projec- tion matrix II Rd + Rdd, where is embedding dimension. We opti- mize II so it projects x to space with minimum dot product with harmful insights (IIz, v), and maximum with the helpful ones (Ilæ, u). More formally, Il arg minn E, C.LFAAIla, u, v)]. The loss given by
Algorithm 2: Label-free adaptation
I: Parameters: Input embedding matrix X, ROBOSHOT projected embedding matrix Xproj. spurious insight representations useful insights representations u,
class embeddings c,,ct, epoch number e
few
2: Initialize II =X Xt
3: for epoch in 1,2...... do
4: 11:+1 arg minn Ex [I.F (Ia,, u, v)]
5: end for
6: 0 = [(II=T<0 < IIr"c'}
7: Returns: Robustified zero-shot prediction 9
R
CLFA 0) = S Hx, vd)
[Lr )
R
where S and R the number of harmful and helpful insights. We observed that the best results are achieved by initializing IT as the ROBOSHOT projection matrix, Un Xproy where X = [21 Ø2 ... xN is the embedding matrix, X its Moore-Penrose pseudo-inverse, and X Xproj is the ROBOSHOT projection matrix. Algorithm details LFA algorithm. We draw inspiration from (Chen et al., (023) where the authors learn an orthogonal feature space from a source domain dataset and adapt to a target domain. In contrast to this approach, our focus is on learning the feature space without any training labels and using insights as the only form of supervision.
4 THEORETICAL ANALYSIS
Next, we provide an analysis that characterizes under what conditions ROBOSHOT can correct zero- shot errors. First, we consider the following error model on the weights the representations. For all benign representations, we assume ab, 36, 7b (O, of heenggg). That is, the magnitudes of benign components are drawn from a Gaussian distribution. The value of a Denign is function of the amount of data and the training procedure for the zero-shot model. Appendix empiriialy shows that in real scenarios, benign components can be canceled out, indicating that this assumption often holds.
Next, assume that the insight embedding v² -Et 2 (where 1 S 5) satisfies the property that for + 8, Tiks (0, of misghh) while Ye.s is constant. In other words, the vectors v', v'...., S spanning the harmful component subspace are well-aligned with genuinely harmful con- cepts, but are also affected by noise. Similarly, we assume that helpful insights v" = Σ 2; (where S - 1 S R) satisfy the same property. We seek to understand the interplay between
5

---

Published as a conference paper at ICLR 2024
this noise, benign noise, and the coefficients of the other vectors (i.e., helpful components). Let the result of ROBOSHOT with insight representations v' S be
S+R
S+R+B
4;2
Y=
We first provide bound on A., the targeted harmful concept coefficient after applying ROBOSHOT. Theorem 4.1. Under the noise model described above, the post- ROss-ROBSSHOT coefficient for harmful
concept s (1 S S S) satisfies
(k 1) insight
insighhht
EA, S
Σ 1=L,s5
72.0
where k is the number of concepts (k = R+ B).
The proof is included in Appendix C3 The theorem illustrates how and when the rejection ponent of ROBOSHOT works-it scales down harmful coefficients at a rate inversely proportional the harmful coefficients of the insight embeddings. As would hope, when insight embed- dings have larger coefficients for harmful vectors (i.c., more precise in specifying non-usefie unn-us terms), ROBOSHOT yields better outcomes. In addition, we observe that the harmful coefficients decrease the
when the insight embeddings have less noise. In fact, we have that iinightt A, case of perfectly identifying harmful, helpful concepts.
Next, we provide a bound on Ar, the Rost-RoBOSHOT coefficient of a targeted helpful concept. Theorem 4.2. With an additional assumption as S 0 (1 I I S) under the described noise model, the post-Roboo Bost-RoBOSHOT coefficient for helpful concept (S 15+15r S R) satisfies
Pr.
EA, 2
11
Cr.
Fp.r + (A:
1)0 issght
Refer to Appendix C.3 for the proof. Theorem 4.2 implies the helpful coefficients are scaled up a rate inversely proportional to the noise rate dinsight- When concepts are perfectly identified, i.e. Finsight = 0, the coefficient o, is doubled, yielding more emphasis the concept 2, desired.
EXPERIMENTAL RESULTS
This section evaluates the following claims:
Improving multimodal models (Section 5.1) ROBOSHOT improves zero-shot classification ro- bustness of various multimodal models, outperforming prompting techniques that include spurious insight descriptions (which we do not have access to) in the label prompts.
Improving language models (Section ROBOSHOT improves zero-shot robustness using LM embeddings for text zero-shot classification, outperforming direct prompting to get predictions.
Label-free adaptation (Section 5.3): LFA (Algorithm
2 can further improve performance with
only small labeled set for validation (100 samples).
Extracting concepts from LM with varying capacities (Section 5.4): ROBOSHOT extract insights from language models with varying capacities. Improvements persist with weaker LMs.
Ablations (Section 5.5): ROBOSHOT benefits from both removing harmful and boosting helpful representations (line 3 and line in ROBOSHOT Algorithm
Metrics. We use three metrics: average accuracy (AVG), worst-group accuracy % (WG), and the gap between the two (Gap). While model that relies harmful correlations may achieve high AVG when such correlations are present in the majority of the test data, may fail in settings where the correlation is absent. A robust model should have high AVG and WG, with small gap between them.
Baselines. We compare against the following sets of baselines:

---

Published as a conference paper at ICLR 2024
Table 1: Main results. Best WG and Gap performance bolded, second best underlined.
ZS
GroupPrompt ZS
ROBOSHOT
Dataset
Model
AVG
WG(1)
Gap(L)
AVG
WG(1)
Gap(L)
AVG
WG(1)
Gap(L)
CLIP (ViT-B-32)
80.7
27.9
52.8
81.6
43.5
38.1
82.0
54.4
28.6
Waterbirds
CLIP (ViT-I--14)
88.7
27.3
61.4
70.7
10.4
60.3
79.9
45.2
34.7
ALIGN
72.0
50.3
21.7
72.5
5.8
66.7
50.9
41.0
9.9
ARCLIP
90.1
35.8
54.3
82.4
29.4
53.0
78.5
54.8
23.7
CLIP (ViT-B-32)
80.1
72.7
7.4
80.4
74.9
5.5
84.8
80.5
4.3
CelebA
CLIP (VIT-L-14)
80.6
74.3
6.3
77.9
68.9
9.0
85.5
82.6
2.9
ALIGN
81.8
77.2
4.6
78.3
67.4
10.9
86.3
83.4
2.9
AIICLIP
82.3
79.7
2.6
82.3
79.0
3.3
86.0
77.2
8.8
CLIP (VIT-B-32)
96.7
82.1
14.6
97.9
82.7
15.2
97.0
86.3
10.7
PACS
CLIP (ViT-L-14)
98.1
79.8
18.3
86.6
11.6
83.9
14.2
ALIGN
95.8
77.1
18.7
96.5
65.0
31.5
95.0
73.8
21.2
ARCLIP
98.5
82.6
15.9
98.6
85.4
13.2
98.7
89.5
9.2
CLIP (ViT-B-32)
75.6
20.5
55.1
76.5
33.0
43.5
VLCS
CLIP (VIT-L-14)
72.6
4.20
68.4
71.1
12.6
58.5
ALIGN
78.8
33.0
45.8
77.6
39.8
37.8
AIICLIP
78.3
24.7
53.6
78.9
25.0
53.9
CXR14
BiomedCLIP
55.3
28.9
26.4
56.2
41.6
14.6
1. Multimodal baselines: (i) vanilla zero-shot classification (ZS) and (ii) ZS with group informa-
tion (Group Prompt ZS). We use a variety of models: CLIP (VTT-B B-33 and ViT-L-14) (Radford 2021 I, ALIGN (Jia et al. 202T], and AIICLIP (Chen al. 2022 Group Prompt ZS assumes access to spurious or harmful insight annotations and includes them in the label prompt. For instance, the label prompts for waterbirds dataset become [waterbi rd with water background, waterbire rd with land background, landbird with water background, landbird with land background], We only report Group Prompt ZS results on datasets where spurious insight annotations are available.
2. Language model baselines: (i) zero-shot classification using language model embeddings, namely BERT (Reimers Gurevych 2019) and Ada (Neelakantan al. 2022) (ZS), (ii) di- rect prompting to LMs, namely BART-MNLI (Lewis et al.) [2019] Williams et al. 2018 and ChatGPT (Ziegler et al.| (2019) (Direct prompting). We also compare with calibration methods for zero-shot text classification (Holzmaan et al. 202 results can be found in Appendix F.1
5.1 IMPROVING MULTIMODAL MODELS
Setup. We experimented on five binary and multi-class datasets with spurious correlations and distribution shifts: Waterbirds (Sagawa et al. 2019). CelebA (Liu et al. 2015), CXR14 (Wang ct al. 2017). PACS (Li et al., 2017). and VLCS (Fang et al.] (013). Dataset details are provided in Appendix D.I For CXR14, on biomedical data. All experiments are conducted using frozen pretrained models embeddings. We evaluate on four model variants: CLIP (ViT-B-32 and ViT-L-14), ALIGN, and AIICLIP.
BiomedCL.IP (Zhang al. [2023] variant of CLIP finetuned
Results. Table D shows that ROBOSHOT significantly improves the worst group performance (WG) and maintains (and sometimes also improves) the overall average (AVG) without any auxiliary information (in contrast to Group Prompt, which requires access to spurious insight annotation). Improved robustness nearly scross-the-board suggests that both the insights extracted from LMs and the representation modifications are useful. We also provide insights into the rare case where our method does not improve the baseline (e.g.. ALIGN model on Waterbirds) in Appendix
5.2 IMPROVING LANGUAGE MODELS
Setup. We experimented on four text classification datasets: CivilComments-WiL. (Borkan ct al. 2019; Koh al. (02T). HateXplain Mathew et al. (02T)) Amazon-WILDS (Ni et al, 2019
Koh et 2021 and Gender Bias classification dataset (Dinan et al. 2020 Miller al. 2017). 1). We

---

Published as a conference paper at ICLR 2024
Table ROBOSHOT zero-shot classification. Best WG bolded, second best underlined. We use inference models comparable to BERT embedding model (i.e., BART-MNLI) and to Ada embedding model (i.e., ChatGPT) for direct prompting experiments.
ZS
Direct prompting
ROBOSHOT
Dataset
Model
AVG
WG(1)
Gap(L)
AVG
WG(t)
Gap(L)
AVG
WG(T)
Gap(L)
BERT
48.1
33.3
14.8
32.5
15.7
16.8
49.7
42.3
7.4
CivilComments
Ada
56.2
43.2
13.0
85.6
19.2
66.4
56.6
44.9
11.7
BERT
60.4
0.0
60.4
61.2
5.3
55.9
57.3
14.0
43.3
HateXplain
Ada
62.8
14.3
48.5
55.4
12.2
43.2
63.6
21.1
42.5
BERT
81.1
64.2
16.8
74.9
36.0
38.9
81.0
64.4
16.6
Amazon
Ada
81.2
63.4
17.8
80.1
73.5
6.6
82,9
63.8
19.1
BERT
84.8
83.7
86.1
78.4
85.1
84.9
0.2
Gender Bias
Ada 77.9
60.0
17.9
90.1
86.6
3.5
78.0
60.1
17.9
the default test splits of all datasets. In text experiments, the distinctions between harmful and helpful insights are less clear than for images- so here we only use harmful vector rejection (line 3 in ROBOSHOT). CivilComments and HateXplain are toxic classification datasets with unwanted correlation between toxicity labels and mentions of demographics (e.g.. male, female, mentions of religions). The datasets are annotated with demographic mentions of each text, and we directly use them to construct v², For Amazon and Gender Bias datasets, we query LMs with task descriptions. All experiments are conducted using frozen pretrained model embedding. We provide full list of prompts used in Direct Prompting experiments in Appendix D.3
Results. Table 2 shows that ROBOSHOT also improves zero-shot text classification, as shown by our consistent boost over the baselines across all datasets on BERT embedding model and BART- MNLI direct prompting. In the Gender Bias and Amazon experiments, RoboShot lifts weaker/older model performance to level comparable to modern LLMs (ChatGPT).
5.3 LABEL-FREE ADAPTATION (LFA)
Next, we evaluate our technique for maximzing robustness when users have access to labeled vali- dation data (as before, we do not use any training data). Setup. We run LFA (Algorithm 2 across
Table 3: LFA CLIP ViT-B-32 embedding. Best WG bolded, second best underlined. Best WG in blue best AVG green
ROBOSHOT
LFA
LFA (100 val)
Dataset
AVG
WG
AVG
WG
AVG
WG
Waterbirds
82.0
54.5
83.8
0.74
55.2
1 0.75
84.2 1: 1.1
53.6 1: 1.76
CelehA
84.8
80.5
86.7
+ 0.811
83.4
+ 1.02
86.5
0.72
83.8
+ 1.17
PACS
95.6
79.7
96.6
+ 0.43
84.3
+ 1.3
96.9
+ 0.38
82.5 + 2.16
VLCS
74.1
25.0
76.3
1.27
36.5
77.0
0.35
37.4
3.34
5 different random seeds and report the mean and standard deviation test results from the model with the best validation performance. Table shows results from using only 100 random validation samples (LFA 100 val) and the full validation set (LFA). We use WILDS (Koh et al.) (021) default splits in Waterbirds and CelebA, and randomly shuffle 70:20: 10 trainntsst.vlliatiion splits in PACS and VLCS. Note that ROBOSHOT performance is slightly different from Table T because there use all samples for test. The training was conducted using two NVIDIA RTX A4000 GPUs, and we report the hyperparameter choices in Appendix D.5]
Results. LFA gives extra improvements on both AVG and WG. Improvement mostly persists even when using only 100 validation samples. If users have validation labels, performance can be
8

---

Published as a conference paper at ICLR 2024
Table 4: ROBOSHOT with LMs of varying capacity. Best WG bolded, second best underlined
Dataset
ZS
Ours (ChatGPT)
Ours (Flan-T5)
Ours (GPT2)
Ours (LLaMA)
AVG
WG
AVG
WG
AVG
WG
AVG
WG
AVG
WG
Waterbirds
80.7
27.9
82.0
54.4
72.1
32.4
88.0
39.9
84.8
36.5
CelebA
80.1
72.7
84.8
80.5
77.5
68.2
80.3
74.1
84.2
82.0
PACS
96.7
82.1
97.0
86.3
96.2
80.3
97.2
74.0
94.8
71.9
VLCCS
75.6
20.5
76.5
33.0
69.6
20.5
75.5
26.1
72.0
18.2
further improved. This indicates that LFA can serve as a lightweight training- baanid alternative to the fully zero-shot approach ROBOSHOT when small set of labeled validation data is available.
5.4 EXTRACTING CONCEPTS FROM LMS WITH VARYING CAPACITIES
Setup.
We use LMs with different capacities: ChatGPT (Ouyang et 2022 Flan-T5 (Chung
et al. 2022). GPT2 (Radford et al., 2019), and LLaMA Touvron et al. 2023), to obtain insights.
Results. Table shows that even though the LM strength/sizes correlate with the performance, ROBOSHOT with weaker LMs still outperforms zero-shot baselines. We hypothesize, based on Theorem 4.1 and 4.2 that insight outputs from weaker/smaller LMs are still precise specifying the useful and non-useful terms and thus ROBOSHOT is able to the insight embeddings.
Table 5: Ablation. Best WG and Gap performance bolded, second best underlined.
zs
Ours (v' only)
Ours (u" only)
Ours (both)
Dataset
Model
AVG WG(1) Gap(U)
AVG WG(f) Gap(
AVG WG(f) Gap(L)
AVG WG(f) Gap())
CLIP (VIT-B-32)
80.7
27.9
52.8
82.0
50.4
31.6
82.6
30.2
52.4
83.0
54.4
28.6
Waterbirds
CLIP (ViT-L-14)
88.7
27.3
61.4
82.7
35.8
46.9
88.3
29.8
58.5
79.9
45.2
34.7
CLIIP (ViT-B-32)
80.1
72.7
7.4
85.2
81.5
3.7
79.6
71.3
8.3
84.8
80.5
4.3
CelebA
CLIP (ViT-L-14)
80.6
74.3
6.3
85.9
82.8
3.1
80.0
73.1
6.9
85.5
82.6
2.9
CLIP (ViT-B-32) 96.7
82.1
14.6
97.0
83.7
13.3
96.6
84.2
12.4
97.0
86.3
10.7
PACS
CLIP (ViT-L-14) 98.1
79.8
18.3
98.0
79.8
18.2
98.1
83.8
14.3
98.1
83.9
14.2
CLIP (ViT-B-32)
75.6
20.5
55.1
75.6
22.7
52.9
76.4
29.5
46,9
76.5
33.0
43.5
VLCS
CLIP (VYT-L-14)
72.6
4.2
68.4
70.9
6.8
64.1
73.4
8.9
64.5
71.1
12.6
58.5
CXR14
BiomedCLIP
55.3
28.9
26.4
55.7
41.8
13.9
54.8
21.8
33.0
56.2
41.6
14.6
5.5
ABLATIONS
Setup. We run ROBOSHOT with only harmful component mitigation (reject of: ROBOSHOT line 3), only boosting helpful vectors (amplify u ROBOSHOT line 7), and both. Due to space constraint, we only include CLIP-based models ablations. Results on all models can be found Appendix F
The combination of both projections often achieves the best performance, as shown in Table Tigure@providee insights into the impact of each projection. Rejecting v² reduces variance in one direction, while increasing n amplifies variance in the orthogonal direction. When both projections are applied, they create a balanced mixture.
Results.
CONCLUSION
We introduced ROBOSHOT,, fine-tuning-free system that robustifies zero-shot pretrained models in a truly zero-shot way. Theoretically. we characterized the quantities required to obtain improvements over vanilla zero-shot classification. Empirically, found that ROBOSHOT improves both multi- modal and language model zero-shot performance, has sufficient versatility to apply to various base models, and can insights from less powerful language models.

---

Published as a conference paper at ICLR 2024
REFERENCES
Prince Osei Aboagye, Yan Zheng, Jack Shunn, Chin-Chia Michael Yeh, Junpeng Wang, Zhongfang Zhuang, Huiyuan Chen, Liang Wang, Wei Zhang, and Jeff Phillips. Interpretable debiasing of vec- torized language representations with iterative orthogonalization. In The Eleventh International Conference on Learning Representations.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez- Pazz Invariant risk minimization. arXiv preprint arXiv:1907.0220 002933, 2019.
Hugo Berg, Siobhan Mackenzie Hall, Yash Bhalgat, Wonsuk Yang. Hannah Rose Kirk, Aleksandar Shtedritski, and Max Bain. A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning. arXiv preprint arXiv:2203. 11933, 2022.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embed- dings. In Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL http:::/pproceeddggg..s.t ccpppperrrcppppppppppppppppppp. httppppprrrfllleeppp/////00
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion pro- ceedings of the 2019 world wide web conference, pp. 491-500, 2019.
Annie S Chen, Yoonho Lee, Amrith Setlur, Sergey Levine, and Chelsea Finn. Project and probe: Sample-efficient domain adaptation by interpolating orthogonal features. arXiv preprint arXiv:2302.05441, 2023.
Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, and Ledell Wu. Alt- clip: Altering the language encoder in clip for extended language capabilities. arXiv preprint SXKi:22111006799 2022.
Kristy Choi, Chris Cundy, Sanjari Srivastava, and Stefano Ermon. Lmpriors: Pre-trained language models as task-specific priors. arXiv preprint vrXiv:2210.12530 2022.
Ching-Yas Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debias- ing vision-language models via biased prompts. arXiv preprint urxiv:2302.00070, 2023.
Hyung Won Chung. Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction fenertion---nnneedddeeee language mod- els. arXiv preprint arXiv:2210.11411 2022.
Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. Discovering latent concepts learned in BERT. In International Conference on Learning Representations, 2022.
Sunipa Dev and Jeff Phillips. Attenuating bias in word vectors. The 22nd International Confer- ence on Artificial Intelligence and Statistics, pp. 879-887. PMLR, 2019.
Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. Multi- dimensional gender bias classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Languuge Processing (EMNLP), pp. 314-331, Online, November 2020. As- URL https:
sociation for Computational Linguistics. doi: 10. 1t0p:6//////22..mmmlll. 7/www. aclweb org http://ahhoooooyy2ooeaaa emn maa...mmai.2........
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and miti- gating unintended bias in text classification. 2018.
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1657-1664, 2013.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep sismalisemantic embedding model. Advances in neural infor- mation processing systems, 26, 2013.
10

---

Published as a conference paper at ICLR 2024
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. arXiv preprint arXiv:2212.06338, 2022.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form compe- tition: Why the highest probability answer isn't always right. arXiv preprint arXiv:2104.08315, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening new frontier for causality. arXiv preprint arXiv:2305. 00050, 2023.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint N2Ni::22244.02337, 2022.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-win wild distribution shifts. In International Conference on Machine Learning, pp. 6664. PMLR, 2021.
David Krueger, Ethan Caballero, loern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribttion generalization via risk extrapo- lation (rex). In International Conference on Machine Learning, pp. 5815-5826. PMLR, 2021.
Anne Lauscher, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulié. A general framework for implicit and explicit debiasing of distributional word vector spaces. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8131 -8188, 2020.
Yannick Le Cacheux, Hervé Le Borgne, and Michel Crucianu. Using sentences as semantic repre- sentations large scale zero-shot learning. In Computer Vision-ECCV 2020 Workshops: Glas- gow, UK, August 23-28, 2020, Proceedings, Part I 16, 641-645. Springer, 2020.
Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. arXiv preprint arXiv:2210.11466, 2022.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequenee pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910. 1911.1911116111 13461, 2019.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference computer vision, pp- 5542-5550, 2017.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In Marina Meila and Tong Zhang (eds.), Proceedings the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6781-6792. PMLR, 18-24 Jul 2021. URL httpt:tt///////tt httpsss//proccede ngs..gg.......... mtw..ml..prees//11555...
press/v139/
1iu21f. html
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Deep learning face attributes in the wild.
Mayug Maniparambil, Chris Vorster. Derek Molloy, Noel Murphy, Kevin McGuinness, and Noel E O' O'Connor Enhancing clip with gpt-4: Harnessing visual descriptions as prompts. arXiv preprint arXiv:2307. 1661, 2023.
11

---

Published as a conference paper at ICLR 2024
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14867-14875, 2021.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07/1: 2022.
Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp- 79-84, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: N0.11653/v1/D17-2001 URL acc antho 1 ogy wtt//oogg/////////
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qim- ing Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.
2022.
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeles reviews and fine-grained aspects. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp. 188- 197, 2019.
Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg. Chils: Zero-shot image classification with hierarchical label sets. In International Conference Machine Learning, pp. 26342-26362. PMLR, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 7748-8763. PMLR, 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv: 1908. 10084, 2019.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint
2019.
Antonio Torralba and Alexei A. Efros. Unbiased look dataset bias. In CVPR 2011, pp. 1521-1528, 2011. doi: 10. 201.VVV....11..99553.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971,
Jialu Wang, Yang Liu, and Xin Eric Wang. Are gender-neutral queries really gender-neutral? miti- gating gender bias in image search. arXiv preprint arXiv: 109.:2009900433. 2021.
Junyang Wang, Yi Zhang, and Jitao Sang. Fairclip: Social bias elimination based on attribute prototype learning and representation neutralization. arXiv preprint arXiv:2210. 14562, 2022.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. Chestx-ray8: Hospital-scte scale chest x-ray database and benchmarks weakly-supervise classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-2106, 2017.
12

---

Published as a conference paper at ICLR 2024
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Lon Papers), pp. 1112-1 1222222, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.1865//1/N1881100 URL
org/ Bxg/N18--12 1101
Yu Yang, Besmira Nushi, Hamid Palangi, and Baharan Mirzasoleiman. Mitigating spurious corre- lations in multi-modal models during fine-tuning. arXiv preprint 3xXi::334.033113 2023.
Michael Zhang and Christopher Ré. Contrastive adapters for foundation model group robustness. arXiv preprint arXiv:2207.07/800 2022.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Matthew Lungren, Tristan Naumann, and Hoifung Poon. Large-scale domain-specific pretraining for biomedical vision-language processing, 2023. URL
https://arxis: http://wwwwwiarrar/o////ooooaa//
00915
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.085900 2019.
APPENDIX
The appendix contains additional related work, details, proofs, and experimental results. The glos- sary contains convenient reminder our terminology (Appendix [A]. Appendix provides the proofs of theorems that appeared in Section 4 In Appendix D give more details and analysis of the experiments and provide additional experiment results. Finally, Appendix Fentails additional experiments combining ROBOSHOT with other methods to highlight its versatility.
A GLOSSARY
The glossary is given in Table 6
Symbol
Definition
input vector
X
embedding matrix
X proj
ROBOSHOT projected embedding matrix class label, prediction
y.
cf
embedding of class i
21, s......
The concept vectors consisting of orthonormal vectors insight representations
ay
The coefficient of input I with respect to the concept 2j (before ROBOSHOT)
Aj
The coefficient transformed input å with respect the concept 2 (after ROBOSHOT) The coefficient of j-th class embedding with respect to the concept 2i The coefficient of j-th insight vector with respect to the concept 2; the number of harmful concepts the number of helpful concepts the number of benign concepts text encoder to get embeddings text string for insight vectors noise rates in the coefficients of benign/insight concepts
S
R
B
g
Obenign, /iisiigtt
Table 6: Glossary of variables and symbols used in this paper.
13

---

Published as a conference paper at ICLR 2024
B EXTENDED RELATED WORK
Using language to improve visual tasks. A large body of work has shown the efficacy of us- ing language improve performance vision tasks Radford al. (202T Frome al. (2013); Le Cacheux et al. 2020). Most relevant are those that focus on robustness, such as Yang et al. (2023). which uses text descriptions of spurious attributes in fine-tuning loss improve robust- ness. In contrast to these works, we focus on using textual concepts to improve zero-shot model robustness without fine-tuning. Other zero-shot works attempt to provide improvements to aceu- racy. For example, (Novack et al.] (203)) increases zero-shot accuracy by first expanding the class options into their subclasses (e.g., dog into labrador and golden retriever) and producing a final prediction by mapping them back to the superclass. The most closely related of these to our work is Menon & Vondrick (2022): Maniparambii et al. (2023), where GPT-3 generated class descrip- tors first generated, then CLIP predictions scores grounded by additive decomposition of scores from the prompts with the descriptors. This approach also does not require fine-tuning. How- ever, it focuses mainly grounding through prompting with class descriptors, while ours focuses on removing harmful concepts and increasing helpful concepts in the embedding space-enabling improved robustness difficult slices.
Language models as priors. The basis of our work is the observation that language models contain information that can serve prior for other tasks. Kıeıman et al. (2023) finds that LLMs can perform causal reasoning tasks, substantially outperforming existing methods. Choi et al. (2022) prompts LLMs for task-specific priors, leading to substantial performance improvements in fea- ture selection, reinforcement learning, and causal discovery. Our work shares the spirit of these approaches in using the insights embedded in language models enhance zero-shot robustness.
C THEORY DETAILS
C.I HARMFUL CONCEPT REMOVAL
As the simplest form of ROBOSHOT,, we consider the case of ROBOSHOT the harmful concept removal only, without boosting helpful concepts. Recall our noise model:
S
S+R Σ r=
S-R+B Σ
T
2 +
Or +
0g
-
b=S+
S+R Σ r=
S+R+B Σ
J' 2 +
2r
(1sts 8).
b R+1
Again, we assume that benign coefficients are drawn from zero-centered Gaussian distribution, ic. 0b, No, ~ (0, Thenign) and also helpful coefficients and non-target harmful coefficients assumed to drawn from a Gaussian distribution, i.e. Ng.t ~ NO dinsight), where 1 S S R, q * t so that only Tt, is constant.
C.1.1 EFFECTS ON HARMFUI. CORFFICIENTS
Now we prove the following theorem.
Theorem C.1. Under the noise model described above, the post-removal coefficient A, for harmful concept 2, satisfies
(k 1)ag0
ag 2g0
EAI
insight
+
Σ
insight
?2.0
2
tyfs
where k is the number of concepts (k- S R B).
14

---

Published as a conference paper at ICLR 2024
Proof. Let & be the output of harmful concept removal procedure such that
22
us
k
> 7g,02)) 21=1
7476
04
01
1=1
#=1
Mi.
j=1
As the first step, we out the coefficients of features. For notational convenience, let T Σ Rh Then,
k
& 147i,
T1,o2j) 2i=1
0421
T.
s=1
1=1
067 Vise
0621
23
Tg
Ok Ti, 7jis
032)
To
5
Σ
Yr
aj
2j
To
==1
Thus get the expression for the coefficient of the target feature
(15s5
S k
A, = 02 Vla
(=1
Next, we get the bound of the absolute expectation [EA,].
0i 7/.t
EAI-
Eag
1=1
1=1
1=1
P
2
s
Σ 1i= To,
Eax,
Ts.t
+
Σ
Σk 2.
Here, the second term on RHS is 0 by independence, i.c.
Σ
1
Σm 2
n1
since Bys,7 by independence. Now split the first term and get the bounds separately.
15

---

Published as a conference paper at ICLR 2024
0x 0s7 2
B.A
Ro's
21=1 1=1 Vi.
a,7s.
9
7
$ Ba,
Σ
E
Σ 21=1 Mis
Σ
71.8
1=1,8
The upper bound for the first term can be obtained by
0g 78.8
i
72.0
Σ /:
11=1
R.
Ei=1 Ri
i Og T5,8
E
0g
Ex
2
1,58
78,8
ipts
(k:
1)a 2
insight
22.
And, for the second term,
Σ
ag
a 2
S
5
IE
a Ta.t
[=1,8
2i=1
k
Pi,
1=1,1
78.t
at E
Ntat
t=1,t
Og insight
7/.
t
Combining two bounds, we get the proposed result.
(k 1)
insight insight
ag insight
EA./ <
22.
72,
While the constant (A: 1) can look daunting since it actually increases as the number of concepts increases, bound less affected by o inssight exists well, scaling down the target coefficient a..
Corollary C.1.1. Under the noise model of Theorem C.I. concept satisfies
the post-removal coefficient for harmful
(k 1)0² meightt
8 Og5
2
EA. < as
insight
18.8 + (I: 1) 1)o? insight
t
where k is the number of concepts (k S R B).
16

---

Published as a conference paper at ICLR 2024
Proof. With the identical steps to the proof of Theorem C.1
obtain
2
Ys,t
BAA| $
Ray,
k
7i,1
9
1s.t
Ea,
|= Ri
Σi=1 7i,t
1,
S Σ
1,73,
0
S Rays
IE
Σ
-1=1 7180
1=1,1
11,1
We improve the first term follows.
as7
Ea,
ag
18
2
Σ 11
11.
Σ
21=1 7(
1
0g
0g 18,8
E3 E &
Jensen's 's inequality
Σt=1
n7.0
EΣ k
R
75.5
0g
1
EΣ 1=1 7.
18.0
72. + (k- - 1) 1 insight
(k: knaiohe
03
15.4 + (k- -1 insight
C.1.2 EFFECTS ON HELPFUL, BENIGN COEFFICIENTS
Based on the coefficient expression
Ag 0g
1=1
11,1
we analyze the bound of |EAy| for S+11++111 9 ful, benign coefficients are less affected than harmful coefficients as long as the harmful coefficients of insight embeddings are significant and the noise small.
Essentially, the following theorem implies help-
Theorem C.2. Under the same noise model described above, the post-removal coefficient for helpful or benign concept q satisfies
8
Qig
2 iniight
E.,, - -agl <
F
17

---

Published as a conference paper at ICLR 2024
Proof. The proof technique is essentially identical to Theorem C.1
Og /g,t + Σ j=1,j 7g.
EAA- - agl
Qq
Eag
Σi=1 ni
1=1
8
ag Ng.t
2
j= T9.
E
+
E
El=1 ni
1=1 7111
8
Og 2
Σ Ej=t,jt Ogg Oq
E
Σ 1= Th
8
Og
7i,
Og insight
7i,
This bound implies the differences of helpful or benign features by harmful concept removal are pro- portional to the noise of insight embeddings o insigher and inversely proportional to the coefficients of harmful coefficients of insight embeddings.
C.2 HELPFUL CONCEPT ADDITION
With similar fashion to the harmful concept removal, we consider the following noise model for the helpful concept addition.
S-R+B Σ
Σ
ag2g
+
Σ
S+1
U=
v'= 5
S+R
S+
Ta +
Σ
Th
(S+151S
S=S-1
b=S+
Again, we assume that benign coefficients are drawn from a zero-centered Gaussian distribution, ie. Ogg Tb, No0, Tbenign) and also harmful coefficients and non-target helpful coefficients assumed to be drawn from another Gaussian distribution, i.e. Tg.t ~ NO0, Tinsight), where < S S R. q A l so that only Tt,t constants.
C.2.1 LOWER BOUND FOR THE COEFFICIENT OF HELPFUL CONCEPT
Theorem C.3. Under the described noise model, the post-addition coefficient for helpful concept satisfies
P (I:
EA, 2
14
1)0²
msight
2
Cr.
Proof. Let è be the output of helpful concept addition procedure such that
TTot 11or/200
$ -
Σ
S=1
k
S+R
Σ amit &
0i
Σ n
b
1=1
t=
==1
18

---

Published as a conference paper at ICLR 2024
As the first step, sort out the coefficients of concepts. For notational convenience, let T₃ Ex R. Then,
k
S+
R
Di
5
04
7g.0
Tg
1=1
1=++1
k
S+R
Σ
04
+
Ti
1=1
1=$+1
i=1 j=1
S+R Σ
Σ
ajzy
23
Te
1=
S+R
Σ
0j
2z.
1=
s=1
Thus we can get the expression for the coefficient of the target concept 2r ($+15r$$+1)
S+R
A, = a, +
T
tm S+1
Then,
S-R
EA, = Ea, +
Tt
1=$+1
S+R
E Tra
ar +
t= 8+1 1 iml
mml
Ti
S+R
k Trd
0g 1r.r
04 Ira
= Oo + E
Σ 11
1= L,
1= N1
1=$11, i=1
S+R
k
a 1r.r
ai7
= Gr + E
Ei=1 1/
Ei=1 ni
im Light
tm S+1,11
S+R
k Ara
Or +
by symmetry
k
11,r
S=
21=1 Ni
a, + E
1. by law of total expectation and symmetry
1==1 1
1
2 0r + 0g 1r.
Σ/= 1==1
1
ar + ar
2
Jensen's inequality
EΣ
1
ar + a Tr 22, + (k- 1)02
insight
Thus, we obtain the result.
Pr.r
EA, 2 14
Cr-
22.r + (I: 1)0
?nniight
19

---

Published as a conference paper at ICLR 2024
C.2.2 EFFECTS ON HARMFUL, BENIGN COEFFICIENTS
For notational convenience, let Ketpf be the non-heipful concept index set such that helpful {i Ni S S R + S S + B). For q E f.. we obtain the bound of effects on harmful, benign coefficients with similar fashion to the harmful concept removal case.
Theorem C.A. Under the same noise model described above, the post- post-addition coefficient for help-
ful or benign concept satisfies
S+R
ag insight
EEAg Ongl 5
7h,
L=S+1
Proof.
8
Oig 19.t
+ Σ, Lj=1,ta Tij
EAg
0gl =
0g Rog +
Eks The
S+R
0g
99.1 14.1
2
Σ 1- Ogg 75.4
1,j44
E
12
El= n8.
S=
2i=1 1=1
71A
S+R
Σ Cン-1يア Na. Tia
Og 19.t
E
PP.A
Σ Th.
S=
0igg Titt
Th,t
S+R
ag Insight
C.3 COMBINED MAIN RESULTS
Now, we are ready to provide the combine main result, i.e. the coefficient bounds with harmful concept removal and helpful concept addition. The noise model can be described as follows.
S
S+R
S-R+B
Σ
Σ
O.a
+
Og +
Oh
r=S+1
b=
S
S+R
S+R+B
v'
2o
+
2r +
To
(LstS S+R)
r=S+1
b=
70.5 NO, (benig)))))
Tg [0, Tinsight),
where 1 1 < + R, 9 # so that only Tt is a constant. We can obtain the expression for cach coefficient as before.
S
k
S R
I
i
0j
Ti.8
Σ
Tir
T.
T.
j=1
8=1
r=
S+R Σ
a. 19.8
No.
A= ag
T.
T,
S=
where Ag is the coefficient of (1 k) after RoBoSHoT(ignoring normalization) and T, Σk_ 22 Using the results from the previous subsections, we provide an upper bound on harmful coefficients, lower bound on helpful coefficients, and an upper bound on the change in the benign coefficients. We restate Theorem 4.1 4.2 and provide proofs.
20

---

Published as a conference paper at ICLR 2024
Under the combined noise model described above, the post-RoBOSHOT coefficient for harmful concept (1 S 9 S S) satisfies
Theorem 4.1
(k-
1) e) 2 insight
S+ R Σ t=
inginsihht 2 7i,
EAAl -
7q.9
t=1,t
where k is the number of concepts (k- = =S R B).
Proof.
S+R
0,7 7g.e
EAAl
Eag
Tg
T
s=1
y= 1=1
(k: 1) 1 of insight
agonniight Pe.
1q.4
==1,05g
1=
The
(k 1) o insight
S+R
Σ
agy insighhht
insight nh
two terms have the same sign dg
74.4
t=L,4a
Next, we state the lower bound for the helpful features. We assume the signs of harmful concepts in input embeddings
0₂ 00 (1=ssS))
keep the appearance of the result clear.
Theorem 4.2
With an additional assumptions as S 0 (1 <. 8 S S) under the combined noise
model, the post- ROBSSOT coefficient for helpful concept qS + 1 q S S R) satisfies
1/.9
EAg 2 1+
+ (k 1) o)
Qq:
79.
insgght
Proof.
S
S+R
k
EAg Eag- Σ)
arTi /g.s
T.
Tr
am1
iml
r=S+1 im
S+R
k
Eaa +
Σ
047iir Yai
E
0, Jq.s
T
r=
S+R Σ
k
04 Ng.r
Oss 1o
1g,9
Bag +
E
E
T.
T.
T.
r=S+1
lm1
sm
1=1.t4g
Here, E
Dk
ar7 0 by symmetry and law of total expectation, and
2im1
T.
2
-E S
====1
0 since a, by assumption, which can be dropped for lower bound.
T,
21

---

Published as a conference paper at ICLR 2024
S+R
8
5
E., = Eag +
Tr
T
r S
i=1
==1
==1
1,89/
S+R
2 Edg Σ
Tr
99.9
2 +
Olgg
+ (k-
1)0 insight
7q.9
Now, we state the upper bound the changes in benign concepts. The proof is straightforward from the previous ones in harmful concept removal and helpful concept addition.
Corollary C.4.1. Under the same combined noise model, the post- Rost-RoBOSHOT coefficient for benign concept q satisfies
S+R Σ
0g
2
IEA., agl S
insight
Ti.
22

---

Published as a conference paper at ICLR 2024
D EXPERIMENTS DETAILS
D.1 DATASETS
Table 7 provides details of the datasets used in experiments. For Gender Bias Mataset[Dinan et al. 2020 Miller et al.] (2017), we test using the train set to get more data. For all other datasets, we the default test set. For Amazon-WilLDD Ni et al. (2019) dataset, convert the original 5-class rating classification into binary, by removing all samples with rating 3, and convert rating and into bad label, and 4 and 5 into good label.
Dataset
Groups
Nau
Nag
Vhclasa
classes
landbird land,
(landbird, waterbird
landbird in water,
Waterbirds
5794
642
2
waterbird on land,
waterbird water }
male & not blond,
(not blond, blond}
female not blond,
CelebA
19962
180
2
male & blond.
female & blond) }
art, cartoons, photos, sketches.)
(dogs, elphant, giraffe, guitar, house, person }
PACS
9991
80
7
Caltech101,
{bird, car, chair, dog, person}
LabelMe,
VLCS
10725
20
5
SUN09, VOC2007 }
no-pneummothorax, pneumothorax }
(no-pneumothorax pneumothorax]
CXR14
2661
20
2
(male, female, I.BTQQ christian, muslim, other religions, black, white
(non-toxic, toxic }
CivilComments-Wl...
133782
520
(hindu, islam, minority, refugee, indian, caucasian, hispanic, women, disability, homosexual, arab, christian, jewish, men, african. nonreligious, asian, indigenous, heterosexual, buddhism. bisexual, asexual)
(normal, offensive]
HateXplain
1921
6
2
(beauty, garden, books, luxury beauty, kindle store. movies and TV. pet supplies. industrial and scientific, office products,
(good,
CDs and vinyl, electronics, cell phones, magazine, clothing, groceries, music, instruments, tools, sports, automotive, toys, arts crafts, kitchen, video games, pantry, software, gift cards) } mmle,, female
Amazon-WILDD
90078
25
2
Gender Bias
22750
3594
2
(female, male)
Table 7: Dataset details
23

---

Published as a conference paper at ICLR 2024
Dataset
Model
harm/ul prompt
helpful prompt
ChatGPT
"List the hiased/spurious differences
"List true visual differences between [classes]."
between [classes]."
Flan-T5 & GPT2
["[class] lypically", "[class] usually")
("a characteristic [class]: "[class] are". ""aa [class] is",
All
"Characteriestics of [class]" "Stereotype of [class]"
"Typical characteristic [class]")
LLaMA
"List the biased/spurious characteristics of [class]"
"List the visual characteristies of [class]"
Table 8: Image dataset prompt details
Dataset
Model
harmful prompt
Amazon-WIL.D WILDS
ChatGPT
"what the biased differences hetween good and bad amazon reviews?"
"what the biased differences
Gender bias
ChatGT
hetween comments about female and comments about male?"
Table 9: NLP dataset prompt details
Dataset
Class prompt
Waterhirds
["a landbird", "a waterbird"]
CelebA
["person with dark hair", "person with blond hair")
PACS
"an image [class]"
VLCS
"this object is [class]"
CXR14
I'non- pneumothorax", "pncumothorax"
WwilComments-WILDDS
["non-toxic", "toxic"]
HateXplain
""normal", "offensive")
Amazon- WILDS
("negative", "positive""
Gender
"female", "male"]
Table 10: Class prompt details
D.2 PROMPT TEMPLATES
We provide details prompts used to get the pharmfal and hhllpful on image datasets in Table 8 As mentioned in the main body, for NLP datasets we only used gharmfal Additionally, we use the demographic mentions annotations to construct hharmful in CivilComments SittmmenttsWWLDDS Borkan ett al. (2019): Koh et al. (2021) and HateXplain Mathew et al. (2021) We provide prompt details to get 0 Parmall for Amazon WILDS Ni al. (2019); Koh et al. (202 | and Gender Bias Dinan et al.
2020 Miller et al. (0017) datasets in Table 9
We also provide class prompts in Table I0
D.3 DIRECT PROMPTING PROMPT TEMPLATES
Table II shows the prompts used for Direct Prompting ChatGPT baseline Table MNL., directly the dataset labels label input the model.
2
For BART-
24

---

Published as a conference paper at ICLR 2024
Dataset
Prompt
CivlCComment-WILLDS
Is this toxic comment? Answer with yes/no
HateXplain
Is this a toxic comment? Answer with yes/no
Amazon- Amazon-WILDS
Is this a positive or negative product review? Answer with pos/neg
Gender Bias
Is this text about male/female? Answer with male/female
Table 11: Direct prompting prompts for ChatGPT
D.4 ROBOSHOT EXPERIMENT DETAILS
All ROBOSHOT experiments are carried out using frozen weights and embeddings from huggingface ALIGN AACLIIP) and open-clip (CLIP ViT-B-32 and ViT-L-14, BiomedCLIP), and no training is involved. There is randomness the ROBOSHOT experiment results reported in the main body of the paper.
D.5 LFA EXPERIMENT DETAILS
Dataset
Batch size
Learning rate
Waterbirds
{1.5e - 2.5e 8 5e 3,.5--,,000 22.56-7
{16, 32, 64]
CelebA
(7.5e le 2,25 "}
{16, 32, 64)
PACS
(2.50 5e 7.5c 1,55e" 8))
{16, 32, 64]
VLCS
(2.5e "ee 9, 7.5. 1.5e
9,
{16, 32, 64)
Table 12: LFA hyperparameter choices
Table 12 shows the choices of hyperparameters we tune over for LEA experiments. use SGD optimizer with fixed default momentum form PyTorch. All training are run for fixed maximum epoch of 300, and we choose model based on validation performance.
25

---

Published as a conference paper at ICLR 2024
E
FULL ABLATION RESULT
Table 13: Ablation. Best WG and Gap performance bolded, second best underlined.
ZS
Ours (e' only)
Ours (u² only)
Ours (both)
Dataset
Model
AVG WG(1) Gap(L)
AVG WG(t) Gap(.)
AVG WG(1) Gap(.)
AVG WG(1) Gap(.)
CLIP (ViT-B-32)
80.7
27.9
52.8
82.0
50.4
31.6
82.6
30.2
52.4
83.0
54.4
28.6
Waterbirds CLIP (ViT-L-14)
88.7
27.3
61.4
82.7
35.8
46.9
88.3
29.8
58.5
79.9
45.2
34.7
ALIGN
72.0
50.3
21.7
56,4
41.6
14,8
62.8
56.4
6,4
50.9
41.0
9.9
AIICLIP
90.1
35.8
54.3
81.4
59.0
22.4
89.1
35.2
53.9
78.5
54.8
23.7
CLIP (VIT-B-32)
80.1
72.7
7.4
85.2
81.5
3.7
79.6
71.3
8.3
84.8
80.5
4.3
CelebA
CLIP (VIT-L-14)
80.6
74.3
6.3
85,9
82.8
3.1
80.0
73.1
6.9
85.5
82.6
2.9
ALIGN
81.8
77.2
4.6
83.9
78.0
5.7
83.9
81.4
2.5
86.3
83.4
2.9
AIILIPP
82.3
79.7
2.6
86.1
75.6
10.5
81.9
79.0
2.9
86.0
77.2
8.8
CLIP (ViT-B-32)
96.7
82.1
14.6
97.0
83.7
13.3
96.6
84.2
12.4
97.0
86.3
10.7
PACS
CLIP (ViT-I-144
98.1
79.8
18.3
98.0
79.8
18.2
98.1
83.8
14.3
98.1
83.9
14.2
ALIGN 95.8
77.1
18.7
95.8
78.0
17.8
95.1
71.1
24.0
95.0
73.8
21,2
AICLIP
98.5
82.6
15.9
98.4
83.0
15.4
98.6
88.8
9.8
98.7
89.5
9.2
CLIP (ViT-B-32)
75.6
20.5
55.1
75.6
22.7
52.9
76.4
29.5
46.9
76.5
33.0
43.5
VLCS
CLIP (ViT-L-14)
72.6
4.2
68.4
70.9
6.8
64.1
73.4
8.9
64.5
71.1
12.6
58.5
ALIGN
78.8
33.0
45.8
78.2
30.7
47.5
78.0
43.2
34.8
77.6
39.8
37.8
AIICLIP 78.3
24.7
53.6
77.5
24.4
53.1
79.0
20.5
58.5
78.9
25.0
53.9
CXRI4
BiomedCLIP
55.3
28.9
26.4
55.7
41.8
13.9
54.8
21.8
33.0
56.2
41.6
14.6
We note that when doing both projections does not improve the baseline, using only uh or v still outperforms the baseline. For instance, the ALIGN model in the Waterbirds dataset achieves the best performance with only u projection. This suggests that in certain cases, harmful and helpful concepts are intertwined the embedding space, and using just one projection can beneficial. We leave further investigation to future work.
F ADDITIONAL EXPERIMENTS
F.1 COMBINATION WITH THE CALIBRATION METHODS
Table 14: Additional baseline: text-classification calibration method Holtzman et al. 2021
Calibration
ROBOSHOT
Calibration + ROBOSHOT
Dataset
Model
AVG
WG(t)
Gap(()
AVG
WG(t)
Gap(L)
AVG
WG(t)
Gap(L)
BERT
51.0
37.3
13.7
49.7
42.3
7.4
53.4
36.9
16.5
CivilComments
Ada
73.3
31.2
42.1
56.6
44.9
11.7
68.3
35.0
33.3
BERT
60.9
15.8
45.1
57.3
14.0
43.3
56.7
22.8
33,9
HateXplain
Ada
61.9
31.6
30.3
63.6
21.1
42.5
59.6
33.3
26.3
BERT
78.0
57.7
20.3
81.0
64.4
16.6
79.0
59.2
19.8
Amazon
Ada
71.2
50.5
20.7
82.9
63.8
19.1
83.2
63.9
19.3
BERT
85.4
83.2
2.2
85.1
84.9
0.2
85.7
82.5
3.2
Gender Bias
Ada
84.2
77.8
6.4
78.0
60.1
17.9
84.2
77.9
6.3
Table 14 shows that ROBOSHOT further benefits from the calibration methods. This further high- lights the versatility of ROBOSHOT-Wo can combine it with such methods with no additional work. To showcase this, show additional results from (1) applying the calibration method alone, (2) our method, (3) the combination.
This result show that the best performing method across the board is either ROBOSHOT or the combination. The underlying reason for this is that as the two methods are orthogonal, adding calibration can further improve the results.
26

---

Published as a conference paper at ICLR 2024
Average Accuracy
Worst Group Accuracy
100
100
25
25
Roboshot
Roboshot
80
80
Accuracy
Accuracy
40
20
20
0
10a
10*
10"
10
1/dlspis
1/dhught
(a)
(b)
Figure 3: Synthetic experiment with varying noise- As expected, the performance improves at a rate inversely proportional to Tnoise.
F.2 SYNTHETIC EXPERIMENTS
Setup. We validate our theoretical claims by performing synthetic experiment where we vary
the noise level in the insight vectors (Finsight). Higher Finsight indicates more noise. We use the
(0, 1, 0), Pbenign (0, 0, 1), and class embedding vectors = 2helpful + Espurious + Zbenign and Co = -helpful
following basis vectors concept vectors Zhelpful (1, 0), 2spurious
Espurious + Zhenign Experiments are repeated 100 times.
Synthetic data input distribution (s denotes spurious feature group)
-yy y s 0 A ([Whelnfut ul W Upperiouss Whenign] Timpur 1), 2500
x y = 1,8 = ~ N ([Whelpful Uspurious, Whenign]: Fingut T), = 2500
-
rry y 0, 0 M([ Whelpful. W. Wspurious Wbenign] Osmput)), [), 2500
- ay = 0, s = V([ Wh Wepurious: Whenign] Timputl)) = 2500
Insight vectors
Uhelpful Telpp Zhelpful Zspurioo + Whheniggs where Vs (1, imisght)
Tb ~ (0, benign))
Uharm Ful Thelfff ul + Tharm fut Espurions + Bbennign where To NO0 imisght) Tb ~ (0, Thenign)
For the experiment reported in Figure 0.5, Thelpfal al = 1, Thormful = 1, T Impput 0.5, Tbenign = 0.01
used Whelpful 1, Cepurious
1, Wbenign
Results. In Figure coefficients = 1). our algorithm works well, recovering worst group accuracy and improving average group accuracy. This result supports our claims Theorems 4.1 and 4.2
observe that up 10 20% of noise level signal (harmful, helpful
F.3 EMBEDDING ANALYSIS
We provide insights into the case where our method does not improve the baseline (ALIGN model on Waterbirds) Fig. Fig. 44a we visualize the original and projected input embeddings (a in green and red points, respectively), and the label embeddings (c" and c²). Fig. 4a (left) shows the embeddings from the ALIGN model. We observe that the projected embeddings (red) still lie within the original embedding space, even with reduced variance. In contrast, when examining the CLIP model embeddings (Figure 4a (right)), we observe that the projected embeddings are significantly distant from the original ones. Unsurprisingly, Figure 4 (left) reveals that ed and uh (harmful and helpful insight embeddings in black and blue stars, respectively) are not distinguishable the text embedding space of ALIGN, collapsing the input embeddings after ROBOSHOT is applied.
27

---

Published as a conference paper at ICLR 2024
ALIGN Waterbirds
CLIP Waterbirds
ALIGN Waterbirds
CLIP Waterbirds
0.1
0.1
0.0
0
-0.1
X
1
-0.2
-2
0.00
0.73
0.25
0.50
0.75
-2
X
e²
original
prrojected
X
e²
(b)
Figure 4: (a) Original (green) and projected (red) input embeddings æ, and label embeddings e and ct. (b) label embeddings e and ct, harmful insight embeddings ve (black star) and helpful insight embeddings w' (blue star)
F.4 ANALYSIS ON THE ROBUSTNESS TO SPURIOUS CORRELATIONS.
We provide in-depth result analysis explain the performance changes in the average accuracy (AVG) and worst group accuracy (WG), especially with respect to spurious correlations. Concretely, consider the distribution of the margin M X' + E given by M(x) == (c 2) (c 22) where ct,c are the correct/incorrect class embeddings. Accuracy can be expressed as (x)).. The margin distributions and the margin changes by roboshot are illustrated in Figure (Waterbirds), 6 (CelebA). We denote data with spurious features as Dxp (i.e. waterbirds with land background, landbirds with water background), and data with non-spurious features as Dnap (i.e. waterbirds with water background, landbirds with land background). In the first column, M (x) denotes the margin distribution zeroshot prediction. In the second column, (ārm) - M(x) represents the margin changes by the roboshot harmful concept removal procedure. In the third column, M (ita)=
M (frm) represents the margin changes by the roboshot helpful concept addition. Typically, inputs with spurious features D.p tend to be closer the decision boundary, inducing expected, we can observe that harmful insight removal procedure increases the margin of D but decreases the margin of inputs with non-spurious features D Pasp- This can explain the potential tradcoff between the accuracy of Dap and Dnsp- If the gain in D outweights the loss in Pneps the average accuracy increases as in most cases. However, if the gain Dap is less the loss in Dnsp. the average accuracy decreases as in ALIGN. In either case, the model performance in D.p is improved by this procedure. In addition step, we expect that margins improve both Dap Dnsp on average. Helpful insight addition procedure turns out be quite effective in CelebA dataset, where visual features can be described more easily by language models.
As
F.5 ISOLATING CONCEPTS BY AVERAGING RELEVANT CONCEPTS
Table 15: Left: Cosine similarity between concept images and original embedding vs. averaged embedding. Right: ROBOSHOT on Waterbirds with original vs. averaged embedding
Concept
Original
Average
Green
0.237
0.241
ZS
ROBOSHOT Original
ROBOSHOT Average
Red
0.236
0.240
AVG
WG
Gap
AVG
WG
Gap
AVG
WG
Gap
Blue 0.213
0.229
86.6
29.6
57.0
87.1
31.5
55.6
78.8
55.1
23.7
Yellow
0,237
0.246
Square
0.214
0.220
We conduct experiments to test the viability of our concept modeling. Specifically, want to find out CLIP input representation z contains harmful, helpful, and benign components (2,, 2,, and 2 respectively in equation 1 and whether it is reasonable assume benign components as noise.
28

---

Published as a conference paper at ICLR 2024
ViT-B-32
Dnso
200
Count
Dep
100
O
0.0 M( - M(x)
-0.1
0.0
0.1
M(x)
M(Xaa) (X7mm
(a) VYT-B-32
14
Dnsp
200
Count
sp
100
0
-1
O
1
-05 0.O
0.5
-05
0.0
0.5
M(x)
m(Xm) M(x)
M(Xaa) M((mm)
(b) ViT-I-14
ALIGN
Drop
200
Count
Dap
100
0
ln-
0.05
0.00
0.05
0.10
-0.025 O. 000 0.025
0.01 0.00 0.01
M(x)
m(Xm) M(x)
M(Xas) M((mm
(c) ALIGN
ALT
200
Dasp
Count
D sp
100
0
0.05
0.00
0.05
0.10
-0.05
0.00
0.05
0.01
0.00
0.01
M(x)
M(Xmm) M())
MiXad) -
(d) AMCCLI
Figure 5: Margin analysis in Waterbirds dataset. Typically, inputs with spurious features D.p tend to As expected,
be closer to the decision boundary, inducing more insight removal procedure increases the margin of D but decreases the margin of inputs with non-spurious features Dnep- This can explain the potential tradeoff between the accuracy of Dap and Dnspp If the gain in D.p outweights the loss in D the average accuracy increases in most cases. However, if the gain in D.p is less the loss in Dasp. the average accuracy decreases as in ALIGN. In either case, the model performance in Dap is improved by this procedure. In addition step, we expect that margin improves both Dap. Dnsp on average as ViT-B-32. However, in most cases, the margin changes are not that crucial, implying extracting helpful insights is not easy in Waterbirds dataset.
observe that harmful
29

---

Published as a conference paper at ICLR 2024
ViT-B-32
1500
Drsp
Count
1000
Dsp
500
-.2 0.0
0.2
0
1
((x
M(Xm) - M(x)
M(Xac) ((
(a) VIT-B-32
VIT-L-14
1000
Dnsp
Count
Dsp
500
2
0
2
2
0
M(x)
M(Xm) M(x)
M((aa) (((mm)
(b) ViT-L-14
ALIGN
600
D nsp
Count
400
Dsp
200
0
0.1
0.0
0.1
0.025 0.000 0.025
0.05
0.00 0.05 0.10
M(x)
M(Xm) M(x)
M(Xae) M((m.))
(c) ALIGN
ALT
600
Dnsp
Count
400
Dap
200
0
-0.1
0.0
0.1
000255 0.000 0.025
-0.1
0.0
0.1
M(x)
M(X - M(x)
M(Xad) M(xm)
(d) AICCIP
Figure 6: Margin analysis in CelebA dataset. Again, inputs with spurious features "blond" tend As expected, we can observe that harmful
induce errors ""meen"-blond", "gil""""non-boneee insight removal procedure increases the margin of DD but decreases the margin of inputs with non-spurious features Dnsp- which may lead the potential tradeoff. However, in CelebA dataset, the helpful insight addition step turns out to be helpful, increasing the margins of both distributions much. can be interpreted as helpful insights can be captured easily in images.
30

---

Published as a conference paper at ICLR 2024
wecerbirds
WG
WG
Poo0Shol
0.6
Accuracys:
0.80
0.3
0.1
0.3
25
0.10
10.0
of insights
insights
(a) Waterbirds
(b) CelebA
Figure 7: Number of insights ablations
Can we partition CLIP input representation into harmful, helpful, and benign concepts? For a particular concept (e.g., "land"), we hypothesize that the true concept component mixed with other concept components due the signal in training data. For instance, land often co-occurs with sky, cattle, and other objects. Thus, the CLIP representation of "land" is entangled with these other concepts. To potentially isolate the helpful concept, we ask LM for an exhaustive list of concepts related to "land" and average the embedding of all related concepts. The intuition here is that a clean "land" component exists each individual embedding. and the remaining is likely to be random, which can be averaged out and leave us with the true concept.
To verify this intuition, we compare the original and averaged embeddings of concepts listed in Table 15 (left). For each concept, we get 100 Google image search results and filter out noisy images (e.g.. images with large text and artifacts) by eyeballing. We then report the average cosine similarity between the images and original embedding vs. the embedding from our averaging procedure. Averaged embedding has higher cosine similarity across the board than original CLIP embedding. To some extent, this indicates that the averaging procedure isolates the true concept. And thus, benign components in embeddings can be canceled out.
Does ROBOSHOT gain improvement with isolated concept? Table 15 (right) compares ROBOSHOT with removing harmful insights using original CLIP embedding vs. averaged embed- ding. We Waterbirds dataset because the harmful insights are known prior. To isolate the effect of our averaging procedure, we use "landbird"" and "waterbird" as labels without additional prompts (e.g., "a picture of [label]"), and we only "land" and "water" as the harmful insights to remove, which causes slight difference with the results reported in Table Confirming our intuition, using the averaged embedding results in better WG performance and smaller Gap.
F.6 ROBOSHOT WITHOUT DECOMPOSITION
To the effectiveness of QR decomposition of insight vectors, we conduct additional ablation ex- periment of decomposition method. In Table 16 w/o QR (uá only), w/o QR (u² only), and w/o QR (both) represents roboshot rejection only, addition only, both without QR decomposition step. Con- trary to our expectation, in binary classification (Waterbirds, CelebA), Roboshot method works well without QR decomposition. This can be interpreted as insights from L.MM provide almost orthogo- nal vectors. However, in multiclass classification, where rejection, addition vectors are generated by combinatorially paring insights for each class, Roboshot method get worse. Especially, addition step collapse. While rejection step wears off the subspace that the insight vectors span and there couldn't be more difference, addition steps can push multiple times to the similar directions. From this ab- be explained by two
lation experiment, the benefits of obtaining subspace via decomposition ways. First, in removal step, it provides a clean way to remove the subspace that spurious features span. Secondly, int addition step, it prevents overemphasis on some helpful insight directions.
31

---

Published as a conference paper at ICLR 2024
Table 16: Ablation QR decomposition
Roboshot w/ QR
w/o QR (e' only)
w/o QR (u* only)
w/o QR (both)
Dataset
Model
AVG WG(T) Gap(L)
AVG WG(()) Cap(J)
AVG WG(f) Gap(U)
AVG
WG(1) Cap(J)
CLIP (ViT-B-32)
83.0
54.4
28.6
79.5
58.3
21.2
83.0
31.2
51.8
79.6
62.5
17.1
Waterbirds CLIP (ViT-L-14)
79.9
45.2
34.7
79.3
36.3
43.0
88.8
31.6
57.2
75.0
45,8
29.2
ALIGN
50.9
41.0
9.9
53.3
36.6
16.7
62.0
50.9
11.1
38.2
36.5
1.7
AICLIIP
78.5
54.8
23.7
70.8
56.1
14.7
89.0
35.0
54.0
64.3
52.8
11.5
CLIP (ViT-B-32)
84.8
80.5
4.3
85.3
81.6
3.7
80.5
73.2
7.3
86.5
83.5
3.0
CelebA
CLIP (ViT-I--14)
85.5
82.6
2.9
86.1
81.7
4.4
79.7
72.5
7.2
85.8
80.0
5.8
ALIGN
86.3
83.4
2.9
84.4
78.9
83.9
81.5
86.8
84.5
2.3
AICLIPP
86.0
77.2
8.8
86.5
75.6
9.9
80.4
75.6
4.8
86.0
77.8
8.2
CLIP (ViT-B-32)
97.0
86.3
10.7
97.0
82.9
14.1
85.5
37.8
47.7
83.8
33.0
50.8
PACS
CLIP (ViT-L-14)
98.1
83.9
14.2
98.0
79.8
18.2
84.9
13.4
71.5
85.8
11.8
74.0
ALIGN
95.0
73.8
21.2
95.7
75.9
19.8
56.9
0.2
56.7
58.0
0.2
57.8
AICLIP
98.7
89.5
9.2
98.4
83.1
15.3
67.8
4.0
63.8
65.0
2.8
62.2
CLIP (ViT-B-32)
75.6
33.0
43.5
75.5
20.5
55.0
21.4
0.0
21.4
30.7
0.0
30.7
VLCS
CLIP (ViT-L-14)
71.1
12.6
58.5
71.1
6.9
64.2
22.3
0.0
22.3
22.1
1.3
20.8
ALIGN
77.6
39.8
37.8
78.1
33.0
45.1
36.2
0.0
36.2
32.7
0.1
32.6
AICLLPP
78.9
25.00
53.9
77.5
25.1
52.4
31.4
0.0
31.4
30.6
2.0
28.6
CXR14
BiomedCLII
41.6
14.6
55.9
36.6
19.3
55.2
23.9
31.3
56.1
37.2
18.9
Average Accuracy
Worst Group Accuracy
100
100
Accuracy
60
Accuracy
60
40
40
20
ZS
20
Robeshot
Roboshat:
Rank of harmful/eeppfll subspace
Rank of harmful/helpfu subspace
10
20
30
35
10
20
30
35
Number insights
Number of insights
(a) AVG
(b) WG
Figure 8: Synthetic experiment on the number of insights
F.7 INSIGHTS ANALYSIS
In Figure 7 we show ablations on the number of insights used ROBOSHOT. We can clearly observe that both average and worst-group accuracy improves (almost) linearly with the number of insights used.
In theory, increasing the number of insights is beneficial until the span of insight vectors cover the subspace of helpful/harmful features - thus for the optimality, we need insight vectors up to the rank of helpful/harmful subspaces. To validate we conduct synthetic experiment with varying the number of insights. Here, the number of helptu//harmfu///enigg concepts is 12 respectively, with the embedding dimension 36 (12 12 +12). We add sequentially increase the number of insights based on similar synthetic experiment setup in Appendix F.2 after all target concepts are covered by least one insight, we resample insights for each target concept.
Figure & we can observe that the AVG/WG performance improves until the number of insights is the same with the rank of harmful/helpful subspace. More detailed synthetic experiment setup is as follows.
-g es where 12 and S S
32

---

Published as a conference paper at ICLR 2024
Er, where R 12 and S+15r$$+R
Cb. where B = 12 and S+R+1
Uspurious Ex=1 kz
S4R Whelpful Σ
1=$+1 S4R4B Ubenign = -b=S+R+1
Synthetic data input distribution (s denotes spurious feature group)
y = 1, 8 = ([W Upprriossspppuriuss
Uspurious: WhelpfulWheipfuu. Whenign Obenign Tmpptt )); n
2500
Ty y 2500
1, 1 <l[ W Wspurious Uspurious: WheipfulPhepp Wbenign Ubenign a Timput 4 )
Tyy- = 0, = 0 ~~ M([ W-- Eispurroouuaprrinnn www.hp/uuueeepffaatt Whenign Whenggg Cimput I); n2
2500
x y = 0, = ~ N Sparrious Esparious:
Whelpp hellpfuly Whenign Obenign O Cimput I),
2500
Insight vectors follow the same assumptions in theory the only target concept has the constant coefficient (?i for 1 S S + R) and other coefficients are sampled from
(0,, (isight) or ((0 benign))
For the experiment reported in Figure 0.5, Whenign = 0.01, Yi = for siss S R, Dimput = 1, Thenign
we used Whelpful
1, sppriious
33


Reviews:
[{"review_id": "bEmoHgtZ4H", "reviewer": "Reviewer_bCze", "summary": "The authors propose ROBOSHOT, a method for improving the robustness of pretrained models in zero-shot settings. It uses language models to obtain insights from task descriptions and uses these insights to remove harmful components and boost useful ones in model embeddings without any supervision.\n\nThe method is evaluated on nine image and NLP classification tasks and shows an average improvement of 15.98% over several zero-shot baselines. It is compatible with various pretrained and large language models and can further boost performance with a zero-shot label-free adaptation variant where there are a large number of unlabeled examples.", "questions": "Please address the weaknesses above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed ROBOSHOT method is an interesting novel approach that improves the robustness of zero-shot models against harmful concepts without the manual identification of harmful concepts. It leverages insights obtained from large language models to refine embeddings, and address inherited biases.  I also find the theoretical arguments interesting  which characterizes the conditions under which ROBOSHOT can outperform existing methods in zero shot learning.\n\n- The  paper presents a well-structured and rigorous experimental evaluation across various datasets and model architectures.\n\n- The paper is written in a clear and accessible manner, I like the detailed explanations of the ROBOSHOT algorithm, the theoretical framework, and the evaluation methodology.\n\n- ROBOSHOT consistently outperforms several zero-shot baselines on multiple datasets. Having a powerful zero-shot learning method can address many real-life image classification tasks where labels are hard to come by.", "weaknesses": "- No large scale datasets like imagenet\n\n- The benchmarks are limited to zero shot classification which is an easy task compared to zero-shot semantic segmentation and instance segmentation where this method could struggle.\n\n- I don't see this work as actual zero shot because the pretrained model has so much information about the classes present in the chosen datasets. This work would be more impactful if the experiments were conducted on rare classes to test whether this method generalizes well.  ChatGPT has been trained on the internet, so this work is far from zero shot learning unless we include classes that are least likely to be seen by ChatGPT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p7QhyMx1Go", "reviewer": "Reviewer_umVa", "summary": "The paper proposes to improve zeroshot performance of various foundation models by trying to segregate the representations into 3 set of orthonormal basis— harmful,helpful and benign vectors. Using a language model, the authors try to identify the set of harmful and helpful basis, and then try to remove/boost those basis accordingly. Overall, the problem is well motivated and gives good results. The method section however needs some efforts in writing to clarify some of the intricacies of the proposed approach.", "questions": "See weakness section", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 4, "strengths": "- The overall motivation and idea of the paper is quite novel with interesting applications across various foundational models like CLIP and LLM.\n- The empirical results also are quite strong.", "weaknesses": "- $X_{proj}$ has not been defined in LFA section.\n- The authors should clarify how the basis vectors ($z$) are identified in the experiments, as the decomposition of insight vectors is based on that.\n- I understand that the proposed approach is poised to give major gains in class imbalance settings or well known setting with spurious features. However, I encourage the authors to also provide results in standard classification tasks like imagenet using CLIP. I fear that in many standard tasks, removing these spurious features might hurt as well. However, one can always choose to not remove these. \n- Do the authors have any insights or ablations as to how many insight vectors ($m$) is needed. I couldn’t see those details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RYxbq9JfRe", "reviewer": "Reviewer_R1Zx", "summary": "The work tackles a very interesting and impactful topic, namely the improvement of zero-shot models without any additional labelled data, training or manual intervention. This goal is accomplished by leveraging existing pretrained Language Models (LMs) to infer positive and negative insigths from the task description, using their embeddings to obtain helpful, harmful and neutral subspaces and finally editing the representations to remove the harmful components while boosting the helpful ones. The paper then presents a theoretical analysis to characterize the conditions under which the framework allows correcting a wrong prediction. Finally, the framework is evaluated in a wide set of experiments, showcasing its benefits on worst-group accuracy when plugged on top of a varied set of baselines.", "questions": "Table 1\n\n- why does the model perform so much worse on Waterbirds except that for CLIP-B?\n\nFigure 2\n\n- what are Y_0 and Y_1? I guess they are some sort of class prototypes, but where are they defined?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "### Originality\n\n- The paper proposes novel methodology to improve the robustness of zero-shot models such as CLIP without fine-tuning or using extra data.\n\n### Quality\n\n- The simplicity of the debiasing techniques makes it a cheap solution that can be easily employed by any practitioner.\n- Accepting the assumptions, the theoretical analysis makes intuitive sense.\n\n### Clarity\n\n- The paper makes for a pleasing read: it is well written, easy to read and mostly clear.\n- The framework is clearly explained in detail with a straightforward formalization and algorithmic outline, making it easy to reproduce. Source code is also provided.\n\n### Significance\n\n- The method is employed on a varied set of baselines (CLIP ViT-B-32, CLIP ViT-L-14, ALIGN, AltCLIP for zero-shot image classification and BERT and ADA for zero-shot text classification) assessing its general applicability\n- The experimental evidence covers multiple datasets, namely Waterbirds, CelebA, PACS, VLCS and CXR14 for zs image classification and CivilComments, HateXplain, Amazon and Gender Bias for zs text classification. The datasets cover different domains.", "weaknesses": "- The assumption of concept embeddings being orthonormal doesn’t seem well motivated; do we expect the concept embedding of ‘waterbird’ to be orthogonal to ‘water’? I find the experiment in Appendix F.5 to be inconclusive due to the simplicity of the considered concepts, and I can’t immediately understand why the average of the images having a higher cosine similarity should give any insight on the decomposition of the space in harmful, helpful and neutral subspaces.\n    - Unfortunately, the overall motivation and analysis seems to lay on this assumption, making it a core criticity of the work.\n- The 15.98% improvement claim in the abstract actually regards the increase in Worst Group accuracy and not the overall improvement which is probably not positive, I find it should be stated clearly to avoid misleading the reader.\n- The qualitative assessment does not really immediately convey the effect of increasing $u^k$. Some quantitative metrics would help, e.g. class separability measure such as the ratio of the inter-class distance to the average intra-class distance $\\frac{d_{\\text{inter}}}{\\frac{1}{2} (d_{\\text{intra}{C_1}} + d{\\text{intra}_{C_2}})}$\n- From the presentation perspective, the captions could be improved. Figure 1 could use a textual description to clarify what’s going on in the image, e.g. how does it go from having two projected embeddings and a single one in the right part.  Analogously, the caption of Figure 2 doesn’t state what are $Y_0$ and $Y_1$\n- The framework fails in several cases to maintain the average accuracy of the baseline, being therefore only advisable when worst group accuracy is the metric of interest. Is this realistic? It would be nice to have a method that fell back to the standard setting when the approach proves to be detrimental.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RFXOFKHPNX", "reviewer": "Reviewer_1hyF", "summary": "This paper proposed a zero-shot method to improve the robustness of pre-trained model embeddings. The key idea is to leverage insights obtained from language models based on task descriptions. After extracting the insights, they use the insights to modify the image embeddings, removing harmful components and enhancing useful ones, without any supervision. To achieve this goal, the method encourages invariant representation (to spurious features) by projecting the pre-trained model embeddings onto the subspace orthogonal to the subspace spanned by spurious feature descriptions. Experiments demonstrate that the proposed method improves multi-modal and language model zero-shot performance.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Novel and useful setting:** The setting of improving the robustness of pretrained model embeddings with task description is novel. RoboShot offers a unique approach that preserves the out-of-the-box usability of pretrained models, which is a key advantage.\n\n- **Extensive experiments and analyses:** The authors demonstrated the efficacy of the proposed method and setting with extensive experiments and analyses, in terms of both datasets and settings.\n\n- The paper is well-written.", "weaknesses": "The robustification relies on the insights provided by language models. However, if the language model does not identify the potential failure cases of the model, the method cannot remedy it. For instance, if the LM does not propose background as a spurious feature, can the method still mitigate such spurious correlation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)