Learning to Compose Soft Prompts for Compositional Zero-Shot Learning
Authors: Nihal V. Nayak
Peilin Yu
Stephen Bach
Venue: ICLR 2023 poster (2023)
Keywords: compositional zero-shot learning
prompts
foundation models

Abstract: We introduce compositional soft prompting (CSP), a parameter-efficient learning technique to improve the zero-shot compositionality of large-scale pretrained vision-language models (VLMs) like CLIP. We develop CSP for compositional zero-shot learning, the task of predicting unseen attribute-object compositions (e.g., old cat and young tiger). VLMs have a flexible text encoder that can represent arbitrary classes as natural language prompts but they often underperform task-specific architectures on the compositional zero-shot benchmark datasets. CSP treats the attributes and objects that define classes as learnable tokens of vocabulary. During training, the vocabulary is tuned to recognize classes that compose tokens in multiple ways (e.g., old cat and white cat). At test time, we recompose the learned attribute-object vocabulary in new combinations to recognize novel classes. We show that CSP outperforms the CLIP on benchmark datasets by an average of 10.9 percentage points on AUC. CSP also outperforms CoOp, a soft prompting method that fine-tunes the prefix context tokens, by an average of 5.8 percentage points on AUC. We perform additional experiments to show that CSP improves generalization to higher-order attribute-attribute-object compositions (e.g., old white cat) and combinations of pretrained attributes and fine-tuned objects. The code is available at https://github.com/BatsResearch/csp.

Published as a conference paper at ICLR 2023
LEARNING TO COMPOSE SOFT PROMPTS FOR COMPOSITIONAL ZERO-SHOT LEARNING
Nihal V. Nayak", Peilin Yu,, Stephen H. Bach
Department of Computer Science Brown University
Providence, RI 02906, USA {nnayak2, pyul2, sbach]@cs wro.n. edu
ABSTRACT
We introduce compositional soft prompting (CSP), parameter-efficient learning technique to improve the zero-shot compositionality large-scale pretrained vision-language models (VLMs) like CLIP. We develop CSP for compositional zero-shot learning, the task of predicting unseen attribute-object compositions (e.g., old cat and young tiger). VLMs have flexible text encoder that represent arbitrary classes as natural language prompts but they often underperform task- specific architectures
the compositional zero-shot benchmark datasets. CSP treats the attributes and objects that define classes as learnable tokens of vocabulary. During training, the vocabulary is tuned recognize classes that compose tokens in multiple ways (e.g., old cat and white cat). At test time, we recompose the learned attribute-object vocabulary in new combinations to recognize novel classes. We show that CSP outperforms the CLIP on benchmark datasets by an average of 10.9 percentage points on AUC. CSP also outperforms CoOp, soft prompting method that fine-tunes the prefix context tokens, by an average of 5.8 percentage points on AUC. We perform additional experiments to show that CSP improves generalization to higher-order sutribute-atribute--bbo white cat) and combinations of pretrained attributes and fine-tuned objects. The code is available at httts:://githbbbco com httpghhhwbbbooo/tteeseerrrrllla Rescarchhhs
compositions (e.g., old
INTRODUCTION
Compositionality the long-stan standitgn goal of artificial intelligence of creating new concepts by combining existing primitive concepts (Chomsky 1956 Fodor & Pylyshyn| 1988; Hupkes et al.) 2020 Lake Baroni 2018 Marcus 2003). The practical advantage compositionality for deep neural networks lies in the ability to build new classifiers by combining existing classifiers. In this work, we consider compositional zero-shot learning, a classification task where the model Icarns to predict unseen novel compositions of primitive concepts (Naeem et al. [021 Nagarajan & Grauman 2018 Purushwalkam et al. 2019). Research on compositional zero-shot learning in language and vision focuses on attribute-object compositions such as old tiger and young tiger, where tiger is the object category described by the attributes old and young.
Existing methods for compositional zero-shot learning typically map attributes and objects pre- trained word embeddings and use a pretrained image encoder backbone to jointly align the image and the attribute-object text representations to learn compositionality (Li al. 2020 Mancini et al. 2021albl Misra et al., (2017; Nacem et al., [2021] Nagarajan & Grauman 2018 8 Purushwalkam et al. 2019 Xu et al., (2021. ). However, the pretraining of the word embeddings and image encoder is disjoint and isolated from each other, i.e., these methods learn to align image and text representations from scratch. These task-specific architectures also are limited in flexibility. For example, to adapt these methods to higher-order compositions with multiple attributes and objects such smal 11 furry cat or old white tiger, the original architecture needs to be modified. The ability to generalize beyond the original training length is key test for compositionality (Hupkes et al. 2020).
"Equal contribution

---

Published as a conference paper at ICLR 2023
Prompt
SSan
Mull-MMdal Embadding Space
Aphoto
Tiger
Oldd Tiger
Young Cat
pphoto of
Young
Cat
Cosine Similarties
Onisein
Aphoto
[ATTRIBUTE]
[OBJECT]
Text representation
Image Representation
Young Tiger
Cat
Vocabulary
Figure I: An overview of compositional zero-shot learning with CSI. We fine-tune the vocabulary for attributes and objects on the seen classes. Then we compose novel soft prompts to test on the unseen classes.
In contrast, we propose to build on large-scale pretrained vision-language models (VLMs), which are trained on massive amounts of aligned images and text (Jain et al. 2021 Jia et al. 2021 Li et al.) 2021; Radford et al., 2021). We focus CLIP (Radford al. 2021p, powerful vision-language model pretrained on 400 million image-text pairs. CLIP has two main components: the image encoder and the text encoder that produce vector representations for images and text in multi-modal embedding space. The text encoder accepts a textual input, or a prompt such as A photo of dog to produce vector representation for the class dog. Taking the cosine similarity with all the class prompts and the image, we get a compatibility score for the classes and pick the one with the highest score. However, CLIP without any fine-tuning underperforms task-specific architectures, even though it has been pre-trained on vastly more data. (See Appendix A for details.) This finding suggests that there sigmificant room for improvement from teaching VLMs like CLIP about composing concepts.
To improve VLMM for compositional zero-shot learning, we introduce compositional soft prompting (CSP), a parameter-efficien learning technique that tunes tokens of vocabulary to represent primitive concepts in a composable way. Fine-tuning large pre-trained models such as CLIP requires huge amounts of compute and may lead to overfitting (Sung et al. 2021 Mitchell et al. 2022) (see also Section 5). This challenge has motivated several soft prompting techniques in both language and
vision (Lester et al. [021] Qin & Eisner 2021] Vu et al. 2021 Zhou et al.] (021). These works tune a single prompt on a downstream supervised task, often a few-shot setting. For instance, they typically use prompts such A photo of [clas ass]] and tune the prefix A photo of the entire dataset. In contrast, CSP is a novel way of soft prompting. We treat the attributes and objects that are composed to define classes as learnable tokens of vocabulary in prompt as A photo of [attribute] [object]. We tune on multiple [attribute] and [object] ] prompt compositions, and then we recompose them into new prompts for zero-shot inference (Figure I.
Our results show that CSP improves over the zero-shot performance of CLIP. CSP significantly improves over CLIP across three benchmark datasets by an average accuracy of 13.7 percentage points in the closed-world setting and 8.0 percentage points in the open-wope world setting (using the AUC metric). CSIP also outperforms CoOp, a soft prompting method that tunes the prefix context, by an average of 7.3 percentage points in the closed- world setting and 4.3 percentage points in the open-world setting on the AUC metric.
In addition to improved benchmark accuracy, CSP has several other advantages when tested on other kinds of zero-shot inferences without any changes to training. We show that the learned attribute vocabulary can be decomposed to better classify attributes in isolation, using prompts of the form photo of [at nnttribbue] ob ject. We also show that training CSP with attribute-object compositions improves CLIP's performance on altribute-attributt-biiee compositions. Finally, we show that CSP improves generalization compositions of unseen attributes and seen objects. Prior work on compositional zero-shot learning typically only evaluates unseen compositions of seen attributes and seen objects.
In summary, main contributions are:
2

---

Published as a conference paper at ICLR 2023
1. We introduce compositional soft prompting (CSP), parameter-efffiient learning technique to improve the compositionality of large-scale language models (VLMs). The at- tributes and objects that are composed to define classes are treated as learnable tokens of vocabulary. Unlike existing work on soft prompting, our learned prompts are tuned on multiple compositions and then recomposed in new combinations for inference.
2. CSP improves the AUC accuracy of CLIP by an average of 10.9 percentage points across three benchmark datasets (Isola et al., 2015 Yu & Grauman 2014 Mancini al. 2021a). It also improves over CoOp, soft-prompting method that tunes the prefix context, by an average of 5.8 percentage points on AUC.
3. We conduct additional experiments to analyze CSIP. We show that training on attributeoojeect compositions improves CLIP's accuracy on attribute classification alone, autribute-attribur object compositions, and compositions of pretrained and fine-tuned vocabulary.
RELATED WORK
We describe the related work in prompting, parameter-efficient learning, and compositional zero-shot learning.
Prompting Prompting a recent focus in the language and vision communities that has shown
benefits in zero-shot and few-shot learning on a wide range of tasks (Bach et al. [022 Bommasani
al.
2021 Brown et al. 2020 Lester al.
2021
Radford et al. 2021| Qin & Eisner 2021 Sanh
et al. 2022 Vu et al. 202 Zhou et al. 2021 2022). Discrete prompts are typically hand written text
inputs that provide guidelines to large ure-trained models such as CLIP, GPT-3 (Brown et al. 2020),
etc. for inference without updating the model parameters. While manually engineering prompts can help achieve better accuracy, it is often cine-consuming and impractical to find the best prompt.
Soft prompting is an alternative to discrete prompts, where a part of the prompt is learned by backpropagating without fine-tuning the entire model. Several works using soft prompts show improved accuracy compared to hand-crafted prompts (Lester al. 2021 Li & Liang. 2021 Qin & Eisner 2021 [Shin et al. 2020; Vu et al. 2021; /Zhou et al. 2021). In all these works, soft prompts a single input concatenated to all inputs for the entire task. In contrast, we learn tokens for each primitive concept from multiple compositions and recompose them in new ways to represent unseen classes for zero-shot inference. We show in Section dhat traditional soft prompting can also improve CLIP on compositional zero-shot learning, but generally not as much as CSP
Recent works have used soft prompting for large-scale vision- language models. Zhou et (2021) propose CoOp (contextual optimization), a soft prompting method for few-shot object classífication. Other applications include visual question answering (Jin et af., 2022) and video understanding (Ju et al. (021). Again, these works tune single soft prompt on the entire dataset. One exception is Ge et al. (2022), which learns multiple soft prompts for cross-domain adaption. While our work shares similarities with their work, there are important differences. Our work decomposes the class labels into multiple parts rather than splitting the prompt into domain-related granularities such as domain-agnostic context, domain-speciff context, and class label. Furthermore, we focus compositional zero-shot learning where we do not have access to labeled examples from the unseen classes in the test set whereas they assume access to all the test classes during training.
Zhou et al. 2022) extend CoOp to CoCoOp (conditional contextual optimization) to reduce overfitting in prompt tuning for few-shot object classification. They learn lightweight network that takes the image representation and produces a conditional vector that is added to the soft tokens in the prompt. In Appendix B we compare CoCoOp and the analogous conditional variant of CSP, CoCSP. While CoCSP improves over CoCoOp across three datasets in the AUC metric, CoCSP does not improve over CSP on these tasks.
Parameter-Efficient Learning Soft prompting is closely related to the growing body of work in parameter-efficient learning (Houlsby et al., 2019 Guo et al. 2021 Mahabadi et al. 202 I, Sung al. 202 Ben-Zaken et al. 2022; Liu et al. 2022 They add small feedforward networks between the layers in the pretrained model, or use sophisticated techniques to select a sparse set of model parameters and update them by fine-tuning
labeled training set. However, unlike CSP. the methods do not assign semantic meaning to the parameters in order to enable composition. In Appendix C experiment with CLIP adapters (Gao al. 2021) for compositional zero-shot
3

---

Published as a conference paper at ICLR 2023
Classes
Attributes
Objects
house
ancient
house
castle
modern
castle
A photo of
[CLASS)
A photo of
[ATTRIBUTE]
[OBJECT]
road
open
CLIP
CS2
Figure 2: Comparison of CLIP zero-shot and CSP in a compositional zero-shot setting.
learning. We find that this approach, which was designed for few-shot learning, reduces accuracy unseen classes in three benchmark tasks.
Compositional Zero-Shot Learning The growing interest in compositional zero-shot learning has
contributed to several architectural innovations (Li et al., 2020; Mancini et al. [222 Misra et al.
2017 Naeem et al. 2021 Nagarajan & Grauman 2018) Purushwalkam et al. [219 Radenović et al.
2021).
Early works compose attributes and objects with transformation function (Misra et al.) 2017
Nagarajan & Grauman 2018). Recent work uses separate encoding layers for attributes and objects, and then combines them with late fusion using linear layer or multilayer perceptron (Purushwalkam
cet al. 2019). The most successful methods represent the attribute and object relationship in graph and learn their compositions via graph convolutional networks (Mancini al. 2021b] Nacem et al., 2021; Ruis et al.) (02T). Yun et al. (2022) study the emergence of compositionalit in CLIP pre- training by learning linear classifiers rather than soft prompts for primitive concepts. [Scialom et al.) (2022) show that continually fine-tuning large language models can learn composable instructions.
Evaluating CLIP in a fair setting compared to the existing task task-specific architectures for compositional zero-shot learning is challenging. On the one hand, CLIP is trained on web-scale dataset to which other methods do not have access, and may even contain some of the classes from the unseen split. On the other hand, existing task-specific architectures fine-tune orders of magnitude more parameters than soft prompting, while using specialized architectures that do not adapt as easily as VLMs to new tasks. For these reasons, our work focuses improving CLIP-based baselines, and we include summary of the results comparing task-specific architectures in Section 5 and extended results in Appendix [
Compositional zero-shot learning is also closely related to the broader goal of compositionality in artificial intelligence (Chomsky 1956; Fodor & Pylyshyn 1988). Existing works have studied compositionality for image generation (Herzig et al. 2020). video synthesis (Ye et al. 2019 Bar al. 202T; Nawhal N et al.) (2020), visual reasoning (Johnson al., 2017), semantic parsing (Drozdov et al. 2022), language grounding (Jin et al., (2022). and question answering (Yuan et al. 2019). For
more in-depth discussion on compositionality, we refer the readers to Hupkes et al.) (2020).
PRELIMINARIES
In this section, we formally define the task of compositional zero-shot learning and describe how to use CLIP for compositional zero-shot inference. Let A = {un, di,,.., } be the set of possible attributes and O {oo. 01, } be the set of possible object categories. Let the label space Y be the Cartesian product of the attribute set and the object category set, Y = A x C. We are given two disjoint label subsets such that Y Yseen C Y, Y Yunseen C V. and Yseen n Y Yunseen where Y Yeen and Y unseen are the set of the seen and unseen classes. At training time, we are given examples Syeen = {(x1,3)).. 91).... (I,, }n to learn some discriminative model f X W seen- During inference, we want the model to predict both seen and unseen classes in the test set, i.e., f: Ytest. In the closed-world evaluation, the test set is defined as Yrest = seen U.Y. Yunseen. In the open-world evaluation, the model has to consider all possible permutations of the attribute-object compositions, i.c., Y Yteat = Y and Y Kunseen Y Y scen-
We can casily adapt CLIP to compositional zero-shot inference by defining prompts appropriately. The prompts used in CLIP for zero-shot inference differ from the prompts used works like GPT- 3 (Brown et al.) 2020) and TO (Sanh et al. 2022). Instead of defining
prompt for N-way classification task, the N classes are transformed into natural language prompts such as A photo of [class] (Figure?) 2 left). This results in N prompt vectors, which are used to compute the cosine similarity with the image representation for prediction. For compositional zero-shot inference,
4

---

Published as a conference paper at ICLR 2023
Gradient Backpropagstion
Forward Pass
Cosine Similarity
To T
Apholo
[ATTRIBUTE]
[OBJECT]
Text Encoder
Image Encoder
T-I
Crass entropy loss
Figure 3: Training setup for CSP. The prompt with the attribute and object vocabulary passed through the text encoder to get the text representation. The example is passed through the image encoder for the image representation. Next, we take the cosine similarity for all the prompts with the image and compute the cross entropy loss. Finally, backpropagate the loss through the text encoder and update the attribute and object vocabulary weights.
replace the [class] placeholder with the attribute and object composition as follows: photo [attribute] [object]. The change in prompt format allows us represent pairs of attribute and objects such a photo of young cat in the text encoder without changes.
4 COMPOSITIONAL SOFT PROMPTING
In this section, we introduce compositional soft prompting (CSP), a parameter-efficient learning technique for fine-tuning large pretrained models for better compositionality.
Motivation
The goal of our work is to improve VLMs such as CLIP on compositional generalization where they seem to underperform the current state-of-the-a art methods (Appendix A). This perhaps because CLIP's pretraining on data crawled from the web does not provide sufficient supervision about attributes and how they combine with different objects. Therefore, we aim to teach VLMs such CLIP how to better compose primitive concepts. We approach this is a vocabulary-learning problem because it parameter efficient and gives us a natural way to compose new classes
Prompt Construction CSP treats the attributes and objects that are composed to define classes as learnable tokens vocabulary and tunes them on multiple prompt compositions. We represent each primitive concept, either attribute or object, as a new, auxiliary token in the VLM's vocabulary. To represent class, we combine fixed prefix and the corresponding primitive concepts, for example photo of young tiger, where young maps to the auxiliary weight vector for the corresponding attribute and tiger maps to the auxiliary weight vector for the correponding object. Then, we use these prompts for compositional zero-shot learning in the same way as we would for CLIP, as described in Section 3 Training and inference with CSP very simple as we only need to swap the vocabulary of the attributes and objects for any desired composition in the prompt (Figure 2 right). As a result, our method tunes only (|A] + (ol) x d parameters where d is the dimension of the vocabulary embedding. This is novel form of soft prompting, because prior work (Lester et al.) 2021 Qin Hisner [2021] Zhou et al. 2021) tune prompts for seen classes and usually one prompt for the whole task, whereas we compose learned prompt tokens to represent unseen classes at test time.
Training We want learn vector embeddings for the new, auxiliary vocabulary: 0 (0x0 Oo] where 0 E [[(||+00)))))) x))x We learn them by composing prompts with them and fine-tuning them on the seen classes. Figure shows the overall learning process for CSP.
First, we initialize the learnable vocabulary with pretrained embeddings from CLIP using the concept names, i.e. attributes and objects. If a concept, such as Faux Fur, has multiple tokens, we average their pre-trained representations construct prompt with the pretrained vocabulary for the prefix context and learnable vocabulary for the primitive concepts:
initialize the learnable vocabulary. Next, for each class, we
to (Wo ... Wp. O.. 0.)
where w, E Idd are the tokens of the prefix context, and 0,, and 0,, are the learnable parameters for the attribute and the object in the prompt.
5

---

Published as a conference paper at ICLR 2023
Composition
Trois
Vulidation
Test
Dotaset
[A]
[D]
|4
[F
[E
[2]]
[X]
Sttt-Steedlool Isola
300
UT-Zappa To & Goumen 2016
16
192
22998
15
5214
18
2911
C-OQA Mancam an 202
413
674
278362
5392
26920
1252
1040
7280
888
923
5058
Table 1: Summary statistics of the datasets used in experiments.
Next, we pass the prompt with learnable parameters through the text encoder to get the text represen- tation:
VLMr (ta,o)
t,o
VLMT (fa,o)]
Then, for some image get the image representation from the image encoder:
VLMy (x)
VLMY (z)]
Finally, we compute the class probability:
exp (x : to,o/T)
p (a, 0) x))
(a,ö)E exp (ar ta,a/T)
where 7 R is the temperature parameter from CLIP. We learn the parameters f by minimizing the cross entropy loss on the training dataset:
Σ
log (x) + A|
S
ssenl
(a.y)
where À E IR is the weight decay.
Inference During inference, we recompose the fine-tuned attribute and object vocabulary in the prompt. We compose the candidate prompts with the tuned 0 with the (attribute, object) pairs in the same way during training. In both closed- wooeddworld world and open-world settings, we only replace attribute and objects with the fine-tuned parameters in the prompt. Finally, we calculate the most likely attribute and object pair as follows:
V arg max Po a:)
We include the pseudocode for inference in Appendix
F
EXPERIMENTAL EVALUATION
In this section, we describe our experiments with CSPP We compare CSP to CLIP-hased baselines in the closed-world and open-world settings of compositional zero-shot learning. We also compare the performance of fine-tuned CLIP with CSIPon the benchmark datasets. Finally, we demonstrate that CSP can generalize beyond these benchmarks to three modified settings: attribute-only classification, composition, and inference with unseen attributes.
Dataset We experiment with three attribute-object composition benchmarks: MIT-states (Isola et al. 2015), UT-Zappos YYu & Grauman, 2014), and C-GQA (Naeem et al. 2021). Table summarizes the statistics of the datasets. states contains images of naturally occurring objects where each object described by adjective. UT-Zappos contains images of shoes paired with fine-grained states. For this dataset, we use the split suggested by Purushwalkam et al. (2019). C-GQA, a newly introduced dataset derived from the Stanford GQA dataset (Hudson & Manning 2019 contains images of objects paired with states.
Benchmark Evaluation We follow the standard closed-worlds and open- ppeldworld evaluation protocols. In the closed-world setting, the unseen classes subset of all possible attribute- combinations and are defined the dataset. the open-world setting, the model considers all possible attribute- object combinations.

---

Published as a conference paper at ICLR 2023
M77-Sties
U7-Zappus
C-GQA
Method
U
ALC
U
AUC
AUC
CLIP
46.0
11,0
25.0
Closed
CoOp
o4.4..
29.8
13.5
52.1 os
49.3 14
14.6
14
20.5 6.
26.8 0.
17.1 os
CS?
46.6
49.9 6.
36.3
19.4
64.27
66.2 1.2
46.6 122
33.0
28.8 p.
26.8 0.1
20.5 0.
6.2 c.
CLIP
10.1
14.3
12.8
3.0
15.7
20.6
11.2
22
4.6
4.0
0.27
Open
CoOp
34.5
9.
a2
2880
52.1
31.5
28.9
13.2
21.0
4.5
5.5
0.70
CSP
46.3
15.7
o7
57
64.1o7
44.
08.9
23.7
26.7
52
6.01
1.20
Table 2: Glosed-world (Closed) and open- world (Open) results on MIT-States, UT-Zappos, and C-GQA. For CoOp and CSP, we report the average performance of the models on 5 random seeds with standard error.
Following prior work (Mancini et al. 2021a), we report the performance in the generalized zero-shot learning for both the closed-world and the open-world settings. In generalized zero-shot learning, we include both the seen and unseen classes in the test set. Several works have noted that zero-shot models are biased towards the seen classes, so we follow the standard of adding a scalar bias to the unseen classes (Chao al., 2016 Ruis et al., 2021) Following prior work, we vary the bias from 00 to +00 to get a curve indicating the seen accuracy on the x-axis and unseen accuracy on the y-axis. We report the area under the curve (AUC) and select the operating point with the best harmonic mean (H) between the seen and unseen accuracy. We also report the best seen accuracy (S) when bias is -00 and the best unseen accuracy (U) when the bias is +oc. We average over five random seeds, selecting the best- pest-performing model after each epoch on the validation data.
Following prior work, we apply feasibility calibration to all methods for prediction in the open- world setting. The open-world setting is particularly challenging as the label space contains all possible combinations of attributes and objects in the dataset. For instance, the label space contains feasible compositions such as young cat and eroded с cliff fff f and infeasible compositions such as eroded cat. Existing work shows significant drop in model performance from the closed- world setting to the open-world setting (Mancini et al.) 202 b). As suggested in Mancini et al. (2021)), we apply a simple feasibility calibration based on GloVe embeddings (Pennington et al. 2014) to filter out infeasible compositions. For more details, see Appendix G
Baselines In our experiments, we primarily compare with CLIP-based baselines. We compare CSP to pretrained CLIP [Rafford et al.) 2021) and CoOp (Zhou et al. (021). CoOp is a soft-prompting method that learns the prefix context with limited labeled examples in few-shot setting. CoOp resembles prompt tuning (Lester et al. 2021) applied to VLMs.
We implement CSP and the baselines with a pretrained CLIP model in Py Torch (Paszke al., 2019). We use the CLIP model ViT-L/14 which is the largest available model in our experiments. Nonetheless, our method is agnostic to the choice of CLIP architecture. The pretrained CLIP VIT-L/14 model has vision transformer (ViT) as the image encoder and transformer the text encoder.
Training Details
We train CSP and CoOp by minimizing the cross entropy loss with the Adam optimizer over the seen split in the dataset for 20 epochs. We use a single NVIDIA RTX 3090 or V100 GPU depending on their availability train all our models. For each dataset, we choose the best hyperparameters based on the performance on the validation split. For more details, refer to Appendix H
Benchmark Results Our results in Table 2 show that CSP significantly improves over CLIP on all the benchmark datasets in the closed-world and open-world settings. In the closed-world setting, we outperform CLIP in the AUC metric by 8.4 points on MIT-states, 28.0 point on UT-Zappos, and 4.8 points on C-GQA. Additionally, we show that CSP beats CoOp, soft- prooptiommtng method, in the AUC metric by 5.9 points on MIT-States, 14.2 points on UT-Zappos, and 1.8 points on C-GQA. In results in the open-world setting show that CSP improves over CLIP by 2.7 points on MIT-States, 20.5 points on UT-Zappos, and 0.93 points C-GQA in the AUC metric. We outperform CoOp on MIT-States by 3.1 points, UT-Zappos by 9.5 points, and C-GQA by 0.50 points in the AUC metric.
We also report additional results on these benchmarks in the appendices. We compare CLIP-base methods to existing compositional zero-shot learning methods in Appendix A CSP outperforms all these methods two out of the three benchmarks the AUC and harmonic mean metrics in open- and closed-wond world settings, but as discussed in Section 2 such comparisons come with several caveats, we only include them in the appendices for additional context. We experiment with
7

---

Published as a conference paper at ICLR 2023
different backbones in Appendix and the trend is consistent with the results reports here. Finally, we qualitatively evaluate CSP in compositional zero-shot image to text retrieval in Appendix
Comparison with Full Fine- Tuning We aim to under- stand the potential benefits of fine-tumi tinettning all the parameters of CLIP instead of small number. To that end, we fine- tune all the parameters in CLIP on the seen classes our benchmark datasets.
Method
MiT-Stater
UT UTppappes
C-GQA
CLIP
11.0
5.0
1.4
CLIP (FT)
22.2 6.1
1.0
0.2
CSP
19.4
33,0 6.0
Table 3:
Closed-world results compar- ing CSP and fine-tuned CLIP (FT). For CSP and CLIP (FT), we report the aver- age AUC on 5 random seeds with stan- dard error.
Table 3 shows that fine-tuning CLIP can improve gener- alization to unseen classes but still achieves lower AUC compared to CSP on UTTZappos. In addition, fine-tuning CLIP requires GPUs with more memory and often metic- ulous hyperparameter selection (Dong et al. 2022).
Comparison with Task-Specific Architectures We
compare CSP to existing task-specific compositional zero-shot learning architectures. We con- sider the following task-specific architectures: CGE (Naeem et al. 2021), Co-CGE (Mancini et al.) 2021b) and ProtoProp (Ruis et al. 2021). These methods are the most recent best-performing methods in the closed-workd setting.
Our results in Table show that CSP outperform existing task-specific architectures MITT-States and C-GQA while being competitive on UT-Zappos in AUC. We see that CSP improves over existing task-specific architectures by 12.8 points on MIT-states and 2.0 points on C-GQA in AUC. We include extended results including open-world setting and additional baselines in Appendix
Method
Mi7-States
UT-Zappos
C-GQA
ProtoProp CGE
34.7
6.5
33.5
4,2
Co-CGE CLIP
33.9
11.0
5.0
1.4
CSP
19.4
33.0
6.2 0.0
Table 4: Closed-worke results compar- ing CSIP with task-specific architectures. For CSP, report the average AUC on 5 random seeds with standard error.
Generalization to Higher-Order Compositions To test the additional flexibility afforded by VLMs, we test if train- ing CSP with attribute-object compositions can generalize to higher-order compositions such as attribute-atribute- object compositions.
We annotate a novel challenge dataset: AAO-MIT-States, subset derived from the MIT-States dataset. In this dataset, we annotate the images in the test split of the MIT-States dataset with an additional attribute, to get an attribute- attribute-objea pair as the class label. More details on the annotation are included in Appendix !
Method
Accuracy
CLIP CoOp CSP
62.7
65.2 0.3
72.6
We compare CLIP, CoOp. and CSI to classify images with object classes. Since these class compositions are not present during training. we treat them as unseen classes and calculate the unseen accuracy. take the best performing models for MIT-States and run inference on the challenge dataset.
Table 5: Unseen ac-
curacy on 5 random seeds with std. error.
shows that CSP improves over CLIP by average 9,9 percentage points unseen accuracy and generalizes to suribbue--atributee results demonstrate that CSP improves the compositionality of CLIP's vocabulary, even in ways that were not explicitly supervised.
compositions without any modifications or training. The
Generalization to Mixed Vocabulary To further test the additional flexibility afforded by VLMs. we also evaluate CSP on compositional zero-shot learning with mixture of pretrained and fine-tuned vocabulary. This evaluation stems from the practical need to combine new unseen attributes with fine-tuned vocabulary. Evaluating in this setting will allow us to assess whether the benefits of CSP extend to classes including vocabulary not during fine-tuning, This setup goes beyond the above benchmarks, which include unseen combinations of attributes and objects, but all attributes and objects seen during training. Now, include completely unseen attributes.
8

---

Published as a conference paper at ICLR 2023
We apply CSP on UT-Zappos with different fractions of attributes seen attributes. We randomly select 25%, 50%, 75%, and 100% of the attributes and all the objects from the training set. Then, we classes the attribute-object pairs that include unseen attribute. Finally,
remove from the train the on the remaining seen attribute-ohject pairs with five random seed values.
For each split of the and unseen attributes, evaluate CSP by dividing the classes into three buck- ets: (1) unseen attribute seen object pairs, (2) seen (i.e., fine-tuned) attribute + seen object pairs in un- seen combinations, and (3) seen attribute seen ob- ject pairs in seen combinations. In this evaluation, we refer to the classes the first and second buckets as the unseen classes and those in the third bucket as the seen classes. This evaluation is more general than typical compositional zero-shot learning, which only evaluates on classes the second and third buck- ets. Similar
Go Generalization with Unseen Attributes
Accuracy
0.4
0.3
Fraction Attributes
CLI: attribute object
Unseen attribute
pair)
Swen aitr bute swen abject (seen pait)
evaluation in Section 5 we add scalar bias to the unseen classes and select the bias that maximizes the harmonic mean between accuracy on the unseen and seen classes in the validation set of UT-Zappos. We then report accuracy on unseen ex- amples in each of the three buckets. To contextualize the performance of CSP, we report the accuracy of CLIP on the unseen attribute + seen object pairs.
Figure 4: Results of CSP and CLIP with dif- ferent fractions of pretrained and fine-tuned vocabulary. In each fraction, we report the average performance of CLIP and CSP on random attribute splits.
Figure shows that the performance on unseen attribute + seen object pairs improves with CSP and sufficient training pairs. Initially, the performance of CLIP and CSP are comparable but by providing more combinations of supervision for the objects CSP significantly outperforms CLIP on the unseen attribute seen object evaluation bucket. These results demonstrate that the fine-tuned vocabulary of CSI can improve the compositional zero-shot performance of pretrained vocabulary.
Additional Experiments We summarize the additional experiments included in the Appendices B and
We compare CSP with other parameter efficient meth- ods (see Figure 5). In Appendix B we experiment
comparisons
CoCoOp CLIP Adspter
30
with CoCoOp, the conditional variant CoOp that in- corporates visual information, on compositional zero- shot learning (Zhou al. 2022). Our results show that CSP outperforms CoCoOp on all three datasets. In Appendix C compare CLIP adapter (Gao 2021) and CSP on compositional zero-shot learn- ing. We see that CSP still achieves the highest AUC across the datasets compared to CLIP adapters.
10
MiT-States
UT-Zappes
C-GQA
Datasets
Figure 5: Closed-worle results on MIT-States, UT-Zappos, and C-GQA comparing CoCoOp (Appendix B). CLIP adapters (Appendix C and CSP. We report the average AUC on random seeds.
In Appendix D to further test their flexibility, we see whether the learned vocabulary generalizes to classifying either attributes or objects alone. Our results show that CSPP consistently improves attribute classification performance but can often reduce object classification accuracy.
Finally, in Appendix [ we compare different ResNet and ViT backhones CLIP. Our results show that CSP performance improves with larger backbones, In particular, we observe the highest gains with VIT backbones.
CONCLUSION
We present a new style of soft prompting, CSP, for compositional zero-shot learning. We show that learning composable components of classes via soft prompting can improve downstream compo- sitional zero-shot performance with a small number of parameters. We also demonstrate that the learned vocabulary generalizes in multiple, useful ways.

---

Published as a conference paper at ICLR 2023
Authors' Note. The first two authors contributed equally. Co-first authors can prioritize their names when adding this paper's reference to their resumes.
ACKNOWLEDGMENTS
We thank Andrew Delworth and Elise Carman for helping annotate the AAO-MIT-States dataset. We appreciate the comments and advice from Cristina Menghini, Wasu Piriyakulkij, and Zheng- Xin Yong on our drafts. This material is based on research sponsored by Defense Advanced Research Projects Agency (DARPA) and Air Force Research Laboratory (AFRL) under agreement number FA8750-19 17-50-19-2-1000 The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted necessarily representing the official policies or endorsements, either expressed or implied, of Defense Advanced Research Projects Ageney (DARPA) and Air Force Research Laboratory (AFRL) the U.S. Govermment. We gratefully acknowledge support from Google and Cisco. Disclosure: Stephen Bach is an advisor to Snorkel AI, company that provides software and services for weakly supervised machine learning.
ETHICS STATEMENT
Zero-shot methods such as CSP should be thoroughly evaluated on in-distribution data before applying them to downstream task. They should also be audited for biases, particuarly if being used to make decisions that directly affect people. Models like CLIP are trained on Internet scale data contain biases that can cause harm to underrepresented groups, if not carefully evaluated. Further, any inference on unseen classes of data carries additional risk because the domain shift between training and application. For more details, refer the readers to Section 7 in Radford et al. (2021).
REPRODUCIBILITY STATEMENT
The code for our experiments is available thttps//githubb.com https://gthub.coo/Bttsssesse Roseeaassssseeerrccsssss We also show the pseudo-code for CSP in Appendix F In Section we provide all the relevant training details for the experiments: pretrained models, training and validation splits, GPUs, and python libraries. Finally, in Appendix we include the hyperparameters for all the datasets.
REFERENCES
Stephen H. Bach, Victor Sanh, Zheng-Xt Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-Jian Jiang, and Alexander M. Rush. PromptSource: An integrated development environment and repository for natural language prompts. In Meeting of the Association for Computational Linguistics (ACL) Demonstration, 2022. URL https: 7/ 1/aclanthology wwwp///2g..222.... acl- 9
Amir Bar, Roei Herzig, Xiaolong Wang, Anna Rohrbach, Gal Chechik, Trevor Darrell, and Amir Globerson. Compositional video synthesis with action graphs. In ICML, 2021.
Elad Ben-Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine- tuning for transformer- masked language-moded
ACL, 2022. URL https://
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon. John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson,
10

---

Published as a conference paper at ICLR 2023
John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee. Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning. Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. Nyarko. Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang. William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang. Xikun Zhang, Yuhui Zhang. Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, abs/2108.07258, 2021. URL https: //axivv htg//bs//// 2108 000.1000..2...
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel werbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bernrr, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewriPS 2020, December 6-12, 2020, virtual, 2020. URL http:::::://ooceeeiiiiiie/ wwwpiinnnnnneurrpepeeee neurips wwweeriss.ccppppeeeeeeeeee cc/ppapet//2222///ah.
Abst html
Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized sero-shot learning for object recognition in the wild. In European conference computer vision (ECCV), 2016. httlittts:///rxiv wwwpl//vorggbb//////oee/o httr/bbs/166000 04253
Noam Chomsky. Three models for the description of language. IRE Transactions on informa- fion theory, 2(3):113-122 1956. _ httpp::/www. its. ca wew.lteehhc... wwwechh.edd/mmat... idee/
Chomsky3Models, pdf
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Gu Shuyang, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Clip itself is strong fine-tuner: Achieving 85.7acurracy with vit-b and vit-l on imagenet. arXiv preprint arXiv:22/2.06/ 2022.
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song. Xinyun Chen, Olivier Bousquet, and Denny Zhou, Compositional semantic parsing with large language models. arXiv preprini 2rX.v: 2022.
Jerry A. Fodor and Zenon W. Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, 18:3-71, 1988. URL
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma. Rongyao Fang. Yongfeng Zhang. Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. ArXiv preprint, abs/2110.04544, 2021.
04544
Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain adaptation via prompt learning. ArXiv preprint, abs/2202.06687, 2022. URL httt::tp:: ///rxiv org/ abs/ 2202 02. 06687
Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efffiient transfer learning with diff pruning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pp. 1884-48966 Online, 2021. Association for Computational Linguistics. doi: 10.18153/11/2011 acI- URL htl.ttts:: //acl antho org.h//looooog///221111 .acl ang.378

---

Published as a conference paper at ICLR 2023
Roei Herzig, Amir Bar, Huijuan Xu, Gal Chechik, Trevor Darrell, and Amir Globerson. Learning canonical representations for scene graph to image generation. In European Conference on Computer Vision. pp. 210-227. Springer. 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficiet transfer learning for NLPP In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2790 2799. PMLR, 2019. URL http:://rroceedinng. mllr. wwwps/veee/ee. http///////u//// houl sby 1 9a. html
Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6700-
6709. Computer Vision Foundation IEEE, 2019. doi: 10. 10.1109/CVP.22019.0000 00666. URL
http:///penacessthheeet com wwnt/ntottetttteeee ott.://see.eaaaatt...oo.o..ooeea for Batte__ellaaae CVPR_2019 2019__aaer___.____ [tmaprrhtml]
GQA
Visual Rewpuaal_eeaaoiinngnn_eeee and wwwpaandddooossitttoo
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed:
How do neural networks generalise? J. Artif. Intell. Res., 67:757-795, 2020. URL https
//arxiv. wttp//////aao/eeee/..e/eee
08351
Phillip Isola, Joseph Lim, and Edward H. Adelson. Discovering states and transformations in image collections. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 1383-1391. IEEE Computer Society, 2015. doi: 10. 109/ CVPR.2015.7298744 htRp:tt:: doi. wttp::......../////////...//.... 2015. 7298744
Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, and Jason Baldridge. Mural: multimodal, multitask retrieval across languages. ArXiv preprint, abs/2109.0511 2021. http:tts//://x...ror////
org.
05125
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, PMLR, 2021. URL
volume 139 of Proceedings of Machine Learning Research, pp. ht.ml
http: http:://proeeeeee ngs ml r.
Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. A good prompt is worth millions of parameters: Low-resource prompt-based learning for vision-language mod- els. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers). Association for Computational Linguistics, 2022. URL sttts://aclantholooy..ooo wwtp:oy..org/oo00/ www.::222..1111.11111.1
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xic. Prompting visual-language models for efficient video understanding. ArXiv preprint, (abs/2112.044777 2021. URL https
04478
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stock- holmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learn- ing Research, pp. 2879-2888, PMLR, 2018. URL http: httt:://proceeddngs.. mlr wtt.piggssmm....r.ss..... press/ v80 aakelaake1a 8a html
12

---

Published as a conference paper at ICLR 2023
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-effciient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republie, 2021. Association for Computational Linguistics. doi: 10. 10.18863/11/222 h2wp./2...lmmmaai225..... URL https://aci antha wtw.h/lllyyoooogoooo/.eoo emn1p 243
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1tth International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 4582-4597, Online, 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021. acl-long.353. URL hattps::/aclantthloooy.
acl-lonn....... 353
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang. Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: data efficient contrastive languugge-e language-image pre-training paradigm. ArXiv, abs/2110.05200. 2021. URL htttppss/aaxxi..org/aaa////
Yonglu Li, Yue Xu, Xiaohan Mao, and Cewu Lu. Symmetry and group in attribute-object composi- tions. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 11313-11322 1322. IEEE, 2020. doi: 10. 10.1 109/CVP42200 2020.01133, URL hothhttp://// /dop:: org 10. 10. 1109/1100011 Ct/R//6CR44200/ 2020. 01133
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. ArXiv preprint, abs/2205 05638, 2022. URL
http://axxiv...g/bbbbaab//// 220.. 05638
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank lofrrient hypercomplex adapter layers. In NeurlPS, 2021. URL http:ptps///aaa...c/////////1// 04647
M Mancini, MF Naeem, Y Xian, and Zeynep Akata. Open world compositional zero-shot learning. In 34th IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2021a. URL
ficts::/aaxxln wwwp/rggg///oooooo//t////...ee/s 2101. 12609
Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, and Zeynep Akata. Learning graph embeddings for open world compositional zero-shot learning. ArXiv, abs/2105.01017, 2021b. URL
wttp//vvoorr/aoooooooooooooo abs/2105. 01017
Gary F Marcus. The algebraic mind: Integrating connectionism and cognitive science. MIT press, 2003.
Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composition with context. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 1160-1169. IEEE Computer Society, 2017. doi: N0.1109/CVPR.2017.1122
110900..111////// 2017 129
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. International Conference on Learning Representations, 2022. URL
/ttpps:/ //tppse//ppenreveeww net. www.:/ewwwweeettttneeoeoooo.. /tt::/ooommo-oDoxeeeeEEEE
Muhammad Ferjad Naeem, Yongqin Xian, Federico Tombari, and Zeynep Akata, Learning graph
embeddings for compositional zero-shot learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 953-962, 2021. URL
abs/2100 0os3/210..00 01987
Tushar Nagarajan and Kristen Grauman. Attributes operators. ECCV, 2018. URL https:
/ waaxxiv.orr//bb//1333.33
Megha Nawhal, Mengyao Zhai, Andreas Lehrmann, Leonid Sigal, and Greg Mori. Generating videos
of zero-shot compositions of actions and objects. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XII. pp. 382-401 Berlin, Heidelberg, 2020. Springer- Verlag. ISBN 000---3300000000--00 doi: 10.
23. URL https: : http://o..ooog////////////////// org / http////....1009111111
5861 58610-2_625
13

---

Published as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen. Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Edward Yang. Zachary VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8035, 2019. URL httpa://proceeeingng//////////// neurips. wwtp:nuurrss.cccppeeeeeeee wwtpppppppp........c 201220999haiii
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pp. 1532-1543, Doha, Qatar, 2014. Association for Computational Linguistics. doi: 10.31
1162. URL https:: pettps://aclanteee wtt.oogg//.4-110.////
ogy
Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Mare' Aurelio Ranzato. Task- driven modular networks for zero-shot compositional learning. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2019, pp. 3592-3666666 --0022 IEFE, 2019. doi: 10. 1110///CCV2001.000000 10. 09 / T CCV. 2019 9. 00369
htl.https:///dl org/
Guanghui Qin and Jason Eisner. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pp. 5203-5212, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v 1 1/2021 1aa22ln.aall-aoe 410...410..... URL https::/aclaanhho
wwwporr//220. naacl ma main..4... 410
Filip Radenović, Animesh Sinha, Albert Gordo, Tamara L. Berg, and Dhruv Kumar Mahajan. Large- scale attribute-object compositions. ArXiv, 2021. URL
11333
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748- 1763. PMLR, 2021. URL http: http::::///occeedinnaana// mlr. press/v139/ radFord2la html
Frank Ruis, Gertjan J Burghouts, and Doina Bucur. Independent prototype propagation for zero-shot compositionality. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, 2021. URL
584b98aac2dddfs wwte:e9eeeeeecce wtt.aaafwee..eeeeeeeeeeeeeee000. Paper pdf
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL
n forum?id- DV..aa--VVVbDDIWWI Bot.e//
Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models continual learners. In Conference on Empirical Methods in Natural Language Processing, 2022.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
14

---

Published as a conference paper at ICLR 2023
pp. 4222-422244 4235, Online, 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020 emnlp-mane main.346. URL
.ttp://oooooooo.../2//.......... mttp://000o..00..mmmaaaaa-aa..a. 346
Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. In NeurIPS, 2021. URL
Tu Vu, Brian Lester, Noah Constant, Rami AI-Rfou, and Daniel Matthew Cer. Spot: Better frozen model adaptation through soft prompt transfer. ArXiv, 2021. URL https::/arxlv.orr org
abs/2110. 07904
Guangyue Xu, Parisa Kordjamshidi, and Joyce Chai. Zero-shot compositional concept learning. In Proceedings of the 1st Workshop Meta Learning and Its Applications to Natural Language Processing, pp. 19-27, Online, 2021. Association for Computational Linguistics. doi: 10. 10.18653/ v1/2021 1e00022meta/pp-1... 1.3. httlattts//aclaattollooo
org/ 2021 met.anl-1..3
Yufei Ye, Maneesh Singh, Abhinav Gupta, and Shubham Tulsiani. Compositional video prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10353-10362, 2019.
Aron Yu and Kristen Grauman. Fine-grained visual comparisons with local learning. In 2014 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pp. 192-199, IEEE Computer Society, 2014. doi: 10.110//VP..2014..222 URL
https: //ooi orr.loorggieee /00 100.110000 1109 CVPR. 2 2014. 32
Xingdi Yuan, Marc-Alexandre Côté, Jie Fu, Zhouhan Lin, Christopher Pal, Yoshua Bengio, and Adam Trischler. Interactive language learning by question answering. arXiv preprint 190i::1908, 2019.
Tian Yun, Usha Bhalla, Ellie Pavlick, and Chen Sun. Do language pretrained models learn primitive concepts? arXiv preprint arXiv:2203. 17271, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. ArXiv preprint, abs/2109.01134, 2021. URL lttppa/t/////x wwwpxiiioor//bb.....o
21 031
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for lision-language models. In CVPR, 2022. URL :tattts://aatai org/ 2203. 05557
15

---

Published as a conference paper at ICLR 2023
MiT-
C-ROA
Meehol
AUC
AUC
AUC
Saszessn.J Ceuuraan
184-
20.1
10.7
61.9
41.0
18.1
138
41200010110 20:9
30.2
20.1
13.0
2.9
58.7
60.0
45.0
29.3
23.1
6.5
7.5
Syate
24.2
16.1
09.8
57.4
-0.4
23.4
36.8
10.3
11.0
2.1
Cherpt
28.1
Close
Co
(0200
32.1
28.3
30,0
02.3
66.3
48.1
33.9
33.3
14.9
15.3
4.1
Razhord 10011
Auf
Vameeran Linggran 2015
16.6
0.7
30.9
34.2
29.4
13.3
00.8
Camg
21.4
7.0
5,5
0.5
53.3
44.5
34.5
18.5
36.7
12
3.3
0.43
Ope
Ctill Nartherenn pitee
12.4
8.0
1.0
61.7
47.3
23.1
R2.7
18
2.9
0-27
CGp Vortnli Fablb
311
5.8
6.4
1.1
62.
64.3
20.3
23.1
32.1
2.0
0.3
CoCCPP9 Mansent
512
0.78
<13P Raston H111
111
CxOp
36
16.5
16
40
61.5
18.3
15.6
19.5
269
45
57
0.7
Culp
36.
93
125
18
$23
31.5
23.93
132
21.022
486
55ed
0.70
46.5
174
37mm
64.
46.
189
22.7
367
52.
69
1,30
Table 6:
Closed-world (Closed) results MIT-Sttates UT-ZZaposs and C-GQA. For CoOp and CSP, we report the average performance of the models on 5 random seeds with standard error. The results for AoP, LE+, TMN, SymNet, CompCos, CGE, and Co- Co-CGE are obtained from Mancini et al. (2021b) and ProtoProp from Ruis et al. (202T). For extended results, see Appendix
A COMPARISON OF EXISTING PASK-SPECIFIC ARCHITECTURES
In this section, we compare CSP existing task-specific compositional zero-shot learning methods.
Baselines We consider the following compositional zero-shot learning methods in closed-works and open-world setting: AoP (Nagarajan & Grauman
2018). LE+ (Misra et al. 2017), TMN
(Purushwalkam et al., 2019), SymNet (Li et al. 2020). CompCos (Mancini al. 2021a). CGE
(Naeem et al. (021). and Co-CGE (Mancini et al. 2021b). We also consider ProtoProp (Ruis et al. 2021 in the closed-world setting.
Results Our results Table show that CSP outperform existing compositional zero-shot learning method on MITT-States and C-GQA while being competitive on UT-Zappos in AUC. In the closed- world setting, CSIP improves over existing compositional zero-shot learning methods by 12.8 points on MIT-states and 2.0 points C-GQA in AUC. In the open-world setting, improve over the existing compositional zero-shot learning methods on MIT-States by 3.4 points and C-GQA by 0.42 points in AUC.
B CONDITIONAL VARIANT OF COOP AND CSP
We experiment with CoCoOp, the conditional variant of CoOp. compositional zero-shot learning. CoCoOp showed improved performance over CoOp in few-shot object classification by using visual information to condition the prompts. We investigate if additional visual information in prompts can help in compositional zero-shot learning.
CoCoOp uses lightweight network to generate an image-specific bias vector and adds them to the learnable vocabulary in the prompt. For fair comparison, we also extend CSP to CoCSP to incorporate visual information into the prompts.
Setup We train CoCoOp and CoCSP with the same hyperparmeters in Appendix H Additionally, the lightweight network in our experiments is a two-layer multilayer perceptron with a ReLU activation between the two layers. The input and the output dimensions of the network are and the hidden dimension is d/16 where d is the dimension of the vocabulary. The generated image-specific bias vector is added to the context vocabulary in CoCoOp and attribute-object vocabulary in CoCSP.
16

---

Published as a conference paper at ICLR 2023
Method
MIT-States
UT-Zuppos
C-GQA
CLIP
11.0
5.0
1,4
CoOp
13.5 0.0
18.8 1.4
4.4 0.1
CSP
19.4 0.1
33.0 1.3
6.2 0.0
CoCoOp
11.3 0.6
18.8 1.1
4.2 0.1
CoCSP
17.6 0.1
32.7 0.3
5,7 0.0
Table 7: Closed-workd results MIT-States, UT-Zappos, and C-GQA. For CSP, CoOp. CoCoOp. and CoCSP, we report the average AUC on 5 random seeds with standard error. denotes bug the soft prompt for CoCoOp which will fix in the final version. In CoCoOp for C-GQA, average the pre-trained vocabulary for attributes and objects with multiple tokens instead of using them as is in the prompt.
Results Table 7 shows CoCSP improves upon CoCoOp across the three datasets, We also observe no benefits over non-conditional methods such as CSP and CoOp in AUC. We suspect that additional parameters for compositional generalization from seen pairs of concepts to unseen pairs.
C CLIP ADAPTERS WITH COMPOSITIONAL SOFT PROMPTS
We extend CLIP adapter (Gao et al. 202 to compositional zero-shot learning. Adapters are an alternate method of parameter- efficcient learning for large- scale models that has shown improved performance on several downstream tasks (Houlsby et al. 2019).
CLIP adapters are variant of the adapters for few-shot object classification (Houlsby et al. 2019). Instead of adding small feedforward networks to all the layers, they learn a multilayer perceptron in the image encoder and the text encoder. They transform the image representation from the encoder with the multilayer perceptron and include a residual connection from the image encoder and the same for the text representation. While the CLIP adapters can be added to the text encoder, the authors show that visual adapters perform better than textual adapters. For this reason, we compare with visual adapters in the experiment.
Setup We add CLIP adapter, two-layer multilayer perceptron with ReLU activation, to the image encoder. We use the same prompt prompt as CSP and train only the CLIP adapter. For fair comparison, we also train a model with CSP and CLIP adapters where we jointly train the prompts and adapter on the task. We train with the same hyperparamters settings for CLIP adapters and CLIP adapters + CSP in Appendix H
Method
MIT-States
UT-Zappps
C-GQA
CLIP
11.0
5.0
1.4
CL.IP adapter
9.5 0.1
31.5 a.9
3.2
CLIP adapter + CSP CSP
8.3 0.2
32.5 1.0
2.7 0.0
19.4 0.1
33.0 1.3
6.2 0.0
Table 8: Closed-word results on MIT-States, UT-Zappos, and C-GQA. For CLIP adapter, CLIP adapter CSP and CSP, we report the average AUC on random seeds with standard error.
Results Table 8 shows that CLIP adapters improve performance over CLIP but CSP performs better on its own. We also observe that combining CLIP adapters with CSI can often hurt performance in AUC.
D DECOMPOSITION OF ATTRIBUTES AND OBJECTS
To further test the flexibility of CSP, we investigate how the learned vocabulary performs on attribute classification and object classification separately. We create prompts of the form photo of aat tribute] object and photo of [object] classify images according either attribute
17

---

Published as a conference paper at ICLR 2023
MIT-States
UT-Zappos
C-GQA
Method
A
A
O
A
0
CLIP
16.6
44.3
8.8
58.5
3.6
25.4
S
CoOp
18.7 0.2
47.6 0.1
25.8 3.4
57.8
7.0 0.6
26.3 6.4
CSP
24.5 0.2
40.2 0.2
66.4 0.5
63.8 1.2
12.5 u.1
24.2 0.1
CLIP
19.4
46.8
11.1
65.6
4.1
25.4
CoOp
22.0 0.2
49.6 0.1
21.5 6.5
37.1
2.9
0.5
25.9 0.4
CSI
23.3 0.1
36.3 0.1
49.2 1.8
46.6
6.0
23.8 0.1
Table 9: Results for decomposition of learned attribute and object vocabulary. We report average top-1 accuracy for attribute (A) and object (O) classification on random seeds with standard error. Evaluation is done seen (S) classes and unseen (U) classes of each dataset.
object. We measure accuracy on two sets of evaluation data. The first is the seen classes of each dataset, meaning that the classification performed on new examples of attributes and objects that previously have been seen in the same combination. The second is the unseen classes of the original datasets, so this can be thought of as a kind of domain shift problem, evaluating how well the learned primitive concepts can be reused in isolation novel attribute-object combinations.
Table shows that CSP consistently improves attribute classification performance, while often reducing object classification accuracy. Sometimes CoOp improves CLIP's object classification accuracy as well. This results indicates that CSP learns useful standalone attribute representations that are also composable with objects, but that the resulting object representations might not be as good on their own.
E MODEL ABLATION
Table 10 shows that CSP with a performance improves with larger backbones. We note that CSP generally improves performance over CLIP. In particular, see the gains are highest with VIT backbones.
Mil-St
UT-Zappes
C'GBA
Method
Bockhose
S
U
AUI.
5
H
AUC
5
U
H
AUC
CLIP
RecNe: 50
21.1
34.4
18.4
5,6
6.1
43.6
6.4
1.4
6.1
17.1
6.1
0.7
CLIP
ResNe-101
25.2
37.4
21.7
7.5
11.2
35.2
11.9
2.8
7,3
19.7
7.6
L.1
CLIP
VIT R/32
25.1
39.1
21.4
7.5
9.6
42.4
10.0
2.4
7,3
22,1
7.4
1.2
CLJP
VIT1/14
15.8
CNP
RosNes-50
1.
38
32.1
25.2
42.2
2.
26
17.0a2
12.0a1
2.
VIT B/32
36
42.5
28.6a1
124
57.1 64
57.3
24.2
30.1
23
19
CSP
VLT
46.6
49.9
36.3
19.4a.
64. 07
66.2 3.2
46.6
33.0 3.3
28.8
26.8
20.5
6.2 cl
Table 10: Closed-world ablation results with respect to different backbone architectures of CLIP, We report the average performance of the model on 5 random seeds with standard error.
F PSEUDOCODE
Figure shows the Torch-like pseudocode for inference with CSP The function accepts the minibatch of images, test pairs, and the clip model with the fine-tuned embeddings and returns the cosine similarities between the image representation and the text representation scaled by constant scalar.
G FEASIBILITY CALIBRATION FOR OPEN-WORLLS SETTING
Feasibility calibration aims filter out infeasible compositions that might be present in the open- world setting. To filter out infeasible compositions, we follow the post-training calibration from Mancini et al. (2021b). They conjecture that similar objects share similar attributes while dissimilar objects are unlikely to share attributes. For example, cat and dog can share the attribute old but cat and cl ff do not share the attribute eroded.
18

---

Published as a conference paper at ICLR 2023
def inference (batch_imagens mn. Tenser, cest_pairss List (List, List], model: :odule);
Function to run inference with the fine-tuned embeddings. Args:
batch_ batch_imagee images test_pairs pairs (tuple): attribute-object pairs in the test test
rensor): minibatch of images [n, h, Wy cl spiit [m, 21
model (nn.Module): model with the fine-tuned embeddings
Returns:
torch. cosine similarties the minibatch images and attribute-object pairs [n, m]
"**
= "a photo of x xx
tokenized prompt tokenize (ereempt_template) tokenized_prompt
bokenized_prenee repeat (len (test_pairs) (tokeni prompt))
token_tensor model.. wawaallttoeeeeemeddeeeeee
fine-tuned embeddings
attr, obj attr_emb = 2
(attr)
obj emb model. wow.ottemmeaiigllo)......
replace the "x x" in prompt template with fine-tuned embeddings token_tensor replace emb (tcken_tensor, attr_emb, obj_emb)
12-normalized
text rep model. text encoder (ensoor
image_rep = model. image_mmmoddes (batch_imagee)
logits (image_rep 8 text_rep) * model. logit t__calleeeeee ()
return logits
Figure 6: Torch-like pseudocode for inference with CSP.
We calculate the feasibility compositions for the composition (a, o) by computing the relationships between the objects and the attributes. First, find the similarities between the objects:
((0) (0)
Po(a, 0) max oE0seen
[|o(o)] |(
where P.).) is the similarity between the object with other objects â and o(-) is an embedding function that maps attributes pretrained embedding. We compute similarities between the attributes in the same way.
Next, we combine the two similarities with pooling function. In our case, we use mean pooling pr: p(a, p(po(a, o), (a)a
where p(a, o) the feasbility score for the composition (a, o). Finally, we filter out infeasible compositions by considering compositions above threshold T calibrated the validation set get our final prediction:
arg tcsts p(a,o)>T
(y
Following prior work, we compute feasibility calibration using GloVe embeddings (Pennington et al., 2014), and filter out the infeasible attribute-object compositions based on the performance on the validation split.
H HYPERPARAMETERS
In our work, we find the best hyperparameters for training CSP via a grid search. We train the ViT-B/32 model for 50 epochs and use the same best performing hyperparameters to train our models including VIT L/14. We run grid search with the following hyperparameters: (1) learning rate: (5e - 03, 5e - 04, 5e - 05}. (2) batch size: {128, 256), (3) attribute dropout: {0.0, 0.1, 0.2, 0.3), and (4) weight decay: {le 05, 5e 05). We choose the hyperparameters for dataset based best unseen accuracy on the validation split. We reduce the number of epochs to with ViT L/14 as found our models tend to converge earlier.
19

---

Published as a conference paper at ICLR 2023
Hyperparameter
MIT-States
UT-7uppos
C-GQA
Learning rate
5e - 05
- 04
5e 05
Batch size
128
128
128
Attribute dropout
0.3
0.2
0.3
Weight decay
le 05
le 05
5e 05
Table 11: Hyperparameters for MIT-States, UT-Zapppss, and C-GQA.
CSP
CLIP
CSP
CLIP
Uiee
More shater
12.25
intaed poot
ataed past
paac
tke pars
4.5
Inflced nubber
velasda tey
pron Jane
Wiloed joans
17.0%
1.0%
50
Vnfatse bul--
wmstt
Blcck jane
Right jeass
spillcd
bocket
placed place
ploctppea
bromn copt
red
11.5%
30.4%
9.7%
trick place
pieced plae
siting bet
bura
bitz
Mloic
stanning place
while
omail dog
sanpy cout
plee
50.%%
6.%
Uhy dag
smuil dog
grecn
grece
41%
5-th
cld
14.7%
codec
pity building
lang prffte
prafti 2.75
larg
building
block gratte
gratt
Figure 7: Additional qualitative comparison for image text retrieval between CSP and CLIP CGQA. Selected samples with concepts correctly identified and top-5 retrieval results by CSI are shown.
Table 11 shows the hyperparameters used train CSP all the datasets. The experiments with CoOp and CLIP adapters do not use dropout as the original architecture did not use dropout in the text the image encoder. For the rest of the architecture, use the same hyperparameters for CoOp and CLIP adapters as CSP.
I ADDITIONAL QUALITATIVE EXAMPLES
In this section, include additional examples from the compositional zero-shot image to text retrieval tasks. Selected samples from CGQA in Figure 7 show that fine-tuning the vocabulary enables CSP to better identify composed concepts compared CLIP. In particular, we observe that CSIP ranks relevant attributes in the attribute-object composition better than CLIP.
DATASET CREATION
We create AAO-MIT-Statte from the MIT-States dataset (Isola et al. 2015). Below we include details on the annotation interface, annotators, and aggregation process for the annotations.
We annotate an additional attribute for the images paired with unseen classes in the test split of MIT-States. The interface has three main components: (1) general instructions, (2) image with caption, and (3) list of populated attributes. The general instructions provide the annotators with a detailed description of the annotation task. To the right of the instructions is randomly sampled
20

---

Published as a conference paper at ICLR 2023
image from the test split of MIT-States Additionally, include caption describing the image. For example, suppose we select an image of wet cat, we ask the users: "which attribute best describes the cat in the image presented?". Since we have large number of attributes in the MIT-States dataset, we need way to reduce the list of attributes the user observes while annotating single image. We CLIP predict the attrihutes except for the original attribute in the image and choose the top-5 attributes as annotation candidates. We also include an option to select none of the above.
Which attributes best describe the stone the image presented?
Select an option
cracked
eroded
2
weathered
3
scratched
ancient
None of the Above
Figure 8: Example annotation interface.
The dataset was annotated by two of the authors and two undergraduate research assistants. We randomly sampled a total of 1200 images from test-split and asked the annotators to annotate from the interface. We received annotations for 1089 instances where each image received exactly three annotations. We aggregate examples for our dataset where all the three annotators agreed on the same attribute other than "None of the above". The total number of examples in the final annotated dataset 193. We have open-sourced the dataset in our code.
21


Reviews:
[{"review_id": "oOo33cEAm0", "reviewer": "Reviewer_m1LH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the compositional zero-shot learning problem using CLIP, a powerful vision-language model that was pretrained on broad data to measure similarity between images and texts. Instead of fine-tuning the entire model, the paper proposes to fine-tune only the attribute and object tokens in a prompt template in the text input space of CLIP. Extensive experiments on three benchmarks, i.e., MIT-states, UT-Zappos and C-GQA, show that the proposed method, despite having a simple design, outperforms several other relevant baselines.", "summary_of_the_review": "The paper provides novel insights on how to apply prompt learning to large vision-language models to solve compositional zero-shot learning. Overall, the paper is well-written, the method is simple and technically sound, and the results are encouraging and useful. So I recommend to accept the paper.\n\n== Post-rebuttal update ==\n\nThe authors have done a good job in addressing the two weakness points. My view about the novelty, results and significance remains the same: the paper should be accepted.\n\nI have also read other reviewers' comments and noticed that some reviewers have concerns about the technical novelty and the resources needed to further tune CLIP. I would like to speak for the authors here.\n- The paper should be evaluated in the context of compositional zero-shot learning. The novelty is then clear because existing methods are mostly based on task-specific models while this paper demonstrates the effectiveness of such a simple prompt-driven paradigm, which is encouraging and could spur a paradigm shift in compositional zero-shot learning.\n- CLIP was indeed trained with massive data but this doesn't mean it must work in any scenario without some adaptation—which is the incomplete nature of foundation models. It would make more sense if we take a look at the counterpart in NLP like GPT or BERT. To adapt large language models, one also needs to apply some adaptation methods, such as learning specialized modules or using prompt learning.\n- It might seem straightforward to apply task-specific learning (e.g., prompt learning) to adapting CLIP to solve compositional zero-shot learning, but such a \"straightforward\" hypothesis has never been scientifically validated—which is done in this work.", "strength_and_weaknesses": "**Strengths**\n\nThe emergence of vision foundation models, particularly vision-language models like CLIP, has offered many opportunities to the computer vision community. This work provides timely insights on how prompt learning, a simple technique that tunes a few parameters in a model's input space, can be used to tackle compositional zero-shot learning. Though soft prompt learning has been applied to CLIP before, the results shown in this work on the compositional zero-shot learning setting are novel and useful to the community.\n\nIt is also worth mentioning that the field of compositional zero-shot learning has been previously dominated by task-specific methods. This work shows that a simple technique based on prompt learning and a large-scale vision-language model could provide significant improvements to the problem. Given the importance of large-scale models, which have become a trend in the community, this work could serve as a strong baseline to build upon for future work.\n\n**Weaknesses**\n\nBelow some minor comments are listed for the authors to address.\n\n1. From the technical point of view, CoOp fine-tunes all context tokens while this work fine-tunes only the attribute token and the object class token. It's good to see that such a seemingly trivial twist can lead to huge improvement. Currently, the paper doesn't provide in-depth analysis in this regard. Would it be possible for the authors to elabroate on this design and explain (perhaps with some evidence) why such a small change could make a big difference?\n\n2. The paper only studies this template, \"a photo of xxx\". Have the authors tried other variants and if so, what are observed in the results?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dagYbAX766", "reviewer": "Reviewer_wi3v", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary: \nThe paper proposes a new method for the task of zero-shot compositionality on top of CLIP style models. The proposed approach learns embeddings for objects and attributes using the CLIP objective. At test time, they are able to recombine the embeddings for attributes and classes for novel compositions and show good zero-shot generalization.\n", "summary_of_the_review": "Overall, the paper was easy to read and the presented idea was simple and intuitive. I am not sure that the paper in its current form presents any new significant insight that is not known to the broader AI community.\n1. We know that CLIP are great at zero-shot tasks.\n2. We know that prompt-tuning works really well instead of fine-tuning in a low-data regime.\n\nI think the paper is missing a good-faith effort of getting CLIP to work for compositional zero-shot without any fine-tuning / prompt-tuning. It's a little dissatisfying that despite training on so much data, CLIP models need further task-specific training to perform well. Doing task-specific prompt-tuning seems like a very natural step, and I am unsure if the paper presents any ideas that will be useful to the broader AI community outside the niche of people who look at the compositional zero-shot learning task.\n\nUpdate after rebuttal: I thank the authors for carefully addressing all the concerns in the rebuttal. I agree with the authors and reviewer  m1LH that the work is interesting in the context of compositional zero-shot learning. I am specially happy to see results for applying CLIP by learning the attribute embeddings without composition, and applying CLIP as it is. To reflect my updated thoughts on the paper, I am updating my score to 8", "strength_and_weaknesses": "\nStrengths: \n- The paper proposes a soft prompting approach for compositional zero-shot learning that improves performance with just a small set of learnable parameters (the tokens corresponding to attributes and classes). \n\n- The paper show many useful experiments -- generalization to higher-order compositions, prompt tuning vs full fine-tuning, etc. Overall, the experiments are enough to demonstrate the effectiveness of the approach on the proposed task. \n\nWeaknesses:\n- Generally speaking, I am unsure if the evaluation is fair / comparable. The authors themselves point out that comparison to existing methods is difficult because they were not trained on large amounts of web data. The authors compare to CLIP and CoOp but the results here are not very surprising either -- task-specific learning should help improve performance.\n\n- CoOp is a weird baseline to compare to. For CoOp, the prefix is learned, while for the proposed approach the token embeddings are learned. I think it will be nice to show if CSP + CoOp help over CSP. If you learn prefix tokens, as well as attribute and class tokens, does that help improve performance over just doing prefix learning / class & attribute token learning. \n\n- Another baseline to compare to would have been learning token embeddings individually instead of together. So instead of using \"A photo of [attribute] [class]\", learn using \"A photo of [attribute object\", and \"A photo of [class]\". Can the model then combine these in interesting ways? \n\n-- Finally, is it possible to avoid training at all on composition of attribute & classes at all. I think it's a little intellectually dissastisfying that despite training on so much web-scale data, we need to do more task-specific prompt tuning for recognizing compositions. Is there a smarter way to combine the two similarity scores (image-attribute, and image-class) that avoids the need to learn these embeddings? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Qj9eSoyReCa", "reviewer": "Reviewer_1rcT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the compositional zero-shot learning through the use of a large-scale pretrained vision-language model, specifically: CLIP. The paper takes the prompt-based classification approach where the cosine similarity between the text and vision embedding is used as the compatibility score. To improve this scheme, the paper takes a soft-prompt approach by fine tuning the prompt tokens' text embeddings. The fine-tuned object/attribute embeddings are then used in prompt-based zero-shot classification. T", "summary_of_the_review": "The paper is a good, mainly-experimental work on the soft-prompting based use of CLIP for compositional classification. The paper is clear and contains good experimental analysis. The technical novelty appears to be rather weak, the work is mainly a simple engineering solution (though I definitely do not question its value). I have concerns regarding the bias-term selection policy. ", "strength_and_weaknesses": "Strengths:\n- The paper presents a rather comprehensive experimental comparison with favorable results. The presented simple approach appears to be effective and overall quite well analyzed in terms of benchmark-based experimental comparisons.\n- The clarity of model selection details is a plus.\n\n\nWeaknesses: \n- The paper appears to be mainly an adaptation of a simple soft-prompting approach to the compositional zero-shot learning case. \n- As noted in the paper,  the underlying CLIP model is trained on a very large training set, even if that's possibly noisy. While CLIP has been used in various works for \"zero-shot classification\", it is hardly zero-shot; therefore, the problem can instead be seen synthesizing MxN-classifiers out of CLIP.  To this end, I find the criticism of \"task-specific architures\" in terms of adaptability and parameter complexity, at the end of Section 2 misleading and imprecise.\n- The model selection discussion emphasizes that all hyper-parameters except the unseen bias are tuned on the validation set wrt unseen performance. Here, I question the validity of tuning the bias term directly on the test set, which breaks the zero-shot setting, even if some prior work has taken this approach. What happens if you were to tune the bias term itself on the validation set, instead of the test set? How much performance drop would it cause? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1BUbgEOOIoW", "reviewer": "Reviewer_43mh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents compositional soft prompting (CSP), a parameter-efficient learning technique to improve the zero-shot compositionality of a large-scale pretrained vision-language model. In addition, the paper demonstrates that the learned vocabulary generalizes across multiple datasets. Finally, the proposed method substantially improves the results on multiple datasets.", "summary_of_the_review": "My main concern is that this paper presented a minor approach to tackles the important problem of the compositionality of VL models. I am open to the authors' feedback and other reviewers' opinions.\n", "strength_and_weaknesses": "**Strengths:**\n\n1) Overall, this paper is well written, and the technical details are easy to follow. \n\n\n2) The main idea of introducing better prompting techniques for zero-shot compositionality is appealing.\n\n\n3) The main contribution of this paper is the compositional soft prompting technique, which treats the attributes and objects that are composed to define classes as learnable tokens of vocabulary in a prompt as “A photo of [attribute] [object]”. Although I consider this a minor novelty in general, it still meets the bar.  \n\n4) The results of the experiment strongly support the proposed approach, including the strong ablations that were performed.\n\n\n**Weaknesses:**\n\n**Experiments.** Recently, prompting has become a hot topic. Why not add more recent prompting techniques such as VPT [1] and others? Essentially, the proposed technique is a standard CLIP training with additional attribute and object prompting. Therefore, I believe it would be useful to have a more comprehensive comparison of prompting baselines.   \n\n[1] Visual Prompt Tuning, ECCV 2022\n\n**Technical Novelty.** As I said above, the main contribution of the paper is the treatment of the attributes and objects that are composed to define classes as learnable tokens of vocabulary in a prompt as “A photo of [attribute] [object]”. I find it difficult to consider the proposed approach (prompting of attributes and objects together) to be a significant contribution, but I acknowledge that it is the first paper to address this important issue.\n\n\n**Relation to Prior Work.** There are several works in that area based on compositionality for videos or scene graphs, which the author could also add and discuss. This topic is very important, and the related work paragraph has a limited scope, which is fine, but here are some additional references the authors may wish to consider:\n\n[1] Bar et al. Compositional Video Synthesis with Action Graphs, ICML 2021.\n\n[2] Generating videos of zero-shot compositions of actions and objects. ECCV 2020.\n\n[3] Compositional video prediction. ICCV 2019.\n\n[4] Learning Canonical Representations for Scene Graph to Image Generation. ECCV 2020.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster