Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?
Authors: Tian Yun
Usha Bhalla
Ellie Pavlick
Chen Sun
Venue: Accepted by TMLR ()

Abstract: Vision-language (VL) pretrained models have achieved impressive performance on multimodal reasoning and zero-shot recognition tasks. Many of these VL models are pretrained on unlabeled image and caption pairs from the internet. In this paper, we study whether representations of primitive concepts–such as colors, shapes, or the attributes of object parts–emerge automatically within these pretrained VL models. We propose a two-step framework, Compositional Concept Mapping (CompMap), to investigate this. CompMap
first asks a VL model to generate concept activations with text prompts from a predefined list of primitive concepts, and then learns to construct an explicit composition model that maps the primitive concept activations (e.g. the likelihood of black tail or red wing) to com-
posite concepts (e.g. a red-winged blackbird). We demonstrate that a composition model can be designed as a set operation, and show that a composition model is straightforward for machines to learn from ground truth primitive concepts (as a linear classifier). We thus
hypothesize that if primitive concepts indeed emerge in a VL pretrained model, its primitive concept activations can be used to learn a composition model similar to the one designed by experts. We propose a quantitative metric to measure the degree of similarity, and refer to the metric as the interpretability of the VL models’ learned primitive concept representations. We also measure the classification accuracy when using the primitive concept activations and the learned composition model to predict the composite concepts, and refer to it as the usefulness metric. Our study reveals that state-of-the-art VL pretrained models learn primitive concepts that are highly useful for fine-grained visual recognition on the CUB dataset, and compositional generalization tasks on the MIT-States dataset. However,
we observe that the learned composition models have low interpretability in our qualitative analyses. Our results reveal the limitations of existing VL models, and the necessity of pretraining objectives that encourage the acquisition of primitive concepts.

Published in Transactions on Machine Learning Research (04/2023)
Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?
Tian Yun
tian han__ynn@@rrumm.cm
Deportment of Computer Science
Brown University
Usha Bhalla
usha bhwllahaaaaaoooooo edu
Deportment Computer Science
Brown University
Ellie Pavlick
ellie_ www.ick@@roum..cc... edu
Department of Computer Science
Brown University
Chen Sun
chensun@broum.ccm edu
Department Computer Science
Brown Universily
Reviewed on OpenReview: https: // openrevie.. net/ forum? id= PLNSI
Abstract
Vision-language (VL) pretrained models have achieved impressive performance multi- modal reasoning and zero-shot recognition tasks. Many of these VL models are pretrained on unlabeled image and caption pairs from the internet. In this paper, we study whether representations primitive concepts such as colors, shapes, or the attributes of object parts-emerge emerge automatically within these pretrained VL models. We propose two-step framework, Compositional Concept Mapping (Comp.Map), investigate this. CompMap first asks VL model to generate concept activations with text prompts from predefined list of primilive concepts, and then learns to construct explicit composition model that maps the primitive concept activations (e.g. the likelihood of black tail or red wing) to com- posite concepts (c.g. a red-winged blackbird). We demonstrate that a composition model can be designed as a set operation, and show that a composition model straightforward for machines learn from ground truth primitive concepts (as a linear classifier). We thus hypothesize that primitive concepts indeed emerge in VL pretrained model, its primitive concept activations can be used to learn composition model similar to the one designed by experts. We propose quantitative metric to measure the degree of similarity, and refer to the metric as the interpretability of the VL models' learned primitive concept repre- sentations. We also measure the classification accuracy when using the primitive concept activations and the learned composition model to predict the composite concepts, and refer to as the usefulness metric. Our study reveals that state-of-the-art VI. pretrained models learn primitive concepts that are highly useful for fine-grained visual rocognition on the CUB dataset, and compositional generalization tasks the MIT-States dataset. However, observe that the learned composition models have low interpretability in our qualitative analyses. Our results reveal the limitations of existing VL models, and the necessity of pretraining objectives that encourage the acquisition of primitive concepts 2
"odde available http::/://ggcabb..c.//tttttttttt wwtpsppiiiiiicceeppp.e..
Prroject homepage: https://vla priniiiivee hww.://itt.iieccoccoosss.tt.....

---

Published in Transactions on Machine Learning Research (04/2023)
Concept Prompting
Composition Learning
"wing
Predicted Concept Activations
wing color blue 0.20
wing black 0.82
bill shape needle 0.75
"bil shape needie""
Lineur Compocition Model
Pretrained Vision-and-Langee Lantgannd--Lagggg Model
wing color blue 0.10
wing black 0.89
bill shape needle 0.56
Predicted Concept Activations
"green violetear"
wing color is blue 0.20
color black 0.82
bill shape is needle 0.75
Figure 1: Illustration of our two-step framework, Compositional Concept Mapping (CompMap). In "Concept Prompting" module, a pretrained vision-and-langnagee (VT.) model is used to generate concept activations from images and & set of concepts, where each concept is represented by one or a few text prompts. In "Composition Learning" module, linear composition model is trained on concept activations (either pre- dicted by a pretrained VL model annotated by human experts) to predict the composite concepts (e.g. green violetear) with supervised lcarning.
1 Introduction
Vision-language (VL) models pretrained on raw images and text (Radford et al. 2021 Lu et al. 2019 Sun
et al. 2019) have revolutionized deep learning in the last few years. They have demonstrated strong transfer learning performance
wide range of tasks (Ramesh et al. 2021 , even under the zero-shot setup, where text prompts (Brown et al., 2020 used to specify task instead of labeled data. Many of these VL models are pretrained on naturally labeled data, such as image and caption pairs from the internet, and thus have the potential to encode commonsense knowledge by processing huge quantities of multimodal data. These models often learn to label complex concepts (c.g., an golden delicious apple, or a bohemian kingfisher) impressively well. However, it not clear whether they do this by implicilly learning to reason over lower-level primitive concepts (e.g., the color, shape, and texture of an apple, or the wing color or bill shape of bird) that humans naturally use to characterize these complex concepts. In this paper, we ask whether pretrained VL models capture representations of such primitive concepts "for free" in the course of their raetingni Tf they do so, it has important implications for the capacity of models to support compositional generalization, and for humans to interpret the reasoning procedures models undertake.
Why are we interested in the primitive concepts if a pretrained VL model can directly recognize composite concepts via prompts? For one reason, primitive concepts provide greater interpretability of and interaction with models (Ghorbani et al. 2019 Mao al. 2019). They can serve as concept bottlenccks (Koh et al. 2020), which allow users to inspect which concepts contribute the most to prediction or to correct model behaviors by correcting mistakes on the primitive concepts. An interpretable notion of primitive concepts also allows users to teach new concepts efficiently, by specifying how new concept (e.g. a red apple) can be derived from primitives (e.g. color red and object apple). Existing systems for these applications require manual annotation of the primitives to build corresponding classifiers [Lampert et al., 2009). Therefore, it would be particularly appealing pretrained VL models are able to learn the primitives automatically.
In this paper, we propose a computational framework, Compositional Concept Mapping (CompMap), quantify the extent to which VL. model has learned representations of primitive concepts. We focus
2

---

Published in Transactions on Machine Learning Research (04/2023)
composite concepts which can be mambiguuussy" described using a collection of non-overlapping primitive c The key motivation of our proposed framework is that this description can be obtained in a data-driven fashion, by learning a linear classifier from the primitive concepts to the composite concepts. We refer to this linear classifier as a composition model. When the true primitive and composite concept annotations are available for a dataset, one can reliably learn a composition model, which we validate experimentally. To align with the more common situation when the primitive concept annotations are not available, we ask & pretrained VL model to automatically annotate the primitive concepts via its natural language prompts (Radford et al., 2021). Intuitively, if VL pretrained model does learn primitive concepts, its concept annotations should allow us to learn the same very similar composition model. An illustration can be found in Figure !
We quantify this with two metrics. First, we measure the usefulness of a learned composition model (e.g., trained with red apple and yellow banana) for recognizing composite concepts (red banana), in terms of its ability to classify compositions of primitive concepts (red, banana). Second, we evaluate the interpretability of learned composition model. We build on our inluition above and make the assumption that, if VL pretrained model's primitive concept predictions are accurate and thus interpretable, it should allow the learning of a composition model that gencralizes well to the ground truth primitive concepts (o.g. provided by a human expert). Thus, we learn a "ground truth composition model" when given true primitive concept annotations as its inputs during training (Figure 2
top). We also learn a composition model with predicted concept activations from a VL pretrained model (Figure 2 middle). We then compute the performance gap between the two models when the true primitive concept annotations are provided during evaluation (Fig- ure bottom): The gap is small when the learned composition correctly maps the true primitive concept annotations to their corresponding composite concepts, as can be achieved by ground truth composition model. The composition concept mapping (CompMap) framework serves as proxy benchmark to measure how well pretrained VL models learn primitive concepts. Our benchmark does not require exhaustive prim- itive concept labeling on the evaluation dataset, since the true composition mapping can be designed by experts for each composite concept (e.g. a red bunana is red and a banana), or learned from a few training examples with true concept annotations.
We focus our study on three recent state-of-tho-art VI. models: (CLIP) (Radford et al. 2021 ViLT (Kim
2021), and ALBEF (i et al. 2021 They represent three categories of VL models: cross-moda fusion (CLIP), early fusion (ViL.T), and late fusion (ALBEF). To learn the composition models, we consider tasks where the target labels are composite concepts, such as in fine-grained visual recognition (Wahl ct al. 2011 (where the primitives are color, shape of the object parts) and object state recognition (where the primitives are objects and their attributes). Our study reveals both limitations and promises of VL. pretrained models. For limitations, we observe that the Iearned composition models are not interprotable, indicating that the model's conceptualization does not align well with how humans prefer think about these composite concepts. For promises, we find that the primitive concept activations preserve the visual information useful for visual recognition tasks, such as few-shot fine-grained visual recognition (Tokmakov
et al.
ct al. 2019 and zero-shot compositional learning (Purushwalkam et al. 2019).
To summarize, we make the following contributions: (1) We propose the Compositional Concept Mapping (CompMap) framework to quantitatively measure how well VL pretrained models learn primitive concepts, via the composition model learning task; (2) We perform extensive quantitative and qualitative studies based on our proposed framework, using range of recent. VL pretrained models; (3) Our analysis shows that composition models learned from pretrained VL models are not interpretable, indicating that the existing VL models do not learn interpretable primitive concepts. The models are nonetheless uscful for visual recognition tasks. Both findings highlight the importance to improve VL pretraining by learning interpretable primitive concepts. We will release the code and models used in the paper.
We acknowledge that this may not be applied to every composite concept in general. IIn this work, suppose that primitive concepts and composite concepts are relative terms, where each composite concept can be decompased into a set of descriptive primitive concepts (I.e., attributes).
3

---

Published in Transactions on Machine Learning Research (04/2023)
(a) A learned ground truth composition model trained with ground truth concept activations. During inference time, the model takes ground truth concept activations and predict the correct bird class.
Ground Truth Concept
Learned Ground Truth Composition Model
"green"
"red- "red-winged blackbird
"black"
"grenn"
"black"
"red"
"red"
rred-winged blackbird"
(b) A composition model learned from concept activations predicted by a vision-and ssioo-and-language (VL) model. The model is affected by the green background, and thus learns weight of 0.5 for concept "green".
Predicted
Concept Activations
Learned Composition Model
"green" 0.62
"red-winged blackbird
"black" 0.31
"green"
"black" "
"red"
W
...
"red" 0.05
"red-wing rred-winged blackbird"
After training this learned composition model with predicted concept activations. We intervene this model with ground truth concept activations during inference time. The model predicts the wrong bird class because the learned weights reflect positive correlation between the green background and red-winged blackbird.
Ground
Concept Activations
Learned Composition Model
"green"
"rusty blackbird"
"black"
"green" 0.5
"black" 0.95
"ved" 0.85
"red""
"red-winged blackbird"
Figure Illustration of a hypothetical scenario for composition learning, which aims to learn a lincar map- ping from primitive concepts (e-g. colors of a bird) into composite concepts (e.g, red-winged blackbird). While the composition model can be reliably learned given true concepts (Figure a), the task be chal- lenging with noisy concept predictions. In this example, a pretrained VI. model confuses the background color (green) with the wing color (black and red), resulting composition model that happens to make a correct prediction with wrong reasons (Figure b). We quantify this phenomenon with "intervention", which feeds the true concept activations to a learned composition model for evaluation (Figure c), and measure the performance gap between the learned composition model and the ground truth composition model.
2 Related Work
Vision and language pretraining. The fast evolution of deep learning hardware enables researchers train large deep neural networks web-scale data (Devlin et al. 2018 Brown et al. 2020 One data source is vision and language pairs from the internet, such images and captions (Sharma et al. 2018 videos and speech transcripts (Miech et al. 2020 Radford et al. 2021). These vision- datasets are usually not manually annotated. Representations learned by VL models can be transferred to a diverse range of tasks, such as image or video captioning (Lu et al.) 2019 Sun et al.) 2019) and visual question answering (Lu) et al. 2019 Bommasani al.) (021)). These pretrained models can directly recognize composite concepts, such human actions (Sun et al.) 2019)) or object categories (Radford al. (2021) with text prompts. To knowledge, make the first attempt to study pretrained VI. models learn primitive concepts.
4

---

Published in Transactions on Machine Learning Research (04/2023)
Primitive concepts and their representation. Based on the theory of concept that emphasizes symbolic representations, previous works found that neural networks satisfy multiple criteria of the theory, such as grounded behavior, but don't show a direct causal connection between the representations of constituent concepts and those of composite concepts. Visual and linguistic concepts are highly compositional and can be represented with primitive concepts. For human activities, the primitives are humans, objects, and their interactions (Ji et al.) 2020). For objects, the primitives can be parts (Hoffman & Richards 1984; Felzenszwalb al. 2008) and their attributes (Parikh & Grauman 2011] Kumar al., [009)] Lampert et al. 2009 Lovering & Pavlick 2022). For sentences, the primitives are often taken to be words and their grammatical relations (Socher et al. 2013).
To study the relations between the representations of composite and primitive concepts, measurements on compositionality have been proposed (Andreas] 2019 Keysers et al. 2019: Tokmakov et al. 2019). One
common technique is to rely on representation arithmetic, where the representation of a composite concept is expected to be reconstructed by its primitives (e.g. summation (Andreas) 2019)). This mcasurement requires having a fixed set of primitive concepts and knowing the composition model from primitive to composite concepts, both of which might not hold for the composite concepts observed by a pretrained VL model. Instead, our paper assumes that the primitives can be selected on-the-fly based on different sets of composite concepts, and the composition model can be learned by a linear classifier.
Using and learning primitive concepts. Primitive concepts have wide applications in machine learning models, such as neural symbolic reasoning (Yi et al. 2018 Mao et al. 2019 Andreas et al. 2016b a). few- shot (Tokmakov al., 2019) or zero-shot (Paikkh & Grauman 2011
Lampert et al. 2009) visual recognition,
building interpretable models (Kim et al., 2018; Ghorbani et al. 2019 Koh et al.) 2020), or simply as visual descriptor (Li et al., 2014]. They require annotations of the primitives to build primitive classifiers before the primitive concepts can be incorporated into the overall framework. In this paper, we are interested in whether the primitive concepts can be recognized by pretrained VL models via zero-shot prompts, a process that does not require any manual annotations, or task-specific prompt tuning (Yao et al. 2021
3 Method
This section describes our proposed framework to measure quality of the learned primitive concept predicted by VL pretrained models. We first introduce the two stages of our proposed framework, concept prompting and composition learning. We name our proposed framework Compositional Concept Mapping, or CompMap. We then discuss our concept prompting setup and our two evaluation benchmarks. An illustration of the overall framework can be found in Figure [
3.1 Concept Prompting
A typical pretrained VL model consists of a text encoder G and an image encoder F. Parts of the encoders can be shared to fuse vision-language signals. During VL pretraining, score s(x, t) is computed to mcasure the "compatibility" between an image and sentence t, such as with cosine similarity between the encoded F(z) and (t). The score should be high for the positive pairs of images and their corresponding descriptions and low for the negative pairs. One appealing property of vision-langua,, pretrained models is that they provide a natural language-based interface to perform visual recognition tasks. This can be achieved via natural language prompls, by asking a VL model to measure how well a prompt, such as "a photo [concept]", is compatible with the content of an image. These natural language prompts are usually evaluated in zero- shot image classification lasks, where composite visual concepts, such as fine-grained object categories in ImageNet (Deng et al. 2009), are provided candidate concepts to the VL model. Our paper follows the same concept prompting stratogy, and assumes that a VL model manages to learn primitive visual concepts, it can perform zero-shot primitive concept recognition via the corresponding natural language prompts. However, due to the lack of suitable benchmarks, it is challenging to directly evaluate the classification accuracy of primitive visual concepts.
We rely on two sources to obtain the vocabularies of primitive concepts: (1) Expert-defined primitive con- cepts, such the color, shape, texture of an object, or the attributes of object parts. An example is
5

---

Published in Transactions on Machine Learning Research (04/2023)
fine-grained bird species recognition; (2) Human-annotated composite concepts. For example, given com- posite concept ripe apple, we use its attribute ripe and object category apple as the primitive concepts. When picking the composite concepts (e.g. fine-grained bird species, or objects and their attributes), we focus on the concepts which can be unambiguously described using a collection of non-overlapping primitive concepts. More formally, we denole a general visual concept as e E C, a primitive concept. as p E Cp: and a composite concept as q E Cz. By definition we have Cp C,C, Cg C C, and Cp CC = 0. Prior work has worked on datasets with human-annotated images with primitive and composite concepts (Wah et al., 2011; Isola et al., 2015), while our goal is to validate if the primitive concepts can be "annotated" automatically with the VL. pretrained models. For a primitive concept p, we fit it into template sentence t(p). A concept representation can be computed with the score function ep = s(x, (p)), as illustrated in Figure | (left). We can repeal the process for all primitive concepts to obtain the predicted primitive concept activalions eored = [ei,..., ex]. In appendix, we study the impact of the choice of prompt template t(-).
3.2 Composition Learning
We denote the process of inferring composite concepts (e.g. fine-grained bird species) from primitive concepts (e.g. colors and shapes of a bird beak) as composition. In this work, we assume that the primitive concepts are composable, and there exists a true composition for any composite concept q, given an expressive set of candidate primitive concepts Cp. As illustrated in Figure 2 a true composition model can be representod in the form of a linear classifier, where the primitive concepts required for the composition are assigned with Wy = 1 (or other positive weights), and the primitive concepts to be ignored are assigned with w, = 0.
We are interested in composition learning: Given a set of paired primitive concept activations e along with composite concept labels, can we learn a composition model H that correctly maps the primitive concepts into the composite concepts, as achieved by the true composition model? Formally, we want to predict a composite concept q with a linear classifier He) = =w] e, where each element e, in e corresponds to how likely a concept ps is to be present, and can be given by an oracle as ground truth eat, or predicted by pretrained VL. model e (see Section 3.1 The classifier can be trained with a contrastive learning objective (Radford et al.) 2021), which pushes the positive pairs of activated primitive concepts and the composition of their corresponding composite concept to be more similar (Hg to be higher), and the negative pairs to be less similar. Given a fixed set of mutually-exclsstn composite concepts, this objective is the same as Softmax cross-entropy.
The composition model is regarded as a "proxy" to measure the quality of primitive concepts learned by VL models. We quantify the quality of the learned concepts with two metrics: usefulness and interpretability. First, the composition model can be directly evaluated for its usefulness accuracy when evaluated on unseen test set. Second, want to measure whether the composition model is interprelable, meaning that the primitive concepts necessary to derive a composite concept are selected by the model and that the learned composition model is not "right for the wrong reason".
classifier, i.c. the classification
Quantifying usefulness. Intuitively, we claim the learned composition model to be useful it can correctly map the predicted primitives erred into composite concepts. A useful composition model should sufficiently learn to compose these primitive concepts such that it would be able to derive unseen composite concepts correctly, even when deriving novel composite concepts. We use multiple evaluation metrics (e.g., classifica- tion accuracy) in composite concept recognition tasks to quantitatively measure the usefulness of a trained composition model; however this is not the main focus of the paper, but rather to validate that the lincar classifier adopted by our composition framework does not come at the cost of performance.
Quantifying interpretability. We assume (and validate experimentally in Section 5.1 that when the ground truth primitive concept activations et and their corresponding composite concept qet are available, the composition from primitive concepts to composite concepts can be reliably learned by a linear composition model Het(e) for any composite concept q (Figure a)))
If pretrained VL models do learn primitive concepts well, their predicted primitives eore should learn a composition model Hpred similar to Het based only on pairs of erred and q (Figure 2b)): Namely, Hpred
6

---

Published in Transactions on Machine Learning Research (04/2023)
should mostly pick the true primitive concepts necessary to compose 9, just like Wi does. We consider this "similarity" to the true composition as interprelability of learned composition model.
To quantitatively measure the interpretability of a composition model Hprod we compare its performance with the corresponding ground truth composition Hg. Rather than forcing Hpred to behave exactly the true composition for arbitrary inputs c, we relax the constraint and only require it to behave simi- larly on true primitives est. Formally, we define the interpretabilityy metric as: A = Metric (745 (est) - Metric (Hprod (e**), where Metric(-) is the performance evaluation metric such classification accuracy, We implicitly assumes that (e*t) would perform nearly perfectly (e.g. accuracy of 100%). Alternatively, we can normalize the interprelability metric by the performance of H (et)). Replacing the model inputs with ground truth concepts given by an oracle can be seen as a special case of model intervention (Koh et al. 2020 (Figure c))
Discussion. A possible alternative to quantitatively measure if VL pretrained models learn primitive concepts is to directly evaluate models' performance when predicting all primitive concepts. However, this evaluation requires exhaustive labeling of primitive concepts on given image, which is often not available (e.g. an image of "red apple" in MIT-States (Dosovitskiy et al.] 2021) can also be "ripe" or "round", which are not labeled.). Therefore, the composition model learning and the usefulness and interpretability metries serve as proxy benchmarks to measure how well pretrained VL models learn primitive concepts: they offer two aggregated metrics by jointly considering the whole set of primilive concepts, as opposed to measuring individual "performance" on each primitive concepts.
3.3 Benchmarks
We consider two image classification benchmarks to learn the composition models, and mcasure model usefulness and interpretability. Our desired benchmarks should provide: (1) annotated image-level labels that are composite concepts; (2) vocabulary for the relevant primitive concepts.
Compositional zero-shot lcarning (CZSL) by Isola et al. (2015) aims to evaluate if a classifier trained to predict composite concepts generalizes to novel compositions, where each composite concept is a pair of object attribute (red) and category (apple). It provides challenging setup for to evaluate the usefulness and interpretability of composition model.
Given the set of composite concept labels C during training and C' during evaluation, CZSL considers the scenario when C * C², but both share the same set of object attributes and categories. We consider the generalized CZSL setup, where C* is a true subset Ct, We also consider both the "orldd" version and the "open-world" version of generalized CZSL. In the closed-world setting, Ct is constraint to be a relatively small set of composite concepts. In the open-world setting, C contains every combination of object attribute and category, and is therefore much more challenging due to the large output space.
Fine-grained visnal recognition. We evaluate whether true compasition model could be learned in a few-shot or full-shot setting for fine-grained visual recognition tasks (Wah et al. 2011). Our intuition is that fino-grained concepts could potentially be defined by the details given by primitive concopts, such as shape or color of bird body parts. We adopt the standard few-shot learning setup, in which the composition model needs to discriminate n composite concepts by training on only & images from each composite concept.
4 Experimental Design
In this section, we introduce our datasets and metrics for the datasets (Section 4.1 and implementation details (Section 4.2 We elaborate how we collect concept activations with different model architectures
(CLIP RRadford et al. 2021). ViLT (Kim et al. 2021) and ALBEF (Li ct al. 2021 in Appendix A.2
4.1 Experimental Setup
Datasets. We conduct experiments on two image classification benchmarks where the target labels are composite concepts. The first dataset. is MIT-States (Dosovitskiy et al. 2021), which contains 53K images of

---

Published in Transactions on Machine Learning Research (04/2023)
115 attributes and 245 objects. Each image is labeled with (object, attribute) tuples. We use the standard splits from Purushwalkam et. al. (2019). The training split has 30K images of 1262 seen altribute-object compositions, the validation split has 10K images of 300 seen and 300 unseen compositions, and the test split has 13K images of 400 seen and unseen compositions. We report results on the validation and test splits.
The other dataset is Caltech-UCST Birds-200-2011 (CUB) (Wah et al., 2011), which contains 11788 pho- tographs of 200 mainly North American bird species. We use the standard training/testing split from Hilliard et al. 2018). Each image is also annotated with attributes corresponding to 28 different categories, such as throat color, wing shape, etc. There are 312 possible binary attributes in total. We adopt the attribute denoising setup from Koh et al. (2020).
Metrics. For the MIT-States dataset, we follow the evaluation protocol from Purushwalkan et al. (2019) whose goal is to mitigate models' bias on seen compositions. The evaluation protocol has four metrics: (I) Unscen-Seen Arca Under the Curve (AUC), (2) best accuracy on data samples of seen compositions (best seen), (3) best accuracy on data samples of unseen compositions (best unseen), and (4) best harmonic mean (best HM) of seen and unseen accuracy (2, 3). For the CUB dataset, we follow the standard practice in n--ay k-shot evaluation (Snell et al. 2017) and report mean accuracy over the 600 sampled tasks run for each setup. In each task, the classes and k examples are chosen at random. All metrics mcasure usefulness, where higher values correspond to being more "useful".
Since a composition is trained only on seen composite concepts, the seen concepts can evaluate much better or worse than the unseen ones on validation and test splits. We follow Purushwalkam et al. (2019) and apply calibration biases, which are scalars to be added to the predicted scores of unseen concepts (See Appendix A.6.2 for more details). Specifically, with larger bias, the accuracy of unscen concepts tends to increase, while that of seen concepts lends to decrease. We vary the values of the calibration bias, and compute a list of accuracy scores of seen concepts and another list for unseen concepts. AUC is the area under the curve of unseen accuracy and seen accuracy; Best seen and best unseen are the best accuracy in the lists of accuracy scores of seen and unseen concepts. Best harmonic mean is the highest harmonic mean computed by
HM = ACC seen ACC unseen
(1)
The interpretability is then quantified as the gaps between ground truth composition model and a learned composition model for the above metrics.
4.2 Implementation Details
We choose three recently open-sourced VL models: CLIP (Radford et al.) 2021), ViLT (Kim et al. 2021 and ALBEF (Li el. al., 2021) for this study. They represent three calegories of VL models: no cross- cross-modal fusion (CLIP), early fusion (ViL.T), and late fusion (ALBEF). For CT.IP, we use the pretrained CT.IP with ViT-B/32 (Dosovitskiy et al.) 2021) visual encoder and a transformer-basee encoder. For ViLT, we use ViLT-B/30 pretrained with masked language modcling and image-text matching objectives. For ALBEF,, we use ALBEF with a ViT-B/16 (Dosovitskiy et. al.] 2021 visual encoder and a BERT (Dvvlnn et al. 2018) text encoder. This checkpoint is trained on Conceptual 12M dataset (Changpinyo et al.
(Vaswani et al. 2017) text
2021). We follow the default setups to compute the concept activations.
For MIT-States, contrastive objective is used to train the composition models. We apply two linear projection layers on the primitive concept activations and the text emboddings of target composite concepts respectively, to embed them into shared space. This approach works with both closed-world and open- settings of CZSL, and maintains the linear composition assumption. During training, subset of composite concepts are supposed to be unknown, so we remove the unscen composite concepts (i.e. |C*1 = 1,262 for training; |C*| = 1,962 for closed-world evaluation, and |C'| = 28,175 for open-world evaluation).
For CUB, a logistic regression model is used to train the composition models since the number of target composite concepts fixed. We used the default sklearn hyperparametors and L2 regularization of 1 for all experiments. We observed that the performances are robust against the choice of logistic regression hyperparameters. Due to annotation inconsistencies, follow Koh et al. (2020) and denoise the primitive
8

---

Published in Transactions on Machine Learning Research (04/2023)
Table 1: Results of CZSL task on MIT-States in the closed-world setting, AUC (%) are computed using precision at k=1,2,3. Best seen and unseen accuracies, and best harmonic mean of the two are reported. Composition models are learned from ground truth primitive concepts e (first row), or from VL model concept prompting epred Training and Evaluation columns spocify model inputs during training and in- ference time respectively, where Prim stands for primitive concept activations, and GT stands for ground truth primitive concept activations. During inference time, when both Prim and GT are checked, it means ground truth primitives are activated, and all dimensions not activated by the ground truth primitives are left as the primitive concopt activations. Usefulness is mcasured when the same predicted concopts e is used for evaluation (rows blue). Interpretability is measured as the performance gap from learned GT composition model (Oracle-Prim) when est is provided "intervention", (See Table 3 for more detailed analysis of this performance gap.)
Training Prim GT
Evaluation Prim
Val AUC 2
Test AUC
Seen
Unseen
HM
Method
GT
k=1
2
Oracle-rrim
99.9
99.7
99.7
99.9
99.9
99.9
100.0
100.0
99.9
CLIP-Prim
8.2
17.0
24.1
6.9
15.6
22.8
34.0
27.9
20.4
CL.P-InteryyFull
32.2
49.1
58.0
30.0
49.3
59.6
52.3
66.0
47.7
(LIP-InterrPPartia)
v
35.8
53.4
62.9
32.6
53.7
63.2
54.1
68.2
49.4
ViLT-Prim
4.7
11.1
16.9
3.8
9.7
15.2
25.4
20.8
15.1
ViL/T-IntervFFul)
8.6
17.1
23.6
4.1
10.2
14.4
21.7
24.8
16.3
ViL:T-nntervParrtia)
20.1
33.8
43.1
12.9
25.5
35.1
39.0
41.8
28.7
ALBEF-Prim
7.1
15.6
23.1
5.8
13.9
21.2
32.9
25.1
18.5
36.8
52.7
63.6
37.9
57.2
64.8
58.5
71.6
53.0
ALBEF LIBEE-Interr(PPartia)
38.7
56.5
68.3
41.1
61.8
70.6
61.3
73.0
55.8
concept annotations via majority voting. Thus, we use class-level concepts, which means that the bird attributes of the same bird species are always the same, regardless of the images of birds.
We design two sets of prompts to extract attribute/object, pparr activations from the VL models. For MIT States, e.g., "this is (ripe)", "this is {apple)", and "this {ripe apple)" are used to produce the correspond- ing activations. For CUB, we compute attribute and class activations by "a photo of bird whose {bill shape} is [needle))" and "the bird is (bohemian warwing)"" These two CUB prompts have different templates since we observe some bird attribute categories can only be understood with the context of birds (c.g. size and shape of birds), and thus we include the language prior "bird" in both prompts. We explore more prompt templates in Appendix A.4 and find our observations to robust.
5 Results
5.1
Interpretability of Learned Composition Models
We first explore if a composition model learned from concept activations is interpretable. We approach this problem by inspecting if the linear classifier learns true composition for the composite concepts or it solves the task with some spurious correlations between the activations and the composite concepts. The true composition is learned from ground truth primitives, while a learned composition is learned from concept activations predicted by VL models.
We intervene on the learned compositions in two ways. The first method is intervention with ground truth primitives (Tntero(Full),, which replaces the primilive concept activations with the binary ground truth prim- itives during inference. The comparison between Interv(Full) and a corresponding ground truth composition can reflect how feasible The second intervention only on attributes activated by the ground truth primitives (Intero(Partial).. This method is similar to Intero(Full), but all dimensions not activated by the ground truth labels are left
to approximate a ground truth composition from primitive concopt activations.

---

Published in Transactions on Machine Learning Research (04/2023)
Table 2: Results of n-way k-shot evaluation on CUB. Mean per-sample accuracy scores are reported. Com- position models are learned from ground truth primitive concepts ext (first row), or from VL model concept prompting eorod, Training and Evaluation columns specify model inputs during training and inference time respectively, where Prim stands for primitive concept activations, and GT stands for ground truth prim- itive concept activations. During inference time, when both Prim and GT are checked, it means ground truth primitives are activated, and all dimensions not activated by the ground truth primitives are left as the primitive concept activations. Full Shot column refers to a 200-way full-shot evaluation that considers all the available data in CUB. Usefulness is measured when the same predicted concepts e is used for evaluation (rows in blue). Interpretability measured as the performance gap from learned GT composition model Prrall-Prrmy when eät is provided as "intervention".
Training Prim GT
Evaluation Prim GT
n
5
n
100
200
Full Shot
Method
k=1
5
1
5
1
5
Oracle-Prim
99.8
100.0
99.9
99.9
98.9
98.9
98.0
97.9
98.0
CLIP-Prim
76.5
87.3
60.9
78.0
21.0
43.0
13.6
33.2
52.6
CLIl-Interv(Full
61.6
66.0
43.1
50.8
9.4
13.4
5.1
7.8
8.0
CLIP-inter(Parriaa)
/
75.4
85.4
59.4
75.2
17.5
32.9
10.9
23.4
32.9
ViLT-Prim
v
/
70.0
83.4
54.0
71.3
14.8
28.8
8.9
19.7
40.1
ViLT-Interv(Full)
54.3
60.7
37.7
45.7
6.5
8.6
3.5
4.7
9.2
ViLTTinterr(Partiaa)
64.9
75.8
48.0
60.0
9.6
15.6
5.3
9.4
19.3
ALBEF-Prim
73.2
85.2
58.3
74.4
17.0
36.3
10.7
27.2
44.6
ALBEFF-nterrxFuul
57.0
63.9
39.3
46.5
7.0
9.5
3.7
5.2
7.6
ALBEF-InterV (Pattv(Partial)
69.9
80.0
52.4
66.3
12.0
24.6
7.0
16.9
30.2
Table 3: Measuring interpretability A on MIT-States (Left) and CUB (Right). A is the performance gap between learned GT composition model (Oracle- Pridee and Interv(Full). On CUB, we show the A for n-way 5-shot evaluation. The lower A is, the more interpretable the composition model is.
A
A
Model
AUC
Seen
Unseen
HM
Model n
5
100
200
FS
CL.IP
69.9
47.7
34.0
52.2
CLIP
34.0
85.5
90.1
90.0
ViLT
95.8
78.3
75.2
83.6
ViLT
39.3
90.3
93.2
88.8
ALBEF
62.0
41.5
28.4
46.9
ALBEF
36.1
89.4
92.7
90.4
the activations predicted by the VL model. The comparison between Intero(Partial) and Interv(Full) shows the spurious correlations learned between non-activated primitives and composite concepts.
Table and Table 2 summarize the results on MIT-States and CUB. First of all, the composition models learned from ground truth primitive concept activations est and their corresponding composite concept lgt perform almost perfectly ( (Oracle-Prim). This validates our hypothesis that a true composition can be learned from e and 9gt- We also observe two consistent patterns on both CZSL, and few-shot learning tasks. First, with Intere(Full) intervention, all models perform worse than the ground truth composition (Table D. which generally reaches perfect performance. This implies that the predicted primitive concept activations and the composition models learned from them not interpretable. Second, by comparing Intere(Partial, intervention with Interv(Full), all models increase in performance, indicating that the models utilize activations from the irrelevant primitive concepts to predict the composite concepts.
When we compare Prim with Intero(Full), we see that Interv(Full) performs better than Prim in MIT-States, but worse than Prim in CUB. This indicates that the learned compositions in MIT-States are able to identify positive correlations between the composite concepts and their primitives, while the learned compositions in CUB not able to identify such positive correlations sufficiently. We attribute this difference to the
10

---

Published in Transactions on Machine Learning Research (04/2023)
Table 4: Quantitative analysis of the learned weights of ground truth composition learned from binary ground truth primitive concepts and the learned weights of compositions learned from concept activations predicted by CLIP. Accuracy measures whether the most activated weights align with the ground truth primitive concepts. If there are & (I: 1) ground truth primitive concepts, we consider the top k most activated weights. Instance-level accuracy (Acc-Instance) considers each aligument between the most activated weights with the ground truth weigths as individual instance. Class-level accuracy to correlate all the primitive concepts of composite concept correctly. Acc-Attribute--Ohiee to the instance-level accuracy over attributes and objects in MIT-States. There are significant gaps between the learned ground truth composition models and composition models learned from CLIP.
seceInnttancccoosdders
requires composition model corresponds
MIT-States
CUB
Method
Acc-Instance
Acc-Attribute
Acc-Objec Object
Acc-Class
Acc- Instanst
Acc-Class
Oracle-Prim 99.6
99.5
99.8
99.3
96.5
97.0
CLIP-Prim 51.4
34.3
68.5
22.5
37.6
0.0
gap of the complexity in these two tasks, as a composition model for MIT-States only needs to capture two primitives (an attribute and an object) for each composite concept, while composition model for CUB must capture multiple primitives for each bird class.
5.2 Analyses of Composition Models
We now directly analyze the weights of the learned composition models. We consider an accuracy metric, which measures the percentage of the primitive concepts with the highest learned weights actually belong to the true composition model. The results are shown in Table 4 The high accuracy scores (e.g. class- level accuracy of 99.3% for MIT-States) for Pracle-Prim show that these models can learn the correlation between the primitive concepts and the composite concepts almost perfectly. We observe that there is significant gap between Oracle-Prim and the composition models learned from CLIP-predicted primitive concept activations (EII-PPrim) indicating that the learned CLIP- Prim models' behaviors are not similar to the true composition model (see Figure 3). We also notice that CLIP-Prin reaches class-level accuracy of 0% on CUB (versus 97.0% for Oracle-Prine We attribute this to the complexity of CUB, since composition model needs to correlate all 28 bird attribute categories correctly for each bird class, which is significantly more difficult than MIT-States (only primitive concepts attribute and object). This may also explain the performance degradation of (LiP-Intero(Full) from CLIP-Prim on CUB (Table 2
In order to understand if extracting the primitive concepts related small regions is the bottleneck, we carry out a quantitative analysis on CUB dataset to study the relationship between the region size of the concepts and the correlation between primitives and composite concepts. The correlation is quantified by accuracy metric, which measures whether the most activated weights align with the ground truth primitive concepts. We observe no strong correlation between the region size of the primitive concepts and how well CLIP-Prim can correlate them to the composite concepts (Figure 4 For instance, even though bird wings are larger than their napes, the accuracy on nape color is significantly higher than on wing color (56.4% versus 41.7%).
5.3 Usefulness of Composition Models
We look into the classification performance based on the primitive concept activations. The primitive concepts are deemed useful if the composition learned on top of them achieves good classification performance.
Table shows the results on MIT-States in closed-world setting. We observe that the zero-shot CLIP already achieves on par or better performance than the previous approaches, which confirms the observation that CLIP learns composite concepts. When we use a pretrained VL model with our CompMap framework, the performance of the composition models perform competitively against previous state-of-the-art methods (i.e., CLIP-Prin has test AUC of 6.9), even though our CompMap has very simple architecture. With
11

---

Published in Transactions on Machine Learning Research (04/2023)
wing color
head pattern
nape color
primary color
crested
macked
eronge
black
orange
0.30
black
black
0:13
033
0.00
"crested auklet"
0.0
syering
pavy
0.25
plain
0.4
114
0.10
"california seagull"
Figure 3: Visualization of the learned weights of composition model learned from CLIP- -LIP-prodicted concept activations (CIP-Prrmm on CUB dalaset. X-axes stands for primitive concepts. Y-axes shows the likelihood for primitive concept to be correlated with composite concept. Tn each subplot, the blue bars highlight the ground truth primitive concepts. We observe that CLIP-Prim models are not similar to the true composition model, which can correlate primitive concepts for each composite concept correctly.
0.6
0.5
0.4
Accuracy
0.3
0.2
0.1
upper
back
nape
primary
crown
breast
under
forehead
upperparts
belly
throat
underparts
wing
leg
bill
Figure Analysis of the correlation between region size of primitive concepts and how well a learned composition model (CLIP-Prim) can match the primitives to the composite concepts on CUB dataset. This analysis focuses on color-rclated concepts. X-axis stands for primitive concepts. Y-axis shows the accuracy metric, which measures whether the most activated weights align with the ground truth primitive concepts. Red dotted line is the random baseline. We observe strong correlation between region size of primitive concepts and how well CLIP-Prim can match the primitives to the composite concepts.
12

---

Published in Transactions on Machine Learning Research (04/2023)
Table 5: Results of generalized CZSL on MIT-States in the closed- world setting, We observe that CLIP with Ca performs the best among the selected VL models, and it outperforms previous methods.
Val AUC
Test AUC
Seen
Unseen
HM
Method
Top
1
2
AOP Nagarajan Grauman
2018
2.5
6.2
10.1
1.6
4.7
7.6
14.3
17.4
9.9
LE+
Nagarajan an Grauman
2018
3.0
7.6
12.2
2.0
5.6
9.4
15.0
20.1
10.7
TMN
Purushwalkam al.
2019)
3.5
8.1
12.4
2.9
7.1
11.5
20.2
20.1
13.0
SymNet
(Li et al. 2020
4.3
9.8
14.8
3.0
7.6
12.3
24.4
25.2
16.1
CompCos [Mnniin et al.
2021
4.5
25.3
24.6
16.4
CGE
Naeem et al. 2021
6.8
5.1
28.7
25.3
17.2
CLIP
(zero-shot)
5.8
13.0
19.1
5.5
12.7
18.8
25.0
30.5
18.3
CLIP-Prim
8.2
17.0
24.1
6.9
15.6
22.8
34.0
27.9
20.4
ViLT-Prim
4.7
11.1
16.9
3.8
9.7
15.2
25.4
20.8
15.1
ALBEF-Prim
7.1
15.6
23.1
5.8
13.9
21.2
32.9
25.1
18.5
CompMap framework, we also observe that CLIP outperforms ALBEF, and ALBEF outperforms ViLT, which matches to the order of their pretraining data size.
Table 6: Ablation study investigate whether the usefulness of concept activations comes from the prim- itive concept prompting, the powerful CLIP visual encoder. We compare CLIP-Prim, which applies projection matrix generated based the text prompt embeddings, with the direct application of CLIP image embedding, and learned or random projection of the CLIP image embedding.
Val AUC
Test AUC
Seen
Unseen
HM
Method
Top
+
2
3
CLIP-Prim
8.2
17.0
24.1
6.9
15.6
22.8
34.0
27.9
20.4
CLIP Image Embed.
8.6
17.4
26.7
7.0
15.8
23.4
36.1
26.4
20.9
CLIP Image Embed. + Learned Projection
7.9
16.4
26.0
6.2
14.8
21.6
34.5
24.9
19.3
CLIP Image Embed. + Random Projection
7.5
16.1
23.0
6.2
14.2
21.2
34.5
25.1
19.3
Table A1 shows the results on MIT-States open-world setting. We mainly focus on CLIP due to its superior performance the wlosed-world setting. With a linear composition model trained on top of concept activations predicted by CLIP, all learned compositions outperform the previous state-of-the-art approach. However, most improvements come from better learning the seen concepts in the training split but not necessarily from better generalizabiliy, to the unseen concepts, implying that the learned compositions are still distracted by the infeasible attribute-object pairs.
Table A2 shows the results on CUB 5-way k-shot learning setting. We observe that the learned composition with CL.P again performs the best among the selected VL. models. Given that VI. models with CompMap have simple model architecture, they still perform competitively against the previous saat-oof-hh-aat in the 1-shot and 5-shot setting, implying that the predicted concept activations by pretrained VL models are useful for such few-shot learning problems.
Finally, we conduct an ablation study to investigate if the uscfulness comes from the application of concept prompting, or comes from the power of raw CLIP image embeddings. We consider three scenarios: First, we directly use the CLIP image embedding with dimension of 512, which has higher dimensions than the primitive concept activations (i.c., dimension of 360 for MIT-States). Since concept prompting can be viewed as applying a projection matrix on the CLIP image embedding, we also consider two alternatives for the projection matrix used by CI.P--rrm,, which is generated based text prompts: end-to-end learned
13

---

Published in Transactions on Machine Learning Research (04/2023)
projection, and frozen random projection. In Table 0 we observe that CLIP-Prim only moderately outperforms the learned and random projection baselines, indicating that the usefulness mainly comes from the CLIP visual encoder, and not from the primitive concept prompting, The results are in line with our carlier observation that the learned composition models are not interpretable according to our metrics.
6 Conclusions
In this paper, we have proposed a framework for measuring how well vision-language pretrained models can learn primitive concepts, in terms usefulness and interpretability. By conducting extensive experiments with recent VL pretrained models CLIP, ViLT, and ALBEF- we observe that these models are able to learn primitive concepts that are useful for visual recognition tasks, but the learned composition models from primitive concepts to composite concepts are often not interpretable. Our study suggests that despite the strong performance of VL pretrained models for visual recognition tasks, more research is needed to improve their ability to better caplure primitive concepts. We anticipate our framework serve meaningful step towards understanding the working mechanisms (and potentially biases) of VL pretrained models.
Limitations. Our interprotability metric measures the unnormalized performance gap between the compo- sition models learned on ground truth primitive concepts (Oracle-Prim) and concept activalions generated by pre-trained VL models. It implicitly assumess that Oracle-Prim achieves near perfect performance when predicting the composite concepts. When the assumption is not true, solution is to normalize the inter- pretability metric by the performance of Oracle-Prim. Our work also makes the assumption that the concept activations predicted by pretrained VT. models are well-calibrated. However, this may not always hold true (example shown in Appendix A.1). A possible solution for this mis-calibration issue is to post-process the predicted concept activations. We leave this as a future work.
7 Acknowledgements
We would like to thank all reviewers and the action editor for their valuable feedback. We would like to thank Calvin Luo and Lishuo Pan for their discussions and insights. This work is in part supported by Adobe, Meta AI, Samsung, and Richard B. Salomon Faculty Research Awards for C.S.
References
Jacob Andreas. Mcasuring compositionality in representation learning. In ICLR, 2019.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. arXiv preprint arXi::1601.01 01700000,,,, 2016a.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In CVPR, 2016b.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXic:2108.0722 2021.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint Xix::0000 14165, 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Concept 12m:
Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 3558 3568. Computer Vision Foundation / IEEE, 2021.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
14

---

Published in Transactions on Machine Learning Research (04/2023)
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprinl or 1rX1i:1810.0440 04805, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Pedro Felzenszwalb, David McAllester, and Deva Ramanan. A discriminatively trained, multiscale, de- formable part model. In CVPR, 2008.
Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-baset explana- tions. aix preprint orXiv:1902.0311 2019.
Nathan Hilliard, Lawrence Phillips, Scott Howland, Artem Yankov, Courtney D Corley, and Nathan O Hodas. Few-shot learning with metric-agnostic conditional embeddings. arxiv preprint or 1 802.00020000 2018.
Donald D Hoffman and Whitman Richards. Parts of recognition. Cognition, 1984.
Yuqing Hu, Vincent Gripon, and Stéphane Pateux. Squeezing backbone feature distributions to the max for efficient few-shot learning. arXiv preprint orXiv:2110.000 10.09446, 2021.
Phillip Isola, Joseph J. Lim, and Edward H. Adelson. Discovering states and transformations in image collections. In CVPR, 2015.
Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In CVPR, 2020.
Dahyun Kang, Heesoung Kwon, Juhong Min, and Minsu Cho. Relational embodding for few-shot classifi- cation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8822-8833, 2021.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momehev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. Mcasuring compositional general- ization: A comprchensive method on realistic data. arXiv preprint arXiv:1912.00 09711, 2019.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In ICML, 2018.
Wonjae Kim, Bokyung Son, and Todoo Kim. Vilt: Vision-and-languuage transformer without convolution region supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 5583-5594. PMLR, 2021.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In JCML, 2020.
Neeraj Kumar, Alexander C Berg, Peter N Belhumeur, and Shree K Nayar. Attribute and simile classifiers for face verification. In ICCV, 2009.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In CVPR, 2009.
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Decpak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NewrIPS, 2021.
15

---

Transactions of the Published in Transactions on Machine Learning Research (04/2023)
Li-Jia Li, Hao Su, Yongwhan Lim, and Li Fei-Fei. Object bank: An object-level image representation for high-level visual recognition. IJCV, 2014.
Yonglu Li, Yue Xu, Xiaohan Mao, and Cewu Lu. Symmetry and group in attribute-object compositions. In CVPR, 2020.
Charles Lovering and Ellie Pavlick. Unit testing for concepts in neural networks. Association for Computational Linguistics, 10:1193-1208, 2022.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lec. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-andd-lnnuuage tasks. arXiv preprint arXiv: 092.00265, 2019.
Massimiliano Mancini, Muhammad Ferjad Nacem, Yongqin Xian, and Zeynop Akata. Open world composi- tional wero-shot learning. In CVPR, 2021.
Puneet. Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vineeth N Balasubramanian. Charting the right manifold: Manifold mixup for few-shot learning. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp.
2020.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904. 12584, 2019.
Antoine Miech, Jean-Baptisle Alayrac, Lucas Smaira, Ivan Laplev, Josel Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, 2020.
Muhammad Ferjad Naeem, Yongqin Xian, Federico Tombari, and Zeynep Akata. Learning graph embeddings for compositional zero-shot learning. In CVPR, 2021.
Tushar Nagarajan and Kristen Grauman. Attributes as operators: Factorizing unseen attribute-object compositions. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), ECCV, 2018.
Devi Parikh and Kristen Grauman. Relative altributes. In ICCV, 2011.
Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc' Aurelio Ranzato. Task-driven modular networks for zero-shot compositional learning. In ICCV, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Nya. Sutskever. Learning transferable visual models from natural language supervision. In JCML, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.112 12092, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hyper- nymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for few-shot learning. arXiv preprint 170.0:1703.051 2017.
Richard Socher, John Bauer, Christopher D Manning, and Andrew Y Ng. Parsing with compositional vector grammars. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 455-465, 2013.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid, Vidcobert: A joint model for video and language representation learning. In ICCV, 2019.
Pavel Toknakov, Yu-Xiong Wang, and Martial Hebert. Learning compositional representations for few-shot recognition. In ICCV, 2019.
16

---

Published in Transactions on Machine Learning Research (04/2023)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998 6008, 2017.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-2011 dataset. Tochnical Report CNS-TR-2011-001, California Institute of Technology, 2011.
Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. arXiv preprint arXie:2109.11997 2021.
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recog- nition, pp. 8808 8817, 2020.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding, arXiv preprint arXiv:1810.0200 2018.
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's s distance and structured classifiers. In Proceedings of the IEEE/CVE conference on computer vision and pottern recognition, pp.
2020.
17

---

Published in Transactions on Machine Learning Research (04/2023)
A Appendix
A.1 Mis-calibration Example
We would like to thank the reviewers for this mis-calibration issue and example. When there are two VL models: A and B. Model A predicts well-calibrated and accurate values (e.g., for red birds, predicts (red: 1, black: 0}). Model B predicts relatively smaller values for the correct concepts (e.g., for red birds, it predicts fred: 0.8, black: 0.2); for black birds, it predicts (rod: 0.2, black: 0.8)). We train composition models using the predicted concept activations from models A and B. For example, their composition models both learn a set of weights {red: 1, black: 0} for red birds, and correspondingly for black birds. Therefore, both composition models reach the same interpretability scores during the intervention. However, based on the predicted concept activations, model A understands primitive concepts better than model does. One possible solution for this mis-calibration issue is to post-process the predicted concept activations.
A.2 Concept Prompting with CLIP, ViLT and ALBEF
2021 is a vision-and-language (VL) pretrained models without cross-modal fusion layers. CLIP learns a textual encoder and an image encoder via the contrastive learning objective. It does not need an explicit architecture component to fuse visual and language signals.
CLIP (Radford et al.
ViLT (Kim et al. 2021 is a single-stream early-fusion VL pretrained model that simplies the visual embed- ding pipeline. The model takes a concatenated sequence of pair of text sequence and image sequence (of patches) as input, and is trained with image-text matching (TTM), masked language modeling (MLM), and word patch alignment objectives.
ALBEF [Li et al. 2021 is a dual-stream late-fusion VL pretrained model that learns a text encoder, an image encoder, and a multimodal encoder. The text input and image input are passed into their corresponding encoders, and the multimodal encoder has a cross-attention layer that fuses the encoded text input and encoded image input together. ALBEF is trained with image-text contrastive learning (TTC), image-text matching (TTM), and masked language modeling (MLM) objectives.
Given set of primitive concepts {pI..... and an image I, obtain concept activation eree of image & with each of CLIP, ViLT, and ALBBE.. For CLIP, we collect the text embedding of the prompted text of a primitive concept pt, and the image embedding of image x. We treat the dot product between the image embedding and text embedding as e, We then repeat the process for all primitive concepts to obtain epred [N,] For ViLT and ALBEF, we extract the ITM score for each concept ps and treat each ITM score as 6;- We then repeat. the process for all primitive concepts to obtain erred = [ei,..., We notice that computing concept activations for ViLT is significantly time-consuming due to its single-stream model input format a concatenated sequence of text and image inputs. We cannot precompute any text image features, and thus need to pass the concatenated sequences for each single ITM score. For dual-stream models (CLIP and ALBEF), we directly cache the encoded images and encoded primitive concepts, and then compute the concept. activations based on these precomputed features.
A.3 Usefulness of Composition Models
Table A1 shows the results of generalized CZSL MIT-States in the open-world setting. Table A2 shows the results of 5-way k-shot learning on CUB. Results both tables demonstrate that the learned primitive concept activations are useful for the image classification tasks. More details are discussed in Section 5.3
A.4 Ablation Study of Prompts
We carry out an ablation study to explore the impact of prompts. Table A3 shows the prompts collected from CLIP. This of prompt templates is the best performing subset on ImageNet Deng et al. (2009) over 80 prompt templates.
18

---

Published in Transactions on Machine Learning Research (04/2023)
Table A1: Results of generalized CZSL on MIT-States in the open-world setting for CLIP with CompMap. We observe that CLIP with Co outperforms previous methods on AUC, but it does not necessarily generalize well unseen composite concepts.
Val AUC
Test AUC
Seen
Unseen
HM
Method
Top k
1
2
3
1
2
3
CompCos
(Mancini et al. 2021
/
/
/
1.6
/
/
25.4
10.0
8.9
CLIP-Prim
2.6
6.0
9.1
2.1
5.2
8.0
33.7
9.9
10.8
Table A2:
Results of 5-way k-shot learning on CUB. Mean accuracy across 600 sampled tasks for CompMap- All are reported for CLIP, ViLT, and ALBEF. CompMap provides competitive performance.
Method
5
1-shot
5-shot
FEAT YYo 2020)
73.3
85.8
DeepEMD Zhang al. 2020
75.7
88.7
RENet
Kang al. 202
79.5
91.1
S2M2 R (Mangla et all 2020
80.7
90.9
PEM,-NCM (Hu et al.
2021
80.8
91.5
CLIP-Prim
74.1
87.5
ViLT-Prim
69.9
84.0
ALBEF-Prim
74.1
86.3
A.4.1 Different Prompt Templates
We first explore how the usefulness and interpretability of learned composition models vary with different prompt templates. We perform concept prompting based on each prompt in Table A3 We then train (7 =) 14 composition models in total. We these models to investigate how robust observations in the main paper are.
Tables A4 and A3 show the results for usefulness as measured on MIT-States in dlosed-oord and open-world settings respectively. We observe that the aggregated results have slightly higher mean values than what were reported in the main paper, and the standard deviations are small. This confirms that our observations are robust and the choice of prompts does not matler much for usefulness of the learned compositions. The gains over the best performing baselines are significant considering the low standard deviations in both settings.
Table A6 shows the results with or without intervention, protocol we use to measure interprelabilily of the learned compositions. We observe that the results with intervention generally exhibit higher variances, indicating that the choice of prompts matters more for the interpretability of the learned compositions. However, the aggregated results are still far from those of the ground truth composition.
A.4.2 Using More Prompts
We further examine the impact of using more prompts. Following what CLIP does to combine prompis, we combine text embeddings from multiple prompts together. We vary the number of prompts to be 1, 4, or 7, and train their corresponding composition models. In Table A7 we observe a positive correlation between the number of prompts used to generate the concept activations and the interpretability of the learned compositions. However, the impact usefulness is marginal.
19

---

Published in Transactions on Machine Learning Research (04/2023)
Table A3: All seven prompt templates we use.
Prompt
ID
Prompt
itap of O
2
a bad photo of the {}
3
a origami {
4
a photo of the large 0
a () in a vidco game
art of the O a photo of the small 0
Table A4: Aggregatod results (mean and standard deviation) over composition models trained on seven prompt templates MIT-States in closed-world setting. Template source of single template means the same prompt template the one used in the main paper.
Template Source
AUC 2
AUC
Seen
Unseen
HM
Top ks +
3
2
3
CGEg
6.8
5.1
28.7
25.3
17.2
Single
CLIP-Prim
8.2
17.0
24.1
6.9
15.6
22.8
34.0
27.9
20.4
Aggregated
CLIP-Prim
+ 0.3
25.5
0.2
7.1 4 16.2
0.1 23.7
0.2
35.1
0.5
27.7 +: 20.7
0.4
Table A5: Aggregated results (mean and standard deviation) over composition models trained on seven prompt templates on MIT-States open-world setting. Template source of single template mcans we use the same prompt lemplate as the one used the main paper.
Template Source
Val AUC
Test AUC
Seen
Unseen
HIM
Top k
2
3
1
2
CompCos
1.6
10.0
8.9
Single
CLIP-Prim
2.6
6.0
9.1
2.1
5.2
8.0
33.7
9.9
10.8
Aggregated
CLIP-Prim
2.9 + 0.1 6.5 + 0.2 9.7 + 0.2
2.2 + 0.1
5.3 + 0.2
8.0 + 0.2
34.0 + 0.7 10.0
0.3
11.3
0.2
Table A6: Aggregated results (mean and standard deviation) composition models trained seven prompt templates on MIT-States in closed-world sctting. We perform intervention to mcasure interpretability
Template
Val AUC
Test AUC
Seen
Unseon
HM
Souree
Top k +
2
3
2
3
Oracle Prim
99.7
Single
CLIP-Prim
17.0
24.1
27.9
[umin
CLIP- (nierv(lFl))))
58.0
66.0
paper)
(Paraaial))
95.8
53.4
62.9
32.6
53.7
63.2
54.1
68.2
49.4
CLIP-Prin
8.7 0.2
18.1 + 0.3
25.5
0.2
0.2
16.2
0.1
35.1 + 0.5
27.7
0.3
20.7
Aggregated
CLIP- (ntl---l
39.3 - 3.9
58.6
+ 3.5
68.6
3.1
39.9
3.0
58.1
=
3.1
68.4 2:14:00 2.6
59.2
2.1
72.7
2.1
55.7 = 2.6
Seell-InteroPPartian
43.6 + 4.3
64.0 + 4.3
73.3
4.2
44.0
+
3.4
63.7
+ 3.5
78.3 + 3.1
62.8
+ 3.2
76.2
2.1
59.0 + 2.9
A.5 Ablation Study of Model Size
We carry out an ablation study to explore the impact of scaling law on how well the VI pretrained model (CLIP) can capture primitive concepts on MIT-States. The results are shown in Table A8 We observe that larger models can predict more useful concept activations. However, larger models do not improve the
20

---

Published in Transactions on Machine Learning Research (04/2023)
Table A7: The impact combining activations from multiple prompt templates on MIT-States in closed- world setting
Template Source
Val AUC
Test AUC
Seen
Unseen
HM
Top h +
1
2
3
3
Composition
99.9
99.7
99.7
99.9
99.9
99.9
100
100
99.9
CLIP w/
CLIP-Prim
8.2
17.4
24.2
7.1
16.3
24.0
35.0
28.2
20.8
prompt
"LIP-Interv(Ful)
43.6
63.1
73.0
40.9
59.8
71.0
59.0
75.1
56.3
CLIP-Interv
47.2
68.4
77.3
45.4
64.4
75.2
63.4
76.9
60.1
CLIP w/
CLIP-Prin P-ii-
8.8
18.5
25.7
7.2
16.3
24.1
35.3
28.3
21.0
4 prompts
CLIP-Interv(Futr
45.9
64.7
75.2
46.3
65.5
74.9
62.7
77.7
62.1
CLIP-InteryvParttinn
49.7
70.4
77.4
69.8 50.1
78.3
66.2
79.5
65.2
CLIP w/
CLIP-Prim
8.9
18.5
25.8
7.2
16.1
24.0
35.3
28.2
20.9
prompts
CLIP-Interv(Ful)
45.8
65.7
76.7
47.7
65.0
76.1
65.0
77.9
62.4
GLlP--nterv(Parttinn
51.5
72.1
80.7
51.1
70.1
80.2
67.3
79.8
65.4
Table A8: Results of ablation study about how scaling law impact CLIP variants to capture primitive concepts CZSL task on MIT-States in the closed-world setting.
Training Evaluation Prim GT Prim GT
Val AUC
Test AUC
Seen Unseen HMM
Method
k=1
3
1
3
Oracle-Prim
99.99 99.7 99.7
99.9 99.9
99.9
100.0
100.0
99.9
CLIP-ResNet5O-Priin
/
6.1 12.9 19.2
4.0
10.5
15.9
28.8
21.1
15.4
CL.P--ResVet50O-nnerrrdlllll
8.7 17.3 22.1
7.9
15.9
22.5
32.1
31.0
23.5
C.IP-ReeNeeSS--lteerrPaaati
11.7 20.9 27.7
10.9
19.9
27.7
36.6
37.2
27.1
CLJP-ResNNe110-Prrim
/
6.6 14.1 20.4
4.1 10.3 15.9
28.9
20.7
15.6
CLIP-RReWee1OO--Ineeeeaalll
/
14.7 29.2 39.3
15.0 29.2 41.1
39.8
46.0
32.0
CIIP-RRsYcct1O0-htter((raaallll
/
1
16.2 31.9 42.3
16.1
31.3
43.5
41.1
47.2
33.8
CLIP-ViT-B/33-Priin
8.2 17.0 24.1
6.9
15.6 22.8
34.0
27.9
20.4
CLIP-WWiT-B/33-Ineerr///
32.2 49.1 58.0
30.0
49.3 59.6
52.3
66.0
47.7
CLIP-VVT-BB33-Ineeer(aatta
1
35.8 53.4 62.9
32.6
53.7 63.2
54.1
68.2
49.4
C.IP-VIT44/144300x-Piin
/
9.3 19.6 27.2
8.0
17.9 26.4
38.0
29.1
21.8
12.7 22.1 29.8
8.2 16.8 24.9
29.6
33.6
23.9
16.3 27.6 37.0
11.1 23.7 33.5
36.7
37.3
27.5
interpretability of the composition models, reflecting that they do not capture primitive concepts better than the smaller models.
A.6
Additional Experimental Details
A.6.1 Baselines
We compare our approach for MIT-States against several state-of-the-art basclines in CZSL. LabelEmbed (LE+) Nagarajan & Graunan 2018)) maps pair embeddings, combinations of word embeddings of altribute and object, and image embeddings into a joint semantic space using two separate MLP. Attribute as Op- erators (AOP) Nagarajan & Grauman (2018) regards attributes as linear transformations and the trans- formed object word embeddings are taken as composition embeddings.
Modular Networks (TMN) Purushwalkam al. (2019) trains feature extraction model and gating model which modifies
21

---

Published in Transactions on Machine Learning Research (04/2023)
the classifier based on the given attribute-object pairs. SymNet Li et al. (020)) utilizes the symmetry prop- erties of attribute-based transformation to learn object embeddinggs Compositional Cosine Logits (Comp- Cos) Mancini et al. (2021) trains a linear layer as composition function that projects a pair of attribute and object embeddings to a compositional space. Compositional Graph Embedding (CGE) Nacem et al. (2021) learns the dependency structure of attributes, objects, and their compositions using Graph Convolut ional Network (GCN) Kipf & Welling (2017). Since the proposed CGE is trained end-to-end and it finetunes its image encoder, we compare with the reported baseline, CompCosH, for MIT-Sttates in closed-world setting, and with CompCos for MIT-States in open- open-world setting. Note that these models are task-specific super- vised, whereas our approach only needs to learn composition models and is capable of performing the task in a zero-shot fashion.
We compare our CUB results to multiple state-of thr--rr baselines. FEAT Ye et al. (2020) adapts instance embeddings the target classification task with a set-to-set Transoformer function. DeepEMD Zhang et al.)
(2020) uses the Earth Mover's Distance as metric to compute structural distances between dense image representations to determine image relevance during training. RENet Kang et al.] [021] combines self- correlational representation and cross-correlational attention to learn relational embeddings. S2M2 Mangla ct al. 2020) regularizes feature manifolds via Manifold Mixup by focusing on learning general purpose representation robust to changes in data distribution. PEMg-NCM Hu et al. (2021) aims at processing feature vectors to be closer to Gaussian-like distributions to boost transfer learning accuracy.
A.6.2 Metrics on MIT-States
Since composition is trained only seen composite concepts, the seen concepts can evaluate much bet- ter or worse than the unseen ones on validation and test splits. We follow the evaluation protocol from Purushwalkam et al. (2019) in order to reduce models' bias on seen compositions by applying calibration bias, which are scalars to be added to the predicted scores of unseen concepts. We vary the values of the calibration bias, and compute a list accuracy scores of seen concepts and another list. for unseen concepts. We compute a list of 20 bias scalars based on the difference between the predicted scores for seen and unseen composite concepts:
Obtain composition model's predicted probability distribution over seen and unseen concepts for each sample on test. set.
2. For each unseen concept in the test set, compute the difference between the predicted score of most probable seen concept and the score of the ground truth concept. Denote this as Aditt
3. Sort Atit and compute list of 20 bias scalars by splitting Aaitt into 20 chunks. Denote the list of 20 bias scalars as B.
For each bias scalar B, add to the predicted scores of unseen concepts and evaluate model's performance. Therefore, we have list of 20 accuracy scores of seen concept, and another list of 20 scores of unseen concept.
Compute the four metrics as introduced in Scction
22


Reviews:
[{"review_id": "K5MNOMs9Vm", "reviewer": "Reviewer_qz9s", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The problem studies here is of high research value.\n2. The reported results are interesting.\n3. The presentation is overall clear.\n\nCons:\n1. Currently, the experiment setup could use further improvement.\n- a. The concepts could be really fine-grained in a very tiny region of the whole image. Therefore, it is possible this is the bottleneck limiting existing models. Comparing text embeddings of prompts and visual embeddings of spatial resolution could provide further justification and understanding the model's behavior.\n- b. The models may interpret the concepts in a different manner than us. Therefore, it is also necessary to check the spatial heat-map to understand whether models really localize to the correct region for a concept or maybe they have their own \"definition\" of the concept by checking if there is a pattern in terms of highly activated regions.\n- c. The current conclusion is that the learned concepts of VL models are not that interpretable. This raise interesting combination and comparison with recent work [1,2]. [1] shows that some learned latent textual concepts could be of more help for the downstream tasks and [2] shows that pretrained language model could be used to generate the helpful query to prompt VL model for finegrained recognition. Therefore, it would be very interesting to show and discuss: 1) if the concepts of VL models should be better not interpretable, could learned latent concept further improve the performance? 2) if the concepts could be actually better retrieved by better query design, can the language model help in this scenario?\n\n\n[1] Visual Classification via Description from Large Language Models, ICLR 23\n\n[2] Learning to Decompose Visual Features with Latent Textual Prompts, ICLR 23", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors study the problem of whether pretrained vision language models can extract primitive concepts and whether the extracted concepts can used to train a composition model in a more interpretable manner. Various studies are reported to empirically study three vision-language models' behavior.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please check the cons for details.", "broader_impact_concerns": "This is not really a concern but more like a question. Since the authors mentioned potential bias research on VL, could the dataset be further extend to datasets with racial or gender annotations to understand the potential bias issue of VL models? Or maybe just given existing datasets, could the authors further discuss on the possible biased concepts in the VL models?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lrgR0qzUaq", "reviewer": "Reviewer_ULW2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper proposes a novel framework called Compositional Concept Mapping (CompMap) to evaluate the effectiveness of visual-linguistic (VL) pre-trained models in learning primitive concepts. The proposed framework enables a quantitative assessment of learned concepts by measuring the performance of composition model learning.\n- The paper proposes the evaluation method of the usefulness and the interpretability. \n- The authors conduct extensive experiments using a range of recent VL pre-trained models, and provide both quantitative and qualitative analysis to reveal that the models lack interpretability, indicating that they do not learn primitive concepts in an understandable way. \n- The authors make their code and models publicly available, which can facilitate further research and development in this field. The paper also provides valuable insights into the limitations of existing VL pre-trained models and suggests potential directions for future research.\n\nWeakness:\n- The relationship between the interpretability metric (i.e., the difference of two models on GT input) and the interpretability of composition model is not clear given the paper. I can get a sense of such relationship but not confident enough, but I think that it is more like a writing concern (although very important for the paper presentation).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a novel framework called Compositional Concept Mapping (CompMap) to evaluate the effectiveness of visual-linguistic (VL) pre-trained models in learning primitive concepts. By measuring the performance of composition model learning, CompMap enables a quantitative assessment of the quality of learned concepts. The authors conduct extensive experiments using a variety of state-of-the-art VL pre-trained models and provide both quantitative and qualitative analysis to reveal that these models lack interpretability, suggesting that they do not learn primitive concepts in a human-understandable way. However, the models still exhibit high performance in visual recognition tasks. The authors highlight the need to improve VL pretraining methods by emphasizing the importance of learning interpretable primitive concepts. The paper provides valuable insights and a useful framework for evaluating VL pretraining, and the authors make their code and models publicly available for further research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The relationship between the interpretability and the score measured by model intervention is not clear to the reviewer (and potentially to the further readers). Thus the reviewer requests a detailed illustration of such relationship. The illustration can be formulated by describing why a small difference can lead to a highly interpretable compositional model, and why a highly interpretable compositional model can be related to the paper's intention on judging the VL model's primitive concepts. It would be great to have a complete step-by-step illustration where no step has confusion to the reader.  More details: for \"why a small difference can lead to a highly interpretable compositional model\", the reviewer expect a explanation on why a small difference can not lead to a low interpretable model as well. \n2. For the evaluation metrics in Sec 4.1, provide a concrete calculation in Appendix. This will help the reproduction of this paper.\n3. [Minor] In Sec 4.2., why the models from different VL model take different ViT backbone? Also, it would be great to also show the trend of usefulness and interpretability as the model size grows. This might be outside the scope of this paper, but the \n4. In Table 4, it would be better to give a break down metric of the accuracy. I think that the current model only considers correct if the whole set of primitive concepts match?\n5. [Minor] Any reasons on why CLIP-Prim has a lower unseen score but a higher seen score? Since one of the goal of incorporating primitive concepts is to improve the generalization. \n6. In Appendix A.1, the CLIP model takes dot product (actually cosine similarity since the vectors are normalized?) and other model takes matching score. Thus the CLIP activations can be negative while the other two models' output is always positive. Would it affect the comparison of these VL models? ", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uUhErYWlYU", "reviewer": "Reviewer_ax67", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper is well motivated and the topic of study regarding machine capability to understand compositional concepts from low-level concepts is important.\n\n2.\tThe proposed method is well explained and easy to follow.\n\nWeaknesses:\n\n1.\tThe paper uses different prompting to extract concepts out of an image but these would only work when the image contains single object or simple scenes. However, VL pretrained models are usually trained with more diverse set of data with complicated scenes. Their power is more about understanding visual scenes as a whole rather than extracting individual concepts. \n\n2.\tI doubt the effectiveness of the proposed model to formulate the problem as a classification task where labels are composite concepts. I also do not think that accuracy is a good metric to measure if the model understands these concepts well as it does not consider the similarity between target labels (e.g., “a white guy” and “a white man” are the same concept but two different labels).\n\n3.\tWhat are the total number of composite labels used in training? I guess the difference in performance between the Interv(Full) and Interv(Partial) are mainly because the large number of classes in the former case. \n\n4.\tThe degradation in performance between “Prim” models and “Interv” models on CUB is interesting. Apart from what has explained in the paper, do you have any quantitative or qualitative analysis to explain this?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the ability of recent vision-language (VL) pretrained model to be able to capture primitive concepts associated with objects presented in an image such as colors, shapes and object’s attributes. It proposes a simple framework called Compositional Concept Mapping (CompMap) that tries to map the primitive concepts to more complicated composite concepts from a finite primitive concepts space. The authors evaluate three state-of-the-art VL models, CLIP, ViLT, and ALBEF, and measure the usefulness and interpretability of the learned composition models. The study shows that VL models can capture primitive concepts to some extent, but there is still room for improvement.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1.\tI would like to see the author’s clarification on their choices of evaluation metrics as mentioned in the weakness section.\n\n2.\tI also welcome further discussion on whether the proposed method can deal with complicated scenes in real-world data where multiple objects/humans or a good mix of objects of different kinds in one scene are present.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a framework for measuring how well vision-language pre-trained models (such as CLIP, ViLT and ALBEF) can learn primitive concepts, in terms of usefulness and interpretability. After author response, it received two Leaning Accept and one Leaning Reject recommendations, so it is really a borderline case. 

On one hand, all the reviewers agree that the research question raised by this paper is important and it will be of interest for those who would like to understand how pre-trained VL models actually work under the hood. The paper is overall well written, and provides new insights and quantitative assessment for evaluating the effectiveness of VL models in learning primitive concepts.

On the other hand, due to the nature that this paper is an analysis paper and aims to provide new quantitative assessment to inspect the current VL models to learn primitive concepts, reviewers have commented that some additional clarification is needed and the paper needs to put more evidence and explanation of the new concepts into writing, such as the definition and measurement of "interpretability" in the paper. Reviewer qz9s also commented that additional experiments and analysis can be added, such as further analysis between region size and the attribute recognition results.

Overall, the editor thinks that the authors have done a relatively good job at rebuttal. In many cases, we care much about pushing the SoTA and here in this paper, the authors take a step back and try to understand better about the current VL models. Surely, the analysis provided in this paper can be further enhanced, but overall, the editor thinks that the merits slightly outweigh the flaws of the paper, and would like to recommend acceptance of the paper. The editor encourages the authors to take all the reviewers' feedback into consideration to prepare the camera-ready version. 
Decision: Accept (poster)