When are Lemons Purple? The Concept Association Bias of Vision-Language Models
Authors: Yingtian Tang
Yutaro Yamada
Yoyo Minzhi Zhang
Ilker Yildirim
Venue: EMNLP 2023 Main (2023)
Keywords: vision and language
bias

Abstract: Large-scale vision-language models such as CLIP have shown impressive performance on zero-shot image classification and image-to-text retrieval. However, such performance does not realize in tasks that require a finer-grained correspondence between vision and language, such as Visual Question Answering (VQA). We investigate why this is the case, and report an interesting phenomenon of vision-language models, which we call the Concept Association Bias (CAB), as a potential cause of the difficulty of applying these models to VQA and similar tasks. We find that models with CAB tend to treat input as a bag of concepts and attempt to fill in the other missing concept crossmodally, leading to an unexpected zero-shot prediction. We demonstrate CAB by showing that CLIP's zero-shot classification performance greatly suffers when there is a strong concept association between an object (e.g. eggplant) and an attribute (e.g. color purple). We also show that the strength of CAB predicts the performance on VQA. We observe that CAB is prevalent in vision-language models trained with contrastive losses, even when autoregressive losses are jointly employed. However, a model that solely relies on autoregressive loss seems to exhibit minimal or no signs of CAB.

When are Lemons Purple? The Concept Association Bias of Vision-Language Models
Yingtian Tang"
Yutaro Yamada*
Yoyo Zhang
llker Yildirim
EPFL
Yale University
yingtian. tang@enfl.cccccc yutaro. yutaro.ymmada@aaleee. edu
Abstract
Large-scale vision-language models such as CLIP have shown impressive performance zero-shot image classification and image-to- text retrieval. However, such performance does not realize in tasks that require finer- grained correspondence between vision and language, such Visual Question Answer- ing (VQA). As a potential cause of the diffi- culty of applying these models to VQA and similar tasks, report an interesting phe- nomenon of vision-language models, which we call the Concept Association Bias (CAB). We find that models with CAB tend to treat in- put as a bag of concepts and attempt to fill in the other missing concept crossmodally, lead- ing to unexpected zero-shot prediction. We demonstrate CAB by showing that CLIP's zero- shot classification performance greatly suffers when there is strong concept association be- tween an object (e-g. eggplant) and an at- tribute (e.g. color purple). We also show that the strength of CAB predicts the performance on VQA. We observe that CAB is prevalent in vision-language models trained with con- trastive losses, even when autoregressive losses are jointly employed. However, model that solely relies on autoregressive loss seems to exhibit minimal or no signs CAB.
CLIP: "In this picture, the color lemon is purple."
Figure I: When ask CLIP the color of the lemon in the above image, CLIP answers "purple". The text prompt we use "In this picture, color of the lemon is [mask]", where CLIP picks one from [red, green, yellow, orange, purple].
curacy for zero-shot performance of CLIP on VQA- v2 (Goyal et al., 2017). a common visual question answering benchmark. However, they simply use "question: [question text] answer: [answer text]" as text input for the text encoder CLIP, which makes the prediction harder than it should be. A subsequent work (Song et al., 2022) proposes better prompt generation method. They convert questions into masked prompts (e.g. "What's in the bowl behind the cake" becomes "The [mask] is in the bowl behind the cake"), and filter impossible answers using a language model, which improves CLIP's zero-shot performance on VQA-v2.
1 Introduction
However, the zero-shot performance of CLIP VQA-v2 is still not state-of-the-an (Shen et al., 2022). In this paper, we report a phenomenon, which we call Concept Association Bias (CAB), one of the reasons why CLIP struggles with VQA.
Recent large-scale vision-language models such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) have shown remarkable performance zero-shot classification and text-image retrieval tasks. These models trained via cross-modal contrastive learning on web-scale image-text pairs and obtain powerful multimodal representations. Encouraged by these strong zero-shot capabilities, several recent papers explored CLIP for more com- plicated vision-language tasks. The initial attempt made by (Shen et al., 2022) reports near chance ac-
To describe this phenomenon, we present a sim- ple image containing a "lemon" and an "eggplant" to CLIP, and ask what color the lemon is, shown in Figure 1. Surprisingly, CLIP predicts "purple" with high confidence. When we instead ask for the color of the eggplant, CLIP answers "yellow"". To cross-ched this phenomenon, we formulate a nary zero-shot image classification task on the same
Equal contribution.

---

image where the two labels are "yellow lemon" and "purple lemon", and find that CLIP predicts "purple lemon" with high confidence.
rather than learning syntactic and semantic ab- stractions. Many benchmarks are proposed evaluate robustness of ImageNet models towards various perturbations including common corrup- tion (Hendrycks and Dietterich, 2019), image style change (Hendrycks et al., 2021), and different view- points (Barbu et al., 2019). Our work differs from these studies that are purely based on language vision, because CAB is cross-modal phenomenon, which occurs when both image and language data are used.
We hypothesize that this phenomenon comes from the discrepancy between what is described in the image and text input, where CLIP attempts fill in the missing concept. The association be- tween "purple" and "eggplant" is strong, so when asked fill in the mask "[mask] lemon", predict- ing "purple" instcad of "yellow" makes more sense for CLIP, because the text description of "purple lemon" is aligned with the image that contains both a lemon and an eggplant more faithfully than "yel- low lemon", which only describes the lemon in the image. In fact, when randomize the color of the lemon and eggplant (e.g. "red" for the lemon and "green" for the eggplant), this bias disappears, and CLIP picks the color almost randomly between the two. We also find that CAB exists for more general object-attribute relationship such as the part-whole relationship (e.g. "humans" tend to have "clothes" " on, and "trees" tend to have "leaves".)
Compositionality in vision and language models The issue of vision and language models strug- gling with complex compositional questions has been studied before, where researchers have pro- posed enhanced training methods and modified ar- chitectures to tackle this problem (Basu et al., 2023; Nayak et al., 2022; Jiang ct al., 2023). Bogin et al. (2021) tests compositional generalization of vision and language models. Thrush et al. (2022) intro- duced a probing dataset called Winoground, which evaluates visuo-linguistic compositionality of vi- sion and language models. They evaluate a diverse range of otate-of-the-ar vision and language mod- els, including CLIP, but all of them perform close to or below random chance. A subsequent work (Di- wan et al., 2022) shows that Wineground requires not only compositional language understanding but also other abilities such as sophisticated common- sense reasoning and locating small objects in low resolution images, which most vision and language models currently lack. The work (Lewis et al., 2023) is the most relevant to our research, although primarily deals with toy datasets. Our work also reveals brittleness of vision-languagg models through the lens of CAB, which has been over- looked in the past.
Does CAB exist in other vision and language models as well? To answer this question, we also test BLIP (Li et al., 2022), BLIP-2 (Li et al., 2023), and OFA (Wang et al., 2022). We find that CAB exists in both BLIP and BLIP-2, but not in OFA, which is trained solely with autoregressive loss. Finally, we demonstrate that enabling deeper interaction between modalities in CLIP can miti- gate CAB. In particular, we show that extending CLIP with an additional Transformer layer on top and fine-tuning it on VQA is particularly helpful. Across such variants of CLIP, we report that the lower the degree of CAB, the higher model per- forms on visual question answering. However, we also find that this fine-tuning method may not be a comprehensive solution for the more general binding problem (Greff et al., 2020), such as accu- rately connecting attribute and object representa- tions, which leaves room for further research.
3
The Concept Association Bias
The zero-shot image classification of CLIP is re- markable for images that contain a single concept. However, when there are multiple concepts in the image but the text input does not cover all of them, the zero-shot classification of CLIP can be signifi- cantly biased towards the missing concept(s). We call this bias the Concept Association Bias (CAB). We first showcase this bias using color recognition tasks. For this analysis, we use the Natural-Color
2 Related Work
Vulnerability of vision and language models There are a number of papers that study the ro- bustness of vision and language models. Some prior work (Sinha al., 2021) shows that Trans- former trained via Masked Language Modeling (Devlin et al., 2019) is insensitive to word orders, suggesting that the success of BERT largely de- pends on learning higher-order word co-occurrence
'For all experiments in the main text, we use the ResNet50- x4 backbone for CI.IP. The results are consistent with the VIT hackbone, which are included in the appendix.

---

Zero-shl transfer from CLIPP to unlor mesgrilion
1,00
0:75
Accurucy
CAB
Figure 2: Example images from Natural- Color Datase (NCD) (Anwar al., 2022), modified for our colo recognition tasks that each image contains two diffe. ent objects.
0.25
0.00
Single object
Two abjects
abjects"
Dataset (NCD) (Anwar et al., 2022), which is a dataset of vegetables and fruits with a white back- ground. We take the following objects: banana, brinjal, broccoli, carrot, cherry, corn, cucumber, lemon, orange, plum, pomegranate, strawberry, tomato. We then randomly sample two images with different vegetable types and place the ob- jects side-by-side, resulting 494 images in total. Examples are shown Figure 2.
Figure 3: Zero-shot performance of CLIP color recognition tasks using NCD (Anwar et al., 2022). CLIP achieves almost perfect accuracy when there is single object in the image, but the accuracy significantly drops with two objects. "Two object* refer to the case which instead measure the accuracy of predicting the color of the object B when it is asked for the color of the object A, where we see 80% zero-shot accuracy. We claim this gap between Two objects and Two objects* is a result of the Concept Association Bias (CAB).
For zero-shot transfer from CLIP to our color recognition task, we ask for the color of one of the objects in the image. The labels "yellow", "purple", "green", and "orange", so it is a 5-way color recognition task. When there is single object in the image, we use the following text prompt: "In this picture, the color of the object is [mask]." When there are two objects in the image, we specify one of these objects in the prompt. For example, if there is lemon and another object in the image, the prompt takes the following format: "In this picture, the color of the lemon is [mask]."
when in reality it is asked predict the color of the object A, we that the zero-shot transfer perfor- mance CLIP much higher ("Two objects in Figure 3), approaching the single object accuracy. To understand this phenomenon, we find it help- ful to consider two variables per object, where each variable represents the object's name in the image and the color attribute of the object, as shown in Figure 4. When the colors are natural (Figure 4 (a)), both the object "lemon" and its attribute "yel- low" in the image are fully explained by the word "lemon" in the text prompt, resulting in the concept of the eggplant remaining. When CLIP performs zero-shot color recognition, we see that placing the color "purple" the prompt can most faithfully explain the remaining concept of the eggplant the image (Figure 4 (b)).
are "red",
The results are shown in Figure 3. We first note that the zero-shot performance of CLIP on our color recognition task is almost perfect when there is single object per image ("Single object" in Figure 3). However, the classification perfor- mance degrades to below chance when there are two objects per image ("Two objects" in Figure 3).
The above explanation suggests that there is a strong association between the color "purple" and the object "eggplant" in CLIP to the point where "purple" can partially explain the concept of the eggplant. What if we break this strong associa- tion? Does the gap between Two objects and Two objects* disappear?
How does this happen? We suggest that CLIP does not have mechanism that stores object- centric representation that correctly binds the ob- ject's name and its attribute. In another words,
CLIP processes its input as a "bag of concepts". To inspect this possibility, we look at what kind of mistakes CLIP makes when there are two objects derived
To test this, generate images of fruit and veg- etable in unnatural color using Stable Diffusion 2.0 (Rombach et al., 2022) with prompt format '[color name] [fruit/vegetable name]', and filter bad images by ourselves. Examples are shown in Figure 5. We call this dataset UNnatural-Color
A and B. We find that many mistakes from common source. That is, when asked for the color of object A, CLIP often predicts the color object the image. fact, when we measure the accuracy of predicting the color the object B

---

Imoaee
Lacon
Capkin
Zero-shot transfer from CLIP unnatural color recognition
color the lemon [maik]]
1,00
Tolke
Pays
(a) Natural color
0.75
Imagge
Text
Accurao
0.50
Capplar
The color of the lemon is purple.
0.25
(b) Natural color ([mask] purple)
0.00
Single object
Two objects
Two objects"
Insag:
Text
Rei
The color of the lemon [mank]
Figure 6: Zero-transfer performance of CLIP to color recognition on UNCD, where we assign non-associated color to each vegetable. CLIP achieves 80% accuracy when there is a single object in the image. While the accuracy drops for Two objects, the drop is not as sig- nificant the NCD case. Furthermore, the gap between Two objects and Two objects* vanishes, compared to the NCD case.
SCBU
Aed
Gava
(c) Unnatural color
Figure 4: The concept binding diagram. Two variables per object represent the object name and its attribute (e.g. color), respectively. We suggest that the text prompt and the image are represented as two separate "bags of con- cepts" in CLIP. When pair of object-attribute concepts are naturally associated with each other, both concepts can be accounted for by including the prompt either of the object or the attribute. When only some of the concepts in the image are included the text, this leaves other concepts in the image unaccounted for.
are paired with random attributes in images? This result arises from a common mechanism that im- pacts both the Two objects and Two objects* tasks. To see this, we back to our diagram in Figure 4 (c). When the colors are unnatural (e.g., a lemon in red color and eggplant in green color), then the remaining bag of concepts that yet to be explained by the text include "red". "green", and "eggplant". This because the color "red" is not as- sociated with the concept of "lemon", and therefore the word "lemon" in the text prompt cannot explain the color "red", unlike the case that uses natural color. As result, CLIP can choose either "red" or "green" for color recognition. And indeed, sur- prisingly, CLIP randomly chooses between the two it does not associate the concept of "red" with the lemon, even though in the image the lemon unambiguously appears in red. Likewise, for the Two objects* task (in which the correct prediction is defined as the color of object B when asked for object A), CLIP essentially randomly picks one of the two colors present in the image, despite the fact that each object has their own very distinct color.
Figure 5: Examples from UNCD. Single object (Top) and Two objects per image (Bottom).
Dataset (UNCD). We repeat the same experiment on UNCD. The results are shown in Figure 6. We see that the zero-shot performance for a single ob- ject is still high, suggesting that CLIP can pick up the color attribute even if the color is not strongly associated with the object itself. However, for the two object cases, we see that there is almost no difference between Two objects and Two o tasks. In other words, CLIP predicts the two non- associated colors in the image with almost equal chance. We also create a version of NCD, which we call UNCD-v2, where we artificially change the color of each fruit and vegetable of NCD to non-associated color. As shown in Appendix, we see a similar pattern of CAB as UNCD.
3.1 CAB exists on real-world dataset
So far, we use NCD to verify the existence of CAB. Here, we test CAB on common visual question an- swering benchmark: VQA-v2 (Goyal et al., 2017). We perform the zero-shot transfer of CLIP the color-related questions in VQA-v2, where our la- bels are beige, black, blue, brown, green, gray, pur- ple, red, orange, pink, white, yellow, and silver.
Why does the CAB gap disappear when objects

---

Black
Thac:
Bmocooh: brom > yellow green
Knie: omane yellow heig
Inocoddi sher ne gay
Figure 8: Example images from ReI3D (Goyal et al., 2020).
Figure 7: The Concept Association Bias (CAB) in VQA- v2. The text prompt is the following format: "The color of the [object] is [color]." The first word top of each image indicates the word used in place of [object] and remaining color names are listed in the order CLIP chooses them for [color].
Prediction accuracy on Rel3D
Two abjects objects"
0.0
02
0.4
Figure 9: Zero-shot transfer performance of CLIP to part-whole recognition on Rel3D (Goyal et al., 2020). Similar to the color recognition task, CAB exists for part-whole recognition.
We the prompt format "The color of the [ob- ject] [color]." We show example images and the CLIP's zero-shot color predictions (top three, in decreasing order) in Figure 7. We can see that these mistakes are result of CAB, For example, when we use the prompt format "The color of the banana is [color]" and the image that contains both banana and black-and-whin portraits of people. CLIP answers "black" instead of "yelloo"". We randomly sample 100 mistakes CLIP makes out of all color-related questions, and manually inspect these images to identify the mistakes are based on CAB. We find that roughly 50 mistakes are due to CAB. In Section 6, we illustrate how the degree of the CAB affects the performance on VQA-v2 in more detail.
are shown in Figure 8.
We select 8 objects and their corresponding part attributes as follows: (person, clothes), (camera, lens), (plant, leaves), (car, wheel), (cat, tail), (com- puter, screen), (bed, sheet), (gum, trigger). In total, collect 356 images from Rel3D that contain of these pairs. Prompt examples include "In this picture, the human has [mask]", "In this picture, the plant has [mask]" etc., and we let CLIP pick one from the part attributes as above (e.g., clothes, lens, leaves, etc.).
The results are shown in Figure 9. We find that indeed, similar to the zero-shot color recognition task, the whote recognition task also shows CAB. This result suggests that CAB more generally applies to CLIP across types of abject-attribut relations.
3.2 CAB exists for attributes other than color
this section, we test whether or not CAB exists for attributes beyond color. While there are various attributes we can evaluate, here we focus on part- whole attributes. Part-whole attributes are suitable for our CAB experiment because just like color, can construct a syntactically reasonable prompt by finding two objects with a part-whole relationship. For example, "A tree has leaves" would be a good example prompt for our test, where the verb "has" indicates the part-whole relationship between the two objects in the sentence. To evaluate the perfor- mance of zero-shot transfer of CLIP on whole recognition,
How does the strength of concept binding affect CAB?
In Section 3, verify CAB for color recognition and part-whole recognition tasks. In this section, we investigate if varying the strength of the binding between two words affects the degree of CAB. We use ConceptNet (Speer et al., 2017) to measure the association strength between two words. Concept- Net is a knowledge graph that connects words with labelled, weighted edges. When selecting words, focus on the edges that are labelled "Relat- edTo"". For each Rel3D object name we used in Section 3.2, we pick related words in the decreas- ing order of association strength, as shown in Table I. For this concept recognition task, we use the following prompt format: "[object] [word]" and
the Re13D dataset (Goyal et al., 2020), which originally proposed test spa- tial relationship understanding of vision models. Rel3D consists of images of 3D scenes, where two objects with particular spatial relationship are sit- uated in each scene. Example images from Rel3D

---

object
treat these models with different heads sepa- rately and abbreviate them as "contrast", "match", and "caption"". OFA unifies various vision and language tasks via a simple -o-eenceee--eeeence framework and employs autoregressive loss for its objective.
person
human
dall
churacter
statue
camera picture
flash
subject
photographer
tripod
plant
seed
free
flower
green
colton
car
drive
vehicle
motor
automobile
wheels
cat
feline animal
pet
kitten
dog
computer
apple desk
print
dell
data
bed
sleeping
furniture
mattress
place
pillows
gun
bullet
weapon
shooting
pistol
For comparison across models, we define the CAB score as:
Table I: Object names from Rel3D (the first column) and the top five related words from ConceptNet (Speer et al., 2017). The smaller the column number is, stronger the association the corresponding object name is.
AcCtwoo object' ACCCsco object +
2
where Acc stands for accuracy. The CAB score ranges from to 1, and higher score indicates a more severe CAB.
Prediction aecuracy for different binding strengths
Two objects
Two objects"
In Table 2, report results NCD for the aforementioned models and order them according to the CAB scores. While these networks all have high single-object recognition performance, they demonstrate a spectrum of levels of CAB.
0.6
Aceracy
The networks trained with contrastive match- ing losses (i.e. CLIP, BLLP-contrass/matchh BLIP- 2-contrast/maachh) have stronger CAB than those trained with autoregressive loss (i.e. BLIP-caption, BLIP-2-captie capptin,, and OFA). Moreover, in compari- son with BLIP, BLIP-2 uses a large language model as the final decoder on top of its sub-networks. mak- ing them more associated with the autoregressive loss and having less CAB scores than their counter- parts in BLIP
0.0
Rank
Rank x
Rank
Rank
Rank
Binding strength
We vary the strength of concept binding, and compute the accuracy for Two objects and Two objects*. We see that as the association strength gets weaker, CAB becomes smaller, although it somewhat noisy.
Figure 10:
the label sets are restricted the words in the same column. The results are shown in Figure 10. While it is noisy, we see that as the concept association becomes weaker, CAB becomes smaller.
Furthermore, we observe that matching losses have lower CAB than contrastive losses for both BLIP and BLIP-2. Although the two losses are similar, the matching task uses cross-attention that jointly processes texts and images, whereas the contrastive task only uses unimodal self-attentio. Based on this observation, we hypothesize that the deeper cross-modal interaction helps mitigate the CAB, We further investigate this in Section 6.
5 CAB widely exists in vision-language models
In Section 3, we demonstrate CAB using CLIP. In this section, we extend our experiments to other vision-language models, including BLIP (Li et al., 2022), BLIP-2 (Li al., 2023), and OFA (Wang al., 2022).
It is worth noting that the amount of CAB in autoregressive models is substantial. For example. Two objects*'s accuracy for BIIP-caption, BLIP-2- caption, and BLIP-2-FlanT5 are 0.471, 0.483, and 0.377 respectively. This means that when joitnly trained with contrastive losses, even autoregressive models are biased by concept association, resulting in Two objects*'s accuracy being much higher than random chance. The only model with minimal or almost CAB is OFA, which sorely relics on autoregressive loss.
BLIP and BLIP-2 are both mudti-modal, multi- task model that has three heads and pre-training objectives: the CLIP-like image-text contrastive loss, the binary image-text matching loss that clas- sifies whether the text corresponds to the image, and the language modeling loss, which autoregres- sively generates the caption for an image. For BLIP-2, there is the second stage of pre-training for visual conditioned language generation, boot- strapped from pre-trained LLM. In our experiment,
In Table 3, we compare CAB scores for CLIP with vision encoders in different sizes. Although

---

Models
Two objects
Two objects"
Single
CAB
Contrastive-los network
CAB
Winoground-group
CLIP 0.011
0.932
0,929
0.961
CLIP
0,961
0.0724
BLIP-contrae
0.086
0.879
0.846
0.896
BLIP-contrast
0.896
0.0800
BLIP-march
0.123
0,841
0,925
0.859
BLIP BLIP-2-contrast
0.851
0.0850
BLIP-2-contrast
0.138
0.840
0,844
0.851
BLIP-2-match
0,330
0.627
0,925
BLIP-2-caption
0.359
0,558
0.775
0.599
Table 4: CAB vs. Winoground with contrastive models.
BLIP-caption
0.438
0.471
0.862
0.516
BLIP-2-FlanTS
0.604
0.377
0,984
0.386
OFA
0.855
0,078
0.879
0.111
Matching-loss network
CAB
Winoground group
Table 2: CAB experiments using Natural Color- ful Dataset (NCD) on multiple architectural variants.
BIII-match
0.859
0.206
BLIP-2-match
0.648
0.235
FlanT5 refers H-P-2-viTg-Flantt
Table CAB Winoground with matching models.
CLIP
Two objects
Two objects"
CAB
controlled experiment using CLIP.
ViT-B/32
0.023
0,944
0,961
The idea of using deep modality interaction on top of image and text embeddings has been ex- plored before in (Kim et al., 2021). However,
ViT-B/16
0.059
0.886
0,913
ViT-L/14
0.057
0.918
0.931
VIT-L/14@336 336
0.058
0,926
0.934
RN50
0.011
0,932
0.961
(Kim et al., 2021), the image and text encoders are shallow unlike CLIP. In (Shen et al., 2022), instead of using CLIP as is, they employ the archi- tecture that uses the image encoder of CLIP, the BERT text embeddings, and Transformer additional modality interaction module. They ap- ply this model for vision and language tasks such as Visual Question Answering, Visual Entailment, and V&L Navigation tasks. The goal (Shen et al., 2022) was to demonstrate that the image encoder of CLIP is more helpful than ImageNet- pretrained image encoders. For our architecture, both image and text encoders from CLIP, and also a Transformer for modality interaction on top of CLIP, as shown in Figure 11.
RN50X4
0.045
0.916
0.936
RN50x16 16
0.121
0.842
0.860
RN50X64
0.074
0.872
0.899
RN101
0.034
0.944
0.955
Table 3: CAB experiments for CLIP with vision en- coders in different sizes.
the size varies greatly, the CAB score stays around the same level for these different CLIP models. In Table 4 and Table 5, we compare CAB with Winoground (Thrush et al., 2022), whose task is to correctly match two images with two captions, but the key aspect is that both captions consist of the exact same words, albeit arranged differently. We compare CLIP, BLIP-come CL.P-cootrast/match. and BLIP- 2-contrast/match because they are more suitable for the Winoground matching task. We roughly see that as CAB decreases, Winoground perfor- mance goes up, which is aligned with what CAB attempts to measure. However, we also observe that using matching loss benefits more for both CAB and Winoground, presumably because the matching loss uses cross-attention between text and image encoders.
We conducted experiments in two settings: 1. Freezing CLIP and only fine-tuning the Trans- former head, and 2. Fine-tuning both CLIP and the Transformer head. Following (Shen al., 2022), we use VQA-v2 fine-tune these model variants. We follow the standard pre-processing of VQA-v2, where we filter less common answers and select
Transformer
Modality
6 How can we mitigate CAB?
Vision Encoder
Vision Encoder
Fine-tuning helps reduce CAB In the last sec- tion, show that CAB can be seen in vision- language models, and it especially prominent for purely contrastively trained models. In this sec- tion, we test our hypothesis from Scction 5 that a deeper modality interaction helps mitigate CAB in
Eneoder
Encoder
Image
Text
Image
Text
CLIP
Figure II: Original architecture of CL.IP (Leff) and the architecture for fine-ture fine-tuning on VQA-v2 (Right).

---

NCD
Two objects
Two objects*
CAB Score
vQ
0.8
(IIP((rozen)
0.352
0.094
0.371
0.542
(FIP(Finemune))
1618
0.390
0.7
Porforraance on VQA-v2
Table 6: Given two objects A and B per image, when
0.6
we ask the color of object A, "Two objects" refer to the case where we use color object A as true labels, and "Two objects*" refer to the case where use the
0.5
color of object B as true labels. For both cases, the
0.4
Transformer head on top of CLIP are fine-tuned. See text for details of the two models.
0.3
3,129 answer vocabularies. We then formulate the VQA task as classification problem over 3,129 answer vocabularies given images and questions as input. After fine-tuning VQA-v2, perform zero-shot color recognition using NCD evaluate CAB a similar manner to Section 3. That is, given two fruits in an image, we ask the color of one of the fruits. The results are shown in Table 6. We can see that adding deeper modality inter- action reduces CAB (See Fig. 3 for comparison). Moreover, we also see that between these model variants, the lower the CAB score is, the higher the accuracy on VQA-v2 is. Does this relationship hold more generally? To see this, we prepare three other baselines: A. CLIP image encoder + BERT text embeddings Transformer head, fine-tuned altogether; B. The same as A. but with the CLIP image encoder frozen; C. The same as A. but uses V&l. pre-trained weights before fine-tuning. These architectures are based on (Shen et al., 2022).
CAB Score
Figure 12: the Concept Association Bias), the higher the models perform on VQA-v2. A-E refers to different model configurations. A-C are detailed in the main text. D and are the in Table 6, respectively.
The lower the CAB Score (less susceptible
(I.oP(Froee)) and CLIP(Finetuned)
such a procedure solve the binding problem (Greff et al., 2020) more generally? The binding prob- lem for neural networks refers the inability of models to dynamically bind information that dis- tributed across the network (Greff et al., 2020). fact, we can view CAB as an instance of the bind- ing problem because one of the causes of CAB is the model's inability to correctly bind object and attribute representations across modalities. If allow- ing for deeper modality interaction and fine-tuning helps more faithfully bind attributes and objects across vision and language inputs, then we should expect to see accuracy improvement for the Two objects setting, even on UNCD. This is because a model that can successfully localize an object and its attributes, and separate the representation of dif- ferent objects, should be able to identify the color of the queried object even if the object is randomly colored. Contrary to this prediction, the accuracy on Two objects for these fine-tuned models is lower than the original CLIP, except for the model C, which uses large pre-training datasets. In fact, we find that these fine-tuned models also have lower accuracy for Two objects* (Table 7), indicating that the fine-tuned models most often choose a color that is not present in the image. This suggests that fine-tuning VQA-v2 simply allows the model to pick up real-world co-occurrence of colors and objects more easily and consistently. Therefore, fine-tuning the Transformer head may not fundamentally solve the problem of correctly binding objects and their attributes.
V&L pre-training used the aggregated data from MS COCO Captions (Chen et al., 2015), Visual Genome Captions (Krishna et al., 2017), VQA (Antol et al., 2015), GQA (Hudson and Manning, 2019), and Visual7W (Zhu et al., 2016), which re- sults in 9.18M image-text pairs. For C. we used the publicly available pre-trained model released by (Shen et al., 2022). For A. and B., train the models on our own. We detail the hyperparameters in the appendix. The results are shown in Figure 12. We see that as the CAB Score becomes lower (i.e., as models become less susceptible to the con- cept association bias), the accuracy on VQA-v2 increases. This encourages us to reduce the bias de- rived from Concept Association to further improve vision and language models in general.
find that
Fine-tuning alone may not necessarily solve the binding problem The last section demonstrates that by introducing deeper modality interaction and fine-tuning, we can mitigate CAB on NCD. Can

---

UNCD
for a wide variety of downstream tasks, we believe our results are important to the community.
Two objects
0.46
0.216
0.266
0,308
0,313
0,574
Two objeces
0.17
QU77
0,332
0.065
0,153
0,005
Table 7: The performance of CLIP with fine-tuned deeper interaction module. We see that fine-tuning rather mostly harms the accuracy for Two objects, in- stead of improving which suggests that fine-tuning may not solve the more general binding problem (Greff et al., 2020). The values for CLIP(Original) are the same as Figure 6. A-C refer to specific fine-tuned model configurations, detailed in the main text.
Ethics statement
Although our findings may not have immediate im- plications for the misuse of Al systems, it is essen- tial to acknowledge that biases and unintended be- haviors exhibited by models like CAB can pose po- tential risks, including social bias and other forms of harm. Addressing these biases and limitations becomes imperative to ensure the ethical and fair utilization of vision-language models in real-world scenarios. We strongly advocate for continued research and development that emphasizes trans- parency, fairness, and accountability in the design and implementation of vision-language models.
7
Conclusion
Every object has set of concepts that are roughly associated with it. For instance, the object "lemon" can be associated with "yellow", "fruit", and so on. Such concept association is automatically learned in vision-language models, to the point where the word "yellow" can partially explain the object "lemon" in certain cases. We establish that the Concept Association Bias (CAB) exists for vision- language models through a series of experiments. and find that the models trained with contrastive loss are especially affected. Furthermore, we verify that the lower the degree of CAB is, the higher the performance of VQA-v2 is. Contrastive models like CLIP is increasingly popular in both computer vision and natural language processing. We hope work raises awareness of the brittleness of con- trastive objectives as develop new vision and language models.
Acknowledgements
YY's research is partially supported by Masason Foundation.
References
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question An- swering. In 2015 IEEE International Conference Computer Vision (ICCV), pages 2425-2433.
Saeed Anwar, Muhammad Tahir, Chongyi Li, Ajmal Mian, Fahad Shahbaz Khan, and Abdul Wahab Muzaffar. 2022. Image Colorization: A Survey and Dataset.
Limitations
Andrei Barbu, David Mayo, Julian Aiverio. William Luo, Christopher Wang. Dan Gutfreund, Josh Tenen- baum, and Boris Katz. 2019. ObjectNet: A large- scale bias- controleed dataset for pushing the limits of object recognition models. Advances in Neural Information Processing Systems, volume 32, pages 9453-9463.
While we verify CAB in zero-shot transfer of con- trastive based vision and language models for color recognition and phrt-whole recognition, there are other object-atribute relationships that are not ex- plored in this paper such as object material, shape. and texture. Additionally, we focus study VQA, as the task format of VQA is directly ap- plicable to our CAB experiments. An interesting future study to complement our work is to explore the effect of CAB on other vision-language tasks (e.g., visual entailment (Song et al., 2022)), and explore other methods to mitigate CAB. Finally, we focus on CLIP, BLIP, BLIP-2, and OFA in this study. Future work should also investigate other vision-language models that incorporate more ex- tensive modality interactions (e.g.. FLAVA (Singh al., 2022) and ALBEF (Li al., 2021). However, given that these models are widely adopted in both computer vision and natural language processing
Samyadeep Basu, Maziar Sanjabi, Daniela Massiceti, Shell Xu Hu, and Soheil Feizi. 2023. Augmenting CLIP with Improved Visio-Linguiste Reasoning.
Steven Bird, Ewan Klein, and Edward Loper. 2009, Nat- ural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O' O'Reilly, Beijing.
Ben Bogin, Shivanshu Gupta. Matt Gardner, and Jonathan Berant, 2021. COVR: A Test-Bed for Vi- sually Grounded Compositional Generalization with Real Images. Proceedings of the 202/ Conference on Empirical Methods in Natural Language Process- ing, pages 9824-9846, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.

---

Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakr- ishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. 2015. Microsoft COCO Cap- tions: Data Collection and Evaluation Server.
Chao Jia, Yinfei Yang, Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le. Sun-Hsuan Sung. Zhen Li, and Tom Duerig. 2021. Scaling Up Vi- sual and Vision-Language Representation Learning With Noisy Text Supervision. In Proceedings of the 38th International Conference on Machine Learning. pages 4904-4916. PMI.R
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Kenan Jiang, Xuehai He, Ruize Xu, and Xin Eric Wang. 2023. CoMCLIP: Training-Free Compositional Im- age and Text Matching.
Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. ViLT: Vision-and- -anguae Transformer Without Convo- lution or Region Supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 5583-5594, PMI.R
Anuj Diwan, Layne Berry. Eunsol Choi, David Harwath, and Kyle Mahowald. 2022. Why is Winoground Hard? Investigating Failures in Visuolinguistic Com- positionality.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. Interna- tional Journal of Computer Vision, 123(1)332-73.
Ankit Goyal, Kaiyu Yang. Dawei Yang. and Jia Deng. 2020. ReI3D: A Minimally Contrastive Benchmark for Grounding Spatial Relations in 3D. In Ad- vances in Neural Information Processing Systems, volume 33, pages 105 10514-10525, Curran Associates, Inc.
Martha Lewis, Nihal V. Nayak, Peilin Yu, Qinan Yu, Jack Merullo, Stephen H. Bach, and Ellie Pavlick. 2023. Does CLIP Bind Concepts? Probing Composi- tionality in Large Image Models.
Yash Goyal, Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. 2017. Making the in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904-6913.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping Language- Image Pre- training with Frozen Image Encoders and Large Lan- guage Models.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmid- huber. 2020. On the Binding Problem in Artificial Neural Networks.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping Language-Imag Pre- training for Unified Vision-Language Understanding and Generation. In Proceedings of the 39th Inter- national Conference on Machine Learning, pages 12888-12900. PMLR
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul De- sai. Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. 2021. The Many Faces of Robustness: A Critical Analysis of Out-of-Distriuttion Generalization. International Conference on Computer Vision (ICCV).
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty. Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before Fuse: Vision and Language Representation Learning with Momentum Distilla- tion. In Advances in Neural Information Processing Systems, volume 34, pages 9694-9705. Curran Asso- ciates, Inc.
Dan Hendrycks and Thomas Dietterich. 2019. Bench- marking Neural Network Robustness to Common Corruptions and Perturbations. In International Con- ference on Learning Representations (ICLR).
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan
Le Bras, and Yejin Choi. 2021. CLIPScore: A Reference-free Evaluation Metrie for Image Caption- ing. In Proceedings of the 202/ Conference on Em- pirical Methods in Natural Language Processing, pages 7514-7528, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Nihal V. Nayak, Peilin Yu, and Stephen Bach. 2022. Learning Compose Soft Prompts for Composi- tional Zero-Shot Learning In The Eleventh Interna- tional Conference on Learning Represenaatinss
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. 2021. Learn- ing Transferable Visual Models From Natural Lan- guage Supervision. In Proceedings of the 38th In- ternational Conference on Machine Learning, pages 8748-8763. PMLR
Drew A. Hudson and Christopher D. Manning, 2019, GQA: A New Dataset for Real-World Visual Reason- ing and Compositional Question Answering. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6700-6709.

---

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical Text- Conditional Image Generation with CLIP Latents.
on Computer Vision and Pattern Recognition, pages
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. OFA: Unifying Architectures, Tasks, and Modalities Through a Sim- Learning Framework. In Proceedings of the 39th International Conference on Machine Learning, pages 28318-23344 PMLR.
Royi Rassin, Shauli Ravfogel, and Yoav Goldberg. 2022. DALLE-2 is Seeing Double: Flaws in Word- to-Concept Mapping Texe2Image Models.
ple Sequeece-to-Sequeene
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High- Resolution Image Synthesis With Latent Diffusion Models. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 10684-10695.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Lu- ong. Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. 2022. Scaling Autoregressive Models for Content-Rich Texl-to-Image Generation.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mah- davi. Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. 2022. Photorealistic Text-to-lmage Diffusion Models with Deep Language Understanding.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei- Fei. 2016. Visual7W: Grounded Question Answering in Images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4995-5004.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai- Wei Chang, Zhewei Yao, and Kurt Keutzer. 2022. How Much Can CLIP Benefit Vision-and- Lonon-un--L Tasks? In International Con- ference Learning Representations.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022. FLAVA: A Foun- dational Language and Vision Alignment Model. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 15638-. 15650.
Koustuv Sinha, Robin Jia, Dicuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre- rre-training for Lit- de. In Proceedings of the 2021 Conference Empir- ical Methods in Natural Language Processing, pages 2888-2913 Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Haoyu Song, Li Dong, Weinan Zhang, Ting Liu, and Furu Wei. 2022. CLIP Models are Few-Shot Learn- ers: Empirical Studies on VQA and Visual Entail- ment. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). pages 6088-6100, Dublin, Ireland. Association for Computational Linguistics.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5:
An open multilingual graph of gen- eral knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. 17 pages 4444 4451, San Francisco, California, USA. AAAI Press.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing Vision and Lan- guage Models for Visio-Linguistic Compositional- ity. In Proceedings of the IEEE/CVE Conference

---

Prediction accuracy of different spatial arrangements
Two objects
Two objects*
left & right
up down
upper-left & down-right
large & small
0.25
0.50
0.75
Accuracy
Figure 13: The Concept Association Bias (CAB) remains regardless of the spatial configurations such as "left &
right", "up & down", "upper-left & down-right", and "large & small". We use the same subset of NCD as in Figure 3.
A Details of UNCD
We used -512-basc-emal
from Stable Diffusion's official github repository². We used the deafult hyperparameters for sampling, except set the size of images to be 512 by 512, and sample 4 images for each prompt. We used the following negative prompt: 'low-res, oversaturated, ugly, cartoon, grain, out of focus, ambiguous, blurred, split frame, out of frame, cropped, multiple frame, split panel, multi panel, people, human, logo'
B
CAB is robust across different conditions
B.1
The spatial arrangement has almost effect CAB
In our earlier experiments NCD and UNCD, two objects are positioned side-by-side. To see if CAB is robust the positioning of objects, we vary the spatial arrangement of the two objects in the image. Concretely, we test the following spatial configurations: left & right, up & down, and upper-left & down-right. also vary the size the two objects for left right, which is denoted as "large & small". As Figure 13 shows, CAB is not affected by either spatial arrangements or the object size.
B.2 CAB persists for prompt variations
The original CLIP paper (Radford et al., 2021) reports that varying text prompts changes zero-shot transfer performance of CLIP. Here, we test if CAB remains effective when we vary the prompt. As Figure 14 shows, CAB is relatively stable across prompt variations. It is interesting that as long as the names of the object and color (denoted as [object] and [color]) are included in the prompt, CAB exists even if text prompts are semantically meaningless such "[object] [color]", "The color of [color] is [object]" and "This prompt random [object] [color]". Moreover, even if negate the sentence (e.g. "The color of [object] is not [color]") CAB still exists, which suggests that the text encoder of CLIP seems ignore the negation.

---

Prediction of different prompts
Two objects
Two objects"
[sbjezt]
The color of [object] is (coloc].
[ebjjut)] non (uoorr)
The color of (culee) [otject]
This prompt is randum [>bjest] (toke).
Figure 14: The Concept Association Bias (CAB) is relatively stable across prompt variations, including semantically meaningless prompts. We use the same subset of NCD as in Figure 3
Figure 15: Examples from UNCD-v2. Single object (Top) and Two objects per image (Bottom).
CAB experiments using UNCD-v2
We utilize the same instances of NCD, but we assign non-associated colors to each vegetable. Figure 15 displays some example images. In Figure 16, we observe that Two objects and Two objects* exhibit similar performance, mirroring the pattern observed with UNCD discussed in the main text.
D Inspecting object representations of CLIP
The section 6 demonstrates that by introducing more modality interaction, can mitigate CAB. However, this requires additional procedure to fine-tune the newly introduced module. Can alleviate CAB by spatially pooling features of the original CLIP? Such an approach would work if CLIP develops localized object-centric representations. To investigate this possibility, we use NCD and take the image tokens that correspond to the left half of images ("Left Pool"), and conduct zero-shot classification to predict the color of the left object of an image. We compare the performance of this procedure with the case where we use the average of all tokens in the image ("Global Pool"). If CLIP develops localized object-level representations, we should see an increase in accuracy compared to Global Pool. However, as shown in Table 8, we see that the accuracy of the left pooling is lower than global pooling. (We also show the accuracy values based on the original CLIP CLS embeddings.) This suggests that the features of object that positioned in the left half of the input image are propagated the right half well, taking the features strictly from the left side reduces the information that is necessary accurately predict the color. Therefore, when there are multiple objects, we can see that CLIP struggles with the binding between object representations and attribute representations. Future work should explore incorporating object-centric learning into CLIP to guide and structure the binding between objects and their attributes.

---

Zero-shot transfer from CLIP to unnatural color recognition
1.00
0.75
Accuracy
0.50
0.25
0.00
Single object
Two objects Two objects*
Zero-transfer performance CLIP to color recognition on UNCD-v2, where we assign non-associated color to each vegetable. CLIP achieves 80% accuracy when there is single object in the image. While the accuracy drops Two objects, the drop is not as significant as the NCD case. Furthermore, the gap between Two objects and Two objects* vanishes, compared to the NCD case.
Figure 16:
CLS
Global Pool
Left Pool
NCD
0.617
0.417
0.209
UNCD
0.474
0.429
0.134
Table 8: 5-way color classification accuracy using spatial pooling. The prompt format we use "The color is [mask]." The original CLIP uses the CLS embedding to compute the similarity between image and text embedding. Global Pool takes the average of image tokens as their image embedding. Left Pool take the average of the tokens corresponding the left side the image. The reason why CLS for NCD higher than 0.5 that sometimes the color of two vegetables in the image are the same. In that case, the zero-shot classifier's prediction is almost always correct, which biases the overall accuracy towards 1.0.

---

Original
Noun swapping
Word Shufling
100
75
Percentage (%)
50
BLEU-2
BLEU-3
BLEU-4
METEOR
ROUGE-L
CIDEr
SPICE
CLIPScore
RefCLIPSeore
Figure 17: Ratio of caption score before and after text manipulation. We can see that CLIPScore operates as a bag of concepts, which affected by either noun swapping word shuffling, compared to other caption metries.
E Hyperparameter details for fine-tuning
In our experiment of fine-tuning the modality interaction architecture (Section 6), we used the exact same hyperparameters as those in (Shen et al., 2022). The specific repository we used is https: //gi thub. com/ htt.a//iiitteeeeataeeeeLL-------
They originally used BERT text encoder and CLIP image encoder. When replacing the BERT text encoder with CLIP text encoder (i.e., the cases D and in Figure 12), add a linear layer on top of the CLIP text encoder map the embedding dimension be the same as the Transformer head. For all fine-tuning experiments, we trained for 5 epochs, which is the default number of epochs (Shen et al., 2022).
F CLIPScore experiment: CAB suggests precaution in downstream applications of CLIP
As we see in Section 3, CLIP tends to treat input bag of concepts, which can have undesired consequences when use CLIP for downstream tasks. Here, examine recent application of CLIP, and find that it suffers from this phenomenon. CLIPScore (Hessel al., 2021) was recently proposed as way to assess the quality image captioning models. contrast reference-based scores, which compare the similarity between generated captions and reference captions, CLIPScore simply compares the similarity between the embedding from input images and the embedding from corresponding captions generated by an image captioning model. The original paper reports high similarity of CLIPScore to human judgement, which is one of the favorable attributes of CLIPScore compared to existing reference- based captioning scores. Here, we show that CLIPScore can be insensitive to swapping and shuffling of words in sentence, although humans
easily tell such differences.
We first use NLTK (Bird et al., 2009) to find part-of-speech tagging, and randomly shuffle the nouns within each sentence ('noun swapping"). We also prepare baseline where we shuffle all words in each sentence ('word shuffling"). We then evaluate the ground truth captions of the validation set of VQA-v2 (Goyal et al., 2017) using CLIPScore, and compare how shuffling procedures affect CLIPScore. In Figure 17, augmented version RefCLIPScore, while other reference-based score methods are affected. For example, given a sentence that reads "A man walking into the room". CLIPScore returns almost the same score for "A room is walking into the man." " This further illustrates that CLIP treats the input sentence as bag concepts, and calls for caution when we use CLIPScore to evaluate image captioning models.
that there is almost effect of our text permutations CLIPScore and its reference

---

Prediction accuracy: object names as GT vs. colors as GT
Color GT
Object GT
0.00
0.25
0.50
0.75
Figure 18: When both color and object names are available for ground truth labels, CLIP tends to pick object names over colors, suggesting that they are not completely interchangeable.
G Are the object name and attribute interchangeable?
We see evidence that the word "purple" serves as a replacement for the word "eggplant" in our CAB experiments so far, which leads to a caption such as "purple lemon" to represent an image of lemon and an eggplant. However, it would be especially surprising if the color attribute is completely interchange- able with the object name. To test this, we expand the labels we use for evaluating zero-shot transfer performance of CLIP. Previously, for color recognition task, use the colors as labels. We now include the object names our labels in addition the color labels. In particular, expand the label set from yellow, red, purple to yellow, red, purple, banana, tomato, ... eggplant. Therefore, if the object names and color attributes are not completely interchangeable and the object names are more suitable explain the image than attributes, then we should see a decrease in accuracy when we use the colors as our ground truth (GT) label. The results are shown in Figure 18. We see that the accuracy for "Color as GT much lower than "Object as GT", which suggests that CLIP only uses colors when object names are not available.
H Additional related work
Peculiarities of CLIP In the image generation community, it has been reported that state-of-the-ae models such as DALL-E (Ramesh et al., 2022) struggle with compositionality (Rassin et al., 2022). One of the potential causes of such failure has been attributed to the use of CLIP-based image encoder (Ramesh et al., 2022). In fact, image generation models that do not use CLIP such as Imagen and Parti are known to be better at generating images that require compositional reasoning (Saharia et al., 2022; Yu et al., 2022). However, few works go into depth analyze the behavior of CLIP in zero-shot image classification and visual question answering. Our analysis based on CAB offers perspective the weakness of CLIP-based models for compositional reasoning.


Reviews:
[{"review_id": "1TxdgZCeDe", "reviewer": "Reviewer_JUDd", "summary": "", "questions": "I had trouble understanding why the models were bad only on the compositional questions and not the individual ones. Can you please clarify that?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that the reviewer finds our work important and interesting. We’ll increase the figure sizes and fix typos in the revision.\n\n*Q. Why the models were bad only on the compositional questions and not the individual ones. Can you please clarify that?*\n\nThank you for a great question! For example, for a setup where we have an image of “purple banana” and nothing else, and ask for the color of banana (\"the color of the banana is [mask]\"), the reason why the model correctly responds as “purple” is that in the image, both the “purple” concept and the “banana” concept are present, and the query text that includes both the words “banana” and \"purple\" matches these visual bag of concepts better than alternatives.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think the paper asks a focused interesting question, exhaustively investigates that, proposes a mitigation mechanism, and clearly points out the limitations of the proposed solution. I like the expository style of writing.", "reasons_to_reject": "I don't see many reasons.", "questions_to_authors": "I had trouble understanding why the models were bad only on the compositional questions and not the individual ones. Can you please clarify that?", "paper_topic_and_main_contributions": "The paper investigates one important reason for the zero-shot failure of Vision Language models on VQA tasks. The main contributions of this paper can be summarized as follows:\n1. The paper identifies an important problem of VL models, specifically on compositional questions in VQA. They identify that VL models tend to treat each image + description as two separate \"bags of concepts\". While they start with CLIP, they experiment with various models and find this to be true in general.\n2. They attribute this to the contrastive loss training of the VL models.\n3.  They provide a fine-tuning-based solution to mitigate this problem.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. L096: sorely to solely\n2. Figures 4 and 7  are too small: if accepted should be increased in camera ready."}, "extra_text": {}}, {"review_id": "qL0BQWLDsq", "reviewer": "Reviewer_ftk9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that the reviewer finds the problem important and our experiments solid.\n\nThank you for suggesting these papers. We’ll make sure to discuss these in the revision. However, we would like to respectfully point out that the focus of our paper is not to propose new algorithms to solve the general problem of compositionality, but rather to provide new insights by reporting the interesting phenomenon of Concept Association Bias (CAB) as one of the reasons behind the frequently observed compositionality issues. In this way, our work aims to deepen our understanding of the source of compositionality issues. CAB targets a specific class of compositionality issues in existing V&L models. Indeed, none of the referenced papers [1-4] explicitly discusses Concept Association (e.g. “lemon” and “yellow”) as one of the reasons behind the compositionality issues of V&L models. To our knowledge, ours is the first work that points out this phenomenon. In addition, we have detailed studies of the properties of CAB, which provide the following insights:\n- We contrast the results from NCD and UNCD to have a more detailed understanding of the binding mechanism in CLIP; we also used ConceptNet to illustrate that the bias is amplified as the binding strength / concept association increases.\n- We illustrate that the CAB problem is not trivial by studying to what extent simple finetuning can fix it;\n- We study the correlation between CAB and contrastive & autoregressive objectives, which provides a better guide of designing objectives for large VL models and raises the caution when applying them to downstream applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The compositionality and attributed binding of VL models is an important problem. This paper systematically studies this problem and gives some insights.\n\nA comprehensive evaluation of different popular VL models ranging from models based on CE loss to models based on regressive loss, is performed.\n\nThe experimental parts are solid and CLIP with different encoders are evaluated.", "reasons_to_reject": "There is no new effective method proposed to address the Concept Association Bias problem even though the work did some study of the effect of finetuning.\n\nThe novelty is limited and many similar works have been done previously, e.g. [1][2][3][4].\n\n[1] Learning to Compose Soft Prompts for Compositional Zero-Shot Learning\n\n[2] Training-Free Compositional Image and Text Matching\n\n[3] Does CLIP Bind Concepts? Probing Compositionality in Large Image Models\n\n[4] Augmenting CLIP with Improved Visio-Linguistic Reasoning", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper find that CLIP’s zero-shot classification performance suffers when there is a strong concept association between an object and an attribute. Multimodal vision and language models as well such as BLIP, BLIP-2, and OFA are evaluated and it is observed that models solely relies on autoregressive loss exhibit minimal or no signs of CAB.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "n7x8k4D8z4", "reviewer": "Reviewer_9Cqi", "summary": "", "questions": "Question A: As described in line 412 - 414, `the CAB score stays around the same level for these different CLIP models`, what does `the same level` means? The scores range from 0.860 to 0.961(the difference is more than 0.1) are around the same level?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that the reviewer finds our work inspiring.\n\n- *Some of the observations can be more solid. For example, as described in line 493 - 495, the relation between the CAB Score and the accuracy can be quantified statistically whether the correlation is significant enough*\n\nThank you for this suggestion. We calculated Pearson's correlation coefficient between VQA-v2 scores and CAB scores and its corresponding p-value (line 493-495), and obtained the following significant relationship: Pearson's correlation coefficient r=-0.94 and p-value=0.017 (statistically significant under the conventional cut-off of .05). We'll include this in the text.\n\n- *As described in line 412-414, the CAB score stays around the same level for these different CLIP models, what does the same level means? The scores range from 0.860 to 0.961(the difference is more than 0.1) are around the same level?*\n\nWhat we mean by \"the same level\" is that the variability of CAB scores in Table 3 is in a similar range (0.860 to 0.961), compared to the variability of CAB scores in Table 2 (0.111 to 0.961), which highlights that the CAB depends more on training objectives & model architectures rather than model sizes. Thank you for pointing this out; we will clarify this in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Enough experiments and many inspiring findings. The Concept Association Bias did not be scrutinized carefully. \n2. Good writing. The structure of the paper is clear and reading it is smooth and natural.", "reasons_to_reject": "1. Some of the observations can be more solid. For example, as described in line 493 - 495, the relation between the CAB Score and the accuracy can be quantified statistically whether the correlation is significant enough.\n", "questions_to_authors": "Question A: As described in line 412 - 414, `the CAB score stays around the same level for these different CLIP models`, what does `the same level` means? The scores range from 0.860 to 0.961(the difference is more than 0.1) are around the same level?", "paper_topic_and_main_contributions": "This paper deeply investigates the Concept Association Bias (CAB) in the task of VQA. Through many experimental results, this paper finds that the models with CAB tend to treat input as a bag of concepts and attempt to fill in the other missing concept. This paper also observes that CAB is prevalent in vision-language models trained with contrastive losses.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work investigates a reason for the zero-shot failure of large-scale Vision-Language Models (such as CLIP) on compositional VQA questions, showing these models treat the image and the text inputs as two separate "bags of concepts" and attempt to fill in the other missing concept crossmodally, and that this issue is more prevalent in models trained with contrastive loss as opposed to an autoregressive loss. The authors further propose a fine-tuning based solution to mitigate this problem.
The work provides enough experiments using various models (including CLIP, BLIP, BLIP-2 and OFA), and more generally provides some interesting insights and inspiring findings for future work. The paper is well-written in an exposatory style and clearly structured.
Decision: Accept-Main