Text encoders bottleneck compositionality in contrastive vision-language models
Authors: Amita Kamath
Jack Hessel
Kai-Wei Chang
Venue: EMNLP 2023 Main (2023)
Keywords: vision-language
text encoders
interpretability

Abstract: Performant vision-language (VL) models like CLIP represent captions using a single vector. How much information about language is lost in this bottleneck? We first curate CompPrompts, a set of increasingly compositional image captions that VL models should be able to capture (e.g., single object, to object+property, to multiple interacting objects). Then, we train text-only recovery probes that aim to reconstruct captions from single-vector text representations produced by several VL models. This approach does not require images, allowing us to test on a broader range of scenes compared to prior work. We find that: 1) CLIP’s text encoder falls short on more compositional inputs, including object relationships, attribute-object association, counting, and negations; 2) some text encoders work significantly better than others; and 3) text-only recovery performance predicts multimodal matching performance on ControlledImCaps: a new evaluation benchmark we collect and release consisting of fine-grained compositional images and captions. Specifically, our results suggest text-only recoverability is a necessary (but not sufficient) condition for modeling compositional factors in contrastive VL models. We release our datasets and code.

Text encoders bottleneck compositionality in contrastive vision-language models
Amita Kamath
Jack Hessel
Kai-Wei Chang
University of California, Los Angeles
Allen Institute for Al
(kamatha, TwchangJ@cs, ucla.edu, jackheallenai. org
Abstract
CaseProvrts
ConnouarlaArs
Performant vision- languae (VL) models like CLIP represent captions using a single vec- tor. How much information about language is lost in this bottleneck? We first curate COMPPROMPTS, set of increasingly composi- tional image captions that VI. models should be able capture (e.g., single object, ob- ject+property, to multiple interacting objects). Then, we train text-only recovery probes that aim to reconstruct captions from single-vector text representations produced by several VL models. This approach does not require im- ages, allowing us to test broader range of scenes compared to prior work. We find that: 1) compo-
cheeah before 4ating batore fying
dhwerzanen
ean eatrader alse atteguual pansidet poddi iin lelping dog wfter sinylng parppaaaal
pawen unbella
Agenst uibrods
Contosthe Just sYoOdAy
Contastive VL model
can probe reconstruct input?? (83)
the model
image
matching? ($4 f)
chwetah cefors eat
predicts
boid button her light
($4.24.55)
bushoerraason an drvingg
Incorrect
prosober andlaal sriing iecturttr addecatec rriping polisiio stalion
CLIP's text encoder falls short sitional inputs, including object relationships, attribute-object association, counting, and nega- tions; 2) some text encoders work significantly better than others; and 3) text-only recovery performance predicts multimodal matching per- formance on CONTrOLLEDMMAArS ation benchmark we collect and release consist- ing of fine-graine compositional images and captions. Specifically, our results suggest text- only recoverability is necessary (but not suf- ficient) condition for modeling compositional factors in contrastive VL models. We release our datasets and code.
Tempora Relations
Spatial Re lations
Figure I: We present COMPPROMETS, a dataset of 18,100 text prompts, and CoNTROLLEDlMCAPS, a dataset of 600 image pairs+captions that differ by only word. The two datasets are grouped by the same set of caption properties, e.g., temporal/spatial relations. Experiments on COMPPROMPTS quantify the information loss of a text encoder, experiments on ConmOLLUMMCArs illustrate that information loss correlates with multimodal errors.
new evalu-
1 Introduction
Underlying several popular multimodal mod- cls like CLIP (Radford et al., 2021), DALL-E 2 (Ramesh et al., 2022) and ALIGN (Jia et al., 2021) is pooled text encoder, i.e., text representation model that outputs single vector for given input caption." In this work, bottleneck as interface to ask: how precise are textual representations of visually-descriptive lan- guage these modern multimodal models?
"A penguin on Mars wearing a spacesuit and walking a robot dog next Santa Claus." Riedl (2022)'s text-to-image query the type that modern multi- modal models should be able to support. is spa- tially precise (the dog is next to Santa, not front), compositional (robot dog, but not robot Santa), and imaginative (it is unlikely such an image exists al- ready). However, several recent works have shown that variety of multimodal models (despite achiev- ing strong benchmark performance) are frequently unable to reason about even simple spatial rela- tions or attribute attachments (Gokhale et al., 2022; Thrush et al., 2022; Yuksekgonul et al., 2023).
this representational
'Pooled text encoders (c.f., bidirectional multimodal en- coders) are used for variety of practical reasons: e.g., for guided diffusion (Dhariwal and Nichol, 2021). for fast k-NN queries over billions of images (Schuhmann et al., 2022), for contrastive objectives dependent on large batch sizes like Rad- ford et al. (2021)'s s 32K example "mini"-batee batch, etc.

---

iguana happy dinosaur
2015; Parcalabescu et al., 2021) and negations (e.g. "a cat that is not yawning"). We compare five text encoders, and find that top contrastive VL models: (1) are broadly ineffective at textually encoding spatial relations, numbers, and negations; (2) fre- quently cannot match attributes to their correspond- ing objects; and (3) fail more inputs grow more compositional. While some text encoders perform significantly better than others, all underperform proof-of- conccppt model which demonstrates that our prompts can indeed be compressed into single vectors with little information loss.
a surfer carrying a lifeguard an orangutan eating and an afficer flying
S CLIP text encoderr
CLIP image encoder
text decoder (inutared
(unusec)
In order to verify that our text-only probe pre- diets performance in multimodal settings, we cu- rate an evaluation set of image-caption pairs, CON- MOLLLOLMCAPS, which operationalizes the compo- sitional factors of CоMPPROMPTS in multimodal setting. Results on this corpus suggest our text- only probe gives necessary condition: if the text-only recovery probe fails to recover a text- only property on COMPPROMPTS, then the associ- ated multimodal model also performs poorly for that property on CONTROLLEDMYAAR, However, our results also suggest that text-only recover- ability is not sufficient condition: model can achieve low text-only information loss on a par- ticular prompt type but not fully solve it on CON- TIOLLEDMCAAS. To facilitate future probing experi- ments, we release our code alongside the newly col- lected COMPPROMPTS and CoNTROLLEDIMCAP cor- pora at httpps://githhe http://iittuu.ocooaaaaaaamao ami takamath/vl_ laaeccdders_rre_oottleeeeeee
iguana
an amusing dinosaur
a lifeguard carrying surfer
XX
orangutan and officer eating orangutan
We probe the representations of single-vector text encoders used in popular VI. models. Using a corpus of increasingly compositional image captions, COMPPReMETS, we attempt to generatively decode the original input sentence. Text encoders popular mod- els like CLIP fail to effectively encode precise aspects their captions like attribute attachments and object relationships (real examples shown, as Figure 1).
Figure 2:
Our strategy is as follows: for fixed pooled text encoder T x + y. which maps from cap- tions & to vectors y E Rd, we test how accurately x can be recovered by an expressive generative de- coder given y. i.e., (==(T((r) In an ideal case, T should result in no information loss, i.e., act reconstruction of the original caption should be possible, account for specific visual factors. However, we hypothesize that if a specific visu- ally descriptive property (e.g., a spatial relation) cannot be accurately decoded from y (using a de- coder trained with significant supervision), then is unlikely multimodal model can effectively use that property of using T. Different from exist- ing probes, ours does not require images, enabling exploration of broader range of captions, e.g., cre- ative text-to-image queries for which there may be no associated image (like "A penguin on Mars...")
Evaluation Corpora
2.1 COMPPROMPTS
We create an evaluation dataset of 18, 18,100 text prompts describing potential visual scenes with varying degrees of specificity and composition. Our starting point is animate nouns with corre- sponding verbs and adjectives from the Webl0K dataset (Kamath et al., 2022). We remove some synonyms to prevent ambiguity in the prompt (e.g. "a rhino to the left of a rhinoceros").
The prompts are increasingly compositional: They have 1 or 2 unique nouns, and 0. 1, or 2 attributes, of which there are 4 types: adjective, verb, spatial, and temporal. Nouns are randomly matched to generate prompts with two unique nouns - this results in unusual and imaginative text inputs that cannot be guessed based on priors learned during model pre-training (e.g., "a crab lift-
We execute our probe using an increasingly- compositional hierarchy of image captions cu- rate, COMPPROMPTS, which covers cases ranging from single object with attributes (e.g. "a cat") to multiple objects with attributes and rela- tions (c.g. "an orange cat to the left dog"). We also test counting (e.g. "three cats") (Seguí et al.,

---

ing a rhino"). The verb and spatial attributes can have either one associated noun (i.e. intransitive, e.g. "a koala yawning". policeman on the left") or two (i.e. transitive, e.g. "a poet chasing a rabbit", "a dinosaur left of a tiger"). We also test multiples and negations in the one-attribute setting.
Embed.
T(x)
Avg. EM (%)
size
CLIP ViT-B/32
512
13.2
CLIP ViT-L/14
768
28.5
negCIIP ViT-B/32
512
28.6
ROBERTACLIP VIT-B/32
512
28.9
Prompt examples of each type are given in Ta- bles 2 and 3. There are 300-500 examples of each prompt type in the dataset.
SBERT
768
41.6
Proof-of-coneepTS
1024
92.9
Table 1: Average EM performance of each text encoder on COMP CGMPPROMPTS, not including multiples and negations (reported Table
2.2 CONTROLLEDIyCAPS
We create a second evaluation dataset to evaluate the overall vision- laiionl model, rather than the text encoder specifically: where COMPPROMPTS con- tains text prompts alone, CONTROLLEDMMCAS con- tains 600 pairs of images, along with a correspond- ing caption for each image.
batch size of 512 for 4 epochs over CC3M; we select checkpoints with the lowest val loss. Mod- els are trained using 4xA60000 GPUs with 48GB of memory using Transformers (Wolf al., 2019) and accelerate At evaluation time, we generate cap- tions for COMPPROMPTS set using beam=5 search.
The images are sourced from the COCO valida- tion set (Lin et al., 2014), and the captions are hand- written to study one of six specific fine-grained textual properties: spatial relations with one asso- ciated noun, spatial relations with two associated nouns, temporal relations, verbs with one associ- ated noun, verbs with two associated nouns, or adjectives. For spatial relations, we evaluate only "left" and "right" (unlike COMPPROMPE which eval- uates also "above", "under", "in front of",, and "be- hind"), due to insufficient presence of other spatial relations clearly depicted the COCO data.
Text Models. We evaluate several T models: CLIP ViT-B/32 (12 layers, 512 dim) and VIT- L/14 (12 layers, 768 dim) (Radford et al., 2021), CLIP with RoBERTa-pretrainnn
text encoder (Liu et al., 2019; Ilharco et al., 2021), and Yuksek- gonul et al. (2023)'s more-order-aware CLIP en- coder finetuned with hard negatives, negCLIP. For comparison, we also consider the uni-modal Sen- tenceBERT (Reimers and Gurevych, 2019) model all-mpnet-bart bbbsee-2, which is trained on several sentence similarity datasets including COCO cap- tions (Lin et al., 2014).
A key property of ConTEoLLEDDMCAN is that only word changes between the two captions associ- ated with given image pair, such that the relation is changed or inverted: e.g., the caption pair "a per- son before opening an umbrella", "a person after opening an umbrella", along with the correspond- ing images for each (as in Figure 1) tests the overall model's understanding of temporal relations alone, without conflating any other types of reasoning.
Proof-of-concepTS5 We also consider a T5-large text encoder that produces a single vector output via mean pooling over the token embeddings. In contrast to the other fixed encoders, we fine-tune this model on CC3M, like an autoencoder". Then, the resulting encoder feature extractor, and hand a dimension-shufffee version of the sulting embeddings the probe. This "proof of concept" encoder is specifically optimized to gener- ate a vector from which a T5 model can decode the full sentence, and serves to validate that our probe setup is even possible.
3 Text-only Recovery
For a given text encoder T, our first step is to obtain a training corpus of representations to a decoding probe (x)) We use (just the text of) Con- ceptual Captions 3M (Sharma et al., 2018) (CC3M) split into 90/10 train/val set; this corpus consists of cleaned alt-texts from web images, and thus is similar to the pretraining corpora of many VL mod- els. For P. we use T5-large: specifically, we con- dition the decoder on T(x). followed by a linear transformation and layer normalization. We train using Adafactor (Shazeer and Stern, 2018) with
We evaluate using exact match (EM). While we report BLEU scores in the Ap- pendix, for our bigh-prrcision setting, partial credit metrics are too generous, e.g., generating "a
Evaluation.
2 Pt hnttps://githhi com/hhugingfaac/accellee. "There is no overlap hetween CC3M and CoMrProMPTS.

---

starilno
2utribuna
+.2-obi
usique +lgeet
RESE
151
197 saage
F-oa conning
orsnge and gooded
cal lof
cat
enangy
CLIP WI--522
34
KIBEKTL
99.4
28.4
SOURT
540
91.5
91.8
7K4
21.2
d
dEaTD do Agg
SAVSSSE
EXT
DESTOV corunda
day/as
of
Exttry
CLIF W5--022
13.4
15.5
6.2
1.5
17.8
6.2
53
1.2
-4
1.2
1.
3,
1.3
CLIP WI1-/144
45.4
10,2
17.
50
59.2
23.6
14.0
14
21.0
155
2.5
05
0.1
21.6
4.3
MIgCT IP NTE-11
12.4
11.4
36.
15.3
364
156
26.8
43
312
76A
16.4
12.4
74
192
1.1
KOBE NTT
45.6
150
34.0
14.6
48.5
240
20.7
50
293
10.5
39.
56
0.4
253
1.6
SBEKT
550
41.8
448
11.2
48.9
32.8
30.3
5.0
302
31.9
23.
3.0
4.4
4.8
10.8
Table 2: Prompt example and exact match (% EM) score of reconstruction from all models, averaged over several hundred instances each. As imputs become more compositional, Lasion-aanguag text encoders perform increasingly poorly text reconstruction.
porter on top of a penguin" as "a penguin on top a hill" scores 48 BLEU-4 points. Similarly for BERT-Score (Zhang et al., 2020), where generat- ing "two rabbits and three shrimps" as "four of the shrimps and rabbit" scores 0.91 F1.
EM), particularly those between two objects (aver- age 13.8 EM). SBERT, in comparison, scores 36.9 and 22.3 EM, respectively.
Temporal relations. VL models perform poorly on temporal relations, scoring on average 17.1 EM. In comparison, SBERT scores 29.6 EM likely because temporal relations appear more frequently in language than in web alt-text.
3.1 Text-only Recovery Results
Table I presents the average exact match of each model over the corpus of prompts in COMPPROMPTS, excluding negations and multiples, which are re- ported Table 3. The proof-of-coneepT5 model's high performance illustrates that is possible in theory to nearly exactly decode all captions in COmpPRoMPtS using T5-large, given the "right" en- coding*. Beyond proof-of-concep15. the best per- forming model is SBERT; and the best performing multimodal model is RoBERTa-CIII..
Transitive intransitive verbs and prepositions. On transitive verbs (e.g.. "'caasin")) CLIP ViT- B/32 and ViT-L/14 do worse by an average of 21 EM than vs. intransitive verbs (e.g.. "yawning"). whereas negCLIP and RoBERTa-CLIP do better by an averaged 18.7 points. On transitive prepositions ("to the left of"") instead of intransitive ("on the left"), all models do worse by an averaged 35 EM.
3.2 Fine-Grained Results on Different Prompt Types
Negations and multiples. Models perform poorly on negations (average EM 13.0) and multi- ples (average EM 5.1). This agrees with previous observations that VL models struggle with count- ing (Seguí et al., 2015; Parcalabescu et al., 2021).
Table 2 contains EM results of all models the various types of prompts in COMPPROMPTS. A sep- arate study on multiples and negations in Table shows that text encoders struggle to encode those as well. These results show that is fairly difficult to decode input sentences from text representations for most VL models, with inereasingly composi- tional categories proving more difficult (e.g., "an orange cat" to "an orange cat yawning" to "an or- ange cat chasing dog").
Prompts where word order matters. VL text encoders struggle to capture word order: on prompts where word order matters less (e.g., "a cat and a dog"), they score an average of 34 EM, but where word order matters more, they score an aver- age of 15.8 EMM. The failure cases are often caused by assigning attributes to nouns incorrectly, high- lighted in the Appendix. This extends Thrush et al. (2022)'s 's and Yuksekgonul al. (2023)'s finding that contrastive VI. models can behave like bags- words - this issue manifests just in the text encoder as well.
Text encoders of VL models struggle to represent spatial relations (average 23.7
Spatial relations.
Moot errors made by proof-of-conceoT55 are minor e.g., "two physicians on the right" - "two physician on the right".

---

aterhoos
EM is 29 points higher. This provides evidence that text pre-training the text encoder helps negations, presumably because negations are less likely alt-texts compared to other settings.
Th-yy
xgg
mallipi
sbject
Unique
Ses ensage
000
skr lyp
Pack extrios
CLIP VITI
CLIP WILHH
10.5
16.1
aGCLIP WIB
11.2
Increasing model size helps overall, but not on spatial relations. The average EM of CLIP ViT- L/14 on prompts that do not include spatial rela- tions is 20.7 points higher than CLIP ViT-B/32. However, on the prompts that do include spatial re- lations, its average EM is only 4 points higher. The modest increase of text encoder size in the CLIP training regime appear less reliable for encoding spatial relations than text pre-training or hard neg- atives (though, more significant scaling could be beneficial, as in Imagen (Saharia et al., 2022)).
unigee
cato
eath saungg
etpcte++
and
fhr Joth of
aslliplo
Staragge
four days
Joar dagr
CLIP VIT-MAI
CLIP VITSUM
MSCLIP WIB
B02
Lmiqu
viat
shat sot
abject
megetiins
shary
the 15
CLIP 545
15.0
CLIP WI--U1
SOCLIP WIFR02
18
umique
Vhat
zoal
sbjecte+
nesetien
the 12
is sky
CLIP VITIR/13
CLIP WILJI
#OCLIP WIB822
Hard negatives from Yuksekgonul et al. (2023) help, especially where word order matters. On average, negCLIP does 15.4 points better than CLIP. On prompts where word order matters (c.g. "a cat chasing a dog"). it scores 16.3 points higher; on prompts where word order does not matter (e.g. "a cat and a dog"), it scores 12.8 points higher.
Table All models' EM on prompts that contain multi- ples or negations. Text recovery of these inputs is very poor, likely because multiples and negations tend to be infrequent image captions.
Adjectives and verbs. VL models perform rel- atively well in the basic one-object, one-attribute setting on both adjectives (average EM 44.8) and verbs (average EM 34.5): even higher than the zero-attribute setting, where error analysis reveals they tend hallucinate information ("a tarantula" "a tarantula in a hand"). While these numbers are well behind SBERT (EM 91.8 and 78.4 respec- tively), they agree with previous observations that VL models exhibit good visual recognition of basic adjectives and actions (Radford et al., 2021).
3.4 Incorrect Model Predictions
We manually inspect models' incorrect predictions. Decoded VL text encoder predictions often come close (e.g. "three shrimps" - "three of shrimp" is pattern shown by CLIP VIT-B/32, CLIP ViT-L/14 and negCLIP)) whereas SBERT's incorrect decod- ings fall further afield (e.g. "three gardeners" "three gardeners and a third man."). Thus, while the superior results of the unimodal SBERT compared to the VI. text encoders when evaluated in the same frozen-encoder setting (including CLIP ViT-L/14, which has the same text embedding size) show that there is significant room for improvement for VL text encoders, the types of errors made by each model may not be fully captured by EM. Nonethe- less, EM remains an appropriate metric for our high- gich-precision setting, as discussed Section
Compositionality. Text encoders struggle with increasingly compositional information, e.g., the probe decodes SBERT("a dentist after examining an ape") - "an ape after examining a dentist". On average, performance on two unique objects drops by 49% from their performance on one unique ob- ject (for CLIP ViT--B32, it drops 71%). VL model performance drops on average by 35% when the prompt contains two attributes compared to one.
4 Experiments and Results in the Multi-modal Setting
3.3 Fine-Grained Results for Different Model Designs
We investigate the hypothesis that if a textual prop- crty cannot be decoded from the VL text encoder's vector representation with highly expressive de- coder (like T5-Large), then it also cannot be readily modeled in the multimodal setting. CONTROLLED- ImCaps studies the attributes from COMPPROMPTS in the multimodal setting. We then compare the text
Pre-training the text encoder helps, especially on negations. The average EM of RoBERTa-CLIP on prompts without multiples or negations is 15.7 points higher than CLIP V/T-B/32. However, on the prompts that do include negations, its average

---

Tpe
Comparable Piompla Confloont
Exmple from CoNsoLunHCAA
EM on
CIC
CIC
Prompt Type
Image
Text
COMPPROMPTS
Spatial 1-obj L/R
28,5
4.0
15.0
Spatial 2-obj I./
4.4
4.0
8.0
car on the rihht
Temporal
26.8
7.0
30.0
Verh 1-ohj
71.6
84.0
89.0
vilee ebjeci 4 udl
Verb 2-obj
46.0
Adjectives
81.8
65.0
85.0
Juarus to
vollo
unipa pojec semporal
Table 4: Performance of CLIP ViT-L/14 text encoder (%) on the equivalent prompts in COMEPhoMPTS, and per- formance of CLIP VYT-L/14 full model on CONTROLLED- ImCArS (CIC). On the types of prompts where the text encoder performs poorly, so too does the overall model.
Tarpoul
sblect semporal seitin of verb
atoeds
abrelling
than the incorrect caption when given an image. and an image score, the fraction of instances where a model scores the correct image higher than the incorrect image when given a caption.
Zuripu objech Bet/vers
ekchhmm
4.1 Multi-modal Results
us Bydlem
Table 4 presents the results of CLIP ViT-L/14 on both COMPPROMPTS and CONTROLLEDIMCAPS (all model results in Appendix). The COMPPROMPTS re- sults correspond to the prompt type(s) most closely matching the captions in COnTiOLLADMMCAr (spec- ified in Figure 3). For the spatial relations, for this table alone, we calculate the EM on the data points in COMPPROMPTS containing "left" and "right" spa- tial relations only due to lack of sufficient support in COCO for other spatial relations, as discussed in Section 2.2. On prompt types where the text encoder performance on COMPPROMPTS is poor, the overall model performance on CONTrOLLEDIMCArS is also poor: showing that the text encoder does indeed bottleneck VL models' compositionality.
Figure 3: Each attribute in CONToLLLll-CAP., with comparable prompts in COMPPROMPTS and an example.
encoder performance of VL model on particular prompt type in COMPPROMPTS with the performance of the overall VL model on that prompt type in CoNTroLLenIwCAIS.. As diseussed in Section 2.2, the two captions in every example differ by only one word which changes or inverts the relation, al- lowing us to perform fine-grained analyses in con- trolled settings without conflating multiple types of compositional reasoning. Figure depicts the six types of attributes studied in CoNTROLLEDIMCAPS,, their corresponding prompt type in COMPPROMPTS, and an example of each.
We see similar findings per prompt type and model design as those discussed in Section 3.3.
VL Models. We evaluate the same VL models in Section 3: CLIP ViT-B/32, CLIP ViT-L/14, CLIP with RoBERTa-pretrainee text encoder (Liu et al., 2019; Ilharco et al., 2021), and negCLIP (Yuksekgonul et al., 2023). Each of these mod- els can return a score when given an image and a caption, representing how well they match.
4.2 Fine-Grained Results Different Prompt Types
We discuss findings on the prompt types in CON- TROLLEDMMCAPS, with 95% confidence intervals.
Models do poorly on spatial relations. On aver- age, VL models perform poorly on spatial relations, achieving average image text score of 2.5 12.4 (=: 2.2 3.7). Their text encoder performance on the corresponding prompts in COMPPROMPTS was similarly poor, with an average EM of 27.8. This agrees with Kamath et al. (2023), which shows that VL models struggle with spatial relations.
Evaluation. We follow the evaluation scheme from Winoground (Thrush al., 2022): for given pair of images with corresponding captions, we measure both a text score, the fraction of instances where a model scores the correct caption higher

---

Models do poorly on temporal relations. VL performs poorly on temporal relations, with an av- erage image text score of 5.3 30.8 (+: 2.7 4.8). Their text encoder performance on COMPPROMPES temporal reasoning was similarly low at 18.9 EM.
4.4 Text reconstruction appears to be necessary...
To study the relationship between text reconstruc- tion and overall model performance beyond Table 4, we evaluate text reconstruction on CONTIOLLEDIY- CAPS. Specifically, we use the trained T5 decoders from Section and try to reconstruct the input when CoNTROLeEDlMAA text inputs are evaluated. On the cases where the reconstruction is incorrect according to human evaluation on either of the two text imputs, the overall model Image Score ControllidiCCas for CLIP ViT-L/14 is zero 96% of the time, and the Text Score is zero 83% of the time. This text reconstruction vs. multimodal matching correlation is more direct compared the similar correlation reported in Table because we compare on the same instances.
Models do well on verbs and adjectives. VL models perform well on verbs (average image text score 65.4 78.1, + 5.0 4.8) and even better on adjectives (average image text score 78.5 89.0, + 7.0 3.5), mirroring their text encoder performance on COMPPROMPTS, where the average EM for verbs and adjectives were 33.7 and 44.8 respectively.
Two-object verbs are more difficult than one- object verbs. We find that for all models, two- object verbs are harder than obe-object verbs, with the former achieving an image text score of 52.3 68.5 and the latter 78.5 87.8 (with p 0.05 under the Wilcoxon signed-rank test). This follows performance on COMPPROMPTS for VIT-B/32 and ViT-L/14, but not for negCLIP and RoBERTa-CLIP, hinting that ability to reconstruct is necessary but not sufficient, as discussed in Section 4.5.
4.5 but insufficient.
Conversely, just because model performs well the COMPPROMPTS probe does not mean it will perform well CONTROLLeDMCAAPS. For exam- ple, ViT-L/14 outperformed ViT-B/32 overall on COMPPROMPTS, but not (statistically significantly) on CONTROLLEDIMCAPS. Also, RoBERTa-CLIP out- performs ViT-B/32 on temporal relations on COMP- PROMPTS, but achieves a worse text score on CON- TROLLEDIMCAPS. When we evaluate text reconstruc- tion on CONTROLLEDIMCAPS, cases where the re- construction is correct for both text inputs, the over- all model Image Score on CoNTROLEEDIMMARS for ViT-L/14 is zero 59% of the time, and the Text Score is zero 47% of the time. This suggests that text recoverability
4.3 Fine-grained results different model design choices
We discuss findings on the model designs in CON- MoILEnMMaAPS. All findings are statistically signif- icant at p < 0.05 using the Wilcoxon signed-rant test to compare models.
Pre-training the text encoder improves text score on verbs. RoBERTa-CLIP obtains a higher text score than CLIP ViT-B/32 (78.0 vs 68.0), as well as a higher EM on the prompts in COMP- PROMPTS corresponding to verbs (39.4 vs 11.0).
necessary but insufficient condition for overall model performance. The in- sufficiency is intuitive, as multimodal errors could potentially stem from the image encoder.
Increasing model size does not help on spa- tial or temporal reasoning, On both spatial and temporal reasoning inputs, ViT-L/14 performance on CoNTROLLEDIMCArS was not statistically signifi- cantly higher than that of ViT-B/32.
4.6 A Note on Winoground
We evaluate our four VL models the Winoground dataset (Thrush et al., 2022). They perform poorly, with an average image text score of 10.3 30.8, where random chance 25.0 25.0. However, on shorter inputs (5 words or less) which cxhibit fewer compositional concepts on average, e.g., "a bird eats a snake" | "a snake eats a bird"', the four models achieve higher scores of 20.4 47.2 on average. On longer (over 10 words), more
Hard negatives from Yuksekgonul et al. (2023) help where word order matters. On prompts where word order matters, negCLIP scores an im- age text score of 36.5 50.0 and COMPPROмPTS EM of 27.2, and other models score an average im- age text score of 24.8 37.5 and a COMPPROMPTS EM of 21.0. negCLIP also outperforms ViT-B/32 on all prompts on average.
'For the simple inputs of CoNmoLLLIlyCAPs, we found human evaluation hy the authors tractable, with the added advantage of not penalizing minor emrrr as EM does.

---

compositional inputs, e.g. "in the stadium, the per- son wearing gray outperformed the one wearing blue"| "in the stadium, the person wearing blue out- performed the one wearing gray", models achieve a much lower score of 3.4 18.5. This mirrors our finding on COMPPROMPTS that VL text encoders struggle with increasingly compositional inputs.
Conclusion and Discussion
We present probing results that suggest significant information loss upon text encoding of composi- tional inputs in vision and language models. This information loss is quantified using COMPPROMPTS, a test set of increasingly compositional image de- scriptions, and CONTROLLEDIMCATT we use to verify that this information loss affects the performance of multimodal models on compo- sitional inputs. Harder negatives, more text pre- training, and larger models all improve encoder quality, but information still lost even for the most performant models, compared to the uni- modal SBERT as well as a T5-based auto-encoder. Going forward, even more difficult test sets than COMPPROMPTS and CONTROLLEDIvCAN might be re- quired to analyze and evaluate vision-language model capabilities. Returning to Riedl (2022)'s tweet from the intro, "A penguin on Mars wear- ing a spacesuit and walking a robot dog next to Santa Claus.", ". even our highly accurate proof-of- concepTTS model struggles, predicting: "compul- sory penguin onexposition wearing spacesuit and walking dog robot next tohoc". To support imagi- native rext-to-image generation queries (for images that may not exist yet), future work would be well- suited to design text encoders that can generalize to captions that contain compositions never-before- seen in web alt-text corpora.
test set that
5 Related work
Building models capable of reasoning jointly about visual and textual inputs is a long-standing goal of AI (Winograd, 1971), with potential applications in the fields of vision-language navigation (Anderson et al., 2018), human-robot interaction (Matuszek et al., 2012), accessible image captioning (Gurari et al., 2020), etc.
Recent challenge datasets have been designed to probe the capacity of multimodal models to rep- resent descriptions of precise visual compositions (Johnson al., 2017; Suhr et al., 2019; Hudson and Manning, 2019; Thrush al., 2022). Yuksek- gonul al. (2023) and Yamada al. (2022) study CLIP specifically, demonstrating its shortcomings (and some potential fixes) in terms of modeling syntax. Ma et al. (2022) study OpenCLIP models for various types of compositional reasoning, with programmatically sourced hard negatives. Differ- ent from these works, our textual probe does not require access to images.
Our probing results suggest two future modeling directions: (1) Modifying contrastive VL models' training objectives additionally encourage abil- ity to reconstruct the text input, either through an additional reconstruction loss on the text encoder during finetuning, or through the addition of even harder negatives than Yuksekgonul et al. (2023) and Ma et al. (2022), would be an exciting avenue for future work. Alternatives to contrastive train- ing, such as captioning. have also shown promise in recent work (Tschannen et al., 2023); and (2) explicitly encouraging linear recovery with a modi- fied loss function: while the gap between SBERT and the VL Text encoders can be partially explained by the superior pooling method and training data, SBERT's training objective does not require linear recoverability (whereas CLIP's dot product inter- action term might): explicitly encouraging linear text-text recoverability might improve multimodal performance. Finally, we hope that CONTROLIER LEDM-- CAPS can facilitate research beyond single-vector bottleneck VL models.
Our image-and-text evaluation most closely re- sembles Thrush et al. (2022). However, we stratify the examples based on type of input (e.g., tem- poral relations) to provide more detailed insights. We also keep our prompts relatively simple, never having more than two objects or two attributes in the input. We believe this is a more realistic goal for our current vision-language models. The word order shuffling aspect is also discussed in Yuksek- gonul et al. (2023). However, as their proposed benchmark does not provide pairs of images with corresponding captions, it is possible to achieve state-of-the-ard with text-only model (specifically, 2-shot ChatGPT (Ouyang et al., 2022), details in Appendix and the recent Hsieh al. (2023)). While this does not detract from their finding that vision- language models ignore word order, our bench- marks have an additional advantage of being insen- sitive to text-only priors.
htttps: //tp:/fo/.....m.mo/om/ openai. cow./oocs/ api-reference/ /hat, using the gpt-3. 5-tubo model

---

Limitations
Danna Gurari, Yinan Zhao, Meng Zhang, and Nilavra Bhattacharya. 2020. Captioning images taken by people who are blind. In ECCV. Springer.
First, our probing method involves a pre-trained T5 decoder. It is possible that language biases from the pre-training emerge while decoding from the VL text embedding, e.g., predicting "a dog chasing a cat" instead of "a cat chasing dog" because the former is more likely under the T5 decoder's priors from pre-training. However, as the methodology is the same across all models we evaluate, we believe that the evaluation is fair. Second, we evaluate with only one probe, whereas probing with com- plementary methods (e.g., especially deterministic ones, like a convex linear probe) could reveal more insights. Third, text encoders that do well on our evaluation may not perform well if directly plugged into a contrastive VL model like CLIP, if the text encoders were not trained encode the informa- tion a manner that linearly recoverable.
Cheng- Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. 2023. Sugarerepe: Fixing hackable benchmarks for vision-language compositionality. In Thirty-Seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Drew A Hudson and Christopher D Manning. 2019. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pages 6700-6709.
Gabriel Ilbarco, Mitchell Wortsman, Ross Wightman. Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Lud- wig Schmidt. 2021. Openclip.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le. Yun-Hsuan Sung. Zhen Li, and Tom Duerig. 2021. Scaling up vi- sual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning.
A Acknowledgements
The authors thank John Hewitt, Akhila Yerukola, and anonymous reviewers for useful discussion and feedback. This work was funded by the Allen In- stitute for AI. AK was additionally supported by the UCLA Computer Science Department First- Year Fellowship. KC was supported in part by U.S. DARPA MCS Program under contract num- ber N66001 1924032, U.S. DARPA ECOLE Pro- gram No. HR001 12390060, and ONR N00014-23- 1-2780, and a Sloan Fellowship. The views and conclusions contained herein are those of the au- thors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. CLEVR: diagnostic dataset for compositional language and elementary visual reasoning. In CVPR.
Amita Kamath, Christopher Clark, Tanmay Gupta, Eric Kolve, Derek Hoiem, and Aniruddha Kembhavi. 2022. Webly supervised concept expansion for gen- eral purpose vision models. ECCV.
Amita Kamath, Jack Hessel, and Kai-Wei Chang. 2023. What's "up"" with vision- language models? inves- tigating their struggle with spatial reasoning. In EMNLP.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In European conference on computer vision, pages 740-755. Springer.
References
Peter Anderson, Qi Wu. Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. 2018. Vision- and-language navigation: Interpreting visually- grounded navigation instructions in real environ- ments. In CVPR.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv,
Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. 2022. Crepe: Can dision-language foundation models reason com- positionally? In CVPR.
Prafulla Dhariwal and Alexander Nichol. 2021. Diffu- sion models beat gans on image synthesis. NewrIPS.
Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vib- hav Vineet, Fric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. 2022. Benchmarking spatial rela- tionships in text-to-image generation, arXiv preprint arXiv:2212.10011 10015.
Cynthia Matuszek, Nicholas FizGGrral, Luke Zettle- moyer, Liefeng Bo, and Dieter Fox. 2012. A joint model of language and perception for grounded at- tribute learning. In ICML.

---

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. NeurlPS.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic im- age captioning. In ACL.
Noam Shazeer and Mitchell Stem. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning. pages 4596 4604. PMLR.
Letitia Parcalabescu, Albert Gatt, Anette Frank, and lacer Calixto. 2021. Seeing past words: Testing the cross modal capabilities of pretrained V&L mod- els counting tasks. In Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR).
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang. Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in pho- tographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. Winoground: Probing vision and lan- guage models for visio-linguistic compositionality. In CVPR.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try. Amanda Askell, Pamela Mishkin, Jack Clark, al. 2021. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pages 8748 8763. PMLR.
Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. 2023. Image captioners are scalable vision learners too.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint 27X
Terry Winograd. 1971. Procedures as a representation for data in computer program for understanding natural language. Technical report.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process- ing and the 9th International Joint Conference on Natural Language Processing GEMNLP-IJCNLA
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi. Pier- ric Cistae, Tim Rault, Rémi Louf, Morgan Funtowiez, et al. 2019, Huggingface' ' transformers: Statc-of- the-art natural language processing. arXiv preprint 190.03791.
Mark Riedl. 2022. A penguin on mars wearing space- suit and walking robot dog next to santa claus. Tweet ID 151174578187051111
Yutaro Yamada, Yingtian Tang, and llker Yildirim. 2022. When are lemons purple? the concept association bias of clip. arXiv preprint arXiv:2212.1 12043.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2023. When and why vision- Jangua-an models behave like bag-of words models, and what to do about it? ICLR.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontij-Lopes, Bureu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. 2022. Pho- torealistic text-to- text-to-image diffusion models with deep language understanding. In Advances in Neural In- formation Processing Systems.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis. Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Lud- wig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022. LAION-5ba An open large-scale dataset for training next generation image-text models. In Thirty- sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Santi Seguí, Oriol Pujol, and Jordi Vitria. 2015. Learn- ing to count with deep object features. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 90-96

---

A Additional Results
Table contains average BLEU-4 scores of the models. Table 6 contains study of model per- formance on object-attribute association in COMP- PROMPTS. Table 7. Table 8 and Table 9 con- tain results of other models on CONTROLLEDIMCArS in comparison to COMPPROMPTS (ViT-L/14 is dis- cussed in Table 4). Table 10 discusses text-only results on the ARO benchmark (Yuksekgonul et al., 2023).
CIC
CIC
EM on
Prompt Type
Image score
Text
CoMPPRONITS
score
Spatial 1-obj L/R
27.2
1.0
10.0
Spatial 2-ohj L/R
4.0
Temporal
4.0
35.0
Verb 1-obj
15.4
71.0
77.0
Verb 2-obj
6.6
43.0
59.0
Adjectives
38.2
74.0
92.0
Embed. size
Table 7: Performance of CLIP VIT-B/32 text encoder
T(x)
Avg. BLEU-4
(%) on the equivalent prompts in COMrPRoMPTS, and per- formance of CLIP ViT-B/32 full model CONTROLLID- InCApS (CIC).
CLIP VIT-B/32
512
33.3
CLIP VIT-L/14
768
46.0
NegCLIP ViT-B/32
512
50.2
RoBERTACLIP VIT-B/32
512
52.0
SBERT
768
56.7
Table 5: Average BLEU-4 performance of each text
including multiples and negations. The trend correlates with EM %, but the eval- uation itself is too lenient for our purposes, as described in the main text.
encoder on COMPPROMITS,
CIC
CIC
EM on
Prompt Type
Image
Text
ComPPRoMPTS
score
Spatial I-obj L/R
26.0
1.0
13.0
Spatial 2-obj L/R
15.0
3.0
13.0
Temporal
23.0
8.0
35.0
T(x)
Embed. size
Shuffled %
Verb 1-obj
20.6
84.0
94.0
Verb 2-obj Adjectives
39.4
70.0
87.0
CLIP ViT-B/32
512
42,2
95.0
92.0
CLIP VIT-L/14
768
55.5
RegCLIP ViT-B/32
512
37.2
Table 8: Performance of negCLIP ViT-B/32 text en-
RoBERTaCLIP VIT-B/32
512
62.8
coder (%) on equivalent prompts in COMrPRoMPTS, and performance of negCLIP ViT-B/32 full model CoNMOLLEOIMCArS (CIC).
SBERT
768
44.2
Table 6: Of the times the model gets the words in the prediction correct. Shuffled the percentage of when it gets the word order incorrect (in the prompts where word order matters, unlike "cat and dog" -pecificall... where attributes must be associated with the correct object). Clearly, negCIIP having heen trained with hard negatives involving word order shuffling allows it to perform the best. All models suffer from poor object attribute association.
CIC
CIC
EM
Prompt Type
Image
Text
CosrPRovPTS
score
score
Spatial I-obj L/R
92.3
1.0
10.0
Spatial 2-obj L/R
28.3
2.0
20.0
Temporal
18.5
2.0
23.0
Verb I-obj
75.0
91.0
Verb 2-obj
48.8
50.0
65.0
Adjectives
17
80.0
87.0
Table 9: Performance ROBERTa-CLIP VIT-B/32
text encoder (%) on the equivalent prompts in Covp- PROMPtS, and performance of RoBERTa-CLIP ViT-B/32 full model on CoNTROLLEDIvCASS (CIC).

---

Dataset IeICT.IP
ChatGPT 2-shot
VG-Relation 0.81
0.90
VG-Atribution 0.71
Flickr30k-PRC
0.91
0,86
COCO-PRC VIT-B/32
0.86
0.86
Table 10: Performance of ChatGPT 2-shot the ARO benchmark (Yuksekgonul et al., 2023). While the dataset was designed to test VL models' sensitivity to word order shuffling (which is orthogonal text-only performance on the same data), the textual priors that exist in ARO (c.g., "horse eating grass" more likely than "grass eating horse") are less relevant to the prob- ing experiments for CoMPProMPTS (because the probe must reconstruct any given caption in COMPPROMITS, in- cluding unusual ones, e.g., "five teenagers riding three butterflies") and do not exist ControulinMICAN due to the maired-image construction.


Reviews:
[{"review_id": "UiKiP7EwVF", "reviewer": "Reviewer_iU94", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank all the reviewers for their detailed and constructive reviews. Thank you to Reviewer iU94 for pointing out our comprehensive analysis, insights, and the potential uses by the community for evaluation and informed model design.\n\n> **Detailed discussion of the results in Table 2:**\n* Thank you for pointing this out! Counter to most of our results, for which more details in the prompt make it harder to recover, the models indeed perform better at recovering adjective-noun pairs than single nouns (as seen in Table 2, “a cat” vs “an orange cat”). Looking at the errors, the models tend to output _more_ than is required for single nouns, i.e. they hallucinate incorrect information (e.g. “a tarantula” → “a tarantula in a hand”, “a schoolboy → “a schoolboy in school” in the case of CLIP ViT-B/32). \n* We will add a more detailed discussion to the final version of the paper that would target more specific phenomena such as the ones you pointed out, along with observations based on error analysis.\n\n> **T5 Decoder training:**\n* Thank you, we will certainly elaborate further on the T5 decoder training. The VL model text embedding is put through a linear transformation+layer normalization (L169-171) to convert it to a uniform embedding size fed into the T5 decoder.\n\n> **Why EM and not, e.g., BLEU?**\n* As suggested, we did consider text generation scores such as BLEU (Appendix Table 5). However, as discussed in L202-210, metrics such as BLEU and BERTScore are too lenient for our precise evaluation setup where one word can imply wildly different circumstances (e.g., “a cat on the left/right”). We think exact match is a fair evaluation metric for measuring information loss in a setup where models should be able to perfectly recover their inputs.\n\n> **Impact of Vision Encoder on ControlledImCaps results:**\n* We agree that an error on ControlledImCaps could be because of the vision encoder: in Sections 4.4-4.5 we discuss text recoverability as a necessary (but not sufficient) condition --- the insufficiency stems from the vision encoder, among other factors. We will add this discussion to the paper.\n* Regarding control over varying vision encoders, three of our four models actually use the same vision encoder architecture: CLIP ViT-B/32, negCLIP ViT-B/32 and RoBERTaCLIP ViT-B/32. The parameters of these vision encoders, however, are different, because they were trained in conjunction with their varying text encoders. \n\nThank you again for your thoughtful comments, and your suggestions to improve the readability of our results. Please let us know if you have any further questions or comments, we would be happy to follow up!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper include its comprehensive analysis of the limitations of current text encoders in contrastive VL models,\nthe proposal of two new evaluation benchmark called CompPrompts and Controlled-ImCaps, and the introduction of text-only\nrecovery method to evaluate the performance of text encoders in VL models on CompPrompts.\n\nIf this paper were to be presented at the conference or accepted into Findings, the main benefits to the NLP community would be\nthe insights it provides into the performance bottlenecks of text encoders in VL models and the proposed methods for evaluating\nand improving their performance.\n\nThe paper's findings could inform the development of more effective text encoders for contrative VL models, which could have\napplications in a wide range of areas. It may also aid the future work in designing text encoders that can generalize to captions that\ncontain compositions never-before-seen in web alt-text corpora and therefore, support imaginative text-to-image generation\nqueries (for images that may not exist yet).\n\nAdditionally, the proposed evaluation benchmarks could be used to compare the performance of different text encoders and VL models, which could facilitate the development of more accurate and robust models in this area.", "reasons_to_reject": "The weakness of this paper is as follows:\n\n\n1. In the experiments on the comp-prompts dataset, it fails to draw effective conclusions in a series of statements in the paper. For example, in Table 2, when comparing the EM scores for column 1 adj +1 spatial or column 1 adj +1-obj-verb with 2 attributes and 1 unique object to those for column 1 spatial or column 1 adj with only 1 attribute, it is found that CLIP ViT-B/32 has a higher score (47) in the former case than all three columns in the latter case. Similar observations can also be seen in negCLIP ViT-B/32.\n\n\n2. Text recoverability of recovery-generated text is used to evaluate Text encoder's performance. Based on EM score metric, conclusions are drawn regarding Text encoder's shortcomings in different factors. However, there exists a gap between text encoder and T5 decoder since authors use decoder of Fine-tuned T5 model to reconstruct input text. I hope authors can provide an datailed explanation on how T5 decoder reconstructs input text from different text encoder embeddings especially when these embeddings have different sizes. Moreover, this gap may potentially affect the content of recovered texts. Since EM score serves as a sensitive metric for evaluation purposes and its measurement might be the reason for performance differences of text encoders when recovering different factors' contents. It is also mentioned by authors in Sec.3.4. but to further analysis. Based on the analysis of EM scores, there is not enough convincing evidence to suggest that the text encoder's ability to capture different factors varies. Therefore, the author should use different metrics for a thorough analysis, especially when Table 2 shows that the EM score may not support the conclusions of the paper. We also noticed in Appendix A that the author provided average results for BLEU-4 Score, which revealed smaller differences in text recoverability among various models. The author should further provide more detailed experimental results and analysis using different metrics.\n\n\n3. In the Controlled-Imcaps experiment, there was no control over the differences in vision encoders among different VL-Models. This means that we cannot rule out the possibility that performance differences are caused by varying capabilities of vision encoders. To investigate this further, the author should consider more ablative experiment. Whether the varying capabilities of vision encoders could be a contributing factor to why text recoverability is not a sufficient condition for VL model performance is also a question. Additionally, this factor may also explain the weak performance of models in winoground.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses the problem of text encoders being a performance bottleneck in contrastive vision-language model.\n\n\nThe paper first proposes a text-only recovery method to reconstruct captions from single-vector text representations generated by\nvarious VL models, and use it to evaluate the information loss caused by the text encoder on the CompPrompts dataset.\nAdditionally, it assesses how this information loss impacts the performance of VL models through the Controlled-ImCaps\nbenchmark.\n\n\nThe main contributions of this paper include:\n1. A detailed analysis of the limitations of current text encoders in VL models.\n2. The introduction of two new evaluation benchmarks, namely CompPrompts and Controlled-ImCaps, to respectively evaluate\ninformation loss in text encoders and its effect on VL model performance.\n3. The proposal of a text-only recovery method for reconstructing captions from single-vector text representations produced by\nmultiple VL models.\n4. The identification that recoverability of textual content is necessary but not sufficient for modeling compositional factors in\ncontrastive VL models.\n\n\nFurthermore, the paper provides an extensive review of related work and discusses potential future research directions in this\nfield.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 2, along with other tables, is difficult to read. The author should consider restructuring the table. For example, they could highlight the data that supports their conclusions instead of making reviewers analyze a large number of tables. Additionally, it would be helpful to include average values in the table rather than only referencing them in the text without displaying them in the table."}}, {"review_id": "owuCFjAyjI", "reviewer": "Reviewer_uEYG", "summary": "", "questions": "I am curious about the rationale behind utilizing the T5 decoder in the Text-only Recovery experiment. Wouldn't it be more appropriate to consider alternative decoder-only models, such as GPT-2, and evaluate their potential effectiveness in this context?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank all the reviewers for their detailed and constructive reviews. Thank you to Reviewer uEYG for pointing out our thorough/meticulous examination, innovative datasets, and extensive experiments.\n\n> **The existence of a bottleneck is somewhat unsurprising.**\n* While it might not be surprising that text encoders bottleneck CLIP-style models to some extent, we are the first, to our knowledge, to quantify the (surprisingly significant!) information loss for various models. We are hopeful our work can inform future model design.\n\n> **Why T5 and not GPT-2:**\n* Decoder-only T5 and GPT-2 share very similar architectures: GPT-2 would also be a reasonable choice for our experiments. However, the high performance of proof-of-concepT5 shows that T5 is sufficient for the claims we make, i.e., that T5 is able to accurately recover captions if they are represented in the input features.\n\nThank you again for your thoughtful comments. Please let us know if you have any further questions or comments, we would be happy to follow up!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "While the discovery that the text encoder in CLIP possesses limited language information may not be groundbreaking, this paper distinguishes itself through meticulous examination and the introduction of innovative datasets for their investigation. Moreover, the authors present an extensive array of experiment results to support their findings.", "reasons_to_reject": "The revelation of the text encoder's limited capacity to retain language information in CLIP may not be considered groundbreaking.", "questions_to_authors": "I am curious about the rationale behind utilizing the T5 decoder in the Text-only Recovery experiment. Wouldn't it be more appropriate to consider alternative decoder-only models, such as GPT-2, and evaluate their potential effectiveness in this context?", "paper_topic_and_main_contributions": "This paper presents a thorough examination of the information loss within the language module of CLIP, aiming to understand its impact on language processing. To facilitate their investigation, the researchers introduce two novel datasets, namely CompPrompts and ControlledImCaps. The compelling analysis leads them to a finding: the text encoders in CLIP act as performance bottlenecks, potentially affecting the model's overall language capabilities.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LCDrAMSYCt", "reviewer": "Reviewer_n92c", "summary": "", "questions": "- Did you explore other CLIP-variants, such as DeCLIP ( + language supervision),  ALBEF, and BLIP, on these probing tasks?  which may tell followers that some directions are useful while some are not.\n- Regarding the objects,  the frequency in the corpus may introduce a significant effect on the performance. Maybe categorize the prompt according to the frequency and check the performance gap as well?\n\nMore discussions and results are always great but I not demanding these  :)  .", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank all the reviewers for their detailed and constructive reviews. Thank you to Reviewer n92c for pointing out our interesting method, our reasonable and insightful findings potentially inspiring future work, and our useful benchmark.\n\n> **Exploring other CLIP-variants:**\n* For non-standard CLIP variants, we experimented with RoBERTaCLIP to determine the effect of text pre-training on the text encoder’s capabilities, and negCLIP to determine the effect of finetuning with hard negatives targeting word order shuffling. But as the reviewer correctly points out, exploring other variants of CLIP would be an interesting avenue for future work. Given our necessary recoverability condition, one could imagine future CLIP variants reporting text-only recoverability as a metric. \n\n> **Relative frequency of objects in the corpus:**\n* In CompPrompts, this is not an issue as the objects and attributes are selected at random from a set of such words. In expectation, there will be no major frequency variations between them. \n* In ControlledImCaps, this would not be an issue as the two captions in any data point vary only by one word, which is the phenomenon being tested (e.g. before/after for temporal prepositions). \n\nThank you again for your thoughtful comments, and your suggestions to improve the readability of our results. Please let us know if you have any further questions or comments, we would be happy to follow up!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed probing method - text reconstruction - is interesting for probing the V+L performance.\n- These findings are intuitively reasonable and insightful,  potentially inspiring future studies on improving the CLIP-like V+L models.\n- The constructed benchmark released would be very useful.", "reasons_to_reject": "- I do not see significant reasons to reject this paper.", "questions_to_authors": "- Did you explore other CLIP-variants, such as DeCLIP ( + language supervision),  ALBEF, and BLIP, on these probing tasks?  which may tell followers that some directions are useful while some are not.\n- Regarding the objects,  the frequency in the corpus may introduce a significant effect on the performance. Maybe categorize the prompt according to the frequency and check the performance gap as well?\n\nMore discussions and results are always great but I not demanding these  :)  .", "paper_topic_and_main_contributions": "This paper reveals a significant language information loss in text encoders of contrastive language-image pretraining models (CLIP).\n\nA text-decoding probing method is introduced, where the encoded CLIP text vector is fed into a generative LM for decoding the original captions, along with a prompt benchmark designed carefully to examine different aspects of objects, attributes & relations between objects. The evaluation results indicate that the text vectors of CLIP struggle to encapsulate spatial, and temporal relations, while do capturing simple attribute knowledge. \n\nThe paper further probes the performance under a multi-modal setting, similar to the setup of Winoground. It shows that the lower text representation indeed bottlenecks the image-text matching, echos most findings in the text-only setup and suggests that the ability to recover the text input is a necessary but insufficient condition for overall results.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Table 3 can be turned in to a bar chart as there are so many blanks.\n\n- Table 4: According to Line 375-379, the EM score of the first two rows is calculated differently from the rest rows. Maybe highlight this difference with * in the table?"}}]

Author rebuttal:

Meta-review: This paper presents a thorough examination of the information loss within the language module of CLIP, aiming to understand its impact on language processing. To facilitate their investigation, the researchers introduce two novel datasets, namely CompPrompts and ControlledImCaps. The compelling analysis leads them to a finding: the text encoders in CLIP act as performance bottlenecks, potentially affecting the model's overall language capabilities.


This is a strong paper, in its observation, experiments and the benchmarks it provides.
Decision: Accept-Main