SlowGuess/NeurIPS_2023_OCR huggingface dataset:
5.3 Finetuning datasets
The Cola training sets are also built in the same way as described in Sec. 3 using the training splits of GQA [17, 28], CLEVR [23] PACO [43], and Visual Genome [28]. For GQA, the training split contains 1381 objects and 601 attributes that compose 27078 queries on 74K images. For CLEVR, we have 3 shapes composed with 8 colors and 2 sizes on 70K images. Finally, for PACO, we have 75 objects and 55 attributes that compose 18696 queries on 37883 images. The Cola multi-object
2we also computed the F1 score and see trends remain the same; more details in the supplementary, Sec. 2, page 4.
3we report "seen" only on 32 classes to avoid disbalance with unseen. "All" MAP is on all 96 classes. "Seen" trends hold same with 64 classes as well; more details in appendix.
GQA	CLEVR	PACO
All	Unseen	Seen	All	Unseen	Seen	All	Unseen	Seen
a. CLIP	36.53	39.06	34.24	15.38	15.01	15.32	12.21	8.64	15.79
b. +Linear	40.44	42.87	38.24	47.96	29.43	46.75	14.22	6.75	21.68
c. +prompt-tune	37.40	40.69	34.43	29.61	23.17	28.05	12.76	5.92	19.61
d. +FT all	38.81	40.85	36.95	52.32	19.00	47.95	14.58	6.49	22.66
e. +FT late	42.19	44.61	40.01	64.06	27.53	67.48	15.66	8.74	22.58
f. +MM-Pred (us)	45.99	48.6	43.64	75.80	51.98	80.72	15.49	8.00	22.94
g. +MM-Adapter (us)	46.83	48.86	44.99	88.21	89.52	77.00	18.56	11.47	25.66
h. FLAVA	39.65	42.18	37.37	15.41	13.27	15.93	12.53	7.29	17.76
i. +Linear	37.07	39.96	34.46	19.30	17.53	18.52	11.65	7.90	15.39
j. +FT-late	39.58	42.26	37.16	77.95	72.72	66.42	12.82	5.79	19.84
k. +MM-Pred (us)	47.12	51.53	43.13	90.43	85.78	86.07	18.57	10.71	26.44
l. +MM-Adapter (us)	48.54	52.55	44.91	91.10	86.64	87.39	19.36	11.16	27.55
Table 1: mAP results on the Cola single object compounds setting. Our multimodal adaptation (MM-Adapter) performs better than common tuning methods. Further, multimodal attention to adapt the image representation (MM-Adapter) generalizes better than using it simply as a prediction head (MM-Pred). MM-Adapter on CLIP is better than tuning the pre-trained multimodal attention layers of the bigger FLAVA (+FT late). MM-Adapter further improves FLAVA.
compounds training split has 551,980 multi-object compounds on 71,174 images. Only the test split is cleaned using human annotations. For datasets built on GQA [17] and Visual Genome [28], we leverage the annotations to explore the effects of different kinds of data queries. We use the region descriptions (denoted as RegionCap in the tables) in Visual Genome to test if linguistic diversity helps over templated captions (Cola single objects and multi-object). We also compare to hard negatives from the Cola multi-object pairs. We finally have a combined setting where we use all data.
Recall that we evaluate on two settings for  $Cola$  - the single-object compounds setting and the multi-object compounds setting as defined in Sec. 3. We discuss the quantitative results below and some qualitative results are shown in Fig. 3.
6.1 Cola Single-object retrieval
Multimodal adaptation is more effective than other tuning methods: In Table 1, compared to prompt-tuning (row c), fine-tuning all of CLIP (row d), or fine-tuning a few of the later layers (row e), tuning a multimodal attention layer of same/lesser parameters has higher mAP (row f and g). Linear probing (row b), although cheaper, significantly underperforms. This is not surprising since multimodal attention over the image regions and text tokens offers more flexibility to the model to learn to bind the right attributes to the right object region. Tuning the whole model is also worse than tuning the later unimodal layers (row d vs e). This might be because fine-tuning the whole model requires larger batch sizes with significantly more data. In Fig 3, we show the comparison of tuning unimodal layers vs our multimodal adaptation (since tuning the unimodal layers is closest in performance). Qualitative examples from each of the other adaptation methods are displayed in Figs. 5-11 in the supplementary.
Using pre-trained multimodal attention layers is not enough - training them on attribute-object compositions is key: In Table 1, we see that MM-Adapter (row g) on CLIP ViT B-32 (151 M params) outperforms tuning the last two multimodal layers of FLAVA B-16 [51] (241M params) model (row j) or tuning a linear probe (row i). Surprisingly, tuning the last two FLAVA multimodal layers (row j) is worse than replacing them and training using MM-Pred and MM-Adapter layers (rows k and l). This suggests that training multimodal layers during adaptation (as opposed to pre-training) is key.
MM-Adapter is better than using multimodal attention as a prediction head: Recall that one of the ablations of our approach is aligning the output of the multimodal encoder to the frozen
Multi-Obj Acc ↑
T2I Cola	I2T CREPE [34]
- Random	25.00	50.00
- Human	83.88	-
o. CLIP	21.42	77.43
a. + Linear	30.47	87.35
b. + Prompt-tune	27.14	80.81
c. + FT all	34.76	82.39
d. + FT late	36.19	87.14
e. + MM-Pred (our)	41.42	77.84
f. + MM-Adapter (our)	40.95	87.02
g. FLAVA	24.76	65.10
h. + Linear	22.38	55.10
i. + FT late	22.38	58.11
j. + MM-Pred (our)	39.04	81.37
k. + MM-Adapter (our)	40.47	74.81
(a)
Data Type	Single-Object Cola GQA
All	Unseen	Seen
a. RegionCap	0.4711	0.4965	0.4481
b. SingleObj	0.4683	0.4886	0.4499
c. + MultiObj	0.4641	0.4795	0.4501
d. + HardNeg	0.4688	0.4843	0.4548
e. Combined	0.4788	0.4983	0.4612
Multi-Object
Cola	CREPE
a. RegionCap	0.3114	0.8833
b. SingleObj	0.2745	0.9023
c. + MultiObj	0.3975	0.8702
d. + HardNeg	0.3483	0.8775
e. Combined	0.3893	0.8798
Table 2: a. Results on our multi-object compounds setting on our Cola task and CREPE. Simpler methods suffice on CREPE, but not on Cola, suggesting that Cola is harder. Red-orange-yellow is in decreasing accuracy order. MM-Adapter and MM-Pred on CLIP perform well on average on both. b. Table showing the effect of the data type used in the contrastive batch training. Having multi-object captions in the data helps Cola performance while maintaining CREPE performance.
text embedding, making it a multimodal image-feature "adapter" (MM-Adapter). This contrasts to using the multimodal module as a prediction head with a linear layer (MM-Pred). As shown in Table 1, MM-Adapter outperforms MM-Pred (row g vs f), especially on unseen classes and on exhaustively annotated datasets like CLEVR [23] and PACO [43]. We posit that aligning to the frozen text representation acts like a regularizer since it was pre-trained on more data.
6.2 Multi-object retrieval
Simpler methods suffice for CREPE, but not for Cola, suggesting that Cola is a harder task: As shown in Table 2a, linear probing (row a) or simple fine-tuning (rows c and d) suffice for CREPE [34]. However, our MM-Adapter improves further on Cola (row f vs row a, b, c, d), while maintaining performance on CREPE. This also suggests that text-to-image matching is harder than image-to-text matching, which is also reflected in Winoground [54].
Baseline CLIP and FLAVA perform below chance: If we evaluate off-the-shelf CLIP [42] and FLAVA [51] on our Cola dataset, we see in Table 2a that it performs below random (row o and g). This is consistent with the findings in Winoground [54].
Training late multimodal layers from scratch help, CLIP+MM-Adapter performs better overall: We improve performance by training multimodal layers from scratch on top of CLIP and FLAVA as shown in Table 2a, row e, f, j, k. Interestingly, as shown in Table 2a, linear probing or tuning the pretrained multimodal layers of FLAVA hurts performance (row g vs row h and i). This could be because tuning adversely perturbs the parameters trained on large-scale data. Finally, CLIP+MM-Adapter (row f) performs comparably well as tuning multimodal layers on FLAVA (row f vs j and k).
6.3 Effect of fine-tuning data
Difference between free-form captions and templated Cola queries is minimal. Hence, Cola templated queries are useful: In Table 2b (row a vs c, d, e), we see that templated queries perform
Figure 3: Qualitative results of multi-object matching (left) and retrieving a single object with multiple attributes (right).
Figure 4: Qualitative results on cases where models struggle with multiple object-attribute compositionality (left). Cases where we see the most improvement and the least on single-object compositional retrieval are shown on the right.
just as well as free-form region descriptions. This shows that  $\mathcal{C}ola$  queries are still useful for attribute-object binding despite not being free-form.
Having multiple objects with multiple attributes in a caption helps: When we combine single object captions with multiple object captions, we see in Table 2b that performance increases especially on the multi-object Cola setting while maintaining performance on the single-object setting (row b vs c). The multi-object and single-object captions are formed on the same number of images. Combining all types of data along with hard negatives (row e) doesn't seem to affect performance much.
6.4 Quality of Cola Benchmark
Our Cola mAPs are harder and are more difficult to improve on: We also computed the performance based on the standard formulation of mAP, where all images are included in the list to
retrieve from. In contrast, Cola mAP only includes hard distractors as defined in Sec. 3. We see that trends remain the same with the standard mAP. However, our Cola mAP is harder to improve on. The standard MAP (supplementary, Table 2) improves by more than  $2\mathrm{x}$  on GQA and  $10\mathrm{x}$  on CLEVR. In contrast, we only improve by  $1.09\mathrm{x}$  on GQA and  $2\mathrm{x}$  on CLEVR with our harder Cola MAP.
Ambiguous colors, spatial relationships, and size are some common themes where models underperform in *Cola* benchmark: We analyze the types of compositional queries that models find difficult on our benchmark. In the qualitative examples shown in Fig. 4, we see that compositions involving ambiguous colors (due to lighting) and spatial relationships are difficult for multiple-object cases. This is likely because spatial relationships are often inconsistently annotated in training data - e.g. "to the left of" can sometimes be from the viewer or image perspective. More examples are in the supplemental (Fig. 4, 16, 17).
Significant improvements on non-salient/occluded objects. Improvements on larger objects are minimal: On attribute-object compositional retrieval for single objects, we see the most improvements using our adaptation method are on non-salient or occluded objects (like a small sign), as shown in Fig. 4 (right). Queries that are commonly represented in training sets (like "clear blue sky" - skies are most commonly blue) have minimal improvements from pre-trained models.
This work finetunes vision-language models to test design choices for compositional attribute-object retrieval. Thus, compared to the original pre-trained models [42], we may lose some other generic capabilities, such as question answering, captioning, etc. For example, our best adaptation scores  $83\%$  zero-shot on CIFAR10 [29], as compared to  $87\%$  for pre-trained CLIP. However, by just using  $5\%$  of CIFAR training data (less than 1 epoch), we reach  $92\%$  accuracy while maintaining performance on our Cola task. We would also like to stress that our goal is not to train a new foundation model but to explore design choices that allow for compositional reasoning. While we focus on attribute-object compositionality, there is still significant room for exploration of other types of compositional structures such as relationships, scene graphs, and counting [14, 12]. A great avenue for future work could be collecting more detailed annotations (about the type of objects and compositions in the queries) on larger sets of images to help pinpoint themes where models are failing. Further testing is also required to see how it fares with sensitive attributes and objects—whether it is predisposed towards attaching incorrect attributes to objects because of racial/political biases in the data [6]. Additionally, it is important to re-evaluate our results in the context of newer vision-language models that are being proposed. Finally, we would like to point out that our finetuning strategy isn't specific to compositions, suggesting its potential applicability for adaptation to other downstream tasks, especially since similar strategies and modules have been used for other computer vision tasks [52, 51, 25].
We present a new task,  $Cola$ , to test the compositional attribute-object binding of vision-language models. This is important for various practical applications like an assistive agent requiring an understanding of fine-grained differences between objects in cluttered workplaces. We explore the architectural choices of adapting large vision-language models that encourage such reasoning. We show that a light-weight multimodal adaptor can improve this capability in a pre-trained vision-language model as a strong baseline for further research. We hope that  $Cola$  serves as a strong benchmark and our adaptation choices as strong baselines for improving compositional vision-language intelligence.
Acknowledgements. We wish to thank Dhruv Mahajan and Jang Hyun (Vincent) Cho for their valuable guidance in the initial phases of the project. We also wish to thank the anonymous reviewers for their thoughtful comments and suggestions. This material is based upon work supported, in part, by DARPA under agreement number HR00112020054 awarded to Kate Saenko and Bryan Plummer at BU. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the supporting agencies.