SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {M I L} ^ {\text {n e g}} = - \frac {1}{B} \sum_ {i} ^ {B} \log \frac {\sum_ {m} S \left(T _ {i , m} , I _ {i}\right)}{\left(\sum_ {m} S \left(T _ {i , m} ^ {\text {n e g}} , I _ {i}\right)\right) + \left(\sum_ {j = 1} ^ {B} \sum_ {m} S \left(T _ {j , m} , I _ {i}\right)\right)} \tag {5}
$$
where  $T_{i,m}^{neg}$  is the result of negative augmentation applied to the captions bag element  $T_{i,m}$ . Finally, the full finetuning loss of our proposed DAC approach can be written as:
$$
\mathcal {L} _ {D A C} = \mathcal {L} _ {\text {c o n t}} + \mathcal {L} _ {\text {n e g}} + \mathcal {L} _ {M I L} ^ {\text {n e g}} \tag {6}
$$
3.7 Implementation details
We used the popular ViT-B/32 OpenAI CLIP [8] original PyTorch implementation as our VL model in all the experiments. For caption quality enhancement (Sec. 3.1), we used the LAVIS implementation of BLIP2 with OPT 6.7B LLM. We used ViT-H SAM model [28] for the Segmentation-based density expansion (Sec. 3.2). Furthermore, we employed the GPT-NEO-2.7B LLM for the LLM-based density expansion (Sec. 3.2). During training, we set the batch size to 128 when training without density expansion (for ablations) and to 32 with density expansions. We used 6 v100 GPUs for 12 hours to train a model. We set the learning rate 5.0e-4, and use the AdamW optimizer over 5 epochs initializing with the CLIP weights. Our code is provided in the Supplementary, and it will be released upon acceptance together with our trained weights.
VL-Checklist	ARO	Avg
Object	Attribute	Relation	VG-R	VG-A	COCO	FLICKR
CLIP[8]	81.58	67.6	63.05	59.98	63.18	47.9	60.2	63.35
BLIP2[10]	84.14	80.12	70.72	41.16	71.25	13.57	13.72	53.27
NegClip[21]	81.35	72.236	63.525	81	71	86	91	78.01
SVLC ([24])	85	71.97	68.95	80.61	73.03	84.73	91.7	79.42
DAC-LLM (Ours)	87.3	77.27	86.41	81.28	73.91	94.47	95.68	85.18
DAC-SAM (Ours)	88.5	75.829	89.75	77.16	70.5	91.22	93.88	83.83
Table 1: Experimental results on the VL-Checklist [20] and ARO [21] datasets to test different compositional reasoning aspects. The two bottom rows present two versions of full approach differing in the type of caption density source, i.e LLM-based and SAM-based. We see that both our methods outperform the baselines by a significant margin in almost all cases. While all methods other than BLIP2 [10] are CLIP-like dual-encoder architectures, BLIP2's heavier encoder-decoder architecture gives it some advantage on the VL-checklist evaluation. Still, we outperform it by a large margin. Our method improves the base CLIP [8] model by over  $20\%$  on average.
Training: We use the Conceptual Captions 3M (CC3M) dataset [75] to finetune CLIP for enhanced compositional reasoning performance using our proposed DAC approach. CC3M is a large set of 3 million image-text pairs automatically crawled from the internet. While the dataset includes captions for the images, we have found that enhancing the quality of the captions before finetuning, as described in Sec. 3.1, greatly improves compositional reasoning. For our full approach, we therefore discard the original captions from the dataset, which (as discussed in Sec. 3.1) uncovers yet another interesting potential of our method - the ability to leverage any unlabeled image collection.
Evaluation: We evaluate our method on two major benchmarks for VL compositional reasoning, VL-Checklist [20] and ARO [21], and also the Elevater [80] for image classification to verify linear transferability of the models (which is of significant importance for usability of the VL models in contemporary downstream applications such as LLaVa [12] and Mini-GPT4 [11]).
VL-Checklist [20] is a benchmark constructed from four datasets, namely Visual Genome [81], SWiG [82], VAW [83], and HAKE [84]. Each image is paired with two captions: a positive and a negative. Positive captions are part of original (paired image-text) datasets. Negative captions were constructed from positive captions by changing one word which changes the meaning of the sentence. The negatives are categorized as changes related to objects, attributes, or relations. Each category is further divided into sub-categories such as size, material, color, etc.
ARO [21] is a new compositional reasoning benchmark that includes positive or negative captions and additionally evaluates sensitivity to word order in the sentence. Word-order negative sentences are created by re-ordering the words, which changes the semantics of the sentence in aspects like attribution, relations, and the general meaning of the word order. ARO is constructed from images and captions from COCO [85], Flickr30K [86], and Visual Genome [81].
Elevator [80] consists of 20 different datasets, including common classification datasets such as CIFAR100 [87], EuroSat [88], and others. We used the Linear-Probing classification protocol described in the Elevater Image Classification Toolkit [80] for evaluating the linear transferability of our resulting DAC models with improved compositional reasoning performance.
4.2 DAC for improving VL model's compositional reasoning performance
Our DAC approach includes caption quality and caption density enhancement steps, followed by a proposed fine-tuning technique that effectively combines several losses, including the proposed MIL loss  $\mathcal{L}_{MIL}^{neg}$  explained in Sec. 3.6 to cope with the inherent noise of caption density expansion. Our main results are summarized in Table 1 and are compared to four strong baselines: (i) CLIP [8] which is also the source model that we improve; (ii) BLIP2 [10] which we also use for caption quality enhancement - to show our resulting models are significantly stronger in compositional reasoning, despite being based on a faster dual-encoder CLIP model (compared to 2-stage encoder-decoder BLIP2); and (iii) two SOTA methods on the compositional reasoning benchmarks, both based on negative text augmentation - NegClip [21] and SVLC [24]. Our DAC-LLM explores the use of LLMs, specifically GPT-NEO-2.7B, to expand the caption and generate additional information that
5-shot	10-shot	20 shot	all-shot
CLIP [8]	66.19%	69.58%	71.90%	78.96%
DAC-LLM	64.92%	69.20%	72.98%	77.44%
DAC-SAM	64.33%	69.41%	72.92%	79.3%
Table 2: Linear probing results on ELEVATER (20 datasets). As desired, we do not observe any significant degradation w.r.t. the baseline in terms of LP accuracy in any setting.
LLM	Neg	MIL	Obj	Attr	Rel
✓	MAX	84.58%	70.99%	60.35%
✓	AVG	85.41%	70.87%	65.57%
✓	MIL	85.74%	71.04%	69.05%
✓	✓	MIL	87.31%	77.22%	86.41%
is plausible to find in a scene described by the source (quality enhanced) caption. Our DAC-SAM leverages semantic over-segmentation [28] and converts image segments into short captions generated from them using [10]. Both these methods increase caption density and add information that was not necessarily available in the quality-enhanced caption. Both employ MIL (Sec. 3.6) combined with negative text augmentation (Sec. 3.3). As can be seen in Tab. 1, both our methods significantly improve on most compositional reasoning metrics with gains of up to  $17\%$  over the current SOTA. In section 5 we analyze the contribution of each individual component.
4.3 Preserving downstream linear transferability with DAC
We used the Linear-Probing classification protocol described in the Elevater Image Classification Toolkit [80] for evaluating the linear transferability to downstream tasks of our resulting DAC models (DAC-LLM and DAC-SAM) with improved compositional reasoning performance. This evaluation is of significant importance, as it demonstrates in a way the usability of the improved VL models in contemporary downstream applications, such as LLaVa [12] and Mini-GPT4 [11] multi-modal conversational agents, which use the linear alignment of VL model features to a strong LLM, e.g., Vicuna [5] in [11, 12]. Table 2 summarizes the Elevater Linear Probing (LP) results. As prescribed by the standard Elevater LP protocol [80], we explore different amounts of training data for LP training, i.e. 5-shot, 10-shot, 20-shot, and all-shot. For all settings, we see that our DAC models are comparable or better than the baseline CLIP [8], only having  $1.5\%$  drop in 5-shot. These experiments show that our DAC VL models compositional reasoning enhancement approach did not degrade the representation power of the original CLIP model.
5 Ablation study
Below we perform an extensive ablation analyzing and highlighting the importance of caption quality and caption density factors towards improving compositional understanding of VL models. We find that each factor separately contributes to compositional reasoning performance, also improving over the baselines, while the best results are obtained by combining the factors. All experiments were conducted for the same number of epochs on the same dataset starting from CLIP pre-trained weights.
Table 3: MIL ablations: we evaluate different strategies for MIL while also examining the effects of negative examples within the MIL loss,  $(\mathcal{L}_{MIL}^{base})$  vs  $\mathcal{L}_{MIL}^{neg}$
Quality	Neg	Density	MIL	VL-Checklist	ARO	Avg
Object	Attribute	Relation	VG-R	VG-A	COCO	FLICKR
CLIP	81.58%	67.60%	63.05%	59.98%	63.18%	47.90%	60.20%	63.36%
A	80.93%	66.28%	55.52%	50.19%	62.48%	21.03%	28.82%	52.18%
LLM	✓	83.54%	68.57%	58.32%	65.49%	62.81%	38.56%	50.84%	61.16%
✓	85%	71.97%	68.95%	80.61%	73.03%	84.73%	91.70%	79.43%
✓	LLM	✓	86.08%	72.15%	70.48%	74.80%	67.80%	84.10%	88.30%	77.67%
B	✓	84.41%	71.03%	63.94%	67.26%	64.76%	29.43%	40.60%	60.20%
✓	LLM	85.00%	70.97%	70.27%	66.44%	65.05%	77.64%	85.30%	74.38%
✓	✓	LLM	87.26%	74.25%	81.97%	72.38%	69.07%	81.22%	80.52%	78.09%
C	✓	✓	LLM	✓	87.30%	77.27%	86.41%	81.28%	73.91%	94.47%	96.12%	85.24%
✓	✓	SAM	✓	88.50%	75.83%	89.75%	77.16%	70.50%	91.22%	93.88%	83.83%
Table 4: Ablation Study: We split the table into three sections: A - examines the effect of LLM expansion without quality improvements. B - analyzes the caption quality and density improvement without the MIL loss, by randomly choosing one example from the bag as the caption. C - Our full method with the two density expansion variants, i.e LLM-based and SAM-based, and MIL.
Figure 3: (a) The empirical probability density function of the CLIP score as a proxy of the caption quality. We see that the improved captions generally have higher scores than the original captions. (b) Examples of images with their original and generated captions. The captioner ([10]) can also make compositional reasoning mistakes (e.g. the zebra image), but our DAC can compensate through the proposed caption density expansion, as also evident from the quantitative evaluation advantage DAC has over [10] in Tab. 1. (c) Analysis of the VL-Checklist results with respect to the percent of captions replaced with our higher quality ones. We see a clear trend favoring quality captions. (d-e) The effects of improving caption density using LLM and SAM methods respectively, by increasing the percent of sampled captions, and thus the density, we see significant improvements.
Caption Quality. Here we validate the benefits of improving caption quality. First, Fig 3a compares the statistics of the CLIP score between captions and images before and after the quality improvement (Sec. 3.1). Clearly, improved captions have better matching statistics. Fig 3b shows some qualitative examples of caption improvement, in all cases original captions hardly describe the image content. Next, Figure 3c shows that gradually increasing the ratio of high-quality captions consistently improves the VL-Checklist metrics. To isolate the effect of quality improvement, we perform this experiment without the caption density enhancements. Finally, block A in Tab. 4 explores the contribution of DAC components without enhancing caption quality. We can see that all of them improve, yet the largest improvements are obtained with quality enhancement (blocks B and C). We have also conducted a caption quality user study. We asked 121 human subjects to review 110 random images from CC3M. For each image, we asked which of 3 choices best fits the image. A and B were captions, either produced by BLIP2 or originating from CC3M (A and B assignment was randomized). Option C was "neither caption fits the image". We found that  $80.7\%$  of responses favored the BLIP2 caption,  $2.2\%$  the original caption, and  $17.1\%$  was "neither". This indicates that BLIP2 captions are indeed better aligned with human perception than the alt-text collected from the Web in CC3M. Intuitively, such better alignment to humans, who are inherently good at compositional reasoning, likely leads to the significant compositional reasoning improvements observed when gradually increasing the percent of the higher quality (BLIP2) captions in the fine-tuning data (Fig. 3c).
Caption Density. Next, we validate the contribution of caption density expansion - increasing the number of captions from different sources (LLM / Segmentation + Caption) produced for each training image (Sec. 3.2). Figure 3d and Figure 3e analyze the improvement in VL-Checklist metrics by expanding the caption density using the DAC-LLM and DAC-SAM methods respectively, while using the proposed MIL techniques to cope with potential noise. Logically, this analysis is done after the quality enhancement and using our full approach. Clearly, increasing caption density improves results. This is also apparent from comparing rows 1 and 2 in block B in Tab. 4.
MIL. In Tab. 3 we explore the different MIL variants for our DAC approach. The MAX - choosing the best match out of MIL-bag, and AVG - averaging text embeddings across MIL-bag, are simple MIL substitutes. We also compare our proposed MIL loss  $\mathcal{L}_{MIL}^{neg}$  (Eq. 5) with its base  $\mathcal{L}_{MIL}^{base}$  (Eq. 4) variant. Clearly the final proposed DAC MIL loss  $\mathcal{L}_{MIL}^{neg}$  has the advantage. This is also evident from comparing blocks B (no MIL) and C (with MIL) numbers in Tab. 4.
DAC-SAM relation extraction quality. We have manually analyzed 100 randomly sampled images from CC3M to check how many: (i) correct-unique relations were contributed by over-segmenting (small object-part segments unavoidable in whole image SAM); and (ii) correct relations generated
for SAM segments in general. Here, correct = visible on the image; unique = not mentioned in any full-object segment for the same image. As some object-parts, such as human hands, are very important, we observed that object-part segments have contributed at least 2 correct-unique relations per 3 images on average. This is potentially a significant boost of relation density for future work applying DAC on large (millions) of images (unlabelled) image collections. Additionally, we measured  $3.4 \pm 1.1$  correct relations per image generated from SAM segments. Since relations (e.g. "in", "on", "holding", "touching", "sitting on", "standing by", etc.) typically involve overlapping or nearly overlapping objects, our segment bounding box crops and the captions resulting from them often do capture those relations and teach them to the VL model, leading to the observed significant performance boost in relation metrics.
Model and data scaling. Experiments analyzing the model and data scaling aspects of DAC can be found in Fig. 4 and Tab. 5.
Object	VL-Checklist Attribute	Relation	VG-R	ARO	Avg	ELEVATER 10-shot
VG-A	COCO	FLICKR
Baseline for (Vit-B/32) CLIP	81.58%	67.6%	63.05%	59.98%	63.18%	47.9%	60.2%	63.35%	69.58%
Baseline for (Vit-L/14) CLIP	83.46%	68.25%	62.86%	61.52%	61.91%	46.1%	55.78%	62.83%	74.9%
CLIP Base (Vit-B/32) + DAC	87.3%	77.27%	86.41%	81.28%	73.91%	94.47%	95.68%	85.18%	69.20%
CLIP Large (Vit-L/14) + DAC	87.92%	79.23%	88.05%	80.71%	74.4%	95.50%	96.6%	86.06%	77.17%
Table 5: Model scaling experiment demonstrating that DAC can also be used to improve the compositional reasoning performance of larger CLIP models. The top two rows present the results for the baseline CLIP models with "Base" (Vit-B/32) and "Large" (Vit-L/14) visual encoders with weights originally released by OpenAI respectively. The bottom two rows demonstrate the improvements attained by applying our DAC approach to improve those models without sacrificing their representation power and vision and language alignment properties (as measured on ELEVATER). It is evident that DAC is able to significantly improves both models with similar strong gains (21.83% for Base and 23.2% for Large) in average compositional reasoning accuracy.
Figure 4: Data scaling experiments evaluating the effect of the amount of (unlabelled) image data used during finetuning with DAC. We can clearly see that the performance (average compositional reasoning accuracy measured on VL-checklist+ARO) increases with adding more image data, and does not seem to be plateauing. We extrapolate from these plots and conclude that it is likely that improvements attained by our DAC approach indeed scale nicely with adding more data.
6 Summary & Conclusions
In conclusion, our work focused on addressing compositional reasoning limitations in Vision and Language Models (VLMs) by considering the factors of caption quality and density. Through our proposed fine-tuning approach applied to the CC3M dataset, we successfully demonstrated a substantial increase in compositional reasoning performance, surpassing both standard VLMs and the strongest baselines. Interestingly, our results indicate that we can effectively tap into visual world knowledge contained in LLMs that have been trained using massive text-only data, and use their expertise to improve VLM compositional reasoning performance. Moreover, our approach can be also applied to any unlabeled image collection without requiring any paired image-text data supervision.
Limitations: Despite the promising results, our work has a limitation. We focused primarily on caption quality and density, and while they contributed to improved compositional reasoning, there may be additional factors that could further enhance VLMs' performance. Exploring other factors and their impact on compositional reasoning would be a worthwhile direction for future improvements. Regarding societal impact, we do not anticipate any specific negative impact, but, as with any Machine Learning method, we recommend exercising caution.
Acknowledgement: This work was partially supported by Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program, Korea University).
[75] P. Sharma, N. Ding, S. Goodman, and R. Soricut, “Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 2556–2565.
[76] S. Santurkar, Y. Dubois, R. Taori, P. Liang, and T. Hashimoto, "Is a caption worth a thousand images? a controlled study for representation learning," arXiv preprint arXiv:2207.07635, 2022.
[77] J. S. Smith, P. Cascante-Bonilla, A. Arbelle, D. Kim, R. Panda, D. Cox, D. Yang, Z. Kira, R. Feris, and L. Karlinsky, "Construct-vl: Data-free continual structured vl concepts learning," arXiv preprint arXiv:2211.09790, 2022.
[78] X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer, "Lit: Zero-shot transfer with locked-image text tuning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18 123-18 133.
[79] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021.
[80] C. Li et al., “Elevator: A benchmark and toolkit for evaluating language-augmented visual models,” Neural Information Processing Systems, 2022.
[81] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al., "Visual genome: Connecting language and vision using crowdsourced dense image annotations," International journal of computer vision, vol. 123, no. 1, pp. 32-73, 2017.
[82] S. Pratt, M. Yatskar, L. Weihs, A. Farhadi, and A. Kembhavi, “Grounded situation recognition,” in European Conference on Computer Vision, Springer, 2020, pp. 314–332.
[83] K. Pham, K. Kafle, Z. Lin, Z. Ding, S. Cohen, Q. Tran, and A. Shrivastava, “Learning to predict visual attributes in the wild,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 13018-13028.
[84] Y.-L. Li, L. Xu, X. Liu, X. Huang, Y. Xu, M. Chen, Z. Ma, S. Wang, H.-S. Fang, and C. Lu, "Hake: Human activity knowledge engine," arXiv preprint arXiv:1904.06539, 2019.
[85] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, "Microsoft coco: Common objects in context," in European conference on computer vision, Springer, 2014, pp. 740-755.
[86] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik, "Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models," in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2641-2649.
[87] A. Krizhevsky, G. Hinton, et al., "Learning multiple layers of features from tiny images," 2009.
[88] P. Helber, B. Bischke, A. Dengel, and D. Borth, "Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification," in IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, IEEE, 2018, pp. 204-207.