SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sivan Doveh $^{1,2}$
Assaf Arbelle
Sivan Harary<sup>1</sup>
Paola Cascante-Bonilla
Amit Alfassy $^{1,5}$
Roei Herzig<sup>1,3</sup>
Donghyun Kim4,6
Raja Giryes<sup>3</sup>
Rogerio Feris
Rameswar Panda
Shimon Ullman
Leonid Karlinsky4
<sup>1</sup>IBM Research, <sup>2</sup>Weizmann Institute of Science, <sup>3</sup>Tel-Aviv University, <sup>4</sup>MIT-IBM Watson AI Lab, <sup>5</sup>Technion, <sup>6</sup>Korea University, <sup>7</sup>Rice University
Vision and Language (VL) models offer an effective method for aligning representation spaces of images and text, leading to numerous applications such as cross-modal retrieval, visual question answering, captioning, and more. However, the aligned image-text spaces learned by all the popular VL models are still suffering from the so-called 'object bias' - their representations behave as 'bags of nouns', mostly ignoring or downsizing the attributes, relations, and states of objects described/appearing in texts/images. Although some great attempts at fixing these 'compositional reasoning' issues were proposed in the recent literature, the problem is still far from being solved. In this paper, we uncover two factors limiting the VL models' compositional reasoning performance. These two factors are properties of the paired VL dataset used for finetuning and pre-training the VL model: (i) the caption quality, or in other words 'image-alignment', of the texts; and (ii) the 'density' of the captions in the sense of mentioning all the details appearing on the image. We propose a fine-tuning approach for automatically treating these factors leveraging a standard VL dataset (CC3M). Applied to CLIP, we demonstrate its significant compositional reasoning performance increase of up to  $\sim 27\%$  over the base model, up to  $\sim 20\%$  over the strongest baseline, and by  $6.7\%$  on average.
Recently, with major discoveries in generative language [1-5] and vision [6, 7] modeling, and Vision & Language (VL) alignment techniques [2, 8-12], we are getting closer than ever to attaining models capable of complete understanding and knowledge of the surrounding world, as well as being capable of acting w.r.t. this knowledge. The 'eyes' of these emerging systems are the VL models that have been shown to exhibit strong performance in a wide range of applications, such as recognition [8, 13], detection [14-16], segmentation [17-19], visual question-answering [9, 10], captioning [9, 10] and many more. However, despite these great advances, the VL models are still known to suffer from significant drawbacks in compositional reasoning - the ability to understand (and properly reflect in the aligned VL representations) the non-object notions appearing on the images and in the text captions, such as object attributes, states, and inter-object relations. This has been publicized and extensively studied in several recent works that also propose metrics for analyzing this phenomenon [20-22]. This drawback of VL models may lead to undesired biases and misconceptions when using them as the 'eyes' of SOTA Large Language Models (LLMs) [23] to build the pinnacle of today's AI systems - multi-modal conversational AI [11, 12]. For example, Figure 1a, illustrates
Figure 1: (a) Current VL models struggle with compositional aspects. In this case, both LLaVA and MiniGPT-4 which are extremely strong combinations of VL models with LLMs, misunderstand some basic spatial relations. (b) We are able to improve VL models' compositional reasoning with our proposed fine-tuning approach including improving caption quality and increasing caption density.
a failure mode of both Mini-GPT4 [11] and LLaVa [12], which successfully align one of the most advanced open-source LLMs (Vicuna [5]) with some of the best open-source VL models (BLIP2 [10] in Mini-GPT4 [11] and CLIP [8] in LLaVa [12]), demonstrating unprecedented capabilities for multi-modal chat, yet in some cases failing to recognize simple inter-object relations appearing on an image (Fig. 1a) despite successfully recognizing all the objects.
In this work, we offer a simple Dense and Aligned Captions (DAC) approach for enhancing the compositional reasoning abilities of VL models, such as the popular CLIP [8] model, without sacrificing their downstream transferability. On top of the previously proposed techniques of negative text augmentation [21, 24], we further study the effects of automatically improving the 'caption quality' and 'caption density' in a paired VL dataset on the compositional reasoning performance of VL models finetuned on the improved dataset (Fig. 1b). Surprisingly, we find that the low quality of the web-crawled captions (alt-texts, not designed to be image-descriptive), that fueled the enormous 400M [8] and 9B [25] paired image-text collections used to train the most widely used VL models [8-10], is one of the main reasons behind degraded compositional reasoning ability in these models. The main source of the low quality seems to be in the loose coupling many of the captions in these datasets have with their respective paired images. Examples include captions mentioning unrelated facts or human emotions, thus contributing a considerable amount of noise to the text-to-image representation spaces alignment learned by the VL model. Another aspect we found important for the degraded compositional reasoning performance seems to be the succinctness and partiality of the web-crawled captions. In the majority of cases, these captions do not describe the paired image in its entirety, but rather mention only a part of the image content, commonly missing object states, relations, and attributes appearing on the image. An intuitive effect of this is that many of the details visible to the model on the image side, are not represented on the text side and hence are being suppressed by the contrastive learning objectives driving the representation spaces alignment, leading to under-representation of those details in the resulting image and text embedding vectors.
Our proposed DAC approach automatically enhances the 'caption quality' and 'caption density' of the text captions in a paired VL dataset by using a combination of an image captioning technique [10], large language models [26, 27], and a foundation segmentation model [28]. In addition, we propose a finetuning approach to best leverage the enhanced VL dataset incorporating Multiple Instance Learning (MIL) losses and a variant of the previous SOTA negative text augmentation compositional reasoning improvement methods [21, 24]. Using our DAC, we are able to significantly boost the compositional reasoning performance of CLIP [8], measured on a variety of VL-checklist [20] and ARO [21] benchmarks, by up to  $\sim 27\%$  in inter-object relations and by over  $6.7\%$  over the highest baseline on average. Moreover, following our proposed VL models' improvements we observe almost no decrease in their linear probing accuracy - arguably the most important metric in light of the recent "linear alignment to LLM" based techniques [11, 12] successfully leveraging these VL models in multi-modal conversational AI systems closely mimicking GPT-4 [2].
To summarize, our contributions are as follows: (i) we propose a Dense and Aligned Captions (DAC) approach for enhancing a VL model's compositional reasoning performance via automatically enhancing the caption quality and density of any off-the-shelf VL dataset and applying the proposed fine-tuning technique utilizing Multiple Instance Learning and negative text augmentation; (ii) applying our proposed approach to the popular CLIP model and finetuning on CC3M enhanced
with our technique, we arrive at DAC-LLM and DAC-SAM, that demonstrate significantly higher compositional reasoning performance on large-scale VL-checklist [20] and ARO [21] benchmarks with up to  $27\%$  absolute improvement in inter-object relations over base CLIP and over  $6.7\%$  average improvement over the strongest baseline; (iii) we perform a detailed analysis and ablation of both our proposed DAC approaches, as well as of the importance of the caption quality and density factors for enhancing the compositional reasoning of VL models.
Vision-language (VL) Models. There have been notable advances in large-scale vision-language (VL) models recently (e.g., CLIP [8] and ALIGN [13]). These models are trained by aligning large-scale image and text pairs obtained from the web with contrastive alignment. Advanced image-text alignments method [9, 10, 13, 29, 30] have been introduced using cross-attention layers with supplementary unsupervised learning objectives including image-text matching, masked, and autoregressive language modeling or learn finer-level alignment and relations between image and text [31-35]. For example, BLIP [9] jointly combines multiple such objectives in a single multi-task training. CyClip [31] adds additional geometrical consistency losses in the image and text alignment. However, recent studies [20-22] reveal that VL models pre-trained on image-text pairs from the web lacks compositional understanding of image and do not understand structured concepts such as object attributes and relations. SLVC [24] and ARO [21] improve the compositional understanding of VL models by augmenting samples with negative texts. In a concurrent work, SGVL [36] leverages a dataset with structural scene graph supervision. Unlike these works, in this paper, we propose to enhance the caption quality (in terms of image alignment) and density (in terms of mentioning all details on the image) in a VL dataset without requiring any additional supervision, as well as propose a negatives-enhanced MIL finetuning method for leveraging the improved VL data to attain compositional reasoning performance significantly improving over the past approaches.
Compositional Reasoning. To achieve a compositional and structured understanding of visual scenes, it is essential to possess the capability to understand visual concepts, including the detection of individual entities and the ability to reason about their interactions and attributes. The ability of compositional reasoning has been successfully utilized in various computer vision applications including vision and language [37-40], scene graph generation [41-45], relational reasoning [46, 47], visual spatial reasoning [48], human-object interactions [49-51], action recognition [52-59], and even image & video generation from graphs [60-62]. Although these approaches have the potential to enhance compositional understanding, they heavily rely on dense and manually curated supervision, such as annotating the precise object locations and their relationships. However, collecting such annotations at scale can be extremely costly, leading to limited-size datasets or the use of synthetic data sources during training. In contrast, our work takes advantage of readily available foundation models, such as vision-language models, large-language models, and segmentation models, to enhance text data derived from noisy web image-text pairs. This augmentation process leads to improved captions, enabling the model to acquire the ability to engage in compositional reasoning regarding the image.
Multiple Instance Learning. Training with noisy labels in the form of 'bags of instances' has been widely researched and applied to many computer vision tasks [44, 63-73]. When dealing with MIL, multiple options have been explored to select a single candidate from the 'bag' such as maximum or random sampling with respect to some predefined measure. Other methods average over the whole bag using different averaging methods such as algebraic mean or MIL-NCE [64]. Recently FETA [74] introduced MIL losses adapted from MIL-NCE [64] to train a VL model with noisy labels generated from automatically from documents. In this work, we extend the prior work by incorporating negative examples generated from the MIL bag into the MIL loss. We demonstrate that this approach plays an important role in compositional reasoning improvement, particularly in VL, and in this capacity is superior to random, maximum, average, or NCE methods.
In this section, we detail our DAC approach (Fig. 2) to enhancing the compositional reasoning performance of VL models (e.g., CLIP [8]). Our approach admits an arbitrary paired VL dataset (e.g., CC3M [75]) and applies automatic caption quality and caption density enhancements (Sec. 3.1 and Sec. 3.2). This enhanced dataset is then used to finetune the VL model by employing negative text augmentation (Sec. 3.3), Multiple Instance Learning (MIL) losses (Sec. 3.6) to cope with additional
noise introduced by the caption density enhancement step, additional losses (Sec. 3.5), and parameter efficient fine-tuning to reduce forgetting (Sec. 3.4).
Figure 2: Detailed flow of our method: (i) The image is first captioned using the strong Captioner to create a high-quality caption. (ii) Then two methods for caption density enhancement (expansion) are applied: the "LLM-Expander" and the "SAM-Expander". Both create additional captions which relate to the image. (iii) the negative generator is applied to all captions, including the expanded ones. (iv) The image and all the captions are encoded using their respective encoders. (v) Finally, three losses are applied to the matching between the visual and textual embeddings.
3.1 Improving caption quality
One of the drawbacks of the Internet-collected VL datasets [25, 75] is the loose alignment the captions commonly have to their paired images. This likely stems from the heuristics commonly employed for this pairing, such as using co-location on the sources web-pages, alt-text, etc. Additional factors that impact the caption-to-image alignment, or in other words 'caption quality', are the ambiguous nature of the captions since different web-page authors refer to the images in different detail and include aspects like the human authors' intent, creativity, and many others. As noted in previous work [76], caption quality may negatively affect VL model transfer to downstream tasks. In this work, we analyze, for the first time, the direct impact that caption quality has on the compositional reasoning performance of VL models.
We employ a straightforward approach to enhancing the quality of the captions. To increase their alignment with their paired images, we use image captioning to generate text directly conditioned on the image. We use the BLIP2 [10] captioner, which in turn translates the image into 32 encoded tokens that are transferred to a language model (OPT [27]) which then outputs a sentence describing the content of the image. Sec. 5 includes an analysis of the effect of this proposed caption-to-image alignment. In particular, we show that: (i) the CLIP matching score statistics are significantly improved by replacing the original captions with BLIP captions (Fig. 3a), and qualitatively they are a better fit in many cases (Fig. 3b); (ii) even enhancing the caption quality alone has a positive impact on the compositional reasoning performance of the model; (iii) DAC full approach significantly outperforms BLIP2 [10] in terms of compositional reasoning (by over  $6.4\%$  on VL-checklist [20] and over  $50\%$  on ARO, Tab. 1), indicating caption quality enhancement alone is insufficient. Interestingly, a bonus property of the proposed caption enhancement method is that by using it one could potentially leverage any unlabeled image collection, without requiring any paired text data, for improving the compositional reasoning performance of a VL model. We believe that our observation, therefore, uncovers an interesting hidden potential of unlabeled image collections for VL training and may have a significant potential impact on the VL compositional reasoning performance scaling in future work, underlining the significance of our findings.
3.2 Improving caption density
Another important drawback of Internet-collected VL datasets is their 'partiality' or 'lack of density'. Intuitively, most of these captions are not intended by web page authors to be descriptive of the image but rather serve different purposes. Consequently, in most if not all of the cases, these captions do not tell the complete story of the image in terms of not mentioning every detail and not imparting
additional knowledge on related concepts from world knowledge. Below, we propose and explore two orthogonal methods for enhancing the density of the captions.
Large Language Model (LLM) knowledge expansion. One solution, is using either the original caption, the higher quality caption produced by an image captioner, or both, and use them as a prompt to a Large Language Model (LLM) effectively tapping on its world knowledge acquired from training on trillions of tokens. In our experiments we used the GPT-Neo-2.7B open LLM, instructing it to generate an expansion from a given caption via zero-shot (in context) prompts. The full prompts are long and provided in the Supplementary, their essence though can be summarized here as: "what should I expect to see in an image of {caption}?". We parsed the resulting text into a collection of individual sentences and employed them in our training via MIL losses (Sec. 3.6) in order to account for the caption noise expected with this kind of LLM expansion (Fig. 2). Indeed, the source caption is the only information provided to the LLM, and some facts are likely to be hallucinated and unrelated to the corresponding image. We provide an analysis of the effects of this way of increasing caption density through LLMs as well as of various related design choices in Sec. 5. Interestingly, LLM knowledge expansion alone has a mixed effect on the compositional reasoning performance, however, it is very effective in combination with other proposed techniques.
Leveraging semantic over-segmentation. An additional, more image-grounded way, to enhance caption density is to decompose the image into a set of segments with consistent language semantics and caption each segment individually or in connected segment groups. Consequently, we employ SAM [28] to produce a decomposition of an image into a collection of segments (seeded by a grid of points). We refer to this as 'semantic over-segmentation' as SAM applied in this way produces segments generally respecting semantic boundaries between objects, but commonly over-segments, highlighting object parts and other sub-object regions. We caption each segment using BLIP2 [10] resulting in a collection of captions for each image. As it is an over-segmentation, some of the resulting segment captions tend to be noisy (Fig. 2). Similar to the LLM-generated captions, we also employ these in our training through MIL (Sec. 3.6). The analysis of the segmentation-based expansion and effects of this way of caption density enhancement on the compositional reasoning performance, in general, is provided in Sec. 5.
3.3 Negative text augmentation
While in the VL datasets, e.g. CC3M [75] and LAION [25], each image-text pair constitutes a single sample, it was shown in recent works that it is possible to significantly enhance the effectiveness of these samples for teaching compositional reasoning to VL models via simple negative text augmentation [21, 24]. Using this approach, every image receives a set of captions, one caption correctly matching the image (the source caption), and one or several captions manipulated to be negative text samples via changing words within the caption, thus highlighting the importance of different compositional reasoning aspects (attributes, relations, actions, states) to the finetuned model. For our DAC approach we also employ this negative text augmentation strategy combining it with the DAC captions enhancement principles outlined in Sec. 3.1 and Sec. 3.2. Specifically, we explore applying this augmentation both to the BLIP2 captions (Sec. 3.5), as well as to the LLM-expanded and Segmentation-expanded captions by their combination with the MIL training losses (Sec. 3.6). Notably, our DAC approach attains significant improvements (e.g. over  $12\%$  on VL-checklist [20]) in compositional reasoning performance over the recent methods that have proposed the negative text augmentation [21, 24], underlining the importance of the DAC factors outlined above. The effects of negative text augmentation are further explored in Sec. 5.
3.4 Avoiding forgetting with Parameter Efficient Fine-Tuning
In this work we explore fine-tuning a large-scale pre-trained VL model (e.g. CLIP [8], pre-trained on 400M image-text pairs) on a relatively smaller scale enhanced VL dataset (e.g. CC3M [75] with 3M image-text pairs). Intuitively, naively finetuning on the smaller dataset may incur 'forgetting' the powerful linear transfer capabilities of the base model (CLIP) that are of major significance for downstream applications, such as LLaVa [12] and Mini-GPT4 [11] enabling open, GPT4-like, multi-modal conversational AI. As this is undesirable, similar to other works [24, 77, 78], we employ parameter efficient finetuning, or more specifically LoRA [79], incorporated into all parameters of both vision and text encoders of the model to finetune the otherwise frozen model. As can be seen in Sec. 4.3, this strategy almost completely eliminates linear transfer performance loss only observing  $\sim 1.5\%$  drop in the 5-shot setting.
3.5 Training Losses
A dual-encoder VLM (e.g., CLIP [8]) admitting text-image pair  $(T, I)$  is comprised of: (i) an image encoder  $e_I = \mathcal{E}_I(I)$ ; (ii) a text encoder  $e_T = \mathcal{E}_T(T)$ . The text-image similarity score is computed as:
$$
\mathcal {S} (T, I) = \exp \left(\frac {\tau e _ {T} ^ {T} e _ {I}}{\| e _ {T} \| ^ {2} \| e _ {I} \| ^ {2}}\right), \tag {1}
$$
where  $\tau$  is a learned temperature parameter.
Contrastive Loss. As most contemporary VL models, we employ the contrastive CLIP-loss [8] as one of our losses for each batch  $\mathcal{B} = \{(T_i,I_i)\}$  where texts  $T_{i}$  of the text-image pairs are quality-enhanced as explained in Sec. 3.1.
$$
\mathcal {L} _ {\text {c o n t}} = \sum_ {i} \log \left(\frac {S \left(T _ {i} , I _ {i}\right)}{\sum_ {j} S \left(T _ {i} , I _ {j}\right)}\right) + \log \left(\frac {S \left(T _ {i} , I _ {i}\right)}{\sum_ {k} S \left(T _ {k} , I _ {i}\right)}\right). \tag {2}
$$
Negatives Loss. In addition to using negative text augmentation as part of our proposed MIL losses (Sec. 3.6), we employ the following 'negatives loss' applied to the quality-enhanced captions  $T_{i}$ :
$$
\mathcal {L} _ {n e g} = \sum_ {i} - l o g \left(\frac {\mathcal {S} \left(T _ {i} , I _ {i}\right)}{\mathcal {S} \left(T _ {i} , I _ {i}\right) + \mathcal {S} \left(T _ {i} ^ {n e g} , I _ {i}\right)}\right). \tag {3}
$$
where  $T_{i}^{neg}$  is the negative text generated from  $T_{i}$  according to the technique of [24].
3.6 Multiple Instance Learning (MIL)
For both kinds of caption density expansions, LLM-based and Segmentation-based, proposed in Sec. 3.2, we arrive with a bag of caption texts tentatively corresponding to the paired image. One approach to using those bags is random sampling or averaging in combination with the other losses. However, due to the inherent noise (e.g.,  $\sim 46\%$  for LLM-expansion measured on 100 random images) in both the density expansion methods (explained in Sec. 3.2), as can be seen from the ablations in Sec. 5, neither produces satisfactory compositional reasoning improvements. To better cope with this noise, we propose the MIL approach discussed next.
The basic MIL setting considers a set of  $M$  captions  $\{T_{i,m}\}_{m = 0}^{M}$  such that at least one of these captions is a positive match to the paired image  $I_{i}$ . In our DAC approach, we extend the MIL-NCE [64] loss that was originally proposed for visual representation learning from uncurated videos. We adopt the MIL-NCE for our needs as follows:
$$
\mathcal {L} _ {M I L} ^ {\text {b a s e}} = - \frac {1}{B} \sum_ {i} ^ {B} \log \frac {\sum_ {m} S \left(T _ {i , m} , I _ {i}\right)}{\sum_ {j = 1} ^ {B} \sum_ {m} S \left(T _ {j , m} , I _ {i}\right)} \tag {4}
$$
For any source of MIL bag (LLM / Segmentation), to combine negative augmentation, explained in Section 3.3, with MIL we modify the base-MIL loss  $\mathcal{L}_{MIL}^{base}$  in Eq. 4 to incorporate the non-matching captions in the denominator of the equation: