Title: Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: computer vision

deep learning

vision and language models
Abstract: Vision and Language (VL) models offer an effective method for aligning representation spaces of images and text allowing for numerous applications such as cross-modal retrieval, visual and multi-hop question answering, captioning, and many more. However, the aligned image-text spaces learned by all the popular VL models are still suffering from the so-called 'object bias' - their representations behave as 'bags of nouns' mostly ignoring or downsizing the attributes, relations, and states of objects described/appearing in texts/images. Although some great attempts at fixing these `compositional reasoning' issues were proposed in the recent literature, the problem is still far from being solved. In this paper, we uncover two factors limiting the VL models' compositional reasoning performance. These two factors are properties of the paired VL dataset used for finetuning (or pre-training) the VL model: (i) the caption quality, or in other words 'image-alignment', of the texts; and (ii) the 'density' of the captions in the sense of mentioning all the details appearing on the image. We propose a fine-tuning approach for automatically treating these factors on a standard collection of paired VL data (CC3M). Applied to CLIP, we demonstrate its significant compositional reasoning performance increase of up to $\sim27$\% over the base model, up to $\sim20$\% over the strongest baseline, and by $6.7$\% on average. Our code is provided in the Supplementary and would be released upon acceptance.

Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models

Abstract

Vision and Language (VL) models offer an effective method for aligning representation spaces of images and text, leading to numerous applications such as cross-modal retrieval, visual question answering, captioning, and more. However, the aligned image-text spaces learned by all the popular VL models are still suffering from the so-called 'object bias' - their representations behave as 'bags of nouns', mostly ignoring or downsizing the attributes, relations, and states of objects described/appearing in texts/images. Although some great attempts at fixing these 'compositional reasoning' issues were proposed in the recent literature, the problem is still far from being solved. In this paper, we uncover two factors limiting the VL models' compositional reasoning performance. These two factors are properties of the paired VL dataset used for finetuning and pre-training the VL model: (i) the caption quality, or in other words 'image-alignment', of the texts; and (ii) the 'density' of the captions in the sense of mentioning all the details appearing on the image. We propose a fine-tuning approach for automatically treating these factors leveraging a standard VL dataset (CC3M). Applied to CLIP, we demonstrate its significant compositional reasoning performance increase of up to ∼ 27% over the base model, up to ∼ 20% over the strongest baseline, and by 6 . 7% on average.

1 Introduction

Recently, with major discoveries in generative language [1-5] and vision [6, 7] modeling, and Vision &Language (VL) alignment techniques [2, 8-12], we are getting closer than ever to attaining models capable of complete understanding and knowledge of the surrounding world, as well as being capable of acting w.r.t. this knowledge. The 'eyes' of these emerging systems are the VL models that have been shown to exhibit strong performance in a wide range of applications, such as recognition [8, 13], detection [14-16], segmentation [17-19], visual question-answering [9, 10], captioning [9, 10] and many more. However, despite these great advances, the VL models are still known to suffer from significant drawbacks in compositional reasoning - the ability to understand (and properly reflect in the aligned VL representations) the non-object notions appearing on the images and in the text captions, such as object attributes, states, and inter-object relations. This has been publicized and extensively studied in several recent works that also propose metrics for analyzing this phenomenon [20-22]. This drawback of VL models may lead to undesired biases and misconceptions when using them as the 'eyes' of SOTA Large Language Models (LLMs) [23] to build the pinnacle of today's AI systems - multi-modal conversational AI [11, 12]. For example, Figure 1a, illustrates

✗

A boy is sitting on a chair

VL

Models

A

cat and a boy in a

living room.

Caption

Density

1.

Sunlight bathes the room,

illuminating the cat's fur as it

lounges on the chair.

2.

A book rests on the living room

table.

3.

A nearby bookshelf holds an

array of books, while the chair

offers comfort for reading.

Prompt

: what can I

see in a scene of

{caption}?

e.g LLM

Dense and Aligned Captions

✓

A cat is sitting on a chair

Original Caption:

'A moment of joy together.'

(b) Compositional Reasoning

Who is sitting on the chair?

LLaVA

A small child,

or a toddler,

is sitting on the chair

The image shows a young child sitting on

a chair while playing with a tablet. There

is a gray and white cat sitting on the floor

in front of the child. The child is not

interacting with anything.

(a) Problem Statement

Caption

Quality

MiniGPT-4

a failure mode of both Mini-GPT4 [11] and LLaVa [12], which successfully align one of the most advanced open-source LLMs (Vicuna [5]) with some of the best open-source VL models (BLIP2 [10] in Mini-GPT4 [11] and CLIP [8] in LLaVa [12]), demonstrating unprecedented capabilities for multi-modal chat, yet in some cases failing to recognize simple inter-object relations appearing on an image (Fig. 1a) despite successfully recognizing all the objects.

In this work, we offer a simple Dense and Aligned Captions (DAC) approach for enhancing the compositional reasoning abilities of VL models, such as the popular CLIP [8] model, without sacrificing their downstream transferability. On top of the previously proposed techniques of negative text augmentation [21, 24], we further study the effects of automatically improving the ' caption quality ' and ' caption density ' in a paired VL dataset on the compositional reasoning performance of VL models finetuned on the improved dataset (Fig. 1b). Surprisingly, we find that the low quality of the web-crawled captions (alt-texts, not designed to be image-descriptive), that fueled the enormous 400M [8] and 9B [25] paired image-text collections used to train the most widely used VL models [8-10], is one of the main reasons behind degraded compositional reasoning ability in these models. The main source of the low quality seems to be in the loose coupling many of the captions in these datasets have with their respective paired images. Examples include captions mentioning unrelated facts or human emotions, thus contributing a considerable amount of noise to the text-to-image representation spaces alignment learned by the VL model. Another aspect we found important for the degraded compositional reasoning performance seems to be the succinctness and partiality of the web-crawled captions. In the majority of cases, these captions do not describe the paired image in its entirety, but rather mention only a part of the image content, commonly missing object states, relations, and attributes appearing on the image. An intuitive effect of this is that many of the details visible to the model on the image side, are not represented on the text side and hence are being suppressed by the contrastive learning objectives driving the representation spaces alignment, leading to under-representation of those details in the resulting image and text embedding vectors.

Our proposed DAC approach automatically enhances the ' caption quality ' and ' caption density ' of the text captions in a paired VL dataset by using a combination of an image captioning technique [10], large language models [26, 27], and a foundation segmentation model [28]. In addition, we propose a finetuning approach to best leverage the enhanced VL dataset incorporating Multiple Instance Learning (MIL) losses and a variant of the previous SOTA negative text augmentation compositional reasoning improvement methods [21, 24]. Using our DAC, we are able to significantly boost the compositional reasoning performance of CLIP [8], measured on a variety of VL-checklist [20] and ARO [21] benchmarks, by up to ∼ 27% in inter-object relations and by over 6 . 7% over the highest baseline on average. Moreover, following our proposed VL models' improvements we observe almost no decrease in their linear probing accuracy - arguably the most important metric in light of the recent 'linear alignment to LLM" based techniques [11, 12] successfully leveraging these VL models in multi-modal conversational AI systems closely mimicking GPT-4 [2].

To summarize, our contributions are as follows: (i) we propose a Dense and Aligned Captions (DAC) approach for enhancing a VL model's compositional reasoning performance via automatically enhancing the caption quality and density of any off-the-shelf VL dataset and applying the proposed fine-tuning technique utilizing Multiple Instance Learning and negative text augmentation; (ii) applying our proposed approach to the popular CLIP model and finetuning on CC3M enhanced

with our technique, we arrive at DAC-LLM and DAC-SAM, that demonstrate significantly higher compositional reasoning performance on large-scale VL-checklist [20] and ARO [21] benchmarks with up to 27% absolute improvement in inter-object relations over base CLIP and over 6 . 7% average improvement over the strongest baseline; (iii) we perform a detailed analysis and ablation of both our proposed DAC approaches, as well as of the importance of the caption quality and density factors for enhancing the compositional reasoning of VL models.

2 Related Work

Vision-language (VL) Models. There have been notable advances in large-scale vision-language (VL) models recently ( e . g ., CLIP [8] and ALIGN [13]). These models are trained by aligning large-scale image and text pairs obtained from the web with contrastive alignment. Advanced image-text alignments method [9, 10, 13, 29, 30] have been introduced using cross-attention layers with supplementary unsupervised learning objectives including image-text matching, masked, and autoregressive language modeling or learn finer-level alignment and relations between image and text [31-35]. For example, BLIP [9] jointly combines multiple such objectives in a single multi-task training. CyClip [31] adds additional geometrical consistency losses in the image and text alignment. However, recent studies [20-22] reveal that VL models pre-trained on image-text pairs from the web lacks compositional understanding of image and do not understand structured concepts such as object attributes and relations. SLVC [24] and ARO [21] improve the compositional understanding of VL models by augmenting samples with negative texts. In a concurrent work, SGVL [36] leverages a dataset with structural scene graph supervision. Unlike these works, in this paper, we propose to enhance the caption quality (in terms of image alignment) and density (in terms of mentioning all details on the image) in a VL dataset without requiring any additional supervision, as well as propose a negatives-enhanced MIL finetuning method for leveraging the improved VL data to attain compositional reasoning performance significantly improving over the past approaches.

Compositional Reasoning. To achieve a compositional and structured understanding of visual scenes, it is essential to possess the capability to understand visual concepts, including the detection of individual entities and the ability to reason about their interactions and attributes. The ability of compositional reasoning has been successfully utilized in various computer vision applications including vision and language [37-40], scene graph generation [41-45], relational reasoning [46, 47], visual spatial reasoning [48], human-object interactions [49-51], action recognition [52-59], and even image & video generation from graphs [60-62]. Although these approaches have the potential to enhance compositional understanding, they heavily rely on dense and manually curated supervision, such as annotating the precise object locations and their relationships. However, collecting such annotations at scale can be extremely costly, leading to limited-size datasets or the use of synthetic data sources during training. In contrast, our work takes advantage of readily available foundation models, such as vision-language models, large-language models, and segmentation models, to enhance text data derived from noisy web image-text pairs. This augmentation process leads to improved captions, enabling the model to acquire the ability to engage in compositional reasoning regarding the image.

Multiple Instance Learning. Training with noisy labels in the form of 'bags of instances' has been widely researched and applied to many computer vision tasks [44, 63-73]. When dealing with MIL, multiple options have been explored to select a single candidate from the 'bag' such as maximum or random sampling with respect to some predefined measure. Other methods average over the whole bag using different averaging methods such as algebraic mean or MIL-NCE [64]. Recently FETA [74] introduced MIL losses adapted from MIL-NCE [64] to train a VL model with noisy labels generated from automatically from documents. In this work, we extend the prior work by incorporating negative examples generated from the MIL bag into the MIL loss. We demonstrate that this approach plays an important role in compositional reasoning improvement, particularly in VL, and in this capacity is superior to random, maximum, average, or NCE methods.

3 Method

In this section, we detail our DAC approach (Fig. 2) to enhancing the compositional reasoning performance of VL models (e.g., CLIP [8]). Our approach admits an arbitrary paired VL dataset (e.g., CC3M [75]) and applies automatic caption quality and caption density enhancements (Sec. 3.1 and Sec. 3.2). This enhanced dataset is then used to finetune the VL model by employing negative text augmentation (Sec. 3.3), Multiple Instance Learning (MIL) losses (Sec. 3.6) to cope with additional

noise introduced by the caption density enhancement step, additional losses (Sec. 3.5), and parameter efficient fine-tuning to reduce forgetting (Sec. 3.4).

ii

iv

SAM

Expander

The elephantis standingon awhitebackground.

Theimage shows the sliding doorin theroom.

The image shows a close up of a toy tiger.

LoRA

LoRA

LMIL

rneg

Miniature zoo of toy animals on a mat.

Plastic farm animals like cows,horses, and chickens.

LLM

Playful accessories like trees and rocks.

Expander

WhatcanIsee

TextEncoder

ImageEncoder

in a scene of

(caption)?

The elephant isrunning on awhitebackground.

Theimage shows the sliding door outside theroom.

The image shows a close up of a real tiger.

Lneg

Captioner

ili

SVLC

Miniature zoo of toy animals inside a mat.

Negative

Stone farm animals likecows,horses,and chickens.

Generation

Playful accessories on trees and rocks.

A child flyingwith toy animals.

Lcont

A child playing with toy animals.

3.1 Improving caption quality

One of the drawbacks of the Internet-collected VL datasets [25, 75] is the loose alignment the captions commonly have to their paired images. This likely stems from the heuristics commonly employed for this pairing, such as using co-location on the sources web-pages, alt-text, etc. Additional factors that impact the caption-to-image alignment, or in other words 'caption quality', are the ambiguous nature of the captions since different web-page authors refer to the images in different detail and include aspects like the human authors' intent, creativity, and many others. As noted in previous work [76], caption quality may negatively affect VL model transfer to downstream tasks. In this work, we analyze, for the first time, the direct impact that caption quality has on the compositional reasoning performance of VL models.

We employ a straightforward approach to enhancing the quality of the captions. To increase their alignment with their paired images, we use image captioning to generate text directly conditioned on the image. We use the BLIP2 [10] captioner, which in turn translates the image into 32 encoded tokens that are transferred to a language model (OPT [27]) which then outputs a sentence describing the content of the image. Sec. 5 includes an analysis of the effect of this proposed caption-to-image alignment. In particular, we show that: (i) the CLIP matching score statistics are significantly improved by replacing the original captions with BLIP captions (Fig. 3a), and qualitatively they are a better fit in many cases (Fig. 3b); (ii) even enhancing the caption quality alone has a positive impact on the compositional reasoning performance of the model; (iii) DAC full approach significantly outperforms BLIP2 [10] in terms of compositional reasoning (by over 6 . 4% on VL-checklist [20] and over 50% on ARO, Tab. 1), indicating caption quality enhancement alone is insufficient. Interestingly, a bonus property of the proposed caption enhancement method is that by using it one could potentially leverage any unlabeled image collection, without requiring any paired text data, for improving the compositional reasoning performance of a VL model. We believe that our observation, therefore, uncovers an interesting hidden potential of unlabeled image collections for VL training and may have a significant potential impact on the VL compositional reasoning performance scaling in future work, underlining the significance of our findings.

3.2 Improving caption density

Another important drawback of Internet-collected VL datasets is their 'partiality' or 'lack of density'. Intuitively, most of these captions are not intended by web page authors to be descriptive of the image but rather serve different purposes. Consequently, in most if not all of the cases, these captions do not tell the complete story of the image in terms of not mentioning every detail and not imparting

additional knowledge on related concepts from world knowledge. Below, we propose and explore two orthogonal methods for enhancing the density of the captions.

Large Language Model (LLM) knowledge expansion. One solution, is using either the original caption, the higher quality caption produced by an image captioner, or both, and use them as a prompt to a Large Language Model (LLM) effectively tapping on its world knowledge acquired from training on trillions of tokens. In our experiments we used the GPT-Neo-2.7B open LLM, instructing it to generate an expansion from a given caption via zero-shot (in context) prompts. The full prompts are long and provided in the Supplementary, their essence though can be summarized here as: "what should I expect to see in an image of {caption}?". We parsed the resulting text into a collection of individual sentences and employed them in our training via MIL losses (Sec. 3.6) in order to account for the caption noise expected with this kind of LLM expansion (Fig. 2). Indeed, the source caption is the only information provided to the LLM, and some facts are likely to be hallucinated and unrelated to the corresponding image. We provide an analysis of the effects of this way of increasing caption density through LLMs as well as of various related design choices in Sec. 5. Interestingly, LLM knowledge expansion alone has a mixed effect on the compositional reasoning performance, however, it is very effective in combination with other proposed techniques.

Leveraging semantic over-segmentation. An additional, more image-grounded way, to enhance caption density is to decompose the image into a set of segments with consistent language semantics and caption each segment individually or in connected segment groups. Consequently, we employ SAM [28] to produce a decomposition of an image into a collection of segments (seeded by a grid of points). We refer to this as 'semantic over-segmentation' as SAM applied in this way produces segments generally respecting semantic boundaries between objects, but commonly over-segments, highlighting object parts and other sub-object regions. We caption each segment using BLIP2 [10] resulting in a collection of captions for each image. As it is an over-segmentation, some of the resulting segment captions tend to be noisy (Fig. 2). Similar to the LLM-generated captions, we also employ these in our training through MIL (Sec. 3.6). The analysis of the segmentation-based expansion and effects of this way of caption density enhancement on the compositional reasoning performance, in general, is provided in Sec. 5.

3.3 Negative text augmentation

While in the VL datasets, e.g. CC3M [75] and LAION [25], each image-text pair constitutes a single sample, it was shown in recent works that it is possible to significantly enhance the effectiveness of these samples for teaching compositional reasoning to VL models via simple negative text augmentation [21, 24]. Using this approach, every image receives a set of captions, one caption correctly matching the image (the source caption), and one or several captions manipulated to be negative text samples via changing words within the caption, thus highlighting the importance of different compositional reasoning aspects (attributes, relations, actions, states) to the finetuned model. For our DAC approach we also employ this negative text augmentation strategy combining it with the DAC captions enhancement principles outlined in Sec. 3.1 and Sec. 3.2. Specifically, we explore applying this augmentation both to the BLIP2 captions (Sec. 3.5), as well as to the LLM-expanded and Segmentation-expanded captions by their combination with the MIL training losses (Sec. 3.6). Notably, our DAC approach attains significant improvements (e.g. over 12% on VL-checklist [20]) in compositional reasoning performance over the recent methods that have proposed the negative text augmentation [21, 24], underlining the importance of the DAC factors outlined above. The effects of negative text augmentation are further explored in Sec. 5.

3.4 Avoiding forgetting with Parameter Efficient Fine-Tuning

In this work we explore fine-tuning a large-scale pre-trained VL model (e.g. CLIP [8], pre-trained on 400M image-text pairs) on a relatively smaller scale enhanced VL dataset (e.g. CC3M [75] with 3M image-text pairs). Intuitively, naively finetuning on the smaller dataset may incur 'forgetting' the powerful linear transfer capabilities of the base model (CLIP) that are of major significance for downstream applications, such as LLaVa [12] and Mini-GPT4 [11] enabling open, GPT4-like, multi-modal conversational AI. As this is undesirable, similar to other works [24, 77, 78], we employ parameter efficient finetuning, or more specifically LoRA [79], incorporated into all parameters of both vision and text encoders of the model to finetune the otherwise frozen model. As can be seen in Sec. 4.3, this strategy almost completely eliminates linear transfer performance loss only observing ∼ 1 . 5% drop in the 5-shot setting.

3.5 Training Losses

A dual-encoder VLM (e.g., CLIP [8]) admitting text-image pair ( T, I ) is comprised of: (i) an image encoder e I = E I ( I ) ; (ii) a text encoder e T = E T ( T ) . The text-image similarity score is computed as:

where τ is a learned temperature parameter.

Contrastive Loss. As most contemporary VL models, we employ the contrastive CLIP-loss [8] as one of our losses for each batch B = { ( T i , I i ) } where texts T i of the text-image pairs are quality-enhanced as explained in Sec. 3.1.

Negatives Loss. In addition to using negative text augmentation as part of our proposed MIL losses (Sec. 3.6), we employ the following 'negatives loss' applied to the quality-enhanced captions T i :

where T neg i is the negative text generated from T i according to the technique of [24].

3.6 Multiple Instance Learning (MIL)

For both kinds of caption density expansions, LLM-based and Segmentation-based, proposed in Sec. 3.2, we arrive with a bag of caption texts tentatively corresponding to the paired image. One approach to using those bags is random sampling or averaging in combination with the other losses. However, due to the inherent noise (e.g., ∼ 46% for LLM-expansion measured on 100 random images) in both the density expansion methods (explained in Sec. 3.2), as can be seen from the ablations in Sec. 5, neither produces satisfactory compositional reasoning improvements. To better cope with this noise, we propose the MIL approach discussed next.

The basic MIL setting considers a set of M captions { T i,m } M m =0 such that at least one of these captions is a positive match to the paired image I i . In our DAC approach, we extend the MIL-NCE [64] loss that was originally proposed for visual representation learning from uncurated videos. We adopt the MIL-NCE for our needs as follows:

For any source of MIL bag (LLM / Segmentation), to combine negative augmentation, explained in Section 3.3, with MIL we modify the base-MIL loss L base MIL in Eq. 4 to incorporate the non-matching captions in the denominator of the equation:

where T neg i,m is the result of negative augmentation applied to the captions bag element T i,m . Finally, the full finetuning loss of our proposed DAC approach can be written as:

3.7 Implementation details

We used the popular ViT-B/32 OpenAI CLIP [8] original PyTorch implementation as our VL model in all the experiments. For caption quality enhancement (Sec. 3.1), we used the LA VIS implementation of BLIP2 with OPT 6.7B LLM. We used ViT-H SAM model [28] for the Segmentation-based density expansion (Sec. 3.2). Furthermore, we employed the GPT-NEO-2.7B LLM for the LLM-based density expansion (Sec. 3.2). During training, we set the batch size to 128 when training without density expansion (for ablations) and to 32 with density expansions. We used 6 v100 GPUs for 12 hours to train a model. We set the learning rate 5.0e-4, and use the AdamW optimizer over 5 epochs initializing with the CLIP weights. Our code is provided in the Supplementary, and it will be released upon acceptance together with our trained weights.

4 Experimental Results

4.1 Datasets

Training: We use the Conceptual Captions 3M (CC3M) dataset [75] to finetune CLIP for enhanced compositional reasoning performance using our proposed DAC approach. CC3M is a large set of 3 million image-text pairs automatically crawled from the internet. While the dataset includes captions for the images, we have found that enhancing the quality of the captions before finetuning, as described in Sec. 3.1, greatly improves compositional reasoning. For our full approach, we therefore discard the original captions from the dataset, which (as discussed in Sec. 3.1) uncovers yet another interesting potential of our method - the ability to leverage any unlabeled image collection.

Evaluation: We evaluate our method on two major benchmarks for VL compositional reasoning, VL-Checklist [20] and ARO [21], and also the Elevater [80] for image classification to verify linear transferability of the models (which is of significant importance for usability of the VL models in contemporary downstream applications such as LLaVa [12] and Mini-GPT4 [11]).

VL-Checklist [20] is a benchmark constructed from four datasets, namely Visual Genome [81], SWiG [82], VAW [83], and HAKE [84]. Each image is paired with two captions: a positive and a negative. Positive captions are part of original (paired image-text) datasets. Negative captions were constructed from positive captions by changing one word which changes the meaning of the sentence. The negatives are categorized as changes related to objects, attributes, or relations. Each category is further divided into sub-categories such as size, material, color, etc.

ARO [21] is a new compositional reasoning benchmark that includes positive or negative captions and additionally evaluates sensitivity to word order in the sentence. Word-order negative sentences are created by re-ordering the words, which changes the semantics of the sentence in aspects like attribution, relations, and the general meaning of the word order. ARO is constructed from images and captions from COCO [85], Flick30K [86], and Visual Genome [81].

Elevater [80] consists of 20 different datasets, including common classification datasets such as CIFAR100 [87], EuroSat [88], and others. We used the Linear-Probing classification protocol described in the Elevater Image Classification Toolkit [80] for evaluating the linear transferability of our resulting DAC models with improved compositional reasoning performance.

4.2 DAC for improving VL model's compositional reasoning performance

Our DAC approach includes caption quality and caption density enhancement steps, followed by a proposed fine-tuning technique that effectively combines several losses, including the proposed MIL loss L neg MIL explained in Sec. 3.6 to cope with the inherent noise of caption density expansion. Our main results are summarized in Table 1 and are compared to four strong baselines: (i) CLIP [8] which is also the source model that we improve; (ii) BLIP2 [10] which we also use for caption quality enhancement - to show our resulting models are significantly stronger in compositional reasoning, despite being based on a faster dual-encoder CLIP model (compared to 2-stage encoder-decoder BLIP2); and (iii) two SOTA methods on the compositional reasoning benchmarks, both based on negative text augmentation - NegClip [21] and SVLC [24]. Our DAC-LLM explores the use of LLMs, specifically GPT-NEO-2.7B, to expand the caption and generate additional information that

is plausible to find in a scene described by the source (quality enhanced) caption. Our DAC-SAM leverages semantic over-segmentation [28] and converts image segments into short captions generated from them using [10]. Both these methods increase caption density and add information that was not necessarily available in the quality-enhanced caption. Both employ MIL (Sec. 3.6) combined with negative text augmentation (Sec. 3.3). As can be seen in Tab. 1, both our methods significantly improve on most compositional reasoning metrics with gains of up to 17% over the current SOTA. In section 5 we analyze the contribution of each individual component.

4.3 Preserving downstream linear transferability with DAC

We used the Linear-Probing classification protocol described in the Elevater Image Classification Toolkit [80] for evaluating the linear transferability to downstream tasks of our resulting DAC models (DAC-LLM and DAC-SAM) with improved compositional reasoning performance. This evaluation is of significant importance, as it demonstrates in a way the usability of the improved VL models in contemporary downstream applications, such as LLaVa [12] and Mini-GPT4 [11] multi-modal conversational agents, which use the linear alignment of VL model features to a strong LLM, e.g., Vicuna [5] in [11, 12].Table 2 summarizes the Elevater Linear Probing (LP) results. As prescribed by the standard Elevater LP protocol [80], we explore different amounts of training data for LP training, i.e. 5-shot, 10-shot, 20-shot, and all-shot. For all settings, we see that our DAC models are comparable or better than the baseline CLIP [8], only having 1 . 5% drop in 5-shot. These experiments show that our DAC VL models compositional reasoning enhancement approach did not degrade the representation power of the original CLIP model.

5 Ablation study

Below we perform an extensive ablation analyzing and highlighting the importance of caption quality and caption density factors towards improving compositional understanding of VL models. We find that each factor separately contributes to compositional reasoning performance, also improving over the baselines, while the best results are obtained by combining the factors. All experiments were conducted for the same number of epochs on the same dataset starting from CLIP pre-trained weights.

(a) CLIP Score

(b) CLIP Score Examples

0.06

0.05

0.04

E0'0

0.02

0.01

0

10 1 12 13 14 15 16 17 18 19 20 22 22 23 4 25 26 77 28 2 30 31 2 3 4 35 6 7 8 939 40 4 42 43 4 45 46 47 48 49 0  72

Original caption scoreCaptioner score

Original text (clip score 18.13):

Original text (clip score 19.42):

‘Person finishes at the top.'

‘Person finallyget their hands with win.

(c) Gradually Improving

(d) Effect of Density in

(e) Effect of Density in

Captioner text (clip score 29.26):

Captioner text (clip score 33.40):

88877万1316665686975

Captions Quality

665

88

LLM

2908648200181647210886462

SAM

‘A group of men standing on a golf

course.

trophy.

The chelsea players are holding up the

84

E8

82

81

80

79

78

77

76

75

Original text (clip score 26.57):

Original text (clip score 39.38):

100% original

50% - 50% 100% improved

74

20% Density50% Densty100% Density

20% Density50% Density100% Density

Captioner text (clip score 37.06):

“The sign for a city.

‘A pencil drawing of a zebra and her baby.

Captioner text (clip score 36.42):

captions

Aallenb

‘A red and white sign with the word

'A drawing of two zebras looking at each

Object

Attribute

Relation

Object

Attribute

—Relation

ObjectAttribute—Relation

other.

Caption Quality. Here we validate the benefits of improving caption quality. First, Fig 3a compares the statistics of the CLIP score between captions and images before and after the quality improvement (Sec. 3.1). Clearly, improved captions have better matching statistics. Fig 3b shows some qualitative examples of caption improvement, in all cases original captions hardly describe the image content. Next, Figure 3c shows that gradually increasing the ratio of high-quality captions consistently improves the VL-Checklist metrics. To isolate the effect of quality improvement, we perform this experiment without the caption density enhancements. Finally, block A in Tab. 4 explores the contribution of DAC components without enhancing caption quality. We can see that all of them improve, yet the largest improvements are obtained with quality enhancement (blocks B and C). We have also conducted a caption quality user study. We asked 121 human subjects to review 110 random images from CC3M. For each image, we asked which of 3 choices best fits the image. A and B were captions, either produced by BLIP2 or originating from CC3M (A and B assignment was randomized). Option C was 'neither caption fits the image'. We found that 80.7% of responses favored the BLIP2 caption, 2.2% the original caption, and 17.1% was 'neither'. This indicates that BLIP2 captions are indeed better aligned with human perception than the alt-text collected from the Web in CC3M. Intuitively, such better alignment to humans, who are inherently good at compositional reasoning, likely leads to the significant compositional reasoning improvements observed when gradually increasing the percent of the higher quality (BLIP2) captions in the fine-tuning data (Fig. 3c).

Caption Density. Next, we validate the contribution of caption density expansion - increasing the number of captions from different sources (LLM / Segmentation + Caption) produced for each training image (Sec. 3.2). Figure 3d and Figure 3e analyze the improvement in VL-Checklist metrics by expanding the caption density using the DAC-LLM and DAC-SAM methods respectively, while using the proposed MIL techniques to cope with potential noise. Logically, this analysis is done after the quality enhancement and using our full approach. Clearly, increasing caption density improves results. This is also apparent from comparing rows 1 and 2 in block B in Tab. 4.

MIL. In Tab. 3 we explore the different MIL variants for our DAC approach. The MAX - choosing the best match out of MIL-bag, and AVG - averaging text embeddings across MIL-bag, are simple MIL substitutes. We also compare our proposed MIL loss L neg MIL (Eq. 5) with its base L base MIL (Eq. 4) variant. Clearly the final proposed DAC MIL loss L neg MIL has the advantage. This is also evident from comparing blocks B (no MIL) and C (with MIL) numbers in Tab. 4.

DAC-SAM relation extraction quality. We have manually analyzed 100 randomly sampled images from CC3M to check how many: (i) correct-unique relations were contributed by over-segmenting (small object-part segments unavoidable in whole image SAM); and (ii) correct relations generated

for SAM segments in general. Here, correct = visible on the image; unique = not mentioned in any full-object segment for the same image. As some object-parts, such as human hands, are very important, we observed that object-part segments have contributed at least 2 correct-unique relations per 3 images on average. This is potentially a significant boost of relation density for future work applying DAC on large (millions) of images (unlabelled) image collections. Additionally, we measured 3 . 4 ± 1 . 1 correct relations per image generated from SAM segments. Since relations (e.g. 'in', 'on', 'holding', 'touching', 'sitting on', 'standing by', etc.) typically involve overlapping or nearly overlapping objects, our segment bounding box crops and the captions resulting from them often do capture those relations and teach them to the VL model, leading to the observed significant performance boost in relation metrics.

Model and data scaling. Experiments analyzing the model and data scaling aspects of DAC can be found in Fig. 4 and Tab. 5.

DataScalingAblation-DAC

86

8

28

8

80

79

78

0

0.5

1

1.5

2

2.5

3

Amount of data #M

6 Summary & Conclusions

In conclusion, our work focused on addressing compositional reasoning limitations in Vision and Language Models (VLMs) by considering the factors of caption quality and density. Through our proposed fine-tuning approach applied to the CC3M dataset, we successfully demonstrated a substantial increase in compositional reasoning performance, surpassing both standard VLMs and the strongest baselines. Interestingly, our results indicate that we can effectively tap into visual world knowledge contained in LLMs that have been trained using massive text-only data, and use their expertise to improve VLM compositional reasoning performance. Moreover, our approach can be also applied to any unlabeled image collection without requiring any paired image-text data supervision.

Limitations: Despite the promising results, our work has a limitation. We focused primarily on caption quality and density, and while they contributed to improved compositional reasoning, there may be additional factors that could further enhance VLMs' performance. Exploring other factors and their impact on compositional reasoning would be a worthwhile direction for future improvements. Regarding societal impact, we do not anticipate any specific negative impact, but, as with any Machine Learning method, we recommend exercising caution.

Acknowledgement: This work was partially supported by Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program, Korea University).

Figure 4: Data scaling experiments evaluating the effect of the amount of (unlabelled) image data used during finetuning with DAC. We can clearly see that the performance (average compositional reasoning accuracy measured on VL-checklist+ARO) increases with adding more image data, and does not seem to be plateauing. We extrapolate from these plots and conclude that it is likely that improvements attained by our DAC approach indeed scale nicely with adding more data.

References

OpenAI, GPT-3.5 , GitHub repository, 2023. [Online]. Available: https://github.com/openai/gpt3.5 .

OpenAI, Gpt-4 technical report , 2023. arXiv: 2303.08774 [cs.CL] .

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. , 'Llama: Open and efficient foundation language models,' arXiv preprint arXiv:2302.13971 , 2023.

R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, Stanford alpaca: An instruction-following llama model , https://github.com/tatsu-lab/stanford_ alpaca , 2023.

W.-L. Chiang et al. , Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality , Mar. 2023. [Online]. Available: https://lmsys.org/blog/2023-03-30-vicuna/ .

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, 'High-resolution image synthesis with latent diffusion models,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 10 684-10 695.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, 'Hierarchical text-conditional image generation with clip latents,' arXiv preprint arXiv:2204.06125 , 2022.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. , 'Learning transferable visual models from natural language supervision,' in International conference on machine learning , PMLR, 2021, pp. 8748-8763.

J. Li, D. Li, C. Xiong, and S. Hoi, Blip: Bootstrapping language-image pre-training for unified visionlanguage understanding and generation , 2022. eprint: arXiv:2201.12086 .

J. Li, D. Li, S. Savarese, and S. Hoi, 'BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models,' in ICML , 2023.

D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, 'Minigpt-4: Enhancing vision-language understanding with advanced large language models,' arXiv preprint arXiv:2304.10592 , 2023.

H. Liu, C. Li, Q. Wu, and Y. J. Lee, 'Visual instruction tuning,' arXiv preprint arXiv:2304.08485 , 2023.

C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig, 'Scaling up visual and vision-language representation learning with noisy text supervision,' 2021. eprint: arXiv:2102.05918 .

A. Zareian, K. D. Rosa, D. H. Hu, and S.-F. Chang, 'Open-vocabulary object detection using captions,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 14 39314 402.

X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui, 'Open-vocabulary object detection via vision and language knowledge distillation,' arXiv preprint arXiv:2104.13921 , 2021.

Y. Zhong, J. Yang, P. Zhang, C. Li, N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y. Li, et al. , 'Regionclip: Region-based language-image pretraining,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 16 793-16 803.

J. Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang, 'Groupvit: Semantic segmentation emerges from text supervision,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 18 134-18 144.

T. Lüddecke and A. Ecker, 'Image segmentation using text and image prompts,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 7086-7096.

G. Ghiasi, X. Gu, Y. Cui, and T.-Y. Lin, 'Scaling open-vocabulary image segmentation with image-level labels,' in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI , Springer, 2022, pp. 540-557.

T. Zhao, T. Zhang, M. Zhu, H. Shen, K. Lee, X. Lu, and J. Yin, 'Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations,' arXiv preprint arXiv:2207.00221 , 2022.

M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou, 'When and why vision-language models behave like bags-of-words, and what to do about it?' In International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id=KRLUvxh8uaX .

T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross, 'Winoground: Probing vision and language models for visio-linguistic compositionality,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 5238-5248.

J. Yang, H. Jin, R. Tang, X. Han, Q. Feng, H. Jiang, B. Yin, and X. Hu, 'Harnessing the power of llms in practice: A survey on chatgpt and beyond,' 2023. arXiv: 2304.13712 [cs.CL] .

S. Doveh, A. Arbelle, S. Harary, R. Panda, R. Herzig, E. Schwartz, D. Kim, R. Giryes, R. Feris, S. Ullman, et al. , 'Teaching structured vision&language concepts to vision&language models,' arXiv preprint arXiv:2211.11733 , 2022.

C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki, 'Laion-400m: Open dataset of clip-filtered 400 million image-text pairs,' arXiv preprint arXiv:2111.02114 , 2021.

S. Black, G. Leo, P. Wang, C. Leahy, and S. Biderman, GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow , version 1.0, If you use this software, please cite it using these metadata., Mar. 2021. DOI: 10.5281/zenodo.5297715 . [Online]. Available: https://doi.org/10.5281/ zenodo.5297715 .

S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. , 'Opt: Open pre-trained transformer language models,' arXiv preprint arXiv:2205.01068 , 2022.

A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. , 'Segment anything,' arXiv preprint arXiv:2304.02643 , 2023.

W. Kim, B. Son, and I. Kim, 'Vilt: Vision-and-language transformer without convolution or region supervision,' in International Conference on Machine Learning , PMLR, 2021, pp. 5583-5594.

J. Yang, J. Duan, S. Tran, Y. Xu, S. Chanda, L. Chen, B. Zeng, T. Chilimbi, and J. Huang, 'Visionlanguage pre-training with triple contrastive learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 15 671-15 680.

S. Goel, H. Bansal, S. Bhatia, R. A. Rossi, V . Vinay, and A. Grover, 'Cyclip: Cyclic contrastive languageimage pretraining,' arXiv preprint arXiv:2205.14459 , 2022.

L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu, 'Filip: Fine-grained interactive language-image pre-training,' arXiv preprint arXiv:2111.07783 , 2021.

A. Fürst, E. Rumetshofer, V. Tran, H. Ramsauer, F. Tang, J. Lehner, D. Kreil, M. Kopp, G. Klambauer, A. Bitto-Nemling, et al. , 'Cloob: Modern hopfield networks with infoloob outperform clip,' arXiv preprint arXiv:2110.11316 , 2021.

Y. Li, F. Liang, L. Zhao, Y. Cui, W. Ouyang, J. Shao, F. Yu, and J. Yan, 'Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm,' arXiv preprint arXiv:2110.05208 , 2021.

Y. Gao, J. Liu, Z. Xu, J. Zhang, K. Li, and C. Shen, 'Pyramidclip: Hierarchical feature alignment for vision-language model pretraining,' arXiv preprint arXiv:2204.14095 , 2022.

R. Herzig, A. Mendelson, L. Karlinsky, A. Arbelle, R. Feris, T. Darrell, and A. Globerson, Incorporating structured representations into pretrained vision & language models using scene graphs , 2023. arXiv: 2305.06343 [cs.CV] .

Y.-C. Chen, L. Li, L. Yu, A. E. Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, 'Uniter: Universal image-text representation learning,' in ECCV , 2020.

L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, 'Visualbert: A simple and performant baseline for vision and language,' ArXiv , vol. abs/1908.03557, 2019.

X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al. , 'Oscar: Object-semantics aligned pre-training for vision-language tasks,' in European Conference on Computer Vision , 2020, pp. 121-137.

H. H. Tan and M. Bansal, 'Lxmert: Learning cross-modality encoder representations from transformers,' in EMNLP , 2019.

D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei, 'Scene Graph Generation by Iterative Message Passing,' in Proc. CVPR , 2017, pp. 3097-3106.

R. Herzig, M. Raboh, G. Chechik, J. Berant, and A. Globerson, 'Mapping images to scene graphs with permutation-invariant structured prediction,' in Advances in Neural Information Processing Systems (NIPS) , 2018.

R. Krishna, I. Chami, M. S. Bernstein, and L. Fei-Fei, 'Referring relationships,' ECCV , 2018.

A. Jerbi, R. Herzig, J. Berant, G. Chechik, and A. Globerson, 'Learning object detection from captions via textual scene attributes,' ArXiv , vol. abs/2009.14558, 2020.

M. Raboh, R. Herzig, G. Chechik, J. Berant, and A. Globerson, 'Differentiable scene graphs,' in WACV , 2020.

F. Baradel, N. Neverova, C. Wolf, J. Mille, and G. Mori, 'Object level visual reasoning in videos,' in Proc. ECCV , 2018, pp. 105-121.

P. W. Battaglia, J. B. Hamrick, V . Bapst, A. Sanchez-Gonzalez, V . Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al. , 'Relational inductive biases, deep learning, and graph networks,' arXiv preprint arXiv:1806.01261 , 2018.

F. Liu, G. E. T. Emerson, and N. Collier, 'Visual spatial reasoning,' Transactions of the Association for Computational Linguistics , 2023.

C. Gao, J. Xu, Y. Zou, and J.-B. Huang, 'Drg: Dual relation graph for human-object interaction detection,' ArXiv , vol. abs/2008.11714, 2020.

K. Kato, Y. Li, and A. Gupta, 'Compositional learning for human object interaction,' in ECCV , 2018.

B. Xu, Y. Wong, J. Li, Q. Zhao, and M. Kankanhalli, 'Learning to detect human-object interactions with knowledge,' 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 2019-2028, 2019.

E. B. Avraham, R. Herzig, K. Mangalam, A. Bar, A. Rohrbach, L. Karlinsky, T. Darrell, and A. Globerson, 'Bringing image scene structure to video via frame-clip consistency of object tokens,' in Thirty-Sixth Conference on Neural Information Processing Systems , 2022.

A. Arnab, C. Sun, and C. Schmid, 'Unified graph structured models for video understanding,' in ICCV , 2021.

J. Materzynska, T. Xiao, R. Herzig, H. Xu, X. Wang, and T. Darrell, 'Something-else: Compositional action recognition with spatial-temporal interaction networks,' in proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2020.

R. Herzig, E. Ben-Avraham, K. Mangalam, A. Bar, G. Chechik, A. Rohrbach, T. Darrell, and A. Globerson, 'Object-region video transformers,' in Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

R. Herzig, E. Levi, H. Xu, H. Gao, E. Brosh, X. Wang, A. Globerson, and T. Darrell, 'Spatio-temporal action graph networks,' in Proceedings of the IEEE International Conference on Computer Vision Workshops , 2019, pp. 0-0.

R. Herzig, O. Abramovich, E. Ben-Avraham, A. Arbelle, L. Karlinsky, A. Shamir, T. Darrell, and A. Globerson, Promptonomyvit: Multi-task prompt learning improves video transformers using synthetic scene data , 2023. arXiv: 2212.04821 [cs.CV] .

J. Ji, R. Krishna, L. Fei-Fei, and J. C. Niebles, 'Action genome: Actions as composition of spatio-temporal scene graphs,' arXiv preprint arXiv:1912.06992 , 2019.

X. Wang and A. Gupta, 'Videos as space-time region graphs,' in ECCV , 2018.

A. Bar, R. Herzig, X. Wang, A. Rohrbach, G. Chechik, T. Darrell, and A. Globerson, 'Compositional video synthesis with action graphs,' in ICML , 2021.

R. Herzig, A. Bar, H. Xu, G. Chechik, T. Darrell, and A. Globerson, 'Learning canonical representations for scene graph to image generation,' in European Conference on Computer Vision , 2020.

J. Johnson, A. Gupta, and L. Fei-Fei, 'Image generation from scene graphs,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 1219-1228.

A. Miech, J.-B. Alayrac, P. Bojanowski, I. Laptev, and J. Sivic, 'Learning from video and text via large-scale discriminative clustering,' 2017 IEEE International Conference on Computer Vision (ICCV) , pp. 5267-5276, 2017.

A. Miech, J.-B. Alayrac, L. Smaira, I. Laptev, J. Sivic, and A. Zisserman, 'End-to-end learning of visual representations from uncurated instructional videos,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 9879-9889.

N. Shapovalova, A. Vahdat, K. J. Cannons, T. Lan, and G. Mori, 'Similarity constrained latent support vector machine: An application to weakly supervised action classification,' in ECCV , 2012.

M. Oquab, on Bottou, I. Laptev, and J. Sivic, 'Is object localization for free? - weakly-supervised learning with convolutional neural networks,' 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 685-694, 2015.

G. Quellec, G. Cazuguel, B. Cochener, and M. Lamard, 'Multiple-instance learning for medical image and video analysis,' IEEE Reviews in Biomedical Engineering , vol. 10, pp. 213-234, 2017.

K. Sirinukunwattana, S.-e.-A. Raza, Y.-W. Tsang, D. R. J. Snead, I. A. Cree, and N. M. Rajpoot, 'Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images,' IEEE transactions on medical imaging , vol. 35 5, pp. 1196-1206, 2016.

K. Ye, M. Zhang, A. Kovashka, W. Li, D. Qin, and J. Berent, 'Cap2det: Learning to amplify weak caption supervision for object detection,' 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 9685-9694, 2019.

B. Zhou, A. Khosla, gata Lapedriza, A. Oliva, and A. Torralba, 'Learning deep features for discriminative localization,' 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 29212929, 2016.

P. Bojanowski, F. R. Bach, I. Laptev, J. Ponce, C. Schmid, and J. Sivic, 'Finding actors and actions in movies,' 2013 IEEE International Conference on Computer Vision , pp. 2280-2287, 2013.

G. Chéron, J.-B. Alayrac, I. Laptev, and C. Schmid, 'A flexible model for training action localization with varying levels of supervision,' in NeurIPS , 2018.

T. Leung, Y. Song, and J. R. Zhang, 'Handling label noise in video classification via multiple instance learning,' 2011 International Conference on Computer Vision , pp. 2056-2063, 2011.

A. Alfassy, A. Arbelle, O. Halimi, S. Harary, R. Herzig, E. Schwartz, R. Panda, M. Dolfi, C. Auer, K. Saenko, et al. , 'Feta: Towards specializing foundation models for expert task applications,' arXiv preprint arXiv:2209.03648 , 2022.

P. Sharma, N. Ding, S. Goodman, and R. Soricut, 'Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,' in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2018, pp. 2556-2565.

S. Santurkar, Y. Dubois, R. Taori, P. Liang, and T. Hashimoto, 'Is a caption worth a thousand images? a controlled study for representation learning,' arXiv preprint arXiv:2207.07635 , 2022.

J. S. Smith, P. Cascante-Bonilla, A. Arbelle, D. Kim, R. Panda, D. Cox, D. Yang, Z. Kira, R. Feris, and L. Karlinsky, 'Construct-vl: Data-free continual structured vl concepts learning,' arXiv preprint arXiv:2211.09790 , 2022.

X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer, 'Lit: Zero-shot transfer with locked-image text tuning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 18 123-18 133.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, 'Lora: Low-rank adaptation of large language models,' arXiv preprint arXiv:2106.09685 , 2021.

C. Li et al. , 'Elevater: A benchmark and toolkit for evaluating language-augmented visual models,' Neural Information Processing Systems , 2022.

R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. , 'Visual genome: Connecting language and vision using crowdsourced dense image annotations,' International journal of computer vision , vol. 123, no. 1, pp. 32-73, 2017.

S. Pratt, M. Yatskar, L. Weihs, A. Farhadi, and A. Kembhavi, 'Grounded situation recognition,' in European Conference on Computer Vision , Springer, 2020, pp. 314-332.

K. Pham, K. Kafle, Z. Lin, Z. Ding, S. Cohen, Q. Tran, and A. Shrivastava, 'Learning to predict visual attributes in the wild,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 13 018-13 028.

Y.-L. Li, L. Xu, X. Liu, X. Huang, Y. Xu, M. Chen, Z. Ma, S. Wang, H.-S. Fang, and C. Lu, 'Hake: Human activity knowledge engine,' arXiv preprint arXiv:1904.06539 , 2019.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, 'Microsoft coco: Common objects in context,' in European conference on computer vision , Springer, 2014, pp. 740755.

B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik, 'Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,' in Proceedings of the IEEE international conference on computer vision , 2015, pp. 2641-2649.

A. Krizhevsky, G. Hinton, et al. , 'Learning multiple layers of features from tiny images,' 2009.

P. Helber, B. Bischke, A. Dengel, and D. Borth, 'Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,' in IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium , IEEE, 2018, pp. 204-207.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of getting models like CLIP to perform compositional reasoning. One observed problem with these models is that they devolve into \"bags of objects\" models, and this work seeks to address their variable-binding ability towards identifying more complex relationships. This paper takes a data-first approach, hypothesizing that the data behind these models might be flawed.\n\nThe paper starts with the Conceptual Captions 3M dataset. The quality of captions is improved by:\n1. Using BLIP2 + OPT to recaption the images\n2. Using a GPT-Neo-2.7B model to perform \"knowledge expansion\". This hallucinates extra text for the given caption, though it should be noted that this is addressed by MIL losses that are introduced.\n3. Using segment-everything to get a bunch of regions, which are then fed to BLIP@.\n4. Using \"negative text augmentation\" to change words around to form captions that don't match the image.\n\nThe paper studies CLIP models that are finetuned on CC3M, using LoRA to reduce catastrophic forgetting. The CLIP losses are extended to incorporate the negative text augmentation strategy as well as to address the knowledge-expansion hallucination issue with MIL.\n\nThe paper evaluates on VL-Checklist, ARO, and Elevater, which are all datasets for compositional reasoning. Linear probing results show that the model performs better on these datasets with benefits on compositionality.\n\n---\nupdate: increasing my score from 6->7 as my main concerns were resolved. I vote to accept this paper. I think I'm in agreement with the other reviewers, except for bpWN whose concerns I'm not really understand (they feel like curious questions IMO but not reason to reject the paper).\n\nSTRENGTHS:\nOverall I am a fan of this paper. I think it addresses an important question: how to make CLIP models more robust to compositional reasoning and variable binding, and it does so by addressing an often-understudied aspect, the role of data.\n\nThe experimental results and ablation study seem solid to this reviewer, and suggest that the various recaptioning / data augmentation strategies help.\n\nWEAKNESSES:\nTo this reviewer, the paper seems strong overall. However, I think the collection of caption augmentations seem a bit complicated and so I'm left wanting to know which is the most effective in terms of overall \"bang for your buck\".\n\nFor Figure 3 I'm not convinced by the use of CLIP score here. If the paper presents ways of exceeding CLIP performance on compositional data then I'm not sure if CLIPScore is a reasonable metric for evaluating captions that are truly better than CLIP.\n\nI would also prefer if these improvements were tried on a larger CLIP model, as ViT-B/32 seems rather small.\n\nSee the questions below - I think also they could improve the paper if addressed.\n\nQUESTIONS:\n* What strategies are the most effective in terms of pretraining/inference compute?\n* Do these results hold across model sizes (e.g. CLIP models and recaptioning models)?\n\nLIMITATIONS:\nthanks for adding a limitations section!", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-23"}

comments: SUMMARY:
The current vision-language (VL) models suffer from "object bias" issues, this paper proposes two components to improve the compositional reasoning capability of the existing VL models. It employs the pretrained LLMs to augment the captions of the images, in terms of quality and density, then finetune the CLIP models on CC3M corpus, and demonstrates promising results on two benchmarks.

STRENGTHS:
1. This paper employs the strong pretrained LLM to enhance the caption quality and quantity of the images, and finetunes the pretrained CLIP on the augmented CC3M corpus, and shows the promising results on two compositional reasoning benchmarks.

WEAKNESSES:
1. An incremental work to improve the compositional capability of the VL models by augmenting text captions with images, and show decent performance on the two benchmark, and weak performance on the Elevater benchmark.

QUESTIONS:
1. I assume the the first row in the Table 4 is the CLIP finetuned on CC3M baseline, right? If yes, it looks hard negative captions are key component to improve the performance (row 3 in A), and quality looks not an important component comparing row 0 in B and row CLIP, it gets worse.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper highlights two problems with existing image-text datasets that make them unsuitable for use as pre-training datasets for evaluating performance on attributes and relations, and claims that the models simply act as bag of words when trained on these datasets. The first problem they identify is that often in these web-scraped datasets, the text is not actually describing the contents of the image, but instead is describing how the captioner feels about the image or unrelated information entirely. The second is that the captions often only describe a part of the image. They propose two approaches - captioning images to improve quality and density as well as a fine-tuning approach that is able to handle the noisy captions generated by their method. \n\n\nEdit: I have updated my score to reflect the rebuttal\n\nSTRENGTHS:\n## Originality and Significance\n* The paper attempts to address two important issues with pre-training on image text datasets (quality and density), which could improve the performance of vision and language models \n* They use two interesting approaches to improve density of captions - using an LLM to probe for other things that can be said about a given caption (without access to an image) and based on SAM. \n* They evaluate performance on image classification benchmarks to see performance compared to the base model (CLIP)\n\nWEAKNESSES:\n## Major issues\n* \"the CLIP matching score statistics are significantly improved by replacing the original captions with BLIP captions (Fig. 3a), and qualitatively they are a better fit in many cases \" The conclusion that BLIP2 based captions are better than the original captions is drawn by computing the CLIP matching score and observing that these are scores higher, which is expected since the BLIP2 model is based on CLIP and would understandably score these outputs higher.  This is insufficient evidence and needs to be supported either by human judgement or other approaches (it is unclear what is meant by qualitatively they are a better fit in many cases <-- was a study conducted to conclude this?)\n* LLM expander approach : \"Indeed, the source caption is the only information provided to the LLM, and some facts are likely to be hallucinated and unrelated to the corresponding image.\" <-- is there some statistic on how often this happens? Some human analysis on a random selection of images that verifies how many expansions have hallucinated facts about the image would be useful to judge the usefulness of this approach. \n* The SAM expander section does not provide any information on how captions are created from the segments. How are they provided as input to the captioning model? Is it just a crop of the single segment? If it is multiple segments how are they handled? \n* It is unclear to me how performance on relations (Fig 3) can improve using the SAM approach when as far as I understand, it is only possible to obtain single object using the pseudolabelling approach described in the paper. \n* It seems odd to me that BLIP-2 does so poorly on the ARO benchmark given that it is a generative model (13% on COCO and FLICKR when CLIP gets more than 3x). Especially when Table 4 Row B(first sub row) shows performance on ARO using captions generated by BLIP and it is around 29 & 40%, and the model on COCO-captions achieves more than 140 CIDEr. Could the authors please describe how BLIP2 was evaluated? (See Table 6 in [1] which shows that even a blind LM decoder can get 99% on the COCO and FLICKR splits that test for word order). My guess is that both ARO and this work use the ITM head - which would not be as good as scoring the likelihood of the caption under the LM head. \n* Using BLIP-2 to caption and then using those captions as data is the same as distilling information from the dataset BLIP-2 was trained on. An experiment that is missing, would be to compare a model trained on BLIP-2 training data (having CC12M, COCO, VG, SBU) with regular fine-tuning vs the approaches described in this paper as the improvements could be due to access to this larger source of image-text data. \n\n## Minor issues \n* Fig 3 (a) is unclear - what are the X-axis and Y-axis depicting? Also (c-e) \n* Negative example generation is not sufficiently described in the paper\n\n[1] Image Captioners Are Scalable Vision Learners Too. Michael Tschannen et al 2023\n\nQUESTIONS:\nIn Table 4 Row B : Could you explain why adding the LLM expansion would increase the score on COCO and FLICKR ordering subsets of ARO by 2x (row 1&2)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for data augmentation on image-text web\ncorpora they call DAC. The main idea is to run both a segmentation\nmodel and an image captioning model, and then use those outputs to\ngenerate captions that are likely to describe the image using an\nLLM. Finally, a MIL loss + LoRA over CLIP is used to align the image\nwith the autogenerated captions, and anti-align with some\nmachine-generated false negatives. The authors achieve strong\nperformance gains over vanilla CLIP (and no performance degradation in\nusual vision tasks) using this method when training on the CC3M\ndataset. Somewhat surprisingly, they actually don't even use the image\ncaptions in CC3M (L261).\n\nSTRENGTHS:\n- The topic of making CLIP-style models better compositional reasoners\n  is interesting.\n\n- The proposed approach is straightforward, creative, and promising. I\n  like that the authors used LoRA to maintain the original strong\n  performance of CLIP (and specifically test for that). The results\n  are strong compared to the CLIP ViT-B/32 baseline.\n\n- I admire that the authors used a public source LLM instead of OpenAI!\n\n- Ablations validate that all the pieces of the proposed pipeline play\n  a role.\n\nWEAKNESSES:\n- I am not the biggest fan of the framing that web captions are \"low\n  quality.\" After all, these captions were presumably written by\n  humans for a purpose other than ML training --- so they may be low\n  quality for that purpose, but L43-44's blanket description (and\n  elsewhere in the paper) I thought could be made more precise like in\n  L160.\n\n- It would have been nice for the authors to front the fact that they\n  actually just use unlabelled images earlier --- it's neat.\n\n- If SAM outputs are handed to BLIP in a single-segment fashion, how\n  are the relations in figure 2 generated? e.g., \"Miniature zoo of toy\n  animals on a mat\" seems to be something that would require multiple\n  segments to parse, and I don't see how this happens given the\n  description in L179.\n\n- It would have been nice to see scaling plots --- do these same\n  results apply to larger versions of CLIP? And, do these results\n  cleanly scale with the number of unlabelled images?\n\nQUESTIONS:\nOverall, I liked this work! It gives a nice,\ndata-bottlenecked/auditable method for transferring the compositional\nknowledge that /feels/ like it should be in/derivable from other\nmodels (like SAM) into CLIP, while maintaining the positives of\nCLIP. My biggest gripe was that the authors didn't apply the method to\nlarger CLIP models, or give scaling results to show how this method\nmight scale with (readily available) unsupervised image data.\n\n\nUPDATE: The authors have addressed my concerns and I have raised my score accordingly.\n\nLIMITATIONS:\nThe authors briefly discussed limitations in a vague sense. But could more be said about the potential risks, e.g., of propagating errors that BLIP2 makes to downstream models? Does DAC result in augmented data with more (social?) biases than the original human-authored data?", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes methods to improve performance on compositional reasoning tasks — by improving the caption quality (alignment) and density (description bias). The paper observes that the poor compositional task performance is due to these limitations in pretraining / fine-tuning data. The approach itself, is quite simple — data augmentation using existing Vision, and Vision-Language models. The noisy augmented data is utilised for fine-tuning with an appropriate multiple-instance loss function, which results in significant improvement in downstream compositional reasoning tasks across datasets.\n\nSTRENGTHS:\n(1) Paper is well-written. The approach is described clearly, and obvious issues (e.g., the hallucination introduced due to LLM-based knowledge expansion) are analyzed and discussed. \n\n(2) It is clear when visualising the noisy web-scraped training data for VLM models that the captions often are not factual descriptions of the contents or activity of the scene. So, many of these samples are unsuitable for learning a good vision-text representation that can be used for reasoning about object-relationships and other tasks. This was explored in detail [76]. The proposed solution — *generating* captions that are better aligned, and fine-tuning the model on that, is simple but somewhat surprisingly, is sufficient to improve downstream reasoning task performance. The paper also validates the generated captions with CLIP matching score. \n\n(3) Increasing the density of captions for a given image by leveraging recent improvements in the segmentation task, is a very reasonable direction to overcome people’s bias to describe only certain “interesting” elements of the image. This aspect has been discussed in the past in the image captioning literature. The paper proposes noisy augmentation techniques, that improve the learned representation for downstream tasks. They key is training with MIL which seems to account for the noise introduced into the training samples. \n\n(4) There is significant performance improvements in the reasoning tasks.\n\nWEAKNESSES:\n(1) Over-segmentation with SAM — could one explicitly filter out noisy captions that correspond only to object parts instead of relying on the MIL to do much of the heavy lifting? For instance, could one evaluate the image features of object-parts with full-objects to automatically recognise (and filter out) over-segmentations? \n\n(2) Noisy knowledge expansion with LLMs. The authors are aware of this, and discuss this. The MIL loss is introduced to handle this noise. Nevertheless, this still means that the approach is training the model with data that we know is quite noisy. Indeed, it results in performance improvement. But might be nicer to not augment w/ so much noise. This is a limitation, and it would be nice to explore a better way to augment this “knowledge”. It’s an open question\n\nQUESTIONS:\n(1) How much worse is these noisy predicted captions algorithm compared to when you may use GT captions? I.e., what may be the upper-bound of this method if we consider high quality captions? I'm not sure if there's a good way to evaluate this. I'm just curious, so I'm raising this point. \n\n(2) What about potential improvement in performance with dataset size? Since the approach uses existing pretrained models, at some point, one would expect the performance to saturate as one keeps increasing the size of the unlabeled dataset.\n\nLIMITATIONS:\nYes, the paper discusses limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}