Comparison Visual Instruction Tuning

## Abstract

Abstract Comparing two images in terms of Commonalities and Differences (CaD ) is a fundamental human capability that forms the basis of advanced visual reasoning and interpretation. It is essential for the generation of detailed and contextually relevant descriptions, performing comparative analysis, novelty detection, and making informed decisions based on visual data. However, surprisingly, little attention has been given to these fundamental concepts in the best current mimic of human visual intelligence - Large Multimodal Models (LMMs). We develop and contribute a new two-phase approach CaD-VI for
collecting synthetic visual instructions, together with an instruction-following dataset CaD-Inst containing 349K image pairs with CaD instructions collected using CaD-VI . Our approach significantly improves the CaD spotting capabilities in LMMs, advancing the SOTA on a diverse set of related tasks by up to 17.5%. It is also complementary to existing difference-only instruction datasets, allowing automatic targeted refinement of those resources increasing their effectiveness for CaD tuning by up to 10%. Additionally, we propose an evaluation benchmark with 7.5K open-ended QAs to assess the CaD understanding abilities of LMMs.

## 1 Introduction

Figure: Figure 1: CaD-VI concept. We collect and pair densely captioned source images to form synthetic CaD instructions using an LLM. The resulting synthetic CaD Visual Instruction dataset is used to train the first CaD enabled LMM that is in turn used in iterative self-refinement by annotating new paired images from additional sources using the CaD LMM, and re-training the model with a growing and more comprehensive CaD-Inst dataset (contributed in this work).
Refer to caption: x1.png

Figure: Figure 2: Pipeline of our two-phase CaD-VI : In Phase-1, we leverage captions for image pairs and an LLM to generate CaD VI data - CaD-Inst^V1 (278K), and perform visual instruction tuning on it to arrive at the Phase-1 model CaD-LLaVA^V1 . In Phase-2, we leverage CaD-LLaVA^V1 to generate CaD VI data on additional image pairs and collect CaD-Inst^V2 (71K). Visual instruction tuning with CaD-Inst^V1 and CaD-Inst^V2 leads to our final model CaD-LLaVA^V2 .
Refer to caption: x2.png

Understanding the Commonalities and Differences (CaD ) between two signals (e.g., images) is a basic capability innate to humans [gestalt]. Spotting change and difference alerts us to interesting events happening in our surroundings, warns us of hazard, and drives us toward learning new concepts exposed after the change or relative movement. Understanding what is common helps structure visual information and allows differences to emerge by elimination. Together, these form powerful tools for human learning and acquiring world knowledge.

The forefront of modern AI shifted with the recent emergence of foundation Large Language Models (LLMs) [bommasani2022opportunities], where the top-performing ones [openai2024gpt4, geminiteam2024gemini, claude, llama3] closely align to human reasoning and world-knowledge capabilities. LLMs’ great performance and wide applicability quickly led to their wide adoption into most of the current ML pipelines. In the Vision community, this impacted the development of Large Multi-modal Models (LMMs) [llava, yang2023dawn, geminiteam2024gemini, huang2023sparkles, li2023otter, internlmxcomposer2, Emu2] largely considered the best available mimic of human visual intelligence to date. While multiple methods for adding multi-modal support to LLMs have been proposed, currently the more popular and better performing open LMMs largely rely on tuning using Visual Instructions (VI) [llava, zhu2023minigpt]. These methods align image tokens produced by visual encoders to be ‘understandable’ by an LLM decoder, allowing images to be seamlessly integrated into the LLM decoder input context stream together with the query text during inference. In most recent methods [llava, huang2023sparkles, li2023otter, internlmxcomposer2], VI takes the form of a multi-turn conversation: with ‘human’ turns providing image context and asking the questions, and LMM turns answering them [llava]. However, the majority of VI data focused on providing merely a single image in the VI conversations [llava], while only a few works included multi-image VI samples [Emu2, awadalla2023openflamingo], and surprisingly, very few included some form of CaD VI data [huang2023sparkles, li2023otter, li2023mimic] to enable CaD support in the resulting LMM.

Due to the fundamental importance of endowing LMMs with CaD capabilities, thus getting them closer to achieving human visual intelligence in all its diversity, we propose CaD-VI - a multi-phase CaD generation approach, for progressive dense and structured CaD VI data collection (concept shown in Fig. 1), which we employ to build CaD-Inst training curriculum and associated CaD-QA benchmark comprised of CaD-related open-ended questions, both contributed in this work. In essence, the final CaD-Inst curriculum associates diverse and large-scale (349K) image pair collection with highly detailed and structured CaD summaries. CaD summaries computed for an additional set of 7.6K image pairs, are used for extracting open CaD-related QA resulting in CaD-QA .

As shown in Fig. 2, the Phase-1 of CaD-VI is a ‘cold start’ where, in the absence of LMMs with substantial CaD capabilities, we leverage image captions and an LLM to hallucinate (coarse) CaD VI data - CaD-Inst^V1 (278K), where we collect structured and detailed CaD summaries for our paired images sourced from a dense & large-scale image collection [pont2020connecting].
Training on the first phase CaD-Inst^V1 data we
arrive at CaD-LLaVA^V1 - an LMM that has strong CaD capabilities compared to a large variety of leading LMMs including the very few trained with some CaD data (see Sec. 5). Next, leveraging our CaD-LLaVA^V1 model to produce non-hallucinated, image-informed CaD data, we generate additional CaD instructions into the collection CaD-Inst^V2 (71K).
Combining CaD-Inst^V1 and CaD-Inst^V2 we form CaD-Inst and train our final CaD-LLaVA^V2 7B and 13B LMMs to achieve (1) significant (up to 17.5%) absolute improvement over a large variety of recent SOTA LMMs over a variety of 5 CaD-related existing closed-QA evaluation benchmarks (namely BISON[hu2019evaluating], SVO Probes[hendricks2021probing], NLVR2[suhr2019corpus], EQBEN[wang2023equivariant], and COLA[ray2023cola]), and (2) strong (up to over 20%) relative improvements on our contributed open-QA CaD benchmark - CaD-QA . Additionally, as CaD-Inst can be safely mixed with the LLaVA VI data [llava1_5], we show in Tab. 3 that our CaD-LLaVA^V2 models effectively avoid forgetting the general capabilities of the corresponding LLaVA LMMs.

Our contributions are as follows: (i) we contribute CaD-Inst - a large-scale visual instruction tuning dataset for enhancing CaD reasoning capabilities of LMMs; (ii) we contribute CaD-QA - an open QA evaluation benchmark for assessing CaD capabilities; (iii) we contribute and open source a CaD-VI methodology for collecting and enhancing CaD instruction tuning data; (iv) we demonstrate significant (up to 17.5%) improvements in CaD reasoning for LMMs trained using CaD-Inst as well as potential to scale CaD-Inst via self-improvement by CaD-Inst -trained models.

## 2 Related Work

Large Multimodal Models. LMMs have shown significant advancements in integrating visual and textual data, enhancing the ability of deep neural networks to understand and generate multimodal content. BLIP-2 employs a bootstrapping approach that leverages frozen image encoders and large language models through a querying transformer, achieving remarkable results on various vision-language tasks with fewer parameters compared to previous models [li2023blip]. Similarly, MiniGPT-4 [minigpt4] and LLaMA-Adapters [zhang2023llama] utilize pretrained visual and language models, with adapters aligning image tokens to language tokens, improving the efficiency and performance of multimodal understanding and generation.
In addition to these early models, the LLaVA series [llava], including LLaVA 1.5 [llava1_5] and LLaVA 1.6 [liu2024llavanext], have enhanced visual instruction tuning, enabling better handling of single-image inputs and more accurate multimodal outputs.
The InternLM XComposer 2.0 VL [zhang2023internlm], EMU2 [sun2023generative], Otter [li2023otter], SparklesChat [huang2023sparkles], and MMICL [zhao2024mmicl] extend these capabilities by incorporating multiple images as input, thereby enriching the models’ understanding and generation of text based on complex visual scenes. These models showcase the evolution from single-image to multi-image inputs, highlighting the progress in multimodal learning architectures and applications.

Visual Instruction Tuning Datasets. The success of LMMs builds on the collection of high-quality visual instruction tuning data, either constructed from existing VQA datasets [gong2023multimodal, goyal2017making, hudson2019gqa, instructblip, li2023m], curated image-text pairs [minigpt4] and LLM-generated instruction-following data with input of rich human annotations [llava, llava1_5, zhang2023llavar, zhao2023svit, li2023mimic]. However, the collection of multimodal data for learning commonalities and differences between two images is still under-explored.

Image Commonalities and Differences.
Only a few datasets contain difference-only related annotation [jhamtani2018learning, li2023mimic].
Spot-the-diff [spotthediff] collects human-annotated short change descriptions for surveillance video frames.
Our CaD-Inst^V1 data collection is partially inspired by the differences-only data collection done by [li2023mimic] as a small part of their VI strategy. However, different from [li2023mimic] we: (i) collect both differences and commonalities (compared to only differences in [li2023mimic]); (ii) we leverage a significantly more dense caption-source of [pont2020connecting] compared to [chen2015microsoft] used in
[li2023mimic]; (iii) we are structuring our differences in CaD according to 6 axes (whichever applicable on case basis) - object types, attributes, counting, actions, locations, and relative positioning, also explicitly asking the LLM to extract (from the dense captions) information along these axes, while [li2023mimic] produced unstructured difference description text; (iv) unlike [li2023mimic] we are not relying on the existence of manually collected object bounding boxes; (v) the scale of our data is approx. 4 times larger than of [li2023mimic].
Due to these differences, as evident from the direct comparison in Tab. 5, training the same model on CaD-Inst^V1 has significant performance advantages over training on CaD instructions of [li2023mimic].
To summarize, our work focuses on CaD understanding, largely neglected by the visual instruction tuning community. We propose a new CaD-VI approach for collecting synthetic visual instructions and enhancing the CaD analysis capabilities in LMMs. CaD-VI not only advances the state-of-the-art in related tasks by significant margins but also complements existing datasets [jhamtani2018learning, li2023mimic] by enabling their automatic targeted refinement, thereby improving their effectiveness for CaD tuning.

## 3 CaD-VI - Two-Phase CaD Visual Instruction Tuning

As illustrated in Fig. 2, our CaD-VI consists of two phases: in Phase-1, we employ an LLM to generate summary of CaD for image pairs (Sec. 3.1) and perform visual instruction tuning on the collected data (Sec. 3.2); in Phase-2, we leverage the Phase-1 model to generate CaD on additional image pairs and perform training with combined instruction data from both phases (Sec. 3.3).

### 3.1 Phase-1a: LLM Instruction Data Collection - CaD-Inst V1

In our first phase, we leverage an LLM to generate a summary of commonalities and differences for a pair of two images, as shown in Fig. 2 (top row). Specifically, we construct image pairs and prompt an LLM, supplying it with two image captions (one per image) and an instruction prompt asking it to summarize all the commonalities and differences according to the provided captions, contributing to our first phase CaD instruction data collection denoted as CaD-Inst^V1 .

Image Source.
We select the Localized Narratives dataset [pont2020connecting] which consists of 873K image-caption pairs with diverse samples sourced from COCO [lin2014microsoft, chen2015microsoft], Flickr30K [young2014image], ADE20K [zhou2019semantic] and Open Images [kuznetsova2020open]. The captions are generated by transcription from spoken descriptions of the image content, which are quite dense, detailed, and descriptive with an average length of 36.5 words.
To cover comprehensive visual contents and increase the diversity in terms of commonalities and differences, we collect 278K image pairs with different levels of similarity between their captions.
We compute similarity by counting the number of overlapping nouns in the corresponding captions.

Figure: Figure 3: (a) Distribution of characteristics (first two words) in the CaD summary collected in CaD-Inst^V1 ; (b) Distribution of question types (first five words) in the evaluation benchmark CaD-QA ; (c) Axis counts in CaD summaries; (d) Two-turn conversation template.
Refer to caption: x3.png

LLM Data Generation.
Inspired by LLaVA [llava] who used an LLM for single images visual instruction collection, we leverage the Mixtral 8$\times$7B open LLM [jiang2024mixtral] for generating detailed and structured summaries of commonalities and differences for pairs of images. As the LLM can only accept text as input, in Phase 1 we use image captions to represent visual content of images. This is a rather crude approximation, which is alleviated in Phase 2 of our CaD-VI approach.
To encourage the diverse and creative generation of commonalities and differences, we do not provide in-context examples of expected output in the prompt to the LLM. Furthermore, we specifically prompt the LLM to structure the commonalities and differences summaries according to the following 6 visual aspects: (i) object types; (ii) attributes; (iii) counts; (iv) actions; (v) locations; and (vi) relative positions; as illustrated in Fig. 2. We provide detailed prompts in the Supplementary. Importantly, LLM is not forced to produce all 6 aspects in every summary;
they are generated adaptively according to the available content.

Generated Data Statistics.
In CaD-Inst^V1 we collected structured summaries of CaD for 278K image pairs, with average length of 157 words (40 for commonalities and 117 for differences). The summaries are structured according to 6 axes, appearing unevenly on a case-to-case basis based on the LLM decision. We illustrate the distribution of data characteristics in Fig. 3(a), and the total observed axis counts in Fig. 3(c). More statistics and details are provided in the Supplementary.

CaD visual instructions data.
We construct a two-turn conversation for each image pair. In the first turn, we define the task of summarizing CaD by providing the encoded visual tokens of the two images and instructing the model to summarize the CaD , where the response part of the turn is the LLM-generated structured summary collected above. In this instruction, we do not provide the image captions, forcing the model to rely only on image tokens to complete the task. In the second turn, we reinforce the image-text alignment by employing a simple task of text-to-image retrieval to avoid forgetting the model’s general capabilities. We randomly sample one of the two captions and request the model to select the image (from the current pair) to which the caption belongs.
The template for the two-turn conversation is illustrated in Fig. 3(d).

### 3.2 Phase-1b: CaD Visual Instruction Tuning

Architecture. As illustrated in Fig. 2, we use our collected CaD-Inst^V1 data to perform visual instruction tuning using the open-sourced code of LLaVA-1.5 [llava1_5] LMM. The LLaVA-1.5 model consists of $\phi_{L}(\cdot;\theta_{L})$ - a pretrained Vicuna 1.5 [zheng2023judging] LLM (finetuned from LLama 2 [llama2]); $\phi_{V}(\cdot;\theta_{V})$ - a pretrained visual encoder CLIP ViT-L/14@336px [clip]; and $\phi_{M}(\cdot;\theta_{M})$ - a two-layer MLP projector converting the visual encoder tokens to post-embedding layer LLM tokens.

Given a pair of two images $x_{V_{1}}$, $x_{V_{2}}$ and the instruction $x_{I}$, the MLP projects the visual features computed by the visual encoder into embedded language tokens, *i.e*. $v_{k}=\phi_{M}(\phi_{V}(x_{V_{k}};\theta_{V});\theta_{M}),k\in\{1,2\}$. Then the projected visual features and instruction text tokens are concatenated and fed into the LLM, where the response text tokens are generated in an autoregressive manner, *i.e*.

$$ $\hat{x}^{i}_{R}=\phi_{L}([v_{1},v_{2},x_{I},\hat{x}^{<i}_{R}];\theta_{L}),$ (1) $$

where $\hat{x}^{i}_{R}$ denotes the $i$-th token in the generated response.

Training. We finetune the LLaVA-1.5 model using the LLaVA [llava] pipeline. Specifically, following LLaVA pre-training,
we finetune only the pretrained projection MLP and the (frozen) LLM with LoRA adapters [hu2021lora].
We minimize the CLM loss of the next token prediction in the responses:

$$ $\mathcal{L}_{CLM}=\sum_{i}-\log p({\hat{x}}^{i}_{R}|{V_{1}},{V_{2}},{x_{I}},{x ^{<i}_{R}})$ (2) $$

To preserve the general VL capabilities of the LMM, we merge our CaD-Inst^V1 with the finetuning data of LLaVA-1.5 (665K samples). In Tab. 3 we show that CaD-VI indeed preserves the general LMM capabilities compared to LLaVA-1.5 as evaluated on the popular SEED benchmark [li2023seed].
The Phase-1 CaD visual instruction tuning results in our cold-start model CaD-LLaVA^V1 which is an LMM that can be leveraged for annotating visual commonalities and differences.

### 3.3 Phase-2: Data Collection and Visual Instruction Tuning

Phase-2a: LMM-based CaD Instruction Collection.
While in Phase 1 we used an LLM to extract a CaD summary based on human-generated captions, for Phase 2 data collection we leverage our Phase 1 model CaD-LLaVA^V1 and additional image pairs to extract the CaD summaries informed by the images directly.
Here we select the Scene-Difference [li2023mimic] collection as an additional image source. It contains 71K pairs of similar images from COCO [lin2014microsoft] and provides annotation of unstructured difference-only summaries (see Fig. 2 bottom left for an example). We feed both the image pairs and the original annotations into our CaD-LLaVA^V1 model, and generate a structured summary of both commonalities and differences. The exact prompt is provided in the Supplementary. This leads to our phase-2 CaD instruction data - CaD-Inst^V2 . As shown in Tab. 5, our collected CaD instructions significantly improve over the utility of the original [li2023mimic] annotations. As part of our analysis in Tab. 5 and 6, and additional experiments provided in Supplementary, we also show that similarly out-of-distribution image pair collections or even unlabeled image pair collections can be effectively leveraged for our Phase-2.

Phase-2b CaD Visual Instruction Tuning
We follow the Phase-1b introduced in Sec. 3.2 for CaD visual instruction tuning. Here we finetune on a combination of LLaVA 1.5 [llava1_5] finetune data (665K), CaD-Inst^V1 data (278K) and CaD-Inst^V2 data (71K). This phase of CaD visual instruction tuning leads to the Phase 2 model, denoted as CaD-LLaVA^V2 .

## 4 CaD-QA - Benchmark of Open-Ended CaD QA

In order to evaluate the capability of LMMs on answering open-ended questions regarding commonalities and differences of a pair of two images, we construct and contribute the CaD-QA benchmark.

Data Collection.
Similar to the data collection pipeline introduced in Sec. 3.1,
we employ Visual Genome [krishna2017visual] and the detailed image captions from SVIT [zhao2023svit] as image & caption source. We collect 7.5K image pairs with 8 or more overlapping nouns in their captions. For each pair, we employ the Mixtral 8$\times$7B LLM to produce the structured CaD summaries from the captions. Next, we prompt Mixtral with both the image captions and the CaD summary, instructing it to generate a multi-turn conversation with several rounds of Q&A, providing some in-context examples of the desired layout (see Supplementary for the prompt). Finally, we randomly select one Q&A per conversation.

Benchmark Statistics.
There are 7520 QA pairs with an average answer length of 26 words. Among these, we also include 2916 questions asking about the content of only one of the two images. It requires the precise attention of the LMM on the corresponding image to correctly answer these questions. Our CaD-QA covers diverse question types as illustrated in Fig. 3(b).

LLM-assisted Evaluation. Motivated by LLMs’ ability to judge response quality consistently with human assessment [zheng2023judging], we employ the Mixtral 8$\times$7B LLM to compare the generated responses to the collected open-ended QA responses. We feed the question, correct answer, and the predicted answer into the LLM and instruct it to provide a rating between 0 and 5 for the predicted answer quality (where higher score indicates a better prediction). We provide the prompt in the Supplementary.

**Table 1: Performance on closed-ended VQA tasks with image pairs in accuracy. Here the method CaD-VI denotes our Phase-2 model CaD-LLaVA^V2 .**
| Dataset | # Instruction<br>Data | BISON | SVO | NLVR2 | EQBEN | COLA |
| --- | --- | --- | --- | --- | --- | --- |
| Random chance |  | 50% | 50% | 50% | 25% | 25% |
| SparklesChat | 6.5K | 56.70% | 43.93% | 58.00% | 19.17% | 20.00% |
| Otter | 2.8M | 40.67% | 47.33% | 52.00% | 08.33% | 08.10% |
| MMICL | 5.8M | 80.00% | 88.13% | 56.67% | 20.83% | 25.71% |
| EMU2-Chat | 1.3M | 46.00% | 47.93% | 60.00% | 07.50% | 13.33% |
| InternLM-XComposer2-VL | >600K | 80.67% | 82.07% | 66.67% | 25.00% | 32.38% |
| LLaVA 1.6 7B | <1M | 66.00% | 70.40% | 58.67% | 20.83% | 11.90% |
| LLaVA 1.6 13B | <1M | 81.33% | 82.13% | 60.00% | 17.50% | 24.76% |
| LLaVA 1.5 7B | 665K | 54.00% | 46.80% | 61.33% | 17.50% | 07.62% |
| LLaVA 1.5 13B | 665K | 59.33% | 56.27% | 66.00% | 16.67% | 12.38% |
| CaD-VI 7B | 1M | 95.33% | 92.73% | 66.67% | 39.17% | 40.95% |
| CaD-VI 13B | 1M | 96.67% | 93.00% | 69.33% | 42.50% | 43.33% |

## 5 Experiments

Evaluation Datasets
We evaluate on several VQA benchmarks of closed-ended and open-ended questions.
For closed-ended VQA on image pairs, we include BISON [hu2019evaluating] and SVO Probes [hendricks2021probing] both consisting of samples with an image pair and a text query that needs to be matched with one of the images in the pair (chance is 50%).
EQBEN [wang2023equivariant] and COLA [ray2023cola] contain samples composed of a pair of two images together with the two textual descriptions.
The goal is to correctly match images with corresponding texts (chance is 25%).
Furthermore, we evaluate on NLVR2 [suhr2019corpus] which comprises samples of a pair of two images and a reasoning sentence. The task is to assess the correctness of the reasoning and has a random chance of 50%. We also evaluate SEED-Bench Video [li2023seed] with two frames sampled from the video to explore the generalization value of our CaD tuning for video understanding. SEED-Bench Video contains three partitions from SEED-Bench and has multi-choice questions on action recognition/prediction or procedure understanding with four answer options per question.
For open-ended tasks, use the LLM-as-a-judge metric (Sec. 4). We evaluate open-ended QAs on our CaD-QA . Furthermore, we also directly evaluate the quality of LMM predicted CaD summaries for 210 image pairs in COLA with shorter summaries generated from brief captions, and for the 7.5K lengthy summaries from CaD-QA generated from detailed VG captions. More details and statistics of the datasets are provided in the Supplementary.

Implementation Details
We leverage the Mixtral 8$\times 7$B Instruct v0.1 and set the maximum token size to 750 data collection and 20 for open-ended task evaluation. For visual instruction tuning, we use the official implementation of LLaVA and tune the LLaVA 1.5 7B model with LoRA. We set the batch size to 128 and LoRA learning rate for LLM and the projector is set to $1\times 10^{-4}$ and $2\times 10^{-5}$ correspondingly. All experiments are run on 4$\times$A100 80G GPUs. More details are in Supplementary.

| # Input Frames | 1 | 2 |
| --- | --- | --- |
| SparklesChat | 21.81% | 19.09% ($\blacktriangledown$-2.72%) |
| Otter | 18.19% | 23.00% ($\blacktriangle$+4.81%) |
| EMU2-Chat | 43.43% | 41.09% ($\blacktriangledown$-2.34%) |
| InternLM-XComposer2-VL | 41.07% | 40.16% ($\blacktriangledown$-0.91%) |
| LLaVA 1.6 7B | 41.95% | 42.03% ($\blacktriangle$+0.08%) |
| LLaVA 1.6 13B | 41.85% | 41.35% ($\blacktriangledown$-0.50%) |
| LLaVA 1.5 7B | 37.43% | 36.68% ($\blacktriangledown$-0.75%) |
| LLaVA 1.5 13B | 40.12% | 38.78% ($\blacktriangledown$-1.34%) |
| CaD-VI 7B | 38.40% | 40.44% ($\blacktriangle$+2.04%) |
| CaD-VI 13B | 40.16% | 43.09% ($\blacktriangle$+2.93%) |

Table: Table 2: Performance on SEED-Bench video partitions by feeding one or two frames into the LMMs.

| Model | SEED-Image |
| --- | --- |
| LLaVA 1.5 7B | 67.34% |
| CaD-VI 7B | 67.48% |
| LLaVA 1.5 13B | 68.83% |
| CaD-VI 13B | 69.11% |

Table: Table 3: Performance on SEED-Bench image partitions for evaluation of general VL capabilities with single-image input.

Comparison to State-of-the-Art LMMs

**Table 4: Performance on CaD-QA and tasks of CaD summary prediction evaluated using LLM-as-a-judge ratings (range 0 to 5). Here the method CaD-VI denotes our Phase-2 model CaD-LLaVA^V2 .**
| Dataset | CaD-QA | VG comm. | VG diff. | COLA comm. | COLA diff. |
| --- | --- | --- | --- | --- | --- |
| SparklesChat | 3.01 | 2.41 | 3.12 | 1.52 | 1.22 |
| Otter | 2.20 | 1.88 | 1.97 | 1.37 | 0.81 |
| MMICL | 2.01 | 1.79 | 1.94 | 1.73 | 0.59 |
| EMU2-Chat | 1.20 | 1.04 | 1.08 | 1.22 | 0.41 |
| InternLM-XComposer2-VL | 2.90 | 2.08 | 2.69 | 1.72 | 1.36 |
| LLaVA 1.6 7B | 3.10 | 2.23 | 2.73 | 1.71 | 1.22 |
| LLaVA 1.6 13B | 3.19 | 2.19 | 2.69 | 1.93 | 1.01 |
| LLaVA 1.5 7B | 2.54 | 1.79 | 1.75 | 1.44 | 1.02 |
| LLaVA 1.5 13B | 2.65 | 2.16 | 2.41 | 1.57 | 1.10 |
| CaD-VI 7B | 3.29 | 2.32 | 3.85 | 2.14 | 1.25 |
| CaD-VI 13B | 3.34 | 2.58 | 3.68 | 2.13 | 1.31 |

We first compare our final model CaD-LLaVA^V2 (denoted by CaD-VI in Table) to state-of-the-art LMMs on closed-ended VQA in Table 1. SparklesChat [huang2023sparkles], Otter [li2023otter], MMICL [zhao2024mmicl], EMU2-Chat [Emu2], InternLM-Xcomposer2-VL [zhang2023internlm] all include samples with multi-image inputs in the visual instruction tuning while LLaVA 1.5 [llava1_5] and LLaVA 1.6 [liu2024llavanext] are tuned with only single image instructions. The evaluated benchmarks are challenging due to the visually very similar image pairs with subtle compositional differences where the LMMs could easily make an incorrect decision leading to performance below random chance.
Our CaD-VI 7B model already outperforms all the other baselines on the five benchmarks and our 13B finetuned model further boosts the performance.

Table 4 demonstrates the comparison to the baseline LMMs on open-ended tasks of CaD-QA and of CaD summary prediction on image pairs.
Our CaD-VI models outperform the baselines on four of the five open-ended tasks, with the exception of COLA difference summary where our 13B model achieves a rating (1.31) close to the best performing InternLM-XComposer2 model (1.36).

Furthermore, we explore whether our CaD instruction tuning improves video understanding evaluated using SEED-Bench Video in Table 2. In the evaluation setting of LLaVA, only one frame per SEED-Bench video is passed to the LMM. To explore the impact of our CaD tuning, we compare this to evaluating using two frames as input. As shown in Table 2, although multiple baseline LMMs achieve better performance in single-frame setting, our CaD-VI 13B model performs the best in the two-frame setting with a significant performance improvement of 2.93% on top of the single-frame performance. The only higher improvement is achieved by Otter, which however struggles below the 25% chance level performance. This underlines that our CaD tuning improves the temporal understanding between video frames.

Additionally, to verify that introducing multi-image CaD data into the tuning does not lead to catastrophic forgetting of general single-image input LMM capabilities, we also evaluate the SEED-Bench Image partitions and report the results in Table 3. Here we directly compare to same architecture baseline of LLaVA 1.5 fine-tuned using its single-image LLaVA mix 665K data.
Table 3 demonstrates that
our CaD tuning indeed preserves the competence in single-image understanding.

**Table 5: Ablation of phase-2 data collection from 71K image pairs in Scene-Difference (ScDiff). We use CaD-LLaVA^V1 to generate CaD on ScDiff either from scratch or by refining from the original annotation of unstructured difference-only summaries. Training settings in E and F lead to our CaD-LLaVA^V1 and CaD-LLaVA^V2 models correspondingly.**
|  | Training Data | BISON | SVO | EQBEN | COLA | CaD-QA |
| --- | --- | --- | --- | --- | --- | --- |
| A: | LLaVA mix (L) | 54.00% | 46.80% | 17.50% | 07.62% | 2.54 |
| B: | L + ScDiff orig. annot. | 92.67% | 90.07% | 22.50% | 33.81% | 2.90 |
| C: | L + ScDiff our annot. (from scratch) | 88.67% | 90.80% | 38.33% | 36.67% | 3.17 |
| D: | L + ScDiff our annot.<br>(refined from orig. annot.) | 94.67% | 91.80% | 32.50% | 34.76% | 3.17 |
| E: | L + CaD-Inst^V1 | 92.00% | 92.27% | 34.17% | 36.67% | 3.27 |
| F: | L + CaD-Inst^V1 + ScDiff our annot. (refined from orig. annot.) | 95.33% | 92.73% | 39.17% | 40.95% | 3.29 |

**Table 6: Ablation of phase-2 data collection from 66K pairs of video frames in Action Genome and GEBC (A/G). We use CaD-LLaVA^V1 to generate CaD on A/G either from scratch or with the prior information from the original frame captions.**
|  | Training Data | BISON | SVO | EQBEN | COLA | CaD-QA |
| --- | --- | --- | --- | --- | --- | --- |
| A: | LLaVA mix (L) | 54.00% | 46.80% | 17.50% | 07.62% | 2.54 |
| B: | L + A/G orig. captions only | 55.33% | 55.67% | 03.33% | 02.86% | 2.78 |
| C: | L + A/G our annot. (from scratch) | 90.00% | 88.53% | 40.83% | 42.86% | 3.21 |
| D: | L + A/G our annot. (given orig. captions) | 88.00% | 86.87% | 43.33% | 30.48% | 3.06 |

## 6 Ablations

Phase-2 Data Collection analysis. Our Phase-2 data collection introduced in Sec. 3.3 can be used to leverage image pairs from various sources for producing effective CaD instructions. We first ablate the data collection from the 71K image pairs in Scene-Difference [li2023mimic] (ScDiff) which contains annotation of unstructured difference-only summaries. As shown in Table 5, training with original annotation of difference-only summaries (row B) significantly improves on the baseline of training with LLaVA data only (row A). Then we show that using CaD-LLaVA^V1 to generate CaD instructions on ScDiff remarkably improves further, either if used from scratch (row C) or by refining from the original annotation (row D, also illustrated in Fig. 2 bottom row). Training with our re-annotation from scratch outperforms the original annotation on all datasets except for BISON. Our re-annotation by refining the original annotation leads to a more balanced performance improvement and is used as the phase-2 instruction data CaD-Inst^V2 . We combine this with our phase-1 data CaD-Inst^V1 and demonstrate the further performance boost in row F of Table 5.

In order to show the robustness of CaD data collection capability using our CaD-LLaVA^V1 model, we also explore applying our phase-2 data collection to visually similar frames from user videos in Action Genome and GEBC (A/G).
In Table 6, we first train a baseline using original frame captions only and a simple instruction task of image description (row B), which leads to a significant performance drop on EQBEN and COLA, and minimal improvement on other datasets. Then we use our CaD-LLaVA^V1 to generate CaD instructions on the frame pairs either from scratch (row C) or conditioned on the frame captions (row D). Interestingly, on most datasets CaD instructions generated by our CaD-LLaVA^V1 from scratch are found to be more effective than ones generated using original captions conditioning, likely due to lack of detail in these captions.
This once again demonstrates that our model is effective in generating CaD instructions on unlabeled data.

**Table 7: Ablation on components in the instruction data. Training settings in F and G lead to our CaD-LLaVA^V1 and CaD-LLaVA^V2 models correspondingly. Here t2i retriev. refers to the text-to-image retrieval task (see Sec. 3.1). Training settings in F and G lead to our CaD-LLaVA^V1 and CaD-LLaVA^V2 models correspondingly.**
|  | Training Data | BISON | SVO | CaD-QA | VG comm. | VG diff. |
| --- | --- | --- | --- | --- | --- | --- |
| A: | LLaVA mix (L) | 54.00% | 46.80% | 2.54 | 1.79 | 1.75 |
| B: | L + t2i retriev. | 58.00% | 51.33% | 2.47 | 1.58 | 1.46 |
| C: | L + comm. | 64.67% | 79.73% | 3.23 | 2.67 | 2.52 |
| D: | L + diff. | 55.33% | 72.13% | 3.24 | 1.97 | 2.89 |
| E: | L + comm. + diff. | 72.00% | 82.60% | 3.24 | 2.13 | 3.42 |
| F: | L + comm. + diff. + t2i retriev. | 92.00% | 92.27% | 3.27 | 2.21 | 3.69 |
| G: | F + CaD-Inst^V2 | 95.33% | 92.73% | 3.29 | 2.32 | 3.85 |

Analysis of CaD Instruction Data Components We verify the effectiveness of the components in our instruction data by ablating on the different combinations of our tuning tasks, including: (i) commonality summary (comm.); (2) difference summary (diff.); and (iii) text-to-image retrieval (t2i retriev.) in Table 7. Training solely on the t2i retrieval task (row B) leads to minimum performance improvement on BISON and SVO Probes, and performance degradation on the three benchmarks of the open-ended tasks due to lacking of any CaD learning. Training with the commonality (row C) and difference summary (row D) tasks separately lead to a significant boost on the VG comm (2.67) and VG diff (2.89) tasks correspondingly. Training with combinations of the three tasks (F) boosts the performance in comparison to the case of each single component, except for VG comm where the commonality training (row C) leads to better results on this task. Finally, combining phase-1 and phase-2 data (row G) leads to further performance boosts on most of the benchmarks.

## 7 Conclusions, Limitations, and Broader Impact

We are contributing CaD-VI - an effective, two-phase strategy for collecting Commonalities and Differences (CaD ) Visual Instruction (VI) data, resulting in the also contributed large scale CaD-Inst with 349K samples for verified improvement of CaD and related image and text comparative capabilities of LMMs. Additionally, we contribute CaD-QA - a benchmark of 7.6K open-ended QA to directly evaluate CaD capabilities between pairs of images. We extensively evaluate and validate our CaD-VI approach, showing it leads to substantial improvements in CaD abilities and related tasks. We further show how the very few existing CaD resources are complementary to our approach and can be further refined automatically using our CaD-VI . We believe that our work contributes to the important investigation and improvement of (currently somewhat missing) CaD abilities of modern LMMs and leads to exciting future work of CaD VI tuning.
Limitations
Currently, our CaD-VI only focuses on the CaD between two images, and we leave the extension of understanding CaD and group relations on three or more images to future work.
Broader Impact
Our CaD-VI , CaD-Inst , and CaD-QA significantly contribute to the understanding and improvement of CaD capabilities in LMMs, and are intended to enhance the applicability and utility of AI across various fields, from robotics to industrial applications. However, this LMM improvement could also lead to job displacement, as these models could increasingly automate complex tasks traditionally performed by humans.

## 8 Acknowledgments

We acknowledge EuroHPC Joint Undertaking for awarding us access to Karolina at IT4Innovations,
Czech Republic, Leonardo at CINECA, Italy, MeluXina at LuxProvide, Luxembourg and LUMI at
CSC, Finland.

## Appendix

In the appendix, we first introduce our dataset release (Sec. A) and the list of assets (Sec. B) used in this project.

For further insights into our approach CaD-VI , we report more statistics on our generated data (Sec. C.1), and statistics on the external evaluation datasets (Sec. C.2). We provide more implementation details (Sec. D) including the specifics of baseline methods, data generation, training and evaluation details.

As additional results, we report the error bars (Sec.E.1), analyze the Phase-2 data collection on Out-Of-Distribution data (Sec. E.2). Finally, we show qualitative results of the collected CaD summaries (Sec. E.3), and compare LMM predictions on our CaD-QA benchmark (Sec. E.4), and LMM predictions on the BISON dataset (Sec. E.5).

## Appendix A Dataset Release

HuggingFace dataset repository: [https://huggingface.co/datasets/wlin21at/CaD-Inst](https://huggingface.co/datasets/wlin21at/CaD-Inst)

We upload both the instruction-following dataset CaD-Inst and the evaluation benchmark CaD-QA to the HuggingFace dataset repository at [https://huggingface.co/datasets/wlin21at/CaD-Inst](https://huggingface.co/datasets/wlin21at/CaD-Inst) in the repository.

The dataset card with the dataset viewer, dataset description with intended use, and instructions for loading the dataset and downloading the image sources are available.

The dataset repository will be hosted in a long term with data access and necessary maintenance ensured.

## Appendix B List of Assets

Our image sources and annotations are obtained from public datasets. We release our data in accordance to the source data licenses.

Here is a list of image sources:

- •
Open Images v6 [kuznetsova2020open] ([https://storage.googleapis.com/openimages/web/download_v6.html](https://storage.googleapis.com/openimages/web/download_v6.html)): The images are under Creative Commons Attribution (CC BY) 2.0 license.
- •
COCO 2017 [chen2015microsoft, lin2014microsoft] ([https://cocodataset.org/#download](https://cocodataset.org/#download)): The images are under a Creative Commons Attribution 4.0 license.
- •
Flicker30K [young2014image] ([https://shannon.cs.illinois.edu/DenotationGraph/](https://shannon.cs.illinois.edu/DenotationGraph/)): The images are the property of SmugMug or its third party licensors and are protected by United States and international intellectual property laws. The images are provided for researchers and educators who wish to use the dataset for non-commercial research and/or educational purposes.
- •
ADE20K [zhou2019semantic] ([https://groups.csail.mit.edu/vision/datasets/ADE20K/index.html#Download](https://groups.csail.mit.edu/vision/datasets/ADE20K/index.html#Download)): The images belong to MIT CSAIL and are licensed under a Creative Common BSD-3 License.
- •
Visual Genome [krishna2017visual] ([https://homes.cs.washington.edu/~ranjay/visualgenome/api.html](https://homes.cs.washington.edu/~ranjay/visualgenome/api.html)): The images are under a Creative Commons Attribution 4.0 license.

Here is a list of image annotation sources:

- •
Localized narratives [pont2020connecting] ([https://google.github.io/localized-narratives/](https://google.github.io/localized-narratives/)): The annotations are released under a Creative Common Attribution (CC BY) 4.0 license.
- •
MIMIC-IT [li2023mimic] ([https://huggingface.co/datasets/pufanyi/MIMICIT](https://huggingface.co/datasets/pufanyi/MIMICIT)): The annotations are released under an MIT license.
- •
SVIT [zhao2023svit] ([https://huggingface.co/datasets/BAAI/SVIT](https://huggingface.co/datasets/BAAI/SVIT)): The annotations are licensed under a Creative Commons Attribution 4.0 license. It should abide by the policy of OpenAI ([https://openai.com/policies/terms-of-use](https://openai.com/policies/terms-of-use)). The use of original images and annotations from Visual Genome and MS-COCO should comply with the original licenses.

Here is a list of implementation sources or model weights:

- •
LLaVA [llava, llava1_5] ([https://github.com/haotian-liu/LLaVA](https://github.com/haotian-liu/LLaVA)): The code is released under an Apache-2.0 license. The project utilizes certain datasets and checkpoints that are subject to their respective original licenses, including but not limited to the OpenAI Terms of Use(^1^11[https://openai.com/policies/eu-terms-of-use/](https://openai.com/policies/eu-terms-of-use/)) for the dataset and the specific licenses for base language models for checkpoints trained using the dataset (*e.g*. LLaMA community license(^2^22[https://ai.meta.com/llama/license/](https://ai.meta.com/llama/license/)) for LLaMA-2 and Vicuna-v1.5).
- •
Mixtral 8$\times$7B model [jiang2024mixtral] ([https://huggingface.co/mistralai/Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)): The model is released under an Apache-2.0 license. Usage is subject to the term of use for Mistral products and services(^3^33[https://mistral.ai/terms/#terms-of-use](https://mistral.ai/terms/#terms-of-use)).

## Appendix C Dataset Statistics

### C.1 Generated Data Statistics

Figure: Figure 4: Distribution of length of CaD summaries (in terms of number of words) in (a) CaD-Inst^V1 and (b) CaD-Inst^V2
Refer to caption: x4.png

CaD-Inst^V1 and CaD-Inst^V2 .
In CaD-Inst^V1 , we collected structured summaries of CaD for 278K image pairs, with an average length of 157 words (40 for commonalities and 117 for differences). In CaD-Inst^V2 , we collected summaries of CaD for 71K images pairs used in Scene-Difference [li2023mimic], with an average length of 156 words (28 for commonalities and 128 for differences). We demonstrate the distribution of CaD summary length (number of words) in CaD-Inst^V1 (Fig. 4(a)) and in CaD-Inst^V2 (Fig. 4(b)).

Figure: Figure 5: Word clouds of CaD summaries in (a) CaD-Inst^V1 and (b) CaD-Inst^V2
Refer to caption: x5.png

In Fig. 5, we also illustrate the cloud of words covered in the CaD summaries in CaD-Inst^V1 (Fig. 5(a)) and in CaD-Inst^V2 (Fig. 5(b)).

Figure: Figure 6: Word cloud of sample-specific characteristics in CaD summaries in CaD-Inst^V1 . The distribution of these sample-specific characteristics is also shown in a Sunburst chart in Fig. 3(a)(main paper).
Refer to caption: x6.png

In the main paper, we mentioned that the collected summaries are structured according to approximate 6 axes of characteristics: object types, attributes, counting, actions, locations and relative positions. Note that the characteristics appear unevenly on a case-to-case basis based on the LLM decision on individual samples. In Fig. 3(a)(main paper), we illustrate the distribution of these sample-specific characteristics in a Sunburst chart. Here in Fig. 6, we also illustrate the cloud of words in these characteristics in CaD summaries in the Phase-1 data collection CaD-Inst^V1 .

Figure: Figure 7: Distribution of (a) number of overlapping nouns between captions in an image pair and (b) image-image similarities in the 278K image pairs in CaD-Inst^V1
Refer to caption: x7.png

In the main paper, we introduced that we collect 278K image pairs with different levels of similarity between their captions. We measure the similarity between two captions by counting the number of overlapping nouns in the corresponding captions.
Here we show the distribution of the number of overlapping nouns in Fig. 7(a). We see that we cover image pairs with different levels of caption-caption similarity. Furthermore, we use the CLIP ViT-B/32 model [clip] to compute the similarity scores between the two images in each pair and report the distribution in Fig. 7(b). We verify that image pairs of diverse similarity levels are covered in our Phase-1 data collection CaD-Inst^V1 .

Figure: Figure 8: Distribution of (a) questions (first 5 words) and (b) answers (first 3 words) in the evaluation benchmark CaD-QA .
Refer to caption: x8.png

CaD-QA . Our CaD-QA benchmark contains 7.5K open-ended questions with answers. Here we show the distribution of questions types (first 5 words) and answer types (first 3 words) in Sunburst charts in Fig. 8. There are diverse question categories covered such as Yes/No questions, What questions on scene characteristics such as objects, attributes and setting, and also requests to describe specific characteristics in details.

### C.2 Statistics of External Evaluation Datasets

We evaluate on several external VQA benchmarks of closed-ended and open-ended questions. Here we give a brief introduction on the contents and statistics.

BISON is a dataset for the binary image selection task [hu2019evaluating]. There are 150 samples in the evaluation benchmark, each sample consisting of a pair of two visually similar images and a query caption. Only one image correctly matches with the query caption. It measures the ability of the LMMs to relate fine-grained text content in the caption to visual content in the images.

SVO Probes is a benchmark designed to probe for subject, verb and object understanding in vision-language models [hendricks2021probing]. In the benchmark, each sample consists of a pair of two images and a query sentence, where only one image correctly matches with the query sentence. The negative image differs from the positive image with regard to either the subject, the verb or the object. There are 36.8K samples in the dataset. For efficient evaluation, we randomly select 1500 samples that can be divided into 3 partitions subject, verb and object where each partition has 500 samples with the image pair contradiction in either subject, verb or object.

EQBEN is a benchmark that focuses on visual minimal change between two images [wang2023equivariant]. Each sample in the benchmark consists of a pair of two images with subtle visual changes and two corresponding captions. The dataset is comprised of frames from natural video datasets such as YouCook2 [zhou2018towards], Action Genome [ji2020action] and GEBC [wang2022geb+], as well as sythetic image pairs with subtle differences generated by the photo-realistic scene generator Kubric [greff2022kubric] and the diffusion model Stable-Diffusion [rombach2022high]. We employ an EQBEN subset(^4^44[https://entuedu-my.sharepoint.com/:u:/g/personal/tan317_e_ntu_edu_sg/ETkpKSsmun1MpBw7FqfUUS8BwTX2gKkTQkDFsfOGCw-9yA?e=KGtpg0](https://entuedu-my.sharepoint.com/:u:/g/personal/tan317_e_ntu_edu_sg/ETkpKSsmun1MpBw7FqfUUS8BwTX2gKkTQkDFsfOGCw-9yA?e=KGtpg0)) which is released by the authors in [wang2023equivariant] for evaluating the performance of LMMs specifically. The subset consists of 120 samples, comprised of frame pairs from Action Genome and GEBC, image pairs with changes in attributes, count and location generated by Kubric, and image pairs with style change generated by Stable-Diffusion. For each sample, we perform the binary image selection task twice, feeding one of the descriptions for image selection at a time. The sample is considered positively answered only when both selection tasks are correctly solved.

COLA is a benchmark for evaluating the capabilities of vision-language models on representing simple compositions by combing objects with their attributes [ray2023cola]. Each sample in the benchmark consists of two images with two corresponding captions. The two images have attributes and objects that are swapped in the captions, *e.g*. large tree to the right of little short green tree, and tall green tree to the right of large tall green tree. We employ the partition of multi-object setting in the benchmark which consists of 210 image pairs and captions. Similar to evaluation on EQBEN, we perform the binary image selection task twice for each sample.

NLVR2 is a benchmark for evaluation of the visual reasoning with natural language task which aesses the ability of LMMs to predict whether a sentence is true about a pair of images [suhr2019corpus]. The task focuses on understanding of compositionalities in terms of relations, comparisons and counting. We use the subset of 150 samples provided in SparklesChat [huang2023sparkles] for a fair comparison.

SEED-Bench is an evaluation benchmark on comprehensive vision-language understanding, consisting of 19K multiple choice questions [li2023seed]. The are two major categories in the benchmark: SEED-Image with 14K samples and SEED-Video with 5K samples. SEED-Image consists of 9 dimensions: scene understanding, instance identity, instance attributes, instance location, instance counting, spatial relation, visual reasoning and text understanding. All samples contain only a single input image. SEED-Video consists of 3 dimensions: action recognition, action prediction and procedure understanding. The videos are from Something-Something-v2 [goyal2017something], EPIC-Kitchen [damen2022rescaling] and Breakfast [kuehne2014language].

## Appendix D Implementation Details

### D.1 Baselines

SparklesChat [huang2023sparkles] is finetuned from the first-stage pretrained model of MiniGPT4 [minigpt4]. The model is finetuned with their collected multi-image dialogue data. SparklesChat follows the architecture of MiniGPT4 and uses Vicuna 7B [vicuna], EVA-CLIP ViT-G/14 [fang2023eva] with a Q-Former from BLIP-2 [li2023blip]. We use the model weights and instruction templates available at [https://github.com/HYPJUDY/Sparkles](https://github.com/HYPJUDY/Sparkles).

Otter [li2023otter] is finetuned from the OpenFlamingo model [awadalla2023openflamingo] with the collected multimodal in-context instruction-response data in MIMIC-IT [li2023mimic]. We use their most recent open-sourced version Otter-Image-LLaMA7B-LA-InContext available at [https://huggingface.co/luodian/OTTER-Image-LLaMA7B-LA-InContext](https://huggingface.co/luodian/OTTER-Image-LLaMA7B-LA-InContext).

MMICL [zhao2024mmicl] is based on the InstructBLIP model [instructblip]. The model is finetuned their own collected multimodal in-context learning datast consisting of interleaved text-image inputs, inter-related multiple image inputs and multimodal in-context learning inputs. We evaluate with their model of the largest scale MMICL-InstructBLIP-T5-XXL, available at [https://huggingface.co/BleachNick/MMICL-Instructblip-T5-xxl](https://huggingface.co/BleachNick/MMICL-Instructblip-T5-xxl).

EMU2-Chat [sun2023generative] is a generative multimodal model trained on large-scale multimodal sequences. The model consists of pretrained EVA-02-CLIP-E-plus [sun2023eva] and LLaMA-33B [llama]. The model weights and inference code are available at [https://huggingface.co/BAAI/Emu2-Chat](https://huggingface.co/BAAI/Emu2-Chat).

InternLM-XComposer2-VL [zhang2023internlm] consists of CLIP ViT-L [clip] and InternLM2-7B [team2023internlm]. The model weights of the InternLM-XComposer2-VL-7B and inference code are available at [https://huggingface.co/internlm/internlm-xcomposer2-vl-7b](https://huggingface.co/internlm/internlm-xcomposer2-vl-7b).

LLaVA 1.5 [llava1_5] is an improved version from LLaVA [llava] with CLIP-ViT-L-336px [clip] as the visual backbone and Vicuna 1.5 [zheng2023judging] as the LLM. Our visual instruction tuning is performed using the open-sourced code of LLaVA 1.5. We train on the first-stage pretrained weights of LLaVA 1.5 via LoRA finetuning. We evaluate both LLaVA 1.5 7B lora and LLaVA 1.5 13B lora as baselines. The models are available at [https://huggingface.co/liuhaotian/llava-v1.5-7b-lora](https://huggingface.co/liuhaotian/llava-v1.5-7b-lora) and [https://huggingface.co/liuhaotian/llava-v1.5-13b-lora](https://huggingface.co/liuhaotian/llava-v1.5-13b-lora).

LLaVA 1.6 [liu2024llavanext] is an improved version from LLaVA 1.5 with increased input image resolution and improved mixture of instruction tuning data. The 7B and 13B versions are avaible on Huggingface at [https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b](https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b) and [https://huggingface.co/liuhaotian/llava-v1.6-vicuna-13b](https://huggingface.co/liuhaotian/llava-v1.6-vicuna-13b). However, the training code is not yet available.

### D.2 Implementation Details

Figure: Figure 9: Prompt for the task of Phase-1 LLM-based CaD summary.
Refer to caption: x9.png

Data Collection. In Phase-1, we leverage the Mixtral 8x7B Instruct v0.1 model(^5^55Huggingface source: [https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)) with 8-bit inference for data generation. We set the batch size to 16 and max new token to 750. The prompt for the task of LLM-based CaD summary is given in Fig. 9. The generation with batch 16 fits to an A100 80G GPU.

Figure: Figure 10: Prompt for the task of Phase-2 LMM-based CaD summary.
Refer to caption: x10.png

In Phase-2, we leverage the Phase-1 model CaD-LLaVA^V1 13B model to generate CaD summary on additional image pairs. The temporature, max new tokens and number of beams are set to 0, 256 and 1. The prompt for the task of LMM-based CaD summary is given in Fig. 10.

Figure: Figure 11: Prompt for the task of generating Q&A pairs based on both image captions and the CaD summary.
Refer to caption: x11.png

For collecting open-ended QAs in CaD-QA , we first use the LMM to generate the CaD summaries based on the image captions (see Fig. 9). Then we prompt the LLM with both the image captions and the CaD summary, instructing it to generate a multi-turn conversation with several rounds of Q&A. We also provide some in-context samples to demonstrate the desired layout. The prompt for the task of generating Q&A pairs based on both image captions and the CaD summary is illustrated in Fig. 11.

Training. We perform visual instruction tuning following the configuration in LLaVA 1.5. We set the batch size to 128 and train for one epoch. The learning rate for LLM with LoRA and for the projector are set to $1\times 10^{-4}$ and $2\times 10^{-5}$ correspondingly. The LoRA rank and alpha values are set to 128 and 256. The training experiments are run on 4$\times$A100 80G GPUs.

Inference. For VQA inference, the temperature, max new tokens and number of beams are set to 0, 256 and 1.

Figure: Figure 12: Prompt for the LLM-assisted evaluation.
Refer to caption: x12.png

LLM-assisted Evaluation We leverage the Mixtral 8$\times$7B model for LLM-assisted evaluation on open-ended questions. We feed the question, correct answer and the predicted answer into the LLM and instruct it to provide a rating between 0 and 5. The prompt for generating the evaluation rating is given in Fig. 12.

## Appendix E Additional Results

### E.1 Error Bars

**Table 8: Average performance of the Phase-1 model CaD-LLaVA^V1 on multiple runs of training.**
| Training Data | BISON | SVO | EQBEN | COLA | CaD-QA |
| --- | --- | --- | --- | --- | --- |
| LLaVA mix + CaD-LLaVA^V1 | 91.78% $\pm$ 1.02% | 92.33% $\pm$ 0.57% | 33.06% $\pm$ 0.96% | 34.64% $\pm$ 2.09% | 3.270 $\pm$ 0.002 |

We run the training of the Phase-1 model CaD-LLaVA^V1 multiple times and report the average performance with standard deviation in Table 8. In most evaluation cases, the standard deviation is within around 1%.

### E.2 Ablation on Phase-2 Data Collection - OOD CaD refinement

**Table 9: Ablation of phase-2 data collection from 15K pairs of video frames in Spot-the-diff (SpotDiff). We use CaD-LLaVA^V1 to generate CaD on SpotDiff by refining from the original human-annotated difference descriptions.**
|  | Training Data | BISON | SVO | Difference Spotting | CaD-QA |
| --- | --- | --- | --- | --- | --- |
| A: | LLaVA mix (L) | 54.00% | 46.80% | 49.50% | 2.54 |
| B: | L + SpotDiff orig. annot. | 51.33% | 52.27% | 60.48% | 2.51 |
| C: | L + SpotDiff our annot. (refined from orig. annot.) | 54.00% | 54.87% | 66.67% | 2.86 |

In Section 6 (main paper), we perform ablation the Phase-2 data collection. Here we further explore applying our phase-2 data collection on out-of-distribution (OOD) data of Spot-the-diff (SpotDiff) dataset. The dataset contains distant-view frame pairs with very subtle changes from video-surveillance footage, which are OOD from most LMM training data.

In Table 9, we train with SpotDiff original human-annotated difference description (row B) and with our CaD-LLaVA^V1 generated CaD summaries which is refined from the original annotation (row C).
We also evaluate on the Difference-Spotting partition on SEED-Bench 2 [li2023seed2] which contains multi-choice questions based on frame pairs from SpotDiff. In data collection and training for this experiment, we only used the 15K training image pairs from SpotDiff which are not included in the Difference-Spotting SEED partition. The results in Table 9 verify that our phase-2 data collection using CaD-LLaVA^V1 is also effective on OOD data.

### E.3 Qualitative Results of CaD Summaries

Figure: Figure 13: Examples of (a) Phase-1 LLM-collected CaD summary and (b) Phase-2 LMM-collected CaD summary
Refer to caption: x13.png

In Fig. 2 (main paper), we illustrate the pipeline of our two-phase CaD-VI together with two examples of Phase-1 LLM-collected CaD summary and Phase-2 LMM-collected CaD summary. Here in Fig. 13, we provide two additional examples. Note that in Fig. 13(a), we only pass the captions with the instruction prompt (in Fig. 9) into the LLM. In Fig. 13(b), we pass the original annotation and both images with the instruction prompt (in Fig. 10) into the Phase-1 model. In the main paper (Table 5), we demonstrate the generated CaD summary without using the original annotation also leads to effective results.

Figure: Figure 14: Examples of Q&A pairs in CaD-QA together with LMM predicted answers and the corresponding LLM evaluation rating for the prediction (Red and green texts denote incorrect and correct description).
Refer to caption: x14.png

Figure: Figure 15: Examples of Q&A pairs in CaD-QA together with LMM predicted answers and the corresponding LLM evaluation rating for the prediction (Red and green texts denote incorrect and correct description).
Refer to caption: x15.png

Figure: Figure 16: Examples of Q&A pairs in CaD-QA together with LMM predicted answers and the corresponding LLM evaluation rating for the prediction (Red and green texts denote incorrect and correct description).
Refer to caption: x16.png

### E.4 Qualitative Results on CaD-QA

In Fig. 14, Fig. 15 and Fig. 16, we show examples of Q&A pairs in our CaD-QA , together with the predicted answers from CaD-LLaVA^V2 model and the vanilla LLaVA 1.5 model. We also report the LLM ratings for the predicted answers. The vanilla LLaVA model has incorrect answers by either mistakenly combining the contents in two images (Fig. 14(b), the man is standing in front of the toilet while holding an umbrella), omitting one of the images (Fig. 15(a), Fig. 16(a)), attending to the incorrect image (Fig. 15(c)) or hallucinating non-existent contents (Fig. 16(b)). The failure demonstrates the lacking of capability of properly comparing two images. At the same time, our CaD-Inst^V2 manages to correctly differentiate between the two images, attend to the corresponding content asked in the question and draw a summary of comparison.

### E.5 Qualitative Results on BISON

Figure: Figure 17: Examples of predictions of the binary image selection task on BISON (red and green texts denote incorrect and correct predictions). We instruct the LMMs to, besides the selection, also give a reasoning for the answer.
Refer to caption: x17.png

In Fig. 17, we illustrate some examples of the binary image selection task on BISON. We instruct the LMMs to give both the selection answer and also the reasoning for the selection. Here we compare the vanilla LLaVA 1.5 and our CaD-LLaVA^V2 .
The LLaVA model, even if it captures the relevant content in some cases, has confusion differentiating the two images (Fig. 17(a)(b)). For our CaD-LLaVA^V2 , the key reasoning that leads to the correct answer is always covered in the structured difference summary.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: see weakness.
soundness: 3
strengths: this paper introduces CaD-VI, a two-phase approach for enhancing visual instruction tuning in large multimodal models (LMMs) with a specific focus on comparing commonalities and differences (CaD) between image pairs. this work addresses an underexplored area in LMMs and providing valuable insights for visual reasoning.

the paper contributes a dataset CaD-Inst containing 349K image pairs for CaD instruction tuning, and CaD-QA, a benchmark of 7.5K open-ended questions designed to evaluate CaD capabilities. These resources are likely to foster advancements in LMM training and evaluation.
summary: the focus of the paper is to improve the model's understanding capability in terms of Commonalities and Differences (CaD) over two images. to address the problem, the paper proposes a two-stage approach, in which the first stage utilizes LLM to generate the CaD information based on the caption only from the Localized Narratives dataset and then train a model. the second stage applies the trained model to generate more training data. a QA dataset is created to further diversity and enhance the data source.
weaknesses: the paper uses the open-sourced LLM or MLLM to generate the training data. what if using the GPT-4 model to generate data directly from the image, rather than from the image descriptions? one less-costly approach could be just to test, say, 5 data points, and see how the gap is, although a more comprehensive way is to replace the open-sourced component with, e.g. gpt-4 in the proposed pipeline. in this way, it may also work to directly use the image rather than the caption to generate the CaD data in the first stage?

in the proposed approach, there are two stages. the first is to generate training data and then do the training on these data. the second is to use the trained model to generate the training data. what if skipping the second phase, but generating more data in the first stage? this sounds like more reasonable and easier, maybe also more effective, way? one reason of why the second stage exists could be to save the cost, as the first stage requires the captioning and LLM to generate data, but this may not be a problem since all components are open-sourced?

in the first stage, the caption is from the dataset itself (this may also be part of the reason why stage 2 exists).  what if applying a MLLM to do the captioning? if it also works, it would make the pipeline more scalable to generate even more data. 

another question is how the scaling curve looks like with more training data in the first stage. that is, if it is 1k, how the perf is; if it is 10k, how the perf is; if it is 100k, .... the reason i'm more interested in this question is that the paper basically curates some training data and then fine-tune the model on these data to improve the performance on the target domain. this is a quite reasonable approach. the dataset scale here could be one of the main factors that impact the performance.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: - How does the model behave when scaling the quantity of commonalities and differences data separately? An investigation into scaling laws for this approach could be insightful.
- What constitutes an ideal or poor image pair for this method? Does random image pairing suffice, or are there specific factors to consider for high-quality data collection?
soundness: 3
strengths: - The concept of recognizing commonalities and differences aligns well with human visual perception and design principles, bringing an innovative perspective to visual data processing.
- The two-phase strategy is technically sound and shows promise for enhancing LMM capabilities.
- Comprehensive statistics and data sources are provided, which strengthens the transparency of the data collection process.
- Implementation details are thoroughly documented, offering useful insights for replication.
- The experimental section includes comparisons with recent models like LLaVA 1.6, 1.5, and InternLM, giving context to the model's relative performance.
summary: This paper introduces a two-phase approach for visual instruction data collection, emphasizing the significance of identifying commonalities and differences in image pairs. The authors argue that understanding these aspects is crucial to improving large multimodal model (LMM) performance.

The proposed methodology is as follows:

- **Phase 1:** Image pairs with captions are initially gathered, followed by prompting a large language model (LLM) to describe their commonalities and differences. This synthetic data is then utilized for instruction tuning, resulting in a baseline instruction-tuned LLaVA variant model.
- **Phase 2:** The model trained in Phase 1 is employed as an annotator to generate additional data, effectively expanding the instruction tuning dataset for further refinement of the LLaVA model.
weaknesses: - The paper offers limited algorithmic innovation, as much of the method depends on synthetic data generation without addressing potential pitfalls.
- The quality of generated data is uncertain, particularly regarding hallucinations or inaccuracies in synthetic outputs. More clarity on quality control measures or validation processes is needed to ensure data reliability.
- The effectiveness of the commonalities and differences data remains ambiguous. In Table 7, results are presented with varied training data sizes, complicating a fair comparison. The authors should consider experiments with controlled dataset sizes across conditions, allowing for a clearer assessment of data type impact rather than quantity alone.
- The manuscript contains several writing issues that hinder comprehension, particularly in the abstract and conclusion sections. A clearer, more concise explanation of the methodology and results would improve readability.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: N/A
soundness: 3
strengths: 1. This paper addresses an overlooked sub-problem in multi-image question answering. The proposed new dataset and benchmark may support the development of versatile multimodal large models and facilitate comprehensive evaluations. 
2. Detailed ablation experiments on the data construction method provide valuable insights, guiding future training data development for multimodal models.
summary: This paper focuses on a specialized task within multi-image question answering: comparing two images to identify their Commonalities and Differences (CaD). The authors propose a two-phase pipeline to generate CaD-formatted data and to train specific CaD-focused LMMs. To evaluate the model’s effectiveness in solving CaD problems, a targeted evaluation benchmark was proposed. Extensive experiments demonstrate that the proposed dataset and methodology significantly enhance the model's performance on CaD-related tasks.
weaknesses: 1. The paper’s scope is limited, focusing narrowly on comparing two images in terms of commonalities and differences (CaD), a sub-capability within the broader multi-image question-answering domain. Its main contribution lies in using LLMs to generate data for CaD, fine-tuning existing VLMs on this data, and performing in-domain evaluations, which constrains its general applicability.
2. The two-phase pipeline is similar to the multi-stage annotation process introduced by Kirillov et al.[1] However, unlike in [1], where extensive manual efforts and model confidence checks ensure data quality, this paper’s Phase 2 lacks additional quality assurance measures. Consequently, CaDv2 (71K) may lack reliability, functioning merely as unlabeled data for self-distillation.
3. Following from point 2, it is misleading for the authors to label CaDv1 and CaDv2 collectively as the CaD-Inst dataset. The authors should clarify the distinctions between the proposed dataset and training method to avoid confusion.

Reference:
[1] Kirillov A, Mintun E, Ravi N, et al. Segment anything. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023: 4015-4026.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: My questions are mentioned in the weakness.
soundness: 3
strengths: 1. The paper introduces a unique two-phase methodology, CaD-VI, specifically designed for training LMMs on commonality and difference (CaD) tasks. This novel approach effectively fills a gap in multimodal AI, where CaD reasoning has received limited focus.
2. By creating and releasing the CaD-Inst dataset with 349K image pairs, the authors provide a valuable resource for training and evaluating LMMs on nuanced visual reasoning tasks. This large dataset enhances model robustness in spotting both differences and similarities between images, a significant step beyond traditional difference-only datasets.
3. The model trained on CaD-Inst shows substantial performance gains, improving up to 17.5% over current state-of-the-art models on CaD-related tasks. This demonstrates the practical effectiveness of the dataset and methodology in advancing LMM performance.
summary: The paper proposes CaD-VI, a novel two-phase approach to improve large multimodal models (LMMs) in spotting and reasoning about commonalities and differences (CaD) between images. The authors introduce CaD-Inst, a dataset containing 349K image pairs with synthetic CaD instructions generated through CaD-VI. They demonstrate that training on CaD-Inst significantly enhances CaD reasoning in LMMs, advancing performance by up to 17.5% on related tasks. Additionally, the authors contribute CaD-QA, a benchmark with 7.5K questions for open-ended CaD evaluations, showcasing substantial improvements over existing state-of-the-art models​
weaknesses: 1. The paper’s experiments mainly focus on benchmark performance gains without a clear demonstration of how the model performs on real-world, uncurated image pairs. Maybe like LVLM benchmarks like MME, hallusionbench, MMMU, MMC-Benchmark and so on. In hallusionbench, MMMU and MMC-Benchmark, they also have multiple image as input.
2. The authors mentioned that they use Mixtral 8 x 7B as the evaluator. How does it compare with human evaluation and GPT4? Is it possible to use non-llm method?
3. Is It possible for the authors to compare with more recently MLLMs like internvl, Cambrain and Eagle?