# Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentation

Gautier Evennou, Antoine Chaffin, Vivien Chappelier, Ewa Kijak

► To cite this version:

Gautier Evennou, Antoine Chaffin, Vivien Chappelier, Ewa Kijak. Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentation. WACV 2025 - IEEE/CVF Winter Conference on Applications of Computer Vision, Feb 2025, Tucson (Arizona), United States. pp.1-11, 10.1109/wacv61041.2025.00143. hal-04889899

HAL Id: hal-04889899

https://hal.science/hal-04889899v1

Submitted on 15 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentation

Gautier Evennou<fnref n="1" />,², Antoine Chaffin³,*, Vivien Chappelier<fnref n="1" />, Ewa Kijak²

¹IMATAG, France, ²IRISA, CNRS, France, ³LightOn, France

`{gautier, vivien.chappelier}@imatag.com, antoine.chaffin@lighton.ai, ewa.kijak@irisa.fr`

## Abstract

The rise of the generative models quality during the past years enabled the generation of edited variations of images at an important scale. To counter the harmful effects of such technology, the Image Difference Captioning (IDC) task aims to describe the differences between two images. While this task is successfully handled for simple 3D rendered images, it struggles on real-world images. The reason is twofold: the training data-scarcity, and the difficulty to capture fine-grained differences between complex images. To address those issues, we propose in this paper a simple yet effective framework to both adapt existing image captioning models to the IDC task and augment IDC datasets. We introduce BLIP2IDC, an adaptation of BLIP2 to the IDC task at low computational cost, and show it outperforms two-streams approaches by a significant margin on real-world IDC datasets. We also propose to use synthetic augmentation to improve the performance of IDC models in an agnostic fashion. We show that our synthetic augmentation strategy provides high quality data, leading to a challenging new dataset well-suited for IDC named Syned<fnref n="1" />.

## 1. Introduction

Misinformation, frequently propagated through manipulated or out-of-context images, poses a significant challenge. Image Difference Captioning (IDC) offers a solution by generating textual descriptions that enable humans to readily ascertain whether an image has undergone semantic alterations. This paper delves into IDC [29], a recent approach that goes beyond traditional image analysis by generating detailed textual descriptions of the differences between two images. IDC finds its application across various fields, from detecting subtle changes in satellite imagery [3, 12, 21, 31] to identifying anomalies in medical im-

<figure><img src="image_2.png" /><figcaption>Figure 1. Results of state-of-the-art IDC models on: (a) a train sample of Emu Edit dataset, (b) a test sample and (c-d) zero-shot samples found in the wild. BLIP2IDC (Ours) is able to capture fine-grained differences, describe complex scenes, and generalize well to unseen data.</figcaption></figure>

agery [22] or misinformation explanation.

Although single-image captioning presents considerable challenges on its own, IDC introduces further complexities, as it involves describing the subtle differences present in a pair of similar images. Ideally, captions should disregard common objects between the images and instead emphasize the nuanced changes between them. The advancement of IDC in recent years relies significantly on the emergence of vision-language models and cross-domain learning techniques.

The second challenge of IDC is the availability of sufficiently large and diverse dataset for the task. Creating a high-quality IDC dataset is particularly difficult and resource-intensive, as it requires image pairs with detailed descriptions of their differences, encompassing various types of changes. This process can rely on expensive and time-consuming crowd-sourced labor [53], or alternatively, on the use of 3D rendered scenes [29] or on capturing temporal variations [41]. Existing IDC training/evaluation pipeline suffer from shortcomings such as suboptimal metrics, inconsistencies in ground truths, small scale of real-world datasets, or the missing categorization of modifications.

In this paper, we propose an innovative use of synthetic data and advanced multimodal architectures to address IDC limitations in terms of both data and model. We propose the application of generative models [2, 34] to produce syn-

<footnote>*Work performed while at IMATAG/IRISA.</footnote>

<footnote n="1">¹The code, weights and dataset are available at https://github.com/gautierevn/BLIP2IDC</footnote>
thetic data, providing IDC models with diverse and com-
plex image pairs. Concerning the advancements in mul-
timodal models, we discuss how the pre-training of mul-
timodal models like CLIP [6] and BLIP2 [16] is instru-
mental for IDC tasks. We explore the potential of BLIP2
in IDC, offering a more 'out-of-the-box' solution contrast-
ing sharply with the previous models reliance on complex,
multi-step training and separate processes for image encod-
ing. Despite its larger size and the associated costs for fine-
tuning, we demonstrate the feasibility and effectiveness of
LoRa [10] to adapt BLIP2 to IDC, capitalizing on its ex-
tensive model capacity while maintaining a low adaptation
cost. The main contributions are :

1. We provide a framework for synthetic augmentation to design well-suited IDC datasets based on diffusion models. We release **Syned**, our synthetic augmented version of Emu Edit [37], an image editing dataset, to provide a challenging benchmark for the IDC task. We demonstrate the significant improvement brought by this targeted synthetic augmentation.
2. We propose **BLIP2IDC**, a new state-of-the-art IDC model based on BLIP2. Extensive experiments conducted on synthetic and real benchmark datasets demonstrate the strong performances of BLIP2IDC in real-world scenarii, and good generalization to unseen content (Fig. 1).
3. We present a comprehensive evaluation of several leading models on a new real IDC dataset based on Emu Edit outputs, contributing to a clearer understanding of the current model capabilities in IDC.

## 2. Related Work

**Image Captioning and Multimodal Models.** Image dif-
ference captioning is closely related to Image Captioning
(IC) and Visual Question Answering (VQA). Image Cap-
tioning [16, 17, 48] aims to describe the content of an image
with fine-grained captions. IC models are trained on web-
scale datasets [8, 20, 52] to harness as much visual knowl-
edge as possible. IC models attempt to connect text and
images with various solutions: syntactic trees from image
features [26], RNN [7] decoding of CNN features with [50]
or without attention [1, 47], pre-training task using object-
based anchors [18], Seq2Seq learning with unified vocabu-
lary for all the linguistic and visual tokens [48] or leveraging
additional embeddings to make a bridge between vision to-
kens and text tokens [16]. Multimodal models specialized
in VQA use cross-modal attention to tune the visual fea-
tures extraction to the text prompt, enabling interaction and
focusing the model on information relevant to the question.
While highly efficient [23, 49], those models are prone to
inconsistency due to their prompt dependency [54].

**Image Difference Captioning (IDC).** IDC focuses on
telling apart the discrepancies between two near-identic im-

ages. The differences changing the meaning of the image
(semantic changes) are the ones the model should focus
upon. On the contrary, editorial changes (non-semantic)
such as compression or rescale should be ignored. Prior
work struggles with two main hardships: how to repre-
sent the difference features and how to gather this specific
kind of data. Deep learning based architectures, such as
CNN, CLIP-based encoding and RNN are widely used in
this topic to learn the features [3, 9, 12, 21, 31, 38, 51]. Pre-
vious work extract features for each image independently,
thus discarding the images correlation in the pixel space.
After this extraction step, they fuse embeddings using con-
catenation [29, 31, 44, 51] before feeding them to whether a
difference encoder for change representation, or directly to
a decoder to generate the text descriptions. The difference
encoder and decoder are transformer-based [46] or RNN-
based [31]. Another axis of work is to define pretraining
tasks to find a suitable representation space for the differ-
ences [44]. Recently, the use of a multimodal model such as
CLIP [32] enables for a better representation, thus achiev-
ing state-of-the-art performances. Yet, current adaptation to
IDC [6] struggles with exploiting all its pretraining. More-
over, because high-quality data is hard to find, most meth-
ods are trained and evaluated on 3D rendered datasets, that
do not accurately represent real-world performance.

**Image Editing.** The recent progress in generative mod-
els [5, 34] enables realistic generation of images. This fur-
ther propell work [2, 25, 30] on image editing, where se-
mantic modifications are performed to an image using a
textual prompt, while keeping the main subject of an im-
age. Recent works [37, 53] focused on using already exist-
ing real-world text-image datasets such as MSCOCO [20]
to perform editions on those images, by building on modifi-
cations of the associated caption. The main hurdle in these
methods is the fully supervised generation and verification
process, where each generation is reviewed by operators
and several level of filters are used to keep the best genera-
tions. While this process produces a high quality dataset,
it is time-consuming and costly. We show in this paper
that such setup nevertheless allows the automatic creation of
IDC datasets: a couple is created using the original image
and its modified version, and the prompt used to perform
the edit is used as the target modification description.

3. BLIP2IDC

BLIP2 [16] is a multimodal model that introduces the Querying Transformer (QFormer) as the main block to connect text and image representations. It relies on a two-stage pretraining: a vision-language representation learning stage with a frozen image encoder, and a vision-to-language generative learning stage with a frozen LLM. The QFormer is composed of two transformers modules. The first one attends to the image through cross attention
<figure><img src="image_3.png" /><figcaption>Figure 2. Classic IDC pipeline. The feature extraction step is performed before training. The training step leverages the features from a frozen image encoder by concatenating them and then learns the difference representation from these concatenated features. The output is then fed to the text decoder to perform the generation of the difference caption in an autoregressive manner.</figcaption></figure>

between learned embeddings, called queries, and vision
encoder outputs. The second module interacts with the
ground truth text and with the queries embeddings to ensure
vision-language alignment. BLIP2 is pre-trained to jointly
optimize three pre-training objectives that are Image-Text
Contrastive Learning, Image-Text Matching, and Image-
grounded Text Generation. As a contribution we show that,
taking advantage of its pretraining for Image Captioning,
BLIP2 can be adapted to IDC at a low computational cost
and without changing its architecture.

**3.1. Adaptation**

We argue that the classic IDC two-streams encoding
scheme featured in Fig. 2 results in a suboptimal compar-
ison of images at the features level. For standard image
captioning, BLIP2 takes one image as input and uses a ViT
model [4, 46] as a frozen image encoder. In the context of
IDC, feeding BLIP2 jointly with the two images to be com-
pared enables the attention layers of the visual encoder and
QFormer to focus early on the differences between the 2
images, and to encode the two images in relation to each
other rather than separately. Our BLIP2 adaptation pipeline
is shown in Fig. 3. We give a single image as input, re-
sulting from the vertical concatenation of two images, al-
lowing the model to attend to the differences early, while
avoiding any modification of the architecture of the model.
Unlike BLIP2, which only trains the QFormer, the ViT and
the LLM should be fine-tuned to compensate for this modi-
fication of the input domain.

Although images are stretched by concatenation, it does
not undermine the performances. We conjecture this be-
haviour is due to the way BLIP2 was pretrained, where im-
ages are randomly cropped then stretched to a fixed square
size, without padding. Thus, the BLIP2 model learned to be
robust to various stretching operations.

**3.2. Efficient fine-tuning**

**Modules fine-tuning.** Although BLIP2IDC does not re-
quire to change the architecture of the model, the model
should be fine-tuned for the IDC task to adapt to the new
task and type of data. Unlike the original training of BLIP2
which only fine-tunes the QFormer, all the components,

<figure><img src="image_4.png" /><figcaption>Figure 3. BLIP2IDC end-to-end pipeline. We first resize and con-
catenate the image inputs before feeding them to the BLIP2 archi-
tecture fine-tuned using LoRA.</figcaption></figure>

including the ViT, QFormer and image-grounded text de-
coder, are finetuned to get the best IDC performance (see
Fig. 6 in the result section).

**Low Rank Adaptation (LoRA).** We use Low Rank Adaptation (LoRA), to reduce training resources while maintaining performance. By fine-tuning just 0.1% of all parameters—specifically, the attention modules’ *Q*, *K*, *V* layers—we achieve top performance with minimal resource use. This approach allows BLIP2IDC to adapt to IDC tasks, balancing high performance with resource efficiency.

**3.3. Advantages over existing IDC models**

The latest leading models for IDC are CLIP4IDC [6] and
SCORER [44]. CLIP4IDC utilizes a two-step training ap-
proach, first to adapt image-pairs visual representations for
captioning and then to generate captions based on visual
differences encoding, through contrastive and cross-entropy
losses respectively. SCORER follows the traditional IDC
framework as depicted in Fig. 2, enhancing performance
with innovative modules for view-shift invariance and cap-
tion informativeness, and is presented as showing particular
strength on the CLEVR-DC dataset.

These models are characterized by complex training pro-
cedures with several steps and separate image encoding pro-
cesses. In contrast, our model BLIP2IDC, simply concate-
nates images for joint encoding employing early attention
mechanisms while learning in only one step. This approach
proves crucial for IDC, as evidenced by our experimental
results in Sec. 5. BLIP2IDC also benefits from knowledge
gained through the large-scale unsupervised pre-training of
BLIP2 for image captioning, making it compelling given
the challenges associated with training an IDC model from
scratch on fewer data.

**4. Datasets**

The structure of IDC datasets is a triplet (*I*<sub>ref</sub>, *I*<sub>modified</sub>, *GT*) which respectively represents the original image *I*<sub>ref</sub>, the modified image *I*<sub>modified</sub>, and the ground-truth set of reference descriptions of the differences *GT*. We compare our approach to recent IDC models on standard IDC datasets: CLEVR-Change [29], CLEVR-DC [13], Spot-The-Diff [31] and Image Editing Request [41]. These datasets are distinguished by various properties, summarized in Tab. 1: the number of image
pairs $(I_{ref}, I_{modified})$, the origin of the $I_{ref}$ images, which may be real images or generated by 3D models, the number of reference captions (ground-truths *GT*) per triplet, the way in which $I_{modified}$ and *GT* are obtained, the number and type of transformations performed, and whether some manual intervention was required to create the dataset. This may happen at the level of triplet filtering or the creation of ground-truth legends.

These properties can limit the creation of these datasets, which can be costly, as well as their usefulness, for example when the diversity of transformation types is reduced. We discuss the limitations of these datasets hereafter, as well as the issues posed by the evaluation of the IDC task. To handle those limitations, we propose a method based on text-guided image editing methods for creating IDC datasets without manual intervention, which can then be adapted to different use cases. Furthermore, instruction-based image editing capabilities are evaluated on some benchmarks (IE datasets), like MagicBrush [53] or Emu Edit test set generations (EE) [37]. Those datasets are also composed of triplets $(I_{ref}, T_{instruction}, I_{edited})$, with $T_{instruction}$ the modification that maps $I_{ref}$ to $I_{edited}$ and $I_{edited}$ the edited image. We thus propose to adapt those datasets for the IDC task.

## 4.1. Existing datasets

**CLEVR-Change.** Introduced by Robust Change Captioning [29], the goal of this dataset is to evaluate primary abilities for visual understanding such as being able to tell the shape, the color, the material and the position of an object and if one or none of those properties changed. This dataset enables to assess if an IDC model understands spatial relationships and if it is robust to non-semantic changes. Five types of scene changes are defined. Editing instruction and ground-thruths captions are automatically constructed following a template. However, this dataset was created with a 3D engine, and all the elements are 3D scenes far from the real world setting, leading to a model that doesn't transfer very well to real-life examples.

**CLEVR-DC.** CLEVR-DC [13] is a derivative of CLEVR-Change with extreme shift in viewpoints. It uses the same types of modifications as CLEVR-Change. The goal of this dataset is to evaluate the view-invariance of the IDC model but it shares the limitations of the original dataset.

**Spot-The-Diff (STD).** Unlike CLEVR-Change, STD [12] uses a real world setting for the pairs of images. To cope with the issue of producing modified images, it uses temporal evolution. Thus, a pair of images is made of one image taken at a given time and another of the same place taken at a later time, with a fixed point of view. Most of the images are set in a parking lot, heavily restricting the diversity of ground-truth as most changes in a parking are add/remove people/car. Although heavily biased, this dataset enables one to test IDC models on real-world images.

**Image Editing Request (IER).** Born from crawling Reddit and Zhopped, the IER dataset [41] relies on instructions from online users in Photoshop specialized forums. The input image and editing instruction were posted, and people sent their modified images on Reddit. Crawling those results of the manual edition led to a really high quality and diverse dataset but with a really small scale. Each image pair of the test set has three GT captions, written by three different annotators. IER provides a wide range of semantic modifications, as for example "add a sailor hat to ducks", "replace the background by a spaceship", or "replaces brooms with lightsabers".

**MagicBrush.** This dataset is the first large-scale, manually-annotated instruction guided image editing dataset [53] covering various global and local editing scenarios. MagicBrush comprises 10K ($I_{ref}$, $T_{instruction}$, $I_{edited}$) triplets, which is sufficient to fine-tune large-scale image editing models with the use of LoRA. For this dataset, modifications are not grouped into categories like in CLEVR-Change, making it hard to explain performance and target specific weaknesses.

**Emu Edit (EE).** This dataset [37] from Meta is composed of pairs of original images from MSCOCO [20] and modified images generated by the Emu Edit model, a state-of-the-art generation model, which takes an image and an editing instruction as inputs. This dataset also contains fine-grained information about the type of modifications generated, grouped in eight different categories: Add, Text, Color, Background, Local Style, Global and Remove. Paired with the real-world setting of original images which enables for greater usability, these properties make it a great candidate for a challenging IDC benchmark.

## 4.2. IDC task issues

IDC models are trained on triplets ($I_{ref}$, $I_{modified}$, *GT*) and evaluated on a test set of pairs ($I_{ref}$, $I_{modified}$) by comparing the generated caption with the reference captions *GT*, using various automatic evaluation metrics, such as BLEU, ROUGE, METEOR and CIDEr [15, 19, 27, 28]. Some aspects of the real-world datasets weaken the quality of the evaluation.

**Suboptimal metrics.** The references-based metrics more accurately evaluate the quality of generated sentences if multiple references are provided. Five references is usually the minimum used. However, none of the available real-world datasets provides a consistent and sufficient number of references, making evaluations less accurate.

**Lack of consistency in ground truths.** In STD and IER, the humanly-annotated ground-truth captions do not consistently refer to the same difference or set of differences, especially when several modifications are made to the image. Some captions describe a single modification, others several. An IDC model trained to describe only the main
<table><caption>Table 1. IDC and IE datasets. Setting indicates the origin of reference images as 3D scenes or from the real world. Editing method refers to the way modified images are created or collected. GT is the number of ground-truths in the validation and test set.</caption><thead><tr><th>Dataset</th><th># image pairs</th><th>setting</th><th>editing method</th><th>curated</th><th># GT</th></tr></thead><tbody><tr><td>CLEVR-Change [29]</td><td>79,606</td><td>3D Scenes</td><td>3D engine</td><td>✓</td><td>5.0</td></tr><tr><td>CLEVR-DC [13]</td><td>48,000</td><td>3D Scenes</td><td>3D engine</td><td>✓</td><td>5.0</td></tr><tr><td>STD [31]</td><td>13,192</td><td>Real-world</td><td>Temporal change</td><td></td><td>1.86</td></tr><tr><td>IER [41]</td><td>3,939</td><td>Real-world</td><td>Human Editing</td><td>✓</td><td>3.0</td></tr><tr><td>MagicBrush [53]</td><td>10,308</td><td>Real-world</td><td>DallE-2 [33] platform</td><td>✓</td><td>1.0</td></tr><tr><td>EE [37]</td><td>5,612</td><td>Real-world</td><td>Automatic generation</td><td>✓</td><td>1.0</td></tr><tr><td>Syned(ours)</td><td>28,720</td><td>Real-world</td><td>Automatic generation</td><td></td><td>5.0</td></tr></tbody></table>

difference will struggle with this type of ground truth, and it is the case for all existing IDC models. Either all the differences should be mentioned in each ground-truth, or just one, consistently, to avoid conflicting objectives.

**Tiny real-world datasets.** As real-world datasets are very time-consuming to make, they are small in size, with at most 10,000 text-image pairs. As a result, they can lead to models specialized in specific subsets of possible modifications, as for STD. On the other side, when modifications are very diverse, as for IER, a type of modification can occur only few times. Metrics then heavily depend on the data splits, according to whether a modification appears only in the training or in the test set. This is all the more possible if the modifications are not grouped into classes allowing stratified sampling.

**Modifications categorization.** Grouping changes into categories is useful for stratified sampling, but also for analyzing results, as it makes it easier to identify difficult changes and the weaknesses of IDC models.

### 4.3. Synthetic augmentation

To cope with the current shortcomings of IDC datasets mentioned above, we propose a pipeline to generate synthetic training samples based on real-world original images. The global pipeline is illustrated in Fig. 4. The idea is to leverage the emerging prompt-based image editing models to produce modified images based on a set of original images and editing instructions belonging to a defined range of modifications. Additional GT captions are generated by a Large Language Model (LLM) as variations of the editing instructions, ensuring consistent GTs.

Such a pipeline can be used either to augment an existing IDC dataset, or to create a new dataset whose images and modifications are tailored to a particular downstream task with new specific types of data or modifications.

To evaluate this pipeline, we applied it to augment the EE dataset [37], as this dataset provides real-world edited images alongside editing instructions well-suited for an image editing model and with a clear distinction between each class of modifications. As image editing model, we choose

<figure><img src="image_5.png" /><figcaption>Figure 4. Synthetic dataset creation pipeline leveraging a prompt-based image editing and large language models.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 5. Comparison of train set samples from EE, generated by the Emu Edit model, and Syned, generated by a fine-tuned InstructPix2Pix model.</figcaption></figure>

the InstructPix2Pix [2] model fine-tuned on the MagicBrush [53] dataset, the state-of-the-art solution for executing real-world edits. The train set is augmented by generating eight new modified images per original ones from the EE dataset and their associated editing instructions, without any manual curation.

To ensure meaningful evaluation, we leverage the Llama-2-7b-chat-hf model to generate 4 additional variations of each editing instructions from the EE test set (cf. supplementary). Each of the 2,022 samples in the test set is therefore described by 5 GT captions: the original instruction and the 4 reference captions provided by the LLM. Note that the train set images (see Fig. 5) were not modified by the same editing model as in the test set.

The resulting Syned dataset comprises:

* a train set of 28,720 pairs of ($I_{ref}$, $I_{modified}$): 8 variations of each of the 3,590 original images from MSCOCO, generated by InstructPix2Pix. The different modifications are categorized in 8 classes.

* a test set of 2,022 samples with 5 references each,
where the modified images were generated by the Emu
Edit model.

Although the modifications can be defined as anything
according to the downstream needs, they have to be aligned
with the image editing model's capabilities. For instance,
editing models currently fall short in adding text to images.
Such instructions introduce bad samples in the training set.
However, as image editing models improve, the proposed
pipeline will allow to include more reliable edits to the train-
ing set of IDC models, although it already allows for cus-
tomized generation of more challenging IDC datasets.

5. Experiments

**5.1. Evaluation protocol**

**Models and datasets.** We compare BLIP2IDC to other state-of-the-art IDC models with the results reported in their respective paper on the four standard IDC datasets: CLEVR-Change, CLEVR-DC, STD, and IER. We also introduce the EE dataset to the IDC task, on which we also train and evaluate CLIP4IDC [6] and SCORER [44] for the sake of comparison.

Finetuning.

BLIP2IDC is fine-tuned as described in Sec. 3.2, with
a LoRA of rank 8. For our own adaptation of BLIP2, we
use the decoder-only BLIP2's LLM version, with opt2.7B
and vit-base-patch16-224 as the ViT model. We use LoRA
implementation from the peft [24] python library to train
BLIP2 and to store adaptation weights in a lightweight 20
MB file. For CLIP4IDC and SCORER, we either used the
hyper-parameters from their respective online repository, or
hyperparameters communicated by the authors (see supple-
mentary).

**Standard data augmentation.** To increase the robustness of the model, we use RandomGaussianBlur and JPEG Compression as non-disruptive data augmentation scheme. Augmentations should indeed leave the ground-truth semantic content untouched. Transformations based on crop, color jitter, horizontal/vertical flip, brightness or contrast can add meaningful differences and thus change the expected difference caption.

**Metrics.** Based on previous work [3, 9, 12, 21, 31, 38, 51], CIDEr (C) is the main metric used to evaluate the generated difference captions. This n-gram-based approach ensures that the evaluated text captures the most relevant aspects of the image as agreed upon by multiple human annotators. In a semantic-dependent context, the CIDEr metric rewards descriptions that accurately reflect the consensus understanding of the image content. BLEU-4 (B), ROUGE-L (R) and METEOR (M) are also used as secondary metrics to assess the sentences quality.

5.2. BLIP2IDC evaluation

We evaluate the performance of BLIP2IDC against other
methods on the four existing IDC datasets. All results
except CLIP4IDC on CLEVR-DC and BLIP2IDC are re-
ported from previous work.

**Results on 3D scenes.** Results per type of semantic changes on CLEVR-Change are given in Tab. 2. BLIP2IDC ranks first on almost all types of semantic changes, with a 10% increase in the captioning performance on the most difficult 'Move' change, and a significant improvement over previous state-of-the-art method on the CIDEr metric. On CLEVR-DC, that introduces extreme viewpoint changes, we observe that CLIP4IDC achieves the best results, followed by BLIP2IDC (Tab. 3). Both models are better than SCORER although the latter was specially designed to ensure extreme-view-shift invariancy.

**Results on real-world images.** According to Table 3, the BLIP2IDC method ranks first on the CIDEr metric by a significant 9.2% margin with respect to state-of-the-art on STD. The IER dataset is made of photoshopped images in a real world setting. Thus, the impressive improvement over previous state-of-the-art is expected since BLIP2 was pre-trained on 129M images in a real-world setting [14, 20, 36]. Our results shows that our adaptation outperforms state-of-the-art models, even those based on CLIP which was trained at large scale too. Overall, those results show that efficiently adapting a powerful pretrained model yields state-of-the-art performance, whereas most existing models fall behind due to the lack of data during training.

**Ablation study.** The performance variations across different BLIP2IDC configurations, depending on the fine-tuned modules, are depicted in Fig. 6. This analysis highlights the significant impact of fine-tuning both the ViT and LLM modules. Given the mismatch between the ViT and LLM modules for both the input and output domains, the QFormer not surprisingly stands out as the less critical component for fine-tuning, as it already had undergone extensive training during the BLIP2 pretraining phase.

**5.3. Automatically generated dataset**

We introduced in Sec. 4.3 Syned, a new dataset designed for the IDC task, built upon the EE dataset. New data are generated by a different instruction-based image editing model, thus broadening the scope and diversity of the new modified images wrt the original EE dataset, as depicted in Fig. 5. We evaluate the usefulness of Syned as data augmentation for different IDC models in Tab. 4. We show an improvement with the proposed synthetic augmentation (EE + Syned) for both CLIP4IDC and BLIP2IDC. Note that the available implementation of SCORER enables to reach at most a CIDEr score of 23.2 which is not significant in this setting and can be achieved only by reward hacking.
<table><caption>Table 2. CIDEr scores on CLEVR-Change for the 5 categories of changes.</caption><thead><tr><th>Method</th><th>Color</th><th>Texture</th><th>Move</th><th>Add</th><th>Drop</th></tr></thead><tbody><tr><td>DUDA [29] (ICCV 2019)</td><td>120.4</td><td>86.7</td><td>56.4</td><td>108.2</td><td>103.4</td></tr><tr><td>VAM+ [39] (ECCV 2020)</td><td>122.1</td><td>98.7</td><td>82.0</td><td>126.3</td><td>115.8</td></tr><tr><td>IFDC [11](TMM 2022)</td><td>133.2</td><td>99.1</td><td>82.1</td><td>128.2</td><td>118.5</td></tr><tr><td>DUDA+ [9] (CVPR 2021)</td><td>120.8</td><td>89.9</td><td>62.1</td><td>119.8</td><td>123.4</td></tr><tr><td>BiDiff [40] (IJIS 2022)</td><td>115.9</td><td>106.8</td><td>71.8</td><td>121.3</td><td>124.9</td></tr><tr><td>IDC-PCL [51] (AAAI 2022)</td><td>131.2</td><td>101.1</td><td>81.7</td><td>133.3</td><td>116.5</td></tr><tr><td>CLIP4IDC [6] (ACL-IJCNLP 2022)</td><td>149.1</td><td>135.3</td><td>91.0</td><td>132.4</td><td><strong>135.5</strong></td></tr><tr><td>SCORER [44] (ICCV 2023)</td><td>143.2</td><td>135.2</td><td>91.6</td><td>129.4</td><td>132.6</td></tr><tr><td>SCORER+CBR [44](ICCV 2023)</td><td>146.2</td><td>133.7</td><td>92.2</td><td>131.1</td><td>133.9</td></tr><tr><td>BLIP2IDC (Ours)</td><td><strong>152.31</strong></td><td><strong>137.0</strong></td><td><strong>99.8</strong></td><td><strong>135.6</strong></td><td>133.1</td></tr></tbody></table>

<table><caption>Table 3. Results on CLEVR-DC, STD, and IER, reported from their corresponding papers, except †: reported from our own experiments.</caption><thead><tr><th rowspan="2">Method</th><th colspan="3">CLEVR-DC</th><th colspan="3">STD</th><th colspan="4">IER</th></tr><tr><th>B</th><th>M</th><th>C</th><th>B</th><th>M</th><th>C</th><th>B</th><th>M</th><th>R</th><th>C</th></tr></thead><tbody><tr><td>Dyn rel-att [41]</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>6.7</td><td>12.8</td><td>37.5</td><td>26.4</td></tr><tr><td>M-VAM [39]</td><td>40.9</td><td>27.1</td><td>60.1</td><td>10.1</td><td>12.4</td><td>38.1</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>M-VAM+RAF [39]</td><td>-</td><td>-</td><td>-</td><td>11.1</td><td>12.9</td><td>43.5</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>VA [13]</td><td>44.5</td><td>29.2</td><td>70.0</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>VACC [13]</td><td>45.0</td><td>29.3</td><td>71.7</td><td>9.7</td><td>12.6</td><td>41.5</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>DUDA [29]</td><td>40.3</td><td>27.1</td><td>56.7</td><td>8.1</td><td>12.5</td><td>34.5</td><td>6.5</td><td>12.4</td><td>37.3</td><td>22.8</td></tr><tr><td>NCT [43]</td><td>47.5</td><td>32.5</td><td>76.9</td><td>-</td><td>-</td><td>-</td><td>8.1</td><td>15.0</td><td>38.8</td><td>34.2</td></tr><tr><td>SRDRL+AVS [45]</td><td>-</td><td>-</td><td>-</td><td>-</td><td>13.0</td><td>35.3</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>IFDC [11]</td><td>-</td><td>-</td><td>-</td><td>8.7</td><td>11.7</td><td>37.0</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>BDLSCR [40]</td><td>-</td><td>-</td><td>-</td><td>6.6</td><td>10.6</td><td>42.2</td><td>6.9</td><td>14.6</td><td>38.5</td><td>27.7</td></tr><tr><td>VARD-Trans [42]</td><td>48.3</td><td>32.4</td><td>77.6</td><td>-</td><td>12.5</td><td>30.3</td><td>10.0</td><td>14.8</td><td>39.0</td><td>35.7</td></tr><tr><td>MCCFormers-D [31]</td><td>46.9</td><td>31.7</td><td>71.6</td><td>10.0</td><td>12.4</td><td>43.1</td><td>8.3</td><td>14.3</td><td>39.2</td><td>30.2</td></tr><tr><td>SCORER [44]</td><td>49.5</td><td><strong>33.4</strong></td><td>82.4</td><td>9.4</td><td>13.8</td><td>38.5</td><td>9.6</td><td>14.6</td><td>39.5</td><td>31.0</td></tr><tr><td>SCORER+CBR [44]</td><td>49.4</td><td><strong>33.4</strong></td><td>83.7</td><td>10.2</td><td>12.2</td><td>38.9</td><td>10.0</td><td>15.0</td><td>39.6</td><td>33.4</td></tr><tr><td>CLIP4IDC [6]</td><td><strong>54.7</strong><sup>†</sup></td><td><strong>33.0</strong><sup>†</sup></td><td><strong>89.9</strong><sup>†</sup></td><td><strong>11.6</strong></td><td><strong>14.2</strong></td><td><strong>47.4</strong></td><td>8.2</td><td>14.6</td><td><strong>40.4</strong></td><td>32.2</td></tr><tr><td>BLIP2IDC (Ours)<sup>†</sup></td><td>49.3</td><td><strong>33.0</strong></td><td><strong>88.5</strong></td><td>11.4</td><td>13.5</td><td><strong>51.4</strong></td><td><strong>17.4</strong></td><td><strong>20.1</strong></td><td><strong>48.5</strong></td><td><strong>74.1</strong></td></tr></tbody></table>

<table><caption>Table 4. CIDEr scores on Emu Edit test set. We show consistent improvements with synthetic augmentation (EE + Syned) on all models.</caption><thead><tr><th>Models</th><th>EE</th><th>Syned+ EE</th><th>Improvement (%)</th></tr></thead><tbody><tr><td>SCORER</td><td>21.1</td><td>23.2</td><td>-</td></tr><tr><td>CLIP4IDC</td><td>32.4</td><td>35.9</td><td>10.8</td></tr><tr><td>BLIP2IDC</td><td>100.0</td><td>106.8</td><td>6.8</td></tr></tbody></table>

In Table 5, we study BLIP2IDC behaviour with respect to its training data, being either Syned, EE or both on different modification categories. Combining datasets is generally the best configuration overall, except for the text (T) category. We assume this is due to the poor capacity of most Image Editing models to generate text in images, which hampers the performance of the synthetic augmentation. The worst categories in terms of CIDEr score performance

are the Style and Global categories, which are very subjective changes. More details are available in the supplementary material.

## 5.4. Qualitative analysis

We present in Fig. 1 a comparative analysis of state-of-the-art IDC models outputs across different scenarios: in-distribution image-pairs from the train and test set of EE, and out-of-distribution images from the web, to assess the zero-shot capabilities of the models.

We observe SCORER's limitations in handling real-world images in all cases. It only succeed in identifying a text modification in the train sample, and confuses actions like adding or removing. CLIP4IDC performs well on samples from EE but struggles with zero-shot generalization. Specifically, it misidentifies the "table tennis set" due to its absence in the training data, confusing it with a similar but incorrect object. Additionnaly, it fails to understand
<table><caption>Table 5. CIDEr Performance Metrics by Change Category and Dataset Version. This table lists the CIDEr score of each following categories: Add (A), Text (T), Background (B), Color (C), Style (S), Global (G), Remove (R), Local (L) and Overall (O) which is the CIDEr computed over all the samples of the test set.</caption><thead><tr><th>Dataset</th><th>A</th><th>T</th><th>B</th><th>C</th><th>S</th><th>G</th><th>R</th><th>L</th><th>O</th></tr></thead><tbody><tr><td>Syned</td><td><u>102.45</u></td><td>147.17</td><td>111.78</td><td>151.87</td><td>12.77</td><td><b>31.56</b></td><td>72.27</td><td>61.00</td><td>94.83</td></tr><tr><td>EE</td><td>101.01</td><td><b>147.69</b></td><td><b>119.48</b></td><td><u>155.76</u></td><td><u>30.53</u></td><td>29.46</td><td><u>93.39</u></td><td><u>76.24</u></td><td><u>100.83</u></td></tr><tr><td>Syned+ EE</td><td><b>107.38</b></td><td><u>147.42</u></td><td><u>112.37</u></td><td><b>164.40</b></td><td><b>37.07</b></td><td><u>30.43</u></td><td><b>111.16</b></td><td><b>77.81</b></td><td><b>106.83</b></td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>(a) IER score</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(b) EE Score</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(c) Syned Score</figcaption></figure>

<caption>Figure 6. Ablation Study of BLIP2IDC's CIDEr according to fine-tuned modules on different datasets.</caption>

the context in which the object is added. BLIP2IDC, on the other hand, excels on data in-distribution and demonstrates superior capability in generalization through effective use of cross-attention. It accurately recognizes and describes the entire scenes in Fig. 1(c-d), including complex additions like a table tennis set, by leveraging its pretraining. This allows BLIP2IDC to not only identify the presence of fire in Fig. 1(d) but also specify the object affected, showcasing

its ability to connect semantic changes with their context.

## 5.5. Limitations

BLIP2 is pre-trained on large image-text datasets sourced from the web, such as LAION [35]. As BLIP2IDC is a finetuned version of BLIP2, it inherits the biases observed on the BLIP2 model due to its pretraining data. Our synthetic augmentation pipeline also relies on several synthetic generation models and thus inherits their limitations, such as ethnics misrepresentation or adversarial vulnerabilities. The LLM used for generating the ground truth variations will hallucinate, even with careful prompting.

Furthermore, Image Editing models, while improving, are still not able to perform every type of modifications. Careless use of these models can deteriorate the dataset quality if the editing instructions are not well aligned with the model capacity. We finally emphasize the in-domain limitation: IDC datasets can hardly be exhaustive in their ability to handle all types of change, due to the multiplicity of modification interpretations and descriptions. For a given domain, a more suitable approach is to curate a limited number of well defined changes and to build upon them, allowing for better control.

## 6. Conclusion

In this paper, we propose a novel framework that adapt existing image captioning model to the IDC task in order to benefit from their extensive pre-training. Image captioning pretraining data are indeed easier to obtain than the one for IDC. This allows to effectively leverage the global world knowledge from large-scale unlabeled dataset and transfer it to IDC using a smaller amount of data. As a demonstration, we adapt BLIP2 to the IDC task. By innovatively encoding image differences at the pixel level rather than relying on the traditional dual-stream scheme, we ensure a more informative and direct approach to understanding image variations. BLIP2IDC allows to outperform existing methods on standard benchmarks. Additionally, we introduce a synthetic augmentation strategy that not only addresses critical challenges of data scarcity and the need for robust, real-world applicable architectures but also sets a new benchmark in IDC performance. This paves the way to the application of IDC to more diverse data and types of edits.
## References

[1] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In *3rd International Conference on Learning Representations, ICLR 2015*, 2015. 2

[2] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In *2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 18392–18402, 2023. 1, 2, 5

[3] Seloua Chouaf, Genc Hoxha, Youcef Smara, and Farid Melgani. Captioning changes in bi-temporal remote sensing images. In *2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS*, 2021. 1, 2, 6

[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In *International Conference on Learning Representations*, 2021. 3

[5] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In *Advances in Neural Information Processing Systems*, 2014. 2

[6] Zixin Guo, Tzu-Jui Wang, and Jorma Laaksonen. CLIP4IDC: CLIP for image difference captioning. In *Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)*, 2022. 2, 3, 6, 7

[7] Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. *Neural Computation*, pages 1735–1780, 1997. 2

[8] Micah Hodosh, Peter Young, and Julia Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. *Journal of Artificial Intelligence Research*, 2013. 2

[9] Mehrdad Hosseinzadeh and Yang Wang. Image change captioning by learning from an auxiliary task. In *2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 2724–2733, 2021. 2, 6, 7

[10] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In *International Conference on Learning Representations*, 2022. 2

[11] Qingbao Huang, Yu Liang, Jielong Wei, Yi Cai, Hanyu Liang, Ho-fung Leung, and Qing Li. Image difference captioning with instance-level fine-grained feature representation. *IEEE Transactions on Multimedia*, 2022. 7

[12] Harsh Jhamtani and Taylor Berg-Kirkpatrick. Learning to describe differences between pairs of similar images. In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, 2018. 1, 2, 4, 6

[13] Hoeseong Kim, Jongseok Kim, Hyungseok Lee, Hyunsung Park, and Gunhee Kim. Viewpoint-agnostic change captioning with cycle consistency. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, 2021. 3, 4, 5, 7

[14] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Yoshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. *International Journal of Computer Vision*, 123:32–73, 2017. 6

[15] Alon Lavie and Abhaya Agarwal. Meteor: an automatic metric for mt evaluation with high levels of correlation with human judgments. In *Proceedings of the Second Workshop on Statistical Machine Translation*. Association for Computational Linguistics, 2007. 4

[16] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In *International Conference on Machine Learning*, 2023. 2

[17] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In *International Conference on Machine Learning*, 2022. 2

[18] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. In *ECCV*, 2020. 2

[19] Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In *Text Summarization Branches Out*. Association for Computational Linguistics, 2004. 4

[20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In *Computer Vision – ECCV 2014*, 2014. 2, 4, 6

[21] Chenyang Liu, Jiajun Yang, Zipeng Qi, Zhengxia Zou, and Zhenwei Shi. Progressive scale-aware network for remote sensing image change captioning. In *IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium*, 2023. 1, 2, 6

[22] Fenglin Liu, Changchang Yin, Xian Wu, Shen Ge, Ping Zhang, and Xu Sun. Contrastive attention for automatic chest X-ray report generation. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021*, 2021. 1

[23] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023. 2

[24] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft, 2022. 6

[25] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In *International Conference on Learning Representations*, 2021. 2

[26] Margaret Mitchell, Jesse Dodge, Amit Goyal, Kota Yamaguchi, Karl Stratos, Xufeng Han, Alyssa Mensch, Alex
Berg, Tamara Berg, and Hal Daumé III. Midge: Generating image descriptions from computer vision detections. In *Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics*, 2012. 2

[27] Gabriel Oliveira dos Santos, Esther Luna Colombini, and Sandra Avila. CIDEr-R: Robust consensus-based image description evaluation. In *Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021)*, 2021. 4

[28] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In *Proceedings of the 40th Annual Meeting on Association for Computational Linguistics*, ACL '02, 2002. 4

[29] Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust change captioning. *2019 IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 4623–4632, 2019. 1, 2, 3, 4, 5, 7

[30] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yi-jun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. *ACM SIGGRAPH 2023 Conference Proceedings*, 2023. 2

[31] Yue Qiu, Shintaro Yamamoto, Kodai Nakashima, Ryota Suzuki, Kenji Iwata, Hirokatsu Kataoka, and Yutaka Satoh. Describing and localizing multiple changes with transformers. *2021 IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 1951–1960, 2021. 1, 2, 3, 5, 6, 7

[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In *International Conference on Machine Learning*, 2021. 2

[33] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. *ArXiv, 2022*. 5

[34] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. *2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2021. 1, 2

[35] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In *Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2022. 8

[36] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, 2018. 6

[37] Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman.

Emu edit: Precise image editing via recognition and generation tasks. *ArXiv, abs/2311.10089*, 2023. 2, 4, 5

[38] Xiangxi Shi, Xu Yang, Jiuxiang Gu, Shafiq R. Joty, and Jian-fei Cai. Finding it at another side: A viewpoint-adapted matching encoder for change captioning. In *European Conference on Computer Vision*, 2020. 2, 6

[39] Xiangxi Shi, Xu Yang, Jiuxiang Gu, Shafiq R. Joty, and Jian-fei Cai. Finding it at another side: A viewpoint-adapted matching encoder for change captioning. In *European Conference on Computer Vision*, 2020. 7

[40] Yaoqi Sun, Liang Li, Tingting Yao, Tongyv Lu, Bolun Zheng, Chenggang Yan, Hua Zhang, Yongjun Bao, Guiguang Ding, and Gregory Slabaugh. Bidirectional difference locating and semantic consistency reasoning for change captioning. *International Journal of Intelligent Systems*, 2022. 7

[41] Hao Tan, Franck Dernoncourt, Zhe Lin, Trung Bui, and Mohit Bansal. Expressing visual relationships via language. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*. Association for Computational Linguistics, 2019. 1, 3, 4, 5, 7

[42] Yunbin Tu, Liang Li, Li Su, Junping Du, Ke Lu, and Qingming Huang. Viewpoint-adaptive representation disentanglement network for change captioning. *IEEE Transactions on Image Processing*, 32:2620–2635, 2023. 7

[43] Yunbin Tu, Liang Li, Li Su, Ke Lu, and Qingming Huang. Neighborhood contrastive transformer for change captioning. *IEEE Transactions on Multimedia*, 25:9518–9529, 2023. 7

[44] Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, Chenggang Yan, and Qingming Huang. Self-supervised cross-view representation reconstruction for change captioning. In *2023 IEEE/CVF International Conference on Computer Vision (ICCV)*, 2023. 2, 3, 6, 7

[45] Yunbin Tu, Tingting Yao, Liang Li, Jiedong Lou, Shengxiang Gao, Zhengtao Yu, and Chenggang Yan. Semantic relation-aware difference representation learning for change captioning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, *Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021*. Association for Computational Linguistics, 2021. 7

[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In *Advances in Neural Information Processing Systems*, 2017. 2, 3

[47] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In *2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2015. 2

[48] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In *Proceedings of the 39th International Conference on Machine Learning*, 2022. 2

[49] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan
Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2024. 2

[50] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In *Proceedings of the 32nd International Conference on Machine Learning*, Proceedings of Machine Learning Research. PMLR, 2015. 2

[51] Linli Yao, Weiying Wang, and Qin Jin. Image difference captioning with pre-training and contrastive learning. In *AAAI Conference on Artificial Intelligence*, 2022. 2, 6, 7

[52] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. *Transactions of the Association for Computational Linguistics*, 2014. 2

[53] Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. In *Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS)*, 2023. 1, 2, 4, 5

[54] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. *International Journal of Computer Vision*, 2022. 2