Differential-Perceptive and Retrieval-Augmented MLLM for Change Captioning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 08:01:55

Abstract: Change captioning involves describing the subtle changes between a pair of similar images. Although existing efforts have achieved compelling success, they overlook the potential of multimodal large language models (MLLMs) in tackling this challenging task. In this work, we aim to empower MLLMs with the capability to perceive subtle differences between paired images and enhance their performance in generating change captions. Specifically, we present a diFferentIal-perceptive aNd rEtRieval-augmented MLLM (FINER-MLLM) tailored for this task. In particular, FINER-MLLM leverages LoRA fine-tuned MLLM's image encoder to extract image patch features, enabling the capture of detailed image information. Subsequently, within MLLM's feature extraction, typically Q-Former, FINER-MLLM incorporates dual constraints: the intra-image feature independence constraint and the inter-image feature alignment constraint. These constraints ensure that the features can comprehensively extract subtle visual information within each image and that corresponding features across images align effectively.Last, we introduced the retrieval augmentation to first retrieve the relevant corpus to facilitate the MLLM's decoder \textit{i.e.}, LLM, in generating accurate change captions. Extensive experiments on three benchmark datasets, \textit{i.e.}, CLEVR-Change, Spot-the-Diff, and Image-Editing-Request, demonstrate the superiority of our proposed method.

Main paper: 
 related work: 
 Our work is closely in line with the studies on change captioning, Multimodal Large Language Models and multimodal retrievalaugmented methods.Change captioning is a vital task in vision-language understanding and generation [5, 14, 20, 29, 34, 40] . Jhamtani et al. [14] initially created this task from image captioning and change detection, and they proposed a model that aligns clusters of pixel-wise difference between image pairs to output sentences. Park et al. [25] introduced a synthetic dataset including samples with viewpoint distractors. They proposed a dual attention mechanism to localize changed object, along with a dynamic speaker that adaptively attends to visual features at each decoding step. Shi et al. [28] proposed a viewpoint-adapted match encoder to capture both changed and unchanged regions by computing feature-level similarity. They introduced a reinforcement attention method to improve the generation process. Sun et al. [29] proposed a bidirectional difference localization method to locate changes by encoding visual features from two directions. To further enhance the performance of change captioning, existing efforts have been combined with additional tasks [1, 10, 12, 38] . Mehrdad et al. [12] introduced a auxiliary composed image retrieval task to improve the training of change caption network, where these two tasks reinforce each other in a unified framework. Another typical example is IDC-PCL [38] , which designs three self-supervised task and contrastive learning strategies to align visual differences and text descriptions at a fine-grained level. Following that, Guo et al. [10] adopted CLIP and the imagetext retrieval task for adaption training, which aligns differences in image pairs based on textual descriptions. Different from the previous work, we aim to advance the change captioning task with MLLMs by empowering MLLMs with the ability to perceive subtle differences between paired images and generate accurate change captioning.Large language models (LLMs) have demonstrated remarkable abilities in various tasks, but they are primarily designed for textbased inputs. Recently, several multimodal large language models (MLLMs) are proposed to incorporate visual information with LLMs, enabling a fusion of visual and textual modalities for improved performance and understanding. Flamingo [2] introduced a perceiver resampler to combine a frozen visual encoder with LLMs, aligning vision and textual modalities. BLIP2 [17] bridged the modality gap with a lightweight Querying Transformer that transforms visual information to textual query embeddings. LLaVa [22] and MiniGPT-4 [42] both employed the fine-tuning approach with well-collected data, resulting in a significant boost in performance. InstructBLIP [7] further improved the instruction-following ability of models by fine-tuning on multi-modal instruction following datasets. However, existing MLLMs lack the capability to inspect differences among multiple images due to the inability to adequately capture subtle visual details and misalignment of features. Our model is devised to refine the differential-perception capabilities of MLLMs for change captioning.Retrieval-augmented methods mainly consist of conditioning generation on additional information that is retrieved from an external datastore [16] , which has proven to be effective plugins for both LLMs and MLLMs [11, 27] . Lewis et al. [16] initially proposed retrieval augmented generation for pre-trained language model, which effectively improved its performance on downstream knowledge-intensive tasks. Ramos et al. [27] proposed an image captioning model augmented with retrieval, which is lightweighttraining and effective. Specially, they integrated vision and language model by cross attention layer and utilized retrieved captions as prompt to instruct language model, leading to an excellent performance. Zhang et al. [41] proposed a open-book video captioning model that retrieved relevant sentences as reference for better generation. While retrieval augmentation has gained traction in other tasks, it remains largely unexplored in the context of image change captioning [27, 39] . Inspired by retrieval augmentation, our method is to employ a retrieval-then-generation fashion in the context of image change captioning, which utilized a emphasize-mechanism on retrieved corpus to generate accurate sentences. 
methdology: 
 In this section, we first formulate the research problem and then detail our proposed method.In this work, we aim to address the challenging change captioning task, which can be formally defined as given a pair of images consisting of a before-change image and an after-change image, the goal is to generate the description of modification between this image pair. Suppose we have a set of triplets, denoted asD = (𝐼 𝑏𝑒 𝑓 , 𝐼 𝑎𝑓 𝑡 ,𝑇 𝑑𝑖 𝑓 𝑓 ) 𝑖 𝑁 𝑖=1, where 𝐼 𝑏𝑒 𝑓 , 𝐼 𝑎𝑓 𝑡 , and 𝑇 𝑑𝑖 𝑓 𝑓 refer to the before-change image, the after-change image, and the caption of difference, respectively. 𝑁 is the total number of triplets. Following the traditional pipeline of retrieval-augmented methods, we first utilize a retriever to obtain semantically relevant corpus 𝑇 𝑟𝑒 𝑓 as reference knowledge, then we train the generator to produce caption 𝑇 𝑑𝑖 𝑓 𝑓 conditioned on retrieval descriptions. The overall process is formally given as follows,EQUATIONwhere 𝑦 is the tokens of 𝑇 𝑑𝑖 𝑓 𝑓 , R represents the function of retriever, and G denotes the function of generator. Following [7] , we adopt EVA-ViT-g/14 [8] as the image encoder, which has shown remarkable success in various vision tasks. We aim to get aligned visual queries through attention constraints.To fulfill this, we need to first obtain the fine-grained features of before-change image and after-change image, and then regularize the attention of visual queries to get aligned queries for accurate difference inference. However, the off-the-shelf image encoder limits its ability to provide differentiated fine-grained features for similar images, thus inhibiting the learning of cross-modal visual queries in Q-Former. Toward this end, we propose to add parallel lightweight low-rank adapters (LoRA) in the frozen Vision Transformer, to extract image patch features of similar images.The core of LoRA is to efficiently fine-tune pre-trained large models by introducing low-rank matrices into the model's parameters. ••• Q ••• ••• Softmax ••• Retrieval Tokens Embedding K W softmax Emphasizing MechanismChangeCaption Encoder "the large blue matte object became yellow" "the" "large" "blue" "matte" "object"…. Here we integrate parallel low-rank matrices into specific linear layers within Transformer blocks to fine-tune the pre-trained image encoder for change captioning. For clarity, here we use the fine-tuning process of a specific linear layer with weight matrix W ∈ R 𝑈 ×𝐾 as an example to introduce the LoRA fine-tuning technique. Specifically, LoRA applies a low-rank decomposition by projection-down and projection-up weights, which is formulated as follows,EQUATIONwhere x ∈ R 𝐾 and h ∈ R 𝑈 refer to the input and output features, respectively. A ∈ R 𝑅×𝐾 is the projection-down weight, B ∈ R 𝑈 ×𝑅 is the projection-up weight, 𝑅 is the rank size (𝑅 ≪ min (𝑈 , 𝐾)), and 𝛼 > 0 is a scaling hyper-parameter. In this way, the fullparameter fine-tuning of optimizing the matrix ΔW is replaced by optimizing the decomposed low-rank matrices B and A, which can greatly reduce the number of parameters to be optimized. Generally, there are four weight matrices in the self-attention module (W 𝑞 , W 𝑘 , W 𝑣 , W 𝑜 ) and two in the feed-forward layer within the Transformer block. Here, we insert LoRA adapters to optimize W 𝑞 and W 𝑣 in the self-attention module, as well as two weight matrices in the feed-forward layer. Given an input image pair comprising a before-change image and an after-change image, we can extract their image patch features using the LoRA image encoder, respectively, which is as follows:EQUATIONwhere𝐼 𝑛 𝑖 ∈ R 𝐷 (𝑛 = 1, • • • , 𝑁) denotes the local image patch feature, 𝐷 is feature dimension, 𝑁 is the number of image patches.After obtaining the image patch features of the before-change image and the after-change image, we feed them into Q-Former to extract query features, respectively. The Q-Former in InstructBLIP [7] utilizes a fixed number of learnable query vectors to interact with image path features through cross-attention layers for aggregating visual information. Additionally, the query vectors interact with each other through self-attention layers, enabling the sharing of visual information among them. When applied in the change captioning task, the query vectors individually interact with every single image of pairs, which presents two issues: (1) The query features extracted from a single image may contain redundancy, which can not obtain comprehensive visual information of the image and hence lose certain details. However, these visual details generally include changed visual information that needs to be focused on change captioning. (2) The query vectors extract features from two images independently, where the features between image pairs in the same location may focus on different visual regions. This discrepancy makes LLMs fail to accurately perceive visual changes based on the extracted features. Hence, it is necessary to eliminate redundancy across different visual queries and align visual regions of the corresponding query of image pairs. Toward this end, we design dual constraints as shown in Figure 4 : (1) intra-image feature independence constraint and (2) inter-image feature alignment constraint. The former intra-image feature independence constraint ensures that queries attend to different image patch features within each image pair, enhancing the extraction of comprehensive visual details. The latter inter-image feature alignment constraint ensures that corresponding queries attend to the same visual regions across image pairs, facilitating interaction between the images to identify similarities and differences.Specifically, we denote the query features extracted from the image patch features asQ 𝑖 = {q 𝑖 1 , q 𝑖 2 , • • • q 𝑖 𝐾 }, where 𝑖 ∈ {𝑏𝑒 𝑓 , 𝑎𝑓 𝑡 } denotes the before-change image and after-change image, respectively. 𝐾 is the number of query features and q 𝑖 𝑘 ∈ R 𝐷 denotes the 𝑘-th query feature. When obtaining q 𝑖 𝑘 , the cross-attention weight distribution2 for aggregating image patch features is denoted asp 𝑖 𝑗 = [𝑝 1 , 𝑝 2 , • • • , 𝑝 𝑁 ] ∈ R 𝑁, where 𝑝 𝑛 is a cross-attention weight scalar for aggregating the 𝑛-th image path feature. The crossattention weight distributions of the 𝐾 query features are denoted asA 𝑖 = {p 𝑖 1 , p 𝑖 2 , • • • , p 𝑖 𝐾 }.Intra-image Feature Independence Constraint. We employ the commonly used InfoNCE [36] loss as the orthogonal loss for extracting comprehensive visual details. Essentially, the InfoNCE loss aims to enforce the identical distributions close to each other and dissimilar distributions away from each other, namely, minimizing the similarity between different distributions. Therefore, we apply this loss to the cross-attention distributions of each query feature, respectively. Mathematically, we have the following loss function for optimizing the extraction of semantic features as follows,EQUATION)where 𝐵 is the batch size, s(•, •) denotes cosine similarity function, and 𝜏 is the temperature factor.Inter-image Feature Alignment Constraint. As mentioned above, each image has 𝐾 cross-attention distributions through Q-Former, denoted as A 𝑏𝑒 𝑓 and A 𝑎𝑓 𝑡 . These distributions manifest the visual regions targeted by the query features. To encourage the query features to concentrate on identical regions, we design the inter-image feature alignment constraint with the Kullback-Leibler (KL) Divergence between A 𝑏𝑒 𝑓 and A 𝑎𝑓 𝑡 as follows,EQUATIONEQUATIONNote that we also integrate LoRA adapters into the Q-Former model instead of conducting full-parameter fine-tuning. In particular, we insert low-rank matrices into the query projection and value projection layer in the self-attention and cross-attention layer of Q-Former.The extracted query features are concatenated and fed into the fully connected layer to map the dimensions the same as the input of LLM. Then, the query features are fed into LLM to generate change captions. As aforementioned above, many relevant sentences in the dataset contain potential descriptions of similar change scenes or the same object-level phrases. Therefore, we resort to retrieval-augmented caption generation to utilize the potentially useful information in the relevant sentences. We initially introduce a visual-to-text retriever to retrieve semantically relevant sentences from training data based on input image pairs. To effectively utilize the retrieved sentences, we design the retrieval-augmented caption generation module at the word level. Specifically, we employ a novel emphasize mechanism on individual words within the retrieved sentences, which aims to copy the useful words in sentences by prioritizing words directly.Caption Decoder. The LLM predicts the right word based on semantic features of image pairs [𝑄 𝑏𝑒 𝑓 ; 𝑄 𝑎𝑓 𝑡 ] at each time step by computing the current hidden state ℎ 𝑐 𝑡 in parallel. The current hidden state of LLM ℎ 𝑐 𝑡 depends on the previous hidden states and input semantic features of images. Then LLM generates the probability distribution of the fixed vocabulary p 𝑣 through a fully connected layer at each time step:EQUATIONwhere p 𝑣 ∈ R 1×𝑉 , W 𝑣 and b 𝑣 are frozen parameters of last output layer in LLM. Generation Augmentation. We first utilize retriever to get the top-𝑘 most relevant sentences based given image pair. Each retrieved sentence consists of a set of words with their embeddings. We aggregate these sentences into one embedding set as retrieved vocabulary by removing repeated words, denoted as 𝑦 = {𝑦 1 , • • • , 𝑦 𝐿 }. At each decoding step 𝑡, the module acts on retrieved vocabulary, uses the current hidden state ℎ 𝑐 𝑡 as the query to attend retrieved words, and produces the word probability distribution over retrieved vocabulary p 𝑟 ∈ R 1×𝐿 of the corresponding sentence, which is formulated as follows:EQUATIONwhere 𝐴𝑡𝑡 (•) is the additive attention module with parameters of 𝜃 𝑟 , p 𝑟 denotes the relevant degree between words in retrieved vocabulary and current hidden state, and c 𝑟 represents the context of the retrieved words that is the weighted summation of 𝑦 by p 𝑟 . Then, we apply a emphasize mechanism to highlight relevant words over p 𝑣 at each time step. Formally, we prioritize the words by adding weight to the corresponding value in the probability distribution over fixed vocabulary:EQUATIONwhere p 𝑟 ∈ R 1×𝐿 is extended to the same size of fixed vocabulary. In this manner, only the prediction of relevant words is improved, without altering remaining values in the prediction. The goal of generation is to minimize the negative log-likelihood of each target word 𝑦 𝑡 :EQUATIONOptimization. Integrating the three key modules, we parameterize the final objective function for our model as follows,EQUATIONwhere Θ denotes the to-be-learned parameters in FINER-MLLM, 𝜆 and 𝜂 are the trade-off hyper-parameters. 
experiment: 
 In this section, we first introduce the experimental settings and then provide the experiment results and corresponding analyses4.1.1 Datasets. We chose three public datasets to evaluate our model, including a synthetic dataset CLEVR-Change [25] , as well as two real-world domain datasets Spot-the-Diff [14] and Image-Editing-Request [30] .CLEVR-Change [14] is a synthetic dataset that includes largescale scene changes of geometry objects and distractors from viewpoint variations. It contains 67, 660, 3, 976 and 7, 970 image pairs for training, validation and testing, respectively. Following [25] we utilized the official dataset split protocol to validate our method.Spot-the-Diff [25] is real-world dataset collected from videosurveillance footage, consisting of 13, 192 image pairs captured from a fixed viewpoint at different timestamps. Following previous work [25] , we evaluated our model in a single caption setting and split image pairs into 80% for training, 10% for validation and 10% for testing.Image-Editing-Request [30] consists of 3, 939 image pairs from real life with 5, 695 editing instructions. Following [33] , we used the official dataset split protocol with 3, 061 image pairs for training, 383 image pairs for validation, and 495 image pairs for testing.Our model is based on Instruct-BLIP [7] , which includes a ViT backbone and a Q-Former to bridge a frozen decoder-based large language model. Following BLIP2 [17] , we adopted EVA-ViT-g/14 [8] and Vicuna-7B [6] as the image encoder and large language model, respectively. We trained model by AdamW optimizer [23] with weight decay of 0.1. We applied a cosine learning rate decay with a fixed peak learning rate and a fixed number of linear warm-up steps. For CLEVR-Change, the peak learning rate and warm-up steps are set to 5𝑒 -5 and 4, 000, respectively. For Spot-the-Diff, the peak learning rate and warm-up steps are set to 3𝑒 -5 and 1, 000, respectively. For Image-Editing-Request, the peak learning rate and warmup steps are set to 3𝑒 -5 and 250, respectively. The batch size is set to 16 for all datasets. The temperature factor 𝜏 in Eqn. ( 4) is set to 0.1. Regarding the trade-off hyper-parameters, we set 𝜆 = 0.25, 𝜂 = 1.0 for CLEVR-Change, 𝜆 = 0.75, 𝜂 = 1.0 for Spot-the-Diff and 𝜆 = 0.75, 𝜂 = 0.1 for Image-Editing-Request. The rank size of low-rank matrices in image encoder is set to 16 for CLEVR-Change, 4 for Spot-the-Diff, and 4 for Image-Editing-Request, respectively. The rank size of low-rank matrices in Q-Former is set to 4 for all datasets. The retrieval augmented generation requires a effective retriever for visual-to-text retrieval. We introduced the pre-trained retrieval model from CLIP4IDC, which is trained to retrieve corresponding change captions based on image pairs. We utilized this model to retrieve top-200 semantically relevant sentences from texts in train set. We utilized top-5 sentences for augmented generation for all datasets. We re-implemented the retrieval model on Image-Editing-Request based on CLIP4IDC, due to the fact that they do not provided pre-trained weights. All the experiments are implemented by PyTorch over a server equipped with 8 A100 GPUs, and the random seeds are fixed for reproducibility. 4.1.3 Evaluation. We used the standard evaluation protocol for each dataset and reported the same metrics for a fair comparison. For change captioning, we reported five standard metrics including BLEU-4(B) [24] , METEOR(M) [4] , ROUGE-L(R) [19] , CIDEr-D(C) [37] and SPICE(S) [3] . Following [10] , we compute the metrics by Microsoft COCO evaluation server.Table 1 : Performance comparison on CLEVR-Change with respect to all metrics. The best results over baselines are underlined, while the overall best results are in boldface.Method B M R C SDUDA [25] (CVPR '19) 47.3 33.9 -112.3 24.5 VAM [28] (ECCV '20) 50.3 37.0 69.7 114.9 30.5 VAM+ [28] (ECCV '20) 51. 3 To validate the effectiveness of our method in the context of change captioning, we compared FINER-MLLM with the following baselines: DDLA [14] , DUDA [25] , VAM [28] , IFDC [13] , VACC [15] , SRDRL [35] , 𝑅 3 Net [34] , BiDiff [29] , IDC-PCL [38] , NCT [32] , VARD [31] , SCORER [33] , and CLIP4IDC [10] . Among them, DDLA [14] initially created this task. It aligns clusters of differential pixels between image pairs to generate change captions. DUDA [25] is the first work to address viewpoint variations, which utilizes dual attention to locate visual changes and a dynamic speaker to generate change captions. CLIP4IDC [10] adopts CLIP [26] as the image feature extraction backbone and designs a two-stage approach including pre-training and fine-tuning to capture differences in image pairs. Tables 1, 2 and 3 summarize the performance comparison on the three datasets. The missing results of some methods are because they do not report their results on the respective datasets. From these tables, we obtained the following observations: 1) FINER-MLLM consistently outperforms all baseline methods across the real-world domain datasets, including Spot-the-Diff and Image-Editing-Request. This confirms the advantage of our model of employing differential-perceptive and retrieval-augmented MLLM in the context of change captioning. 2) We also noticed that our FINER-MLLM exhibits inferior performance on the CLEVR-Change dataset. This may be attributed to two reasons. Firstly, the images in CLEVR-Change are all synthesized geometries and are not from real-world data. The pre-training datasets of MLLM may contain only a few synthesized geometrical images, leading to the poor generalization ability of MLLM on this dataset. Secondly, the CLEVR-Change dataset includes changes in viewpoint with a high percentage (around 60%), such as slight rotations or scaling of the image, whereas the other two real-world scene datasets hardly contain changes in viewpoint. Through our designed differentialperceptive feature modeling, we reduced the model's capability to perceive changes in viewpoint, resulting in poor performance in the synthesized dataset. While we want to claim that in realworld scenarios, change captioning often does not involve changes in viewpoint, such as surveillance cameras and medical image lesion comparisons. This further highlights the potential application values of our FINER-MLLM in real-world scenarios. 3) We also observed that in the Image-Editing-Request dataset, the quality of the retrieved relevant corpus is not as good as the other two datasets. Accordingly, we reported the results of FINER-MLLM †, which does not utilize the retrieved relevant corpus in text generation. It can be seen from Table 3 that FINER-MLLM † yields slightly better performance compared to FINER-MLLM. This may be due to the diverse nature of change captions in this dataset, where the retrieved relevant corpus cannot provide assistance and may even confuse the LLM.To verify the importance of each component in our model, we compared FINER-MLLM with its following derivatives.• w/o_LoRA-ViT: To explore the necessity of fine-tuning the image encoder, we froze the parameters of EVA-ViT-g/14. • w/o_RAG: To study the effect of the retrieval-augmented approach, we discard the retrieved relevant corpus in the caption generation process. Table 4 shows the ablation results of FINER-MLLM on Spotthe-Diff. From this table, we gained the following observations: 1) w/o_LoRA-ViT is inferior to FINER-MLLM, which proves that fine-tuning the image encoder is beneficial for providing image patch features with more detailed visual information for the latter differential-perceptive feature extraction. 2) w/o_Dual-constraints yields the worst performance, while both w/o_Independence and w/o_Alignment perform inferior to FINER-MLLM. This highlights the importance of differential-perceptive capabilities for change captioning in real-world scenarios. The designed dual constraints ensure both intra-image feature independence and inter-image feature alignment. 3) FINER-MLLM outperforms w/o_RAG, which confirms the benefit of utilizing retrieved relevant corpus to effectively enhance the change caption generation process.To delve deeper into retrieval-augmented caption generation, we examined the impact of the quality of the retrieved relevant corpus on FINER-MLLM's performance, as depicted in Figure 5 . The x-axis represents the utilized relevant corpus, with ranking results of the retrieved relevant corpus sorted from high to low similarity. "Rank-0" denotes not using the retrieved corpus, while "Random" indicates randomly selecting 5 sentences as assistance during the generation process. It can be observed that caption generation performance significantly improves with the top 1 -5 retrieved sentences, indicating high-quality relevant corpus. However, as the quality of the retrieved relevant corpus decreases, the performance degrades. Notably, random selection ("Random") even yields worse performance than not using the retrieved corpus ("Rank-0"). These findings underscore the effectiveness of the retrieval-augmented approach in enhancing caption generation, particularly with highquality retrieved relevant corpus.Figure 6 illustrates six examples by FINER-MLLM across three datasets. Ground-truth change captions and our FINER-MLLM generated change captions are provided for comparison, with changed regions denoted by red boxes. Additionally, the cross-attention distribution between the query features and the image patch features is illustrated, where multiple cross-attention distributions of query features are averaged and presented as a heatmap. Regarding the cases in Figure 6 (a), the top case demonstrates FINER-MLLM successfully recognizing the color change of the green ball and providing more detailed location information than the ground-truth caption. While in the bottom case, our method fails to generate the correct change caption, as denoted in red fonts. Further analysis reveals that in this case, the two images are slightly rotated, causing the ground-truth caption to label it as "no difference. " However, our FINER-MLLM struggles to recognize these viewpoint changes due to the feature alignment design, instead describing the location change of the green cylinder, which is more concise when ignoring the viewpoint changes. In Figure 6 (b), all images are from video-surveillance footage with a fixed viewpoint, and our method successfully captures subtle person changes in both cases. This can also be observed in Figure 6 (c). Moreover, we can intuitively observe from all six heatmaps that attention covers almost every detail within intra-images, with well-aligned attention distributions across paired images. Overall, these observations demonstrate the effectiveness of our designed dual constraints in improving differential-perceptive capabilities and retrieval-augmented caption generation, enhancing generation effectiveness. Moreover, they highlight the real-world application potential of our FINER-MLLM. 
conclusion: 
 In this work, we introduce a differential-perceptive and retrievalaugmented MLLM, enhancing multimodal large language models' ability to discern visual differences across multiple images for change captioning. In particular, we first extracted the image patch features of paired images to facilitate the following differential-perceptive feature extraction. This process incorporates intra-image feature independence and inter-image feature alignment constraints, ensuring that query features capture detailed information and remain aligned with each other, which is crucial for accurately perceiving subtle changes. Additionally, we employed retrieval augmentation to enhance the change caption generation capability of LLM, prioritizing the prediction of relevant words from the retrieval corpus. Extensive experiments have been conducted on three public datasets, and the results demonstrate the effectiveness and real-world application potential of FINER-MLLM.In the future, we plan to extend our method to address tasks involving multiple images, such as video caption, which is an essential problem in multimodal applications. 
: 
 a more challenging task of vision and language. This task requires not only comprehending the contents of two similar images but also describing their differences using natural language. Given its great potential value in various real-world applications, such as anomaly detection [9, 20] and pathological change report [18, 21] , change captioning has garnered increasing research attention in recent years.Although existing efforts have shown promising performance in change captioning, none have explored the application of advanced multimodal large language models (MLLMs) [2, 7, 17, 42] in this task. In mainstream MLLM paradigms such as BLIP-2 [17] and InstructBLIP [7] , a pivotal strategy involves utilizing a learnable Q-Former [17] to integrate a pre-trained frozen image encoder and a frozen large language model (LLM). The learnable Q-Former introduces a set of queries to capture fine-grained image features, aligning them effectively with the semantic space of the LLM. Consequently, both the image encoder and LLM do not require further fine-tuning yet exhibit remarkable success in various multimodal tasks necessitating both visual comprehension and text generation. With this in mind, our work aims to leverage MLLMs to harness the power of MLLMs to tackle the challenging task of change captioning more effectively.Essentially, the key to the change captioning lie in two points: (1) extracting visual features from two images to perceive subtle visual differences, and (2) generating natural language descriptions for these differences. However, directly applying existing MLLMs to change captioning presents challenges for both of these points.For visual feature extraction, when employing existing MLLMs directly, the image undergoes processing through a frozen visual encoder and a learnable Q-Former, which generate a set of visual queries. However, some of the learned visual queries may focus on the same prominent information in the image, resulting in redundancy and the inability to adequately capture subtle visual details. For example, subtle visual details, such as the pedestrians highlighted within the red boxes in Figure 1 (a), are prone to change and may be overlooked due to their small size, leading to an incomplete representation of the image's content. Furthermore, whenthe small purple metal cube behind the brown cylinder changed to red. the tiny purple metal cube that is behind the small brown thing changed to red. the tiny purple shiny block that is behind the large blue matte object changed to red. the tiny purple shiny block that is left of the brown cylinder turned red. the small purple shiny cube behind the red thing changed to red. MLLMs are applied to extract image features separately from the two images including before-change image and after-change image, the two forward propagation processes may not align features of the same visual query to the same regions within the images. This misalignment is problematic because changes often occur in the same region of both images, as shown in Figure 1(b) , where the color of the ball changes at the same position. Accordingly, the inability to consistently map the same image regions into same visual queries hinder the model's effectiveness in perceiving differences. Therefore, how to enhance the capability of existing MLLMs to comprehensively extract subtle visual information from images and ensuring that the same visual queries correspond to the same regions in both input images poses the first challenge.Semantic ChangeAs for differential description generation, the LLM in MLLMs has predominantly pre-trained on complete image descriptions, with limited exposure to differential descriptions. Employing the LLM without fine-tuning, as commonly done in existing methods, would result in suboptimal performance. However, fine-tuning the LLM would incur high computational costs and is prone to overfitting due to its large parameter size. Consequently, how to enhance the performance of LLM in generating differential descriptions in a cost-effective manner is also a crucial challenge.To tackle the aforementioned challenges, we aim to refine the differential-perception capabilities within the feature extraction process of paired images. Additionally, we endeavor to develop a retrieval-augmented methodology to enhance the generation of differential descriptions. We present a diFferentIal-perceptive aNd rEtRieval-augmented MLLM (FINER-MLLM) for change captioning. Specifically, FINER-MLLM employs InstructBLIP as the backbone and consists of three key components: image encoding, differential-perceptive feature modeling, and retrieval-augmented caption generation. The first component involves extracting image patch features with LoRA fine-tuning of the image encoder. The second component focuses on extracting paired image features through Q-Former, where we introduce dual constraints to enhance the differential perception capabilities. Specifically, we devise an intra-image feature independence constraint to eliminate redundancy across different visual queries, ensuring that each subtle visual detail is captured. Additionally, we propose an inter-image feature alignment constraint to ensure that corresponding regions in the two input images are mapped to the same visual queries. The third component aims to enhance the generation of differential descriptions by leveraging the assistance of relevant corpus retrieval.Our motivation stems from the observation that many relevant sentences share highly similar semantics suitable for change captioning. As illustrated in Figure 2 , we employ an off-the-shelf model that utilizes the two images as search queries, providing the top-5 relevant sentences. Notably, the semantic content of the text highlighted in green accurately describes the difference between the two images. Therefore, we adopt a retrieval-augmented approach, first retrieving the relevant corpus as assistance, and then generating the differential descriptions.The contribution of this work can be summarized in three folds: • 
 
 Appendix: https://anonymous.4open.science/r/FINER-MLLM-7010/.To simplify the explanation, we default to using a single attention head for illustration purposes.mm'24, October 28-November 1, 2024, Melbourne, Australia Anon. 


Decision: The paper initially received for positive indications (1WA, 3BA), which have been confirmed after the rebuttal period, also with a slight increase of the ratings (1BA moved to a 1WA). The paper can be accepted as a poster.

Reviews: review: paper_summary: paper_summary: The paper presents Differential-Perceptive and Retrieval-Augmented MLLM (FINER-MLLM) to enhance the capabilities of multimodal large language models (MLLMs) in perceiving subtle differences and generating accurate change captions. The FINER-MLLM incorporates a fine-tuned image encoder, dual constraints (intra-image feature independence and inter-image feature alignment) for detailed feature extraction, and a retrieval-augmented strategy to aid the generation of differential descriptions. The method is validated through extensive experiments on three benchmark datasets: CLEVR-Change, Spot-the-Diff, and Image-Editing-Request, showing significant improvements over existing methods.
strength_weakness: strength_weakness: 1. The integration between differential-perceptive capabilities and retrieval-augmented strategies within MLLMs is a novel approach and has not been explored previously.
2. The dual constraints applied to the feature extraction process (intra-image feature independence and inter-image feature alignment) are theoretically sound and ensure comprehensive extraction of visual details and proper alignment of corresponding features across image pairs.
3. The paper provides extensive experimental validation on three benchmark datasets.
limitations: limitations: 1. The method shows inferior performance on the synthetic CLEVR-Change dataset compared to real-world datasets. This limitation is acknowledged in the paper, but a deeper analysis or proposed solution for this discrepancy would be beneficial.
2. The retrieval-augmented approach's performance heavily relies on the quality of the retrieved relevant corpus. This dependency might limit the method's applicability in scenarios where high-quality retrieval data is not available or difficult to obtain.
paper_summary: paper_summary: This paper introduces FINER-MLLM, a model designed for Change Captioning tasks. It enhances multimodal large language models through refined feature extraction and the integration of retrieval-augmented techniques. By incorporating LoRA fine-tuning and dual constraints, the model effectively captures subtle visual details and ensures alignment across image regions. The adoption of a retrieval-augmented approach further enhances caption accuracy, demonstrating the model's effectiveness in addressing change captioning tasks on benchmark datasets. 

Overall, FINER-MLLM presents a comprehensive solution that leverages advanced techniques to improve the generation of change descriptions in image pairs, showcasing promising results in the field of change captioning
strength_weakness: strength_weakness: 1, The introduction of FINER-MLLM represents a novel approach in the domain of change captioning by empowering multimodal large language models with differential-perceptive features and retrieval-augmented techniques. 

2, The paper provides a comprehensive evaluation of the proposed model on three public datasets, including a synthetic dataset and two real-world domain datasets . By reporting standard metrics such as BLEU-4, METEOR, ROUGE-L, CIDEr-D, and SPICE, the evaluation ensures a fair comparison with existing methods and demonstrates the effectiveness of the proposed approach in generating change captions.

3, The paper is well-structured and clearly articulates the motivation, methodology, experimental settings, and results. The use of figures and examples further enhances the clarity of the presentation, making it easier for readers to understand the concepts and contributions of the research.
limitations: limitations: 1, While FINER-MLLM introduces novel aspects such as dual constraints and retrieval-augmented methodology, some individual components like have been utilized in prior works . This may limit the overall novelty of the model in terms of individual components.

2, The effectiveness of the retrieval-augmented approach in generating differential descriptions is contingent on the quality and relevance of the retrieved corpus. In scenarios where the quality of the retrieved corpus is suboptimal, the model's performance may be negatively impacted.

Overall, the paper demonstrates a good work in the field of vision-language understanding and generation. However, concerns arise regarding the novelty and performance parity, particularly in the utilization of components like LoRA fine-tuning and Q-Former, as mentioned earlier.  To enhance understanding of the contribution, it is advisable to provide a clearer explanation of the key insights and technical novelty introduced by the model. This will help highlight the unique approach and innovative aspects of the research. Additionally, for a fair comparison with existing methods, it is recommended to include detailed information on the model settings, backbones, and image resolutions used for performance measurement.
paper_summary: paper_summary: This paper introduces an approach to the task of change captioning, which involves describing subtle changes between pairs of similar images. The authors propose FINER-MLLM, a model that enhances multimodal large language models (MLLMs) with the capability to perceive and describe changes in images. The key innovations include the use of LoRA fine-tuned image encoders for detailed image feature extraction, dual constraints for intra-image feature independence and inter-image feature alignment, and a retrieval-augmented approach to facilitate the generation of accurate change captions. The model was evaluated on three benchmark datasets and demonstrated superior performance compared to existing methods.
strength_weakness: strength_weakness: 1. The paper presents a unique integration of differential perception and retrieval augmentation in the context of change captioning, which is a novel contribution to the field.

2. The introduction of dual constraints for feature extraction is technically sound and enhances the model's ability to capture subtle changes.

3. Utilizing an MLLM to accomplish this task represents a promising trend.
limitations: limitations: 1. The authors should discuss the performance in general tasks of the fine-tuned MLLM. Does fine-tuning significantly degrade the performance of an MLLM on basic dialogue tasks? The authors can use cases or add experiments to discuss.

2. The improvement in performance appears to be limited, which diminishes the novelty of the paper.
paper_summary: paper_summary: The paper introduces a Multimodal Language Model (MLLM) designed for the change captioning task, which involves describing subtle differences between pairs of similar images.  The key innovation in the proposed model, named FINER-MLLM (diFferentIal-perceptive aNd rEtRieval-augmented MLLM), is the inclusion of a retrieval-augmentation step. This step enhances the model's ability to detect and describe changes by retrieving relevant information from a corpus, which is then used to guide the language model in generating accurate change captions. The effectiveness of FINER-MLLM is demonstrated through extensive experiments on three benchmark datasets: CLEVR-Change, Spot-the-Diff, and Image-Editing-Request. The results show that the proposed model outperforms existing methods, generating more accurate and detailed change captions.
strength_weakness: strength_weakness: * The claim is clear and also well supported through a range of experiments. 
* The comprehensive ablation studies effectively illustrate the importance of each architecture component and support the paper.
* The introduction of retrieval in the pipeline is quite novel.
limitations: limitations: * It is unclear whether the retrieval of the top-k words uses an exact index or libraries such as Faiss.
* It would be interesting to see the impact of when varying the number of k retrieved elements
* It would be interesting to see the impact of passing the retrieved tokens concatenated to the input of the LLM instead of using them in the cross attention
* Given that the CIDEr metric has known issues when evaluating captions, particularly with longer and more detailed ones, it would be beneficial to evaluate the proposed approach using more recent metrics like CLIP-S [1] and PAC-S [2].

Minor:
* The authors did not adhere to the review template, and the margins lack line numbers.

Typos:
* The title of Section 3 “Methdology” → “Methodology”


[1] Hessel, Jack et al. Clipscore: A reference-free evaluation metric for image captioning, 2021

[2] Sarto, Sara et al. Positive-augmented contrastive learning for image and video captioning evaluation, 2023