Title: Difference-aware Visiolinguistic Regularization for Image Change Captioning
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Change Captioning, Image Captioning, Vision-Language Models

Abstract: Image Change Captioning (ICC) has emerged as an important task in multi-modal generative AI, aiming to generate natural language descriptions that  reflect the differences between two similar images. Unlike traditional image captioning, ICC requires strong cross-image difference reasoning and language generation capabilities to handle diverse and complex scenarios. Recent advances have introduced MLLM-based methods for ICC, achieving impressive results. However, these approaches rely solely on caption-level supervision to implicitly infer and describe changes, which often results in the omission of fine-grained differences and suboptimal caption quality. To address this, we propose a Difference-Aware Visiolinguistic Regularization (DAVIR) paradigm that jointly regularizes the fine-tuning of MLLM from both visual and linguistic perspectives, enabling better adaptation to  ICC. Specifically, we first introduce a fine-grained attention control  module to regularize the final-layer self-attention maps of the MLLM’s encoder, guiding it to focus on subtle changes during feature extraction. Second, we propose an entity prompt construction  scheme to guide the MLLM’s decoder and enhance caption generation quality. Extensive experiments on three benchmark datasets across different scenarios demonstrate that our method achieves state-of-the-art performance. The code will be released publicly.

TL;DR: We propose DAVIR which improves MLLM fine-tuning by guiding the encoder to focus on subtle visual changes and enhancing decoder caption generation via entity prompts, achieving state-of-the-art results across multiple benchmarks.