Context-aware Difference Distilling for Multi-change Captioning (2024)

Abstract: Multi-change captioning aims to describe complex and coupled changes within an image pair in natural language. Compared with singlechange captioning, this task requires the model to have higher-level cognition ability to reason an arbitrary number of changes. In this paper, we propose a novel context-aware difference distilling (CARD) network to capture all genuine changes for yielding sentences. Given an image pair, CARD first decouples context features that aggregate all similar/dissimilar semantics, termed common/difference context features. Then, the consistency and independence constraints are designed to guarantee the alignment/discrepancy of common/difference context features. Further, the common context features guide the model to mine locally unchanged features, which are subtracted from the pair to distill locally difference features. Next, the difference context features augment the locally difference features to ensure that all changes are distilled. In this way, we obtain an omni-representation of all changes, which is translated into linguistic sentences by a transformer decoder. Extensive experiments on three public datasets show CARD performs favourably against state-of-the-art methods. The code is available at https: //github.com/tuyunbin/CARD.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7941–7956 August 11-16, 2024 ©2024 Association for Computational Linguistics
Multi-change captioning aims to describe complex and coupled changes within an image pair in natural language. Compared with singlechange captioning, this task requires the model to have higher-level cognition ability to reason an arbitrary number of changes. In this paper, we propose a novel context-aware difference distilling (CARD) network to capture all genuine changes for yielding sentences. Given an image pair, CARD first decouples context features that aggregate all similar/dissimilar semantics, termed common/difference context features. Then, the consistency and independence constraints are designed to guarantee the alignment/discrepancy of common/difference context features. Further, the common context features guide the model to mine locally unchanged features, which are subtracted from the pair to distill locally difference features. Next, the difference context features augment the locally difference features to ensure that all changes are distilled. In this way, we obtain an omni-representation of all changes, which is translated into linguistic sentences by a transformer decoder. Extensive experiments on three public datasets show CARD performs favourably against state-of-the-art methods. The code is available at https: //github.com/tuyunbin/CARD.

heading: 1 Introduction
text: Change captioning aims to describe differences between a pair of similar images, which enables many important applications, such as automatic report generation about change conditions of surveillance areas (Hoxha et al., 2022) and pathological changes between medical images (Liu et al., 2021). On the other hand, this task is more challenging than image captioning (Yang et al., 2023; Rotstein et al., 2024; Zhao and Xiong, 2024). This is because machines need to understand the contents of
*Corresponding authors
two images simultaneously, and further reason and caption all genuine changes between them, while resisting irrelevant viewpoint/illumination changes.
Recently, single-change captioning has made remarkable progress (Tu et al., 2021b; Hosseinzadeh and Wang, 2021; Yao et al., 2022; Yue et al., 2023; Tu et al., 2024). In a dynamic environment, however, the changes are usually the many-in-one, where multiple changes exist in an image pair. For instance, there are multiple object/background changes (Figure 1 (a) (b)). In other cases, object and viewpoint changes simultaneously appear (Figure 1 (c)). In above cases, unchanged objects commonly mingle with changed ones and even appear position misalignment under viewpoint changes. Such distractors pose a great challenge to identify and caption the genuine changes.
There are a few attempts to address multi-change captioning. The pioneer work (Jhamtani and Berg-
7941
Kirkpatrick, 2018) computed pixel differences of two images, which is sensitive to noise. Latest works tried to capture differences at representation space: some of them (Hoxha et al., 2022; Liu et al., 2022, 2023a) computed difference features by subtraction, while the others (Qiu et al., 2021; Chang and Ghamisi, 2023) built the correlations between the patches of two images to model the change features for caption generation.
Despite progress, the above endeavors in multichange captioning have several limitations. (1) Direct subtraction between two images generalizes poorly to unaligned image pairs under viewpoint changes (Figure 1 (c)). (2) Directly correlating two images fails to sufficiently mine locally unchanged features as multiple objects change, because such features might mingle with the features of changed objects. (3) These methods focus on modeling locally difference features, which are useful to catch conspicuous changes. Nevertheless, certain local changes with weak features might be overlooked, e.g., the car occluded by its shadows in Figure 1 (a). These limitations would result in obtaining unreliable difference features for the language decoder.
We notice that the above methods capture differences between two images only based on local features, while neglecting the use of more comprehensive features. We argue that, to learn locally unchanged/changed features of two images, the model should first encapsulate their context features of commonality and difference. Such context features aggregate all similar/dissimilar semantics, termed common/difference context features. The former can help correlate and mine locally common features for deducing locally difference features, while the latter can augment the locally difference features to ensure all changes are distilled.
In this paper, we propose a Context-Aware DiffeRence Distilling (CARD) network to learn the robust difference features under multi-change scenes. Specifically, given the featuers of two images, we first build intra-image interaction to help the model understand each image content of the pair. Then, we use CARD to decouple the common/difference context features from the image pair. Herein, the common context features of two images summarize joint semantics in between; the difference context feature in each image provides an independent space to preserve its all changed semantics. Besides, the consistency and independence constraints are designed to enforce
the alignment and discrepancy of common and difference context features, respectively. Next, guided by the common context features, CARD models inter-image interaction to mine locally common features, which are removed from the pair to distill locally difference features. Subsequently, CARD augments the locally difference features via the difference context features, so as to construct an omni-representation of all changes, for generating descriptions by a transformer decoder.
Our key contributions are: (1) We propose CARD to first decouple common and difference context features, and then use them to facilitate modeling an omni-representation of all changes for multi-change captioning. (2) The consistency and independence constraints are customized to guarantee the alignment and discrepancy of decoupled common and difference context features. (3) Extensive experiments show our method achieves the state-of-the-art results on three public datasets.

heading: 2 Related Work
text: Change captioning is an emerging task in the community of multi-modal learning (Cong et al., 2022, 2023; Tu et al., 2022). In the following, we introduce the relevant works about single-change captioning and multi-change captioning, respectively.
Single-change Captioning has been widely studied by most existing methods. The prior work (Park et al., 2019) collects a dataset about geometric objects under viewpoint changes. This work computes the difference representation by direct subtraction, which generalizes poorly between two unaligned images. To remedy this limitation, MVAM (Shi et al., 2020), VACC (Kim et al., 2021) and R3Net (Tu et al., 2021a) match local features to predict difference features, which has been a classic paradigm. The latest work SCORER+CBR (Tu et al., 2023c) further improves this paradigm by maximizing cross-view contrastive alignment between two images, so as to learn a more stable difference representation. In addition, these works (Hosseinzadeh and Wang, 2021; Kim et al., 2021; Tu et al., 2023c; Yue et al., 2024) introduce the idea of cross-modal consistency constraint to improve captioning quality. Besides improving architecture, recent works (Yao et al., 2022; Guo et al., 2022) propose the strategy of pre-training to fine-tuning for facilitating change location and caption. However, it is seldom that only one change appears in a dynamic environment, so a powerful model should
has the capability to describe multiple changes. Multi-change Captioning has been explored by a few attempts. The pioneer work (Jhamtani and Berg-Kirkpatrick, 2018) proposes to describe multiple changes between the image pairs from the surveillance cameras, where it captures changes at pixel level. Recent works (Hoxha et al., 2022; Liu et al., 2022, 2023a; Chang and Ghamisi, 2023) propose to caption changes between remote sensing images, where they first compute the difference features and then use them for change detection and description. Nevertheless, the above methods only describe differences between two well-aligned images, while ignoring the cases of unaligned image pairs under varied viewpoints. To this end, Qiu et al. (Qiu et al., 2021) collect a dataset to caption multiple changes under viewpoint changes, where they adopt the classic paradigm of local feature matching to detect changes for caption generation. However, such a matching paradigm may fail to mine the fine-grained common features and thus learn error-prone difference features. Meanwhile, the current methods focus on modeling local difference features, which risks ignoring certain local changes with weak features. On the other hand, Qiu et al. (Qiu et al., 2023) develop a new synthetic dataset to describe the multiple changes and their orders. However, recording the order of changed objects is laborious in the real world, making it hard to test such a capability in a real-world scene.
In short, different from previous methods computing difference features only based on local features, CARD first decouples common and difference context features from an image pair. The common context features guide the model to fully extract locally unchanged features for computing the features of local differences, while the difference context features augment the locally difference features to construct an omni-representation of all changes, for generating accurate sentences.

heading: 3 Methodology
text: The overall architecture of our method is shown in Figure 2. Given a pair of images, our method is to generate linguistic sentences that detail all of the changes. Architecture-wise, our method contains three components: (a) image pair encoding; (b) context-aware difference distilling; (c) caption generation. We provide an overview of two basic components (a) and (c) in Sec. 3.1 and Sec. 3.3, while elaborating our key ingredient (b) in Sec. 3.2.

heading: 3.1 Image Pair Encoding
text: Formally, given a pair of “before” Ibef and “after” Iaft images, we first leverage an off-the-shell encoder (e.g., ResNet-101 (He et al., 2016)) to extract n local features for each of them, and then introduce a trainable [CLS] feature to represent the global content of each image: Xo = {xocls, x1, x2, ..., xn}, where o ∈ (bef, aft) and xi ∈ Rd. Besides, the trainable position encodings are added into the features of each image to help the model perceive the relative position changes of objects. Next, we exploit a multi-head self-attention layer (Vaswani et al., 2017) to capture the relationships among the features of each image, which helps the model sufficiently understand the image content of the pair. Through the above manner, we can obtain the relation-embedded features for each image, denoted as X̃o = {x̃ocls, x̃1, x̃2, ..., x̃n}.

heading: 3.2 Context-Aware Difference Distilling

heading: 3.2.1 Context Feature Decoupling
text: To obtain common and difference context features, we first devise a common encoder CE (·; θC) and two difference encoders DEo (·; θo), where three encoders are based on the linear projection and o ∈ (bef, aft). The common encoder CE (·; θC) shares the parameters θC between two images, while the difference encoders DEo (·; θo) learn the parameters θo for each image. Then, we feed x̃ocls into these encoders to decouple the common and difference context features, respectively:
Co = CE (x̃ocls; θC) , Do = DEo (x̃ocls; θo) ,
(1)
where Cbef , Caft, Dbef , and Daft ∈ Rd. Consistency Constraint. To make two common context features Cbef and Caft embedded in a shared space, we tailor the consistency constraint based on contrastive learning. Given a training batch, we sample B pairs of common context features. For the common context feature in the k-th “before” image Cbefk , the common context feature in the r-th “after” image Caft +
r(r=k) is its positive, while common context features in the other “after” images Caft −
r(r ̸=k) will be the negatives in this batch. Then, we project these positive/negative pairs into a shared embedding space, normalize them by L2normalization, and compute their similarity. Next, we introduce the InfoNCE loss (Oord et al., 2018) to optimize their contrastive alignment, i.e., pulling
semantically close pairs of Cbefk and C aft+
r(r=k) together and pushing away non-related pairs:
Lb2a = − 1
B
B∑
k
log e
( sim ( Cbefk ,C aft+ r(r=k) ) /τ )
∑B r e
( sim ( Cbefk ,C aft r ) /τ ) ,
La2b = − 1
B
B∑
k
log e
( sim ( Caftk ,C bef+ r(r=k) ) /τ )
∑B r e
( sim ( Caftk ,C bef r ) /τ ) ,
Lcon = 1
2 (Lb2a + La2b),
(2) where “sim” is the dot-product function to measure the similarity between two context features. τ is the temperature hyper-parameter. Through this consistency constraint, we enforce the common context features of two images to be projected into a shared semantic space with aligned distributions.
Independence Constraint. Each decoupled difference context feature can learn the unique characteristics of its corresponding image within the image pair. These unique characteristics represent the semantic differences between the two images, so each difference context feature should be distinct from the other. To this end, we design an independence constraint, which ensures that the two difference context features are projected into separate feature spaces. Here, we opt for the Hilbert-Schmidt Independence Criterion (HSIC) (Song et al., 2007), a proven method for testing
feature independence, to design the loss of the independence constraint. A lower HSIC score between the two difference context features indicates a higher independence between them: each difference context feature adequately encapsulates the semantic changes in each image.
Concretely, we first project the difference context feature of each image into a separate space, normalize each by L2-normalization, and define the independence (HSIC) constraint between Dbef and Daft as:
HSIC (Dbef , Daft) = (B−1)−2 tr (PKbefPKaft) , (3) where Kbef , Kaft ∈ RB×B are the Gaussian kernel matrices with kbef,ij = kbef ( Dibef , D j bef ) and
kaft,ij = kaft ( Diaft, D j aft ) . B is batch size. Dibef refers to the difference context feature in the i-th “befor” image. P = I − 1B eeT , where P ∈ RB×B , I is an identity matrix and e is an all-one column vector. If the HSIC score between Dbef and Daft is lower, their disparity is more significant. We define the independence loss as:
Lind = HSIC (Dbef , Daft) . (4)

heading: 3.2.2 Difference Distilling
text: With the local features of each image L̃o = {x̃1, x̃2, ..., x̃n}, where o ∈ (bef, aft), we first broadcast the common context feature of each im-
age Co ∈ Rd to Co ∈ Rn×d. Then, we concatenate it with L̃o on the channel dimension to obtain X̃ ′o ∈ Rn×2d, where Co can guide inter-image interaction to mine locally unchanged features. Next, we transform X̃ ′o ∈ Rn×2d to X̃ ′o ∈ Rn×d by a non-linear function with ReLU activation. Further, we compute the locally common features on each image by the multi-head cross-attention (MHCA) mechanism (Vaswani et al., 2017):
X̃cbef = MHCA ( X̃ ′bef , X̃ ′aft, X̃ ′aft ) , X̃caft = MHCA ( X̃ ′aft, X̃ ′bef , X̃ ′bef ) . (5)
Subsequently, we respectively subtract each X̃co from L̃o to compute the locally difference features of each image. These locally difference features are further augmented by difference context feature of each image, so as to distill all of the genuine changes in each image:
X̃dbef = [L̃bef − X̃cbef ;Dbef ], X̃daft = [L̃aft − X̃caft;Daft],
(6)
where [;] is a concatenation operation. Both X̃dbef and X̃ d aft are then concatenated as an omnirepresentation of all changes between two images, which is implemented by a non-linear transformation with the ReLU function:
X̃d = ReLU ([ X̃dbef ; X̃ d aft ] Wc + bc ) . (7)

heading: 3.3 Caption Generation
text: After obtaining the omni-representation X̃d ∈ Rhw×d, we use a transformer decoder (Vaswani et al., 2017) to decode it into sentences. First, we obtain the embedding features of all m words of these sentences, where each sentence is separated by a special token [SEP]. Then, we use the masked self-attention to model relationships among these word features. Next, we model the interaction between the word features and omni-representation by cross-attention, so as to locate the most related difference features during word generation. Subsequently, we feed the selected features into a feedforward network to obtain the enhanced difference representation, denoted as Ĥ ∈ Rm×d. Finally, the probability distributions of words in these sentences are calculated via a single hidden layer:
S = Softmax ( ĤWs + bs ) , (8)
where Ws ∈ Rd×u and bs ∈ Ru are the learnable parameters. u is the dimension of vocabulary size.

heading: 3.4 Joint Training
text: Our method is trained in an end-to-end manner by maximizing the likelihood of the observed word sequence. Given the ground-truth words (s∗1, . . . , s ∗ m), we minimize the negative loglikelihood loss:
Lcap(θ) = − m∑
t=1
log pθ (s ∗ t | s∗<t) , (9)
where pθ (s∗t | s∗<t) is computed by Eq. (8), and θ are all the learnable parameters. Our method is also self-supervised by the losses of consistency and independence constraints. Hence, the total loss is optimized as follows:
L = Lcap + λc(Lcon + Lind), (10)
where λc is a trade-off parameter to balance the contribution between the caption generator and constraints, which is discussed in the appendix.

heading: 4 Experiments

heading: 4.1 Datasets
text: CLEVR-Multi-Change Dataset (Qiu et al., 2021) is about basic object scene with multiple changes. Since original dataset has not been released, we regenerate this dataset based on the authors’ released code. The regenerated dataset has 45,044 valid image pairs/captions with viewpoint changes. Based on the official split, we split it into training, validation, and testing with a ratio of 4:1:1.
LEVIR-CC Dataset (Liu et al., 2022) is about remote sensing scene, which contains 10,077 pairs of bi-temporal images and 50,385 ground-truth captions. We use the official split with 6,815 image pairs for training, 1,333 for validation, and 1,929 for testing, respectively.
Spot-the-Diff Dataset (Jhamtani and BergKirkpatrick, 2018) has 13,192 image pairs from surveillance cameras, and on an average there are 1.86 ground-truth sentences per image pair. According to the official split, we split it into training, validation, and testing with a ratio of 8:1:1.

heading: 4.2 Evaluation Metrics
text: We follow the existing methods of multi-change captioning to use the five metrics for evaluating the generated sentences: BLEU-4 (B) (Papineni et al., 2002), METEOR (M) (Banerjee and Lavie, 2005), ROUGE-L (R) (Lin, 2004), CIDEr (C) (Vedantam et al., 2015) and SPICE (S) (Anderson et al., 2016).
We compute all the results by the Microsoft COCO evaluation server (Chen et al., 2015).

heading: 4.3 Implementation Details
text: For fair-comparison, we follow previous multichange captioning methods to use a pre-trained ResNet-101 (He et al., 2016) to extract the local features of a pair of images, with the dimension of 1024 × 14 × 14. We project them into a lower dimension of 512, while the dimension of trainable [CLS] features is also set to 512. The hidden size of the model and word embedding size are set to 512 and 300. Temperature τ in Eq. (2) is set to 0.07. We train the model to converge with 10K iterations in total. We use Adam optimizer (Kingma and Ba, 2014) to minimize the negative log-likelihood loss of Eq. (10). More details are shown in the appendix.

heading: 4.4 Performance Comparison
text: Results on CLEVR-Multi-Change. We compare CARD with the following SOTA methods: DUDA (Park et al., 2019), M-VAM (Shi et al., 2020), MCCFormers-S / MCCFormers-D (Qiu et al., 2021), VARD-Trans (Tu et al., 2023a), and SCORER+CBR (Tu et al., 2023c). On this regenerated dataset, we re-implement the above methods based on their papers and released codes.
The results are shown in Table 1. Our CARD performs favourably against these SOTA methods on all metrics, showing that CARD can better describe multiple changes under viewpoint changes. In addition, our CARD outperforms MCCFormersD and MCCFormers-S by a large margin, which are classic match-based methods to directly capture inner/inter-patch correlations between two image representations. On the caption-specific metric CIDEr, CARD significantly surpasses both methods, in particular with increases of 2.2% and 3.4%.
Results on LEVIR-CC. We compare CARD with the SOTA methods: DUDA (Park et al., 2019), MCCFormers-S/D (Qiu et al., 2021), RSICCFormer (Liu et al., 2022), PSNet (Liu et al., 2023a), Prompt-CC (soft/hard) (Liu et al., 2023b), and Chg2Cap (Chang and Ghamisi, 2023).
The experimental results are shown in Table 2. Our CARD achieves the best results on all metrics. This indicates that it can detect whether there are semantic changes and what have changed between two remote sensing images. Besides, we notice that the match-based methods (MCCFormers-D / MCCFormers-S) cannot generalize well in this remote sensing scenario. Our conjecture is that there are usually most changed areas (e.g., Figure 1 (b)), so directly matching two images might fail to extract fine-grained unchanged objects and thus capture the difference features with noise.
Results on the Spot-the-Diff Dataset. Most previous works (Hosseinzadeh and Wang, 2021; Huang et al., 2022; Tu et al., 2023b; Yue et al., 2023) tested the models based on single-change setup, where the models are only required to randomly describe one of the changes. Different from them, we require the model to caption all changes.
Under this setting, the compared SOTA methods are: DDLA (Jhamtani and Berg-Kirkpatrick, 2018), DUDA (Park et al., 2019), MCCFormers-S
/ MCCFormers-D (Qiu et al., 2021), VARD-Trans (Tu et al., 2023a), and SCORER+CBR (Tu et al., 2023c). The experimental results are shown in Table 3. Our method achieves the best results on most metrics, particularly with an increase of 9.1% on CIDEr. As shown in Figure 1 (a), the changed objects are not well captured by surveillance cameras and are even occluded by shadows. Our method still achieves encouraging performance, which validates its good generalization in surveillance scenes.
Performance Analysis. On the three datasets, our CARD outperforms the existing methods by a large margin, which shows its good generalization. This superiority benefits from that decoupled context features facilitate learning difference features. Instead, the compared methods only compute the differences based on the local features, which fails to mine fine-grained common objects and thus disregards inconspicuous changes.

heading: 4.5 Ablation Study and Analysis
text: To figure out the contribution of each component, we conduct ablation studies on two large-scale datasets: CLEVR-Multi-Change and LEVIR-CC. The image pairs on CLEVR-Multi-Change contain basic geometric objects and are unaligned due to viewpoint changes, while the pairs on LEVIR-CC are well-aligned and from the real world.

heading: 4.5.1 Ablation Study for Context Features
text: We study the effectiveness of decoupled common and difference context features, denoted as CCF and DCF, respectively. The results are shown in Table 4. Baseline directly matches two image features to extract the locally common features and
difference features for caption generation. We find that on the both datasets, 1) the model’s performance is enhanced when it is augmented by either CCF or DCF; 2) when we augment the model with both context features, the model’s performance is significantly boosted, especially the CIDEr score is enhanced from 362.3 to 391.6. These show that 1) CCF guides the model to sufficiently interact and mine locally common features for computing locally difference features, while DCF augments the locally difference features to ensure all changes are distilled; 2) each kind of context feature not only plays its unique role, but also supplements each other for better reasoning genuine changes.

heading: 4.5.2 Ablation Study for Constrains
text: To study the effect of the consistency constraint (CC) and independence constraint (IC), we make the ablation study in Table 5. First, we enforce CC or IC respectively on CARD, and find that each of them improves the performance of CARD. Then, we impose both constraints and observe that the results are enhanced significantly on two datasets. The increased results indicate that both constraint losses are essential to learn the context features.
We further visualize the common/difference context features decoupled without/with the constraints, which are shown in Figure 3. Without constraints, the common/difference context features cannot be well learned on two datasets. With both constraints, the common context features are blended on CLEVR-Multi-Change, while the difference context features are learned better on LEVIRCC. The results show that consistency constraint
helps align distributions of shared properties, while independence constraint makes difference context features of two images more separable.

heading: 4.5.3 Generalization to unaligned image pairs
text: To verify the generalization to unaligned image pairs under viewpoint changes, we compare CARD with two ablative variants on the CLEVR-MultiChange dataset. (1) Direct Subtraction first performs direct subtraction between the features of two images to compute the locally difference features, which are then fed into a transformer decoder for multi-change captioning. (2) Feature Matching directly matches two image features to extract the locally common features and difference features for multi-change captioning. The experimental results are shown in Table 6.
It is noted that the performance of Feature Matching is better than that of Direct Subtraction on every metric, which validates the generalization of feature matching paradigm to the unaligned image pairs under viewpoint changes. Our proposed CARD outperforms both models by a large margin.
This not only indicates a better generalization of our method to unaligned image pairs, but also verifies the effectiveness of context-aware difference distilling to help capture genuine changes.

heading: 4.6 Qualitative Analysis
text: To obtain an overall evaluation of our method, we conduct qualitative analysis on the three datasets. The compared method MCCFormers-D (Qiu et al., 2021) performs well on the three datasets, which directly correlates two images to predict locally common and difference features. In Figure 5, we visualize the alignment of common properties between two images, to validate whether context features help mine locally common information. We find that MCCFormers-D fails to align the common properties and even misjudges changed objects as unchanged objects. Instead, our CARD can better match the joint objects. For instance, in Figure 5 (a), the unchanged object is only the brown object. MCCFormers-D wrongly identifies some changed objects as unchanged ones. By contrast, our CARD can pinpoint the unchanged brown object. This superiority benefits from the guidance of decoupled common context features, during the matching of two image features.
In Figure 4, we further visualize the captions yielded by MCCFormers-D and CARD, as well as the change localization results from CARD. In the first three cases, MCCFormers-D either only describes one of the changes or misidentifies changed objects. Contrarily, our CARD can accurately locate and describe all changed objects. Particularly, we notice that our method performs better in detecting subtle changes. For instance, in Figure 4 (c) that is from surveillance scene, the moved car and disappeared person are very tiny, and the car is occluded by the shadow of building. In this hard case, MCCFormers-D fails to recognize the moved car in the “after” image, thus generating a wrong sentence. By contrary, our CARD can locate and describe this tiny change. For the failure reason of MCCFormers-D, our conjecture is that it directly interacts two images, which cannot sufficiently identify locally unchanged features and compute the locally difference features. Besides, MCCFormers-D captures differences between two images only based on local features, which risks overlooking certain tiny changes with weak features. Compared with MCCFormers-D, the superiority of our method is mainly attributed to the
guiding and augmenting of common and difference context features. Guided by the common context features, CARD models inter-image interaction to sufficiently mine locally common features and compute locally difference features. Further, the difference context features augment the locally difference features to ensure that all changes are distilled. Through this manner, the model can learn genuine changes for caption generation. More qualitative examples are shown in the appendix.

heading: 5 Conclusion
text: In this paper, we propose the CARD to reason and describe genuine changes under various multichange scenarios. CARD first decouples the common and difference context features from the image pair. Then, two kinds of constraints are designed to ensure the alignment and discrepancy of the common and difference context features, respectively. Further, we use the common context features to guide the mining of locally common
features for deducing locally difference features. In addition, we leverage the difference context features to augment the locally difference features, thereby constructing an omni-representation of all changes for multi-change captioning. Extensive experiments conducted on the three datasets show that the proposed CARD outperforms the current state-of-the-art methods by a large margin.
Limitations
The last case in Figure 4 shows that the trees are replaced by a cross road with villas. Our CARD successfully locates the changed objects, which validates the effectiveness of context-aware difference distilling. However, it yields a sub-optimal sentence that does not well express the change process: trees are removed and a cross road with villas built along. A more proper sentence should be: trees are removed and a cross road with villas are built. In the future work, we will try to introduce linguistic knowledge (e.g., syntactic dependencies between words) that regularizes the process of sentence generation, in order to generate the optimal sentences well elaborating the change process.

heading: Acknowledgements
text: This work was supported in part by National Key R&D Program of China under Grant (2023YFB4502800), National Natural Science Foundation of China: 62322211, 61931008, 62236008, 62336008, U21B2038, 62225207, Fundamental Research Funds for the Central Universities (E2ET1104), “Pionee” and “Leading Goose” R&D Program of Zhejiang Province (2024C01023, 2023C01030).

heading: A Appendix
text: In this appendix, we provide more details about the caption generation and more experimental results, as well as the qualitative analyses.
A.1 Caption Generation After obtaining the representation of all changes X̃d ∈ Rhw×d, we use a transformer decoder to decode it into target sentences. First, we obtain the embedding features of all m words of these sentence E[S] ∈ Rm×d, where each sentence is separated by a special token [SEP]. Then, we use the masked self-attention (Vaswani et al., 2017) to model relationships among these word features, which is defined as:
Ê[S] = LN (E[S]+ MHSA (E[S], E[S], E[S])), (11) where LN is short for layer normalization (Ba et al., 2016). Next, we model the interaction between these word features and difference representation X̃d by multi-head cross-attention (MHCA) (Vaswani et al., 2017), so as to locate the most related difference features H̃:
H̃ = LN (E[Ŝ] + MHCA (E[Ŝ], X̃d, X̃d)). (12) Subsequently, we feed the selected features H̃ into a feed-forward network to obtain the enhanced difference representation, denoted as Ĥ ∈ Rm×d.
Ĥ = LN((H̃ + FFN(H̃)). (13)
Finally, the probability distributions of words in these sentences are calculated via a single hidden layer:
S = Softmax ( ĤWs + bs ) , (14)
where Ws ∈ Rd×u and bs ∈ Ru are the learnable parameters. u is the dimension of vocabulary size.
A.2 Experiments
A.2.1 Implementation Details For fair-comparison, we follow the previous multichange captioning methods (Jhamtani and BergKirkpatrick, 2018; Qiu et al., 2021; Liu et al., 2022) to use a pre-trained ResNet-101 (He et al., 2016) to extract the local features of a pair of images, with the dimension of 1024 × 14 × 14. We project them into a lower dimension of 512, while the dimension of trainable [CLS] features is also set to 512. The hidden size of overall model and word embedding size are set to 512 and 300, respectively. Temperature τ in Eq. (2) of main paper is set to 0.07. The attention layers in CARD is set to 1 on the CLEVRMulti-Change and Spot-the-Diff datasets; and 3 on the LEVIR-CC dataset. We train the model with PyTorch (Paszke et al., 2019) on a single RTX 3090 GPU, and use Adam optimizer (Kingma and Ba, 2014) to minimize the negative log-likelihood loss of Eq. (10) in the main paper. The training details about batch size and learning rate are shown in Table 7. The used training resources about time and GPU memory are shown in Table 8. We find that training CARD does not require much more time and GPU memory. Hence, it would be easily reimplemented by other researchers and be a strong baseline for the future works.
A.3 Study on the Trade-off Parameter λc We discuss the influence of trade-off parameter λc in Eq. (10). The results under varied values are shown in Table 9-11. Note that λc=0 means there are no constraints upon the decoupled results. We observe that imposing the constraint losses does improve the model’s performance. Besides, there is a trade-off between the captioning loss and constraint losses, because too large λc may lead to deterioration in caption performance. Based on the results, we choose the value as 0.3 on CLEVRMulti-Change, 0.1 and 0.001 on the LEVIR-CC and Spot-the-Diff, respectively.
A.4 Qualitative Analysis
In this appendix, we will show more qualitative examples on the CLEVR-Multi-Change, LEVIRCC, and Spot-the-Diff datasets, which are shown in Figure 6-9. To intuitively understand whether the common context features help mine reliable common properties, we visualize the alignment of common properties between two images on the three datasets, which are shown in Figure 6-7. The compared method is MCCFormers-D (Qiu et al.,
2021), which has the stable performance on the three datasets. From these examples, we can observe that MCCFormers-D is unable to align the common properties properly and even misjudges changed objects as unchanged objects. Compared with it, the proposed CARD can better match the common objects, so as to validate the effectiveness of the decoupled common context features.
Further, in Figure 8-9, we visualize the captions yielded by MCCFormers-D and CARD, as well as the change localization results from CARD on the three datasets. In Figure 8, on the CLEVRMulti-Change dataset under one-to-four changes, we find that MCCFormers-D either only describes partial changes or misidentifies changed objects. Instead, the proposed CARD is able to accurately locate and describe all changed objects. In Figure 9, on the LEVIR-CC and Spot-the-Diff datasets, it is noted that MCCFormers-D fails to describe all the changes within each image pair that is from the real-word environment. Different it, our CARD is capable of locating and describing all changed objects, which show a good generalization and robustness of our method. The superiority benefits from that the proposed CARD can provide the model with a overview of potential changed/unchanged semantics within an image pair, which helps learn genuine changes for caption generation.

heading: Future Work
text: In the future work, we will try to introduce linguistic knowledge (e. g. , syntactic dependencies between words) that regularizes the process of sentence generation, in order to generate the optimal sentences well elaborating the change process