Imagine How To Change: Explicit Procedure Modeling for Change Captioning
Authors: Jiayang Sun
Zixin Guo
Min Cao
Guibo Zhu
Jorma Laaksonen
Venue: ICLR 2026 Poster (2026)
Keywords: dynamic procedure understanding
confidence-guided sampling
change captioning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Change captioning generates descriptions that explicitly describe the differences between two visually similar images. Existing methods operate on static image pairs, thus ignoring the rich temporal dynamics of the change procedure, which is the key to understand not only what has changed but also how it occurs. We introduce ProCap, a novel framework that reformulates change modeling from static image comparison to dynamic procedure modeling.
ProCap features a two-stage design: The first stage trains a procedure encoder to learn the change procedure from a sparse set of keyframes. 
These keyframes are obtained by automatically generating intermediate frames to make the implicit procedural dynamics explicit and then sampling them to mitigate redundancy.
Then the encoder learns to capture the latent dynamics of these keyframes via a caption-conditioned, masked reconstruction task.
The second stage integrates this trained encoder within an encoder-decoder model for captioning.
Instead of relying on explicit frames from the previous stage---a process incurring computational overhead and sensitivity to visual noise---we introduce learnable procedure queries to prompt the encoder for inferring the latent procedure representation, which the decoder then translates into text. The entire model is then trained end-to-end with a captioning loss, ensuring the encoder's output is both temporally coherent and captioning-aligned. Experiments on three datasets demonstrate the effectiveness of ProCap. Code and pre-trained models are available at https://github.com/BlueberryOreo/ProCap.

Published as a conference paper at ICLR 2026
IMAGINE How TO CHANGE: EXPLICIT PROCEDURE MODELING FOR CHANGE CAPTIONING
Jiayang Sun Zixin Guo Min Cao'! Guibo Zhu Jorma Laaksonen
School of Computer Science and Technology. Soochow University, Jiangsu, China Department of Computer Science, Aalto University, Espoo, Finland Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China
*School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China WWuhan AI Research
Jysun028stu. suda. edu. cn mcao@suda edu.en
zixin. mixin.guc@aalto... fi gbzhuEn1pr. ac.
ABSTRACT
Change captioning generates descriptions that explicitly describe the differences between two visually similar images. Existing methods operate on static im- age pairs, thus ignoring the rich temporal dynamics of the change procedure, which is the key to understand not only what has changed but also how it oc- curs. We introduce ProCap, novel framework that reformulates change mod- eling from static image comparison to dynamic procedure modeling. ProCap features a two-stage design: The first stage trains procedure encoder to learn the change procedure from a sparse set of keyframes. These keyframes are obtained by automatically generating intermediate frames to make the implicit procedural dynamics explicit and then sampling them to mitigate redundancy. Then the encoder learns capture the latent dynamics of these keyframes via caption-conditioned, masked reconstruction task. The second stage integrates this trained encoder within an decoder model for captioning. Instead relying explicit frames from the previous stage- a process incurring computa- tional overhead and sensitivity to visual noise-we we introduce learnable procedure queries to prompt the encoder for inferring the latent procedure representation, which the decoder then translates into text. The entire model then trained end-to-end with a captioning loss, ensuring the encoder's output both tem- porally coherent and aeptioning-alignedd Experiments on three datasets demon- strate the effectiveness of ProCap. Code and pre- trained models are available com http://tw..a...cooo///eeeeeeeeee ott.
INTRODUCTION
Change captioning aims to generate textual descriptions that emphasize differences between two similar images. It has attracted growing interest due its wide applications, like monitoring tem- poral changes in remote sensing (Chouaf et al. 2021), supporting medical diagnosis by leveraging comparisons between abnormal and normal medical images (Bian al.) 2025), supporting urban planning via intelligent surveillance (Sun et al., 2024), and improving industrial quality control (Xie) ct al. 2024). Despite its promise, the task remains challenging due to (1) subtle appearance changes often being obscured by variations in viewpoint, illumination, background clutter, and (2) the dif- ficulty of transforming fine- gree-grrindd visual differences into coherent, accurate language descriptions.
To address them, existing methods follow encoder-decoder framework, where the encoder cap- tures visual differences and the decoder generates descriptive captions. Early works (Park et al.] Shi et al. 2020) model lixel-level differences via patch features, while later works (Qiu al., Yao et al. 2022 Tu et af. 2023c) introduce intricate difference extractors with alignment
2019
2021
Both authors contributed equally to this research.
Corresponding authors.

---

Published as a conference paper at ICLR 2026
mechanisms better localize change regions. More recently, the field has seen shift towards in- tegrating Large Language Models (LLMs) as decoders (Yang et al. 2023 Hu et al. 20244 Zhang et al. 2024), leading to substantial gains in caption quality. Furthermore. recent ad vancements in applying reinforcement learning to bolster L.LM reasoning (Peng et al. 2025 Wu et al. 2025b)
present promising avenue for further enhancing change captioning. Although promising. these methods typically focus on static image pairs, neglecting dynamic context and temporal cues critical for robust change perception. ate frames that capture rich spatio-temporal dynamics, explicitly revealing appearance and motion changes only implicitly encoded in the static pair (see Figure [T). Explicitly modeling this transition process thus offers a more principled basis for change understanding and captioning.
practice, the transition between images often involves intermedi-
Static Pair Modeling
The two scenes scem identical.
iitimta Priiioeeriann www. Dynamic Procedure Modeling
The tiny yellow nahber cylinder changed its location.
Figure 1:
Comparison between static image pair modeling and our proposed dynamic procedure modeling. Dynamic procedures offer temporal cues: the yellow cylinder, initially partly obscured by the green cube, changes its location.
In this work, we make the first attempt beyond change captioning on static image pairs by formulat- ing a procedure-modeling then-captioning paradigm. We explicitly model the dynamic change pro- cedure between static image pairs and perform captioning on the modeled spatio-temporal change procedure. We present ProCap, innovative two-stage framework: (1) explicit procedure mod- cling, which captures latent spatio-temporal dynamics between image pairs, and (2) implicit pro- cedure captioning, which generates rich descriptions by leveraging learnable queries implicitly reason over the modeled change procedure.
Explicit procedure modeling. Our framework models the underlying change procedure via three key components. Procedure Generation Module: This component synthesizes intermediate frames to transform the implicit transformation between input images into explicit and observable tem- poral sequence. However, the generated sequence tends to be dense and temporally redundant, often containing low-information content that incurs unnecessary computational overhead. Confidence- Based Frame Sampling Module: To address this, we introduce confidence-awand sampling module to distill the sequence into sparse set of informative keyframes. Each frame is assigned a con- fidence score based on temporal and semantic importance. By retaining only the highest- sighest-scoring frames, our module focuses learning on pivotal transition moments, thereby improving efficiency and representational quality during training. Procedure Modeling Module: Finally, we employ pro- cedure encoder to learn a compact latent representation the sampled keyframe sequence. We cast this caption-conditioned masked frame reconstruction task, where multi- cuulti-raanulaitty masking strategy- strategy-ranging from local patches to entire frames introduced. This encourages the model to capture aligned spatio-temporal dynamics across multiple scales, while mitigating overfitting to superficial visual cues and enhancing generalization procedural understanding.
Implicit procedure captioning. The key challenge for captioning lies in leveraging the procedu- ral knowledge learned by the procedure encoder for efficient and effective text generation. naive high computational
grrroachhggnerattnn and encoding intermediate frames at inference-ineut cost and introduces sensitivity to synthesis noise. To address them, we present the implicit proce- of learnable procedure queries between static image pairs, acting as "slots" to replace explicit intermediate frames. By leveraging the understanding of opatio-tempone dynamics learned during the first stage, the procedure encoder is prompted to infer the latent change
dure captioning, inserting
2

---

Published as a conference paper at ICLR 2026
procedure implicitly encoded within the image pair. The resulting procedural representation is de- coded into textual deseription, enabling end-to-end optimization via a captioning loss. This yields temporally coherent, task-aligned representation without costly frame synthesis at inference.
Our contributions are summarized: (1) We introduce ProCap, a two-stage framework reformulat- ing change captioning from static comparison to dynamic procedure modeling, directly addressing limitation of prior works: their reliance on static image pairs, overlooking rich temporal dynam- ics. (2) We propose explicit procedure modeling, where procedure encoder is trained on sampled keyframes from a synthesized explicit procedure, with a caption-conditionn
masked reconstruc- tion task to capture change dynamics. (3) We develop implicit procedure captioning, introducing learnable queries to enable the encoder to model the procedure implicitly, bypassing the costly and noise-prone frame synthesis inference for efficient and effective captioning.
2 RELATED WORK
Existing change captioning methods primarily operate on static image pairs, treating the task as a spatial comparison problem. Pioneering works by Jhamtani & Berg- Kirg-KKikpttrikk (2018) and Park
et al. (2019) establish a foundational encoder-decoder framework. Subsequent efforts enhance this
static comparison with two main paradigms: (1) designing intricate change encoders for fine- -ra-indi
localization and robustness to distractors like viewpoint or illumination changes (Kim et al. 2022] Tu
al. 2023a Yue et al. 2023 Tu al. 2024a Li et al. 2025 Hu et al.
2025 Zhong et al. 2025); )
and (2) adopting advanced training strategies, such as auxiliary retrieval tasks (Hosseinzadeh &
Wang 2021) or multi-stage alignment (Guo et al. 2022; Yao et al. 2022 Rahmanzadehgervi al.,
2025). to guide the learning process, Beyond model design, recent vision- Laiionla studies (Menon & Vondrick 2022 Pratt et al., 2023] Guo et al. 2023) have increasingly explored prompt-driven methodologies to harness lcage-scale data. Inspired by these advancements, recent studies like Liu cet al. (2025) and Di et al. (2025) have focused on constructing large-scale, high-quality datasets to further advance the field. Furthermore, to mitigate hallucinations arising from large-scale datasets, Guo et al. (2025) explores a noise-robust pre- pre-training framework for change captioning. Although promising. these methods infer changes directly from "before" and "after" images, ignoring the un- derlying continuous and dynamic transition process. In contrast, we propose to explicitly model the change procedure, shifting the paradigm from spatial comparison spatio-temporal procedure modeling. We argue that the intermediate sequence contains rich temporal dynamics critical for ro- inherently missing in static pairs. While recent advances in video understanding have focused on improving temporal grounding in LLMs through explicit frame identifiers (Wu al., [2025a). the application of such dynamic modeling in change caption- ing remains underexplored. The most closely related work is Zhu et al. (2025), which implicitly models temporal dynamics in remote sensing using -specff change maps. Our approach differs fundamentally: (1) we explicitly generate and model intermediate transitions to reason about how changes unfold, enhancing dynamic representation; and (2) we eliminate reliance on domain- specific supervision, enabling generalization to complex, unconstrained natural scenes. Additional related work, particularly on frame interpolation, is included in Appendix B
bust change inweersttnddnggnnfrmmmaeee
3 METHODOL.OGY
Relying solely on two static images, existing methods neglect the rich spatio-temporal procedure that connects an image pair. Our key insight is that such procedure is crucial for understanding not only what has changed but also how it occurs, thereby improving the change dynamics mod- eling. Given an image pair (Iuet, lan)) containing objects O {or, 02..... on.) each object o, is represented by threc continuous attributes (p;, a;, w;) corresponding to position. appearance, and existence, respectively. valid change procedure with respect to a change caption T formalized as a mapping "Tr [0. 1] + I, where I denotes the space of all possible images, satisfying: (1) boundary conditions Yr = lnef and (1) = Iati (2) continuous evolution of each object's at- tributes (p., 0;, w;) over time f, such that (t()) = (p)((), (t), w; (()) (3) consistency with the semantic constraints imposed by caption T; and (4) invariance unchanged objects throughout the process. Our objective is to derive an informative sequence P I that approximates '2. To address this, we introduce procedure modeling for captioning (ProCap), illustrated in Figure Our proposed ProCap formulates change captioning as a two-stage learning process: (1) explicit proce-
3

---

Published as a conference paper at ICLR 2026
Training Explicit Procedure Modeling
Training 2: Implicti Procedure Captioning
Fresddars Comeratio www.aaiiaoeaaeoooeeoo
Cuekleen Prose Aumphay Vodier
Frane:
Inoegelativn Medfll
Image Festune
Fartiddmoo Soon
Aut
Frwender Maaking Markk
Procedure Queries
Boniy yohree ofadd daaae
Imsgee Fisture Esiractot
*
K2
Ficco
Eimbeddling Layer
graniarre Maskisz
Ioug:
Ø Mark
Procedure Bncoder
IZA
Procedure Encoder
Drooder
[ALl]
Alpmnc
Miskal Sequenee Moscling lind
the yellow rubber cyllider changed its location
Our two-stage ProCap framework. In the first stage, Explicit Procedure Modeling, procedure encoder learns change dynamics from keyframes sampled from the generated explicit procedure frames. In the second stage, Implicit Procedure Captioning, learnable procedure queries, instead of explicit frames, prompt the encoder to infer an implicit representation for captioning.
Figure 2:
dure modeling stage learning capture the latent dynamics of the change procedure, and (2) implicit procedure captioning stage learning to generate descriptions based on the modeled procedure.
3.1 EXPLICIT PROCEDURE MODELING
This stage incorporates three key components: procedure generation module that produces con- tinuous frames between given static image pairs, a confidence-based frame sampling module that identifies and selects keyframes from the produced continuous frames, and procedure modeling module that models the latent change dynamics these keyframes.
3.1.1 PROCEDURE GENERATION MODULE
The first step is to make the change procedure explicit. To achieve this, we employ pre-trained, off-the-shelf Frame Interpolation (FI) model (Luu al.) (002) synthesize the procedure. Given an image pair (Tbe,, Jan), the FI model first uses a CNN to predict bidirectional optical flows (Or-heef ). which are applied to the images and their features to generate warped image pairs (Tee,, Jan) and warped feature pairs (Fret, Fan). These warped pairs, along with the original images, then fed into Transformer that produces soft mask H and image residual A/, The intermediate frame synthesized J H Toar+ H) @ Jan + AT, where denotes the Hadamard product. Typically, I, represents an intermediate state within the overall change pro- cedure. To construct sequence of gpeudo-framed explicit procedure:
recursively apply the FI model, yielding
p = FI an) = (T,, (2,,2...... J2,...,T4 Tt)-
However, the generated dense sequence PPl is not optimal for direct modeling (Appendix M.2 Fig- ure The primary challenge inherent temporal redundancy. Owing to the recursive nature the synthesis process, redundancy in single intermediate bles the input images propagates to adjacent regions of the sequence, thereby providing minimal novel information about the change. Modeling this entire sequence is not only computationally in- efficient but also risks diluting the critical moments of the change with trivial, redundant frames. Therefore, distilling the sequence into sparse set of keyframes that relatively more informative about the change dynamics is critical for efficient and effective procedure modeling.
caanme-prrtccllatte when it closely resem-
3.1.2 CONFIDENCE-BASEE BASED FRAME SAMPLING MODULE
To achieve this, introduce confidence-barse frame sampling module. It identifies and selects the keyframes using a sccore-then-sampee strategy. Specifically, each frame in pil is assigned a confidence quantifying its informativeness, which then guides the sampling process.
4

---

Published as a conference paper at ICLR 2026
Score. To quantify frame's informativeness, formalize the intuition that the more critical frames are those that represent the semantic midpoint of the change- the point where a frame is semantically equidistant from the initial (Terr and final (Lan) states. Conversely, frames that are highly similar to either endpoint are generated set PFI, compute the confidence score vector
Given the image pair (Teer, Tan) and the
(2) where (-) is the softmax function, and s(..)) is a semantic similarity function. The squared differ- ence term ensures that frames are penalized regardless of which endpoint they are closer to. This score assigns high values to frames that semantically equidistant from the start and end im- ages, effectively identifying the "peak" of the change.
a( [s(Thee PF) <Tan, s(Tnn,Pi))3))
We explore two strategies to compute the similarity s(.,,) leveraging different sources of informa- tion: (1) visual-only: based solely on visual frames, and (2) visual-text: incorporating both visual frames and textual change caption. Further details of these strategies presented in Appendix C
Sample. Guided by the confidence score vector w, we then sample sparse subset of & keyframes, p = [/ ] C phl, This sampled set prepended with lbef and appended with an to construct the procedure P:
P= ( 1 15 I Jan-
(3)
This sequence serves as the input for the procedure modeling module, which learns to encode the dynamics of the change.
3.1.3 PROCEDURE MODELING MODULE
The core of our method is the procedure modeling module, designed to learn rich, unified repre- sentation of the ppatio-tempora dynamics within the procedure list P. Inspired by Han al. (2022). we employ a Transformer-basee encoder as the backbone to model the change procedure, and utilize a pre-trained image tokenizer (Esser et al., 2021) quantize into discrete tokens, which serve as the targets for our masked fulti-frame reconstruction objective. This encourages the model to infer missing spatio-temporal information, ensuring a deep understanding the procedural dynamics.
Input representation. To prepare the encoder input, we first create a multi-modal sequence.
Visual stream: Each frame in the sequence P (length k++2) is passed through a frozen CNN backbone (Esser et al.) 2021) extract grid of ny patch-level visual features with d-dimensional vectors. yielding sequential embeddings e E IR (R(k=2)nxxd
Textual stream: Concurrently, the corresponding change caption is tokenized into ny tokens and embedded into sequential embeddings e Rey where each token is embedded in d-dimensional space.
Special tokens: To structure this sequence for joint modeling. we prepend two learnable token embeddings: e E IRd to e' to capture frame consistency, and eilign E IRd to e" to facilitate visual-textual alignment.
The concatenated input embeddings are (eil "e e ee..
granuiigranulrrity masking. To learn both coarse-grained semantics and fine-grained details, we introduce multi-granularity masking strategy. This strategy is applied to the visual patch embed- dings e', while leaving the caption fully visible. This encourages the encoder to learn the underlying ppatio-temporal dynamics by reconstructing masked regions under textual guidance. The strategy comprises four distinct masking schemes. The first operates at a coarse, frame-level granularity, while the remaining three focus fine-grained, patch-level details:
Entire masking masks the entire frame embeddings. It forces the encoder to reconstruct them using cross-modal context from the change caption.
Random patch masking masks individual patches across the frames. This strategy encourages
the encoder to learn distributed visual representations.
In-block masking (Tan al.) (221)) masks contiguous rectangular block of patches. This forces the encoder to learn the appearance and texture of local regions by "filling in" the masked area from its surrounding context.
5

---

Published as a conference paper at ICLR 2026
Out-of-block masking (Tong et al., 2022) masks all patches "outside" specific block. This encourages the encoder to learn how to represent a region while understanding its relationship to the broader surrounding scene.
During each training step, every visual stream within the batch is independently masked using one of four randomly selected guatuligaaulrrtty strategy. This selected strategy then generates a binary mask index set . E R (k++2)),, where a value of indicates a patch to be masked. The chosen patch embeddings in e' are replaced with learnable mask embedding em E Rd, creating the masked visual sequence emsk- This sequence then fed into the procedure encoder, which outputs sequence of contextualized hidden states mi for subsequent optimization:
mmsk (halign h²" h llnsk).
(4)
Details on the formulation of these masking operations are provided in Appendix D
3.1.4 OPTIMIZATION
The objective of procedure modeling, LPRO, comprises three components: (1) masked sequence modeling for reconstructing the masked regions in each frame, Lmmm's (2) cross-modal alignment between the visual frames and the textual change caption, Laligni and (3) temporal consistency within the procedure sequence, Lesy. The overall training objective is formulated as:
Pro = Lmsm + Lalign + -osy-
(5)
Masked sequence modeling. We leverage pre-trained image tokenizer (Esser et al. 2021 to tokenize each frame in procedure sequence P into n, discrete tokens. This tokenization process yields a corresponding discrete token sequence E IR (k+2),, serving as the ground truth. Given the modeled masked frame representations hysk from Eq. 44. we apply linear projection layer as the masked sequence modeling head map each position the representation sized logits vector, yielding the predicted token sequence " E [K++2)n, Given the masked frame embeddings enst and the change caption T,, the masked sequence modeling loss is defined as:
vocabulary-
mmsm
logr(s, 2z e e²))
[
sEInk
where Imsk MM 1} denotes the index set of positions masked in multi-granulariin masking.
Cross-modal aligmment. We incorporate an alignment loss to bridge the visual change procedure and its corresponding textual change caption. Using the special token representation haiggn which captures the relevance between visual and linguistic modalities as defined in Eq. ( we optimize the encoder effectively differentiate between aligned and non-aligned caption-procedde pairs:
Catign = log c/nat e - log p(0 c/nat e e²),
where the change caption paired with frame embedding e and T is a negative sample not aligned with el
Temporal consistency. To mitigate the impact of temporal incoherence on the modeled procedure, we incorporate a consistency loss that encourages coherent representations across frames in the sequence. Using the special token representation h that captures frame consistency from Eq. (4) we optimize the encoder to differentiate between consistent and non-consistent frame sequences:
Las = pog e e²)) - loo p(0 c e²),
(8)
where I denotes the temporally warped version of I, intentionally disrupting the temporal consis- tency of the sequence. This warped version serves negative sample, encouraging the model to learn to distinguish between temporally coherent and incoherent sequences and generate more temporal coherent sequences. More details about constructing / are provided in Appendix E

---

Published as a conference paper at ICLR 2026
3.2 IMPLICIT PROCEDURE CAPTIONING
The captioning stage employs encoder-decoder architecture, with the encoder sharing weights with the procedure encoder from the prior stage. Directly leveraging the synthesized intermediate frames for captioning may introduce additional computational overhead well irrelevant noise. Therefore, wc propose learnable procedure queries as dynamic "slots" inserted between start and end image features, which guide the encoder to implicitly infer change dynamics from the image pair. This design supports end-to-end training via captioning loss and yields encoder outputs that both temporally coherent and task-relevant.
Processing. First, we process the image pair (Tbel, Ian) with a CNN backbone to extract their respective visual patch features, e Rorxd and e/a E R To bridge these two static representations, we introduce learnable procedure queries that replace the k sampled intermediate frames within the previous stage. Since each frame represented by ny patch features, we insert sets of queries, where each set contains n learnable embeddings. This results queries. Each of these queries learnable masked embedding (em) used in the previous stage. The input sequence for the procedure encoder is constructed as follows:
total of
e inp = (clha "en,, " Cm, e
The procedure encoder processes i"P to produce representations that capture the underlining dy- namic change procedure. Given these encoded representations, Transformer-bar -base textual decoder then learns to generate the change caption.
Optimization.
The objective of captioning, LCAP, is an autoregressive language modeling loss that trains the entire model using ground-truth change captions as supervision:
LCAP Σ
PTT Tei; e)
(10)
where T; denotes i-th word the caption sequence.
3.3 PROCAP INFERENCE FOR CAPTIONING
For inference incoming image pair, the procedure encoder takes inp in Eq. the input. The output, a latent procedural representation, is then translated into the text caption by the textual decoder. Compared other approaches, ProCap introduces only additional n-n matrix for pro- cessing. As n, remains fixed throughout our experiments, the variation in computational overhead primarily governed by k. With & this overhead is negligible, and we further analyze its effect in the following experiments.
4 EXPERIMENTS
4.1 DATASETS AND METRICS
Datasets. We conduct experiments three widely-used benchmark datasets: CLEVR-
Change (Park et al. 2019). Spot-the-Diff (Jhamtani Berg-Kirkpatrick 2018), and Image-Editing:
Request (Tan al. 2019 These datasets cover a diverse range of change domains, from synthetic changes CLEVR-Changee to subtle differences in natural scenes (Spot-the-Diff and Image-Editing- Request), allowing for comprehensive evaluation of our model's capabilities. Additional details about dataset introduction are presented in Appendix
Metrics. To evaluate the quality of the generated captions, report four standard metrics: BLEU- (B) (Papineni et al. 2002), METEOR (M) (Banerjee & Lavie, 2005). ROUGE-L (R) LLn 2004). and CIDEr (C) (Vedantam al. 2015). All scores obtained using the official Microsoft COCO evaluation toolkit (Chen et al. 2015). To evaluate the trade-off between captioning accuracy and our procedure modeling efficiency, we also measure inference efficiency in Tokens Per Second (TPS).

---

Published as a conference paper at ICLR 2026
Table 1: Comparison with SOTA CLEVR-Change, Spot-the-Diff and Image-Fditing-
CLEVR-Change Mt
Spot-the-Diff M
Image-Fditing-i
Methods
B
R
C
B
M1 R
Ct
LLM based Methods
Qwen-Vi VL 2023
48.9
36.0
71.2
119.8
LLaVA-1.5 2025
49.7
35.4
70.8
122.4
VIXEN 2024
8.6
15.4
42.5
38.1
FINER (2024
55.6
36.6
72.5
137.2
12.9
14.7
61.8
13.3
14.6
39.6
50.5
LLaVA-
I.SRRP -LAVA-TT+RP
2025
9.7
13.0
30.8
43.2
16.2
19.5
46.7
60.9
Non I.M based Methods
DUDA 2019
47.3
33.9
112.3
9.1
11.8
32.5
6.5
12.4
37.3
22.8
DUDA AuX
2021
51.2
37.7
70.5
115.4
8.1
12.4
31.3
38.1
2021
69.1
NCT
20Z5B
55.1
40.2
73.8
124.1
8.1
15.0
38.8
34.2
VARD- Trans 2023a
55.4
40.1
73.8
126.4
10.0
14.8
39.0
35.7
SCORER 12023g
56.3
74.5
126.8
10.2
10.0
15.0
39.6
33.4
MURAT+GCM
2024
10.2
13.1
33.1
39.4
SMART 2024b)
56.1
40.8
74.2
127.0
13.5
31.6
39.4
10.5
15.2
39.1
37.8
DIRL+CCK (2124a
34.1
RDD+ACR 2025
56.1
41.3
75.0
128.1
9.2
13.9
31.0
43.6
MCT-CCDif (2025
57.5
40.6
75.6
131_7
10.8
14.5
35.5
41.7
10.2
15.4
41.2
38.3
ProCap (Ours)
56.7
41.7
74.7
135.6
11.0
13.6
42.7
11.7
15.9
43.2
40.6
4.2 PERFORMANCE COMPARISON
4.2.
BASELINES
We compare ProCap against a set of state-of-the-arr methods, which are grouped into two cate- gories: 1) Non-LLM-based methods are the conventional paradigm, where pre-trained CNN extracts visual features from input image pairs. These features are then fed into Transformer- based encoder-decoder for captioning. We compare method with: DUDA (Park et al.) 2019) DUDA+Aux (Hosseinzadch & Wang 2021), IFDC (Huang et al.) 2021). NCT (Tu et al. 2023b) VARD-Trans (Tu et al. (023a), SCORER+CBR (Tu et al. 2023c). MURAT+GCM Yue et al. 2024). SMART (Tu et al. 2024b), DIRL+CCR (Tu et al. 2024a). RDD+ACR (Li et al., 2025) and MCT-CCDiff (Hu al.) 2025). 2) LLM-based methods leverage LLMs powerful decoders, capitalizing on their vast knowledge and strong generative capabilities to improve caption quality. 2023), LLaVA-1.5 (Liu al. 2023), VIXEN-
We compare our method with: Qwen-VL (Bai
C Black et al, 2024), FINER (Zhang et al. (0024) and LLaVA-1.5+RP (Jiao et al.
2025
Our ProCap falls into the non-LLM-based category. While LLM-based methods benefit from rich prior knowledge, they typically entail heavy computation and large parameter sizes. In contrast, ProCap achieves strong performance with a lightweight, efficient architecture, avoiding reliance on large external language models.
4.2.2 RESULTS
We analyze ProCap's performance across three challenging scenarios, each testing specific ca- pability, in Table I Additional qualitative comparisons with state-of-the-art methods, extensive visualizations and case studies, are provided Appendix Mo illustrate ProCap effectiveness.
Robustness to viewpoint changes. First. we evaluate ProCap's robustness to viewpoint shifts on the CLEVR-Change dataset. As shown in Table I ProCap substantially outperforms all non-LLM methods on CIDEr and achieves competitive results on other metrics, indicating stronger semantic understanding. This improvement stems from our procedure modeling, which disentangles object transformations (the "what"" of change) from camera movements (distractors) by analyzing the full transition path. Compared with L-Med methods, ProCap surpasses Qwen-VI and LLaVA- and even outperforms FINER most metrics, demonstrating strong reasoning capability without relying on large-scale decoders. A detailed comparison across different change categories is pro- vided in Appendix
8

---

Published as a conference paper at ICLR 2026
Application to multiple changes in complex scenes. Next, we evaluate ProCap the Spot-the- Diff dataset, a more challenging real-world benchmark with cluttered scenes and multiple subtle changes. As shown in Table I ProCap achieves competitive CIDEr score of 42.7. This demon- strates a key strength of our approach: by modeling change as a stepwise procedure, ProCap can "re- play" the transformation process disentangle concurrent changes and generate accurate captions. To better capture the rich dynamics in this setting, the frame interpolation module is pre-trained on a specialized video dataset (Oh al. 2011) before ProCap's main training.
Generalization to open-ended scenarios. Finally, we assess ProCap's generalization abilities the Image-Editinge -editing-Requesst dataset, which is characterized by its open-ended nature with largely un- seen vocabulary. The results Table [ show that ProCap consistently outperforms all non-LLM baselines across all metrics, This suggests that by learning the "how" of a change (the procedure), our model develops a core understanding of the transformation itself, making more resilient to variations in vocabulary and phrasing. While the LLM- basdd LLaVA-1.5+BB edge base, still leads in overall accuracy. ProCap significantly narrows the performance gap. This demonstrates that procedural modeling is a powerful strategy for achieving robust generalization. It highlights a key distinction: whereas LLM-based methods obtain generalization by infusing external knowledge, ProCap's ability stems directly from its architectural innovation.
with its vast knowl-
4.3 ABLATION STUDY
We study the impact of key components within the procedure modeling stage. Additional ablations on other components are detailed in Appendices
Table 3: Effectiveness and performance com- parison CLEVR- CEEnn-h dataset with vary- ing procedure query set length k.
Table 2: Ablation study for explicit procedure modeling (EPM) and implicit procedure cap- tioning (IPC) on SLEVR-Chang. dataset.
EPM IPC
k
B
1
1
R
C1
Methods
k
TPS
1
M1
R 11
Ct
0
47.2
35.8
68.6
108.4
766.02
56.5
41.9
75.5
128.5
52.6
38.0
70.1
112.7
ProCap
2
699.04
56.7
41.7
74.7
135.6
47.3
36.3
68.8
106.2
4
461.24
57.4
42.3
75.5
128.7
56.5
41.9
75.5
128.5
7
270.55
56.8
41.8
75.5
130.5
Impact of introducing explicit procedure modeling and implicit procedure captioning. Ta- analyzes the introduction of the explicit procedure modeling stage. We begin with a baseline encoder-decoder model trained on static image pairs from scratch. We then compare two enhance-
ments to this bascline: (l) applying a pre-training stage (explicit procedure modeling), and (2) introducing set of learnable procedure queries enable implicit procedure captioning (implicit procedure captioning). Finally, we extend the model with both pre-training and learnable procedure queries. Compared to the baseline initialized randomly, applying the learnable queries directly (line 3) introduces random vectors of learnable queries, therefore lacking any temporal or procedural con- text. In this case, the model cannot effectively reason about the evolution from the "before" the "after" image. Besides, applying explicit procedure modeling without the learnable queries (line 2) demonstrates that pre-training alone provides only limited gains, far smaller than the improvement observed when both pre-training and learnable queries are used together (line 4), with the CIDEr score significantly increasing 128.5. This remarkable gain highlights our key insight: explicitly modeling the procedural dynamics of change is far more effective than simply comparing static im- age pairs. Notably, Table 14 in Appendix D further presents the advantages of implicit procedure captioning reducing computational overhead and exhibiting greater robustness to visual noise, once the explicit modeling stage has provided the rich temporal understanding.
Impact of procedure query set length k. TableBishows the effect of varying the procedure query set length & on both accuracy and computational efficiency (using one NVIDIA A40 GPU). Overall, efficiency decreases as the sequence length increases due to the heavier computational load. Consid- ering the overall performance across the four evaluation metries, C reaches its peak value of 135.6 at = 2. while the other metrics exhibit a non mon-oooooiiic trend. Although the model achieves its the TPS drops substantially. Therefore, select k 2 as
best scores B, M, and R

---

Published as a conference paper at ICLR 2026
it offers the optimal balance between capturing sufficient procedural detail for accuracy and main- taining computational efficiency. We further compare the performance and effectiveness of different procedure query set lengths with LLM-based methods in Appendix K.4
Table 4: Ablation study for combinations of training objectives in explicit procedure modeling on CLEVR-Chan Change and Spot-the-Dire Diff.
FLEVR-Change
Spot-the-DDe Diff-
Lisan
Lalign
Lay
C
B
M
R
C
55.1
40.6
73.9
127.5
8.1
11.8
28.1
29.7
55.5
40.6
73.8
127.1
7.9
11.7
28.0
28.9
56.1
40.9
74.5
128.6
9.3
12.5
31.2
36.3
56.7
41.7
74.7
135.6
11.0
13.6
33.7
42.7
Integration all the objectives. Table44 the contribution of each objective function from Eq. () within this stage. Building on the foundation of Lmmms the full model-jointly optimized with all objectives- achieves peak performance, reaching a CIDEr score of 135.6 on the CL.EVR-Chan dataset and 42.7 the Spot-the-Dii tieffifff dataset. This corresponds improvements of 8.5 CLEVR- Change and 13.8 on Spot-the-Diff when removing Calign. and gains 7.0 and 6.4 on the two datasets when removing Luy.. This further improvement highlights the integration of the other two losses, each targeting a specific aspect of the procedure representation: The alignment loss (Catign) acts as a crucial bridge, grounding the visual procedure representation in the linguistic domain. It explicitly enforces that the learned procedure is not just visually coherent, but also semantically aligned with its corresponding textual description. Meanwhile, the consistency loss (Lesy ensures the temporal order of the procedure, specifically penalizing temporally incoherent (e.g., shuffled) sequences. This forces the model to be sensitive to the correct order of events within the change.
CONCLUSION
In this paper, we introduce ProCap, a novel two-stage paradigm that shifts change captioning from modeling static image comparison the dynamic change procedure. The first stage learns pro- cedure encoder that models change dynamics by performing caption-condittooed masked recon- struction on sparse set of intermediate frames, distilled from the synthesized explicit procedure. The second stage, captioning, introduces efficient and learnable procedure queries represent the implicit process within the image pair. This design enables end-to-end training without costly inter- mediate frame synthesis during inference. Experiments across diverse datasets demonstrate ProCap effectiveness.
ACKNOWLEDGEMENTT
This work is supported by the National Natural Science Foundation of China under Grants 62476188, the National Key R&D Program of China (No. 2022ZD0160601,, and the Key Lab- oratory of Computing Power Network and Information Security, Ministry of Education under Grant No.2024PY024.
REFERENCES
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile Dision-language model for understanding, local- ization, text reading, and beyond, 2023.
Satanjeev Banerjee and Alon Lavie. Meteor:
An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Minghao Bian, Kun Zhang, Dexin Zhao, and S Kevin Zhou. DiffRGennet: Difference-aware medi- cal report generation. In Medical Imaging with Deep Learning, 2025.
10

---

Published as a conference paper at ICLR 2026
Alexander Black, Jing Shi, Yifei Fan, Tu Bui, and John Collomosse. Vixen: Visual text comparison network for image difference captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 846-854, 2024.
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015.
Seloua Chouaf, Gene Hoxha, Youcef Smara, and Farid Melgani. Captioning changes in bi-temporal remote sensing images. In 202/ IEEE International Geoscience and Remote Sensing Symposium IGARSS, pp. 2891-2894. IEEE, 2021.
Zonglin Di, Jing Shi, Yifei Fan, Hao Tan, Alexander Black, John Collomosse, and Yang Liu. Difftell: A high-quality dataset for describing image manipulation changes. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 25580-24590, 2025.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x words: Transformers for image recognition at scale. In International Conference Learning Reprexentations, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 12873-12883, 2021.
Tsu-Jui Fu, Licheng Yu, Ning Zhang, Cheng- Fu, Jong-Chy's Su, William Yang Wang, and Sean Bell. Tell me what happened: Unifying text-guided video completion via multimodal masked video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10681-10692, 2023.
songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vggan and time-sensitive transformer. In European Conference on Computer Vision, pp. 102-118. Springer, 2022.
Zixin Guo, Tzu-Jui Wang, and Jorma Laaksonen. Clip4ide: Clip for image difference captioning. In Proceedings of the 2nd Conference of the Asia Pic--Pccific Chapter of the Association for Computa- tional Linguistics and the 12th International Joint Conference on Natural Language Processing, pp. 33-42, 2022.
Zixin Guo, Tzu-Jui Julius Wang, Selen Pehlivan, Abduljalil Radman, and Jorma Laaksonen. Pitl: Cross-modal retrieval with weakly-supervised vision-language pre-training via prompting. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2261-2265. 2023.
Zixin Guo, Jiayang Sun, Tzu-Jui Julius Wang, Abduljalil Radman, Selen Pehlivan, Min Cao, and Jorma Laaksonen. Learning to describe implicit changes: Noise-robust pre-training for image difference captioning. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 10125-10145, 2025.
Ligong Han, Jian Ren, Hsin-Yon Ying Lee, Francesco Barbieri, Kyle Olszewski, Shervin Minaee, Dim- itris Metaxas, and Sergey Tulyakov. Show what and tell me how: Video synthesis via mul- timodal conditioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3615-3625, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Mehrdad Hosseinzadeh and Yang Wang. Image change captioning by learning from auxiliary task. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2725-2734, 2021.
Erdong Hu, Longteng Guo, Tongtian Yue, Zijia Zhao, Shuning Xue, and Jing Liu. Onediff: A generalist model for image difference captioning. Proceedings of the Asian Conference on Computer Vision pp. 2439-2455, 2024.
11

---

Published as a conference paper at ICLR 2026
Jinhong Hu, Guojin Zhong, Jin Yuan, Wenbo Pan, and Xiaoping Wang. Met-cediff: Context-aware contrastive diffusion model with mediator-bridging cross-modal transformer for image change captioning. IEEE Transactions on Image Processing, 2025.
Qingbao Huang, Yu Liang. Jielong Wei, Cai Yi, Hanyu Liang, Ho-fung Leung, and Qing Li. Image difference captioning with instance-level fine-gratd one-grained feature representation. IEEE Transactions on Multimedia, 2021.
Junhwa Hur, Charles Herrmann, Saurabh Saxena, Janne Kontkanen, Wei-Sheng Lai, Yichang Shih, Michael Rubinstein, David J Fleet, and Deqing Sun. High-resolution frame interpolation with patch-based cascaded diffusion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 3868-3876 2025.
Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling, 2022.
Harsh Jhamtani and Taylor Berg-Kirkpatriie Learning to describe differences between pairs of sim- ilar images. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4024-4034, 2018.
Qirui Jiao, Daoyuan Chen, Yilun Huang, Bolin Ding, Yaliang Li, and Ying Shen. Img-diff: Con- trastive data synthesis for multimodal large language models. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 9296-9307, 2025.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Hoeseong Kim, Jongseok Kim, Hyungseok Lee, Hyunsung Park, and Gunhee Kim. Agnostic change captioning with cycle consistency. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2095-2104, 2021.
Rong Li, Liang Li, Jiehua Zhang, Qiang Zhao, Hongkui Wang, and Chenggang Yan. Region-aware difference distilling with attribute-guided contrastive regularization for change captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 4887-4895, 2025.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 34892-34916, 2023.
Yuan Liu, Saihui Hou, Saijie Hou, Jiabao Du Shibei Meng, and Yongzhen Huang. Omnidiff: A comprehensive benchmark for fine- imee-graned image difference captioning. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11140-214444 October 2025.
Liying Lu, Ruizheng Wu, Huaijia Lin, Jiangbo Lu, and Jiaya Jia. Video frame interpolation with transformer. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recog- nition, pp. 3532-3542, 2022.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXhv:2210.071833 2022.
Sangmin Oh, Anthony Hoogs, Amitha Perera, Naresh Cuntoor, Chia-Chih Chen, Jong Taek Lee, Saurajit Mukherjee, Jake K Aggarwal, Hyungtae Lee, Larry Davis, et al. A large-scale benchmark dataset for event recognition surveillance video. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3153-3160. IEFE, 2011.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077/93, 2023.
12

---

Published as a conference paper at ICLR 2026
Enrico Pallotta, Sina Mokhtarzadeh Azar, Shuai Li. Olga Zatsarynna, and Juergen Gall. Synevp: Joint diffusion for synchronous multi-modal video prediction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13787-13797. 2025.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 31 1-1188 2002.
Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust change captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4624-4633, 2019.
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning 2025.
abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.072:
Sarah Pratt, lan Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? gener- ating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15691-15701, 2023.
Yue Qiu, Shintaro Yamamoto, Kodai Nakashima, Ryota Suzuki, Kenji Iwata, Hirokatsu Kataoka, and Yutaka Satoh. Describing and localizing multiple changes with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1971-1980, 2021.
Pooyan Rahmanzadehgervi, Hung Huy Nguyen, Rosanne Liu, Long Mai, and Anh Totti Nguyen. Tab: Transformer attention bottlenecks enable user intervention and debugging in vision-language models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 22551-22561 2025.
Xiangxi Shi, Xu Yang, Jiuxiang Gu, Shafiq Joty, and Jianfei Cai. Finding it at another side: A viewpoint-adapted matching encoder for change captioning. In European Conference on Com- puter Vision, pp. 574 590. Springer, 2020.
Yanjun Sun, Yue Qiu, Mariia Khan, Fumiya Matsuzawa, and Kenji Iwata. The stychrono dataset: Towards continuous change recognition in time, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1411 11111-18 1-14220, 2024.
Hao Tan, Franck Dernoncourt, Zhe Lin, Trung Bui, and Mohit Bansal. Expressing visual relation- ships via language. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pp. 1873-1883, 2019.
Hao Tan, Jie Lei, Thomas Wolf, and Mohit Bansal. Vimpac: Video pre-training via masked token prediction and contrastive learning. 2021.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. Advances in neural information process- ing systems, 35: 15:10078-1009. 2022.
Yunbin Tu, Liang Li, Li Su, Junping Du, Ke Lu, and Qingming Huang. Adaptive representation disentanglement network for change captioning. IEEE Transactions on Image Processing, 32: 2620-2635, 2023a.
Yunbin Tu, Liang Li, Li Su, Ke Lu, and Qingming Huang. Neighborhood contrastive transformer for change captioning. IEEE Transactions on Multimedia, 2023b.
Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, Chenggang Yan, and Qingming Huang. Self-supervised cross-view representation reconstruction for change captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2805-2815, 2023c.
Yunbin Tu, Liang Li. Li Su, Chenggang Yan, and Qingming Huang. citractors-immmnn represen- tation learning with cross cross-modal modal contrastive regularization for change captioning. European Conference Computer Vision, pp. 3113288. Springer, 2024a.
13

---

Published as a conference paper at ICLR 2026
Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, and Qingming Huang. Smart: Syntax- calibrated multi- aspect relation transformer for change captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence, (6(7):49264494)) 2024b.
Syntax-calibrated
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-baud based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Vikram Voleti, Alexia Martoouurr,Mtnnuuuu and Chris Pal. Mevd-masked conditional video diffusion for prediction, generation, and interpolation. Advances in neural information processing systems, 35-23371-23333 2022.
Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, and Xu Yang. Number it: Temporal grounding videos like flipping manga. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13754-13765, 2025a.
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508. 05629, 2025b.
Jiayuan Xie, Zhiping Zhou, Zihan Wu, Xinting Zhang, Jiexin Wang, Yi Cai, and Qing Li. Automated defect report generation for enhanced industrial quality control. Proceedings of the AAAI Confer- ence on Artificial Intelligence, 38(17):1930--19314 Mar. 2024. doi: 10.1609/aaai. www.w..281111000..
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers, 2021.
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 30XV::2505.093888, 2025.
Xu Yang. Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng. Exploring diverse in- context context configurations for image captioning. Advances in Neural Information Processing Systems, 36:40924-40943, 2023.
Linli Yao, Weiying Wang, and Qin Jin. Image difference captioning with pre-training and contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 3 3108- 3116, 2022.
Shengbin Yue, Yunbin Tu, Liang Li, Ying Yang. Shengxiang Gao, and Zhengtao Yu. 13n: Intra-and inter-representtatio interaction network for change captioning. IEEE Transactions on Multime- dia, 2023.
Shengbin Yue, Yunbin Tu, Liang Li, Shengxiang Gao, and Zhengtao Yu. Multi- represen- tation aggregating transformer with gating cycle for change captioning. ACM Transactions on Multimedia Computing, Communications and Applications, 2024.
Guozhen Zhang, Yuhan Zhu, Yutao Cui, Xiaotong Zhao, Kai Ma, and Limin Wang. Motion-aware aware generative frame interpolation. arXiv preprint arXiv:2501. 033699 2025a.
Xian Zhang, Haokun Wen, Jianlong Wu, Pengda Qin, Hui Xue', and Liqiang Nie. Differential- perceptive and retrieval- augmented mllm for change captioning. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 4148-4157 2024.
Zihao Zhang, Haoran Chen, Haoyu Zhao, Guansong Lu, Yanwei Fu, Hang Xu, and Zuxuan Wu. Eden: Enhanced diffusion for high-quality large-motice motion video frame interpolation. Proceed- ings of the Computer Vision and Pattern Recognition Conference, pp. 2105-2115 15, 2025b.
14

---

Published as a conference paper at ICLR 2026
Guojin Zhong, Jinhong Hu, Jiajun Chen, Jin Yuan, and Wenbo Pan. Decider: Difference-aware contrastive diffusion model with adversarial perturbations for image change captioning. In Pro- ceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 10662-10670, 2025.
Duowang Zhu, Xiaohu Huang, Haiyan Huang, Hao Zhou, and Zhenfeng Shao. Change3d: Revis- iting change detection and captioning from a video modeling perspective. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24011-24000 2025.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX OVERVIEW
The appendix provides the following details:
More related work about frame interpolation.
Semantic Similarity Function: A detailed description of the function s(;))) used our Confidence-based Frame Sampling Module (see Eq. (2 in the main paper).
Multi-gramularity Masking Schemes: An overview of the four masking schemes employed in our Procedure Modeling Module.
Warping Strategies: The description of the warping strategies to enhance temporal consis- tency in Explicit Procedure Modeling.
Asymptotic Upper Bound:
The derivation of the asymptotic upper bound for ProCap.
Introduction of Datasets: The details three datasets evaluated in our experiment.
Implementation Details: The description hyperparameters and settings used in our ex-
periments.
I
Comparison on Varied Change Categories: The performance comparison of different change categories on CLEVR-Change with SOTA methods.
U Extended Comparison with MCT-CCDiff: An extended analysis on Spot-the-Diff and com- parison with MCT-CCDiff on effectiveness and inference efficiency.
Ablation on Explicit Procedure Modeling: An analysis of component contributions to our Explicit Procedure Modeling.
Ablation Implicit Procedure Captioning: An analysis of component contributions to Implicit Procedure Captioning.
M
Qualitative comparisons with SOTA methods and visualization of procedure modeling.
Limitation and Future Work: The discussion of limitations in ProCap, and future work.
0
The statement of ethics.
P The statement of reproducibility.
The statement of using LLMs in the paper.
B
RELATED WORK
B.1 FRAME INTERPOLATION
Frame Interpolation (FI) aims to synthesize a dynamic visual transition between a given start and end frame. Existing approaches have achieved remarkable progress with powerful generative models, in- cluding denoising diffusion models that generate intermediate frames from noise (Voleti et al. 2022) Höppe et al.) 2022 Pallotta et al. [025 [Zhang al.) 2025b) Hur et al.
2025) and Transformer- based 2021 Ge et al. 2022). A no- exx-cconiiiooeed interpolation (Han et al. [022] et al.) (023). which uses textual descriptions to guide the synthesis in a controllable manner. However, existing Fl research primar- ily focuses on generating visually realistic videos. rather than supporting reasoning for downstream tasks such as captioning. To enhance change captioning, we draw inspiration from FI techniques explicitly synthesize a procedural sequence and model the underlying change dynamies, thus providing richer foundation for downstream reasoning.
architectures that predict missing content autoregressively (Yan et al.
table solution
C SEMANTIC SIMILARITY FUNCTION
To quantify the informativeness of the intermediate frame, we investigate two strategies for com- puting the similarity metric, s(..) -) in Eq. (2) in the main paper. These strategies are defined by the modalities they incorporate: (1) visual-only, which relies solely visual frame information, and
(2) visual-text, which integrates both visual frames and the corresponding textual change caption. The effectiveness of these strategies is experimentally presented in Appendix K.2
16

---

Published as a conference paper at ICLR 2026
C.I VISUAL-ONLY
Modeling fine-gratn fine-grained semantic similarity in images a task that requires detailed comparison of object attributes and context-poses a challenge for conventional feature extractors. Extractors like ResNet (He et at.] 2016), which pre-trained classification tasks, tend produce coarse, global feature representations that overlook subtle semantic distinctions. To capture them, we employ DI- NOv2 (Oquab al.) 2023), a powerful Vision Transformer (ViT) (Dosovitskiy et al. 2021) pre- trained through self-supervision. Its attention-based architecture and training objective encourage the extraction of features that are highly sensitive local details and object-level semantics. Conse- quently, we employ DINOv2 to extract features from each image and compute their cosine similarity, providing robust measure of their semantic alignment.
We formalize visual similarity using features from a pretrained DINOv2 model with a ViT-L/14 backbone, denoted as the encoder & (-). Given target image (where & E {bef, aft)) and the generated frame set 1DF,, we define the visual similarity score set Svis (Ti PDF))
as:
Svis (1,, PFI) {s(1,, 1:) E E s(,, sim[EDINO (It), (DIIOO(I.)
(11)
where sim[;,) represents the cosine similarity between the extracted features.
C.2
VISUAL-TTEXT
While visual similarity with pFl serves to measure information redundancy, it is insufficient for verifying the semantic correctness of the change transformation. A purely visual metric is text- agnostic; thus, pseudo-frame can be visually plausible interpolation yet fail to represent the specific change conveyed by the ground-truth caption. To resolve this issue and enforce semantic validity, incorporate the ground-truth -ounndttutth-oe change caption explicitly model the informativeness of each pseudo-frame.
To this end, employ the pretrained CLIP-based model from Guo et al. (2022). which is specif- ically designed to measure semantic alignment between an image-pair transformation and tex- tual description. The model provides dedicated image-pair encoder ECLIP (., ), and a text encoder (.).P()). The similarity function Svis-eex(""" (.;+) between target image I, and pseudo-frame candidates pii under caption I' is defined
8 (, PHl 7) = [s(Te, J;, T) 1 T PP)), s 1, 1, T) i (Te, T.), & (T)),
(12)
where is the change caption corresponding to the image pair (Aeet, lat). If a pscudo-frame 14 is semantically misaligned with the caption T, it will receive a lower similarity score, indicating that it contains incorrect or irrelevant information about the change transformation.
D MULTI-GRANULARITY MASKING SCHEMES
We adopt four masking strategies illustrated in Figure during the training of Explicit Procedure Modeling: (1) entire masking, (2) random patch masking, (3) in-block masking (Tan et al., (02T)) and (4) out-of-block masking (Tong et al. 2022). During training, one masking strategy is randomly
selected with a probability of 0.1, 0.7, 0.1, 0.1. respectively, and applied to each sample in a batch.
Given an input image embedding el E (k+2)nn xd the binary mask index set is denoted as M E (k++2)), where value of I on index i indicates the i-th patch to be masked.
Entire Masking. This strategy masks all embeddings in the process sequence, forcing the model to reconstruct the entire process solely based on the accompanying text sequences the alignment setting. Formally, the masking probability is defined
p(e e of E el 11
(13)
17

---

Published as a conference paper at ICLR 2026
Masking Posh
Modking Block
Keeping Blcck
(1) Ensire Maskia
(2) Raadon Paxch Marking
Makling
Figure 3: Four masking schemes in the proposed multi- multi-granulann granularity strategy. We mask visual patch embeddings for reconstruction during training; the masks are visualized at the patch level for clarity.
Random Patch Masking. Given interval (a, b), the masking probability for index is sampled from a uniform distribution over this interval, denoted as U(a 6), where a and are set to 0.2 and 0.5 respectively in experiments. Specifically, for ef e², the probability of replacing ef with mask token e is given by:
p(e! =jusk le/ ee) Ps where p, (),
(14)
In-block and Out-of-block Masking. Given an unflattened image embedding ej, E Rxxwxd
rectangular region, whose area ratio the whole image is randomly sampled within [0.2, 0.8] (with an expected value of approximately 0.5), is randomly selected with bottom-left corner at (æ1, V1) and top-right corner (I2, 82):
m1 S <is 22, Si R = {(i,])
(15)
where 0 81 < J2 < w and 0 < 1/1 < y/ h. In In-block masking, all embeddings within this region are masked:
e man
mmak e m. e & (l...., k + 2}. (m, n) E R) =
(16)
Conversely, in Out-of-block masking, all embeddings outside the selected region masked:
p e
m.n = Emsk m.n E e ho E: {l,..., k + 2}, (m,n) n) & R 1.
(17)
E WARPING STRATEGIES
We apply four widely used warping strategies for disrupting the temporal consistency of the frame sequence for training in Explicit Procedure Modeling stage: (1) batch procedure frame shuffle, (2) frame shuffle, (3) color shifting, and (4) affine transformation.
Batch Procedure Frame Shuffle. Given batch of procedures P2,,,, the sequence frame shuffle strategy randomly selects two positions and j from two different procedure Po, and Pby. respectively. It then replaces the frame I; E Pb, with I, Pog.
Frame Shuffle. Given a procedure P. random permutation is applied to its frames to produce a shuffled sequence P', which serves as the augmented data.
18

---

Published as a conference paper at ICLR 2026
Color Shifting. Given procedure P RTx/Xxxx3, randomly select single RGB channel and add random scalar value a to all the pixels in that channel across the entire sequence. This results in color shifting augmentation:
Shin
1 + a,
(18)
where 1 ERTHHx represents the selected RGB channel of all images in P.
Affine Transformation. We apply a random affine transformation to the input image 1 E P. Specifically, we sample:
rotation angle (l(aaaa)) a))
horizontal and vertical transitions to ty ~ ((-T,T),
and a scaling factor 8 L1(1 - p4, 11).
where a, т and μ are user-defined hyperparameters, which is set to 30, 0.1 and 0.1 in out experiments respectively. An affine transformation matrix is defined as:
s Cos 0
8 sin A s cos
1x
A
(19)
sin
by
For each position of the input image
[x yl. the augmented output can be denoted as:
s cos sin A
-8 s sin 0 cos 0
x 3/
tx
(20)
F ASYMPTOTIC UPPER BOUND
In this section, we will discuss the asymptotic upper bound in inference. Let ny denote the length of image embeddings and nr the length of text embeddings. For simplicity, we assume all embeddings have a uniform dimensionality d. The asymptotic upper bound of the entire model in inference can be divided into two compnents: one corresponding to procedure encoder, and the other to the text decoder.
Analysis for Procedure Encoder. Before sending to the Transformer-based procedure encoder, image pairs are first encoded into embeddings via CNN. These embeddings are then concatenated with masked embeddings to form sequence of shape (k 2)m, x d. For clarity in complexity + 2 and denote K ny as npp The time complexity of CNN can be denoted as On1 channels? kernels). Assuming constant kernel size and fixed number of channels, the time complexity of a convolutional layer scales linearly with the number of output pixels, i.e., Onn)). For each layer the Transformer architecture, the input embeddings are linearly projected to obtain queries, keys, and values, incurring a time complexity of O(up x d²). The self-attention mechanism, as introduced by Vaswani al. (2017), computes attention as follows:
analysis, we
QK
Attention(Q. K, V) softmax
V.
(21)
Vd
This step dominates the computational cost of the attention mechanism, with a time complexity of (n d).
As result, the final asymptotic upper bound of procedure encoder of lg layers can be denoted as: O(ns + le (np d² + n x d)).
(22)
Given that d > 1, the lower-order term O(nz) becomes negligible, and the complexity can be approximated by:
Olle (np x & + n d)).
(23)
19

---

Published as a conference paper at ICLR 2026
Analysis for Text Decoder. The text decoder for captioning is la-layer Transformer decoder, which includes both self-attention and cross-attention mechanisms. For the self-attention mecha- nism, the time complexity per layer is given by On X de + n x d). For the cross-attention mechanism, where attention is computed between the change procedure sequence and the text se- quence, the time complexity can be expressed as:
O(np X nr d - np x &f + T&T x d
(24)
accounting for the projections of both input sequences and the attention computation. As np >> NT and d >> N: our experiments, Eq. (24) can be approximated by:
(ne x d + -p x d² + dd) = O(np dd).
(25)
Therefore, the final asymptotic upper bound of a la-layer text decoder can be denoted as:
O (NT d² + + d + nip x d²)),
(26)
which can be approximated by: Oll x nep x d²).
(27)
Asymptotic Upper Bound in Inference. Comprising Eq. 23 and Eq. 27 the final asymptotic upper bound of the entire model in inference can be denoted as:
Ole x (np d' nip d) la np
(28)
Since la a small constant, it can omitted from the asymptotic expression. Therefore, the asymp- totic upper bound can be denoted as:
Olle (np x df + n² d)).
(29)
Substituting np = K x n2 into the above expression yields:
Oll x (K x x x &e + K2 x n x d)).
(30)
It can be noted that the inference computation scales quadratically with respect procedure length
K. Therefore, it is necessary to reach a balance between performance and inference computation cost.
G INTRODUCTION OF DATASETS
We conduct experiments on three widely used benchmark datasets: Spot-the-Diff (Jhamtani & Berg- Kirkpatrick 2018), CLEVR-Change (Park et al. 2019), and Lae--iiinnn-Rqqsee (Tan et al.) 2019). In this section, we provide detailed overview of each dataset.
Spot-the-Diff is the first dataset specifically designed for change captioning. is constructed by sampling from VIRAT (Oh al. (01TT, a realistic video surveillance dataset. The dataset comprises 13,192 pairs of similar images, each paired with a aanaatttee change caption. Since the image pairs derived from surveillance videos, they are well-aligned, and each pair contains at least one semantic change. The dataset is split to training, validation and testing sets with an 8:1:1 distribution.
is a synthetic dataset generated using CLEVR (Johnson al. 2017 rendering engine capable of producing images of objects with complex relationships. It consists of 79,606 pairs of similar images with 493,735 change caption annotations, which split into 67,660, 3,976, and 7,970 training/validaiionnnes image pairs, respectively. Unlike soot-the-Difff CLEVR-Change introduces distractors alongside semantic changes- for example, variations in viewpoint that do not alter object positions. This design poses greater challenges for change captioning, requiring models to distinguish genuine semantic changes from irrelevant visual differences and be more robust in reasoning about visual transformations.
CLEVR-Change
provides similar image pairs with image editing approaches guided by instructions. It comprises 3,939 similar image pairs with 5,695 human--nnotadl instructions as change captions. The dataset segmented into 3,061 training pairs, 383 validation pairs, and 495 testing pairs.
20

---

Published as a conference paper at ICLR 2026
Table 5: Evaluation CLEVR-Change with varied change categories by METEOR.
Method
Color
Texture
Add
Drop
Move
DUDA 2019
32.8
27.3
33,4
31.4
23,5
DUDA+ AUD 2021
36.1
30.4
37.8
36.7
27.0
IFDC 2021
33.1
27.9
36.2
31.4
31.2
NCT 20230
39.1
36.3
39.0
37.2
30.5
SMARI 2024b
40.2
37.8
39.3
38.1
31.5
DIRL+ URR+ 22024a
40.7
38.2
40.0
37.9
33.5
ProCap (Ours)
39.7
41.0
Table 6:
Extended comparison with MCT-CCDifT on the Spot-the-Diff dataset, where model training with LLM-augmented captions.
denotes
Method
Speed (s/caption)
B 1
M f
R 1
C1
MCT-CCDiff (0225)
0.91
10.8
14.5
35.5
41.7
ProCap (Ours)
0.04
11.0
13.6
33.7
42.7
ProCap' (Ours)
11.7
14.2
34.6
H IMPLEMENTATION DETAILS
synthesize pseudo change procedures, with the process length set as = 7. To balance captioning quality and inference efficiency, we sample t = 2 intermediate frames. For image representation, we fine-tune pre-trained VQGAN on the change captioning datasets via an image reconstruction task. The VQGAN is configured with a codebook size of K 1024 and latent dimension d, 256. Input images are resized to 224 x 224 and encoded into latent resolution of 14 14. The procedure encoder is configured with le 12 layers on CLEVR ETEVR-Change and Image-Editing- Seitting-Request. and le 4 layers on Spot-the-Diff. The hidden size is fixed at 768. The caption decoder consists of la = 2 layers on CLEVR-CUE Rhyange and Image-Editing- Seetingg-Request datasets, and consists la = 3 layers on Spot-the-Diff dataset, with common hidden size of 512.
We employ pre-trained frame interpolation model, VFIformer (Lu et al. 2022).
In the Explicit Procedure Modeling stage, we train our model for 200,000 steps on 2 NVIDIA A40 GPUs using warm-up strategy that linearly increases the learning rate from 1 10 to 1 10" over the first 5,000 steps. The total batch size set to 8. In the Implicit Procedure Captioning stage, train model for 40 epochs with the total batch size of 16 on NVIDIA A40 GPU. The procedure encoder is optimized with fixed learning rate of 5 x 10-5 on the CLEVR-Change and Image-Editing- E--tiiin-eeuuuss datasets, and 2 10-5 the Spot-the-Dit -Dff dataset. Meanwhile, the caption decoder adopts a warm-up schedule that linearly increases the learning rate from 0 to x 10-3 during the first 10% of total training steps for all datasets.
Code and data for our experiments will be made publicly available.
I COMPARISON ON VARIED CHANGE CATEGORIES
In this section, we present detailed comparison of performance across different change categories on CLEVR- -Change, evaluated with METEOR against SOTA methods. Table 5 shows that our ap- proach achieves competitive results on color and texture changes, and attains the best performance on addition, removal, and movement changes. In particular, it significantly outperforms the cur- rent SOTA method on movement changes, indicating superior ability to distinguish action-related changes in the presence of environmental distractors.
J EXTENDED COMPARISON WITH MCT-CCDIFF
To better understand the performance characteristics of ProCap on the Spot-the-Diff dataset, we conducted extended analysis comparing method with the current SOTA approach, MCT-
21

---

Published as a conference paper at ICLR 2026
CCDiff (Hu al. 2025 We observed that MCT-CCDiff reports notably higher METEOR and ROUGE scores on this dataset, while ProCap achieves superior CIDEr performance. Upon exami- nation, we found that this discrepancy is primarily attributable to differences in the richness of the training captions rather than limitations the model architecture itself.
As documented in MCT-CCDift, their training pipeline expands the original Spot-the-Diff training set with GPT-generated captions, substantially enriching the linguistic diversity of the supervision. In contrast, our primary experiments strictly follow the original, unaugmented annotations. Since METEOR and ROUGE are highly sensitive to caption diversity and surface-level phrasing, this difference in training data preparation naturally affects these metries.
To isolate the effect of caption richness, we conducted an additional experiment in which we aug- mented the Spot-the-Din tiff training captions using Qwen3 (Yang et al.) 2025), following the strategy introduced in MCT-CCDiff. As shown in Table O under this matched setting, ProCap achieves com- parable METEOR and ROUGE scores and surpasses MCT-CCDiff on the more semantically aligned measures, including CIDEr and BLEU-4 (with improvements of +7% and +8%, respectively). These results indicate that the gap previously observed on METEOR and ROUGE stems largely from linguistic properties of the training set rather than from the robustness of the model.
In addition accuracy, we also compare inference efficiency. Under identical conditions, Pro- Cap is 22x faster than MCT-CCDiff while maintaining superior CIDEr performance. This demon- strates that ProCap offers not only competitive captioning quality but also a significantly better Snnciency-effecttveee
trade-off compared to existing non-LLM SOTA approaches.
K ABLATION ON EXPLICIT PROCEDURE MODELING
This section extends the ablation study from Sec. 4.3 of the main paper with detailed compo- nent analysis on the three commonly used datasets. We specifically evaluate the contributions of individual components within the Procedure Generation, Confidence-basse Frame Sampling, and Procedure Modeling Modules.
K.1 MORE ABLATION ON SPOT-THE- -IFF DATASET
Tables 7 and andesntt present additional ablation studies on the Spot-the- dataset, which contains more realistic scenarios compared with FLEVR-Changee Consistent patterns emerge across these experi- ments, further demonstrating the effectiveness of our method and its strong generalization ability in real-world settings.
Table 8: Effectiveness and performance com- parison on Spot-the-Diff dataset with varying procedure query set length
Table 7: Ablation study for explicit procedure modeling (EPM) and implicit procedure cap- tioning (IPC) on Spof-the-Diff dataset.
Methods
f
1
f
f
EPM IPC
M
8.6
12.5
32.2
36.0
7.9
11.7
28.0
11.0
13.6
33.7
42.7
0
8.5
12.1
27.8
30.6
ProCap
8.3
12.1
27.5
29.8
4
8.5
12.4
27.7
31.3
12.5
32.2
36.0
7
7.5
11.8
25.7
29.2
K.2 PROCEDURE GENERATION MODULE
We investigate the interaction between the number of generated pseudo-frames, and the choice of semantic similarity function for keyframe sampling. To this end, we evaluate the two functions (see Appendix C) within Confidence- Frame Sampling Module, benchmarking them against random sampling baseline that selects frames uniformly.
Varying number of generated pseudo-frames! 1. Figure examines how varying the number of generated frames affects captioning performance, while keeping the number of sampled keyframes in the Procedure Modeling Module fixed at = 2. The results highlight a clear trade-off: increas- ing enriches spatio-temporal cues but simultaneously introduces substantial redundancy and noise.
22

---

Published as a conference paper at ICLR 2026
This trade-off is most pronounced in the random sampling strategy the CLEVER Change dataset and in the visual-only sampling strategy on the Spot-the-Dii dataset. In both cases, performance improves increases from 3 to 7, but then noticeably degrades when / rises to 15. Although proposed sampling strategy also experiences a slight decline on the Spot-the-Diff dataset as I contin- grow, it consistently outperforms the other strategies. This suggests that, without semantic guidance, redundant and irrelevant frames can easily overwhelm the model, reinforcing the need for more robust sampling mechanisms capable of isolating truly informative temporal cues while filter- ing out misleading ones. Based on these observations, we set = 7 as the default configuration in our experiments.
CLEVER-Change
140
Randan
Viraal
135
40
CLDRY
130
125
Randam
30-
Only Text
15
133
20
Figure 4: Comparison of CIDEr scores across four sampling strategies with respect the number of pseudo-frames / on CLEVR-Change dataset (left) and Spot-the-Diie dataset (right). Each strategy set to sample two key frames from the pseudo-frames.
Measure of constraint in Fl model. As defined in Sec. 3 we formalized the change procedure mapping ?r [0, 1] + I, where T is referred change caption and I denotes the space of all possible images. As the mapping is non-bijective, without additional constraints, there exist infinite procedures for the same image pair. In our experiment, to restrict the solution space, adopt off- the-shelf optical-flow- wiw--floooweeeeee frame interpolation method to synthesize change procedures, where the optical flow serves as strong constraint: the intermediate frame is obtained by warping the before and after images according to the linearly interpolated optical flow, rather than being generated from scratch. To empirically demonstrate the necessity of these constraints, we compared our approach on the Saaee--diiinneeeee Request dataset with one diffusion-bal based frame interpolation, which operates within a significantly less constrained solution space. As shown in Table y relaxing the constraints leads to noticeable performance degradation compared to the optical- fioicca--fowbbased approach. We at- tribute this drop to the stochastic nature of diffusion models. Unlike pptical-flow-basee methods that enforce strict pixel-wise correspondence, diffusion models inherently introduce unpredictable and uncontrollable visual variations in the intermediate frames (as shown in Figure These unintended variations make procedure modeling more difficult, hindering effective model training.
Table 9: Performance comparison with different constraints of the FI model on the Image- Request dataset.
Models
B f
M
R
C
Ours flafed (025a))
9.9
15.3
41.3
37.8
Ours coptical-flos flow-based (2022)
11.7
15.9
43.2
40.6
K.3 CONFIDENCE-BIL BAASDD FRAME SAMPLING MODULE
Impact of semantic similarity functions. Figure illustrates the comparative performance of three distinct semantic similarity functions for keyframe selection. Our analysis yields the fol- lowing observations.
(1) Random Sampling vs Visual Only Strategies: Compared with random sampling, Visual Only demonstrates benefits, particularly when sampling a larger number of pseudo- frames, such as 1 = 15. This highlights the effectiveness of filtering out redundant frames in long frame sequences. However, Visual Only strategy still exhibits clear performance decline creases, indicating its sensitivity to irrelevant visual content when textual grounding is absent. (2) Visual+Tess Strategy: In contrast, Visual+ Text strategy consistently outperforms other strategies
in-
23

---

Published as a conference paper at ICLR 2026
(b)
(d)
Figure 5: Uncontrollable predicted intermediate frames examples of diffusion baased FI models. Samples (a) and (b) show an unexpected object prediction, while samples (c) and (d) show an unex- pected motion generation.
across most evaluated values of l. Its performance remains robust even increases, suggesting that the integration of textual cues provides strong guiding signal for identifying informative and relevant frames. This makes Visual+Text strategy resilient to noisy redundant frames within the temporal sequence. (3) Overall: These results collectively highlight the effectiveness of leveraging multimodal coanals-particulaart textual grounding for key frame selection under varying tempo- ral resolutions. As result, we select Visual+ Text strategy for our model.
140
ProCup (c
CIDEr (1 is hetter)
ProCap (k=7)
130
ProCap
ProCap (k-1)
125
LLaVA-155
LLM-based
Owen-VL.
ProCup (Ours)
ProCap variants
100
200
300
400
500
600
700
800
TPS (Tokens per Second, 1 is better)
Figure 6: Effectiveness and performance comparison with LLM-based methods on CLEVR-Change
dataset.
K.4 PROCEDURE MODELING MODULE
Comparison with LLM--baed methods on different query set lengths k. Oresents the performance comparison with LLM-b approaches the CLEVR-Change dataset, using the same query set configurations as in Sec, 4.3) Our method achieves clear improvements over general multi-modal large language models QW and 1A-15. demonstrating its strong capabil-
24

---

Published as a conference paper at ICLR 2026
50
Madding
45
Intbick Mosking
Ntaddnng
Madding
Oei-sf Masking
40
335
30
0.1
0.25
0.4
0.7
Mask Probability Setting
Figure 7:
Comparison of CIDEr scores on the Spot-the-Diff dataset under different masking strate- gies across varying probability settings. When the probability of one strategy is set to p, the proba- bilities of the remaining three strategies are each set to (1 p)/3.
ity in change captioning. Although the specifically trained LLM-based method FINER performs well on CLEVR- it suffers from substantial computational cost due to its large number of parameters. contrast, our approach attains competitive overall performance while maintaining remarkable effectiveness at k 2.
Impact of caption-conditioning. Table 10 presents the benefit of incorporating ground-truth cap- tions as a condition during procedure modeling. A significant performance boost is observed when the model is conditioned on the text, compared to using visual inputs alone. This shows the power of cross-modal learning in our procedure modeling. The caption acts as a powerful semantic prior, achieving two key objectives: (1) helping understand the nature of visual changes, and (2) achieving an early alignment between visual dynamics and linguistic contents. By learning to generate pro- cedure that is consistent with the target description, the model produces representation that is not only visually coherent but also semantically aligned with the captioning stage. Therefore, we utilize ground-trufe captions conditional guidance for training the procedure modeling module.
Table 10:
Ablation study for caption-conditioning in explicit procedure modeling CLEVR- Change and Spot-the-Diff.
ELEVR-Change M Rt
Spot-the-Diff M
Settings
R
C1
w/o caption
57.0
40.9
74.7
128.8
8.0
11.6
28.1
28.9
caption
56.7
41.7
74.7
135.6
11.0
13.6
33.7
42.7
Table 11: Ablation study for multi-granularity masking strategy in explicit procedure modeling stage on tpof-the-Diff.
Settings
M f
R
C
w/o Entire Masking
8.8
11.9
30.2
32.5
w/o Random Patch Masking
10.3
12.5
32.7
40.7
w/o in-block Masking
7.9
12.0
28.0
30.0
w/o Out-of-block Masking
8.0
12.1
27.6
30.5
w/ All Masking Strategies
11.0
13.6
33.7
42.7
Impact of multi-granularity masking strategy. Table [1 shows the contribution of each mask- ing strategy described in Sec 3.1.3] Without the entire masking strategy, the model cannot ade- quately learn to reconstruct intermediate frames solely from change captions, thereby weakening its cross-modal understanding ability. In contrast, incorporating random patch masking yields bet- ter performance by promoting the learning of distributed visual representations. Furthermore, the
25

---

Published as a conference paper at ICLR 2026
significant performance drop observed when either in-block out-of-block masking is removed highlights the crucial role of these strategies in facilitating spatial-temporal understanding. Figure illustrates the performance comparison actoss different probability configurations of the four mask- ing strategies. Together with Table T1 the observations consistently reveal three key findings: (1) Stronger learning of distributed visual representations leads better performance. Random patch masking plays a central role by providing broad and dense visual coverage, and therefore receives the highest probability. (2) Entire masking, in-block masking, and out-of-block masking are es- sential for modeling global context and localized structural cues. However, overemphasizing any of these structured strategies removes too many fine-grained visual details, which hampers detailed feature learning and ultimately degrades change-detection performance. This is evident from the steady performance drop observed when the probability of any of these three strategies increased. (3) The four masking strategies work synergistically, jointly supporting both coarse-grained and fine-grained representation learning. In contrast, relying solely on random patch masking yields only marginal improvements.
Impact of the procedure encoder's depth. We investigate the impact of the procedure encoder's depth on the CLEVR-Change and Spot-the-Diff datasets, with results presented in Tables 12 and The results reveal that the optimal encoder depth is dataset-dependent. On CLEVR -hyahe, per- formance consistently improves with a deeper encoder, peaking with a 12-layer architecture. This suggests that modeling the changes in FLEVR-Change benefits from higher-capacity encoder. In contrast, a shallower 4-layer encoder is optimal for Spot-the-Diff, as an overfitting observed with deeper encoders.
Table 12: procedure encoder layers on CLEVR-Change.
Ablation results of using different
Table 13: Ablation results of using different procedure encoder layers on Spot-the-Diff.
Layers
M
R
C
Layers
M
R
52.6
38,9
718
117.4
13.0
28.4
30.2
4
53.9
39,9
73.1
124.3
11.0
13.6
33.7
42.7
8
54.2
41.0
73.9
133.2
8
9.4
12.0
32.1
42.2
12
56.7
41.7
74.7
135.6
13.5
27.8
L ABLATION ON IMPLICIT PROCEDURE CAPTIONING
We further evaluate the contributions of two key components within the implicit procedure caption- ing on the CLEVR-Change dataset and the Spot-the-Diff dataset.
Explicit and implicit procedure captioning. We compare our proposed Implicit Procedure Cap- tioning (using learnalle queries) against a baseline that performs Explicit Procedure Captioning (directly encoding synthesized frames). Table [4 shows that our implicit approach with procedure queries achieves superior performance the CLEVR-Chame dataset. The explicit baseline, which relies on synthesized frames, not only incurs higher computational costs but also suffers in perfor- mance. We attribute the lower accuracy of explicit procedure modeling to the redundant and noisy temporal information in the generated frames. In contrast, our learnable queries provide a more robust representation of procedural dynamics, leading to more accurate change descriptions.
Table 14: Impact of implicit procedure captioning using procedure queries. The first line denotes explicit procedure captioning using synthetic pseudo-frant frames generated from Procedure Generation Module directly.
Settings
TPS
B
M
R
C
Explicit procedure captioning
421.03
74.4
128.5
Implicit procedure captioning
699.04
56.7
41.7
74.7
135.6
26

---

Published as a conference paper at ICLR 2026
Impact of the text decoder's depth. We analyze the effect of decoder depth the CLEVR- Change and Spot-the-Dife datasets (Tables 15 16). observing a general trend overfitting with excessive layers. The optimal decoder depth for Spot-the-Diff (3 layers) is greater than for CLEVR- Change (2 layers). We attribute this to the nature of the target change deseriptions. Unlike the highly structured descriptions for CLEVR-Change Spot-the-Diff requires more descriptive power. Its surveillance-style scenes feature non-canonical object poses and complex background clutter, demanding greater linguistic capacity from decoder.
Table 15: Ablation results of using different text decoder layers on FLEVR-Change
Table 16: Ablation results of using different text decoder layers on Spot-the-Dift.
Layers
M
C
Layers
R
C
56.7
41.7
74.7
135.6
9.4
12.0
32.6
37.1
129.5
11.0
33.7
42.7
4
56.5
40.7
74.6
129.7
7.1
10.7
28.1
31.7
5
56.8
41.0
74.7
130.4
5
8.1
11.7
27.5
28.5
M QUALITATIVE RESULTS
M.1 COMPARISON OF CAPTIONING GENERATIONS
Figure presents the qualitative results of our ProCap. We compare our model with two non-LLM- based approaches (DIRL. (Tu al. 2024a) and SCORER (Tu et al. 2023c)) and one L.LM-based method (FINER (Zhang et al. 2024)) to highlight its generation capabilities. Our model demon- strates robust performance across variety of change scenarios. Moreover, by incorporating tempo- ral information into the change captioning process, our model better captures the temporal order of events, enabling it to generate more accurate and coherent captions, exemplified in Figure
M.2 VISUALIZATION OF CHANGE PROCEDURES
Figures 9 12 present qualitative visualizations of the explicit change procedures generated by model on three datasets: CLEVR-Change, Spot-the-Diff and Image-Editing Eee-iiiiin-RRsssts.. Our model leverages the synthetic procedures from the Procedure Generation Module and the key frames lected by the Confidence-bll basedd Frame Sampling Module to effectively capture the transformation process between image pairs. Notably, it remains robust even when the synthesized procedures hibit temporal redundancy in the third and fourth samples, which is a critical prerequisite for the subsequent Implicit Procedure Captioning.
M.3 CASES WITH SIGNIFICANT VIEWPOINT SHIFT
Figure shows several cases exhibiting significant viewpoint shifts in the CLEVR-Change dataset. Following Park et al. (2019), we use the loU between similar image pairs to quantify the degree of viewpoint change. The mean IoU in CLEVR- CLEVR-Change is 0.51 with variance of 0.02; therefore, an loU around 0.2 regarded as indicating a substantial viewpoint shift (as illustrated in the first two rows). Notably, even under such drastic viewpoint differences, a plausible intermediate process, demonstrating the robustness of our procedure modeling module. We attribute this robustness our proposed consistency loss, which explicitly promotes spatial- temporal consistency in the reconstructed intermediate frames.
model is able to reconstruct
M.4 FAILURE CASES
Figure 14 presents several failure cases produced by our proposed ProCap. For most failures on the CLEVR-Change dataset, the modifications are extremely subtle, which makes it difficult for the model to reliably detect the change throughout the procedure. In contrast, the primary source of errors in the Eearee-diiing--eee the intermediate procedure, which subsequently leads to incorrect change captions. We attribute this issue overfitting, these two datasets are more open and unconstrained compared with the
and Spot-the-Diff datasets lies in inaccurate reconstruction of
27

---

Published as a conference paper at ICLR 2026
(c) Object Add
(a) Color Change
Texture Change after
before
after
before
before
aller
Ground Truth: the yellow ball has been newly placed
shiny behind the large gray rubber object be- came rubber
Ground
Ground Truth: the tiny yellow cube turned gray
ProCap (Ours); the small yellow metal ball that is in front of the big blue object has been newly placed
ProCap (Ours): the small yellow rubber cube that behind big yellow matte object changed to gray
ProCap (Ours): the small blue metal cube that behind the large gray matte ball changed rubber
DIRL: the small cyan matte ball that is front
DIRL: scene remains same SCORER: the scene remains the same FINER: there no change
big red metal thing becamee
DIRL: the scene remains the same
yellow
SCORER; the
beforee FINER: there no change
FINER: the small eyan matte
is
in front of the hig red cube changed to motallic
(d) Object Remove before
(f) Complex Scenes
afler
(e) Object Move
be
after
hefore
after
Ground Truth: the tiny green cylinder is longer there
Ground Truth: shiny cylinder changed its location
ProCap (Ours): the small green matte cylinder that behind the big gray matte cylinder is no longer there
Ground Truth: the white vehicle driving the the left gone
ProCap (Ours): the small green metal cylinder that is in front of the small red ohject changed
ProCap (Ours): silver car has left the parking lot
DIRL: scene remains same SCORER: the small yellow rubber cube that behind
DIRL: the soene remains the same before FINER: there no change
DIRL: there is person walking in the parking lat
small brown rubber block became red
SCORER: scene
FINER: there is person walking in the parking lot
FINER: there no change
(g) Complex Scenes
(i) Composite Change
before
(h) Fine-Grained Attribule Change after
before
after
before
Ground Truth: black suv has appeared ProCap (Ours): black vehicle driving down the road the second image DIRL: there is a person walking in the parking lot
Ground Truth: Remove the text from the entire image
Ground Truth: Change the wall color white blue
ProCap (Ours): remove the text from the photo
ProCap (Ours): change the color of the image blue
DIRL: Change the snake the snake
FINER: there is a person walking in the parking lot
FINER: change the background to pink
DIRL: change the background from a lit.
FINER:
zuom in on the girl
(l) Pattern Change
(k) General Attribute Change before
before
after
(i) General Attribute Change before
after
after
L
Ground Truth: photo
Ground Truth: remove the text "louis vuitton" and the "LV" sign, add the area that is behind and above instead of the text and sign
Ground Truth: Darken the entire image
ProCap (Ours): crop the image
ProCap (Ours): darken the image DIRL: Brighten the image
DIRL: Change the background from that it looks like a little
FINER: brighten the entire image
FINER: remove the people the back- ground
ProCap (Ours): remoove the text DIRL: remove hackground
FINER:
remove the background
Figure 8: Comparison of captioning generations. We compare our model against two non-LLM- based approaches (DIRL and SCORER) and one LLM-based method (FINER). The examples are grouped into 10 change types, and (a)-(e) are from the CLEVR -hhnng dataset, (f)-(g) from Spot- the-Diff, and (h)-()) from Image-Fdiiingg Request.
TLEVR-Change dataset. In future work, we plan further investigate the generation and modeling of more coherent and semantically reasonable intermediate transformation processes to improve the robustness of change captioning.
28

---

Published as a conference paper at ICLR 2026
Caption: the big matte cube changed to purple
Caption: the tiny rubber cube changed to metal
Caption: the tiny brown thing been added
Caption: the small sphere is longer there
Figure 9: Visualization of change procedures on CLEVR-Change For each sample, the top row displays the synthetic procedure generated by the Procedure Generation Module. The bottom- left shows key frames selected from this synthetic procedure using the Confidence- Frame Sampling Module, while the bottom-right visualizes the reconstructed procedural representation produced by the Procedure Encoder within Procedure Modeling Module.
29

---

Published as a conference paper at ICLR 2026
Caption: the large cylinder is different location
Caption: there is difference
Caption: the scene same hefore
Caption: change has occurred
Figure 10: Additional visualizations of change procedures on CLEVR-Change
30

---

Published as a conference paper at ICLR 2026
Caption: there's 3 fewer boys on left photo
Caption: there is person
Caption: a car now parked in the spot next to the building
Caption: there is only one person shown
Figure 11: Visualization of change procedures on Spot-the-Dii
31

---

Published as a conference paper at ICLR 2026
Caption: color his hair blonde
Caption: lighten the image
Caption: make solid white hackground brighten up the whole image of
Caption: sharpen this image
Figure 12: Visualization of change procedures on Image- image-Editing Seitting-Request
32

---

Published as a conference paper at ICLR 2026
before
overlaid
Caption: blue object a different location
before
overlaid
Caption: large
metallic
befooe
everleid
Caption:
big sphere moved
before
overlaid
Caption: green matte cylinder changed to shiny
Visualization of cases with significant viewpoint shift. The left shows the original image pair with the overlaid image. The right visualizes the reconstructed procedural representation pro- duced the Procedure Encoder within the Procedure Modeling Module.
Figure 13:
Reference: the tiny brown shiny sphere right of the tiny yellow thing became yellow ProCup: no change has occurred
changed ProCap: change has occurred
Reference: Adjust the hrightness
ProCap: change the background
Reference: red car has left the soene ProCup: there car middle
Figure 14: Visualization of failure cases generated by ProCap. The left shows key frames selected from the synthetic procedure using the Confidence-based Frame Sampling Module, while the right visualizes the reconstructed procedural representation produced by the Procedure Encoder within the Procedure Modeling Module.
33

---

Published as a conference paper at ICLR 2026
N LIMITATION AND FUTURE WORK
In this work, we propose novel two-stage framework, ProCap, which reformulates change cap- tioning from static comparison to dynamic procedure modeling. While experiments demonstrate that method achieves strong performance across three widely-used benchmark datasets, certain challenges remain in specific scenarios.
For instance, when scenes exhibit dramatic changes, for example, where transformations exceed the variations in position, appearance, and existence defined in Sec.B] or drastic viewpoint changes hap- pen, generating perfectly physically grounded intermediate frames becomes inherently ill ill-posed for any current generative model, as pixel-level correspondence no longer preserved. In such cases, 21) generative models, such optical-flow anfical-llow-based approaches (Lu et al. 2022), face fundamental limitations due to the lack of explicit geometric depth reasoning. We believe that a paradigm shift toward 3D scene modeling to maintain geometric consistency is beneficial to maintain geometric consistency and produce physically grounded intermediate frames under such extreme variations. Consequently, identify 3D-aware representation as a critical direction extreme geometric dis- continuities in future exploration.
Another open problem lies in defining what constitutes a theoretically optimal informative point. While our current formulation provides practical solution, more rigorous theoretical definition remains unexplored. Future work could investigate a principled mathematical characterization of this optimal point within the broader context of change analysis, potentially leading to more robust and generalizable criteria.
Finally, integrating LLMs represents a natural and valuable extension of our framework. We plan to explore LLM-based architectures- as instruction-tuning strategies-to combine the high- level reasoning capability of LLMs with the explicit dynamic modeling strengths of ProCap. Such integration may enable richer semantic guidance and more adaptive dynamic understanding in future systems.
Collectively, we believe these limitations highlight several promising avenues for continued devel- opment. With more refined model design and deeper theoretical grounding, ProCap can be extended to address these challenges more effectively.
O ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. No human subjects or animal experiments were involved in this study. All datasets used, including CLEVR-Change, Spot-the-Diff, and Image- Editing- Request, were obtained in accordance with their respective usage guidelines, ensuring full compliance with privacy standards. We have taken care to minimize potential biases and avoid discriminatory outcomes throughout the research process. No personally identifiable information was utilized, and no experiments were conducted that could raise privacy or security concerns. We committed to upholding transparency, fairness, and integrity in all aspects of this research.
P REPRODUCIBILITY STATEMENT
We have taken extensive measures to ensure the reproducibility of our results. All code and data used in the experiments will be released publicly to facilitate replication and independent verifica- tion. The experimental setup ioppl-dnniiinggg training procedures, model configurations, and hardware specifications-is is detailed in Appendix H In addition, provide comprchensive description of ProCap to further support reprodueibility.
Furthermore, the three change captioning datasets used in our work CLEVR-Change Spot-the- Diff, and Image-Editing-- Editing-Request Request-uuee are publicly available, ensuring consistent and reproducible eval- uation.
We believe these efforts will enable other researchers to faithfully reproduce our findings and con- tribute to advancing the field.
34

---

Published as a conference paper at ICLR 2026
Q STATEMENT OF USING LLMS IN THE PAPER
Large Language Models (LLMs) were employed to assist in writing and refining this manuscript, specifically for grammar checking and sentence polishing, with the aim of enhancing overall read- ability.
Importantly, the LLM not involved in the ideation, research methodology, experimental design, or data analysis. All research concepts, ideas, and analyses were independently developed and carried by the authors. The role of the LLM was strictly limited improving the linguistic quality of the text, without contributing to the scientific content.
The authors take full responsibility for the manuscript, including any portions refined with LLM assistance. We have ensured that the use of LLMs complies with ethical standards and does not involve plagiarism scientific misconduct.
35


Reviews:
[{"review_id": "jlgoGs1Ih2", "reviewer": "Reviewer_syPv", "summary": "This paper proposes ProCap, a two-stage framework that reformulates change captioning from static image comparison to dynamic procedure modeling. Stage 1 uses a pre-trained Frame Interpolation model to generate intermediate frames, samples informative keyframes via confidence scoring, and trains a Procedure Encoder through caption-conditioned masked reconstruction with three losses (reconstruction, alignment, consistency). Stage 2 replaces explicit frames with learnable procedure queries for efficient inference, fine-tuning the encoder end-to-end with a text decoder. The method achieves competitive results on three datasets, claiming that modeling \"how\" changes occur improves understanding beyond just comparing \"what\" changed.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Novel Paradigm and Strong Motivation: The paper presents a fresh perspective by shifting from static image comparison to dynamic procedure modeling, addressing a limitation in existing methods that ignore temporal dynamics. \n\n2. Two-Stage Design Decoupling Learning and Inference: The framework separates explicit procedure learning (Stage 1) from implicit inference (Stage 2). Using learnable queries instead of generating frames at test time is technically sound.\n\n3. Comprehensive Experimental Validation: The paper provides thorough evaluation across three diverse datasets, extensive ablations, rich visualizations (Figures 6-10), and detailed analysis of individual components. The experiments demonstrate consistent improvements over non-LLM baselines and competitive performance against LLM-based methods while being more efficient.", "weaknesses": "1.\tThis paper models the change procedure to improve captioning, but (132-137) the mapping γ_T: [0,1] → I is inherently non-bijective with an exponentially large solution space. For any given (I_bef, I_aft) pair, infinitely many valid procedures exist, yet the method relies on a generated sequence from FI without justifying why this particular realization should be canonical or optimal. This makes procedure modeling more difficult than directly describing the change—a small model must navigate a continuous, high-dimensional function space rather than a discrete caption space. Additionally, the confidence-based sampling assumes frames at the \"semantic midpoint\" (Eq. 2) are most informative, but no experiments validate this hypothesis. \n2.\tTable 2's ablation shows that adding implicit procedure queries alone (row 2, k=1) yields almost no improvement over the baseline (+2.2). The gain only appears in row 3 after incorporating the full Stage 1 pre-training. This demonstrates that the learnable queries contribute negligibly. The actual performance boost comes from the expensive Stage 1 pre-training itself, not from the \"procedure modeling\" concept. \n3.\tStage 1 trains the encoder to reconstruct sampled frames, yet Stage 2 replaces them with learnable queries. If the frames are good training targets, why not use them directly for captioning as a simpler end-to-end approach? Additionally, the multi-granularity masking rationale (lines 236-237) is vague, and the specific schemes/probabilities (0.1, 0.7, 0.1, 0.1) appear without ablation justification.\n4.\tThe model design is relatively complex, consisting of two training stages, with the first stage further divided into three submodules. Compared to other non-LLM-based models, the overall performance improvement is not significant, and some metrics (M, R) are relatively low.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aaMmEX6wuP", "reviewer": "Reviewer_REM5", "summary": "In this paper, the authors focused on the task of change captioning. Considering the existing methods ignore the temporal dynamics of the change procedure, the authors proposed the ProCap framework. Specifically, ProCap explicitly models the spatio-temporal changes between image pairs. Based on the first stage, the second stage can generate rich descriptions by implicitly reasoning over the modeled change procedure.", "questions": "As listed above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The designed framework reformulates change captioning from static comparison to dynamic procedure modeling, which can capture the rich temporal dynamics.\n2. The proposed explicit procedure modeling module can produce continuous frames between static image pairs, which facilitates the change captioning.", "weaknesses": "1. The method’s performance across the three datasets does not consistently outperform baseline methods, indicating that its overall robustness may not be sufficiently strong.\n2. ProCap employs a non-LLM-based backbone, and it remains unclear how its procedure modeling module would still perform when integrated with a powerful LLM-based backbone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7lULGt8ADr", "reviewer": "Reviewer_weBZ", "summary": "This paper addresses the task of Change Captioning, which aims to generate textual descriptions of the differences between two similar images. The authors argue that existing methods, which rely on static image pair comparison, fundamentally fail to model the rich temporal dynamics of how a change occurs. To address this limitation, the paper proposes ProCap, a novel two-stage framework that reformulates the task from static comparison to dynamic procedure modeling. \nStage 1: Explicit Procedure Modeling: This stage trains a \"procedure encoder\" to capture the latent dynamics of a change.\nStage 2: Implicit Procedure Captioning: This stage handles the final captioning task.", "questions": "1.\tWhen there are significant scene differences between the two images, how does the procedure modeling module generate intermediate frames that are meaningful for the change analysis?\n2.\tThe ablation studies were performed exclusively on the CLEVR-Change dataset, which is insufficient for evaluating the proposed method's effectiveness in real-world scenarios (such as Spot-the-Diff).", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tNovel Problem Formulation: The paper introduces a conceptual shift by reformulating change captioning from a static image comparison task into a dynamic procedure modeling problem. This directly addresses a key limitation of prior work, which largely ignores the rich temporal dynamics of how a change unfolds.\n2.\tEfficient Architecture: \n(1)\tStage 1 effectively learns a rich representation of spatio-temporal dynamics by training an encoder on explicitly generated and sampled keyframes.\n(2)\tStage 2 introduces learnable procedure queries as an efficient solution. This allows the model to implicitly reason about the change process during inference, bypassing the computational cost and noise associated with synthesizing frames.", "weaknesses": "1.\tIt does not show a clear performance advantage on Spot-the-Diff over other SOTA methods.\n2.\tAs shown in Table 4, the consistency loss does not yield a substantial performance gain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IkxKaTX8wb", "reviewer": "Reviewer_hLpn", "summary": "The paper presents a novel perspective on the task of Change Captioning. Instead of merely focusing on static image pairs, it emphasizes modeling the changing process between similar image pairs. This process captures rich spatial-temporal information, enabling the model to better understand how changes occur and to generate more precise descriptions of them. Extensive experiments demonstrate that, even without a complex framework design, the proposed method effectively leverages the learning of dynamic change processes to overcome challenges caused by distractor variations, achieving strong and robust performance.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a fresh and meaningful perspective on the change captioning task by modeling the dynamic process of change instead of focusing on static image pairs, which makes the approach both novel and insightful. \n2. The proposed framework, ProCap, demonstrates an elegant design that effectively captures rich spatial–temporal information by explicitly modeling the changing process. Moreover, its use of implicit procedure queries enables efficient and effective inference. Experiments show that ProCap surpasses existing non-LLM-based methods on two datasets and achieves competitive results on the remaining one.", "weaknesses": "1.As shown in Table 1, the method performs slightly worse on Spot-the-Diff; further analysis would be informative.\n2.Presenting some failure cases would provide deeper insights into its limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)