MIMIC: Mask-Injected Manipulation Video Generation with Interaction Control
Authors: Tianxiao Chen
Jintao Rong
Huajin Chen
Jingya Wang
Tao Zhou
Jiming Chen
Qi Ye
Venue: ICLR 2026 Poster (2026)
Keywords: video diffusion model
manipulation video
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Embodied intelligence faces a fundamental bottleneck from limited large-scale interaction data. Video generation offers a scalable alternative, but manipulation videos remain particularly challenging, as they require capturing subtle, contact-rich dynamics. Despite recent advances, video diffusion models still struggle to balance semantic understanding with fine-grained visual details, restricting their effectiveness in manipulation scenarios. Our key insight is that reference videos provide rich semantic and motion cues that can effectively drive manipulation video generation. Building on this, we propose MIMIC, a two-stage image-to-video diffusion framework. (1) We first introduce an Interaction-Motion-Aware (IMA) module to fuse visual features from the reference video, producing coherent semantic masks that correspond to the target image. (2) then utilize these masks as semantic control signals to guide the video generation process. Moreover, considering the ambiguity of the motion attribution,  we introduce a Pair Prompt Control mechanism to disentangle object and camera motion by adding the reference video as an additional input. Extensive experiments demonstrate that MIMIC significantly outperforms existing methods, effectively preserves manipulation intent and motion details, even when handling diverse and deformable objects. Our findings underscore the effectiveness of reference-driven semantics for controllable and realistic manipulation video generation.

Published as a conference paper at ICLR 2026
MIMIC: MASK-INJECTED MANIPULATION VIDEO GENERATION WITH INTERACTION CONTROL
Tianxiao Chen'
Jintao Rong Huajin Chen
Jingya Wang
Tao Zhou' Jiming Chen
Qi Yel.dt
"Zhejiang University *Zhejiang University of Technology "ShanghaiTech University
'Zhejiang Key Laboratory of Airspace Awareness and Autonomous Unmanned Systems
Show
Reference Example
Interaction Videos Generation By MIMIC
Images
"A block gripper fold's the cloth
"A white gripper picks apple
Maving blue pen comera
Figure We propose MIMIC, a novel approach for video generation in manipulation scenarios. Given reference video, MIMIC conditions it to generate new video that preserves the same operational semantic information.
ABSTRACT
Embodied intelligence faces fundamental bottleneck from limited large-scale interaction data. Video generation offers scalable alternative, but generating manipulation videos remain particularly challenging, as they require capturing subtle, contact-rich dynamics. Despite recent advances, video diffusion models still struggle balance semantic understanding with fine-grained visual details, restricting their effectiveness in manipulation scenarios. Our key insight is that reference videos provide rich semantic and motion cues that can effectively drive manipulation video generation. Building this, we propose MIMIC, two-stage image-to-vitee toviieoo diffusion framework: (1) we first introduce an Interaction -ieeactioo-MMotion- Aware (IMA) module to fuse visual features from the reference video, producing coherent semantic masks that correspond to the target image, (2) then utilize these masks control signals to guide the video generation process. Considering the ambiguity the motion attribution, we further introduce a Pair Prompt Control mechanism disentangle object and camera motion by adding the reference video as an additional input. Extensive experiments demonstrate that MIMIC signifi- cantly outperforms existing methods, effectively preserving manipulation intent and motion details, even when handling diverse and deformable objects. Our find- ings underscore the effectiveness of reference-driven semantics controllable and realistic manipulation video generation.
INTRODUCTION
Embodied intelligence has made notable progress Black al. Bjorck al. (2025); Agarwal et al. 2025), but its development still hindered by the scarcity of large-scale, high-quality interaction
Corresponding author.

---

Published as a conference paper at ICLR 2026
data A promising alternative is to learn from manipulation videos, which naturally encode rich interaction cues and can provide valuable guidance for embodied agents et al. (2025). Building on this idea, video generation He et al. (2022): Wan et al. (025) offers scalable solution by not only leveraging existing videos but also simulating realistic new ones, thereby augmenting training data and promoting more generalizable robot learning. Generative models in this setting can capture intrinsic vidco patterns and synthesize target scenes from language descriptions, e.g. "a person fold- ing clothes at home". Yet, language alone is insufficient for manipulation scenarios, where subtle motions and contact dynamics must be faithfully represented.
Generating realistic manipulation videos is particularly challenging because they involve complex, contact-rich interactions between hands (or grippers) and objects. Although recent video diffusion models Xing et al.] (2024); Guo et al.] (2023); Yang et al. (2024) have advanced rapidly, they still struggle to balance abstract semantic understanding with fine-grained visual detail (Tan et al. (2025, making it difficult to capture the nuances of manipulation behaviors. Demonstrations, in contrast, naturally convey both high-level semantics (e.g. folding clothes) and fine-grained interaction cues. This observation inspires our key idea: Show a Reference Example to the Model-that is, guiding diffusion models with reference video alongside textual descriptions to generate manipulation sequences.
Existing general-purposs methods incorporate additional control signals, such drag points Yin et al. (2023), object depth Xu et al. (2024b), or hand meshes Fan et al. (2025), to explicitly constrain motions. While effective for certain tasks, such strong constraints often reduce flexibility and may even produce physically implausible results. FlexiAct ZZhng et al. (2025) adopts a reference video strategy similar ours and extracts global motion representations from the reference video for gen- eral video generation. However, when applied to manipulation generation, these methods struggle to effectively handle manipulation scenarios due to complex motions between multiple objects. In addition, the generated videos demonstrate scale inaccuracies and incorrect modeling of interac- tions as (1) the reference and target scenes commonly exhibit large misalignments in manipulated objects, initial poses, and background contexts; and (2) the model often produces physically implau- sible motions, since it strictly follows the imposed control signals while neglecting the underlying causal dependencies of real-world interactions. These limitations highlight the intrinsic difficulty of manipulation generation, as success demands both structural alignment across scenes and explicit reasoning about the physical dynamics of interactions.
To address the above challenges, we propose a novel manipulation generation framework MIMIC by opening the black box of single-stage generation and explicitly inject the capability of manipulation-centric understanding. improving interpretability and controllability. Specifically, given a reference video, the first frame of the target scene, and textual description, we decompose generation into two stages. The first stage is trained to jointly identify the object to be manipulated in the target initial frame and synthesize a temporally-coherent, physically-plausible interaction mo- tion trajectory, which is represented as sequence of masks. A novel diteraction-MooionAAweeeee embed interaction semantics to guide subsequent video synthesis by
(IMA) layer is proposed learning IMA embeddings from visual and mask embeddings from reference videos and injecting the embeddings into the generation process via IMA attention.
Given the generated mask sequence and the target initial image, the second stage renders the final realistic video. We observe that using a single mask as the control signal couples object motion with camera motion due to the lack of background information, which makes it challenging to capture interactive motions within the video. To accommodate scenarios with possible camera motion, introduce Pair Prompt Control mechanism that conditions the rendering stage on both the predicted interaction mask and the original reference video. This approach resolves the inherent ambiguity in mask-based control, allowing the model to disentangle object motion from camera motion and generate temporally coherent videos that respect the global scene dynamics.
We construct dedicated benchmark for video generation of manipulation scenarios that includes human hands and grippers to evaluate the performance of MIMIC Experiments demonstrate that our method effectively transfers manipulation information from the reference video the generated video, demonstrating both plausible motion patterns and the fine visual details of video. As shown in Fig. [] MIMIC exhibits strong capabilities across diverse manipulation scenarios and maintains high-quality generation even when handling deformable objects.
2

---

Published as a conference paper at ICLR 2026
Our key contributions summarized as follows:
We propose MIMIC, novel image-to-video diffusion framework tailored for manipulation sce- narios, which leverages semantic extraction from reference videos combined with explicit inter- action masks to generate physically plausible and semantically accurate manipulation videos.
We design Interaction-Notion-AWie nipulation semantics into the video generation process, addressing the challenges of complex mo- tion representation without relying on predefined control signals.
attention mechanism that effectively embed latent ma-
We propose Pair Prompt Control mechanism that integrates reference videos with predicted interaction masks, enabling the model to effectively incorporate semantic information and reduce ambiguities inherent mask-based control, thereby enhancing the coherence and realism of gen- erated manipulation videos.
RELATED WORK
2.1 IMAGE-TO- VIDEO DIFFUSION MODELS
Leveraging the advantages of diffusion models Ho et al. 20200): Song et al. (2020), the field of im- age generation has witnessed remarkable advancements in content creation Rombach et al. (2022); ; Nichol al. (2021): Betker et al. (2023). Following this success, video diffusion He al. (2022)
rapidly evolves by integrating high-fidelity image priors with temporal modeling to synthesize co- herent short videos. Early Image-to- Video (I2V) diffusion work begins with AnimateDiff Guo al. (202)), which introduces a lightweight Motion Adapter and MotionLoRA to animate Stable Dif- fusion Rombach et al. (2022). followed by SVD Blattmann et al. (023), which augments image- conditioned diffusion with temporal convolutions and inter-frame attention. Building on these UNet- based methods, DynamiCrafter Xing al.) (2024) employs a dual-stream injection of visual detail and CLIP-aligned context to animate open-domain images into videos. More recently, DiT-based architectures Peebles & Xie (2023) such as VideoX Yang et al. (2024) combine a 3D VAE with an Expert Transformer to generate high-fidelity video, and Wan2. I Wan et al. (2025) fuses causal 3D VAE with a Diffusion Transformer and shared MLP temporal embeddings to scale generation to arbitrary lengths. We inherit the 12V diffusion backbone and its strong video priors, but apply it to manipulation by explicitly modeling hand-object interactions.
2.2 VIDEO MOTION CUSTOMIZATION
Motion customization seeks to generate videos that replicate specific motion patterns from refer- ence videos while aligning with textual semantics. Early works such as Tune-A-Vides Wu et al. (2023) enable shot video generation by fine-tuning Stable Diffusion Rombach et al. (2022). ControlVideo Zhao et al.] (2025b) and Vext2Video-Zero Baadeo-ZZeroKKaccattyaaaae Khachatryan et al. (2023) extend Control- Net Zhang et al. (2023) with cross-frame interactions for zero-shot controllable video synthesis, while Control-A Video Chen al. (2023) incorporates trainable motion layers to model temporal dynamics conditioned on diverse cues. To better decouple appearance and motion, methods like VMC Jeong et al.] (2024), MotionDirector Zhhao et al.] (202)). and MotionInversion [Wagg et al. scenes and prompts. FlexiAct /Zhang 2025) enables one-shot complex action transfer by combining spatial adapters and a trainable frequency-aware embedding. Training-free strategies, including DMT Yatim et al. (2024) and Mo- tionClone Ling et al. (2024), extract motion priors from latents for inference-ting customization. Unlike the above approaches, we employ an in-context video generation paradigm that leverages motion information from a reference video to produce the corresponding manipulation videos.
2024
train motion-specific modules that generalize
et al.
2.3 INTERACTION VIDEO GENERATION
Generating interactive manipulation videos faces significant challenges, including ensuring physical plausibility and handling occlusions among multiple objects. To address these issues, some meth- ods Xu et al. (2024b); Pang al. (2025); Fan et al. (2025) incorporate fine-grained control signals to specify detailed manipulation processes. For instance, AnchorCrafter Xu et (2024b) uses hand meshes and object depth maps inputs, while Re-Hold Fan et al. (2025) describes interactions
3

---

Published as a conference paper at ICLR 2026
through hand-object bounding boxes. Other approaches aim for the model implicitly learn phys- ical motion patterns. CosHand Sudhakar et al. (2024) and InterDyn Akkerman et al. (2025) utilize explicit hand masks control signals guide the learning of object motions influenced by hand movements, and Toste-Rob Zhao et al. (2025a) similarly controls via hand keypoints. Beyond direct video generation, recent work has also explored inpainting- baade editing hand-object interaction content. AffordanceDiffussion Ye et al. (2023) edits object-only images by inserting plausible hand configurations to create HOI scenes, while HOI-Swap Xue al. (2024) and Re-Hold Fan et al. 2025) edits HOI videos by replacing the interacted object while keeping the hand motion intact. These methods focus on modifying existing content rather than generating novel interaction videos from scratch. Furthermore, recognizing the difficulty of directly generating manipulation videos with diffusion models, several methods adopt a coarse-to-fine learning strategy by first predicting coarse control signal, then generating videos conditioned on it. For example, Ang2Flow2Ach Xu itt al. (2024a) generates object motion trajectories via diffusion models, and FLIP Gao et al. (2024) performs uniform pixel-space sampling and predicts trajectories for each point. Existing methods rely on a single control signal to constrain the limited motions of generated videos. In contrast, our approach decouples multiple motion features from reference videos as control signals, thus avoiding complicated inputs while accurately capturing the manipulation process.
3 PRELIMINARY
Our method is built upon the image-to-Vidde model DynamiCrafter/Xing et al. (2024), which mainly comprises a diffusion UNet Ronneberger et al. (2015) éş with spatial and temporal layers, and a vari-
ational autoencoder (VAE) Kingma & Welling (2013) composed of an encoder & (-) and decoder (-). During training, a video with Irames space as =f E(x R RFxxxxxx,
x 1.ff Xxxxxx is encoded into latent
The forward diffusion process corrupts the video via
I:F = Vār 2 + VI a N(0,I,
(1)
where / E (1....,7 T indexes time steps and o, controls the noise scale. The reverse process is modeled by the diffusion UNet eg, which estimates the noise given noisy latent, conditioning on the initial image of the target scene Har and the textual description c. Diffusion model is trained using the following loss:
Ldifs Egg. .ae. || - co(2) ,c, (etrr 1)ll
During inference, starting from Gaussian noise e, the model iteratively denoises with T timesteps to obtain an estimated clean latent sl=F, which is then decoded to the realistic video by D (-).
4 METHODOLOOGY
4.1 OVERVIEW OF MIMIC
As shown in Fig. we suggest an in-context video-generation paradigm: conditioned on a refer- ence manipulation video Vref. an initial image Ieer of the target scene, and textual description the model is required to produce corresponding manipulation video Vtar in the target environ- ment. The generation process is explicitly divided into two stages to promote the understanding of interactive dynamics and enhance the controllability of the manipulated object state. The first stage aims at jointly identifying the manipulated object the target initial frame Itar and synthesizing a temporally-coherent, physically-plause quence of masks Mear to achieve pixel-level control of the state of the object while accommodating possible non-rigid deformations. Given the predicted mask sequence Mtar and the target initial im- age hor. the second stage renders the final realistic video, where we propose a Pair Prompt Control mechanism to accommodate scenarios with potential camera motion.
motion trajectory. The trajectory is represented by a
4

---

Published as a conference paper at ICLR 2026
Stage I: Inferaction Motion Mask Generation
Stage II: Video Generation
Prosen
:rairable
Concetonals
IMA Loyer
Motion Extractor
IMA Attention
Munpyy
JANA Erooddng
CLIP Vilce Crocder
Aftantion
Vicee Rebbedd eg Mrt
FRN
Refarenca Foul
Quary
Projestion
VIEE
Repeot
Target for
Internation Naak Creadding
Image
Pre- trained UNet
Reference
Isteraction Mast
Maf
Control Module
Ten
VAE
Example Pair
Query Encoder
Pair Ensodar
Image lur
Torget Video
Roi lneerc ViSto Dnfancchinn Moa
Referance Video Vis!
Mar
UNet
Pair Prompt Control
Torget
Figure 2: Pipeline of MIMIC. (1) Stage illustrates interaction motion mask generation. We construct Motion Extractor to capture the interaction motion information from the reference video. And then we utilize transformer layer after the spatial layer to inject this motion information into the UNet by Interacion-MottoonAAvan (IMA) attention. (2) Stage II illustrates video generation with interaction masks. We integrate a set of example pairs, each consisting of a reference video and its corresponding interaction mask, with the predicted interaction mask of the target video through the Pair Prompt Control module. This integration is injected into the UNet to facilitate video generation.
4.2 STAGE I: JOINT PERCEPTION AND INTERACTION MOTION GENERATION
Given reference manipulation video Vref, reference manipulation masks Mre and textual de- scription c, Stage is explicitly trained jointly recognize the manipulated object the target initial frame lar and synthesize a temporally-coherent by masks M
motion trajectory represented
To fully leverage the interactive semantic information and motion information contained in the ref- erence video Vref, we introduce reference video embeddings fV.f and interaction mask embeddings frey, which are respectively extracted from the reference video Vref and the corresponding inter- action mask Myef by frozen CLIP Radford et al.) (021) visual encoder t. Complementary to these semantic embeddings, lightweight Motion Extractor is further employed inject reference motion cues into the denoising U-Net. To further enhance the alignment of interactive semantics and motion information, we fuse learnable query embedding q E Fxxxx with interaction mask embedding fin via clement-wise addition, yielding an accumulated query 4m = + fiet as the input to the extractor. The accumulated query interacts with the frozen video embeddings through a cross-attention layer (CA), followed by a feedforward network (FFN), obtaining the Interaction- Moti)n-Aware(IMA) embedding rIMA"
JIMA (FN((CA(mms))) Prets (N
This IMA embedding FIMA then interacts with the diffusion model via another cross-attention layer, so that the diffusion process is guided by the understanding of manipulation semantics. To maintain the stability of training, the output projection layer inside this attention layer is aaro-initialized and equipped with residual connection.
For the training strategy, we train the Stage I model in a two-step manner. Firstly, we construct static videos with repeated first frames, allowing the model focus exclusively on learning to recognize the hand-object interactions occurring in the first frame of the target scene. Subsequently, we restore temporal dynamies and train the model generate manipulation motions. Both phases optimize the objective in Eq. 2
For data without annotated masks, utilize SAMunding-SAM22RRR et al. with language inputs to generate corresponding masks.
5

---

Published as a conference paper at ICLR 2026
4.3 STAGE II: VIDEO GENERATION WITH INTERACTION MASKS
The objective of Stage II is to generate a temporally coherent and detail-rich video from the interac- tion masks Mter predicted in Stage I. Mask-only conditioning is inherently ambiguous: it specifies where an interaction occurs but cannot disambiguate object versus camera motion or capture how the manipulation unfolds. In manipulation centric scenarios, this limitation often results in weak consistency along the interaction trajectory and unrealistic rendering of hands or grippers.
To address these issues, we propose Pair Prompt Control, which conditions generation on both the target mask sequence Maar and a reference pair Mref. Vref. While the target mask provides spa- tial alignment, the reference pair contributes semantic and motion priors, thereby reducing mask ambiguity and enabling manipulation-awaree synthesis. Architecturally, we adopt ControlNet- style Zhang et al.] (2023) control branch, where lightweight convolutional Query Encoder and Pair Encoder modules are used process the target mask sequence Miar and the example pair Mref. Vref respectively, The encoded features are fused within a control module, which then in- jects multi-scale guidance into the UNet backbone, ensuring reference-drie -driven conditioning through- out the generation process.
To enhance fidelity and consistency within interaction regions, we use the Stage I predicted mask sequence Mzar together with the target image Iar to form a masked image maskkod = 1,ar @ m that preserves only the interaction areas. This masked image is concatenated with the original target image as input to the diffusion model, providing explicit appearance guidance. Additionally, we reweight the diffusion loss Laifs with an adaptive region loss that emphasizes mask-aligned areas across time by combining the current interaction mask m f = 1,.., F and the first-frame mask mtari
S
S Maar) @ Laiss,
Lregion
SMar Mear +
SMtar
(4)
Mlar Repeat(m F),
where Repeat(rr, n) repeats x along the temporal dimension, and SMaan SMy... denote the corre- sponding mask areas. The final training objective is defined as:
Lfinal = -Mar Miar) O Laifs + XLregion-
(5)
By combining iasked-image conditioning with adaptive region loss, the model focuses learning on the relevant regions, reducing ghosting artifacts and improving both visual fidelity and temporal consistency of generated videos.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETTING
Datasets. We curate subsets from three public benchmarks Something Something-v2 v2 (SSv2) Goyal ct al.) (2017), BridgeV2 Walke et al.] (2023), and Fractal Brohan et al.] (2022) to ensure sufficient temporal coverage. We annotate total of 20,000 human hand interaction videos and 40,000 gripper interaction videos for training. We organize all samples into structured manipulation templates and randomly sample two videos from the same template, one as the reference and the other as the target. We collect 240 samples for evaluation, Both reference videos and corresponding target videos of evaluation pairs are unseen during training. Considering that some comparative methods ZZhng et al.] (2025); Zhao et al. (2024) require additional training for different reference videos, we further split the evaluation samples into 48 groups, where each group contains one reference video and five target images to ensure uniform coverage across different manipulation classes. Detailed data curation is provided in the Appendix A.
Comparison Methods We compare our method with representative image-to-video motion trans- fer approaches: (1) DynamiCrafter Xing et al. (2024) and Cog VideoX Yang et al. (2024), further adapted via one-shot fine-tuning the reference video; (2) MotionClone Ling et al. (2024),
training- free motion cloning framework; (3) MotionDirector Zhao et al. (2024), a dual- path LoRA- based model that decouples appearance and motion learning; and (4) FlexiAct Zhang et al. (0255)
a learnable global motion transfer method. Except for MotionClone, all baselines require extra fine- tuning with the reference video. Training details and hyperparameters are provided in Appendix B

---

Published as a conference paper at ICLR 2026
Tonget Image
Torget image
Dynami- Crafter
CogVideoX
Motion Clone
Motion Director
FlexiAct
Language prompt
Language prompt
liting up drop dowe.
MIMIC
plares upright
MIMIC
Figure 3: Qualitative comparison of video motion transfer results. Each row shows frames gen- erated by a different method conditioned on the same reference video in two challenging scenarios. The yellow box marks the object state in the reference video; the red box marks the object state in the comparison video; and the green box marks the object state in our video.
Implementation Details. Our training dataset consists of 16-frame videos with resolution of 320 x 512 pixels. Both stages are initialized from pretrained DynamiCrafter Xing et al. (2024) weights and optimized with AdamW on two NVIDIA H100 (8GGB) GPUs. The batch size set to 4 and the learning rate to 1 x 10-5 throughout training. In Stage I, we freeze the temporal layers and finetune only the spatial and IMA layers of the UNet backbone. We first train for 5,000 steps on repeated first frames to preserve spatial fidelity, and then train for 50,000 steps on full video sequences to learn motion. Stage II, we finetune the video generation model for 25, 25,000 steps. The trainable components consist of the query encoder, the pair encoder, and the control module initialized from UŇet weights. To stabilize training, we introduce the adaptive region loss after 5,000 steps and apply sigmoid-base based nonlinear warm-up over 2,000 steps mitigate abrupt loss changes. At inference time, we adopt 50 DDIM sampling steps and set the CFG scale to 7.5.
5.2 EVALUATION
Evaluation Metrics. Our metrics consist of four aspects. Firstly, we employ CLIP-based Radford jet al. (202I) text alignment and appearance consistency to reflect Perceptual Similarity. Sec- ondly, Temporal Quality is evaluated using subject consistency and background stability from VBench (uang et al. (2024). Thirdly, we utilize the multimodal large language model (MLLM) Bai et al. (2023); Lin et al. (2023) in evaluation, as it possesses superior semantic understanding capabil- ities. Finally, we conduct extensive Human Preference evaluations via user studies. More details can be found in Appendix. and C.3
Quantitative and Qualitative Analysis. Our quantitative evaluation results are summarized in Tab. [ and Fig. B Our method achieves the best performance in terms of temporal quality and appearance consistency, demonstrating its superior capability in preserving visual fidelity. It also performs strongly in text alignment, ranking second only to MotionClone. However, MotionClone underperforms on other metrics and exhibits weaker preservation of input image fidelity. Traditional quantitative metrics tend to emphasize low-level pixel alignment, but they are limited in their ability to assess aspects that require higher-level semantic understanding. For example, such metrics cannot reliably determine whether an object with a specific pose has been lifted correctly, or whether the model interacts precisely with the intended object multi-object scenes. To address this limita- tion, we additionally incorporate evaluations using multimodal large language model(MLLM) Lin ct al. (2023), which possesses strong high-level semantic reasoning capabilities. Due to space con- straints in Tab. we report two dimensions that most relevant manipulation tasks, Interaction

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative comparison of manipulation video generation. We report several automatic evaluation metrics alongside human preference rates. Participants are asked to select the top2 videos, making the subjective metric more robust and reliable.
Perceptunl Similarity
Temporal Quality
MLLM Evalustion
Method
Extra Finetune
Text Aligament
Appeurance Consistency
Subject Condstency
Background Stabality
Interaction Rationality
Semantic Similarity
Preference
Human
DynamiCrafter
0.2684
0.8781
0.9185
0.9331
3,0543
2.4346
8.86%
CogVideoX
0.2667
0.8537
0.8126
3.1318
2.3736
MotionClone
0.7400
0.6833
2,1277
0.909)
MotionDirettor
0.2658
0.8336
0.8542
0.9160
3.1489
2.11.19
0,96%
FlexiAct
0,2694
0.8999
0.8921
0.9230
3,5529
2,5238
27.8%
Ooe Sin-Sttge
0.2688
0.8709
0.8591
0.9130
3,6170
2.4468
weo IMA Atteation
0.2548
0.8537
0.8418
019029
3,6216
2.4134
Pair Prompt Coutrol
0.2677
0.8862
0.9172
3.8789
2.2526
MIMIC
0.3721
0.9084
0.9291
0.9385
4.1381
2,9127
42,88%
Rationality and Semantic Similarity, while additional dimensions are provided in Fig. In the MLLM-based evaluation, method demonstrates superior performance in terms of operational completeness, accuracy, and semantic consistency with the reference video. Finally, we conduct human preference study. To mitigate extreme biases, participants are asked select their top2 preferred results out of 6 candidates. Our method consistently receives a clear majority of prefer- ences, further validating its effectiveness in generating manipulation videos that align with human judgment. Details on the MLLM evaluation can be found in the Appendix,
Action
Smoothness
Physical
Ours
Rationality
Interaction
39.8%
Authenticlty 4004
DynamiCreftar
Sementic Similarity
aR Intersction Rationaity
CogVideex
479%
Mollord
Consistency
Temponal
40.6%
Temporal Consistency
Mot ondirecter
Physical
2.9127
Semantic Siiliority
PladAct
Authenticity
Human Preference
MLLM Evaluation
Figure 4: Qualitative results of Human Preference and MILM Evaluation. Right: human preference selection ratios, where participants asked choose the top2 videos from all options from various perspectives. Left: MLLM scores for the generated videos across different evaluation dimensions.
5.3 ABLATION STUDY
Two-Stage One-Stage We first validate the rationale behind the proposed two-stage generation strategy, which progresses from motion patterns to visual details, by conducting an ablation study on single-stage direct video generation. In this experiment, all training settings remain consistent with those used in our Stage I model training, the ground truth video frames are used as supervision signals instead of masks. As shown Fig. [ videos generated by the diffusion model in a single stage suffer from severe visual quality issues. However, we observe that the interaction motion information conveyed roughly matches the reference videos and our results. This further confirms the rationality of using Stage I to learn interaction motion patterns by generating masks.
Video 1
Torger Troge
"iiiing and letting down."
Motion
Latguoge promp!
Reference
One-Stage
(wa-Stage(Ours)
Figure 5: The results of one-stage and two-stage(ours) generation strategies.
Effeet of IMA and Pair Prompt Control To verify the effectiveness of our approach in generat- ing semantically consistent videos from reference videos, conduct ablation studies the IMA
8

---

Published as a conference paper at ICLR 2026
Torget Image
Reference
w/o IMA Attention
Video
IMA Attention
prompt
FA white gripper move
can"
Predktted (5tsgel)
(5
w/o Pair Prompt Control
Beckground
Pair Prompt Control
Languige promp?
"Moving down"
Ground Truth Masks
(Shogazi)
Figure 6: Qualitative results of ablation study. We show the interaction mask from Stage and the generated video from Stage II. On the left, IMA attention enables semantic motion alignment with the reference video, while without it the video fails to match the language prompt. Right: Pair Prompt Control improves mask understanding: without it, the background moves with the mask, re- sembling camera motion rather than interaction. With Pair Prompt Control, the background remains stable, and changes arise only from true motion.
Attention module in Stage and the Pair Prompt Control module Stage II. Quantitative results presented in Tab. [ It be observed that removing the IMA Attention module (w/o IMA) leads to a clear drop in Semantic Similarity (2.41 vs. 2.91) and degrades other metrics due to the lower-quality masks predicted in Stage I. As shown in the top of Fig- 6 the model misinterprets the prompt and manipulates the wrong object. illustrating the semantic inconsistency caused by the absence of IMA. Without Pair Prompt Control, the explicit mask struggles capture complete ma- nipulation information due to the coupling of object motion and camera movement, which leads to slight performance decline across various metrics. As shown the example (Fig. o bottom), the background drifts with the mask, indicating that the model generates camera motion instead of the hand-object interaction. Overall, IMA preserves high-quality semantic mask prediction, whereas Pair Prompt Control disentangles camera motion and injects fine-grained appearance information. Additionally, we provide further ablation studies on the region loss used in Stage II, detailed in the Appendix. D
CONCLUSION
In this paper, present MIMIC, two-stage image-to-video generation framework designed for manipulation scenarios, which effectively leverages semantic information extraction and explicit in- teraction masks to produce physically plausible and semantically consistent videos. Our approach overcomes key challenges of existing methods by disentangling camera and object motions through Pair Prompt Control mechanism, and enhancing temporal stability with an adaptive region loss. Limitations and future work: Limited by the capacity of the base model Xing et al. (2024), videos generated by our method are currently restricted maximum 16 frames, which precludes the generation of long-horizon videos depicting complex operations. Adopting more powerful foun- dational models can facilitate our approach to synthesizing longer temporally coherent videos with more complex action compositions.
7
ACKNOWLEDGMENT
This work was supported in part by NSFC under No.62233013, 62293511, Key Research and Development Program of Zhejiang Province under No. 2025C01064, Fundamental and Inter- disciplinary Disciplines Breakthrough Plan of the Ministry of Education of China under No.

---

Published as a conference paper at ICLR 2026
JYB2025XDXM103. We extend sincere gratitude to Qingtao Liu and Haoming Li for their insightful discussions and writing assistance.
REFERENCES
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chat- topadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03577 2025.
Rick Akkerman, Haiwen Feng, Michael J Black, Dimitrios Tzionas, and Victoria Fernández Abre- vaya. Interdyn: Controllable interactive dynamics with video diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 12467-12479, 2025.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023.
Johan Bjorek, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, al. Gr0Ot nl: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.144 14734, 2025.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. T0: A vision- Dision-language-acte flow model for general robot control. corr, abs/2410.24164, 2024. doi: 10.48550. arXiv preprint ARXIV.2410.2414
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominík Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15112 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control scale. arXiv preprint 22xv:2221..068) 2022.
Weifeng Chen, Yatai Ji, Jie Wu, Hefeng Wu, Pan Xie, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video diffusion models with motion prior and reward feedback learning. arXiv preprint 2rii:22305,13844 2023.
Yingying Fan, Quanwei Yang, Kaisiyuan Wang, Hang Zhou, Yingying Li, Haocheng Feng, Errui Ding, Yu Wu, and Jingdong Wang. Re-hold: Video hand object interaction reenactment via adaptive layout-instructed diffusion model. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 17550-17500 2025.
Chongkai Gao, Haozhuo Zhang, Zhixuan Xu, Zhehao Cai, and Lin Shao. Flip: Flow-centric gener- ative planning as 2024.
manipulation world model. arXiv preprint arXiv:2412.08261,
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. arXiv preprint arXiv:2307.04725. 2023.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.1321 2022.
10

---

Published as a conference paper at ICLR 2026
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-683 6851, 2020.
Ziqi Huang. Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024.
Hyeonho Jeong, Geon Yeong Park, and Jong Chul Ye. Vme: Video motion customization using temporal attention adaption for text-to-video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9211-9222211 2024.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In European Conference on Computer Vision, pp. 35. Springer, 2024.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15954-15964, 2023.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 2013.
arXiv preprint
Bin Lin. Yang Ye, Bin Zhu, Jiaxi Cui. Munan Ning, Peng Jin, and Li Yuan. Video- Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023.
Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-free motion cloning for controllable video generation. arXiv preprint arXiv:2406.05333 005338 2024.
Tyler Ga Wei Lum, Olivia Y Lee, C Karen Liu, and Jeannette Bohg. Crossing the human- robot embodiment gap with sim-to-real using one human demonstration. arXiv preprint arXiv:2504. Stti::2004122000 12609, 2025.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 2021.
Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. Mofa- video: Controllable image animation via generative motion field adaptions in frozen image-to- video diffusion model. In European Conference Computer Vision, pp. 111-128. Springer, 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199. 2023.
Youxin Pang, Ruizhi Shao, Jiajun Zhang, Hanzhang Tu, Yun Liu, Boyao Zhou, Hongwen Zhang, and Yebin Liu. Manivideo: Generating hand-object manipulation video with dexterous and gen- eralizable grasping. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 12209-12219, 2025.
William Peebles and Saining Xie. Scalable diffusion models with transformers. pp. 4195-4203 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. PmLR, 2021.
11

---

Published as a conference paper at ICLR 2026
Nikhila Ravi, Valentin Gabeur, Yuan- Minn-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXivv2408.000 007144 2024.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-wer open-world -wolld models for diverse visual tasks. arxiv 2024. arXiv preprint arXiv:2401.144 141199
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234-241. Springer, 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv
preprint arXivv2010.02502, 2020.
Sruthi Sudhakar, Ruoshi Liu, Basile Van Hoorick, Carl Vondrick, and Richard Zemel. Controlling the world by sleight of hand. In European Conference on Computer Vision, pp. 414 430. Springer, 2024.
Shuai Tan, Biao Gong, Yutong Feng, Kecheng Zheng, Dandan Zheng, Shuwei Shi, Yujun Shen, Jingdong Chen, and Ming Yang. Mimir: Improving video diffusion models for precise text un- derstanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 23978-23988, 2025.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pp. 402-419. Springer, 2020.
Homer Rich Walke, Kevin Black, Tony Z. Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning, pp. 1723-1736. PMLR, 2023.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arxiv preprint arXiv:2503. 203 14, 2025.
Luozhou Wang, Ziyang Mai, Guibao Shen, Yixun Liang, Xin Tao, Pengfei Wan, Di Zhang, Yijun Li, and Yingcong Chen. Motion inversion for video customization. arXiv preprint arXiv:2403.20193, 2024.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7623-7633, 2023.
Jinbo Xing. Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open- open-domain images with video diffusion priors. In European Conference on Computer Vision, pp. 397-417. Springer, 2024.
Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. arXiv preprint arXiv:2407. 15508, 2024a.
Ziyi Xu, Ziyao Huang, Juan Cao, Yong Zhang, Xiaodong Cun, Qing Shuai, Yuchen Wang, Linchao Bao, Jintao Li, and Fan Tang. Anchorcrafter: Animate cyberanchors saling your products via human-object interacting video generation. arXiv preprint arXiv:2411. 1...221111.7333 2024b.
Zihui Sherry Xue, Romy Luo, Changan Chen, and Kristen Grauman. Hoi-swap:
Swapping objects in videos with hand-object interaction awareness. Advances in Neural Information Processing Systems, 37:77132-77166 164, 2024.
12

---

Published as a conference paper at ICLR 2026
Zhuoyi Yang. Jiayan Teng. Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06/0 060772 2024.
Danah Yatim, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, and Tali Dekel. Space-time diffusion features for zero-shot text-driven motion transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, pp. 8466 8476, 2024.
Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song, Shub- ham Tulsiani, and Sifei Liu. Affordance diffusion: Synthesizing hand-object interactions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp-
22479-22489, 2023.
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Drag- nuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint S.Ki:23308088033 2023.
Xiaohang Zhan, Xingang Pan, Ziwei Liu, Dahua Lin, and Chen Change Loy. Self-supervised learning via conditional motion propagation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1881-1889, 2019.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference computer vision, pp. 3836-3847, 2023.
Shiyi Zhang, Junhao Zhuang, Zhaoyang Zhang, Ying Shan, and Yansong Tang. Flexiact: Towards flexible action control in heterogeneous scenarios. arXiv preprint WxXiv:2505.03736 2025.
Hongxiang Zhao, Xingchen Liu, Mutian Xu, Yiming Hao, Weikai Chen, and Xiaoguang Han. Taste-rob: Advancing video generation of task-oriented hand-object interaction for generaliz- able robotic manipulation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 27683 27683-27693, 2025a.
Min Zhao, Rongzhen Wang, Fan Bao, Chongxuan Li, and Jun Zhu. Controlvideo: conditional control for one-shot text-driven video editing and beyond. Science China Information Sciences, 68(3):132107, 2025b.
Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jia- Wei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization text-to-video diffusion models. In European Conference on Computer Vision, pp. 273-290. Springer, 2024.
A DATA CURATION
A. DATA PROCESS
Due to limitations of the pre-trained model, we only use videos with a length of 16 frames for training, which generally requires temporal sampling of the original data. However, this often leads to issues such as excessively large frame-to-frame variations and difficulty in capturing the full extent of the language prompt within the video, both of which significantly impact our training process. Additionally, motion- -luurred frames frequently occur in video data, especially in the Something- Something-ve (SSv2) Goyal al. (2017) dataset. Such blurring poses challenges for our training process, as it can affect the consistency and accuracy of learned motion representations. We have processed the data as follows:
(1) Redundant Frame Elimination: We utilize OpenCV to detect the magnitude of motion both of the initial frames and the final frames of each video segment to determine whether the scene remains static. Frames exhibiting negligible motion at either end are identified as stationary and subsequently removed to eliminate redundant static frames.
(2) Blurred Frame Elimination: We utilize OpenCV to implement method that calculates the variance of the Laplacian of each frame, providing an indicator its blur level. When this indicator falls below preset threshold, the frame is identified as blurry and consequently discarded.
13

---

Published as a conference paper at ICLR 2026
(3) Temporal Sampling: After removing redundant and blurry frames, we perform uniform sampling over the entire video and set a maximum allowable sampling interval. When the interval between sampled frames exceeds this threshold, indicating excessive variation between frames, the corre- sponding segment is discarded.
A.2 DATA ANNOTATION
Given a raw video V paired with a language prompt e, we generate the annotations following the stream below:
(1) Objects Segmentation: We employ Grounded SAM2 Ren et al. to achieve open-set segmen- tation of interactive objects, categorizing target objects into two types: active manipulators (e.g.. hands, grippers) and passive manipulated objects. For segmenting active manipulators, we prede- fine comprehensive language prompts according the data set used. For example, 'human hand' in SSy2l et al. (2017) and 'white robotic gripper' in Fractal Brohan et al. (2022). For passive ob- ject segmentation, 'placeholder' language labels have been provided in SSv2. For the Fractal Fractal|Brohan Brohan et al. (2022) and Bridge V2 Walke et al. (2023) datasets, we extract the first noun from the prompt as input to Grounded SAM2.
(2) Manipulation Template: We adopt a classification approach consistent with SSv2 Goyal et al. (2177). which categorizes videos using templates such as "move [something] down." " Specifically, we implement natural language processing pipeline with spaCy to extract structured action-object pairs from textual descriptions. The pipeline first normalizes the input text and then identifies the main verb in each sentence, including phrasal verbs like "place [something] upright" and "pick up [something]"' enabling precise action extraction aligned with established dataset standards. The structured language is used solely for this categorization step; at inference time, the model only requires a natural-languas description and a reference video, without relying on structured prompts.
A.3 DISSUSION ABOUT MOTION REPRESENTATION
We adopt a mask-based motion representation to model human-object interactions. This choice is motivated by the need for reliable supervision: high-quality global optical flow (Ted & Deng (2020), point trajectories Karaev et al. (2024), or part-level optical flow obtained by combining Co-Tracke(Karaev et al. (2024) with CMP Zhan et al. (2019) as used in MOFA-Videe NNu et al. (2024) are difficult to obtain for large-scale videos, particularly when objects undergo non-rigid or highly deformable motion. As shown in Fig. 7 these representations all degrade significantly in cloth-folding scenarios, where global flow, sparse tracking points, and part-level flow fail to capture coherent motion cues. In contrast, segmentation masks either manually annotated or automatically extracted via SAM2 Ravi el al. (2024) remain stable and consistent even under such challenging conditions.
Moreover, masks inherently encode the semantic separation between the manipulated object and the hand gripper, a property essential for role-conditioned motion generation in our pipeline. Alternative motion representations lack this role-aware structure and are more susceptible to failures under large motions complex deformations. While mask-based signals may limit the modeling of fine-grained shape dynamics, they provide interpretable and robust motion cues well suited to our task.
B DETAILS ON BASELINES
DynamiCrafter Xing al. (2024). We fine-tune the spatial layers of DynamiCrafter for 5,000 steps on ground truth videos and captions from the specific dataset mitigate the existing domain gap. And we further fine-tune on the reference video for an additional 300 steps. During inference, we 50 DDIM sampling steps and set the guidance rescale to 0.0.
Cog VideoX Yang et al. 2024). Unlike our method, which uses 16-frame videos for training, to preserve the prior knowledge of the pre-trained model, fine-tune VideiX using 49-frame videos. Similar to DynamiCrafter we employ LoRA to fine-tune Cog VideoX on the reference video for 5,000 steps to bridge the domain gap, and we further train on the reference video for additional
14

---

Published as a conference paper at ICLR 2026
Start
End
Frames
X
Optical
Flow
x
Tracking
Points
x
Part
Flow
Mask
Figure 7: Motion representation comparison in a cloth-folding scenario. Global optical flow (Row 2) deteriorates rapidly under large non-rigid deformation, leading to unstable missing es- timates in critical regions. Tracking points (Row 3) fail to maintain consistent correspondences on the deforming cloth, resulting in drifting jittery trajectories that do not capture coherent motion. Part-level flow (Row 4) suffers from even greater instability, as part assignments fluctuate across frames and produce noisy, visually inconsistent flow fields. In contrast, segmentation masks (Row 5) remain temporally stable and accurately preserve the global structure of the cloth throughout the folding sequence.
300 steps for one-shot scenarios. During inference, we the LoRA rank 128 and use 50 DDIM sampling steps.
MotionClone Ling et al. (2024). MotionClone is a training-free method, which extracts motion priors from the temporal attention matrix of a reference video and constructs an energy function to guide the sampling process. During inference, we set the number inference steps to 100 and the number of guidance steps to 40.
MotionDirector Zhao et al. (2024). We follow the setup of MotionDirector by first training the spatial LoRA on the target image for 300 steps, followed by training the temporal LoRA on the 16-frame reference video for 150 steps. During inference, we set noise prior to and use DDIM with 50 sampling steps.
FlexiAct Zhang et al. (2025). We follow the FlexiAct recommendation by training the frequency- the 49-frame reference video for 3,000 steps. During inference, we set
aware embedding(FAAB the transition timestep for the FAE to 0.8, the additional attention weight FAE to 1.0, and the guidance scale to 6.0.
C EVALUATION
C.1 AUTOMATIC METRICS
We provide a detailed explanation of the principles and meanings behind the automatic evaluation metrics used in our experiments:
(1) Perceptual Similarity: The metric includes Text Alignment as the average cosine similarity between the text prompt embedding and frame embeddings encoded by CLIP et al. (021
15

---

Published as a conference paper at ICLR 2026
Fromes uniformly sampled From Return the oction order in video. Here the template.
video lests for
approaches apple and grabs with its gripper. 2. The robot arm moves away from the holding the apple.
User
LLavA 4-
XXXC
helpful assistant and brilliant action similarity any given focus exclusively the actions (and directly related objects) they describe. Ignore differences subjects, context; wording-only assess whether
they depict the same type of action.
System
Do not infer actions that merely implied: consider what explicitly stated. Rate their action similarity scale from 5,
= completely different actions; 2 = loosely related actions; moderstely similar actions;
highly similar actions; 5 = identical actions.
with
first,
brief explanstion.
Few-shot Scoring
User
User
cuts a rope scissors.
User
picks up a metal cup.
1. hand sponge. drops ball.
2. hand
cable
A hand grasps a metol cup, lifting
Qwen
Squeazing sponge and dropping ball completely different actions.
Qwen
Both involve cutting with handheldd fool, and torget differ.
Qwen
5, Both describe the identical action of picking up metal cup with a hand.
Figure 8: Detailed pipeline for computing semantic similarity based on large-scale pretrained mod- els.
and Appearance Consistency as the average visual similarity between the first frame and subsequent frames to capture intra-video coherence.
(2) Temporal Quality Huang et al. (2024): A human-aligned spatiotemporal benchmark assessing Subject Consistency and Background Stability, For Subject Consistency, it measures whether the appearance of a subject remains consistent throughout the video by calculating DINO Oquab et al. (2023) feature similarity across frames. For Background Stability, it evaluates the temporal con- sistency of background scenes by calculating CLIP Radford et al. (2021) feature similarity across frames.
C.2 DETAILS OF MILM EVALUATION
Semantic Similarity To establish a reliable metric for evaluating the semantic consistency be- tween the reference video and the generated output, we utilize the prior knowledge embedded in large-scale pretrained models. Specifically, we temporally concatenate the reference and generated videos, inserting a buffer of two blank frames between the two segments to construct a combined video Veoncat Leveraging the video understanding capability of L.L we generate two captions that respectively describe the content of each video segment within concat-
Lin al. (003),,
Next, to assess the similarity between these captions at a fine-grained semantic level, employ Owenn.5-77BIInstrueiin Bai et al. (2023). a large language model with advanced language com- prehension. This model evaluates the action similarity based solely on the explicitly stated tions ignoring differences in subjects, context, or wording gasdinngassigning a score on 1 to 5 scale, where 1 denotes completely different actions and indicates identical actions.
The scoring process starts by presenting the model with pairs of captions and instructing it to rate the similarity based exclusively on the described actions. The model is asked to first provide numeric score, followed by concise explanation. Fig. [ illustrates the detailed pipeline of this semantic similarity computation, including some of the example prompts used to guide the models and the standardized scoring templates designed for Qwen.
Other MILM Metries For the other four aspects, only use LLaVA-Video-7B Lin al. 2023)) which has video understanding capabilities. To improve the reasoning behind the MLLM scoring, guide it with multiple related questions before requesting score and provide clear criteria cor- responding to each score. We show the prompt input for Interaction Rationality as an example in Fig. 9
16

---

Published as a conference paper at ICLR 2026
Chain-of-Query
Criteria for Scoring
Please corefully andlyae the ralionaliyy of the interoctione the video by considrring the following
(poor) (excellent) according
below:
It there contect betwees the egent (hand tool)
logizel, purposeful,
only
and objects when expected?
algn
- If no contact occura and the scens remzirs nestly
accurately with applied infersctione accesional ninor position
still assign low score.
Are these contacts lgically timed and purpocelui?
mismotches that signficontly affect perceived in Saveral contocts, or delayed respanass couse modercla daruphan. Score i: Enteractions lack clear soherence, Frequent unnaturd touches, missed contects, or unexpetted object motions incons shent with appited forces Score Objects move withour touch, contacts occur mid oir, there continupus fallure to make contect resutting in highly implousile intersctions. If the scene largaly atill with no connect casign aimtarly low score.
Do oeject movements correspend acourately to the applied forces contectel
interactions, but llogiaal miiplazed
User
there any unnetund unnecesiary touches
User
misedd contects)
sequence
consistent
miraughout?
and legically
the interschions?
Figure 9: Example of MILM Evaluation. Taking Interaction Rationality
example, we
present the specific content used when employing the MILLM to evaluate manipulation videos.
Manipulation Video Generation Evalution Survery
Nottes to read before you start:
-Manipulation Generation Task Definition: Given single input image and reference manipulation video, generates
vidco whose manipulation semantic consistent with the reference.
given video used
evaluation survey, you may see:
CCortiin models generate vidoos with highly distorted sections; this is model artifict, playback error.
Yoo may also see unnatural hands or objects, and we appreciati your patience with such artifacts.
*Simlarrl,, reference objects might unnatural with holes, irregular boundaries, irregular motion due sogmentation artifacts,
this survey, please evaluate the videos comparatively select the Two best options.
Question for you to answer:
Refervace
B
D
Qi-Which Two videos have the most rational and plausible motion?
DA CB =C nD DB oF
Q2: Whihh Two vidcos beat match the action from Reference Video?
Q3: Which Two vidcos the most temporally consistent?
DA aB 2C nD oE oF
Q4:Which Two videos the most physically believable?
DA
Q5:Which Two videos have the most logical object interactions?
DA OB 2C mD mB
oF
Figure 10: Human evaluation for video generation. We provide a reference video and an target image, users are asked evaluate and select their preference. videos based on various video editing criteria.
C.3 DETAILS OF HUMAN EVALUATION
We design a user study by randomly selecting 25 groups from our generated video library. Each group consists of target image, a reference video, and videos generated by eight different meth- ods, including ours. We invite 25 participants to perform human evaluation. They are asked to select their two preferred videos among the eight generated ones based on three criteria: Action Smoothness, Interaction Rationality, Temporal Consistency, Physical Authenticity, and Semantic Consistency with the reference video. As shown in the Fig. 10 we present the content of the user study along with the specific question settings. The Tab. 2 presents the detailed results of the human evaluation. Our method demonstrates a clear advantage in the human preference study, outperform- ing competing approaches across all evaluated criteria.
17

---

Published as a conference paper at ICLR 2026
Table Quantitative comparison on human preference evaluation.
Method
Action Smoothness
Ineraction Rationality
Physical Authenticity
Tempural Comsitencyy
Semuntic Similarity
DynamiCrafier
13.8%
6.9%
8.4%
7.7%
6.6%
VidVide
22.4%
22.15%
18.4%
11.8%
19.2%
MotionClone
0.4%
0.7%
0.5%
2.4%
0.5%
MotionDireetor
0.5%
0.9%
1.2%
0.5%
1.7%
FlexiAct
24.9%
29.6%
37.0%
24.1%
MIMIC
38.0%
39.8%
48.1 1%
40.6%
47.9%
Table Quantitative comparison on MILM evaluation.
Method
Action Smoothness
Ineraction Rationality
Physical Authendicity
Temporal Consistoncy
Semantic Similarity
DynamiCrafter
3.2173
3.0543
3.5217
3.8801
2,4348
CogVideoX
3,6923
3.1318
3.6681
4.1468
2.3736
Motionl Cone
2,8404
3.0957
3,7021
4.0425
2.1277
MoretnDirector
3.2978
3.1489
3.5957
4.0744
2.4149
FlexiAct
3,7761
3.5529
3.6441
4.1323
2.5238
MIMIC
3,9411
4,1381
4,0841
4,0913
2.9127
D ADDITIONAL ABLATION STUDY
D.1 ADAPTIVE REGION Loss
In Stage II, design Adaptive Region Loss to address consistency issues and introduce hy- perparameter as shown in Equation. 4 We conduct additional ablation experiments to validate the effect of this loss. To reduce the influence of other factors, and quantitatively analyze the impact of the region loss using metrics that evaluate visual qual- ity, specifically Appearance Consistency, Subject Consistency, and Background Stability. Due to computational resource limitations, the ablation experiments requiring retraining for the Adaptive Region Loss are conducted on subset of approximately 10,000 videos from the SSv2 Goyal et al.) (2017) dataset, rather than on our full-scale dataset. And we only evaluate on human hand interac- tion data. We first validate the composition and specific effect of the Adaptive Region Loss, which,
ground truth masks inputs
Table 4: Ablation study on the different compositions and weights of Adaptive Region Loss.
X
Appearance Consistency
Subject Consistency
Background Stability
ori. w/o FM
0.8927
0.8894
0.9296
0.9042
0.9118
0.9360
0.25
0.9001
0.8975
0.9328
0.50
0.8872
0.8709
0,9225
0.75
0.9043
0.9039
0.9325
1.00
0.9054
0.9090
0.9323
1.25
0.9033
0.8937
0.9339
1.50
0.8972
0.9118
0.9326
as formulated, consists of two mask components: the overall motion mask and the temporally re- peated first-frame mask. We design three experimental groups for evaluation: the original diffusion loss(ori.), the Adaptive Region Loss without the first-frame mask(w/o FM), and the complete Adap- tive Region Loss(full). Furthermore, we perform hy pyrrparametter ablation for À in the range of to 1.5 with increments of 0.25. The detailed experimental results are presented in the accompanying Tab.
As illustrated in Fig. I1 of the loss function. When the mask for the first frame is omitted, noticeable ghosting artifacts appear. Conversely, the weight increases, the visual details contained within the mask improve significantly.
we present visualization results under different compositions and weights
18

---

Published as a conference paper at ICLR 2026
Target Image
Ground Truth
Mask
Original Loss
w/o First Mask
Reference Video
1 = 0.0
1 0.5
2 1.0
= 1.5
Language prompt
PPullig apple from left to right."
Figure 11: The generation results under different compositions and weights of the loss function.
D.2 DIFFERENT INPUT TYPE
we conduct ablation study comparing two types of conditioning inputs for the motion generation model: (i) interaction masks only, and (ii) the full input image. The results are reported in Tab. 55 We observe that using mask-only input leads to significantly worse performance than using the image. This degradation stems from the fact that masks provide only coarse spatial localization and lack essential appearance cues required for understanding the underlying scene context. Without access to the image content, the model tends to overly rely the temporal changes of the masks themselves and fails to infer how the object should be manipulated in a physically plausible manner.
Table 5: Ablation study of different input type.
Perceptual Similarity
Temporal Quality
MLLM Evaluation
Input Type
Text Alignment
Appearance Consistency
Subject Consistency
Background Stability
Interaction Rationality
Semantic Similarity
t
Mask Image
0.2672
0.8804 0.9084
0.8745
0.9168 0.9385
3.5689
2.5862
0.2721
0.9291
4.1381
2.9127
As illustrated in Fig. 12 models conditioned solely masks often attempt replicate the tion pattern observed in the reference video (e.g., unfolding a piece of cloth) rather than generating contextually appropriate manipulation trajectory for the target image. This indicates that mask- only conditioning encourages the model to focus on mask dynamics rather than action semantics or scene geometry, leading to incorrect or unrealistic motion synthesis. contrast, using the full im- age provides rich appearance information, enabling the model to better understand object attributes, hand-object configurations, and scene layout. This contextual grounding is essential for generating plausible motion trajectory rather than merely propagating mask deformations. These findings jus- tify the necessity of Stage although models like Grounding-SAAn -AM2 Ren et al. can directly provide interaction masks, conditioning the full image allows the generative model to build its own derstanding of the scene, leading to more accurate and contextually appropriate motion generation.
19

---

Published as a conference paper at ICLR 2026
Start
End
Reference Video
SHAR
Stage I
Input: Mask
Stage II
JARAN
x
Stage I
Input: Image
Stage II
unk
Figure 12: Motion representation comparison in a cloth- cloth-folding scenario.
E NETWORK ARCHITECTURE
E. QUERY ENCODER AND PAIR ENCODER
In Stage II, we design two encoders, Ftar and Fref- separately capture the motion information of the target video and the semantic information of the reference pair. Both Eaar and Eref have similar lightweight network architecture, as illustrated in Fig. 13 The architecture mainly consists of convolutional layers interleaved with SiLU activation layers. Through convolutional downsampling operations, the input conditional information is aligned to the size of the latent code 24. Notably, in the last convolutional layer, employ the zero convolution layer from ControlNet [Zhang et al. (202).. This zero convolution layer is initialized with zero weights and biases, ensuring that the initial output of the model remains identical to the output of the pretrained model.
Torget Misi (charneize))
Eximple Pair (chantelee))
comaaaia karnil poddingal, stridet)
Convädtcharrntte-Rit ternel dizew), stride=2)
SLU
SILU
Comfidtsnomels-1,, poddingsl, stridel)
Carviadichameets... ternel stzea3, strideat)
SLU
Comvätiinameb-12
Canvädtcharreets-25. kemel stze3, stride=))
Convldiinomali--3, paddingsi,
(charnds-256,
Figure 13: Network Architecture of Query Encoder and Pair Encoder.
20

---

Published as a conference paper at ICLR 2026
F ADDITIONAL INTERACTION SCENARIOS
To further examine the flexibility of the proposed framework beyond the settings covered in the main experiments, we evaluate behavior under broader range of interaction scenarios involving changes in embodiment, coordination structure, and temporal horizon.
Transfer. We examine whether motion cues extracted from human-hane -aan demon- strations can be transferred to robotic end-cffectors. As illustrated in Fig. [4] the model is able to interpret high-level manipulation intent from human-hand videos and generate plausible inter- action sequences for a parallel-jaw gripper. Despite the substantial differences in appearance and morphology, the generated motions maintain consistent contact patterns and task-relevant dynamics, indicating that the representation accommodate changes in manipulator embodiment.
Torget Imoge
Reference
Video
Target Image
water baftie"
Figure 14: Cross-domain generation from human-hand demonstrations to a robotic gripper. The model preserves task-relevant contact patterns and motion intent across embodiments.
Two-Hand Coordination. We analyze the performance of method in coordinated two-hand interactions. As illustrated in Fig. [5] the model generates coherent predictions for scenarios involv- object. The outputs preserve stable spatial relations
ing two human hands jointly manipulating between the hands and exhibit coordinated motion trajectories, suggesting that the MIMIC remains effective even when the interaction involves multiple effectors operating shared workspace.
Long-Horizon Generation. We further examine whether longer videos can be produced by se- quentially composing multiple generations. As shown in Fig. [6] we generate long-horizon teraction sequence by using the final frame of generated segment the initial frame for the next generation. This process repeated across three segments guided by different reference videos. The resulting long video maintains smooth transitions between segments, preserves consistent object and manipulator configurations, and exhibits coherent temporal evolution throughout the sequence. These observations indicate that the proposed framework can be extended produce temporally longer interaction videos through iterative composition.
G MORE RESULTS
We present additional comparisons with other methods, including interaction videos between human hands mechanical grippers and objects, shown in Fig. 18 21 We showcase additional gener- ated videos of human hands and mechanical grippers interacting with objects Fig, 22 and 23 MIMIC not only produces videos that semantically match the reference video but also distinguishes
21

---

Published as a conference paper at ICLR 2026
Video
Target Imoge
into piacel
Referance Video
Torgef Image
"ruating pipper dele
Figure 15: Generation results for coordinated two-hand manipulation, showing stable spatial rela- tions and synchronized motion between the two hands.
Reference
Targef
Image
Video
Target Image
Reference
Torget Image
Long-horizon video obtained by iteratively composing segments, using final frame of each segment as initial frame of the next segment to ensure smooth transitions.
Figure 16:
between similar operations, such as pulling and pushing. Moreover, our method capable han- dling interactions with non-rigid objects, such as unfolding.
H FAILURE CASES
While our method achieves strong results across various interaction scenarios, still struggles in some challenging cases, as illustrated in Fig. 17 The upper portion of the figure shows
22

---

Published as a conference paper at ICLR 2026
with significant occlusion, where large parts of the hand and object not visible. This leads to noticeable appearance instability and incorrect hand-object scale in the first predicted frame, which then propagates through subsequent frames and degrades the overall generation quality. The lower portion of the figure depicts an interaction involving fine-grained dexterous manipulation, specifically twisting open bottle cap. This task requires precise rotational control, stable contact maintenance, and detailed reasoning about fingertip-object interactions. Our model is not yet able fully capture these subtle motion dynamics, resulting in imperfect motion synthesis.
Reference
Video
Targef Image
Vemalan P2 boby
Marabee
powder
Reference Video
Target Image
"hvistin
Figure 17: Failure cases. Top: severe occlusion leads to incorrect hand-object scale and unstable generation. Bottom: fine-grained dexterous manipulation such as twisting bottle cap remain chal- lenging, resulting in inaccurate motion synthesis.
I BROADER IMPACT
This project addresses the critical bottleneck of scarce and costly real interaction data in the field of embodied intelligence by generating a large volume of plausible hand or gripper manipulation videos. This effectively facilitates research and applications in robotic perception and manipulation, accelerating technology dissemination and cost reduction. Meanwhile, the diversity of generated data enhances the generalization capabilities of robotic systems and fosters interdisciplinary inno- vation. By providing rich, low-cost data resources, the approach opens new technical pathways and practical possibilities for the development of embodied intelligence, yielding significant social and industrial impacts.
23

---

Published as a conference paper at ICLR 2026
Target Image
Dynami- Crafter
Motion Director
Reference Videa
FlexiAct
Ours
Language
prompt
Mask
"Moving power tank down."
Figure 18: More comparison results of human hand-object interaction video from the reference video.
Target Image
Dynami- Crafter
Motion Director
Reference Video
FlexiAct
Ours
Language prompt
Mask
"Pulling bottle from right left.
Figure 19: More comparison results human hand-object interaction video from the reference video.
24

---

Published as a conference paper at ICLR 2026
Target Image
Crafter
Dynami-
Motion Director
Reference
Video
FlexiAct
Ours
Language prompt black gripper
Mask
folds yellow cloth."
Figure 20: More comparison results of robotic gripper-object interaction video from the reference video.
Target
Dynami- Crafter
Image
Motion Director
Video
FlexiAct
Ours
Language prompt white gripper
Mask
picks orange can."
Figure 21: More comparison results of robotic gripper-objes interaction video from the reference video.
25

---

Published as a conference paper at ICLR 2026
Reference
up pret dep
Ganeration
Generation
Reference
Yalling kitts free right bet"
Tuullly dia lee fun right is sif
Generation
right
palla rihtt
Generation
Tauling al zitts right
"Veving the ansurr
Generation
Figure 22: Results of human hand-object interaction video from reference videos.
Reference
Generation
Generation
Reference
A Med groper urbsh biss cut"
bact ofoe viai due del""
Generation
gioprr
esag
Generation
griper cirt"
yper aipas lias dut!
Generation
Figure 23: Results of robotic gripper-object interaction video from reference videos.
26

---

Published as a conference paper at ICLR 2026
Reference
Generation
"iiing tee esser fra right ts let
"Noiing the ersee of*
Generation
Reference
Generation
Reference
Toiding beard Juner lil"
Moring bean dater to right"
Generation
Figure 24: Results of human hand video on same target image with different reference videos.
27


Reviews:
[{"review_id": "QtfxrUsSWb", "reviewer": "Reviewer_uEB3", "summary": "The paper proposes MIMIC, a two-stage image-to-video diffusion framework MIMIC designed for manipulation video generation. The propsoed training framework consists of two-stages: 1) Condition on a reference video, reference video interation mask, and target image, generte the target video interation mask. 2) condition on reference video, reference video interation mask, target image and masked target image, generate the target video. The paper use DynamiCrafter as the base model. The model was trained and evaluated on human-hand and robotic-gripper datasets built from SSv2, BridgeV2, and Fractal datasets. It was compared against major baselines such as DynamiCrafter, CogVideoX, MotionClone, FlexiAct, MotionDirector, etc.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow, presenting its ideas and methodology clearly.\n\n- The paper combines quantitative metrics, MLLM-based semantic evaluation, and human preference studies, providing a comprehensive assessment of model performance.\n\n- The paper conducts extensive ablation studies, including analyses of the Two-Stage vs. One-Stage framework and the effectiveness of each key component, further validating the proposed approach.", "weaknesses": "- **Design choice of motion representation**: I was curious about the motivation for adopting masks as the abstraction for motion information rather than alternative representations such as tracking points or optical flow [1]. While masks provide clear spatial localization, they might be too rigid in preserving the object shape from the reference video, potentially limiting flexibility in motion adaptation.\n\nReference: [1] VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models (ICML).\n\n- **Impact of base model**: Would employing a more powerful base model (e.g., a larger or more recent video diffusion backbone) further improve the overall generation quality and temporal coherence? Since the paper mentions limitations due to the capacity of the current base model, it would be interesting to discuss how the proposed approach might scale with stronger foundational architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FxXkwEUGqQ", "reviewer": "Reviewer_xAXU", "summary": "This paper proposes MIMIC, a novel two-stage image-to-video diffusion framework designed to generate realistic and controllable manipulation videos. The method first utilizes an Interaction-Motion-Aware (IMA) module to generate a sequence of semantic interaction masks guided by a reference video. Subsequently, it introduces a Pair Prompt Control mechanism that conditions the final video generation on both the predicted masks and the original reference video, effectively disentangling object motion from camera motion and enhancing controllability.", "questions": "1. The experiments only show in-domain cases. Have the authors tested cross-domain generalization? For example, can a video of a human hand be used to guide a robot arm?\n\n2. Figure 6 shows that lacking the IMA module leads to failure. With the IMA module, how well does it work if the reference motion is complex or the object is occluded?\n\n3. The paper mentions that the input of Stage I includes text, but it appears to be unused in the architecture (Figure 2) and the IMA module description. Could the authors clarify how exactly the text is used in Stage I?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors' decomposition of the task into two stages, \"mask generation\" and \"video rendering,\" is a reasonable design. Furthermore, the authors demonstrate the necessity of this decomposition through ablation studies (such as \"Two-Stage vs. One-Stage\" and Figure 6) and clearly show the contributions of each key module (IMA and PPC).\n\n2. The authors identified the problem with using only masks for control. The Pair Prompt Control (PPC) part is a clever solution. Figure 6 (background drift vs. static background) shows how this part is effective at separating object motion from camera motion.", "weaknesses": "1. The IMA module just copies the motion from the reference video and does not understand the physics of the target scene. The predicted masks in Figure 6 support this, appearing to be a direct motion transfer that ignores the specific geometry and constraints of the target object.\n\n2. The examples in the paper are all very similar, showing only robot-to-robot or hand-to-hand tasks. This makes it unclear if the method can work for more general cases where the reference and target are different. For instance, it is not shown if a human hand video can be used to control a robot arm.\n\n3. The method's first stage requires reference manipulation masks as input, but the paper does not explain how these are obtained. Class-agnostic models like SAM are difficult to automatically and accurately prompt to segment only the manipulated object in complex videos. Meanwhile, class-specific semantic segmentation models (e.g., Mask R-CNN) would restrict the method to only predefined object categories, causing it to fail on any new objects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IyLbOj3kDJ", "reviewer": "Reviewer_Kzh4", "summary": "This paper proposes MIMIC, a two-stage diffusion framework to generate manipulation videos by leveraging a reference video for semantic and motion cues. The method first generates a sequence of interaction masks (Stage I) and then renders the final video (Stage II), aiming to provide a scalable data source for embodied AI. The authors demonstrate strong quantitative and qualitative performance over existing methods.", "questions": "My overall feeling is that the paper presents a strong method, but the presentation is unclear in several critical areas, as detailed in the comments. I hope the authors can clarify these points in their response. Happy to increase my score once these are clarified.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses the important and challenging problem of data scarcity for training embodied intelligence systems.\n- The proposed two-stage framework is logical. Decoupling the generation process into (1) motion/interaction understanding (mask generation) and (2) high-fidelity rendering (video generation) is a sensible approach to decompose a complex problem.\n- The qualitative comparisons provided in the Supp. video are compelling.\n- The ablation studies are informative and effectively validate the contributions of the proposed components, i.e., the IMA Attention and the Pair Prompt Control mechanisms .", "weaknesses": "- Relation to Video Editing: The related work section focuses on \"Video Motion Customization\" and \"Interaction Video Generation\" but does not explicitly discuss the field of generative video editing. The task in this paper shares conceptual overlap with reference-based video editing. It would strengthen the paper to include this discussion, clarify the key distinctions, and situate the work relative to recent highly relevant papers in the HOI image/video editing domain, such as [1][2].\n- The paper does not clearly explain how the (reference, target) video pairs are constructed for training. Is a video simply used as its own reference? Are pairs formed in some other way (based on action label)? This is a critical methodological detail that is currently missing and needs to be explicitly stated.\n- The paper's description of Stage I is confusing. It's trained to \"identify\" the manipulated object , but Grounding-SAM2 is already used for annotation. Why train a generative model for a recognition task? Why Grounding-SAM2 isn't used at test time to provide the initial object mask, allowing the diffusion model to focus on its core task: generating the new motion pattern (i.e., the mask trajectory)?\n- It is unclear what the scope of the tested actions and objects is. For instance, do the experiments demonstrate generalization to unseen object categories, or new instances of seen categories? The experimental setup and data splits should be described more explicitly to clarify what level of generalization is being evaluated.\n- About the use of language: Appendix A.2 says that the method relies on structured language templates (e.g., \"move [something] down\") and an NLP pipeline to \"extract structured action-object pairs\". I feel this should be explicitly stated in the main paper, as it clarifies that the model is not conditioned on free-form, natural language prompts, right?\n- About the masks: the text repeatedly describes them as \"binary masks\". However, numerous figures visualize multi-class masks, with hands/grippers and manipulated objects colored differently. The authors need to clarify whether the masks are binary (e.g., interaction vs. background) or multi-class (e.g., hand vs. object vs. background). Also Ln204 “soft binary mask”, what does “soft” mean?\n\n[1] Affordance Diffusion: Synthesizing Hand-Object Interactions\n\n[2] HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1UKSCWhDwG", "reviewer": "Reviewer_Sdzw", "summary": "This paper proposes a two-stage, image-to-video diffusion framework for generating manipulation videos with fine control over object interactions. First, an Interaction-Motion-Aware (IMA) module extracts semantic masks from a reference video to guide future frames. Then, these masks are injected into a diffusion model to steer the video generation process. A Pair Prompt Control mechanism helps disentangle object motion from camera motion by conditioning on both the target image and the reference video. \nThe method is shown to preserve manipulation intent and detail, even when dealing with deformable or diverse objects, outperforming existing baselines in both visual fidelity and control.", "questions": "1. How did the authors choose the reference videos? \n2. Did the authors try to generate longer videos by concatenating generations? (e.g. use the last frame of a previously generated video as the initial frame of a new generation)\n3. Did the authors try multi-object or multi-hand interaction scenarios? e.g. manipulate different objects with 2 hands or a hand and a gripper in the same video?\n4. Are there failure cases you exhibited, such as specific objects/motions/lighting conditions etc.?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper addresses an interesting and important problem with a novel solution.\n2. The paper proposes several new ideas that could be leveraged for future research in different direction, including the IMA attention, Pair Prompt Control module and the idea of using ICL for learning motion from motion masks of similar tasks.\n3. Experiments are sound and the visual results show the superiority of MIMIC over previous work.", "weaknesses": "1. The method depends on relevant reference videos. This might limit its applicability in scenarios where such references are not available or are very different from the target.\n2. It is unclear how the reference videos are chosen.\n3. The method is limited to generating 16-frame videos.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)