SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios
Authors: Lingwei Dang
Ruizhi Shao
Hongwen Zhang
Wei MIN
Yebin Liu
Qingyao Wu
Venue: NeurIPS 2025 spotlight ()
Keywords: hand-object interaction
synchronized diffusion
visual prior
motion dynamic
Primary area: applications
TL;DR: A novel framework that combines visual priors and dynamic constraints within a synchronized diffusion process for HOI video and motion generation.

Abstract: Hand-Object Interaction (HOI) generation has significant application potential. However, current 3D HOI motion generation approaches heavily rely on predefined 3D object models and lab-captured motion data, limiting generalization capabilities. Meanwhile, HOI video generation methods prioritize pixel-level visual fidelity, often sacrificing physical plausibility. Recognizing that visual appearance and motion patterns share fundamental physical laws in the real world, we propose a novel framework that combines visual priors and dynamic constraints within a synchronized diffusion process to generate the HOI video and motion simultaneously. To integrate the heterogeneous semantics, appearance, and motion features, our method implements tri-modal adaptive modulation for feature aligning, coupled with 3D full-attention for modeling inter- and intra-modal dependencies. Furthermore, we introduce a vision-aware 3D interaction diffusion model that generates explicit 3D interaction sequences directly from the synchronized diffusion outputs, then feeds them back to establish a closed-loop feedback cycle. This architecture eliminates dependencies on predefined object models or explicit pose guidance while significantly enhancing video-motion consistency. Experimental results demonstrate our method's superiority over state-of-the-art approaches in generating high-fidelity, dynamically plausible HOI sequences, with notable generalization capabilities in unseen real-world scenarios. Project page at [https://droliven.github.io/SViMo_project](https://droliven.github.io/SViMo_project).

SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios
Lingwei Dang" Ruizhi Shao"?, Hongwen Zhang
Wei MIN 4, Yebin Liu
2, Qingyao Wut!
School of Software Engineering, South China University of Technology
DDpartment of Automation, Tsinghua University
School of Artificial Intelligence, Beijing Normal University "Shadow AI
Abstract
Hand-Object Interaction (HOI) generation has significant application potential. However, current 3D HOI motion generation approaches heavily rely prede- fined 3D object models and lab-captured motion data, limiting generalization ca- pabilities. Meanwhile, HOI video generation methods prioritize pixel-level visual fidelity, often sacrificing physical plausibility. Recognizing that visual appear- ance and motion patterns share fundamental physical laws in the real world, we propose a novel framework that combines visual priors and dynamic constraints within a synchronized diffusion process to generate the HOI video and motion simultaneously. To integrate the heterogeneous semantics, appearance, and mo- tion features, method implements tri-modal adaptive modulation for feature aligning, coupled with 3D full-attention for modeling inter- and intra-modal de- pendencies. Furthermore, we introduce vision-aware 3D interaction diffusion model that generates explicit 3D interaction sequences directly from the synchro- nized diffusion outputs, then feeds them back to establish closed-loop feedback cycle. This architecture eliminates dependencies on predefined object models or explicit pose guidance while significantly enhancing video-motion consistency. Experimental results demonstrate our method's 's superiority over state-of-the ap- proaches in generating high-fidelity, dynamically plausible HOI sequences, with notable generalization capabilities in unseen real-world scenarios. Project page at http://rrliieeegggibbbo..ooo///o [
1 Introduction
Human-object or hand-object interaction (HOI) generation serves critical applications across gam- ing, animation, digital human creation, and robotic action retargeting [ 37, 53, 58]. Some stud- ies [7A, [3 [2 construct high-precision 3D interaction datasets through laboratory-based multi- view camera arrays and motion capture systems, then train diffusion-based motion generators. These object-centric approaches typically predict parametric human/hand motions and the corre- sponding object pose sequences given object meshes and initial configurations. However, existing datasets [39 A [8] collected in laboratory environments lack diversity in object types and inter- action patterns, constraining model generalization and resulting in ambiguous object boundaries, implausible or inconsistent actions (Fig. MI left). Moreover, the reliance on precise 3D object mod- els fundamentally limits their zero-shot generation capabilities.
"Equal contributions. Email: levondang @ 163.com, jial saurus @ gmmail.com. Corresponding Author, Email: qyw www.scu..dd..c.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

use kettle into
use spatula to plate
Kef. lin
Promptt
Img
Posr Condition
Ref. ling
Pronpts (Optional)
Motion Diffusion Model
Video Model
Synchrumized Diffusion
Generated HOT Motion Sequenee
Gieneaated HOI Video tranes
Generited HUI video frumes
Unclerr Boundary
Implussihility
Incresistinay
Distuttion
Helloiistion
Generated Minion Sequeree
Modified MDM
Animute Anyune
Ours
Figure I: Different HOI generation methods. Approaches like MDM [57] rely on limited mocap data without visual guidance, resulting in blurred boundaries and compromised plausibility and consis- tency, Methods like Animate Anyone [2T]] leverage large-scale visual priors but exhibit distortions and inconsistencies because inadequate physical awareness. Our method marries visual priors with 3D motion constraints and eliminates dependency pre-defined object models or pose guid- ance.
Recent advances in large video foundation models based on Diffusion Transformers (DiT) [BE]] (e.g., Sora [S], CogVideo [20, 69), Hunyuan Video [26]), have shown impressive capabilities in modeling physical dynamics through large-scale video training. These models can generate inter- action videos with high visual fidelity end-to-end from text reference images. However, their pixel-level generation approaches often struggle to produce accurate and coherent hand-object inter- actions due to limited explicit modeling of motion dynamics and physical constraints. To address this, some methods extend image-based diffusion models (e.g., SVD [[]] by adding pose-guided pipelines |66, 211, IZ5, 655]. These approaches combine pose conditions and appearance features to improve human-objee object interaction generation. While effective, they require pose sequences or ex- ternally estimated motion trajectories inputs, preventing full end-to-end text/image-conditnn generation. Additionally, their single-frame generation leads to poor temporal coherence, causing flickering and identity inconsistencies (Fig. II, middle).
These challenges reveal longstanding methodological contradiction: motion generation systems excel at physical constraint modeling but suffer from limited data scales. In contrast, video genera- tion models leverage massive visual priors but lack motion plausibility. We argue that this division stems from neglecting the co-evolution mechanism between visual appearance and motion patterns: they not only share the same foundation of physical dynamics but also could leverage similar diffu- sion processes. Based this insight, propose SViMo, Synchronized Video-Motie diffusion framework that enables synchronous HOI video generation and motion synthesis within a unified architecture (Fig. M. right). The core innovation lies in extending pretrained image-to-vided foun- dation model into the multimodal joint generation framework through the scalable DiT architecture. To better integrate the heterogeneous features of text semantics, visual appearance, and motion dy- namics, we introduce the triple modality adaptive modulation to align feature scales and employ a 3D full-attention mechanism to learn their synergistic and complementary dependencies. Addition- ally, it is difficult for video foundation models to learn explicit 3D interaction motions directly. To bridge the representation gap and reduce optimization complexity, project 3D motions onto 2D) image planes, constructing "rendered motion videos" as SViMo's motion representation.
To further enhance the video-motion consistency, design Vision-aware 3D Interaction Dif- fusion model (VID). This model generates explicit 3D hand poses and object point clouds using denoised latent codes from the synchronized diffusion, which are then reinjected into the SViMo as interaction guidance and gradient constraints. Unlike methods requiring pre-specified action se- ries, approach integrates video synthesis and 3D interaction generation within end-to-end denoising pipeline. This creates a closed-loop feedback mechanism where motion guidances refine video generation while video latents update motion results, enabling synergistic co-evolution of both modalities.
In summary, our contributions are threefold:

---

A novel synchronized diffusion model for joint HOI video and motion denoising, effec- tively integrating large-scale visual priors with motion dynamic constraints.
A vision-aware 3D interaction diffusion that generates explicit 3D interaction sequences, forming a closed-loop optimization pipeline and enhancing video-motion consistency.
Our method generates HOI video and motion synchronously without requiring pre-defined poses or object models. Experiment results demonstrate superior visual quality, motion plausibility, and generalization capability to unseen real-world data.
2
Related Work
3D interaction synthesis relies on high-precision motion capture datasets, some of which focus on human action conditioned on static objects [TR] [SS, while others simultancously capture interactions of both human and dynamic objects [8, 33% 67, 771. B8, 54, [14]. Building upon these datasets, existing 3D interaction generation methods employ diffusion models to either introduce intermediate contact maps or milestones for modeling kinematic features [[Z] [3 2S,, 26, gS. 72, 62. 28, 27 (30, B6, 311, B7]
or leverage physical simulations to ensure physical dynamics plausibility [6-], 60. EL X Iill. How- ever, due to the limited availability of 3D interaction data, the generalization capability of these approaches remains constrained. In contrast, our method leverages large-scale visual priors, oper- ates conveniently without requiring 3D object models, and demonstrates promising generalization potential.
Interaction Video Generation. The success of image generation models [T 50-557 has inspired significant advances in video generation. Several approaches [C. 66, A3, 75, 65. 6l]] extend 2D image denoising U-Nets to video by incorporating temporal attention layers, while improving controllability via pose guidance and reference networks. Alternatively, native large video mod- els [5], 20. 69, (26, [25] directly generate videos using spatiotemporal DiT architectures [[ A sepa- rate line of work aims to generate both video and interactive motion dynamics [SL III. For instance, VideoJAM [9]] produces video frames alongside 2D optical flow to improve visual fidelity and mo- contrast, our method generates videos with explicit 3D motion representation. This enables direct modeling of object geometry, scale, and spatial relationships, leading to superior physical plausibiliyparticclaary
tion coherence.
complex scenarios involving occlusions or intricate interactions. Notably, Vlogger [[ employs two-stage pipeline: it first generates 3D human poses using one net- work, then synthesizes video via separate model conditioned on the predicted motions. In contrast, unify video and motion generation within single, synchronized diffusion framework. Since human-object interaction videos and their corresponding motion dynamics inherently obey the same physical laws, joint modeling allows the system to learn intrinsic co-evolutionary patterns, thereby enhancing the quality and consistency both modalities.
Multimodal Generative Models. Driven by advancements in vision-language models [10) 34
59
68), researchers have explored versatile generative models that align with the multimodal essence of the physical world. Some works M [i] cascade single-modality generators into asynchronous pipelines, yet suffer from complex workflows and noise accumulation. Others develop end-to-end multimodal joint generation frameworks that rely massive aligned multimodal data (II, 32, I77 766),
indirect bridging mechanisms [[5]], or intricate hierarchical attention strategy [65]] to ensure cross- modal synchronization. Differently, our method extends native large video model into synchro- nized video-motin generation systems, aligning heterogeneous modality features through multi- modal adaptive modulation and closed-loop feedback strategies.
3 Methodology
We define the HOI video and motion generation task as follows: Given a reference image frame I I2 #x x3 and a textual prompt P, generate the future video V E R Xxx and 3D motion sequence M = ((h,, [ with N time steps. Here h, E Jx3 and o, E R are the hand joints trajectories and object point clouds, each has J and K nodes respectively. The following sections detail each component of our approach. See the Appendix for training/inference pseudo- code and additional details.
3

---

Sumple (((.T))
Sinascohal
Tiunttep
Lyves ||4 -Rboll 2 ||=05" #Mell
Evor
Sipatall, scrpe plate
T5 Enceder
Triple Modality Alaapine Modulation
Promps
Yi
den
Yin
P
Gule
S&S
Paint Freelorrnn
Cane
YAE
Sooo
Noddon Video
Reflug
Cato
SBS
Cale
Roskeed
YAL
Calz
S&S
Calte
Interaction Guidunce
2
Gradient
Synchronized Video-Mooe Mnteo-Motion Diffusion (SViMo)
Constrain
Vision-aware 3D Interaction Diffusion (VID)
Evm ||A Ro.p [
D Detuntee )n
Grad, Beckpropegnte
Fucion
Projection
Mullicclk 3D Cor
Generated Interaction (g,,
0011000000000
S&S
Scale Shift
Maikan
Troining Loss Fanction
Denshing NoNet
Our method comprises: (1) A synchronous diffusion model that jointly generates HOI videos and motions (Sec. B..)). (2) A vision-aware interaction diffusion model that generates 3D hand pose trajectories and point clouds from the former's outputs (Sec. (.4 then feeds them back into the synchronized denoising process to establish closed-loop optimization (Sec.
Figure 2:
3.1 Preliminary: Basic Large Video Generation Model
The core of large video generation models diffusion and its reverse denoising process [19]] The diffusion process adds Gaussian noise to video latent codes step-by-step using q(=x|z(-1) N(zi Va.zt-1, 1, 3) where B. E [0, 1] increases monotonically with time step t, and at 3r. Repeated application of this formula yields the marginal distribution:
g<>,|=0) = N(z Va,zo, (1 - o.) ā = II (1- - Br).
The above distribution converges to an isotropic Normal distribution when grows large.
Correspondingly. the reverse denoising generator G with trainable weights 0 predicts the clean latent code f0.0, and is optimized by:
Lxp prediction -x [||zo - Go(z,, e, t)1
(2)
During the inference phase, the trained model is utilized to iteratively denoise from pure Gaussian noise step by step, following the procedure:
20.0 = Go(2,,c,) C,,),,
2c-1 Vare
2z Vat-1( and
1 1- az)
20,01
Cz
1- Cr
Finally, we obtain the clean denoised latent code eg. which then decoded to the raw video space.
3.2 Framework Overview
Based on the insight that visual appearance and motion dynamics share inherent physical laws of the real world, we propose an end-to-end framework that unifies HOI video and 3D motion generation by integrating visual priors with dynamic constraints. Our framework consists of two key components. The first is SViMo, which focuses on video generation (Fig. A top: See. B.3). It exeeendds a pre-trained imagc-to-video video foundation model into joint video-motion generation architecture. During joint

---

denoising, SViMo dynamically aligns visual appearance with high-level motion signals, improving both visual quality and motion plausibility. Notably, the motion representations SViMo refer not to explicit 3D hand trajectories or object point cloud sequences. Because there is significant gap between 3DD motion data and 2D video representations, directly modeling explicit 3D motions would disrupt the pre-trained foundation model's visual priors and degrade performance. Instead, we project explicit 3D interactions onto the 2D image plane to create "rendered motion videos" as intermediate representations for SViMo. The second component is VID, which generates explicit 3D motions (Fig. Q bottom; Sec. B.4D) maps the output SViMo, both the generated video and the 2D rendered motion video, into target 3D interactions, i.e., hand-object trajectories and object point clouds. Additionally, the collaboration between SViM and VID forms a closed-loop feedback pipeline, ensuring consistency between generated videos and 3D motions (Sec. (3.5).
3.3 Synchronized Video-Motion Diffusion
The SViMo learns to predict the video and motion, given the time step, text prompt, and reference image: (2) . Gp ((z) ). (P. I). The entire video-motion generation process com-
prises feature embedding, multimodal feature modulation and fusion, and joint denoising.
Feature Embedding. Time step is sampled from a uniform distribution and then added sinusoidal position encoding, followed by simple two-layer linear mapping to obtain the time embedding e, e Rda.., For the text prompt, we use frozen pre-trained language model, Google T5 [[8]. extract text embedding Sxx E IR 4xdd, and then calculate the semantics features fexi with linear projection, where L is the max length of textual tokens. Thirdly, for the original reference image I, to ensure a certain level of robustness, we first add random noise yielding I noised. Then, we compute the compressed latent code 2 through 3D video VAF, N = E(T noised) E RR A x W. xdvxx Additionally, for the target video V, we also map it to the latent space with the same VAE. Then the noised latent code z E RR +1)==+ A x W
Fie xdvar are obtained based on the forward diffusion process in Eq. M. To better align it with the image condition signal during the video denoising process, we repeat the image feature 21 along the temporal axis, and concatenate them along the channel-axx to get video feature embedding ey R +1xx
x(Xxdvx))
xY + 21. Then the visual feature H R N- +=)++1xx axna x W xd The
is patchified through a 2D stride convolution, esulting in fv E rendered motion video encoded through the same VAE and diffused to obtain 2 In contrast, to get motion embedding ews the diffused latent code here are concatenated with interaction guidance M provided by VID (Sec. B.4, rather than being combined with z₁. After that, the motion feature fM obtained in the same way as that of fv.
Multimodal Feature Modulation and Fusion. In our SViMo framework, the DiT token sequence comprises three distinct modalities: text tokens f fext video tokens fy and motion tokens f M which differ significantly both feature spaces and numerical scales. To bridge these disparities while preserving modality-ppecific characteristics, we adopt a triple modality adaptive modulation method that learns modulation parameters from the timestep signal é.. These parameters determine the scaling, shifting, and gating operations of each modality's features separately. Additionally, 3D full-attention mechanism is employed to capture intra- and inter-modal relationships. Take the processing of text features as an example, the DiT Block B proceeds as follows:
(aieet Beoee / 1= MLP(e:), f'xI - LN(S (ox) (1 alea)) + +Beext
B(-)-
f Fees f + Tex Otex (3DFA (U(flext F'v f' f'm)))
same for f. and fMS (4)
f = LN( (em)
@ (1 + azeu) + B
f" M' + Peex @ Orext (FFD (U( I'v. fhr)))
where "3DFA" and "FFD" unified multi-head 3D full-attention and feedforward layers, U and n denote token concatenation and segmentation along the sequence dimension, respectively.
Video-Motion Joint Denoising. The video and motion features output by the final DIT block then go through an MLP to reconstruct the VAE latent codes, yielding = for video and 2 for motion. Finally, the SViMo is optimized according to Eq.
LsVIMo = E (M,,,
[[|>]] - Go(z) (P,I), 1 Go(zM (P. (5)
5

---

3.4 Vision-aware 3D Interaction Diffusion
The VID Ma generates the explicit 3D hand pose trajectories hg and object point cloud sequences On given latent codes of videos z and motions (rendered motion video) z any time. The framework operates follows: First, a dual-stream 3D convolutional module encodes multi-scale spatiotemporal features from both video and motion codes. Then they are fused and subsequently injected into the 3D interaction denoising modules through cross-attention mechanisms to synthesize 3D HOI trajectory sequences. Following the Xo xo-prediction formula Eq. D. the model optimized through the following loss function:
{ (hoos Ø) = =Ma [[hr. or), (zY, 2x
(6)
CVID = (x) (at a,,e
MSE Nu, he.p)) + D Detumfer (Ou, 0u.s)]
3.5 Close-loop Feedback and Training Objectives
Close-loop Feedback. To enhance the mutual promotion and co-evolution between SViMo and VID, and improve the consistency between video and 3D motions. we design closed-loop feed- back mechanism. This mechanism includes two pathways: interaction guidance and gradient constraint. Specifically, for the straightforward interaction guidance strategy (Eq. M. row 1st), we first generate the 3D interaction from the video and motion inputs of SViMo: (ho, op) = Man-grad ((hy, or), , (). then projected it onto the 2D image plane obtain rendered mo- tion video M, which are subsequently embedded into the VAE latent space yielding 2)" Finally, concatenated with the noised motion latent code z,M mentioned in Sec. B forming an additional interaction guidance for the SViMo. On the other hand, the input of VID could come from the output of the SVMo. Therefore, the gradient of VID will be backpropagated into the SViMo during the training process, forming a gradient constraint path and thereby promoting its optimization (Eq. M row 2nd).
(x) MM) (z) 2t =04
VIDaogul
(ho, oo)
Proj.
M
VAB
xM 20
Inter. Guid, to SVIMo
(z), 2
+ 20 )
SMMM
VID
:
20)
(ho, 00)
LVID-
Gradient
Gradient
Gradient
Training Objectives. The training process of our method involves two phases: initially warming up the VID based on Eq. G. followed by closed-loop training where the SViMo and the VID are jointly optimized according to Eq. B and Eq. S L = wi LiVMM + w22LVD.
4
Experiments
We conducted extensive experiments to validate the effectiveness of our proposed method. More information, such as additional results and limitation discussions, is provided in the Appendix.
4.1 Experimental Setup
TACO dataset [GX] is large-scale bimanual hand-object interaction dataset capturing diverse tool- use behaviors via multi-view video recordings and high-fidelity 3D motion annotations. Each task is defined as a triplet <tool category, action type, target object category>, describing tool-mediated interactions with objects. The dataset includes 2.5k interaction sequences, covering 20 object cat- egories, 196 3D models, 14 participants, and 15 daily interaction types. It provides allocentric (4096x3000) and egocentric (1920x 1080) video streams, totaling 5.2M frames at 30 Hz. To re- duce computational load, we crop hand-object interaction regions, adjust their aspect ratio to 3:2, and downsample to 49 frames at FPS. This results in videos with spatiotemporal resolution 416x624 x9, aligning with Cog VideoX's 's [69] default settings while lowering spatial resolution. To mitigate overlap between test and training sets that might compromise evaluation generalization, we implemented two-stage data partitioning strategy. First, we reserved all instances involving specific actions (e.g., hit), tools (e.g., glue gun), and objects (e.g., toy) as the initial test set, ensur- ing these elements are absent from the training set. Second, from the remaining instances grouped

---

Table 1: Comparison of video generation results. The best and second-best results are highlighted with bold and underline formatting.
Content Quality
Dynamic Quality
Method
Type
Training
Overall f
Subj.
Bkg. t
TSmoo.
Dyn. f
Hunyuan-1 13 [26]]
0.9632
0.9627
0.9889
0,4900
0.4493
3D V
w/o
Wan-14B [
0,9576
0.9620
0.9829
0.8476
0.7675
Animate Anyone [2
0,9206
0,9302
0.9671
0,9867
0,8172
2.5D V
Easy Animate [6]]
0,9243
0,9358
0.9657
0.9933
0.8297
Cog VideoX-BB
[5
3D V
0.9404
0.9477
0,9858
0,9933
0.8727
Ours
3D V&M
0,9500
0,9533
0.9898
0.9801
0.8785
by <action, tool, object> triplets, we applied weighted sampling according to group sizes to obtain additional test data. Finally, the ratio of the training set to the test set is 9:1.
Evaluation Metric. For video evaluation, we use VBench [22]] to assess two key dimensions: Con- tent Quality (including Subject Consistency and Background Consistency) and Dynamic Quality (Temporal Smoothness and Dynamic Degree). More details about these metrics can be found in the Appendix. To address the partiality of individual metries, we multiply them to derive Overall score for holistic evaluation. For 3D interaction evaluation, we separately assessed hand poses and object point cloud sequences. For the former, we calculated MPJPE (Mean Per Joint Position Er- ror) and Motion Smoothness metrics. For object evaluation, we measured the Chamfer Distance between generated and ground-truth point clouds. Additionally, compute a comprehensive FID score via pretrained interaction autoencoder.
4.2 Implementation Details
Network Architecture. Our proposed model generates hand-object interaction videos with resolu- tion [H. W, .N] = [416, 624, 49] and 3D motion sequences containing J/ = 42 hand keypeints and K 298 object nodes. During training. timesteps uniformly sampled from [0, 1000], and the embedding dimension is duime = 512, while only 50 steps are sampled during inference for accelera- tion. Text prompts (max length I. 226) embedded into features of dimension d, 4096. The video VAE uses spatial-temporal compression ratios [rw: rh, rn] = [8. 8, 4], producing latent codes of size [52 78 x 13) with dvae 16 channels. After patchification via two-step stride convolution, video and motion tokens are compressed to 13182 tokens each. This results in a total multimodal token sequence length of 226 + 13182 2 26590 (text video motion). The DiT backbone comprises 42 DiT Blocks, each with 48 attention heads, totaling 6.02B parameters.
Training Details. All models are trained on 4 NVIDIA A800-80G GPUs With memory opti- mization techniques including DeepSpeed ZeRO-3 [19)), gradient checkpointing, and BF16 mixed- precision trick, achieve per-GPU batch size of 4. We first warm up the VID for 5k steps, then conduct joint training with the SViMo. To enhance computational efficiency, we initially train at reduced resolution [H', W [240, 368) for 30k steps before fine-tuning at full resolution for 5k steps. The weights of SViMo and VID terms in the training objectives are w1 = and W2 = 0.05.
4.3 Comparison with Previous Approaches
Baselines. For video generation performance, we compare with video models that follow the imagc-animation paradigm (2.5D Video Models), including Animate Anyone [2.0]] and Easy Ani- mate [6]]] well native 3D large video models, including Hunyuan-13B [26]. Wan-14BB 14B [2S]. and CogVideoX-555 [6У]] Particularly, due to the high training costs of the first two 3D models, we directly utilized them for zero-shot inference. For 3D motion generation quality, we compare with
the classic MDM [57]] and its latest improved version EMDM [T4]]. Our method generates motion from both images and text (image+text-to-monl mottionn), whereas MDM and EMDM are inherently text-
to-motion systems. For fair comparison, we adapted both baselines into rext-and-in and-image--eee wwl---imaaattttiiiiee to-motion generators by supplying additional reference frames their CLIP encoders.
Quantitative and Qualitative Evaluation. As shown in Table MI our method achieves the highest overall score in video generation. Notably, individual metrics often conflict: high scores in one
7

---

lung Prompe
Animate Anyune
Easy Animate
CogVideoX-50
scrape off pan
Figure 3: Visualization of videos. Red boxes highlight artifacts such deformation, hallucinations, distortion, and implausible motions.
Rrt Ing Pronspt
MDW
EMDM
Figure 4: Quantitative comparison of motion generation results. Our method achieves smoother motions, sharper object point clouds, and higher instruction compliance.
aspect may compromise others. For instance, Hunyuan- 13B [26] attains top subject/background consistency and hiihhet temporal smoothness, but its near-static outputs (dynamic degree: 0.49) yield the lowest overall score. Wan-14B [e5]] also exhibits the same phenomenon, and the lack of TACO dataset fine-tuning results in poor instruction adherence and hallucinations (Fig. B. Row 1). The 2.5D image animation methods [2II, [i]] achieve high dynamic degree yet exhibit inadequate content consistency and temporal smoothness, visually manifesting as distortions and temporal flickering (Fig. E. Row 2). Cog VideoX-SBB [69]] achieves the second-
overall score, but the generated videos still exhibit inconsistencies, as shown on the left the last row of Fig. In contrast, our method benefits from the synchronized modeling of visual and dynamic, resulting better comprehensive performance.
For motion generation, our method achieves su- perior performance across all metrics, as shown in Tab. D. Qualitative results in Fig. reveal that MDM [57]]] and EMDM [4]] produce motions with poor instruction compliance and frame consistency. This stems from two limitations: 1) They compress both reference images and text prompts into 512 dimensions through the CLIP encoder, then simply concatenate them as denoising conditions, which dilutes the instruction signal. 2) Their motion models lack vision aware- ness, causing large discrepancies between generated point clouds and references. Contrastingly, our approach not only preserves input condition effectiveness through a triple-modality adaptive modu- lation mechanism, but also enhances object point cloud consistency with low-level visual priors.
Table 2: Quantitative comparison against motion generation methods. The best results are bold.
Hand
Object
HOI
Method
MPJPE 1 MSmoo.
Cham. J
FID
MDM [52]]
0.3382
0.0366
0.7915
0.4056
EMDM (M4]
0.3255
0.0306
0.7788
0.3681
Ours
0.1087
0.0255
0,1577
0.1050
User Study. To validate our method's effectiveness, we conducted user studies for video and motion generation (Fig. D) For video generation, 26 image-prompp pairs were used to generate videos with six models each, yielding 1,066 valid responses from 41 participants. Our method achieved 78.42% preference rate, significantly outperforming all baselines. In motion generation, 10 image-prompt pairs produced 410 valid responses, with our results surpassing the baseline in 97.56% of cases. These results demonstrate the clear advantages of our video-motion synchronous diffusion model.

---

11.0%
10.6%%
6.19%
B Hany-an-138
Wan-
B Animatchayyse
Animats
MOV EMDD
CogVifroN-511
(b)
Figure 5: User studies for generated videos (a) and motions (b). We received 1,066 and 410 valid responses respectively, and our method significantly outperforms other baselines.
Ref, Lin Prompes
Generated Video Trames
Generated Motien Sequences
dethe colle
so quaiil
the
Figure 6: Zero-shot inference on real-worke data. We collect data with everyday objects and gen- erate fighfiidlityy, plausible HOI videos and 3D motions. demonstrating the generalizability of our method.
Table 3: Ablation studies on the synchronized diffusion and the vision-aware 3D interaction diffu- sion.
||
Content
Dynamic
Hand
Object
HOI
Varients
Overall
|| Subj.
Bkg. 1
TSmoo.
Dyn. 1
MPJPE
MSmoo.
Cham.
FID
SViMo VID (Ours)
0.9534
0.9546
0.9883
0.9784
0,8800
0.0121
0.0053
0.0019
0.0100
SVIMo w/ Inter, Guid.
0.9522
0.9546
0.9877
0.9768
0.8770
0.0157
0.0060
0.0022
0.0100
SVIMo w/ Grad. Cons.
0,9525
0.9881
0.9757
0.8723
0.0141
0.0058
0.0124
SViMo wlo VID
0.9543
0,9545
0.9883
0.9686
0.8719
0.0195
0.0070
0.0037
0.0546
VModel w/ Pred. Mot.
0.9356
0,9392
0.9858
0.9675
0.8381
0.0202
0.0074
0.0040
0.0575
4.4 Generalization and Extensibility
We evaluate our method on manipulation tasks involving common household objects such rollers, spatulas, spoons, and bowls, for which we collect image-prompp pairs. These inputs are processed by our synchronized diffusion model to generate human-object interaction (HOI) videos and corre- sponding 3D interactions, as shown in Fig. H. The successful generation of plausible interactions on real-world object categories demonstrates the generalization capability of our approach. This gen- cralization stems from two key design principles. First, our plug-and-play synchronized diffusion architecture effectively leverages the visual-semantie priors encoded in large video foundation mod- els. Second, the use of 3D point clouds as object representations provides detailed spatial geometry and enhanced physical awareness, which supports robust reasoning across different viewpoints.
Furthermore, the framework inherently extensible to diverse HOI scenarios. By adjusting point cloud configurations, it readily handles multiple objects, complex geometries, or articulated struc- tures. For example, adding objects only requires including additional point clouds, and articulated objects (e.g., drawers) can be decomposed into rigid sub-components modeled separate point clouds. Increasing point density captures fine details in deformable complex shapes. This flexi- bility highlights the method's practical potential for modeling varied real-world interactions.

---

4.5 Ablation Study
Effectiveness of Synchronized Diffusion. We argue that integrating visual priors and physical dynamics into synchronized diffusion process is essential for HOI video and motion genera- tion. To validate our synchronized diffusion mechanism: (1) We first remove VID to avoid con- founding factors (SViMo w/o VID). (2) Then we decompose it into two independent components: a motion generation model with only motion loss and video generation model conditioned on groundtruth motion (VModel w/ GT Mot. Guid.). After training these models independently, we use the predicted motions from the former as conditions for the latter during inference (VModel w/ Pred. Mot.). The last two rows of Tab. B show that modeling video and motion indepen- dently not only leads to a 3.88% decrease in video overall score (0.8719 vs. 0,8381), but also re- sults in a 5.31% degradation in motion FID (0.0546 0.0575). This highlights the importance of our synchronous diffusion model in enabling feature-level synergy between video and motion. This highlights the advantage of integrating visual priors and motion dynamics for our method.
SViMe wio VID
Impact of Vision-aware 3D Interaction Diffusion. The vision-aware 3D interaction diffusion model forms a closed-loop feedback and co-evolution mechanism with the synchronized video-motion diffusion, by injecting in- teraction guidance and gradient constraints into the latter. To validate its effectiveness, we conduct four variants: re- moving VID entirely (SViMo w/o VID), applying only gradient constraints (SViMo w/ Grad. Cons.), provid- ing only interaction guidance (SViMo w/ Inter. Guid.), and preserving the full VID (SViMo w/ VID). Evaluation results in Tab. & and training loss curves in Fig. show that direct interaction guidance slightly outperforms gradient constraints, while the complete VID achieves the best performance.
SViMo Grid.
Shitde Grrd Cons.
VMdeei so Mee
206
1300
Skep
Figure 7: Video loss curves of different variants during the training process.
Influence of 3D Data on Convergence Speed and Performance. Our method simultaneously generates 2D videos and 3D motions. To evaluate the contribution of 3D motion data to training convergence and output quality, we conduct an ablation study in which 3D motion are replaced with 2D data (Ours-2D-Ve) 2D-VDD.
The comparison is performed using three met- ries: motion FID, video loss at IK training steps, and overall video quality. As shown in Tab. AL incorporating 3D motion diffusion leads to more plausible motions, faster convergence, and a higher overall video score. In contrast, the variant that completely removes the explicit motion generation module (S ViM w/o VID) exhibits further degradation attributed to the fact that 3D motion representation enhances the model's understanding of object scale and spatial occlusion relationships.
Table 4: Ablation analysis of the impact of 3d data on convergence speed and performance.
Method
Mo. FID
Vid. Loss @1K
Over. Qual.
SVIMo wio VID
-3,4883
-41912
0,8719
Ouns-2D. VID
0.0522
33. 3.8266
0.8721
0.0100
.4.9119
0.8800
both convergence speed and final performance. This can be largely
5 Conclusion
In conclusion, we propose synchronized diffusion model that unifies hand-object interaction video generation and motion synthesis in a single diffusion process. By jointly modeling the co-evolution of appearance and motion, our method produces visually realistic and dynamically plausible results. A key component is the vision-aware 3D interaction diffusion model, which guides the denoising process via gradient constraints in a closed-loop pipeline, greatly improving video-motion consis- tency. The framework requires no predefined conditions and shows strong zero-shot generalization in real-world settings. It is also inherently extensible, easily adapting to diverse HOI scenarios, such as those with multiple, complex, or articulated objects, through simple adjustments to the 3D point cloud representation. This paradigm offers a promising direction for multimodal alignment and building world models that understand complex physical interactions.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China (NSFC) No.62125107 and No.62272172.
References
Tenglong Ao. Body of her: preliminary study end-to-end humanoid agent. arXiv preprint
arXiv:2408. 02879, 2024.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 2025.
[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:311.15127,, 2023.
[4] Jona Braun, Sammy Christen, Muhammed Kocabas, Emre Aksan, and Oumar Hilliges. Physically plau- sible full-body hand objeet interaction synthesis. In 2024 International Conference on Vision (3DV), pages 464 473. IEEE, 2024.
[5] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Tay- lor, Troy Luhman, Eric Luhman, et al. Vidco generation models as world simulators. 2024. URL http:://openai. com/researclliideegeeeeee weaeeeo--geeerttoon-meeeee mocooon-oddllaassee worolddd------ld------oo sinnddd-mmlators, 3:1, 2024.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sulf-super vision transformers. Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[7] Junuk Cha, Jihycon Kim, Jac Shin Yoon, and Seungryul Back. Text2hoi: Text-guided 3d motion gener- ation for hand-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1577-1585, 2024.
[8] Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar lqbal, Stan Birchfield, al. Dexych: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pages 9044-9053, 2021.
[9] Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. Videojam: Joint appearance-motion representations for enhanced motion generation in vidco models. arXiv preprint Xiiv:2502.02492.22 2025.
[10] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et Interuvl: Scaling up vision foundation models and aligning for generic visnal-linguistic tasks. Proceedings of the IEEF/CVF conference on computer vision and pattern recog- nition, pages 24185-24198 2024.
[11] Enric Corona, Andrei Zanfir, Eduard Gabriel Bazavan, Nikos Kolotouros, Thiemo Alldieck, and Cristian Sminchisescu. Vlogger: Multimodal diffusion for embodied avatar synthesis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15896-15908, 2025.
[12] Lingwei Dang, Yongwei Nic, Chengjiang Long, Qing Zhang, and Guiqing Li. Diverse human motion prediction via gumbel-softmas sampling from an auxiliary space. Proceedings of the 30th ACM inter- national conference on multimedia, pages 5162-5171, 2022.
[13] Christian Diller and Angela Dai. Cg-hoi: Contact-guided 3d human-object interaction generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19888- 19901, 2024.
[14] Zicong Fan, Omid Taheri, Dimitrios Tzionas. Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Oumar Hilliges. Arctie: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12943-12954 2023.
[15] Rao Fu, Dingxi Zhang, Alex Jiang, Wanjia Fu, Austin Funk, Daniel Ritchie, and Srinath Sridhar. Giga- hands: A massive annotated dataset of bimanual hand activities. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17461-17474, 2025.

---

[16] Jiawei Gao, Ziqin Wang, Zeqi Xiao, Jingbo Wang. Tai Wang. Jinkun Cao, Xiaolin Hu, Si Liu, Jifeng Dai. and Jiangmiao Pang. Coohoi: Learning cooperative human-object interaction with manipulated object dynamics, Advances in Neural Information Processing Systems, 37:79741-79763, 2024.
[17] Yanjiang Guo, Yucheng Hu, Jianke Zhang, Yen-Jen Wang, Xiaoyu Chen, Chaochao Lu, and Jianyu Chen. Prediction with action: Visual policy learning via joint denoising process. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[18] Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang. Yi Zhou, and Michael J Black. Stochastic scene-aware motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11374-11384, 2021.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 0801,,6811118 2020.
[20] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In The Eleventh International Conference on Learning Representations, 2023.
[21] Li Hu. Animate anyone: Consistent and controllable image-to to video synthesis for character animation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8153- 8163, 2024.
[22] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang. Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video gener- ative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
pages
[23] Yujin Jeong, Wonjeong Ryoo, Seunghyun Lec, Dabin Sco, Wonmin Byeon, Sangpil Kim, and Jinkyu Kim. The power of sound (tpos); Audio reactive video generation with stable diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7822-7832, 2023.
[24] Nan Jiang, Zhiyuan Zhang. Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1737-1747, 2024.
[25] Zeyinzi Jiang, Zhen Han, Chaojje Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing. arXiv preprint arXiv:2503.07598. 2025.
[26] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arx: 233003333 2024.
(27] Nilesh Kulkamni, Davis Rempe, Kylc Genova, Abhijit Kundu, Justin Johnson, David Fouhey, and Lconidas Guibas. Nifty: Neural object interaction fields for guided human motion synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 947-957, 2024.
[28] Jihyun Lee, Shunsuke Saito, Giljoo Nam, Minhyuk Sung, and Tac-Kyun Kim. Interhandgen: Two-hand interaction generation via cascaded reverse diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 527-537, 2024.
[29] Jiaman Li, Alexander Clegg. Roozbeh Mottaghi, Jiajun Wu, Xavier Puig, and Karen Liu. Controllable human-object interaction synthesis. In European Conference on Computer Vision, pages 54-72, Springer, 2024.
[30] Jiaman Li, Jiajun Wu, and Karen Liu. Object motion guided human motion synthesis. ACM Transac- tions on Graphics (TOG), 12 2023.
[31] Quanzhou Li, Jingbo Wang. Chen Change Loy, and Dai. Task-oriented human-object interactions generation with implicit neural representations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3035-3044, 2024.
[32] Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. In The Twelfth International Conference on Learning Representations, 2024.
[33] Zhen Li. Zuo ZuonLiang Zhu, Ling-Han Hao Han, Qibin Hou, Chun- Guo, and Ming-Ming Cheng. Amt: All- pairs multi-field field transforms for efficient frame interpolation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9801-9810, 2023.
12

---

[34] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-3496,, 2023.
[35] Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Rongxin Jiang, Jiebo Luo, Hao Fei, et al. Javisdit: Joint audio-video diffusion transformer with hicrarchical spatio-temporal prior synchronization. arXiv preprint arXiv:2503.0 23377, 2025.
[36] Siqi Liu, Yong-Lu Li, Zhou Fang, Xinpeng Liu, Yang You, and Cewu Lu, Primitive-based 3d human- object interaction modelling and programming. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 3711-3719, 2024.
[37] Xucyi Liu and Li Yi. Gencoh diffusion: Towards generalizable hand-object interaction denoising via denoising diffusion. In The Twelfth International Conference on Learning Representations, 2024.
[38] Yun Liu, Haolin Yang, Xu Si, Ling Liu, Zipeng Li, Yuxiang Zhang. Yebin Liu, and Li Yi. Taco: Bench- marking generalizable bimanual tool-action-object understanding. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Puttern Recognition, pages 21740-21751, 2024.
[39] Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category- human-object interaction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 21013- 21022, 2022.
[40] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong. Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning, arXiv preprint arXiv:2503.07785 2025.
[41] Zhengyi Luo, Jinkun Cao, Sammy Christen, Alexander Winkler, Kris Kitani, and Weipeng Xu. Omni- grasp: Grasping diverse objects with simulated humanoids. Advances in Neural Information Processing Systems, 37:2161-21 218444 2024.
[42] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106 106, 2021.
[43] Youxin Pang. Ruizhi Shao, Jiajun Zhang, Hanzhang Tu, Yun Liu, Boyao Zhou, Hongwen Zhang, and Yebin Liu. Manivideo: Generating hand-object manipulation video with dexterous and generalizable grasping. arXiv preprint arXiv:2412. 16212, 12, 2024.
[44] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 4195-4205, 2023.
[45] Xiaogang Peng, Yiming Xie, Zizhao Wu, Varun Jampani, Deqing Sun, and Huaizu Jiang. Hoi- diff: Text-driven synthesis of 3d human-object interactions using diffusion models. arXiv preprint arXiv:2312.0655. 2023.
[46] Huaijin Pi, Sida Peng, Minghui Yang, Xiaowei Zhou, and Hujun Bao. Hierarchical generation of human- object interactions with diffusion probabilistic models. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pages 15061-15073, 2023.
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin. Jack Clark, al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages 8748-88888 44483-763..88 PmLR, 2021.
[48] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 1164 2020.
[49] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1 1-16. IEEE, 2020.
[50] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821 8821-8831. Pmnl, 2021.
[51] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2022.
13

---

[52] Chitwan Saharia, William Chan, Saurabh Saxena. Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text- to-image diffusion models with deep language understanding. Advances in neural information prucessing systems, 35:36479-36498 -6499, 2022.
[53] Himanshu Gaurav Singh, Antonio Loquercio, Carmelo Sterrazza, Jane Wu, Haozhi Qi. Pieter Abbeel, and Jitendra Malik. Hand-object interaction pretraining from videos. arXiv preprint arXiv:2409.08273, 2024.
[54] Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas. Grab: A dataset of whole-body human grasping objects. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part IV 16. pages 581-600. Springer, 2020.
[55] Zineng Tang. Ziyi Yang. Chenguang Zhu, Michael Zeng, and Mohit Bansal, Any-to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 06:16083-16099) 2023.
[56] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. pages 402-419. Springer, 2020.
[57] Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano, Human motion diffusion model. In The Eleventh International Conference on Learning Representations, 2023.
[58] Rong Wang, Wei Mao, and Hongdong Li. Deepsimho: Stable pose estimation for hand-object interaction via physics simulation. Advances in Neural Information Processing Systems, 36:79685-79697, 2023.
[59] Weihan Wang. Qingsong Lv. Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang. Lei Zhao, Song XiXuan, et al. Cogylm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 37:121475-12149, 2024.
[60] Yinhuai Wang, Jing Lin, Ailing Zeng, Zhengyi Luo, Jian Zhang, and Lci Zhang. Physhoi: Physics- based imitation of dynamic human-object interaction. arXiv preprint arXiv:3312.0430 2023.
[61] Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang. Fasyanimate: A high-performance long video generation method based on transformer architecture. arXiv preprint arXiv:2405. 18991, 2024.
[62] Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang- Yan Gui. Interdiff: Generating 3d human-object interactions with physics-informed diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14928-1490, 2023.
[63] Sirui Xu, Hung Yu Ling, Yu- Xioni Wang, and Liang- Yan Gui. Intermimic: Towards universal whole- body control for physics-based human-object interactions. 2025.
[64] Sirui Xu, Yu-Xiong Wang, Liangyan Gui, al. Interdreamer: Zero-shot text 3d dynamic human-object interaction. Advances in Neural Information Processing Systems, 37:52858-52888 2024.
[65] Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia- Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1481- 2024.
[66] Ziyi Xu, Ziyao Huang, Juan Cao. Yong Zhang, Xiaodong Cun, Qing Shuai. Yuchen Wang. Linchao Bao, Jintao Li, and Fan Tang. Anchorcrafter: Animate cyberanchors saling your products via human-object interacting vidco generation. arXiv preprint arXiv:2411. 171.:331 2024.
[67] Lixin Yang. Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A large-scale knowledge repository for understanding ence computer vision and pattern recognition, pages 20953-20962, 2022.
interaction. In Proceedings of the IEEE/CVF confer-
[68] Zhengyuan Yang. Linjie Li, Kevin Lin, Jianfeng Wang. Chung-Chine Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint Xiv::2300 (740), 9()1, 2023.
[69] Zhuoyi Yang. Jiayan Teng. Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng et al. Cogvideox: Text-to-video diffusion models with an expert transformer. In The Thirteenth International Conference Learning Representations, 2025.
14

---

[70] Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi. Diverse and aligned audio-to- video generation via text-to-video model adaptation. Proceedings the AAAI Conference Artificial Intelligence, volume 38. pages 6639-6647, 2024.
[71] Xinyu Zhan, Lixin Yang. Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: dataset bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 445 456, 2024.
[72] Jiajun Zhang, Yuxiang Zhang. Liang An, Mengeheng Li, Hongwen Zhang, Zonghai Hu, and Yebin Liu. Manidext: Hand-object manipulation synthesis via continuous correspondence embeddings and residual- guided diffusion. arXiv preprint arXiv:2409.0940 2024.
[73] Xiaohan Zhang, Bharat Lal Bhatnagar, Sebastian Starke, Vladimir Guzov, and Gerard Pons-Moll, Couch: Towards controllable human- mhaii interactions. In European Conference on Computer Vision, pages 518- 535, Springer, 2022.
[74] Wenyang Zhou, Zhiyang Dou, Zeyu Cao. Zhouyingcheng Liao, Jingbo Wang. Wenjia Wang, Yuan Liu, Taku Komura, Wenping Wang, and Lingjie Liu. Emdm: Efficient motion diffusion model for fast and high-quality motion generation. In European Conference Computer Vision, pages 18-38. Springer, 2024.
[75] Shenhao Zhu, Junming Chen, Zuozhuo Dai, Zilong Dong, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu, Champ: Controllable consistent human image animation with parametric guidance. In European Conference on Computer Vision, pages 145-162. Springer, 2024.
[76] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. R1-2: lasoo-language-action models transfer web knowledge to robotic control. In Conference Robot Learning, pages 2165-2183. PMLR, 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract clearly summarizes the core idea and highlights key innovations. The introduction explicitly states the main contributions, supported by theoretical analysis (Sec. B and experimental results (Sec. H 'he claims match the paper's findings, and the scope is well-defined.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings. is fine to include aspirational goals motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper does not have a dedicated limitations section in the main body. We addressed potential limitations in the appendix due to space constraints, allowing the main text to focus on methodological details and results.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-texx system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
The paper presents the theoretical componentsincluding synchronized video- motion diffusion (SViMo), viston-aware 3D interaction diffusion (VID), and their closed- loop co-evolution mechanismin Sec. BL This section describes the network architecture and loss functions. Additional training and inference pseudocode included in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification: The paper provides detailed experimental setup, including datasets, hyper- parameters, and evaluation metrics. Reproduction steps are clearly outlined in the main text (Sec. M) Both qualitative and quantitative results are presented for comparison. We commit to releasing the source code upon paper acceptance.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
17

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The TACO dataset [+0] and the CogVideoX [69] foundation model we already open-sourced, and we commit to releasing the source code upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://mips.ce)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:)
y haapss.c//uullicguuddeess otw.a/dddeeoddeetbbbsssssooooo.o
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the training and test details in the experimental section (Sec. E!) and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the convention in prior works and report the performance number the standard benchmarks.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources and other train- ing details in Sec 2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
19

---

Justification: Due to space limitations, we discuss both potential positive societal impacts and negative societal impacts in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credit the creators or original owners of the assets (e.g., code, data, models) used in the paper, and explicitly mention and respect the corresponding li- censes and terms of use the references section.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
20

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects requiring IRB Ap- provals.
Guidelines:
21

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM for what should or should not be described.
22

---

SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios (Appendix)
In this supplementary material, we provide additional details on our methodology and experiments, and discuss the limitations, thereby enabling a more comprehensive and in-depth understanding of our proposed Synchronized diffusion for video and motion generation (SViMo).. Below is the outline of all contents.
A Pseudo-code for the Training and Inference Phases
In the main text, introduced the Synchronized Video-Motion Diffusion Go (SViMo, Sec. ( which focuses on video generation. We also presented the Vision-aware 3D Interaction Diffusion Ms (VID, Sec. (3,4), designed for motion generation. Additionally, feedback mechanism between these two components (Sec. B Next. we will provide a more detailed description of the training (Alg. M and inference process (Alg. Q using pseudocode. Note that our method does not require the rendered motion video as input during inference. We only require a reference image and prompt as inputs, and we initialize the target outputs (2D video, 2D motion video, and 3D motion) with pure noise, and iteratively denoise them to obtain the final results. As detailed in Alg. C. Line 4, denote the triple noisy latents at timestep as {2/ 2t (h+, oc)) we first apply single-step denoising using VID Ma to produce the refined (but still noisy) 3D motion oo). The refined 3D motions are then rendered back into noisy 2D motion video latent = Since 2, represents a more refined estimate than the original noisy latent z we concatenated with the latter as auxiliary information and input them into SViMo Go.
described closed-loop
Algorithm 1 Joint training process of SViMo and VID.
Input: Reference image I, text prompt P. target video V, target 3D motion (h, o), frozen video VAE encoder E, SViMo network Go. VID network Ma.
Output: Optimized parameters of SViMo (0*) and VID (Ø*).
1: M Proj(h, o)
D rendered motion video projection b calculate latent codes
2: z E(V), =0 E(M), 21 E(I)
3: while not converged do
4: I 1~11
sample time step / D following Eq. b following Eq.
5: Calculate diffused data 2Y M hr, Ot
6:
(ho, ōu) = Mao- ((h 01), (z) 22 ),t)
M
7:
M Proj(ho, on). 2M [(M
direct interaction guidance following Eq. inverse denoising
(E), 20.0) Go(z) 21, 2t ① 2/ .,t)
8:
9:
(ha Ōo.0) = Ma (hi, 0t), (20,0= 2M 20.0).
b. indirect gradient constraint (Eq. R. M
L L= LsviMo W2LvID
calculate loss following Eq.
11: update parameters 0 and Φ by gradient descent
end while
13: return 4r = 0, 00 =
B More Implementation Details
B.1 VBench Metrics
We use some of the evaluation metrics from VBench [22] for quantitative analysis of the generated videos. The details of its implementation could not be fully presented in Sec. of the main text due to page constraints, so provide supplement here.
Subject Consistency score used to evaluate whether a subject (e.g., a person or an object) re- mains consistent throughout video. Specifically, we first extract image features from each frame of the video using DINO [6]. Then we calculate the cosine similarity between features of consec-
utive frames and between each frame and the first frame to characterize subject consistency. The
23

---

Algorithm 2 Generation process of videos and motions.
Input: Reference image I, text prompt P, parameters of the noise scheduler (at, B ) frozen video VAE encoder & and corresponding decoder D, trained SViMo network Go and VID net- work MMg.
Output: Generated HOI video V and 3D motion (ho, Oo).
{() 2: 2 ~ (0 1). zM ~ ~N, 1) (hrr, or) ~ I)
21_
calculate latent codes initialization
for =T.t-T... ... T.......
do
4: (ho, ōo) Mao-gnd ((hr 0c), (z) =t
AMA )t))
following Eq.
M - Proj(ha, on), 20 = {(M)
direct interaction guidance following Eq.
6:
(2) 20 G0 @ 21, 21 M
@ &M P.t)
D inverse denoising
(ho: 0o) = =Ma (hr, or), (20 2M
b. indirect gradient constraint (Eq. E M
8:
BV
yar(l- 2 +
1-û:
/ 21)
1-6c
20
same as MM and Ph.o
0² 1- (1 a1))
constant value
10:
z\=1 N (HV, T2I),
M ~N (H (h1-1, 01-1) N (N o2
end for
12: V ((2))
> decode into raw video
return V. (ho. Op)
calculation formula is as follows:
Ssabj. Σ 2=
((d de) + (de-1 . de)).
(8)
where d, is the DINO feature of the hhh video frame.
Background Consistency. High-quality videos should not only ensure the subject's appearance remains consistent throughout the video, but also should not overlook the consistency of the back-
ground, such scenes, rooms, and tabletops. VBench [22] computes the background consistency
score by utilizing the cosine similarity of CLIP [7 image features, with the calculation process being similar Eq. &
SBKg.
Σ (fe a)) + (c--1 ci))
T
where is the CLIP feature of the thh video frame. Notably, crop the HOI region, following the common practice in HOI generation [[a]], because the global background context has limited influence fine-grained local interactions. Our experiments demonstrate that baseline models still exhibit hallucinated artifacts even with cropped regions. These artifacts, which include the spon- taneous generation of objects, violate temporal coherence (see Fig. E. rows 1-2). Therefore, this metric remains valuable for assessing video consistency.
Temporal Smoothness score is based on prior knowledge that video motion should be smooth, i.e.. linear or quadratic, over very short time intervals few consecutive video frames). quantify the smoothness of a generated video V (V1, U2, 12,... U2r) }. VBench [[2]] first removes odd-numbered
frames (V1, U3, -1) Then uses a video frame interpolation model [[X]] to generate the removed odd-numbered frame sequence (θ1, ....... -1)
Next, it calculates the mean absolute
error between the reconstructed frames and the original removed frames to measure whether the
generated video satisfies the prior knowledge embedded in the video frame interpolation model.
Finally, this value is normalized to the range [0, 1]:
255 MAE 022 - 121-1)
STSmoo. T
(10)
255
Dynamic Degree. The three metries mentioned above, iee. Subject Consistency, Background Con- sistency, and Temporal Smoothness, can individually quantify the performance of generated videos
24

---

in certain specific aspects. However, completely static video may achieve very high scores in them, which is not the desired outcome. To evaluate the dynamic degree generated videos, VBench [22]] employs RAFT [S6]] to calculate optical flow strengths between adjacent frames. The average of the top 5% highest optical flow intensities is taken as the video's 's dynamic score. Once this score exceeds predefined threshold Top. the video is considered dynamic, otherwise, it is deemed static. The overall dynamic score is calculated as follows:
.V "E(AAv Avg (Top: (RAFT (U--1 (V:,, Top)
SDyn.
(11)
N
==1
where N is the number of generated videos.
B.2
Training Loss for 3D Motions
We generate 3D motions using vision-aware 3D interaction diffusion model. During training, apply separate supervision for hand and object motions. The hand loss is computed as:
+ h h I 0.05 h2" hh-h"
Lhand
(12)
where the three terms represent the hand pose and its first-order and second-order differences be- tween frames. For the 3D object point cloud sequence o E KK,,, we first split the object into tool Qwal E (B.,NN and target Ourget E RRIB.N.K/233 then calculate losses separately for these two components and average them obtain the final object loss. Firstly, we directly compute the Chamfer distance between each frame's generated point cloud and the ground truth to ensure consistency in shape contours. Furthermore, to enforce smoothness of the inter-frame motion, we first reshape the data into B x N, K/ // 3], then calculate pairwise Chamfer distances between con- secutive frames for both ground truth and generated motion, which captures the movement dynamic in the point cloud sequence. Finally, we compute the mean absolute error (MAE) between predicted values and ground truth motion dynamics:
L1 Avg (Dehamt (o,
C2 Avg N Σ MAE (Dehamt On-1), 1), ebbamt ō on-1))
(13)
Latj L1 + 0.1 L2,
where consists of Owol and Ourget-
B.3 Pretrained 3D Interaction Reconstruction Model
To assess the difference between the distribution of the generated 3D interactions and that of the ground truth motions, we trained 3D action reconstruction model on the TACO [38] dataset using the variational autoencoder architecture from Diverse Sampling [I22]. During evaluation, we utilized features from the residual graph convolutional layers of this model to compute the FID score.
C Additional Experimential Results
C.1 More Ablation Studies
Our model is built upon the Cog VideoX-5B [ model and extended to a video-motion joint genera- tion model. Additional parameters include the input and output projection layers, and triple modality modulation modules in all 42 DiT Blocks, resulting in a total of 6.02 billion parameters. To reduce training computational consumption, we tested two model variants. First, added motion modal- ity modulation modules only to even-numbered DiT Blocks while retaining only the original text and video modulation modules in odd-numbered DiT Blocks, i.e., Only Even Layers. This de- sign slightly reduced the total parameter count. Second, we applied LoRA Training, which uses low-rank decomposition the original model parameters to significantly decrease the number of trainable parameters. Experimental results in Tab. E show that both variants produce lower-quality videos and motion compared to our default settings. This indicates that effective alignment and fusion of video and motion modality features essential for generating high-quality outputs.
25

---

Table 5: Ablation studies on the synchronized diffusion and the vision-aware 3D interaction diffu-
sion.
Content
Dynamic
Hand
Object
HOI
Varients
Overall
Subj.
Bkg.
TSmoo.
Dyn.
MPJPE
MSmoo. !
Cham. !
FID
Ours
0.9534
0,9546
0,9883
0.9784
0.8800
0.0121
0.0053
0.0019
0.0100
Only Even Layers
0,9552
0.9884
0,9757
0.8783
0.0297
0.0103
0.0127
0.1368
LORA Training
0.9458
0.9523
0.9892
0.9688
0.8631
0.0671
0.0211
0.0555
0.4671
Table 6: Generalization performance on more distant HOI scenarios.
HOI Data for Evaluation
Overall Video Quality
3D Motion FID
Ours diverse camera viewpoint data)
0.8703
0.0214
Ours completely unseen verbs)
0.8712
0.0197
Ours TACO data)
0,8800
0.0100
C.2 More Results of Our SViMo
Qualitative Results of Our SViMo. We present two additional videos and motion generation results
shown in Figure 8 These results demonstrate videos and motions that both reasonable and consistent. Considering that image frames cannot effectively showcase video effects, we provide videos in the supplementary materials for more vivid demonstration.
Iog Prompts
Video Sequences
scrape plate
bui looel
Figure 8: Visualization of the generated videos and motions. We provide vivid demonstrations in the video of the supplementary materials.
Generalization to More Distant HOI Scenarios. To enable a more rigorous evaluation of gener- alization capability, we curated an additional test set of 100 instances, consisting entirely of unseen human-object interaction (HOI) categories with novel verbs. 'hese were sourced from TACO [[8], OakInk2 [ZI]], and GigaHands [I5]. While TACO provides 12 camera views, we further enhanced viewpoint diversity by selecting approximately 70 instances from OakInk2 (4 views) and GigaHands (51 views). As shown Tab. E. both video and motion metrics on this unseen HOI set remain com- parable to those on the main TACO test set. For the generation involving novel verbs, although slight performance drop is observed, the results stay reasonably close to the main set.
C.3 Comparisons of Our SViMo and Other Methods
Diversity and Physical Metrics of 3D Motions. We add three new evaluation metrics: diversity, penetration rate, and contact rate. The results Tab. show that our method achieves lower diver- sity scores than baselines. This occurs because baseline methods produce 3D motions with visible artifacts, including chaotic object point clouds and significant shape/scaling distortions relative to reference images (Fig. B). Consequently, higher diversity scores don't 't necessarily indicate better
26

---

Table 7: Additional metrics of the generated 3D motion.
Method
Diversity 1
Penetration Rate +
Contact Rate f
MDM [>
6,5308
0.0904
0.8107
EMDM 72
5.7847
0.8251 0.0844
Ours
2.4135
0.0473
0.8328
Table 8: The influence of motion information on baseline video models.
Method
Motion Availability
Motion Type
Overall Video Quality 1
Easy Animate [6]]]
0.8330
w/o
w/o
CogvidenX-5B
0.8684
Animate Anyone [M]]
Input GT Motion
0.8209
Easy Animate 6
w/
0,8366
CogvideoX
Train with motion data
0,8698
Ours
0,8800
Table 9: Comparison on memory and time consumption.
Truining
Inference
Method
Params (B)
Peak VRAM (G)
Time Per Step (s)
Peak VRAM (G)
Latency for steps (s)
Animate Anyone
2.17
39,98
2.45
6.84
20.81
Cog CogVideoX-SB (69
5.57
56.32
11.61
20.59
48.35
Ours
5.82
61.52
18.06
23,35
71.31
motion quality. Additionally, our method achieves superior performance in both penetration rate and contact rate metrics, indicating superior physical plausibility.
Influence of Motion Information on Baseline Video Models. Compared to baseline methods performing video generation alone, approach jointly generates video and motion, and learns visual-motion joint representation through adaptive multimodal modulation. To enable fair compar- isons with baselines and assess the contribution of the modulation mechanism our performance gains, We augmented baseline video models with the motion modality. Specifically, Animate Any- one [2I]] inherently uses Motion GT input. For EasyAnimate [6J0] and Cog VideoX [69]. we trained them using both the original video and the rendered motion video. Results in Tab. & that Animate Anyone produces videos of notably lower overall quality due severe inter-frame flicker- ing (see Fig. B in the main text and supplementary videos). For Easy and CogVideoX, while motion-ave aatre training yielded marginal gains, improvements remained modest. This suggests that multimodal modulation fusion plays more critical role in our approach than the motion modality alone.
VRAM Consumption and Training/Inferecee Time. We compared VRAM consumption and training/inference time with Animate Anyone [2I] and Cog VideoX-5B [69]. Results in Tab. show that Animate Anyone requires significantly lower VRAM consumption and shorter time for both training and inference. This efficiency stems from its incremental design: it extends 2D image generation model with temporal attention modules to enable video synthesis. In contrast, both CogVis VideoX and our method employ unified spatial-temporal full attention. Compared with CogVideoX our approach simultaneously generates both 2D videos and 3D motion sequences, intro- ducing only minimal additional parameters and maintaining computational cost within a reasonable range.
Additional Comparisons of Video Generation Results. We present additional video generation results from three more cases, as shown Figures II and II Other baseline methods' generated results exhibit certain artifacts, such flickering, distortion, and implausible movements, which highlighted in red. Our method demonstrates superior performance compared to theirs.
C.4 Failure Cases
While our approach can generate high-realistic videos and plausible motions most cases, it oc- casionally exhibits certain artifacts such as penetration, low dynamics, and object inconsistency, as
27

---

Honyean- 13B; Wane-44 14B
Haller.
dyn.
Impplan mooy.
Animate Anyone
Distortion
1 Ohii Incouusis.
Antintte
Eas
Cogg VideoX
Impphan mov.
Ours
Figure 9: Comparison of video generation results. The artifacts in the videos generated by the baseline methods highlighted in red. Refer the video in the supplementary material for vivid demonstrations.
Honynan- LMP
Low--dyn dlyn.
Wine 148
Halluc
Anyone
Animate
Flickerinng Distortion
Flickering Implon moo..
Animute
Eay
Cog VideoX
Penctration
Our
Figure 10: Comparison of video generation results. The artifacts in the videos generated by the baseline methods are highlighted in red. Refer to the video in the supplementary material for vivid demonstrations.
shown in Figure 12 However, the overall performance is still better than the baseline method. Ad- dressing these issues, we need explore stronger foundational models on one hand and leverage larger-scale video-motion datasets for training the other hand.
28

---

Cumn Fonee 188
Low Low-dyn.
Min 4P
Camm. shake
Animute Anyune
Distortion
1
Animute
Easy
Halluc.
Cog Videoo
Obj inconnsi.
Our
Figure 11:
Comparison of video generation results. The artifacts in the videos generated by the baseline methods highlighted in red. Refer the video in the supplementary material for vivid demonstrations.
Ref lon Prompts
Generated Video Fromes and Motion Sequences
pour plate
dyaamic
to
seraccoff gluse
kuitle
Oboject disortiim
Figure 12: Visualization of failure cases. Red boxes highlight artifacts.
D Limitations and Future Work
In fine ine-grained hand-object interaction scenarios, simultaneously generating visually high-fidelity videos and physically plausible motions remains a challenging task. While our synchronized dif- fusion model has made significant progress in addressing this challenge, three key limitations still exist. First, our method relies on foundation model trained on large-scale video data, which is then fine-tuned on smaller dataset of video-3D motion pairs. Although the latter has relatively smaller
29

---

data scale, it remains essential for the expansion. Second, while our approach can generate diverse interaction motions, the produced 3D object point clouds are currently restricted rigid, simple objects and struggle with structurally complex geometries. Third, the capabilities of the pre-trained foundation model directly impact both training efficiency and final performance. For instance, using lightweight LoRA (low-rank adaptation) strategies with the Cog VideoX [69] foundation model re- sults suboptimal outcomes, even a full-parameter finetuned model has potential blurring artifacts when sampling at reduced resolution.
To address these challenges, future work should focus on three directions. First, replacing the non- differentiable 3D trajectory representation with differentiable neural representations (e.g., NeRF- style formulations [ could enable video-only supervision without requiring explicit 3D annota- tions. This would transform our vision-aware 3D interaction diffusion model (VID) into a large reconstruction model, potentially resolving the second limitation. Second, continuous following of advanced open-source foundation models is necessary, as their evolving capabilities directly affect training stability and output quality. Third, integrating visual reinforcement learning strategies [40] could further enhance generation fidelity. These improvements would collectively advance the field toward more robust and scalable hand-object interaction synthesis.
E
Broader Impacts
Our proposed synchronized video-man motion diffusion model generates hand-object interaction videos alongside corresponding motion sequences, achieving both visual realism and physically plausible dynamics. This overcomes the limitations of prior approaches that prioritized only visual quality or physical accuracy in isolation. The model demonstrates generalization capabilities to real-world sce- narios, making significant progress in this research domain. Moreover, the joint diffusion paradigm offers valuable insights for cross moos- information integration in multimodal generative models. Practical applications span multiple fields, including game development, digital human animation, and embodied robotics, with potential societal benefits across these domains.
While acknowledging the positive impacts of this technology. also recognize its dual-use nature. The system could be misused to generate synthetic media that violates privacy or perpetuates biases. To address these concerns, we commit ethical development practices and transparent implementa- tion protocols. Our research emphasizes proactive risk mitigation to ensure responsible innovation that maximizes social benefits while minimizing potential harms.
30


Reviews:
[{"review_id": "J0jOh0R9v5", "reviewer": "Reviewer_gC6v", "summary": "This paper presents SViMo, a synchronized diffusion model for video and motion generation in hand-object interaction (HOI) scenarios. The authors observe that HOI involves complex physical dynamics, which pixel-level video generation often fails to capture with high fidelity. They note that visual appearance and motion patterns are governed by shared physical laws, and thus propose synchronized generation of three representations: 3D motion, rendered 2D motion, and video. The SViMo model takes as input text, a reference image, and a rendered motion video, using a triple modality adaptive module to share information across these three modalities, and generates both HOI video and rendered motion video.\nA vision-aware 3D interaction model then takes the two video latents as input, diffusing noise over hand and object joint sets to produce 3D hand skeletons and object surface point clouds. These generated 3D interactions can be used as closed-loop supervision through gradient guidance or rendered to 2D for rendered motion supervision. Quantitative and qualitative experiments show that the proposed method outperforms prior video generation models in terms of quality, dynamics, and motion plausibility. Perceptual studies indicate that the generated videos are preferred by users. Ablation studies evaluate the effectiveness of the VID model and the impact of synchronized training of both HOI RGB and motion video streams.", "questions": "* How is the train-test split performed—by object, by action, or by instance?\n* The paper uses TACO, where in all scenes the subject interacts with exactly two objects. How does the proposed method generalize to scenarios with more objects or with greater variability in object numbers and shapes?", "limitations": "* As noted in the supplementary material, the method has limitations in training efficiency and relies heavily on access to 3D ground truth data.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* The paper addresses an important and challenging problem: generating physically plausible HOI videos.\n* It introduces the novel idea of leveraging rendered motion videos as strong guidance to improve physical plausibility, supported by a tailored training and inference pipeline.\n* The method makes effective use of explicit 3D supervision during training, ensuring consistency and quality in generated motion.\n\n**Weaknesses**\n\n* The comparison with other video generation models is somewhat unfair, as these models do not utilize the additional motion information available in SViMo. For models not trained on this information, it would be interesting to see if they could animate motion videos from the first frame and generate plausible results. For other baselines, it would be useful to include rendered motion as an additional condition, or to show how fine-tuning with this modality affects their performance. This would help ablate the contribution of the extra motion video modality and the triple modality adaptive module to overall video generation quality.\n* Robustness to camera viewpoint is not thoroughly evaluated. The mocap data (TACO) used contains only 12 camera views, and the supplementary video primarily shows results from two views. In real-world settings, camera perspectives are more varied, and since the video encoding does not explicitly incorporate camera information, it is unclear how well the method generalizes. I recommend evaluating the method’s performance across a broader range of viewpoints, possibly using datasets like OAKINK2 or GigaHands, which feature diverse view settings.\n\n  * *OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion*\n  * *GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities*\n* After reviewing the supplementary material, it remains unclear whether the pipeline requires a rendered motion video as input during inference. For example, in Supplementary Algorithm 2 (line 4), Z\\~^{M}\\_{0} is derived directly from h0 and o0. But h0 and o0 are inferred from Ht, Ot, Zt; is this feasible without further guidance? More explanation or clarification would be helpful.", "was_revised": false, "final_justification": "", "rebuttal": "- **Q1: The influence of motion information on baseline video models.** Compared to baseline methods performing video generation alone, our approach jointly generates video and motion, and learns a visual-motion joint representation through adaptive multimodal modulation. To enable fair comparisons with baselines and assess the contribution of the modulation mechanism to our performance gains, we augmented baseline video models with the motion modality. Specifically, Animate Anyone inherently uses GT Motion as input. For EasyAnimate and CogVideoX, we trained them using both the original video and the rendered motion video. Results show that Animate Anyone produces videos of notably lower overall quality due to severe inter-frame flickering (see Fig. 3 in the main text and supplementary videos). For EasyAnimate and CogVideoX, while motion-aware training yielded marginal gains, improvements remained modest. This suggests that multimodal modulation fusion plays a more critical role in our approach than the motion modality alone.\n\n|    **Method**    | **Motion Availability** |     **Motion Type**    | **Overall Video Quality ↑** |\n|:---:|:---:|:---:|:---:|\n|   Easy Animate   |           w/o           |           w/o          |           0.8330            |\n| CogVideoX-I2V-5B |           w/o           |           w/o          |           0.8684            |\n|  Animate Anyone  |            w/           |     Input GT Motion    |           0.8209            |\n|   Easy Animate   |            w/           | Train with motion data |           0.8366            |\n| CogVideoX-I2V-5B |            w/           | Train with motion data |           0.8698            |\n|       Ours       |            w/           | Train with motion data |         **0.8800**         |\n\n\n---\n\n- **Q2: Evaluation on HOI datas with more camera viewpoints.** The TACO dataset we used contains 12 camera views. To assess our method on HOI datasets with greater viewpoint diversity, as suggested by the reviewer, we curated approximately 70 instances from OakInk2[1] (4 views) and GigaHands[2] (51 views). As shown below, both the video and motion metrics on this unseen set are comparable to those on the main TACO test set. This robust generalization capability to camera views stems from two critical design principles: (1) Our plug-and-play synchronized diffusion architecture leverages the visual-semantic priors of large video foundation models. (2) Our 3D point cloud object representation offers enhanced spatial geometry and physical dynamic awareness, and supports viewpoint generalization.\n\n|           **Evaluation Data**           | **3D Motion FID ↓** | **Overall Video Quality ↑** |\n|:---:|:---:|:---:|\n| Ours (on diverse camera viewpoint data) |       0.0214        |           0.8703            |\n|         Ours (on TACO test data)        |     **0.0100**     |         **0.8800**         |\n\n---\n\n- **Q3: Requirement of the rendered motion video during inference.** \nOur method does not require the rendered motion video as input during inference.\nWe only require a reference image and a prompt as inputs, and we initialize the target outputs (2D video, 2D motion video, and 3D motion) with pure noise, and iteratively denoise them to obtain the final results.\nAs detailed in Appendix Alg. 2, Line 4, denote the triple noisy latents at timestep $t$ as $[z_t^V, z_t^M, (h_t, o_t)]$ , we first apply single-step denoising using VID $\\mathcal{M}_\\phi$ to produce the refined (but still noisy) 3D motion ($\\tilde{\\boldsymbol{h}}_0$,$\\tilde{\\boldsymbol{o}}_0$). The refined 3D motions are then rendered back into a noisy 2D motion video latent $\\tilde{\\boldsymbol{z}}_0^M$. \nSince $\\tilde{z}_0^M$ represents a more refined estimate than the original noisy latent $z_t^M$, we concatenate it with the latter as auxiliary information and input them into SViMo $\\mathcal{G}\\theta$.\n\n---\n\n- **Q4: Train-test dataset splitting.** We implemented a two-stage data partitioning strategy. First, to mitigate overlap between test and training sets that might compromise evaluation of generalization, we reserved all instances involving specific actions (e.g., hit), tools (e.g., glue gun), and objects (e.g., toy) as the initial test set, ensuring these elements are absent from the training set. Second, from the remaining instances grouped by <action, tool, object> triplets, we applied weighted sampling according to group sizes to obtain additional test data. Finally, the ratio of the training set to the test set is 9:1 (Line 203 in the main text).\n\n---\n\n- **Q5: Extension potential to HOI scenarios with objects of different numbers and shapes.** Our method inherently supports extension to HOI scenarios involving diverse object numbers, shapes, complex geometries, or even articulated objects. Extending the approach requires only minor adjustments to point cloud density or quantity in our vision-aware 3D interaction diffusion model. Specifically, **(1) Object number expansion** merely needs additional point clouds. **(2) Articulated objects** (e.g., drawers, flip phones) can be decomposed into sub-components and handled via approach (1). **(3) Complex geometries or deformable objects** require increased point density per object for precision.\n\n\n---\n\n- **Q6: Limitations about training efficiency and the reliance on 3D data.** We discussed the limitations in training efficiency of the video foundation model, and the requirement for paired video-3D motion data in Appendix Sec. D. These issues will be alleviated with the development of video foundation models, as well as advances in 3D hand pose recovery and object pose estimation methods.\n\n---\n\n  - [1] Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 445–456, 2024.\n  - [2] Rao Fu, Dingxi Zhang, Alex Jiang, Wanjia Fu, Austin Funk, Daniel Ritchie, and Srinath Sridhar. Giga-hands: A massive annotated dataset of bimanual hand activities. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17461–17474, 2025."}, {"review_id": "3zu3qiHigf", "reviewer": "Reviewer_rh5e", "summary": "This paper introduces SViMo, a two-stream diffusion framework that jointly generates a hand-object interaction (HOI) video and its corresponding 3D motion (hand pose and object point cloud) from an image-and-text prompt.", "questions": "- Could you elaborate on the training and inference efficiency of your model, including memory consumption and runtime costs?\n\n- How does your model generalize to novel verbs (e.g., “crack”, “unscrew”) that are not present in the TACO dataset?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The paper is clearly written and easy to follow.\n\n2. The proposed model, which jointly denoises video and motion in a closed-loop feedback cycle, presents a somewhat novel approach within the domain. Additionally, the removal of pre-defined meshes or pose conditions increases its general applicability.\n\n3. The experimental evaluation is comprehensive, covering diverse baselines, user studies, and real-world zero-shot settings. The results also demonstrate that the proposed method achieves superior performance.\n\n\nWeaknesses\n\n- For motion generation evaluation, the chosen metrics are limited. MPJPE and Chamfer distance are commonly used in prediction tasks, but for generative tasks, metrics like Diversity should also be included. Furthermore, for assessing physical plausibility, metrics such as Contact and Penetration would help substantiate the claims.\n\n\nMinor issues:\n\n- The vertical spacing below Figures 5 and 6 is slightly excessive.", "was_revised": true, "final_justification": "The author's rebuttal has addressed my concerns. As my current score already indicates support for accepting the paper, I will retain it.", "rebuttal": "- **Q1: Diversity and physical metrics of 3D motions.** As suggested by the reviewer, we added three new evaluation metrics: diversity, penetration rate, and contact rate. The following table shows that our method achieves lower diversity scores than baselines. This occurs because baseline methods produce 3D motions with visible artifacts, including chaotic object point clouds and significant shape/scaling distortions relative to reference images (Fig. 4). Consequently, higher diversity scores don't necessarily indicate better motion quality. Additionally, our method achieves superior performance in both penetration rate and contact rate metrics, indicating superior physical plausibility.\n\n| **Method** | **Diversity ↑** | **Penetration Rate ↓** | **Contact Rate ↑** |\n|:---:|:---:|:---:|:---:|\n|     MDM    |   **6.5308**   |         0.0904         |       0.8107       |\n|    EMDM    |     5.7847      |         0.0844         |       0.8251       |\n|    Ours    |     2.4135      |       **0.0473**      |     **0.8328**    |\n\n---\n\n- **Q2: Memory and time consumption for training and inference.** We compared the VRAM usage and training/inference time with our backbone CogVideoX-5B, and found that our method simultaneously generates both 2D video and 3D motion sequences while introducing only minimal additional parameters, and maintaining computational cost within a reasonable range.\n\n|          **Method**         | **Params (B)** | **Training Peak VRAM (G)** | **Training Time Per Step (s)** | **Inferance Peak VRAM (G)** | **Inferance Latency for 50 steps (s)** |\n|:---:|:---:|:---:|:---:|:---:|:---:|\n| CogVideoX-5B (Our Backbone) |     5.57      |           56.32           |              11.61             |            20.59            |                  48.35                 |\n|             Ours            |     5.82      |           61.52           |              18.06             |            23.35            |                  71.31                 |\n\n---\n\n- **Q3: Generalization to actions with novel verbs.** Our train-test split mitigates HOI category overlap by reserving all instances involving specific actions (e.g., hit), tools (e.g., glue gun), and objects (e.g., toy) exclusively for the test set, ensuring these categories remain unseen during training. Thereby the evaluation on test data (Tables 1 & 2 in the main paper) could demonstrate generalization capabilities to novel verbs to some extent. For more rigorous generalization evaluation, we constructed an additional test set of 100 instances comprising entirely unseen HOI categories with novel verbs, sourced from TACO, OakInk2 [1], and GigaHands [2] datasets. As shown below, while video and motion metrics for this unseen set show a slight performance decline compared to the main TACO test set, they remain reasonably close. This robust zero-shot generalization capability stems from two critical design principles: (1) Our plug-and-play synchronized diffusion architecture leverages the visual-semantic priors of large video foundation models. (2) Our 3D point cloud object representation offers enhanced spatial geometry and physical dynamic awareness, and supports viewpoint generalization.\n\n\n|    **Evaluation Data**    | **Overall Video Quality ↑** | **3D Motion FID ↓** |\n|:---:|:---:|:---:|\n| Ours (on completely unseen verbs) |           0.8712            |       0.0197        |\n|      Ours (on TACO test data)     |         **0.8800**         |     **0.0100**     |\n\n\n---\n\n  - [1] Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 445–456, 2024.\n  - [2] Rao Fu, Dingxi Zhang, Alex Jiang, Wanjia Fu, Austin Funk, Daniel Ritchie, and Srinath Sridhar. Giga-hands: A massive annotated dataset of bimanual hand activities. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17461–17474, 2025."}, {"review_id": "QNlTJL56Ek", "reviewer": "Reviewer_Lhcy", "summary": "This paper presents a framework that integrates HOI action generation models with HOI video generation models, enabling closed-loop collaborative optimization between the two. The approach facilitates the transfer of useful information and physical principles between 2D and 3D data, thereby enhancing the generalization and effectiveness of HOI video and motion generation.", "questions": "From my perspective, exploring how to combine the strengths of 3D and 2D models is an interesting topic. This direction becomes even more appealing in the complex HOI scenarios, where leveraging both modalities can be especially beneficial.\n\nI am also highly interested in the generalizability of the proposed framework. Specifically, it would be valuable to understand whether it can be extended to more diverse object and human interaction scenarios, such as human-scene interactions, full-body representations, deformable or flexible objects, and architectural objects (e.g., flip phones or earbud cases).\n\nIn addition, it would be helpful to include results or analysis regarding the convergence behavior of the model on both 3D and 2D data.\n\nThe comparison with 3D motion generation models also requires clarification. To my knowledge, the compared 3D motion models, MDM and EMDM, are text-to-motion, whereas the 3D motion generation in this work appears to be conditioned on input images.\n\nMoreover, a comparison of resource and time consumption with methods like AnimateAnyone under the same resolution setting would be informative for understanding the efficiency and scalability of the proposed approach.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThis paper explores how to effectively combine 3D and 2D models within the HOI (Human-Object Interaction) context to leverage complementary information from both modalities. This is an interesting and potentially impactful direction for the community.\n\nThe authors propose a unified optimization pipeline and validate its effectiveness on both action generation and video generation tasks. The visual quality of the generated videos appears promising.\n\nThe paper is clearly written, with sufficient methodological details, making the work seem reproducible.\n\nWeaknesses:\n\nThe Multimodal Feature Modulation and Fusion's validation is somewhat insufficient. Balancing features across different modalities is a critical issue. To better demonstrate the effectiveness of the feature alignment in this module, it is recommended that additional ablation or validation experiments be included.\n\nSome technical details are not clearly explained. For example, the Chamfer loss in Eq. (2) should be further clarified or properly cited in the main text.\n\nSome related works that combine motion and video, although not in a joint optimization setting, should be cited and discussed for completeness. Relevant examples include:\n\nCorona, Enric, et al. \"Vlogger: Multimodal diffusion for embodied avatar synthesis.\" CVPR 2025.\n\nChefer, Hila, et al. \"VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models.\" arXiv preprint arXiv:2502.02492 (2025).", "was_revised": true, "final_justification": "The authors have addressed the concerns I raised, and after also considering the feedback from other reviewers, I believe that this paper tackles an interesting and important problem. The generation results are also promising. On the other hand, I encourage the authors to further explore scenarios in more sophisticated HOI scenarios. Overall, I keep my score of accept.", "rebuttal": "- **Q1: Ablation study of the multimodal fusion mechanism.** We replaced our adaptive modulation with a simpler linear projection (Linear Proj. (Simpler)) mapping features to a shared embedding space. The results below demonstrate that the linear projection leads to significant degradation in motion FID scores, slower convergence speed, and moderate declines in overall video quality. That is because our triple-modality adaptive modulation has the advantage of inherently decoupling appearance and motion, thereby enabling the extraction of discriminative and complementary video-motion representations.\n\n|      **Multimodal Fusion**     | **3D Motion FID ↓** | **Video Loss @1K (log)** | **Overall Video Quality ↑** |\n|:---:|:---:|:---:|:---:|\n| Simpler Linear Proj. (Simpler) |       0.1573        |         -2.9739          |           0.8643            |\n|        Adap. Mod. (Ours)       |     **0.0100**     |       **-3.4883**       |         **0.8800**         |\n\n---\n\n- **Q2: Chamfer distance loss** is employed in Eq. 6 (not Eq. 2) in the main text with its detailed formulation presented in Eq. 13 in the Appendix. This loss measures the Chamfer distance between the generated object point clouds and the ground-truth point clouds, which also serves as a metric for 3D motion generation quality.\n\n---\n\n- **Q3: Discussion of concurrent works Vlogger[1] (CVPR 2025) and VideoJAM[2] (ICML 2025).** VideoJAM generates video alongside 2D optical flow to enhance visual fidelity and motion plausibility. In contrast, our approach generates video with explicit 3D motion. This enables direct perception of object geometry, scale, and spatial relationships, resulting in superior physical plausibility for complex scenarios involving occlusions or intricate interactions. Vlogger utilizes a two-stage pipeline: first, it generates 3D human poses with one network, then feeds the motions into another separate network for video synthesis. In contrast, we unify video and motion generation within a single synchronized diffusion model. Since HOI video and motion inherently share the same physical laws of the real world, this joint modeling enables the learning of intrinsic co-evolutionary relationships, enhancing the quality of both outputs. We will cite and discuss Vlogger and VideoJAM in the revised manuscript.\n\n---\n\n- **Q4: Extension potential to more sophisticated HOI scenarios.** Our method inherently supports extension to HOI scenarios involving objects of complex geometries, or even articulated objects, and full-body interactions. Extending the approach requires only minor adjustments to point cloud density or quantity, or pose representation in our vision-aware 3D interaction diffusion model. Specifically, **(1) Complex geometries or deformable objects** require increased point density per object for precision. **(2) Articulated objects** can be decomposed into sub-components and increase the total number of point clouds. **(3) Full-body interaction** requires extending the current hand pose to a full-body skeleton pose. Notably, **human-scene interaction** is distinct from our task. Those works typically take static 3D scenes as input and generate plausible human motions[3][4].\n\n---\n\n- **Q5: Convergence behavior on the 2D and 3D data.** Our model comprises two core components: synchronized diffusion for generating 2D video and motion video, and 3D interaction diffusion for synthesizing 3D HOI motions. To analyse the convergence behavior, we conduct comparative experiments using only 2D data to train the synchronized diffusion versus using joint 2D/3D data to train the whole model. Results show that at identical training steps (1K, 3K), integration of 3D data achieves lower training loss, faster convergence, and superior final performance.\n\n|      **Method**      | **Video Lose @1K (log) ↓** | **Video Lose @3K (log) ↓** | **Over Video Quality ↑** |\n|:---:|:---:|:---:|:---:|\n| Ours w/ Only 2D Data |          -3.4883           |          -4.1912           |          0.8719          |\n| Ours w/ 2D & 3D Data |        **-4.9119**        |        **-5.9522**        |        **0.8800**       |\n\n---\n\n- **Q6: Comparison setting for 3D motions generation between ours and baselines.** Our method generates motion from both images and text (image+text-to-motion), whereas the original MDM and EMDM are text-to-motion systems. For fair comparison, we adapted both baselines into text-and-image-to-motion generators by additionally adding the reference image into their CLIP encoders (details in lines 253-260).\n\n---\n\n- **Q7: Memory and time consumption for training and inference.** We compared VRAM consumption and training/inference time between Animate Anyone (as suggested by the reviewer) and CogVideoX-5B (our backbone). Results show that Animate Anyone requires significantly lower VRAM consumption and shorter time for both training and inference. This efficiency stems from its incremental architecture design: it extends a 2D image generation model with temporal attention blocks to enable video synthesis. In contrast, both CogVideoX and our method employ unified spatial-temporal full attention modules. Compared with CogVideoX, our approach can simultaneously generate both 2D videos and 3D motion sequences while introducing only minimal additional parameters and keeping computational cost within a reasonable range.\n\n\n|          **Method**         | **Params (B)** | **Training Peak VRAM (G)** | **Training Time Per Step (s)** | **Inferance Peak VRAM (G)** | **Inferance Latency for 50 steps (s)** |\n|:---:|:---:|:---:|:---:|:---:|:---:|\n| Animate Anyone (Suggested)  |     2.17      |           39.98           |              2.45              |            6.84           |                  20.81                 |\n| CogVideoX-5B (Our Backbone) |     5.57      |           56.32          |              11.61             |            20.59            |                  48.35                 |\n|             Ours            |     5.82      |           61.52           |              18.06             |            23.35            |                  71.31                 |\n\n\n---\n\n\n  - [1] Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. VideoJAM: Joint appearance-motion representations for enhanced motion generation in video models. In Forty-second International Conference on Machine Learning, 2025.\n  - [2] Enric Corona, Andrei Zanﬁr, Eduard Gabriel Bazavan, Nikos Kolotouros, Thiemo Alldieck, and Cristian Sminchisescu. Vlogger: Multimodal diffusion for embodied avatar synthesis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15896–15908, 2025.\n  - [3] Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1737–1747, 2024.\n  - [4] Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Move as you say interact as you can: Language-guided human motion generation with scene affordance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 433–444, 2024."}, {"review_id": "xmi49n6k7r", "reviewer": "Reviewer_T4nk", "summary": "SViMo proposes a joint video and motion diffusion pipeline that extends image-to-video generative models for Hand-Object Interaction (HOI) scenarios. The framework is trained on the TACO dataset, which includes 20 object categories and 15 types of daily hand-object interactions. The pipeline introduces a triple-modality adaptive module that enables integration of text, video, and motion tokens despite differences in scale and feature space. In addition, a vision-aware 3D interaction diffusion module maps and optimizes the generated motion into a 3D point cloud space. The results are evaluated using two types of metrics: (1) VBench-based perceptual metrics for subject/background consistency and temporal smoothness in the generated video, and (2) standard 3D interaction metrics such as mean per-joint position error (MPJPE) and motion smoothness, where SViMo shows improved performance over prior baselines.Strengths\n•\tThe paper is clearly written and the proposed SViMo framework is intuitively motivated, especially in how it bridges video appearance and motion dynamics through a unified joint generation process.\n\n\n•\tThe modular architecture and use of intermediate 2D-rendered motion videos as proxies for 3D motion is practical and well-reasoned, given the domain gap between 3D point clouds and video generation.\n\n\nWeaknesses\n•\tThe related work section could better acknowledge and position this work relative to prior works such as VideoJAM [1], which tackles related challenges in joint video-motion modeling.\n\n\n•\tWhile quantitative 3D evaluation metrics are used, they may not correlate well with perceptual quality in generated videos. User studies or perceptual metrics would strengthen claims of motion plausibility and realism.\n\n\n•\tThe zero-shot generalization results remain limited in scope, although the model is evaluated on unseen HOI prompts, the object categories (e.g., bowl, spoon) remain closely tied to the fine-tuning distribution. It would be more compelling to include qualitative or quantitative results on semantically distant HOI categories (grabbing a phone and dialing numbers).\n\n\n•\tOne of the VBench metrics used is background consistency which is affected by the authors’ choice to crop hand regions, which could confound the metric’s validity and should be acknowledged more clearly.\n\n\n•\tIt would be beneficial to have more supplementary video examples available outside of the curated video presentation to allow reviewers to explore the full diversity of generations independently.", "questions": "Zero-shot Generalization Scope\n Were any test prompts explicitly filtered to ensure no overlap with the object or interaction types seen during training? While the paper claims out-of-domain HOI generation,prompts involving bowls and spoons still appear semantically close to training categories. Could the authors include examples involving semantically unrelated HOIs to better demonstrate true generalization?\n\n\nAlignment Between Evaluation Metrics and Human Perception\nThe current evaluation relies heavily on structural 3D metrics like Chamfer distance. However, these metrics may not correlate well with human-perceived motion plausibility or overall video quality. Would it be possible to provide even a small-scale user preference study, or discuss whether such perceptual evaluation was considered or piloted?\n\n\nAblation on Fusion Strategy Simplicity\nThe proposed triple-modality adaptive modulation with scaling, shifting, and gating is a compelling approach to multimodal alignment. Could the authors clarify whether they explored simpler fusion baselines, such as projecting all modalities into a shared embedding space via linear layers.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n•\tThe paper is clearly written and the proposed SViMo framework is intuitively motivated, especially in how it bridges video appearance and motion dynamics through a unified joint generation process.\n\n\n•\tThe modular architecture and use of intermediate 2D-rendered motion videos as proxies for 3D motion is practical and well-reasoned, given the domain gap between 3D point clouds and video generation.\n\n\nWeaknesses\n•\tThe related work section could better acknowledge and position this work relative to prior works such as VideoJAM [1], which tackles related challenges in joint video-motion modeling.\n\n\n•\tWhile quantitative 3D evaluation metrics are used, they may not correlate well with perceptual quality in generated videos. User studies or perceptual metrics would strengthen claims of motion plausibility and realism.\n\n\n•\tThe zero-shot generalization results remain limited in scope, although the model is evaluated on unseen HOI prompts, the object categories (e.g., bowl, spoon) remain closely tied to the fine-tuning distribution. It would be more compelling to include qualitative or quantitative results on semantically distant HOI categories (grabbing a phone and dialing numbers).\n\n\n•\tOne of the VBench metrics used is background consistency which is affected by the authors’ choice to crop hand regions, which could confound the metric’s validity and should be acknowledged more clearly.\n\n\n•\tIt would be beneficial to have more supplementary video examples available outside of the curated video presentation to allow reviewers to explore the full diversity of generations independently.", "was_revised": true, "final_justification": "My final score is Borderline Accept. The justification is found in the sequence of initial review / rebuttal / comments.", "rebuttal": "- **Q1: Discussion of VideoJAM [1] (ICML 2025).** The concurrent work VideoJAM generates video alongside 2D optical flow to enhance visual fidelity and motion plausibility. In contrast, our approach generates video with explicit 3D motion. This enables direct perception of object geometry, scale, and spatial relationships, resulting in superior physical plausibility for complex scenarios involving occlusions or intricate interactions. The core challenge in 3D joint video-motion generation lies in the significant domain gap between visual and dynamic information, as well as the difficulty in ensuring consistency between both outputs. To address this, we bridge the multimodal domain gap through adaptive modulation and establish a closed-loop co-evolutionary pipeline. This framework effectively captures richer complementary information between video and motion. To validate our improvement over VideoJAM, we replace our default synchronized video-3D-motion diffusion model with a video-2D-motion variant to mimic VideoJAM. Results demonstrate that incorporating 3D motion diffusion achieves more reasonable motions, faster convergence speed, and improved overall video score, and significantly higher human perceptual quality.\nWe will cite and discuss VideoJAM in the revised manuscript.\n\n\n|                **Method**                | **Motion FID ↓** | **Video Lose @1K (log) ↓** | **Over Video Quality ↑** |\n|:---:|:---:|:---:|:---:|\n|     Ours (Video-2D-Motion Diffusion)     |      0.0522      |          -3.8265           |          0.8721          |\n| Ours (Video-3D-Motion Diffusion,Default) |    **0.0100**   |        **-4.9119**        |        **0.8800**       |\n\n\n\n---\n\n- **Q2: Human perceptual evaluation of 3D motions.** For 3D motion evaluation, we employed MPJPE, motion smoothness, chamfer distance, and FID metrics, detailed in Sec. 4.1 and Tables 2 & 3. To provide evaluation more consistent with human perception, we have conducted a user study (Fig. 5(b) in the main paper) involving 410 participants, which demonstrates our method's superiority over the baselines in 97.56% of cases.\n\n---\n\n- **Q3: Zero-shot generalization to more distant HOI categories.** Our train-test split mitigates HOI category overlap by reserving all instances involving specific actions (e.g., hit), tools (e.g., glue gun), and objects (e.g., toy) exclusively for the test set, ensuring these categories remain unseen during training. Thereby our evaluation on the TACO test data could demonstrate zero-shot generalization capabilities to some extent. To further validate this capability in real-world scenarios, we conducted experiments using daily HOI examples (Fig. 6). For more rigorous generalization evaluation, we constructed an additional test set of 100 instances comprising entirely unseen HOI categories (novel objects, actions, and diverse camera viewpoints), sourced from TACO, OakInk2[2], and GigaHands[3] datasets. As shown below, while video and motion metrics for this unseen set show a slight performance decline compared to the main TACO test set, they remain reasonably close. This robust zero-shot generalization capability stems from two critical design principles: (1) Our plug-and-play synchronized diffusion architecture leverages the visual-semantic priors of large video foundation models. (2) Our 3D point cloud object representation offers enhanced spatial geometry and physical awareness, and supports viewpoint generalization.\n\n|        **Evaluation Data**       | **Overall Video Quality ↑** | **3D Motion FID ↓** |\n|:---:|:---:|:---:|\n| Ours (on completely unseen data) |           0.8712            |       0.0197        |\n|     Ours (on TACO test data)     |         **0.8800**         |     **0.0100**     |\n\n\n---\n\n- **Q4: Demonstration of more diverse videos.** As suggested by the reviewer, we will include more representative video samples, including both successful demonstrations (cases mentioned in Q3) and systematically analyzed failure cases, in the upcoming supplementary material to further substantiate the practical versatility of our approach.\n\n---\n\n- **Q5: Validity of the VBench metric (background consistency).** Firstly, as defined in VBench [4], the \"background consistency\" metric calculates global CLIP image feature differences between consecutive video frames, reflecting overall video stability across both foreground and background. Secondly, in the field of HOI generation, cropping the HOI region is a common practice[5], as the global background context has limited impact on fine-grained local interactions. Our experiments reveal that despite cropping the HOI region, baseline models still exhibit hallucinated artifacts, such as spontaneous object emergence, violating temporal coherence (See Appendix Fig. 9, rows 1-2). Therefore, this metric remains valuable for video consistency evaluation.\n\n\n\n- **Q6: Ablation study of the multimodal fusion machinism.** As suggested by the reviewer, we replaced the adaptive modulation with a simpler linear projection (Linear Proj. (Suggested)) mapping features to a shared embedding space. The results below demonstrate that the linear projection leads to significant degradation in motion FID scores, slower convergence speed, and moderate declines in overall video quality. That is because our triple-modality adaptive modulation has the advantage of inherently decoupling appearance and motion, thereby enabling the extraction of discriminative and complementary video-motion representations. \n\n|       **Multimodal Fusion**      | **3D Motion FID ↓** | **Video Loss @1K (log) ↓** | **Overall Video Quality ↑** |\n|:---:|:---:|:---:|:---:|\n| Simpler Linear Proj. (Suggested) |       0.1573        |         -2.9739          |           0.8643            |\n|         Adap. Mod. (Ours)        |     **0.0100**     |       **-3.4883**       |         **0.8800**         |\n\n---\n\n  - [1]  Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. VideoJAM: Joint appearance-motion representations for enhanced motion generation in video models. In Forty-second International Conference on Machine Learning, 2025.\n  - [2] Xinyu Zhan, Lixin Yang, Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 445–456, 2024.\n  - [3] Rao Fu, Dingxi Zhang, Alex Jiang, Wanjia Fu, Austin Funk, Daniel Ritchie, and Srinath Sridhar. Giga-hands: A massive annotated dataset of bimanual hand activities. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17461–17474, 2025.\n  - [4] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21807–21818, 2024.\n  - [5] Youxin Pang, Ruizhi Shao, Jiajun Zhang, Hanzhang Tu, Yun Liu, Boyao Zhou, Hongwen Zhang, and Yebin Liu. Manivideo: Generating hand-object manipulation video with dexterous and generalizable grasping. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12209–12219, 2025."}]

Author rebuttal:

Meta-review: Strengths
* Novel closed-loop architecture for generating hand-object interaction videos - unlike many others, which are more or less "ControlNet" variants
  * SViMo: synchronized, all-in-one video generation and motion generation diffusion model for plausible hand-object interaction video generation
  * VID: vision-aware 3D interaction diffusion generates 3D interaction from the SViMo video and motion latent codes to feed into the SViMo interaction guidance to form a closed-loop feedback prompting optimization
* Convincing results qualitatively and quantitatively. Ablations support the design choices.

Weaknesses
* Some clarifications and more experiments, e.g., generalization capabilities, as suggested by reviewers, will strengthen the paper.

I believe this multi-modal generation with closed-loop feedback has great potential beyond hand-object interaction. Because of this general impact of the paper, I recommend accepting it as a spotlight.
Decision: Accept (spotlight)