Title: SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios
Venue: NeurIPS 2025
Accepted: yes
Keywords: hand-object interaction

synchronized diffusion

visual prior

motion dynamic
Abstract: Hand-Object Interaction (HOI) generation has significant application potential. However, current 3D HOI motion generation approaches heavily rely on predefined 3D object models and lab-captured motion data, limiting generalization capabilities. Meanwhile, HOI video generation methods prioritize pixel-level visual fidelity, often sacrificing physical plausibility. Recognizing that visual appearance and motion patterns share fundamental physical laws in the real world, we propose a novel framework that combines visual priors and dynamic constraints within a synchronized diffusion process to generate the HOI video and motion simultaneously. To integrate the heterogeneous semantics, appearance, and motion features, our method implements tri-modal adaptive modulation for feature aligning, coupled with 3D full-attention for modeling inter- and intra-modal dependencies. Furthermore, we introduce a vision-aware 3D interaction diffusion model that generates explicit 3D interaction sequences directly from the synchronized diffusion outputs, then feeds them back to establish a closed-loop feedback cycle. This architecture eliminates dependencies on predefined object models or explicit pose guidance while significantly enhancing video-motion consistency. Experimental results demonstrate our method's superiority over state-of-the-art approaches in generating high-fidelity, dynamically plausible HOI sequences, with notable generalization capabilities in unseen real-world scenarios. Project page at [https://droliven.github.io/SViMo_project](https://droliven.github.io/SViMo_project).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSViMo proposes a joint video and motion diffusion pipeline that extends image-to-video generative models for Hand-Object Interaction (HOI) scenarios. The framework is trained on the TACO dataset, which includes 20 object categories and 15 types of daily hand-object interactions. The pipeline introduces a triple-modality adaptive module that enables integration of text, video, and motion tokens despite differences in scale and feature space. In addition, a vision-aware 3D interaction diffusion module maps and optimizes the generated motion into a 3D point cloud space. The results are evaluated using two types of metrics: (1) VBench-based perceptual metrics for subject/background consistency and temporal smoothness in the generated video, and (2) standard 3D interaction metrics such as mean per-joint position error (MPJPE) and motion smoothness, where SViMo shows improved performance over prior baselines.Strengths\n•\tThe paper is clearly written and the proposed SViMo framework is intuitively motivated, especially in how it bridges video appearance and motion dynamics through a unified joint generation process.\n\n\n•\tThe modular architecture and use of intermediate 2D-rendered motion videos as proxies for 3D motion is practical and well-reasoned, given the domain gap between 3D point clouds and video generation.\n\n\nWeaknesses\n•\tThe related work section could better acknowledge and position this work relative to prior works such as VideoJAM [1], which tackles related challenges in joint video-motion modeling.\n\n\n•\tWhile quantitative 3D evaluation metrics are used, they may not correlate well with perceptual quality in generated videos. User studies or perceptual metrics would strengthen claims of motion plausibility and realism.\n\n\n•\tThe zero-shot generalization results remain limited in scope, although the model is evaluated on unseen HOI prompts, the object categories (e.g., bowl, spoon) remain closely tied to the fine-tuning distribution. It would be more compelling to include qualitative or quantitative results on semantically distant HOI categories (grabbing a phone and dialing numbers).\n\n\n•\tOne of the VBench metrics used is background consistency which is affected by the authors’ choice to crop hand regions, which could confound the metric’s validity and should be acknowledged more clearly.\n\n\n•\tIt would be beneficial to have more supplementary video examples available outside of the curated video presentation to allow reviewers to explore the full diversity of generations independently.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n•\tThe paper is clearly written and the proposed SViMo framework is intuitively motivated, especially in how it bridges video appearance and motion dynamics through a unified joint generation process.\n\n\n•\tThe modular architecture and use of intermediate 2D-rendered motion videos as proxies for 3D motion is practical and well-reasoned, given the domain gap between 3D point clouds and video generation.\n\n\nWeaknesses\n•\tThe related work section could better acknowledge and position this work relative to prior works such as VideoJAM [1], which tackles related challenges in joint video-motion modeling.\n\n\n•\tWhile quantitative 3D evaluation metrics are used, they may not correlate well with perceptual quality in generated videos. User studies or perceptual metrics would strengthen claims of motion plausibility and realism.\n\n\n•\tThe zero-shot generalization results remain limited in scope, although the model is evaluated on unseen HOI prompts, the object categories (e.g., bowl, spoon) remain closely tied to the fine-tuning distribution. It would be more compelling to include qualitative or quantitative results on semantically distant HOI categories (grabbing a phone and dialing numbers).\n\n\n•\tOne of the VBench metrics used is background consistency which is affected by the authors’ choice to crop hand regions, which could confound the metric’s validity and should be acknowledged more clearly.\n\n\n•\tIt would be beneficial to have more supplementary video examples available outside of the curated video presentation to allow reviewers to explore the full diversity of generations independently.\n\nQUESTIONS:\nZero-shot Generalization Scope\n Were any test prompts explicitly filtered to ensure no overlap with the object or interaction types seen during training? While the paper claims out-of-domain HOI generation,prompts involving bowls and spoons still appear semantically close to training categories. Could the authors include examples involving semantically unrelated HOIs to better demonstrate true generalization?\n\n\nAlignment Between Evaluation Metrics and Human Perception\nThe current evaluation relies heavily on structural 3D metrics like Chamfer distance. However, these metrics may not correlate well with human-perceived motion plausibility or overall video quality. Would it be possible to provide even a small-scale user preference study, or discuss whether such perceptual evaluation was considered or piloted?\n\n\nAblation on Fusion Strategy Simplicity\nThe proposed triple-modality adaptive modulation with scaling, shifting, and gating is a compelling approach to multimodal alignment. Could the authors clarify whether they explored simpler fusion baselines, such as projecting all modalities into a shared embedding space via linear layers.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a framework that integrates HOI action generation models with HOI video generation models, enabling closed-loop collaborative optimization between the two. The approach facilitates the transfer of useful information and physical principles between 2D and 3D data, thereby enhancing the generalization and effectiveness of HOI video and motion generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis paper explores how to effectively combine 3D and 2D models within the HOI (Human-Object Interaction) context to leverage complementary information from both modalities. This is an interesting and potentially impactful direction for the community.\n\nThe authors propose a unified optimization pipeline and validate its effectiveness on both action generation and video generation tasks. The visual quality of the generated videos appears promising.\n\nThe paper is clearly written, with sufficient methodological details, making the work seem reproducible.\n\nWeaknesses:\n\nThe Multimodal Feature Modulation and Fusion's validation is somewhat insufficient. Balancing features across different modalities is a critical issue. To better demonstrate the effectiveness of the feature alignment in this module, it is recommended that additional ablation or validation experiments be included.\n\nSome technical details are not clearly explained. For example, the Chamfer loss in Eq. (2) should be further clarified or properly cited in the main text.\n\nSome related works that combine motion and video, although not in a joint optimization setting, should be cited and discussed for completeness. Relevant examples include:\n\nCorona, Enric, et al. \"Vlogger: Multimodal diffusion for embodied avatar synthesis.\" CVPR 2025.\n\nChefer, Hila, et al. \"VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models.\" arXiv preprint arXiv:2502.02492 (2025).\n\nQUESTIONS:\nFrom my perspective, exploring how to combine the strengths of 3D and 2D models is an interesting topic. This direction becomes even more appealing in the complex HOI scenarios, where leveraging both modalities can be especially beneficial.\n\nI am also highly interested in the generalizability of the proposed framework. Specifically, it would be valuable to understand whether it can be extended to more diverse object and human interaction scenarios, such as human-scene interactions, full-body representations, deformable or flexible objects, and architectural objects (e.g., flip phones or earbud cases).\n\nIn addition, it would be helpful to include results or analysis regarding the convergence behavior of the model on both 3D and 2D data.\n\nThe comparison with 3D motion generation models also requires clarification. To my knowledge, the compared 3D motion models, MDM and EMDM, are text-to-motion, whereas the 3D motion generation in this work appears to be conditioned on input images.\n\nMoreover, a comparison of resource and time consumption with methods like AnimateAnyone under the same resolution setting would be informative for understanding the efficiency and scalability of the proposed approach.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper introduces SViMo, a two-stream diffusion framework that jointly generates a hand-object interaction (HOI) video and its corresponding 3D motion (hand pose and object point cloud) from an image-and-text prompt.

STRENGTHS AND WEAKNESSES:
Strengths

1. The paper is clearly written and easy to follow.

2. The proposed model, which jointly denoises video and motion in a closed-loop feedback cycle, presents a somewhat novel approach within the domain. Additionally, the removal of pre-defined meshes or pose conditions increases its general applicability.

3. The experimental evaluation is comprehensive, covering diverse baselines, user studies, and real-world zero-shot settings. The results also demonstrate that the proposed method achieves superior performance.


Weaknesses

- For motion generation evaluation, the chosen metrics are limited. MPJPE and Chamfer distance are commonly used in prediction tasks, but for generative tasks, metrics like Diversity should also be included. Furthermore, for assessing physical plausibility, metrics such as Contact and Penetration would help substantiate the claims.


Minor issues:

- The vertical spacing below Figures 5 and 6 is slightly excessive.

QUESTIONS:
- Could you elaborate on the training and inference efficiency of your model, including memory consumption and runtime costs?

- How does your model generalize to novel verbs (e.g., “crack”, “unscrew”) that are not present in the TACO dataset?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SViMo, a synchronized diffusion model for video and motion generation in hand-object interaction (HOI) scenarios. The authors observe that HOI involves complex physical dynamics, which pixel-level video generation often fails to capture with high fidelity. They note that visual appearance and motion patterns are governed by shared physical laws, and thus propose synchronized generation of three representations: 3D motion, rendered 2D motion, and video. The SViMo model takes as input text, a reference image, and a rendered motion video, using a triple modality adaptive module to share information across these three modalities, and generates both HOI video and rendered motion video.\nA vision-aware 3D interaction model then takes the two video latents as input, diffusing noise over hand and object joint sets to produce 3D hand skeletons and object surface point clouds. These generated 3D interactions can be used as closed-loop supervision through gradient guidance or rendered to 2D for rendered motion supervision. Quantitative and qualitative experiments show that the proposed method outperforms prior video generation models in terms of quality, dynamics, and motion plausibility. Perceptual studies indicate that the generated videos are preferred by users. Ablation studies evaluate the effectiveness of the VID model and the impact of synchronized training of both HOI RGB and motion video streams.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The paper addresses an important and challenging problem: generating physically plausible HOI videos.\n* It introduces the novel idea of leveraging rendered motion videos as strong guidance to improve physical plausibility, supported by a tailored training and inference pipeline.\n* The method makes effective use of explicit 3D supervision during training, ensuring consistency and quality in generated motion.\n\n**Weaknesses**\n\n* The comparison with other video generation models is somewhat unfair, as these models do not utilize the additional motion information available in SViMo. For models not trained on this information, it would be interesting to see if they could animate motion videos from the first frame and generate plausible results. For other baselines, it would be useful to include rendered motion as an additional condition, or to show how fine-tuning with this modality affects their performance. This would help ablate the contribution of the extra motion video modality and the triple modality adaptive module to overall video generation quality.\n* Robustness to camera viewpoint is not thoroughly evaluated. The mocap data (TACO) used contains only 12 camera views, and the supplementary video primarily shows results from two views. In real-world settings, camera perspectives are more varied, and since the video encoding does not explicitly incorporate camera information, it is unclear how well the method generalizes. I recommend evaluating the method’s performance across a broader range of viewpoints, possibly using datasets like OAKINK2 or GigaHands, which feature diverse view settings.\n\n  * *OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion*\n  * *GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities*\n* After reviewing the supplementary material, it remains unclear whether the pipeline requires a rendered motion video as input during inference. For example, in Supplementary Algorithm 2 (line 4), Z\\~^{M}\\_{0} is derived directly from h0 and o0. But h0 and o0 are inferred from Ht, Ot, Zt; is this feasible without further guidance? More explanation or clarification would be helpful.\n\nQUESTIONS:\n* How is the train-test split performed—by object, by action, or by instance?\n* The paper uses TACO, where in all scenes the subject interacts with exactly two objects. How does the proposed method generalize to scenarios with more objects or with greater variability in object numbers and shapes?\n\nLIMITATIONS:\n* As noted in the supplementary material, the method has limitations in training efficiency and relies heavily on access to 3D ground truth data.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}