Title: Restage4D: Reanimating Deformable 3D Reconstruction from a Single Video
Venue: NeurIPS 2025
Accepted: yes
Keywords: Gaussian Splatting; Motion Editing; Monocular Reconstruction
Abstract: Motion is one of the key components in deformable 3D scenes. Generative video models allow users to animate static scenes with text prompts for novel motion, but when it comes to 4D reconstruction, such reanimations often fall apart. The generated videos often suffer from geometric artifacts, implausible motion, and occlusions, which hinder physically consistent 4D reanimation. In this work, we introduce \textbf{Restage4D}, a geometry-preserving pipeline for deformable scene reconstruction from a single edited video. Our key insight is to leverage the unedited original video as an additional source of supervision, allowing the model to propagate accurate structure into occluded and disoccluded regions.
To achieve this, we propose a video-rewinding training scheme that temporally bridges the edited and original sequences via a shared motion representation. We further introduce an occlusion-aware ARAP regularization to preserve local rigidity, and a disocclusion backtracing mechanism that supplements missing geometry in the canonical space. Together, these components enable robust reconstruction even when the edited input contains hallucinated content or inconsistent motion.
We validate Restage4D on DAVIS and PointOdyssey, demonstrating improved geometry consistency, motion quality, and 3D tracking performance. Our method not only preserves deformable structure under novel motion, but also automatically corrects errors introduced by generative models, bridging the gap between flexible video synthesis and physically grounded 4D reconstruction.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a new task, video restaging, and builds a novel framework to achieve controllability and geometric fidellity in 4D reconstruction. The task is to generate new motion for the object in input video. The main idea of this work is to re-animate the first frame of input video and leverage base video to correct the artifacts from generative videos during 4d reconstruction. Video-rewinding joint training strategy is proposed with occlusion-aware ARAP and disocclusion\n\nSTRENGTHS AND WEAKNESSES:\nstrength:\n1. this work propose a novel task, re-animating 4D object in the wild \n2. The paper is easy to follow, and the result is good.\n\nweakness:\n1. lack comparison the state-of-the-art 4D reconstruction methods, e.g., shape-of-motion (the baseline in this work), SCGS and Vidu4D. It is suggested to take sololy new video as input for these methods for comparison.\n2. The evaluation presented in Table 1 is conducted solely on training views, which is not a reasonable approach. It would be more appropriate to evaluate novel view synthesis capabilities. While I acknowledge that generated videos lack ground truth for novel views, the authors could leverage public multi-view video datasets for evaluation, such as the Neural 3D Video Synthesis dataset[1] and Dycheck[2]. A possible evaluation strategy would be to split these videos into two parts: using the first segment as the base video and the second as the reference for generated content. Although this approach would yield artifact-free \"generated\" videos, it's worth noting that state-of-the-art video generation models can also produce artifact-free results after careful cherry-picking.\n[1] Neural 3D Video Synthesis from Multi-View Video\n[2] Monocular Dynamic View Synthesis: A Reality Check\n\nIf my concerns are adequately solved during rebuttal, I'm glad to raise my assessment.\n\nQUESTIONS:\nWere the generative videos cherry-pick? If so, what was the picking criteria?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper presents Restage4D, a pipeline for reanimating deformable 3D scenes by conditioning on a single real-world video. It first generates a synthetic driving video based on the input’s first frame, then the method uses the original sequence as a supervisory signal to correct artifacts from synthetic motion. The input real-world video provides the shape and articulation model, while the generated video provides new dynamics.

STRENGTHS AND WEAKNESSES:
Strengths

1. The paper is well written and the task is highly engaging.

2. The video-rewinding joint-training scheme is both well motivated and insightful.

Weaknesses

1. All examples involve a single foreground object. How does the method handle scenes with multiple interacting objects—for instance, “two people dancing together”?

2. Some minor typos to fix: Line 176: “bas” → “base”, Video at 4:12 (bus case): the wrong prompt is shown.

3. Additional qualitative results—especially on more complex scenes—would strengthen the empirical validation of your approach.

QUESTIONS:
N.A.

LIMITATIONS:
The authors have discussed the method’s limitations, but they omit the societal impact—please include it in the final version.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRestage4D proposes a task to generate a new 3D sequence based on a given video. It proposes a framework that first does image-to-video generation, and then reconstructs the new video together with the original video into 3D.  To do so, a video concatention strategy and occlusion-aware rigidity handling are proposed.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The 4D reanimation task sounds interesting and new. It could be useful for some downstream simulations like autonomous driving and robotics tasks.\n2. The paper identifies the geometry inconsistency as the challenge and tackles it with video rewinding, occlusion-aware rigidity, and occlusion backtracking, which makes a lot of sense to me. The results also show that the method is able to fix some inconsistencies in the newly generated video.\n\nWeakness:\n1. Using image-to-video generation as the first step seems a debatable design choice. The first frame of a video does not necessarily capture all important information, for example, the subject might not be present in the first frame, or textures are not observable in the first frame. To tackle this task, it seems that how to generate videos that are consistent with the input deserves more discussion. For example, we might want to let the video model take the full input video as a condition. Instead, the paper relays the inconsistency problem to the reconstruction step and tries to fix it with rigidity constraints, which I think is less optimal.\n\n2. The task seems to be defined to be too complex and specific, with a mix of reconstruction and generation. Currently, the input is a video, and the output is an edited 4D reconstruction of the input video. It seems to involve several more fundamental tasks, like single-view video reconstruction, video editing, and 3D editing. The complex task setting also makes the evaluation difficult. In the paper, the evaluation mainly reports 3D consistency to the original video and PSNR to the generated video. First, as an editing task (\"restage\"), the editing accuracy is not evaluated at all. One could achieve a perfect score if the video stays identical before and after restaging. Second, evaluating PSNR to the generated video seems a bit weird, since the task does not necessarily require having a generated video as an intermediate output. Overall, my concern is if the task is significant enough to be separated topic and if the current evaluation is really suitable for it.\n\n3. The technical part mainly makes improvements on \"shape-of-motion\" to better serve the purpose of reconstructing one video using the geometry of another video. All the technical parts bring solid improvements, but they are also not significantly novel or transferable to other domains.\n\nMinor:\n\n- L276: \"We evaluate our pipeline on and PointOdyssey\"\n\nQUESTIONS:\nPlease see the weakness section.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the task of video-conditioned 4D content generation. Specifically, given a video depicting an object and a target motion text, the goal is to generate novel videos that are conditioned on (1) the object's appearance from the input video and (2) motion synthesized via image-to-video generation models.\n\nThe key idea is to jointly optimize the reversed input video and the synthesized video, allowing motion coefficients to be shared across both domains.\n\nIn addition, the authors introduce an occlusion-aware rigidity regularization, which leverages the original video to recover occluded regions, as well as a disocclusion backtracing mechanism to handle newly revealed areas in the synthesized video.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1) The paper introduces a new and meaningful task—video-conditioned 4D content generation—effectively bridging generative models with 4D reconstruction.\n2) The proposed occlusion-aware rigidity regularization and disocclusion backtracing are thoughtfully designed to address visibility challenges such as occlusion and disocclusion in dynamic scenes.\n3) The method demonstrates strong generalizability across diverse object types, including both articulated and rigid objects.\n\nWeakness\n1) **[ARAP Regularization and Missing Analysis]**\n ARAP regularization is widely used in 4D reconstruction, so its inclusion alone is not novel. Although the paper introduces an occlusion-aware variant, its impact is unclear without ablation comparing it to standard ARAP. The lack of citations (e.g. SC-GS) and comparison with existing ARAP-based methods further limits the clarity of the contribution.\n\n2) **[Additional ablation on each component]**\nQualitative comparisons showing the effect of each module—e.g., synthetic video optimization only, joint optimization, +ARAP, +occlusion-aware ARAP, and +disocclusion backtracking—would help clarify how each part contributes under different conditions (e.g., occlusion, disocclusion). As it stands, the effectiveness of individual modules difficult to recognize.\n\n3) **[Missing Comparison with Image + Text Methods]**\nThe paper lacks comparison with existing methods that generate video content conditioned on a single image and a text prompt. Including such a baseline would help contextualize the performance of the proposed approach.\n\n4) **[Unconvincing Visual Results in Figure 6]**\nThe output in Figure 6, particularly the camel example, is not visually compelling. Artifacts appear between the camel's humps, which makes the result look worse than the driving video itself. It is also unclear which part of the image demonstrates the claimed motion artifact improvements—highlighting or marking these regions would be helpful.\n\n5) **[Missing Detailed Description on Prompt Refinement Using VLM]**\nThe paper needs justification for why smooth camera motion is necessary. An ablation on VLM refinement is missing, and the instructions used for VLM are not provided, limiting clarity and reproducibility.\n\n6) **[Missing Detailed Description on Experimental Settings]**\nThe paper needs sufficient details on key experimental settings. For example, it is unclear how the \"naive reconstruction\" baseline in Figure 4 is implemented. Visualizations of both training and occlusion views would help clarify the effectiveness of the method. In Table 1, the baseline method referenced is not clearly specified—further explanation is needed. Additionally, it is ambiguous whether the ARAP loss is applied to all points or only to occluded regions. Lastly, the number of input video frames and the number of frames in the synthesized video should be explicitly stated for reproducibility.\n\nQUESTIONS:\n1) **[Clarification on Disocclusion Backtracing]**\nAs I understand it, frames of the synthetic video are jointly optimized during initialization, meaning that disoccluded regions should also be optimized alongside the base video. Given this, it is unclear why the disocclusion backtracing module is necessary I would appreciate it if the authors could elaborate on why these regions are not sufficiently addressed through joint optimization, and clarify the specific motivation for introducing this component.\n\n2) **[Clarification on Evaluation Protocol in Section 4.3]**\nThe evaluation in Section 4.3 raises some questions, as the motion generated by Sora may not precisely align with the ground-truth motion. It would be helpful if the authors could explain how alignment between Sora-generated motion and the ground truth is handled when computing the metrics.\n\n3) **[Assumptions Regarding Input Video]**\nI am also curious whether there are any assumptions or constraints regarding the input video. Since the proposed method leverages the input to handle occlusions, how does it perform when the occluded regions are not visible in the input video either? If the quality of the results is sensitive to the occlusion coverage in the input, I would suggest providing some analysis or discussion on this aspect.\n\n4) **[Request for Additional Ablations]**\nFinally, I believe that additional ablation studies with qualitative comparisons—as mentioned in the Weakness section—would greatly strengthen the paper and help better assess the contribution of each component. Including such analysis would positively impact my overall evaluation.\n\nLIMITATIONS:\nThis paper does not discuss potential societal impacts, which is important given the nature of generative video models.\nAdditionally, the rationale behind L316–317 in the Limitations section is unclear. Why are highly deformable or textureless objects less temporally consistent? Further explanation would be helpful.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}