Restage4D: Reanimating Deformable 3D Reconstruction from a Single Video
Authors: Jixuan He
Chieh Hubert Lin
Lu Qi
Ming-Hsuan Yang
Venue: NeurIPS 2025 poster ()
Keywords: Gaussian Splatting; Motion Editing; Monocular Reconstruction
Primary area: deep_learning

Abstract: Motion is one of the key components in deformable 3D scenes. Generative video models allow users to animate static scenes with text prompts for novel motion, but when it comes to 4D reconstruction, such reanimations often fall apart. The generated videos often suffer from geometric artifacts, implausible motion, and occlusions, which hinder physically consistent 4D reanimation. In this work, we introduce \textbf{Restage4D}, a geometry-preserving pipeline for deformable scene reconstruction from a single edited video. Our key insight is to leverage the unedited original video as an additional source of supervision, allowing the model to propagate accurate structure into occluded and disoccluded regions.
To achieve this, we propose a video-rewinding training scheme that temporally bridges the edited and original sequences via a shared motion representation. We further introduce an occlusion-aware ARAP regularization to preserve local rigidity, and a disocclusion backtracing mechanism that supplements missing geometry in the canonical space. Together, these components enable robust reconstruction even when the edited input contains hallucinated content or inconsistent motion.
We validate Restage4D on DAVIS and PointOdyssey, demonstrating improved geometry consistency, motion quality, and 3D tracking performance. Our method not only preserves deformable structure under novel motion, but also automatically corrects errors introduced by generative models, bridging the gap between flexible video synthesis and physically grounded 4D reconstruction.

Restage4D: Reanimating Deformable 3D Reconstruction from a Single Video
Jixuan He' Chieh Hubert Lin?, Lu Ming- Yang²
Cormell Tech, 2 University of California, Merced, $ Wuhan University h jh2926@corne11.cem
Abstract
Creating deformable 3D content has gained increasing attention with the rise of text-to-image and image-to-video generative models. While these models provide rich semantic priors for appearance, they struggle to capture the physical realism and motion dynamics needed for authentic 4D scene synthesis. In contrast, real- world videos provide physically grounded geometry and articulation cues that are difficult to hallucinate. One question is raised: Can we generate physically consistent 4D content by leveraging the motion priors of the real-world video? In this work, we explore the task reanimating deformable 3D scenes from a single video, using the original sequence supervisory signal to correct artifacts from synthetic motion. We introduce Restage4D, a ceametry-preserving pipeline for video-conditioned 4D restaging. Our approach uses video-rewinding training strategy to temporally bridge a real base video and a synthetic driving video via a shared motion representation. We further incorporate an occlusion-aware rigidity loss and a disocclusion backtracing mechanism to improve structural and geometry consistency under challenging motion. We validate Restage4D on DAVIS and PointOdyssey, demonstrating improved geometry consistency, motion quality, and 3D tracking performance. Our method not only preserves deformable structure under novel motion, but also automatically corrects errors introduced by generative models, revealing the potential of video prior 4D restaging task. Source code and trained models will be released.
Introduction
Recent advances foundational generative models images [1, 2], videos [3], and 3D scenes [4, 5] drive growing interest in the creation of 4D content, which brings time-varying dynamics to 3D. In practical applications, controllability remains a valuable and continuing challenge in all generative frameworks. Inspired by video generative models, early controllable 4D generative frameworks condition on text prompts [6] or an image [7]. This intrigues us to explore other conditioning sources that are beneficial and intuitive for 4D content creation. In particular, we are interested in video conditioning, which is easily accessible on the Internet, exhibits physical properties of objects, and maintains visible regions that are broader than single image. Meanwhile, existing works in 4D synthesis [8] mostly source the dynamics from video foundation models. Despite these models being able to synthesize visually appealing results with seemingly correct dynamics, they struggle to ensure physical plausibility, such as inconsistent appearance (e.g., geometry modified after occlusion or out-of-view) and infeasible deformation (e.g., limbs swapping during intersection). In Section 4.3, we show that extracting explicit 3D deformation from infeasible physics leads to floaters and broken geometry. These observations motivate us to develop new paradigm for creating 4D content,
"Corresponding author: Qi
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Deformable Scene with new motion
Video Input
Training View
Occlusion View
Let the horve nurn left curefully avoid the obstacle.
Naive Reconstruction
Restage4D
Sequence
Let the car drive to the edge of the rond, facing the eliff.
Sequence
Figure 1:
We show the input and output for the Restage4D in the left. Given base video and text prompt, we reanimate the scene according to the specified motion, while keeping the geometry consistency for occlusion part and motion coherent for the restaged 4D scene as shown in the right.
leveraging two promising information sources: the geometry and physical constraints distilled from a real-world video and the diverse pseudo-motion motion synthesized by video foundation models.
One intuitive way to build 4D representation is using single-video deformable 3D reconstruc- tion [9, 10], which achieves photorealistic results with temporally coherent and physically plausible deformations. Such methods commonly model dense 3D deformation using low-rank motion bases, then linearly combine via motion coefficients. These motion coefficients determine how the bases are weighted to produce the final deformation. Their similarity can be interpreted an indicator of how tightly coupled the two spatial points are in the object's 's motion. We observe that such design conceptually similar the skinning weights of articulated meshes [11]. In graphics workflow, the artists assign deformations to only a few key points, and per-vertex skinning weights map those deformations to the full mesh and drive the articulated meshes to perform desired motions. In this view, the input base video provides informed skinning weights in the real world, effectively turning the reconstructed object into puppet, whose strings can be manipulated by altering the low-rank motion components. This intrigues us to explore controlling the puppet with dynamics generated by a video diffusion model.
We propose Restage4D towards a new 4D restaging task to create video-conditioned 4D content. Given a monocular video, Restage4D aims to synthesize a new 4D motion sequence that maintains the original appearance while generating novel and physically plausible deformations. The restaged 4D sequence driven by a synthetic video generated by an image-conditioned video diffusion model based on the first frame of the base video. More specifically, the base video provides the shape and articulation model (i.e., the puppet to control), while the driving video supplies new dynamics (i.e., manipulate the puppet) and the appearance of the additional dis-occlusion revealed by novel motions.
The first challenge lies in sharing consistent shape and motion representation across both videos. We observe that the deformable 3D reconstruction frameworks are invariant to temporal direction, reconstructing from a video playing in reversed temporal order (i.e., rewind) would result in the same reconstruction. In combination with the assumption that our base video and driving video share the first frame, we propose video-rewinding joint-training scheme. We rewind the base video into the reversed temporal order, temporally concatenate with the driving video, and then jointly reconstruct both sequences as a single video. This enables smooth motion transition across videos when initializing deformable 3D reconstruction frameworks with smooth tracking, and allows intuitive sharing of motion coefficients across two videos. However, the simple joint optimization does not sufficiently leverage all the available information. As the synthetic video presents novel motions, certain geometries visible the input video become partially or even completely invisible throughout the synthetic video. Lacking visibility makes these geometries unable to receive gradients to preserve the smoothness and continuity of the appearance. To leverage the geometry recovered in the input video, we introduce an occlusion-aware rigidity regularization to preserve the local rigidity of the less visible geometry. addition, we use a disocclusion backtracing mechanism to recover missing canonical geometry by tracing observed points from the driving video back to the base video.

---

We validate our method on two datasets with monocular video settings and complicated motions, DAVIS [12] and PointOdyssey [13], well as several self-collected Internet videos to test the gener- alization. Our experimental results show that our Restage4D) can correct both infeasible motion and inconsistent geometries created by video diffusion models under complex deformations. Under rigor- ous ablation and physically- -aased metrics evaluation, we show that our method achieves consistent improvements across reconstruction quality, appearance consistency, and physical consistency.
The main contributions of this work are:
We introduce a new task, 4D restaging, which aims to reanimate deformable 4D scenes from video-driven motion.
We propose a geometry-preserving -preervinng pipeline that leverages a video-rewinding joint training scheme, combined with occlusion-aware ARAP and disocclusion backtracing.
We demonstrate how real-world supervision can correct artifacts from generative videos, bridging controllability and geometric fidelity in 4D reconstruction.
2
Related Work
Deformable 3D Neural Rendering. Neural rendering techniques based on NeRF [14, 15, 16, 17, 18, 19, 20), and Gaussian Splatting [21, 22] have demonstrated impressive performance in reconstructing high-fidelity static 3D scenes. Building upon this success, recent works have explored how to extend these representations to handle deformable 3D scenes with complex motion. A prominent line of work, including D-NeRF [23], Nerfies [24), and HyperNeRF [25], learns a time-conditioned deformation field via MLPs, enabling per-frame warping from canonical space. However, because of the implicit nature of NeRF, maintaining coherent motion and preserving geometry over time remains challenging. To address this, recent Gaussian Splatting-based approaches [26, 27, 28, 29] adopt similar deformation-field formulations to model temporal variation, using multi-view videos as input. In contrast, methods like Shape-of-Motion [9], Mosca [10] and DGS-LRM [30] explicitly represent motion via rigid-body transformations and shared motion bases by utilizing depth [31], tracking [32] and camera-pos [33] priors. Although occlusion remains a significant challenge for such approaches, the explicit nature of the representation enables finer-grained control over motion and facilitates geometric reasoning. However, existing monocular approaches suffer from preserving geometric information under occlusion introduced by the complex motion. Therefore, our work focuses on maintaining geometric realism and motion coherence during monocular video-conditioned 4D content creation, which requires strong regularization and guidance from real observations.
Motion Retargetting. Motion retargeting is fundamental problem in human-centric motion analysis and plays a central role in motion capture (MoCap) [34), scene animation [35], and motion transfer [36]. The goal of retargeting is to animate the static object using a reference input. Early works formulate retargeting as an optimization problem with kinematic constraints on articulationd body models [37], often solved via inverse kinematics (IK) [38, 39]. Recent approaches leverage deep learning by incorporating skeleton-aware modules [40, 41], enabling high-quality real-time motion retargetting in variety of subjects. Beyond human motion, retargeting deformable 3D scenes presents new challenges. Under the Gaussian Splatting framework, methods such as SC-GS [42] and D-MiSo [43] introduce keypoint- based control to transfer motion from one 4D scene another. Although effective in constrained scenarios, these methods rely on carefully designed control points and often require meticulous tuning to produce natural motion, especially when handling complex or long-range deformations. In contrast, our approach adopts an example-driven paradigm. Instead of designing explicit keypoint trajectories, we reanimate the scene by conditioning on a generated video sequence. This allows for retargeting complex and subtle motions directly from video while preserving geometric and temporal coherence.
4D generation. Recent advances in large-scale generative models such as Stable Diffusion [1], Flux [44], and Sora [45] have catalyzed significant progress in image, video and 3D generation. Natu- rally, this momentum has extended to 4D content generation, where the goal is to synthesize dynamic 3D scenes with time-varying geometry and appearance. One line of work directly uses generative models to create multiview observations or spatiotemporal sequences for the object condition on image or text, such as Zero-1-to-3 [46] and 4D-Diffusion [47]. Another line of research integrates synthetic supervision via score distillation sampling (SDS): methods like DrcamGaussian [48] and
3

---

Generation Stage
(5) Video- Rewinding Geometry Prosriing Join Training
Oreluiin-Awcrss Photometric Furen Stage
X Bowr Vikeo
Rowinded Box Video
Drivng
Belliing Comera Certee Concortio
Georselyy Presening
Stuge
Sooelaerr Fooi
User
Op
Video
Occlusio View
Deformable Soone
Training View
Figure 2: Pipeline of Restage4D. To perform 4D restaging task, base video and text prompt are utilized to generate driving video the generation stage. Then, in video-rewinding stage, the base video is back back-played and concatenate with the driving video to perform joint optimization. A geometry-preserving preserving rigidity loss applied in the motion initialization stage. Then, photometric fusion stage, occlusion-aware rigidity loss and a backtracing mechanism is incorporated.
4D-FY[8] use SDS to guide 3D or 4D reconstruction from text or reference images. Style-conditioned generation and editing also benefit from generative models, demonstrated by works such as Instruct- NeRF2NeRF [49] and DreamBooth3D [50], where image-generated priors are used to control object appearance, geometry, or identity across views. However, few works explore using video input for the objects' articulation and using text as motion source to generate realistic 4D scene. By using semantic prior in the video, our work tackles the problem of inconsistent appearance and motion in 4D restaging, helping to preserve physical plausibility and temporal coherence.
3 Method
With an input base video providing the context and text prompt providing the description of motion, our goal is to create a 4D scene in which the object follows the specified motion. Since we use monocular video as reference for the objects' motion articulation and appearance and a text prompt as the source of motion, it is very likely that the novel motion introduces occlusion and artifacts, leading wrong geometry. To have better geometry in the restaged 4D scene, the key to utilize the prior knowledge in the base video. Therefore, we use low-rank decomposed representation combined with geometry-preserviee training pipeline to obtain authentic 4D content.
3.1 Preliminaries: Low-rank Decomposed 4D Gaussian Splatting Representation
We adopt a motion representation similar to Shape-of-Morte [9] to model dynamic 4D Gaussian Splatting. Specifically, the scene is represented by two components: static background point set B (bo. bN,--,,,,,11 and a dynamic foreground point set X = { IN, where each point is associated with a 3D Gaussian (mean, scale, rotation, opacity and color).
To capture temporal motion, each foreground point æ, is associated with a canonical position μ4 E R shared time- -imeiindeeendent motion coefficient 33 E IR and set of time-varying motion bases M = (t) where each M₁ (t) E SE(3) is a smooth rigid body transformation over time.
The deformation of each foreground point at time f. is obtained by composing the motion bases weighted by its coefficients:
(t(t) 7(
3R Mt)))),
2x(() TT(t) 14
where the transformation is applied in SE(3), and (t) denotes the 3D position of point 2; at time t. An orthogonalization function 7 applied to ensure the legality T,. During training, the parameters of the Gaussians and the motion bases Mz(()) are optimized to match multi-view input observations. The training procedure Shape-of-Motion consists of two stages. In the motion initialization phase, pre-trained tracking model is applied to detect dynamic foreground points across frames. These

---

tracked 2D points are then unprojected into 3D to estimate a coarse scene flow, providing supervision for the initial motion. Based on this, the motion coefficients 34x and the time-varying motion bases (t) are initialized and optimized to best fit the scene flow. In the photometric fusion phase, the model is fine-tuned using a standard Gaussian splatting-style inverse rendering pipeline, which jointly refines motion, appearance, and structure using photometric losses and 2D tracking consistency.
3.2 Driving Video Generation & Rewinding
To achieve 4D scene creation with novel motion, we propose an example-driven pipeline that incorporates the deformable object's s geometry and motion articulation from the base video and the motion dynamics from the text prompt. As shown in the first stage of Figure 2, the input of the pipeline base video and motion prompt describing the desired motion. To create the dynamics from the text prompt to drive the object, instead of using some key points to simulate the proposed motion, we leverage an Image-to- vidVideo (I2V) diffusion model such as Sora [45] to synthesize a driving video for complex and plausible motion as the source of dynamics. To be specific, the first frame of the base video extracted as the input for the 12V models. Moreover, to obtain a stable source of dynamics, a smooth and steady camera motion in the driving video is desired. Therefore, another Vision Language Model (VLM) such as ChatGPT-40 is utilized to refine the text prompt, especially for explicit control of camera motion. After that, the refined text prompt, along with the extracted frame, is sent to the 12V models to generate a relatively high quality driving video.
However, driving video may introduce additional occlusion, disocclusion, and artifacts, To address these issues, we incorporate geometry and articulation priors from the base video as additional super- vision. This helps stabilize the reconstruction, preserve geometry, and improve motion consistency in the final 4D scene. A key insight is that the original and generated sequences typically share the same first frame, enabling us to connect them through a shared canonical scene. As such, we rewind the base video (i.e., play it backward), concatenate with the driving sequence, and perform joint optimization over the entire temporal span as shown in the second of Figure 2. After optimization, we can truncate the reconstructed scene to get the final result.
This joint training strategy brings in strong geometry and motion constraints from the original clip and allows them to propagate into the edited segment via shared motion coefficients. We formally state this motivation in the following lemma and defer the detailed proof to Appendix A.
Lemma 1. Let d(t) denote the distance between two 3D points I (t), (f) defined via shared motion coefficients and time-dependent bases, as in Shape of Motion. Suppose that the motion bases are trained jointly on an edited sequence [0, t1] (without supervision) and an original sequence [t,, te] (with supervision), with the temporal smoothness regularization applied.
Then, the variance of d((t) over [O, tı] is lower than if the model were trained on [0, tı] alone:
Varic(o,a. (d(t)) < 0o
where do is the variance from training without supervision.
3.3 Occlusion-Aware Rigidity Regularization
During the Motion Initialization Stage, we achieve geometry preservation by effectively propagating geometry supervision throughout the sequence to achieve better initialization. Following the shape of motion [9], we use a low-rank decomposition of the motion bases (t) and the motion coefficients B; to fit the estimated flow of the scene. This scene flow is often noisy, especially in the driving video due to occlusions or motion artifacts. Inspired by physical-
regularization such a global Rigidity Loss to regularize the deformation. Specifically, we first construct a k-NN graph Ω = {(I,, among foreground Gaussians, and define the loss
as As- As-Rigid-Ass Srgiid-As-Psssiiee As-Possible (ARAP) [51],
T
L Rrggidity init
Dzyg siy (1ldd(a((0,=;;(1))) - d(ps, 15)l
t=1
Here, we use the frame with the most visible points as canonical frame. /4s is the canonical position of the point 24. Siz measures motion similarity based on the cosine similarity of motion coefficients
5

---

3 3 |311111
8ij
Once the motion bases and coefficients are reasonably initialized, the object's appearance will be refined during the Photometric Fusion Stage. The photometric and tracking losses are used to further optimize the Gaussian parameters. To effectively propagate geometry supervision from the base sequence to occluded and inconsistent regions the driving sequence, while avoid oversmoothing visible regions, adapt the Rigidity Loss to this occlusion-aware scenario. The intuition is that the occluded regions in the driving video should correspond to visible regions in the base sequence under rigid transformation, while the visible part should rely more on the photometric information.
We first estimate the invisibility score (t) E [o, 1] for each point (t))), based on its depth difference from the rendered depth buffer (t). The depth buffer is computed similar to color rendering as:
1-1
D(t) = Σ di(t)a;Ti, where T; = III o;) EN
j=1
where dt(t) is the depth of each splat (t) from the camera. We then define a smooth step function over the depth difference 8(t) di (t) D(t) to compute invisibility
i To
(t)
3
Ti 8-70 )
-
5- 3
<
(5)
2
where to and Ti a pre-defined boundary between visible and invisible part. This formulation allows a soft occlusion mask to guide the selective regularization of rigidity.
Finally, the occlusion-aware Rigidity loss is applied during refinement
Regiditt refine
E (t)))) ((),))))))) d(u4, d(u4,1y)l2 py)
(6)
This encourages local rigidity only in regions that are not directly supervised, improving geometry propagation in occluded areas without compromising visible-region quality, We observe that overly strong rigidity constraints may encourage motion coefficient sparsity and suppress uighffreqeeccy motion. To address this, we increase the number of motion bases to 50-100 in our implementation, which empirically balances rigidity and expressiveness.
3.4 Disocclusion Backtracing
While the Rigidity Loss during initialization helps align the edited region motion with the original sequence, we observe that newly disoccluded parts that appear only in the driving video hard to recover, since they were never visible in the base video and thus absent from the canonical frame.
To address this, propose disocclusion backtracing mechanism to recover such missing geometry by supplementing the canonical representation after initialization. We examine the visibility (t()) of each tracked point p5 (t), using the output of standard tracking models such as TAP-AIR [32] or Bootstap [52]. A point is considered disoccluded at time t E Tedu it becomes visible in the edited video but was occluded D in the canonical frame:
uy A -U (teano)
(t())= E D(t)
For each such disoccluded point, we insert new Gaussian splat (t( at its 3D location and assign its color directly from the input. We then interpolate its motion coefficients & from nearby Gaussians and backtrace the point to the canonical frame by using the inverting of Equation 1:
4 = 8y (a)(t)--a(f1 Vp-(t) D(t)
(8)
This complements the occlusion-aware rigidity regularization by explicitly recovering disocclusion, allowing both occluded and disoccluded regions to be consistently aligned in the canonical space.

---

Input Video
Deformable 3D Scene with Specified Motion
Sequence
The elephant walky away."
The elephant tarms left and wulls the nec."
The clephaus torns face fowardd us."
Sequence
The con is grating
The cow valks to the wendows,
The can walks backoards.
Figure 3: Samples of 4D motion creation. For the same input base video, create different motion variation based on the input text prompt, while preserving visual context in the base video.
Experiments
We evaluate our pipeline both curated and in-the-win wild video datasets, including DAVIS [12] and PointOdyssey [13]. For 4D restaging evaluation, we use 20 video sequences from the DAVIS dataset. Following the pipeline described in the previous section, we apply text-driven video generation to reanimate the input videos and reconstruct them using our Soometry-preserving evaluate geometric consistency, we further test our method on the PointOdyssey dataset. We apply our full pipeline to reconstruct dynamic scenes and compare the resulting 3D trajectories with the ground-truth object tracking annotations, demonstrating our method's ability preserve accurate geometry. The DAVIS sequences used in our experiments will be released to facilitate future research. All experiments are conducted on single NVIDIA A100-SXM4 with 15GB GPU RAM. Each sequence typically takes 40 minutes for a sequence with 100 frames on 500 training epochs.
framework. To
4.1 Reanimation and Geometry Preserving
We evaluate our pipeline in the DAVIS data set to demonstrate its ability to preserve geometry in the 4D reconstruction task. Given an input video and a text prompt describing a new motion, we use a pre-trained video generation model (e.g.. Sora) to reanimate the object and then reconstruct the 4D scene using our pipeline. As shown Figure 3, our method can generate a variation of motion based on a different text prompt. The restaged video presents novel motion while keeping the context of the input scene. Please refer to the supplementary for the input sequence and motion prompts.
A key challenge in 4D restaging is occlusion, as parts of the object may become invisible due to the new motion trajectory. Figure shows the visualization for the restaged 4D scene in different views and time steps. Significant improvement in quality can be observed for occluded regions.
To assess geometry consistency, we introduce three metries: (1) CLIP [53]-based object-centered view (OCV) consistency, (2) volume consistency, and (3) edge length stability. Existing 3D quality metrics such as PointSSIM [54] and GraphSSIM [55] are not suitable for our setting, they are sensitive to spatial deformation and assume static correspondence. Since our pipeline explicitly models deformation, such frame-level metrics fail capture global geometry stability. For OCV CLIP, we render each scene from fixed relative viewpoint to the object (e.g., behind or side) and compute the CLIP similarity to clean reference frame. This measures whether the occluded geometry is preserved and visually plausible. Volume consistency C, is computed as:
C =
loog
Σ (v, (v-- (v-
1.5
[Tariving
tETomin
where V is the volume of the foreground voxelized at time f, and V is the mean volume over the driving clip. Higher values indicate lower variance and stronger volume conservation. Similarly, we measure the frame-to-frane consistency of local geometry by sampling 1000 Gaussians and tracking the standard deviation of their edge lengths to neighbors over time. Our proposed metries better reflect the consistency of deformable 3D reconstructions under the 4D restaging task.
7

---

Deformable 3D Scene
Rendered Foreground From Occlusion View
Timestep
Timestep
Timestop 3
Recomstruction
The train backs off along the raibway."
Restage4D
Noive Reconstruction
Restoge4D
The black swan is turning around in the water."
Figure Samples of geometry preserving for occlusion view. For each restaged 4D scene, we visualize the foreground observing from the occlusion view at different time step. The geometry of object can better preserved using our method, compared to naive reconstruction.
Tracking
Visuolization
Table 4: Quantitative measure on PointOdessy Dataset. Restage4D can achieve higher tracking quality with geometry-preserving strategy.
Sprce
Graurd Truffs
Model
3D Tracking Loss
Space
All
0.05
w/o Backtracing
0.06
Helve
Botage4D
Badage4D
w/o Rigidity
0.09
Figure 5: Visualization on PointOdessy dataset.
Table I summarizes the results. These results reflect our core motivation for designing the occlusion- aware ARAP. While the naive ARAP improves geometric consistency metrics (Vol./Edge Consistency) the occluded part, applied strong rigidity constraint globally, which significantly degrades the reconstruction quality (PSNR drops by 5.24). This is especially problematic in regions with high-frequency motion, where excessive regularization results visually unnatural and frozen movements, leading to incorrect motions. Through empirical observations, Enforcing ARAP uniformly helps preserve occluded geometry, but overly constrains visible regions and suppresses natural motion. (2) Reducing regularization improves visible motion, but sacrifices occluded region stability.
found trade-off: (I)
By providing a base video, we enable the model to transfer accurate articulations and geometry. Combined with selective ARAP regularization, our approach optimizes the trade-off between ge- ometric consistency and motion fidelity. Our method achieves the highest PSNR the training views. By leveraging priors from the base video, it also achieves strong consistency scores in terms of CLIP-score and occlusion geometry, demonstrating its ability to maintain perceptual coherence.
In Table 2, we also compare our model with other 4D reconstruction baselines on generated video condition on videos from DAVIS [12]. It's hard to get ground truth for novel view in monocular video, so we conduct additional evaluation using the Neural3DV [56], which provides 20 calibrated camera views of dynamic scenes, from which we can derive ground truth for novel views. Please refer to the appendix for construction details. We integrate other image-to-video[[2) models into our pipeline, and it works on multiple models. In Table 3, we evaluated Caus Vid [57] and LTX- Video [58] for video generation and compared them with Sora. We applied our restaging pipeline two sequences: cat-jump and cows-graze, using each 12V model the video generation component. These results demonstrate that our Restage4D framework performs effectively across diverse I2V backbones, with consistent reconstruction quality and visual plausibility.
video(12V)

---

Table 1: Ablation study on 20 DAVIS sequences. We report the mean + std of PSNR, SSIM and LPIPS for foregrounds, as well as CLIP similarity for occlusion views (OCV), and geometry consistency metrics. Improvements (A) are computed per-sequence relative to the Baseline. We also evaluated the effect of using naive ARAP and occlusion-aware ARAP (o-ARAP).
Method
PSNR1
SSIM
I.PPS.
OCV CLIP 1
Vol. Consist
Edge Consist
Baveline
26.71 1: 0.9765 1 0.004
4.91
3,80
3.01
w/o Bose Viden
Naive ARAP
23.67 + 1.78
0.9581 = 0.007
0.1733 + 0.091
84.78 + 4.39
9.14 + 3,31
15.91 + 3.84
A. from Baseline
-3.04
-0.0175
+0.0739
+3.1
+1.73
+3.25
-0-ARAP
26.80 +: 2.26
0,9812 + 0.006
0.0754 + 0,044
82.65 + 4.29
7.92 d: 3,97 +0.51
13.64 3.38
A from Baseline
+0.19
+0.0056
-0.024
+0.97
+0,98
w/ Base Video
w/a ARAP
26.58 1: 2.21
0,9614 1 0.004
0.1429 1 0.063
82.29 1: 3.58
8.50 3.81
13:24 3.54
A from Baseline
-0.13
-0.0152
+0.0435
+0.61
+1.09
+0.58
ARAP
21.47 +
0.9094 + 0.005
0.083 0.1917
87.31 3,05
11.43 4.27
17.30 3,95
À. from Baseline
-5.24
-0.0671
+0.0923
+5.63
+4,02
+4,64
O-ARAP
27.12 + 2.27
0.9832 = 0.006
0.0632 0.042
85.40 = 4.11
10.32 = 4.23
16.28 3,10
À from Baseline
+0.41
+0.0067
-0.0362
+3.72
+2.91
+3.62
Table 2: Quantitative measure on PointOdessy Dataset. Restage4D can achieve higher tracking quality with eeometry-preserviee strategy.
Table 3: Reconstruction PSNR using different video generation models. Restage4D can create content of similar quality using different models.
Method
DAVIS [12]
Neural3DV [56]
12V Model
cat-jump
cows-graze
Vidu4D [59]
19.77
14.74
Sora [59]
26.74
26.25
SC-GS [42]
22.69
16.71
Caus Vid [42]
23.68
26.45
Ours
27.38
17.78
I.T--video
25.08
28.05
4.2 3D Tracking
The advantages of geometry preservation are also demonstrated in 3D tracking. We evaluate our pipeline on DAVIS and PointOdyssey to show that our occlusion-awand rigidity constraint improves the tracking quality of occluded regions, As shown in the left part of Figure 5, our method can faithfully track points that become occluded during motion, well as obtain smoother tracking trajectories, while the baseline without rigidity loss fails to maintain consistent trajectories and geometry for those regions. PointOdyssey provides ground-truth 3D point tracks, allowing to quantitatively evaluate tracking accuracy. We compute the I.1 error between the predicted and ground-trutt 3D trajectories. As shown in Table 4, incorporating the occlusion-awaad rigidity loss and back-trac tracing mechanism leads to lower tracking error, particularly in previously occluded areas, confirming its effectiveness in maintaining geometric consistency over time. We further visualize the learned motion coefficients in the right part of Figure 5. With loss of occlusion-sensitive rigidity, the coefficients become sparser and exhibit smoother spatial transitions. Occluded regions are encouraged to share similar motion coefficients with their neighbors, which helps to enforce local rigidity and preserve geometry through occlusion. This in turn improves the stability of 3D trajectories over time, leading to better tracking quality for both visible and invisible parts of the object.
4.3 Automatic Geometry Correction
Since our motion editing pipeline relies on videos generated by image-to-video models, it common for these synthesized sequences to contain incorrect geometry or motion artifacts. This highlights an additional advantage of incorporating the original video as an auxiliary input during training: by using accurate observations as supervision, the model can automatically correct geometric and motion-related errors present the generated video. As shown in Figure 6, our method successfully fixes several failure cases introduced by the video generation model. In the first example, the model mistakenly attaches a road sign to the bus, corrupting the appearance of the foreground. In the second case, part of the background is erroneously fused to the camel's 's hump, and the motion of its legs is incorrectly changed. Through joint training with the original sequence, Restage4D is able to correct these inconsistencies, recovering plausible appearance and producing more realistic deformation,

---

Correct Geometry in Base Video
Driving Video
Restaged Output
Motion Artifacts in Driving Video
Reasonable Restaged Output
Figure 6: Samples geometry and motion correction. Conditioning on the base video, Restage4D can use the learned articulation to correct some appearance and motion artifacts in the driving video.
even from flawed motion inputs. These results demonstrate that leveraging accurate prior observations from the original sequence not only improves geometric consistency, but also enables automatic correction of hallucinated artifacts and implausible motion, bridging the gap between generative and physically faithful reconstructions.
Conclusion and Limitation
In this work, introduce 4D restaging, aims to reanimate deformable 3D reconstructions from a single real-world video. By combining a pre-trained video generation model with our geometry--reserving video-rewinding training pipeline, Restage4D enables the transfer of motion dynamics from synthetic video while preserving the physical plausibility and geometric fidelity of the original scene. Our approach leverages the input video source of shape and articulation priors, allowing us to generate high-quality 4D scenes that remain consistent even in occluded or disoccluded regions.
video-conditioned 4D content creation task that
However, our method has limitations. the video generation model produces sequences with severe geometric artifacts unrealistic motions, the resulting supervision may not be sufficient recover accurate 4D geometry. Additionally, while we leverage geometry supervision from real videos, assume the input object remains temporally consistent across sequences, which may not hold for highly deformable or textureless objects. Our framework involves generative video models, which may pose risks when applied to human-centric content, potentially leading the synthesis of misleading or harmful videos. One possible solution is incorporating prompt filtering mechanisms during the prompt refinement stage detect and block inappropriate or sensitive prompts.
We hope that our work offers insights into how worldw video properties (e.g., geometry priors, articulation, visibility) can be used to stabilize and guide 4D content creation, bridging generative modeling with reconstruction-driven pipelines.
Acknowledgement
This work supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOI/IBC) contract number 140D0423C0074, The reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should be interpreted as necessarily representing the official policies or endorsements, either expressed implied, of IARPA, DOI/BB,, or the U.S. Government. This work is also supported in part by Sony Faculty Award.
U.S. Government is authorized
10

---

References
[1] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High- riightresolution image synthesis with latent diffusion models," in Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, 2022, pp. 696 1,3
[2] I. J. Goodfellow, J. Pouget-Abadic, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and
Y. Bengio, "Generative adversarial nets," Advances in neural information processing systems, vol. 27, 2014.
[3] U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang. O. Ashual, O. Gafni et al., "Make-a-video: Text-to-video generation without text-video data," arXiv preprint wxXiv22209.4792, 2022.
[4] S. Li, Ci W. Zhu, B. Yu, Y. Zhao, C. Wan, H. You, H. Shi, and Y. Lin, "Instant-3d: Instant neural radiance field training towards on-device ar/vr 3d reconstruction," in Proceedings of the 50th Annual International Symposium on Computer Architecture, 2023, pp. 1-13.
[5] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, "Dreamfusion: Text-to-3d using 2d diffusion," arXiv
preprint arXiv:2209. 14988, 2022. 1
[6] U. Singer, S. Sheynin, A. Polyak, O. Ashual, I. Makarov, F. Kokkinos, N. Goyal, A. Vedaldi, D. Parikh,
J. Johnson et al., "Text-to-4d dynamic scene generation," arXiv preprint arXiv:2301. /280, 2023.
[7] Y. Zhao, Z. Yan, F. Xie, L. Hong, Z. Li, and G. H. Lee, "Animate 124: Animating one image to 4d dynamic scene," arXiv preprint arXiv:2311. 14600033,, 2023. 1
[8] S. Bahmani, I. Skorokhodov,, V. Rong, G. Wetzstein, L. Guibas, P. Wonka, S. Tulyakov, J. J. Park,
A. Tagliasacchi, and D. B. Lindell, "4d-fy: Text-to-4d generation using hybrid score distillation sampling," " in Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 7996-800 8006. 1,4
[9] Q Wang. V. Ye, H. Gao. J. Austin, Z. Li, and A. Kanazawa, "Shape of motion: 4d reconstruction from single video," arXiv preprint arXiv:2407. 137644 2024. 2, 5,4,
[10] J. Lei, Y. Weng, A. Harley, L. Guibas, and K. Danillidis, "Mosca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds," arXiv preprint 2rxiv:2405.17421, 2024. 2,
[11] D. Vlasic, I. Baran, W. Matusik, and J. Popavić, "Articulated mesh animation from multi-view silhouettes," "
in Acm Siggraph 2008 popers, 2008, pp. 1-9. 2
[12] F. Perazzi, J. mostt, B. Mc McWilliams, L.. Van Gool, M. Gross, and A. Sorkine- Hornung, "A benchmark dataset and evaluation methodology for video object segmentation." in Proceedings of the IEEE conference
computer vision and paltern recognition, 2016, pp. 724-732. 3. 7, 8,9
[13] Y. Zheng. A. W. Harley, B. Shen, G. Wetzstein, and L. J. Guibas, "Pointodyssey: A large-scale synthetic dataset for long-term point tracking," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 19 855 19 865. 3,7
[14] B. Mildenhall, P. P. Srinivasan, M. Tancik. J. T. Barron, Ramamoorthi, and R. Ng. "Nerf: Representing scenes as neural radiance fields for view synthesis," Communications of the ACM, vol. 65, no. 1, pp.
99-106, 2021. 3
[15] A. Chen, Z. Xu, A. Geiger, Yu, and H. Su, "Tensorf: Tensorial radiance fields." in European conference on computer vision. Springer, 2022, pp. 333-350. 3
[16] T. Müller, A. Evans, C. Schied, and A. Keller, "Instant neural graphics primitives with a multiresolution
hash encoding." ACM transactions on graphics (TOG), vol. 41, no. 4. pp. 1-15, 2022. 3
[17] R. Martin-Brualla N. Radwan, M. S. Sajjadi. J. T. Barron, A. Dosovitskiy, and D. Duckworth, "Nerf in the wild: Neural radiance fields for unconstrained photo collections." " in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021, pp. 7210-7219. 3
[18] S. Fridovich-Keil A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa, "Plenoxels: Radiance fields without neural networks." in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2022, pp. 5501 -5110.
[19] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman. R. Martin- Brualla, and P. P. Srinivasan, "Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields." in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 5855-5864,
[20] S. Fridovich-Keil, G. Mcanti, F. R. Warburg, B. Recht, and A. Kanazawa, "K-planes: Explicit radiance fields in space, time, and appearance." in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023, pp. 12479-1 1799-12444 18888.
[21] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, "3d gaussian splatting for real-time radiance field rendering." ACM Trans. Graph., vol. 42, no. 4, pp. 139 1, 2023. 3
11

---

[22] J. You, C. H. Lin, W. Lyu, Z. Zhang. and M.-HI Yang. "Instainpaint: Instant 3d-scene inpainting with masked large reconstruction model," in Adv. Neural Inform. Process. Syst., 2025. 3
[23] A. Pumarola, E. Corona, Pons-Moll, and Moreno-Noguee "D-nerf: Neural radiance fields for dynamic scenes," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 10:31--103277 3
[24] K. Park, U. Sinha, T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and Martin- Martin-Brualla, "Nerfies Deformable neural radiance fields," in Proceedings of the IEEE/CVF international conference on computer vision, 2021. pp. 5865-5874. 3
[25] K. Park, U. Sinha, P. Hedman, J. T. Barron, S. Bouaziz, D. B. Goldman, R. Martin-Brualla, and S. M. Seitz, "Hypemerf: higher-dimensional representation for topologically varying neural radiance fields," arXiv preprint ar arXiv:2 1.328,, 2021. 3
[26] Z. Yang, X. Gao, W. Zhou, S. Jiao, Y. Zhang, and X. Jin, "Deformable 3d gaussians for high-fidelity monocular dynamic seene reconstruction," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 20 20131-20 341.
[27] Z. Lu, X. Guo, L. Hui, T. Chen, M. Yang. X. Tang, F. Zhu, and Y. Dai, "3d geometry-aware deformable gaussian splatting for dynamic view synthesis." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, 2024, pp.
3
[28] G. Wu, T. Yi, J. Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian, and X. Wang. "4d gaussian splatting for real-time dynamic scene rendering," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 20 310-20 320. 3
[29] H. Jung, N. Brasch, J. Song, F. Perez-Pellitero, Y. Zhou, L., N. Navab, and B. Busam, "Deformahle 3d gaussian splatting for animatable human avatars," arXiv preprint arXiv:23 15059, 2023. 3
[30] C. H. Lin, Z. Lv, S. Wu, Z. Xu, T. Sguyen-Phuooe H.-Y. Tseng, J. Straub, N. Khan, L. Xiao, M.-H. Yang et al., "Dgs-Irm Real-time deformable 3d gaussian reconstruction from monocular videos," in Adv. Neural Inform. Process. Syst., 2025. 3
[31] L. Yang. B. Kang. Z. Huang. X. Xu, J. Feng, and H. Zhao, "Depth anything: Unleashing the power of large-scale unlabeled data," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, 2024, pp. 10.371-10381.1 3
[32] C. Doersch, Y. Yang, M. Vecerik, D. Gokay, A. Gupta, Y. Aytar, J. Carreira, and A. Zisserman, "Tapir: Tracking any point with per-frame initialization and temporal refinement," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 1061-10072. 3, 6
[33] Z. Teed and J. Deng, "Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras," Advances in neurul information processing systems, vol. 34, pp. 16558-16 1669, 2021.
[34] J. Deutscher, A. Blake, and I. Reid, "Articulated body motion capture by annealed particle filtering." " Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.
in Proceedings IEEE Conference PR00662), vol. 2. IEEE, 2000. pp. 126-133. 3
[35] Y. Horry, K.-I. Anjyo, and K. Arai, "Tour into the picture: using a spidery mesh interfuce to make animation from a single image," in Proceedings of the 24th annual conference on Computer graphics and interactive techniques, 1997, pp. 225-232. 3
[36] C. Farhat, M. Lesoinne, and Le Tallec, "Load and motion transfer algorithms for fluid/structure inter- action problems with non matching discrete interfaces: Momentum and energy conservation, optimal discretization and application to aernelasticity," Computer methods in applied mechanics and engineering, vol. 157,, no. 1-2, pp. 95-114, 1998. 3
[37] M. Gleicher, "Retargetting motion to new characters," in Proceedings the 25th annual conference on Computer graphics and interactive techniques, 1998, pp. 33-42. 3
[38] J. Lee. Chai, P. S. Reitsma, J. K. Hodgins, and N. S. Pollard, "Interactive control of avatars animated with human motion data," in i Proceedings of the 29th annual conference on Computer graphics and interactive techniques, 2002, pp. 491-500. 3
[39] R. W. Sumner and J. Popović. "Deformation transfer for triangle meshes," ACM Transactions on graphics (TOG), vol. 23, no. 3. pp. 399-405, 2004. 3
[40] K. Aberman, Li, D. Lischinski, O. Sorkine-Horunng,
D. Cohen-Or, and B. Chen, "Skeleton-aware networks for deep motion retargeting." " ACM Transactions on Graphics (TOG), vol. 39, no. 4, pp. 62-1, 2020. 3
[41] J. Zhang, Z. Tu, J. Weng, J. Yuan, and B. Du, "A modular neural motion retargeting system decoupling skeleton and shape perception," " IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[42] Y.H. Huang. Y.-T. Sun, Z. Yang, X. Lyu, Y.P. Cao, and X. Qi, "Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 4230.4230. 3,9
12

---

[43] J. Waczynska, P. Borycki, J. Kaleta, S. Tadeja, and P. Spurek, "D-miso: Editing dynamic 3d scenes using multi-gaussians soup." Advances in Neural Information Processing Systems, vol. 37, pp. 107 865-107 889,
2024. 3
[44] B. F. Labs, "Flux," https://ggthubb.combbaaakkka labs//tx,, 2024. 3
[45] T. Brooks, B. Peebles, Holmes, W. DePue, and Y. Guo, "Video generation models as world simulators." cono//ndee/viidoo-eeeeaattoooo generation-models as- wor--woordddiieee simulatossttrrs// 2024, openAl Technical Report.
3,5
[46] R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick, --roo-Itoo3: sero-shot one image to 3d object," in Proceedings of the IEEE/CVF international conference on computer vision, 2023,
pp. 9298-9309. 3
[47] H. Liang, Y. Yin, D. Xu, H. Liang, 7. Wang, K. N. Plataniotis, Y. Zhao, and Y. Wei, "Diffusion4d: Fast spatial-temporal consistent 4d generation via video diffusion models," arXiv preprint arXiv:2405. 16645, 2024. 3
[48] J. Tang, Ren, H. Zhou, Z. Liu, and G. Zeng, "Dreamgaussian: Generative gaussian splatting for efficient content creation," " arXiv preprint arXiv:2309. 16653, 2023. 3
[49] A. Haque, M. Tancik, A. A. Efros, A. Holynski, and A. Kanazawa, "Instruct-ner(nnerf. Editing 3d scenes with instructions." in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023,
pp. 19 19400-9770.
[50] A. Raj. S. Kaza, B. Poole. M. Niemeyer, N. Ruiz, B. Mildenhall, S. Zada, K. Aberman, M. Rubinstein. J. Barron et al., "Dreambooth3d: Subject-driven text-to-3d generation." in Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 3349-2359. 4
[51] T. Igarashi, T. Moscovich, and J. F. Hughes, ses-riiddaa-possiiie shape manipulation." ACM transactions Graphics (TOG), vol. 24, no. 3, pp. 1134 1141, 2005. 5
[52] C. Doersch, Lue, Y. Yang, D. Gokay, S. Koppula, A. Gupta, J. Heyward, I. Rocco, R. Goroshin, J. Carreira et al., "Bootstap: Bootstrapped training for tracking-any-point"" in Proceedings of the Asian Conference Computer Vision, 2024, pp. 3257-3274. 6
[53] A. Radford, W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin.
J. Clark et al., "Learning transferable visual models from natural language supervision," in International conference machine learning. PmLR, 2021, pp. 8748-8763. 7
[54] D. Lazzarotto and T. Ebrahimi. "Towards a multiscale point cloud structural similarity metric," 2023 IEEE 25th International Workshop on Multimedia Signal Processing (MMSP), 2023, pp. 1-6. 7
[55] Q. Yang, Z. Ma. Y. Xu, Z. Li, and J. Sun, "Inferring point cloud quality via graph similarity." IEEE Transactions on Puttern Analysis and Machine Intelligence, vol. 44, no. pp. 3015-3029, 2022.
[56] T. Li, M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele. R. Newcombe et al., "Neural 3d video synthesis from multi-view video." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 5521-5531111 8,9
[57] T. Yin, Q Zhang, R. Zhang, W. T. Freeman, F. Durand, E. Shechtman, and X. Huang, "From slow bidircctional to fast autoregressive video diffusion models," in /EEE Conf. Comput. Vis. Pattern Recog., 2025. 8
[58] Y. HaCohen, N. Chiprut, B. Brazowski, D. Shalem, D. Moshe, E. Richardson, E. Levin, G. Shiran, N. Zabari, O. Gordon, P. Panet, S. Weissbuch, V. Kulikov, Y. Bitterman, Z. Melumian, and O. Bibi, "Lix-viden: Realtime video latent diffusion." arXiv preprint arXiv:2501.0011 000100, 2024.
[59] Y. Wang, X. Wang, Z. Chen, Z. Wang, F. Sun, and J. Zhu, "Vidu4d: Single generated video to high-fidelity 4d reconstruction with dynamic gaussian surfels." " in Ady. Neural Inform. Process. Syst., 2024. 9
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We proposed the 4D restaging task, aiming at reanimating the 3D reconstruc- tion using single video input. We conducted several experiments prove motivation using video information to improve the restaging quality. We want to explore the pos- sibility of cideoocondiioneed 4D content creation, and provide insights of learning motion articulation from real-world video and use them for down-stream tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We used Lemma for the main motivation, see Lemma 2. The corresponding proof is in the supplementary.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We listed the training details in Section 4. For the training methods we use, we listed the algorithm in Section 3. For all the metrics we used, we provide their calculation formulation in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
15

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release code and data we use after the double-blinding review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
public/guide Salelee/oooSSuumissS/Se oonooocy)))) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: // /nips..c. cc/ wwpplic/gguideeeeeeeeeeeeppbeeee httpg//dddd//eeedummmiiiieee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See the experiment table.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: stttps:/neurippl: wwtprrips.ccppupupee cctppblic//eceeccccccecccccc httpaa////hissciideeeeeeeeee
Answer: [Yes]
Justification: Our research focuses on reanimating 4D content from single video input using publicly available datasets (e.g., DAVIS, PointOdyssey) and pre-trained generative models. The videos used do not contain personally identifiable information or sensitive content. We do not involve human subjects or deploy the model in way that affects people or communities. The purpose of our work is academic, and we have taken care to avoid generating or reinforcing any harmful biases. Thus, we believe our research adheres to NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
17

---

Justification:
We discuss the societal impact of our work in the supplementary materials.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimesss considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve the release of any high-risk models, such as large- scale language models, generative adversarial networks for face synthesis, or web-scraped datasets that may contain unsafe content. The data used in our research is either publicly available or synthetically generated using existing safe models. As such, we believe no special safeguards are necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have properly cited DAVIS and PointOdessy.
Guidelines:
The answer NA means that the paper does not existing assets.
18

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswithcode. thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We plan release the full implementation of our Restage4D pipeline, along with example input videos and the reconstructed 4D scenes for academic use. Documenta- will provided alongside the code and data to describe usage instructions, training configuration, data formats, and licensing. All released assets are either synthetic or derived from publicly available datasets that do not involve personal or sensitive information. Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
19

---

Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Answer: [Yes]
Justification: In the pipeline used LVM for prompt refinement, and described the usage and showed its input and output in the supplementary.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: http://eeurp.p..ee wcc.onnfeeeeccce/22255555555555/ for what should or should not be described.
A Proof of Lemma 2
We introduce Lemma to demonstrate the necessity of using base video as additional input, which is
Lemma 2. Let x A(t) (t) R be the positions two 3D points time E [0, T), defined by shared time-iine ineeinndeeendnt motion coefficient vector 3 E IR and a set of time-dependent, spatially smooth motion bases {t)
Assume the following:
(i) The scene is trained over two segments: an edited clip [0, 4i] without supervision, and an original clip [4, 12] with ground-truth geometry supervision;
(ii) A temporal smoothness regularization is applied on the motion bases M2 i.e. Exe [| + M=
(iii) Each point's 'ss position at time t given by a transformation composed from the bases and coefficients:
x(1)
Bx MM(t)(zo).
where Do is the canonical position.
Then, the variance of the pairwise distance d(t) = (()= xt(())] within the edited clip satisfies:
02 == Varrejo.t.)(d(t)) < do.
where dn is the variance obtained by training only on the edited clip without supervision.
We prove this lemma using
20

---

Proof. Let - TB # E R be fixed canonical positions of two points. At time t. their positions in world coordinates are given by:
zi(1) Σ 3k (x)t) for {A,
where B E RK shared and fixed, and (t) R + R are time-varying motion basis functions.
Define the pairwise squared distance:
(())= - (1)==1oa(0-an(((1
- =n(0 = Σ BR ( -M00000002M00000))))
Let (+) = (x(()=2) Mx(()(2)) E TR. then:
(C)
Σ 3x (1)
This is quadratic form in the temporal functions A& (t), linearly combined by fixed weights Bg. Its vime-nariance depends on the temporal variability of (t)...
Now, consider the regularized training setup:
(i) The motion bases (M&(!) are supervised only in [4, [2]. enforcing accurate deformation there;
i11) A temporal smoothness regularization is imposed:
K
smooth Σ Σ
Due this regularization, each M&(t) evolves smoothly over time. temporal second difference of A (t) as:
Let denote the discrete
(t) -)
Then, smoothness implies that [t)ll is small, especially near the boundary = f1, where (t) is influenced by supervision at > f1- Since the functions A₂ (t) more constrained in this case (compared to unregularized training), their fluctuations in [0, fi] are suppressed.
Let us define:
f(t) = Σ As(t), then (t) = 11
The temporal variance of d(t) satisfies:
(arre(o.,,((())) = Var
Since f(t) is fixed linear combination of smoother functions (t)) its temporal fluctuation is reduced by the regularization. That is:
(l))
as Csmooth 4
In contrast, training |0, f1] alone without regularization leads to unconstrained Mt)t), and thus larger temporal variability in f(t). Therefore:
02 Var,cjo.,, (d(t)) < do.
where op is the variance from training without supervision or smoothness.
21

---

Tracking 3D Loss Ridiily Loss
1930
330
0.00
0.20
TroeaggD
Figure 7: Effect of changing the number of bases.
B Training Details
B.1 2D Priors
To enable 4D reconstruction from monocular video, we leverage a set of 2D foundation models to provide reliable initialization. Following the pipeline of Shape-of-Mottee for foreground segmentation and Track Anything to propagate the masks across frames. For camera motion and point correspondence, we adopt Tapir for 2D point tracking and MegaSAM for estimating camera poses. Additionally, we incorporate dense depth maps from VideolDeptAAnything to provide richer geometric details for the scene.
we use Segment Anything
B.2 Hyperparameters
To achieve found essential carefully tune the rigidity loss. Since the overall motion primarily determined during the motion initialization stage, we evaluate the effect of different combinations of rigidity loss weights and the number of motion bases.
reconstruction while maintaining high-frequency motion details, we
As shown in Figure 7, each curve represents the trade-off between object rigidity and tracking accuracy for given number of motion bases, vary the rigidity loss weight from 0 to 10-1, Our results show that increasing the number motion bases can improve this trade-off curve, and that a loss weight of 10-3 offers good balance. Based this analysis, use 100 motion bases and set the rigidity loss weight to 10 30 in all experiments.
C Samples from DAVIS Dataset
To qualitatively evaluate our 4D restaging pipeline, we present several representative samples from the DAVIS dataset in Figure 8. For each sequence, we show: (1) two keyframes from the base video (original monocular input): (2) the corresponding motion prompt describing the desired new motion; and (3) two keyframes from the generated driving video, synthesized using pre-trained image-to-vides model (e.g., Sora) conditioned on the prompt and the first frame. These examples demonstrate the diversity of motion prompts supported by our pipeline, including changes in direction, speed, and posture. They also highlight the realism and coherence of the generated driving video, which serves the motion source for our 4D reconstruction. For each example, visualize the base and drive videos using two keyframes (top and bottom) to illustrate the overall temporal dynamics.
D Evaluation on Neural 3D Video Synthesis dataset
Evaluating the novel view by comparing with ground truth is important for validating view general- ization in monocular reconstruction, but the groundtruth is hard to get. To address this, we conduct additional evaluation using the Neural 3D Video Synthesis dataset, which provides 20 calibrated camera views of dynamic scenes, from which we can derive ground truth for novel views We adopt the following protocols:
To simulate monocular causal video with natural camera motion, we perform depth-first search (DFS) to find smooth camera trajectory across the 20 views. We then sample 100 frames along
22

---

this path to construct a pseudo-monoeular the remaining 40 frames the driving video. For evaluation, we select a camera that is not on the sampled path and render from this novel viewpoint, which has ground truth available for comparison. We compute PSNR scores for three training strategies: (i) training only on the generated video, (ii) joint training with the base video, and (iii) joint training with ARAP refinement.
monocular video. We treat the first 60 frames as the base video and
E
More Results
To demonstrate generalization of the pipeline, we also provide visualization on some web-collected sequences shown in Figure 3.
F
Video Demo
Please refer to the video demo in the attachment.
23

---

Base Video
Prompt
Drive Video
"A small silver slowly drives up onto sidewalk from the street in a quiet urban area. The camera stays fixed in position but smoothly pans to follow the vehicle's 's motion it mounts the curb and continues onto the pedestrian pavement.
Surrounding buildings are modern and light-colored, with crosswalk lines, bollards, few pedestrians in the background. The motion feels slightly unusual yet calm, with ahrupt changes in camera movement."
"A brown and white cow slowly turns its body and walks toward the grassy field to graze. The remains fixed in physical position but gently pans follow the cow's 's movement as it turns and begins cating grass. The scene takes place quiet rural pasture under bright daylight, with patches soil, stones, and fencing visible in the foreground.
The motion is natural and unhurried, with the cow staying centered in the frame."
"A light-colored dog wearing a harness suddenly dashes into the ncarby bushes on the right side of the frame. The camera remains completely still, capturing the dog 's quick movement as disappears into the foliage. The setting dry grassy area with sparse leaves, and fence and
road visible the background. The scene feels spontaneous and natural, with no camera tracking zooming."
Figure 8: Examples of motion prompts and corresponding base/drive videos. Each column shows the input video, the motion prompt, and the generated video. Each video is visualized using two keyframes (top and bottom).
24

---

Timestep
Timestep
Timestep
Timestep 4
Base Video
Restaged Scene
Base Video
Restaged Scene
Figure 9: Some samples of 4D restaging on in-the-wild video, which are collected from Internet.
25


Reviews:
[{"review_id": "WbdbnsBosz", "reviewer": "Reviewer_dprx", "summary": "This paper addresses the task of video-conditioned 4D content generation. Specifically, given a video depicting an object and a target motion text, the goal is to generate novel videos that are conditioned on (1) the object's appearance from the input video and (2) motion synthesized via image-to-video generation models.\n\nThe key idea is to jointly optimize the reversed input video and the synthesized video, allowing motion coefficients to be shared across both domains.\n\nIn addition, the authors introduce an occlusion-aware rigidity regularization, which leverages the original video to recover occluded regions, as well as a disocclusion backtracing mechanism to handle newly revealed areas in the synthesized video.", "questions": "1) **[Clarification on Disocclusion Backtracing]**\nAs I understand it, frames of the synthetic video are jointly optimized during initialization, meaning that disoccluded regions should also be optimized alongside the base video. Given this, it is unclear why the disocclusion backtracing module is necessary I would appreciate it if the authors could elaborate on why these regions are not sufficiently addressed through joint optimization, and clarify the specific motivation for introducing this component.\n\n2) **[Clarification on Evaluation Protocol in Section 4.3]**\nThe evaluation in Section 4.3 raises some questions, as the motion generated by Sora may not precisely align with the ground-truth motion. It would be helpful if the authors could explain how alignment between Sora-generated motion and the ground truth is handled when computing the metrics.\n\n3) **[Assumptions Regarding Input Video]**\nI am also curious whether there are any assumptions or constraints regarding the input video. Since the proposed method leverages the input to handle occlusions, how does it perform when the occluded regions are not visible in the input video either? If the quality of the results is sensitive to the occlusion coverage in the input, I would suggest providing some analysis or discussion on this aspect.\n\n4) **[Request for Additional Ablations]**\nFinally, I believe that additional ablation studies with qualitative comparisons—as mentioned in the Weakness section—would greatly strengthen the paper and help better assess the contribution of each component. Including such analysis would positively impact my overall evaluation.", "limitations": "This paper does not discuss potential societal impacts, which is important given the nature of generative video models.\nAdditionally, the rationale behind L316–317 in the Limitations section is unclear. Why are highly deformable or textureless objects less temporally consistent? Further explanation would be helpful.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1) The paper introduces a new and meaningful task—video-conditioned 4D content generation—effectively bridging generative models with 4D reconstruction.\n2) The proposed occlusion-aware rigidity regularization and disocclusion backtracing are thoughtfully designed to address visibility challenges such as occlusion and disocclusion in dynamic scenes.\n3) The method demonstrates strong generalizability across diverse object types, including both articulated and rigid objects.\n\nWeakness\n1) **[ARAP Regularization and Missing Analysis]**\n ARAP regularization is widely used in 4D reconstruction, so its inclusion alone is not novel. Although the paper introduces an occlusion-aware variant, its impact is unclear without ablation comparing it to standard ARAP. The lack of citations (e.g. SC-GS) and comparison with existing ARAP-based methods further limits the clarity of the contribution.\n\n2) **[Additional ablation on each component]**\nQualitative comparisons showing the effect of each module—e.g., synthetic video optimization only, joint optimization, +ARAP, +occlusion-aware ARAP, and +disocclusion backtracking—would help clarify how each part contributes under different conditions (e.g., occlusion, disocclusion). As it stands, the effectiveness of individual modules difficult to recognize.\n\n3) **[Missing Comparison with Image + Text Methods]**\nThe paper lacks comparison with existing methods that generate video content conditioned on a single image and a text prompt. Including such a baseline would help contextualize the performance of the proposed approach.\n\n4) **[Unconvincing Visual Results in Figure 6]**\nThe output in Figure 6, particularly the camel example, is not visually compelling. Artifacts appear between the camel's humps, which makes the result look worse than the driving video itself. It is also unclear which part of the image demonstrates the claimed motion artifact improvements—highlighting or marking these regions would be helpful.\n\n5) **[Missing Detailed Description on Prompt Refinement Using VLM]**\nThe paper needs justification for why smooth camera motion is necessary. An ablation on VLM refinement is missing, and the instructions used for VLM are not provided, limiting clarity and reproducibility.\n\n6) **[Missing Detailed Description on Experimental Settings]**\nThe paper needs sufficient details on key experimental settings. For example, it is unclear how the \"naive reconstruction\" baseline in Figure 4 is implemented. Visualizations of both training and occlusion views would help clarify the effectiveness of the method. In Table 1, the baseline method referenced is not clearly specified—further explanation is needed. Additionally, it is ambiguous whether the ARAP loss is applied to all points or only to occluded regions. Lastly, the number of input video frames and the number of frames in the synthesized video should be explicitly stated for reproducibility.", "was_revised": true, "final_justification": "My primary concern was the novelty over the naive ARAP.\nThe authors have sufficiently addressed this with ablations and baseline comparisons.\nI believe the paper merits acceptance.", "rebuttal": "We appreciate that Reviewer `dprx` considers the proposed task 4D Restaging meaningful, the components are thoughtfully designed, and the pipeline generalizable.\n\n## [Weakness 1] Comparison with existing ARAP methods\n- **The goal of Occlusion-aware ARAP is to better serve a unified pipeline that leverages monocular input videos for controllable deformable scene generation.** \nThe design of occlusion-aware ARAP is tailored to the task of 4D restaging, because we aim to transfer structure and articulation from the base video into occluded or disoccluded regions during restaging. Unlike existing methods that rely on multiview or synthetic data, we operate on monocular real-world videos with newly generated motions, where occlusion patterns can differ significantly from the original sequence. By weighting the ARAP loss with visibility cues, we can preserve shape consistency even under occlusion, which is critical for high-quality 4D restaging.\n- **Compare with other methods.** \nWe compared our Restage4D with SC-GS and Vidu4D on the DAVIS dataset and reported the PSNR to evaluate reconstruction quality. We ran a sequence of `horsejump-high` and `flamingo`. \nWe describe the qualitative and quantitative results here:\n- *Sequence 1*:\n    - *Input video: A rider jumps a horse over an obstacle.*\n    - *Text prompt: \"The horse turns around.\"*\n    - *Reconstruction results:*\n        - *SC-GS fails to model realistic deformation; the horse remains static, with rotating artifacts.*\n        - *Vidu4D reconstructs the foreground reasonably well but lacks background and suffers from blurred edges.*\n        - ***Ours** successfully reconstructs the full scene (foreground and background) with realistic deformation and consistent motion.*\n- *Sequence 2*:\n    - *Input video: A flamingo drinks water with head motion.*\n    - *Text prompt: \"The flamingo moves its leg up and down.\"*\n    - *Reconstruction results:*\n         - *SC-GS fails to animate the leg, which remains static. The entire flamingo even disappears due to camera motion in some frames.*\n         - *Vidu4D only reconstructs the foreground with a gray background. The flamingo follows the prompt.*\n         - ***Ours** produces a full-scene reconstruction with consistent leg motion and high visual quality.*\n    |**Method**   | **horsejump-high-turn** | **flamingo-leg** |\n    | -------- | -------- | -------- |\n    | SC-GS     | 22.41     | 21.57     |\n    | Vidu4D     | 24.2 (foreground)      | 26.4 (foreground)      |\n    | **Ours**     | **25.8**     | **28.91**     |\n\nThe results show that existing ARAP-based methods are more suitable for \"effective multiview\" [1] synthetic data like D-NeRF [2], and fail to reconstruct the motion and geometry for the objects in monocular causal video. In addition, Vidu4D concentrates on reconstructing the foreground, and we only got a background-free object.\n## [Weakness 2 & Q1 & Q4] Qalitative Ablation\nWe provide qualitative description to illustrate the contribution of each module in our pipeline.\n- **Setup.**\nWe use the bear sequence from DAVIS with the prompt “The bear walks to the heap.” In the base video, the left side of the bear is visible, while the right side is occluded. In the generated video, the bear turns right and reveals the right-back side of its body (i.e., a disocclusion). This setup naturally introduces both occlusion and disocclusion regions. Then we visualize:\n    - Frame 15 with the camera rotated to the left. It renders the face of the bear, which is occluded now due to its turning right.\n    - Frame 40 with the camera rotated to the right. It renders the right part of the bear, which is disoccluded, revealed by its motion.\n- **Qualitative Analysis.** \nFrom the visual results, we observe\n  1. **Synthetic video optimization only** will lead to the wrong geometry for the occluded part. The entire face is missing in frame 15. The geometry of the right part in frame 40 is right.\n  2. **1 + ARAP** can help keep a part of the geometry for occlusion. The nose of the bear is missing in frame 15, and the right part in frame 40 is similar to case 1.\n  3. **Joint Optimization** visually it seems not to bring direct benefits, compared to case 1, though, it connects the base video that serves as the chance to transfer the geometry and articulation.\n  4. **3 + ARAP** can utilize the information in the base video, which leads to a good geometry for occlusion in frame 15, keeping the entire head. However, splats are missing for the disoccluded part in frame 40, since it's too rigid with the canonical frame, where these splats are invisible.\n  5.  **3 + Occlusion-aware ARAP** loses the regularization for the dis-occlusion part, leading to a better geometry in frame 40 compared to case 4, and the geometry for occlusion in frame 15 is slightly worse than case 4 for the weaker regularization.\n  6.  **5 + Backtracing** helps to improve the quality of dis-occlusion part and speeds up convergence for its geometry. The splats are denser for the right part in frame 40.\n- **Motivation for Backtracing.** \nAlthough joint training and occlusion-aware ARAP are still limited in recovering disoccluded regions, which often lack initialization due to not being visible in the base video. Moreover, ARAP tends to over-regularize these areas and prevent new geometry from emerging, especially combined with low-rank motion. Therefore, disocclusion backtracing helps effectively add new points during initialization.\n\n## [Weakness 3] Other I2V Models\n- **We integrate other image-to-video models into our pipeline, and it works on multiple models.** \nWe evaluated CausVid [3] and LTX-Video [4] for video generation and compared them with Sora. We applied our restaging pipeline to two sequences: cat-jump and cows-graze, using each I2V model as the video generation component. Below are the PSNR scores of the final restaged outputs: \n    | I2V Model | cat-jump | cows-graze |\n    | -------- | -------- | -------- |\n    | Sora     | 26.74     | 26.25     |\n    | CausVid     | 23.68    | 26.45     |\n    | LTX-video     | 25.08     | 28.05     |\n\n    These results demonstrate that our Restage4D framework performs effectively across diverse I2V backbones, with consistent reconstruction quality and visual plausibility. \n## [Weakness 4] Result in Figure 6\nIt's difficult to show the dynamic results using only images. We provide the video illustration for Figure 6 in the supplementary (3:53-4:10).\n## [Weakness 5 & Q2] VLM Refinement & Evaluation\n- **Criteria to pick the video**\nCurrently, the generated driving videos are manually filtered to ensure basic quality:\n\t1.\tCamera: We discard sequences with hard cuts or abrupt viewpoint changes, as these cause failures in dense point tracking and violate the motion consistency assumption.\n\t2.\tMotion: We also exclude sequences where the motion does not align with the input text prompt, in order to ensure edit fidelity.\n- **Effect of smooth camera**\nThe smoothness of the camera is essential for motion initialization. We use the tracking point from Tapir [5] to initialize the foreground Gaussian Splats, so we evaluate Tapir on sequences generated conditionally on `car-roundabout` in DAVIS with the prompt \"The car drives away\". One of the sequences has smooth camera motion and another has hard cuts. Tapir performs well on the smooth sequence, but on the hard-cut one, it fails to find the correspondence before and after the cut, leading to some floating points.\n- **Ablation on VLM Refinement**\nUsing VLM to refine the prompt can provide a more precise description of the camera and motion control, which accelerates the filtering for videos. In the `car-roundabout` case, with VLM refinement, there are 5 videos out of 8 that satisfy the criteria using Sora, and only 1 out of 8 without refinement. We will provide the prompt for refinement in our final version.\n## [Weakness 6] Experimental Settings\n- **Figure 4 & Table 1**\nNaïve reconstruction and baseline refer to Shape-Of-Motion. \n- **Range of ARAP** \nDuring the motion initialization stage, ARAP is applied to all points for better rigid motion and geometry initialization, while during the photometric fusion stage, ARAP is applied to occlusion points.\n- **Training Details**\nThe base video has 50 to 60 frames, and the driving video has 40 to 50 frames. We will provide the precise number.\n## [Q3] Assumptions Regarding Input Video\nSince we utilize the input video to learn the geometry and articulation, the pipeline works best when the motion and geometry in the input video are richer than those in the generated video. If the occluded regions are also not visible in the input video, the model can't reconstruct the geometry since we don't incorporate a generative prior during reconstruction. \n## [Limitation]\n- **Social Impact** \nOur framework involves generative video models, which may pose risks when applied to human-centric content, potentially leading to the synthesis of misleading or harmful videos. One possible solution is incorporating prompt filtering mechanisms during the prompt refinement stage to detect and block inappropriate or sensitive prompts.\n- **Performance Limitation**\nOur method relies on existing dense point tracking for motion initialization. However, for highly deformable or textureless objects, such tracking often fails due to ambiguous correspondences across frames. This leads to temporal inconsistencies in the reconstructed geometry and motion. Improving robustness under such cases remains a promising direction for future work.\n\nWe will add these discussions and show all the figures in the revised version.\n\n[1] Monocular dynamic view synthesis: A reality check.\n\n[2] D-nerf: Neural radiance fields for dynamic scenes.\n\n[3] From slow bidirectional to fast autoregressive video diffusion models.\n\n[4] Ltx-video: Realtime video latent diffusion.\n\n[5] Tapir: Tracking any point with per-frame initialization and temporal refinement."}, {"review_id": "XQ6ZyZlkI8", "reviewer": "Reviewer_gQQs", "summary": "Restage4D proposes a task to generate a new 3D sequence based on a given video. It proposes a framework that first does image-to-video generation, and then reconstructs the new video together with the original video into 3D.  To do so, a video concatention strategy and occlusion-aware rigidity handling are proposed.", "questions": "Please see the weakness section.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The 4D reanimation task sounds interesting and new. It could be useful for some downstream simulations like autonomous driving and robotics tasks.\n2. The paper identifies the geometry inconsistency as the challenge and tackles it with video rewinding, occlusion-aware rigidity, and occlusion backtracking, which makes a lot of sense to me. The results also show that the method is able to fix some inconsistencies in the newly generated video.\n\nWeakness:\n1. Using image-to-video generation as the first step seems a debatable design choice. The first frame of a video does not necessarily capture all important information, for example, the subject might not be present in the first frame, or textures are not observable in the first frame. To tackle this task, it seems that how to generate videos that are consistent with the input deserves more discussion. For example, we might want to let the video model take the full input video as a condition. Instead, the paper relays the inconsistency problem to the reconstruction step and tries to fix it with rigidity constraints, which I think is less optimal.\n\n2. The task seems to be defined to be too complex and specific, with a mix of reconstruction and generation. Currently, the input is a video, and the output is an edited 4D reconstruction of the input video. It seems to involve several more fundamental tasks, like single-view video reconstruction, video editing, and 3D editing. The complex task setting also makes the evaluation difficult. In the paper, the evaluation mainly reports 3D consistency to the original video and PSNR to the generated video. First, as an editing task (\"restage\"), the editing accuracy is not evaluated at all. One could achieve a perfect score if the video stays identical before and after restaging. Second, evaluating PSNR to the generated video seems a bit weird, since the task does not necessarily require having a generated video as an intermediate output. Overall, my concern is if the task is significant enough to be separated topic and if the current evaluation is really suitable for it.\n\n3. The technical part mainly makes improvements on \"shape-of-motion\" to better serve the purpose of reconstructing one video using the geometry of another video. All the technical parts bring solid improvements, but they are also not significantly novel or transferable to other domains.\n\nMinor:\n\n- L276: \"We evaluate our pipeline on and PointOdyssey\"", "was_revised": true, "final_justification": "I see value in the proposed techniques, particularly the strategies for handling occlusion and enforcing rigidity constraints. I’m convinced that these components are effective for reanimating a video in 3D space, and for that reason, I am inclined to slightly raise my score to a borderline accept.\n\nHowever, I still believe the problem formulation and evaluation methodology require improvement. The current setup would make more sense if the task were framed as, for example, reconstructing multiple videos that occur concurrently.", "rebuttal": "We thank reviewer `gQQs` for finding that our new task 4d restaging is interesting and new, and the problem identification makes sense.\n\n## [Weakness 1] Design Choice\n- **Flexible Frame Selection**  \nOur pipeline does not assume that the first frame is always used for video generation. In practice, we select a frame that contains sufficient semantic and appearance information, and it can be any frame from the input sequence. For example, in our cat sequence experiment, we use the 48th frame as the reference frame to generate a video of a jumping cat in the `[Weakness 3]` part for reviewer `dprx`. We will include this example in the final version.\n- **Research Scope and Motivation**\nWhile we agree that leveraging the full input video during generation is a promising direction, our work focuses on the reconstruction stage, which is orthogonal and complementary to advances in video generation. Even with powerful video generation models, occlusion, disocclusion, and temporal inconsistencies are inevitable when introducing novel motion. Our insight is that these challenges can be effectively addressed during reconstruction by leveraging input video geometry and structure. This separation of concerns allows our framework to remain compatible with different generation models and improve results in a plug-and-play fashion.\n\n## [Weakness 2] Task and Evaluation\n- **Task Necessity**\nOur goal is to explore a novel paradigm for 4D content creation, where motion is conditioned on both a reference video and a text prompt (L26–28). This goes beyond typical text-to-4D or image-to-4D settings by allowing users to control motion and appearance jointly through intuitive and accessible inputs. The video-based control provides richer motion priors and more precise specifications, making it particularly valuable for downstream applications such as scene animation, 4D data synthesis, and artistic content creation. We believe this justifies treating “restaging” as a distinct and meaningful research problem.\n- **Evaluation Metrics**\n  1. Editing Accuracy: Evaluating prompt-following behavior in generative models is still an open problem in both image and video generation. In our current setup, we mitigate this by manually filtering driving videos to ensure alignment with the text prompt. As shown in Figures 1, 3, and 4, the resulting motion diverges clearly from the base video, demonstrating successful editing rather than identity preservation. \n  2. PSNR: This metric serves a clear purpose in our reconstruction-based framework. Since the driving video is pre-filtered to match the desired motion, PSNR reflects how well the reconstructed 4D scene conforms to the intended motion and appearance. It measures the faithfulness of the reconstruction to a desired visual target, which is particularly relevant when using generative video as input. We acknowledge that more semantic-aware metrics would strengthen evaluation, and we plan to explore automatic prompt-following metrics (e.g., using vision-language models) as future work.\n\n## [Weakness 3] Novelty and Transferability\n- **Novelty**\n  - As stated in L81–86, **our core contribution lies in introducing a new paradigm for video-conditioned 4D content creation**, where motion and geometry are jointly guided by the base video and edited via a generated driving video. While ARAP-based regularization is not new in isolation, our occlusion-aware variant is specifically tailored to tackle challenges unique to this setting, especially how to reconstruct geometry across occlusion/disocclusion boundaries introduced by new motions. Therefore, the technical novelty lies in how these components interact synergistically in our pipeline rather than in isolation.\n  - As detailed in our response to reviewer `dprx`, we visualize the effect of each module using the bear sequence from DAVIS. This example involves both occluded (bear’s face turning away) and disoccluded (right side of the bear turning visible) regions. A detailed qualitative analysis shows how each component incrementally improves performance:\n    - Synthetic-only optimization fails to recover geometry in occluded areas.\n    - Adding ARAP preserves shape better, but remains limited.\n    - Joint optimization brings the base video into the loop, enabling geometry transfer.\n    - Occlusion-aware ARAP loosens rigidity for disoccluded regions.\n    - Backtracing explicitly supplements sparse areas, speeding up convergence.\n\n    **These insights collectively demonstrate that each technical module plays a novel and indispensable role when integrated for the specific goal of controllable 4D restaging.**\n\n\n- **Transferability** \nWhile our pipeline is designed for restaging, the core ideas are generalizable:\n  1. Using video-conditioned articulation as a source of geometry/motion supervision.\n  2. Using occlusion-aware regularization to handle incomplete or one-sided observations.\n  3. Leveraging joint optimization with deformable priors to fuse geometry and motion across input modalities.\n  These strategies can be directly applied to tasks such as video-to-3D reconstruction, editable 4D avatar generation, and controllable NeRF editing, particularly in settings where occlusions and motion-driven deformation are challenging.\n  \n## [Minor] Typo\nWe will fix all the typos in the revised manuscript."}, {"review_id": "1L7MzyOmXe", "reviewer": "Reviewer_duUh", "summary": "This paper presents Restage4D, a pipeline for reanimating deformable 3D scenes by conditioning on a single real-world video. It first generates a synthetic driving video based on the input’s first frame, then the method uses the original sequence as a supervisory signal to correct artifacts from synthetic motion. The input real-world video provides the shape and articulation model, while the generated video provides new dynamics.", "questions": "N.A.", "limitations": "The authors have discussed the method’s limitations, but they omit the societal impact—please include it in the final version.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The paper is well written and the task is highly engaging.\n\n2. The video-rewinding joint-training scheme is both well motivated and insightful.\n\nWeaknesses\n\n1. All examples involve a single foreground object. How does the method handle scenes with multiple interacting objects—for instance, “two people dancing together”?\n\n2. Some minor typos to fix: Line 176: “bas” → “base”, Video at 4:12 (bus case): the wrong prompt is shown.\n\n3. Additional qualitative results—especially on more complex scenes—would strengthen the empirical validation of your approach.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed my concerns, I will keep my positive score.", "rebuttal": "We thank the reviewer `duUh` for finding our task engaging and pipeline insightful. We address the concerns by providing more experiments.\n\n## [Weakness 1] Multiple Objects\n**Our monocular reconstruction pipeline does not impose any restriction on the number of objects in the scene.**\nIn practice, the use of K-Nearest Neighbor (KNN) graph construction for the ARAP loss leads to localized connectivity among nearby splats. As a result, splats naturally group into object-level clusters, since intra-object distances are smaller than inter-object distances. This behavior helps maintain per-object rigidity and avoids enforcing unrealistic deformation across different objects. \n\nAt object interaction boundaries (e.g., contact surfaces between two objects), the proximity-based KNN may occasionally link splats across object boundaries. While this typically has a limited impact, it may affect fine-grained interaction modeling (e.g., relative motion at contact points). Addressing this challenge remains an important direction for future work.\n## [Weakness 2] Typos\nWe will fix all the typos in the revised version. \n\n## [Weakness 3] More Complex Scene\nWe run our restaging pipeline on a more complex scene with two interacting objects, a cat and a dog. \n- *Input video content: A dog and a cat are in the bathroom. The cat is standing on the toilet lid, and the dog is by its side on the floor. They first sniff each other, then the cat turns around at the toilet lid.*\n- *Text prompt: \"The cat jumps towards the dog.\"*\n- *Restaged video: The video follows the prompt, and there are two sets of dynamic Gaussian Splats representing the cat and dog, respectively.*\n|  | PSNR | OCV CLIP | Vol. Consist | Edge Consist |\n| -------- | -------- | -------- | -------- | -------- |\n| Cat-jump-dog     | 31.53     | 84.27     | 8.84 |  17.25 |\n| DAVIS Sequence | 27.12 | 85.40 | 10.32 | 16.28 |\n\nThis case demonstrates that our pipeline can generalize to more complex scenes with multiple objects. We will include this sample in our final version.\n\n## [Limitation]\n**Social Impact**\nOur framework involves generative video models, which may pose risks when applied to human-centric content, potentially leading to the synthesis of misleading or harmful videos. One possible solution is incorporating prompt filtering mechanisms during the prompt refinement stage to detect and block inappropriate or sensitive prompts."}, {"review_id": "mAyM9EPoLS", "reviewer": "Reviewer_Nx6n", "summary": "This work introduces a new task, video restaging, and builds a novel framework to achieve controllability and geometric fidellity in 4D reconstruction. The task is to generate new motion for the object in input video. The main idea of this work is to re-animate the first frame of input video and leverage base video to correct the artifacts from generative videos during 4d reconstruction. Video-rewinding joint training strategy is proposed with occlusion-aware ARAP and disocclusion", "questions": "Were the generative videos cherry-pick? If so, what was the picking criteria?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "strength:\n1. this work propose a novel task, re-animating 4D object in the wild \n2. The paper is easy to follow, and the result is good.\n\nweakness:\n1. lack comparison the state-of-the-art 4D reconstruction methods, e.g., shape-of-motion (the baseline in this work), SCGS and Vidu4D. It is suggested to take sololy new video as input for these methods for comparison.\n2. The evaluation presented in Table 1 is conducted solely on training views, which is not a reasonable approach. It would be more appropriate to evaluate novel view synthesis capabilities. While I acknowledge that generated videos lack ground truth for novel views, the authors could leverage public multi-view video datasets for evaluation, such as the Neural 3D Video Synthesis dataset[1] and Dycheck[2]. A possible evaluation strategy would be to split these videos into two parts: using the first segment as the base video and the second as the reference for generated content. Although this approach would yield artifact-free \"generated\" videos, it's worth noting that state-of-the-art video generation models can also produce artifact-free results after careful cherry-picking.\n[1] Neural 3D Video Synthesis from Multi-View Video\n[2] Monocular Dynamic View Synthesis: A Reality Check\n\nIf my concerns are adequately solved during rebuttal, I'm glad to raise my assessment.", "was_revised": true, "final_justification": "The authors have provided extensive experiment results, both comparison and ablation, to address my concerns. So I decide to raise the recommendation.", "rebuttal": "We appreciate reviewer `Nx6n` finds our proposed task novel and the result good. We address the concerns with additional experiments.\n\n## [Weakness 1] Comparison with Other Models\n\nWe have conducted comparisons between Restage4D and two recent methods, SC-GS and Vidu4D, on generated sequences from the DAVIS dataset. Specifically, we use two videos: horsejump-high and flamingo. For each, we apply a motion prompt and evaluate reconstruction quality both qualitatively and quantitatively via PSNR.\n- *Sequence 1*:\n  - *Input Video: A rider jumps a horse over an obstacle.*\n  - *Text Prompt: \"The horse turns around.\"*\n  - *Reconstruction Result:*\n      - *SC-GS fails to model realistic deformation; the horse remains static, with rotating artifacts.*\n      - *Vidu4D reconstructs the foreground reasonably well but lacks background and suffers from blurred edges.*\n      - ***Ours** successfully reconstructs the full scene (foreground and background) with realistic deformation and consistent motion.*\n\n- *Sequence 2*:\n  - *Input Video: A flamingo drinks water with head motion.*\n  - *Text Prompt: \"The flamingo moves its leg up and down.\"*\n  - *Reconstruction Result:*\n      - *SC-GS fails to animate the leg, which remains static. The entire flamingo even disappears due to camera motion in some frames.*\n      - *Vidu4D only reconstructs the foreground with a gray background. The flamingo follows the prompt.*\n      - ***Ours** produces a full-scene reconstruction with consistent leg motion and high visual quality.*\n\n**Quantitative Results (PSNR)**: \n| Method | horsejump-high-turn | flamingo-leg |\n| -------- | -------- | -------- |\n| SC-GS     | 22.41     | 21.57     |\n| Vidu4D     | 24.2 (foreground)     | 26.4 (foreground)      |\n| **Ours**     | **25.8**     | **28.91**     |\n    \nThese results show that:\n\n1.\tSC-GS, while effective on synthetic multiview datasets (e.g., D-NeRF [citation]), struggles with monocular, real-world video due to occlusion and motion complexity.\n2.\tVidu4D focuses solely on reconstructing foregrounds and lacks full-scene fidelity.\n3.\tRestage4D handles deformable motion and scene-level reconstruction holistically, achieving higher PSNR and better visual quality under diverse motion prompts.\n\nWe will include the comparison in our final version.\n## [Weakness 2] Evaluation\nWe agree that evaluating the novel view by comparing with ground truth is important for validating view generalization in monocular reconstruction. \nTo address this, we conduct additional evaluation using the Neural 3D Video Synthesis dataset [1], which provides 20 calibrated camera views of dynamic scenes, from which we can derive ground truth for novel views\nWe adopt the following protocols:\n\n1.\tTo simulate a monocular causal video with natural camera motion, we perform depth-first search (DFS) to find a smooth camera trajectory across the 20 views. We then sample 100 frames along this path to construct a pseudo-monocular video.\n2.\tWe treat the first 60 frames as the base video and the remaining 40 frames as the driving video.\n3.\tFor evaluation, we select a camera that is not on the sampled path and render from this novel viewpoint, which has ground truth available for comparison.\n4.\tWe compute PSNR scores for three training strategies: (i) training only on the generated video, (ii) joint training with the base video, and (iii) joint training with ARAP refinement.\n\n\nWe evaluate on two sequences: sear-steak and cut-roasted-beef. The PSNR results are as follows:\n\n|  | sear-steak | cut-roasted-beef |\n| -------- | -------- | -------- |\n| Driving Video   |   12.87 ± 0.37   |  17.84 ± 0.56    |\n| Joint     |   17.18 ± 0.56   |  19.14 ± 0.63  | \n| Joint + ARAP     | **17.70 ± 0.46**  |   **19.56 ± 0.72**  |\n\nThese results demonstrate the benefits of using the base video as supervision and incorporating geometry-aware refinement. Joint training notably improves generalization to unseen viewpoints, and the addition of ARAP yields further gains.\n\n## [Question] Criteria to select video\n\nCurrently, the generated driving videos are manually filtered to ensure basic quality:\n\n1.\tCamera: We discard sequences with hard cuts or abrupt viewpoint changes, as these cause failures in dense point tracking and violate the motion consistency assumption.\n2.\tMotion: We also exclude sequences where the motion does not align with the input text prompt, to ensure edit fidelity.\n\nWe are exploring the use of vision-language models (VLMs) to automate this filtering process, and plan to incorporate it into our pipeline in future work."}]

Author rebuttal:

Meta-review: This paper proposes a new task of generating 4D content by conditioning on a single real-world video. 
It receives two acceptance and two boardline acceptance. 
Main concerns are about ablation studies, state-of-the-art comparisons.
The authors provided extensive experiments to resolve the concerns.
The AC recommends acceptance for this paper.
However, the AC found the abstract in the openreview is not consistent with the version in the submitted pdf file. The authors are urged to update and revise any inconsistencies.
Decision: Accept (poster)