Title: Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos
Venue: NeurIPS 2025
Accepted: yes
Keywords: novel-view synthesis

dynamic scenes

monocular videos

video diffusion

video inpainting

test-time finetuning
Abstract: We explore novel-view synthesis for dynamic scenes from monocular videos. Prior approaches rely on costly test-time optimization of 4D representations or do not preserve scene geometry when trained in a feed-forward manner. Our approach is based on three key insights: (1) covisible pixels (that are visible in both the input and target views) can be rendered by first reconstructing the dynamic 3D scene and rendering the reconstruction from the novel-views and (2) hidden pixels in novel views can be ``inpainted" with feed-forward 2D video diffusion models. Notably, our video inpainting diffusion model (CogNVS) can be self-supervised from 2D videos, allowing us to train it on a large corpus of in-the-wild videos. This in turn allows for (3) CogNVS to be applied zero-shot to novel test videos via test-time finetuning. We empirically verify that CogNVS outperforms almost all prior art for novel-view synthesis of dynamic scenes from monocular videos.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a framework for novel view synthesis given a monocular video. The proposed method utilizes rasterized point clouds in the novel views to condition a video diffusion model, which is trained to fill in the missing information. The main novelty of the paper is to design a training algorithm to utilize only monocular videos (without concurrent multiviews) to train the video diffusion model, utilizing covisibility. Test-time fine-tuning is also conducted and important to improve the quality of the results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n- The method is simple and makes sense. Utilizing the known information (reconstructed point clouds from structure from motion) and fill in the missing information with a generative model is intuitive and effective.\n\n- The paper provides results on various types of scenes (real, synthetic, with/without scene dynamics). \n\n\nWeaknesses:\n\n- The paper may benefit from more discussions about why using co-visibility when the actual novel-view image and depth map is unknown can simulate the mask at source view if novel view were presented. Also in what condition does it hold?  If my understanding is correct, it seems to be that the proposed way to create the mask does not consider if the novel view has an occluder that additionally blocks point clouds from the source view -- e.g., imagine the novel view is partially behind a corner but the corner is not visible in the source view.  Also, the proposed method to create a mask seems to assume pinhole camera models -- if the camera has a sizable aperture, more information may be available (through defocus blur).   Since the self-supervision training by simulating masks is the main novelty of the paper, it seems that the discussion is a bit light in the paper. \n\n\n- The paper does not answer a few questions:  \n(1) Does the method produce consistent multiview videos?  If I sample twice the generative model using the same input monocular video but from two different camera trajectories, will the content of the unobserved areas in the two trajectories be consistent with each other?  Does the test-time finetuning improve the consistency between multiple views?  \n\n(2) How long does the test-time finetuning take, how many gpus does it need? \n\n(3) What is the performance of the model before test-time finetuning on DyCheck? \n\n(4) How long do the comparison methods take in Table 1 and Table 2 (including test-time optimization and rendering)? \n\n\n- While utilizing rasterized point clouds in the novel view is effective, it also makes the system depend heavily on the quality of the structure from motion (ie, depth map, camera pose and intrinsics).  The difference in quality can be observed easily between the Kubric and in-the-wild results. The in-the-wild results (eg, Superman and Friends) seem to have camera breathing that makes the characters’ faces look unnatural.  In comparison, when using ground-truth depth and camera parameters the Kubric results do not have the problem.  \n\n\n- The writing can be slightly improved.  When I was reading the paper, I was not sure whether the method requires concurrent multiview videos or not until I finished reading the method part (e.g, Line 39).  It was also not clear in Line 108 and 114 that whether g_src contains the entire information about the scene (ie, full 3D) or not.   Line 122 seems to have a type (V_src or V_novel). \n\n- It seems from Figure 5, the finetuning reduces the contrast of the results. Any reason why? \n\n- Would the proposed method to generate camera trajectories create cameras inside objects?\n\nQUESTIONS:\nI asked the questions above, as an outline: \n- Would having occlusion / defocus effect in the novel views and source videos affect the masks?  \n- How long does the test-time finetuning take (on how many gpus) and how does it compare to baselines? \n- Does the proposed method produce consistent multiview videos when sample the model multiple times with different trajectories, and does the test-time finetuning help? \n- What is the performance of the model before finetuning on DyCheck?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a warping-then-inpainting paradigm for dynamic novel view synthesis from monocular video. Begin with reconstructed dynamic scenes (e.g., MegaSAM ), and novel views can be rendered with different camera trajectories. The main challenge lies in inpainting disoccluded regions in rendered novel views. To address this, the authors fine-tune the video generation model CogVideo for video inpainting, using constructed training pairs of masked and source images. Additionally, they further fine-tune the model on test scenes to reduce domain gaps and enhance inpainting quality. Experimental results demonstrate that the proposed method achieves superior performance in most cases.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n(1) This paper proposes a simple pipeline for dynamic novel view synthesis from a monocular video, yielding good results.\n(2) This paper fine-tunes a video inpainting model, suitable for filling in disoccluded regions in novel view synthesis tasks. \n(3) The paper is well organized and easy to follow.\n\nWeakness:\n(1) Since the warping and then inpainting pipeline is widely used in view synthesis and 3D scene creation (e.g., Text2Room, etc.), it is not suitable to claim it as a contribution.\n(2) The generated content lacks 3D consistency. The output of this method is a camera-controlled video (cannot support free-view exploration), which means that disoccluded regions can be generated with different content when conditioned on a new camera trajectory in the second generation.\n(3) The technique contributions seem to be limited. Fine-tune a video generation model to inpaint the disoccluded regions in novel views without insightful designs during the training. Other components in this paper are built upon existing approaches.\n\nQUESTIONS:\n(1) Do the performance improvements come from more advanced 4D initialization and video generation models? As mentioned in the paper, the quality of the reconstructed scene influences the performance. Given the pipeline’s similarity to TrajCrafter (warping + novel view rendering + finetuned video inpainting), how does the proposed method quantitatively and qualitatively outperform it? Are gains attributable to the choice of CogVideo, warping strategy, fine-tuning approach, and reconstruction methods?\n\n(2) By warping the monocular video into novel views (some pixels are occluded and discarded) and then warping it back, we can also create the training pairs. This approach has been widely used in TrajCrafter and previous methods, such as [1]. I believe the underlying motivation for the warp-back strategy is the same as the design of this paper. Or what are the advantages of this paper's mask creation design? \n\n(3) About per-scene optimization. Will the finetuning on a specific scene cause the overfitting problem?\n\n[1] Single-View View Synthesis in the Wild with Learned Adaptive Multiplane Images\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers novel view synthesis from monocular videos. The method can be summarized by first creating a 3D reconstruction which is rendered to the target views followed by a fine-tuned video diffusion model which refine the rendering and in-paint the missing regions. The diffusion model is based on a pre-trained CogVideoX model and then fine-tuned on a large set of videos for which the authors present a method to create plausible occlusion masks from monocular videos which the diffusion model is conditioned on. Furthermore, in addition to the general pre-training, the diffusion model is refined in the same way for each video at test time. Experimental results show excellent qualitative results and quantitative results outperforming previous state-of-the-art for the task.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The main strength of the paper is that results look very good qualitatively, and better than the previous state-of-the-art which the method is compared to.\n- The key difference methodologically compared to prior work such as NVSsolver [84] or Gen3C (which both use 3D reconstruction+inpainting with video diffusion models for NVS for monocular videos) is to fine-tune for each test video. This is shown quantitatively to greatly improve the performance.\n- The data generation strategy is well designed to resemble point cloud renderrings by using co-visible regions in reconstructed videos, and it is shown to improve metrics compared to training with simpler masking heuristics. It should be noted though that the data curation method shares many similarities with TrajectoryCrafter [85], something which should be highlighted clearer in the paper.\n\nWeaknesses:\n- The method shares many similarities with TrajectoryCrafter [85], in the sense that a 3D reconstruction is created and a video diffusion model is conditioned on renderings of the 3D reconstruction in the target poses, and the data curation strategy used to create data for pre-training is similar. An interesting experiment would be to use TrajctoryCrafter and the proposed test-time fune-tuning. This would be interesting since the difference between CogNVS and TrajCrafter in Table 1 is in line with the difference between using test-time fine-tuning or not in the ablation in Table 3.\n- While the results are improved by the fine-tuning of the video diffusion model per test video, it also significantly increases the run-time. It would have been good to also include run-time comparisons of the different methods. The run time of the test-time fine-tuning is not stated directly but can be calculated. To my understanding the diffusion model sampling takes 5 minutes per video while the per-video diffusion model fine-tuning takes about 1,2/2,4 hours (short vs long videos) with 8 GPUs, for each video, so about 10-20 GPU hours for a single video. (inferred from lines 188-192 in the paper - 200 or 400 steps of test-time fine-tuning vs 12,000 steps for the pre-training which takes 3 days with 8 A6000 GPUs - please correct in the rebuttal if something is misunderstood or incorrect)\n\nQUESTIONS:\n- Please see weaknesses.\n- To my understanding, $V_{nvs}^{cov}$ is encoded by the VAE encoder $E$ to $z_{cond}$, where pixels that are not co-visible are simply zeros. Are missing regions simply treated as zero-values in $V$, and were alternatives such as providing masks as input to the diffusion models tested? The diffusion model denoises in the latent space which has a reduced spatial resolution, e.g. 1/8th. How are regions in 8x8 patches in the image treated when there are a few missing values only? Are they treated simply as black pixels among the co-visible pixels in the 8x8 patch and encoded as such by the VAE?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper tackles the important task of dynamic novel-view synthesis. It introduces CogNVS, a video diffusion framework to inpaint hidden pixels in novel views and is able to generalize to novel videos via test-time finetuning. Experiments show state-of-the-art results across multiple benchmarks (Kubric-4D, ParallelDomain-4D, DyCheck) without requiring multi-view training data or ground-truth geometry.

STRENGTHS AND WEAKNESSES:
### Strength
- I like the idea of constructing self-supervised learning pairs on 2D videos to avoid costly 3D geometric data supervision. It is easier to scale up and learn the scene dynamics in a data-driven way.
- Qualitative results show improved temporal consistency and 3D plausibility over baseline methods.

### Weakness
- Almost all human and animal characters in the generated videos look somewhat distorted (see the in-the-wild bear case, superman case, and man hiking case in the webpage). The face and body feel flattened. Is this a result of inaccurate MegaSAM estimation? 
- Given the randomly constructed pairs for training, what is the maximum rotation angle or the maximum portation of unseen pixels can this model generate robust outputs?

QUESTIONS:
- Almost all human and animal characters in the generated videos look somewhat distorted (see the in-the-wild bear case, superman case, and man hiking case in the webpage). The face and body feel flattened. Is this a result of inaccurate MegaSAM estimation? 
- Given the randomly constructed pairs for training, what is the maximum rotation angle or the maximum portation of unseen pixels can this model generate robust outputs?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02