Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos
Authors: Kaihua Chen
Tarasha Khurana
Deva Ramanan
Venue: NeurIPS 2025 poster ()
Keywords: novel-view synthesis
dynamic scenes
monocular videos
video diffusion
video inpainting
test-time finetuning
Primary area: applications

Abstract: We explore novel-view synthesis for dynamic scenes from monocular videos. Prior approaches rely on costly test-time optimization of 4D representations or do not preserve scene geometry when trained in a feed-forward manner. Our approach is based on three key insights: (1) covisible pixels (that are visible in both the input and target views) can be rendered by first reconstructing the dynamic 3D scene and rendering the reconstruction from the novel-views and (2) hidden pixels in novel views can be ``inpainted" with feed-forward 2D video diffusion models. Notably, our video inpainting diffusion model (CogNVS) can be self-supervised from 2D videos, allowing us to train it on a large corpus of in-the-wild videos. This in turn allows for (3) CogNVS to be applied zero-shot to novel test videos via test-time finetuning. We empirically verify that CogNVS outperforms almost all prior art for novel-view synthesis of dynamic scenes from monocular videos.

Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos
Kaihua Chen"
Tarasha Khurana" Deva Ramanan
Carnegie Mellon University
https://cog- wwwcoooognvv...o nvs. wws.github.ii io/
Input views
Dynemic novel views
Reconstruct
Inpaint
Temt-Time Finetune
We present CogNVS, video diffusion model that enables novel-view synthesis dynamic scenes. Given in-the-wild monocular video of dynamic scene, first reconstruct the scene, render from the target novel-view and inpaint any unobserved regions. Because CogNVS can be pre-trained via self-supervision, it also be test-time-finetur to zero-shot generalize to novel domains. Our simple pipeline outperforms almost all prior state-of- the-art for dynamic novel-view synthesis. We show outputs from CogNVS from unseen videos; a real-world video above, and a generated video below.
Figure 1:
given target video, enabling it
Abstract
We explore novel-view synthesis for dynamic scenes from monocular videos. Prior approaches rely on costly test-time optimization of 4D representations or do not preserve scene geometry when trained in a feed-forward manner. Our approach is based on three key insights: (I) covisible pixels (that are visible both the input and target views) can be rendered by first reconstructing the dynamic 3D scene and rendering the reconstruction from the novel-view and (2) hidden pixels novel views be "inpainted" with feed-forward 2D video diffusion models. Notably, our video impainting diffusion model (CogNVS) can be self-supervised from 2D videos, allowing to train a large corpus of in-the-wild videos. This in turn allows for (3) CogNVS to be applied zero-shot to novel test videos via test-fime finetuning. We empirically verify that CogNVS outperforms almost all prior art for novel-view synthesis of dynamic scenes from monocular videos.
'Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Rapid advances in static 3D scene representations [50] 31 have paved the way for spacetime understanding of the dynamic world. This has enabled photorealistic content creation and immersive virtual reality applications. In this work, we focus on the problem of novel-view synthesis from casually-captured monocular videos of dynamic scenes.
Why is this hard? Prior work on dynamic view synthesis addresses this task from two extremes. The first class of methods "test-time" optimize a new 4D representation from scratch for every new test video. While this ensures
plaausill scene geometry, careful choices in modeling scene motion- in the form of an independent deformation field, or learnable temporal offsets have to be made (74, 41, [T]. More importantly, it can take on the order of hours to optimize and render a novel-view video. An attractive alternative is to train large feed-forward video models directly for view synthesis [95] 25 While inference on such models is dramatically faster (on the order of milliseconds), the resulting renderings often are not as accurate as their test-time optimized counterparts. From a pragmatic perspective, such models need to be trained on mega-scale multi-view training data, which is difficult to obtain for dynamic scenes.
Our method addresses the above challenges by decomposing the problem of dynamic view- synthesis into three distinct stages. First, we lean on the success of non-rigid structure from motion [46] 94 [40] approaches that produce reconstructions of visible scene regions, sometimes known as "2.5D" reconstructions (since occluded regions are not reconstructed), We point out that such reconstructions can be trivially produced for casual mobile videos captured with depth sensors and egomotion [20). When such reconstructions are rendered from a target novel view, previously-hidden regions will not not be rendered. To "inpaint" these regions, we train a 2D video- iiteo-inpainter-- - CogNVS by fine-tuning a video diffusion model Vogee (89) to condition on the partially-bseervable novel-view pixels. Importantly, we allow CogNVS to also update the appearance of previously-visible pixels, allowing our pipeline to model view nie-ddeenddent (dynamic) scene effects.
The key insight of our work is that CogNVS can be trained on any 2D video via self-supervision. However, rather than training our inpainter with random 2D masks, we make use of 3D multi-view supervision that better captures 3D scene visiblity, similar to prior art [78). Specifically, given a 2D training video, we first reconstruct it (with an aff-the-shelf method such MegaSAM) and then render the reconstruction from a random camera trajectory. This rendering is used to identify co-visible pixels from the source video that remain visible in the novel views. This original source video and its co-visible-only masked variant can now form a training pair for 3D-consistent video inpainting. Importantly, because such a training pair does not require ground-truth 3D supervision, CogNVS can be trained on diverse in-the-wild 2D videos. We use dynamic scenes from TAO ([3), SA-V [59). Youtube-VOS (86), and DAVIS (54). Equally as important, we use the same paradigm to test-time finetume CogNVS on the test video-of-interest. We show that this allows our pipeline to "zero-shot" generalize test videos that were never seen during training. We argue that our test-time finetuning of 2D diffusion models can be seen as the bbest-of-both-wordd training data (for data-driven robustness) and test-time optimization (for accuracy).
by leveraging large-scale
In summary, our contributions are as follows: (1) We decompose dynamic view synthesis into three stages of reconstruction, inpainting and test-time finetuning, (2) we use a large corpus of only 2D videos for training CogNVS, and (3) we do extensive zero-shot benchmarking on three evaluation datasets against statc-of-the-ar methods and show improvements on dynamic view synthesis.
2 Related Work
Novel-view synthesis has seen recent advancements with the rise of implicit scene representations
like NeRFs (50) and Gaussian primitives [31 32, 12]. We have seen widespread efforts in scaling
these representations to model larger scenes [22 83 68). making them faster to fit [36] 22
15
31
5T D anti-aliased (42, 2 3 26
4 and extend to representing dynamic scenes 64 21 56
36 The most popular paradigms have been the adoption of dynamic NeRFs [5 56 19 52 and
deformable Gaussian primitives [3 88
47
for modeling scene dynamics, apart from using
voxel grids 34| 33) or learnable tokenzation 96
Most approaches need multi-view posed videos

---

Renders for pre-training
Pencere for test-time fine-turing
Renders for inference
Vo
CogNVS
CogNVS A
Sure
CogNVS @
Loss
Loss
V
V
luvs
(a) Training
(b) Zero-shot Tasting
Figure 2: CogNVS overview. During training (left), given 2D source video (in blue) of a dynamic scene, we first reconstruct the scene using off-the-shelf monocular reconstruction algorithms like MegaSAM [40] to obtain the 3D scene geometry, Sare and camera odometry, Csre- We then sam- ple a set of arbitrary camera trajectories {c1.... CN } to simulate plausible occluded geometries, (S - which when rendered from original camera trajectory, Care produces mask of source pixels that are co-visible in the sampled trajectory (in orange). The source video and its masked variant produce a self-supervised training pair for learning CogNVS, our video inpainting diffusion model (visualized in Fig. 3) At inference (right), we finetune CogNVS on the given input sequence by similarly constructing self-supervised training pairs, The final novel-view is then generated using the finetuned CogNVS in a feed-forward manner.
as input, and only recently monocular view synthesis has gained traction 17 37 74 39} However,
each of the aforementioned approaches have be test-time optimized separately for every new test video, are slow to optimize and yet fail to recover highly- dynamic scene content (37]. Moreover, there is focus on predicting the unobservable scene content, which is exacerbated by benchmarking metrics that only evaluate co-visible pixels (20) in training and inference views and therefore encourage benchmarking novel views that are not too far apart from the training views. Our approach instead reformulates dynamic view synthesis as an inpainting task, which specifically focuses generating parts of the scene that were occluded from the training views, thereby facilitating extreme novel view synthesis for dynamic scenes. Our large-scale pretraining for feed-forward novel-view inpainting enables data-driven robustness.
Data-driven novel-view synthesis approaches have emerged 73 train for view synthesis in feed-forward manner with large-scale data. One class of methods is based on transformer architectures, more often than not trained with multi-view supervision and rendering in the loop 25 95| 29
82
92
45
87 which
85 60). Another class of methods reformulate novel-view synthesis as a conditional generation task and use diffusion-basee generative architectures [45] 62) for
80
the same. Initially, the focus was developing data-driven pipelines for static novel-view synthesis
45
44 43 93 95 80 7 76). or exploiting data-driven priors [55] 75 66
9 69
35| 57 90). using
multi-viev posed image inputs. However, the focus is now shifting to dynamic view synthesis of
38 28 These approaches allow greater level of hallucination of unseen scene components which instills the capability of view synthesis for camera poses that are far apart from the training views. We fit into this setting. While the data-driven learning provides faster inference times and broad generalization, it compromises 3D geometric accuracy and thysical-plausibiie outputs (e.g., objects suddenly exist or cease to exist) (92]. In this work, we highlight that test-time finetuning is crucial to preserving the 3D geometry of the scene and reducing implausible artifacts.
casually captured videos in an unconstrained setting [73 61
92
87
which introduces unrealistic artifacts in the synthesized
Test-time finetuning is a lingsstndding paradigm to curb distribution shifts in machine learning algorithms and improve their generalization. It's origin lies in early-age algorithms for optical char- acter recognition [5) and text classification (30], where the algorithm adjusts itself after observing the test data. A decade back, popularly resurfaced for super-resolution (63) where learning to super-resolve an image achieved by downsampling and super-resolving the test image. Domain
3

---

generalization approaches for vision 66 through and recently, chain-of-thought prompting 79 and general LLM reasoning [T] in natural language processing adapted this paradigm. The most recent adoption was seen in 4D) reconstruction and tracking (16), and we similarly explore this paradigm further in our work.
27
10
18 97
soon took inspiration from this break-
3
Method
Given a monocular video of a dynamic scene, Vic (V T we want to generate a novel from target camera pose. As discussed (c.f. Fig. 2). we achieve this by decomposing the task into three distinct stages - (1) obtain an off-the-shelf reconstruction of the observed scene over time, (2) render the scene from the novel views and inpaint the non-co-visiin visible regions, and (3) curb the train-test distribution shift with test-time finetuning, Note that the first two stages of our pipeline are similar to concurrent work [92 [T] but importantly we find that test-time finetuning is crucial stage to allow generalization. We now describe each of the stages in detail.
view of the observed scene, Viv== {Vi Nive)<== )
3.1 Dynamic view synthesis structured inpainting
We use off- the-shelf SLAM frameworks, like MegaSAM (40), to obtain a reconstruction of the given scene. Formally, let the underlying 3D structure of the world as observed by V be represented by. S = {X time, t. Any physical properties of the primitives the recovered camera poses from which V Vae was observed be, Care = ( where с denotes camera pose and is formulated as,
17" I==. where X are the evolving 3D primitives (points, Gaussians, etc.) across omitted from this discussion for simplicity. Let
(R, t) E SE(3) lie group. The source video Vsre can be
obtained by using a rendering function R as,
V Vac = R( (SSre; Care)
Learning to inpaint novel views For obtaining Vuvs, we note that a subset of 3D primitives that must be visible from Cares are already available the reconstructed scene geometry. Iare- Therefore, partial observation of the world in the form of co-visible pixels (20) from novel views, Cavs = [chvss 1 can rendered as follows,
V = ( Cavs)
At this point, the novel view synthesis is incomplete, and all missing regions have to be generated. To this end, we train a conditional video diffusion model, CogNVS (denoted by eạ) built on top of recently proposed transformer-bast based video diffusion model [89]. CogNVS takes in the partially observed novel view video and generates an inpainted novel-view of the scene. The overall CogNVS pipeline first employs 3D causal VAE compress the conditioning Veov and target novel-view Var into latent representations Zeond = &(V nvs and Zg = &(V respectively, enabling efficient training while preserving temporal coherence and photometric fidelity. Here, & is the VAE encoder. Gaussian noise is then added to the target latent Zo, and the resulting noisy latent is concatenated with the conditional latent Zeond. This joint representation is passed through a self-attention transformer equipped with 3D rotary positional embeddings (3D-RoPE) (65) and adaptive layer normalization, which predicts the added noise. The training objective follows a score matching formulation:
min E #z (rrr), Beond= (V kw (K,,,),,,,,))) c~N((0,))
6o 6o k Zeond) < <|2
Here, Zy: = -o0 VT O'g denotes the noisy latent at a uniformly sampled timestep k where Ok is the cumulative signal preserving factor. While Cog VideoX originally designed as an image-to-video diffusion model that zero-pads conditional image patches to match the length of the target video, we adapt for iideo-to-vides setting, where the shapes of the conditional and target inputs are inherently aligned and no padding is needed. In practice, CogNVS is trained with datasets of 2D videos which used generate self-supervised training pairs. We discuss this below.
3.2 Data generation for self-supervised training
We propose to train CogNVS in a self-supervised manner. This allows us to use a large corpus of 2D videos. For each casually captured monocular video V similar to prior work 23 48 92

---

Scurce camera trajectory Care
Self sarr-suervised training
Input
video
SLAM
hamowork
Aibiirary camera trajectory Cn
Voov arG.
V.n.
V
Figure 3: Self-supervised training data generation. To curate a large training set for video inpainting, we first reconstruct an input source 2D video (in blue) with an off-the-shelf monocular SLAM system. After reconstruction, we randomly sample N pairs of 'start" and 'end' camera poses around a spherical region, 8 of the estimated camera pose in the given 2D video. S is bounded by predefined deviation in the spherical coordinate axes, similar to prior work (93). We sample SE(3) camera trajectory that interpolates the start and end poses while looking at the center of the scene. We render the reconstruction from this novel trajectory (in dotted-orange) and use the rendering to identify co-visible pixels in the original source view (in orange). The source video and its masked variant are used to produce a self-supervised training pair for training CogNVS, our "3D-aware" video inpainting diffusion model.
obtain its 3D reconstruction Sare and odometry Carc from off-the-shelf SLAM frameworks [40]. As demonstrated in Fig. 3 sample N arbitrary camera trajectories in order create training pairs from 2D videos, described as follows.
We first obtain the "center" of the scene by considering the pixel at the optical center in the first frame of the given video, similar to prior work (92). We then construct a bounded region spherical coordinates, around the camera center of Cere- Within this region, end spherical coordinates of each new camera trajectory, and then again sample two intermediate
uniformly sample start and camera locations between the start and end spherical coordinates to interpolation. Camera poses are obtained by converting the spherical coordinates into euclidean space to get translations, and camera rotations the center of the scene. Using the four sampled camera poses, we do bicubic interpolation on the SE(3) manifold. This results in of smooth camera trajectories, {c,, J which are then used to construct the training pairs. With N trajectories, we can obtain "partial" novel-view renderings as,
smoothness during
obtained such that the look-at vector always points to
reov R(Sares Cp)
Between Veow and Vare, only a subset of primitives from Sare are co-visible. Let this subset be Drccn for the th trajectory. Then, partial renderings the source video are given by,
denoted by goov
Cov (SSOy Care)
s.l.
D ((Veoy
sre, Vn [1
the set of training pairs created by one monocular video. We repeat this for all 2D videos considered.
3.3 Test-time finetuning for target domain adaptation
At test time, to reduce domain gap arising due to different scene properties (lighting, appearance, motion) we use the source test video Vare to adjust the priors of CogNVS and create self-supervised finetuning pairs, D as described above. We therefore adapt the model weights 0 on-the-fly with M
5

---

gradient steps with step size as follows,
с e Voll co k, Zoond) <l1
where aond the latent of the n²t self-supervised training pair input. At the end of finetuning. we obtain the desired novel view Vuvs from CogNVS by using the partially observed novel-view, (S Cavs) the input conditioning, and running a reverse diffusion process.
Empirical Analysis
4.1 Experimental setup
Datasets We train CogNVS on four in-the-wild video datasets, SA-V (59). TAO [3], Youtube-VOS [86), and DAVIS [53]. We sample 3000, 3000, 4000 and 100 videos respectively from each of the datasets, giving us a total training video pool of Ra 10,000 videos. For pretraining, we randomly select new subsequence of 49-frames in every epoch and construct its training pairs. For benchmarking, we follow prior work (37 733 and use a combination of Kubric-4D, ParallelDomain-4D (73] and Dycheck (20). These have held-out test set of 20, 20 and 5 videos each. Note that our evaluation on Kubric-4D, ParallelDomain 4DD and Dycheck is zero-shot as the datasets are not seen during training. Since the Kubric-4D and ParallelDomain odometry for fair comparison to baselines. For Dycheck, we use MegaSAM for reconstruction and align the estimated point cloud with the groundtruth solve for scale ambiguity.
are synthetie, we use their groundtruth point clouds and
Baselines For Kubric-4D, we consider GCD 73 and Gen3C [61]. alongside a concurrent work, TrajectoryCrafter [92]. For ParalleIDomain-4D, we consider the same baselines except Gen3C, which only evaluates on Kubric-4D, as there is no open-source implementation available yet. For Dycheck, we consider recent work like Shape-of- benchmark test-time optimization approaches on Kubrie-4D and ParallelDomain-41 because their performance degrades catastrophically on novel views that are far apart from training views. For
(74), MoSca (37], CAT4D (8).. Note that we do not
more quantitative analysis of CAT4D, see appendix.
Metries For pixel-wise photometric evaluation, we adopt the widely used PSNR, SSIM, and LPIPS family of metrics for evaluating reconstruction quality via novel-view synthesis. We additionally benchmark the generation quality with FID and KID. This is in line with the benchmarking proposed in several diffusion-based view synthesis works
73 61 45 71
Implementation details During pretraining, we load the official CogVideoX-SBBIIV checkpoint and fully finetune all 42 transformer blocks. We use the Adam optimizer with 31 = 0.9, 82 0.95, and By 0.98, learning rate of 2 x 10e 5, and a batch size of 8 for 12,000 steps. To fit within 48GB VRAM, we employ DeepSpeed ZeRO-2 (58) partition model states across A6000 Ada GPUs in a distributed setting. Pretraining completes in approximately 3 days.
During test-time finetuning, of steps to 200 for shorter sequences (e.g., Kubric-4D) and 400 for longer ones (e.g.. DyCheck). For all experiments, we use an input resolution of x set the classifier-free guidance scale and run 50 inference steps. A single novel-view sequence generates in 5 mins on an A6000 Ada. We provide additional implementation details and evaluation protocols in appendix.
maintain the same optimizer and learning rate but reduce the number
4.2 Comparison to state-of-the-art
Kubric-4D and ParallelDomain-4D We first do zero-shot benchmarking of CogNVS on two synthetic datasets that come with high-fidelity dense depth and accurate camera odometry annotations. For fair comparison to all baselines, we use the depth and poses to backproject the given scene into a canonical coordinate frame. Given this scene, we generate self self-supervieed pairs for test-time finetuning. Upon inference (see Tab. [.. we find that CogNVS beats prior work on photometric evaluation with PSNR, SSIM, LPIPS, even when baselines are not evaluated zero-shot (GCD is trained on Kubric-4D and ParallelDomain-4D and Gen3C is trained on Kubric-4D). In Fig. 4 we show the plausible and realistic novel-views predicted by our method on both datasets as compared to the basclines. This is quantitatively demonstrated by better FID and KID scores. A concurrent

---

Input point
TraCroator
CogM/S (Ours)
Nee
Kubric-de
ParallelDomaain--e
Inout view
MeQaSAM
Shape-ofMMttin
Mosca
CAT4D
Tra(Cratter
CogNvS (Clun))
GT nowel view
Dycheck
Figure 4: We show a qualitative comparison with state-of-the-art approaches for dynamic novel-view synthesis Kubric-4D (top). ParallelDomain-4D (middle) and DyCheck (bottom). Note how reconstruction alone, either by groundtruth depth, MegaSAM [40). Shape of Motion (74). or MoSca 37 cannot synthesize complete novel view. Optimization based approaches like Shape Motion, and MoSca, blur the dynamic regions when fitting 4D representations. CAT4D (82. whose visuals taken from its project page due to unavailable code, struggles to generalize. TrajectoryCrafier [92] over-hallucinates the occluded regions and does not preserve geometry. GCD (73) performs well because was trained on Kubric-4D and ParallelDomain-41 Our method can instead produce photorealistic and 3D 3D-consistent rovel-views for the given scenes in a zero-shot manner with test-time finetuning, even starting from point cloud renders are incomplete and noisy (e.g., from MegaSAM for DyCheck). It is consistently able to synthesize sharp dynamic objects, which the other baselines struggle with. Please see the video in the appendix.
work, TrajectoryCrafter (84) performs competitively. We also evaluate the rendered visible scene structure from groundtruth depth, for establishing a lower bound on dynamic-view synthesis.
DyCheck We evaluate the performance of our method on real-world dataset of casually captured iPhone videos in Tab. First, note that since CogNVS can be applied on top of reconstructions from any method, we show two variants. Better initial reconstruction (in this case, with MoSca rather than MegaSAM allows for better dynamic view synthesis. Second, of approaches, our method produces the most visually plausible novel views, captured by drastically better FID and KID.

---

Table I: Comparison to state-of-the-ars art for dynamic view synthesis Kubric-4D and ParallelDomain- 4D. We find that our method, that operates zero-shot unlike Gen3C and GCD, achieves state-of-the-an
performance across majority of metrics. Note that Gen3C only evaluates Kubric-4D and there is no open-source code that would allow us to benchmark it on ParallelDomain-4D.
Kubric-4D LPIPS
ParallelDomain-4 4D
Method
PSNR
SSIM
FID 1
KID 1
PSNR
SSIM
LPIPS L
FID 1
KID L
GT
15.12
0.671
0.328
175.01
0.063
18.79
0.499
0.409
197.99
0.129
GCD
73
18.59
0.555
0.383
121.57
0.020
21.77
0.665
0.400
90.58
0.022
Gen3C
19.41
0.630
0.290
98.58
n/a
n/a
n/a
n/a
n/a
n/a
TrajCrafter 92
20.93
0.730
0.257
130.20
0.024
21.46
0.719
0.342
95.38
0.026
CogNVS
22.63
0.760
0.232
102.47
0.008
24.34
0.797
0.302
102.43
0.033
Table 2: Comparison to state-of-the-art for dynamic novel-view synthesis on Dycheck. First, we note that our method can be run on top of any reconstruction approach and the better the reconstruction (e.g., replacing MegaSAM with MoSca), the better the view synthesis. Second, we see that our method can achieve state-of-the-ard FID/ / KII scores because test-time optimization approaches [74] 37 40] result in blurry dynamic regions and cannot hallucinate new scene content, and completely feed-forward approaches [92]
cannot return precise geometry. Our method instead gets the "best of
both worlds".
Method
PSNR
SSIM 1
LPIPS
FID 1
KID 1
Mc 440
12.16
0.299
0.698
239.57
0.148
Shape-of-Ne of-Motion
74
15.30
0.476
0.494
164.29
0.073
MoSca 37
16.22
0.472
0.586
148.18
0.063
TrajCrafter 92
12.74
0.337
0.749
140.35
0.059
CogNVS (MegaSAM)
15.19
0,382
0.622
94.48
0,030
CogNVS (MoSca)
16.94
0.449
0.598
92.83
0.031
Third, note how TrajectoryCrafter [92], also based on video diffusion which was the second-best method on Kubric-4ID, is unable to handle the distribution shift in Dycheck (shallow held-of-view. close-up videos of moving objects) and fails to generalize. Whereas our method benefits from test-time finetuning and is able to adjust to any new dist-distribution at test-time. Other test-time optimization approaches lilape-of-Motion MoSca) do better as long evaluation views are close to training views, because there is only one distribution they need to fit to.
Runtime analysis We conduct a runtime analysis of CogNVS against other state-of-the-art art base- lines, as shown in Tab. E covering both stages of test-time optimization and rendering/inference. Specifically, we measure runtime by evaluating each method on the DyCheck evaluation set, with an average video length of 400 frames. Under our default hardware configuration. CogNVS requires 140 min for 400 steps of test-time fine-tuning on these long videos, followed by an additional minutes for rendering. Note that this duration be flexibly traded off with computational cost, which has recently been referred to as "inference- time scaling" in diffusion models [49]. For example, as shown in Fig. 8 (left) in the appendix, our method achieves 96% of its final performance with only half the fine-tuning steps.
4.3 Ablation studies
Effect of test-time finetuning We study the effectiveness of the test-time finetuning stage of our method. Row 2 vs. 3 in Tab. [ssow that proposed self-supervised finetuning is crucial for adaptation of CogNVS target video's distribution test-time. Once the self-supervised test-time finetuning stage is completed, our method yields outputs with high fidelity, showcasing improved precision, and more contextually and geometrically consistent 3D appearances, shown in Fig. 5
Effeet of large-scale pretraining We also study the usefulness of the large-scale pretraining stage with 2D videos from 4 training datasets. In this case, test-time finetuning alone with a self-supervised objective, cannot pull CogNVS out of the local minima it reaches without a good initialization. This is a common failure mode of many test-time optimization approaches that overfit to the training views
8

---

Table 3: Runtime analysis of test-time optimization and feed-forward approaches on DyCheck. We report the time taken for optimization test-time finetuning for all approaches in addition to the final inference rendering duration. In general, our optimization is faster than some test-time optimization approaches with the additional benefit of being able to inpaint/hallucinate unknown regions spatiotemporally consistent manner. Additionally, our inference is on par with other feed-forward methods.
Method
Optimization min
Rendering / Inference
McgaSAM 440
Real-time
MoSca
37
66 min
Real-time
Shape of Motion 74
237 min
Real-time
GCD 73
min
TrajCrafter
92
5 min 5 min
CogNVS
140 min
Table 4: We ablate our design choices of large-scale pretraining and test-time finetuning on three randomly chosen sequences from Kubric-4D test set. We find that no pretraining is detrimental to the performance of CogNVS, so much so that the PSNR drops by points, thereby devoiding CogNVS of data-driven robustness. Test-time finetuning is also essential as without the adaptation of CogNVS the test video, the performance terms of PSNR drops by 3 points.
Pretrain
Finetunc
PSNR 1
SSIM
LPIPS L
FID 1
KID
X
/
18.62
0.691
0.318
201.29
0.051
X
20.06
0.662
0.284
185.48
0.038
/
23.29
0.779
0.240
158.98
0.036
but default to rendering artifacts such blurry dynamic regions 37 3) and Fig. 5 that pretraining essential for data-driven robustness.
We show in Tab. (Row 1 vs.
Effect of reconstruction quality Although we touch upon how the initial reconstruction affects the quality of dynamic view synthesis, we describe in detail here. We create pertubed version of the Kubric-4D dataset, by obtaining reconstruction and odometry from MegaSAM and aligning the reconstruction to groundtruth to solve for scale ambiguity. Quantitative results show a ~ 3 points drop on PSNR and a consistently worse performance on all metrics with sub-optimal reconstructions and cameras. This also addresses the gap in the photometric performance (with PSNR, SSIM, LPIPS) of MegaSA.M-basee CogNVS on DyCheck. For the quantitative and qualitative analysis of this ablation, please the appendix.
Discussion
In this work, we focus on the problem of dynamic novel-view synthesis from monocular videos. Contrary to prior state-of-the-as that approaches this task from two extremes (cither test-time optimization for every new video from scratch, or large-scale feed-forward novel view synthesis) we propose simple setup that is the "best-of-booe worlds". We reformulate dynamic view synthesis as an inpainting task and lean on the success of reconstruction algorithms like MegaS that can estimate the structure and geometry of in-the-wild videos. We first train video inpainter, CogNVS, on pairs co-visible novel-view pixels and target novel-views via self-supervision on only 2D videos. At test-time, we propose finetune CogNVS, again via self-supervision, to adjust to the target video distribution. The proposed setup provides data-driven robustness with the large-scale pretraining of video inpainting model, and enhances 3D accuracy of the predictions with test-time finetuning.
Limitations CogNVS does not currently take advantage of open-source 3D and 4D video datasets and trains on a relatively small set of 2D videos. While the zero-shot evaluation can achieve better photorealistic performance than prior state-of-the-art even with this unprivileged training data, the model and its geometric inpainting capabilities can be enhanced by adding more training data from all three- 2D, 3D and 4D data sources. Additionally, the performance of CogNVS is dependent

---

Inou render
No pretrain
No fireture
Pretrsin Finelune
Figure 5: We qualitatively analyze the effect of pretraining and test-time finetuning. We note that without the data-driven robustness and generalization of pretraining (second column), CogNVS cannot hallucinate missing regions properly (e.g., inpainted region in first row is still black in top left corner). Finally, without test-time finetuning (third column), 3D consistency and adherence to scene lighting and appearance properties cannot be ensured (e.g. overall darker scene in second row, and output off by a few pixels at the bottom and right side of the image in first row, thereby inhibiting geometric consistency).
on the quality of dynamic scene reconstruction obtained from off-the-shelf structure from motion algorithms. When groundtruth structure and odometry is available, such as from ubiquitous depth sensors, CogNVS's performance can be increased. A limitation of the data generation pipeline is that the sampled arbitrary camera trajectories are not able to mimic the diversity of camera trajectories that are encountered in real-life, which is a bottleneck to the performance of CogNVS. A better strategy would be create "data-driven" trajectory sampler that samples from set of real-world trajectories observed in the training set.
Acknowledgements: This work was supported in part by Bosch and by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/Interior Business Center (DOI/IBC) contract number 140D0423C0074. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DOI/IBC, the U.S. Government.
References
[1] Ekin Akyürek, Mehul Damani, Adam Zweiger. Linlu Qiu, Han Guo. Jyothish Pari, Yoon Kim, and Jacob Andreas. "The Surprising Effectiveness of Test-Time Training for Few-Shot Learning". In: arXiv preprint arXiv:2411.00 1.07279 (2024). [2] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin Buutin-Bruuala,. and Pratul P Srinivasan. "Mip-nerf;; A multiscale representation for anti-aliasing neural radiance fields". In: ICCV. 2021.
[3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. "Mip-nerf 360: Unbounded anti-aliased neural radiance fields". In: CVPR. 2022.
[4] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. "Zip-nerf: Anti-aliased grid- neural radiance fields". In: ICCV, 2023.
[5] Léon Bottou and Vladimir Vapnik. "Local learning algorithms". In: Neural computation 4.6
(1992), pp. 888-900 900.
[6] Ang Cao and Justin Johnson. "Hexplane: A fast representation for dynamic scenes". In: CVPR. 2023.
[7] Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittlla, Shalini De Mello, Tero Karras, and Gordon Wetzstein. "Generative novel view synthesis with 3d-aware diffusion models". In: ICCV, 2023,
10

---

[8] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. "Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo". In: ICCV. 2021.
[9] Kaihua Chen, Deva Ramanan, and Tarasha Khurana. "Using Diffusion Priors for Video Amodal Segmentation". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2025.
[10] Liang Chen, Yong Zhang, Yibing Song. Ying Shan, and Linggiao Liu. "Improved test-time adaptation for domain generalization". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 24172 24182..
[11] Liang Chen, Yong Zhang. Yibing Song, Jue Wang, and Lingqiao Liu. "Ost: Improving gen- eralization of deepfake detection via one-shot test-time training". In: Advances in Neural Information Processing Systems 35 (2022), pp. 24597-24610.
[12] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. "Mysplat: Efficient 3d gaussian splatting from sparse multi-view images". In: ECCV. 2024.
[13] Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, and Deva Ramanan. "Tao: A large-scale benchmark for tracking any object". In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16. Springer. 2020, pp. 436-454.
[14] Kangle Deng, Andrew Liu, Yan Zhu, and Deva Ramanan. "Depth-supervised nerf: Fewer views and faster training for free". In: CVPR. 2022.
[15] Zhiwen Fan, Wenyan Cong, Kairun Wen, Kevin Wang. Jian Zhang. Xinghao Ding, Danfei Xu, Boris Ivanovic, Marco Pavone, Georgios Pavlakos, et al. "Instantsplat: Unbounded sparse-view pose-free gaussian splatting in seconds". In: arXiv:2203. 20309 (2024).
[16] Haiwen Feng, Junyi Zhang, Qianqian Wang. Yufei Ye, Pengcheng Yu, Michael J. Black, Trevor Darrell, and Angjoo Kanazawa. "St4RTrack: Simultaneous 4D Reconstruction and Tracking in the World". In: arXiv preprint arxiv:2504.1315 (2025).
[17] Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. "Dynamic neural radiance fields for monocular 4d facial avatar reconstruction" In: CVPR. 2021.
[18] Yossi Gandelsman, Sun, Xinlei Chen, and Alexei Efros. "Test-time training with masked autoencoders". In: Advances in Neural Information Processing Systems 35 (2022), pp. 29374- 29385.
[19] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. "Dynamic View Synthesis from Dynamic Monocular Video". In: ICCV. 2021.
[20] Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. "Monocular Dynamic View Synthesis: A Reality Check"". In: NeurlPS. 2022.
[21] Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Le Chen, Danhang Tang, and Ulrich Neumann. "Gaussianflow: Splatting gaussian dynamics for 4d content creation". " In: arXiv preprini arXiv:2403.. /2365 (2024).
[22] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin.
"Fastnerf: High-fidelity neural rendering at 200fps". In: ICCV. 2021.
[23] Yuxuan Han, Ruicheng Wang, and Jiaolong Yang. "Single-view view synthesis in the wild with learned adaptive multiplane images". In: ACM SIGGRAPH. 2022.
[24] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. "Text2Room: Extracting textured 3D meshes from 2D text-to-image models". In: ICCV. 2023, pp. 7909- 7920.
[25] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. "LRM: Large Reconstruction Model for Single Image to 3D". In: ICLR. 2024.
[26] Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. "Tri-MipRF: Tri-Mip Representation for Efficient Anti-Aliasing Neural Radiance Fields". In: ICCV. 2023.
[27] Yusuke Iwasawa and Yutaka Matsuo. "Test-time classifier adjustment module for model- agnostic domain generalization". In: Advances in Neural Information Processing Systems 34 (2021), 1), pp. 2427-2440.
11

---

[28] Hyeonho Jeong.
Jinho Chang, Geon Yeong Park. and Jong Chul Ye. "Dreammotion: Space- time self-similarity score distillation for zero-shot video editing". In: arXiv e-prints (2024), arXiv-2403.
[29] Hanwen Jiang, Qixing Huang, and Georgios Pavlakos. "Real3d: Scaling up large reconstruction models with real-world images". In: arXiv preprint arXiv:2406.08477 08479 (2024).
[30] Thorsten Joachims et al. "Transductive inference for text classification using support vector machines". In: Ieml. Vol. 99. 1999, pp. 200-209.
[31] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. "3d gaussian splatting for real-time radiance field rendering". In: ACM TOG (2023).
[32] Leonid Keselman and Martial Hebert. " "Approximate differentiable rendering with algebraic surfaces". In: European Conference on Computer Vision. Springer. 2022, pp. 596-614.
[33] Tarasha Khurana, Peiyun Hu, Achal Dave, Jason Ziglar, David Held, and Deva Ramanan. "Differentiable raycasting for self-supervised occupancy forecasting". In: European Conference Computer Vision. Springer. 2022, pp. 353-369.
[34] Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. "Point cloud forecasting as a proxy for 4d occupancy forecasting". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 1116-1122 1124.
[35] Tarasha Khurana and Deva Ramanan. "Predicting Long-horizon Futures by Conditioning on Geometry and Time". In: arXiv preprint arXiv:2404. 7754 (2024).
[36] Yao-Chih Lee, Zhoutong Zhang, Kevin Blackburr-Matzen, Simon Niklaus, Jianming Zhang, Jia-Bin Huang, and Feng Liu. "Fast View Synthesis of Casual Videos with Soup-of-Planess
In: ECCV. 2025.
[37] Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, and Kostas Daniilidis. "MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds". In: arXiv preprint arXiv:2405. 1444505..72.. /742/ (2024).
[38] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. "Spacetime gaussian feature splatting for real-time dynamic view synthesis". In: CVPR. 2024.
[39] Zhengqi Li. Simon Niklaus, Noah Snavely, and Oliver Wang. "Neural scene flow fields for space-time view synthesis of dynamic scenes". In: CVPR. 2021.
[40] Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski. and Noah Snavely. "Megasam: Accurate fast, and robust structure and motion from casual dynamic videos". In: arXiv preprint arXi::2412.04465 (2024).
[41] Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. "DynIBaR: Neural Dynamic Image-Based Rendering". In: CVPR. 2023.
[42] Zhihao Liang, Qi Zhang, Wenbo Hu, Lei Zhu, Ying Feng, and Kui Jia. "Analytic-splattin::
Anti-aliased 3d gaussian splatting via analytic integration" " In: ECCV. 2024.
[43] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. "One-2-3-45+4: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion". In: Proceedings of the IEEE/CVF conference on computer vision and puttern recognition. 2024, pp. 10072-10083.
[44] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. "One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization". In: Advances in Neural Information Processing Systems 36 (2023), pp. 22226-22246.
[45] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. "Zero- 1-to-: Zero-shot one image to 3d object". In: ICCV. 2023.
[46] Jiahao Lu, Tianyu Huang, Peng Li, Zhiyang Dou, Cheng Lin, Zhiming Cui, Zhen Dong, Sii-Kit Yeung, Wenping Wang, and Yuan Liu. "Align3R: Aligned Monocular Depth Estimation for Dynamic Videos". In: arXiv preprins arXiv:24122 2412.03079 (2024).
[47] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. "Dynamic 3d gaus- sians: Tracking by persistent dynamic view synthesis". In: 2024 International Conference on 3D Vision (3DV). IEEE. 2024, pp. 800-809.
[48] Zhen Lv, Yangqi Long, Congzhentao Huang. Cao Li, Chengfei Lv, Hao Ren, and Dian Zheng. "SpatialDreamer: Self-supervised Sterco Video Synthesis from Monocular Input". In: CVPR. 2025, pp. 1-12..
12

---

[49] Nanye Ma, Shangyuan Tong. Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang. Yandong Li, Tommi Jaakkola, Xuhui Jia, and Saining Xie. "Inference- Time Scaling for Diffusion Models beyond Scaling Denoising Steps". In: arXiv preprint arXiv:250.009732 (2025).
[50] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. "Nerf: Representing scenes neural radiance fields for view synthesis". In: ECCV. 2020.
[51] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. "Instant neural graphics primitives with multiresolution hash encoding". In: ACM Transactions on Graphics (ToG) 41.4 (2022). Pp. 1-15.
[52] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Brrtin-rraaala. "Nerfies: Deformable Neural Radiance Fields". In:
ICCV. 2021.
[53] Federico Perazzi, Jordi Pont-Tuset, Brian McWiliiams, Luc Van Gool, Markus Gross, and Alexander Horiung. "A benchmark dataset and evaluation methodology for video object segmentation". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 724-732.
[54] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung and Luc Van Gool. "The 2017 davis challenge on video object segmentation". In: arXiv preprint rrXiv:170400061 (2017). [55] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. "DreamFusion: Text-to-3I) using 2D Diffusion". In: ICLR. 2023.
[56] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. "D-nerf: Neural radiance fields for dynamic scenes". In: CVPR. 2021.
[57] Amit Raj. Srinivas Kaza, Ben Poole, Michael Niemeyer, Ben Mildenhall, Nataniel Ruiz, Shiran Zada, Kfrr Aberman, Michael Rubenstein, Jonathan Barron, Yuanzhen Li, and Varun Jampani. "DreamBooth3D: Subject- Driven Text-to-3D Generation". In: ICCV. 2023.
[58] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. "Zero: Memory Opti- mizations Toward Training Trillion Parameter Models". In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC). IEEE, 2020, pp. 1-16.
[59] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloc Rolland, Laura Gustafson, et al. "Sam 2: Segment anything in images and videos". In: arXiv preprint arXiv:2408. 007 14 (2024).
[60] Jiawei Ren, Cheng Xie, Ashkan Mirzaei, Karsten Kreis, Ziwei Liu, Antonio Torralba, Sanja Fidler, Seung Wook Kim, Huan Ling, et al. "L4gm: Large 4d gaussian reconstruction model". In: Advances in Neural Information Processing Systems 37 (2024), pp. 56828-56600 008228-6658.
[61] Xuanchi Ren, Tianchang Shen, Jiahui Huang, Huan Ling, Yifan Lu, Merlin Nimier-Davin Thomas MÄlaller, Alexander Keller, Sanja Fidler, and Jun Gao. "GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control", In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2025.
[62] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. "Zero 123++ a single image consistent multi-view diffusion base model". In: arXiv preprint wrXivv22310.5/11
(2023). [63] Assaf Shocher, Nadav Cohen, and Michal Irani. "">err-shot" super-resolution using deep internal learning". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2018, pp. 3113-11
[64] Colton Stearss, Adam Harley, Mikaela Uy, Florian Dubost, Federico Tombari, Gordon Wet- zstein, and Leonidas Guibas. "Dynamic gaussian marbles for novel view synthesis of casual monocular videos".. In: SIGGRAPH Asia 2024 Conference Papers. 2024.
[65] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. "Roformer: Enhanced Transformer with Rotary Position Embedding". In: Neurocomputing 568 (2024), p. 127063.
[66] Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang. Wen Liu, Zhenda Xic, and Yebin Liu. "Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior". In: ICLR. 2024.
13

---

[67] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. "Test-time training with self-supervision for generalization under distribution shifts". In: International conference machine learning. PMLR. 2020, pp. 9229-9248
[68] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. "Block-nerf: Scalable large scene neural view synthesis". In: CVPR. 2022.
[69] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. "Make-It-3D: High-fidelity 3D Creation from A Single Image with Diffusion Prior". In: ICCV. 2023.
[70] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. "Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training". In: Advances in neural information processing systems 35 (2022), pp. 10078-10093.
[71] Joseph Tung, Gene Chou, Ruojin Cai, Guandao Yang. Kai Zhang, Gordon Wetzstein, Bharath Hariharan, and Noah Snavely. "Megascenes: Scene-level view synthesis at scale". In: European Conference on Computer Vision. Springer. 2024, pp. 197-214.
[72]
Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. "Mega-NeRF: Scalable Con- struction Scage-Scale NeRFs for Virtual FlyTThru.gh"". In: CVPR. 2022.
[73] Basile Van Hoorick, Rundi Wu, Ege Ozguroglu, Kyle Sargent, Ruoshi Liu, Pavel Tokmakov, Achal Dave, Changxi Zheng, and Carl Vondrick. "Generative camera dolly: Extreme monocular dynamic novel view synthesis". In: ECCV. 2024.
[74] Qianqian Wang. Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. "Shape of motion: 4d reconstruction from a single video". In: arXiv preprint 2xXXv:2407.13766 (2024).
[75] Zhengyi Wang. Cheng Lu, Yikai Wang. Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. "Pro- lificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation". In: Advances in Neural Information Processing Systems 36 (2023), pp. 1406-8441.
[76] Daniel Watson, William Chan, Ricardo Martin Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. "Novel View Synthesis with Diffusion Models". In: ICLR. 2023.
[77] Daniel Watson, Saurabh Saxena, Lala Li, Andrea Tagliasacchi, and David J. Fleet. "Controlling space and time with diffusion models". In: ICLR. 2025.
[78] Ethan Weber, Aleksander Holynski, Varun Jampani, Saurabh Saxena, Noah Snavely, Abhishek Kar, and Angjoo Kanazawa. "Nerfiller: Completing scenes via generative 3d inpainting". In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 2024, pp. 20731-20741.
[79] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. "Chain-of-thougg" prompting elicits reasoning in large language models". In: Advances in neural information processing systems 35 (2022), pp. 24824-24444 248377.
[80] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. "MeshLRM: Large Reconstruction Model for high-Quality Meshes". In: arXiv preprini arXiv:2404. 12385 (2024).
[81] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. "4d gaussian splatting for real-time dynamic scene rendering".
In: CVPR. 2024.
[82] Rundi Wu, Ruiqi Gao, Ben Poole, Alex Trevithick, Changxi Zheng, Jonathan T Barron, and Aleksander Holynski. "Cat4d: Create anything in 4d with multi-view video diffusion models". (2024).
In: arXiv preprint arXiv:2411.
[83] Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. "BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering". In: ECCV. 2022.
[84] Zeqi Xiao, Wenqi Ouyang, Yifan Zhou, Shuai Yang, Lei Yang, Jianlou Si, and Xingang Pan. "Trajectory Attention for Fine-grained Video Motion Control". In: ICLR. 2025.
[85] Desai Xie, Sai Bi, Zhixin Shu, Kai Zhang, Zexiang Xu, Yi Zhou, Sören Pirk, Arie Kaufman, Xin Sun, and Hao Tan. "Lum-zero: Training large reconstruction models with synthesized data". In: arxiv preprint arXiv:2406.097: (2024).
14

---

[86] Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue. Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. "Youtube-voss "qquence-to-sequeee video object segmentation". In: Proceedings of the European conference on computer vision (ECCV). 2018,
pp. 585-601.
[87] Jiawei Yang, Jiahui Huang, Yuxiao Chen, Yan Wang, Boyi Li, Yurong You, Apoorva Sharma, Maximilian Igl, Peter Karkus, Danfei Xu, al. "STORM: Temporal Reconstruction Model for Large-Scale Outdoor Scenes". In: arXiv preprint arXiv:2501 00602 (2024).
[88] Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang. "Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting". In: ICLR. 2024.
[89] Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuamming Yang. Wenyi Hong, Xiaohan Zhang. Guanyu Feng, et al. "Cogvideox: Text-to-vides diffusion models with an expert transformer". In: arXiv preprint arXiv:2408. 06072 (2024).
[90] Meng You, Zhiyu Zhu, Hui Liu, and Junhui Hou. "NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer". In: ICLR. 2025.
[91] Hong-Xing Yu, Haoyi Duan, Junhwa Hur, Kyle Sargent, Michael Rubinstein, William
Freeman, Forrester Cole, et al. "Wonderlourney: Going from anywhere to everywhere". In:
CVPR. 2024, pp. 6658-6667.
[92] Mark YU, Wenbo Hu, Jinbo Xing, and Ying Shan. "TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Models", In: arXiv preprint 2533:2503.006638
(2025).
[93] Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang. Xiangjun Gao, Tien-Tsin Wong, Ying Shan, and Yonghong Tian. "ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis". In: arXiv preprint arXiv:2409.02044 (2024).
[94] Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Hsuan Yang. "Monst3r: A simple approach for estimating geometry in the presence of motion". In: arXiv preprint arXiv:2410..m 12.0850 (2024).
[95] Kai Zhang. Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. "Gs-Im: Large reconstruction model for 3d gaussian splatting". In: European Conference on Computer Vision. Springer. 2024, pp. 1-19.
[96] Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun. "Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion". In: arXiv preprint arXiv:2311.0100
(2023).
[97] Yizhou Zhao, Hengwei Bian, Kaihua Chen, Pengliang Ji, Liao Qu, Shao-yu Lin, Weichen Yu, et al. "Metric from Human: Zero-shot Monocular Metric Depth Estimation via Test-time Adaptation". In: NewrlPS, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All claims are verified by the experiments in Sec. 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations is in Sec. the main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
We don't have any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Most details are in the main paper in Sec. 3 and 4, some have been included in the supplement.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

reproducible, they Answer: [Yes]
Justification: We include a zip file of the code in supplement with some starter instructions. More in-depth details about running all code will be publicly released on GitHub after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co cc/
public/guiden /tt////Souummssioooo...
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 7/nips. //nips.cc/pubb httppss.ccpubllc/guudeeee Sttp:e////oooeSeeessiiisssssesee
ssioiioliiiiiiioccicc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All details provided exhaustively in Sec. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Computing error bars is computationally expensive for video diffusion models which what we are building upon.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
18

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Sec. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: It does.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This does not apply to our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work only generates novel views of the provided input video.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Citations are in the main paper, licenses included in the supplement.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our asset is the code and model, which we have already said is included in supplement with documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
We do not do any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We did not work with any human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not use an LLM for this purpose.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy httt:p///eeerrreeccc///ooerrrere for what should or should not be described.
22

---

Appendix
In this appendix, we extend our discussion of dynamic view synthesis in casual monocular videos. First, discuss the intricacies in the training and evaluation protocols adopted. This is followed by an in-depth ablation study on various design decisions in the proposed pipeline. Finally, we show more qualitative results, both on the considered benchmarks and on in-the-win whil examples.
A Implementation Details
Training pair details To generate self-supervised training pairs, we randomly perturb the source camera trajectory to create diverse camera paths. In the spherical coordinate system, we sample random elevations from [-15", 15 azimuths from [-30", 30"], and radius deviations from [-0.15, 0.15], followed by bicubic interpolation. This procedure enables flexible generation of training pairs across arbitrary camera trajectories. For pretraining, we use N camera views per training videos. For test-time finetuning, set N = 5 for DyCheck and N 9 for both Kubric-4D and ParallelDomain-4D. due to their wider novel-view gaps. When a video sequence exceeds Cog VideoX's default input length of 49 frames, each epoch. On DyCheck, we additionally apply a noise injection strategy to simulate real-world degradation on training pairs, analyzed in Section B.
randomly sample 49-frame subsequence in
Evaluation protocol For Kubric-4D and ParallelDomain we follow the official GCD evaluation protocol, using the 20-sequence sub-test set and an input resolution of 576 (width) x 384 (height). On DyCheck, we follow the official Mftpeonf-Motion and Mosca evaluation protocols, and we consistently report evaluation metrics an image resolution of 360 (width) x 480 (height) on the five standard test sequences: apple, block, paper- windmill, spin, and teddy. We render dynamic Gaussian representations from Mosca and Shape-of- Motoon with black background for fair comparison. Both methods optimize camera poses using the ground-trume -tuth novel views to improve photometric metrics; retain this step stay consistent with their original implementation. CAT4D, although diffusion- based, fits 4D-GS representation (with minor extensions) after synthesizing multi-view videos. When evaluating CogNVS on MegaSAM renders, append static background extracted from the full input video to better capture long-term context. The effectiveness of background stacking is validated in Section B. Also note that Shape-of-Motion and MoSca optimize for evaluation camera poses during evaluation using ground-truth novel view videos. Whether CAT4D adopts this step is unknown. We do not do this camera pose optimization test-time. Since the DyCheck evaluation sequences are more than 49-frames in length, we isolate the static scene regions and stack them in 3D across time. This accumulated background then rendered onto each frame which helps, large extent, "pre-inpaint" the static background regions using fused information from multiple 49-frame length sequences.
Table 5: Effect of reconstruction quality on Kubric-4D. We quantitatively evaluate CogNVS's performance with the use of two different reconstructions for Kubric-4D. Groundtruth depth gives an upperbound on view synthesis performance by CogNVS. Our first observation, perhaps unsurprisingly. that the quality of McgaSAM reconstruction subpar to that of the groundtruth. This difference quality is also translated to the novel-view synthesis task with CogNVS, where CogNVS used with groundtruth depth does 3 and 45 points better PSNR and FID respectively as compared to CogNVS used on top of MegaSAM.
Method
PSNR
SSIM 1
LPIPS $
FID $
KID 1
MegaSAM
12.73
0.299
0.644
280.62
0.164
GT CogNVS (MegaSAM)
15.12
0.671
0.328
175.01
0.063
19.62
0.621
0.313
147.83
0.033
CogNVS (GT)
22.63
0.760
0.232
102.47
0.008
B
Ablation Study
Effect reconstruction quality In Tab. 5 we show the effect of using different sources of reconstructions on the entire Kubric-4D evaluation set. Specifically, we compare the structure and
23

---

MegaSAM
GT
CogNVS (MegaSAM)
CogNVS (GT)
Figure 6: We show the effect using different qualities of reconstruction. Note that the groundtruth depth of the scene is perfect because it is derived synthetically. This re-rendered depth results in more realistic object placements in the scene as compared to the predictions using the depth from MegaSAM. This is because the MegaSAM depth is noisy at the object edges and therefore results in smeared objects in the novel view predictions.
Table 6: Effect of masking strategy on Kubric-4D. We study the effect of building CogNVS as an inpainting model using other masking strategies, specifically, random and tube masking (70.. We find that random masking is the least optimal as it does not mimic the rest-time scenario, tube masking does better, but our structured masking strategy is the best for the proposed structured inpainting task.
Mask
PSNR
SSIM t
LPIPS
FID
KID
Random
20.62
0.755
0.310
187.79
0.059
Tube
21.75
0.778
0.236
173.55
0.041
Ours
23.29
0.779
0.240
158.98
0.036
odometry from MegaSAM 44 and the synthetic depth groundtruth from Kubric-4D [73] We
find that both quantitatively and qualitatively (c.f. Fig. [6]. our pipeline benefits more from better reconstructions. This is because the quality of reconstruction directly affects the input to CogNVS, and if the input point cloud is noisy (e.g., smearing at the object borders), the prediction of the novel view also becomes inaceurate.
Ablation masking strategy Since CogNVS is an inpainting model, we ablate different masking strategies to train CogNVS on three sequences from Kubric-4D, instead of the proposed structured we use random and tube masking from VideoMAE (70) and apply
masking. In Tab. and Fig. 7 them with a 50% masking ratio on the input video sequences divided into 16 x 16 patches. We find that random masking is the least optimal at test-time. Tube masking is more amenable to the test-time inpainting pattern, which reflects as better photometric and generative metrics. Of all, structured masking obtained by rendering scene reconstructions into the novel views performs the best.
does not resemble the structured inpainting task
Ablation on test-time finetuning epochs Following the same data setup as above, we assess how the length of test-time finetuning affects the final prediction from CogNVS. In Fig. 8 (left), as expected we observe that the performance improvement the first few epochs is high (both in terms of PSNR and FID going from 0 to 50 epochs) and saturates as the number of epochs are increased further (up to 200).
Top-K evaluation Following the same data setup as above, we compute probabilistic PSNR and FID metrics for CogNVS's performance Kubric-4D in the form of Top-k metrics (where the best of k number is reported) in Fig. & (right). As we sample multiple modes from CogNVS's learnt distribution, the Top-k metrics for PSNR and FID become better and start to saturate near k=8.
Ablation on background stacking and noise addition We conduct an ablation on the MegaSAM reconstructions of DyCheck for the effect of static background stacking described in the previous section. In Tab. 7 and Fig. 9 we see that stacking the background on DyCheck provides a large in photometric performance. Secondly, we propose to add noise to dynamic object depths during training, especially for out-o--distribution data. This is essential as our creation of self- training pairs only masks out certain pixels from the source video which leaves no room for CVS to be able to real-world noise. To simulate real noise, at say object edges, estimate the
24

---

Random masking
Tube masking
Ours
Figure 7: We illustrate the different masking strategies considered, proposed by a prior work (70]. For random masking (left), the masked out patches are different in each frame of the input video. For tube masking (center), random set of patches is masked but this set is constant across multiple frames of the video. For our structured masking (right), we derive the mask by rendering visible scene reconstruction from the novel views.
24.0-
100
20.3
100
23.5
175
PENE
20.5-
21.0-
20.4
205
150
Spochs
We conduct ablations on the number of epochs used for test-time finetuning (left) and number of samples drawn from CogNVS for probabilistic evaluation (right). Both experiments suggest similar trends; performance improves with an increase in the number of finetuning epochs and increase in the number of samples drawn from our diffusion model. Performance saturates once a threshold reached.
Figure 8:
noise between (pseudo) groundtruth depth (coming from iPhone LiDARs or state-of-the-art depth estimator, say, MoGe) and the predicted depth (coming from a SLAM framework like MegaSA This estimated noise for the source pixels, is added to the visible scene reconstruction but in the ray direction the pixels visible in the arbitrary cameras. This results in noisy visuals that make CogNVS training more robust, especially to out-of-distributio cases. In Tab. 7 and Fig. demonstrate the improvements in performance by training CogNVS to inpaint in the presence of distracting noise artifacts.
Table 7: We quantitatively evaluate the effect of static background stacking and noise addition on DyCheck. Note that background stacking helps DyCheck because the video sequences are longer than 49-frames that CogNVS can handle. This gives us points performance boost PSNR. Adding real-world noise to dynamic objects helps make CogNVS robust to noise and therefore it reduces artifacts like smeared object edges, reflected a much lower FID metric.
Background stacking
Noise addition
PSNR
SSIM
LPIPS
FID 1
KID
X
11.55
0.304
0.848
197.93
0.204
X
14.27
0.352
0.737
180.67
0.171
/
14.30
0.354
0.740
156.83
0.141
Evaluation with masked metrics on DyCheck In addition to the metrics reported in the main paper, we also report masked photometric errors as proposed by a prior work (20). While this metric only evaluates the visible scene content and how any view-dependent changes were handled during novel-view synthesis, it does not encourage the generation of unseen scene regions. On this metric, CogNVS performs competitively compared to basclines.
25

---

No bkg stacking noise
No noise
bkg stacking noise
Figure 9: We visualize the 'apple' evaluation sequence from DyCheck for analysis of the effect of background stacking over time and noise addition strategy during training to simulate realistic in-the-wild scenarios. First (column I vs. 2), we see that for longer videos, stacking the static background from the entire input video helps accumulate multi-view cues about the static background. Second (column vs. 3), we see that due to the noise addition strategy during training, CogNVS is more robust to real-world noise patterns like smearing across object (in this case, apple) edges.
Table 8: We report masked perceptual quality metrics as proposed by prior work 20 This metric only evaluates the visible regions of the scene and does encourage generation unseen scene components, Note that our method performs competitively as compared to the baselines which only focus on modeling the visible scene content.
Method
mPSNR
mSSIM
mLPIPS
McgaSAMM
40
14.60
0.517
0.609
Shapc-of-Morte Motion
74
16.47
0.639
0.409
MoSca 37
17.82
0.635
0.507
CAT4D
82
17.39
0.607
0.341
TrajCrafter
92
13.60
0.518
0.663
CogNVS (MegaSAM)
15.35
0.549
0.557
CogNVS (MoSca)
17.33
0.607
0.530
Maximum novel view synthesis angle As demonstrated in Kubric 4D evaluation and several in-the-wild examples, CogNVS can plausibly generate novel views with up 90 degrees of variation (up, down, left, right) in general with single forward pass. Moreover, model can produce more extreme viewpoints by progressively generating new views conditioned on past generations - fairly straightforward and common strategy adopted by many generative models [24] 91 77). We illustrate such a progressively generated panoramic view in Fig. 10 Additionally, we want to point out that method is not able to generate table top views of object-centric scenes (inward facing 360 degree view) beyond 90 degrees of camera deviation, likely because such training data was not seen the model during pretraining.
Figure 10: We create panoramic view of the outdoor scene through progressive generation. Specifi- cally, we divide the 360 camera trajectory into four 90º segments of 49 frames each, reconstructing the previously generated views before reprojecting and inpainting the next 90" novel view. This qualitative analysis suggests that CogNVS can still generate 3D plausible and temporally coherent novel-views even in such extreme cases.
26

---

C
Qualitative comparison on evaluation datasets
Please see our webpage for videos in addition the qualitative visuals below.
Input view
point cloud
OCD
TreyCratter
CogNVS (Ours)
novel view
We show supplementary qualitative comparison Kubric-4D. Note that Trajectory Crafter able to generate reasonable background for the unseen scene regions, but is not able to inpaint the shadows / masks created by foreground objects. GCD is trained Kubric-4D so performs reasonably well but struggles to preserve the precise geometry. CogNVS achieves better performance compared to baselines and is the closest is geometric consistency to the groundtruth novel view.
Figure 11:
27

---

Input view
MegaSAM
Shape-of-Motion
Mosca
CAT4
TreyCrafter
CogNvS (Curs)
GT novel view
We show supplementary qualitative comparison DyCheck with CogNVS which surpasses the performance of all prior sfate-of-the-arr, Note that baselines either do not hallucinate the unseen regions in the novel-view (Shape-of-Ne -Shape-of-Motion, Motion, MegaSAM), show blurry dynamic regions (MoSca, CAT4D), or are not able to preserve the underlying geometry of the scene (TrajectoryCrafter))
Figure 12:
28

---

D Qualitative results on in-the-wild examples
Please see our webpage for videos in addition the qualitative visuals below.
Input
CogNVS
Input
CogNVS
Input
CogNVS
Inputt
CogNVS
Input
CogNvS
Figure 13: Qualitative results on in-the-wild examples. Part of 2.
29

---

Imput
CogNVS
Input
CogNVs
Imput
CogNVs
Input
CogNVs
Figure 14: Qualitative results on in-the-wild examples (static scenes included in last two rows). Part
2 2.
30

---

Input
CogNVS
Input
CogNVS
Input
CogNVS
Input
CogNVs
Imput
CogNVS
Input
CogNVS
Figure 15: Qualitative results on synthetic videos from SORA.
31


Reviews:
[{"review_id": "O6QPG50Okh", "reviewer": "Reviewer_DsAK", "summary": "The paper tackles the important task of dynamic novel-view synthesis. It introduces CogNVS, a video diffusion framework to inpaint hidden pixels in novel views and is able to generalize to novel videos via test-time finetuning. Experiments show state-of-the-art results across multiple benchmarks (Kubric-4D, ParallelDomain-4D, DyCheck) without requiring multi-view training data or ground-truth geometry.", "questions": "- Almost all human and animal characters in the generated videos look somewhat distorted (see the in-the-wild bear case, superman case, and man hiking case in the webpage). The face and body feel flattened. Is this a result of inaccurate MegaSAM estimation? \n- Given the randomly constructed pairs for training, what is the maximum rotation angle or the maximum portation of unseen pixels can this model generate robust outputs?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n- I like the idea of constructing self-supervised learning pairs on 2D videos to avoid costly 3D geometric data supervision. It is easier to scale up and learn the scene dynamics in a data-driven way.\n- Qualitative results show improved temporal consistency and 3D plausibility over baseline methods.\n\n### Weakness\n- Almost all human and animal characters in the generated videos look somewhat distorted (see the in-the-wild bear case, superman case, and man hiking case in the webpage). The face and body feel flattened. Is this a result of inaccurate MegaSAM estimation? \n- Given the randomly constructed pairs for training, what is the maximum rotation angle or the maximum portation of unseen pixels can this model generate robust outputs?", "was_revised": true, "final_justification": "The rebuttal addresses all of my concerns, therefore I raise my rating to accept.", "rebuttal": "We appreciate your comment that we solve an \"important\" task of dynamic novel-view synthesis by \"avoiding costly 3D geometric data\", building a pipeline that is \"easier to scale up\" and \"improves temporal consistency and 3D plausibility over the baselines\". We address your concerns below.\n\n**Q1: Distorted foreground objects**  \n\nThis is correct. The distortion that is visible in in-the-wild examples (e.g. left leg of a man hiking) is because of the distortion in the MegaSAM outputs. We will include visualization of renders from MegaSAM that are input to CogNVS to make this clear. Additionally, we note that there is no such distortion in the Kubric 4D objects as ground truth metric depth is available in this case.\n\n**Q2: Maximum rotation angle**  \n\nAs we show in Kubric 4D evaluation and some in-the-wild examples, CogNVS can plausibly generate novel views with up to 90 degrees of variation (up, down, left, or right). We also attempted to generate panorama views of scenes (outward facing 360 degree view) by dividing the camera trajectory into 49-frame chunks and progressively generating the new views conditioned on the past generations – a fairly straightforward and common strategy adopted by many generative models [R1, R2, R3]. Qualitative analysis suggests that CogNVS can still generate 3D plausible and temporally coherent novel-views even in such extreme cases. Unfortunately, visuals cannot be uploaded as part of this rebuttal but we have verified this on our end. \n\nAdditionally, we want to point out that our method is not able to generate table top views of object-centric scenes (inward facing 360 degree view) beyond 90 degrees of camera deviation, likely because such training data was not seen by the model during pretraining.\n\n[R1] Text2room: Extracting textured 3d meshes from 2d text-to-image models, ICCV23.\\\n[R2] Wonderjourney: Going from anywhere to everywhere, CVPR24.\\\n[R3] 4DiM: Controlling space and time with diffusion models, ICLR25."}, {"review_id": "oNBb6vvs9X", "reviewer": "Reviewer_N7JL", "summary": "This paper considers novel view synthesis from monocular videos. The method can be summarized by first creating a 3D reconstruction which is rendered to the target views followed by a fine-tuned video diffusion model which refine the rendering and in-paint the missing regions. The diffusion model is based on a pre-trained CogVideoX model and then fine-tuned on a large set of videos for which the authors present a method to create plausible occlusion masks from monocular videos which the diffusion model is conditioned on. Furthermore, in addition to the general pre-training, the diffusion model is refined in the same way for each video at test time. Experimental results show excellent qualitative results and quantitative results outperforming previous state-of-the-art for the task.", "questions": "- Please see weaknesses.\n- To my understanding, $V_{nvs}^{cov}$ is encoded by the VAE encoder $E$ to $z_{cond}$, where pixels that are not co-visible are simply zeros. Are missing regions simply treated as zero-values in $V$, and were alternatives such as providing masks as input to the diffusion models tested? The diffusion model denoises in the latent space which has a reduced spatial resolution, e.g. 1/8th. How are regions in 8x8 patches in the image treated when there are a few missing values only? Are they treated simply as black pixels among the co-visible pixels in the 8x8 patch and encoded as such by the VAE?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The main strength of the paper is that results look very good qualitatively, and better than the previous state-of-the-art which the method is compared to.\n- The key difference methodologically compared to prior work such as NVSsolver [84] or Gen3C (which both use 3D reconstruction+inpainting with video diffusion models for NVS for monocular videos) is to fine-tune for each test video. This is shown quantitatively to greatly improve the performance.\n- The data generation strategy is well designed to resemble point cloud renderrings by using co-visible regions in reconstructed videos, and it is shown to improve metrics compared to training with simpler masking heuristics. It should be noted though that the data curation method shares many similarities with TrajectoryCrafter [85], something which should be highlighted clearer in the paper.\n\nWeaknesses:\n- The method shares many similarities with TrajectoryCrafter [85], in the sense that a 3D reconstruction is created and a video diffusion model is conditioned on renderings of the 3D reconstruction in the target poses, and the data curation strategy used to create data for pre-training is similar. An interesting experiment would be to use TrajctoryCrafter and the proposed test-time fune-tuning. This would be interesting since the difference between CogNVS and TrajCrafter in Table 1 is in line with the difference between using test-time fine-tuning or not in the ablation in Table 3.\n- While the results are improved by the fine-tuning of the video diffusion model per test video, it also significantly increases the run-time. It would have been good to also include run-time comparisons of the different methods. The run time of the test-time fine-tuning is not stated directly but can be calculated. To my understanding the diffusion model sampling takes 5 minutes per video while the per-video diffusion model fine-tuning takes about 1,2/2,4 hours (short vs long videos) with 8 GPUs, for each video, so about 10-20 GPU hours for a single video. (inferred from lines 188-192 in the paper - 200 or 400 steps of test-time fine-tuning vs 12,000 steps for the pre-training which takes 3 days with 8 A6000 GPUs - please correct in the rebuttal if something is misunderstood or incorrect)", "was_revised": true, "final_justification": "The authors provided detailed answers to all questions. I keep my borderline accept rating and expect the authors to include all run-time results in the main paper.", "rebuttal": "We appreciate your comment that our data generation strategy is \"well-designed\", our proposed self-supervised finetuning of test videos is a \"key difference methodologically\" and that our method’s \"results look very good qualitatively, and better than the previous state-of-the-art\". We address your concerns below.\n\n**Q1: Test-time finetuning with TrajectoryCrafter**\n\nWe understand the similarity between the data generation strategy of our method and TrajectoryCrafter, but we want to clarify that unlike TrajectoryCrafter we train our model with *only 2D videos*. While both methods finetune a video generation model to an inpainting model, TrajectoryCrafter’s contribution is a “multiview” model that requires the input view as a conditioning. Our architecture is simpler as it simply inpaints in. Because of this we are able to finetune CogNVS at test-time, and it is *impossible* to do so with TrajectoryCrafter as you cannot have access to the groundtruth novel view at test-time. Finally, we want to point out that TrajectoryCrafter will be published at ICCV in October and therefore, should be regarded as concurrent work.\n\n**Q2: Runtime analysis with baselines**\n\nYour analysis of the runtime is correct – CogNVS takes 70/140 mins for test-time finetuning on short/long videos. However, per the graph in Fig. 8 (left) in supplement, note that our method achieves 96% of its performance in just half the number of finetuning steps. This duration can always be traded-off with the required compute. This has recently been referred to as “inference-time scaling” for diffusion models [R4].\n\nNevertheless, we provide a more detailed analysis of optimization and rendering/inference runtime in comparison to baselines below. These are average runtimes for the DyCheck evaluation set. We run finetuning on 8 A6000 GPUs, and for a 400-frame sequence, our method takes ~2 hrs for test-time finetuning and another 5 mins for final rendering. In general, our optimization is faster than some test-time optimization approaches with the additional benefit of being able to inpaint/hallucinate unknown regions in a spatiotemporally consistent manner. Additionally, our inference is on par with other feed-forward methods.\n\n| Method                       | Optimization | Rendering / Inference |\n| ---------------------------- | ------------ | --------------------- |\n| MegaSAM (CVPR '25)           | 9 min        | Real-time             |\n| MoSca (CVPR '25)             | 66 min       | Real-time             |\n| 4DGS (CVPR '24)              | 151 min      | Real-time             |\n| Shape of Motion (ICCV '25)   | 237 min      | Real-time             |\n| GCD (ECCV '24)               | -            | 2 min                 |\n| TrajectoryCrafter (ICCV '25) | -            | 5 min                 |\n| CogNVS                         | 140 min      | 5 min                 |\n\nTable A. Run time analysis of ours and baselines.\n\n[R4] Inference-time scaling for diffusion models beyond scaling denoising steps, arXiv 2501.09732.\n\n**Q3: VAE encoding of missing pixels**\n\nYes, we simply treat missing regions as zeros in the image-space. While there are other ways to encode missing pixels, e.g. providing explicit masks as you suggest (which TrajectoryCrafter also uses), it is not clear what advantage this brings. For instance, we find that TrajectoryCrafter is unable to inpaint unknown regions in some Kubric 4D examples even though they have been explicitly marked as unknown in their input masks. We instead take a data-driven approach – we rely on the model to infer where and how to inpaint missing regions based on context. \n\nThe latent space of the VAE can capture semantic information, including the context provided by neighboring pixels. Therefore, black pixels or missing pixels (even though both are represented by 0s) should have different latent embeddings. Similarly, 8x8 patches with few or many missing pixels should have different latent embeddings. In practice, we empirically observe that such patches are handled well by CogNVS without explicit masking."}, {"review_id": "mVqTK14mfO", "reviewer": "Reviewer_Tsgf", "summary": "This paper proposes a warping-then-inpainting paradigm for dynamic novel view synthesis from monocular video. Begin with reconstructed dynamic scenes (e.g., MegaSAM ), and novel views can be rendered with different camera trajectories. The main challenge lies in inpainting disoccluded regions in rendered novel views. To address this, the authors fine-tune the video generation model CogVideo for video inpainting, using constructed training pairs of masked and source images. Additionally, they further fine-tune the model on test scenes to reduce domain gaps and enhance inpainting quality. Experimental results demonstrate that the proposed method achieves superior performance in most cases.", "questions": "(1) Do the performance improvements come from more advanced 4D initialization and video generation models? As mentioned in the paper, the quality of the reconstructed scene influences the performance. Given the pipeline’s similarity to TrajCrafter (warping + novel view rendering + finetuned video inpainting), how does the proposed method quantitatively and qualitatively outperform it? Are gains attributable to the choice of CogVideo, warping strategy, fine-tuning approach, and reconstruction methods?\n\n(2) By warping the monocular video into novel views (some pixels are occluded and discarded) and then warping it back, we can also create the training pairs. This approach has been widely used in TrajCrafter and previous methods, such as [1]. I believe the underlying motivation for the warp-back strategy is the same as the design of this paper. Or what are the advantages of this paper's mask creation design? \n\n(3) About per-scene optimization. Will the finetuning on a specific scene cause the overfitting problem?\n\n[1] Single-View View Synthesis in the Wild with Learned Adaptive Multiplane Images", "limitations": "yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n(1) This paper proposes a simple pipeline for dynamic novel view synthesis from a monocular video, yielding good results.\n(2) This paper fine-tunes a video inpainting model, suitable for filling in disoccluded regions in novel view synthesis tasks. \n(3) The paper is well organized and easy to follow.\n\nWeakness:\n(1) Since the warping and then inpainting pipeline is widely used in view synthesis and 3D scene creation (e.g., Text2Room, etc.), it is not suitable to claim it as a contribution.\n(2) The generated content lacks 3D consistency. The output of this method is a camera-controlled video (cannot support free-view exploration), which means that disoccluded regions can be generated with different content when conditioned on a new camera trajectory in the second generation.\n(3) The technique contributions seem to be limited. Fine-tune a video generation model to inpaint the disoccluded regions in novel views without insightful designs during the training. Other components in this paper are built upon existing approaches.", "was_revised": true, "final_justification": "My concern will not be solved.", "rebuttal": "We appreciate your comment that we propose a \"simple pipeline\" yielding \"good results\". We address your concerns below.\n\n**Q1: Claiming inpainting as a contribution**\n\nThank you for your comment. We want to highlight that we *do not* claim \"warping then inpainting\" as a contribution (L52-55). In fact, we sufficiently cite other works which employ a similar inpainting strategy (c.f. MegaScenes, NVS-Solver, Gen3C, TrajectoryCrafter). We apologize for the oversight on Text2Room. We will gladly cite it and clarify these details further in the manuscript.\n\n**Q2: Multimodal generations lack 3D consistency**\n\nIt is correct that every time CogNVS is run with a new camera trajectory, it would result in a different plausible output. Having said that, we observe that test-time finetuning does increase the consistency of the inpainted regions with respect to the content of the visible scene. If 3D multi-view consistency across samples is desired, we suggest the following: CogNVS should be used iteratively by inferring and fusing the point clouds from all previous novel-view generations. This is a fairly straightforward and common strategy adopted by many generative models [R1, R2, R3]. We have verified this on our end by generating a panorama trajectory in chunks for an outdoor scene which looks spatiotemporally consistent. Unfortunately, visuals cannot be uploaded as part of this rebuttal.\n\n[R1] Text2room: Extracting textured 3d meshes from 2d text-to-image models, ICCV23.\\\n[R2] Wonderjourney: Going from anywhere to everywhere, CVPR24.\\\n[R3] 4DiM: Controlling space and time with diffusion models, ICLR25.\n\n**Q3: Limited technical contribution**\n\nAs elucidated in the main paper, we would like to reiterate that our main technical contributions are, (1) training a dynamic novel-view synthesis method *purely on 2D videos*, and (2) self-supervised finetuning with input videos at *test time* using our data generation strategy. These help our method achieve state-of-the-art performance on dynamic novel-view synthesis. Reviewer N7JL in their review regards these contributions as \"key differences\" that are methodologically novel as compared to prior works, and to the best of our knowledge, these contributions have not appeared in the literature before. \n\nTrajectoryCrafter is one similar and concurrent work (to be published at ICCV in October). While both TrajectoryCrafter and CogNVS finetune a video generation model to an inpainting model, TrajectoryCrafter’s contribution is a “multiview” model that requires the input view as a conditioning. Our architecture is simpler as it simply inpaints in. Because of this we are able to finetune CogNVS at test-time, and it is *impossible* to do so with TrajectoryCrafter as you cannot have access to the groundtruth novel view at test-time. Empirically, this test-time finetuning enabled by our architecture design manifests in improved performance metrics. See below for quantitative comparison.\n\n**Q4: Primary source of performance improvement**\n\nPlease note that as stated above, our primary source of improvement comes from the last stage of training – self-supervised finetuning of CogNVS at test time on a test video. For example, on Kubric 4D, TrajectoryCrafter’s PSNR↑/FID↓ is 20.9/130.2. Our pretrained CogNVS without test-time finetuning (with a similar CogVideoX backbone, warping strategy and groundtruth point cloud as TrajectoryCrafter) achieves a PSNR↑/FID↓ of 20.1/114.0, which is on par with the performance of TrajectoryCrafter out of the box. Upon test-time finetuning, CogNVS achieves a PSNR↑/FID↓ of 22.6/102.5, which surpasses the performance of TrajectoryCrafter. As we cover in the main text, test-time finetuning helps CogNVS adjust to the target scene of interest, and adapt the weights to suit the scene’s appearance, texture, lighting and 3D cues.\n\n**Q5: Mask creation design**\n\nThe underlying motivation for the warp-back strategy is indeed the same as TrajectoryCrafter and [1]. On another note, two distinctions of our work with respect to TrajectoryCrafter (as touched upon in **Q3**) are that, (1) we train CogNVS *purely on 2D videos*, and (2) use a self-supervised test-time finetuning phase to adapt CogNVS to the test video, which is not possible to do with TrajectoryCrafter's architecture design. Finally, we want to point out that TrajectoryCrafter will be published at ICCV in October and therefore, should be regarded as concurrent work.\n\n**Q6: Would finetuning cause overfitting?**\n\nYes, overfitting to the target scene is the intention with test-time finetuning. This is the same concept by which NeRFs or Gaussian Splats are overfit to a single scene at test-time. In practice, we do not find that this overfitting is detrimental to novel-view synthesis in any way. Regardless, the duration of test-time finetuning can always be adjusted to favor generating a more diverse scene at the cost of lower adherence to the target scene."}, {"review_id": "5FoTLLo2Ke", "reviewer": "Reviewer_PugC", "summary": "The paper proposes a framework for novel view synthesis given a monocular video. The proposed method utilizes rasterized point clouds in the novel views to condition a video diffusion model, which is trained to fill in the missing information. The main novelty of the paper is to design a training algorithm to utilize only monocular videos (without concurrent multiviews) to train the video diffusion model, utilizing covisibility. Test-time fine-tuning is also conducted and important to improve the quality of the results.", "questions": "I asked the questions above, as an outline: \n- Would having occlusion / defocus effect in the novel views and source videos affect the masks?  \n- How long does the test-time finetuning take (on how many gpus) and how does it compare to baselines? \n- Does the proposed method produce consistent multiview videos when sample the model multiple times with different trajectories, and does the test-time finetuning help? \n- What is the performance of the model before finetuning on DyCheck?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n- The method is simple and makes sense. Utilizing the known information (reconstructed point clouds from structure from motion) and fill in the missing information with a generative model is intuitive and effective.\n\n- The paper provides results on various types of scenes (real, synthetic, with/without scene dynamics). \n\n\nWeaknesses:\n\n- The paper may benefit from more discussions about why using co-visibility when the actual novel-view image and depth map is unknown can simulate the mask at source view if novel view were presented. Also in what condition does it hold?  If my understanding is correct, it seems to be that the proposed way to create the mask does not consider if the novel view has an occluder that additionally blocks point clouds from the source view -- e.g., imagine the novel view is partially behind a corner but the corner is not visible in the source view.  Also, the proposed method to create a mask seems to assume pinhole camera models -- if the camera has a sizable aperture, more information may be available (through defocus blur).   Since the self-supervision training by simulating masks is the main novelty of the paper, it seems that the discussion is a bit light in the paper. \n\n\n- The paper does not answer a few questions:  \n(1) Does the method produce consistent multiview videos?  If I sample twice the generative model using the same input monocular video but from two different camera trajectories, will the content of the unobserved areas in the two trajectories be consistent with each other?  Does the test-time finetuning improve the consistency between multiple views?  \n\n(2) How long does the test-time finetuning take, how many gpus does it need? \n\n(3) What is the performance of the model before test-time finetuning on DyCheck? \n\n(4) How long do the comparison methods take in Table 1 and Table 2 (including test-time optimization and rendering)? \n\n\n- While utilizing rasterized point clouds in the novel view is effective, it also makes the system depend heavily on the quality of the structure from motion (ie, depth map, camera pose and intrinsics).  The difference in quality can be observed easily between the Kubric and in-the-wild results. The in-the-wild results (eg, Superman and Friends) seem to have camera breathing that makes the characters’ faces look unnatural.  In comparison, when using ground-truth depth and camera parameters the Kubric results do not have the problem.  \n\n\n- The writing can be slightly improved.  When I was reading the paper, I was not sure whether the method requires concurrent multiview videos or not until I finished reading the method part (e.g, Line 39).  It was also not clear in Line 108 and 114 that whether g_src contains the entire information about the scene (ie, full 3D) or not.   Line 122 seems to have a type (V_src or V_novel). \n\n- It seems from Figure 5, the finetuning reduces the contrast of the results. Any reason why? \n\n- Would the proposed method to generate camera trajectories create cameras inside objects?", "was_revised": true, "final_justification": "I like the proposed method and find it interesting.  The rebuttal addressed my questions. I recommend the authors to include the discussions in the rebuttal to the revision.", "rebuttal": "We appreciate your comment that our proposed method for dynamic novel-view synthesis is \"simple\" and the \"test-time finetuning is important\". We also show \"results on various types of scenes (real, synthetic, with/without scene dynamics)\". We address your concerns below.\n\n**Q1: Consistent multi-view generation**\n\nEvery time CogNVS is run with a new camera trajectory, it would result in a different plausible output. Having said that, we observe that test-time finetuning does increase the consistency of the inpainted regions with respect to the content of the visible scene. If 3D multi-view consistency across samples is desired, we suggest the following: CogNVS should be used iteratively by inferring and fusing the point clouds from all previous novel-view generations. This is a fairly straightforward and common strategy adopted by many generative models [R1, R2, R3]. We have verified this on our end by generating a panorama trajectory in chunks for an outdoor scene which looks spatiotemporally consistent. Unfortunately, visuals cannot be uploaded as part of this rebuttal.\n\n[R1] Text2room: Extracting textured 3d meshes from 2d text-to-image models, ICCV23.\\\n[R2] Wonderjourney: Going from anywhere to everywhere, CVPR24.\\\n[R3] 4DiM: Controlling space and time with diffusion models, ICLR25.\n\n**Q2: Performance on DyCheck before test-time finetuning**\n\nWe include the performance on DyCheck before test-time finetuning below. For completeness, we also include a comparison on Kubric 4D and ParallelDomain 4D. Across all datasets, the takeaway is that test-time finetuning benefits all metrics by adapting the weights of CogNVS to suit the test scene’s appearance, texture, lighting and 3D cues. This is one of our primary contributions in this work.\n\n| Method                   | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FID ↓  | KID ↓ |\n| ------------------------ | ------ | ------ | ------- | ------ | ----- |\n| w/o test-time finetuning | 15.00  | 0.375  | 0.664   | 172.02 | 0.073 |\n| w/ test-time finetuning  | 15.18  | 0.382  | 0.622   | 94.48  | 0.030 |\n\nTable A. Test-time finetuning improves performances on DyCheck.\n\n| Method                   | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FID ↓  | KID ↓ |\n| ------------------------ | ------ | ------ | ------- | ------ | ----- |\n| w/o test-time finetuning | 20.14  | 0.654  | 0.260   | 114.00 | 0.012 |\n| w/ test-time finetuning  | 22.63  | 0.760  | 0.232   | 102.47 | 0.008 |\n\nTable B. Test-time finetuning improves performances on Kubric-4D.\n\n| Method                   | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FID ↓  | KID ↓ |\n| ------------------------ | ------ | ------ | ------- | ------ | ----- |\n| w/o test-time finetuning | 23.16  | 0.741  | 0.320   | 108.58 | 0.038 |\n| w/ test-time finetuning  | 24.34  | 0.797  | 0.302   | 102.43 | 0.033 |\n\nTable C. Test-time finetuning improves performances on ParallelDomain-4D.\n\n**Q3: Duration of test-time finetuning**\n\nCogNVS takes 70/140 mins for test-time finetuning on short/long videos and another 5 mins for final rendering. A detailed analysis of average using 8 A6000 GPUs for our method and all baselines on DyCheck evaluation set with 400-frame sequences is included below. In general, our optimization is faster than some test-time optimization approaches with the additional benefit of being able to inpaint/hallucinate unknown regions in a spatiotemporally consistent manner. Additionally, our inference is on par with other feed-forward methods.\n\n| Method                       | Optimization | Rendering / Inference |\n| ---------------------------- | ------------ | --------------------- |\n| MegaSAM (CVPR '25)           | 9 min        | Real-time             |\n| MoSca (CVPR '25)             | 66 min       | Real-time             |\n| 4DGS (CVPR '24)              | 151 min      | Real-time             |\n| Shape of Motion (ICCV '25)   | 237 min      | Real-time             |\n| GCD (ECCV '24)               | -            | 2 min                 |\n| TrajectoryCrafter (ICCV '25) | -            | 5 min                 |\n| CogNVS                         | 140 min      | 5 min                 |\n\nTable D. Run time analysis of ours and baselines.\n\nNote that per the graph in Fig. 8 (left) in supplement, our method achieves 96% of its performance in just half the number of finetuning steps. This duration can always be traded-off with the required compute. This has recently been referred to as “inference-time scaling” for diffusion models [R4].\n\n[R4] Inference-time scaling for diffusion models beyond scaling denoising steps, arXiv 2501.09732.\n\n**Q4: Effect of occlusion / defocus in novel-views on mask**\n\nDuring mask creation, we do not explicitly hallucinate new occluders in the novel view to reject more points from the point cloud during reprojection. However, this is not a bottleneck for training, or for inference, as the random nature of our training mask creation, in expectation, covers cases where an occluder would be visible in the novel view. Even if this was not the case, the inference from CogNVS would be unaffected as all that CogNVS does is pixel-level inpainting given a set of masked pixels. The resulting inpainted region can hallucinate new objects which act as occluders in the novel view. Note that all occlusion scenarios except “containment” are covered in the generated training data. As you notice correctly, for the scope of the paper, we only model a pinhole camera as the methods we use for preprocessing (e.g., MegaSAM) are also trained for a pinhole camera assumption. We will gladly add this discussion to the main paper.\n\n**Q5: Dependent on quality of structure-from-motion**\n\nThis is correct. Since our method takes as input a partial rendering of the input scene from the novel-view, its performance will increase with better structure-from-motion approaches which can give us better depth maps, camera poses and intrinsics. As you notice correctly, our generations are better when groundtruth depth and cameras are available, e.g., in Kubric 4D.\n\n**Q6: Improved writing**\n\nThanks for your suggestion. g_src only contains information about the scene from the source view visible in V_src (i.e., not full 3D). Line 122 in fact does not have a typo and it should be V_src because we use the source video as groundtruth during finetuning. Nevertheless, we will make the writing more clear as suggested.\n\n**Q7: Contrast is reduced after finetuning**\n\nThis can happen in some cases when the scene is out-of-distribution (e.g. the textured ground plane in this case) and finetuning is not enough for adaptation. We will highlight this as a limitation of our method more clearly in the main text.\n\n**Q8: Cameras inside objects**\n\nOur current setup does not allow generating cameras inside objects because the training datasets we consider do not cover these cases."}]

Author rebuttal:

Meta-review: This paper introduces a method for dynamic novel view synthesis from monocular video. 
It received one acceptance, two boardline acceptance and one rejection.
Reviewers praise the good qualitative results, self-supervised learning paradigm with 2D videos to avoid costly 3D geometric data supervision, well designed test-time finetuning strategy.
However, Reviewer Tsgf participates extensively in the discussion worries about the novelty of the whole pipeline and suggests that fine-tuning an inpainting model from monocular input is not new.
Both Reviewer N7JL and Reviewer Tsgf suggest the experiment of fine-tuning on TrajctoryCrafter. 
The authors explained in the rebuttal and discussion that why TrajectoryCrafter cannot be test-time finetuned on *dynamic* videos and the motivation to remove the input video as a condition.
The ACs had an in-depth discussion on this point and suggest the concern is a minor issue but the authors are urged to describe what are the unique aspects of the proposed research comparing to TrajectoryCrafter in the final version.
Decision: Accept (poster)