Pseudo-Generalized Dynamic View Synthesis from a Video
Authors: Xiaoming Zhao
R Alex Colburn
Fangchang Ma
Miguel Ángel Bautista
Joshua M. Susskind
Alex Schwing
Venue: ICLR 2024 poster (2024)
Keywords: dynamic novel view synthesis
generalized novel view synthesis
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Rendering scenes observed in a monocular video from novel viewpoints is a challenging problem. For static scenes the community has studied both scene-specific optimization techniques, which optimize on every test scene, and generalized techniques, which only run a deep net forward pass on a test scene. In contrast, for dynamic scenes, scene-specific optimization techniques exist, but, to our best knowledge, there is currently no generalized method for dynamic novel view synthesis from a given monocular video. To explore whether generalized dynamic novel view synthesis from monocular videos is possible today, we establish an analysis framework based on existing techniques and work toward the generalized approach. We find a pseudo-generalized process without scene-specific \emph{appearance} optimization is possible, but geometrically and temporally consistent depth estimates  are  needed. Despite no scene-specific appearance optimization, the pseudo-generalized approach improves upon some scene-specific methods.For more information see project page at https://xiaoming-zhao.github.io/projects/pgdvs.

Published as a conference paper at ICLR 2024
PSEUDO-GENERALIZED DYNAMIC VIEW SYNTHESIS FROM A VIDEO
Xiaoming Zhao 1.2 Alex Colburn' Fangchang Ma'
Miguel Ángel Bautista'
Joshua M. Susskind' Alexander G. Schwing
Apple
University of Illinois Airbana-Champan
x2 x2239illinois edu; edu; falexcolburn fangchang, mbaut mbautistamartinn jsusskind, ag ag.schwingggaappee...
ABSTRACT
Rendering scenes observed in a monocular video from novel viewpoints is a chal- lenging problem. For static scenes the community has studied both scene-specific optimization techniques, which optimize on every test scene, and generalized tech- niques, which only run a deep net forward pass on test scene. In contrast, for dy- namic scenes, scene-specific optimization techniques exist, but, to our best knowl- edge, there currently no generalized method for dynamic novel view synthesis from a given monocular video. To explore whether generalized dynamic novel view synthesis from monocular videos is possible today, we establish an analy- sis framework based on existing techniques and work toward the generalized ap- proach. We find a pseudo-genen generallzeed process without scene-specific appearance optimization is possible, but geometrically and temporally consistent depth esti- mates are needed. Despite no scene-specific appearance optimization, the pseudo- generalized approach improves upon some scene-specific methods. For more information see project page at https://iaammingzhaa http:///oooiiinniiaiiiiiiiiiiiee gittub.io/projects/pgdss
INTRODUCTION
Rendering (static or dynamic) 3D scenes from novel viewpoints is a challenging problem with many applications across augmented reality, virtual reality, and robotics. Due to this wide applicability and the compelling results of recent efforts, this area has received significant amount of attention. Recent efforts target either statie dynamic scenes and be categorized into scene-specifie or scene-agnostic, i.e., generalized, methods. -sene-specific methods for static scenes, like neural ra- diance fields (NeRFs) (Mildenhall et al., 2020) have been continuously revised and latest advances attain compelling results. More recently, generalized methods for static scenes have also been devel- oped and are increasingly competitive, despite the fact that they are not optimized on an individual scene (Yu et al., 2020; Wang et al., 2021; Suhail et al., 2022; Varma et al., 2023).
While this transition from scene-specific optimization to generalized methods has been in progress for static scenes, little work has studied whether generalized, i.e., scene-agnostic, methods are pos- sible for dynamic scenes. As dynamic novel view synthesis from monocular videos is highly ill-posed task, many scene-specific optimization methods (Li et al., 2021; Xian et al., 2021; Gao et al., 2021; Li et al., 2023) reduce the ambiguities of a plethora of legitimate solutions and generate plausible ones by constraining the problem with data priors, e,g., depth and optical flow (Gao et al., 2021; Li et al., 2021). With such data priors readily available and because of the significant recent progress in those fields, cannot help but ask natural question:
"Is generalized dynamic view synthesis from monocular videos possible today?"
In this work, we study this question in a constructive manner. We establish an analysis framework, based on which we work toward generalized approach. This helps understand what extent we can reduce the scene-specific optimization with current state-of-the-e data priors. We find
Work done as part of an internship at Apple. "Generalized" is defined as need of optimization fitting / training/ fine-tuning on scenes.

---

Published as a conference paper at ICLR 2024
Neriap
LPS5
TRT
Oyn
Ourr
Scoroiee Opirrattst Time 1GN0 TPu moant
(b)
Ours full rendering
Ground-truth
NSFF
DyniBaRt
Ours
Figure 1: (a) We find that is possible get rid of scene-specific appearance optimization for novel
view synthesis from monocular videos, which could take up to hundreds of GPU hours per video, while still outperforming several scene-specific approaches on the NVIDIA Dynamic Scenes (Yoon et al., 2020). (b) Qualitative results demonstrate the feasibility of generalized approaches as our rendering quality on-par or better (see details of the dragon balloon), even though we do not use any scene-specifie appearance optimization. The face is masked out to protect privacy.
"A ggeudo-generalizee approach, i.c.. no scene-specifie appearance optinization, is possible, but geometrically and temporally consistent depth estimates are needed." 2
Concretely, our framework is inspired from scene-specific optimization (Li et al., 2021; Xian et al., 2021: Gao et al., 2021; Li et al., 2023) and extends their underlying principles. We first render static and dynamic content separately and blend them subsequently. For the static part, we utilize pre- trained generalizable NeRF transformer (GNT) (Varma et al., 2023). As it was originally developed for static scenes, extend it to handle dynamic scenes by injecting masks, which indicate dynamic content, into its transformers. For the dynamic part, we aggregate dynamic content with the help of two commonly-used data priors, i.e., depth and temporal priors (optical flow and tracking), and study their effects.
Applying our analysis on the challenging NVIDIA Dynamic Scenes data (Yoon al., 2020) we find: 1) we are unable to achieve completely generalized method with current state-of- the- monocular depth estimation or tracking; but 2) with the help of consistent depth (CD) estimates, we can get rid of scene-specific appearance optimization, which could take up to hundreds of GPU hours per video, while still outperforming several acene-specifer approaches on LPIPS (Zhang et al., 2018) as shown in Fig. 1. Such CD estimates can come from affordable scene-specine specific optimization approaches (Luo et al., 2020; Kopf et al., 2020; Zhang et al., 2021b; 2022), taking around three GPU hours per video. able to achieve pseudo-generalize generalized dynamic novel view synthesis. We use the
All in all, word 'pseudo' due to the required scene-specific CD optimization, and 'generalized' because of no need for costly scene-specific appearance fitting. Note, such a mheudo-generalized approach does not 'cheat' by exploiting additional information, because CD estimates are also required by many scene-specific approaches (Xian al., 2021; Li al., 2023). We hence refer to CD as a sufficient condition for generalized dynamic novel view synthesis from monocular videos.
Moreover, we verify that our finding holds when consistent depth from physical sensors is avail- able, e.g., from an iPhone LiDAR sensor, as we still outperform several scene-specific baselines with respect to mLPIPS on the DyCheck iPhone data (Gao et al., 2022a). Note, sensor depth obvi- ates the need for scene-specific optimizations, enabling a completely generalized process.
To summarize, our contributions are:
1. We carefully study the use of current oftte-of-thearr monocular depth estimation and track- ing approaches for generalized dynamic novel view synthesis from monocular videos, sug- gesting assessment of the downstream usability for those fields.
2. We find sufficient condition and propose a pseudo-generalized approach for dynamic novel view synthesis from monocular videos without any scene-specific appearance op- timization. We verify that this approach produces higher-quality rendering than several scene-specific methods that require up hundreds of GPU hours of appearance fitting.
2 RELATED WORK
Scene-specific static novel-view synthesis. Early work on static scene-specific novel-view syn- thesis reconstructs a light field (Levoy & Hanrahan, 1996; Gortler et al., 1996) or layered repre- sentation (Shade et al., 1998). Recent works (Suhail et al., 2021; Attal et al., 2021; Wizadwongsa
We borrow the definition of "geometrically and temporally consistent depth estimates" from Zhang et al. (2021b): "the depth and scene flow corresponding points should be consistent over frames, and the scene flow should change smoothly in time".
2

---

Published as a conference paper at ICLR 2024
et al., 2021) extend such representations to be data-driven. Also recently, neural radiance fields (NeRFs) (Mildenhall et al., 2020) were proposed to represent the scene as an implicit neural net- work. uollow-up works further improve and demonstrate compelling renderrin results (Barron et al., 2021; 2022; Verbin et al., 2022; Barron et al., 2023). All these works focus on static scenes and re- quire scene-specific fitring, i.e. learned NeRF cannot be generalized to unseen scenes. Different from these efforts, we focus on dynamic scenes and aim for a scene-agnostic approach.
Generalized static novel-view synthesis. Prior works for scene-agnostic novel-view synthesis of static scenes utilized explicit layered representations, e.g., 3D photography (Shih et al., 2020) and multiplane images (Tucker & Snavely, 2020; Han al., 2022). Works like free view syn- thesis (Riegler & Koltun, 2020) and stable view synthesis (Riegler & Koltu, 2021) require a 3D geometric scaffold, e.g., a 3D mesh for the scene to be rendered. Recently, static scene-agnostic novel-view synthesis has received lot of attention in the community. Prior works tackle it with either geometry-basee approaches (Rematas et al., 2021; Yu et al., 2020: Wang et al., 2021; Chibane et al., 2021; Trevithick & Yang, 2021; Chen et al., 2021; Johari et al., 2022; Liu et al., 2022; Wang et al., 2022a; Du et al., 2023; Lin et al., 2023; Suhail et al., 2022; Varma et al., 2023) or geometry- free ones (Rombach al., 2021; Kulhánek et al., 2022; Sajjadi et al., 2022b;a; Venkat et al., 2023; Bautista et al., 2022; Devries et al., 2021; Watson et al., 2023). Differently, we focus on generalized dynamic view synthesis while the aforementioned approaches can only be applied to static scenes.
Scene-specific dynamic novel-view synthesis. Various scene representations have been developed in this field, e-g-, implicit (Pumarola et al., 2021; Li et al., 2021; Xian et al., 2021), explicit (Broxton et al., 2020; Fridovich-Keil et al., 2023), or hybrid (Cao & Johnson, 2023; Attal et al., 2023). Prior works in this area can be categorized into two setups based on the input data: multiview input data (Lombardi et al., 2019; Bemana et al., 2020; Bansal et al., 2020; Zhang et al., a0 Wang et al., 2022b; Li et al., 2022: Attal al., 2023; Wang et al., 2023a) and monocular input data (Park al., 2021b; Du et al., 2021; Tretschk et al., 2021; Gao et al., 2021; Park et al., 2021a: Cao & Johnson,
2023; Fridovich-Keil et al., 2023; Li et al., 2023; Liu et al., 2023; Fang et al., 2022; Yoon et al., 2020; Büsching et al., 2023). All these approaches require scene-specific optimization. Different from these efforts, we focus on a scene-agnostic method using monocular input data.
Generalized dynamic novel-view synthesis. To our best knowledge, few prior works target novel- view synthesis of dynamic scenes from monocular videos in a scene-agnostic way. In the field of avatar modeling, some methods are developed for generalizable novel-view synthesis for animatable avatars (Wang et al., 2023b; Kwon et al., 2023; Gao et al., 2022b). However, these approaches re- quire multiview input and strong domain geometric priors from human templates, e.g., SMPL (Loper et al., 2015). Differently, we care about dynamic novel-view synthesis in general, which domain- specific approaches are unable to handle. The concurrent work of MonoNeRF (Tian et al., 2023) tackles this challenging setting via 3D point trajectory modeling with a neural ordinary differential equation (ODE) solver. Notably, MonoNeRF only reports results for a) temporal extrapolation on training scenes; or b) adaptation to new scenes after fine-tuning, Both setups require scene- speccfcc appearance optimization on test scenes. Thus, it unclear whether MonoNeRF is entirely general- izable and can remove scene-specific appearance optimization. In contrast, we show improvements upon some scene-specific optimization methods without scene-specific appearance optimization.
3 METHODOLOGY FOR STUDYING THE QUESTION
3.1 OVERVIEW
We assume the input is a monocular video of N frames {(T,, ti, K2, Ed) where I; E [0, 1] [|HxW..3 is an RGB image of resolution H; x W, recorded at time 4 E IR with camera intrin- sics K; e 3x3 and extrinsics E, SE(3). Our goal is to synthesize a view Iug E [0, 1] x x3 given camera intrinsics Keg:
3x3 extrinsics Enge E SFE(3), and desired time lags E [N]
To study to what extent we can reduce scene-specific optimization and to develop a generalized ap- proach, we separate rendering of static and dynamic content as shown in Fig. 2. Note that this is also a common principle exploited in prior scene-specific works, either through explicitly decomposed rendering (Li al., 2021; Gao al., 2021; Li et al., 2023) or via implicit regularization (Xian al.,
*Cao & Johnson (2023); Fridovich- Keil al. (2023) operate on either multiview or monocular input.
3

---

Published as a conference paper at ICLR 2024
Stuti Rendering
Prior M' network
Data prior
Target fug
Fival Rendoring fas
Target canera (Fog)
Source 6d
Prior Piinetwck
Dota prior
Full-a
Source time (cl)
Detomrt Vng lasaey
Figure 2: Framework overview (Sec. 3.1). Our analysis framework separately renders static (Sec. 3.2) and dynamic content (Sec. 3.3). We focus exploiting depth and temporal data pri- ors, which are commonly utilized in scene-specific optimization methods.
2021; Yoon et al., 2020). Formally, we obtain the target rendering as follows:
Lug = Mgt 4gg. dy + (1 Migi.y) Ligt. st:
(1)
where Mgt dy E 10.13*M. MMx Wis is binary mask indicating the dynamic content on the target render- ing. lig. at Figt dy E [0, 1] Hy W are target static and dynamic content renderings respectively.
For the static rendering, we want to benefit from the compelling progress in the field of generalized static novel view synthesis. Specifically, to handle static content in dynamic scenes, we adapt the recently proposed generalizable NeRF transformer (GNT) (Varma et al., 2023) (see Sec. 3.2 for more). For rendering of dynamic content, we study how to exploit depth and temporal data priors (see Sec. 3.3 for more). This is inspired by the common practice of scene-specific approaches which distil priors from optimization losses, e.g., depth consistency (Xian et al., 2021; Li et al., 2021; 2023) and 2D-3D flow consistency (Gao et al., 2021; Li et al., 2021; 2023). Notably, to understand the limitations of data priors, intentionally avoid any (eaatue-bbased) learning for dynamic content rendering which would hide issues caused by data priors.
3.2 STATIC CONTENT RENDERING
In many prior scene-specific methods for dynamic novel view synthesis from monocular videos, re-rendering the static content often already requires costly scene-specific appearance optimization. Given the recent progress of generalized static novel view synthesis, we think it is possible avoid scene-specific appearance optimization even if the monocular video input contains dynamic objects. However, this non-trivial. Generalized approaches for static scenes usually rely on well-defined multiview constraints, e.g., epipolar geometry (Ma et al., 2004), which not satisfied in the pres- ence of dynamic content, e.g., due to presence new content and disappearance Therefore, modifications are required. For this, in this study, we carefully analyze and adapt pretrained GNT (Varma et al., 2023), transformer- approach that is able to synthesize high- quality rendering for unseen static scenes. In the following, we briefly describe GNT's structure and our modifications. Please see Sec. A for formal description.
existing objects.
GNT alternatingly stacks from source views and from samples on a ray respectively. Concretely, to obtain the RGB values for a given pixel location in the target rendering. GNT first computes the pixel's corresponding ray r with the help of Kut and Eig (Sec. 3.1). It then uniformly samples N₂ points {x],.. x on the ray. Given desired camera extrinsies Figt. GNT selects Neputial source views with indices
view and ray transformers successively aggregate information
Sppatill = ) Nauia In the p-th iteration (p (I,..., PJ), GNT first uses the p-th view transformer to update each sample's 's representation based on the raw feature from Sepatial as well as the feature from the previous (p- 1)-th iteration. It then utilizes the p-th ray transformer to exchange repre- sentations between (xf...., xVv, } along the ray. The final RGB values for the ray r are obtained by transforming the aggregated feature after P iterations.
3.2.1 GNT ADAPTATION
Our goal is to adapt pretrained GNT to render high-quality static content appearing in monocular video with dynamics. For this, we need to identify when GNT fails and what causes it. In order to build direct connection between the GNT-produced rendering and source views, inspired by MVSNet (Yao et al., 2018), we compute the standard deviation between features from all source views for each sample x and aggregate them along the ray. Fig. (i(c).(i) and (c). provide
4

---

Published as a conference paper at ICLR 2024
[X]
IN
Vie
0.3
Source view
Source view
0.1
Targe: klow
Samplie's projection Epipoiar 0/00/ trrroforner turaformer
Figure 3: Static content rendering (Sec. 3.2). Samples on a target ray (shown in (a)), project to epipolar lines in source views (see (b)). Vanilla GNT does not consider dynamic content, i... no utilization of dynamic masks (green arrows) in (b), and produces artifacts as shown in (c).(i, e.g., the rendering for static walls is contaminated by the foreground balloon. We find those artifacts correlate with the standard deviation of sampled features across source views as visualized in (c).(ii). Our proposal of using masked attention (Sec. 3.2.1) based on dynamic masks in GNT's view transformer improves the static content rendering as shown in (ii.. The face is masked to protect privacy.
the qualitative result when GNT is naively applied on a video with dynamics and the visualization of the standard deviation. A strong correlation between high standard deviations and low-quality renderings is apparent. In other words, GNT aims to find consistency across source views Sppatial- Due to dynamic content and resulting occlusions on some source views, GNT is unable to obtain consistency for some areas in the target rendering, resulting in strong artifacts. However, occlusions may not occur on all source views, meaning consistency can still be obtained on a subset of S
Following this analysis, we propose simple modification: use masked atlention in GNT's view transformer for dynamic scenes. Essentially, the sample x projects into potentially dynamic content of source view Iey, that source view will not be used in the view transformer's attention computation and its impact along the ray through the ray transformer will be reduced. As a result, the probability of finding consistency is higher. To identify potentially dynamic areas, compute a semantically-segmemmie mentation and tracking with optical flow (see Sec. C.4). Fig. (iic shows the result after modification. We observe that we successfully render more high-quality static content than before. We use the adapted GNT to produce the static rendering Igt. which is needed in Eq. (I). In an extreme case, if x is projected onto potentially dynamic content in all source views, it is impossible render static content and we rely on dynamic rendering for the target pixel.
dynamic mask for each source view based on single-frame semantic seg-
3.3 DYNAMIC CONTENT RENDERING
One of the key challenges in dynamic novel view synthesis from monocular videos (even for scene- specific methods) is arguably modeling of dynamic motion. This is a highly ill-posed problem plenty of solutions exist that explain the recorded monocular video. Via regularization, prior works encourage consistency between 1) attributes induced from learned motions; and 2) generic data priors. Two typical such priors are depth (Xian et al., 2021; Li al., 2021; 2023) and optical flow (Gao et al., 2021; Li et al., 2021; 2023). Notably, even scene-specific approaches heavily rely on such priors (Gao al., 2021; Li et al., 2021; Gao et al., 2022a). This encourages us to exploit generic data priors for generalized approach.
3.3.1 USING DEPTH AND TEMPORAL PRIORS
We utilize depth and optical flow for dynamic content rendering. For each frame 4, we com- pute a depth estimate D; E &.W. Note, for scene-specific optimizations, learned motions can only be supervised at observed time (tI....,ty t.... To produce renderings for temporal in- terpolation, i.e., figt & (f1.....ttx tN). previous works assume that motion between adjacent ob- served times is simply linear (Li et al., 2021; 2023). Inspired by this, for any desired time tigs. first find two temporally closest source views ligi max(ilt; fegss i {1,..., . N)) and ii = min(i] 2 lyg, E (I,...,N]} We then compute two associated point clouds from T,_ and with the help of dynamic masks, depths, and optical flows. Formally, we use point clouds P
O, M [1z] > 0, F.Flow F.Floggei (nn uz) }. fI (u1), Li Lift, (1z)))))))) M oy [u.]
(2)
where u1 E [0, kg
x [0, H and us W [0, H+) mark pixel coordinates. R
Ln fift((u) [u(u) F. K computes a 2D coordinate u's corresponding 3D point. Here
5

---

Published as a conference paper at ICLR 2024
O00
fie hig fet
Mga
Oytua flow
m
Truck
Light Ao
Figure 4: Dynamic content rendering (Sec. 3.3). (a) For two temporally closest source views (indices 'gg and &ig)), with the help of depth, dynamic mask, well as camera information, 2D dynamic content, e.g. human and balloon in this figure, is lifted into two point clouds. They are then connected based on optical flow between L.g and 1. (Sec. 3.3.1). Correspondences are highlighted as "rainbow" lines in the point cloud set P. We then obtain the point cloud Pogr for target time figt based on the linear motion assumption. (b) We utilize temporal priors of 2D tracking and lift visible trajectories to 3D with depth priors (Sec. 3.3.2). A complementary target point cloud is obtained based on the linear motion assumption as well and aggregated with Pigi- For example, for the shown 3D track, we linearly interpolated 3D positions from iigi and iigi +2 as they are temporally closest to dgg in the visible tracking (track on + + is invisible). The face is masked to protect privacy.
slightly abuse notation by omitting homogeneous coordinates. Di[u] denotes the interpolated depth value at coordinate u. Mi,ay [u] is the interpolated value at u from the mamaniically-segmennn dynamic mask Mi,d (Sec. 3.2. whose positive value denotes that belongs to potentially dynamic content. Besides, Flowsei (u1, U2) indicates that coordinates by optical flow and they pass cycle consistency check. Essentially, P refers paired two point clouds, lifting dynamic content from 2D to 3D (see Fig. 4.(a)).
on I and on I₃ are connected
For target time lig,, based on the linear motion assumption, we obtain dynamic content by interpo- lating between the two point clouds in P if
{ ligt lig
ligt
Pigt
X2
+
big 1
X1 (x) (x) P
1:1 t.
t
In the case of t. t.t tigs P contains two identical point clouds and we define Pig [xi](x,, X2) E PJ. Further, we conduct statistical outlier removal for Pig to mitigate inaccura- cies in depths and masks. We then render the point cloud Pgg with either SoftSplat (Niklaus & Liu, 2020) or point/mesh- based renderer to obtain the dynamic RGB Igt, dy and mask Mg. dy in Eq. (1). See Sec. C.3 for more details about the renderer.
3.3.2 USING MORE TEMPORAL PRIORS
source views render dynamic objects. To study the of information from more views we track a dynamic object's motion and integrate tracked points for final rendering based on the linear motion assumption. Please see Fig. 4.(b) and Sec. for more details.
Sec. 3.3.1
EXPERIMENTS
We aim answer: how far is the performance from scene-specific methods (Sec. 4.2): 2) to what extent can scene-specific optimization be reduced by which priors (Sec. 4.3).
4.1 EXPERIMENTAL SETUP
Dataset. We conduct quantitative evaluations on the NVIDIA Dynamic Scenes data (Yoon et al., 2020) and the DyCheck iPhone data (Gao et al., 2022a). The former consists of eight dynamic
'TiNeuVox needs consistent depth to compute 3D points of static background for the background loss: f3adb/1 wwad_hhperr.ee 1.9
https: httgggttthb....og wwt///st./teeeeeeeeeeeeee httph//wwwww////////////////aaae

---

Published as a conference paper at ICLR 2024
Table I: Results on NVIDIA Dynamic Scenes (13992 images). LPIPS is reported by multiplying
with 1000. Results for Row 1 1 2 are from Li et al. (2023). Results for Row 3 3-5 are reproduced by us (see Sec. E for clarifications). Row 6-1 and 6-2 evaluate on checkpoints after 20k and 40k steps respectively. DVS, NSFF. DynIBaR, TiNeu Vox, and ours utilize the same consistent depth (CD) estimates.* Though originally DVS and NSFF only utilize monocular depth, for fair compari- son, Li al. (2023) retrain DVS and NSFF with CD estimates instead. Appearance fitting time and hardware information are either transcribed from corresponding papers or provided by authors.
Appearance Fitting (h)
Appeorance Fitting Hardware
Full Image
Dynamic Area
Static Area
PSNR
SSIMI
LPIPS)
PSNR
SSIM
LPIPS)
PSNR
SSIM1 LPIPS
1-1 Vox (20k) (Pang al. 2022)
0.75
I V100
18.94
0.581
254,8
16.87
0.468
316.6
19:45
0,59]
245.7
1-2 ToNewVox (40k) (Fang al, 2022)
1.5
V100
18.86
0.578
244.2
16.77
0.464
292.3
19.38
0.588
237.0
2 Nerfies (Park et al., 202 la)
16
8 V100s
20.64
0.609
204.0
17.35
0.455
258.0
HyperNeRR (Park al., 2021b)
4TPU
20.90
182.0
17.56
0.446
4 DVS (Gao et al., 2021)
36
I V100
27.96
0.912
81.93
22.59
0.777
144.7
29.83
0.930
72.74
NSFF et al., 2021)
48
V100s
29.35
0.914
62.11
23.14
0.784
158.8
32.06
0.956
46.73
6 DyalBaR (Li et al., 2023)
48
A100s
29.08
0.952
31.20
24.12
0.823
62.48
31.68
0.971
25.81
Ours
Not Needed
26.15
0.922
64.29
20.64
0.744
104.4
28.34
0.947
57.74
Our rendering
Ground-truth
DVS
NSFF
DynlBaR
Ours
Figure 5: Qualitative results on NVIDIA Dynamic Scenes. Even without scene-speciff appear- ance optimization, the method still produces on-par or higher quality renderings than some scene- specific approaches: our background is aligned well with the ground-truth, our highlighted fore- ground is sharper than DVS and NSFF, and we do not miss the right arm as DynIBaR does in the bottom row. Faces are masked to protect privacy.
scenes captured by a synchronized rig with 12 forward-facing cameras. Following prior work (Li et al., 2023), we derive monocular videos by selecting frames at different time steps in a round robin manner. The resulting monocular video covers most timesteps. Note, this different from the setup in (Li et al., 2021; Gao et al., 2021) where only 24 timesteps from the full video are selected. We evaluate novel view synthesis with respect to the remaining 11 held-out views at each time step. The DyCheck iPhone data contains seven dynamic scenes captured by three synchronized cameras: one hand-held moving iPhone and two stationary cameras with a large baseline. Following Gao et al. (2022a), we evaluate on frames captured from the latter two viewpoints.
Preprocessing. For the NVIDIA dynamic scenes data, we use COLMAP (Schönberger & Frahm, 2016) to estimate camera poses. When consistent depth (CD) is needed, utilize dynamic video depth estimates (Zhang et al., 2021b). For the DyCheck iPhone data, all camera poses and depths are obtained from the iPhone sensors. For both datasets, we compute optical flow with RAFT (Teed dynamic mask with OneFormer (Jain et al., (Kirillov et al., 2023) (see Sec. C.4 for details).
& Deng, 2020) and obtain a comantically-seememee 2023) and Degmeet-Anything-MMone
Implementations. To render static content, we directly utilize GNT's pretrained weights. The possibility to re-use pretrained weights one advantage of our GNT adaptation (Sec. 3.2.1). We emphasize that the pretrained GNT is never exposed to the dynamic scenes we are evaluating on. Please see Sec. C for more implementation details.
Baselines and evaluation metrics. We compare to scene-specific methods to assess whether scene- specific optimization can be reduced. On both datasets we compare to Nerfies (Park et al., 2021a), HyperNeRF (Park et al., 2021b). NSFF (Li et al., 2021), and TiNeuVo: Vox (Fang et al., 2022). We run Vox on each video for 40k steps, double the default of 20k iterations. This gives TiNeu- Vox additional advantage. On NVIDIA Dynamic Scenes data, additionally compare to DVS (Gao et al., 2021) and DynIBaR (Li et al., 2023). We use PSNR, SSIM (Wang et al., 2004) and LPIPS (Zhang et al., 2018) on the full image, dynamic area, and static following prior work (Li
7

---

Published as a conference paper at ICLR 2024
Table 2: Results on DyCheck iPhone data (3928 images). T-NeRF, Nerfies, and HyperNeRE are supervised by metric depth from the iPhone LiDAR during optimization. TiNeuVox needs metric depth to compute loss on background 3D points during optimization. Appearance fitting time, hardware information, and results for row evaluate on checkpoints after 20k and 40k steps respectively. Due to the small effective multi-view factors (EMRs) (Gao et al., 2022a), all approaches (including ours) struggle.
are from Gao et al. (2022a). Row 5-1 and 5-2
Appearance Fitting (h)
Appearance Fitting Hardware
mPSNR
MSSIM ml PIPS!
1-1 Vox (20k) (Pang al,, 2022)
0.75
V100
14,03
0.502
0.538
1-2 TiNeu Vox (40k) (Pang al., 2022)
1.5
V100
13.94
0.500
0.532
2 NSFF (Li et al, 2021)
24
A4000s 22 A100s
15.46
0.551
0.396
T-NeRF (Gao et al., 2022a)
12
A4000s 2 A100s
16.96
0.577
0.379
Nerties (Park al., 2021a)
4 A4000s 2 A100s
16.45
0.570
0.339
5 HyperNeRF (Park et al,, 2021a)
72
A4000/ 1 A100
16.81
0.569
0.332
Ours
Not Needed
15.88
0.548
0.340
Grundeith
T-keer
Nerfec
Hyperbads
Grundenthh
Nerfles
Hypartwe:
Dure
Figure 6: Qualitative results on DyCheck iPhone data. Due to the small effective multi-view fac- tors (EMFs) (Gao et al., 2022a), all approaches struggle to produce high-quality renderings, which is reflected in Tab. 2. However, ours still produces on-par or better renderings than scene-specific ones: background aligns well with the ground-truth and foreground is sharper. Whitish areas indi- cate regions that are not considered during evaluation as they are invisible in training frames.
et al., 2023). On the DyCheck iPhone data, we also compare to T-NeRF (Gao et al., 2022a) and report masked PSNR (mPSNR), masked SSIM (mSSIM), and masked LPIPS (mLPIPS) following the dataset protocol (see Gao et al. (2022a) Appendix C). For all metrics, to avoid a result skewed by the scene with the most frames, we first compute the averaged metric over all frames for a scene, and then compute the mean across all scenes' averaged metrics.
4.2 COMPARISON TO SCENE-SPECIFIC APPROACHES
Tab. 1 and 2 summarize results on the NVIDIA Dynamic Scenes and DyCheck iPhone data. We compare our study-found best performing model, which utilizes consistent depth, to scene-specific methods. As expected, a generalized method does not improve upon all scene-specific baselines due to the huge disadvantage of no scene-specific appearance optimization compared with 100 to 384 GPU hours of fitting in scene-specific baselines. However, it is encouraging that a general- ized method can perform par even better than some baselines LPIPS, major indicator of perceptual quality (Zhang et al., 2018). Fig. 5 and 6 show qualitative results. Note, compared to baselines do not additional information the same consistent depth priors used dur- ing optimization of Tab. I's DVS, NSFF, DynIBaR, and Vox and Tab. 2's T-NeRF, Nerfies, HyperNeRR, and TiNeuVox.
4.3 ABLATIONS
We aim to understand what contributes to the achievement of reducing scene-specific optimizations presented in Sec. 4.2. For this ablate the NVIDIA dynamic scenes data (Tab. 3). Here only list the main ablations. Please refer to Sec. D for more extensive ablation studies.
Consistent depth is not enough. Although our best-performing model (row 3) utilizes consistent depth (CD), CD itself is not enough for high-quality rendering. This can be concluded by comparing row 1 vs. in Tab. 3, differing in the static rendering strategy. In row 1, we lift depths for static areas in each frame to point clouds and aggregate them for a 3D representation for static regions in the scene. Then a static image I, produced via paint-based rendering. In row 2, we use the vanilla GNT without any adaptation for static rendering. The main reason for inferior performance of row stems from the sparsity of depth and inaccuracies for background/static regions while data-driven and ray marching based GNT bypasses this issue.
NSFF is not supervised with metric depth due to its NDC formulation. See Sec. 5.2 "Benchmarked results" in Gao et al. (2022a) for more details.
8

---

Published as a conference paper at ICLR 2024
Table 3: Ablations on NVIDIA Dynamic Scenes (Yoon et al., 2020) (13992 images). LPIPS reported by multiplying with 1000. 'Masked Input' means masking out dynamic content in in- put frames; 'Masked Attention' is from Sec. 3.2.1; "CD' means consistent depth and ZoeD means depth from Bhat al. (2023); Pag Clean' marks statistical outlier removal (see Sec. 3.3.1): 'Track' indicates the method for dynamic tracking. Please see Tab. S4 for more ablations.
GNT
Masked
Inpur
Atlention
Masked
Depth
Pig Clan
Track
PSNRT
Fall Image
SSIM
LPIPS)
PSNRT
Dynamic Area
SSIMP
LPIPS]
PSNRT
Statie Area
SSIM
LPIPS]
21.10
0.717
211.2
19.35
0.688
167.0
21.78
0.726
212.8
25.86
0.919
69.70
20.86
0,943
63.46
0,922
64.29
20.64
104.4
0.947
x
ZoeD
21.15
0.814
142.3
15.93
0.179
233.5
23.36
0.854
129.9
5-1
y
x
CD CD
/
TAPIR
25.87
0.917
70.34
20.61
0.736
114.8
27.67
0.942
63.37
5-2
/
x
Colmecker
25.80
0.917
69.65
20.51
0.731
117.6
27.57
0.942
62.53
(a) ZoeD
(b) w/ ZoeD
(c) w/ TAPIR
(d) Ours
(e) Ground-truth
Figure 7: Qualitative results for ablations. (a) two views of a point cloud from aggregating frames's ZoeDepth: apparent inconsistency; (b) rendering with ZoeDepth: duplicated content; (c)
rendering with tracking with TAPIR: floating artifacts. Faces are masked to protect privacy.
[0] dog
(b) stroller
(c) train
Figure 8: Qualitative results DAVIS. Our approach is able to produce reasonable renderings various scenes. Highlighted artifacts of missing parts in foreground objects and a blurry background suggest future work on context-aware refinement. Faces are masked to protect privacy.
GNT adaptation is needed. This is verified by row 3's superiority over row 2. Note the difference for the static region. Please see Fig. 3.(c) for qualitative results on improved rendering static areas.
Consistent depth is necessary as verified by row 3 vs. 4 in Tab. and Fig. 7.(b). In row 4, we replace the consistent depth with predictions from ZoeDepph (Bhat et al., 2023), the state-of-the-art single- image metric depth estimator. We further mitigate depth scale-shift ambiguities by aligning 1) depth induced from the sparse point cloud from COLMAP's structure- concturr-frommmotion and 2) ZoeDepth's predictions on corresponding 2D projections of sparse point clouds. Nevertheless, we still observe inferior performance with Zepthp.h. We think the inconsistencies across frames shown in Fig. (.(a) the root cause, indicating room for monocular depth estimation improvements.
Simply using tracking not good enough as validated by row 4 vs. 5-1/5-2 and Fig. 7.(c). In Row 5-1/5-2, we utilize state-of-the-art point tracking approaches TAPIR (Doersch et al., 2023) and CoTrracker (Karaev et al., 2023) to aggregate dynamic content information from more frames. We notice consistent performance drops in Row 5-1 and 5-2. We hypothesize it is because inaccura- cies are accumulated by chaining tracking and depth estimates; and 2) our linear motion assumption too coarse when aggregating long-term dynamic information across several frames, indicating the need for sophisticated design to obtain accurate temporal information aggregation.
4.4 MORE QUALITATIVE RESULTS ON DAVIS
We render spatio-temporal interpolation videos on DAVIS (Perazzi al., 2016), where we COLMAP to estimate camera poses and consistent depth estimates (Zhang et al., 2021b). Qualitative results in Fig. 8 demonstrate the capabilities of our framework handle various scenes. Meanwhile, we observe artifacts for 1) missing parts the foreground caused by occlusions in source views or inaccurate depths; and 2) blurriness in the background stemming from out-of-boundary sampling in source views. These suggest context-aware inpaining could be a valuable future work.
5 CONCLUSION
We study the problem of generalized dynamic novel view synthesis from monocular videos. We find a generde-enrr method which avoids costly scene-speeific appearance training consistent depth estimates are used. Improvements in monocular depth estimation and tracking needed.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank Zhengqi Li for fruitful discussions and for providing rendered images from DVS and NSFF for evaluations in Tab. 1.
REFERENCES
Benjamin Attal, Jia-Bin Huang, Michael Zollhoefer, Johannes Kopf, and Changil Kim. Learning Neural Light Fields with Ray-Space Embedding. In CVPR, 2021. 2
Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf, Matthew O'Toole, and Changil Kim. HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sam- pling. CVPR, 2023.
Aayush Bansal, Minh Vo, Yaser Sheikh, Deva Ramanan, and Srinivasa Narasimhan. 4D Visualiza- tion of Dynamic Events from Unconstrained Multi-Vien Videos. In CVPR, 2020.
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. In ICCV, 2021. 3
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. In CVPR, 2022. 3
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields. In ICCV, 2023.
Miguel Ángel Bautista, Pengsheng Guo, Samira Abnar, Walter A. Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, Afshin Dehghan, and Joshua M. Susskind. GAUDI: A Neural Architect for Immersive 3D Scene Generation. In
NeurlPS, 2022.
Mojtaba Bemana, Karol Myszkowski, Hans-Peter Seidel, and Tobias Ritschel. X-Fields: Implicit Neural View-, Light- and Time-Image Interpolation. ACM TOG, 2020. 3
Shariq F. Bhat, Reiner Birkl, Diana Wofk. Peter Wonka, and Matthias Müller. ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth. arXiv, 2023. 9, 20
Michael Broxton, John Flynn. Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew Du Vall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Dehevec. Immersive Light Field Video with Layered Mesh Representation. ACM TOG, 2020. 3
Marcel Büschingg, Josef Bengtson, David Nilsson, and Märten Björkman. FlowIBR: Leveraging Pre- Trainrinnng for Efficient Neural Image-Basee Based Rendering of Dynamic Scenes. arXiv, 2023.
Ang Cao and Justin Johnson. HexPlane: A Fast Representation for Dynamic Scenes. In CVPR, 2023. 3
Anpei Chen, Zexiang Xu, Fugiang Zhao, Xiaoshuai Zhang. Fanbo Xiang, Jingyi Yu, and Hao Su. MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-Viens Stereo. In ICCV, 2021.
Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll. Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes. In CVPR, 2021. 3
Terrance Devries, Miguel Ángel Bautista, Nitish Srivastava, Graham W. Taylor, and Joshua M. Susskind. Unconstrained Scene Generation with Locally Conditioned Radiance Fields. In ICCV, 2021. 3
Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens Continente, Kucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. TAP-Vid: A Benchmark for Tracking Any Point Video. In NewrIPS Datasets Track, 2022. 16
10

---

Published as a conference paper at ICLR 2024
Carl Doersch, Yi Yang, Mel Vecerík, Dilara Gokay, Ankush Gupta, Yusuf Aytar, João Carreira, and Andrew Zisserman. TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement. arXiv, 2023. 9, 16
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In ICCV, 2021. 3
Yilun Du, Cameron Smith, Ayush Kumar Tewari, and Vincent Sitzmann. Learning to Render Novel Views from Wide-Baseline Stereo Pairs. In CVPR, 2023. 3
Jiemin Fang, Taoran Yi, Xinggang Wang. Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast Dynamic Radiance Fields with Time-Aware Neural Voxels. In SIG- GRAPH Asia, 2022. 7,7.
Sara Fridovich- Giacomo Meanti, Frederik Warburg, Benjamin Recht, and Angjoo Kanazawa. K-Planes: Explicit Radiance Fields in Space, Time, and Appearance. In CVPR, 2023. 3
Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic View Synthesis from Dy-
namic Monocular Video. In ICCV, 2021. 1, 2, 3, 5,
Han Gao, Ruilong Li. Shubham Tulsiani, Bryan C. Russell, and Angjoo Kanazawa. Monocular Dynamic View Synthesis: A Reality Check. In NeurlPS, 2022a. 5, 6, 7, 17
Xiangjun Gao, Jiaolong Yang, Jongyoo Kim, Sida Peng, Zicheng Liu, and Xin Tong. MPS-NeRF: Generalizable 3D Human Rendering from Multiview Images. TPAMI, 2022b. 3
Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. The lumigraph. In SIGGRAPH, 1996. 2
Yuxuan Han, Ruicheng Wang, and Jiaolong Yang. Single- View View Synthesis in the Wild with Learned Adaptive Multiplane Images. In ACM SIGGRAPH, 2022. 3
Jitesh Jain, Jiacheng Li, Man Chun Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. OneFormer:
One Transformer to Rule Universal Image Segmentation. In CVPR, 2023. 7, 16, 18
Mohammad Mahdi Johari, Yann Lepoittevin, and Franccois Fleuret. GeoNeRF: Generalizing NeRF with Geometry Priors. In CVPR, 2022. 3
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Rup- precht. CoTracker: It is Better to Track Together. arXiv, 2023. 9, 16
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything. arXiv, 2023. 7, 16, 19
Johannes Kopf, Xuejian Rong, and Jia-Bin Huang. Robust Consistent Video Depth Estimation. In CVPR, 2020.
Jonáš Kulhánek, Erik Derner, Torsten Sattler, and Robert Babuška. ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformers. In ECCV, 2022.
Young Chan Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. Neural Image-based Avatars: Generalizable Radiance Fields for Human Avatar Modeling. In ICLR, 2023. 3
Mare Levoy and Pat Hanrahan. Light field rendering. In SIGGRAPH, 1996. 2
Tianye Li, Miroslava Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, S. Lovegrove, Michael Goesele, Richard A. Newcombe, and Zhaoyang Lv. Neural 3D Video Synthesis from Multi-view Video. In CVPR, 2022. 3
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural Scene Flow Fields for Space- Time View Synthesis of Dynamic Scenes. In CVPR, 2021. 1, 2, 3, 4, 5, 7, 8,
Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. DynIBaR: Neural Dynamic Image-Based Rendering. In CVPR, 2023. 1, 2, 3, 4, 5, 7, 17, 19,21

---

Published as a conference paper at ICLR 2024
Kai-En Lin, Yen-Chen Lin, Wei-Sheng Lai, Tsung- Yi Lin, Yichang Shih, and Ravi Ramamoorthi. Vision Transformer for NeRF-Basee View Synthesis from a Single Input Image. In WACV, 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In ECCV, 2014. 18
Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf, Changil Kim, Yung- Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust Dynamic Radiance Fields. In CVPR, 2023.
Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural Rays for Ocelusion-aware Image-based Rendering. In CVPR, 2022.
Stephen Lombardi, Tomas Simon, Jason M. Saragih, Gabriel Schwartz, Andreas M. Lehrmann, and Yaser Sheikh. Neural Volumes: Learning Dynamic Renderable Volumes from Images. ACM TOG, 2019. 3
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: skinned multi-person linear model. ACM TOG, 2015. 3
Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf. Consistent video depth estimation. ACM TOG, 2020.
Yi Ma, Stefano Soatto, Jana Košecká, and Shankar Sastry. An Invitation to 3D Vision: from Images to Geometric Models. Springer, 2004.
James MacQueen. Some methods for classification and analysis of multivariate observations. In
Fifth Berkeley Symposium Mathematical Statistics and Probability, 1967. 17
Ben Mildenhall, Pratul P. Srinivasan, Matthew Taneik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, 2020. 1, 3
Simon Niklaus and Feng Liu. Softmax Splatting for Video Frame Interpolation. In CVPR, 2020. 6, 18
Simon Niklaus, Long Mai, Jimei Yang, and Feng Liu. 3D Ken Burns effect from single image.
ACM Transactions Graphics (TOG), 2019. 17
Keunhong Park, Utkarsh Sinha, Jonathan Barron, Sofien Bouaziz, Dan B. Goldman, Steven M. Seitz, and Ricardo 7,8
Nerfies: Deformable Neural Radiance Fields. In ICCV, 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B. Gold- man, Ricardo Martin-Brualla, and Steven M. Seitz. HyperNeRF: A sentation for Topologically Varying Neural Radiance Fields. ACM TOG, 2021b. 3,7
Repre-
Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Lue Van Gool, Markus H. Gross, and Alexander Sorkine-Horuung A Benchmark Dataset and Evaluation Methodology for Video Ob- ject Segmentation. In CVPR, 2016. 9
Albert Pumarola, Enrie Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In CVPR, 2021. 3
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision Transformers for Dense Prediction.
In ICCV, 2021. 19
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot 2022. 19
Transfer. TPAMI,
12

---

Published as a conference paper at ICLR 2024
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yet Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3D Deep Learning with PyTorch3.. arXiv, 2020. 18
Konstantinos Rematas, Ricardo Martin-Brualla, and Vittorio Ferrari. ShaRF: Shape-conditioned Radiance Fields from Single View. In ICML, 2021. 3
Gemot Riegler and Vladlen Koltun. Free View Synthesis. In ECCV, 2020.
Gemot Riegler and Vladlen Koltun. Stable View Synthesis. CVPR, 2021.
Robin Rombach, Patrick Esser, and Björ Ommer. Geometry-Free View Synthesis: Transformers and no 3D Priors. In ICCV, 2021. 3
Mehdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučié, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object Scene Representation Transformer. In NeurIPS, 2022a. 3
Mchdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs M. Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas A. Funkhouser, and Andrea Tagliasacchi. Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations. In CVPR, 2022b. 3
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-mottie revisited. In CVPR,
2016. 7
Jonathan Shade, Steven Gortler, Li wei He, and Richard Szeliski. SIGGRAPH, 1998. 2
Layered depth images. In
Meng-Li Shih, Shih- Yang Su, Johannes Kopf, and Jia-Bin Huang. 3D Photography Using Context- Aware Layered Depth Inpainting. In CVPR, 2020.
Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Light Field Neural Ren- dering. In CVPR, 2021. 2
Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable Patch-Based Neural Rendering. In ECCV, 2022. 1,3
Zachary Teed and Jia Deng. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In ECCV, 2020.
Fengrui Tian, Du, and Yueqi Duan. MonoNeRF: Learning a Generalizable Dynamic Radiance Field from Monocular Videos. In ICCV, 2023. 3
Edgar Tretschk, Ayush Kumar Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video. In ICCV, 2021.
Alex Trevithick and Bo Yang. GRF: Learning General Radiance Field for 3D Scene Representation and Rendering. In ICCV, 2021. 3
Richard Tucker and Noah Snavely. Single-Vien View View Synthesis With Multiplane Images. In CVPR, 2020. 3
Mukund Varma, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, and Zhangyang Wang. Is Attention All NeRF Needs? In ICLR, 2023. 1, 2, 3, 15, 16
Naveen Venkat, Mayank Agarwal, Maneesh Singh, and Shubham Tulsiani. Geometry-biased Trans- formers for Novel View Synthesis. arXiv, 2023.
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd E. Zickler, Jonathan T. Barron, and Pratul Srinivasan. Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields. In CVPR, 2022.
Dan Wang. Xinrui Cui, Septimiu E. Salcudean, and Jane Wang. Generalizable Neural Radiance Fields for Novel View Synthesis with Transformer. In arXiv, 2022a.
13

---

Published as a conference paper at ICLR 2024
Feng Wang, Sinan Tan, Xinghang Li, Zeyue Tian, and Huaping Liu. Mixed Neural Voxels for Fast Multi-view Video Synthesis. In ICCV, 2023a.
Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Lan Xu, and Jingyi Yu. Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time. In CVPR, 2022b. 3
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P. Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin- Bartlla,, Noah Snavely, and Thomas Funkhouser. IBRNet: Learning Multi-View Image-Based Rendering. In CVPR, 2021. 1,
Yiming Wang, Qingzhe Gao, Libin Liu, Lingjie Liu, Christian Theobalt, and Bao Xin Chen. Neu- ral Novel Actor: Learning a Generalized Animatable Neural Representation for Human Actors. TVCG, 2023b. 3
Zhou Wang, Alan Bovik, Hamid R. Sheikh, and Eero Simoncelli. Image Quality Assessment: from Error Visibility to Structural Similarity. TIP, 2004. 7
Daniel Watson, William Chan, Ricardo Martin-Brualle Jonathan Ho, Andrea Tagliasacchi, and Mo- hammad Norouzi. Novel View Synthesis with Diffusion Models. ICIR, 2023. 3
Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasom Suwa- janakom. NeX: Real-time View Synthesis with Neural Basis Expansion. In CVPR, 2021.
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time Neural Irradiance Fields for Free- Viewpinnn Video. In CVPR, 2021. 1, 2, 3, 5,
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang. and Long Quan. MVSNet: Depth Inference for Unstruc- tured Multi-view Stereo. In ECCV, 2018. 4, 16
Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. Novel View Synthesis of Dynamic Scenes With Globally Coherent Depths From Monocular Camera. In CVPR, 2020. 2, 3, 4, 6, 9, 17, 18, 20
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixeINeRF: Neural Radiance Fields from One or Few Images. In CVPR, 2020. 1,3
Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu. Editable fiee-viewpoint video using layered neural representation. ACM TOG, 2021a.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features Perceptual Metric. In CVPR, 2018. 2, 7,8
Zhoutong Zhang, Forrester Cole, Richard Tucker, William T. Freeman, and Tali Dekel. Consistent Depth of Moving Objects in Video. ACM TOG, 2021b. 2, 7,9,, 19
Zhoutong Zhang, Forrester Cole, Zhengqi Li, Michael Rubinstein, Noah Snavely, and William T. Freeman. Structure and Motion from Casual Videos. In ECCV, 2022. 2
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torrabba Scene Parsing through ADE20K Dataset. In CVPR, 2017. 18
14

---

Published as a conference paper at ICLR 2024
APPENDIX PSEUDO-GENERALIZED DYNAMIC VIEW SYNTHESIS FROM A VIDEO
This appendix is organized follows:
1. Sec. A provides a formal description for the static rendering discussed in Sec. 3.2;
2. Sec. describes the temporal priors mentioned in Sec. 3.3.2 in more detail;
3. Sec, gives implementation details;
4. Sec. presents additional ablations complementing Sec. 4.3;
5. Sec. E clarifies baselines's ' results NVIDIA Dynamic Scenes in Tab, 1;
6. Sec. F analyzes reasons for rendering artifacts.
A FORMAL DESCRIPTION OF STATIC RENDERING
Here we provide formal description of the GNT analysis and our adaptation discussed in Sec. 3.2.
A.1 GNT REVISITED
GNT (Varma et al., 2023) a cunsformer-based approach that synthesizes a high-quality rendering for unseen static scenes. It alternatingly stacks P = 8 view and ray transformers to successively aggregate information from source views and from samples on a ray respectively, Formally, to obtain the corresponding RGB values for given pixel location in the target rendering, GNT first computes its corresponding ray r with the help of Ki and Egg (Sec. 3.1). It then uniformly samples N, points {xi...., :......,jjy xxx on the ray. Hereafter, omit the superscript for simplicity. A pixel's 's RGB value is computed as C. fIORGB N En' Fx..P. where fIORGB is fully-connected layer. F Fx,P R" marks the d-dimensional feature for computed via the P-th ray transformer. Specifically, we have
Nr = TRay.D ((Fx..)) } ):m1) {F Fxx.ps (x.,
Vp E [I.
(SI)
where (-) the p-th ray transformer based on self-attention and ax.p is the attention value, i.e., Oxee E [0, 1] and Σm 0x,, (see discussion by Varma et al. (2023) in Sec. E). F xxip E RRd the d-dimensional feature from the p-th view transformer. Essentially, given desired camera extrinsics Egg (Sec. 3.1), GNT selects Napunial source views with indices Sepatil {sy} M Each sample x; extracts its own information F Fx.pp from Sspatial and the ray transformer aims to exchange such information along the ray. To obtain Fx.. we have
xp Tview, (Fx ( }
(oau)) ) Vp E (L....,PP.
(x j=1
(S2)
where (v denotes the p-th view transformer, and the first and second arguments are for query and key / value respectively. Idd 3 Fx..,; fRGB (I,,) [[(x; Eay, Ks;)) is x, 's feature vector ex- tracted from the j-th closest source view I,, Here FRGB (-) a feature extractor and II(x; E.,, K.,) x, marks the perspective projection of x; to Tyy- (b denotes interpolating value from a based on coor- dinate b. In essence, view transformer updates each sample's attached information based on the raw feature F Fx..,, as well as Fx.p-11 that considers communication along the ray in the previous iteration. Note, GNT initializes sample features as IRd 3 Fx..0 #1emmaxx( e(M x..s; ) where E. (m refers element-wise maximum pooling.
A.2 GNT ADAPTATION
Our goal for adapting pretrained GNT is to render static content appearing in monocular video in as high of quality as possible. For this, we need to understand when GNT fails and what causes the failure. In order to build direct connection between the GNT-produced rendering and source views,
15

---

Published as a conference paper at ICLR 2024
inspired by MVSNet (Yao et al., 2018), we compute the standard deviation between features from all source views for each sample and aggregate them along the ray using attention value from Eq. (S1):
Nr Oxip std (f KKey
( ( }
Nevis
Or.
(S3)
where Key is the projection of the "key" in the p-th view transformer and sta(-) is the standard deviation computation. This essentially mimics the volume rendering equation to aggregate point-wise attributes similar the discussion by Varma et al. (2023) in their Sec. E.
Fig. (.(c).(i and (ii) shows a qualitative example when naively applying GNT on a dynamic video and the visualization of or.-- We can easily observe strong correlation between high standard deviations and low-quality renderings. In other words, GNT essentially finds consistency across source views Sapeiial As a result, due to the dynamic content and its resulting occlusions in some source views, GNT is unable to find consistency for some areas in the target rendering, resulting in strong artifacts. However, occlusions may not occur on all source views, which means consistency can still be obtained on a subset of Squaial- Following this analysis, we propose a simple adaptation by using masked attention in GNT's view transformer. Concretely, we modify Eq. (S2) to
F xx.p TNiow.p (Fx..--1 {Fx., Fx... (Exi,ee > 0)} Nqquial
1=1
Vp E (1....,P), P}:
(S4)
where 1-) is an indicator function. Sx,,, = M2y [T(,,, Fay: K K3y)] is the interpolated value from a semaatically-segmentee dynamic mask Ma, which is computed from Le; with One- Former (Jain et al., 2023) and AntteentAntthing--Mee sentially, Sx..ay > O indicates that the sample x; projects into the potentially dynamic content on source view Tzy. Consequently, Fx., will not be used in the view transformer's attention computa- tion and its impact on the communication along the ray through the ray transformer will be reduced. As a result, the probability of finding consistency will increase. In an extreme case, if &x,.; > for all source views, it is impossible to render static content and we rely on dynamic rendering for the target pixel. As a result, we just return to the unmodified version that does not use any masked attention for x,. Fig. (ii) showcases the result after our modification. We can see that we suc- cessfully render more static content in high quality than before. We use the adapted GNT to produce the static rendering /gg. in Eq. (1).
(Kirillov et al., 2023) (see Sec. C.4). Es-
B DETAILS FOR RENDERING WITH MORE TEMPORAL PRIORS
We provide detailed description for utilizing more temporal priors mentioned in Sec. 3.3.2. In this work, we study the effect of generalized track-any-poie (TAP) approaches due to recent impressive progress (Doersch al., 2022; 2023; Karaev et al., 2023).
Specifically, we select Nemporal temporally-close source views. Their indices make up a set Siemporal- Given pixel coordinate in one of the source views's dynamic areas, we run 2D tracking and obtain a 2D temporal tracking trajectory:
T2D = 23.5) E Scampornt /.
(S5)
where j denotes the source view's index. u, E [0, W,] x [0, is the 2D coordinate on source view I5. 1/ (0, 1) is binary value indicating whether the tracking is visible. U/ = O means the target is occluded or bfuundayy on I,. e.g., Fig. 4.(b)'s tracking on +1 We lift this 2D temporal trajectory's 's visible parts to 3D with the Lift(-) function defined in Eq. (2). However, such a 3D temporal trajectory does not contain 3D position at lig. which we need to estimate. Intuitively, in absence of any additional information, the temporally closer the time to fig, the more likely the time's iss corresponding 3D position is similar to that at figt. Following this, we find two 3D positions on the trajectory whose time is closest to ligt and estimate 3D positions for by based on the linear motion assumption (similar to Eq. (3)). We repeat the above procedure for all pixels in the dynamic mask areas in all source views of Stemporal and obtain complementary point cloud Pige to Pigt- Combining Pegs and Pags we can conduct dynamic content rendering to obtain the dynamic RGB Tig. dy and mask My. dy in Fq.. (1).
16

---

Published as a conference paper at ICLR 2024
Figure S1: Effects of statistical outlier removal. For each scene, we show the dynamic rendering 1) without outlier removal on the left; 2) with outlier removal on the right. Apparently, statistical outlier removal produces much cleaner results. Faces are masked to protect privacy.
C IMPLEMENTATION DETAILS
Here provide implementation details: we describe 1) the spatial source view selection strategy for GNT in Sec. C.1; 2) how to conduct statistical outlier removal for Pig: in Sec. C.2; 3) the renderer for dynamic content in Sec. C.3; 4) how compute the semantically-segmeetee in Sec. C.4; 5) how to compute consistent depth estimates in Sec. C.5; 6) how to set depth range for GNT's samples the ray in Sec. C.6.
dynamic mask
C.1 SPATIAL SOURCE VIEW SELECTION FOR GNT
As mentioned in Sec. 3.2 and Sec. A. GNT selects N spattal source views with indices Sppatal { Niquial Here we provide the details about how to select Seputial- Throughout our experiments, we 10.
have Nspatial
For NVIDIA Dynamic Scenes data (Yoon et al., 2020), we choose the spatially closest source views. As mentioned in Sec. 4, NVIDIA Dynamic Scenes data is collected with rig of 12 cameras and the monocular video is composed by selecting one of the 12 cameras at different time steps in a round robin manner (Li et al., 2021: 2023). Therefore, for any target camera Ligt with time figr, only consider source views from time 1 12] to fa + 127. Without such a time window, Sepatial will always be chosen from frames captured the same camera. Concretely, we have Sspatial N
im - 12j Noputial and 8j E {l 12),..., Pl 12]), Vj. Here, NN(a, b, c) denotes the indices of the top e elements in b that are spatially closest to a.
Nquial NN( Euga, ( (E,)
+1
For the DyCheck iPhone data (Gao et al., 2022a), 1) we first cluster all cameras of a monocular video into Netuster (Netuser 2 Nopataa) groups with K-means (MacQueen, 1967) operating on posi- tions of the cameras. 2) detect Nppatial clusters that are spatially closest to the target camera Eg based on distances between cluster centers and the position of the target camera; 3) for each of the selected Nyuatill clusters, choose the member that temporally closest to ligr- Such design is motivated by the small effective multi-view factors (EMFs) (Gao et al., 2022a) of the dataset: we simply select the spatially closest source views, projected epipolar lines in all source views will capture very similar content, meaning consistency can be found almost everywhere along the epipolar lines. discussed in Sec. 3.2.1 and shown Fig. 3.(c), GNT's rendering relies on finding consistency across source views. However, 'Consistency almost everywhere' causes low-quality rendering. Our source view selection mitigates this situation. In this work, we use Netuster 40 in all our experiments.
C.2 STATISTICAL OUTLIER REMOVAL FOR Prer
Here we explain how to use statistical outlier removal to clean up Pys as mentioned in Sec. 3: 1) for each point E Pugs we average the distances between and its Nm. = 50 nearest neigh- bours, Le., we obtain Nm. = 50 distances per point and compute their mean to obtain |Pugll averaged distances in total; 2) we compute the median medianp, and standard deviation stdp,, of the |Pue averaged distances to understand their statistics; 3) for any point E Pig.. if its averaged distance to Nm nearest neighbours is larger than medianp, + ståpp we mark it as an outlier and experiments,
remove it. Throughout neural network to achieve similar outlier removal effects. On the contrary, we rely on point cloud's statistical properties. Fig. S1 illustrates the effect of outlier removal.
use 8 = 0.1. Niklaus al. (2019) utilizes learned
17

---

Published as a conference paper at ICLR 2024
Table S1: Ablations on rendering mechanism for dynamic content on NVIDIA Dynamic
Scenes (Yoon et al., 2020) (13992 images). LPIPS is reported by multiplying with 1000. Row 3 corresponds Tab. 3's corresponding row.
Dynamic Renderer
Full
Dynamic
Static
PSNR1
SSIM
LPIPS|
PSNR
SSIMI
LPIPS)
PSNRT
SSIM
LPIPS
3 SoftSplat
26.15
0.922
64.29
20.64
0.744
104.4
28.34
0.947
57.74
3-1
PointRenderer
25.61
0.916
65.67
19.97
0.707
110.6
27.87
0.944
58.66
3-2
MeshRenderer
25.78
0.919
65.63
19.77
0.709
111.7
28.32
0.947
58,40
Table S2: Ablations on rendering mechanism for dynamic content on DyCheck iPhone data (Yoon et al., 2020) (3928 images). LPIPS is reported by multiplying with 1000. Row 5 corresponds to Tab. 2's corresponding row.
Dynamic Renderer
mPSNR mSSIM mLPPPS.,
PointRenderer
15.88
0.548
0.340
5-1 SofiSplat
15.83
0.550
0.342
C.3 RENDERER FOR DYNAMIC CONTENT
Here we describe choices of dynamic renderers. We study three types of renderers:
Splatting based renderer. Using the target camera intrinsics Kur and extrinsics Egg, we obtain the pixel coordinates for each point in Pige via a projection onto the target image plane. Meanwhile, since Pur is obtained by lifting the temporally closest source views !. and + we also know the source view's pixel coordinate for each point in Put. As result, we can connect 1) pixels in the dynamic area of the target image; and 2) pixels in the dynamic area of the source view. This enables us to use SoftSplat (Niklaus & Liu, 2020) to transfer RGB values for dynamic content from the source views to the target view. To determine the weights of splatting, we use weights of 'importance metric' type (see Eq. (14) by Niklaus & Liu (2020)). In our experiments, we set the coefficient for the imporrance metric' to 100 throughout all experiments.
Point-based renderer. Since Pig is point-based representation, it intrinsically supports point- based rendering. In our experiments, we utilize PoTochhD'ss ' renderer with alpha-composition (Ravi al., 2020). We set the point radius to 0.01.
Mesh-based renderer. Since Pig is computed by lifting dynamic content pixels from temporally closest source views, cretely, we split every four neighboring pixels that form a rectangle into two triangles. The topology of triangles across 2D pixels be directly transferred to Puy.. We then utilize PyTorc33D's mesh rasterizer (Ravi et al., 2020) to render the image.
utilize the topology between pixels to convert Pig: into mesh. Con-
Tab. S1 and Tab. $2 present ablations of renders NVIDIA Dynamic Scenes and DyCheck iPhone data. We observe that the rendering quality doesn't 't differ much. We choose the best-performing one for each dataset, i.e., splatting-baser rendering for NVIDIA Dynamic Scenes and point-based rendering for DyCheck iPhone data.
C.4 COMPUTATION OF
DYNAMIC MASK
Here we provide details regarding the semantically-segmentee dynamic mask mentioned in Sec. 3.2.1 and Sec. 4.1.
The goal is to obtain binary-valued dynamic mask M; for the i-th frame. We can use single-frame semantic segmentation to compute potential dynamic masks Specifically, we pre-select a set of semantic labels as potentially dynamic objects, e.g., person. We provide the semantic labels we use in Tab. S3. After obtaining semantic segmentations, we treat areas with pre-selected labels as dynamic and everything else static. In this study, we OneFormer (Jain al., 2023) twice to cover more dynamic objects: one is trained on ADE20K (Zhou et al., 2017) while the other is trained on COCO (Lin et al., 2014). This is helpful since ADE20K and COCO have different
18

---

Published as a conference paper at ICLR 2024
Table S3: Semantic labels for potential dynamic categories.
Dataset
Categories
ADE20K
person, car, boat, bus, truck, airplane, dress/clothes, van, ship, toy, bag, motorbike, cradle, ball, animal, bicycle, fan, flag
COCO
person, car, motorcycle, airplane, bus, train, truck, boat, bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe, umbrella, ski, snowboard, skateboard, surfboard, tennis racket
semantic labels. We let M, E (0. H refer to the binary-valued mask of the i-th frame obtained from this procedure.
As we rely on the dynamic mask for lifting of dynamic objects to 3D, we need high-precision masks. However, if we naively set M; M,, find 1) single-frame semantic segmentation can miss dynamic objects in some frames while the same model detects the same object successfully in other frames; the mask boundary precision of pure semantic segmentation does not meet our needs.
To address the first issue, we develop a streaming- mask tracking approach to temporally prop- agate mask information via optical flow. We keep track of how many times a pixel is classified as be- ing dynamic via real-valued matrix M,. Specifically, 1) when i O, we have Mo float (Mo), where float denotes converting the binary mask to a floating- saating-point-bseed matrix. 2) when i > 0,
1) 2 6). where WaxP((-1)-+ warps mask tracking information from frame i- to frame i. Hence, (M
M. M1 AND ((aIP(-1)-
represents a normal- ized value in [0, 1]. A value of I indicates that this pixel is always treated as containing dynamic content in previous frames while value of denotes that the pixel always belonged static gions. We then use a threshold to convert the
matrix to a binary matrix. Such information is useful because taking the 'AND' operation with M, permits to correct mistakes made by single-frame semantic segmentation. The smaller 8, the higher the chance that the current M, will be corrected by the tracking information. In all experiments, we set 8 = 0.5. Finally, we update the tracking with M, M,-1 + float (M,).
To address the second issue, we combine M, computed above with segmentations predicted by the Anythen--nyyhingg- Model (SAM) (Kirillov al., 2023). Since SAM does not predict semantic labels, we rely on M, to relate SAM's predicted segmentations to semantic categories. Specifically, binury-valued mask MSAM, For each of the segmentations predicted by SAM we assess its overlap with dynamic areas M, For each segmentation from SAM, if more than 10% of its area overlaps with dynamic areas in Mj, we use a value of one in MSAM, highlight this area. We find using MSAM,; leads to more precise boundaries. As we use MsAM,; as the final dynamic mask, we update the tracking information with M, Mi-1 + float (MsAM,) instead of M, = M + f float(M) as stated previous paragraph.
create
C.5 CONSISTENT DEPTH ESTIMATION
Here we describe how to obtain consistent depth for monocular videos. We use work by Zhang (2021b) to complete this processing, but modify two steps: 1) we replace MiDaS (Ranftl et al., 2022) with DPT (Ranftl et al., 2021) for better quality; 2) we don't change camera poses. Originally, as a preprocessing, Zhang et al. (2021b) modify the provided camera translation to align 1) neural-network predicted depth; and 2) depth induced from sparse cructure-frommmootie point clouds. However, we wish to keep the camera poses unchanged for convenient rendering and evaluation. Therefore, following Li et al. (2023), we align the neural network predicted depth and depth obtained from SfM by computing a scale and shift value. We always use such precomputed scale and shift values during training.
(SIM)
19

---

Published as a conference paper at ICLR 2024
Table S4: Ablations on NVIDIA Dynamic Scenes (Yoon et al., 2020) (13992 images). Row numbers are aligned with Tab. 3. LPIPS is reported by multiplying with 1000. 'Masked Input' means masking out dynamic content in input frames; 'Masked Attention' is from Sec. 3.2.1; 'CD' means consistent depth and ZoeD means depth from Bhat et al. (2023); 'Pigt Clean' marks statistical outlier removal in Sec. 3.3.1; 'Track' indicates the method for dynamic tracking.
GNT Masked Input
Masked Aneetion
Depth
Pai Clean
Track
Full Image
Dynamic Area
Stutic Area
PSNR1
SSIM
LPIPS)
PSNR1
SSIM
I.PIPS)
PSNR
SSMM
IPPPS_
0
x
x
24.90
0.906
97.85
17.79
0.589
264.0
28.38
0.947
74.53
1-1
x
x
x
CD
x
21.08
0.713
220.2
19.61
0.674
192.6
21.68
0.723
220.2
1
CD
/
21.10
0.717
211.2
19.35
0.688
167.0
21.78
0.726
212.8
2-1
/
x
x
CD
x
25.44
0,909
82.27
20.66
0.717
141.6
26.83
0,934
74.36
2
x
CD
25.86
0,919
69.70
20.86
0.744
110.6
27.49
0.943
63.46
3-1
CD
x
23.75
0.895
96.62
19.08
0.679
182.3
24.99
0,923
85.72
3-2
/
x
CD
23.17
0.899
86.25
18.00
0.680
155.8
24.72
0.927
77.21
3.3
CD
24.93
0.914
72.41
19:31
0.717
126.5
27.10
0.941
64.60
3
/
CD
26.15
0.922
64,29
20.64
0.744
104.4 4
28.34
0,947
57.74
4
ZooD
21.15
0.814
142.3
15.93
0.479
233.5
23.36
0.854
129.9
5-2
5-1
/
CD
CD
CaTrucker
TAPIR
25.80
25.87
0,917
0.917
69.65
70. 34
20.51
20.61
0.731
0.736
117.6
114.8
27.57
27.67
0.942
0.942
62.53
63.37
C.6 DEPTH RANGE SELECTION FOR SAMPLES ON THE RAY
In general, we compute the depth range for the target view based on depth maps corresponding to spatial source views Sspaiill selected Sec. C.I with the help of camera extrinsics. Specifically, for each spatial source view's 's depth map, compute point cloud in the world coordinate system. Then we transform the point cloud into the target view's 's camera coordinate system. Repeating the above procedure for all spatial source views, we attain set of depth values Ds Seg for the target view.
For NVIDIA Dynamic Scenes data, use 0.8 x min D Squaa -gt the near value and 1.2 x Quant i ( -tgg: 90} as the far value, where Quant puantile(,g) indicates 9% quantile value of the first argument.
For the DyCheck iPhone data, we use (uantile(Dspaan 10} as the near value and Quanti prantile(D)) (D 90) as the far value. Additionally, since we have depth from the physi- cal sensor this dataset,, we simplify the rendering by setting the depth range for the static part to be centered around its depth in the target view. To be concrete, with the help of the dynamic mask computed from Sec. C.4, we can easily identify depth values for the static part from Ds computed above. We project the transformed point cloud in the target view to the image plane to pin down the corresponding pixels for the static part. We then just set the depth range for those pixels as interval of length 2 centered around the depth value in the target view.
D ADDITIONAL ABLATIONS
In Tab. S4 we provide more complete ablation studies complementary to Sec. 4.3.
Statistical outlier removal for Pug is useful. This consistently verified by row 1 vs. 1-1, row 2 vs. 2-1, and row 3-2 vs. 3-1, indicating again that the computed dynamic masks and depth estimates are not yet accurate enough for the task of novel view synthesis. See Fig. S1 for qualitative results.
Masked attention for GNT adaptation is superior. To obtain high-quality renderings of static areas from GNT operating on videos with dynamic content, we explore three ways: 1) directly masking dynamic objects in the source views (row 3-2): this results in apparent artifacts of black boundaries as shown in Fig. S2.(a): 2) applying both masked attention and masked input (row 3-3): this still causes the black boundary artifacts but mitigates it when compared to only using masked input. See Fig. S2.(b) for more details. 3) using only masked attention (row 3): this produces the most compelling rendering.
20

---

Published as a conference paper at ICLR 2024
(a) Masked Input
[b) Masked Input Attention
[c] Masked Attention
Ground-truth
Different strategies for adapting GNT. We explore various ways adapt GNT to dynamic scenes. (a) simply masking out dynamic content on source views results in strong artifacts like empty boundaries; (b) combining masked input and masked attention slightly alleviates the artifacts; (c) using masked attention solely produces the most compelling rendering.
Figure S2:
Table S5: Clarifications of baseline results on NVIDIA Dynamic Scenes (13992 images). LPIPS is reported by multiplying with 1000. "Reproduced" means the results are obtained from run- ning DynIBaR's official code with pretrained checkpoints or rendered images provided by authors. "DynIBaR's ' Mask" refers to DynlBaR's binary rendering masks, whose pixel values of true indicate that DynlBaR is able to produce content for that pixel. "?" denotes that it is unknown since not mentioned in the paper.
DynIBaR's Mask
Full Image
Dynamic
Statie
Approach
Type
PSNR1
SSIM
LPIPS)
PSNR1
SSIM
LPIPS,
PSNR SSIM LPIPS)
1-1
From paper (v))
?
26.05
0,913
81.00
22.63
0.778
144.0
1-2
From the paper (v3)
?
27.44
0.921
70.00
22.63
0.778
144.0
1-3
DVS
Reproduced
V
27.96
0,912
81.63
22.59
0.777
144.6
29.84
0,931
72.23
1-4
Reproduced
x
27.96
0.912
81.93
22.59
0.777
144.7
29.83
0.930
72.74
2-1
From the paper Reprodacdd
?
28.90
0.927
62.00
23.08
0.783
159.0
2-2
NSFF
/
29.34
0.934
61.90
23.15
0.784
158.7
32.07
0.956
46.35
2-3
Reproduced
29.35
0.934
62.11
23.14
0.784
158.8
32.06
0.956
46.73
3-1
From the paper Reproduced
?
30,86
0.957
27.00
24.24
0.824
62.00
3-2
DynlBaR
V
30.77
0.957
26.88
24.21
0.824
61.73
34.21
0.976
20.98
3-3
Reproduced
X
29.08
0.952
31,20
24.12
0.823
62.48
31,68
0.971
25.81
E CLARIFICATION FOR BASELINE RESULTS ON NVIDIA DYNAMIC SCENES
Here we elarify the difference for baseline results in Tab. and the original paper (Li et al., 2023). We transcribe baseline results from Li al. (2023). However, we find:
1) DynlBaR is unable to render full images in some cases. The official code only evaluates areas of renderings where DynlBaR produces content. It hence does not necessarily evaluate on full images. We argue this favors DynlBaR, especially for "full image" and "static area" evaluation. In contrast, for "dynamic area" evaluation, the dynamic mask already eliminates the need for evaluation on areas that are not rendered by DynlBaR, Tab. S5 Row 3-2 vs. 3-3 verifies this hypothesis: the difference between evaluating with or without DynlBaR's rendering mask more apparent on "full image" and "static area" evaluations. Note, for DVS and NSFF, the results are maintained no matter whether we use DynIBaR's mask or not (see row 1-3 vs. 1-4 and 2-2 vs. 2-3). This is because DVS and NSFF are able to render full images. Based on this finding, in Tab. 1, we use Row 3-3 from Tab. S5 for DynIBaR results since Tab. conducts evaluation on full images.
2) There are discrepancies between quantitative results of full image evaluations for DVS from different public versions of Li et al. (2023): row 1-1 and 1-2 in Tab. S5 are for results from version 1/2 and 3 respectively." We find that reproduced results in row 1-3's "full image" evaluation section do not match any of the public versions entirely. Specifically, our reproduced PSNR (row 1-3) aligns well with the corresponding PSNR from v3 (row 1-2) while our reproduced SSIM and
httppp/////ttth...c./oo///////// Tvl: https///www..ooo///////2/1111111 v2: https://arxiv. 2211. 20111110022...ttthhttt/e////rorr 11182223.....
21

---

Published as a conference paper at ICLR 2024
[a] Source Dynamic Rendering
(b) Dynamic Content
Image
Ground-truth Rendering
(d) Static Rendering from Adapted GNT
(e) Dynamic Rendering w/ SoftSplat
(t) Dynamic Rendering w/ Point Renderer
lg) Dynamic Rendering w/ Mesh Renderer
(h) Final Rendering: (d) (e)
(i) Finall Rendering: (4) (t)
(i) Final Rendering: (d) (e)
Figure S3: Artifact analysis. Due to occlusions in the source image (a), our pseudo-generalle generalized approach unable to inpaint those highlighted missing areas, resulting artifacts highlighted in (h) (i) (e) - (g) display the dynamic content RGB rendering Tug. dy and mask Mg. dy respectively. When rendering video, occluded areas change across frames, causing flickering artifacts. Faces masked to protect privacy.
LPIPS (row 1-3) match the corresponding metries 1/2 (row 1-1). Since we reproduce quantitative results by rumning the official code on rendered images provided by authors, we choose to use our reproduced results for the DVS results Tab. Namely, we use row 1-4 for DVS results since we evaluate without DynIBaR's masks. Similarly, we use row 2-3 in Tab. S5 for the NSFF results Tab.
F FURTHER ANALYSIS FOR ARTIFACTS
When rendering videos, we observe flickering artifacts. We investigate and find they mainly come from occlusions. To be specific, we display one rendered frame with artifacts in Fig. S3. Fig. S3.(h)- (i exhibit results from different renderers (see Sec. C.3). This frame rendered from a camera pose and time step corresponding to the ground-truth in Fig. S3.(c). We also showcase the source image for dynamic rendering in Fig. S3.(a), which is captured at the same time step Fig. S3.(c). As result, to obtain the dynamic content rendering, content (Fig. S3.(b)) with depth, and render it to the target frame. However, comparing Fig. S3. and (c), easily detect there exist areas that are unable to be rendered as they missing
need lift the source image's 's dynamic
22

---

Published as a conference paper at ICLR 2024
in (a), e.g., the forehead. The dynamic content rendering's s mask in Fig. S3.(e) (g) corroborate our observations. Since occluded areas vary in given monocular videos, such artifacts will change accordingly in renderings, resulting in flickering.
Based on our observations, scene-specific approaches suffer less from flickering artifacts. We believe temporal aggregation techniques be used to mitigate flickering as from more frames for those missing areas. However, as we discussed in Sec. 4.3, simple temporal aggregation with state-of-the-art tracking models does work well (see Tab, 3 and Tab. S4). More sophisticated designs are required. We leave this to future work.
gather information
23


Reviews:
[{"review_id": "ZGiCtMDaqS", "reviewer": "Reviewer_YL5S", "summary": "This paper presents a method to render novel views of a dynamic scene with much less per-scene optimization than competing techniques such as NSFF [Li et al. 2021] and Dynlbar [Li et al. 2023] . The input to the method is a video of a scene and a set of new camera poses over time. The output is a rerendered video. \n\nThe method works by computing a mask of the dynamic parts of the scene using existing methods. It also computes depth and optical flow. The dynamic parts of the scene are then modified by turning the dynamic pixels into point clouds and rerendering according to the new camera poses. The static parts of the scene are rerendered using a modified version of the generalizable NeRF transformer [Varma et al. 2023]. The dynamic and static parts of the scene are then combined.\n\nAccording to the quantitative metrics, the proposed method seems to perform slightly worse than NSFF and better than some other baselines that work on dynamic video inputs. This would be acceptable as the proposed method is much faster than NSFF and Dynlbar. However, qualitatively, according to the supplemental videos, the proposed method seems much worse than all competing methods, with substantial flickering.\n\n\nZhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural Scene Flow Fields for SpaceTime View Synthesis of Dynamic Scenes. In CVPR, 2021.\n\nZhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. DynIBaR: Neural Dynamic Image-Based Rendering. In CVPR, 2023.", "questions": "The video results presented in the supplemental seem much worse than both NSFF and Dynlbar. Specifically, there is a lot of temporal flickering in the dynamic parts of the scene. Do the authors know why this is? I would be very interested in seeing the dynamic mask, the optical flow and the depth estimates to better understand why there is so much flickering. I wonder if the authors uploaded the wrong set of results?\n\nI would be interested in seeing video results on the DyCheck dataset.\n\nHow does the proposed method handle cases where the new camera poses peer behind an object to a location that was not seen in any of the input frames? There’s no explicit inpainting step, but my guess is that GNT will not do a good job in these locations? You can kind of see this issue in the presented videos at the edge of the frames, where there are parts of the scene not seen in any of the input views.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "The paper tries to solve a worthwhile problem that would be very impactful to many groups and companies (rerendering a dynamic video without per-video optimization). The paper reads a bit like a systems paper where there are a dozen components (dynamic mask generation, depth estimation, optical flow, static scene rendering, dynamic scene rendering) that contribute to the final solution.\n\nThe overall algorithm makes a lot of sense and seems like it should work. There are also a lot of comparisons to other methods and an ablation study.", "weaknesses": "Given that Dynlbar exists, the answer to the title’s question seems like a resounding yes? I suggest the authors change the title to not be a general question and to be something specific about how their method works. I know that the authors are not planning on using scene-specific optimizations, which is how they distinguish their work from Dynlbar, but the title does not make this clear.\n\n\nThe supplementary results seem much, much worse than Dynlbar or NSFF. In the presented results, the dynamic portion of the scene flickers in and out of existence. The LPIPS metrics reported in the paper (Figure 1, Table 1) are only slightly worse than NSFF, but the actual results seem much worse. Perhaps this is because LPIPS doesn’t capture any notion of temporal consistency between the frames? The presented results are very inconsistent while those in NSFF and Dynlbar are not that inconsistent.\n\nI am not sure if the metrics evaluated make sense since they don’t take into account temporal consistency.\n\nThere is a minor missing citation. Consider discussing Figure 5 of https://arxiv.org/pdf/1909.05483.pdf [Niklaus et al. 2019] when presenting the statistical outlier removal technique (Fig.~S1). Niklaus et al. 2019 solve a similar problem where inaccurate depth estimates at object boundaries cause a similar problem to the one presented in Figure S1.\n\nMy relatively negative rating is based on the seemingly low quality results presented in the supplemental. It seems like the proposed technique does not work that well? The overall algorithm makes sense to me, so I am very surprised at how low quality the results are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zU6pCsGjgI", "reviewer": "Reviewer_BQLw", "summary": "The paper aims to solve and study the novel view synthesis problem for a general dynamic scene. It argues that we can have a generalized approach to dynamic novel view synthesis modeling from monocular videos by overcoming the dependence on the scene appearance. The proposed method uses scene depth, optical flow, and dynamic and static content masks, assuming that dynamic motion is linear and spatially consistent. Results on a few datasets are shown to back up the claims made in the paper.", "questions": "Kindly refer weakness section.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "* The paper aims at solving a very challenging problem and studies existing bottlenecks.", "weaknesses": "## Abstract\n- We find a pseudo-generalized … is possible -> We found that … is possible.\n\n## Introduction\n- Authors have given explanations justifying the keywords such as generalized, scene-specific optimization, scene consistent depth, etc., used in the paper. Yet, it is rather weak as the approach itself relies on consistent depth estimates of a dynamic scene, which, in fact, is a very open problem and acceptable solutions generally rely on appearance cues and scene flow.\n\n- Furthermore, with years of practice with physical depth sensors—be it iPhone depth sensing modalities or recent LiDAR, it's very hard, if not impossible, to recover consistent depth estimates for outdoor and indoor cluttered scenes. Even for static scenes, it is highly dependent on the subject material type, lighting condition, and other physical phenomena to have acceptable depth from a physical sensor, and here we are dealing with dynamic scenes. This is precisely the reason for methods such as \"Stable View Synthesis\" CVPR 2021, \"Enhanced Stable View Synthesis\" CVPR 2023, and Enhancing photorealism enhancement, TPAMI 2022 papers to make use of multi-modal 3D data to train the model. For completion, TPAMI 2022 could also work for dynamic scenes. The paper should emphasize such intrinsic details, detailing the papers mentioned above and the role of 3D data in novel view synthesis.  \n\n- Authors should also clarify why MonoNeRF does not qualify the definition of pseudo-generalized approach, given that the paper mentions “it is unclear whether MonoNeRF is entirely generalizable and can remove scene-specific appearance optimization”. It is better to test and present clarity in the rebuttal phase.\n\n## Scene Content Rendering\n- “we think it is possible to avoid scene-specific appearance optimization even if the monocular video input contains dynamic objects.” This argument is provided despite the paper relies on Varma et al. 2023 pretrained GNT which greatly benefits from appearance. Please clarify in the rebuttal as it is inconclusive as to how far the proposed methods benefit from Varma et al. 2023 work, given that the current method is aware of the dynamic subject mask. Hence, in my view, the contribution looks very little.\n\n## Using Depth and temporal priors\n- The assumptions about linear motion and use of optical flow is mentioned later in the paper. This must be highlighted in the introduction. Also, the assumption about linear dynamics of a scene is not convincing for a paper oriented towards a generalized or pseudo-generalized approach. \n\n## Experiments\n\n- Results are considerably lower in performance. This makes me conclude appearance is indeed an important cue for neural rendering. Of course, it could take more time, yet it helps gain realism. So, I am not sure whether the research presented in the paper is about time optimization or towards photorealistic rendering of dynamic scenes. Please clarify.\n\n\n- Missing experiments on outdoor dynamic scene dataset such as Cityscapes. Kindly evaluate results on this dataset and compare it with Enhancing photorealism enhancement, TPAMI 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vt83zqUBNp", "reviewer": "Reviewer_3K58", "summary": "The paper presents a study on generalized dynamic novel view synthesis from monocular videos, a challenge yet to be addressed in the literature. The authors establish an analysis framework, developing a \"pseudo-generalized\" technique that doesn't require scene-specific appearance optimization. The study found that geometrically and temporally consistent depth estimates are crucial to achieve this approach. Interestingly, this pseudo-generalized method outperformed some scene-specific techniques.", "questions": "- Can the authors clarify the specific role and impact of consistent depth estimates in their method?\n- How does the proposed method compare to generalized techniques not mentioned in the paper?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality in addressing the generalized dynamic novel view synthesis from monocular videos.\n- Introduction of the pseudo-generalized process without scene-specific appearance optimization.\n- A comprehensive set of experiments and detailed ablations to validate the approach.", "weaknesses": "- Presentation and clarity can be enhanced.\n- A broader range of related works should be included in the comparisons.\n- Ambiguity about the role of consistent depth estimates in the final result.\n- Experimental validation seems limited to certain datasets, potentially affecting generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PNzSmz5vuj", "reviewer": "Reviewer_bWXg", "summary": "The submission aims to find minimally necessary requirements for a generalized dynamic NeRF method for monocular RGB video input. It finds that running off-the-shelf methods for consistent depth estimation of the RGB video is sufficient. Using the RGB input video and these depth estimates, dynamic novel view synthesis is possible without further per-scene optimization at test time (e.g. for appearance). As additional input annotations, semantic segmentation masks (to identify dynamic objects in the input images) and optical flow are obtained via pretrained, off-the-shelf methods. These inputs can then be combined with a pretrained generalizable static NeRF transformer and novel special handling for dynamic content to render novel views. Experiments on existing benchmark datasets show that the method outperforms or is on par with many prior scene-specific dynamic NeRFs.", "questions": "I am confused by what's happening in Table 3. The final method is 3? And 5-1 and 5-2 differ in what way? Does the final method not use Sec. 3.3.2, while 5-1 and 5-2 do?\n\nOther than that, the weaknesses cover the four points I think need to be addressed in a rebuttal. I am open to arguments as to why these experiments might not be necessary.\n\n===\n\nThe rebuttal addressed my concerns very well and I am hence updating my score to Accept.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is an intriguing problem setting and while the paper does not find a method that is truly fast (due to the consistent depth estimation), it is a carefully executed study with good experiments. It is informative for researchers in the field to see where things currently stand.\n\nThe paper is extremely well written. The experiments of the method by itself are very thorough, only comparisons to other methods are a bit lacking (see below). \n\nThe appendix is thorough and covers all the questions I had about finer details of the method.", "weaknesses": "The most obvious downside of the paper is the result quality, unfortunately. The supplemental videos show rather low-quality results. Given that this is the first method in its problem setting, this is not necessarily a reason for rejection, as long as the experimental evaluation is great. The remaining weaknesses all concern the evluation.\n\n(1) Qualitative video comparisons: I don't understand the results on the Nvidia Dynamic Scenes dataset. Quantitatively, Neural Scene Flow Fields seems to be about on par with the submission. However, qualitatively, the results of NSFF (on their website) are much better. Where does that large discrepancy come from? Qualitative video comparisons to other works would help.\n\n(2) Fast scene-specific dynamic NeRFs: I would like to see a comparison with fast scene-specific dynamic NeRF methods. In terms of utility for novel view synthesis, generalizable NeRFs have two main advantages over scene-specific NeRFs: speed and learned prior knowledge. The latter is not exploited by the submission, as the results show novel view synthesis that sticks closely to the input camera path (instead of revealing hidden areas that scene-specific methods could not handle). Which leaves speed and I'd hence like to a see a comparison with Fang et al. TiNeuVox '22 (code is available), which optimizes a dynamic NeRF in a few minutes, unless there is a reason why such a comparison is unnecessary.\n\n(3) Static generalizable per-frame NVS: Also, given the rather limited quality and temporal instability of the results, I'd like to see a comparison with static single-image generalizable novel view synthesis methods. For example, Sajjadi et al. Scene Representation Transformer '22. MIT-licensed code is here https://github.com/stelzner/srt and the authors say on their project page that this code is reliable.\n\n(4) All qualitative results of the proposed method: Why are the qualitative video results on the DyCheck iPhone dataset not included in the supplement? \n\n== Minor Comments ==\n\nWhat is \"hundreds of GPU hours per video\" in the introduction referring to? It sounds as if existing per-scene methods take hundreds of hours per video (incl. appearance optimization). But that's not the case with most monocular dynamic NeRFs, especially the recent fast methods that take minutes rather than hours.\n\nThe related work is covered very thoroughly. Only recent diffusion-based approaches for novel view synthesis could additionally be cited, e.g. Watson et al. Novel View Synthesis with Diffusion Models (ICLR '23).\n\nThe first citation in A.1 is for transformers in general, not for GNT.\n\nPlease add a sentence to the main text that splat-/point-/mesh-based rendering of the point cloud is used to get the dynamic image rendering. The section on dynamic rendering feels incomplete currently, with some context/framing missing. \n\nI would not call the results \"high-quality\" (e.g. caption of Figure 8). The videos are not high-quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)