DynPoint: Dynamic Neural Point For View Synthesis
Authors: Kaichen Zhou
Jia-Xing Zhong
Sangyun Shin
Kai Lu
Yiyuan Yang
Andrew Markham
Niki Trigoni
Venue: NeurIPS 2023 poster (2023)
Keywords: View Synthesis
Monocular Video

Abstract: The introduction of neural radiance fields has greatly improved the effectiveness of view synthesis for monocular videos. However, existing algorithms face difficulties when dealing with uncontrolled or lengthy scenarios, and require extensive training time specific to each new scenario.
To tackle these limitations, we propose DynPoint, an algorithm designed to facilitate the rapid synthesis of novel views for unconstrained monocular videos. 
Rather than encoding the entirety of the scenario information into a latent representation, DynPoint concentrates on predicting the explicit 3D correspondence between neighboring frames to realize information aggregation.
Specifically, this correspondence prediction is achieved through the estimation of consistent depth and scene flow information across frames.
Subsequently, the acquired correspondence is utilized to aggregate information from multiple reference frames to a target frame, by constructing hierarchical neural point clouds. 
The resulting framework enables swift and accurate view synthesis for desired views of target frames. 
The experimental results obtained demonstrate the considerable acceleration of training time achieved - typically an order of magnitude - by our proposed method while yielding comparable outcomes compared to prior approaches. Furthermore, our method exhibits strong robustness in handling long-duration videos without learning a canonical representation of video content.

DynPoint: Dynamic Neural Point For View Synthesis
Kaichen Zhou, Jia-Xing Zhong, Sangyun Shin, Kai Lu, Yiyuan Yang, Andrew Markham, Niki Trigoni
Department of Computer Science University of Oxford
{rui zhou, jiaxing zhong, sangyun shin, kai. dei.lu)@cs. ox.ac. uk lyiyuan.yang. andrew. andrew.markham, niki trigorii](ee ox ox. ac.uk uk
Abstract
The introduction of neural radiance fields has greatly improved the effectiveness of view synthesis for monocular videos. However, existing algorithms face diffi- culties when dealing with uncontrolled or lengthy scenarios, and require extensive training time specific to each new scenario. To tackle these limitations, DynPoint, an algorithm designed to facilitate the rapid synthesis of novel views for unconstrained monocular videos. Rather than encoding the entirety of the scenario information into a latent representation, DynPoint concentrates on predicting the explicit 3D correspondence between neighboring frames to realize information aggregation. Specifically, this correspondence prediction is achieved through the estimation of consistent depth and scene flow information across frames. Subse- quently, the acquired correspondence utilized to aggregate information from multiple reference frames to target frame, by constructing hierarchical neural point clouds. The resulting framework enables swift and accurate view synthesis for desired views of target frames. The experimental results obtained demonstrate the considerable acceleration of training time achieved typically an order of mag- nitude by our proposed method while yielding comparable outcomes compared to prior approaches. Furthermore, our method exhibits strong robustness handling long-duration videos without learning canonical representation of video content.
propose
Introduction
The computer vision community has directed significant attention towards novel view synthesis (VS) due to its potential for both emerging techniques in artificial reality and also to enhance a machine's ability comprehend the appearance and geometric properties of target scenarios (37 g 39] State-
of-the-art techniques leveraging neural rendering algorithms, as demonstrated in studies such as
37 58 61
have successfully attained photorealistic reconstruction of views in static scenarios.
However, the dynamic characteristics inherent to most real-world scenarios present a formidable challenge to the suitability of existing approaches that rely the epipolar geometric relationship. traditionally applicable to static scenarios [22 45
Recent studies have primarily focused on the synthesis of views in dynamic scenarios by employing one or multiple multilayer perceptrons (MLPs) to encode the essential spatiotemporal information of the scene 45
B1 43 42). In one approach, latent representation is generated to encompass the
comprehensive per- frame details of the target video [T 16 62). Although this method is capable of producing visually realistic results, its applicability is limited to short videos due to the constrained memory capacity MLPs other representation mechanisms [2]. Alternatively, another approach seeks to construct latent representation of canonical scenarios and establish correspondences between individual frames and the canonical scenario 45 63). This alternative approach allows for the processing of long-term videos; however, it imposes specific requirements on the video
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

characteristics being learned. For instance, the videos should consistently feature similar objects across frames, and some algorithms may require prior knowledge about the video 660 44
In order to address this challenge, we introduce DynPoint, a novel approach designed to achieve efficient view synthesis of lengthy monocular videos without the need for learning latent canonical representation. Unlike conventional methods that encode information implicitly. DynPoint employs an explicit estimation of consistent depth and scene flow for surface points. These estimates subsequently utilized to aggregate information from reference frames into the target frame. Sub- sequently, hierarchical neural point clouds are constructed based the aggregated information. This hierarchical point cloud set is then employed to synthesize views of the target frame. Our contributions can be summarized as follows:
We introduce novel module to estimate consistent depth information for monocular video with the help of proposed regularization and training strategy.
We propose an efficient approach to estimate smooth scene flow between adjacent frames with the proposed training strategy by leveraging the estimated consistent depth maps.
We present representation to aggregate information from reference frames to target frame, facilitating rapid view synthesis of the target frame within a monocular video.
Comprehensive experiments conducted on datasets including Nerfie, Nvidia, HyperNeRF, Iphone, and Davis showcasing the speed and accuracy of DynPoint for view synthesis.
Related Works
2.1
Static View Synthesis
The generation of photo-realistic views from arbitrary input viewing angles has been longstanding challenge in the field of computer vision. In the scenario of static scenes, early approaches addressed the challenge of synthesizing photo-realistic views by employing local warping techniques to handle
densely sampled views [19 30
5 7 Additionally, the gradient domain was utilized to handle the single-view case (29. In order to tackle the challenges associated with view synthesis, several subsequent works have been developed. These works aim to address issues such as reflections,
2
repetitive patterns, and untextured regions 28 15
24
14 48 34
36
41
50
51
52 53
59 69. In
more recent developments, researchers have explored the representation of scenes as continuous neural radiance fields (NeRF) using fully connected neural networks. These works, such as [8 66 25),
have demonstrated remarkable outcomes with a trade-off between accuracy and computational complexity (i.e., one network for one scene). To tackle the computational burden associated with neural radiance fields, recent works have focused on developing approaches that generalize the representation across multiple scenes using a single network. These methods employ various techniques, such as fully-convolutional architectures 665 plane-swept cost volumes [8], image-based rendering 58, disentanglement of shape and texture utilization of local features in 2D [56, as
well as generative methods 6
49
40
20 3
2.2 Dynamic View Synthesis
The emergence of static scene advancements has spurred interest in the exploration of view synthesis for dynamic scenes within the field. Previous research efforts have expanded upon studies conducted in static scenes, incorporating clements such as globally coherent depth (64] and 3D mask volume [33] Recent research direction builds upon the concept of NeRF and extends it to dynamic scenes by integrating time into the learning process of netio-ttmmpora radiance fields. One category of research focuses on the assumption of a canonical scenario that spans the entire video 4
10 21]. Deformable
radiance field-based approaches, described in previous works 45 55 warping techniques to adapt a base NeRF representation of the canonical space for dynamic scenes. This enables the synthesis of novel views in the context of long monocular videos. Another category of algorithms aims to encode the temporal dynamics of the scene into a global representation. [31] employs a MLP to model the 3D dense motion, resulting in a spatial-temport NeRF. Building upon this, 12 [6) demonstrate that introducing additional regularization techniques that encourage consistency and physically plausible solutions can enhance the accuracy of view reconstruction. The first category of methods, while achieving impressive photorealistic results, constrained by the requirement for object-centric videos, thus limiting their generalizability to diverse scenarios.
42
33
35), employ temporal

---

Surface Point
Scene
A
Flow
Reference Frame {Ir, dr}
Flow Loss
Scene Flow APP r-
Rendering
MLP
Scene Flow
Optic Flow
Color & Density (6, c)
MLP
Reference Frame
Aggregate
I
Radius R
Flow
Depth Loss
Neural
Target
Points Ps
4
Frame
Viewpoint
Target Frume
{T,,
Monocular
Depth Net
{ar, 8c]
(2,2,)
Depth
(Stage 1)
(Stage
Figure Structure of DynPoint. The Stage 1 shows the pipeline consistent depth estimation Initially, the frames are employed in the Flow Net, Depth Net, and Scale Parameters to produce optic flows and depth. Then, surface points are calculated based on the estimated depth and utilized in the Scene Flow MLP. The Stage 2 shows the process information aggregation presented in Sec, 3.4 Neural Point Clouds firstly generated based on pre-computed scene flow. The Rendering MLP utilizes all neural points located within specified radius from the queried point as inputs predict the final color and density.
in Sec. 3.2 and scene flow estimation in Sec. 3.3
Conversely, the second category of methods encounters difficulties when dealing with long videos, leading to limitations in effectively handling such scenarios. To address the aforementioned issue, our algorithm focuses achieving novel synthesis by aggregating information from the reference frame to the target frame. This information aggregation process is accomplished by explicitly modeling the object movement and depth information. As result, our algorithm exhibits significant advancements both accuracy and speed.
Methodology
3.1 Overview
Our algorithm designed to realize view synthesis for a dynamic scenario by utilizing monocular video {4, lz,.., Jr). The frames in the video, denoted by 4, are captured by a known camera C2 MKK,e|[R,, t..e).. where denotes known camera viewpoint. The objective is generate the novel view from a specified viewpoint y at desired time frame Ct,g (K,,g [R,, (r,])-
adopt training paradigm in which our model trained on the input monocular video with the assistance of pre-trained optic flow and monocular depth models [47] 54 26). During the training process, the RGB information obtained from the observed viewpoint is utilized as the supervision signal, without relying on any canonical information. Subsequently, the trained model is evaluated the task of synthesizing corresponding RGB, depth and scene flow information for unobserved viewpoints.
Consistent with previous researches 31 16 17
32 35| 62).
In contrast to previous methods that utilize one or multiple MLPs to encode the 3D information of each frame, our work focuses establishing correspondence relationships, i.e., scene flow, between the 3D surface points of the current frame and those of adjacent frames. We can infer 3D information about unobserved points of the current frame by aggregating the information from adjacent frames.
To realize the aforementioned concept, our proposed model should undertake three key tasks. The first task is to estimate depth information consistently for each frame, outlined in Sec. 3.2 The second task involves learning the 3D scene flow between the current frame and its adjacent frames, as detailed in Sec. 3.3 These two processes are demonstrated in the stage of Fig. The final task is

---

Corrected
Depth
Frame
Targot Frame
(b) Map
Scene
(..)
Optic
Cotrected Dopth
Flow
Socne Flow
MLP
Frame
(c) Map
(d)
Figure 2: Demonstration of Geometric Edge Mask and Scene Flow Estimation. The left section depicts the conceptual basis for designing the Geometric Edge Mask. The right part demonstrates the construction of the scene flow objective function shown in Sec. 3.3
to aggregate information based on learned correspondence and subsequently use this information to realize view synthesis, discussed in Sec. 3.4 This process is shown in the stage 2 of Fig.
Consistent Depth Estimation
Despite the ability of current monocular depth methods [6 to produce accurate depth priors, the predicted depth maps d' suffer from scale-variance and shift problems when compared to the ground truth depth [18 68). This characteristic renders d' inconsistent across the temporal axis of monocular video. Moreover, since the optic flow fi-+ between frames could be predicted from pretrained model, depth value de can be computed using the triangulation relation between frames, along with the camera parameters and Mid- point method. Compared to d', de can have consistent scaling across frames. However, d, heavily relies on fi-- and Epipolar constraint, and it can only generate accurate depth information for limited portion of the frame. The first module of DynPoint aims combine d' and de to generate the final consistent depth estimation de.
Regularization: To address this issue, DynPoint identifies the accurate region of d, by utilizing three masks: the corresponding mask, the geometric edge mask, and the dynamic object mask.
Correspondence Mask Me..orr: The purpose of the optic flow is to capture the pixel-wise correspon- dence between two frames, making it accurate only for the corresponding regions. These regions can be identified by masking out areas of occlusion caused by both ago-motion and object movement. This Correspondence Mask could be formulated as follow:
5 iʃ ()rr(()) S (c)r( Ec: [1 if ()) fe Sa (p) co.
(I)
where e, is predefined threshold for the correspondence mask.
Geometric Edge Mask My: Furthermore, it has been observed that the reliability of d diminishes when applied to geometric edges as in Fig. _ particularly in areas where the optic flow exhibits non-smooth characteristics. To compute the mask My. first estimate the normal vector map of the surface (- dy
dz 1)/lI- ds dr dy 1)Il with the help of dr Then we apply the Canny edge detector [1 on the n E Hx to generate the estimation of geometric edge. The intuition of the geometric edge mask demonstrated in the left part of Fig. From the error map shown Fig. 2)b), it evident that the errors primarily manifest around the geometric boundaries of the scenario.
Dynamic Object Mask Ma In addition the masks mentioned above, Dynamic Object Mask is necessary exclude the dynamic regions of the frame where the triangulation relationship does not hold. Similar to [16), we combine the MASK R-CNN (3 with the Sampson error to generate the My. To obtain valid mask with high accuracy, apply image erosion on the inverse of Ma and

---

Me to mitigate inaccurate boundary detection. We also perform image dilation on Mg to include the surrounding region of the geometric edge.
Objective Function:
Based on the above three masks, the reliable region of the de could be masked out by using the final mask M = =Me n M. MMa. To combine the information of d' and de we assume that the consistent depth map de E R#* could be approximated by using d2 a scale variable Or E In and a shift variable 3t E R. The parameters at and 3t can be generated by optimizing:
0t2 B argmin M O Idg (andy + B-)1-
Training Strategy: Dependent solely on the nearest frame for the computation of d, would lack sufficient reliability due to the combined influence of the accuracy of the camera matrix and the accuracy of the optic flow on the triangulation process. We employ a series of adjacent 2K frames to calculate the triangulated depth set denoted as {d, JK=1 Instead of directly utilizing all {dz y2 in Eqn. 2 we perform reevaluation by recomputing the intersection mask as M, M) n n. NM7, which further refines the triangulated depth de ensuring the appropriate scale constraint for dy. It's worth noting that pose estimation may not be accurate for dynamic scenarios in certain datasets, as relies on COLMAP--asse estimation. In such cases, we utilize the algorithm presented in our Supplementary material to enhance and fine-tune the pose estimation.
3.3 Scene Flow Estimation
Combined with estimated consistent depth map de and optic flow fi-e, DynPoint also aims to infer the scene flow &t-L' to build the 3D correspondence between the current frame and adja- cent frames. Unlike previous works 42 45 55 [3], which estimate the trajectory of all points (hundreds sampled points on the ray each point) the scenario, DynPoint only infers the trajectory of surface point (one point on the ray of each point) of the frame to accelerate both training and inference process. To realize this process, we use MLP to estimate the scene flow, which can be Fa (P, t) where P E R denotes input 3D point; APr-ee denotes the trajectory of P from i L'. The weight 0 be optimized by using the relationship among the depth de, the optic flow fi-r and the scene flow S
written as A AP:-t-1
Given a pixel Pr in frame t. its corresponding pixel in frame t', denoted as pr. can be obtained by adding the 2D flow fe (Pr) to Pt- Additionally, utilizing the depth map d, and camera matrix C., which available frame the 3D point corresponding Pt can be expressed as R,K," de ty. The same method can be used compute Pr. Thus, we've:
Objective Function:
St (p:) = (pe (pe + fe-e (p:)) K2 (pt + fiee(())) + tor Pt.
(3)
This process is demonstrated in the right part of Fig. 2 For static part masked by Ma. set si (pt) = 0. For the dynamic part, the loss function can be written as
C.
Σ
Ist (pr) AP,
P: P.EMANM,n-A
In order to enhance the accuracy of scene flow estimation for the dynamic elements within the scenario, we employ the cycle constraint, a well-established technique utilized in prior studies The cycle constraint can be expressed as follows:
16 35
C.
Σ + PiE-Ma
+ AP
AP
(5)
Training Strategy: We have observed that optimizing Eqn. solely with the near frames, where or t + 1, does not produce accurate results for further information aggregation. To reconstruct the 3D information of the current frame, it is necessary to compute the correspondence between the current frame and 2K adjacent frames, in order to aggregate sufficient information. During the training process, compute the scene flow between frame / and its 2K adjacent frames, where K E (1,.., K). We then utilize Eqn. 4 to form the loss function between current frame and K adjacent frames. The scene flow A Pt-se+& could be written as:
APrLE (P,,, [0] + ... ...++.. Fo(P + AP A 1)(0].
(6)
5

---

Table 1: Novel View Synthesis Results on Nvidia Dataset. We report the average PSNR and LPIPS results with comparisons to existing methods on Nvidia dataset [6]. denotes the number adopted from DynamicNeRF [16]. The best performance is highlighted. The second-best is also emphasized.
PSNR |
Jumping
Skating
Truck
Ulmbrolla
Ralloon
Rolloce?
Playground
Average
NoRF
hours)
20.90 00.90/0.305
23:67
70.311
22.73/0.22
19 82 00.000
24.37
11.99/ /0
NeRF
hours)
18.04 0.455
00.320.382
17.69/00 0.726
00.26
16.68 10.421
hours)
22.36/0.193
22.48
21.47:0200
19.06
20.76/00 00.27
20.18
21.48/1 0.228
NSPP
hours)
24.65 70.151
09:29/0.129
25.96/0.167
22.97/0.95
21.96/0.215
21.23
0..19
NeR [6] 36 hours)
24.63 0.000
32.66 0.035
28.56/0.062
23.26/0.337
22:36 22:36/0.00
27.06 /0.049
24.1570.080
26. 0.082
HyperNeKR [33 (> hours)
18.3470.502
21.97 00.183
20.61 /0.205
13.59/0.44)
13.96 01.50
16:57/00 /0.411
15.37 10,895
17.50 0.367
TiNeuNox [33 (ow mins)
20.83 (7.3770,247 002247
23.32 00.152
23.86/0.175
20.00/
0.355
17:30 0.353
19.00/ /0.279
13.84 70,437
19:74 0.285
RoblYN [S] P 36 hourss)
25.66 0.077
28.08/
/ 0.140
29.13/0.06
24.26
0.099
22.37 / 10.103
26.19 8:004
24.96 11.044
25.89/ 0JIGS
DyaPaini (- 30 mins)
24.02 1.:02/0.000 0.099
31.34 70.047
29.30 0.066
24.59 00086
22.77 00.099
27.63/0.049
25.37 11.0399
26.53 / DJB6S
(a) GT
(b) NSFF
(c) HyperNeRF
(d) RoDynRF
(e) DynPoint
Figure 3: Demonstration of View Synthesis Results on Nvidia Dataset. This demonstration compares the view synthesis outcomes of DynPoint with those of NSFF, HyperNeRF, and RoDynRF.
It's important to mention that achieving better results can be accomplished by fine-tuning the refinement layers of the pre-trained depth network.
3.4 View Synthesis
Information Aggregation: In order to collate data from the 2K reference frames to the target frame the pixels in the reference frame Pr4-I are transformed into 3D space as P.+.& through the utilization of the corresponding depth value d and the matrix C.+k. Next, the DynPoint evaluates the scene flow AP(+k=1 between frame i and frame t + k by implementing Equ. 66] By utilizing both the scene flow APt+k- and the 3D point Pe+k, we are able calculate the 3D point Pe+k-+t that corresponds to reference frame t ++A ks at the target frame t. This is achieved by applying the following computation: P Pi+++ + P Additionally, a pre-trained 2D convolutional neural network (CNN) is utilized to generate a 2D image feature vector f, which is subsequently allocated to each of the points corresponding to pixels.
Finally, we propose the introduction of a hierarchical neural point cloud construction approach to enhance the "perception field" of individual points. This technique aims to address the problem of surface irregularities that may arise from using current monocular depth estimation methods in multi view depth map fusion. Thus, the point cloud of current frame could be generated by combining 2K + point clouds as follows:
K ph Σ P(P k=-K
P= Σ ph P k=-K
(7)
where H is the number of hierarchical levels. our case, we set H 3. At each step within this framework, we perform downsampling operation the neural point cloud (the downsampling procedure implemented within the point cloud generation process, whereby the depth map and
scene map are downsampled using linear interpolation technique; it is important to note that when

---

Table 2:
Novel View Synthesis Results Nerfie Dataset. We report the average PSNR and LPIPS
results with comparisons to existing methods on Nerfie dataset [42
PSNR T/LPIPS 4
CURLS
TOBY SIT
TAIL
BROOM
Average
NeRF 38
14.40/ 0.616
22.80/ 0.463
23.00 0.571
21.00/0.60
20.30/ 0.579
NeRF + time 38
17.30 0.539
19.40/ 0.385
24.90 0.404
21.90/ 0.576
20.87 0.476
NSFF 31
03.00/0.432
26.90 0.208
30.60 0.245
28.20 / 0.202
25.93 0.272
Nerfies 42
24.90 0.312
22.80 0.174
23.60 0.175
21.00 0.270
23.08 0.233
DynPoint
24.33 0.339
24.90 0.186
29.12 0.218
27.28 0.222
26.41 0.241
Source view
NSFF Mew
Dyn New
NSFF Vvew
DyrPoit Vicw
NSFF (ilww 30
DonPlint (V
Soure Views
NSFT 2iamn:
DunPa (Maw 11
NSFT Wiew 2;
DyePoitt (View 23
NSF (dlew 3)
Dynitint (Mwac 3)
Figure 4: Demonstration of View Synthesis Results Nerfie Dataset. This demonstration compares the view synthesis outcomes of DynPoint with those of NSFF.
downsampling the depth map, the intrinsic matrix should also be adjusted accordingly to maintain accurate spatial information.)
View Synthesis: Given 3D position q and view direction d, we leverage M proximate neural points
within radius of R to generate the corresponding density and color data ö, c as in [6]]. This process shown in the right part of Fig. [ This can be expressed as follows:
(8, (a d, Pl. ft Te.... P.M P.MM 77 M)
(8)
where y/ is the per-pont per-point confidence introduced in [61]. Finally, make use the rendering process in (38] to predict final RGB value C and depth D as:
C=
Σ (1 expl- cxp(-o,0,))cp ==1
where 7, = erp(- Σ o,ō,): 65 denotes the distance between adjacent shading samples; cj is the color information and 6, is the density information. The L2 loss function used to supervise our rendered pixel values similar the setting of [37]. For further information regarding the network architecture, please consult our supplementary materials.
-
3.5
Discussion
The purpose of this section is to explicate the dissimilarities between the DynPoint and recently published concurrent work, namely DynIBaR (2) which was released in March 2023 and currently lacks available code. Both DynIBaR and DynPoint harness information aggregation mechanisms to realize the synthesis of novel views. However, DynlBaR predominantly centers around the aggregation of information through two-dimensional (2D) pixel units. This approach draws inspiration from image-based rendering principles, entailing the synthesis of novel perspectives from collection of reference images via weighted fusion of reference pixels. In contrast, DynPoint's focal point lies in the information aggregation achieved by constructing three-dimensional (3D) neural point clouds. The final novel view synthesis is realized by using neural points surrounding queries' position.
7

---

Table 3: Novel View Synthesis Results of HyperNeRF Dataset. We report the average PSNR and
LPIPS results with comparisons to existing methods on HyperNeRF dataset 43
Broom
3D printer
Chicken
Expressions
Peel Banana
PSNR LPPPS 1
Average
197 Frames
207 Frames
164 Frames
259 Frames
513 Frames
NSFF 31
26. 10/ / 0.284
27.70 / 0.125
26.90 / 0.106
26.70/ 0.157
24.60 / 00198
26.40 00.174
Nerfies [42]
19.20 0.325
20.60 / 0.108
26.70/ / 0.078
21.80 0.0
22.40/ 0.147
22.10/ 0.166
Hyper-NeRF
20.60 0.613
21.40 0.212
27.60/ / 0.108
22.00 /.. 196
24.30/ 0.170
23.20 /0.260
DynPoint
27.40 0.248
27.60/ / 0.163
28.10 / 0.089
27.90/ 0.147
26.50 / 0.129
27.50 / 0.155
Table 4: Novel View Synthesis Results of Iphone Dataset. We compare the mPSNR and MSSIM
scores with existing methods on the iPhone dataset [17].
mPSNR 1// /
Apple
Block
Paper-windmil windmill
Space out
Spia
Toddy
Wheel
Average
NSFF PD
17.54/0.750
16.61 706639
17.34/0.377
17.29/ / 0.622
18.38 0..55
13.65/0.55
13.82 0.455
15.46/0.560
Netties HypurNeRF
17.61
17.51
17.38
17.93/ 0.605
19.20 /0.56
13.97/ 10.576
13.99 13.95/0:155
/6.669
16.47/0..755
14.71/0.600
14,94/0.272
17.65 10636
17.26 17.56/0.540
12.59/0.537
14.59/0.51
1681 70,350
DynPuint
17.78/0.743
17,67/0660
17.32/0.366
17.78 / 0.603
19.34/0.50
13.95 /0.551
14.72/ 0.515
16.89 /0,572
Moreover, DynPoint introduces an efficacious strategy for the seamless integration of monocular depth estimation within the ambit of monocular video view synthesis. In contrast to DynlBaR, which endeavors to model the trajectory of all samples (128) traversing each ray, DynPoint exclusively focuses on the trajectory of surface points, thereby yielding a substantial acceleration in both training and inference stage.
Experiment
4.1 Experimental Setting
To evaluate the view synthesis capabilities of DynPoint, we performed experiments on four extensively utilized datasets, namely Nvidia dataset in 31., Nerfie (42) HyperNeRF in 45 and Iphone in 177. It is noteworthy to mention that the official website of Nerfie [42 only provides four specific scenarios. Consequently, our experiments were solely conducted on provided four scenarios as in [17]. Additionally, we also assessed DynPoint's performance on a recent dataset Iphone [17], which specifically addresses the challenge of camera teleportation. Furthermore, we examined the efficacy of monocular depth estimation and scene estimation by visualizing the results obtained from the Davis dataset, as in [67]
We conducted a comparative analysis of our work with several recent methods, based on the reported results in their original papers or the reimplementation results of their official code. The methods we compared with include NeRF (38), NeRF + time (38] (which directly utilizes embedded time information
input to encode all information of the target dynamic scenario), D-NeRF (5)
NSFF 31,, DynamicNeRF [6). HyperNeRR [33, Nerfie 42), Vox [33] and RoDYN
35
In research, employed the pretrained Deep Pruning Transformer (DPT) network in for monocular depth estimation. For optic flow estimation, we utilized the pretrained FlowFormer model [26]. To facilitate the fine-tuning process, initialized the weights of the Rendering MLP by pretraining it on the DTU dataset, employing a similar training set to that used in [61].
46
The per-scenario training time is shown in Tab. I A notable observation was made regarding the reduced per-scenario training time of DynPoint in comparison to other algorithms. This enhancement can be attributed to the implementation of two-step strategy. In the initial stage, the optimization process focuses on limited set of parameters pertaining to monocular depth estimation and scene flow estimation. Subsequently, for the second stage, leveraging the outcomes obtained from the first stage, the generation of the neural point cloud for subsequent view synthesis is achieved successfully, Furthermore, the pretraining stage also contributes to the efficiency of our view synthesis stage, requiring only a few iterations produce desirable results on novel scenarios.
Nvidia (Tab. I): During this experiment, it was observed that DynPoint exhibited superior perfor- mance compared to other algorithms in terms of peak signal-to-noise ratio (PSNR). Additionally, DynPoint achieved the second highest ranking among all algorithms when evaluated based on the Learned Perceptual Image Patch Similarity (LPIPS) metric. Even for scenarios involving multiple objectives, such as Jumping, DynPoint demonstrates reasonable performance without requiring the learning of any canonical representation extensive training time. The results of specific partial
8

---

Table 5: Ablation studies of Nvidia Dataset. We report the average PSNR and LPIPS results with comparisons to existing methods on Nvidia dataset [64].
PSNK /LPIPS
Jumping
Skating
Track
Unbbvolla
Hailoon1
Balloon22
Playground
Average
multiplessep step strategies
15.01
17.43/00
70.583
15,46
11.35
14,78
w/ K=3
22.53/
22.38
19.1370.88
/0.413
16.92/0.45
0.416
(0,35) 58,58/11 0
Hierarchial Point Cloud
21.02/0.174
0.055
27.50/0.085
22.73/0.392
/0.256
24.67
22.14/ 0.189
DynPoint (K 6]
24.69/ IM097
31.34/ 0.045
29.30 / /0.061
24.59 0.086
22:77 /0,099
27.63/0.049
25.37 0.039
26.53 (L068
Target
Reference
Target
Reference
Depth Flow
Error Map
Depth Flow
Error Map
Scene Flow
Error Map
Scene Flow
Error Map
Figure 5: Demonstration of Depth and Scene Flow Estimation. This figure presents the output of the target images obtained by warping the reference image using depth estimation (second row) or using both depth and scene flow estimation (third row). It is important to clarify that the figure is not intended for comparing view synthesis results. The synthesized figures generated based on scene flow inherently incorporate object motion as input, resulting in observable motion blur within the synthesized figures. Additionally, an map represented by the intensity of red is provided to visualize the performance, where deeper shades of red indicate larger errors (in terms of pixel movement compared corrected optic flow).
scenarios, namely Playground, Skating, Truck, and Umbrella, are presented in Fig. 3 Notably, leveraging monocular depth estimation, DynPoint generally produces views with improved geometric features, as evident in the Skating case shown in the second row of Fig. 33 Even in challenging scenarios like Umbrella, DynPoint successfully generates high-quality views while minimizing blurring effects. Nerfie (Tab. 2): In the case of the extended scenario, DynPoint exhibits superior performance in terms of PSNR and achieves comparable results those obtained in the short video Nvidia, as presented in Tab. 2 This achievement can be attributed to our information aggregation approach, which focuses on effectively aggregating information from the target frame by establishing associations between its points and those in the reference frames. Novel view synthesis results for scenarios TAIL and TOBY SIT are depicted in Figure 4 It is evident that for longer videos, DynPoint continues to produce more realistic frames. Notably, the generated views in this figure do not exist in either the training or test dataset. HyperNeRF (Tab. (3:: In the case of longer video sequences, DynPoint showcases superior performance in PSNR, as highlighted in Table 3 Iphone (Tab. (4): In the context of monocular videos without camera teleportation, as demonstrated in the work (17]. DynPoint attains comparable outcomes to previous algorithms. Due to the inherent limitations of having few multi-view perspectives and overlapping information, establishing correspondences between adjacent frames becomes challenging. Consequently, the view synthesis task based on monocular videos proves to be more arduous on this dataset compared to other datasets.
4.2 Ablation Studies
In order to evaluate the effectiveness the strategies proposed in our paper, we conducted three ablation studies. These studies include: (I) the absence of the multiple-step training strategy outlined in Sec. and and Sec. [3.3 (2) the utilization of only six frames in the vicinity (K = 3); and (3) the exclusion of the hierarchical point cloud. The results are presented in Tab. S it evident from the results that the omission of the multiple-step training strategy leads to the largest drop in performance. Without this strategy, the first stage produces noisy outcomes for both monocular depth and scene

---

GT
DynPoint
GT
DynPoint
Figure 6: Demonstration of Failure Results on Nvidia Dataset. In this demonstration, we present the failure results generated by DynPoint.
flow estimation, consequently hindering the generation of the neural point cloud and impacting the performance the second stage. Moreover, using a limited number of adjacent frames also adversely affects the final performance, which aligns with our expectations limited inputs correspond to limited information. Although the removal of the hierarchical point cloud does not significantly degrade performance, it still plays role in generating finer results.
4.3 Consistent Depth & Scene Flow Estimation On Davis
In order to validate the efficacy of the monocular depth estimation technique and scene flow estimation method, we present the following visualization results: (1) Reconstructed Target Image using Monocular Depth Estimation: We demonstrate the reconstruction of the target image by warping the reference image based on the monocular depth estimation. (2) Reconstructed Target Image using Scene Flow and Depth Estimation: We present the visual reconstruction of the target image achieved through warping the reference image using solely depth estimation or combination of scene flow estimation and depth estimation. The results are displayed in Fig. 5 Upon observing the second row, it is evident that DynPoint demonstrates commendable performance in reconstructing the static background of the target frame through monocular depth estimation. However, the primary errors (deep red part) occur in the dynamic region, where accounting for object movement becomes crucial for accurate warping. Moving to the third row, our scene flow estimation proves be effective in capturing the movement of dynamic objects, as in the red error map Fig. 5
4.4 Failure Cases
Despite the notable achievements of DynPoint, certain failure cases were observed during the view synthesis process, as demonstrated in Fig. 6 By comparing the first and second images, becomes apparent that generating realistic facial features in regions with intricate details proves be challenging. Furthermore, when comparing the third and fourth images, is evident that DynPoint struggles with handling fine objects and reflections, these aspects heavily rely the accurate geometry inference obtained in the first stage.
Conclusion
In this research paper, present DynPoint, algorithm specifically designed to address the view synthesis task for monocular videos. Rather than relying on learning a global representation encompassing color, geometry, and motion information of the entire scene, propose approach that aggregates information from neighboring frames. This aggregation process is facilitated by learning correspondences between the target frame and reference frames, aided by depth and scene flow inference. The experimental results demonstrate that our proposed model exhibits improved performance in terms of both accuracy and speed compared existing approaches.
Acknowledgements. Our research is supported by Amazon Web Services in the Dofford-Siggapee Collaboration Programme and by the ACE-OPS project (EP/S030832/1), We are grateful to all of the anonymous reviewers for their valuable comments.
10

---

References
[1] T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, "Mip-nerf:
A multiscale representation for anti aliasing neural radiance fields," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 3855-58644
[2] C. Buehler, M. Bosse, L.. McMillan, S. Gortler, and M. Cohen, "Unstructured lumigraph rendering," in Proceedings of the 28th annual conference Computer graphics and interactive techniques, 2001, pp. 425-432
[3] S. Cai, A. Obukhov, D. Dai, and L. Van Gool, "Pix2nerf: Unsupervised conditional p-gan for single image to neural radiance fields translation." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 3981-3990.
[4] J. Carranza, C. Theobalt, M. A. Magnor, and H. Seidel, "Frreevieepoon video of human actors," ACM transactions on graphics (TOG), vol. 22, no. pp. 569-577, 2003.
[5] J.-X. Chai, X. Tong, S.-C. Chan, and H.-Y. Shum, "Plenoptic sampling," in Proceedings of the 27th annual conference Computer graphics and interactive techniques. 2000, pp. 307-318.
[6] E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein, "pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 5799-5809.
[7] G. Chaurasia, S. Duchene, O. Corkine-Hornung, Hornung, and G. Drettakis, "Depth synthesis and local warps for plausible image-haser navigation," ACM Transactions Graphics (TOG), vol. 32, no. 3, pp. 1-12, 2013.
[8] A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su, "Mysnerf: Fast generalizable radiance field reconstruction from multi-view stereo," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 14 124-14 14-111133 133.
[9] I. Choi, O. Gallo, A. Traccoli, M. H. Kim, and J. Kautz, "Extreme view synthesis," in Proceedings of the IEEE/CVF International Conference Computer Vision, 2019, pp. 7781-7790.
[10] E. De Aguiar, C. Stoll, C. Theobalt, N. Ahmed, H.-P. Seidel, and S. Thrun, "Performance capture from sparse multi-view video," in ACM SIGGRAPH 2008 papers, 2008, pp. 1-10.
[11] I.. Ding and A. Goshtasby, "On the canny edge detector," " Pattern recognition, vol. 34, no. 3, pp. 721-725, 2001.
[12] Y. Du, Y. Zhang, H.-X. Yu, Tenenbaum, and Wu, "Neural radiance flow for 4d view synthesis and video processing," in 2021 /EEE/CVF International Conference on Computer Vision (ICCV). IEEE Computer Society, 2021, pp. 14304-14314.
[13] Fang, Yi, X. Wang, L. Xie, X. Zhang, Liu, M. Nießner, and Q. Tian, "Fast dynamic radiance fields with aware neural voxels," in SIGGRAPH Asia 2022 Conference Papers, 2022, pp. 1-9.
[14] J. Flynn, M. Broxton, P. Dehevee, M. DuVall, G. Fyffe, Overbeck, N. Snavely, and R. Tucker, "Deepview: View synthesis with learned gradient descent," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2367-2376.
[15] J. Flynn, Neulander, Philbin, and N. Snavely, "Deepstereo: Learning predict new views from the world's s imagery," " in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 5515-5524.
[16] C. Gao, A. Saraf, J. Kopf, and J.-B. Huang, "Dynamic view synthesis from dynamic monocular video," Proceedings of the JEFF/CVF International Conference on Computer Vision, 2021, pp. 5712-5721.
[17] H. Gao, R. Li, S. Tulsiani, B. Russell, and A. Kanazawa, "Monocular dynamic view synthesis: A reality check," in Advances in Neural Information Processing Systems, 2022.
[18] C. Godard, O. Mac Aodha, and G. J. Brostow, "Unsupervised monocular depth estimation with left-right consistency." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 270-279.
[19] S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, "The lumigraph." in Proceedings of the 23rd annual conference Computer graphics and interactive techniques, 1996, pp. 43-54.
[20] J. Gu, L. Liu, P. Wang, and C. Theobalt, "Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis." arXiv preprint arXiv:2 110.0885, 2021.
[21] K. Guo, P. Lincoln, P. Davidson, J. Busch, X. Yu, M. Whalen, G. Harvey, S. Orts-lsco,ano, R. Pandey,
J. Dourgarian, al., "The relightables: Volumetric performance capture of humans with realistic relighting." ACM Transactions Graphics (ToG), vol. 38, no. 6. pp. 1-19, 2019.
[22] R. Hartley and A. Zisserman, Multiple view geometry in computer vision. Cambridge university press. 2003.
11

---

[23] K. He, G. Gkioxari. P. Dollár, and R. Girshick, "Mask r-cnn." in Proceedings of the IEEE international conference on compuder vision, 2017. pp. 2961-2969.
[24] P. Hedman, J. Philip, T. Price, J.MM. Frahm, G. Drettakis, and G. Brostow, "Deep blending for free- viewpoint image-based rendering," ACM Transactions on Graphics (TOG), vol. 37, no. 6, pp. 1-15, 2018.
[25] S. Hu, K. Zhou, K. Li, I. Yu, I., Hong. T. Hu, 7. Li, G. H. Lee, and 7. Liu, "Consistentnerf: Enhancing neural radiance fields with 3d consistency for sparse view synthesis," arXiv preprint orXiv: 1031, 2023.
[26] Z. Huang, X. Shi, C. Zhang. Q. Wang, K. C. Cheung. H. Qin, J. Dai, and H. Li, "Flowformer: A transformer architecture for optical flow," in Computer Vision-ECCV October 23-27, 2022, Proceedings, Part XVII. Springer, 2022, pp. 668-685.
ECCV 2022: 17th European Conference, Tel Aviv, Israel,
[27] W. Jang and L. Agapito, "Codenerf: Disentangled neural radiance fields for object categories." in Proceed- ings of the IEEE/CVF International Conference Computer Vision, 2021, pp. 12949 12958.
[28] N. K. Kalantari, T.-C. Wang, and R. Ramamoorthi, "Learrnigg-haeed view synthesis for light field cameras," ACM Transactions on Graphics (TOG), vol. 35, no. 6. pp. 1-10, 2016.
[29] J. Kopf, F. Langguth, D. Scharstein, R. Szeliski, and M. Goesele, "Image-basse rendering in the gradient domain," ACM Transactions on Gruphics (TOG), vol. 32, no. 6, pp. 2013.
[30] Levoy and Hanrahan, "Light field rendering," " in Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 1996, pp. pp. 31-42.
[31] 7. Li, Niklaus, N. Snavely, and O. Wang. "Neural scene flow fields for space-time view synthesis of dynamic scenes," in Proceedings of the IEEÉ/CVF Conference on Computer Vision and Puttern Recognition. 2021. pp. 6498-6508
[32] Z. Li, Q. Wang, F. Cole, R. Tucker, and N. Snavely, "Dynibar: Neural dynamic image-based rendering." " arXiv preprint arXiv:2211.11022 2022.
[33] K.-E. Lin, L. Xiao, F. Liu, G. Yang. and R. Ramamoorthi, "Deep 3d mask volume for view synthesis of dynamic scenes," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1749-1758.
[34] L. Liu, J. Gu, Zaw Lin, T.-S. Chua, and C. Theobalt, "Neural sparse voxel fields." Advances in Neural Information Processing Systems, vol. 33, pp. 15 651--5 663, 2020.
[35] Y.-L. Liu, C. Gao, A. Meuleman, H.-Y. Tseng, A. Saraf, C. Kim, Y.-Y. Chuang, J. Kopf, and J.-B. Huang, "Robust dynamic radiance fields," arXiv preprint arXiv:2301.02239) 2023.
[36] S. Lombardi, Simon, J. Saragih, G. Schwartz, A. Lehrmann. and Y. Sheikh, "Neural volumes: Learning dynamic renderable volumes from images," ACM Trans. Graph., vol. 38, no. 4, pp. 65:1-66:14, July 2019.
[37] B. Mildeahall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. "Nerf: Representing scenes as neural radiance fields for view synthesis." in European conference on computer vision. Springer, 2020, pp. 405-421.
[38]
"Nerf: Representing scenes as neural radiance fields for view synthesis," " Communications of the ACM, vol. 65, no. 1, pp. 99 199, 2021.
[39] M. Niemeyer, J. Barron, B. Mildenhall, M. S. Sujjadi, A. Geiger, and N. Radwan, "Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4490-
[40] M. Niemeyer and A. Geiger, "Giraffe: Representing scenes as compositional generative neural feature fields," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2021,
pp. 11 413--11 464.
[41] M. Niemeyer, L. Mescheder, M. Occhsle, and A. Geiger, "Diffcrentiable volumetric rendering: Learning implicit 3d representations without 3d supervision," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2020, pp. 3504-3515.
[42] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla, "Nerfies: Deformable neural radiance fields," " in Proceedings of the IEEF/CVF International Conference on Computer Vision, 2021, pp. 5865-5874.
[43] K. Park, U. Sinha, P. Hedman, J. T. Barron, S. Bouaziz, D. B. Goldman, R. Martin-Brually and S. M. Seitz, "Hypemerf: A preprint arXiv: 206. 13228, 2021.
representation for topologically varying neural radiance fields," arXiv
[44] Y. Peng, Y. Yan, S. Liu, Y. Cheng, S. Guan, B. Pan, G. Zhai, and X. Yang, "Cagenerf: Cage-based neural radiance field for generalized 3d deformation and animation," Advances in Neural Information Processing Systems, vol. 35, pp. 31 402-3 415, 2022.
12

---

[45] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer, "D-nerf: Neural radiance fields for dynamic scenes," in Proceedings of the IEEE/CVE Conference Computer Vision and Puttern Recognition, 2021, pp. 10318-10327,
[46] R. Ranftl, A. Bochkovskiy, and V. Koltun, "Vision transformers for dense prediction." in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 12 179- 12 188.
[47] R. Ranftl, K. Lasinger, D. Hafner. K. Schindler, and V. Koltun, "Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer." IEEE transuctions on pattern analysis and machine intelligence, vol. 44, no. 3. pp. 1623-1631 2020.
[48] G. Riegler and V. Koltun, "Free view synthesis." in Computer Vision- ECCV 2020: 16th European Conference, Glasgow. UK, August 23-28, 2020, Proceedings, Part XIX 16. Springer, 2020, pp. 623-640.
[49] K. Schwarz, Y. Liao, M. Niemeyer, and A. Geiger, "Graf: Generative radiance fields for 3d-aware image synthesis," Advances in Neural Information Processing Systems, vol. 33, pp. 20 154-20 166, 2020.
[50] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, "Implicit neural representations with periodic activation functions," Advances in Neural Information Processing Systems, vol. 33, pp. 7462-7473. 2020.
[51] V. Sitzmann, J. Thies, F. Heide, M. Nießner, G. Wetzstein, and M. Zollhofer, "Deepvoxels: Learning persistent 3d feature embeddings." in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2019, pp. 2437 4446.
[52] V. Sitzmann, M. Zollhöferr and Wetzstein, "Scene representation networks: Continuous 3d-structure- aware neural scene representations," Advances in Neural Information Processing Systems, vol. 32, 2019.
[53] P. P. Srinivasan, R. Tucker, J. T. Barron, R. Ramamoorthi, R. Ng, and N. Snavely, "Pushing the boundaries of view extrapolation with multiplane images," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019. pp. 175-184.
[54] 7. Teed and J. Deng, "Raft: Recurrent all pairs field transforms for optical flow," in Computer Vision-FCCCV 2020: I 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Purt II 16. Springer. 2020, pp. 402-419.
[55] E. Tretschk, A. Tewari, V. Golyanik, M. Zollhöfer, C. Lassmer, and C. Theobalt, "Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. pp. 12959- 12 970.
[56] A. Trevithick and B. Yang. "Grf: Learning a general radiance field for 3d representation and rendering," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 15 182-15 192.
[57] C. Wang, B. Eckart, S. Lucey, and O. Gallo, "Neural trajectory fields for dynamic novel view synthesis," " arXiv preprint arXiv:2105,00 059944 2021.
[58] Q. Wang, Z. Wang, K. Genova, P. Srinivasan, H. Zhou, J. T. Barron, R. Martin Brualla, N. Snavely, and T. Funkhouser, "Ibrnet: Learning multi-view image- based rendering," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. (69004699.
[59] S. Wizadwongsa, P. Phongthawee, J. Yenphraphai, and S. Suwajanakorn, "Nex: Real-time view synthesis with neural basis expansion." in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021. pp. 8534-8543.
[60] H. Xu, T. Alldieck, and C. Sminchisescu, "H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion," Advances in Neural Information Processing Systems, vol. 34, pp. 14955- 166, 2021.
[61] Q. Xu, Z. Xu, J. Philip, S. Bi, Z. Shu. K. Sunkavalli, and U. Neumann, "Point-nerf: Point-based neural radiance fields." in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2022, pp. 5438-5448.
[62] G. Yang, M. Vo, N. Neverova, D. Ramanan, A. Vedaldi, and H. Joo, "Banmo: Building animatable 3d neural models from many casual videos." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2863-2873.
[63] G. Yang. C. Wang, N. D. Reddy, and D. Ramanan, "Reconstructing animatable categories from videos." arXiv preprint 2xXiv:2305.06.33
2023.
[64] J. S. Yoon, K. Kim, O. Gallo, H. S. Park, and Kautz, "Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 5336-5345.
[65] A. Yu. V. Ye. M. Tancik. and A. Kanazawa, "pixelnerf: Neural radiance fields from few images." "" in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 478-4587.
[66] K. Zhang, G. Riegler, N. Snavely, and V. Koltun, "Nerft+: Analyzing and improving neural radiance fields," arXiv preprint arXiv:2010.0749,, 2020.
13

---

[67] Z. Zhang, F. Cole, Z. Li. M. Rubinstein, N. Snavely, and W. T. Freeman, "Structure and motion from casual videos." in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Ivruel, October 23-27, 2022, Proceedings, Part XXXIII. Springer, 2022, pp. 20-37.
[68] K. Zhou, L. Hong, C. Chen, H. Xu, C. Ye, Q. Hu, and Z. Li, "Devnet: supe-supe monocular depth learning via density volume construction," in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 2022, Proceedings. Part XXXIX. Springer, 2022, pp. 125-142.
[69] T. Zhou, R. Tucker, Flynn, G. Fyfle, and N. Snavely. "Stereo magnification: Learning view synthesis using multiplane images," arXiv preprint arxiv: /80.:0888000877777 2018.
14


Reviews:
[{"review_id": "WEEjxGMc5u", "reviewer": "Reviewer_J3AP", "summary": "The paper proposes a method for dynamic scene synthesis from monocular video. The authors aim to speed up training and deal with long-duration videos. To do this, they propose to estimate scene flows of surface points supervised by the signals which are generated by the proposed consistent depth algorithm. Then the information is aggregated to form a neural point cloud and novel view images are rendered by sampling amount this point cloud.", "questions": "1.\tWhat is the render speed of the method, considering the relative complex neural point cloud construction process.\n2.\tAn open question: is it possible to extend this method to a generalizable method, which could generalize to new scenes without training? For example, replace scene flow MLP with an un-learnable method and use a generalizable renderer.\n3.\tEquation 7 is confused. The warped 2K point clouds are summed or combined?\n", "limitations": "The authors addresed the limitation I concerned, which is the potential failure caursed by the flaws in explicit depths and flows.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper proposes a novel solution to dynamic scene. Instead of estimating density and color of the scene for each time, the paper explicitly warps the pixel features of reference images to target time according to the explicitly reconstructed 3D geometry. This would reduce the time for training, as there less learnable components.\n2.\tThe experiment shows SOTA results on Nvidia dataset.\n", "weaknesses": "1.\tPlease consider to rewrite the Equation 3.\n2.\tThe results on Nerfie dataset and Iphone dataset are less consistent compared to Nvidia dataset. Why not test TiNeuVox on Nerfie and Iphone dataset? Or test the proposed method on HyperNeRF dataset?\n3.\tPlease highlight all best results in Table 2 & Table 3.\n4.\tThe paper claim that the method could handle long-duration video, which I agree. However, no experiment is conducted to prove this ability.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Section 1 - Reviewer J3AP\n\n\nWe thank the reviewer for the constructive assessment of our work. In the following, we address the concerns point by point. Please feel free to use the discussion period if you have any additional questions.\n\n## 1.1 Weakness\n\n**1.1.1 Equation 3**\n\nThank you for pointing it out. The formulation will be adjusted to incorporate the predefined mathematical symbols as follows: \n$s_{t-t'}(p_{t})= \\hat{d_{t'}}(p_t+f_{t-t'}(p_t)) R_{t'} K_{t'}^{-1}  (p_t+f_{t-t'}(p_t)) + t_{t'} - P_t.$\n\n**1.1.2 Performance on iPhone dataset**\n\nThank you for your careful assessment. The smaller gaps in the iPhone dataset are due to its limited information from various angles, as indicated by \"effective multi-view factors\" proposed by DynCheck [17]. This dataset naturally has fewer diverse camera views, making the reconstruction of appearance more challenging compared to the Nvidia and Nerfie.\n\n**1.1.3 Tested on HyperNeRF**\n\nWe greatly appreciate your reminder regarding HyperNeRF's significance as a pivotal benchmark for evaluating monocular video view synthesis algorithms. We have indeed included an exposition of our algorithm's performance on this dataset within the \"global\" response PDF document. For clarity, we've included a subset of results here.\n\n### Novel View Synthesis Results of HyperNeRF Dataset\n\n|  PSNR ↑ / LPIPS ↓                   | Broom | 3D Printer | Chicken | Expressions | Peel Banana | Average |\n|----------------------------|--------------|--------------|--------------|---------------|---------------|----------------|\n| Hyper-NeRF        | 20.60 / 0.613      | 21.40 / 0.212            | 27.60 / 0.108       | 22.00 / 0.196            | 24.30 / 0.170            | 23.20 / 0.260 |\n| NSFF        | 26.10 / 0.284      | 27.70 / 0.125            | 26.90 / 0.106       | 26.70 / 0.157            | 24.60 / 0.198            | 26.40 / 0.174 |\n| DynPoint          | **27.40 / 0.248**       | **27.60 / 0.163**            | **28.10 / 0.089**       | **27.90 / 0.147**            | **26.50 / 0.129**            | **27.50 / 0.155** |\n\n\n**1.1.4 Table 2 \\& Table 3**\n\nCertainly, highlighting all the best results will improve reader understanding. We will ensure that all the best results are emphasized in our final version.\n\n**1.1.5 Long video**\n\nWe acknowledge your astute observation. Owing to the constraints imposed by length considerations, the answer could be found for the second weakness posed by reviewer Rm9i (Sec. 3.1.2).\n\n\n## 1.2 Questions\n\n**1.2.1 Rendering speed**\n\nIndeed, while our emphasis does not lie in rendering speed optimization, our method outperforms standard NeRF-based approaches in this regard. Demonstrating around 2.78-fold acceleration relative to D-NeRF, our technique capitalizes on the point cloud representation to efficiently bypass unoccupied space. Furthermore, the construction of the neural point cloud is remarkably swift, benefiting from the scene flow computation focusing on surface points. This stands in contrast to the computational demands of D-NeRF and NSFF, which necessitate the prediction of motion for all sampled points (192 samples for D-NeRF and 128 samples for NSFF) along the ray originating from each pixel.\n\n**1.2.2 Generalizability**\n\nThank you sincerely for raising this valuable point. Designing a generalizable model for a monocular-video-based view synthesis task is always challenging. Just as you said, to make DynPoint generalizable, we also need a generalizable scene flow estimation model. We are currently dedicating our efforts to devising a generalizable approach for estimating scene flow without relying on per-scene fine-tuning in our upcoming work.\n\nNevertheless, we acknowledge that monocular scene flow estimation presents a formidable challenge due to occlusion caused by both the moving camera and dynamic objects, which can hinder accurate estimation. In this context, we have found significant insights from the prevalent usage of diffusion models in 3D scenarios [69,70], which realize generalizable monocular 3D reconstruction. We hope that the development of such a generalizable scene flow estimation model will greatly enhance the applicability of our method,  which could serve as a direct replacement for our current first stage.\n\n[69] Zero-1-to-3: Zero-shot One Image to 3D Object.\n\n[70] One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization.\n\n**1.2.3 Equation 7**\n\nThank you for bringing up this crucial observation. The central concept of our paper is aggregating information from adjacent frames to improve the inference of appearance and geometry information for the current frame. While both summation and combination methods could potentially be utilized for information aggregation, our experiments reveal that summation is not feasible in practice. This is primarily due to the challenge of identifying two perfectly overlapping points in the 3D space with predicted scene flow, which makes the summation unreliable. \n\nTo address this concern, we have opted for the combination method, which effectively densifies the neural point cloud of the current frame by integrating information from nearby frames. This approach offers a viable solution to achieve information fusion without encountering the issues associated with direct summation."}, {"review_id": "BiLVVstxdq", "reviewer": "Reviewer_nc3b", "summary": "This paper proposes an algorithm for novel-view synthesis in dynamic scenes. According to the abstract and introduction section, this paper takes a monocular video as an input (L.98) and cope with uncontrolled or length scenarios (L.3). Using geometric priors, such as monocular depth and optical flow from off-the-shelf methods, this paper exploits the pixel-wise correspondence to encode sceneflow within the network.  ", "questions": "I hope that the authors properly address my concerns which are listed in the weakness section.\n\nOverall, this paper does not provide the whole information that is necessary for the reviewers to properly judge its validity. \n", "limitations": "It's okay to me.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "Overall, I am a bit not that positive for this paper. Let me put more comments on the weakness section.", "weaknesses": "_1. Wrong experimental setup._\n\nThis paper claims that they focus on rendering scenes from a monocular video to handle lengthy video. However, this is not true. NVIDIA dataset [63] captures the scenes using a camera rig (2x8 array camera? not sure), which is not a monocular video. I think that the correct meaning of the monocular video is addressed by the Dycheck dataset [17] which is called iPhone dataset [17] in this manuscript. Moreover, NVIDIA dataset [63] only has a few training frame which is not well aligned with the addressed points by the authors in the abstract and introduction. At least, HyperNeRF dataset [42] provides more than 200~ frames per scene, where I could say this is a bit lengthy video. Such inconsistency in addressed problems and the proposed solution is quite not good. I am not that satisfied with the quality of the writing.\n\nMoreover, where can I find the qualitative results on Dycheck dataset [17]? While table 3 provides the quantitative results, I could not find any qualitative results. \n\n\n_2. No video demo_\n\nMore seriously, though this paper tackles the neural rendering in dynamic scenes, I could not find any video demo even in the supplementary material. __Lack of video demo is a big problem__.\n\n\n_3. What is the difference in technical contribution in comparison with NSFF [30]?_\n\nNSFF [30] also exploit the sceneflow understanding within the network, similar to this paper. What is the main difference? At least authors mainly discuss this issue and provide some quant/qual difference within their experimental section. But I could not find any difference or novelty in this part.\n\n\n_4. Lack of related papers to address the speed in dynamic NeRFs._\n\nThis paper is not the pioneering work that provides fast rendering/training speed in dynamic NeRF. I hope that the authors to look at the paper of TiNeuVox, Siggraph Asia 2022 (https://github.com/hustvl/TiNeuVox). If this paper wants to claim the novelty in speed, the authors should have fairly compared with TiNeuVox.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Section 2 - Reviewer nc3b\n\n\nWe thank the reviewer for the constructive assessment of our work. In the following, we address the concerns point by point. Please feel free to use the discussion period if you have any additional questions.\n\n## 2.1 Weakness\n\n**2.1.1 Wrong experimental setup**\n\nWeakness Regarding Nvidia Dataset: Your observations are greatly appreciated. We agree with your assessment regarding the limitations of the Nvidia dataset, particularly within the context of monocular video-based view synthesis, a facet also addressed by Dycheck [17]. Despite this, our utilization of the Nvidia dataset is driven by its widespread adoption across related literature in this domain. This deliberate choice enhances the comparability of our algorithm with other approaches.\n\nInquiries Regarding HyperNeRF Dataset: We extend our gratitude for bringing this matter to our attention. The performance of our algorithm on the HyperNeRF dataset has been incorporated into our comprehensive response, available in the appended PDF document of \"global\" response. These newly added experimental results on the HyperNeRF dataset consistently demonstrate the improvement achieved by our approach. For clarity, we’ve included a subset of results in the reviewer J3AP (Sec. 1.1.3). The video link of this dataset could be found on \"Official Comment\" to Associate Chair (AC). \n\nInquiries Regarding Dycheck (iPhone) Dataset: We appreciate your keen observations. Our response includes the results produced by our algorithm when we tested it on the iPhone dataset. It's important to note that the iPhone dataset presents challenges, and our algorithm struggles to generate clear views in most situations. The demo for Dycheck (iPhone) and Nerfies Dataset could be found on the PDF document of rebuttal to Associate Chair (AC) at the top of the review page.\n\n**2.1.2 Video demo**\n\nWe're grateful for your insightful observation, and we recognize how a video demo could really make our paper clearer. Following the instructions for the rebuttal process, we've added an anonymous link to the video demo we've made for the HyperNeRF dataset. You can find this link in our response to the Associate Chair (AC) under the \"Official Comment\" box at the top of the review page.\n\n**2.1.3 Difference with NSFF**\n\nWe want to express our gratitude for your perceptive observation. The NSFF paper has undeniably made a significant contribution to the realm of monocular view synthesis by integrating scene flow into the framework of monocular video-based algorithms. However, there are distinct differences that set NSFF and DynPoint apart.\n\nFirstly, DynPoint introduces a corrective mechanism to adjust the depth scale obtained from monocular videos. This correction greatly improves the consistency of geometry across different frames.\n\nFurthermore, NSFF predicts flow for all points (128) along a ray, while DynPoint focuses exclusively on predicting scene flow for surface points. This approach brings two important advantages: firstly, it adds specific constraints to the scene flow of surface points by using predicted optic flow and depth information, which significantly enhances the learning process; secondly, by concentrating on surface points, the scene flow prediction process is made faster.\n\nLastly, unlike NSFF, which encodes information extensively within the MLP, DynPoint gathers information from neighboring frames using 3D correspondence. This unique approach empowers DynPoint to more effectively learn from sequences of long videos.\n\n**2.1.4 Comparison with TiNeuVox**\n\nWe want to acknowledge your input, which is truly valuable. In our submitted work, we've referenced TiNeuVox and carried out a comparison of its performance with our method using the Nvidia dataset. TiNeuVox is a notable effort that aims to create a more effective way to represent geometry, appearance, and motion information in dynamic situations. While it has its strengths, there are noticeable differences between DynPoint and TiNeuVox.\n\nFirstly, TiNeuVox doesn't explicitly learn about scene flow. Instead, it adopts a method that learns about motion in an indirect manner through a specially designed voxel-style representation. This implicit way of representing motion can be challenging to control effectively during training, which can contribute to its decreased performance in the Nvidia dataset.\n\nSecondly, TiNeuVox tries to capture the entire dynamic scenario within an improved representation. However, this comprehensive approach can face difficulties when dealing with long videos, as it might require more computational resources and memory due to its complexity.\n\nLastly, we've included a speed comparison in Table 1 of the main text. From this table, we could notice that our model performs faster and achieves higher accuracy when compared to TiNeuVox on the Nvidia dataset."}, {"review_id": "E6MRDz4mDQ", "reviewer": "Reviewer_Rm9i", "summary": "The paper proposes a novel method for performing novel view synthesis from monocular captured videos. This is done by learning scene flow and depth parameters which enable the accurate aggregation of appearance information from nearby frames. The paper proposes a novel method of acquiring both consistent depth estimations for a video, and consistent scene flow represented by an MLP. These two sources of information are used to synthesize novel views at a specific time from an arbitrarily specified viewpoint by sampling from corresponding information in a window of nearby frames. The proposed method is demonstrated to outperform existing methods based on representation learning in synthesizing novel views from monocular videos, and this is shown for a wide variety of datasets.\n\nAFTER REBUTTAL:\n\nI have read the author rebuttal, and believe that it has addressed my questions. I appreciate the detailed discussion on the comparison to DynIBaR. I also believe the additional videos provided have definitely helped improve my opinion of the method, although I wish the method was tested with more qualitative video examples (as mentioned by another reviewer). However, I believe the testing now is sufficient.", "questions": "- How does the generated consistent video depth estimation compare to other work in this field, for example: https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/. Could this method be dropped in and used for this estimation step?", "limitations": "The paper adequately addresses the limitations of the method. One additional limitation that discussion of would significantly improve the quality of the paper, is the amount of differing from the monocular video path which can be taken and still synthesize reasonable novel views. Obviously, since this method is using appearance information in nearby frames, it is likely to not be able to hallucinate information beyond these. However, NeRF-based methods are able to put something there and potentially reconstruct better results due to the smoothness of the representation learned by the MLP. Some comparison of the existing methods and the proposed method on this axis (range of possible novel views synthesized) would be helpful, and if this is a limitation of the proposed method then it should be discussed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "In my opinion, the main strengths of the paper are that:\n1. The presentation of the paper is concise, clear, and thorough.\n    - The introduction and related work I find to be comprehensive, and accurately describe what the contributions are and what the current issue is with the practice.\n    - The methods section is described in detail, and is relatively clear to understand and follow.\n    - The figures are all informative and demonstrate the capabilities of the method. One thing I wish could be added is a video result demonstrating consistent novel view synthesis.\n2. The proposed contributions are original and impactful.\n    - The module for estimating consistent depth for a monocular video seems novel, and while there is existing work tackling this (see questions), the method proposed seems to generate good enough results for novel view synthesis with DynPoint.\n    - The method for estimating and representing the scene flow between adjacent frames and using this to aggregate information from reference to target frames seems novel (although based on image-based rendering ideas), and learning this from the video seems like an important contribution.\n3. The evaluations are sound.\n    - The evaluations are done for many (4) different datasets, and compare to many different dynamic scene representation methods (Nerfies, D-NeRF, NSFF, HyperNeRF among others). Because of how thorough these evaluations are, and DynPoint outperforms all of the baseline methods, it is very convincing that the proposed method is robust and is able to generate high-quality results.\n    - The ablations are relatively thorough, and Table 4 shows how the individual parts of the representation and aggregation pipeline contribute to the final quality. I found it to be informative about the method.", "weaknesses": "I do not view there to be many major weaknesses of the paper. One thing which would be interesting is a comparison to the method DynIBaR [31]. The paper mentions that this method is limited to short videos due to the capacity of MLPs or other representation mechanisms (L34), but from my understanding this method uses a similar philosophy as this work where new frames are synthesized as inspired by image-based rendering and scene flow estimation. This seems to me to be the current state-of-the-art. Additionally, it’s not clear that the videos this method is evaluated on are long enough for this to be a concern, considering NeRF-based methods can be evaluated and these are constrained by the capacity of the MLP being used to represent the scene. Additionally, I think there could be a bit more discussion on the limitations of the proposed method (see limitations).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Section 3 - Reviewer Rm9i\n\nWe thank the reviewer for the constructive assessment of our work. In the following, we address the concerns point by point. Please feel free to use the discussion period if you have any additional questions.\n\n## 3.1 Weaknesses\n\n**3.1.1 Comparison with DynIBaR**\n\nThis is an insightful observation. Indeed, our perspectives align closely, and we are concurrent work. Nevertheless, discernible distinctions between DynPoint and DynIBaR exist, serving to underscore the robustness of the proposed DynPoint methodology.\n\nBoth DynIBaR and DynPoint harness information aggregation mechanisms to realize the synthesis of novel views. However, DynIBaR predominantly centers around the aggregation of information through two-dimensional (2D) pixel units. This approach draws inspiration from image-based rendering principles, entailing the synthesis of novel perspectives from a collection of reference images via a weighted fusion of reference pixels. In contrast, DynPoint's focal point lies in the information aggregation achieved by constructing three-dimensional (3D) neural point clouds. The final novel view synthesis is realized by using 3D neural points surrounding the queries' position. \n\nMoreover, DynPoint introduces an efficacious strategy for the seamless integration of monocular depth estimation within the ambit of monocular video view synthesis. In contrast to DynIBaR, which endeavors to model the trajectory of all samples (128) traversing each ray, DynPoint exclusively focuses on the trajectory of surface points, thereby yielding a substantial acceleration in both the training and inference stages. \n\n**3.1.2 Long video**\n\nWe value your insight. To address this, we examined our model using the iPhone dataset, encompassing numerous images (several hundred frames) across different scenarios in comparison to Nerfie and Nvidia datasets. Our model demonstrated stable performance within this context. However, we acknowledge a minor improvement in the iPhone dataset compared to the significant enhancements seen in Nvidia and Nerfie datasets. This discrepancy can be attributed to the limited information available from various angles in the iPhone dataset, as indicated by the \"effective multi-view factors\" proposed by DynCheck [17].\n\nTo further explore DynPoint's capabilities with longer videos, we followed Review J3AP's suggestion and conducted relevant experiments on the HyperNeRF dataset in the Reviewer J3AP (Sec. 1.1.3), which also contains hundreds of frames per scenario. The results and our algorithm's proficiency in handling longer videos are presented in the attached PDF document. For clarity, we’ve also included a subset of results in the reviewer J3AP (Sec. 1.1.3).\n\n## 3.2 Questions\n\n**3.2.1 Consistent-Video-Depth-Estimation**\n\nThank you for bringing this up. It's a notable point, as the paper you mentioned indeed provides valuable insights. Both our methods draw inspiration from traditional structure-from-motion reconstruction principles to establish the connection between monocular depth and optical flow. It's possible that their algorithm could be integrated into our depth estimation process.\n\nHowever, there are distinct differences between our DynPoint approach and theirs. While their focus is primarily on achieving accurate depth estimation, which is also a key aspect of our work. Our method focuses on estimating scene flow and depth information together. \n\nAnother difference lies in the requirements of our methods. While their approach involves fine-tuning a complex convolutional neural network-based depth estimation architecture tailored for each scenario, our method calls for fine-tuning a simpler scene flow MLP structure and learnable scale factors of the depth map. This divergence significantly reduces the computational demands during the optimization process.\n\n## 3.3 Limitations\n\n**3.3.1 Limitation concerning global information**\n\nThank you for bringing this up. Initially, we were concerned about the extent to which global information might impact our method. Unlike approaches such as NSFF or Nerfie, which capture comprehensive scene details for predictions, our method showed promising performance on the iPhone dataset, which contains multiple frames for each scenario. Additionally, the newly added experimental results on the HyperNeRF dataset during the rebuttal period consistently demonstrate the improvement achieved by our approach. The reason for this success might be that even though we only use information from nearby frames to predict stuff, we actually train our scene flow (MLP structure) and render (MLP structure) parts using the whole video, which encodes some global information across frames into our model and empowers our model to make a reasonable inference. More demos on Nerfies and iPhone datasets could be found on the PDF document of the rebuttal to the Associate Chair (AC) at the top of the review page. The video link of HyperNeRF could be found in our official comment.\n\nWe also experimented with our model using different numbers of nearby frames, and we explain these results in our ablation studies. What we found is that when we use fewer frames, our model doesn't work as well. But when we increase the number of frames up to a certain point, the improvement becomes less noticeable. One reason for this could be that keeping track of things over a long time doesn't always work perfectly. Another reason could be that the camera and objects move in ways that are hard to predict, so having a longer video might not necessarily give us much more useful information.\n"}, {"review_id": "easjsrXAtG", "reviewer": "Reviewer_BEKz", "summary": "DynPoint is an algorithm proposed to enhance the synthesis of novel views in monocular videos using neural radiance fields. It focuses on predicting explicit correspondence between neighboring frames, enabling information aggregation for view synthesis. The demonstrated experimental results show a significant reduction in training time, comparable outcomes to previous approaches, and robustness in handling uncontrolled and lengthy videos without requiring a canonical representation of video content. To achieve it authors propose to improve the estimated scene flow from the optical flow with corrected depth values ", "questions": "1. How do you measure the time? Does it include phase 1 and the hardware is the same as competitors?\n2. “Neural point cloud” technique is similar to [1, 2, 3] representations in some sense (humans & static scene examples). Could you extend related works in the neural point cloud as well? Does hierarchy is an important contribution for the neural point cloud? If so, it can be \n3. What is the main drawback of the method for iPhone dataset? Is it possible to extend the comparison on this benchmark with others, since the gap is much smaller than on others?\n4. Do you consider including video metrics for the results (e.g. VMAF)?\n5. How issues in Figure 5 can be addressed and what is causing them?\n\n[1] PointAvatar: Deformable Point-based Head Avatars from Videos\n[2] Self-Improving Multiplane-To-Layer Images for Novel View Synthesis\n[3] Pulsar: Efficient Sphere-based Neural Rendering", "limitations": "Fast changes in the scene as well as thin objects can be a problem for the method, which is quite common for this class of models. Compared with other methods this one needs more information and additional pretraining steps. the category-specific methods can be better for non-rigid parts. The scalability of the method is limited due to two-stage training for each of the scenes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "the proposed technique for scene flow estimation is a solid contribution. It takes into account two inconsistencies in depth and flow that are denoised with the method.  The improvements can be applied to any baseline as a separate technique to improve baseline (e.g. NSFF). Moreover, this allows extracting correspondences for the arbitrary 3D scene.\nhierarchical scheme to render the semi-explicit structure is a novel and a trustable way to get the fast method based on existing research\nExperiments demonstrate the substantial of the proposed method and provides a strong contribution to the field.\n\n", "weaknesses": "The proposed method relies heavily on existing models and it is unclear whether the quality depends if you replace them with others\nAuthors don't investigate the quality of the correspondence, which can be interesting by itself. The easiest way is to extend section 4.4 with a comparison versus method like \"Deformable Sprites\" Ye et al.\nThe presentation of the visual results can be improved. Figure 3 should have a close-up by emphasizing the region of interest, otherwise, results are not clear as well as others.\nNot compared with “DynIBaR: Neural Dynamic Image-Based Rendering”. Yes, I understand that it is a recent work. At least add numbers to the table for Nvidia dataset.\nWithout video results, it is difficult to reason even with such metric improvements on average.\nNo limitation or ethical section to give a reader explicit statements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Section 4 - Reviewer BEKz\n\nWe thank the reviewer for the constructive assessment of our work. In the following, we address the concerns point by point. Please feel free to use the discussion period if you have any additional questions.\n\n## 4.1 Weakness\n\n**4.1.1 Deformable sprites**\n\nUnlike our initial focus on 3D relationships, \"Deformable Sprites\" aims to learn a non-rigid 2D pixel-wise transformation using video data, enabling motion separation without supervision. This makes a direct comparison between our learned 3D correspondences and their results challenging. But, both approaches utilize predicted optical flow from RAFT [53] to assist in determining the correspondence between two frames. However, the concept of using video to understand unsupervised motion for generating new views is attractive, which enables specific rules for different motion groups.\n\n**4.1.2 Region of interest** \n\nCertainly, highlighting the key area will improve reader understanding. We'll include this in the final version.\n\n**4.1.3 DynIBaR**\n\nDynIBaR is indeed a relevant study to ours. We'll add their metrics in the final version and we also provide a comparison in the Reviewer Rm9i (Sec. 3.1.1).\n\n**4.1.4 Video results** \n\nWe're grateful for your insightful observation. Following the instructions for the rebuttal process, we've added an anonymous link to the video demo we've made for the HyperNeRF dataset. You can find this link in our response to the AC under the \"Official Comment\" box at the top of the review page. More demos could be found in the PDF document to AC.\n\n**4.1.5 Limitation or ethical section**\n\nOur method's primary limitation lies in its generalizability. Addressing this issue would significantly enhance the feasibility of DynPoint. Further details can be found in the Reviewer J3AP (Sec. 1.2.2). \n\n## 4.2 Questions\n\n**4.2.1 Measurement of time** \n\nWe've tested two stages of DynPoint using the same computer setup. \n\n**4.2.2 Neural point cloud** \n\nYour insights are appreciated. Recent findings in Sec. 4.3 (main text), Ablation Studies, confirm the positive impact of our introduced hierarchical structure on overall performance. The three referenced papers offer interesting viewpoints on addressing view synthesis challenges.\n\nIn reference to [1], it highlights neural points for video-based Head Avatars reconstruction. Our approach reconstructs point clouds from refined depth and predicted scene flow, differing from [1]'s emphasis on direct learning of canonical point cloud representation. Our research seeks a universal solution for monocular video-based view synthesis tasks, diverging from [1]'s specific focus. Implementing our hierarchical structure might enhance [1]'s approach.\n\nRegarding [2] and [3], their strategies differ from our neural point methodology. [2] introduces front-parallel planes for static scenarios, and [3] proposes spheres for a similar purpose. \n\n**4.2.3 iPhone dataset**\n\nThe smaller gaps in the iPhone dataset are due to its limited information from various angles, as indicated by \"effective multi-view factors\" proposed by DynCheck [17]. This dataset naturally has fewer diverse camera views, making the reconstruction of appearance more challenging compared to the Nvidia and Nerfie.\n\n**4.2.4 Video metrics** \n\nYour question prompted us to consider a new perspective. We've realized that VMAF has broader applications beyond creating new views from videos. It can be useful for evaluating view synthesis in static scenarios as well. It's intriguing that this aspect hasn't been widely explored in existing research. Therefore, we acknowledge the significance of reevaluating and thoroughly examining the suitability of VMAF's features for view synthesis tasks. \n\n**4.2.5 Figure 5**\n\nFigure 5 highlights three failure instances: facial expression, reflection-induced, and fine object view synthesis. Facial expression reconstruction faces challenges due to humans being sensitive to these artifacts. Models like 3DMM improve this aspect.\n\nReflections can change surface appearances, making it hard to distinguish actual geometry from reflections. Illumination and viewing angles further complicate achieving consistent appearances. Advancements like \"NeRFReN\" Guo, Yuan-Chen, et al. integrate reflections for better results.\n\nView synthesis for fine objects or high-resolution images is an active research area. Innovative architectures, such as \"3D Gaussian Splatting for Real-time Radiance Field Rendering,\" are designed to solve it.\n\n## 4.3 Limitations\n\n**4.3.1 Fast change \\& Thin object**\n\nIndeed, dealing with fast changes and thin objects in scenes poses challenges. Fast changes mean less matching between frames, and thin objects involve complex geometry. Both of these aspects make monocular-video-based view synthesis more difficult.\n\n**4.3.2 Pretraining steps**\n\nYes, the initial pretraining is vital for our approach. However, we use fewer learnable weights, leading to shorter total training times than other methods.\n\n**4.3.3 Category-specific method**\n\nIt's true that incorporating category-specific details can improve the performance of view synthesis models in specific scenarios, e.g., SMPL for human reconstruction and 3DMM for facial expression reconstruction. Our paper focuses on introducing a generalizable algorithm that doesn't rely on specific prior assumptions about objects.\n\n**4.3.4 Scalability**\n\nScalability, ensuring optimal performance with bigger and more complex tasks or datasets, is a consideration for our model. This affects our model's initial stage in DynPoint, involving explicit correspondence learning. But it's worth noting that other one-stage models face similar hurdles. Our experiments with the iPhone Dataset revealed performance declines across models. One-stage approaches like NSFF aim for a universal video representation, which requires implicit frame correspondence. However, the lack of explicit correspondence adds complexity to implicit correspondence learning.\n"}]

Author rebuttal:
# Section 0 - Author rebuttal to ACs

Dear Reviewers, we truly appreciate your thoughtful review, which has been immensely valuable in refining our paper. Your insights have contributed significantly to the enhancement of our work.

## 0.1 Experiments on HyperNeRF
In order to comprehensively evaluate the performance of our method, we undertook an additional experiment using the HyperDataset. This dataset offers several hundred frames for each scenario, enabling a thorough assessment. Our findings from this experiment reinforce that our algorithm consistently exhibits improvements across nearly all scenarios. Some of the experiment's results are in the reviewer J3AP (Sec. 1.1.2) and you can find the full results in the PDF attached.

## 0.2 Video demo

Following NeurIPS 2023's guidelines for responding, we're able to provide the video by sharing a link in the official comment to the Associate Chair (AC). As a result, we've included the link to our demonstration video in our "official comments" to ACs. This video showcases our perspective on the HyperNeRF dataset. In the video, we start by training our model and NSFF (The second best-performing model on the HyperNeRF dataset) using the left video. Then, we show how NSFF generates the middle video by keeping the view constant but changing the input time. Similarly, the right video is generated by DynPoint, where the view is fixed, but the input time changes.

## 0.3 Qualitative results on Nerfies and iPhone

To better illustrate our algorithm, we've included additional examples of the results it produces on the Nerfies and iPhone datasets in our attached PDF (Figure 1 for Nerfies and Figure 2 for iPhone). When comparing DynPoint and NSFF on the Nerfies dataset, we can clearly see the noticeable differences, which align with the findings we presented in our main paper. Moreover, the comparison on the iPhone dataset showcases our algorithm's edge in handling lengthy and intricate dynamic scenarios. Moreover, we recognize that the iPhone dataset poses a unique challenge due to its intentional design. This design intentionally limits the available multi-view information for each frame. As a result, this complexity makes it more challenging for both the correspondence learning and 3D reconstruction processes.

Once again, we extend our sincere gratitude for your insightful review, and we are fully committed to addressing any further inquiries or suggestions you may have.



Meta-review: This paper presents a DynPoint for novel view synthesis for Monocular video. This paper is rated as 6/3/7/7. Most reviewers acknowledged the technical contributions of the paper. In the rebuttal period, the authors and the reviewers have in-depth discussions. Most of the reviewer's questions have been addressed. The remaining concern of the reviewer who rated this paper as 3 is about the insufficient demos, and the authors have promised to do more video comparisons and release the videos later.  Thus I recommend accepting the paper. But I also strongly suggest authors show more video results of different methods in the supplementary.
Decision: Accept (poster)