Title: DynPoint: Dynamic Neural Point For View Synthesis
Venue: NeurIPS 2023
Accepted: yes
Keywords: View Synthesis

Monocular Video
Abstract: The introduction of neural radiance fields has greatly improved the effectiveness of view synthesis for monocular videos. However, existing algorithms face difficulties when dealing with uncontrolled or lengthy scenarios, and require extensive training time specific to each new scenario.
To tackle these limitations, we propose DynPoint, an algorithm designed to facilitate the rapid synthesis of novel views for unconstrained monocular videos. 
Rather than encoding the entirety of the scenario information into a latent representation, DynPoint concentrates on predicting the explicit 3D correspondence between neighboring frames to realize information aggregation.
Specifically, this correspondence prediction is achieved through the estimation of consistent depth and scene flow information across frames.
Subsequently, the acquired correspondence is utilized to aggregate information from multiple reference frames to a target frame, by constructing hierarchical neural point clouds. 
The resulting framework enables swift and accurate view synthesis for desired views of target frames. 
The experimental results obtained demonstrate the considerable acceleration of training time achieved - typically an order of magnitude - by our proposed method while yielding comparable outcomes compared to prior approaches. Furthermore, our method exhibits strong robustness in handling long-duration videos without learning a canonical representation of video content.

DynPoint: Dynamic Neural Point For View Synthesis

Abstract

The introduction of neural radiance fields has greatly improved the effectiveness of view synthesis for monocular videos. However, existing algorithms face difficulties when dealing with uncontrolled or lengthy scenarios, and require extensive training time specific to each new scenario. To tackle these limitations, we propose DynPoint, an algorithm designed to facilitate the rapid synthesis of novel views for unconstrained monocular videos. Rather than encoding the entirety of the scenario information into a latent representation, DynPoint concentrates on predicting the explicit 3D correspondence between neighboring frames to realize information aggregation. Specifically, this correspondence prediction is achieved through the estimation of consistent depth and scene flow information across frames. Subsequently, the acquired correspondence is utilized to aggregate information from multiple reference frames to a target frame, by constructing hierarchical neural point clouds. The resulting framework enables swift and accurate view synthesis for desired views of target frames. The experimental results obtained demonstrate the considerable acceleration of training time achieved - typically an order of magnitude - by our proposed method while yielding comparable outcomes compared to prior approaches. Furthermore, our method exhibits strong robustness in handling long-duration videos without learning a canonical representation of video content.

1 Introduction

The computer vision community has directed significant attention towards novel view synthesis (VS) due to its potential for both emerging techniques in artificial reality and also to enhance a machine's ability to comprehend the appearance and geometric properties of target scenarios [37, 8, 39]. Stateof-the-art techniques leveraging neural rendering algorithms, as demonstrated in studies such as [37, 58, 61], have successfully attained photorealistic reconstruction of views in static scenarios. However, the dynamic characteristics inherent to most real-world scenarios present a formidable challenge to the suitability of existing approaches that rely on the epipolar geometric relationship, traditionally applicable to static scenarios [22, 45].

Recent studies have primarily focused on the synthesis of views in dynamic scenarios by employing one or multiple multilayer perceptrons (MLPs) to encode the essential spatiotemporal information of the scene [45, 31, 43, 42]. In one approach, a latent representation is generated to encompass the comprehensive per-frame details of the target video [31, 16, 62]. Although this method is capable of producing visually realistic results, its applicability is limited to short videos due to the constrained memory capacity of MLPs or other representation mechanisms [32]. Alternatively, another approach seeks to construct a latent representation of canonical scenarios and establish correspondences between individual frames and the canonical scenario [45, 43, 63]. This alternative approach allows for the processing of long-term videos; however, it imposes specific requirements on the video

characteristics being learned. For instance, the videos should consistently feature similar objects across frames, and some algorithms may require prior knowledge about the video [60, 44].

In order to address this challenge, we introduce DynPoint, a novel approach designed to achieve efficient view synthesis of lengthy monocular videos without the need for learning a latent canonical representation. Unlike conventional methods that encode information implicitly, DynPoint employs an explicit estimation of consistent depth and scene flow for surface points. These estimates are subsequently utilized to aggregate information from reference frames into the target frame. Subsequently, hierarchical neural point clouds are constructed based on the aggregated information. This hierarchical point cloud set is then employed to synthesize views of the target frame. Our contributions can be summarized as follows:

We introduce a novel module to estimate consistent depth information for monocular video with the help of a proposed regularization and training strategy.

We propose an efficient approach to estimate smooth scene flow between adjacent frames with the proposed training strategy by leveraging the estimated consistent depth maps.

We present a representation to aggregate information from reference frames to target frame, facilitating rapid view synthesis of the target frame within a monocular video.

Comprehensive experiments are conducted on datasets including Nerfie, Nvidia, HyperNeRF, Iphone, and Davis showcasing the speed and accuracy of DynPoint for view synthesis.

2 Related Works

2.1 Static View Synthesis

The generation of photo-realistic views from arbitrary input viewing angles has been a longstanding challenge in the field of computer vision. In the scenario of static scenes, early approaches addressed the challenge of synthesizing photo-realistic views by employing local warping techniques to handle densely sampled views [19, 30, 2, 5, 7]. Additionally, the gradient domain was utilized to handle the single-view case [29]. In order to tackle the challenges associated with view synthesis, several subsequent works have been developed. These works aim to address issues such as reflections, repetitive patterns, and untextured regions [28, 15, 24, 14, 48, 9, 34, 36, 41, 50, 51, 52, 53, 59, 69]. In more recent developments, researchers have explored the representation of scenes as continuous neural radiance fields (NeRF) using fully connected neural networks. These works, such as [38, 1, 66, 25], have demonstrated remarkable outcomes with a trade-off between accuracy and computational complexity (i.e., one network for one scene). To tackle the computational burden associated with neural radiance fields, recent works have focused on developing approaches that generalize the representation across multiple scenes using a single network. These methods employ various techniques, such as fully-convolutional architectures [65], plane-swept cost volumes [8], image-based rendering [58], disentanglement of shape and texture [27], utilization of local features in 2D [56], as well as generative methods [6, 49, 40, 20, 3].

2.2 Dynamic View Synthesis

The emergence of static scene advancements has spurred interest in the exploration of view synthesis for dynamic scenes within the field. Previous research efforts have expanded upon studies conducted in static scenes, incorporating elements such as globally coherent depth [64] and 3D mask volume [33]. Recent research direction builds upon the concept of NeRF and extends it to dynamic scenes by integrating time into the learning process of spatio-temporal radiance fields. One category of research focuses on the assumption of a canonical scenario that spans the entire video [4, 10, 21]. Deformable radiance field-based approaches, as described in previous works [45, 55, 42, 13, 35], employ temporal warping techniques to adapt a base NeRF representation of the canonical space for dynamic scenes. This enables the synthesis of novel views in the context of long monocular videos. Another category of algorithms aims to encode the temporal dynamics of the scene into a global representation. [31] employs a MLP to model the 3D dense motion, resulting in a spatial-temporal NeRF. Building upon this, [12, 57, 16] demonstrate that introducing additional regularization techniques that encourage consistency and physically plausible solutions can enhance the accuracy of view reconstruction. The first category of methods, while achieving impressive photorealistic results, is constrained by the requirement for object-centric videos, thus limiting their generalizability to diverse scenarios.

Surface

Point

Color &

Density

Flow Net

MLP

Scene

Flow

Monocular

Depth

Optic

Flow

Reference

Frame

Target

Frame

Depth

Loss

Flow

Loss

Target

Frame

Reference

Frame

Rendering

MLP

Scene

Flow

Radius

Viewpoint

Aggregate

Neural

Points

Depth Net

(Stage 2)

(Stage 1)

P

△Pt→r

Pt

?

{It,dt}

{ft-→r,fr→t}

It

(8, c)

{at,βt}

Conversely, the second category of methods encounters difficulties when dealing with long videos, leading to limitations in effectively handling such scenarios. To address the aforementioned issue, our algorithm focuses on achieving novel synthesis by aggregating information from the reference frame to the target frame. This information aggregation process is accomplished by explicitly modeling the object movement and depth information. As a result, our algorithm exhibits significant advancements in both accuracy and speed.

3 Methodology

3.1 Overview

Our algorithm is designed to realize view synthesis for a dynamic scenario by utilizing a monocular video { I 1 , I 2 , ..., I T } . The frames in the video, denoted by I t , are captured by a known camera C t,c = { K t,c | [ R t,c , t t,c ] } , where c denotes known camera viewpoint. The objective is to generate the novel view from a specified viewpoint q at a desired time frame C t,q = { K t,q | [ R t,q , t t,q ] } .

Consistent with previous researches [31, 16, 17, 32, 35, 62], we adopt a training paradigm in which our model is trained on the input monocular video with the assistance of pre-trained optic flow and monocular depth models [47, 54, 26]. During the training process, the RGB information obtained from the observed viewpoint is utilized as the supervision signal, without relying on any canonical information. Subsequently, the trained model is evaluated on the task of synthesizing corresponding RGB, depth and scene flow information for unobserved viewpoints.

In contrast to previous methods that utilize one or multiple MLPs to encode the 3D information of each frame, our work focuses on establishing correspondence relationships, i.e., scene flow, between the 3D surface points of the current frame and those of adjacent frames. We can infer 3D information about unobserved points of the current frame by aggregating the information from adjacent frames.

To realize the aforementioned concept, our proposed model should undertake three key tasks. The first task is to estimate depth information consistently for each frame, as outlined in Sec. 3.2. The second task involves learning the 3D scene flow between the current frame and its adjacent frames, as detailed in Sec. 3.3. These two processes are demonstrated in the stage 1 of Fig. 1. The final task is

Frame t

Frame t

'

p

p

t

'

P

P

t

'

p

(

P

, t)

Optic

Flow

Corrected

Depth

Corrected

Depth

MLP

(a) Target Frame

(b) Error Map

(c) Normal Vector Map

(d) Geometric Mask

to aggregate information based on learned correspondence and subsequently use this information to realize view synthesis, as discussed in Sec. 3.4. This process is shown in the stage 2 of Fig. 1.

3.2 Consistent Depth Estimation

Despite the ability of current monocular depth methods [47, 46] to produce accurate depth priors, the predicted depth maps d ′ t suffer from scale-variance and shift problems when compared to the ground truth depth [18, 68]. This characteristic renders d ′ t inconsistent across the temporal axis of monocular video. Moreover, since the optic flow f t → t ′ between frames could be predicted from a pretrained model, a depth value d t can be computed using the triangulation relation between frames, along with the camera parameters and Mid-point method. Compared to d ′ t , d t can have consistent scaling across frames. However, d t heavily relies on f t → t ′ and Epipolar constraint, and it can only generate accurate depth information for a limited portion of the frame. The first module of DynPoint aims to combine d ′ t and d t to generate the final consistent depth estimation ˆ d t .

Regularization: To address this issue, DynPoint identifies the accurate region of d t by utilizing three masks: the corresponding mask, the geometric edge mask, and the dynamic object mask.

Correspondence Mask M c,t → t ′ : The purpose of the optic flow is to capture the pixel-wise correspondence between two frames, making it accurate only for the corresponding regions. These regions can be identified by masking out areas of occlusion caused by both ego-motion and object movement. This Correspondence Mask could be formulated as follow:

where ϵ c is a predefined threshold for the correspondence mask.

Geometric Edge Mask M g : Furthermore, it has been observed that the reliability of d t diminishes when applied to geometric edges as in Fig. 2, particularly in areas where the optic flow exhibits non-smooth characteristics. To compute the mask M g , we first estimate the normal vector map of the surface n = ( -dz dx , -dz dy , 1) / || ( -dz dx , -dz dy , 1) || with the help of d t . Then we apply the Canny edge detector [11] on the n ∈ R H × W to generate the estimation of geometric edge. The intuition of the geometric edge mask is demonstrated in the left part of Fig. 2. From the error map shown in Fig. 2(b), it is evident that the errors primarily manifest around the geometric boundaries of the scenario.

Dynamic Object Mask M d : In addition to the two masks mentioned above, a Dynamic Object Mask is necessary to exclude the dynamic regions of the frame where the triangulation relationship does not hold. Similar to [16], we combine the MASK R-CNN [23] with the Sampson error to generate the M d . To obtain a valid mask with high accuracy, we apply image erosion on the inverse of M d and

M c to mitigate inaccurate boundary detection. We also perform image dilation on M g to include the surrounding region of the geometric edge.

Objective Function: Based on the above three masks, the reliable region of the d t could be masked out by using the final mask M f = M c ∩ M g ∩ M d . To combine the information of d ′ t and d t , we assume that the consistent depth map ˆ d t ∈ R H × W could be approximated by using d ′ t , a scale variable α t ∈ R and a shift variable β t ∈ R . The parameters α t and β t can be generated by optimizing:

Training Strategy: Dependent solely on the nearest frame for the computation of d t would lack sufficient reliability due to the combined influence of the accuracy of the camera matrix and the accuracy of the optic flow on the triangulation process. We employ a series of adjacent 2 K frames to calculate the triangulated depth set denoted as { d k t } 2 k K =1 . Instead of directly utilizing all { d k t } 2 k K =1 in Eqn. 2, we perform a reevaluation by recomputing the intersection mask as M f = M 1 f ∩ ... ∩M 2 f K , which further refines the triangulated depth d t , ensuring the appropriate scale constraint for d ′ t . It's worth noting that pose estimation may not be accurate for dynamic scenarios in certain datasets, as it relies on COLMAP-based estimation. In such cases, we utilize the algorithm presented in our Supplementary material to enhance and fine-tune the pose estimation.

3.3 Scene Flow Estimation

Combined with estimated consistent depth map ˆ d t and optic flow f t → t ′ , DynPoint also aims to infer the scene flow s t → t ′ to build the 3D correspondence between the current frame and adjacent frames. Unlike previous works [42, 45, 55, 13], which estimate the trajectory of all points (hundreds of sampled points on the ray of each point) in the scenario, DynPoint only infers the trajectory of surface point (one point on the ray of each point) of the frame to accelerate both training and inference process. To realize this process, we use a MLP to estimate the scene flow, which can be written as ∆ P t → t +1 , ∆ P t → t -1 = F θ ( P, t ) where P ∈ R 3 denotes input 3D point; ∆ P t → t ′ denotes the trajectory of P from t to t ′ . The weight θ can be optimized by using the relationship among the depth ˆ d t , the optic flow f t → t ′ and the scene flow s t → t ′ .

Objective Function: Given a pixel p t in frame t , its corresponding pixel in frame t ′ , denoted as p t ′ , can be obtained by adding the 2D flow f t → t ′ ( p t ) to p t . Additionally, utilizing the depth map ˆ d t and camera matrix C t , which are available at frame t , the 3D point corresponding to p t can be expressed as P t = R t K -1 t ˆ d t ( p t ) p t + t t . The same method can be used to compute P t ′ . Thus, we've:

This process is demonstrated in the right part of Fig. 2. For static part masked by M d , we set s t ( p t ) = 0 . For the dynamic part, the loss function can be written as

In order to enhance the accuracy of scene flow estimation for the dynamic elements within the scenario, we employ the cycle constraint, a well-established technique utilized in prior studies [16, 35]. The cycle constraint can be expressed as follows:

Training Strategy: We have observed that optimizing Eqn. 4 solely with the near frames, where t ′ = t -1 or t + 1 , does not produce accurate results for further information aggregation. To reconstruct the 3D information of the current frame, it is necessary to compute the correspondence between the current frame and 2 K adjacent frames, in order to aggregate sufficient information. During the training process, we compute the scene flow between frame t and its 2 K adjacent frames, where K ∈ { 1 , ..., K } . We then utilize Eqn. 4 to form the loss function between current frame and 2 K adjacent frames. The scene flow ∆ P t → t + k could be written as:

PSNR

↑

/ LPIPS

↓

Jumping

Skating

Truck

Umbrella

Balloon1

Balloon2

Average

NeRF* [38] (

∼

24 hours

)

20.99 / 0.305

23.67 / 0.311

22.73 / 0.229

21.29 / 0.440

19.82 / 0.205

24.37 / 0.098

21.07 / 0.165

21.99 / 0.250

NeRF + time*[38](

∼

24 hours

)

18.04 / 0.455

20.32 / 0.512

18.33 / 0.382

17.69 / 0.728

18.54 / 0.275

20.69 / 0.216

14.68 / 0.421

18.33 / 0.427

D-NeRF [45] (>

20 hours

)

22.36 / 0.193

22.48 / 0.323

24.10 / 0.145

21.47 / 0.264

19.06 / 0.259

20.76 / 0.277

20.18 / 0.164

21.48 / 0.232

NSFF* [31](

∼

24 hours

)

24.65 / 0.151

29.29 / 0.129

25.96 / 0.167

22.97 / 0.295

21.96 / 0.215

24.27 / 0.222

21.22 / 0.212

24.33 / 0.199

DynamicNeRF* [16] (>

36 hours

)

24.68 / 0.090

32.66

/

0.035

28.56 / 0.082

23.26 / 0.137

22.36 / 0.104

27.06 /

0.049

24.15 / 0.080

26.10 / 0.082

HyperNeRF [43] (>

24 hours

)

18.34 / 0.302

21.97 / 0.183

20.61 / 0.205

18.59 / 0.443

13.96 / 0.530

16.57 / 0.411

13.17 / 0.495

17.60 / 0.367

TiNeuVox [13] (

∼

45 mins)

20.81 / 0.247

23.32 / 0.152

23.86 / 0.173

20.00 / 0.355

17.30 / 0.353

19.06 / 0.279

13.84 / 0.437

19.74 / 0.285

RoDYN [35] (>

36 hours

)

25.66

/

0.071

28.68 / 0.040

29.13 / 0.063

24.26 / 0.089

22.37 / 0.103

26.19 / 0.054

24.96 / 0.048

25.89 /

0.065

DynPoint (

∼

30 mins

)

24.69 / 0.097

31.34 / 0.045

29.30

/

0.061

24.59

/

0.086

22.77

/

0.099

27.63

/

0.049

25.37

/

0.039

26.53

/ 0.068

(a) GT

(b)NSFF

(c)HyperNeRF

(d) RoDynRF

(e) DynPoint

It's important to mention that achieving better results can be accomplished by fine-tuning the refinement layers of the pre-trained depth network.

3.4 View Synthesis

Information Aggregation : In order to collate data from the 2 K reference frames to the target frame t , the pixels in the reference frame p t + k are transformed into 3D space as P t + k through the utilization of the corresponding depth value ˆ d t + k and the matrix C t + k . Next, the DynPoint evaluates the scene flow ∆ P t + k → t between frame t and frame t + k by implementing Eqn. 6. By utilizing both the scene flow ∆ P t + k → t and the 3D point P t + k , we are able to calculate the 3D point P t + k → t that corresponds to reference frame t + k at the target frame t . This is achieved by applying the following computation: P t + k → t = P t + k +∆ P t + k → t . Additionally, a pre-trained 2D convolutional neural network (CNN) is utilized to generate a 2D image feature vector f , which is subsequently allocated to each of the points corresponding to pixels.

Finally, we propose the introduction of a hierarchical neural point cloud construction approach to enhance the "perception field" of individual points. This technique aims to address the problem of surface irregularities that may arise from using current monocular depth estimation methods in multi-view depth map fusion. Thus, the point cloud of current frame t could be generated by combining 2 K +1 point clouds as follows:

where H is the number of hierarchical levels. In our case, we set H = 3 . At each step within this framework, we perform a downsampling operation on the neural point cloud (the downsampling procedure is implemented within the point cloud generation process, whereby the depth map and scene map are downsampled using linear interpolation technique; it is important to note that when

SourceView

NSFF (View1)

DynPoint(View1)

NSFF(View2)

DynPoint(View2)

NSFF(View3)

DynPoint(View3)

SourceView

NSFF (View 1)

DynPoint(View1)

NSFF (View 2)

DynPoint(View2)

NSFF(View 3)

DynPoint(View3)

downsampling the depth map, the intrinsic matrix should also be adjusted accordingly to maintain accurate spatial information.)

View Synthesis : Given a 3D position q and view direction d , we leverage M proximate neural points within a radius of R to generate the corresponding density and color data δ, c as in [61]. This process is shown in the right part of Fig. 1. This can be expressed as follows:

where γ 1 t is the per-point confidence introduced in [61]. Finally, we make use the rendering process in [38] to predict final RGB value C and depth D as:

where τ j = exp ( -∑ j -1 t =1 σ t δ t ) ; δ j denotes the distance between adjacent shading samples; c j is the color information and δ j is the density information. The L2 loss function is used to supervise our rendered pixel values similar to the setting of [37]. For further information regarding the network architecture, please consult our supplementary materials.

3.5 Discussion

The purpose of this section is to explicate the dissimilarities between the DynPoint and a recently published concurrent work, namely DynIBaR [32] which was released in March 2023 and currently lacks available code. Both DynIBaR and DynPoint harness information aggregation mechanisms to realize the synthesis of novel views. However, DynIBaR predominantly centers around the aggregation of information through two-dimensional (2D) pixel units. This approach draws inspiration from image-based rendering principles, entailing the synthesis of novel perspectives from a collection of reference images via a weighted fusion of reference pixels. In contrast, DynPoint's focal point lies in the information aggregation achieved by constructing three-dimensional (3D) neural point clouds. The final novel view synthesis is realized by using neural points surrounding queries' position.

Moreover, DynPoint introduces an efficacious strategy for the seamless integration of monocular depth estimation within the ambit of monocular video view synthesis. In contrast to DynIBaR, which endeavors to model the trajectory of all samples (128) traversing each ray, DynPoint exclusively focuses on the trajectory of surface points, thereby yielding a substantial acceleration in both training and inference stage.

4 Experiment

4.1 Experimental Setting

To evaluate the view synthesis capabilities of DynPoint, we performed experiments on four extensively utilized datasets, namely Nvidia dataset in [31], Nerfie in [42], HyperNeRF in [43] and Iphone in [17]. It is noteworthy to mention that the official website of Nerfie [42] only provides four specific scenarios. Consequently, our experiments were solely conducted on provided four scenarios as in [17]. Additionally, we also assessed DynPoint's performance on a recent dataset Iphone [17], which specifically addresses the challenge of camera teleportation. Furthermore, we examined the efficacy of monocular depth estimation and scene estimation by visualizing the results obtained from the Davis dataset, as in [67].

We conducted a comparative analysis of our work with several recent methods, based on the reported results in their original papers or the reimplementation results of their official code. The methods we compared with include NeRF [38], NeRF + time [38] (which directly utilizes embedded time information as an input to encode all information of the target dynamic scenario), D-NeRF [45], NSFF [31], DynamicNeRF [16], HyperNeRF [43], Nerfie [42], TiBeuVox [13], and RoDYN [35]. In our research, we employed the pretrained Deep Pruning Transformer (DPT) network in [46], for monocular depth estimation. For optic flow estimation, we utilized the pretrained FlowFormer model [26]. To facilitate the fine-tuning process, we initialized the weights of the Rendering MLP by pretraining it on the DTU dataset, employing a similar training set to that used in [61].

The per-scenario training time is shown in Tab. 1. A notable observation was made regarding the reduced per-scenario training time of DynPoint in comparison to other algorithms. This enhancement can be attributed to the implementation of a two-step strategy. In the initial stage, the optimization process focuses on a limited set of parameters pertaining to monocular depth estimation and scene flow estimation. Subsequently, for the second stage, leveraging the outcomes obtained from the first stage, the generation of the neural point cloud for subsequent view synthesis is achieved successfully. Furthermore, the pretraining stage also contributes to the efficiency of our view synthesis stage, requiring only a few iterations to produce desirable results on novel scenarios.

Nvidia (Tab. 1): During this experiment, it was observed that DynPoint exhibited superior performance compared to other algorithms in terms of peak signal-to-noise ratio (PSNR). Additionally, DynPoint achieved the second highest ranking among all algorithms when evaluated based on the Learned Perceptual Image Patch Similarity (LPIPS) metric. Even for scenarios involving multiple objectives, such as Jumping, DynPoint demonstrates reasonable performance without requiring the learning of any canonical representation or extensive training time. The results of specific partial

PSNR

↑

/ LPIPS

↓

Jumping

Skating

Truck

Umbrella

Balloon1

Balloon2

Average

w/o multiple-step strategies

15.01 / 0.711

17.43 / 0.691

15.36 / 0.606

14.90 / 0.837

13.98 / 0.583

15.46 / 0.624

11.35 / 0.613

14.78 / 0.666

w/ K = 3

20.22 / 0.262

22.53 / 0.274

22.38 / 0.291

19.13 / 0.383

16.59 / 0.413

16.92 / 0.459

14.83 / 0.416

18.94 / 0.357

w/o Hierarchical Point Cloud

23.82 / 0.174

28.74 / 0.053

27.50 / 0.085

22.73 / 0.192

21.08 / 0.256

24.67 / 0.180

22.14 / 0.189

24.38 / 0.161

DynPoint (K = 6)

24.69

/

0.097

31.34

/

0.045

29.30

/

0.061

24.59

/

0.086

22.77

/

0.099

27.63

/

0.049

25.37

/

0.039

26.53

/

0.068

Target

Reference

Target

Reference

DepthFlow

Error Map

DepthFlow

ErrorMap

Error Map

Error Map

Figure 5: Demonstration of Depth and Scene Flow Estimation. This figure presents the output of the target images obtained by warping the reference image using depth estimation (second row) or using both depth and scene flow estimation (third row). It is important to clarify that the figure is not intended for comparing view synthesis results. The synthesized figures generated based on scene flow inherently incorporate object motion as input, resulting in observable motion blur within the synthesized figures. Additionally, an error map represented by the intensity of red is provided to visualize the performance, where deeper shades of red indicate larger errors (in terms of pixel movement compared to corrected optic flow).

scenarios, namely Playground, Skating, Truck, and Umbrella, are presented in Fig. 3. Notably, leveraging monocular depth estimation, DynPoint generally produces views with improved geometric features, as evident in the Skating case shown in the second row of Fig. 3. Even in challenging scenarios like Umbrella, DynPoint successfully generates high-quality views while minimizing blurring effects. Nerfie (Tab. 2): In the case of the extended scenario, DynPoint exhibits superior performance in terms of PSNR and achieves comparable results to those obtained in the short video Nvidia, as presented in Tab. 2. This achievement can be attributed to our information aggregation approach, which focuses on effectively aggregating information from the target frame by establishing associations between its points and those in the reference frames. Novel view synthesis results for scenarios TAIL and TOBY SIT are depicted in Figure 4. It is evident that for longer videos, DynPoint continues to produce more realistic frames. Notably, the generated views in this figure do not exist in either the training or test dataset. HyperNeRF (Tab. 3): In the case of longer video sequences, DynPoint showcases superior performance in PSNR, as highlighted in Table 3. Iphone (Tab. 4): In the context of monocular videos without camera teleportation, as demonstrated in the work [17], DynPoint attains comparable outcomes to previous algorithms. Due to the inherent limitations of having few multi-view perspectives and overlapping information, establishing correspondences between adjacent frames becomes challenging. Consequently, the view synthesis task based on monocular videos proves to be more arduous on this dataset compared to other datasets.

4.2 Ablation Studies

In order to evaluate the effectiveness of the strategies proposed in our paper, we conducted three ablation studies. These studies include: (1) the absence of the multiple-step training strategy outlined in Sec. 3.2 and Sec. 3.3; (2) the utilization of only six frames in the vicinity ( K = 3 ); and (3) the exclusion of the hierarchical point cloud. The results are presented in Tab. 5. It is evident from the results that the omission of the multiple-step training strategy leads to the largest drop in performance. Without this strategy, the first stage produces noisy outcomes for both monocular depth and scene

GT

DynPoint

GT

DynPoint

flow estimation, consequently hindering the generation of the neural point cloud and impacting the performance of the second stage. Moreover, using a limited number of adjacent frames also adversely affects the final performance, which aligns with our expectations as limited inputs correspond to limited information. Although the removal of the hierarchical point cloud does not significantly degrade performance, it still plays a role in generating finer results.

4.3 Consistent Depth & Scene Flow Estimation On Davis

In order to validate the efficacy of the monocular depth estimation technique and the scene flow estimation method, we present the following visualization results: (1) Reconstructed Target Image using Monocular Depth Estimation: We demonstrate the reconstruction of the target image by warping the reference image based on the monocular depth estimation. (2) Reconstructed Target Image using Scene Flow and Depth Estimation: We present the visual reconstruction of the target image achieved through warping the reference image using solely depth estimation or a combination of scene flow estimation and depth estimation. The results are displayed in Fig. 5. Upon observing the second row, it is evident that DynPoint demonstrates commendable performance in reconstructing the static background of the target frame through monocular depth estimation. However, the primary errors (deep red part) occur in the dynamic region, where accounting for object movement becomes crucial for accurate warping. Moving to the third row, our scene flow estimation proves to be effective in capturing the movement of dynamic objects, as in the red error map Fig. 5.

4.4 Failure Cases

Despite the notable achievements of DynPoint, certain failure cases were observed during the view synthesis process, as demonstrated in Fig. 6. By comparing the first and second images, it becomes apparent that generating realistic facial features in regions with intricate details proves to be challenging. Furthermore, when comparing the third and fourth images, it is evident that DynPoint struggles with handling fine objects and reflections, as these aspects heavily rely on the accurate geometry inference obtained in the first stage.

5 Conclusion

In this research paper, we present DynPoint, an algorithm specifically designed to address the view synthesis task for monocular videos. Rather than relying on learning a global representation encompassing color, geometry, and motion information of the entire scene, we propose an approach that aggregates information from neighboring frames. This aggregation process is facilitated by learning correspondences between the target frame and reference frames, aided by depth and scene flow inference. The experimental results demonstrate that our proposed model exhibits improved performance in terms of both accuracy and speed compared to existing approaches.

Acknowledgements. Our research is supported by Amazon Web Services in the Oxford-Singapore Human-Machine Collaboration Programme and by the ACE-OPS project (EP/S030832/1). We are grateful to all of the anonymous reviewers for their valuable comments.

References

J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, 'Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 5855-5864.

C. Buehler, M. Bosse, L. McMillan, S. Gortler, and M. Cohen, 'Unstructured lumigraph rendering,' in Proceedings of the 28th annual conference on Computer graphics and interactive techniques , 2001, pp. 425-432.

S. Cai, A. Obukhov, D. Dai, and L. Van Gool, 'Pix2nerf: Unsupervised conditional p-gan for single image to neural radiance fields translation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 3981-3990.

J. Carranza, C. Theobalt, M. A. Magnor, and H.-P. Seidel, 'Free-viewpoint video of human actors,' ACM transactions on graphics (TOG) , vol. 22, no. 3, pp. 569-577, 2003.

J.-X. Chai, X. Tong, S.-C. Chan, and H.-Y. Shum, 'Plenoptic sampling,' in Proceedings of the 27th annual conference on Computer graphics and interactive techniques , 2000, pp. 307-318.

E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein, 'pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 5799-5809.

G. Chaurasia, S. Duchene, O. Sorkine-Hornung, and G. Drettakis, 'Depth synthesis and local warps for plausible image-based navigation,' ACM Transactions on Graphics (TOG) , vol. 32, no. 3, pp. 1-12, 2013.

A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su, 'Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 14 124-14 133.

I. Choi, O. Gallo, A. Troccoli, M. H. Kim, and J. Kautz, 'Extreme view synthesis,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 7781-7790.

E. De Aguiar, C. Stoll, C. Theobalt, N. Ahmed, H.-P. Seidel, and S. Thrun, 'Performance capture from sparse multi-view video,' in ACM SIGGRAPH 2008 papers , 2008, pp. 1-10.

L. Ding and A. Goshtasby, 'On the canny edge detector,' Pattern recognition , vol. 34, no. 3, pp. 721-725, 2001.

Y. Du, Y. Zhang, H.-X. Yu, J. B. Tenenbaum, and J. Wu, 'Neural radiance flow for 4d view synthesis and video processing,' in 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . IEEE Computer Society, 2021, pp. 14 304-14 314.

J. Fang, T. Yi, X. Wang, L. Xie, X. Zhang, W. Liu, M. Nießner, and Q. Tian, 'Fast dynamic radiance fields with time-aware neural voxels,' in SIGGRAPH Asia 2022 Conference Papers , 2022, pp. 1-9.

J. Flynn, M. Broxton, P. Debevec, M. DuVall, G. Fyffe, R. Overbeck, N. Snavely, and R. Tucker, 'Deepview: View synthesis with learned gradient descent,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 2367-2376.

J. Flynn, I. Neulander, J. Philbin, and N. Snavely, 'Deepstereo: Learning to predict new views from the world's imagery,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 5515-5524.

C. Gao, A. Saraf, J. Kopf, and J.-B. Huang, 'Dynamic view synthesis from dynamic monocular video,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 5712-5721.

H. Gao, R. Li, S. Tulsiani, B. Russell, and A. Kanazawa, 'Monocular dynamic view synthesis: A reality check,' in Advances in Neural Information Processing Systems , 2022.

C. Godard, O. Mac Aodha, and G. J. Brostow, 'Unsupervised monocular depth estimation with left-right consistency,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 270-279.

S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, 'The lumigraph,' in Proceedings of the 23rd annual conference on Computer graphics and interactive techniques , 1996, pp. 43-54.

J. Gu, L. Liu, P. Wang, and C. Theobalt, 'Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis,' arXiv preprint arXiv:2110.08985 , 2021.

K. Guo, P. Lincoln, P. Davidson, J. Busch, X. Yu, M. Whalen, G. Harvey, S. Orts-Escolano, R. Pandey, J. Dourgarian, et al. , 'The relightables: V olumetric performance capture of humans with realistic relighting,' ACM Transactions on Graphics (ToG) , vol. 38, no. 6, pp. 1-19, 2019.

R. Hartley and A. Zisserman, Multiple view geometry in computer vision . Cambridge university press, 2003.

K. He, G. Gkioxari, P. Dollár, and R. Girshick, 'Mask r-cnn,' in Proceedings of the IEEE international conference on computer vision , 2017, pp. 2961-2969.

P. Hedman, J. Philip, T. Price, J.-M. Frahm, G. Drettakis, and G. Brostow, 'Deep blending for freeviewpoint image-based rendering,' ACM Transactions on Graphics (TOG) , vol. 37, no. 6, pp. 1-15, 2018.

S. Hu, K. Zhou, K. Li, L. Yu, L. Hong, T. Hu, Z. Li, G. H. Lee, and Z. Liu, 'Consistentnerf: Enhancing neural radiance fields with 3d consistency for sparse view synthesis,' arXiv preprint arXiv:2305.11031 , 2023.

Z. Huang, X. Shi, C. Zhang, Q. Wang, K. C. Cheung, H. Qin, J. Dai, and H. Li, 'Flowformer: A transformer architecture for optical flow,' in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII . Springer, 2022, pp. 668-685.

W. Jang and L. Agapito, 'Codenerf: Disentangled neural radiance fields for object categories,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 12 949-12 958.

N. K. Kalantari, T.-C. Wang, and R. Ramamoorthi, 'Learning-based view synthesis for light field cameras,' ACM Transactions on Graphics (TOG) , vol. 35, no. 6, pp. 1-10, 2016.

J. Kopf, F. Langguth, D. Scharstein, R. Szeliski, and M. Goesele, 'Image-based rendering in the gradient domain,' ACM Transactions on Graphics (TOG) , vol. 32, no. 6, pp. 1-9, 2013.

M. Levoy and P. Hanrahan, 'Light field rendering,' in Proceedings of the 23rd annual conference on Computer graphics and interactive techniques , 1996, pp. 31-42.

Z. Li, S. Niklaus, N. Snavely, and O. Wang, 'Neural scene flow fields for space-time view synthesis of dynamic scenes,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 6498-6508.

Z. Li, Q. Wang, F. Cole, R. Tucker, and N. Snavely, 'Dynibar: Neural dynamic image-based rendering,' arXiv preprint arXiv:2211.11082 , 2022.

K.-E. Lin, L. Xiao, F. Liu, G. Yang, and R. Ramamoorthi, 'Deep 3d mask volume for view synthesis of dynamic scenes,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 1749-1758.

L. Liu, J. Gu, K. Zaw Lin, T.-S. Chua, and C. Theobalt, 'Neural sparse voxel fields,' Advances in Neural Information Processing Systems , vol. 33, pp. 15 651-15 663, 2020.

Y.-L. Liu, C. Gao, A. Meuleman, H.-Y. Tseng, A. Saraf, C. Kim, Y.-Y. Chuang, J. Kopf, and J.-B. Huang, 'Robust dynamic radiance fields,' arXiv preprint arXiv:2301.02239 , 2023.

S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh, 'Neural volumes: Learning dynamic renderable volumes from images,' ACM Trans. Graph. , vol. 38, no. 4, pp. 65:1-65:14, July 2019.

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, 'Nerf: Representing scenes as neural radiance fields for view synthesis,' in European conference on computer vision . Springer, 2020, pp. 405-421.

--, 'Nerf: Representing scenes as neural radiance fields for view synthesis,' Communications of the ACM , vol. 65, no. 1, pp. 99-106, 2021.

M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. Sajjadi, A. Geiger, and N. Radwan, 'Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 5480-5490.

M. Niemeyer and A. Geiger, 'Giraffe: Representing scenes as compositional generative neural feature fields,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 11 453-11 464.

M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger, 'Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 3504-3515.

K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla, 'Nerfies: Deformable neural radiance fields,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 5865-5874.

K. Park, U. Sinha, P. Hedman, J. T. Barron, S. Bouaziz, D. B. Goldman, R. Martin-Brualla, and S. M. Seitz, 'Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields,' arXiv preprint arXiv:2106.13228 , 2021.

Y. Peng, Y. Yan, S. Liu, Y. Cheng, S. Guan, B. Pan, G. Zhai, and X. Yang, 'Cagenerf: Cage-based neural radiance field for generalized 3d deformation and animation,' Advances in Neural Information Processing Systems , vol. 35, pp. 31 402-31 415, 2022.

A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer, 'D-nerf: Neural radiance fields for dynamic scenes,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 10 318-10 327.

R. Ranftl, A. Bochkovskiy, and V. Koltun, 'Vision transformers for dense prediction,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 12 179-12 188.

R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V . Koltun, 'Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,' IEEE transactions on pattern analysis and machine intelligence , vol. 44, no. 3, pp. 1623-1637, 2020.

G. Riegler and V. Koltun, 'Free view synthesis,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIX 16 . Springer, 2020, pp. 623-640.

K. Schwarz, Y. Liao, M. Niemeyer, and A. Geiger, 'Graf: Generative radiance fields for 3d-aware image synthesis,' Advances in Neural Information Processing Systems , vol. 33, pp. 20 154-20 166, 2020.

V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, 'Implicit neural representations with periodic activation functions,' Advances in Neural Information Processing Systems , vol. 33, pp. 7462-7473, 2020.

V. Sitzmann, J. Thies, F. Heide, M. Nießner, G. Wetzstein, and M. Zollhofer, 'Deepvoxels: Learning persistent 3d feature embeddings,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 2437-2446.

V. Sitzmann, M. Zollhöfer, and G. Wetzstein, 'Scene representation networks: Continuous 3d-structureaware neural scene representations,' Advances in Neural Information Processing Systems , vol. 32, 2019.

P. P. Srinivasan, R. Tucker, J. T. Barron, R. Ramamoorthi, R. Ng, and N. Snavely, 'Pushing the boundaries of view extrapolation with multiplane images,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 175-184.

Z. Teed and J. Deng, 'Raft: Recurrent all-pairs field transforms for optical flow,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 . Springer, 2020, pp. 402-419.

E. Tretschk, A. Tewari, V. Golyanik, M. Zollhöfer, C. Lassner, and C. Theobalt, 'Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 12 959-12 970.

A. Trevithick and B. Yang, 'Grf: Learning a general radiance field for 3d representation and rendering,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 15 182-15 192.

C. Wang, B. Eckart, S. Lucey, and O. Gallo, 'Neural trajectory fields for dynamic novel view synthesis,' arXiv preprint arXiv:2105.05994 , 2021.

Q. Wang, Z. Wang, K. Genova, P. P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla, N. Snavely, and T. Funkhouser, 'Ibrnet: Learning multi-view image-based rendering,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 4690-4699.

S. Wizadwongsa, P. Phongthawee, J. Yenphraphai, and S. Suwajanakorn, 'Nex: Real-time view synthesis with neural basis expansion,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 8534-8543.

H. Xu, T. Alldieck, and C. Sminchisescu, 'H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion,' Advances in Neural Information Processing Systems , vol. 34, pp. 14 955-14 966, 2021.

Q. Xu, Z. Xu, J. Philip, S. Bi, Z. Shu, K. Sunkavalli, and U. Neumann, 'Point-nerf: Point-based neural radiance fields,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 5438-5448.

G. Yang, M. Vo, N. Neverova, D. Ramanan, A. Vedaldi, and H. Joo, 'Banmo: Building animatable 3d neural models from many casual videos,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 2863-2873.

G. Yang, C. Wang, N. D. Reddy, and D. Ramanan, 'Reconstructing animatable categories from videos,' arXiv preprint arXiv:2305.06351 , 2023.

J. S. Yoon, K. Kim, O. Gallo, H. S. Park, and J. Kautz, 'Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 5336-5345.

A. Yu, V. Ye, M. Tancik, and A. Kanazawa, 'pixelnerf: Neural radiance fields from one or few images,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 4578-4587.

K. Zhang, G. Riegler, N. Snavely, and V. Koltun, 'Nerf++: Analyzing and improving neural radiance fields,' arXiv preprint arXiv:2010.07492 , 2020.

Z. Zhang, F. Cole, Z. Li, M. Rubinstein, N. Snavely, and W. T. Freeman, 'Structure and motion from casual videos,' in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII . Springer, 2022, pp. 20-37.

K. Zhou, L. Hong, C. Chen, H. Xu, C. Ye, Q. Hu, and Z. Li, 'Devnet: Self-supervised monocular depth learning via density volume construction,' in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX . Springer, 2022, pp. 125-142.

T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely, 'Stereo magnification: Learning view synthesis using multiplane images,' arXiv preprint arXiv:1805.09817 , 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDynPoint is an algorithm proposed to enhance the synthesis of novel views in monocular videos using neural radiance fields. It focuses on predicting explicit correspondence between neighboring frames, enabling information aggregation for view synthesis. The demonstrated experimental results show a significant reduction in training time, comparable outcomes to previous approaches, and robustness in handling uncontrolled and lengthy videos without requiring a canonical representation of video content. To achieve it authors propose to improve the estimated scene flow from the optical flow with corrected depth values\n\nSTRENGTHS:\nthe proposed technique for scene flow estimation is a solid contribution. It takes into account two inconsistencies in depth and flow that are denoised with the method.  The improvements can be applied to any baseline as a separate technique to improve baseline (e.g. NSFF). Moreover, this allows extracting correspondences for the arbitrary 3D scene.\nhierarchical scheme to render the semi-explicit structure is a novel and a trustable way to get the fast method based on existing research\nExperiments demonstrate the substantial of the proposed method and provides a strong contribution to the field.\n\nWEAKNESSES:\nThe proposed method relies heavily on existing models and it is unclear whether the quality depends if you replace them with others\nAuthors don't investigate the quality of the correspondence, which can be interesting by itself. The easiest way is to extend section 4.4 with a comparison versus method like \"Deformable Sprites\" Ye et al.\nThe presentation of the visual results can be improved. Figure 3 should have a close-up by emphasizing the region of interest, otherwise, results are not clear as well as others.\nNot compared with “DynIBaR: Neural Dynamic Image-Based Rendering”. Yes, I understand that it is a recent work. At least add numbers to the table for Nvidia dataset.\nWithout video results, it is difficult to reason even with such metric improvements on average.\nNo limitation or ethical section to give a reader explicit statements.\n\nQUESTIONS:\n1. How do you measure the time? Does it include phase 1 and the hardware is the same as competitors?\n2. “Neural point cloud” technique is similar to [1, 2, 3] representations in some sense (humans & static scene examples). Could you extend related works in the neural point cloud as well? Does hierarchy is an important contribution for the neural point cloud? If so, it can be \n3. What is the main drawback of the method for iPhone dataset? Is it possible to extend the comparison on this benchmark with others, since the gap is much smaller than on others?\n4. Do you consider including video metrics for the results (e.g. VMAF)?\n5. How issues in Figure 5 can be addressed and what is causing them?\n\n[1] PointAvatar: Deformable Point-based Head Avatars from Videos\n[2] Self-Improving Multiplane-To-Layer Images for Novel View Synthesis\n[3] Pulsar: Efficient Sphere-based Neural Rendering\n\nLIMITATIONS:\nFast changes in the scene as well as thin objects can be a problem for the method, which is quite common for this class of models. Compared with other methods this one needs more information and additional pretraining steps. the category-specific methods can be better for non-rigid parts. The scalability of the method is limited due to two-stage training for each of the scenes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel method for performing novel view synthesis from monocular captured videos. This is done by learning scene flow and depth parameters which enable the accurate aggregation of appearance information from nearby frames. The paper proposes a novel method of acquiring both consistent depth estimations for a video, and consistent scene flow represented by an MLP. These two sources of information are used to synthesize novel views at a specific time from an arbitrarily specified viewpoint by sampling from corresponding information in a window of nearby frames. The proposed method is demonstrated to outperform existing methods based on representation learning in synthesizing novel views from monocular videos, and this is shown for a wide variety of datasets.\n\nAFTER REBUTTAL:\n\nI have read the author rebuttal, and believe that it has addressed my questions. I appreciate the detailed discussion on the comparison to DynIBaR. I also believe the additional videos provided have definitely helped improve my opinion of the method, although I wish the method was tested with more qualitative video examples (as mentioned by another reviewer). However, I believe the testing now is sufficient.\n\nSTRENGTHS:\nIn my opinion, the main strengths of the paper are that:\n1. The presentation of the paper is concise, clear, and thorough.\n    - The introduction and related work I find to be comprehensive, and accurately describe what the contributions are and what the current issue is with the practice.\n    - The methods section is described in detail, and is relatively clear to understand and follow.\n    - The figures are all informative and demonstrate the capabilities of the method. One thing I wish could be added is a video result demonstrating consistent novel view synthesis.\n2. The proposed contributions are original and impactful.\n    - The module for estimating consistent depth for a monocular video seems novel, and while there is existing work tackling this (see questions), the method proposed seems to generate good enough results for novel view synthesis with DynPoint.\n    - The method for estimating and representing the scene flow between adjacent frames and using this to aggregate information from reference to target frames seems novel (although based on image-based rendering ideas), and learning this from the video seems like an important contribution.\n3. The evaluations are sound.\n    - The evaluations are done for many (4) different datasets, and compare to many different dynamic scene representation methods (Nerfies, D-NeRF, NSFF, HyperNeRF among others). Because of how thorough these evaluations are, and DynPoint outperforms all of the baseline methods, it is very convincing that the proposed method is robust and is able to generate high-quality results.\n    - The ablations are relatively thorough, and Table 4 shows how the individual parts of the representation and aggregation pipeline contribute to the final quality. I found it to be informative about the method.\n\nWEAKNESSES:\nI do not view there to be many major weaknesses of the paper. One thing which would be interesting is a comparison to the method DynIBaR [31]. The paper mentions that this method is limited to short videos due to the capacity of MLPs or other representation mechanisms (L34), but from my understanding this method uses a similar philosophy as this work where new frames are synthesized as inspired by image-based rendering and scene flow estimation. This seems to me to be the current state-of-the-art. Additionally, it’s not clear that the videos this method is evaluated on are long enough for this to be a concern, considering NeRF-based methods can be evaluated and these are constrained by the capacity of the MLP being used to represent the scene. Additionally, I think there could be a bit more discussion on the limitations of the proposed method (see limitations).\n\nQUESTIONS:\n- How does the generated consistent video depth estimation compare to other work in this field, for example: https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/. Could this method be dropped in and used for this estimation step?\n\nLIMITATIONS:\nThe paper adequately addresses the limitations of the method. One additional limitation that discussion of would significantly improve the quality of the paper, is the amount of differing from the monocular video path which can be taken and still synthesize reasonable novel views. Obviously, since this method is using appearance information in nearby frames, it is likely to not be able to hallucinate information beyond these. However, NeRF-based methods are able to put something there and potentially reconstruct better results due to the smoothness of the representation learned by the MLP. Some comparison of the existing methods and the proposed method on this axis (range of possible novel views synthesized) would be helpful, and if this is a limitation of the proposed method then it should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an algorithm for novel-view synthesis in dynamic scenes. According to the abstract and introduction section, this paper takes a monocular video as an input (L.98) and cope with uncontrolled or length scenarios (L.3). Using geometric priors, such as monocular depth and optical flow from off-the-shelf methods, this paper exploits the pixel-wise correspondence to encode sceneflow within the network.\n\nSTRENGTHS:\nOverall, I am a bit not that positive for this paper. Let me put more comments on the weakness section.\n\nWEAKNESSES:\n_1. Wrong experimental setup._\n\nThis paper claims that they focus on rendering scenes from a monocular video to handle lengthy video. However, this is not true. NVIDIA dataset [63] captures the scenes using a camera rig (2x8 array camera? not sure), which is not a monocular video. I think that the correct meaning of the monocular video is addressed by the Dycheck dataset [17] which is called iPhone dataset [17] in this manuscript. Moreover, NVIDIA dataset [63] only has a few training frame which is not well aligned with the addressed points by the authors in the abstract and introduction. At least, HyperNeRF dataset [42] provides more than 200~ frames per scene, where I could say this is a bit lengthy video. Such inconsistency in addressed problems and the proposed solution is quite not good. I am not that satisfied with the quality of the writing.\n\nMoreover, where can I find the qualitative results on Dycheck dataset [17]? While table 3 provides the quantitative results, I could not find any qualitative results. \n\n\n_2. No video demo_\n\nMore seriously, though this paper tackles the neural rendering in dynamic scenes, I could not find any video demo even in the supplementary material. __Lack of video demo is a big problem__.\n\n\n_3. What is the difference in technical contribution in comparison with NSFF [30]?_\n\nNSFF [30] also exploit the sceneflow understanding within the network, similar to this paper. What is the main difference? At least authors mainly discuss this issue and provide some quant/qual difference within their experimental section. But I could not find any difference or novelty in this part.\n\n\n_4. Lack of related papers to address the speed in dynamic NeRFs._\n\nThis paper is not the pioneering work that provides fast rendering/training speed in dynamic NeRF. I hope that the authors to look at the paper of TiNeuVox, Siggraph Asia 2022 (https://github.com/hustvl/TiNeuVox). If this paper wants to claim the novelty in speed, the authors should have fairly compared with TiNeuVox.\n\nQUESTIONS:\nI hope that the authors properly address my concerns which are listed in the weakness section.\n\nOverall, this paper does not provide the whole information that is necessary for the reviewers to properly judge its validity.\n\nLIMITATIONS:\nIt's okay to me.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for dynamic scene synthesis from monocular video. The authors aim to speed up training and deal with long-duration videos. To do this, they propose to estimate scene flows of surface points supervised by the signals which are generated by the proposed consistent depth algorithm. Then the information is aggregated to form a neural point cloud and novel view images are rendered by sampling amount this point cloud.\n\nSTRENGTHS:\n1.\tThe paper proposes a novel solution to dynamic scene. Instead of estimating density and color of the scene for each time, the paper explicitly warps the pixel features of reference images to target time according to the explicitly reconstructed 3D geometry. This would reduce the time for training, as there less learnable components.\n2.\tThe experiment shows SOTA results on Nvidia dataset.\n\nWEAKNESSES:\n1.\tPlease consider to rewrite the Equation 3.\n2.\tThe results on Nerfie dataset and Iphone dataset are less consistent compared to Nvidia dataset. Why not test TiNeuVox on Nerfie and Iphone dataset? Or test the proposed method on HyperNeRF dataset?\n3.\tPlease highlight all best results in Table 2 & Table 3.\n4.\tThe paper claim that the method could handle long-duration video, which I agree. However, no experiment is conducted to prove this ability.\n\nQUESTIONS:\n1.\tWhat is the render speed of the method, considering the relative complex neural point cloud construction process.\n2.\tAn open question: is it possible to extend this method to a generalizable method, which could generalize to new scenes without training? For example, replace scene flow MLP with an un-learnable method and use a generalizable renderer.\n3.\tEquation 7 is confused. The warped 2K point clouds are summed or combined?\n\nLIMITATIONS:\nThe authors addresed the limitation I concerned, which is the potential failure caursed by the flaws in explicit depths and flows.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}