SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $H$  is the number of hierarchical levels. In our case, we set  $H = 3$ . At each step within this framework, we perform a downsampling operation on the neural point cloud (the downsampling procedure is implemented within the point cloud generation process, whereby the depth map and scene map are downsampled using linear interpolation technique; it is important to note that when
Table 2: Novel View Synthesis Results on Nerjie Dataset. We report the average PSNR and LPIPS results with comparisons to existing methods on Nerjie dataset [42].
PSNR ↑ / LPIPS ↓	CURLS	TOBY SIT	TAIL	BROOM	Average
NeRF [38]	14.40 / 0.616	22.80 / 0.463	23.00 / 0.571	21.00 / 0.667	20.30 / 0.579
NeRF + time[38]	17.30 / 0.539	19.40 / 0.385	24.90 / 0.404	21.90 / 0.576	20.87 / 0.476
NSFF [31]	18.00 / 0.432	26.90 / 0.208	30.60 / 0.245	28.20 / 0.202	25.93 / 0.272
Nerfies [42]	24.90 / 0.312	22.80 / 0.174	23.60 / 0.175	21.00 / 0.270	23.08 / 0.233
DynPoint	24.33 / 0.339	24.90 / 0.186	29.12 / 0.218	27.28 / 0.222	26.41 / 0.241
Figure 4: Demonstration of View Synthesis Results on Nerfie Dataset. This demonstration compares the view synthesis outcomes of DynPoint with those of NSFF.
downsampling the depth map, the intrinsic matrix should also be adjusted accordingly to maintain accurate spatial information.)
View Synthesis: Given a 3D position  $q$  and view direction  $d$ , we leverage  $M$  proximate neural points within a radius of  $R$  to generate the corresponding density and color data  $\delta, c$  as in [61]. This process is shown in the right part of Fig. 1. This can be expressed as follows:
$$
(\delta , c) = F _ {\phi} \left(q, d, \hat {P} _ {t} ^ {1}, \mathbf {f} _ {t} ^ {1}, \gamma_ {t} ^ {1}, \dots , \hat {P} _ {t} ^ {M}, \mathbf {f} _ {t} ^ {M}, \gamma_ {t} ^ {M}\right). \tag {8}
$$
where  $\gamma_t^1$  is the per-point confidence introduced in [61]. Finally, we make use the rendering process in [38] to predict final RGB value  $C$  and depth  $D$  as:
$$
C = \sum_ {j = 1} ^ {N} \tau_ {j} \left(1 - \exp \left(- \sigma_ {j} \delta_ {j}\right)\right) c _ {j}, \tag {9}
$$
where  $\tau_{j} = \exp (-\sum_{t = 1}^{j - 1}\sigma_{t}\delta_{t})$ ;  $\delta_{j}$  denotes the distance between adjacent shading samples;  $c_{j}$  is the color information and  $\delta_{j}$  is the density information. The L2 loss function is used to supervise our rendered pixel values similar to the setting of [37]. For further information regarding the network architecture, please consult our supplementary materials.
3.5 Discussion
The purpose of this section is to explicate the dissimilarities between the DynPoint and a recently published concurrent work, namely DynIBaR [32] which was released in March 2023 and currently lacks available code. Both DynIBaR and DynPoint harness information aggregation mechanisms to realize the synthesis of novel views. However, DynIBaR predominantly centers around the aggregation of information through two-dimensional (2D) pixel units. This approach draws inspiration from image-based rendering principles, entailing the synthesis of novel perspectives from a collection of reference images via a weighted fusion of reference pixels. In contrast, DynPoint's focal point lies in the information aggregation achieved by constructing three-dimensional (3D) neural point clouds. The final novel view synthesis is realized by using neural points surrounding queries' position.
Table 3: Novel View Synthesis Results of HyperNeRF Dataset. We report the average PSNR and LPIPS results with comparisons to existing methods on HyperNeRF dataset [43].
PSNR ↑ / LPIPS ↓	Broom 197 Frames	3D printer 207 Frames	Chicken 164 Frames	Expressions 259 Frames	Peel Banana 513 Frames	Average
NSFF [31]	26.10 / 0.284	27.70 / 0.125	26.90 / 0.106	26.70 / 0.157	24.60 / 0.198	26.40 / 0.174
Nerfies [42]	19.20 / 0.325	20.60 / 0.108	26.70 / 0.078	21.80 / 0.150	22.40 / 0.147	22.10 / 0.162
Hyper-NeRF [43]	20.60 / 0.613	21.40 / 0.212	27.60 / 0.108	22.00 / 0.196	24.30 / 0.170	23.20 / 0.260
DynPoint	27.40 / 0.248	27.60 / 0.163	28.10 / 0.089	27.90 / 0.147	26.50 / 0.129	27.50 / 0.155
Table 4: Novel View Synthesis Results of Iphone Dataset. We compare the mPSNR and mSSIM scores with existing methods on the iPhone dataset [17].
mPSNR ↑ / mSSIM ↑	Apple	Block	Paper-windmill	Space-out	Spin	Teddy	Wheel	Average
NSFF [31]	17.54 / 0.750	16.61 / 0.639	17.34 / 0.378	17.79 / 0.622	18.38 / 0.585	13.65 / 0.557	13.82 / 0.458	15.46 / 0.569
Nerfies [42]	17.64 / 0.743	17.54 / 0.670	17.38 / 0.382	17.93 / 0.605	19.20 / 0.561	13.97 / 0.568	13.99 / 0.455	16.45 / 0.569
HyperNeRF [43]	16.47 / 0.754	14.71 / 0.606	14.94 / 0.272	17.65 / 0.636	17.26 / 0.540	12.59 / 0.537	14.59 / 0.511	16.81 / 0.550
DynPoint	17.78 / 0.743	17.67 / 0.667	17.32 / 0.366	17.78 / 0.603	19.04 / 0.564	13.95 / 0.551	14.72 / 0.515	16.89 / 0.572
Moreover, DynPoint introduces an efficacious strategy for the seamless integration of monocular depth estimation within the ambit of monocular video view synthesis. In contrast to DynIBaR, which endeavors to model the trajectory of all samples (128) traversing each ray, DynPoint exclusively focuses on the trajectory of surface points, thereby yielding a substantial acceleration in both training and inference stage.
To evaluate the view synthesis capabilities of DynPoint, we performed experiments on four extensively utilized datasets, namely Nvidia dataset in [31], Nerjie in [42], HyperNeRF in [43] and Iphone in [17]. It is noteworthy to mention that the official website of Nerjie [42] only provides four specific scenarios. Consequently, our experiments were solely conducted on provided four scenarios as in [17]. Additionally, we also assessed DynPoint's performance on a recent dataset Iphone [17], which specifically addresses the challenge of camera teleportation. Furthermore, we examined the efficacy of monocular depth estimation and scene estimation by visualizing the results obtained from the Davis dataset, as in [67].
We conducted a comparative analysis of our work with several recent methods, based on the reported results in their original papers or the reimplementation results of their official code. The methods we compared with include NeRF [38], NeRF + time [38] (which directly utilizes embedded time information as an input to encode all information of the target dynamic scenario), D-NeRF [45], NSFF [31], DynamicNeRF [16], HyperNeRF [43], Nerfie [42], TiBeuVox [13], and RoDYN [35]. In our research, we employed the pretrained Deep Pruning Transformer (DPT) network in [46], for monocular depth estimation. For optic flow estimation, we utilized the pretrained FlowFormer model [26]. To facilitate the fine-tuning process, we initialized the weights of the Rendering MLP by pretraining it on the DTU dataset, employing a similar training set to that used in [61].
The per-scenario training time is shown in Tab. 1. A notable observation was made regarding the reduced per-scenario training time of DynPoint in comparison to other algorithms. This enhancement can be attributed to the implementation of a two-step strategy. In the initial stage, the optimization process focuses on a limited set of parameters pertaining to monocular depth estimation and scene flow estimation. Subsequently, for the second stage, leveraging the outcomes obtained from the first stage, the generation of the neural point cloud for subsequent view synthesis is achieved successfully. Furthermore, the pretraining stage also contributes to the efficiency of our view synthesis stage, requiring only a few iterations to produce desirable results on novel scenarios.
Nvidia (Tab. 1): During this experiment, it was observed that DynPoint exhibited superior performance compared to other algorithms in terms of peak signal-to-noise ratio (PSNR). Additionally, DynPoint achieved the second highest ranking among all algorithms when evaluated based on the Learned Perceptual Image Patch Similarity (LPIPS) metric. Even for scenarios involving multiple objectives, such as Jumping, DynPoint demonstrates reasonable performance without requiring the learning of any canonical representation or extensive training time. The results of specific partial
Table 5: Ablation studies of Nvidia Dataset. We report the average PSNR and LPIPS results with comparisons to existing methods on Nvidia dataset [64].
PSNR ↑ / LPIPS ↓	Jumping	Skating	Truck	Umbrella	Balloon1	Balloon2	Playground	Average
w/o multiple-step strategies	15.01 / 0.711	17.43 / 0.691	15.36 / 0.606	14.90 / 0.837	13.98 / 0.583	15.46 / 0.624	11.35 / 0.613	14.78 / 0.666
w/ K = 3	20.22 / 0.262	22.53 / 0.274	22.38 / 0.291	19.13 / 0.383	16.59 / 0.413	16.92 / 0.459	14.83 / 0.416	18.94 / 0.357
w/o Hierarchical Point Cloud	23.82 / 0.174	28.74 / 0.053	27.50 / 0.085	22.73 / 0.192	21.08 / 0.256	24.67 / 0.180	22.14 / 0.189	24.38 / 0.161
DynPoint (K = 6)	24.69 / 0.097	31.34 / 0.045	29.30 / 0.061	24.59 / 0.086	22.77 / 0.099	27.63 / 0.049	25.37 / 0.039	26.53 / 0.068
Depth Flow
Scene Flow
Figure 5: Demonstration of Depth and Scene Flow Estimation. This figure presents the output of the target images obtained by warping the reference image using depth estimation (second row) or using both depth and scene flow estimation (third row). It is important to clarify that the figure is not intended for comparing view synthesis results. The synthesized figures generated based on scene flow inherently incorporate object motion as input, resulting in observable motion blur within the synthesized figures. Additionally, an error map represented by the intensity of red is provided to visualize the performance, where deeper shades of red indicate larger errors (in terms of pixel movement compared to corrected optic flow).
Error Map
Scene Flow
scenarios, namely Playground, Skating, Truck, and Umbrella, are presented in Fig. 3. Notably, leveraging monocular depth estimation, DynPoint generally produces views with improved geometric features, as evident in the Skating case shown in the second row of Fig. 3. Even in challenging scenarios like Umbrella, DynPoint successfully generates high-quality views while minimizing blurring effects. Nerfie (Tab. 2): In the case of the extended scenario, DynPoint exhibits superior performance in terms of PSNR and achieves comparable results to those obtained in the short video Nvidia, as presented in Tab. 2. This achievement can be attributed to our information aggregation approach, which focuses on effectively aggregating information from the target frame by establishing associations between its points and those in the reference frames. Novel view synthesis results for scenarios TAIL and TOBY SIT are depicted in Figure 4. It is evident that for longer videos, DynPoint continues to produce more realistic frames. Notably, the generated views in this figure do not exist in either the training or test dataset. HyperNeRF (Tab. 3): In the case of longer video sequences, DynPoint showcases superior performance in PSNR, as highlighted in Table 3. Iphone (Tab. 4): In the context of monocular videos without camera teleportation, as demonstrated in the work [17], DynPoint attains comparable outcomes to previous algorithms. Due to the inherent limitations of having few multi-view perspectives and overlapping information, establishing correspondences between adjacent frames becomes challenging. Consequently, the view synthesis task based on monocular videos proves to be more arduous on this dataset compared to other datasets.
In order to evaluate the effectiveness of the strategies proposed in our paper, we conducted three ablation studies. These studies include: (1) the absence of the multiple-step training strategy outlined in Sec. 3.2 and Sec. 3.3; (2) the utilization of only six frames in the vicinity  $(K = 3)$ ; and (3) the exclusion of the hierarchical point cloud. The results are presented in Tab. 5. It is evident from the results that the omission of the multiple-step training strategy leads to the largest drop in performance. Without this strategy, the first stage produces noisy outcomes for both monocular depth and scene
Figure 6: Demonstration of Failure Results on Nvidia Dataset. In this demonstration, we present the failure results generated by DynPoint.
flow estimation, consequently hindering the generation of the neural point cloud and impacting the performance of the second stage. Moreover, using a limited number of adjacent frames also adversely affects the final performance, which aligns with our expectations as limited inputs correspond to limited information. Although the removal of the hierarchical point cloud does not significantly degrade performance, it still plays a role in generating finer results.
4.3 Consistent Depth & Scene Flow Estimation On Davis
In order to validate the efficacy of the monocular depth estimation technique and the scene flow estimation method, we present the following visualization results: (1) Reconstructed Target Image using Monocular Depth Estimation: We demonstrate the reconstruction of the target image by warping the reference image based on the monocular depth estimation. (2) Reconstructed Target Image using Scene Flow and Depth Estimation: We present the visual reconstruction of the target image achieved through warping the reference image using solely depth estimation or a combination of scene flow estimation and depth estimation. The results are displayed in Fig. 5. Upon observing the second row, it is evident that DynPoint demonstrates commendable performance in reconstructing the static background of the target frame through monocular depth estimation. However, the primary errors (deep red part) occur in the dynamic region, where accounting for object movement becomes crucial for accurate warping. Moving to the third row, our scene flow estimation proves to be effective in capturing the movement of dynamic objects, as in the red error map Fig. 5.
4.4 Failure Cases
Despite the notable achievements of DynPoint, certain failure cases were observed during the view synthesis process, as demonstrated in Fig. 6. By comparing the first and second images, it becomes apparent that generating realistic facial features in regions with intricate details proves to be challenging. Furthermore, when comparing the third and fourth images, it is evident that DynPoint struggles with handling fine objects and reflections, as these aspects heavily rely on the accurate geometry inference obtained in the first stage.
In this research paper, we present DynPoint, an algorithm specifically designed to address the view synthesis task for monocular videos. Rather than relying on learning a global representation encompassing color, geometry, and motion information of the entire scene, we propose an approach that aggregates information from neighboring frames. This aggregation process is facilitated by learning correspondences between the target frame and reference frames, aided by depth and scene flow inference. The experimental results demonstrate that our proposed model exhibits improved performance in terms of both accuracy and speed compared to existing approaches.
Acknowledgements. Our research is supported by Amazon Web Services in the Oxford-Singapore Human-Machine Collaboration Programme and by the ACE-OPS project (EP/S030832/1). We are grateful to all of the anonymous reviewers for their valuable comments.
[67] Z. Zhang, F. Cole, Z. Li, M. Rubinstein, N. Snavely, and W. T. Freeman, "Structure and motion from casual videos," in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII. Springer, 2022, pp. 20-37.
[68] K. Zhou, L. Hong, C. Chen, H. Xu, C. Ye, Q. Hu, and Z. Li, "Devnet: Self-supervised monocular depth learning via density volume construction," in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX. Springer, 2022, pp. 125-142.
[69] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely, "Stereo magnification: Learning view synthesis using multiplane images," arXiv preprint arXiv:1805.09817, 2018.