Monocular Dynamic View Synthesis: A Reality Check
Authors: Hang Gao
Ruilong Li
Shubham Tulsiani
Bryan Russell
Angjoo Kanazawa
Venue: NeurIPS 2022 Accept (2022)
Keywords: dynamic view synthesis
novel-view synthesis
single-view 3D
dynamic 3D
3D vision
NeRF

Abstract: We study the recent progress on dynamic view synthesis (DVS) from monocular video. Though existing approaches have demonstrated impressive results, we show a discrepancy between the practical capture process and the existing experimental protocols, which effectively leaks in multi-view signals during training. We define effective multi-view factors (EMFs) to quantify the amount of multi-view signal present in the input capture sequence based on the relative camera-scene motion. We introduce two new metrics: co-visibility masked image metrics and correspondence accuracy, which overcome the issue in existing protocols. We also propose a new iPhone dataset that includes more diverse real-life deformation sequences. Using our proposed experimental protocol, we show that the state-of-the-art approaches observe a 1-2 dB drop in masked PSNR in the absence of multi-view cues and 4-5 dB drop when modeling complex motion. Code and data can be found at http://hangg7.com/dycheck.

Monocular Dynamic View Synthesis: A Reality Check
Hang Gao
Ruilong Li
Shubham Tulsiani?
Bryan Russell
Angjoo Kanazawa'
UC Berkeley
Carnegie Mellon University
Adobe Research
Abstract
We study the recent progress on dynamic view synthesis (DVS) from monocular video. Though existing approaches have demonstrated impressive results, we show discrepancy between the practical capture process and the existing experimental protocols, which effectively leaks in multi-view signals during training. We define effective multi-view factors (EMFs) quantify the amount of multi-view signal present in the input capture sequence based on the relative camera-scene motion. We introduce and correspondence accuracy, which overcome the issue existing protocols. We also propose new iPhone dataset that includes more diverse real-life deformation sequences. Using our proposed experimental protocol, we show that the state-of-the-arr approaches observe 1-2 dB drop in masked PSNR in the absence of multi-view cues and 4-5 4-5 dB drop when modeling complex motion. Code and data can be found at http: httt:///hangg.... dycchck
new metrics: co-visibility masked image metrics
Introduction
Dynamic scenes are ubiquitous in our everyday lives people moving around, cats purring, and trees swaying in the wind. The ability capture 3D dynamic sequences in "casual" manner, particularly through monocular videos taken by smartphone in an uncontrolled environment, will be a cornerstone scaling up 3D content creation, performance capture, and augmented reality.
Recent works have shown promising results in dynamic view synthesis (DVS) from a monocular video [1, 2, 3, 4, 5, 6, 7, 8]. However, upon close inspection, we found that there is discrepancy between the problem statement and the experimental protocol employed. As illustrated in Figure 1, the input data to these algorithms either contain frames that "teleport" between multiple camera viewpoints at consecutive time steps, which is impractical to capture from a single camera, or depict quasi-static scenes, which do not represent real-life dynamics.
In this paper, we provide a systematic means characterizing the aforementioned discrepancy and propose better set of practices for model fitting and evaluation. Concretely, we introduce effective multi-view factors (EMFs) to quantify the amount of multi-view signal monocular sequence based the relative camera-scene motion. With EMFs, we show that the current experimental protocols operate under an effectively multi-view regime. For example, our analysis reveals that the aforementioned practice of camera teleportation makes the existing capture setup akin to an Olympic runner taking a video of a moving scene without introducing any motion blur.
The reason behind the existing experimental protocol is that monocular DVS is challenging problem that also hard to evaluate. Unlike static novel-view synthesis where one may simply evaluate on held-out views of the captured scene, in the dynamic case, since the scene changes over time, evaluation requires another camera that observes the scene from a different viewpoint at the same time. However, this means that the test views often contain regions that were never observed in the input sequence. Camera teleportation, i.e., constructing a temporal sequence by alternating samples
Work partially done as part of HG's internship at Adobe.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

D-NeRF
NSFF
Norfies
Hype
Nerlies guee
HyperNeRF
HyperNeRF
Teleporting camera motion (A 0.2s)
Quasi-static scene motion (A 2s)
Figure I: Visualizing the training data of existing benchmarks. Existing datasets operate under the effective multi-view regime: The sequences either have (a) teleporting camera motion or (b) quasi-static scene motion. These motions leak multi-view cues, e.g., the model can observe the human (1* column) and hands (last column) roughly the same pose from different viewpoints.
from different cameras, addresses this issue at the expense of introducing multi-view cues, which are unavailable in the practical single-camera capture.
We propose two sets of metrics to overcome this challenge without the use of camera teleportation. The first metric enables evaluating only on pixels that were seen in the input sequence by computing the co-visibility of every test pixel. The proposed co-visibility mask can be used to compute masked image metrics (PSNR, SSIM [9] and LPIPS [10]). While the masked image metries measure the quality of rendering, they do not directly measure the quality of the inferred scene deformation. Thus, we also propose a second metric that evaluates the quality of established point correspondences by the percentage of correctly transferred keypoints (PCK-T) [L]]. The correspondences may be evaluated between the input and test frames or even within the input frames, which enable evaluation on sequences that are captured with only single camera.
We conduct extensive evaluation existing datasets [5, 7] as well challenging motion and diverse scenes. When tested on existing datasets without camera teleportation, the state-of-the-arr methods observe 1-2 dB drop in masked PSNR and -5% drop in PCK-T. When tested on complex motion with the proposed dataset, existing approaches observe another 4-5 dB drop in masked PSNR and -30% drop in PCK-T, suggesting large room for improvement. We encourage future works to report EMFs on new data and adopt our experimental protocol to evaluate monocular DVS methods. Code and data are available at our project page.
dataset that includes more
2 Related work
Non-rigid structure from motion (NR-SIM). Traditional NR-SfM tackles the task of dynamic
3D inference by fitting parametric 3D morphable models [12, 13, 14, 15, 16, 17, 18, 19], or fusing non-parametric depth scans of generic dynamic scenes [20, 21, 22, 23, 24]. All of these approaches aim recover accurate surface geometry at each time step and their performance is measured with ground truth 3D geometry or 2D correspondences with PCK [25] when such ground truth is not available. In this paper, we analyze recent dynamic view synthesis methods whose goal a photo-realistic novel view. Due to their goal, these methods do not focus on evaluation against ground truth 3D geometry, but we take inspiration from prior NR-SIM works evaluate the quality of the inferred 3D dynamic representation based on correspondences. We also draw inspiration from previous SR-SfM work that analyzed camera/object speed and 3D reconstruction quality [26, 27].
generate
Monocular dynamic neural radiance fields (dynamic NeRFs). Dynamic NeRFs reconstruct moving scenes from multi-view inputs given pre-defined deformation template [28, 29, 30, 31, 32, 33, 34, 35]. In contrast, there is series of recent works that seek to synthesize high-quality novel views of generic dynamic scenes given a monocular video [1, 2, 3, 4, 5, 6, 7,8). These works can be classified into two categories: a deformed scene directly modeled as time-varying NeRF the world space [1, 4, 6] or as a NeRF in canonical space with time-dependent deformation [2, 3. 5, 8). The evaluation protocol these works inherit from the original static-scene NeRF [36] that quantify the rendering quality of held-out viewpoints using image metrics, e.g., PSNR. However, in dynamic

---

scenes, PSNR from an unseen camera view may not be meaningful since the novel view may include regions that were never seen in the training view (unless the method can infer unseen regions using learning based approaches). Existing approaches resolve this issue by incorporating views from mul- tiple cameras during training, which we show results in an effectively multi-view setup. We introduce metrics to measure the difficulties of an input sequence, monocular dataset with new evaluation protocol and metrics, which show that existing methods have large room for improvement.
Effective multi-view in a monocular video
We consider the problem of dynamic view synthesis (DVS) from a monocular video. monocular dynamic capture consists of single camera observing moving scene. The lack of simultaneous multi-view in the monocular video makes this problem more challenging compared to the multi-view setting, such reconstructing moving people from multiple cameras [30, 34, 37].
Contrary to the conventional perception that the effect of multi-viww is binary for capture (single versus multiple cameras), we show that it can be characterized on continuous spectrum. Our insight that monocular sequence contains effective multi-view cues when the camera moves much faster than the scene, though technically the underlying scene is observed only once at each time step.
3.1 Characterizing effective multi-view in monocular video
Although monocular video only sees the scene from one viewpoint at a time, depending on the capture method, it can still contains cues that are effectively similar to those cap- tured by multi-view camera rig, which we call as effective multi-view. As shown in Figure 2, when the scene moves significantly slower than the camera (to the far right end of the axis), the same scene observed from multiple views, resulting in multi- view capture. In this case, DVS re- duces well-constrained multi-view stereo problem each time step. Consider another case where the camera moves significantly faster compared to the scene so that it observes roughly the same scene from different viewpoints. As the camera motion approaches the infinity this again reduces the monocular capture to a multi-view setup. We therefore propose to characterize the amount of multi-view cues by the relative camera-scene motion.
Slow cam. Fast scene
Fast cam. Slow scene
KAA
Striet monocular
Effective
Strict multi-view
multi-view
The spectrum of effective multi-view in monocular video. A video captured by single camera can still have multi- view when the camera moves much faster than the scene. We disentangle recent advances DVS given monocular video from such phenomena.
Figure 2:
3.2 Quantifying effective multi-view in a monocular video
For practicality, we propose two metrics, referred to effective multi-view factors (EMFs). The first metric, full EMF Q is defined as the relative ratio between the motion magnitude of the camera to the scene, which in theory characterizes the effective multi-view perfectly, but in practice can be expensive and challenging to compute. The second metric, angular EMF w is defined as the camera angular velocity around the scene look-at point, which only considers the camera motion; while approximate, it is casy to compute and characterizes object-centric captures well.
Full EMF Ω: ratio of camera-scone motion magnitude. Consider monocular video of moving scene over a set of time steps T. At each discrete time E T, let the camera's 3D location be Or- We consider each point x, on the domain of observed seene surface S? c IR3. We define the camera-scene motion as the expected relative ratio,
0
- x;
Ω
E 6,+1ET
E
X.
X++1
where the denominator X - X, denotes the scene flow and the the numerator 0,+1 O2 denotes the 3D camera motion, both over time step forward. The 3D scene flow can be estimated via the
3

---

#Train cam.
Duration
FPS
Depth
Kpt.
Sequences
D-NeRF
- 150
3s
60
MV
HyperNeRF
2
8
15s
15
MV + 13 SV
Nerfies [5]
2
8
15s
5/15
4 MM
NSFF [4]
24
3s
15/30
Estimated [39]
MV
iPhone (proposed)
8 15s
30/60
Lidar
7 MV + SVV
Table Summary of the existing and proposed iPhone datasets. Existing datasets operate under the effective
multi-view regime, teleporting between multiple cameras during training to generate a synthetic monocular video. Unlike previous protocols, the proposed iPhone dataset consists of dynamic sequences captured by single smoothly moving camera. also has accompanying depth maps for training supervision and labeled keypoints for evaluation. "MV" denotes multi-camera capture, and "SV" denotes single-camera capture.
2D dense optical flow field and the metric depth map when available, or monocular depth map from off the---e-eef approaches [38, 39] in the general case. Please the Appendix for more details. Note that f2 in theory captures the effective multi-view factor for any sequence. However, in practice, 3D scene flow estimation is an actively studied problem and may suffer from noisy costly predictions.
Angular EMF wi camera angular velocity. We introduce a second metric w that easy to compute in practice. We make additional assumption that the capture has single look-at point in world space, which often holds true, particularly for captures involving a single centered subject. Specifically, given look-at point by triangulating the optical-axes of all cameras per [5] and the frame rate N, the camera angular velocity w is computed as scaled expectation,
(a O2, oc+1)
)} N.
I.H+ITT
arecos
| 0..|
0
Note that even though w only considers the camera motion, it is indicative of effective multi-view in the majority of existing captures, which we describe Section 4.1.
For both and w, the larger the value, the more multi-view cue the sequence contains. For future works introducing new input sequences, we recommend always reporting angular EMF for its simplicity and reporting full EMF when possible. Next we inspect the existing experimentation practices under the lens of effective multi-view.
4 Towards better experimentation practice
In this section, we reflect on the existing datasets and find that they operate under the effective multi-view regime, with either teleporting camera motion quasi-static scene motion. The reason behind the existing protocol is that monocular DVS is challenging from both the modeling and evaluation perspective. While the former challenge is well known, the latter is less studied, expand below. To overcome the existing challenge in the evaluation and enable future research to experiment with casually captured monocular video, propose better toolkit, including two new metrics and a new dataset of complex motion in everyday lives.
4.1 Closer look at existing datasets
We investigate the datasets used for evaluation in D-NeRF [3], HyperNeRF [7], Nerfies [5], and NSFF [4]. Table shows their statistics. We evaluate the amount of effective multi-view cues via the proposed EMFs, shown in Figure 3. We find that existing datasets have large EMF values on both metrics. For example, the HyperNeRF dataset has an w as large as 200 /s. To put these numbers in context, a person imaging object 3m away has move at Im/s get w 20% (close to the statistics in the proposed dataset). Some datasets exhibit w higher than 120º /s, which is equivalent to camera motion faster than the Olympic 100m sprint record, without incurring any motion blur.
Visualizing the actual training data shown in Figure reveals that existing datasets feature non-practical captures of either (1) teleporting/fast camera motion or (2) quasi-static/slow scene motion. The former not representative of practical captures from a hand-held camera, egg., smartphone, while the latter is not representative of moving objects in daily life. Note that, out

---

EMF 4
EMF @
14.98
2135.45
10
10a
2.52
1.31
134.54
10
102
53.24
0.20
15.44
D-NeRF
HyperNeRF
Nerfies
NSFF
iPhone (Ours)
D-NeRF
HyperNeRF
Nerfies
NSFF
iPhone (Ours)
Figure 3: Statistics of effective multi-view factors (EMFs) across different datasets. Existing datasets have high EMFs, indicating the abundance of multi-view cues during training (note that the y-axis is in log scale). The proposed iPhone dataset features single camera, capturing the moving scene with smooth motion, and thus has smaller EMFs. Our results help ground the difficulty of each dataset for DVS from a monocular video.
of the 23 multi-camera sequences that these prior works used for quantitative evaluation, 22 have teleporting camera motion, and has quasi-static scene motion - the CURLS sequence shown at the 5th column in Figure 1. All 13 single-camera sequences from HyperNeRF used for qualitative evaluation have quasi-static scene motion.
The four datasets also share similar data protocol for generating effective multi-view input sequences from the original multi-camera rig capture. In Fig- ure 4, we illustrate the teleporting protocol used in Nerfies [5] and HyperNeRF [7] canonical ex-
Left cam. Right cam.
AAAA
Moving scene
Capture seq. Input seq.
ample. They sample alternating frames from two physical cameras (left and right in this case) mounted on a rig to create the training data. NSFF [4] samples alternating frames from 24 cameras based the data released from Yoon et al. [40]. D-NeRF [3] experi- ments on synthetic dynamic scenes where cameras are randomly placed on a fixed hemisphere at every time step, in effect teleporting between 100-200 cameras. We encourage you to visit our project page to view the input videos from these datasets.
Figure 4: Camera teleportation from a multi- camera rig. As an example, we show the data protocol used in Nerfies [5] and HyperNeRF [7].
Existing works adopt effective multi-view capture for two reasons. First, makes monocular DVS more tractable. Second, it enables evaluating novel view on the full image, without worrying about the visibility of each test pixel, as all camera views were visible during training. We show this effect in Fig- ure 5. When trained with camera teleportation (3 column), the model generate high-quality full image from the test view. However, when trained without camera teleportation (4th column), the model struggles hallucinate unseen pixels since NeRFs [5, 36] are not designed to predict completely unseen portions of the scene, unless they are specifically trained for generalization [41]. Next, we pro- pose new metrics that enable evaluation without using camera teleportation. Note that when the model is trained without camera teleportation, the rendering quality also degrades, which we also evaluate.
4.2 Our proposed metrics
While the existing setup allows evaluating the full rendered image from the test view, the perfor- mance under such evaluation protocol, particularly with teleportation, confounds the efficacy of the proposed approaches and the multi-view signal present the input sequence. To evaluate with an actual monocular setup, we propose two new metrics that evaluate only on seen pixels and measure the correspondence accuracy of the predicted deformation.
Co-visibility masked image metrics. Existing works evaluate DVS models with image metrics on the full image, e.g., PSNR. SSIM [9] and LPIPS [10]. following novel-view synthesis evaluation on static scenes [36]. However, in dynamic scenes, particularly for monocular capture with multi-camera validation, the test view contains regions that may not have been observed at all by the training camera. To circumvent this issue without resorting to camera teleportation, for each pixel in the test image, we propose co-visibility masking, which tests how many times a test pixel has been observed in the training images. Specifically, we use optical flow to compute correspondences between every test image and the training images, and only keep test pixels that have enough correspondences in the training images via thresholding. This results in a mask, illustrated in Figure 5, which we use to confine the image metrics. We follow the common practice from the image generation literature and adopt masked metries, mPSNR and mLPIPS [42, 43J. Note that NSFF [4] adopts similar metrics
5

---

3.13 111//51.4 222225133 36
1.22/59. 59. 10
3D PRINTER
Nerfies w/ mask (non-teleporting) Figure 5: Evaluation without teleportation with the co-visibility mask. 2/Ww metrics of the input sequence are annotated on the top-left. Existing works avoid evaluating on unseen pixels by camera teleportation (3 column). Naively training with the non-teleporting (smooth) camera trajectory causes evaluation issues on the full image (1th column), since NeRF model cannot hallucinate unseen regions. We propose to only evaluate seen regions during training by co-visibility mask (we show non-visible regions in green the last column). Note that the model performs poorly on seen regions well when trained without camera teleportation.
Nerfies
Nerfies
Training view
Test view
(leleporting) [5]
(non-teleporting)
3.30/ 160.55
TOBY SIT
12px
2s
+ 2N
Training view
Test view
Prediction [5]
Source Kpt.
Target Kpt.
Pred. Kpt. [5]
Figure 6: High-quality novel-view synthesis does not imply accurate correspondence modeling. 2/W metrics of the input sequence shown the top-left. The time steps of the ground-truth data and predictions are shown on the bottom-left. Using Nerfies [S] as an example, we show that the model renders high-quality results (3 column) without modeling accurate correspondences (last column). Transferred keypoints are colorized by heatmap of end-point error, overlaid on the ground-truse target frame.
but for evaluating the rendering quality on foreground versus background regions. We additionally report mSSIM by partial convolution [44], which only considers seen regions during its computation. More details are in the Appendix. Using masked image metrics, we quantify the performance gap in rendering when model trained with or without multi-view in Section 5.1.
Percentage of correctly transferred keypoints (PCK-T). Correspondences lie at the heart of traditional non-rigid reconstruction [21], which is overlooked in the current image-based evaluation. We propose evaluate 2D correspondences across training frames with the percentage of correctly transferred keypointss (PCK-T) [11], which directly evaluates the quality of the inferred deformation. Specifically, we sparsely annotate 2D keypoints across input frames ensure that each keypoint is fully observed during training. For correspondence readout from existing methods, we use either root finding [45] or scene flow chaining. Please the Appendix for details correspondence readout, and metric computation. As shown in Figure 6, evaluating correspondences reveal that high quality image rendering does not necessarily result in accurate correspondences. which indicates issues in the underlying surface, due to the ambiguous nature of the problem.
keypoint annotation,
4.3 Proposed iPhone dataset
Existing datasets can be rectified by removing camera teleportation and evaluated using the proposed metrics, as we do in Section 5.1. However, even after removing camera teleportation, the existing datasets are still not representative of practical in-the-wild capture. First, the existing datasets are limited in motion diversity. Second, the evaluation baseline in existing datasets small, which can hide issues in incorrect deformation and resulting geometry. For these reasons, we propose new dataset called the iPhone dataset shown in Figure 7. In contrast to existing datasets with repetitive object motion, we collect 14 sequences featuring non-repetitive motion, from various categories such as generic objects, humans, and pets. We deploy three cameras for multi-camera capture one hand-held moving camera for training and two static cameras of large baseline for evaluation. Furthermore, our iPhone dataset comes with metric depth from the lidar sensors, which we use to provide ground-trute tuuth depth for supervision. In Section 5.2, we show that depth supervision, together

---

HARU
CREEEER
(a) Capture setup
(b) Sampled sequences
Figure 7: Capture setup and sampled sequences from our proposed iPhone dataset. The proposed iPhone dataset has single hand-held camera for training and multiple cameras with large baseline for validation. It has accompanying depth from the iPhone sensor and features diverse and complex real-life motions.
with other regularizations, beneficial for training DVS models. Please see the Appendix for details on multi-camera capture setup, data processing, and more visualizations.
Reality check: re-evaluating the state of the art
In this section, we conduct series of empirical studies to disentangle the recent progress in dynamic view synthesis (DVS) given a monocular video from effective multi-view in the training data. We evaluate current state-of-the-an -hrt methods when the effective multi-view factor (EMF) is low.
Existing approaches and baselines. We consider the following state-of-the-arr art approaches for empirical studies: NSFF [4], Nerfies [5] and HyperNeRF [7]. We choose them canonical examples for other approaches [1, 2, 3, 6, 8, 34, 35J, discussed in Section 2. We also evaluate time-conditioned NeRF (T-NeRF) common baseline [1, 4J. Unlike the state-of-the-art methods, is not possible to extract correspondences from a T-NeRF. summary of these methods can be found in the Appendix.
Datasets. We evaluate on the existing datasets as well as the proposed dataset. For existing datasets, we use the multi-camera captures accompanying Nerfies [5] and HyperNeRF [7] for evalulation. Due to their similar capture protocol, we consider them single dataset in our experiment (denoted as the Jerfies-HyperNer annotations. Our dataset has 7 multi-camera captures and 7 single-camera captures. We evaluate novel-view synthesis on the multi-camera captures and correspondence on all captures. Our data adopts the data format from the Nerfies-HyperNGr dataset, with additional support for depth and correspondence labels. All videos are at 480p resolution and all dynamic scenes are anward-facing
dataset). It consists of 7 sequences in total, which we augment with keypoint
Masked image and correspondence metrics. Following Section 4.2. evaluate co-viss visibility masked image metrics and the correspondence metric. We report masked image metrics: mPSNR, mSSIM [9, 44], and mLPIPS [4, 10, 42, 43]. We visualize the rendering results with the visibility mask. For the correspondence metric, we report the percentage of correctly transferred keypoints (PCK-T) [11] with threshold ratio 0.05. Additional visualizations of full image rendering and inferred correspondences can be found in the Appendix.
Implementation details. We consolidate Nerfies [5] and HyperNeRF [7] in one codebase using JAX [46]. Compared to the original official code releases, our implementation aligns all training and evaluation details between models and allows correspondence readout. Our implementation reproduces the quantitative results in the original papers. We implement T-NeRF in the same codebase. For NSFF [4], we tried both the official code base [47] and public third-party re-implementation [48], where the former fails to converge our proposed iPhone dataset while the latter works well. We thus report results using the third-party re-implementation. However, note that both the original and the third-party implementation represent the dynamic scene in normalized device coordinates (NDC). As NDC designed for forward-facing but not considered haward-facing scenes, layered artifacts may appear due to its scalscale sampling rate in the world space, shown in Figure 9, More details
7

---

mPSNR
mS5 IM
mLPPPS
PCK-T
teleperting
22.00
non-teleporting
0.302
20.00
0.200
0.403
Norting RppertoR
T-ear
Noriies AipeRaer
74teer
Hemma Myoooooe
Nerios
Hypoliatt
3.40/120 12..4
2.57/ /60.36
BROOM
3.13/251.37
1.22/59.44
3D PRINTER
Training view
Test view
Nerfies (teleporting) [5] Nerfies (non-teleporting)
Figure 8: Impact of effective multi-view the Berfies-yperrNete dataset. 22ww metries of the input sequence are shown the top-left. We compare the existing camera teleporting setting and our non-teleporting setting. (Top): Quantitative results different models using our proposed evaluation metrics. (Bottom): Qualitative comparison using Nerfies as an example. Two settings use the same set of co-visibility masks computed from common training images. Visualizations of other models are in the Appendix.
1.52/33 02//33.46
CHICKEN
0,33/33. 033/33.77
PEEL BANANA
Training view
Test view
T-NeRF
NSFF [34]
Nerfies [5]
HyperNeRF [7]
Figure 9: Qualitative results on the Nerfies-HyperNeRF dataset without camera teleportation. 2/Ww metrics
of the input sequence shown the top-left. Existing approaches struggle modeling dynamic regions.
about aligning the training procedure and remaining differences provided in the Appendix. Code, pretrained models, and data are available on the project page.
5.1 Reality check the Nerfies-HyperNefe dataset
Impact of effective multi-view. We first study the impact of effective multi-view on the Nerfies- HyperNeRF [5. 7] dataset. In this experiment, rectify the effective multi-view sequences by only using the left camera during training as opposed to both the left and right cameras, illustrated in Figure 4. We denote the original setting as "teleporting" and the rectified sequences as "non- teleporting". We train all approaches under these two settings with the same held-out validation frames and same set co-visibility masks computed from common training frames. In Figure (Top), all methods perform better across all metrics when trained under the teleporting setting compared the non-teleporting one, with the exception of PCK-T for NSFE We conjecture that this is because that NSFF has additional optical flow supervision, which is more accurate without

---

Ω = 1.30
1 = 0.24
mPSNRT
mSSIMT
mLPIPS}
PCK-TT
MPSNRT
MSSIMT
mLPIPS
PCK-T1
T.NeRF
21.55
0.595
0.297
T.NeRF
16.96
0.577
0.379
NSFF
19.53
0.521
0.471
0.422
NSFF [-]
15.46
0.551
0.396
0.256
Nerties
Nerlis [5]
0.570
0.453
HyperNeRF171
HyperNeRF17
0.569
0.400
Table 2: Benchmark results the rectified Nerfies- HyperNeRE dataset. Please see the Appendix for the breakdown over multi-camera sequences.
Table 3: Benchmark results the proposed iPhone dataset. Please see Appendix the breakdown over 7 multi-camera sequences of complex motion.
mPSNR
mSSIM
mLPIPS
PCK-T
original
17.
0.459
16.00
0.459
0.409
0.350
15.00
0.300
0500
0.250
Hyperloots
livrlee
Hyperllott
Vhenles
Hypestott
Nertler
RyperNcop
0.20/7.66 2.6/7.62
TEDDY
PAPER WINDMILL
0.38/10.7 100.1
Training view
Test view
HyperNeRF [7]
+B
+B+D
+B+D+S
Figure 10: Ablation study improving the state of the art the proposed iPhone dataset. 92/w metries of the input sequence shown on the top-left. +B, +D, +S denotes random background compositing [32], additional metric depth supervision [l. 4] from iPhone sensor, and surface sparsity regularizer [49]. respectively.
camera teleportation. In Figure 8 (Bottom), we show qualitative results using Nerfies (we include visualizations of the other methods in the Appendix). Without effective multi-view, Nerfies fails at modeling physically plausible shape for broom and wires. Our results show that the effective multi-view the existing experimental protocol inflates the synthesis quality of prior methods, and that truly monocular captures are more challenging.
Benchmark results without camera teleportation. In Table 2 and Figure 9, we report the quanti- tative and qualitative results under the non-teleporting setting. Note that our implementation of the T-NeRF baseline performs the best among all four evaluated models terms of mPSNR and mSSIM. In Figure 9, we confirm this result since T-NeRF renders high-quality novel view for both sequences. HyperNeRF produces the most photorealistic renderings, measured by mLPIPS. However it also produces distorted artifacts that do not align well with the ground truth (e.g., the incorrect shape in the CHICKEN sequence).
5.2 Reality check on the proposed iPhone dataset
Ablation study on improving the state of the art. We find that existing methods perform poorly out-of-the-box on the proposed iPhone dataset with more diverse and complex real-life motions. In Figure 10 (Bottom), we demonstrate this finding with HyperNeRF [7] for it achieves the highest mIPP metric on the Nerfies- HyrfeeeR dataset. Shown in the 3rd column, HyperNeRF produces visually implausible results with ghosting effects. Thus we explored incorporating additional regu- larizations from recent advances in neural rendering. Concretely, we consider the following: (+B)

---

0.75/3.79
APPLE
0.13/6.37
SPACE OUT
Training view Test view
Depth re-proj. T-NeRF++
NSFF [4]
Nerfies+ HyperNeRI++
Figure 11: Qualitative results on the proposed iPhone dataset. fi/w metrics of the input sequence are shown the top-left. The models shown here trained with all the additional regularizations (+B+D+S) except NSFF However, existing approaches still struggle to produce high-quality results.
random background compositing [32]: (+D) depth loss on the ray matching distance [1, 4]: and (+S) a sparsity regularization for scene surface [49]. In Figure 10 (Top), we show quantitative results from the ablation. In Figure 10 (Bottom), show visualizations of the impact of cach regularization. Adding additional regularizations consistently boosts model performance. While we find the random background compositing regularizations particularly helpful, extra depth supervision and surface regularization further improve the quality, egg., the fan region of the paper windmill.
Benchmarked results. In Figure 11, we show qualitative results from our benchmark using the best model settings from the ablation study, denoted "++". Note that it is non-trivial to apply the same enhancements to NSFF for its NDC formulation so we keep it as-is. We visualize the lidar depth re-projection from the training view (1st column) to the test view (2nd column), reference for qualitative comparison (3td column). Note that the white region is occluded from the input view, whereas the green region is occluded from the most of input video frames. We observe that existing approaches do not handle complex deformation well. For example, all models fail at fusing valid human shape the SPACE OUT sequence. In Table Nerfies- Herfies-HypeNNeRF dataset: the baseline T-NeRF performs the best in terms of mPSNR and mSSIM while HyperNeRR produces the most photorealistic renderings in terms of mLPIPS. The overall synthesis quality and correspondence accuracy of all methods drop considerably compared to the results on the Nerfies-HyperNeRF dataset. Taking Nerfies as an example, drops 4.4 dB mPSNR, 69.6% in mLPIPS, and 40. 1 % in PCK-T. Our study suggests an opportunity for large improvement when modeling complex motion.
find similar trend as in the
6 Discussion and recommendation for future works
In this work, we expose issues in the common practice and establish systematic means to calibrate performance metries of existing and future works, in the spirit of papers like [50, 51, 52]. We provide initial attempts toward characterizing the difficulty of monocular video for dynamic view synthesis (DVS) in terms of effective multi-view factors (EMFs). In practice, there are other challenging factors such variable appearance, lighting condition, motion complexity and more. We leave their characterization for future works. We recommend future works to visualize the input sequences and report EMFs when demonstrating the results. We also recommend future works to evaluate the correspondence accuracy and strive for establishing better correspondences for DVS.
We would like to thank Zhengqi Li and Keunhong Park for valuable feedback and discussions; Matthew Tancik and Ethan Weber for proofreading. We are also grateful to our pets: Sriracha, Haru, and Mochi, for being good during capture. This project is generously supported in part by the CONIX Research Center, sponsored by DARPA, as well as the BDD and BAIR sponsors.
Acknowledgements.
10

---

References
[1] Wenqi Xian, Jia-Bin Huang. Johannes Kopf, and Changil Kim. Space-timm since-timeeae neural irradiance fields for free-viewpoint video. In CVPR, 2021.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis a dynamic scene from monocular video. In ICCV, 2021.
[3] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer, D-NeRF: Neural Radiance Fields for Dynamic Scenes. In CVPR, 2021.
[4] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021.
[5] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Scitz, and Ricardo Martin- Brualla. Nerfies: Deformable neural radiance fields. In ICCV, 2021.
[6] Chen Gao, Ayush Saraf. Johannes Kopf, and lia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In ICCV, 2021.
[7] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensionall representation for copologically varying neural radiance fields. ACM TOG, 2021.
Tianhao Wu, Fangeheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli. Denerf: Self-supervised decoupling dynamic and static objects from a monocular video. NeurlPS, 2022.
[9] Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 2004.
[10] Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[11] Nilesh Kulkarni, Abhinav Gupta, and Shubham Tulsiani. Canonical surface mapping via geometric cycle consistency. In /CCV, 2019.
[12] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In CVPR, 1999.
[13] Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Reconstructing 3d human pose from 2d image landmarks. In ECCV, 2012.
[14] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep smpl: Automatic estimation of 3d human pose and shape from single image. In ECCV, 2016.
[15] Abhishek Kar. Shubham Tulsiani, Joao Carreira, and Jitendra Malik. sategory-speciite object reconstruc- tion from single image. In CVPR, 2015.
[16] Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik. Learning category-specific mesh reconstruction from image collections. In ECCV, 2018.
[17] Nilesh Kulkarni, Abhinav Gupta, David Fouhey, and Shubham Tulsiani. Articulaton-awaal canonical surface mapping. In CVPR, 2020.
[18] Gengshan Yang. Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William Freeman, and Ce Liu. Lasr: Learning articulated shape reconstruction from a monocular video. In CVPR, 2021.
[19] Gengshan Yang, Minh Vo, Neverova Natalia, Deva Ramanan, Vedaldi Andrea, and Joo Hanbyul. Banmo: Building animatable 3d neural models from many casual videos. In CVPR, 2022.
[20] Brian Curless and Mare Levoy. A volumetric method for building complex models from range images. In ACM CGIT, 1996.
[21] Richard Newcombe, Dieter Fox, and Steve Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In CVPR, 2015.
[22] Hao Li, Robert W. Sumner, and Mark Pauly. Global correspondence optimization for non-rigid registration of depth scans. SGP, 2008.

---

[23] Wei Gao and Russ Tedrake. Surfelwarp: Efficient non-volumetric single view dynamic reconstruction. In RSS, 2018.
[24] Aljaz Bozic, Pablo Palafox, Michael Zollöfer, Angela Dai, Justus Thies, and Matthias Nießner. Neural non-rigid tracking. In NeurfPS, 2020.
[25] Yi Yang and Deva Ramanan. Articulated human detection with flexible mixtures of parts. IEEE TPAMI, 2012.
[26] Hyun Soo Park, Takaaki Shiratori, lain Matthews, and Yaser Sheikh. 3d reconstruction of moving point from a series of 2d projections. In ECCV. 2010.
[27] Kaan Yucer, Oliver Wang, Alexander Horkine-Hormung. and Olga Borkine-Homungg Reconstruction of articulated objects from a moving camera. In ICCVW, 2015.
[28] Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. Neural human performer: Learning generalizable radiance fields for human performance rendering. NewrIPS, 2021.
[29] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field. ICCV, 2021.
[30] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang. Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021.
[31] Shin-Yang Su, Frank Yu, Michael Zollhoefer, and Helge Rhodin, A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose. In NeurlPS, 2021.
[32] Chung- Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Semnelaache-Shhizeemeee Humannerf: vicwpvin rendering of moving people from monocular video. In CVPR, 2022.
[33] Guy Gafni, Justus Thies. Michael Zollhöfer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In CVPR, 2021.
[34] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, and Zhaoyang Lv. Neural 3d video synthesis from multi-view video. In CVPR, 2022.
[35] Yilum Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and vidco processing. In /CCV, 2021.
[36] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[37] Ruilong Li, Julian Tanke, Minh Vo, Michael Zollhofer, Jurgen Gall, Angjoo Kanazawa, and Christoph Lassner. Tava: Template-free animatable volumetric actors. In CVPR, 2022.
[38] Zachary Teed and Jia Deng. Raft: Recurrent pairs field transforms for optical flow. In ECCV, 2020.
[39] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. ICCV, 2021.
[40] Jac Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Seo Park, and Jan Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera. In CVPR, 2020.
[41] Alex Yu, Vickic Ye, Matthew Tancik, and Angjoo Kanazawa. pixeINeRF: Neural radiance fields from one or few images. In CVPR, 2021.
[42] Leon A Gatys, Alexander S Ecker, Matthias Bethge, Aaron Hertzmann, and Eli Shechtman. Controlling perceptual factors in neural style transfer. In CVPR, 2017.
[43] Minyoung Huh, Richard Zhang, Jun-Yan projecting images into class-conditional generative networks. In ECCV, 2020.
Yan Zhu, Sylvain Paris, and Aaron Hertzmann. Transforming and
[44] Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In ECCV, 2018.
[45] Xu Chen, Yufeng Zheng, Michael J Black, Otmar Hilliges, and Andreas Geiger. Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In ICCV, 2021.
12

---

[46] James Bradbury. Roy Frostig, Peter Hawkins. Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations Python+NumPy programs, 2018.
[47] Zhengqi Li. Simon Niklaus. Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes: Official code release, 2021.
[48] kweal23. Neural scene flow fields for space-time view synthesis of dynamic scenes: Pytorch re- implementation, 2021.
[49] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
[50] Santosh K Divvala, Derek Hoiem, James H Hays, Alexci A Efros, and Martial Hebert. An empirical study of context in object detection. In CVPR, 2009.
[51]
Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. What do single-view 3d reconstruction networks learn? In CVPR, 2019.
[52] Kevin Musgrave, Serge Belongie, and Ser-Nam Lim, metric learning reality check. ECCV, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] in Section.
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "yPNVkq5NFJv", "reviewer": "Reviewer_2BcV", "summary": "This paper studied the problem of dynamic 3D scene synthesis from monocular video sequence and found flaws of over-representation of slow-moving objects with a fast-moving camera in existing datasets. The authors then proposed a new metric called effective multi-view actors (EMF) to quantify the amount of multi-view signal in the image sequence. The authors also introduced a new dataset with very low EMF and argued that the new dataset should be more suitable for evaluation of dynamic 3D scene synthesis methods. Finally the authors evaluated four representative algorithms on the new dataset and find performance gap not being noticed with previous existing datasets.", "questions": "When evaluate on different sequences with different EMF value, do we always find lower loss for high EMF sequences on existing methods?", "limitations": "The authors answered YES for the question “Did you describe the limitations of your work” but not stating which section contains it.", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors delve into the characteristic of existing dataset and managed to produce innovative metric to evaluate the difficulty of dataset in terms of monocular dynamics.\n\nWeaknesses\n- While the ability of build 3d representation from monocular dynamics is desirable, real word video sequences could also contain a well proportion of high EMF data. So the argument that low EMF datasets is better for evaluation may not hold unconditionally. The difficulty of dataset may also come from other aspects, such as shape complexity and surface property of objects.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Xqe-Mv7xpxx", "reviewer": "Reviewer_dLU6", "summary": "This paper proposes a dataset of monocular videos to evaluate non-rigid novel view synthesis methods. A factor for measuring the multi-view effect is proposed, and two evaluation metrics are proposed. ", "questions": "If the authors propose a new method, it is good to evaluate it in the proposed datasets and evaluation metrics. However, from my perspective, the contribution is not significant enough for presentation in NeurIPS.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well-organized, and it is easy to read.\n\nThe datasets with evaluation metrics are proposed.\n \nSeveral methods of non-rigid novel view synthesis are evaluated on the proposed datasets. \n\n\n\nWeakness:\n\n1. Using a multi-camera setup is more reliable than single-camera setup for performance analysis. After all, the goal is to evaluate methods instead of training models, I don’t think that it is the drawback of existing methods. We do not always need to capture new data for non-rigid novel view synthesis evaluation, so it is difficult for me to understand the strengths of the proposed solution.\n\n2. The idea of Masked-PSNR is not surprising. First, if our goal is to evaluate the regions that are observed, it would be straightforward to mask other regions out. Second, we may also hope that the methods can predict and synthesize the regions that are not observed in training data. In this scenario, the mask should not be used. Overall, I don’t think that the masking is novel.\n\n3. Although Nerfies trained models on images sampled from two cameras, the method is able to train models using a single-camera setup. \n\n4. The factor of the multi-view effect is just a simple trick, and there exist many other variants if we want. It is correct, but I don’t think that it is sufficiently significant in this problem. \n\nPost rebuttal,\n\nAfter discussion, I agree with Reviewer vDbH, and I would strongly suggest the authors use the comments by vDbH in the final submission to position the work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PDN1Abae72", "reviewer": "Reviewer_vDbH", "summary": "This paper studies the effective metrics for evaluating recent works on novel view synthesis from monocular videos. The proposed effective multi-view factors (EMF) measures multi-view signals in the evaluation of view synthesis. It also comes with a new dataset which tries to mitigate the multi-view signals in captures. Two new metrics are proposed to measure the quality of view synthesis and motion. Several state-of-the-art methods are further improved, but still struggle in the cases of motion and few multi-view signal.", "questions": "See above", "limitations": "Limitations are not discussed. The author can discuss the use case of EMF and two new metrics more.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- EMF\n\nThe effective multi-view factors measure the amount of ''multi-viewness'' in the capture. Such ''multi-viewness'' makes the task of novel view synthesis on dynamic scenes easier as the problem degrades into a multi-view setup. EMF consists of scene motion and camera angular velocity. And higher value in either suggests there is a strong multi-viewness in the capture. EMF basically tells how easy/difficult the capture is to reconstruct. Such metrics is nice to have in the future dataset release.\n\n- Masked PSNR and Correspondence\n\nTwo new metrics are proposed to further measure the quality of the synthesized image. Both are tailored for dynamic scenes, but I see them also useful in static scenes as well. Masked PSNR only calculates PSNR in the valid region, and PCK-T tells whether the predicted motion is correct. The design choices are smart.\n\n# Weaknesses\n\n- Correspondence\n\nI see masked PSNR can be applied to any method as long as the ground truth pose and depth are available. However, the correspondence metrics lacks generalizability. As shown in the paper, only methods that explicitly models motion can calculate the PCK-T score. It does not apply to methods like T-NeRF.\n\n- Camera angular velocity\n\nNerfies and HyperNeRF have a high camera angular velocity in Tbl. 1. As far as I see, this is because it is switching frames between two cameras. If the captures are reordered in a way that uses frames from camera 1 first and camera 2 second (or only frames from camera 1), will Nerfies still work? If so, it will have a much smaller velocity, thus less EMF. It would be interesting to see how the method performs vs. EMF.\n\n- No view direction or appearance encoding\n\nIt is mentioned in L297 both are turned off during training. But the goal of PSNR_M is to evaluate the quality of NVS in seen region. In practice, view direction and appearance encoding are essential in synthesizing new views. Turning them off will hurt the PSNR_M score a lot. Is there a specific reason doing so aside from overfitting?\n\n- Typo\nL32 agnitude -> magnitude", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "H5_3FlzWouL", "reviewer": "Reviewer_A6gt", "summary": "The paper does a complete review for existing approaches recovering dynamic 3D scenes from monocular videos, especially Nerfies, HyperNeRF and NSFF. It studies the camera trajectory, proposes a metric Effective Multiview Factors (EMF) to quantify multi-view cues in a dynamic scene with moving cameras. Existing datasets typically have high multi-view cues. Therefore, a new dataset captured by iPhone is introduced with little multi-view cue. Additional metrics such as Masked PSNR and PCK-T are also introduced to ensure the fairness of evaluation. The new benchmark brings additional challenges for existing approaches of dynamic 3D capture.", "questions": "> (L213 - 215) Each training sequence is annotated with 10 to 20 keypoints in every 10 frames\n\nHow do you select these keypoints?\n", "limitations": "Limitations are not discussed in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Quantifying multi-view cues using EMF is neat. Different datasets in dynamic 3D capture proposes different data including different camera trajectories. And EMF quatifies the difficulties of all the data.\n- I also like PCK-T correspondence besides normal PSNR. Right now a lot of novel view synthesis approaches focus on PSNR but PSNR does not directly reflect how well the model understands the 3D world. PCK-T is more explicit and I think reporting both masked PSNR and PCK-T is a good idea.\n- Supp webpage is fantastic. Thank you!\n\nWeaknesses:\n\nI don't see any weaknesses, although I think the paper is more like a new benchmark -- It might be more suitable for the benchmark track.\n\nAdditional comments (no need to address):\n\n> Page 8 footnote: We find that this code base performs better than the original code release\n\nmissing a period.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept