Improving the Convergence of Dynamic NeRFs via Optimal Transport
Authors: Sameera Ramasinghe
Violetta Shevchenko
Gil Avraham
Hisham Husain
Anton van den Hengel
Venue: ICLR 2024 poster (2024)
Keywords: Dynamic NeRF
Optimal transport
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Synthesizing novel views for dynamic scenes from a collection of RGB inputs poses significant challenges due to the inherent under-constrained nature of the problem. To mitigate this ill-posedness, practitioners in the field of neural radiance fields (NeRF) often resort to the adoption of intricate geometric regularization techniques, including scene flow, depth estimation, or learned perceptual similarity. While these geometric cues have demonstrated their effectiveness, their incorporation leads to evaluation of computationally expensive off-the-shelf models, introducing substantial computational overhead into the pipeline. Moreover, seamlessly integrating such modules into diverse dynamic NeRF models can be a non-trivial task, hindering their utilization in an architecture-agnostic manner. In this paper, we propose a theoretically grounded, lightweight regularizer by treating the dynamics of a time-varying scene as a low-frequency change of a probability distribution of the light intensity. We constrain the dynamics of this distribution using optimal transport (OT) and provide error bounds under reasonable assumptions. Our regularization is learning-free, architecture agnostic, and can be implemented with just a few lines of code. Finally, we demonstrate the practical efficacy of our regularizer across state-of-the-art architectures.

Published as a conference paper at ICLR 2024
IMPROVING THE CONVERGENCE OF DYNAMIC NERFS VIA OPTIMAL TRANSPORT
Sameera Ramasinghe', Violetta Shevchenko', Gil Avraham, Hisham Husain, Anton van den Hengel Amazon, Australia
ABSTRACT
Synthesizing novel views for dynamic scenes from collection of RGB inputs poses significant challenges due to the inherent umder-constrained nature the problem. To mitigate this ill- ill-posedness, practitioners in the field of neural ra- diance fields (NeRF) often resort to the adoption of intricate geometric regular- ization techniques, including scene flow, depth estimation, learned perceptual similarity. While these geometric cues have demonstrated their effectiieneess, their incorporation leads evaluation of computationally expensive off-the-shelff mod- els, introducing substantial computational overhead into the pipeline. Moreover, seamlessly integrating such modules into diverse dynamic NeRF models can be a non-trivial task, hindering their utilization in an architecture-agnue this paper, we propose theoretically grounded, lightweight regularizer by treating the dynamics of time-varying scene as low-frequency change of a probability distribution of the light intensity. We constrain the dynamics of this distribution using optimal transport (OT) and provide error bounds under reason- able assumptions. Our regularization is learning- free, architecture agnostic, and can be implemented with just few lines of code. Finally, demonstrate the practical efficacy of our regularizer across state of-tth--rtttttt architectures. Our code is at htttps://ggii http://////ogggggoooo///////////
manner.
com/
INTRODUCTION
Synthesizing novel views of a dynamic scene, given a sequence of monocular frames (Park et al, 2010 Gao et al.) 2022) set of iimessyncroonized cameras (Li et al. 2022). has recently gained significant traction in the field (Brickwedde et al. 2019b] Park et al. 2222 a Pumarola et al., 2021 Cai et al. 2022 Li al. 2023a). The concept of novel-view synthesis originates from the realm of image-based rendering (IBR) (Gortler et al. 1996). where the objective is to accurately aggregate light intensity information from a dense collection of images to generate desired novel view. One of the most noteworthy recent developments in deep learning for IBR is NeRF (Mildenhall et al., 2027], which can replicate the capabilities of a meticulously designed array of cameras using state- of-the-art Structure from Motion (SfM) algorithms (Schonberger & Frahm 2016).
Despite NeRF's impressive results and robust performance, numerous recent works have embarked on the challenging task of extending NeRF to dynamic scenes (Pumarola et al.) 2021 Li et al.) 2023b. Paar et al. 2021b Fang et al. 2022 Ramasinghe et al.) 2023). However, transitioning from static dynamic settings presents non-trivia and challenging problem replete with complexities (Deng et al. 2022). Notably, disentangling camera motion from object motion inherently constitutes an ill-posed problem. further complicated by variations in lighting, non-rigid object deformations, and their interplay. To address these additional complexities, explicit regularizations are imperative to constrain the problem. Most existing approaches that employ such regularizations rely geomet- ric constraints (e.g., depth losses, flow-based losses) or learned priors (e.g., maintaining consistent perceptual similarity across rendered frames over time) guide dynamic NeRFs toward plausible solutions.
However, these regularization techniques often entail adding additional deep networks on top of already bulky NeRF models or involve expensive preprocessing steps. For instance, perceptual
Authorr contrihuted equally.

---

Published as a conference paper at ICLR 2024
regularizers require backpropagation through pre-trained deep networks, while depth-based losses demand capturing depth information (e.g., using LIDAR) or running off-the-shelf depth estimation models during data preprocessing. Flow-based losses similarly necessitate extra preprocessing steps involving external models. Furthermore, these regularizers enforce priors that are averaged over large datasets, which may lead to domain gaps when applied to specific scenes with unique geome- tries. Errors in estimating geometric cues can directly propagate from external models to the NeRF optimization process.
In contrast, we introduce a straightforward plug-in regularizer that leverages statistical regularities of rendered frames from fixed camera views to constrain optimization for unseen views during training. Our central hypothesis is that the pixel intensity distribution of a scene, as rendered from a specific fixed camera view, should remain approximately consistent within short time intervals. However, due to object motion, pixel coordinates can change over time, making it challenging to minimize a pixel-to-pun ni-eexee distance function between two rendered frames. To address this limitation, we pro- pose to minimize a dissimilarity measure between pixel intensity distributions to maintain statistical regularity. One effective approach, in this vein, is to use a geometric divergence metric between distributions. Two primary candidates include optimal transport (OT) based metries and maximum mean discrepancy (MMD), each with its own merits and downsides.
OT-based metries exploit the geometry of the underlying ground space, making them more favourable over MMD. However, there are two key challenges: 1) solving an OT problem is gen- erally of O(n²) complexity, making it impractical to compute in each iteration, whereas MMD is computationally efficient, since it can be expressed as a finite-dimensional inner product; and 2) ren- dering the entire pixel distribution of a particular frame at each iteration consumes significant mem- ory in the NeRF framework. To address the former, we employ a which reduces OT computation complexity O(n log n), significantly enhancing computational ef- ficiency even compared to MMD. To overcome the latter, we obtain an unbiased estimator of the full pixel distribution and further refine the estimator with interpolation for unsampled pixel coordinates. Our experiments demonstrate that the sliced- Wasserstein approximator offers both faster computa- tion and superior performance compared to MMD variants. We also show that the Wasserstein dis- tance in our setting converges at competitive rates of O (1/2) with poly-logarithmic dependence on the number of observed values in the image. Importantly, our regularizer can be seamlessly integrated into any dynamic NeRF architecture without altering the existing model. Our method assumes no prior knowledge about the scene and adapts dynamically at training. Empirically, we demonstrate that the proposed regularizer enhances the performance of various dynamic NeRFs. Moreover, ablation studies illustrate that our simple regularizer surpasses existing approaches that rely on resource-intensive deep models or costly preprocessing steps.
approximator,
2 RELATED WORK
D-NeRF Pumaaola al. 2021), demonstrated that in the context of dynamic object-centric scene, a canonical representation assumption could be made. This assumption allows for the optimization of implicit motion and ray interactions with respect to reference frame while treating neighboring frames as small deformations. This paradigm involves the coordination of two Multilayer Percep- trons (MLPs): the first one is responsible for deforming rays from the required view the canonical view, essentially inferring the delta ray bend; the second MLP applies the delta ray-bending from the canonical reference frame the desired view. The ray-bending paradigm with respect canonical frame is effective, but, inherently under-constrained. Works by [Tretschk et al. (2021) Park et al.) (2021a). and Park et al. (202Tb) introduced several regularizations to constrain the al- lowed ray deformations. NR-NeRF by TTettchk et al. (2021) regularized the magnitude of the ray offsets, effectively limiting their deformation to reasonable changes, assuming that the motion is not abrupt. However, due to the absence of explicit geometry, this regularization is applied to the entire deformation volume, including occluded and unseen areas, requiring specific scheduling to stabilize. Similarly, Park et al. (2021a) employed elastic regularization, which enforces constraints on the singular values of the Jacobian of the deformation MLP, driving them towards zero. By doing so, they achieve uniform scale of changes in the deformation directions.
Nevertheless, it was observed that elastic regularization faced challenges in scenarios with complex topological deformations. HyperNeRF (Park et al. 2021b), addressed discontinuities in the defor-
2

---

Published as a conference paper at ICLR 2024
mation field, particularly when changes in surface connectivity occur (e.g., opening mouth, cutting a lemon) or transient objects appear (c.g., fire or flame). To address this, HyperNeRF models frames as the zero-level set of digheer-dmensional ambient space. Consequently, to infer novel views in practice, HyperNeRR requires accurate slicing of the ambient space and obtaining a latent repre- sentation, which is then used to model the ray-deformation. Similar NR-NeRF, this approach deals with complexity through prerterramttiiaaiion, necessitating careful scheduling during opti- mization. Drawing inspiration from image-Based Rendering (IBR), researchers have also explored the use of explicit geometry, such as monocular depth estimation (Ranfil et al. (202T) and scene flow (Brickwedde et al. 2019a), as active regularization techniques for the deyoorooootiion operation.
3
METHODOL.OGY
Sec.
3.1
presents a brief exposition of statistical divergence metrics and motivates our approach. delves deeper into the proposed regularizer.
Sec. 3.2
3.1
STATISTICAL DIVERGENCES
Let Σ denote the standard Borel o-algebra on Re and P(RT))) to denote the set of Borel probability measures (positively signed Borel measures with each μ E ((RR)) satisfying (( = 1. When comes to comparing two probability distributions, there are plethora of different options, referred to as divergences. A classic choice is the f-divergence (or the Czisar divergence (Csiszár) 1975)) which for two probability distributions v, P(R") is defined as Iy( : 2) fge fldu/ dv dd if is absolutely continuous with respect to and do otherwise where [ (-00, 00 - TR is convex lower semicontinuous function such that f(1) = O. In particular, this divergence is a generalization of the well-known Kullback-Leibler (KL) divergence for the choice of f(t = llo t. Despite the widespread use of f-divergences, they are computationally intractable in the setting when only samples are available and require absolute continuity between μ and v which often surfaces stringent requirement.
Natural remedies this issue is to consider Integral Probability Metrics (IPM), which is the motiva- tion of works in other domains such as generative models (Mroueh & Sercu 2017) and distributional
robustness (Staib & Jegelka| 2019 Husain (2020). There are two prevalent choices of IPMs that are
used: the (kernel) Maximum Mcan Discrepancy (MMD) (Gretton et al. (012) and the p- p-Wasserstein
distance (Villami et al., 2009). The MMD typically favoured for its computational tractability as
it can be computed in (n²) time where n is the number of samples however relies on an appro- priate choice of characteristic kernel function. On the other hand the p- p-Wasserstein distance utilizes ground metric on the support of the distributions and characterizes divergence at the statistical level. More formally, the p- Wassersttti distance between two probability measures μ1, v E P((Rd) is
f ||I yl (1)
W p (2) = inf infven(e.)) RdxRd
(1)
where I[(H,)) v)) is the set of all couplings: joint probability measures on Fd x Rd with marginals and v, and ||| ||| is the Fuclidean norm. The Wasserstein distances solve for a mass transporta- tion problem by finding the minimal coupling and as such, alleviate issues with f-divergence such as non-overlapping support. However Wasserstein distances remain analytically intractable for all but a few cases. One straightforward way solve the discrete OT problems is to use linear programming based algorithms such as the Hungarian method Kuhn (1955), the auction algorithm Bertsekas (1988) and the network simplex Waissi (1994), which are typically numerically robust. Unfortunately, these methods pay large price in algorithmic complexity, especially the memory requirements for solving larger problems. However, in the 1-dimensional setting, we can com- pute the p- y~Wasserstein distance in O(n logn) n) time. Consider two finitely support distributions: A = 1 Σn= 82,, D = 1 _i_ 6y. where 24, Di E IR where d. (z') is the dirac delta function that corresponds to 1 if 2' = and 0 otherwise. In this case, the p- Wasserstein distance can be computed with (B, 2)P = 1 17, 1 -il". where (ā,) and {ū,} are sorted variants of (x;) and {
n
such that 31 .... < In and 01 < ... < Gn- This problem can be solved at O log n)) cost, which is the sorting cost. This property of the Wasserstein distance has been heavily leveraged in many practical applications, using the Sliced- Wasserstein (SW) distance Rabin et al. (2012); Bon-
neel et al. (2155): Avraham et al. (2019) More specifically, Sliced Wasserstein distance provides
3

---

Published as a conference paper at ICLR 2024
practical univariate approximation to the multi-dimensiona. case, with robust theoretical guarantees. Let gdt be the d-dimensional unit sphere and y the uniform distribution on the surface of gd For 8 E gd-1, 8 Rd + R denotes the linear form + (x, 0) with (:,-) the Euclidean inner-product. Then, SW of order is
(WI, (2) = / Wp Wp 11,00,00111100 (2)))
(2)
gdd-
where for any measurable function f Rd + +R and P(R4) ) is the push-forward measure of by f: for any Borel set A the o-algebra of R. (=m(A) = n (()-)), where (A) = (r E Rd [(x) E A}.
In this work, we treat images as distributions with finite support. For a grayscale image R RwxH* xHx1 of size W x H. let Ω C R denote the set of observed values in the true image. We use P((r)) to denote the distribution from images whose observed values are {r}. Then, we present the following theorem.
Theorem For any two sets of samples P({+')) and ((f(')) drawn i.i from t base distributions P((r")) and P( P((r'a) supported on finite set of values Q, the following holds
|0/2
[22
Wi (P((E") (,(( S <4V
log
(3)
2n
with probability at least
(Proof in Appendix). It should be noted here that the rate convergence depends on the size of [S2]. We can see from above that the I-Wasserstein distance from empirical samples at two time-stamps converges competitively. In practice we observe that the performance increases with n (Table 7 possibly due to the lower error in OT computation predicted by the above result.
La Sittneo
Lap
MND
Sinkhcen
An example illustration of different divergence metries over temporarily varying pixel dis- tributions captured from fixed camera. Top row: An image sequence extracted from the space-oud sequence of the iPhone dataxet. Bottom row: The heatmaps indicate the metric distance between the corre- sponding images. As shown, pixel-to-pixel L2 loss does not reasonably capture the similarity between images as the objects in the scene are dynamic. In comparison, geometric distances better capture the similarity of pixel distributions, where sliced- Wasserstein distance performs best (zoom in for better view). We notice that this slight enhancement results a notable boost performance within the context of the more intricate dynamic NeRF scenario, potentially due to ability to effectively handle subtle variations in lighting conditions.
Figure 1:
3.2 OPTIMAL TRANSPORT FOR IMPROVING THE CONVERGENCE OF DYNAMIC NERFs
We consider the scenario in which the provided data consists solely of a series of snapshots and corresponding camera poses of dynamic scene, captured by single moving camera. This problem highly underconstrained, as it provides only one observation from a specific camera pose at any given timestamp. Consequently, the NeRF model potentially converge to degenerate solutions, violating multi-view consistency, unless explicit regularization measures are imposed.
Notably, observe that for scene with reasonably smooth temporal dynamics, the pixel intensity distributions when rendered from a fixed camera pose should remain similar within a short time interval. However, the pixel coordinates change over time due object movements. For example,
4

---

Published as a conference paper at ICLR 2024
consider the pixel distribution p(r of rendered image from camera pose p at time where Fi,p represents RGB intensities. Within a small time interval we assume At, P(r(p) = P(rr+arpp) Therefore, the minimization of (),(rt,)) P(rt+Ar.p)). where -(..)) is a suitable divergence metric that remains invariant with respect to pixel coordinates, should assist the model in converging to more robust solution. In fact, we have found that in real-world scenes, this approach effectively regularizes the training of dynamic NeRFs on-the-fly, resulting in improved 3D reconstructions.
Fig. D is a toy example that illustrates the validity of this hypothesis. Note that as the person moves, the L2 distance rapidly diverges from zero the indices move away from the diagonal of the heatmap. In comparison, geometric losses are able to maintain a low metric distance across larger area. We observe that the better performance of Bice-Wasserseen metric translates to a sig- nificant performance improvement in the more complex dynamic NeRF setting, possibly owing to its robustness to subtle light changes (see Sec. 4). Interestingly, we observe that sliced- Wasserstein distance performs better even compared to the Sinkhorn divergence which is an efficient approxi- mation the Wasserstein distance and can be considered as an interpolation between OT and MMD (Feydy et al. 2019). This particular behaviour allows to enjoy both faster estimations (O log n) as opposed O(n)) complexity of MMD) and better convergence with sliced- Wasserstein distance.
One critical obstacle that impedes the regularization of dynamic NeRFs using rendered images as inputs is their computational complexity. Rendering a complete projection given camera angle involves tracing a ray for each pixel and densely sampling the density and color fields in 3D space. This necessitates the exhaustive evaluation of a NeRF backbone (typically an MLP a dense feature grid) for each rendering, often demanding compute power that is infeasible. Consequently, methods utilizing image statistics for regularization are compelled to resort to using small patch renderings for this purpose, which frequently provide inadequate approximations of the global statistics of the entire projection. As these regularization methods require well-structured pixel information as imputs, rendering sparse pixel samples to obtain better global statistics is not a viable option (c.g., calculating LPIPS similarity between randomly sampled pixels from images lacks meaning). In contrast, our regularization method based on reducing the OT cost between pixel distributions, and it is not hindered by the aforementioned obstacle. Instead, we can obtain an unbiased estimator of the true population by rendering pixel values at random coordinates, enabling us to efficiently regularize the scene with theoretical guarantees and manageable computational re- quirements. Fig. illustrates our method graphically.
3D scene from a
Further, observed that interpolating to unsampled pixel coordinates leads to better performance. Therefore, we interpolate to the full image resolution after rendering random pixels before computing the distance. It is important to note that this only minimally affects the computational overhead since we are not actually rendering the missing pixels. We conducted experiments with more complex interpolation kernels: however, bilinear interpolation with soft smoothing yielded the best results. For an ablation, please see Table Our method is outlined in Algorithm
D1
Figure graphical demonstration of our approach. We min- imize the OT distance D between the pixel intensity distributions P(rt.p) and P(rr+ where Ft,p is the set of pixels rendered from camera p time t.
See Sec. 5 for limitations.
5

---

Published as a conference paper at ICLR 2024
Algorithm The proposed regularization loss for each optimization iteration.
Input:
(ri "==a Training camera poses, At Time interval, W Width of rendered images, H Height of rendered images, B Scalar loss weight, Number random used for sliced-We Wasserstein distance, d Number pixels sampled for compating sliced Wassersennntnnnn distance, Lphoto Thee usual photometric loss of dynamic NeRF
Output:
Lental - 'The result the algorithm.
1: Randomly sample camera pose: P;
2: Sample f1 (0, and (0, Ar)
3: Calculate 22 =
4: Sample 2D coordinates: ((r) yn (rr, pa)...., (FF ya)) (10 W) U(0
5: Render pixels at and fa from p: sampled courdimates: R1,
Ry E Rdxa > n, E E
6: Sample vectors: u1j ug...... ~ (7( 1)a
7: Normalize vectors: ||u|||
8: Create matrix: U [U,; U2i ...::..:0 [up]
gkxa
9: Compute sliced Wasserstein distance:
U_r U. reshape (a, 1, 3)
R1_r = R1 Re.reshape (1, d, 3)
R2_r R2. R2.reshape (1, d, 3)
3
P (0_r Rl_r) sum (dim 2)
= ( r2_r) sum (dim 2)
P, toren. sort (p, dim=1)
q torch sort (q, dim-1)
10
RegLoss torch. mean (torch. abs (p - -q) / (1 + (p - q) ** )
10: Irroial Phooto
LRgg
Caleulate total loss
11: Return Lootat
EXPERIMENTS
In this section, we empirically assess the effectiveness of the proposed regularization method by integrating it into several recent dynamic NeRF models. We used B 0.1. Although augmenting At per scene leads to improved results, we opted to fix to 0.1 across all scenes and datasets to better demonstrate the robustness of our method. We used 256 or 512 for and 2048 or 4096 for d, depending on the model size.
4.1 DATASETS AND BASELINES
We use the iPhone dataset proposed by Gao et al. (2022), HyperNeRF interpolation dataset, and the HyperNeRF vrig dataset (see Appendix) proposed by Park et al. (2021b) for evaluation. We choose
DNeRF (Gao et al. 2021 HyperNeRF (Park et al. 2021b). Nerfies (Park et al. (021a). TiNeuVox
Fang et al. 2022). and Hexplanes (Cao & Johnson
2023 baselines.
4.2
EVALUATION ON THE IPHONE DATASET
Many datasets used in dynamic NeRF setting do not represent practical in-the-wild capture. For instance, the datasets contain limited motions and the test sets are small. hiding issues in incorrect deformation and resulting geometry. For these reasons, Gao et al. (2022) proposed new dataset dubbed the iPhone dataset. It consists of sequences with non- repetitieee motion, from various cat- egories such as generic objects, humans, and pets. The dataset deploys three cameras for multi- camera capture one hand-held moving camera for training and two static cameras for evaluation. Table and Fig. depict quantitative and qualitative improvements by our regularization, respec- tively. We use the masked metrics proposed by the dataset that use covisibility masks.
4.3 EVALUATION ON THE HYPERNERF INTERPOLATION DATASET
HyperNeRF interpolation dataset contains six sequences with densely captured images. To make the task more challenging, we train all the models by taking only every fourth frame for training. For evaluation, fix the first camera pose, and render frames from all the training time stamps. This setting measures how robustly the models have captured the scene dynamics when deviating from the ground truth pose, time pairs. Since do not have the ground truth images for this eval-

---

Published as a conference paper at ICLR 2024
Apple
Block
Paper windmill MSSIM mLPIPS.
Method
mPSNR1
MSSIM1 mLPIPS} mPSNR1 MSSIM mLPIPS!
mPSNR
DNeRF
12.635
0.671
0.669
14.628
0.330
0.375
DNcRF w/ Reg
13.454
0.679
0.649
16.383
0,381
0.351
Nerfies Nerfies w/ Reg
13.853
0.709
0,570
13.860
0.572
0.523
12.817
0.223
0.570
17.719
0.762
0.480
14.398
0.583
0.527
16.854
0.342
0.273
HyperNeRF
13.440
0.704
0.594
14.535
0.593
0.502
13.628
0.256
0.499
HyperNeRF w/ Reg
18.056
0.757
0.464
16,562
0.639
0,404
16.946
0.337
0.265
HexPlane
16.606
0.712
0.620
15.888
0.622
0.542
16.989
0.356
0,516
HexPlane w/ Reg
17.189
0.718
0.633
15.893
0.625
0.537
17.144
0.365
0,512
TiNeuVox
9.259
0.682
0.704
9.793
0.607
0.502
11.486
0.289
0.359
TiNeu Vox w/ Reg
12.561
0.716
0.596
11.114
0.625
0.481
12,233
0.317
0.349
Space out
Spin
Teddy
Method
mPSNR1
mSSIM1 mLPIPS
mPSNR1
MSSIM1
mLPIPSJ mPSNR MSSIM mLPIPS.
DNeRF
16.740
0.606
0.410
16.384
0.502
0.451
DNeRF w/ Reg
17.080
0.628
0.363
15.825
0.484
0.506
Nerfies Nerfies w/ Reg
15.709
0.571
0.397
13,900
0.471
0.505
13.073
0.542
0.442
16.545
0.607
0.349
13.324
0.485
0.547
14.147
0.556
0.417
HyperNeRE
16,000
0.560
0.422
14.441
0.479
0.497
13.382
0.546
0.438
HyperNeRF w/ Reg
17.090
0,613
0.337
15.314
0.502
0,486
14.361
0.565
0.389
HexPlane
17.081
0.581
0.540
16.125
0.493
0.563
13.051
0.517
0.627
HexPlane w/ Reg
17.197
0.593
0.560
16.170
0.495
0.558
13.126
0,519
0.629
TiNeu TiNeu Vox w/ Reg
11.377
0.610
0.422
10.179
0.473
0.616
8,219
0.543
0,551
12.372
0.627
0.411
14.799
0.635
0.531
9.161
0.562
0.514
Wheel
Mcan
Method
mPSNR MSSIM mLPIPS mPSNR1
MSSIM mLPIPS}
DNcRF
15.097
0.527
0,476
DNeRF w/ Reg
15.685
0.543
0.467
Nerfies Nerfies w/ Reg
10.109
0.333
0.511
13.617
0.489
0.503
11.308
0.392
0.381
14.899
0.532
0.425
HyperNeRF
9.741
0.314
0.537
13.881
0.493
0.498
HyperNeRF w/ Reg
11.863
0.389
0.386
15.742
0.543
0.390
HexPlane HexPlane w/ Reg
13.044
0.407
0.577
15.541
0.527
0.569
13.228
0.434
0.564
15.707
0.536
0.570
TiNeuVox
5.801
0.330
0.604
9.444
0.504
0.536
ToNeuVox w/ Reg
5.995
0.335
0.590
11.176
0.545
0.496
Table 1: Evaluation of the proposed regularizer over the iPhone dataset. The proposed regular- izer improves the performance of all the baselines across all scenes. We were not able to converge DNeRF on some sequences.
Chickchicken SSIM1
Torchocolate SSIM
Aleks-teapor
Method
LPIPS
LPIPS
SSIM
LPIPS
TiNeu
0.723
0.551
0.844 0.859
0.384 0.377
0.770
0.486
TiNeu Vox w/ Reg
0.796
0.381
0.834
0.441
Lemon SSIM
Hand
Mean
Method
LPIPS4
SSIM
mLPIPSL
SSIM
LPIPSJ
TiNeu Vox TiNeu Vox w/ Reg
0.816 0.826
0.662
0.718 0.824
0.667 0.650
0.774
0.550
0.648
0.872
0.499
Table 2: Evaluation of the proposed regularizer over the HyperNeRF interpolation dataset.
7

---

Published as a conference paper at ICLR 2024
Regularization
mPSNRT
mSSIMI
mLPIPS.
TiNeuVox (baseline)
9.444
0.504
Scene-flow
9.457
0.514
0.336
Kernel
mPSNR
MSSIM
LPIPS
9,991
0.521
0.512
Depih
9,477
0.319
0.518
Spline
10.113
0.441
0.599
RB
9.332
0.518
0.518
Inverse quadratic
10.441
0.423
0.501
0.523
Gaussian
11.001
0.521
0.622
RB SR Depth
10.311
0.535
0.508
Cubic
9,993
0.425
0.513
Quintic 10.661
9.336
0.415
0.581
RB SR Depth Ours
11.221
0.565
0.490
Multiquadric
0.445
0,522
Linear
11.176
0.545
0.496
Table 4: Comparison over different regulariza- tions. Depth and Scene-flow regularizations include expensive preprocessing steps that involve evaluating off the-shelf models. LPIPS regularization requires stacking pre-trained deep network on NeRF train- ing time. In contrast, regularization is more cffi- cient and performs better.
Table 3: Comparison different interpolation
kernels. A linear kernel performed best.
Apple
Block
Pupaor- milmmmml
Apppa
GT
HyperNeKI
HyperNest Reg
TiNeuVox Reg
Figure 3: A qualitative illustration of the effect of the proposed regularization. As illustrated, our regular- izer significantly stabilizes the 3D reconstruction task.
uation setting, we measure perceptual similarity between the ground truth image at time zero and the rendered images different time stamps where the camera pose fixed at the first ground truth pose. The quantitative and qualitative results are shown in Table and Fig. 4 respectively. Inter- estingly, we observed that HyperNeRF and Nerfies baselines already performed well in this setting and our regularization did not have much effect. In contrast, TiNeu Vox demonstrated difficulty in converging to plausible solution and our regularizer significantly improved the results.
4.4 ABLATION AGAINST OTHER REGULARIZATIONS AND DESIGN CHOICES
8

---

Published as a conference paper at ICLR 2024
T.Nastox Eivastee voi
1ivestte
Whe Ves
Theerroo Aog
Figure 4: A qualitative illustration of the effect of the proposed regularization the HyperNeRF interpolation dataset.
We compare the efficacy of the proposed method against other regularizations. Results are shown Table For depth regulariza- tion, we use the LIDAR-based depth maps pro- vided by the iPhone dataset for computing the photometric eтror between the rendered depth and ground-truth depths. For LPIPS regulariza- tion, minimize the LPIPS distance between RGB patches rendered from a fixed camera, in different time stamps (similar our regular- ization, but LPIPS is used instead of the OT distance). We used a pre-trained AlexNet for computing LPIPS. For scene- scene-flow, we use the method proposed in Wang al. (2023) which uses RAFT (Ted & Deng 2020) for estimat- ing optical flow. We also compared against
Method
MSSIMf mPSNR
mLPIPS
Baseline
9.444
0,504
0.536
8.313
0.407
0.697
KL-divergence
9.414
0.461
0.570
Total variation norm
9.471
0,426
0.597
Caussian-MMM
10.011
0.523
0.511
Laplacian-Min -MMD
10.114
0.522
0,500
Energy-MMa
10.746
0.513
0.521
Sinkhorn
10.888
0.533
0.498
Sliced Wasserstein
11.176
0.545
0.496
Table 5: Comparison
different distant met- rics used with the proposed regularization. Sliced- Wasserstein yields the hest results.
sparcity regularizer [Barron et al. (2022) and random background regularizer Weng et al. (2022) which has been previously been used
literature. Our regularization outperforms the above methods that either additional devices, off-the-shelf models, deep networks. Table 5 and Table depict comparison against other distance metrics and interpolation kernels. Interestingly, sliced- Wasserstein outperformed all other metrics by significant margin. We TiNeu on the iPhone dataset for these experiments.
LIMITATIONS
Our method assumes smooth motions and can fail with abrupt scene dynamics. However, note that in most real world scenes, this is a fair assumption. Additionally, due to our method's 's reliance on an approximate OT distance, a pixel averaging effect blur. Interestingly, this effect can potentially lead to a minor degradation in renderings, particularly when the baseline model already performs well in given sequence. Moreover, if the baseline model fails to converge entirely in a specific sequence, our regularization method may not yield significant improvements in the results.
occasionally occur, resulting in slight
CONCLUSION
We propose an architecture-agnosee simple regularizer that can be easily integrated into dynamic NeRF models. We do not rely on deep architectures or expensive learned priors across a large dataset. We leverage optimal transport (OT) to learn instance-based statistical priors on-the-fly during training. To circumvent the need to solve an OT problem in each iteration, we employ a sliced- Wasserstein approximation and derive theoretical bounds for to the convergence with respect to sampling complexity. We show the effectiveness our regularizer by evaluating it across chal- lenging real-world dynamic scenes.

---

Published as a conference paper at ICLR 2024
REFERENCES
Gil Avraham, Yan Zuo, and Tom Drummond. Parallel optimal transport gan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4411-4420, 2019.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5470-5479, 2022.
Dimitri P Bertsekas. The auction algorithm: A distributed relaxation method for the assignment problem. Annals of operations research, 14(1):105-123, 1988.
Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and radon wasserstein barycenters of measures. Journal of Mathemutical Imaging and Vision, 51:22-45, 2015.
Fabian Brickwedde, Steffen Abraham, and Rudolf Mester. Mono-sf: Multi-view geometry meets single-view depth for monocular scene flow estimation of dynamic traffic scenes. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2780-2790, 2019a.
Ferdinand Brickwedde, Benjamin Stammen, Christof Meyer, Carlo Curio, and Konrad Schindler. Monoloco: Monocular 3d pedestrian localization and uncertainty estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision 2019b.
Hongrui Cai, Wanquan Feng, Xuetao Feng, Yan Wang, and Juyong Zhang. Neural surface re- construction of dynamic scenes with monocular rgb-d camera. Advances in Neural Information Processing Systems, 35:967-981, 2022.
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. CVPR, 2023.
Imre Csiszár. I-divergence geometry of probability distributions and minimization problems. The annals of probability, pp. 146-158, 1975.
Bailin Deng, Yuxin Yao, Roberto M Dyke, and Juyong Zhang. A survey of non-rigid 3d registration. In Computer Graphics Forum, volume 41, pp. 559-589. Wiley Online Library, 2022.
Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPHI Asia 2022 Conference Papers, 2022.
Jean Feydy, Thibault Séjourné, François-Xavier Vialard, Shun-ichi Amari, Alain Trouvé, and Gabriel Peyré. Interpolating between optimal transport and mmd using sinkhorn divergences. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2681-2699 PMLR, 2019.
Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. Pp15712-555 5712-5721, 2021.
Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dy- namic view synthesis: A reality check. Advances in Neural Information Processing Systems, 35: 33768-33780, 2022.
Steven J Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F Cohen. The lumigraph. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pp.
43 43-54, 1996.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Hisham Husain. Distributional robustness with ipms and links to regularization and gans. Advances in Neural Information Processing Systems, 33:11816- 11827, 2020.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
10

---

Published as a conference paper at ICLR 2024
Rui Li, Dong Gong, Wei Yin, Hao Chen, Yu Zhu, Kaixuan Wang, Xiaozhi Chen, Jinqiu Sun, and Yanning Zhang. Learning to fuse monocular and multi- cues for mulli--rame depth estimation in dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 21539-21545, 2023a.
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 5521-5531 2022.
Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neu- ral dynamic image-based rendering. In Proceedings of the IEEE/CVE Conference on Computer Vision and Paltern Recognition, pp. 4273-4284, 2023b.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99 106, 2021.
Youssef Mrouch and Tom Sercu. Fisher gan. Advances in neural information processing systems, 30, 2017.
Hyun Soo Park, Takaaki Shiratori, lain Matthews, and Yaser Sheikh. 3d reconstruction of a moving point from a series of 2d projections. In Computer Vision-ECCV 2010: 11th European Confer- ence on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part III 11, pp. 158-171. Springer, 2010.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan Goldman, Steven M Seitz, and Ricardo the IEEE/CVF International Conference on Computer Vision, pp. 5865-5874, 2021a.
Nerfies: Deformable neural radiance fields. In Proceedings of
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Gold- man, Ricardo Marrinn-ruulli and Steven M. Seitz. Hypernerf: A digher-dimensional representa- tion for topologically varying neural radiance fields. ACM Trans. Graph., 40(6), dec 2021b.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10318-10327, 2021.
Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bernot. Wasserstein barycenter and its appli- cation to texture mixing. In Scale Space and Variational Methods in Computer Vision: Third International Conference, SSVM 2011, Ein-Gedi, Israel, May 29-June 2, 2011, Revised Selected Papers 3, pp. 435-446. Springer, 2012.
Sameera Ramasinghe, Violetta Shevchenko, Gil Avraham, and Anton Van Den Hengel. Bali-rf: Bandlimited radiance fields for dynamic scene modeling. arXiv preprint 1xXx::3302.135:: 2023.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference computer vision, pp. 12179 121888 2021.
Johannes L Schonberger and Jan-Michael Frahm. Structure-from-mottie revisited. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4104-4113 2016.
Matthew Staib and Stefanie Jegelka. Distributionally robust optimization and generalization in ker- nel methods. Advances in Neural Information Processing Systems, 32, 2019.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer AisionEECCT 2020: 16th European Conference, Glasgow, UK, August 23-28 2020, Proceedings, Part 11 16. pp. 402-419. Springer, 2020.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Con- ference Computer Vision, pp. 12959-12970, 2021.

---

Published as a conference paper at ICLR 2024
Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2009.
Gary R Waissi. Network flows: Theory, algorithms, and applications, 1994.
Chaoyang Wang. Lachlan Ewen MacDonald, László A. Jeni, and Simon Lucey. Flow supervision for deformable nerf. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21128-21137, June 2023.
Chung-Ya Yi Weng. Brian Curless, Pratul P Srinivasan, Jonathan Barron, and Kemelmacher- Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF conference on computer vision and pattern Recognition, pp- 16210-16220, 2022.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 PROOF OF THEOREM 1
In order prove our result, introduce some additional notation. For any probability measure E ((R),, the support of the measure is
(4) Furthermore, for any set A R, we use La(x) if E A and 0 otherwise to denote the characteristic function of the set A. Furthermore, we use [+] to denote the Iverson bracket of an event: [A] if A is true and otherwise. We will require the of Lemma that will aid in
supp(r)
if E Nx open = P(Nx)>00
the main proof
Lemma For any measure E P([O, 1]) with finite support: |supp(u)] < 0o, let An = 82, where y i.idd then have
lu(x)) - An(x)) S
log
(5)
2n
for a fixed E IR with probability ô.
Proof Note that uxx) = (z)] and (x) E [0, Thus by standard concentration inequality
such as Hoeffding's s inequality, we get
IP (x) μ(x)] 2 s 2 exp (-2nt²).
Setting & =
1. log(2/6) completes the proof. 2n
V
Lemma's For any measure E P((O, 1)) with finite support: E 8z where Fs o I i.idd, then we have
[supp(u)]
< co, let Pr.
supp(())
(2 (2-|sun) [-(sup((u))
(i( S
log
2n
with probability at least
Proof We first invoke the Centtroviic-Ruubessee dual of the Wasserstein distance (Villani al.
2009 :
Wi(u, Pm)
(=) (E Ep, [h]
sup
(8)
sup
(E [4] - Ea, [h])
(9)
k: (((n) 1
sup 0C
(E,, [1A] - Ep,,, [1A])
(10)
sup
Σ lm(x) 1a(x)]
Σ #Esupp(H)
(An()) Ia(x)]
(11)
Ag (Esupp(a)
sup AC
Σ (m(x) [x)) 1 (x)
(12)
#Esupp(u)
where (1) is due to fact that Ja x'1 for xn [0. 11, (2) is by dual formulation of the Total
Variation, and (3) is by the fact that fi, is support on supp( by construction. Consider then the following boolean variable
1
2 [supp[u)]
H = Ilu(xr) -(nn(x))
log
(13)
2n
13

---

Published as a conference paper at ICLR 2024
then have by Union bound inequality of probability:
U
H
S
Σ
[[Hz]
(14)
[eSsup(())
#εsupp(k)
Σ xÉsupp(u)
(15)
(supp(n)
6.
(16)
Therefore, we have
(supp(u) 2
(2
2 (supp(u))
P
Wi (HL 2
log
(17)
2n
5
suppp(u))
2
2 (supp(u))
P
sup
Σ
[u(x)) -n(()) 1A(x)
log
(18)
AC
2n
8
r)Supp(M)
(1)
2
U
H₂
(19)
=ESupp(a)
IV
(20)
where (1) is due to the union bound argument, completing the proof.
We now ready complete the proof:
Wi (P((F') P){(((()))) s ()(P(fFt)) +)
(21)
< Wi(P((E P(( }))
(22)
+ Wi(P((F }), Wi((((r P),
(23)
(supp(u))
2 |supp(u))
< $ <AV
log
2n
5
(24)
Similarly, we can apply the above argument to get
(sup((u))
2 |supp(u))
Wi(P(E=')),P( s <
log
(2
2n
8
(25)
Merging these two together allows us to complete the proof.
A.2 EVALUATION ON HYPERNERF VRIG DATASET
The HyperNeRF dataset, initially introduced by Park et al. (021 and Park et al. (2021b), un- derwent revisions after Gao et al. (2022) identified certain limitations. These limitations included frames that transitioned abruptly between multiple camera viewpoints in consecutive time steps, scenario challenging to capture from single camera, as well as scenes portraying quasi-static sce- narios that do not accurately represent real-world dynamics. In response, [ao et al. (2022) proposed an enhanced and more demanding version this dataset, which we employ for our evaluation. This augmented dataset comprises seven sequences in total, each enriched with keypoint annotations. It encompasses 7 multi-cann mulli--cmmra captures and single-camera captures, all featuring 480p resolution videos. It is noteworthy that all dynamic scenes within this dataset are wwward-fciing.. For evalu- ation, we apply masked metrics as introduced in Gao et al. (2022), which utilize covisibility masks. Results shown in Table and Fig.
14

---

Published as a conference paper at ICLR 2024
Broom
Chicken
Peel-banana
Method
mPSNR
mSSIMt mLPIPS. mPSNRt mSSIM LPIPS
mPSNR1 mSSIM m
TiNeu Vox
19.633
0.609
0.659
24.103
0.777
0.264
21.558
0.863
0.267
TiNeu Vox w/ Reg
20.955
0.774
0.628
26.006
0.949
0.196
22.313
0.891
0.274
3dprinter
Tail
Toby sit mSSIM mLPIPS.
Method
mPSNR mSSIM mLPIPS.,
mPSNR1 mSSIM mLPIPS. mPSNR1
TiNeu Vox
19.184 19.323
0.780 0.775
0.296
22.593
0.822
0.511
18.191
0.783 0.790
0.611
TiNeu Vox w/ Reg
0.318
23.191
0.891
0.438
19.385
0.536
Table 6: Evaluation of the proposed regularizer over the HyperNeRF vrig dataset.
d 128
256
Method
mPSNR mSSIM mLPIPS.
mPSNR1 mSSIM
LPIPS
mPSNR1 mSSIM mLPIPS.
TiNeu Vox w/ Reg
9.441
0.409
0.575
10.020
0.478
0.510
10.190
0.479
0.536
d 1024
2048
4096
Method
mPSNR
MSSIM mLPIPS! mPSNRT mSSIM mLPIPS! mPSNR1 MSSIM1 mLPIPS,
TiNeu Vox w/ Reg
10.325
0.489
0.544
10.841
0.525
0.555
11.176
0.545
0.496
Table 7: Ablation over the number of samples used for OT distance computation. Metrics are calculated over the iPhone dataset. As the number of samples grow, the performance improves, possible due to the lower error in OT computation as predicted by Theorem
A.3 EFFICIENCY OF OUR REGULARIZER
The efficiency of regularization approach is attributed two key factors: 1) the elimination of data pre-processing and 2) superior run-time computational efficiency.
For depth regularization, the main efficiency bottleneck is the necessity pre-compute depth maps for all frames. Although this adds minimal computational overhead during run-time, it is signifi- cant pre- processing step. In contrast, our method requires no such pre-processing, offering clear advantage.
More importantly, our method demonstrates substantial training time efficiency compared both scene-flow and LPIPS regularizations. The following table illustrates the average run-time for the
TiNeu Vox on the iPhone dataset;
Method
Time Required
Ours
hours
LPIPS
hours
Scene-flow
10 hours
Table 8: Comparison of Time Required for Different Methods
Scene-flow regularization demands pre-computing optical flows and solving integration problems using the Runge- Kutta method during training, which is computationally intensive. Similarly, LPIPS involves backpropagating through an additional deep network at training, adding to its computational load.
In contrast, our method is significantly more lightweight, as it avoids these computationally demand- ing steps.
A.4 FAILURE CASES
Since our method relies on an approximate OT distance, there can be instances where a pixel aver- aging effect occurs, resulting in subtle blur. Interestingly, this effect can potentially cause slight decline in the quality of renderings, especially when the baseline model already performs excep- tionally well in a given sequence. Additionally, the baseline model fails to converge completely in particular sequence, regularization technique may not yield substantial improvements in the results. Fig. shows some examples.
15

---

Published as a conference paper at ICLR 2024
TiNeuVox
ToNeuVox \w reg
TiNeuVox
TiNeuVox \w reg
Figure 5: qualitative illustration of the effect of the proposed regularization on the HyperNeRF vrig dataset.
16

---

Published as a conference paper at ICLR 2024
TiNeuVox
TiNeuVox \w reg
TiNeuVox
TiNeuVox kw reg
Figure 6: qualitative illustration of the effect of the proposed regularization the HyperNeRF vrig dataset.
17

---

Published as a conference paper at ICLR 2024
Figure 7: Example failure cases. Top row: baseline, botton row: haseline with regularization. Due to the pixel averaging effect that stems from the OT approximation sometimes degrade or does not improve the results. We observed cases where the baseline model performs too poorly, leading our regularization having much effect.
18


Reviews:
[{"review_id": "j1wAcCuE2x", "reviewer": "Reviewer_cmoJ", "summary": "This work presents a regularizer for dynamic NeRFs to improve the rendering quality. Specifically, it is based on the hypothesis that the pixel intensity distribution of a scene, which is rendered from a specific fixed camera view, should remain consistent within short time intervals. Then, this work uses the dissimilarity measure between pixel intensity distributions as the alternative to the pixel-to-pixel distance function to formulate the problem as an Optimal Transport (OT) problem. Additionally, this work employs a sliced-Wasserstein approximation to reduce the OT computation complexity from O(n^3) to O(nlogn), making the proposed regularizer more lightweight as compared to baselines. The experiments on 3 dynamic scene reconstruction datasets valide that the proposed regularizer can further improve SOTA dynamic NeRF models' rendering quality.", "questions": "Minor question: this regularizer seems to be compatible with all dynamic NeRF models because it is defined in the pixel space. Currently, there are more dynamic NeRF models based on Gaussian Spatting (e.g., https://arxiv.org/abs/2308.09713 or https://arxiv.org/abs/2310.08528), will such a new representation cause new challenges in applying the proposed regularize from the optimization side?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "> + Well-Motivated: I think the central hypothesis, i.e., the pixel intensity distribution of a scene should remain approximately consistent within short time intervals, is valid for most cases and existing dynamic scene datasets unless scenes with objects in very high-speeds. Also, the related works are discussed in an organized way. Specifically, the regularizers in static NeRF as disentangling camera motion from object motion inherently constitute an ill-posed problem, and existing regularizers in dynamic NeRFs with DNN-based solutions are costly and face the domain gap issue. Thus, a lightweight regularize for dynamic NeRF is highly desirable.\n\n> + Interesting problem formulation: I am not familiar with optimal transport (OT) problems, but this work seems to be the first one to formulate the dynamic NeRF regularization as an OT problem and further optimize the corresponding computation complexity in OT to make it lightweight.\n\n> + Solid experiments: the experiments consider 3 dynamic NeRF datasets and 5 dynamic NeRF models, showing impressive improvement across different datasets.", "weaknesses": "> + Discussion on Failure Cases: Since the hypothesis of this work is that the pixel intensity distribution of a scene should remain approximately consistent within short time intervals, failure cases may happen when some high-speed objects appear in the scene. However, the limitation section did not add sufficient details about it. It would be better if the author could provide some analysis or experimental results to provide a more direct way to show when the hypothesis will be invalid (e.g., under what speed or in some specific scenes).\n\n> + Comparision with the Strongest Baseline: From Table 1, HexPlane seems to be the strongest baseline, but it was not listed in Tab. 2 and Fig. 3. Could the author add more details about it? \n\n> + Comparison of Efficiency Metrics: The author claimed that their proposed regularization is more efficient. It could be more solid to add the training time to better verify it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OwT29ZjuS6", "reviewer": "Reviewer_C1xG", "summary": "This paper proposes a regularization method to enhance the quality of novel view synthesis for dynamic NeRFs. The regularization is based on the notion that the light intensity distribution should remain approximately constant from a fixed camera view within short time intervals. Specifically, the method implements distance metrics such as sliced-Wasserstein and serves as a robust plug-in to improve the quality for several state-of-the-art methods across multiple dynamic scene view synthesis benchmarks.", "questions": "- How does the proposed method compare to more recent regularization techniques?\n- Can the proposed method be combined with existing regularization techniques?\n- Do we need to compromise performance (reduce batch size to fit into GPU memory) and/or speed (increase per-iteration time) to enable the proposed method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The intuitive idea of exploiting light distribution consistency.\n- The proposed regularization is easy to implement and can be integrated into various methods.", "weaknesses": "- Baseline comparison. Previous literature, including random background compositing [2], additional metric depth supervision [3] from iPhone sensor, and surface sparsity regularizer [4], presents more advanced regularization techniques. Quantitative results reported in Table 3 and Figure 10 in [1] suggest that the baseline methods Nerfies and HyperNeRF gain a significant performance boost from these techniques. Therefore, it is unclear how well the proposed regularization performs in comparison\n- Robustness in combination with other regularizations. It is uncertain whether the proposed regularization term can be effectively combined with those mentioned above [1][2][3] to further enhance the performance.\n- Efficiency. The paper lacks data points illustrating the speed and memory usage of the proposed method.\n\n[1] Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dy-\nnamic view synthesis: A reality check. *Advances in Neural Information Processing Systems*, 35:\n33768–33780, 2022.\n\n[2] Chung-YiWeng,BrianCurless,PratulPSrinivasan,JonathanTBarron,andIraKemelmacher-Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In *CVPR*, 2022.\n\n[3]  Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In *CVPR*, 2021.\n\n[4] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In *CVPR*, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KH7OKjMLV9", "reviewer": "Reviewer_HrLW", "summary": "This paper introduces a regularization method for dynamic view synthesis by reformulating the regularization task to an optimal transportation problem. The paper assumes that, for a fixed camera pose, the intensity of the pixel of images within a small interval should be constant. Hence, the authors propose to minimize the divergence metric of two distributions at two times. The method is architecture-agnostic, simple and can be easily integrated into existing frameworks. The experiments show that the proposed regularization improve the performance of different frameworks.", "questions": "1.\tWhat are the settings of the baseline methods? Do you remove all other losses and only use the photometric loss, or you use all the proposed losses of the original papers.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1.\tThe work reformulates the problem of regularization of dynamic nerf based on the assumption proposed by the paper. The Fig 1 clearly shows the difference between the proposed method and the simple metrics (like L2 distance).\n2.\tThe proposed method seems technically sound with the theory of optimal transportation.\n3.\tThe unbiased estimator is a clear advantage over existing regularization, like depth and optical flow.", "weaknesses": "1.\tI am not the expert of the optimal transportation area. I cannot judge the novelty of the designed method, including the proposed theorem 1, and the sliced-Wasserstein approximator. It seems that the optimal transport method proposed here is not original and lack novelty. But again, I am not an expert for this area, and I would be open mind for other reviewers’ opinions for this part.\n2.\tThe proposed regularizations only based on 2D distributions and cannot regularize 3D deformation directly. For me, it is equal to the idea that keep the image as static as possible. Is it true that the improvement of the regularization come from the improvement of the static background of the images, which are not related to the dynamic objects? The chichchicken video in supp supports this point.\n3.\tThis method does not have much effect when the performance of the framework is above the certain level.\n4.\tThis method could not deal with scenes with high frequency details, as such situation could break the assumption of the paper.\n5.\tThere should be more visual results, especially for videos. Combine the two videos side by side in one video would be much helpful for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oK19CaN5zJ", "reviewer": "Reviewer_bXej", "summary": "This paper proposed a regularization method for dynamic NeRFs using the optimal transport of pixel values at two different time steps, exploiting consistent structures between the video frames at different time steps. Although they assumed smooth motions without abrupt scene dynamics, they showed consistent results for iPhone and HyperNeRF interpolation datasets applying to various dynamic NeRFs.", "questions": "- Q1. In Tbl. 1. Hexplane w/ Reg on the Block scene had a significant gain while deteriorating the other scenes (i.e., Paper windmill, Teddy, Wheel in PSNR). How do you speculate on this result?\n\n- Q2. In Sec. 3.1, the theoretical statement makes the paper unnecessarily complicated. And I do not follow why Thm. 1 is necessary for the argumental context.\n\n- Q3. In Fig. 1, how do you decide which metric is the best in the manner of quantitativeness?\n\n- Minors:\n  - In Sec. 2, Related works -> Related work\n  - In Sec. 3.1, from an images -> from images\n  - The caption of Tbl. 5 goes beyond the content boundary of the bottom.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A simple yet effective regularization for dynamic NeRFs. \n\n- Thank you for the neat presentation of your work in the code of Alg. 1.", "weaknesses": "- As the authors mentioned, this method asserted smooth motions without abrupt scene dynamics in the dynamic scene. The proposed method is rather an ad-hoc method for a subset of dynamic scenes. The authors may introduce a method to determine whether a given scene is appropriate to apply this method before full training and evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)