Supplemental Materials:
Neural 3D Video Synthesis from Multi-view Video
Tianye Li1,2,∗
Mira Slavcheva2,∗
Michael Zollhoefer222Simon Green
Christoph Lassner
Changil Kim3
Tanner Schmidt2
Steven Lovegrove2
Michael Goesele2
Richard Newcombe2
Zhaoyang Lv2
1

2

University of Southern California

1. Supplemental Video

Reality Labs Research

3

Meta

capture videos using the linear camera mode at a resolutionof 2028×2704 (2.7K) and frame rate of 30 FPS. The multiview inputs are synchronized by a timecode system, and thecamera intrinsic and extrinsic parameters are obtained by
COLMAP [6] and are kept the same throughout the capture.

We strongly recommend the reader to watch our supplemental video, hosted at the project website: https://neural-3d-video.github.io/, to better judge
the photorealism of our approach at high resolution, whichcannot be represented well by the metrics.
The supplemental video includes:

Details on the capture setup. We build a mobile multiview capture system using 21 GoPro Black Hero 7 cameras,
as shown in Fig. 2. For all results discussed in this paper, we

Our collected data can provide sufficient synchronizedcamera views for high quality 4D reconstruction of challenging dynamic objects and view-dependent effects in anatural daily indoor environment, which did not exist in
public 4D datasets. Our captured data demonstrates a variety of challenges for video synthesis, including objectsof high specularity, translucency and transparency. It also
contains scene changes and motions with changing topology (poured liquid), self-cast moving shadows, volumetriceffects (fire flame), and an entangled moving object withstrong view-dependent effects (the torch gun and the pan),various lighting conditions (daytime, night, spotlight fromthe side), multiple people moving around in open livingroom space with outdoor scenes seen through transparent
windows with relatively dark indoor illumination. We visualize one snapshot of the sequence in Fig. 1. Unless otherwise stated, we use keyframes that are 30 frames apart.
In total, we trained our methods on a 60 second video sequence (flame salmon) in 6 chunks with each 10 seconds
in length, five other 10 seconds cooking videos captured
at different time with different motion and lighting, andone 25 seconds video in indoor videos in 5 chunks. We
also trained a few additional videos of outdoor scenes inchunks of 5 seconds with denser keyframes, which are 10
frames apart. In the end, we employ a subset of 18 camera views for training, and 1 view for quantitative evaluation for all datasets except one sequence observing multiple
people moving, which only uses 14 cameras views for training. We calculate a continuous interpolated spiral trajectory
based on the training camera views, which we employ for
qualitative novel view evaluation.

∗ Equal contribution. TL’s work was done during an internship at
Reality Labs Research.

We found that the GoPro linear FOV mode sufficiently
well compensates for fisheye effects, thus we employ a pin-• 3D video synthesis results on various dynamic scenes
including challenging dynamic topology change fast
motion, view-dependent effects such as specularity
and transparency, varying illuminations and shadows,
and volumetric effects such as steam and fire;• A short presentation on the method (the DyNeRF representation and the efficient training method);• Video comparisons to baseline methods: NeRF-T,
DyNeRF-noIS, LLFF [4], NeuralVolume [3];• Visualization of the estimated geometry (rendered as
depth maps);• Slow-motion and bullet-time effects by our DyNeRF;• More results on more challenging indoor scenes;• Results on immersive video datasets [1];• Demonstration of interactive playback of our 3D
videos in commodity VR headset Quest 2 using layered meshes distilled from our pretrained DyNeRF
model;• Limitation of our results on more challenging outdoor
scenes.

2. Datasets

1

Figure 1. Frames from our captured multi-view video flame salmon sequence (top). We use 18 camera views for training (downsized on
the right), and held out the upper row center view of the rig as novel view for quantitative evaluation. We captured sequences at different
physical locations, time, and under varying illumination conditions. Our data shows a large variety of challenges in high quality wide angle
3D video synthesis.

Figure 2. Our multi-view capture setup using synchronized GoPro Black Hero 7 cameras.

hole camera model for all our experiments. For all training,
we hold out the top center camera for testing, and use therest of the cameras for training. For each captured multiview sequence, we removed a particular camera if the timesynchronization did not work. We also notice there aresome inconsistent appearance in the some video streams
caused by different lighting sources observed from different view angles, which we excluded in training.
Additional Immersive Videos from [1]. We also demonstrate our method using the multi-view captured videos
from [1] which have been made publicly available recently.
Due to the time constraints, we train DyNeRF models individually on a few 5s video clips from “Welder”, “Flames”,
and “Alexa Meade Face Paint” to validate our algorithm.

There are a few differences in their capture setup whichpose different opportunities and challenges to our method.
First, different from our captured linear camera videos
which are front-facing, their videos are captured on a half
spherical inside-out rig with heavy distortion in each view.
Second, their rig is composed of 46 cameras in each scene,
which contains more than two times more numbers of cameras in training. Successfully training on this scene using
Dynerf requires us to compress a larger dynamic view space
and utilizing all training video pixels more efficiently. During training, we sample the rays directly from the raw resolutions of the distorted multi-view videos and render thenovel view video using a pinhole camera. We demonstrate
our algorithm can work on this type of data to create an
immersive 3D video experience without any change in the
representation.

3. Importance Sampling Schemes
Sampling Based on Global Median Maps (DyNeRFISG). For each ground truth video, we first calculate theglobal median value of each ray for all time stamps C(r) =median C(t) (r) and cache the global median image. Durt∈T

ing training, we compare each frame to the global median
image and compute the residual. We choose a robust normof the residuals to balance the contrast of weight. The norm
measures the transformed values by a non-linear transfer
function ψ(·) that is parameterized by γ to adjust the sensitivity at various ranges of variance:


1
.
(1)
W(t) (r) =ψ C(t) (r) − C(r); γ312Here, ψ(x; γ) = x2x+γ 2 is the Geman-McLure robust function [2] applied element-wise. Intuitively, a larger γ willlead to a high probability to sample the time-variant region,and γ approaching zero will approximate uniform sampling.
C(r) is a representative image across time, which can also
take other forms such as a mean image. We empirically validated that using a median image is more effective to handle
high frequency signal of moving regions across time, whichhelps us to approach sharp results faster during training.
Sampling Based on Temporal Difference (DyNeRFIST). An alternative strategy, DyNeRF-IST, calculates theresiduals by considering two nearby frames in time ti andtj . In each training iteration we load two frames within a
25-frame distance, |ti − tj | ≤ 25. In this strategy, we focus
on sampling the pixels with largest temporal difference. We
calculate the residuals between the two frames, averaged
over the 3 color channels
1
C(ti ) (r) − C(tj ) (r) , α . (2)W(ti ) (r) = min
3
1
To ensure that we do not sample pixels whose values
changed due to spurious artifacts, we clamp W(ti ) (r) witha lower-bound α, which is a hyper-parameter. Intuitively, asmall value of α would favor highly dynamic regions, whilea large value would assign similar importance to all rays.
Combined Method (DyNeRF-IS⋆). We empirically observed that training DyNeRF-ISG with a high learning rate
leads to very quick recovery of dynamic detail, but resultsin some jitter across time. On the other hand, training
DyNeRF-IST with a low learning rate produces a smooth
temporal sequence which is still somewhat blurry. Thus,
we combine the benefits of both methods in our final strategy, DyNeRF-IS⋆ , which first obtains sharp details via
DyNeRF-ISG and then smoothens the temporal motion via
DyNeRF-IST.
Training Details with the Important Sampling Schemes.
We apply global median map importance sampling(DyNeRF-ISG) in both the keyframe training and fullvideo training stage, and subsequently refine with temporal
derivative importance sampling only for the full video. For
faster computation in DyNeRF-ISG we calculate temporal
median maps and pixel weights for each view at 14 th of theresolution, and then upsample the median image map to the
input resolution. For γ in the Geman-McClure robust norm,
we set 1e−3 during keyframe training, and 2e−2 in the fullvideo training stage. Empirically, this samples the background more densely in the keyframe training stage thanfor the following full video training. We also found out thatusing importance sampling has a larger impact in the fullvideo training, as keyframes are highly different. We set

α = 0.1 in DyNeRF-IST. In the full video training stage wefirst train for 250K iterations of DyNeRF-ISG with learning
rate 1e−4 and then for another 100K iterations of DyNeRFIST with learning rate 1e−5.

4. More Results
4.1. Details on Baseline Methods.
• Multi-View Stereo (MVS): We reconstruct the textured 3D meshes using commercial photogrammetry
software RealityCapture * and render the novel view
with from the textured 3D meshes frame-by-frame.
This baseline demonstrates the challenges using traditional geometry based approaches.
• Local Light Field Fusion (LLFF) [4]: LLFF is oneof the state-of-the-art Multiplane Images based methods tailored to front-facing scenes. We apply the pretrained network in LLFF to produce the multiplane images and render the novel views using default parameters. To work with videos in our datasets, we produce
the novel view frame-by-frame by query the inputs ateach corresponding time.
• Neural Volumes (NV) [3]: NV is one of the state-ofthe-art learning based volumetric methods can generate novel view videos. We use the same training videos
and apply the default parameters to train the network.
We set the bounding volume according to the geometry of the scene. We use 1283 voxel grid for the RGBαvolume and 323 for the warping grid. It renders a novel
view image via ray marching a warped voxel grid ateach timestamps.
• NeRF-T: Refers to the version in Eq. 1. in the main
paper, which is a straight-forward temporal extensionof NeRF. We implement it following the details in [5],with only one difference in the input. The input concatenates the original positionally-encoded location,
view direction, and time. We choose the positionallyencoded bandwidth for the time variable to be 4 and
we do not find that increasing the bandwidth further
improves results.
• DyNeRF† : We compare to DyNeRF without our proposed hierarchical training strategy and without importance sampling, i.e. this strategy uses per-frame latentcodes that are trained jointly from scratch.
• DyNeRF with varying hyper-parameters: We vary
the dimension of the employed latent codes (8, 64,
256, 1024, 8192). We also apply ablation studies on different versions of DyNeRF with important* https://www.capturingreality.com/

Table 1. Quantitative comparison of our proposed method tobaselines of existing methods and radiance field baselines trained
at 200K iterations on a 10-second sequence. DyNeRF-IS⋆ uses
both sampling strategies (ISG and IST) and thus runs for moreiterations: 250K iterations of ISG, followed by 100K of IST; it isshown here only for completeness.
MethodPSNR ↑MSE ↓DSSIM ↓LPIPS ↓FLIP ↓MVS
NeuralVolumes
LLFF
NeRF-T
DyNeRF†
DyNeRF-ISG
DyNeRF-IST
DyNeRF-IS⋆

19.121322.7975
23.2388
28.4487
28.499429.462329.7161
29.58080.01226
0.005250.004750.001440.001430.001130.001070.001100.1116
0.06180.07620.02280.02310.02010.01970.01970.25990.2951
0.23460.10000.09850.08540.08850.08320.25420.20490.18670.14150.14550.13750.13400.1347

sampling methods: DyNeRF-ISG, DyNeRF-IST, and
DyNeRF-IS⋆.

4.2. Quantitative Comparison to the Baselines.
Tab. 1 shows the quantitative comparison of our methods to the baselines using an average of single frame metrics. We train all the neural radiance field based baselines and our method the same number of iterations forfair comparison. Compared to the existing methods, MVS,
NeuralVolumes and LLFF, our method is able capture andrender significant more photo-realistic images, in all thequantitative measures. Compared to the time-variant NeRF
baseline NeRF-T and our basic DyNeRF model without
our proposed training strategy (DyNeRF† ), our DyNeRF
model variants trained with our proposed training strategy
perform significantly better in all metrics. DyNeRF-ISG
and DyNeRF-IST can both achieve high quantitative performance, with DyNeRF-IST slightly more favorable in termsof the metrics. Our complete strategy DyNeRF-IS⋆ requires
more iterations and is added to the table only for completeness.

4.3. The Impact of Importance Sampling
In Fig. 3 we evaluate the effect of our importance sampling strategies, DyNeRF-ISG, DyNeRF-IST and DyNeRFIS⋆ , against a baseline DyNeRF-noIS that also employsa hierarchical training strategy with latent codes initialized from trained keyframes, but instead of selecting rays
based on importance, selects them at random like in standard NeRF [5]. The figure shows zoomed-in crops ofthe dynamic region for better visibility. We clearly seethat all the importance sampling strategies manage to recover the moving flame gun better than DyNeRF-noIS in
two times less iterations. At 100k iterations DyNeRF-ISG
and DyNeRF-IST look similar, though they converge differently with DyNeRF-IST being blurrier in early iterations and DyNeRF-ISG managing to recover moving de-Table 2. Ablation studies on the latent code dimension on asequence of 60 consecutive frames. Codes of dimension 8 are insufficient to capture sharp details, while codes of dimension 8,192
take too long to be processed by the network. We use 1,024 forour experiments, which allows for high quality while converging
fast. *Note that with a code length of 8,192 we cannot fit the same
number of samples in the GPU memory as in the other cases, so
we report a score from a later iteration when roughly the same
number of samples have been used.
Dimension

PSNR ↑MSE ↓DSSIM ↓LPIPS ↓FLIP ↓

8
64
256
1,024
8,192*

26.4349
27.1651
27.3823
27.628627.4100

0.002280.001930.001840.001730.00182

0.04380.0401
0.04210.04080.03480.26230.2476
0.26690.25280.19320.15620.16530.15000.15560.1616

Table 3. Comparison in model storage size of our method (DyNeRF) to alternative solutions. For HEVC, we use the default GoPro 7 video codec. For JPEG, we employ a compression ratethat maintains the highest image quality. For NeRF, we use aset of the original NeRF networks [5] reconstructed frame byframe. For HEVC, PNG and JPEG, the required memory may vary
within a factor of 3 depending on the video appearance. For NeuralVolumes (NV), it only accounts the neural network size without
counting its dependency on additional input streams. For NeRF,
NeuralVolume and DyNeRF, the required memory is constant. All
calculation are based on 10 seconds of 30 FPS videos captured by
18 cameras.

Size (MB)

HEVC

PNG

JPEG

NeRF

NV

DyNeRF

1,406

21,600

3,143

1,080

773

28

tails slightly faster. The visualizations of the final results
upon convergence in Fig. 3 demonstrate the superior photorealism that DyNeRF-IS⋆ achieves, as DyNeRF-noIS remains much blurrier in comparison. We notice that without
importance sampling, the system cannot reach an acceptable visual quality within extended training time, indicating the necessity of the importance sampling scheme. In
Fig. 4, we compare various settings of the dynamic neural radiance fields. NeRF-T can only capture a blurry motion representation, which loses all appearance details in themoving regions and cannot capture view-dependent effects.
Though DyNeRF† has a similar quantitative performanceas NeRF-T, it has significantly improved visual quality inthe moving regions compared to NeRF-T, but still struggles to recover the sharp appearance details. DyNeRF with
our proposed training strategy, DyNeRF-ISG, DyNeRF-ISTand DyNeRF-IS⋆ , can recover sharp details in the moving
regions, including the torch gun and the flames.

25000 iterations

50000 iterations

100000 iterations

DyNeRF-IS⋆ final

DyNeRF-noIS final

DyNeRF-IST

DyNeRF-ISG

DyNeRF-noIS

5000 iterations

Figure 3. Comparison of importance sampling strategies over training iterations.

Comparisons on of Model Compression. Our model iscompact in terms of model size. In Tab. 3, we compare ourmodel DyNeRF to the alternatives in terms of storage size.
Compared to the raw videos stored in different images, e.g.,
PNG or JPEG, our representation is more than two ordersof magnitude smaller. Compared to a highly compact 2D
video codec (HEVC), which is used as the default video
codec for the GoPro camera, our model is still 50 times
smaller. It is worth noting that these compressed 2D representations do not provide a 6D continuous representation

as we do. Though NeRF is a compact model for a single
static frame, representing the whole captured video without
dropping frames requires a stack of frame-by-frame reconstructed NeRF networks, which is more than 30 times larger
in size compared to our single DyNeRF model. Comparedto the convolutional model used in NeuralVolume, DyNeRF is more compact in size and can represent the dynamic
scene with better quality.

Dynamic region zoom-in

DSSIM

FLIP

0.14870.03920.1294

DyNeRF-ISG

0.0252

0.1174

DyNeRF-IST

0.02750.11990.02350.1144

DyNeRF†

0.0531

DyNeRF-IS*

NeRF-TRGB rendering

Figure 4. Qualitative comparisons of DyNeRF variants on one image of the sequence whose averages are reported in Tab. 1. From left
to right we show the rendering by each method, then zoom onto the moving flame gun, then visualize DSSIM and FLIP for this region
using the viridis colormap (dark blue is 0, yellow is 1, lower is better). The three hierarchical DyNeRF variants outperform these baselines:
DyNeRF-ISG has sharper details than DyNeRF-IST, but DyNeRF-IST recovers more of the flame, while DyNeRF⋆ combines both of thesebenefits.

Impact of Latent Embedding Size on DyNeRF We runan ablation on latent code length on 60 continuous frames
and present the results in Table 2. In this experiment, we do
not include keyframe training or importance sampling. We
ran the experiments until 300K iterations, which is whenmost models are starting to converge in rendering qualities. Note that with a code length of 8,192 we cannot fit

the same number of samples in the GPU memory as in theother cases, so we report a score from a later iteration whenroughly the same number of samples have been used. We
use 4× 16GB GPUs and network width 256 for the experiments with this short sequence. From the metrics we clearlyconclude that a code of length 8 is insufficient to represent the dynamic scene well. Moreover, we have visuallyobserved that results with such a short code are typicallyblurry. With increasing latent code size, the performance
also increases respectively, which however saturates at a dimension of 1024. A latent code size of 8192 has longer
training time per iteration. Taking the capacity and speed
jointly into consideration, we choose 1024 as our default
latent code size for all the sequences in this paper and the
supplementary video.
Additional Discussions on the Latent Codes Besides
all the above findings, we also observe some failure cases
to manipulate the latent codes. Extrapolating the latentcodes in time cannot directly create high quality extrapolated views. We have extensively investigated latent code
optimization with various combinations of parameter learnability for latent codes (keyframe / remaining frames) andthe network. With frozen keyframe latent codes, we observe
blurrier results than the all-learnable case. Therefore learning both latent codes (keyframe / remaining frames) and the
network is necessary for producing sharp and high-quality
renderings.
View-dependent Effects in Dark Indoor Scenes. DyNeRF can represent view-dependent effects as well as motion in one continuous representation. When input cameras
streams have slightly different appearance differences in observation, we find DyNeRF will model this difference aspart of the view-dependent effects when generating novel
views. We can observe this artifact in all of our dark indoor
scenes where there are more obvious color inconsistencyfrom wide-angle input video streams. Incorporating more
careful color calibration and learning color calibration may
address this problem, which we leave for future work.
3D Video Editing via Manipulating the Latent Codes
DyNeRF represents a continuous spatial-temporal dynamic
scene which supports rendering any view within the interpolation boundary of space and time. We can create a latent code at a sub-frame time via interpolation and rendera “slow motion” 3D video with any given FPS rate. DyNeRF can enable smooth interpolation from 30fps to 60fps oreven 150fps. Furthermore our method can render “bullettime” effect by by freezing the latent code at any arbitrary
time and manipulating camera views in space. We include
the video effects of “slow motion“ and “bullet-time“ fromarbitrary time in our supplementary videos.
Rendering Time The rendering time of our dynamic neural radiance fields is on par with NeRF due to the structuralsimilarity of the approaches. Our current, not fully optimized version achieves a rendering time of 45 seconds forone 1080p frame using two V-100 GPUs with 16 GB memory.

References[1] Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew Duvall, Jason Dourgarian, Jay
Busch, Matt Whalen, and Paul Debevec. Immersive light field
video with a layered mesh representation. ACM Transactions
on Graphics (TOG), 39(4):86–1, 2020. 1, 2
[2] Stuart Geman and D McClure. Bayesian image analysis:
An application to single photon emission tomography. Amer.
Statist. Assoc, pages 12–18, 1985. 3[3] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel
Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images.
ACM Trans. Graph., 38(4):65:1–65:14, July 2019. 1, 3
[4] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon,
Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, andAbhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions
on Graphics (TOG), 38(4):1–14, 2019. 1, 3
[5] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik,
Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf:
Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. 3, 4
[6] Johannes Lutz Schönberger and Jan-Michael Frahm.
Structure-from-motion revisited. In Conference on Computer
Vision and Pattern Recognition (CVPR), 2016. 1