Patent ID: 12243273

DESCRIPTION OF EXAMPLE EMBODIMENTS

Particular embodiments described herein relate to systems and methods of generating a spatial-temporal representation of a dynamic scene with moving objects based on videos that capture scene information in both spatial and temporal domains. The spatial-temporal representation may be a Neural Radiance Field (NeRF). Once trained, NeRF may be used to create new views of the dynamic scene from arbitrary viewpoints and arbitrary times. The space-time NeRF may include a multi-layer perceptron (MLP) which can be trained using training samples during the training stage and queried at the inference stage to generate images for desired viewpoints and times to represent the dynamic scene. NeRF may use a continuous volume rendering method that allows the color of a pixel to be determined by integrating the radiance as modulated by the volume density along the camera ray.

More formally, embodiments described herein may be used to tackle the problem of reconstructing dynamic 3D scenes from video inputs from multiple cameras, i.e., {C(t)} for time index t∈T={1, 2, . . . , T} with known camera intrinsic and extrinsic parameters. The resulting representation should be able to render photorealistic images from a wide range of camera views and at arbitrary times in the recordings. For example, a dynamic scene may include any number of objects in motion, such as a person performing some task, like cooking. The dynamic scene may be simultaneously recorded by several cameras (e.g., 5, 10, or 20 cameras) over a period of time. The cameras each has a location in 3D space and a corresponding view direction. The viewpoint of each of the recorded videos may be static. Despite the static camera positions, embodiments described herein learn a representation of the dynamic scene using the videos recorded and enable a novel view of the dynamic scene to be reconstructed for any time instant. In other words, the reconstructed video may show the dynamic scene from a perspective different from any of the cameras used to record the original videos.

We achieve this goal with a continuous space-time Neural Radiance Fields (NeRF), controllable by a series of learned temporal latent embeddings/codes. The learned spatial-temporal representation of NeRF is able to compress the vast discrete amount of input videos from multiple cameras to a compact 6D representation that can be continuously queried. The learned temporal embedding/codes capture detailed temporal effects, such as topological changes, radiance changes, such as self-casted shadows, volumetric effects, such as a flame. This is a simple yet effective design that is able to represent one order of magnitude longer 3D video than the current state-of-the-art methods. Once the continuous space-time NeRF learns the 3D dynamic scene, it could be used to reconstruct a new video of the dynamic scene from any novel viewpoint. For example, each frame of the new video may be generated by querying NeRF using any desired viewpoint (including view position and direction in 3D space), field of view, and/or time. The desired viewpoint has a 3D position (akin to a virtual camera). The image plane of the virtual camera has a plurality of pixels. The pixel value (e.g., color and/or opacity) of each pixel may be queried from NeRF using a ray cast from the viewpoint through that pixel in the image plane. The ray may be defined by the viewpoint and a ray direction. The ray, along with a time representation specifying the desired time, may be used to query NeRF for several samples along the ray for color and opacity data. The samples along the ray may be aggregated to provide a final pixel value (color and opacity) for the pixel of interest. This process may be repeated for every pixel within the desired frame to reconstruct the whole frame. Different pixels of each frame are generated using the same viewpoint and time representation but different ray directions. Different frames within the new video would be queried from NeRF using different time representations.

FIG.1illustrates a conceptual representation of certain features of a continuous space-time Neural Radiance Fields (NeRF) according to particular embodiments. During training, the volume100may include a dynamic scene with moving objects (e.g., a person cooking, pets running around, etc.). The dynamic scene may be simultaneously captured in training videos by multiple cameras over time. For example, one camera may have a camera viewpoint110a, and another camera may have a camera viewpoint110b. For simplicity,FIG.1illustrates two camera viewpoints110a,110b, but any other number of camera viewpoints may be used instead (e.g., 10, 18, 20, etc.). The camera viewpoints110a-bhave their respective image planes120a-b. Each of the image planes120a-bhas multiple pixels. For simplicity,FIG.1shows an example of a pixel130ain image plane120aand another example of a pixel130bin image plane120b. For each pixel, a ray may be drawn from the camera viewpoint through that pixel.FIG.1shows a ray140abeing drawn from the camera viewpoint110athrough the pixel130a. Similarly, a ray140bis drawn from the camera viewpoint110bthrough the pixel130b. Each ray (e.g.,140a) associated with each pixel may be defined by the camera viewpoint (e.g.,110a) and a ray direction stemming from that viewpoint.

During training, the pixel value (e.g., color and/or opacity) of a pixel (e.g.,130a) may be generated using NeRF and volume rendering and compared to a ground truth pixel value of that pixel captured in a frame of a training video. In a training iteration, the computing system used for training may determine that a frame in a training video was captured by a camera from a camera viewpoint110aat time t. Based on the camera's viewpoint110aand intrinsic and extrinsic camera parameters, the computing system may determine the 3D locations of the corresponding image plane120aand the pixels within it. A pixel (e.g.,130a) within the image plane120amay be associated with a ray140adefined by (1) the viewpoint110a(the ray's origin) and (2) a ray direction pointing at the pixel. Since the frame used for training is associated with time t, the pixel is additionally associated with a latent code to represent the context at time t, as will be described in more detail below. The viewpoint, ray direction, and latent code are used to query NeRF for the color and opacity values at various sampling points150a-galong the ray140a. The color and opacity values of the sampling points150a-gmay be aggregated using volume rendering to generate a final pixel value for the pixel130a. The final pixel value may then be compared to the corresponding pixel value in the ground truth frame t. The result of the comparison is then used to update NeRF and the latent code. This process may be repeated pixel-by-pixel for any number of temporal video frames of the dynamic scene captured from any number of camera viewpoints.

Once NeRF is trained, it may be used to reconstruct any frame of the dynamic scene at any desired time and from any desired viewpoint. A frame is reconstructed pixel-by-pixel. As an example, let's assume the desired frame is viewed from the perspective of the desired viewpoint110aand at time t. For a particular pixel130awithin the image plane120aassociated with the desired viewpoint110a, the rendering system may associate the pixel130awith a ray140adefined by the desired viewpoint110aand a ray direction pointing toward the pixel130a. The desired viewpoint110a, ray direction, and latent code for time t may be used to query the trained NeRF for the color and/or opacity values at a series of sample points150a-galong the ray140a. The color and/or opacity values of the sample points150a-gmay be aggregated, such as using volume rendering techniques, to generate a rendered pixel value for the pixel130a. This process may be repeated for each pixel within the desired frame. As will be described in further detail below, the desired time and/or desired viewpoint may be different from the times and camera viewpoints associated with ground truth frames used during training.

More formally, the problem of representing 3D video boils down to learning the 6D plenoptic function that maps a viewpoint x∈3, ray direction d∈2from the viewpoint, and time variable t∈, to RGB radiance c∈3and opacity/alpha a∈.

Conventional NeRF approximates the 5D plenoptic function of a static scene with a learnable function:
FΘ:(x,d)→(c,σ)  (1)
The function is given by a Multi-Layer Perceptron (MLP) with trainable weights Θ. NeRF can learn a static scene based on multi-view images of that scene (e.g., 15 photos of the same static scene captured from different viewpoints). Once learned, NeRF may be queried to generate an image of the scene from a novel viewpoint (i.e., the viewpoint may be different from any of the viewpoints of the 15 photos used for training NeRF). The desired image of the scene has a desired viewpoint x and a corresponding image plane. The image plane has a plurality of pixels (e.g., 1000×1000 pixels). To determined the color value of each pixel, a ray may be cast from the viewpoint toward that pixel in the image plane, which is the ray direction d. The viewpoint x and ray direction d define a ray used for querying NeRF for the pixel value of the pixel of interest. Color c and opacity σ values sampled along the ray may be queried from NeRF and aggregated to form the final pixel value for the pixel. This design has been shown to synthesize high fidelity novel views for static scenes only, and it is not trivial to extend it to dynamic 3D scenes.

In particular embodiments, one potential solution is to add a time dependency t to the function:
FΘ:(x,d,t)→(c,σ)  (2)
The 1-dimensional time variable t can be mapped via positional encoding to a higher-dimensional space, in a manner similar to how NeRF handles the inputs x and d. However, empirical experiments have found that it is challenging for this design to capture complex dynamic 3D scenes with challenging topological changes and time-dependent volumetric effects, such as flames (e.g., in a dynamic scene that involves cooking).

Instead of a 1-dimensional time variable t, particular embodiments model the dynamic scene by latent codes zt∈D. A set of time-dependent latent codes, indexed by discrete time variable t, is learned with the function:
FΘ:(x,d,zt)→(c,σ)  (3)
The time-dependent latent codes {zt} provide a compact representation of the state of a dynamic scene at a certain time, which can cope well with variants of complex scene dynamics and radiance changes implicitly. We apply positional encoding to the input position coordinates (e.g., x and d) to map them to a higher-dimensional based on a series of sinusoidal functions. In particular embodiments, no positional encoding is applied to the time-dependent latent codes.

In particular embodiments, before training, the latent codes {zt} are randomly initialized in an independent manner across the frames. For example, before training, the latent code z1corresponding to the frame at time t=1 may be randomly initialized to some vector of values with D dimensions. Similarly, the latent code z2corresponding to the frame at time t=2 may be randomly initialized to another vector of values, and so on. During training, the latent codes will be iteratively updated based on the loss computed after each training iteration. After NeRF is trained, the latent code ztmay be used to specify a particular desired state of the dynamic scene at a certain time t and used to query NeRF for the radiance observed from a particular desired viewpoint and view direction.

FIG.2illustrates an embodiment for using NeRF to synthesize the value of a pixel. NeRF200is a machine learning model conditioned on a compact yet expressive time-variant latent code210and a view ray220, which may be specified as a viewpoint (x, y, z) and view direction (θ, Φ). The latent code210represents the state of a dynamic scene at a particular time. The latent code210that is input to NeRF200may be selected from any number of latent codes (e.g., zt−1,211a, zt211b, or zt+1211c) corresponding to different instances in time (e.g., time t−1, t, or t+1, respectively). By processing a given latent code210and a desired view ray220for a pixel, NeRF200outputs a color230and opacity240of the pixel corresponding to the view ray220for the moment in time that corresponds to the latent code210. The final pixel value (color230and opacity240) may be the result of aggregating (e.g., using volume rendering) sampled values at multiple sample points along the ray220. NeRF may be used in this manner to generate the values for every pixel in a frame to complete the frame. For example, one may desire to generate a frame depicting the dynamic scene at time t from a desired view position p (represented as a 3D coordinate). For each pixel in the frame, a view ray220may be determined based on the desired view position p and the position of that pixel in 3D space (i.e., the view ray220extends from the view position p and through the pixel in 3D space). The view ray220of each pixel in the frame and the latent code ztcorresponding to time t may be processed by NeRF to generate the color230and opacity240values for the pixel. The output frame250bis an example of the dynamic cooking scene at time t as viewed from the perspective of the view position p. In a similar manner, NeRF may be used to generate the frame250afor time t−1 using the latent code zt−1211aand frame150cfor time t+1 using the latent code zt+1211c.

AlthoughFIG.1shows NeRF being used to render frames that correspond to discrete integer frames (e.g., frames at time t−1, t, and t+1), NeRF can also be used to render subframes (i.e., frames between the integer frames). In other words, NeRF allows for continuous time interpolation. During training, the dynamic neural radiance field may be trained on video datasets that capture appearance only at discrete integer frame numbers. To render at arbitrary and continuous time indices, we can linearly interpolate the neighboring latent codes that have been found for the integer frame indices. For example, if one desires to render a subframe at time t+0.5, which is halfway between time t and time t+1, the corresponding learned latent codes ztand zt+1may be linearly interpolated to generate an interpolated latent code zt+0.5for time t+0.5. The interpolated latent code zt+0.5, along with a desired view position and view direction, may be provided to NeRF as input to generate the desired pixel value. Rendering with interpolated latent codes results in a smooth and reasonable dynamic behavior between the two close temporal training frames. This can enable rendering of special visual effects such as slow-motion by interpolating sub-frame latent codes between two discrete time-dependent latent codes and the ‘bullet time’ effect with view-dependent effect by querying any latent code at any continuous-time within the video.

In particular embodiments, NeRF may use volume rendering techniques to produce photorealistic images from arbitrary camera views and time from the dynamic neural radiance field. The rendering process is used both at run time after NeRF has been trained and during training. Conceptually, NeRF learns and encodes the radiance and opacity values of a dynamic scene over time based on video frames captured by multiple cameras. The learned representation of NeRF may be thought of as a volume representation of the spatial-temporal features of the dynamic scene. When provided with a desired latent code zt, a viewpoint, and a view direction, NeRF would return the radiance and opacity values of various sample points along a corresponding ray that traverse the volume representation. The radiance and opacity values sampled along the ray may be blended/fused to generate the final output color and opacity value of the pixel.

More formally, given a ray r(s)=o+sd (origin o and direction d defined by the specified camera pose and camera intrinsics), the rendered color of the pixel corresponding to this ray C(r) is an integral over the radiance weighted by accumulated opacity:
C(t)(r)=∫snsfT(s)σ(r(s),zt)c(r(s),d,zt))ds.(4)
Here, T(s)=exp(−∫snsσ(r(p),zt))dp) and sn, and sfdefine the bounds of the volume depth range. The quadrature can be approximated by evaluating radiance and opacity at discrete sampled points along the rays. In particular embodiments, a hierarchical sampling strategy is used with first stratified sampling on the coarse level followed by importance sampling on the fine level.

In particular embodiments, dynamic neural radiance field is trained on video datasets that capture appearance at discrete integer frame numbers (e.g., a video in the training dataset may have 30 evenly spaced frames in a second, with each frame having a corresponding integer index). To render at arbitrary and continuous time indices, particular embodiments can interpolate (e.g., linearly or any other suitable interpolation scheme) the neighboring latent codes that have been found for the integer frame indices. For example, to render a subframe at time index 2.6 between integer frames 2 and 3, the latent codes for frames 2 and 3 may be linearly interpolated to generate an interpolated latent code for the subframe. The interpolated latent code for the subframe may be used along with a desired viewpoint and view direction to have NeRF render a subframe corresponding to the time index at 2.6. Experiments have shown that rendering with interpolated latent codes results in a smooth and reasonable dynamic behavior between the two close temporal training frames. This can enable rendering of special visual effects such as slow motion by interpolating sub-frame latent codes between two discrete time-dependent latent codes and the ‘bullet time’ effect with view-dependent effect by querying any latent code at any continuous time within the video.

The learnable function weights Θ and the latent codes {zt} are simultaneously trained by minimizing the2-loss between the rendered images C(r) and the ground truth images Ĉ(r), and summed over all rays r that correspond to the image pixels from all training camera views R and throughout all time indices t of the recording:
L=[∥Ĉc(t)(r)−C(t)(r)∥22+∥Ĉf(t)(r)−C(t)(r)∥22]  (5)

FIG.3illustrates a training process for temporal latent codes and a spatial-temporal NeRF, according to particular embodiments. For example, for a given pixel320in a frame310used for training, the training system determines a corresponding ray220(defined by a viewpoint and ray direction) for the pixel location, as previously described. If the frame310used for training is associated with time t, the training system would retrieve the corresponding latent code zt211. At the start of training, the latent code211may be a randomly generated vector of values (e.g., the vector may have a predetermined number of values, such as 1024 or any other suitable number). In subsequent training iterations, the latent code211used would be an updated latent code that has been updated based on previous training iterations. The latent code211and ray220are used to query NeRF200for the pixel value of the pixel of interest, which may include a color230component and/or an opacity240component. The generated pixel value (color230and/or opacity240) may be compared to the pixel value of the corresponding pixel320in the ground-truth frame310using a loss function300. For example, the loss function may be Equation (5) above, which aggregates the loss of multiple rays across multiple frames and camera views. The resulting loss value is backpropagated340to update NeRF200and the latent codes211used during training. This training process may continue until NeRF200and latent codes211converge (e.g., the rendering loss is below a certain threshold) and/or the training data set is exhausted.

One challenge of the ray-tracing-based neural rendering is the significant amount of training time. For example, training NeRF requires about 50 GPU hours for a single frame captured from about twenty 1K resolution images. It becomes infeasible to scale up this computation to train 3D video frames and achieve similar photorealistic quality.

The appearance changes in the natural video between adjacent frames are typically small and locally consistent. To explore how temporal redundancy can be exploited in the context of 3D video, we propose two strategies to accelerate the training process: 1) importance sampling that prefers rays around regions of higher temporal variance and 2) hierarchical training that optimizes data over a coarse-to-fine frame selection. These two strategies combined can be regarded as an adaptive sampling approach for reconstructing the 6D plenoptic function, contributing to significantly faster training and improved rendering quality. The two strategies, either alone or in combination, may be used with the training process described above (e.g., with reference toFIGS.1-3) to improve training speed and rendering quality.

Ray Importance Sampling

Existing ray-based neural rendering methods, including NeRF network, are trained on randomly sampled rays. The number of iterations it takes in training per epoch scales linearly with the number of the pixels in the multi-view videos. For a 10 second 30 FPS video sequence observed from 18 camera views, it takes 18 billion ray sampling per epoch. To train one epoch using 8 GPU, it takes about 600K iterations or about one week. This sampling process is one of the biggest challenges for ray-based neural rendering methods to train 3D video at scale.

However, for a natural video, a large proportion of the dynamic scene is either time-invariant or only contains a small radiance change at a particular timestamp across the entire observed videos. This fact leads to a significant imbalance in the pixel observation and their contribution to the final representation. On the one hand, the perceptual quality in the time-invariant region saturates after a certain number of pixels. On the other hand, reconstructing the time-variant regions with high photorealism requires sampling every single moving pixel observed every single timestamp, which has a significantly lower ratio in every epoch to be sampled and more epochs in training. Therefore, training all pixels with an equal amount of attention using uniform pixel sampling would be a waste of computational resources to achieve photorealistic 3D video.

To address this issue, particular embodiments sample the rays across time with different importance based on the temporal variation of the input videos. An example is shown inFIG.4, which illustrates sampling probability with heatmap visualizations, according to particular embodiments.FIG.4visualizes the heatmaps of a series of keyframes400in a video and the full sequence of frames450in the video. The heatmap of each frame shows the temporal variance of each pixel within the frame. Pixels with high temporal variance are opaque and shown in a warm color, such as red, while pixels with relatively lower temporal variance are shown in a cold color, such as blue. A pixel's temporal variance is directly proportional to the probability of that pixel being selected for training NeRF200and latent codes211. For example, “hot” pixels may be 90% likely selected for training, whereas “cold” pixels may be 15% likely selected.

In particular embodiments, for each observed ray r at time t, a weight W(t)(r) is computed to represent the ray/pixel's temporal variance. In each iteration during training, particular embodiments randomly sample all the views at time t. The weight of the rays from all the input views is first normalized. Then, we apply inverse transform sampling to sample those pixels using the normalized weights of all the rays at time t.

To calculate the weight of each ray, we propose two strategies based on two different insights. In the first one, we calculate the weight map of each ray based on the residual difference of that ray compared to the global median pixel value of that ray across time. We call this strategy importance sampling based on the global median map (NeRF-ISG). In the other one, we consider two temporal frames close in time and calculate the weight based on the residuals of the two frames. This approximates the motion derivative between the temporal frames, which we call importance sampling based on motion derivative (NeRF-ISMD). We explain the details of the two strategies below.

Importance Sampling based on the Global Median Map (NeRF-ISG)). For each ground truth video, we first calculate the global median value of each pixel across time

C_⁡(r)=mediant∈T⁢⁢C(t)⁡(r).
and cache the global median image. During training, we compare each frame to the global median image and compute the residual. We choose a robust1-norm of the residuals to balance the contrast of weight. The norm measures the transformed values by a non-linear transfer function ψ(⋅) that is parameterized by θ to adjust the sensitivity at various ranges of variance:
W(t)(r)=⅓∥ψ(C(t)(r)−C(r);θ)∥1(6)

Here,

ψ⁡(x;θ)=x2x2+θ2
is the German-McLure robust function applied element-wise. Intuitively, a larger θ will lead to a high probability to sample the time-variant region, and θ approaching zero will approximate uniform sampling. The1-norm is applied along the RGB channel dimensionC(r) is a representative images across time, which can also take other form such as mean. We empirically validated that using median is more effective to handle high-frequency signal of moving regions across time, which helps to us approach sharp results faster during training.

Importance Sampling based on Motion derivative (NeRF-ISM). An alternative strategy, NeRF-ISM, calculates the residuals by considering two nearby frames in time tiand tj. In this strategy, we focus on sampling on the pixels with the largest temporal difference. If they are within a 25-frame distance in a batch, |ti−tj|≤25, we calculate the residuals between the temporal frames, averaged over the 3 color channels
W(t)(r)=min(⅓∥C(ti)−C(tj)∥1,α)  (7)
To ensure that we do not sample pixels whose values changed due to spurious artifacts, we clamp W(t)(r) with a lower-bound α, which is a hyper-parameter.
Hierarchical Training

Keyframes are one of the important foundations of video compression techniques. We adapt this idea to our 3D video scenario by firstly training for a set of keyframes and then training for all in-between frames together with the keyframes.

We first train a NeRF model on the keyframes, which we sample equidistant from the multi-view image sequence at fixed intervals K.FIG.5Aillustrates keyframes used in hierarchical training, according to particular embodiments. Each circle corresponds to a frame in a video. The black circles represent the keyframes500a-dused for training, and the white circles represent frames that are not selected for training during the initial training phase. Each frame500a-dselected for training (e.g., the black circles) may be used in the manner described above, such as with reference toFIG.3. In particular embodiments, every pixel within a selected frame is used to train NeRF200and the latent codes211. In other embodiments, only certain pixels within a selected frame are used. Such pixels may be selected based on the importance sampling technique described above.

After the model (e.g., NeRF200and the latent codes211) converges based on the keyframes500a-d, the pre-trained model may be used to initialize the final model to have the same temporal resolution of the full video.FIG.5Billustrates the full sequence of frames used in hierarchical training, according to particular embodiments. InFIG.5B, all the frames in the video are selected for training (i.e., all the circles are black). Alternatively, less than all the frames may be selected for training (e.g., only 80% or 90% of all the frames are selected). In particular embodiments, each selected frame may be assigned a fine-level latent code z211. Since the per-frame motion of the scene within each segment (divided by neighboring keyframes) is smooth, particular embodiments may initialize the fine-level latent codes by linearly interpolating between the coarse latent codes of the keyframes500a-d. For example, after pre-training, each of the keyframes500a-dshown in black inFIG.5Amay have an updated latent code. The latent codes of the in-between frames shown in white inFIG.5Amay be initialized by interpolating the latent codes of the keyframes500a-d. For example, the latent code of the first frame after500amay be initialized based on an interpolation of the latent codes of keyframes500aand500b(e.g., 80% of the latent code of keyframe500aand 20% of the latent code of keyframe500b). Finally, we train with data from all the selected frames jointly, further optimizing the network weights of NeRF200and the latent codes z211. The keyframe models captured an approximation to the time-invariant information across the video, which also occupies a significant portion of the radiance fields at every frame. With a photo-realistically reconstructed keyframe model, the full-frame training only needs to further learn the time-variant information per frame. In particular embodiments, every pixel within a selected frame is used to train NeRF200and the latent codes211. In other embodiments, only certain pixels within a selected frame are used. Such pixels may be selected based on the importance sampling technique described above.

FIG.6illustrates a flow diagram for using hierarchical training and importance sampling, according to particular embodiments. The method600may firstly use keyframes within a video to pre-train NeRF and update the latent codes for the keyframes, followed by full training based on all or a larger subset of the frames in the video. For example, the method may begin at step610, where the latent codes of a set of keyframes in a video are initialized. For example, the latent codes may be randomly initialized. At step620, the pixels in the keyframes may be selected based on their respective temporal variance values. Techniques for computing the temporal variance values (e.g., the aforementioned importance weights) are described above. At step630, the selected pixels in the keyframes may be used in a pre-training process to update NeRF and the latent codes of the keyframes. At step640after the pre-training process concludes, the latent codes of the frames between the keyframes may be initialized. In particular embodiments, the latent codes of the non-keyframes may be initialized based on an interpolation of the latent codes of the neighboring keyframes. At step650, the pixels in the frames may be selected based on their respective temporal variance. Then at step660, the selected pixels in the frames may be used to update NeRF and the latent codes of the frames. Details on how the selected pixels of a frame/keyframe are used to train and update NeRF and the latent codes are described with reference toFIG.7.

FIG.7illustrates a flow diagram for training and using a spatial-temporal NeRF, conditioned on latent codes, that represents both the special and temporal domains of a dynamic scene, according to particular embodiments. The method700may begin at step710, where a computing system may initialize the latent codes respectively associated with times associated with frames in a training video of a scene captured by a camera. Then, for each of the frames, the computing system may use it to update NeRF and the latent codes. For example, at step720, the computing system may generate rendered pixel values for a set of pixels in the frame by querying NeRF using the latent code associated with the frame, a camera viewpoint associated with the frame, and ray directions associated with the set of pixels. At step730, the computing system may update the latent code associated with the frame and the NeRF based on comparisons between the rendered pixel values and original pixel values for the set of pixels. At step740, the computing system may determine whether to continue training. The process described above may continue until each frame selected for training has been used (e.g., the keyframes selected during the pre-training process or all the frames during the full training process) or until NeRF and the latent codes converge (e.g., the loss is below a predetermined threshold). If training is to continue, the process may repeat, starting at step720. If training is complete, NeRF may be used at inference/runtime. For example, at step750after training completes, the computing system may render output frames for an output video of the scene. Each output frame may be rendered by querying the updated NeRF using one of the updated latent codes corresponding to a desired time associated with the output frame, a desired viewpoint for the output frame, and ray directions associated with pixels in the output frame.

We have proposed a novel neural 3D video synthesis approach that is able to represent real-world multi-view video recordings of dynamic scenes in a compact, yet expressive representation. Our model-free representation enables both high-quality view synthesis as well as motion interpolation. At the core of our approach is an efficient algorithm to learn dynamic latent-conditioned neural radiance fields that significantly boosts training speed, leads to fast convergence, and enable high-quality results. We see our approach as a first step forward in efficiently training dynamic neural radiance fields and hope that it will inspire follow-up work in the exciting and emerging field of neural scene representations.

FIG.8illustrates an example computer system800. In particular embodiments, one or more computer systems800perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems800provide the functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems800performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems800. Herein, reference to a computer system may encompass a computing device, and vice versa, where appropriate. Moreover, reference to a computer system may encompass one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems800. This disclosure contemplates computer system800taking any suitable physical form. As example and not by way of limitation, computer system800may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (such as, for example, a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile telephone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented/virtual reality device, or a combination of two or more of these. Where appropriate, computer system800may include one or more computer systems800; be unitary or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems800may perform without substantial spatial or temporal limitation one or more steps of one or more methods described or illustrated herein. As an example and not by way of limitation, one or more computer systems800may perform in real time or in batch mode one or more steps of one or more methods described or illustrated herein. One or more computer systems800may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein, where appropriate.

In particular embodiments, computer system800includes a processor802, memory804, storage806, an input/output (I/O) interface808, a communication interface810, and a bus812. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor802includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor802may retrieve (or fetch) the instructions from an internal register, an internal cache, memory804, or storage806; decode and execute them; and then write one or more results to an internal register, an internal cache, memory804, or storage806. In particular embodiments, processor802may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor802including any suitable number of any suitable internal caches, where appropriate. As an example and not by way of limitation, processor802may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory804or storage806, and the instruction caches may speed up retrieval of those instructions by processor802. Data in the data caches may be copies of data in memory804or storage806for instructions executing at processor802to operate on; the results of previous instructions executed at processor802for access by subsequent instructions executing at processor802or for writing to memory804or storage806; or other suitable data. The data caches may speed up read or write operations by processor802. The TLBs may speed up virtual-address translation for processor802. In particular embodiments, processor802may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor802including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor802may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors802. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

In particular embodiments, memory804includes main memory for storing instructions for processor802to execute or data for processor802to operate on. As an example and not by way of limitation, computer system800may load instructions from storage806or another source (such as, for example, another computer system800) to memory804. Processor802may then load the instructions from memory804to an internal register or internal cache. To execute the instructions, processor802may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor802may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor802may then write one or more of those results to memory804. In particular embodiments, processor802executes only instructions in one or more internal registers or internal caches or in memory804(as opposed to storage806or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory804(as opposed to storage806or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor802to memory804. Bus812may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor802and memory804and facilitate accesses to memory804requested by processor802. In particular embodiments, memory804includes random access memory (RAM). This RAM may be volatile memory, where appropriate. Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory804may include one or more memories804, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

In particular embodiments, storage806includes mass storage for data or instructions. As an example and not by way of limitation, storage806may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage806may include removable or non-removable (or fixed) media, where appropriate. Storage806may be internal or external to computer system800, where appropriate. In particular embodiments, storage806is non-volatile, solid-state memory. In particular embodiments, storage806includes read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. This disclosure contemplates mass storage806taking any suitable physical form. Storage806may include one or more storage control units facilitating communication between processor802and storage806, where appropriate. Where appropriate, storage806may include one or more storages806. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

In particular embodiments, I/O interface808includes hardware, software, or both, providing one or more interfaces for communication between computer system800and one or more I/O devices. Computer system800may include one or more of these I/O devices, where appropriate. One or more of these I/O devices may enable communication between a person and computer system800. As an example and not by way of limitation, an I/O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I/O device or a combination of two or more of these. An I/O device may include one or more sensors. This disclosure contemplates any suitable I/O devices and any suitable I/O interfaces808for them. Where appropriate, I/O interface808may include one or more device or software drivers enabling processor802to drive one or more of these I/O devices. I/O interface808may include one or more I/O interfaces808, where appropriate. Although this disclosure describes and illustrates a particular I/O interface, this disclosure contemplates any suitable I/O interface.

In particular embodiments, communication interface810includes hardware, software, or both providing one or more interfaces for communication (such as, for example, packet-based communication) between computer system800and one or more other computer systems800or one or more networks. As an example and not by way of limitation, communication interface810may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface810for it. As an example and not by way of limitation, computer system800may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system800may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless network or a combination of two or more of these. Computer system800may include any suitable communication interface810for any of these networks, where appropriate. Communication interface810may include one or more communication interfaces810, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

In particular embodiments, bus812includes hardware, software, or both coupling components of computer system800to each other. As an example and not by way of limitation, bus812may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Bus812may include one or more buses812, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

Herein, a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) (such, as for example, field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM-drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these, where appropriate. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

Herein, “or” is inclusive and not exclusive, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A or B” means “A, B, or both,” unless expressly indicated otherwise or indicated otherwise by context. Moreover, “and” is both joint and several, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A and B” means “A and B, jointly or severally,” unless expressly indicated otherwise or indicated otherwise by context.

The scope of this disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the example embodiments described or illustrated herein that a person having ordinary skill in the art would comprehend. The scope of this disclosure is not limited to the example embodiments described or illustrated herein. Moreover, although this disclosure describes and illustrates respective embodiments herein as including particular components, elements, feature, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that a person having ordinary skill in the art would comprehend. Furthermore, reference in the appended claims to an apparatus or system or a component of an apparatus or system being adapted to, arranged to, capable of, configured to, enabled to, operable to, or operative to perform a particular function encompasses that apparatus, system, component, whether or not it or that particular function is activated, turned on, or unlocked, as long as that apparatus, system, or component is so adapted, arranged, capable, configured, enabled, operable, or operative. Additionally, although this disclosure describes or illustrates particular embodiments as providing particular advantages, particular embodiments may provide none, some, or all of these advantages.