Patent Publication Number: US-2023154101-A1

Title: Techniques for multi-view neural object modeling

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims priority benefit of the United States Provisional patent application titled, “MULTI-VIEW NEURAL FACE MODELING,” filed on Nov. 16, 2021 and having Ser. No. 63/280,101. The subject matter of this related application is hereby incorporated herein by reference. 
    
    
     BACKGROUND 
     Technical Field 
     Embodiments of the present disclosure relate generally to computer science and image generation and, more specifically, to techniques for multi-view neural object modeling. 
     Description of the Related Art 
     Realistic digital images of heads (e.g., human heads) are required for various computer graphics and computer vision applications. As used herein, an image of a head can include regions corresponding to the skin of a face as well as non-skin regions that can correspond to eyes, ears, scalp hair, facial hair, inside of the mouth, parts of the neck and shoulder, etc. For example, digital images of heads are oftentimes used in the virtual scenes of film productions and in video games, among other things. 
     One approach for generating digital images of heads involves capturing faces and rendering the captured faces in images. However, conventional facial capture techniques are limited to capturing the skin regions of faces, while non-skin regions are typically not captured. To generate an image of a head that includes both skin and non-skin regions, the non-skin regions that are not captured need to be filled in, or “inpainted,” after the captured skin regions are rendered. Conventional techniques for inpainting the non-skin regions oftentimes cannot generate photorealistic heads. In addition, to generate images of heads having desired inpainting details (e.g., eyes and hair that look realistic), manual effort is typically required to tune the parameters used by conventional three-dimensional (3D) modeling and two-dimensional (2D) inpainting techniques, which can be tedious and time consuming. 
     Another conventional approach for generating images of heads uses trained machine learning models to generate synthetic images of heads. While the synthetic images can look photorealistic, one drawback of conventional machine learning models is that, as a general matter, those models can only generate single images. In the context of heads, the single images are typically limited to nearly frontal views of the heads. When conventional machine learning models are used to generate images of heads from other viewpoints, such as profile views of heads, those images can suffer from inconsistencies in the head identity and appearance across viewpoints. As used herein, a “head identity” refers to aspects of the appearance of a head that are considered distinct and help differentiate one head from another head. In addition, conventional machine learning models require relatively large data sets, such as a large number of images of mostly frontal heads, to train. 
     As the foregoing illustrates, what is needed in the art are more effective techniques for generating images of heads. 
     SUMMARY 
     One embodiment of the present disclosure sets forth a computer-implemented method for rendering an image of an object. The method includes tracing a ray through a pixel into a virtual scene, and sampling one or more positions along the ray. The method further includes applying a machine learning model to the one or more positions and an identifier (ID) code associated with an object to determine, for each position included in the one or more positions, a density, a diffuse color, and specular color. In addition, the method includes computing a color of a pixel based on the density, the diffuse color, and the specular color corresponding to each position included in the one or more positions. 
     Another embodiment of the present disclosure sets forth a computer-implemented method for training a machine learning model. The method includes receiving a first set of images of one or more objects that are captured from a plurality of viewpoints. The method further includes generating a second set of images of the one or more objects from another plurality of viewpoints. In addition, the method includes training, based on the first set of images and the second set of images, a machine learning model, where the machine learning model includes a neural radiance field model and an identity model, and where the identity model maps an identifier (ID) code to (i) a deformation ID code that encodes a geometric deformation from a canonical object geometry, and (ii) a canonical ID code that encodes an appearance within a space associated with the canonical object geometry. 
     Other embodiments of the present disclosure include, without limitation, one or more computer-readable media including instructions for performing one or more aspects of the disclosed techniques as well as one or more computing systems for performing one or more aspects of the disclosed techniques. 
     At least one technical advantage of the disclosed techniques relative to the prior art is the disclosed techniques can be used to generate photorealistic images of objects, such as images of heads that include non-skin regions in addition to skin regions. The disclosed machine learning model can also generate images of multiple different heads from various viewpoints, while requiring less data to train than is required by conventional machine learning models that generate images of heads. In addition, the disclosed machine learning model can be adapted to generate images of a new target head (or other object) by fitting the machine learning model to one or more images of the new target head (or other object). These technical advantages represent one or more technological improvements over prior art approaches. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       So that the manner in which the above recited features of the disclosure can be understood in detail, a more particular description of the disclosure, briefly summarized above, can be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this disclosure and are therefore not to be considered limiting of its scope, for the disclosure can admit to other equally effective embodiments. 
         FIG.  1    illustrates a system configured to implement one or more aspects of the various embodiments; 
         FIG.  2    illustrates the frontal view of an exemplar camera system for capturing heads of individuals, according to various embodiments; 
         FIG.  3    is a more detailed illustration of the morphable radiance field (MoRF) model of  FIG.  1   , according to various embodiments; 
         FIG.  4 A  illustrates exemplar captured views of a head that can be used to train a MoRF model, according to various embodiments; 
         FIG.  4 B  illustrates exemplar synthetically generated views of a head that can be used in addition to the captured views of  FIG.  4 A  to train a MoRF model, according to various embodiments; 
         FIG.  5    illustrates different types of exemplar images of heads generated from multiple viewpoints using a MoRF model, according to various embodiments; 
         FIG.  6    illustrates exemplar images of synthetic heads generated by mixing the deformation identifier (ID) codes and canonical ID codes associated with different heads, according to various embodiments; 
         FIG.  7    illustrates exemplar images of synthetic heads generated by interpolating the deformation ID code and the canonical ID code of two heads, according to various embodiments; 
         FIG.  8    illustrates an approach for fitting a MoRF model to exemplar images of heads, according to various embodiments; 
         FIG.  9    sets forth a flow diagram of method steps for training a MoRF model, according to various embodiments; 
         FIG.  10    sets forth a flow diagram of method steps for generating an image of a head using a MoRF model, according to various embodiments; and 
         FIG.  11    sets forth a flow diagram of method steps for fitting a MoRF model to one or more images of a head, according to various embodiments. 
     
    
    
     DETAILED DESCRIPTION 
     In the following description, numerous specific details are set forth to provide a more thorough understanding of the present invention. However, it will be apparent to one of skill in the art that embodiments of the present invention can be practiced without one or more of these specific details. 
     System Overview 
       FIG.  1    illustrates a system  100  configured to implement one or more aspects of the various embodiments. As shown, the system  100  includes a machine learning server  110 , a data store  120 , and a computing device  140  in communication over a network  130 , which can be a wide area network (WAN) such as the Internet, a local area network (LAN), or any other suitable network. 
     As shown, a model trainer  116  executes on a processor  112  of the machine learning server  110  and is stored in a system memory  114  of the machine learning server  110 . The processor  112  receives user input from input devices, such as a keyboard, a mouse, a joystick, a touchscreen, or a microphone. In operation, the processor  112  is the master processor of the machine learning server  110 , controlling and coordinating operations of other system components. In particular, the processor  112  can issue commands that control the operation of a graphics processing unit (GPU) (not shown) that incorporates circuitry optimized for graphics and video processing, including, for example, video output circuitry. The GPU can deliver pixels to a display device that can be any conventional cathode ray tube, liquid crystal display, light-emitting diode display, or the like. 
     The system memory  114  of the machine learning server  110  stores content, such as software applications and data, for use by the processor  112  and the GPU. The system memory  114  can be any type of memory capable of storing data and software applications, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash ROM), or any suitable combination of the foregoing. In some embodiments, a storage (not shown) can supplement or replace the system memory  114 . The storage can include any number and type of external memories that are accessible to the processor  112  and/or the GPU. For example, and without limitation, the storage can include a Secure Digital Card, an external Flash memory, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. 
     It will be appreciated that the machine learning server  110  shown herein is illustrative and that variations and modifications are possible. For example, the number of processors  112 , the number of GPUs, the number of system memories  114 , and the number of applications included in the system memory  114  can be modified as desired. Further, the connection topology between the various units in  FIG.  1    can be modified as desired. In some embodiments, any combination of the processor  112 , the system memory  114 , and a GPU can be replaced with any type of virtual computing system, distributed computing system, or cloud computing environment, such as a public, private, or a hybrid cloud. 
     In some embodiments, the model trainer  116  is configured to train one or more machine learning models, including a morphable radiance field (MoRF) model  150 . The MoRF model  150  is a generative machine learning model that outputs densities, diffuse colors, and specular colors for positions in three-dimensional (3D) space given the positions and an identifier (ID) of a head as input. The exemplar architecture of the MoRF model  150 , and techniques for training the same, are discussed in greater detail below in conjunction with  FIGS.  3 ,  4 A,  4 B, and  8 - 11   . Training data and/or trained machine learning models, including the MoRF model  150 , can be stored in the data store  120 . In some embodiments, the data store  120  can include any storage device or devices, such as fixed disc drive(s), flash drive(s), optical storage, network attached storage (NAS), and/or a storage area-network (SAN). Although shown as accessible over the network  130 , in some embodiments the machine learning server  110  can include the data store  120 . 
     Once trained, the MoRF model  150  can be deployed for use in rendering images of heads from various viewpoints. Illustratively, an image generating application  146  that utilizes the MoRF model  150  is stored in a system memory  144 , and executes on a processor  142 , of the computing device  140 . In some embodiments, components of the computing device  140 , including the system memory  144  and the processor  142  can be similar to corresponding components of the machine learning server  110 . 
     It will be appreciated that the system  100  shown herein is illustrative and that variations and modifications are possible. For example, the number of machine learning servers and computing devices can be modified as desired. Further, the functionality included in any of the applications can be divided across any number of applications or other software that are stored and executed via any number of computing systems that are located in any number of physical locations. 
       FIG.  2    illustrates the frontal view of an exemplar camera system  200  for capturing heads of individuals, according to various embodiments. In some embodiments, the camera system  200  is used to capture data for training the MoRF model  150 , as discussed in greater detail below in conjunction with  FIGS.  4 A- 4 B and  9   . As shown, the camera system  200  includes polarized light sources  202 ,  204 ,  206 , and  208 ; cross-polarized cameras  2101 - 4  (collectively referred to herein as “cross-polarized cameras  210 ” and individually referred to herein as “a cross-polarized camera  210 ”), and cameras that are not cross-polarized  2121 - 8  (collectively referred to herein as “cameras that are not cross-polarized  212 ” and individually referred to herein as “a camera that is not cross-polarized  212 ”). 
     In operation, the polarized light sources  202 ,  204 ,  206 , and  208  produce light having a particular polarization that is directed toward an individual who is seated in front of the camera system  200 . In some embodiments, the particular polarization can be any linear polarization (e.g., horizontal polarization or vertical polarization), circular polarization (e.g., left or right circular polarization), or elliptical polarization, and any suitable light sources can be used. For example, the polarized light sources  202 ,  204 ,  206 , and  208  could include light-emitting diodes (LEDs) or studio flashes (e.g., a floor-standing light), with horizontal polarizing filters placed in front of the LEDs or studio flashes. In some embodiments, the polarized light produced by the polarized light sources  202 ,  204 ,  206 , and  208  can be substantially uniform, i.e., light that is incident on a head from different directions (e.g., front, above, below, left, and right) and that does not have any patterns. 
     The cross-polarized cameras  210  capture light having a polarization orthogonal to the particular polarization of light produced by the polarized light sources  202 ,  204 ,  206 , and  208 . For example, if light produced by the polarized light sources  202 ,  204 ,  206 , and  208  is horizontally polarized, then the cross-polarized cameras  210  could be configured to capture vertically polarized light, or vice versa. In such a case, the cross-polarized cameras  210  could be digital cameras (e.g., digital single-lens reflex (DSLR) cameras) with linear polarizing filters placed in front of the digital cameras and oriented to pass light having an orthogonal polarization to the linear polarization of light produced by the polarized light sources  202 ,  204 ,  206 , and  208 . As another example, if light produced by the polarized light sources  202 ,  204 ,  206 , and  208  is left-circularly polarized (or left-handed elliptically polarized), then the cross-polarized cameras  210  can be configured to capture right-circularly polarized (or right-handed elliptically polarized) light, or vice versa. The cross-polarized cameras  210  are color cameras (as opposed to monochrome cameras) in some embodiments. 
     The cameras that are not cross-polarized  212  capture light produced by the polarized light sources  202 ,  204 ,  206 , and  208 , including light that is not orthogonally polarized with respect to the particular polarization of light produced by the polarized light sources  202 ,  204 ,  206 , and  208 . For example, the cameras that are not cross-polarized  212  could be unpolarized cameras that capture all of the light produced by the polarized light sources  202 ,  204 ,  206 , and  208 . As another example, if light produced by the polarized light sources  202 ,  204 ,  206 , and  208  is horizontally polarized, then the cameras that are not cross-polarized  212  could be parallel-polarized cameras that capture the horizontally polarized light. In such cases, the parallel-polarized cameras can be parallel-polarized digital cameras (e.g., digital single-lens reflex (DSLR) cameras) with linear polarizing filters placed in front of the digital cameras and oriented to pass through horizontally polarized light from the light sources  202 ,  204 ,  206 , and  208 . Although some examples are described herein with respect to parallel-polarized cameras, it should be understood that techniques disclosed herein are also applicable to other types of cameras that are not cross-polarized in other ways. The cameras that are not cross-polarized  212  can either be monochrome (La, grayscale) cameras or color cameras in some embodiments. 
     Images captured by the cross-polarized cameras  210  do not include specular highlights, in contrast to images captured by the cameras that are not cross-polarized  212 . As a result, the images captured by the cross-polarized cameras  210  can be used to determine appearance properties, such as diffuse albedo, that are caused by subsurface scattering. During subsurface scattering, light bounces under the skin and is absorbed by skin pigments before bouncing back out, which gives skin a “soft” appearance. On the other hand, images captured by the cameras that are not cross-polarized  212  can be used to determine appearance properties including specular intensity and specular lobe. In contrast to subsurface scattering, specular intensity as well as the shape of specular lobes represents highlight caused by light reflecting from the surface of skin. Such highlights are white in color, assuming the light being reflected is white. 
     Because images captured by the cross-polarized cameras  210  and the cameras that are not cross-polarized  212  can be used separately to determine the non-specular and specular properties of a head, it is not necessary to strobe lights to take multiple shots. Thus, a situation in which strobing of lights is uncomfortable to an individual whose head is being captured can be avoided. Rather than taking multiple shots using strobing in a time-multiplexed manner, the camera system  200  enables view multiplexing, in which only one shot is required, because the polarization of some views are different from the polarization of others. 
     As shown, the cross-polarized cameras  210  and the cameras that are not cross-polarized  212  are arranged as triplets of cameras, each of which includes a pair of cameras that are not cross-polarized  212  and one cross-polarized camera  210 . In operation, one of the triplets of cameras can be used to capture the front of a head, another of the triplets of cameras can be used to capture the bottom of the head that includes the region under the chin as well as the chin itself and a region around the mouth, another of the triplets of cameras can be used to capture the left side of the head, and yet another of the triplets of cameras can be used to capture the right side of the head. Accordingly, the cross-polarized cameras  210  and the cameras that are not cross-polarized  212  provide substantially full coverage of the front and sides of a head. 
     Images captured by the pairs of cameras that are not cross-polarized  212  can be used to determine the 3D geometry of a head using triangulation techniques. Alternatively, the 3D geometry can be obtained in any technically feasible manner. For example, the 3D geometry could be received from another facial capture system that uses a different set of cameras, a depth camera, or some other scanning system. 
     Although four triplets of cross-polarized cameras  210  and cameras that are not cross-polarized  212  are shown for illustrative purposes, other embodiments can employ one or more cross-polarized cameras and one or more cameras that are not cross-polarized, arranged in any suitable manner, depending on the amount of facial coverage and specular information that is desired. More cross-polarized cameras, more cameras that are not cross-polarized, or an equal number of cross-polarized cameras and cameras that are not cross-polarized can be used in embodiments. In addition, any of the cross-polarized cameras and the cameras that are not cross-polarized can be arranged inside the boundaries formed by light sources, outside those boundaries, or in any other technically feasible manner (e.g., if the light sources do not form a boundary). It should be understood that diffuse color remains constant when captured by cameras at different viewpoints, but specular information can change when captured by cameras at different viewpoints. For example, one cross-polarized camera and one camera that is not cross-polarized could be used if partial facial coverage and a limited amount of specular information is acceptable (e.g., if only part of the head needs to be reconstructed). As another example, fewer than four triplets of cross-polarized cameras and cameras that are not cross-polarized could be used if the cameras are wide-angle cameras. On the other hand, more than four triplets of cross-polarized cameras and cameras that are not cross-polarized can be used to provide redundancy. In addition, the cross-polarized cameras and cameras that are not cross-polarized can be separated from each, rather than placed together in triplets, so long as complementary image data is captured by the cross-polarized cameras and the cameras that are not cross-polarized. However, pairs of cameras that are not cross-polarized should be close to each other if stereo reconstruction is required. 
     The camera system  200  can be used to capture diffuse and specular images of a head (e.g., a human head) from multiple viewpoints. The captured images can also be used to reconstruct 3D geometry corresponding to skin regions of the head, as well as to create appearance maps (e.g., diffuse albedo, specular intensity, and roughness maps), which can in turn be used to render images of skin regions of the head from different viewpoints. 
     Generating Images of Heads Using a Morphable Radiance Field Model 
       FIG.  3    is a more detailed illustration of the morphable radiance field (MoRF) model  150  of  FIG.  1   , according to various embodiments. As shown, the MoRF model  150  is a generative machine learning model that includes a deformation field model  306 , an identity model  308 , and a canonical neural radiance field (NeRF) model  316 . In some embodiments, each of the deformation field model  306 , the identity model  308 , and the canonical NeRF model  316  is a multi-layer perceptron (MLP) neural network. In operation, the MoRF model  150  takes an identifier (ID) code  304  associated with a particular head and a world space position  302  as input and outputs a density  326 , a diffuse color  328 , and a specular color  330  corresponding to the particular head and the world space position  302 . 
     As shown, the identity model  308  takes as input the ID code  304  associated with a particular head. The ID code  304  can be associated with a particular head identity and/or a particular expression (e.g., a neutral expression or a non-neutral expression). Given the ID code  304 , the identity model  308  generates a deformation ID code  310  and a canonical ID code  312 . The deformation ID code  310  is a latent code encoding a geometric deformation from a canonical head geometry. The canonical head geometry is the geometry of a standard head, which can be computed by, e.g., averaging the geometry of skin regions of multiple heads captured via a camera system (e.g., camera system  200 ). Other non-skin regions of the canonical 3D volume are either learned from training data or are defined in terms of the input canonical ID code  312 . The geometric deformation from the canonical head geometry is a non-linear mapping (i.e., a warp) of points in the 3D world space to points in a 3D space associated with the canonical head geometry, which is also referred to herein as the “shape-normalized space.” Use of the geometric deformation removes the variability in the shapes of different heads, permitting the variability in appearances across the different heads to be modeled in the shape-normalized space. The canonical ID code  312  is a latent code encoding the appearance of the head associated with the ID code  304  within the shape-normalized space. Intuitively, the deformation ID code  310  indicates where to obtain color and density information within the shape-normalized space for the head associated with the ID code  304 , and the canonical ID code  312  indicates what the colors (i.e., appearance) and densities of the head associated with the ID code  304  are in the shape-normalized space. 
     More formally, for a latent ID code  304  id∈   D  associated with a particular head, the identity model  308  maps the ID code  304  id to the deformation ID code  310  and the canonical ID code  312  as follows: 
       [ id   w   ,id   c ]= ID ( id ),  (1)
 
     where id w  is the deformation ID code  310  that encodes the geometric deformation from a canonical head geometry, and id c  is the canonical ID code  312  that encodes the canonical appearance within the shape-normalized space. Accordingly, the design of the identity model  308  defines the latent space model of equation (1). It should be understood that the identity model  308  explicitly aligns (spatially and semantically) corresponding head regions within the shape-normalized space to facilitate modeling of observed variability across different heads. Experience has shown that directly optimizing for per-head deformation ID codes id w  and canonical ID codes id c  becomes poorly constrained as the number of heads in the training data and the dimensionality of the deformation and canonical ID codes increases. Instead, the MoRF model  150  can be trained to predict the deformation and canonical ID codes, resulting in smoother and relatively disentangled embeddings for the deformation and canonical ID codes. The disentangled deformation and canonical ID codes mean that the geometry and appearance of heads can be controlled separately using the MoRF model  150 , including to generate images of heads that mix and match various geometries and appearances, discussed in greater detail below in conjunction with  FIGS.  6 - 7   . 
     The deformation field model  306  is a MLP that takes as input a concatenation of a world space position  302  and the deformation ID code  310 . During the rendering of an image via ray tracing, the world space position  302  can be a position sampled along a ray that is traced, through a pixel of the image, into a virtual scene. Based on the deformation ID code  310 , the deformation field model  306  maps the world space position  302  to a canonical 3D space position  314  in the shape-normalized space. 
     More formally, the deformation ID code  310  id w  output by the identity model  308  conditions the non-rigid deformation field MLP of the deformation field model  306  that warps the 3D world space position  302  x=(x, y, z) into the canonical 3D space, as follows: 
         x′=DF ( x;id   w ),  (2)
 
     where x′ is the canonical 3D space position  314 . The deformation field model  306  models a learned 3D deformation field that maps all heads onto the shape-normalized space that better constrains the training of each position of the canonical NeRF model  316 . Effectively, geometric variability of heads is modeled by the deformation field model  306  as smooth deformations of a canonical 3D head geometry. In some embodiments, the deformation field model  306  outputs a spatially-varying 6-dimensional vector that encodes a 3D rotation R x  and 3D translation t x . The canonical 3D space position  314  can then be obtained as x′=R x (x+t x ). The smoothness of the warp by the deformation field model  306  can be controlled based on the number of frequency bands used to position-encode the input 3D world space position  302  x. One benefit of the design of the deformation field model  306  is that the deformation field for each ID code can be partially supervised by leveraging a training dataset captured by the camera system  200  that includes multiple heads with dense semantic alignment of the skin surface across the heads, as provided by a registered common 3D geometry. 
     The canonical NeRF model  316  takes as input a concatenation of the canonical 3D space position  314  and the canonical ID code  312 . The canonical NeRF model  316  outputs a density  326 , a diffuse color  328 , and a specular color  330 . The canonical NeRF model  316  includes a MLP trunk  318 , a density MLP branch  320 , a diffuse MLP branch  322 , and a specular MLP branch  324 . The density MLP branch  320 , the diffuse MLP branch  322 , and the specular MLP branch  324  output the density  326 , the diffuse color  328 , and the specular color  330 , respectively, for the world space position  302 . 
     When a ray is traced through a pixel during rendering, the densities (e.g., density  326 ) output by the MoRF model  150  for sampled positions along the traced ray indicate where the surface of a head is located, because the density at the surface will be higher relative to the densities at other sample points. The diffuse colors (e.g., diffuse color  328 ) and specular colors (e.g., specular color  330 ) at the sampled positions along the traced ray can be averaged based on the densities at the sampled positions to compute a color for the pixel through which the ray was traced. Such a rendering process, which is also sometimes referred to as “volumetric rendering,” is discussed in greater detail below in conjunction with  FIG.  10   . 
     More formally, given the concatenation of the canonical 3D space position  314  and the canonical ID code  312  [x′, id c ], an appearance in the shape-normalized space, including a diffuse color, a specular color, and a density, can be modeled by the canonical NeRF model  316  as follows: 
       [σ, c   d   ,c   s ]= CN ( x′;id   c ),  (3)
 
     where σ is the density  326 , c d  is the diffuse color  328 , and c s  is the specular color  330 . In some embodiments, the density σ is a per-point density, the diffuse color c d  is a viewpoint-invariant diffuse RGB (red, green, blue) color, and the specular color c s ∈   k  is a local distribution of specular radiance over the full 3D sphere of allowed viewing (ray) directions r. In such cases, the distribution of the specular color c s  can be modeled within a low-frequency spherical harmonics (SH) subspace of order K. Experience has shown that the constraint K=3 for the low-frequency SH subspace can be used to enforce smoothness across viewpoints and permit the MoRF model  150  to be trained with as few as K 2  viewpoints. In addition, the canonical NeRF model  316  becomes an omnidirectional NeRF that can be evaluated once at any point x′ to provide a single output for all possible viewing directions r. The full color observed from a particular r can then be computed as c(r)=c d +c s   T SH(r), where SH(r) is the SH basis along r. For training images that are captured under (or synthetically rendered with) white light, a (scalar) neutral specular color can be added to the diffuse RGB color c d . 
     In some embodiments, the concatenation of the canonical 3D space position  314  and the canonical ID code  312  [x′, id c ] that is input into the MLP trunk  318  is also supplied to a mid layer of the MLP trunk  318  via skip links. In such cases, the task of the initial layers of the MLP trunk  318  can be understood as transforming the input into a localized canonical ID code, with the subsequent layers of the MLP trunk  318  then being conditioned by both the global and local canonical ID codes. Although the deformation field model  306  helps constrain the canonical NeRF model  316 , the warp supervision of the deformation field model  306  lacks semantic correspondences for complex variations in hair styles of different heads and optional accessories. As a result, the canonical NeRF model  316  can be required to generate new densities in some large areas that would otherwise encode empty space (e.g., areas corresponding to long hair). To enable the canonical NeRF model  316  to generate such densities, all outputs of the canonical NeRF model  316  can be conditioned on the canonical ID code  312  id c , rather than only conditioning color outputs of the canonical NeRF model  316  on the canonical ID code  312  id c . 
       FIG.  4 A  illustrates exemplar captured views of a head that can be used to train the MoRF model  150 , according to various embodiments. As shown, images of heads, such as image  404 , can be captured from multiple viewpoints  402   i  (referred to herein collectively as viewpoints  402  and individually as a viewpoint  402 ) to use as training data. For example, in some embodiments, the camera system  200  described above in conjunction with  FIG.  2    can be used to capture 12 real views of a head. In some embodiments, the images from multiple viewpoints can include images that include diffuse colors as well as specular information, as well as corresponding images that include only diffuse colors. For example, the images that include specular information and the images that include only diffuse colors can be captured using the cameras of the camera system  200  that are not cross-polarized  212  and the cameras that are cross-polarized  210 , respectively. The images that include specular information and the images that include only diffuse colors can be used to train the MoRF model  150  to output diffuse colors (e.g., diffuse color  328 ) and specular colors (e.g., specular color  330 ) separately. 
       FIG.  4 B  illustrates exemplar synthetically generated views of a head that can optionally be used in addition to the captured views of  FIG.  4 A  to train the MoRF model  150 , according to various embodiments. As shown, synthetic images of heads, such as synthetic image  414 , can be rendered from multiple viewpoints  412   i  (referred to herein collectively as viewpoints  412  and individually as a viewpoint  412 ) to use as training data. For example, given the appearance and geometry of a head captured via a camera system, such as the camera system  200 , synthetic images can be rendered from a number (e.g., 35) of viewpoints using any technically feasible rendering technique. In some embodiments, the synthetic images (e.g., synthetic image  414 ) include only skin pixels that are indicated by a skin mask (e.g., skin mask  416 ). Also shown is a diffuse image  420  corresponding to the synthetic image  414  that does not include specular information (shown in a specular image  418 ). In some embodiments, the synthetically generated images can include diffuse color images (e.g., diffuse image  420 ) that correspond to synthetic images that include both diffuse colors and specular information. 
     In some embodiments, training of the MoRF model  150  is supervised using captured images (e.g., image  404 ) of multiple heads from different viewpoints and optionally synthetically generated images (e.g., synthetic image  414 ) from additional viewpoints, with a rendering loss on foreground pixels (e.g., foreground pixels indicated by the matting  406  for the image  404  and the skin mask  416  for the synthetic image  414 ). In such cases, the training can also be supervised with density losses applied on the background, shown in black in the matting  406 , and in front of the surface using multiview stereo (MVS) depth maps and confidence maps, such as MVS depth map  408  and confidence map  410 , corresponding to the image  404 . Any suitable number of captured images and synthetically generated images can be used to train the MoRF model  150 . For example, 12 images can be captured of the head of each individual via the camera system  200 , and 35 synthetic views that include only skin pixels can be generated via ray tracing and appearance parameters captured in the 12 images. The synthetic images may not provide matting or RGB supervision for non-skin areas (e.g., hair, eyes, etc.) of heads, but experience has shown that use of synthetic images can improve convergence during training, while also constraining facial silhouettes and per-point specular outputs of the MoRF model  150 . It should be noted that synthetic images may not be required if, for example, real images are captured from enough viewpoints. 
     More formally, in some embodiments, per-point diffuse and specular colors c d  and c s  that are output by the MoRF model  150  according to equation (3) are supervised using training images I c  from parallel- and cross-polarized cameras, such as the cameras that are not cross-polarized  212  and the cross-polarized cameras  210 , described above in conjunction with the camera system  200 , as well as images from synthetic viewpoints that are generated via rendering. For a pixel p seen on camera c, let i denote the index of a point sampled along the camera ray r through p. In such a case, the rendering loss can be computed as: 
           RGB =Σ s,c,p ∥   cp −Σ i ω pi ( c   dpi   +c   spi   T   SH ( r ) )∥ 2   2 ,  (4)
 
     where ω pi  are volumetric rendering weights defined by the densities σ pi  for integrating along the ray, the indicator   is 0 when the condition   is false, and   is the set of cross-polarized cameras, and s is the index of a subject in the training image dataset. 
     In some embodiments, training of the MoRF model  150  is additionally supervised by the following deformation loss: 
           DF =λ DF Σ s,v ∥( v′+ξn ′)− DF ( v+ξn;id   w   s )∥ 2   2 ,  (5)
 
     where λ DF  is a weight and v is a vertex (with normal n) on the s-th subject&#39;s head geometry that correspond to vertex v′ (with normal n′) in the canonical 3D head geometry, and ξ˜ (0,σ) is a normal random variable with small deviation, such as σ=0.05 mm. The deformation loss of equation (5) supervises training of the deformation field model  306  within a thin volume near the surface of the mesh (face and neck), where reliable correspondences across heads are typically available. In other areas, semantic correspondence can be lacking due to the large variability in both geometry and appearance, such as different hair styles, accessories, etc. In such cases, the deformation field model  306  learns correspondences guided by a smoothness constraint (on the whole volume) and the other losses described herein. 
     In some embodiments, training of the MoRF model  150  is additionally supervised by the following density loss that promotes sparsity along the ray that is traced to render each pixel p: 
           σ =Σ s,c,p (λ m Σ i |σ pi | +λ z λ cp Σ i |σ pi |1 [z     pi     &lt;z     cp     −δ     z     ] ),  (6)
 
     where σ pi  is the density and z pi  is the depth at the i-th 3D point sampled along the ray that is traced to render pixel p, while Z cp  is the depth computed using MVS reconstruction. The weight λ z  is modulated by the per-point confidence derived from the MVS reconstruction, which typically depicts skin areas with high confidence λ cp ≈1.0. The two terms in equation (6) make use of (i) the set of foreground (head) pixels in a matting mask    c  (e.g., matting  406 ), and (ii) a MVS depth map (e.g., MVS depth map  408 ). 
     In some embodiments, training of the MoRF model  150  is additionally supervised by the following ID loss: 
           ID =λ ID Σ s   ∥id   s ∥ 2   2 ,  (7)
 
     where λ ID  is a weight that controls the id space regularization near the origin. In some embodiments, compact ID codes id are initialized from Gaussian white noise and optimized along with weight parameters of the identity model  308 , the deformation field model  306 , and the canonical NeRF model  316 , using the ID loss of equation (7). 
     In some embodiments, training of the MoRF model  150  includes performing backpropagation with gradient descent and minimizing a loss function that is a sum of the losses of equations (4)-(7), described above. In the sum, the weights λ ID , λ DF , λ m , and λ z  in equations (4)-(7) can be set to, for example, λ ID =0.1, λ DF =10, λ m =0.1, and λ z =0.5. In some embodiments, the input points x and x′, described above, can be position-encoded using 8 frequency bands, and training of the MoRF model  150  can employ a training scheme that masks higher frequency bands. In some embodiments, only the rendering loss    RGB  is applied during the early stages of training to allow the canonical NeRF model  316  to more quickly learn sufficient head geometry and appearance. In such cases, all loss terms can be enabled after a certain period of training, such as 20% of training epochs. Additionally, in some embodiments, a large distance δ z =100 mm is initially set in the density loss of equation (6) to constrain only points far from the surface of a head, and the distance δ z  can be gradually decreased during training. 
       FIG.  5    illustrates different types of exemplar images of heads generated from multiple viewpoints using the MoRF model  150 , according to various embodiments. As shown, images  502  that include new views of a head can be rendered in diffuse color using only the diffuse colors output by the MoRF model  150 . Illustratively, the images  502  are photorealistic and demonstrate multiview consistency, as the head identity and appearance are consistent across viewpoints in the images  502 . In addition, the head identity closely matches the head identity in captured images (not shown) used to train the MoRF model  150 . 
     In addition to the images  502 , images  504  that include new views of the head can be rendered in full color using diffuse and specular colors output by the MoRF model  150 . Also shown are images  506  indicating the head depth that is derived from the densities output by the MoRF model  150 , as well as images  508  that include only specular colors output by the MoRF model  150 . Illustratively, while the geometry and diffuse appearance of the head in the images  502  and  504  remain consistent, the view-dependent specular highlights in the images  502  and  508  change realistically across the rendered viewpoints. 
       FIG.  6    illustrates exemplar images of synthetic heads generated by mixing the deformation ID codes and canonical ID codes associated with different heads, according to various embodiments. As shown, by mixing deformation ID codes associated with heads  602  and canonical ID codes associated with heads  604 , images  606  rendered using the MoRF model  150  include the geometry of the heads  602  and the appearance of the heads  604 . As described above in conjunction with  FIG.  3   , deformation ID codes and canonical ID codes are intermediary latent codes for the 3D geometry and the appearance associated with a head, respectively. The deformation ID codes associated with the heads  602  and the canonical ID codes associated with the heads  604  can be input into the deformation field model  306  and the canonical NeRF model  316 , respectively, to generate the images  606 . Illustratively, the images  606  are photorealistic renderings of novel head identities, showing good disentanglement of the deformation and canonical ID codes. Although not shown, consistent images of the novel head identities can also be rendered from multiple viewpoints. 
       FIG.  7    illustrates exemplar images of synthetic heads generated by interpolating the deformation ID code and the canonical ID code of two heads, according to various embodiments. As shown, by interpolating the deformation ID codes and the canonical ID codes associated with heads  702  and  704 , images  706  can be generated that include various mixtures between the geometry of the heads  702  and  704  and between the appearances of the heads  702  and  704 . Once again, the interpolated deformation ID codes and the canonical ID codes can be input into the deformation field model  306  and the canonical NeRF model  316 , respectively, to generate the images  706 . In the images  706 , the deformation ID codes associated with the heads  702  and  704  have been interpolated along the vertical axis, and the canonical ID codes associated with the heads  702  and  704  have been interpolated along the horizontal axis. Illustratively, the images  706  are photorealistic renderings of novel head identities. Although not shown, consistent images of the novel head identities can also be rendered from multiple viewpoints. 
       FIG.  8    illustrates an approach for fitting the pre-trained MoRF model  150  to new exemplar images of heads, according to various embodiments. In some embodiments, given one or more input images and optional 3D geometry of a new head that is a target, the MoRF model  150  can be fit to the new head. Four images  802  of a head that the MoRF model  150  can be fit to are shown for illustrative purposes. 
     In some embodiments, an iterative nonlinear optimization technique is first performed to determine a fit for the ID code input into the MoRF model  150  based on the one or more images (and optional 3D geometry) of the new head. The ID code is fit first in order to initialize the deformation ID code and the canonical ID code, which are further optimized in their own subspaces via the iterative nonlinear optimization technique. Images  804  of different views of a head rendered by the MoRF model  150  after fitting of the ID code, deformation ID code, and canonical ID code are shown in  FIG.  8   . In some embodiments, the fitting is applied to downscaled versions of the one or more images of the new head (e.g., downscaled by a factor of 6) and is based on coarse renderings. To obtain relatively robust fits to images of new heads, whose pixels may correspond to “outliers” that the previously trained MoRF model  150  cannot ordinarily represent, the fitting can minimize the following losses. First, the rendering loss of equation (4) can be modified by replacing the L2-norm therein with the L1-norm. Second, a perceptual loss, such as the    LPIPS  loss, can also be applied: 
           LPIPS =λ P Σ c ∥Φ(   c ′)−Φ(Σ i ω pi ( c   dpi   +c   spi   T   SH ( r ) ))∥ F ,  (8)
 
     where λ P =0.05 and a complete image is rendered before the perceptual loss    LPIPS  is applied; Φ(⋅) denotes the set of feature activations from layers conv1-1, conv1-2, conv3-3 of a pre-trained VGG-16 network; and ∥⋅∥ is the Frobenius norm. Third, if the fitting input includes 3D geometry (in addition to one or more images), the deformation loss    DF  of equation (5) can also be applied. Fourth, the ID loss    ID  can be enforced during the initial stage of fitting the ID code. Fifth, during the subsequent stage of fitting the deformation ID code and the canonical ID code, the following ID loss can be used: 
           ID ′=λ ID (∥ id   w   −id   w   0 ∥ 2   2   +∥id   c   −id   c   0 ∥ 2   2 ),  (9)
 
     where [id w   0 , id c   0 ]=ID(id 0 ) are the codes initially predicted from the optimized code input into the identity model  308  of the MoRF model  150 . 
     Subsequent to fitting of the ID code, the deformation ID code, and the canonical ID code, the deformation ID code and the canonical ID code are frozen, and the MoRF model  150  is re-trained to fine tune the MoRF model  150  for generating images of the head in the images  802 . The re-training is similar to the initial training of the MoRF model  150 , described above in conjunction with  FIG.  4   , and the re-training updates weight parameters of the MoRF model  150  based on the images  802 . In some embodiments, the re-training uses the same rendering loss    RGB , deformation loss    DF , and density loss    σ , described above in conjunction with  FIG.  4 B , except the depth-map component of the density loss is not used. Experience has shown that the re-training can take less than 5% of the total time to train the MoRF model  150  from scratch for the head in the images  802 . After re-training using 12, 2, and 1 input images, the MoRF model  150  was applied to render images  806 ,  808 , and  810  of the head, respectively. 
       FIG.  9    sets forth a flow diagram of method steps for training the MoRF model  150 , according to various embodiments. Although the method steps are described in conjunction with the systems of  FIGS.  1 - 2   , persons of ordinary skill in the art will understand that any system configured to perform the method steps, in any order, is within the scope of the present disclosure. 
     As shown, a method  900  begins at step  902 , where the model trainer  116  receives images of one or more heads that are captured from multiple viewpoints and geometry associated with the one or more heads. In some embodiments, the images and geometry can be captured using the camera system  200 , described above in conjunction with  FIG.  2   . As described, the camera system  200  can capture images of a head from multiple viewpoints. Further, the images can include images that include diffuse colors as well as specular information and corresponding images that include only diffuse colors. In addition, the images captured by the camera system  200  can be used to reconstruct 3D geometry corresponding to skin regions of the head. 
     At step  904 , the model trainer  116  (optionally) generates synthetic images of the one or more heads from additional viewpoints using color data from the captured images and the geometry associated with the one or more heads. The synthetic images can be generated using any technically feasible rendering techniques, and can also include images that include diffuse colors as well as specular information and corresponding images that include only diffuse colors. 
     At step  906 , the model trainer  116  trains the MoRF model  150  using the captured and (optional) synthetic images. In some embodiments, training the MoRF model  150  includes performing backpropagation with gradient descent and minimizing a loss function that is a sum of the losses of equations (4)-(7), described above in conjunction with  FIG.  4   . In some embodiments, ID codes associated with the one or more heads are also learned during training. 
       FIG.  10    sets forth a flow diagram of method steps for generating an image of a head using the MoRF model  150 , according to various embodiments. Although the method steps are described in conjunction with the system of  FIGS.  1  and  3   , persons of ordinary skill in the art will understand that any system configured to perform the method steps, in any order, is within the scope of the present disclosure. 
     As shown, a method  1000  begins at step  1002 , where the image generating application  146  receives an ID code associated with a head. The ID code can be associated with one of the heads used to train the MoRF model  150 , an interpolation of multiple ID codes associated with such heads, or another ID code in the latent space of ID codes, such as a randomly sampled ID code near an origin of the latent space. 
     At step  1004 , the image generating application  146  traces a ray through a pixel into a virtual scene. The pixel is one of the pixels of an image being rendered. The ray is traced from a virtual camera placed at a desired viewpoint. 
     At step  1006 , the image generating application  146  determines densities, diffuse colors, and specular colors at a number of sample points along the traced ray by inputting the ID code and positions of the sample points into the MoRF model  150 . As described herein, given the ID code and the positions of the sample points, the MoRF model  150  outputs densities, diffuse colors, and specular colors at the sample point positions. 
     At step  1008 , the image generating application  146  computes averages of the diffuse and specular colors at the sample point positions to obtain diffuse and specular colors for the pixel. In some embodiments, each average is an alpha blending average that weights each diffuse or specular color at a sample point based on a density at that sample point, with a higher density leading to a higher weight. Aside from averaging colors, the same set of weights can be used to average depths at the sample point positions to compute and render a depth map, such as one of the images  506  described above in conjunction with  FIG.  5   . 
     At step  1010 , if there are additional pixels in the image, then the method  1000  returns to step  1002 , where the image generating application  146  traces another ray through another pixel into the virtual scene. If there are no additional pixels in the image, the method  1000  ends. 
       FIG.  11    sets forth a flow diagram of method steps for fitting the MoRF model  150  to one or more images of a head, according to various embodiments. Although the method steps are described in conjunction with the system of  FIG.  1   , persons of ordinary skill in the art will understand that any system configured to perform the method steps, in any order, is within the scope of the present disclosure. 
     As shown, a method  1100  begins at step  1102 , where the model trainer  116  receives one or more images of a head to which the MoRF model  150  will be fit. Any number of images can be used in some embodiments. More images will, as a general matter, produce better fitting results. In some embodiments, 3D geometry of the head is also received and used to fit the MoRF model  150 . 
     At step  1104 , the model trainer  116  performs an iterative nonlinear optimization technique to fit an ID code to the one or more images of the head. The iterative nonlinear optimization technique for fitting the ID code, including losses used during the optimization, are described above in conjunction with  FIG.  8   . 
     At step  1106 , the model trainer  116  performs the iterative nonlinear optimization technique to fit a deformation ID code and a canonical ID code to the one or more images of the head. In some embodiments, the iterative nonlinear optimization technique at step  1106  is initialized using a deformation ID code and a canonical ID code that correspond to the ID code determined at step  1104 . The iterative nonlinear optimization technique, including losses used during the optimization, for fitting the deformation ID code and the canonical ID code are described above in conjunction with  FIG.  8   . 
     At step  1108 , the model trainer  116  re-trains the MoRF model  150  using the one or more images of the head as training data. In some embodiments, the deformation ID code and the canonical ID code determined at step  1106  are frozen during the re-training, which is described above in conjunction with  FIG.  8   . The re-training updates parameters of the MoRF model  150 , thereby fine tuning the MoRF model  150  for generating images of the head in the one or more images received at step  1102 . 
     Although described herein primarily with respect to heads as a reference example, techniques disclosed herein can also be used to train and apply a MoRF model to render digital images of other objects, such as full bodies, body parts other than heads, internal organs, etc. 
     In sum, techniques are disclosed for generating photorealistic images of objects, such as heads, from multiple viewpoints. In some embodiments, a MoRF model that generates images of heads includes an identity model that maps an ID code associated with a head into (1) a deformation ID code that encodes a geometric deformation from a canonical head geometry, and (2) a canonical ID code that encodes a canonical appearance within a shape-normalized space. The MoRF model also includes a deformation field model that maps a world space position to a position in the shape-normalized space based on the deformation ID code. In addition, the MoRF model includes a canonical NeRF model that includes a MLP trunk and a density MLP branch, a diffuse MLP branch, and a specular MLP branch that output densities, diffuse colors, and specular colors, respectively. The MoRF model is trained using images of one or more heads that are captured from multiple viewpoints and (optionally) additional synthetically rendered images of the one or more heads. Subsequent to training, the MoRF model can be applied to render images of the one or more heads, or combinations thereof, from different viewpoints via volumetric rendering. In addition, the MoRF model can be fit to a new target head based on one or more images of the new target head and (optionally) 3D geometry of the new target head. 
     At least one technical advantage of the disclosed techniques relative to the prior art is the disclosed techniques can be used to generate photorealistic images of objects, such as images of heads that include non-skin regions in addition to skin regions. In particular, the disclosed machine learning model can generate images of multiple different heads from various viewpoints, while requiring less data to train than is required by conventional machine learning models that generate images of heads. In addition, the disclosed machine learning model can be adapted to generate images of a new target head (or other object) by fitting the machine learning model to one or more images of the new target head (or other object). These technical advantages represent one or more technological improvements over prior art approaches. 
     1. In some embodiments, a computer-implemented method for rendering an image of an object comprises tracing a ray through a pixel into a virtual scene, sampling one or more positions along the ray, applying a machine learning model to the one or more positions and an identifier (ID) code associated with an object to determine, for each position included in the one or more positions, a density, a diffuse color, and a specular color, and computing a color of a pixel based on the density, the diffuse color, and the specular color corresponding to each position included in the one or more positions. 
     2. The computer-implemented method of clause 1, wherein the machine learning model comprises an identity model that maps the ID code to (i) a deformation ID code that encodes a geometric deformation from a canonical object geometry, and (ii) a canonical ID code that encodes an appearance within a space associated with the canonical object geometry. 
     3. The computer-implemented method of clauses 1 or 2, wherein the machine learning model comprises a neural radiance field (NeRF) model that comprises a multi-layer perceptron (MLP) trunk, a first MLP branch that computes densities, a second MLP branch that computes diffuse colors, and a third MLP branch that computes specular colors. 
     4. The computer-implemented method of any of clauses 1-3, wherein computing the color of the pixel comprises averaging the diffuse color corresponding to each position included in the one or more positions based on the density corresponding to the position to determine an averaged diffuse color, averaging the specular color corresponding to each position included in the one or more positions based on the density corresponding to the position to determine an averaged specular color, and computing the color of the pixel based on the averaged diffuse color and the averaged specular color. 
     5. The computer-implemented method of any of clauses 1-4, further comprising training the machine learning model based on a set of images of one or more objects that are captured from a plurality of viewpoints. 
     6. The computer-implemented method of any of clauses 1-5, wherein the set of images include a first set of images that include diffuse colors and specular information and a second set of images that include the diffuse colors. 
     7. The computer-implemented method of any of clauses 1-6, wherein the machine learning model is further trained based on a generated set of images of the one or more objects from another plurality of viewpoints. 
     8. The computer-implemented method of any of clauses 1-7, further comprising fitting at least one of the ID code or the machine learning model to one or more images of another object. 
     9. The computer-implemented method of any of clauses 1-8, further comprising fitting the at least one of the ID code or the machine learning model to geometry associated with the another object. 
     10. The computer-implemented method of any of clauses 1-9, wherein the object is a head. 
     11. In some embodiments, one or more non-transitory computer-readable storage media include instructions that, when executed by one or more processing units, cause the one or more processing units to perform steps for rendering an image of an object, the steps comprising tracing a ray through a pixel into a virtual scene, sampling one or more positions along the ray, applying a machine learning model to the one or more positions and an identifier (ID) code associated with an object to determine, for each position included in the one or more positions, a density, a diffuse color, and a specular color, and computing a color of a pixel based on the density, the diffuse color, and the specular color corresponding to each position included in the one or more positions. 
     12. The one or more non-transitory computer-readable storage media of clause 11, wherein the machine learning model comprises an identity model that maps the ID code to (i) a deformation ID code that encodes a geometric deformation from a canonical object geometry, and (ii) a canonical ID code that encodes an appearance within a space associated with the canonical object geometry. 
     13. The one or more non-transitory computer-readable storage media of clauses 11 or 12, wherein the machine learning model comprises a neural radiance field (NeRF) model that comprises a multi-layer perceptron (MLP) trunk, a first MLP branch that computes densities, a second MLP branch that computes diffuse colors, and a third MLP branch that computes specular colors. 
     14. The one or more non-transitory computer-readable storage media of any of clauses 11-13, wherein computing the color of the pixel comprises averaging the diffuse color corresponding to each position included in the one or more positions based on the density corresponding to the position to determine an averaged diffuse color, averaging the specular color corresponding to each position included in the one or more positions based on the density corresponding to the position to determine an averaged specular color, and computing the color of the pixel based on the averaged diffuse color and the averaged specular color. 
     15. The one or more non-transitory computer-readable storage media of any of clauses 11-14, wherein the instructions, when executed by the one or more processing units, further cause the one or more processing units to perform the step of training the machine learning model based on a set of images of one or more object that are captured from a plurality of viewpoints. 
     16. The one or more non-transitory computer-readable storage media of any of clauses 11-15, wherein the set of images include a first set of images that include diffuse colors and specular information and a second set of images that include the diffuse colors. 
     17. The one or more non-transitory computer-readable storage media of any of clauses 11-16, wherein the instructions, when executed by the one or more processing units, further cause the one or more processing units to perform the step of fitting at least one of the ID code or the machine learning model to one or more images of another object. 
     18. The one or more non-transitory computer-readable storage media of any of clauses 11-17, wherein the instructions, when executed by the one or more processing units, further cause the one or more processing units to perform the step of fitting the at least one of the ID code or the machine learning model to geometry associated with the another object. 
     19. In some embodiments, a computer-implemented method for training a machine learning model comprises receiving a first set of images of one or more object that are captured from a plurality of viewpoints, generating a second set of images of the one or more object from another plurality of viewpoints, and training, based on the first set of images and the second set of images, a machine learning model, wherein the machine learning model comprises a neural radiance field model and an identity model, and wherein the identity model maps an identifier (ID) code to (i) a deformation ID code that encodes a geometric deformation from a canonical object geometry, and (ii) a canonical ID code that encodes an appearance within a space associated with the canonical object geometry. 
     20. The method of clause 19, wherein the training is based on at least one of a rendering loss, a deformation loss, a density loss, or an ID loss. 
     Any and all combinations of any of the claim elements recited in any of the claims and/or any elements described in this application, in any fashion, fall within the contemplated scope of the present invention and protection. 
     The descriptions of the various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. 
     Aspects of the present embodiments can be embodied as a system, method or computer program product. Accordingly, aspects of the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that can all generally be referred to herein as a “module” or “system.” Furthermore, aspects of the present disclosure can take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon. 
     Any combination of one or more computer readable medium(s) can be utilized. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. 
     Aspects of the present disclosure are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions/acts specified in the flowchart and/or block diagram block or blocks. Such processors can be, without limitation, general purpose processors, special-purpose processors, application-specific processors, or field-programmable. 
     The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block can occur out of the order noted in the figures. For example, two blocks shown in succession can, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. 
     While the preceding is directed to embodiments of the present disclosure, other and further embodiments of the disclosure can be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.