Patent Publication Number: US-11657546-B2

Title: End-to-end relighting of a foreground object technical

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation of U.S. patent application Ser. No. 16/823,092, filed on Mar. 18, 2020 and issued as U.S. Pat. No. 11,380,023. The aforementioned application is hereby incorporated by reference in its entirety. 
    
    
     FIELD 
     The disclosed teachings generally relate to a technique for end-to-end relighting of a foreground object (e.g., human) relative to a background of an image, in real-time or near real-time. 
     BACKGROUND 
     With the development of machine intelligence in recent years, photographs are no longer simply static images for display. In fact, digital images are a popular type of media captured by mobile devices and are routinely subject to editing. Typical image editing tools include conventional computer systems running sophisticated software (e.g., Adobe Photoshop®, After Effects™) that has a steep learning curve and, therefore, offer limited features for non-professional consumers. Popular implementations of image editing tools include applications (“apps”) for mobile devices, which are designed for user interactions on small screens (e.g., touchscreens). The interactions for image editing on mobile devices should be intuitive because of the limited interface on the small screens. Thus, image editing tools for mobile apps are substantially more limited compared to professional image editing tools. Accordingly, the algorithms to edit photographic images on modern mobile platforms need to at least partially be automated to avoid the need for intensive or professional inputs. For example, editing an image on a mobile device should preferably only require a few clicks or swipes on a touchscreen. 
     Images of human subjects are commonly modified by consumers, and a large portion of the images that are captured on mobile platforms include humans. For example, consumers would like to change the faces on an image, transfer facial expressions, replace a background, or beautify makeup on a face. However, observers can readily perceive an unrealistic effect on images that were generated with existing editing algorithms that render the images with conventional models in a conventional image editing process. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1 A  is a block diagram that illustrates a framework operable to perform end-to-end relighting of a foreground object in an image. 
         FIG.  1 B  is a flowchart that illustrates a process of end-to-end relighting of a foreground object in an image. 
         FIG.  2    is a block diagram that illustrates a segmentation process to generate segmentation data based on an image. 
         FIG.  3    is a block diagram that illustrates an example of an encoder. 
         FIG.  4    is a block diagram that illustrates an example of a residual block. 
         FIG.  5    is a block diagram that illustrates an albedo decoder and a light transport decoder. 
         FIG.  6    is a block diagram that illustrates a deconvolution block. 
         FIG.  7    is a block diagram that illustrates a light coefficients block. 
         FIG.  8    is a block diagram that illustrates a refinement block. 
         FIG.  9    is an example showing results of synthetic images that have undergone end-to-end relighting. 
         FIG.  10    is an example showing a result of a real photograph of a human that has undergone end-to-end relighting. 
         FIG.  11    illustrates a network environment that includes an editing platform that can perform end-to-end relighting in real-time or near real-time. 
         FIG.  12    is a block diagram illustrating an example of a computing system in which at least some operations described herein can be implemented. 
     
    
    
     Various features of the technologies described herein will become more apparent to those skilled in the art from a study of the Detailed Description in conjunction with the drawings. Embodiments are illustrated by way of example and not limitation in the drawings, in which like references may indicate similar elements. While the drawings depict various embodiments for the purpose of illustration, those skilled in the art will recognize that alternative embodiments may be employed without departing from the principles of the technologies. Accordingly, while specific embodiments are shown in the drawings, the technology is amenable to various modifications. 
     DETAILED DESCRIPTION 
     The disclosed techniques include an end-to-end solution to relight a foreground object on the background of an image, in real-time or near real-time. Although the disclosure includes examples that relate to a human subject in a photographic image, any other foreground object on a background of any image could benefit from the relighting techniques disclosed herein. Examples of other foreground objects include animals, vehicles, homes, food, or any other objects in an image that users desire to modify with relighting. In one example, a framework can automatically perform a segmentation process to separate a human subject from a background in a photographic image. The automatic relighting of the human subject or any other foreground object in an image can be achieved without manual inputs, or with fewer manual inputs compared to existing systems. The disclosed techniques can include different component operations that can model or infer the shading on a human subject in a photographic image, including environmental lighting, global radiance transport, and material properties to relight the image. For example, as described further below, the global radiance transport can include indirect inter-reflection between a virtual ground plane and the human body in an image. 
     As used herein, “image relighting” may refer to changing the illumination of an image to a target illumination effect without necessarily knowing the original scene geometry, material information, and an illumination condition. A relighting operation is fundamental among various image editing operations, especially in the context of editing images that include human subjects. For example, relighting may be desired when replacing a background image, for illumination beautification, or to place human subjects in a new virtual environment. 
     A conventional computer-implemented process for relighting an image of a human subject typically involves complicated manual inputs. The conventional relighting process is complicated because an editing platform is fed only a single, colored image, usually without any information related to illumination, materials, or geometries of the content in the image. Thus, a user has to manually edit the image to provide a relighting effect. As a result, an observer can readily perceive an unrealistic relighting effect because conventional relighting algorithms render an image based on unreliable manual inputs. 
     One technique for relighting includes using deep learning, which exploits a large data set prior to processing an image, to understand a setup for the image (e.g., content, features). Deep learning can provide improved results for human relighting, when built on a set of rendered images of synthesized human subjects. Existing deep learning algorithms estimate self-shadowing of a human body other than local shading of materials. That is, the algorithms are capable of rendering only direct illumination when performing a relighting process. Moreover, prior implementations for deep learning are limited because they require manual segmentation of the human body from the background image. That is, a user has to manually separate the human object in an image from the background of the image. Thus, utilizing a deep learning algorithm for relighting is still not feasible and there is no existing relighting application for broad consumer use. 
     Various embodiments and implementations of the disclosed framework include systems and methods described herein. This description provides specific details for a thorough understanding and an enabling description of these implementations. One skilled in the art will understand, however, that the invention can be practiced without many of these details. Additionally, some well-known structures or functions may not be shown or described in detail for the sake of brevity. The terminology used in the description presented below is intended to be interpreted in its broadest reasonable manner, even though it is being used in conjunction with a detailed description of certain specific implementations of the invention. 
     Although not required, implementations are described below in the general context of computer-executable instructions, such as routines executed by a general-purpose data processing device, e.g., a networked server computer, mobile device, or personal computer. Those skilled in the relevant art will appreciate that the invention can be practiced with other communications, data processing, or computer system configurations, including: Internet appliances, handheld devices, wearable computers, all manner of cellular or mobile phones, multi-processor systems, microprocessor-based or programmable consumer electronics, set-top boxes, network PCs, mini-computers, mainframe computers, media players and the like. Indeed, the terms “computer,” “server,” and the like are generally used interchangeably herein, and refer to any of the above devices and systems, as well as any data processor. 
     While aspects of the disclosed implementations, such as certain functions of the framework, can be performed exclusively or primarily on a single device, some implementations can also be practiced in distributed environments where functions or modules are shared among disparate processing devices, which are linked through a communications network, such as a local area network (LAN), wide area network (WAN), or the Internet. In a distributed computing environment, program modules can be located in both local and remote memory storage devices. 
     Aspects of the invention can be stored or distributed on tangible computer-readable media, including magnetically or optically readable computer discs, hard-wired or preprogrammed chips (e.g., EEPROM semiconductor chips), nanotechnology memory, biological memory, or other non-transitory data storage media. In some implementations, computer implemented instructions, data structures, screen displays, and other data under aspects of the invention can be distributed over the Internet or over other networks (including wireless networks), on a propagated signal on a propagation medium (e.g., an electromagnetic wave(s), a sound wave) over a period of time, or they can be provided on any analog or digital network (e.g., packet switched, circuit switched, or other scheme). 
     In one example of the disclosed solution, a framework with algorithms can provide automatic human relighting by introducing a segmentation operation in an inference neural network. The solution can perform both direct and indirect illumination. An implementation includes an application (“app”) for a mobile platform that offers an end-to-end solution for human relighting. For example, a photo-editing mobile app can include advanced lighting effects. In one instance, only direct illumination is rendered, which is assumed and limited in both training data and an inference model. The disclosed technique extends this solution to global transport such that indirect illumination is also considered. In particular, the disclosed techniques assume that there is a virtual ground plane where human subjects in images are standing and then renders with differential rendering. The virtual ground plane is a non-distant environmental setting for relighting of a human object. Therefore, the bottom part of environment lighting or a background image is taken as the illumination from ground plane, and the inter-reflection between a human body and ground plane is also rendered in accordance with the disclosed solution. 
     Radiance Transport 
     In computer graphics, realistic photographic rendering aims to solve a rendering equation. The rendering equation describes radiance leaving a point as the emitted radiance, and the sum of the reflected radiance over a hemisphere around the surface normal in that point. The rendering equation can be written in the following form:
 
 L   s ( x,ω   o   ,λ,t )= L   e ( x,ω   o   ,λ,t )+∫ Ω   f   r ( x,ω   i ,ω o   ,λ,t ) L   i ( x,ω   i   ,λ,t )(ω i   ·n ) dω   i   (1)
 
where
         x is the location in space.   ω o  is the direction of outgoing light.   λ is the wavelength of the light.   t is time.   L s (x, ω o , λ, t) is the spectral radiance of wavelength λ, along direction at time t, from the position x.   L e (x, ω o , λ, t) is the emitted spectral radiance.   n is the surface normal at x.   Ω is the unit hemisphere around the surface normal n.   ω i  is the negative direction of incoming light.   ∫ Ω ( . . . )dω i  is the integral over Ω.   f r (x, ω i , ω o , λ, t) is the bidirectional reflectance distribution function (BRDF). Amount of light reflected from ω i  to ω o , at position x, time t, and wavelength λ.   L i (x, ω i , ω o , λ, t) is the spectral radiance of wavelength λ, coming inward towards x, at time t, from direction ω i .   (ω i ·n) is a decay factor of L i  due to the incident angle.       

     Implementations of the disclosed techniques involve various assumptions. For example, the original rendering equation is assumed to have a dependence in light spectra and time. Moreover, light is assumed to travel at infinite speed and in a straight line. 
     Other assumptions relate to the visible spectrum of light, using a single frame, distant illumination, and non-emissive objects. For example, regarding the visible spectrum, the disclosed technique can sample L o  to lie only in the visible spectrum of the light wavelength. Therefore, the λ notation is avoided. Regarding a single frame, the disclosed techniques are assumed not to account for motion blur effects and aim to obtain a static image. As such, the value of time t is constant and can thus be ignored. Regarding distant illumination, the disclosed techniques use illumination that reliably represents the real-world. Therefore, L i  can be represented by an environment map captured from the real-world. Accordingly, the disclosed techniques assume that a light source has sufficient distance to a surface such that all points are at the same distance to a light source. In this way, incoming illumination becomes a function of the incident direction alone, L i (x, ω i )≈L i (ω i ). Regarding non-emissive objects, the disclosed technique assumes that humans or the clothes they wear do not emit light. Therefore, the term L e (x, ω o )=0. Based on these assumptions, and omitting the surface position, the rendering equation has the resulting form:
 
 L   s (ω o )=∫ ω   f   r (ω i ,ω o ) L   i (ω i )(ω i   ·n ) dω   i .  (2)
 
     An unbiased estimation of Equation 1 can be achieved using Monte Carlo integration, at the expense of introducing variance. However, variance can be reduced by increasing the sampling rate and providing wider sampling distributions that resemble the shape of f r . Nevertheless, achieving an unbiased, and low variance estimation in a real-time (or interactive-time) application is a challenging task. 
     Spherical Harmonics Lighting 
     The disclosed technique works with real spherical harmonics. Spherical harmonics are a mathematical method analogous to the Fourier transform over a 1D circle but defined across the surface of a sphere. They are usually defined on imaginary numbers but, for light intensity, as here, an approximation over a sphere is utilized. Hence, unless indicated to the contrary, reference to spherical harmonics herein refers to their real values. 
     Spherical harmonics are advantageous for capturing low-frequency signals and can be used in order to obtain a real-time approximation of the rendering equation. One formulation projects the spherical signals of the incoming illumination distribution L i  and the cosine decay term (ω i ·n) of the irradiance:
 
 E ( n )=∫ Ω   L   i (ω i )(ω i   ·n ) dω   i   (3)
 
and then approximates the rendering equation by calculating the radiosity on each point B(x,n), which corresponds only to the light diffusely scattered
 
 B ( x,n )=ρ( x ) E ( n )  (4)
 
where ρ is the value of the albedo at location x.
 
     By projecting the spherical harmonics of the irradiance E and then calculating the radiosity of a scene B, an approximation of the rendering equation is obtained where only the diffuse reflections are handled and, as such, the final appearance of the materials lack realism. Thus, the spherical harmonics of the radiance L s  defined in Equation (2) takes into account the BRDF of the materials f r . By using the elevation θ and azimuth ϕ angles to parametrize a unit direction vector ω i =(θ, ϕ), the incoming illumination can be decomposed as: 
     
       
         
           
             
               
                 
                   
                     
                       L 
                       i 
                     
                     ( 
                     
                       θ 
                       , 
                       ϕ 
                     
                     ) 
                   
                   = 
                   
                     
                       ∑ 
                       
                         l 
                         , 
                         m 
                       
                     
                     
                       
                         
                           ( 
                           
                             L 
                             i 
                           
                           ) 
                         
                         
                           l 
                           , 
                           m 
                         
                       
                       ⁢ 
                       
                         
                           
                             Y 
                             
                               l 
                               , 
                               m 
                             
                           
                           ( 
                           
                             θ 
                             , 
                             ϕ 
                           
                           ) 
                         
                         . 
                       
                     
                   
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
           
         
       
     
     The cosine term, together with the BRDF, can be defined as f r (ω i , ω o )(ω i ·n)=f r (ω i , ω o )cos θ=A(θ, ϕ). This can be decomposed as 
                     A   ⁡   (     θ   ,   ϕ     )     =       ∑   l         A     l   ,   m       ⁢         Y     l   ,   m       (     θ   ,   ϕ     )     .                 (   6   )               
where Y l,m  are spherical harmonics with band index l∈l≥0, and m≤2. The terms (L i ) l,m  and A l  are the coefficients for the illumination, and cosine decay respectively.
 
     The integral of Equation (2) can be rewritten in terms of spherical harmonics 
     
       
         
           
             
               
                 
                   
                     
                       L 
                       s 
                     
                     ( 
                     
                       θ 
                       , 
                       ϕ 
                     
                     ) 
                   
                   = 
                   
                     
                       ∑ 
                       
                         l 
                         , 
                         m 
                       
                     
                     
                       
                         
                           
                             
                               A 
                               ˆ 
                             
                             
                               l 
                               ⁢ 
                               m 
                             
                           
                           ( 
                           
                             L 
                             i 
                           
                           ) 
                         
                         
                           l 
                           , 
                           m 
                         
                       
                       ⁢ 
                       
                         
                           Y 
                           
                             l 
                             , 
                             m 
                           
                         
                         ( 
                         
                           θ 
                           , 
                           ϕ 
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
           
         
       
       
         
           
             
               with 
               ⁢ 
                   
               
                 A 
                 
                   l 
                   , 
                   m 
                 
               
             
             = 
             
               
                 
                   
                     4 
                     ⁢ 
                     π 
                   
                   
                     
                       2 
                       ⁢ 
                       l 
                     
                     + 
                     1 
                   
                 
               
               ⁢ 
               
                 
                   A 
                   
                     l 
                     , 
                     m 
                   
                 
                 . 
               
             
           
         
       
     
     Lastly, in order to have a fast approximation to Equation (2), the coefficients as vectors can be rewritten, where:
 
{( L   i ) l,m   }=L,  
 
and the basis function:
 
{ Â   l,m   Y   l,m   }=T,  
 
the radiance L s  of the final image can directly calculate as a dot product:
 
 L   s   =T·L.   (8)
 
     In some implementations, this should be done for each channel RGB to obtain the final image ψ. 
     Spherical Harmonics Approximation 
     Equation (8) allows for efficient approximations of L s  only when L l,m  and A l,m  have been previously precomputed. For any given distant illumination L i , (L i ) l,m  coefficients can be calculated as:
 
( L   i ) l,m =∫ θ=0   π ∫ ϕ=0   2π   L   i (θ,ϕ) Y   l,m  sin(θ) dθdϕ   (9)
 
and the coefficients A l,m  for a fixed ω o  can be calculated as:
 
 A   l,m =∫ θ=0   π ∫ ϕ=0   2π   f   r ({θ,ϕ}, w   o ) Y   l,m  sin(θ)cos(θ) dθdϕ.   (10)
 
Where Y l,m  are constant values. An unbiased estimation of (L i ) l,m  and A l,m  can be obtained using Monte Carlo integration. Nevertheless, to obtain a low variance estimation, a large number of samples are required, making this approximation step burdensome to run in real-time (or interactive-time) applications. As such, the rendering equation can be approximated in real-time only for scenes that are known beforehand (L i  and f r ({θ, ϕ}, w o )cos(θ)) are known and the coefficients have been precomputed).
 
     Generalize to Global Transport 
     The formulation of spherical harmonics lighting is based on Equation (1), which is in the form of local shading. In other words, L i  (ω i ) can include direct lighting from the emitter if not occluded, and it can also include indirect lighting because of inter-reflections. L i  (ω i ) varies with respect to different positions. Further, this is difficult to evaluate without an expensive Monte Carlo simulation. Therefore, the spherical harmonics lighting can be formulated directly with distant environmental lighting instead of local incoming radiance. Because the local incoming radiance is linearly dependent on the light source, the following equation follows:
 
 L=M{circumflex over (L)},   (11)
 
where {circumflex over (L)} is the spherical harmonics coefficients of distant lighting, and M is a square matrix to convert environmental lighting to local incoming radiance. Taking Equation (11) into Equation (8), the global illumination result as the product of spherical harmonics coefficients follows as:
 
 L   s =( M   T   T )· {circumflex over (L)}.   (12)
 
     In some embodiments, the formulation is with respect to distant environmental lighting instead of local incoming radiance. Thus, it is denoted that (M T T) and {circumflex over (L)} with T and L, respectively, for brevity. 
     The transport coefficients T vary according to individual algorithms. In some algorithms, T represents the self-occlusions of a human body. In other algorithms, T is a full global transport including inter-reflections, as well as the transport with a virtual ground plane. 
     Relighting 
     Relighting involves changing the incoming illumination L i  in a scene, for a target illumination specified by the user L′ i . As such, the integral of Equation (2) is solved with the new L′ i . However, a new image can be generated using the spherical harmonics decomposition of Equation (8), where the relighting problem consists on finding a new vector L′ from the target illumination and then evaluating the dot-product again. This will yield a new image L′ s  relight with the target illumination L′ i :
 
 L′   s   =T   T   ·L′   (13)
 
     Example Framework 
     An objective of the disclosed techniques is to relight an image with a human in it ψ, with a user-specified target illumination L′ i :
 
 R (ψ, L′   i )=ψ′  (14)
 
where R is a perfect relight function and ψ′ is the resulting relight image with illumination L′ i . Using spherical harmonics, an approximation is obtained as:
 
 L′   s   ≈T   T   ·L′.   (15)
 
     However, due to the spherical harmonics decomposition, this approximation can fail to recover high-frequency details. Moreover, the vector T encodes most information about the scene (e.g., shading, albedo, and materials), which makes it challenging to approximate the information about the scene. Therefore, the vector T can be additionally decomposed as:
 
 T=T   s ρ  (16)
 
where ρ refers to the albedo. In order to approximate the relight radiance L′ s , a dot product is calculated between the coefficients and scaled by the albedo
 
 L′   s ≈( T   s   T   ·L ′)ρ.  (17)
 
     Finally, an approximation to a ground-truth radiance for each channel can be
 
 L′   s   ≈D (( T   s   T   ·L ′)ρ)  (18)
 
where D is an enhancement function that alleviates the problems of the spherical harmonics decomposition approximating high-frequencies and brings it closer to the path-traced result. However, computing the coefficients T and L′ is not necessarily a fast operation. Therefore, solving the integral Equations (9) and (10) is avoided and, instead, the solution is approximated with the following function:
 
 T   s   ,L′,ρ≈G (ψ, L′   i ,α)  (19)
 
where G is a parametric function that takes the input image ψ and the target illumination L′ i  as the input, together with its parameters α. In particular, G can be a convolutional neural network (CNN) that can perform the relighting task in other domains or with human input and more expressive materials. Later, it will be necessary to only obtain T and L′ for each RGB channel, and perform D (T T ·L′,β) to get the final image. D is also a parametric function with parameters β. Additionally, the albedo ρ of the input image ψ is predicted. This allows for fully reconstructing the final relight radiance L′ s  for each channel and, therefore, the relight image ψ′. The variable T, as used herein, may refer to the light transport coefficients and L, as used herein, may refer to the illumination coefficients.
 
     The disclosed framework, which is based on the decomposition proposed by spherical harmonics, has two advantages. First, processing an image decomposition based on spherical harmonics (albedo, light transport, and illumination coefficients) allows relighting of new images in real-time (or near real-time), where only the coefficients of the target illumination need to be multiplied by the light transport and shading map to obtain the predicted image. The disclosed framework splits the problem of image relighting into a set of smaller problems that are easier for the model to process. Therefore, this allows for higher quality reconstructions compared to using a direct pixel-to-pixel transformation. 
       FIG.  1 A  is a block diagram that illustrates a framework operable to perform end-to-end relighting of a foreground object in an image. As shown, the framework  10  has an architecture that includes a segmentation block  12  that receives an input image  14  and outputs a segmentation mask  16 . An encoder  18  receives a masked image and outputs a feature space. An albedo decoder  20  outputs an albedo map  22  based in part on target illumination coefficients  24 - 1  L. A light transport decoder  26  outputs a light transport matrix  28  T based in part on target illumination coefficients  24 - 2  L. A light coefficients block  30  predicts the coefficients for the illumination of the image. Two refinement blocks  32 - 1  and  32 - 2  can remove noise and improve the quality of a predicted shading map  34 , and the predicted relight image  36  ψ′, respectively. 
       FIG.  1 B  is a flowchart that illustrates a process  100  of the framework  10  to perform end-to-end relighting of a foreground object in an image. As shown, the end-to-end relighting process involves receiving the image  14  including a foreground object and a background (step  102 ). In particular, the segmentation block  12  receives the image  14  to generate the segmentation mask  16  of the image  14 . In the illustrated example, the image  14  includes a human object and a background. The image  14  is received from, for example, an image editing application, a cloud-configured storage or computing system, or an additional computing system. In some cases, the image  14  includes pixels (e.g., a .jpeg, .gif, or bitmap image file). In some embodiments, the image  14  includes vector-based graphical information (e.g., an image in a .svg, .ai, or .dwg file format). In some embodiments, an image including vector-based data is converted to an image including pixel-based data. 
     The image  14  undergoes a segmentation process to automatically segment the foreground object (e.g., human object) relative to the background to thereby produce a segmented mask for the masked image (step  104 ). For example,  FIG.  2    is a block diagram that illustrates a segmentation process to generate segmentation data from an image. The segmentation process allows for automatic generation of a segmentation mask for a particular foreground object in an input image. The segmentation process obviates the need for user interaction to generate the segmentation mask. In the illustrated example, the segmentation process is composed of a multi-branch convolutional neural network that combines feature maps and refines them to predict an output. 
     More specifically, the multi-branch neural network  202  receives an input image  204  (e.g., image  14 ). The multi-branch neural network  202  includes multiple encoders, such as encoder  206  and encoder  208 . The encoders  206  and  208  are trained to extract features from an image, and to provide respective feature maps  210  and  212  based on the extracted features. In some cases, the encoders  206  and  208  have a similar structure and training, and provide feature maps  210  and  212  that include similar data describing the extracted features. In some embodiments, the encoder  206  has a structure or training, or both, that are different from the structure and training of encoder  208 . For example, the encoder  206  can have a structure that includes multiple neural network layers that are trained to extract high-level features from an input image. The encoder  206  can have a structure that includes few or one neural network layer that are trained to extract low-level features from an image. In some embodiments, additional branches of the multi-branch neural network  202  include additional neural networks, such as additional encoders having training or structure suitable for extracting additional features or types of features. 
     The encoders  206  and  208  can extract similar features, and the feature maps  210  and  212  can include similar data. In some cases, the encoders  206  and  208  extract different features or different types of features, and each of the feature maps  210  and  212  can include respective data describing the features extracted by the respective encoder. In some cases, additional branches of the multi-branch neural network  202  generate additional feature maps based on the received input image  204 . 
     The multi-branch neural network  202  combines the feature maps generated by the multiple branches. For example, the multi-branch neural network  202  generates a combined feature map  214  based on a combination of the feature maps  210  and  212 . Any suitable combination technique may be used, such as concatenation, convolution, a mathematical operation, analysis by an additional neural network, or any other suitable technique. In the multi-branch neural network  202 , the combined feature map  214  is provided to a boundary refinement module  216 . The boundary refinement module  216  can include one or more neural networks. Based on analysis of the feature data included in the combined feature map  214 , the boundary refinement module  216  determines boundaries of one or more regions of the input image  204 . For example, the combined feature map  214  includes one or more channels that are associated with one or more pixels of the input image  204 . The boundary refinement module  216  can analyze the channels, and determines whether the associated pixel(s) are included in a region of the input image  204 , such as a region depicting a human. 
     The multi-branch neural network  202  can provide segmentation data, such as segmentation data  218 . The segmentation data  218  is provided, for example, by the boundary refinement module  216 . The segmentation data  218  can be based on analysis of the combined feature map  214 . In some cases, the segmentation data  218  includes information describing related areas of the input image  204 . For example, the segmentation data  218  can include label data associated with pixels included in the input image  204 , e.g., each label indicating whether an associated pixel is part of a region depicting a human figure. 
     Referring back to  FIG.  1   , the segmentation data can be provided as the segmentation mask  16  or another digital mask. An embodiment of the segmentation mask  16  is a black-and-white image in which a pixel value indicates a label value (e.g., black corresponds to outside a region and white corresponds to inside the region). The segmentation data can include numerical data associated with the image  14 , such as a set of probabilities. Each probability can indicate, for example, a likelihood of whether a given pixel is included in a region depicting a foreground object. The segmented image is input to an encoder  18  that transforms the segmented image into a feature space (step  106 ). For example, the encoder  18  can have multiple residual blocks (e.g., 3, 4, 5) that convert the image  14  into a feature space (e.g., feature map, feature vector). 
       FIG.  3    is a block diagram that illustrates an example of an encoder  300 . The encoder  300  receives the input  302  with the foreground object that is going to be relighted in the image. As shown, the encoder  300  performs a convolution  304  followed by a series of residual blocks  306  that sequentially increase by a factor of two for the number of channels, and decrease the dimensions of the input  302  (e.g., width, height) by a factor of two. The output includes a feature map  308  of the input image that will be used in subsequent processes by the framework illustrated in  FIG.  1   . 
       FIG.  4    is a block diagram that illustrates an example of a residual block  400 . The input  402  can be the segmented image output by the encoder  300 . As shown, a residual block can be composed of two convolution branches  404 - 1  and  404 - 2  with kernel sizes of one and three, respectively. That is, one convolution branch has a greater kernel size than the other convolution branch. The output of both branches is summed up and forwarded through a non-linear leaky rectified linear unit (LReLU)  406 . 
     The feature space (e.g., feature map  308 ) output by the encoder  18  is concatenated with the precomputed coefficients  24 - 1  and  24 - 2  of a target illumination, and input to the albedo decoder  20  and the light transport decoder  26  to predict an albedo map  22  and a light transport matrix  28 , respectively (step  108 ). The feature space is concatenated with the target illumination to disentangle the decoders  20  and  26  from the target illumination. That is, this process yields ground-truth illumination coefficients to avoid the decoders needing to spend resources learning and, instead, allows them to focus on respective tasks. 
       FIG.  5    is a block diagram that illustrates processes of an albedo decoder (e.g., the albedo decoder  20 ) and a light transport decoder (e.g., light transport decoder  26 ). The albedo and transport decoders include two types of components, the residual and the deconvolutional part. The decoders receive as the output of the encoder concatenated with the illumination coefficients as their input. The residual component includes two residual blocks similar to that shown in  FIG.  4   . The output of the residual component is input to the convolutional component, which includes several deconvolutional components. 
       FIG.  6    is a block diagram that illustrates processes of a deconvolution block. As illustrated, a deconvolutional block can have a deconvolution with a kernel of size four, and an LReLU non-linearity. Each input to a deconvolutional block is concatenated with its equivalent output of the residual block in the encoder and by the equivalent output in the light transport decoder. This is done in order to disentangle each branch from solving the task of the other branches. The albedo decoder will output an image with three channels while the light transport decoder will output a tensor with as many channels as illumination coefficients used for the discussed spherical harmonics. 
     Referring back to  FIG.  1   , the output of the encoder  18  is also concatenated with the output of a residual part of each decoder  20  and  28  and fed to the light coefficients block  30 , which will predict the coefficients for the illumination of the image  14  (step  110 ). In particular, the illumination coefficients L are obtained by concatenating the output of two residual blocks in the albedo decoder  20  and the light transport decoder  26  with the feature space output by the encoder  18 , which is then forwarded to the light coefficients block  30  that outputs a tensor with size 3×Co, where Co is the number of coefficients l used in the spherical harmonics lighting decomposition. 
     In one example, the albedo decoder  20  and the light transport decoder  26  can each have two residual blocks and then five deconvolutional blocks. The albedo decoder  20  outputs the albedo map  22  ρ with size 3×H×W where H and W is the height and width of the image  14  while the light transport decoder  26  outputs the light transport matrix L with size Co×H×W. All the deconvolutional layers in the decoders  20  and  26  are concatenated with their respective convolutional layer in the encoder. Moreover, the equivalent layers are concatenated between both decoders  20  and  26  to disentangle each decoder  20  and  26 . As a result, an albedo map  22  and a light transport matrix  28  are obtained. 
       FIG.  7    is a block diagram that illustrates a process of a light coefficients block (e.g., light coefficients block  30 ). The light coefficients block predicts the illumination coefficients of the input image L i . It has a similar structure to the encoder in  FIG.  3   . The input of the light coefficients block is the output of the encoder concatenated with the output of the residual part of the albedo and light transport decoders. As shown, the light coefficients block has a convolutional layer with a kernel of size seven and then it is composed of three residual blocks. Additionally, the light coefficients block includes an average pooling after the residual blocks that ensures that the output has a size of 3×Co. 
     The light transport matrix  28  and the predicted illumination coefficients are multiplied to obtain the shading map  34  of the image  14 , which can be forwarded through a refinement block  32 - 1  that will sharpen the details of the shading (step  112 ). In particular, a matrix multiplication is performed between the light transport matrix  28  and the predicted illumination coefficients T s   T ·L to obtain the shading map  34 . This is a similar calculation as the one performed in spherical harmonics lighting, as described above. 
     The relight image  36  of the image  14  is then produced based on the albedo map  22  and the shading map  34 . In particular, the shading map  34  is scaled by the predicted albedo ρ (e.g., albedo map  22 ) to produce the relight image ψ′=(T s   T ·L)ρ (e.g., relight image  36 ). The relight image  36  can also be processed through a refinement block D (e.g., refinement block  32 - 2 ) that will denoise the relight image and improve the overall output quality. That is, the shading map  34  can go through a refinement block that generates sharper shadows and reduces the noise. 
     For example,  FIG.  8    is a block diagram that illustrates a process of a refinement block. The refinement block has an architecture that is responsible for denoising pixels of a predicted image. The framework illustrated in  FIG.  1    includes two refinement blocks. A first refinement block receives the shading as the input and the second refinement block receives the final relight image as the input. As shown, the refinement block has a simple architecture composed of 3 convolutional components. The output of the refinement component has the same size as the input. 
     Training 
     The machine learning models described herein can be trained based on synthetic data, simulated using the algorithms described above. Realistic synthetic data allows for obtaining enough ground-truth samples to train data-driven models such as convolutional neural networks. The synthetic data can include several human figures in different poses under different real-world illuminations. For each foreground object (e.g., human), the framework computes its ground-truth path-traced image, a segmentation mask, albedo map, and light transport coefficients. 
     In some embodiments, an Adam optimization algorithm can be used for training by adding a term to rectify the variance of the adaptive learning rate. In one instance, the initial learning rate has a value of 0.0002, reduced by a factor of 10 after 15, 25 and 30 epochs. The input image can be normalized to be in the range [−1, +1]. In some embodiments, the framework does not use any normalization layers in the model. In one embodiment, the batch size is one during the training. 
     Loss Function 
     To optimize a model, the framework can use a loss function composed of three losses: a total variation loss (L TV ) that reduces the noise produced by convolutional neural networks in image prediction, a reconstruction loss (L R ) that calculates a smooth L1 error (that varies between an L1 and L2 loss regarding the magnitude of the error) between the prediction and the ground-truth and allows to predict images similar to the target ones, and a perceptual loss (L P ) that compares the features of the prediction and the ground-truth produced by a convolutional network for classification and detection and improves the quality of predictions with a high-level comparison of the predicted images at the feature level. 
     The total variation loss is applied on the albedo, L TV (A), and light transport map, L TV (T). The reconstruction loss is applied on the albedo, L R (A′,A), and light transport map, L R (T′,T), refined shading, L R (S′,S), illumination coefficients, L R (L′,L), and on the final relight image, L R (I′,I). The perceptual loss is applied to the same images as the reconstruction loss: albedo, L P (A′,A), and light transport map, L P (T′,T), refined shading, L P (S′,S), illumination coefficients, L P (L′,L), and on the final relight image, L P (I′,I). The final loss is a sum of all the previous terms, equally weighted. 
     Examples 
     An application of the disclosed framework is to automatically relight an image (made, for instance, by a commercial mobile phone) with a target illumination specified by the user. For example,  FIG.  9    is an example showing results of synthetic images that are relit from a dataset.  FIG.  10    is an example showing a result of relighting of a real photograph of a human. As shown, the human is relight with three different illuminations. 
     Also, the framework has a light estimation block that is capable of estimating the input illumination L i . This can be leveraged to predict the coefficients L of the input image ψ. This feature could have an impact on immersive media such as virtual reality or augmented reality, where one could obtain the illumination of the scene and automatically relight new objects so that they appear more realistic. 
     The framework has several differences compared to prior relighting methods. For example, as described earlier, the disclosure technology uses a different deep learning architecture, which is also trained using different loss functions. Further, the framework automatically segments the human figure of an image; thus, user input is not required for segmentation. Moreover, the BRDF is included in the spherical harmonics decomposition, thereby allowing to have a much more realistic representation of materials. In contrast, prior technologies can only model diffuse objects. 
     The disclosure introduces a wider variety of materials in a training dataset, including glossy and diffuse, metallic materials and translucent materials (skin) with high levels of detail. Further, the disclosure material includes introduce a smooth L1 loss that is more robust during training than the standard L1 loss used in existing systems. The disclosure is operable to disentangle each branch (albedo, transport and albedo) by stacking the illumination coefficients to the input and by concatenating features in the decoders. This allows each branch to focus on a particular task to improve the quality of the results. Further, the disclosed encoders contain residual blocks instead of convolutional blocks which has proven to yield better results since they learn a residual mapping. Another difference is the introduction of a refinement component that significantly improves the quality of the predicted images. 
     Although the disclosed techniques describe examples in the context of images, the disclosed techniques are similarly applicable to other forms of media. For example, embodiments of the framework can output consistent and accurate video relighting. The illumination coefficients of the target illumination can be automatically predicted with the disclosed model instead of being pre-computed. 
     Editing System 
       FIG.  11    illustrates a network environment  1100  that includes an editing platform  1102  to perform relighting of an image. Individuals can interface with the editing platform  1102  via an interface  1104 . An example of an editing platform  1102  is any computing device such as a mobile device. The editing platform  1102  may be responsible for performing the end-to-end relighting of an image. For example, an individual may access the editing platform  1102  and then select, via an interface of the editing platform  1102 , images from a memory for relighting. 
     In some embodiments, the editing platform  1102  may reside in a network environment  1100 . Thus, the editing platform  1102  may be connected to one or more networks  1106   a - b . The network(s)  1106   a - b  can include personal area networks (PANs), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), cellular networks, the Internet, etc. Additionally or alternatively, the editing platform  1102  can be communicatively coupled to computing device(s) over a short-range communication protocol, such as Bluetooth® or near-field communication (NFC). 
     The interface  1104  is preferably accessible via a web browser, desktop application, mobile application, and/or over-the-top (OTT) application. Accordingly, the interface  1104  may be viewed on a personal computer, tablet computer, mobile phone, game console, music player, wearable electronic device (e.g., a watch or a fitness accessory), network-connected (“smart”) electronic device, (e.g., a television or a home assistant device), virtual/augmented reality system (e.g., a head-mounted display), or some other electronic device. 
     Some embodiments of the editing platform  1102  are hosted locally. That is, the editing platform  1102  may reside on the computing device used to access the interface  1104 . For example, the editing platform  1102  may be embodied as a desktop application executing on a personal computer. Other embodiments of the editing platform  1102  are executed by a cloud computing service operated by Amazon Web Services® (AWS), Google Cloud Platform™, Microsoft Azure®, or a similar technology. In such embodiments, the editing platform  1102  may reside on a host computer server that is communicatively coupled to one or more content computer servers  1108 . The content computer server(s)  1108  can include different types of data (e.g., graphics), user information (e.g., profiles and credentials), and other assets. Such information could also be stored on the host computer server. 
     Certain embodiments are described in the context of network-accessible interfaces. However, those skilled in the art will recognize that the interfaces need not necessarily be accessible via a network. For example, a computing device may be configured to execute a self-contained computer program that does not require network access. Instead, the self-contained computer program may cause necessary assets (e.g., graphics, matching algorithms, and processing operations) to be downloaded at a single point in time or on a periodic basis (e.g., weekly, daily, or hourly). 
     Computing System 
       FIG.  12    is a block diagram illustrating an example of a computing system  1200  in which at least some operations described herein can be implemented. For example, some components of the computing system  1200  may be hosted on a computing device that includes an editing platform (e.g., the editing platform  1102  of  FIG.  11   ). 
     The computing system  1200  may include one or more central processing units (also referred to as “processors”)  1202 , main memory  1206 , non-volatile memory  1210 , network adapter  1212  (e.g., network interface), video display  1218 , input/output devices  1220 , control device  1222  (e.g., keyboard and pointing devices), drive unit  1224  including a storage medium  1226 , and signal generation device  1230  that are communicatively connected to a bus  1216 . The bus  1216  is illustrated as an abstraction that represents one or more physical buses and/or point-to-point connections that are connected by appropriate bridges, adapters, or controllers. The bus  1216 , therefore, can include a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard  1194  bus (also referred to as “Firewire”). 
     The computing system  1200  may share a similar computer processor architecture as that of a personal computer, tablet computer, mobile phone, game console, music player, wearable electronic device (e.g., a watch or fitness tracker), network-connected (“smart”) device (e.g., a television or home assistant device), virtual/augmented reality systems (e.g., a head-mounted display), or another electronic device capable of executing a set of instructions (sequential or otherwise) that specify action(s) to be taken by the computing system  1200 . 
     While the main memory  1206 , non-volatile memory  1210 , and storage medium  1226  (also called a “machine-readable medium”) are shown to be a single medium, the term “machine-readable medium” and “storage medium” should be taken to include a single medium or multiple media (e.g., a centralized/distributed database and/or associated caches and servers) that store one or more sets of instructions  1228 . The term “machine-readable medium” and “storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the computing system  1200 . 
     In general, the routines executed to implement the embodiments of the disclosure may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively referred to as “computer programs”). The computer programs typically comprise one or more instructions (e.g., instructions  1204 ,  1208 ,  1228 ) set at various times in various memory and storage devices in a computing device. When read and executed by the one or more processors  1202 , the instruction(s) cause the computing system  1200  to perform operations to execute elements involving the various aspects of the disclosure. 
     Moreover, while embodiments have been described in the context of fully functioning computing devices, those skilled in the art will appreciate that the various embodiments are capable of being distributed as a program product in a variety of forms. The disclosure applies regardless of the particular type of machine or computer-readable media used to actually effect the distribution. 
     Further examples of machine-readable storage media, machine-readable media, or computer-readable media include recordable-type media such as volatile and non-volatile memory devices  1210 , floppy and other removable disks, hard disk drives, optical disks (e.g., Compact Disk Read-Only Memory (CD-ROMS), Digital Versatile Disks (DVDs)), and transmission-type media such as digital and analog communication links. 
     The network adapter  1212  enables the computing system  1200  to mediate data in a network  1214  with an entity that is external to the computing system  1200  through any communication protocol supported by the computing system  1200  and the external entity. The network adapter  1212  can include a network adaptor card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multilayer switch, a protocol converter, a gateway, a bridge, bridge router, a hub, a digital media receiver, and/or a repeater. 
     The network adapter  1212  may include a firewall that governs and/or manages permission to access/proxy data in a computer network and tracks varying levels of trust between different machines and/or applications. The firewall can be any number of modules having any combination of hardware and/or software components able to enforce a predetermined set of access rights between a particular set of machines and applications, machines and machines, and/or applications and applications (e.g., to regulate the flow of traffic and resource sharing between these entities). The firewall may additionally manage and/or have access to an access control list that details permissions including the access and operation rights of an object by an individual, a machine, and/or an application, and the circumstances under which the permission rights stand. 
     The techniques introduced here can be implemented by programmable circuitry (e.g., one or more microprocessors), software and/or firmware, special-purpose hardwired (i.e., non-programmable) circuitry, or a combination of such forms. Special-purpose circuitry can be in the form of one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), etc. 
     Remarks 
     The foregoing description of various embodiments of the claimed subject matter has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to one skilled in the art. Embodiments were chosen and described in order to best describe the principles of the invention and its practical applications, thereby enabling those skilled in the relevant art to understand the claimed subject matter, the various embodiments, and the various modifications that are suited to the particular uses contemplated. 
     Although the Detailed Description describes certain embodiments and the best mode contemplated, the technology can be practiced in many ways no matter how detailed the Detailed Description appears. Embodiments may vary considerably in their implementation details, while still being encompassed by the specification. Particular terminology used when describing certain features or aspects of various embodiments should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the technology with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the technology to the specific embodiments disclosed in the specification, unless those terms are explicitly defined herein. Accordingly, the actual scope of the technology encompasses not only the disclosed embodiments, but also all equivalent ways of practicing or implementing the embodiments. 
     The language used in the specification has been principally selected for readability and instructional purposes. It may not have been selected to delineate or circumscribe the subject matter. It is therefore intended that the scope of the technology be limited not by this Detailed Description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of various embodiments is intended to be illustrative, but not limiting, of the scope of the technology as set forth in the following claims.