Patent Publication Number: US-11663798-B2

Title: System and method for manipulating two-dimensional (2D) images of three-dimensional (3D) objects

Description:
TECHNICAL FIELD 
     The present disclosure generally relates to image processing, and more specifically to a system and method for manipulating two-dimensional (2D) images of three-dimensional (3D) objects. 
     BACKGROUND 
     Nowadays, image manipulation of two-dimensional (2D) images of human faces is gaining popularity in a wide range of applications. These include varying illumination conditions to make an appealing portrait image, changing identity of a person to anonymize an image, exchanging hairstyles in a virtual try-out setting, etc. In face image manipulation, careful handling of details of a face image is crucial to achieve photorealism in a synthesized face image. However, the human visual system may be sensitive to very small artifacts in the synthesized face images. Typically, in order for a 2D image of an object to be photorealistic, should be equivalent to a 2D view of a realistic 3D object (e.g., a 3D face whose shape and reflectance properties are within the range of typical human experience). Thus, in order for a modified 2D face image to remain photorealistic, it should correspond to realistic variations in a corresponding 3D object or its environment, such as variations in the apparent shape, expression, and appearance attributes of a corresponding 3D face. State-of-the-art existing methods for photorealistic automated image manipulation of 2D face images are inherently 2D—that is, such methods modify one 2D face image to form another 2D face image, without explicitly determining a 3D face that corresponds to either image. This limits the ability of such 2D methods to synthesize large variations, such as pose changes (e.g., from frontal view to profile view) of the 2D images. 
     On the other hand, existing methods for image manipulation that are inherently 3D and are trained from 3D scans of human faces can accurately perform 3D manipulations, but they are limited in their ability to generate photorealistic 2D images of the resulting 3D synthesized faces. For instance, a 3D face model may be used for reconstructing a 3D face from a 2D image of a human face. The 3D model may be obtained from a training 3D dataset of the human face. The training 3D dataset may have limited variations in the attributes of the 2D images, which may result in limited expression and limited pose control over the faces. Furthermore, the 3D training dataset may not have sufficient detail or sufficiently accurate information about the faces in the 3D training dataset to render photorealistic 2D images. Such limitation may cause lack of realism in the reconstructed 2D face images. In some cases, high-quality 3D scans of the human face may be performed for increasing 3D data of the training 3D dataset. However, the high-quality 3D scans may consume excessive amounts of time and storage resources. 
     Accordingly, there is a need for a technical solution for reconstructing and manipulating 2D images with photorealism. More specifically, there is need for manipulating 2D images of 3D objects to achieve photorealism in an efficient and feasible manner. 
     SUMMARY 
     Accordingly, it is an objective of the present disclosure to provide a system for manipulating 2D images of 3D objects. An image of a 3D object is a 2D representation that is affected by one or multiple physical properties of the 3D object. The one or multiple physical properties of the 3D object include a 3D shape of the 3D object, an albedo of the 3D shape, a pose of the 3D object, and lighting illuminating the 3D object. When there is change in a value of each of these physical properties, the 2D image representation of the object is affected. To that end, the 2D image representation of the object may be manipulated based on manipulation of each of the physical properties. Additionally or alternatively, prior to the manipulation, the 2D image may be masked by a mask image. The mask image may be automatically generated for the 2D image using a segmentation method such as a semantic segmentation network, landmark localization, or face parsing. The mask image segments regions, such as segmenting a background region from a foreground region corresponding to the 3D object in the 2D image by masking the background region from the foreground region. For instance, a human face in a 2D image may be segmented by masking background region in the 2D image using a corresponding face mask image of the human face, and may be further segmented into a hair region using a corresponding hair mask image of the hair. Such mask images may improve further processing of the 2D image representation of the 3D object. 
     Some embodiments are based on the realization that the one or multiple physical properties may be extracted from the 2D image representation of the 3D object. For example, some embodiments are based on the recognition that a neural network may be trained in one or a combination of a supervised manner and an unsupervised manner to extract the one or multiple physical properties of 3D objects from training 2D images of the 3D objects, and to reconstruct the training images back from the extracted one or multiple physical properties of the objects. The one or multiple physical properties may include a 3D shape of the 3D input object, an albedo of the 3D shape of the 3D input object, a pose of the 3D input object, and lighting illuminating the 3D input object. In some example embodiments, the neural network is pretrained in one or a combination of a supervised manner and an unsupervised manner based on a dataset comprising synthetically generated human face images. Such pretraining process may enable the neural network to capture one or more salient facial features from the synthetically generated human face images. 
     Some embodiments are based on the realization that manipulation of these extracted physical properties of 3D objects prior to reconstruction of the image may be used for a realistic image manipulation, by manipulating the properties of the 3D object and then rendering the manipulated 3D object as a 2D image, as contrasted with image manipulation directly in the 2D domain of the image. Some embodiments are based on another realization that the neural network may include multiple and/or different subnetworks. In some example embodiments, the multiple subnetworks extract different physical properties of 3D objects from the 2D images. The extraction of the different physical properties by the subnetworks allows disentangling the different physical properties in an independent or separate manner. The independent disentanglement of each of the physical properties enable in representing different variations in one or more corresponding physical properties, which in turn contribute to achieving realism in reconstructing a 2D output image from a 2D input image of a 3D object. To that end, the disentangled physical properties may be rendered into a 2D image. 
     Some embodiments are based on the realization that due to limitations in factors such as the rendering process, the representation of one or multiple physical properties, or the level and type of variations present in training data, the resulting rendered 2D image may not have a desired level of photorealism. Some embodiments are based on further realization that photorealism of the rendered 2D image may be enhanced by 2D refinement of the rendered 2D image, which is performed in some embodiments using a 2D refiner network that may be trained using an adversarial approach such as one used in training a generative adversarial network (GAN). 
     In some example embodiments, each of the one or multiple physical properties is disentangled by the trained neural network from pixels of the 2D input image using the multiple subnetworks. Each of the multiple subnetworks disentangles one or more of the one or multiple physical properties of the 3D object. The disentangled one or multiple physical properties produced by the multiple subnetworks is combined into the 2D output image using a differentiable renderer. For instance, the differentiable renderer renders a reconstructed 2D image, such as the 2D output image using the disentangled physical properties of lighting and pose of the 3D object. Such disentanglement allows designing different specialized subnetwork architectures that may adapt for different physical properties. 
     Some embodiments are based on the realization that using a differentiable renderer, rather than a more classical computer graphics renderer, enables a combination neural network that includes one or more of the subnetworks and the differentiable renderer to be trained end-to-end, so that errors in rendered images can be propagated back through the combination neural network to train one or more of the subnetworks. 
     In one embodiment, an albedo subnetwork may be trained to extract the albedo of the 3D object from pixels of the 2D input image. To that end, the albedo subnetwork may include a style-based generative adversarial network (GAN) architecture, and the network may be trained adversarially using a GAN framework. The style-based GAN may be trained using a set of photographic images of objects from a predetermined class. In some embodiments, the 3D objects may be selected from the predetermined class of human faces. 
     The style-based GAN architecture may be advantageous for generating realistic 2D images. 
     Some embodiments are based on the realization that style-based GANs, such as the StyleGAN and StyleGAN2 neural network architectures, typically generate photorealistic 2D images of objects from a predetermined class (e.g., faces) after being trained on a large dataset of 2D photographic images of objects from the predetermined class (e.g., a large database of face images) by allowing the style-based GAN&#39;s input to modulate intermediate layers of the style-based GAN generator neural network. Typically, the style-based GANs are used to directly generate 2D images that are trained, using adversarial training with a discriminator network, to be indistinguishable from photographic images from the training dataset. However, in order to generate 3D object models by the style-based GANs using only 2D photorealistic image data, output space of the style-based architecture may be part of the 3D model rather than the 2D image space of the real-world dataset. Some embodiments are based on the further realization that style-based GANs may be used to generate an albedo of a 3D object, e.g., a 2D albedo map. Such 2D albedo map may be applied to a 3D shape of the 3D object that may be further inputted into a renderer to generate a synthetic 2D output image of the 3D object. However, training a style-based GAN to generate albedo maps may require high-quality albedo maps to train the style-based GAN by applying a discriminator to its output space. Such high-quality albedo maps may be difficult or costly to obtain and may require specialized equipment. 
     Some embodiments are based on the additional realization that to generate photorealistic synthetic 2D output images without high-quality albedo maps, a discriminator network used for adversarial training may be applied after using a differentiable renderer to render the 3D object model into a 2D synthetic image. The differentiable renderer rendering the 3D object model into the 2D synthetic image before application of a discriminator may alleviate the need to apply a discriminator directly to the 2D albedo map that is output by the style-based GAN. In this way, the style based GAN architecture may be used to generate the albedo maps for 3D objects, while the discriminator network is used on the output at the end of the rendering process, in a space of 2D images. It may be possible to obtain a large amount of real-world training data (e.g., photographic images of faces) in the 2D image space. 
     Some embodiments are based on the additional realization that to generate photorealistic synthetic 2D output images without high-quality albedo maps, a discriminator network may be used for adversarial training of the style-based GAN. The discriminator network may be applied after using a differentiable renderer to render the 3D object model into a 2D synthetic image. The differentiable renderer rendering the 3D object model into the 2D synthetic image prevents directly applying the 2D albedo map to the style-based GAN. In this way, the stytle-based GAN architecture may be used to the generated albedo maps for 3D objects, while the discriminator network is used for the output at the end of the rendering process, in a space of 2D images. The 2D image space may be used for obtaining a large amount of real-world training data (e.g., photographic images of faces.) 
     Additionally or alternatively, some embodiments are based on the realization that the neural network may be trained in an unsupervised manner. The unsupervised neural network may extract the different physical properties of 3D objects into a latent space. The latent space may be used for reconstructing a physical space for the corresponding physical property. For example, an albedo subnetwork, as well as a 3D shape subnetwork, may have GAN architectures that encode pixels of the 2D image into corresponding latent spaces of each of an albedo of the object and a 3D shape of the object. These latent spaces may be used for generating physical spaces for the albedo and the 3D shape of the object. The latent space and the physical space enable a dual representation of the object. Such dual representation allows manipulating the physical property in either or both of the latent and physical space for achieving a photorealistic 2D image. In contrast, in some embodiments, the lighting and the pose subnetworks are encoded from the pixels of the 2D image directing into the corresponding physical spaces. The unsupervised training of the neural network for the extraction of the different physical properties into the latent space may preclude the need for real 3D data, which may improve overall performance and storage efficiency of the image processing system. 
     Additionally or alternatively, the image processing system may be used for generating a plurality of realistic synthetic images from the 2D images of the 3D objects. The plurality of realistic synthetic images includes variations in the one or multiple physical properties of the poses, expressions, and lighting illumination of the 3D objects. The plurality of realistic synthetic images may be used in different applications. The different applications may include an application for generating a cartoon or animated version of a user, an application for trying out different hairstyle looks on the user, a camera-enabled robotic system application, a face recovery application, and/or the like. In some example embodiments, the plurality of realistic synthetic images may be used as training data sample images for a different system (e.g., the robotic system, the face recovery application) for generating realistic training images in an efficient manner, while alleviating a need to collect the sample images using a camera associated with the corresponding system. For instance, an existing dataset for training the different system may include images of faces that do not have the same variations in the one or multiple physical properties such as pose, expression, and lighting as images that the different system processes when it is deployed. In such cases, the plurality of realistic synthetic images may be used to augment or replace the existing training dataset with images that include more examples for the different system. 
     Additionally or alternatively, the image processing system may be used to generate an anonymized version of an input image. The anonymized version may be generated using anonymization techniques, for example, replacing a face in the input image by a realistic synthetic image generated by manipulating some of the one or multiple physical properties of the input image, such as one or both of shape and albedo, while preserving others of the one or multiple physical properties such as pose and lighting. This may be used, for example, in applications in which images such as face images are to be made public but in which the users of the system do not have permission to share photographs or videos of one or more persons in the input image. 
     Additionally or alternatively, the image processing system may be used to generate an anonymized version of an input image. The anonymized version may be generated using anonymization techniques, for example, similarity-based metric. This may be used, for example, in applications in which images such as face images are to be made public but in which the users of the system do not have permission to share photographs or videos of one or more persons in the input image. 
     Accordingly, one embodiment discloses an image processing system for manipulating two-dimensional (2D) images of three-dimensional (3D) objects of a predetermined class. The image processing system comprises an input interface, a memory, a processor and an output interface. The input interface is configured to receive a 2D input image of a 3D input object of the predetermined class, and one or multiple manipulation instructions for manipulating one or multiple physical properties of the 3D input object. The one or multiple physical properties include a 3D shape of the 3D input object, an albedo of the 3D shape of the 3D input object, a pose of the 3D input object, and lighting illuminating the 3D input object. The memory is configured to store a neural network. The neural network is trained to reconstruct the 2D input image by disentangling the one or multiple physical properties of the 3D input object from pixels of the 2D input image using multiple subnetworks. The disentangled one or multiple physical properties produced by the multiple subnetworks are combined into a 2D output image using a differentiable renderer. There is one subnetwork for each subset of one or more of the one or multiple physical properties of the 3D input object, and each subnetwork of the multiple subnetworks is trained to extract a corresponding physical property from the pixels of the 2D input image into a one or a combination of latent space and physical space of the corresponding physical property. An albedo subnetwork is trained to extract the albedo of the 3D input object from the pixels of the 2D input image has a style-based generative adversarial network (GAN) architecture trained using a set of photographic images of objects from the predetermined class. The processor is configured to submit the 2D input image received from the input interface to the neural network and manipulating the latent space, the physical space, or both, of the one or multiple physical properties of the 3D input object according to the one or multiple manipulation instructions. The manipulation is performed on the disentangled one or multiple physical properties produced by the multiple subnetworks to modify the 3D input object before the differentiable renderer reconstructs the 2D output image. The output interface is configured to output the 2D output image. 
     Accordingly, another embodiment discloses a method for manipulating two-dimensional (2D) images of three-dimensional (3D) objects of a predetermined class. The method includes receiving a 2D input image of a 3D input object of the predetermined class, and one or multiple manipulation instructions for manipulating one or multiple physical properties of the 3D input object. The one or multiple physical properties include a 3D shape of the 3D input object, an albedo of the 3D input object, a pose of the 3D input object, and lighting illuminating the 3D input object. The method includes submitting the 2D input image to a neural network and manipulating the latent space, the physical space, or both, of the one or multiple physical properties of the 3D input object according to the one or multiple manipulation instructions, wherein the manipulation is performed on the disentangled one or multiple physical properties produced by the multiple subnetworks to modify the 3D input object before the differentiable renderer reconstructs the 2D output image. The neural network is trained to reconstruct the 2D input image by disentangling the physical properties of the 3D input object from pixels of the 2D input image using multiple subnetworks, and combine the disentangled physical properties produced by the multiple subnetworks into a 2D output image using a differentiable renderer. There is one subnetwork for each subset of one or more of the physical properties of the 3D input object. Each subnetwork of the multiple subnetworks is trained to extract a corresponding physical property from the pixels of the 2D input image into a one or a combination of latent space and physical space of the corresponding physical property. An albedo subnetwork trained to extract the albedo of the 3D input object from the pixels of the 2D input image has a style-based generative adversarial network (GAN) architecture trained using a set of photographic images of objects from the predetermined class. The method further includes outputting the 2D output image. 
     Further features and advantages will become more readily apparent from the following detailed description when taken in conjunction with the accompanying drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present disclosure is further described in the detailed description which follows, in reference to the noted plurality of drawings by way of non-limiting examples of exemplary embodiments of the present disclosure, in which like reference numerals represent similar parts throughout the several views of the drawings. The drawings shown are not necessarily to scale, with emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments. 
         FIG.  1    shows an exemplary representation depicting an image manipulation of an image of a 3D object, according to some embodiments of the present disclosure. 
         FIG.  2    shows a block diagram of a system for manipulating two-dimensional (2D) images of three-dimensional (3D) objects of a predetermined class, according to one example embodiment of the present disclosure. 
         FIG.  3    shows a framework of a neural network for manipulating 2D images of 3D objects of a predetermined class, according to one example embodiment of the present disclosure. 
         FIG.  4    shows a flowchart illustrating a process for manipulating 2D images of 3D objects of a predetermined class, according to one example embodiment of the present disclosure. 
         FIG.  5    shows a framework corresponding to a hair-manipulation algorithm, according to one example embodiment of the present disclosure. 
         FIGS.  6 A and  6 B , collectively, show a method for manipulating 2D images of 3D objects of a predetermined class, according to one example embodiment of the present disclosure. 
         FIG.  7    shows a block diagram of a system for manipulating 2D images of 3D objects of a predetermined class, according to one example embodiment of the present disclosure. 
         FIG.  8    illustrates a use case scenario for manipulating 2D images of 3D objects of a predetermined class, according to one example embodiment of the present disclosure. 
         FIG.  9    illustrates a use case scenario for manipulating 2D images of 3D objects of a predetermined class, according to another example embodiment of the present disclosure. 
         FIG.  10    illustrates a use case scenario for manipulating 2D images of 3D objects of a predetermined class, according to another example embodiment of the present disclosure. 
         FIG.  11    illustrates a use case scenario for manipulating 2D images of 3D objects of a predetermined class, according to yet another example embodiment of the present disclosure. 
     
    
    
     While the above-identified drawings set forth presently disclosed embodiments, other embodiments are also contemplated, as noted in the discussion. This disclosure presents illustrative embodiments by way of representation and not limitation. Numerous other modifications and embodiments can be devised by those skilled in the art which fall within the scope and spirit of the principles of the presently disclosed embodiments. 
     DETAILED DESCRIPTION 
     In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, to one skilled in the art that the present disclosure may be practiced without these specific details. In other instances, apparatuses and methods are shown in block diagram form only in order to avoid obscuring the present disclosure. 
     As used in this specification and claims, the terms “for example,” “for instance,” and “such as,” and the verbs “comprising,” “having,” “including,” and their other verb forms, when used in conjunction with a listing of one or more components or other items, are each to be construed as open ended, meaning that the listing is not to be considered as excluding other, additional components or items. The term “based on” means at least partially based on. Further, it is to be understood that the phraseology and terminology employed herein are for the purpose of the description and should not be regarded as limiting. Any heading utilized within this description is for convenience only and has no legal or limiting effect. 
     Overview 
     The proposed image processing system enables reconstruction of a 2D output image from a 2D input image of a 3D object of a predetermined class, such as a human face. It is an objective of the image processing system to manipulate the 2D input image for the reconstructing of the 2D output image. The 2D input image is manipulated by disentangling one or multiple physical properties of the 2D input image, such as a 3D shape of the 3D object, an albedo of the 3D object, a pose of the 3D object, and lighting illuminating the 3D object, which is further explained in  FIG.  1   . 
       FIG.  1    shows an exemplary representation  100  depicting an image manipulation of an image  102  of a 3D object, according to some embodiments of the present disclosure. The image  102  is a 2D representation of the 3D object, such as a human face. The image  102  possesses one or multiple physical properties that may be extracted from the 2D representation of the human face. In some example embodiments, the one or multiple physical properties may be extracted from pixel values of the 2D image  102 . The extracted one or multiple physical properties may be explicitly represented as: an estimated 3D shape  104 A of the 3D input object; an estimated albedo  104 B of the 3D input object, an estimated pose  104 C of the 3D input object, and an estimated lighting  104 D illuminating the 3D input object. The explicit representation of each of the one or multiple physical properties, collectively referred to hereinafter as one or multiple physical properties  104 A- 104 D, enable manipulation of each of the one or multiple physical properties  104 A- 104 D in an independent manner. The manipulation herein corresponds to change in values (e.g., pose angles or lighting parameters) of each of the corresponding one or multiple physical properties  104 A- 104 D. When there is change in values of the one or multiple physical properties, the 2D representation of the image  102  is manipulated. In some example embodiments, such independent manipulation of the one or multiple physical properties  104 A- 104 D may be used for generating a plurality of realistic synthetic images from the 2D images of the 3D objects. The plurality of realistic synthetic images may correspond to variations in the one or multiple physical properties  104 A- 104 D of the 3D objects. 
     The one or multiple physical properties  104 A- 14 D are combined to generate a 3D model of the object, which may be rendered to produce a 2D synthetic image  106 A that resembles the input image  102 . Manipulation of one or more of the one or multiple physical properties  104 A- 104 D corresponds to changes in the 3D object model, which cause corresponding manipulations in the 2D synthetic image  106 A. Such manipulation prior to rendering the 3D model as a reconstructed synthetic 2D image  106 A enables an image manipulation of the image  102  in a realistic manner. In some embodiments, the reconstructed synthetic 2D face model  106 A is combined with a hair model  106 B to generate a final output image, such as an output image  108 , which is a photorealistic synthetic 2D image corresponding to a manipulated version of the input image  102 . 
     Further, the changes in each of the one or multiple physical properties  104 A- 104 D generate different manipulations, such as a lighting variation  110 , a facial expression variation  112 , a pose variation  114 , a shape variation  116 , and a texture variation  118  of the image  102 . The different manipulations of the physical properties  104 A- 104 D may be used to synthesizing photorealistic 2D images that correspond to variations in properties such as pose, facial expression, and lighting conditions of the input image  102 . 
     Such image manipulation of the 2D representation of the image  102  through the disentanglement of each of the one or multiple physical properties  104 A- 104 D is performed by a system, which is described further with reference to  FIG.  2   . 
       FIG.  2    shows a block diagram of a system  200  for manipulating 2D images of 3D objects of a predetermined class, according to one example embodiment of the present disclosure. The system  200  corresponds to an image processing system that includes an input interface  202 , a memory  204 , a processor  206  and an output interface  208 . The input interface  202  is configured to receive a 2D input image, such as the image  102  of a 3D input object, i.e., the human face. The input interface  202  is also configured to receive one or multiple manipulation instructions for manipulating one or multiple physical properties, such as the one or multiple physical properties  104 A- 104 D of the 3D input object that include a 3D shape of the 3D input object (e.g., the shape  104 A), an albedo of the 3D input object (e.g., the albedo  104 B), a pose of the 3D input object (e.g., the pose  104 C), and a lighting property illuminating the 3D input object (e.g., the lighting  104 D). 
     The memory  204  is configured to store a neural network  210 . The neural network  210  is trained to reconstruct the image  102  by disentangling the one or multiple physical properties  104 A- 104 D. The one or multiple physical properties  104 A- 104 D are disentangled from pixels of the image  102 . In some embodiments, multiple subnetworks are used for disentangling the one or multiple physical properties  104 A- 104 D from the pixels. There is one subnetwork for each of one more subsets of the one or multiple physical properties  104 A- 104 D of the 3D object. Each subnetwork of the one or multiple subnetworks is trained to extract a corresponding physical property from pixels of the image  102  into a one or a combination of latent space and physical space of the corresponding physical property. For instance, an albedo subnetwork is trained to extract the albedo  104 B of the 3D shape  104 A of the 3D input object from the pixels of the image  102 . In some embodiments, the albedo subnetwork may have a style-based generative adversarial network (GAN) architecture. In some example embodiments, the style-based GAN may be trained using a set of photographic images of objects from the predetermined class. 
     Further, each of the individual disentangled physical properties is combined into a 2D output image. In some embodiments, the individual disentangled physical properties are combined using a differentiable renderer. The differentiable renderer renders the 2D output image of the image  102  using physical spaces of the one or more physical properties  104 A- 104 D. 
     The processor  206  is configured to submit the image  102  received from the input interface  202  to the neural network  210 , while manipulating the latent space, the physical space, or both, of the one or multiple physical properties of the 3D input object according to the one or multiple manipulation instructions. The processor  206  is configured to manipulate the disentangled physical properties by the multiple subnetworks to modify the 3D input object so that the differentiable renderer reconstructs the 2D output image. The 2D output image is outputted via the output interface  208 . 
     The disentangling of each of the one or multiple physical properties by the corresponding subnetwork of the neural network  210  is described further with reference to  FIG.  3   . 
       FIG.  3    shows a framework  300  of the neural network  210  for manipulating 2D images of 3D objects of a predetermined class, according to one example embodiment of the present disclosure. When the image  102  is received, the processor  206  generates a mask image  302  for the image  102 . In some example embodiments, the processor  206  may estimate the mask image  206  using a semantic segmentation network. The semantic segmentation network may generate the mask image  302  that includes some pixels that represent background (e.g., pixels with value zero) and remaining pixels that represent foreground (e.g., pixels with non-zero values such as the value 1.0). The image  102  is masked with the mask image  302  such that the values of pixels that represent the 3D object in the image  102  are retained in the masked image  304 , while the values of pixels that represent the background and (in the case of faces) hair in the image  102  are masked by the pixel values that are zero. The masking of the segmented background (and hair) pixel values yields a masked image  304  that contains the pixel values that represent the object of the predetermined class (e.g., face) in the image  102 . This masked image  304  is submitted to the neural network  210 . 
     The neural network  210  includes multiple subnetworks disentangling the one or multiple physical properties  104 A- 104 D of the image  102 . In some example embodiments, the multiple subnetworks correspond to encoder architectures that includes a shape encoder  306 A, an albedo encoder  306 B, a pose encoder  306 C, and a lighting encoder  306 D (referred to hereinafter as encoders  306 A- 306 D), as shown in  FIG.  3   . Each of the multiple subnetworks, i.e., each of the encoders  306 A- 306 D, processes the image  102  to extract each of the corresponding one or multiple physical properties  104 A- 104 D from pixels of the masked image  304 . For instance, the encoder  306 A extracts the shape  104 A, the encoder  306 B extracts the albedo  104 B, the encoder  306 C extracts the pose  104 C and the encoder  306 D extracts the lighting  104 D. 
     The one or multiple physical properties  104 A- 104 D that are extracted from the image  102  may be represented in one or a combination of latent space and physical space of the corresponding physical property. For instance, the shape encoder  306 A extracts a shape code  308 A in a shape latent space, and the albedo encoder  306 B extracts an albedo code  308 B in an albedo latent space. In contrast, the pose encoder  306 C extracts a pose representation  308 C in a pose physical space, and the encoder  306 D extracts a lighting representation  308 D in a lighting physical space. Each of the shape code  308 A and the albedo code  308 B represents a compressed state of the corresponding physical property, i.e., the shape  104 A and the albedo  104 B. 
     Further, the shape code  308 A and the albedo code  308 B are respectively given as input to a corresponding shape generator  310 A and albedo generator  310 B, to respectively generate a shape representation  312 A in shape physical space and an albedo representation  312 B in an albedo physical space. In some example embodiments, the shape generator  310 A corresponds to a convolutional generator that generates a 3D shape  312 A of the image  304  from the shape latent space  308 A. For instance, the generated 3D shape  312 A may be composed of 3 channels in a texture mapping space, such as a UV-space that represents 3D coordinates of vertices of the 3D shape  312 A. In some example embodiments, the albedo representation  312 B is represented as an albedo map, which can be used to determine an albedo at specific or arbitrary locations on the 3D shape  312 A. For example, the albedo representation  312 B may correspond to a Red-Green-Blue (RGB) albedo map in the UV space. In some embodiments, the albedo representation  312 B includes one or more channels for specular reflection. For example, the albedo representation  312 B may correspond to a Red-Green-Blue-Specular albedo map with 4 channels (red diffuse, green diffuse, blue diffuse, and specular) in the UV space. Different variations in the albedo code  308 B in the albedo latent space may correspond to different variations of the albedo map of the albedo representation  312 B in the albedo physical space for achieving a photorealistic image in a final output image of the image manipulation. To that end, in some embodiments, the albedo generator  310 B corresponds to a style-based generative adversarial network (GAN). 
     In some example embodiments, the lighting physical space  308 C may be represented using a spherical harmonic parameterization. The spherical harmonic parameterization enables recovery of a physical property corresponding to lighting illuminating a 3D object (such as the lighting  104 D) of the image  102  that can approximate arbitrary unknown lighting conditions. In some example embodiments, the pose latent space  308 D may correspond to a 6-degree-of-freedom (6-DOF) pose vector that includes 3 parameters for 3D rotation via an axis-angle representation and 3 parameters for a 3D translation. 
     The processes starting from processing the image  102 , yielding the masked image  304 , extracting each of the corresponding one or multiple physical properties  104 A- 104 D from pixels by the encoders  306 A- 306 D, generating shape representation  312 A and albedo representation  312 B, to outputting the rendered image  316  corresponds to a face reconstruction pipeline  350 , as shown in  FIG.  3   . In some embodiments, the face reconstruction pipeline  350  may not include using a mask image  302 . 
     The generated 3D shape  312 A, the albedo map  312 B, the pose physical space  308 A and the lighting physical space  308 D are given as input to a differentiable renderer  314 . The differentiable renderer  314  produces a rendered image  316  of the 3D face model from the 3D shape  312 A and the albedo representation  312 B using the physical space pose representation  308 C and the physical space lighting representation  308 D. 
     The processes starting from processing the image  102 , yielding the masked image  304 , extracting each of the corresponding one or multiple physical properties  104 A- 104 D from pixels by the encoders  306 A- 306 D, generating shape representation  312 A and albedo representation  312 B, to outputting the rendered image  316  corresponds to a face reconstruction pipeline  350 , as shown in  FIG.  3   . 
     Further, the rendered image  316  of the 3D face model is masked using the mask image  302  to generate a reconstructed masked rendered face image  318 . The reconstructed masked rendered face image  318  is combined with a reconstructed 2D masked hair image  330 . In parallel, the masked hair image  330  is reconstructed from the image  102  using a hair-manipulation algorithm. In some embodiments, the hair-manipulation algorithm corresponds to an encoder-generator architecture that includes a hair encoder  320  and a hair generator  324 . The encoder  320  extracts hair features  322  corresponding to hair from the image  102 . The extracted hair features  322  are provided to the hair generator  324  as input. The hair generator  324  generates a hair image in 2D, such as hair image  326 . The hair image  326  is masked with a hair mask  328  to generate the masked hair image  330 . 
     The combination of the masked rendered face image  318  and the masked hair image  330  generate a combined synthetic face image  332  from the face in the image  102 . In some embodiments, the reconstructed combined synthetic face image  332  is inputted to a refiner network  334  to generate a photorealistic refined synthetic face image  336  as output of the image manipulation of the image  102 . 
     For a given set of N portrait images (x i ) along with face masks (M f   i ) and hair masks (M h   i ) denoted as {(x i ,M f   i ,M h   i )} i=1   N , the following optimization problem is solved: 
                                     arg   ⁢        min       {       E   ◻     ,     E   β     ,     E   γ     ,   ,     G   ◻     ,     G   β     ,   R     }                i   =   1                X   N             ❘   &#34;\[RightBracketingBar]&#34;             ❘   &#34;\[RightBracketingBar]&#34;         ⁢         x     i     ⊙     (       M   f   i     +     M   h   i       )         -     x   i           ❘   &#34;\[RightBracketingBar]&#34;             ❘   &#34;\[RightBracketingBar]&#34;         1           (   1   )               
where each final image (e.g., photorealistic refined synthetic face image  336 ) is denoted as {circumflex over (x)}=R({circumflex over (x)} f  ⊙M f +{circumflex over (x)} h  ⊙M h ) with {circumflex over (x)} f =Φ(G α (E α (x′)),G β (E β (x′)), E γ (x′), E θ (x′)) representing a reconstructed image, such as the rendered face image  316 , and {circumflex over (x)} h =G h  (E h  (x)) represents the reconstructed hair image, such as the hair image  326 . The term x denotes an input image, such as the image  102 , M f  denotes a face mask (e.g., the mask image  302 ), and x′:=x⊙M f  denotes the masked image  304 . The term M h  denotes a hair mask, such as the hair mask image  328 , and thus the reconstructed 2D masked hair image  330  may be expressed as {circumflex over (x)} h ⊙M h . The term E α  denotes the shape encoder  306 A that extracts the shape code  308 A (denoted by α), E β  denotes the albedo encoder  306 B that extracts the albedo code  308 B (denoted by β), E θ  denotes the pose encoder  306 C that extracts the pose representation  308 C (denoted by {circumflex over (θ)}) and E γ  denotes the lighting encoder  306 D that extracts the lighting representation  308 D (denoted by {circumflex over (γ)}). Further, the term G α  denotes the shape generator  310 A, which generates the 3D shape representation  312 A (denoted by Ŝ) and G β  denotes the albedo generator  310 B, which generates the albedo representation  310 B (denoted by Â). The differentiable renderer  314 , denoted ϕ, renders the rendered image  316  of the 3D face model {Ŝ, Â} using the physical space pose representation  308 C ({circumflex over (θ)}) and the physical space lighting representation  308 D ({circumflex over (γ)}) to output the rendered face image denoted as {circumflex over (x)} f ) {circumflex over (x)} f =Φ(Ŝ, Â, {circumflex over (γ)}, {circumflex over (θ)}).
 
     In some example embodiments, the albedo generator  310 B corresponds to a style-based Generative Adversarial Network (style-based GAN) generator, for example a (StyleGAN or StyleGAN2 neural network architecture). To that end, the system  300  may be trained adversarially with a GAN framework using a set of photographic images of human faces to generate photorealistic output images  336  that exhibit different variations of the human face in the image  102 . 
     The style-based GAN generator may be used to generate realistic images from a predetermined class by allowing the input of the style-based GAN (e.g., the albedo code  308 B) to modulate intermediate layers of the generator neural network (e.g., albedo generator  310 B). When data from albedo maps are available (e.g., during pretraining using synthetic data), an adversarial discriminator network may be applied directly to the output (e.g., the albedo map  312 B) of the style-based GAN (e.g., the albedo generator  310 B). In some embodiments, an adversarial discriminator network is applied to an output or intermediate product of the framework  300  that is downstream of or derived from the output of the style-based GAN. For example, an adversarial discriminator that is used to help train the style-based GAN may be applied to an output or intermediate product that is in the space of 2D images of 3D objects (e.g., 2D images of faces), such as the refined synthetic face image  336 . 
     In some embodiments, one or more discriminator networks may be used to train the style-based GAN architecture of the albedo generator  310 B as well as other generators in the framework  300 , such as one or more of the shape generator  310 A and the hair generator  324 . The discriminator networks may be applied at one or multiple stages of the framework  300 . In some embodiments, for example, a discriminator network is applied to outputs or intermediate images of the framework  300  after the differentiable renderer, such as one or more of the rendered image  316 , the masked rendered face image  318 , the combined synthetic face image  332 , and the refined synthetic face image  336 . 
     In some embodiments, part or the entire neural network  210 , for example, the face reconstruction pipeline  350 , may be pretrained, which is further described in  FIG.  4   . 
       FIG.  4    shows a flowchart illustrating a process  400  for manipulating 2D images of 3D objects of a predetermined class, according to one example embodiment of the present disclosure. The process  400  starts at step  402 . The steps of the process  400  may be executed by the processor  206 . 
     Step  404  corresponds to pretraining the face reconstruction pipeline  350  of the framework  300  of  FIG.  3   . The face reconstruction pipeline  350  is performed by encoders and generators, such as the encoders  306 A- 306 D and the shape generator  310 A and the albedo generator  310 B, as well as the differentiable renderer  314 . To that end, at step  404 , the encoders  306 A- 306 D and the generators  310 A and  310 B of the neural network  210 , are pretrained using a pretraining dataset. For instance, the face reconstruction pipeline  350  is trained to input images from the pretraining dataset as the input image  304  and output rendered images, such as the rendered image  316 , as a reconstruction of the input image  304 . In some embodiments, the pretraining input images  304  and the pretraining output images  316  are not masked. The face reconstruction pipeline  350  may be pretrained using a dataset of synthetically generated face images, such as ones generated by a linear 3D morphable model (3DMM). In such pretraining, the face reconstruction pipeline  350  learns to capture features of faces using the synthetically generated face images. For instance, 80,000 2D face images for pretraining may each be rendered by randomly sampling a 3D head model from a 3DMM, such as the Faces Learned with an Articulated Model and Expressions (FLAME) model, and rendered the sampled model into a 2D image. Before each sampled 3D model is rendered to produce a training set image, it may be translated in 3D so that the resulting 2D face image in the training set correspond to the same 2D alignment as the faces in a dataset of real face images. The dataset of real face images is also used during training step  406 . Further, ground truth values for disentangled one or multiple physical properties are used to generate each of the images in the pretraining set, such as ground truth values corresponding to the one or multiple physical properties  104 A- 104 D. Such ground truth values may be used to guide the pretraining process by imposing reconstruction losses between the ground truth and estimated values of the physical properties, including the shape representation  312 A (L shape   syn )  312 A, the albedo representation  312 B (L albedo   syn ), the lighting representation in the lighting physical space  308 D (L lighting   syn ), and the pose representation in the pose physical space  308 C (L pose   syn ), which are represented in the following equations.
 
 L   image   syn   =P   x   −{circumflex over (x)}   f   P   2   2  
 
 L   albedo   syn   =PA−ÂP   2   2  
 
 L   shape   syn   =Pw   s   T ( S−Ŝ ) P   2   2  
 
 L   pose   syn   =Pθ−{circumflex over (θ)}P   2   2 .
 
 L   lighting   syn   =Pγ−{circumflex over (γ)}P   2   2  
 
 L   reg   syn =λ α   PαP   2   2 +λ β   PβP   2   2  
 
 L   image   syn   =∥x−{circumflex over (x)}   f ∥ 2   2   (2)
 
 L   albedo   syn   =∥A−Â∥   2   2   (3)
 
 L   shape   syn   =∥w   s   T ( S−Ŝ )∥ 2   2   (4)
 
 L   pose   syn =∥θ−{circumflex over (θ)}∥ 2   2   (5)
 
 L   lighting   syn =∥γ−{circumflex over (γ)}∥ 2   2   (6)
 
 L   reg   syn =λ α ∥α∥ 2   2 +λ β ∥β∥ 2   2   (7)
 
     The loss function for pretraining  404  includes: reconstruction losses for the reconstructed face image (L image   syn , represented by equation (2)); reconstruction losses for the one or multiple physical properties including the shape representation  312 A (L shape   syn  represented by equation (4)), the albedo representation  312 B (L albedo   syn , represented by equation (3)), the lighting representation in lighting physical space  308 D (L lighting   syn , represented by equation (6)), and the pose representation in the pose physical space  308 C (L pose   syn , represented by equation (5)) a regularization for the shape code  308 A and the albedo code  308 B (L reg   syn , represented by equation (7)); and a non-saturating logistic GAN loss to improve the photorealism. The shape reconstruction loss in equation (4) may involve assigning weights (w s ) to the locations of vertices in regions surrounding salient facial features, such as eyes, eyebrows, mouth, and the like of the image  102 . 
     During training  406 , the neural network  210  is trained on real 2D images, such as the image  102 , to generalize the model that was pretrained at step  404  on synthetic face images to work on real face images. In some example embodiments, the neural network  210  may be trained using dataset of face photographs and images. In some cases, the face photographs and images may include faces with accessories, such as glasses. In such case, the images with accessories may be eliminated from the dataset. A face mask image  302  (M f ) is obtained for each image in the dataset automatically using a semantic segmentation network. Additionally or alternatively, the face mask image  302  may be obtained using a different method, such as face parsing, facial landmark localization, or manual human annotation. The masked 2D face images  304  are inputted to the neural network  210 . In some embodiments, during the training process  406 , ground truth values of the one or multiple physical properties  104 A- 104 D for the real face images is not known, and no reconstruction loss is applied to the one or multiple physical properties  104 A- 104 D. 
     The pixelwise reconstruction loss between the masked input face image  304  and the masked rendered face image  318 , is represented by equation (8),
 
 L   image   real   =∥x⊙M   f   −{circumflex over (x)}⊙M   f ∥ 2   2   (8)
 
     In addition, we may impose an identity loss between the input image  102  and a reconstructed face image {circumflex over (x)}:=x⊙(1−M f )+{circumflex over (x)}⊙M f , which combines the masked rendered face image  318  with the hair and background from the input image  102  (x). The identity loss is represented by equation (9),
 
 L   identity   real =1−cos( f   id ( x ), f   id ( {circumflex over (x)} ′))  (9)
 
where f id (·) denotes the feature vector extracted by a face recognition network, e.g., Arcface face recognition network, and cos(·,·) denotes a cosine similarity.
 
     Further, a landmark loss that measures a distance between an image-plane projection of 3D facial landmark locations in the input image  102  f lmk   (1) (x) and the corresponding locations f lmk   (2) (Ŝ) in the reconstructed 3D shape model  312 A (Ŝ) is defined by following equation (10),
 
 L   landmark   real   =∥w   1   T [ f   lmk   (1) ( x )− f   lmk   (2) ( Ŝ )]∥ 2   2  
 
 L   landmark   real   =Pw   1   T [ f   lmk   (1) ( x )− f   lmk   (2) ( Ŝ )] P   2   2   (10)
 
where weighting term wl may be included to place more weight on important landmarks such as those contained in the lip outlines, which helps to keep the learned model faithful to the topology of a 3DMM face model (e.g., FLAME).
 
     In some cases, during the training of the neural network  210  for the extraction of the one or multiple physical properties  104 A- 104 D, there may be ambiguities in the extracted one or multiple physical properties  104 A- 104 D. The ambiguities may correspond to relative contributions of color lighting intensities and surface albedo to an RGB appearance of a skin pixel in the images, such as the image  102 . To that end, the ambiguity may be overcome based on an albedo regularization loss corresponding to the albedo map  312 B. The albedo regularization loss is represented as,
 
 L   albedo   real   =P ( B   T   B ) −1   B   T ( Â−Ā ) P   2   2  
 
 L   albedo   real =∥( B   T   B ) −1   B   T ( Â−Ā )∥ 2   2   (11)
 
     The albedo regularization loss (equation 11) may minimize a projection of the albedo map  312 B into a principal component space for albedo from a 3DMM that was trained on albedo scans of real face (e.g., the FLAME model), where Ā and B respectively represent mean and basis vectors of the principal component space for the albedo. Additionally or alternatively, to address the same ambiguity, a lighting regularization loss may be used that may be represented as,
 
 L   lighting   real   =P ({circumflex over (γ)}− γ ) T Σ −1 ({circumflex over (γ)}− γ ) P   2   2  
 
 L   lighting   real =∥({circumflex over (γ)}− γ ) T Σ −1 ({circumflex over (γ)}− γ )∥ 2   2   (12)
 
     The lighting regularization loss minimizes a log-likelihood of reconstructed lighting parameters ({circumflex over (γ)}) under a multivariate Gaussian distribution over lighting conditions. The term  γ  denotes a sample mean and Σ denotes a sample covariance of the reconstructed lighting parameters. Such a multivariate Gaussian distribution over lighting parameters may be obtained using a large dataset of photographic images with known or inferred lighting parameters. Further, the shape code  308 A (α) and albedo code  308 B (β) are regularized. The regularization of the shape code  308 A (α) and the albedo code  308 B (β) is represented as,
 
 L   reg   real =λ α   PαP   2   2 +λ β   PβP   2   2  
 
 L   reg   real =λ α ∥α∥ 2   2 +λ β ∥β∥ 2   2   (13)
 
     At step  408 , the masked rendered face image  318  is combined with the masked hair image  330  to obtain the combined synthetic face image  332 . In some embodiments, a hair-manipulation pipeline modifies an appearance of the hair to correspond to the modified pose of the face, which is further explained in  FIG.  5   . 
     During refining  410 , the combined synthetic face image  332  is further processed by the refiner neural network  334  to generate the refined synthetic face image  336 . The combined synthetic face image  332  may not exhibit sufficient variation in regions, such as eye regions, and may lack certain details such as eyelashes, facial hair, and teeth. Furthermore, since face and hair are processed separately, some combined synthetic face images, such as the combined synthetic face image  332 , may have blending issues between the face and the hair. To that end, the refiner network  334  generates the photorealistic refined synthetic face image  336  that overcomes the realism gap between the combined synthetic face images  332  and their corresponding original images  102 , while making only a minimal change to the reconstructions. The refined synthetic image  336  may be more photorealistic than, but only minimally different from, the combined synthetic face image  332 . In some example embodiments, the refiner neural network  334  may correspond to a convolutional neural network architecture, such as a U-Net. The U-Net manipulates the reconstructed combined synthetic image  332  and outputs the final image  336 . The refiner neural network  334  may be trained with pairs of original images from a training dataset and reconstructed face images  332 . 
     The refiner neural network  334  may be trained using a set of loss functions. These loss functions may include a perceptual loss that measures the perceptual similarity between the refined synthetic image  336  and the input image  102 , an identity loss that measures the identity similarity between the refined synthetic image  336  and the input image  102 , and a non-saturating logistic GAN loss to improve the photorealism of the refined synthetic image  336 . 
     At step  412 , the reconstructed refined synthetic image  336  is generated as an output, comprising a realistic portrait human face image based on the image  102 . In some alternate embodiments, the refined synthetic image  336  may be generated as an anonymized version of a face image in the input image  102 . The anonymized version may be generated using anonymization techniques, for example, replacing a face in the input image  102  by a realistic synthetic image, such as the combined rendered face image  332 , that may be generated by manipulating some of the one or multiple physical properties  104 A- 104 D of the input image, such as one or both of shape and albedo (which may be manipulated for example by randomly sampling a new shape code  308 A and a new albedo code  308 B), while preserving others of the one or multiple physical properties such as pose and lightin. The generated anonymized version may enable sharing photographs or video of one or more persons in an input image, such as the input image  102 , to a public forum that may restrict a user to share actual face image. In such case, the user may share the anonymized version corresponding to the refined synthetic image  336  in the public forum, while protecting actual identity of the user. At step  414 , the process  400  ends. 
       FIG.  5    shows a framework corresponding to a hair-manipulation pipeline  500 , according to one example embodiment of the present disclosure. In some example embodiments, the hair-manipulation pipeline  500  corresponds to manipulating one or more of the hair image  326 , the hair mask  328 , and the masked hair image  330 . The hair-manipulation algorithm  500  may include a Multi-Input-Conditioned Hair Image GAN (MichiGAN) that processes attributes of hair, such as hair shape, structure, and appearance, from the image  102  in a disentangled manner. The hair shape corresponds to a 2D binary mask of hair region of the image  102 . The structure represents a 2D hair strand orientation map. The appearance refers to a global color and style of the hair, which may be encoded as a hair appearance code  504  in a hair appearance latent space. 
     In some example embodiments, the hair-manipulation pipeline  500  may be used to manipulate the shape and the structure of the hair from an input image  102 , without changing the hair color and style from the original image  102 . To that end, the hair-manipulation pipeline  500  extracts from the input image  102  a hair code  504  in a hair appearance latent space. 
     In some example embodiments, the hair manipulation pipeline  500  is coupled with the face reconstruction pipeline  350 . The face reconstruction pipeline  350  is used to extract a 3D model  502  of the face in the image  102 . The extracted 3D model  502  includes information about the one or multiple physical properties  104 A- 104 D, such as one or more of the shape code  308 A, the albedo code  308 B, the shape representation  312 A, the albedo representation  312 B, the pose representation  308 C, and the lighting representation  308 D. 
     In some example embodiments, the hair-manipulation pipeline  500  is implemented iteratively, for example, achieving a large pose manipulation by manipulating the pose by several intermediate steps that each achieves a small pose manipulation. The iterative approach for the hair manipulation pipeline  500  may improve an output of photorealism of the hair image  326 , in particular for large pose manipulations. From the input image  102 , the 3D model  502 , and hair code  504  are extracted once and remain constant throughout all the iterations of the hair manipulation pipeline  500 . 
     In each iteration, the hair-manipulation algorithm  500  may use a reference pose  504 A from a previous iteration and a target pose  506 A. In the first iteration, the reference pose  504 A may be the pose  308 C that was extracted from the input image  102 . In the final iteration, the target pose  506 A may be the final desired pose for the final output image. In each intermediate iteration, call it iteration t, the reference pose  504 A is equal to the target pose from the previous iteration (i.e., iteration t−1), and the target pose  506 A is one step closer to the final desired pose for the final output image. In each iteration, the reference pose  504 A and target pose  506 A are used to calculate a 2D warp field  508  that represents how the image pixels that correspond to hair may move in 2D as a result of the 3D head model  502  transitioning from the reference pose  504 A to the target pose  506 A. The 2D warp field  508  may be calculated based on the projection into a 2D image plane (e.g., the 2D image plane of the input image  102 ) of 3D vertices of the 3D model  502 , as the 3D model  502  pose changes from the reference pose  504 A to the target pose  506 A. For instance, the 3D vertices of the 3D model  502  in the reference pose  504 A may be represented by a face mesh  504 B, and the 3D vertices of the 3D model  502  in the target pose  506 A may be represented by a face mesh  506 B, as shown in  FIG.  5   . The 2D warp field  508  may be obtained by extrapolating the warp field at image locations that are part of the face to image locations that are not part of the face. 
     A reference image  510  that is input to each iteration is a face image or rendering in the reference pose  504 A, while a target image  520  that is output from each iteration is a rendering of the face in the target pose  506 A. In a first iteration, the reference image  510  may be the original input image  102 . In each intermediate iteration, iteration t, the output image  520  from the previous iteration, t−1, may be used as the input reference image  510  for iteration t. Alternatively or additionally, the combined image  518  from the previous iteration, t−1, may be used as the reference image  510  for iteration t. The warp field  508  is used to warp a reference mask  512 A and a reference orientation map  512 B of a reference image  510 . The reference image  510  along with the reference mask  512 A and the reference orientation  512 B may be obtained from a previous iteration of the hair-manipulation algorithm  500 . The warp field  508  warps the reference mask  512 A into a warped mask  514 A and the warps the reference orientation map  512 B into as a warped orientation map  514 B. Each of the warped mask  514 A and the warped orientation map  514 B is regularized to obtain a corresponding target mask  516 A and target orientation  516 B, respectively. The hair-manipulation algorithm  500  combines the target mask  516 A and the target orientation  516 B along with the hair appearance latent space  504  to generate a target image  518 . The target image  518  is inputted to the refiner neural network  334  to generate a photorealistic refined target image  520 . The refined target image  520  from an iteration t of the hair manipulation algorithm may be used as the reference image  510  in the next iteration, iteration t+1. In a final iteration, the refined target image may correspond to the reconstructed 2D image  336 . 
     Each of the reference pose  504 A, the hair appearance code  504 B, the face mesh  506 A, the face mesh  506 B, the 2D warp field  508 , the reference image  510 , the reference mask  512 A, the reference orientation map  512 B, the warped mask  514 A, the reference orientation map  512 B, the warped orientation map  514 B, the target mask  516 A and target orientation  516 B, the target image  518 , and the refined target image  520 , is updated with every iteration. 
       FIGS.  6 A and  6 B , collectively, show a method  600  for manipulating 2D images of 3D objects of a predetermined class, according to one example embodiment of the present disclosure. The method  600  starts from operation  602  and ends at operation  606 , which is performed by the system  200 . 
     At operation  602 , the method  600  includes receiving a 2D input image (e.g., the image  102 ) of a 3D input object of the predetermined class, and one or multiple manipulation instructions for manipulating one or multiple physical properties (e.g., the one or multiple physical properties  104 A- 104 D) of the 3D input object. The one or multiple physical properties include a 3D shape of the 3D input object, an albedo of the 3D input object, a pose of the 3D input object, and lighting illuminating the 3D input object. In some example embodiments, the object is selected from the predetermined class of human faces. 
     At operation  604 , the method  600  includes submitting the 2D input image to a neural network (e.g., the neural network  210 ) while manipulating the latent space, the physical space, or both, of the one or multiple physical properties of the 3D input object according to the one or multiple manipulation instructions. In some embodiments, the neural network is trained to reconstruct the 2D input image by disentangling the physical properties of the 3D input object from pixels of the 2D input image using multiple subnetworks and combining the disentangled physical properties produced by the multiple subnetworks into a 2D output image using a differentiable renderer. Each of the multiple subnetworks disentangles one or more of the one or multiple physical properties of the 3D object. In some embodiments, there is one subnetwork for each of the physical properties of the 3D input object, such that each subnetwork of the multiple subnetworks is trained to extract a corresponding physical property from the pixels of the 2D input image into a one or a combination of latent space and physical space of the corresponding physical property. The albedo subnetwork trained to extract the albedo of the 3D input object from the pixels of the 2D input image has a style-based generative adversarial network (GAN) architecture trained using a set of photographic images of objects from the predetermined class. The manipulation is performed on the disentangled physical properties produced by the multiple subnetworks to modify the 3D input object before the differentiable renderer reconstructs the 2D output image. In some example embodiments, a plurality of realistic synthetic images may be generated from the 2D images of the 3D objects. The plurality of realistic synthetic images may include variations in the one or multiple physical properties of the poses, expressions, and lighting illumination of the 3D objects. 
     At operation  606 , the method  600  includes outputting the 2D output image. In some example embodiments, the 2D output image may be utilized in at least one of a 3D animation of the 3D object from the 2D input image, a hairstyle trial on a human face of the 2D input image, a reconstruction of a photorealistic synthetic image of the human face from the 2D input image, and a robotic assistance using the reconstructed 2D output image. 
       FIG.  7    shows a block diagram of a system  700  for manipulating 2D images of 3D objects of a predetermined class, according to one example embodiment of the present disclosure. The system  700  corresponds to the system  200  of  FIG.  2   . The system  700  includes a processor  702  and a memory  704 . The memory  704  can include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory system. The memory  704  is configured to store a neural network  706 . The neural network  706  corresponds to the neural network  210 . The processor  702  can be a single core processor, a multi-core processor, a computing cluster, or any number of other configurations. 
     The system  700  also includes an input interface  720  configured to receive a 2D input image of a 3D input object of the predetermined class, and one or multiple manipulation instructions for manipulating one or multiple physical properties of the 3D input object. The one or multiple physical properties correspond to the one or multiple physical properties  104 A- 104 D that include a 3D shape of the 3D input object, an albedo of the 3D shape of the 3D input object, a pose of the 3D input object, and lighting illuminating the 3D input object. Additionally or alternatively, the input interface  720  may receive the 2D images from a camera, such as a camera  724 . A few examples of the camera  724  may include an RGBD camera. 
     The processor  702  is configured to submit the 2D images to the neural network  706  stored in the memory  704 . The neural network  706  corresponds to the neural network  210 . 
     In one implementation, a human machine interface (HMI)  714  within the system  700  connects the system  700  to the camera  724 . Additionally or alternatively, a network interface controller (NIC)  716  may be adapted to connect the system  700  through a bus  712  to a network  718 . In one implementation, the 2D images may be accessed from an image dataset  710  via the network  718 . The image dataset  710  may be used pretraining and training the neural network  706 . 
     Additionally or alternatively, the system  700  may include an output device  726  configured to output a final image, such as the reconstructed 2D image  336 . The output device  726  may be connected with the system  700  via an output interface  722 . 
     Additionally or alternatively, the system  700  may include a storage  708  configured to store one or multiple physical properties, such as the one or multiple physical properties  104 A- 104 D of the image, reconstructed images of the image (e.g., reconstructed images after the one or multiple physical properties  104 A- 104 D have been manipulated), latent spaces of the one or multiple physical properties  104 A- 104 D, and/or the like. 
     The data stored in the storage  708  may be accessed through the network  718  for further processing. For instance, the processor  702  may access the storage  708  via the network  718  for utilizing the 2D output image in at least one of a 3D animation of the 3D object from the 2D input image, a hairstyle trial on a human face of the 2D input image, a reconstruction of a photorealistic synthetic image of the human face from the 2D input image, and a robotic assistance using the reconstructed 2D output image. 
       FIG.  8    illustrates a use case scenario  800  for manipulating 2D images of 3D objects of a predetermined class, according to one example embodiment of the present disclosure. The use case scenario  800  corresponds to an application for generating a cartoon or an animated version of a user, such as a user  802  using the system  700 . For the generation of such animated version of the user  802 , the system  700  may be trained using animated training dataset. In some cases, the animated version of the user  802  may correspond to an anonymized version of the user  802 . The application may be installed in a user device  804 . In an illustrative example scenario, the user  802  may capture a 2D image that includes a frontal face of the user  802 . The 2D image may be shared to the system  700  via a network, such as the network  718 . The system  700  processes the 2D image of the user  802 . After the processing, the system  700  generates one or more animated versions  806 . The one or more animated versions  806  may include face images of the user  802  in different poses and/or expressions, different animated pictures of the user  802  and/or the like. 
       FIG.  9    illustrates a use case scenario  900  for manipulating 2D images 3D objects of a predetermined class, according to another example embodiment of the present disclosure. In the use case scenario  900 , a 2D image of a user, such as a user  902  may be used for trying out different hairstyle looks in a realistic manner. In an illustrative scenario, a 2D face image of the user  902  is shared to an electronic device  904 . In some cases, the electronic device  904  may be remotely connected with the system  700 . In some other cases, the system  700  may be embedded within the electronic device  904 . The electronic device  904  sends the 2D image of the user  902  to the system  700 . The system  700  generates different images of the user  902  with corresponding different hairstyles in a photorealistic manner, such that even when the user  902  changes different head poses, the different images with the corresponding different hairstyles remain realistic. The user  902  may select the desired hairstyle from the generated different images with different hairstyles. 
       FIG.  10    illustrates a use case scenario  1000  for manipulating 2D images 3D objects of a predetermined class, according to yet another example embodiment of the present disclosure. The use case scenario  1000  corresponds to a reconstruction of a photorealistic synthetic image of the human face from a 2D input image. In an illustrative example scenario, a surveillance camera may capture a 2D image of a person. The captured 2D image of the person may be a side view of person&#39;s face. In such scenario, the 2D image may be shared to the system  700 . The system  700  may generate different images  1002  of the person from different angles of view, different poses and/expressions of the person. The different images  1002  may include images with a reconstructed frontal view of the person, which may be used to track down a suspect by a security official. For instance, an image  1004  with the frontal view of the person may be selected from the different images  1002 . 
       FIG.  11    illustrates a use case scenario  1100  for manipulating 2D images of 3D objects of a predetermined class, according to yet another example embodiment of the present disclosure. The use case scenario  1100  corresponds to a robotic system using a reconstructed 2D output image. Such robotic system may be used in an industrial automation application for picking objects from a conveyor belt  1104 . There may be times that lighting condition on the object may be poor, which may hinder functionality of the robotic system to pick the objects from the conveyer belt  1104 . In such poor lighting conditions, the system  700  may generate reconstructed images of objects, such as the object  1102  under the poor light conditions, and aid the robotic system in picking the object  1102  from the conveyer belt  1104 . To that end, a 2D image of a 3D object, such as object  1102 , may be captured using a device  1106 . The device  1106  may include a camera. The 2D image of the object  1102  is shared from the device  1106  to the system  700  for further processing. The system  700  may reconstruct the 2D object with manipulated lighting illumination to aid the robotic system in picking the object  1104  from the conveyer belt  1104 . 
     In this manner, the system  700  may be used for manipulating 2D images for 3D objects using a neural network trained in an unsupervised manner. The unsupervised training of the neural network precludes the need for 3D data, which may be difficult and/or expensive to obtain, and improves overall processing and storage efficiency. Further, such manipulation of the 2D images helps in achieving photorealistic images, in an efficient and feasible manner. 
     The above description provides exemplary embodiments only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the above description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Contemplated are various changes that may be made in the function and arrangement of elements without departing from the spirit and scope of the subject matter disclosed as set forth in the appended claims. 
     Specific details are given in the above description to provide a thorough understanding of the embodiments. However, it can be understood by one of ordinary skill in the art that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the subject matter disclosed may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments. Further, like reference numbers and designations in the various drawings indicated like elements. 
     Also, individual embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process may be terminated when its operations are completed, but may have additional steps not discussed or included in a figure. Furthermore, not all operations in any particularly described process may occur in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the function&#39;s termination can correspond to a return of the function to the calling function or the main function. 
     Furthermore, embodiments of the subject matter disclosed may be implemented, at least in part, either manually or automatically. Manual or automatic implementations may be executed, or at least assisted, through the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine-readable medium. A processor(s) may perform the necessary tasks. 
     Various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages and/or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments. 
     Embodiments of the present disclosure may be embodied as a method, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts concurrently, even though shown as sequential acts in illustrative embodiments. Further, use of ordinal terms such as “first,” “second,” in the claims to modify a claim element does not by itself connote any priority, precedence, or order of one claim element over another or the temporal order in which acts of a method are performed, but are used merely as labels to distinguish one claim element having a certain name from another element having a same name (but for use of the ordinal term) to distinguish the claim elements. 
     Although the present disclosure has been described with reference to certain preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the present disclosure. Therefore, it is the aspect of the append claims to cover all such variations and modifications as come within the true spirit and scope of the present disclosure.