Patent Description:
Recent advances in generative adversarial networks (GANs) have enabled the production of realistic high resolution images of smooth organic objects such as faces. Generating photorealistic human bodies, and faces in particular, with traditional rendering pipelines that do not use neural networks is notoriously difficult, requiring hand- crafted three dimensional (3D) assets. However, once these 3D assets have been generated it is possible to use a conventional renderer to render the face from different directions and in different poses. In contrast, GANs can be used to easily generate realistic head and face images without the need to author expensive 3D assets, by training on curated datasets of 2D images of real human faces. However, it is difficult to enable meaningful control over this generation without detailed hand labelling of the dataset. Even when conditional models are trained with detailed labels, they struggle to generalize to out-of-distribution combinations of control parameters such as children with extensive facial hair or young people with gray hair. Thus it has not previously been possible for GAN based rendering techniques to replace traditional rendering pipelines.

Rendering and animation of realistic objects such as human faces is a longstanding problem in the field of computer graphics. To create an animation of a specific actor's face one usually requires a 3D capture of the actor's face performing various expressions. The capture is then used to create a rigged 3D model which can be animated. To render the face, additional artistic work is necessary to recreate the elements of the face that are difficult to capture in 3D, such as hair. The whole process is very time-consuming and expensive.

The embodiments described below are not limited to implementations which solve any or all of the disadvantages of known image processing methods.

<NPL> describes a generative adversarial network (GAN) for the task of unsupervised learning of 3D representations from natural images. Most generative models rely on 2D kernels to generate images and make few assumptions about the 3D world. These models therefore tend to create blurry images or artefacts in tasks that require a strong 3D understanding, such as novel-view synthesis. HoloGAN instead learns a 3D representation of the world, and to render this representation in a realistic manner. Unlike other GANs, HoloGAN provides explicit control over the pose of generated objects through rigid-body transformations of the learnt 3D features. Using explicit 3D features enables HoloGAN to disentangle 3D pose and identity, which is further decomposed into shape and appearance, while still being able to generate images with similar or higher visual quality than other generative models. HoloGAN can be trained end-to-end from unlabelled 2D images only. Particularly, pose labels, 3D shapes, or multiple views of the same objects are not required. This shows that HoloGAN is the first generative model that learns 3D representations from natural images in an entirely unsupervised manner.

The detailed description provided below in connection with the appended drawings is intended as a description of the present examples and is not intended to represent the only forms in which the present example are constructed or utilized. The description sets forth the functions of the example and the sequence of operations for constructing and operating the example. However, the same or equivalent functions and sequences may be accomplished by different examples.

Although the present examples are described and illustrated herein as being implemented in an image processing system for images of faces, the system described is provided as an example and not a limitation. As those skilled in the art will appreciate, the present examples are suitable for processing images of other types of object where there is a synthetic rendering apparatus available for rendering synthetic images of the type of object concerned. Any known synthetic rendering apparatus is used which takes parameters as input to specify attributes of the synthetic images it generates.

<FIG> is a schematic diagram of a neural renderer <NUM> deployed as a cloud service and/or at an end user computing device <NUM>. The neural renderer comprises one or more neural networks used for various image processing tasks including: generating an image of an object using neural networks and where it is possible to control attributes of the image using semantically meaningful parameters.

Where the neural renderer is deployed as a cloud service it is implemented at one or more web servers or other computing resources which are in communication with client devices such as end user computing device <NUM> via a communications network <NUM>. A client device is able to send inputs to the neural renderer <NUM> comprising images and/or parameter values and, in response, receives one or more output images. The output images are stored or displayed.

Where the neural renderer is deployed at an end user device it is stored in local memory of the end user device and/or embodied in hardware or firmware at the end user device. In <FIG> a smart phone is illustrated as comprising (indicated by dotted lines) a neural renderer <NUM>, a processor <NUM>, an optional camera <NUM> and a memory <NUM>. It is possible to have hybrids between the cloud service deployment and the end user device deployment. That is, the functionality of the neural renderer is distributed between the client device and other computing devices in some examples. A non-exhaustive list of suitable end user computing devices <NUM> is: smart phone, wearable computer, tablet computer, desktop computer, laptop computer, game console.

In the example of <FIG> an end user computing device <NUM> such as a smart phone shown on the left hand side of the figure displays a real image of a child's face in a neutral expression with eyes open and with no facial hair. A user inputs values of parameters including "no smile", "no beard" and "eyes shut". The neural renderer <NUM> generates an output image which is displayed at the smart phone on the right hand side in <FIG>. The output image depicts the child's face with eyes shut, no smile and no beard. Previously it has not been possible to achieve this type of functionality using neural network technology. A significant level of control over generative neural network technology is achieved without sacrificing realism. Previous approaches using conditional models trained with detailed hand labelling of the dataset struggle to generalize to out of distribution combinations of control parameters such as children with facial hair. In contrast the present technology does not need detailed hand labeled datasets and performs well for combinations of control parameters such as children with facial hair.

In an example the trained neural renderer <NUM> takes as input an image of a face of a person to be animated such as an actor's face in one example. In this example, the neural renderer <NUM> also takes as input one or more of the following attributes: head pose, face expression, facial hair style, head hair style, hair colour, illumination, beard style, eyebrow style, eye colour, eye rotation, hair colour, head shape, lower eyelash style, texture, upper eyelash style. These attributes are parametrized in semantically meaningful ways that are commonly used in computer graphics. For example, face expression is parametrized as a sum of individual expression (smile, eyebrows raised, eyes open) with individually specified intensities.

The neural renderer <NUM> computes an output image depicting the person in the input image with the specified attributes. This allows for face animation and edition with minimal effort. The output image is stored or displayed.

The neural renderer <NUM> is also used to generate novel images in some examples as described in more detail with reference to <FIG>.

<FIG> is a schematic diagram of a real image of a child's face <NUM> with a neutral expression. <FIG> also shows schematically four images <NUM>, <NUM>, <NUM>, <NUM> of the child's face computed by the neural renderer and where different individual attributes have been controlled by setting parameter values. An end user is able to set the parameter values using a graphical user interface or in other ways. Image <NUM> was generated with a parameter for facial hair selected. Image <NUM> was generated with a parameter for a smile selected. Image <NUM> was generated with a parameter for eyes closed selected. Image <NUM> was generated with a parameter for head pose set to facing right.

As explained in more detail below with reference to <FIG> and <FIG> the neural renderer <NUM> comprises two encoders <NUM>, <NUM> and a decoder <NUM> which together are sometimes referred to as an autoencoder. The two encoders <NUM>, <NUM> correspond to two types of data used for training: real images with no labels and synthetically generated images with labels for attributes. A first one of the encoders is referred to herein as a real data encoder <NUM> since it has been trained using real images. When it is used at test time it can be used to encode real or synthetic images. In <FIG> it is shown as taking an image as input. The real data encoder <NUM> computes an embedding <NUM> by mapping the input image to the embedding which is typically expressed as a vector specifying a location in a multi-dimensional space. However, it is not essential to use a vector format as other formats are used in some cases.

A second one of the encoders is referred to herein as a synthetic data encoder <NUM> since it has been trained using synthetic images. It takes parameter values <NUM> as input as explained in more detail below and it computes a mapping from the parameter values <NUM> to an embedding <NUM> which is typically expressed as a vector specifying a location in a multi-dimensional space but is given in other formats in some cases. The multi-dimensional space is referred to as a latent space since it is learnt by the neural renderer <NUM> during training and is not observed.

To animate an object shown in an input image I, the image is first passed to the real data encoder <NUM> to generate an embedding z_0. Given z <NUM>, the decoder <NUM> generates an output image that is very close to I. To generate an output image with a different attribute, the part of z_0 that corresponds to that attribute is modified. The synthetic data encoder <NUM> is factorised into separate parts that correspond to different attributes. To modify, for example, illumination, pass the desired illumination parameters to the synthetic data encoder <NUM>, which generates a part v of the latent embedding that corresponds to that illumination. The embedding z_1 that corresponds to the same object as z_0 but with a different illumination is generated by swapping out the part that corresponds to illumination with v. The embedding z_1 is then input to the decoder which generates an output image <NUM> depicting the object and with the illumination as specified by the parameter values <NUM>.

The encoders and the decoder comprise neural networks which have been trained as described in more detail below. The first and second encoders have been trained using adversarial training such that a distribution of embeddings computed by the first encoder is substantially the same as a distribution of embeddings computed by the second encoder. The second encoder is factorized so that the embeddings it computes have separate parts, each part corresponding to a factor of the factorization, and where each part corresponds to an attribute of a synthetic image rendering apparatus. The term "substantially the same" means near-enough identical to give a good working result.

The neural renderer is trained on both real and synthetically generated images. Since the synthetic images were generated with a traditional graphics pipeline, the renderer parameters for those images are readily available. The known correspondences between the renderer parameters and synthetic images are used to train a generative model that uses the same input parametrization as the graphics pipeline used to generate the synthetic data. This allows for independent control of various attributes of objects depicted in images. By simultaneously training the model on unlabeled images, it learns to generate photorealistic looking images, while enabling full control over the outputs.

The encoders and decoder of the disclosure operate in an unconventional manner to achieve controllable image generation.

The encoders and decoder of the disclosure improve the functioning of the underlying computing device by computing a factorized embedding and modifying one or more factors of the factorized embedding according to attributes desired in an output image generated from the modified factorized embedding.

The neural renderer <NUM> treats synthetic images IS and real images IR as two different subsets of a larger set of all possible face images. Hence, the neural renderer consists of a decoder G <NUM> and two encoders ER <NUM> and ES <NUM> that embed real and synthetic data into a common factorized latent space z. The following description refers to z predicted by ER and ES as zR and zS respectively. While the real data is supplied to the encoder as images IS ∈ IR, the synthetic data is supplied as vectors θ ∈ Rm that fully describe the content of the corresponding image IS ∈ IS. During training, to optionally increase the realism of the generated images two discriminator networks DR and DS are optionally used for real and synthetic data respectively.

Assume that the synthetic data is a reasonable approximation of the real data so that IS ∩ IR ≠ Ø. Hence, it is desirable for ES(Θ) and ER(IR), where Θ is the space of all θ, to also be overlapping. To do so, a domain adversarial loss is introduced on z, that forces zR and zS to be close to each other.

Alternatively, or in addition, the functionality of the encoders and the decoder described herein is performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that are optionally used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), Graphics Processing Units (GPUs).

<FIG> shows another example of a neural renderer described in an example where the images are face images. The example of <FIG> is readily modified to operate for images depicting other classes of object. The first encoder is implemented as a convolutional neural network <NUM> which receives as input a real face image <NUM>. The first encoder <NUM> has been trained to map an input image to an embedding <NUM> as described above with reference to <FIG>.

The second encoder <NUM> comprises a plurality of parameterized functions which in this example are multi-layer perceptrons (MLPs) <NUM>. Each parameterized function maps a parameter to a factor of a factorized embedding. Other types of parameterized function are used in some examples. Each parameterized function corresponds to a parameter of a synthetic image rendering apparatus. In the example of <FIG> there is one MLP for head pose, one MLP for hair style, one MLP for expression and one MLP for illumination. The particular parameters used depends on the type of objects the neural renderer has been trained to work with and on the parameters of the synthetic image rendering apparatus. The factors computed by the parameterized functions are concatenated using a concatenator <NUM> to form an embedding <NUM>.

Each synthetic data sample θ is factorised into k parts θi to θk, such that: <MAT>.

Each θi corresponds to semantically meaningful input of the synthetic image rendering apparatus used to generate IS. The synthetic data encoder ES <NUM>, <NUM> maps each θi to zi, a part of z, which thus factorizes z into k parts.

The factorized latent space allows for easy modification of various aspects of the output images <NUM>. For example, one might encode a real image into z using ER and then change the illumination by swapping out the part of z that corresponds to illumination. Note that the part of z that is swapped in might come from θi, which is semantically meaningful, or it may come from a different real face image encoded by ER <NUM>.

The decoder <NUM> is implemented using neural networks. It takes as input the embedding <NUM> and computes an output image <NUM> as described above.

<FIG> shows an example of a neural renderer with components used to train the neural renderer. <FIG> also shows the decoder <NUM> in two parts for ease of understanding the technology. Note that the two parts of the decoder <NUM> are neural networks with shared weights so that the two parts function as a single decoder as illustrated in <FIG> and <FIG>. <FIG> shows two output images <NUM>, <NUM> because during training there are two output images, one from the part of the decoder associated with the real data encoder <NUM> and one from the part of the decoder associated with the synthetic data encoder <NUM>. After training, when the neural renderer is in operation, the output image <NUM> associated with the real data encoder is not present. Training is done using both real images and synthetic images. Each training batch consists of a portion of real images and a portion of synthetic images.

The components used to train the neural renderer include: domain discriminator <NUM>, latent regressor <NUM>, real image discriminator <NUM> and synthetic image discriminator <NUM>. Note that the latent regressor <NUM>, real image discriminator <NUM> and synthetic image discriminator <NUM> are optional. After training the domain discriminator <NUM>, latent regressor <NUM>, real image discriminator <NUM> and synthetic image discriminator <NUM> are omitted. Each of the domain discriminator <NUM>, latent regressor <NUM>, real image discriminator <NUM> and synthetic image discriminator <NUM> are neural networks.

The function of the domain discriminator <NUM> is to enable adversarial training of the real data encoder <NUM> and synthetic data encoder <NUM> so that the distributions of embeddings computed by the real data encoder <NUM> and synthetic data encoder are substantially the same. Since the embeddings <NUM> computed by the synthetic data encoder are factorized by virtue of having the plurality of parameterized functions in the synthetic data encoder <NUM>, the embeddings <NUM> computed by the real data encoder <NUM> are divisible into the same factors as for the factors of the synthetic data encoder. Each factor is a part of an embedding vector identified by the location of entries in the vector. The domain discriminator is trained with a domain adversarial loss between embeddings produced by the two encoders. It forces the distributions generated by the two encoders to be similar.

The latent regressor <NUM> is optional. The purpose of the latent regressor <NUM> is to encourage the interpretation of the latent space to be similar for real and synthetic data. The loss function used by the latent regressor is between embeddings predicted by the latent regressor and input embeddings.

The real image discriminator <NUM> is used to enable adversarial training so as to improve performance of the real data encoder <NUM> and the decoder <NUM>. The real image discriminator <NUM> is trained using an adversarial loss between the images generated from the real data encoder <NUM> predictions and a real image training set.

The synthetic image discriminator <NUM> is used to enable adversarial training so as to improve the performance of the synthetic data encoder and the decoder <NUM>. The synthetic image discriminator <NUM> is trained using an adversarial loss between the images generated from synthetic encoder <NUM> predictions and a synthetic image training set.

During training an image loss is used between the input real image I and output image produced with embedding predicted from I.

During training an image loss is used between a synthetic image corresponding to a set of attributes and an output image produced with a embedding predicted from those attributes.

In a particular embodiment, where the images are face images, the neural renderer is trained using the following loss functions:.

To ensure that the output image G(z) <NUM> is close to the corresponding ground truth image IGT, a perceptual loss Lperc, which is the mean squared error between the activations of a pre-trained neural network computed on G(z) and IGT. In an example the pre-trained neural network is a <NUM> layer convolutional neural network trained on ImageNet.

An additional loss is optionally used to preserve eye gaze direction as follows: <MAT>
where M is a pixel-wise binary mask that denotes the iris, only available for IS. Thanks to the accurate ground truth segmentation that comes with the synthetic data, similar losses are added for any part of the face if necessary.

The adversarial blocks are trained with a non-saturating GAN loss: <MAT> <MAT>
where LGAND is used for the discriminator and LGANG is used for the generator, D is the discriminator, x is a real sample and y is the generated sample.

<FIG> is a flow diagram of a method of operating a trained neural renderer to embed an existing image and then manipulate it. A new image is generated which depicts the object in the existing image but with one or more attributes changed according to parameter values input to the synthetic data encoder. No person-specific 3D assets are required for the method of <FIG>. The realism of the generated images is found to be higher that that of the synthetic training data used during training. The use of a parameterization derived from a traditional graphics pipeline makes the neural rendered easy to use for people familiar with digital animation.

The neural renderer accesses <NUM> a real image such as a digital photograph, a frame of a video, or other real image. The real image depicts an object of a specified type such as human faces, human bodies, automobile, laptop computer, animal or any other type of object. The real image is input to a trained real image encoder which computes <NUM> a factorized embedding. The factorized embedding denotes a location in a multi-dimensional latent space which has been learnt by the neural renderer in advance during a training process. The embedding is factorized in that it is separable into parts, called factors, where each part corresponds to a parameter of a synthetic rendering apparatus. The synthetic rendering apparatus is configured to render synthetic images of object of the specified class and it takes as input values of parameters specifying attributes of the synthetic images it generates. In the example of <FIG> the synthetic rendering apparatus is the synthetic data encoder and the decoder which together are able to generate synthetic images depicting objects of the specified class.

The neural renderer checks <NUM> whether it has received values of one or more parameters of the synthetic renderer. If not it waits to receive those. The values are received as a result of user input or from another automated process such as a computer game. If one or more values have been received the neural renderer computes <NUM> an embedding factor for the received value. To compute the embedding factor the values are input to the synthetic data encoder which computes a prediction which is a predicted embedding factor for the received value.

The neural renderer modifies <NUM> the factorized embedding with the embedding factor by swapping a part of the factorized embedding, which corresponds with the parameter value input to the synthetic data encoder, with the embedding factor. The modified factorized embedding is input to the decoder.

The decoder decodes <NUM> the modified factorized embedding to generate an output image. The output image is the same as the real image accessed at operation <NUM> except that one or more attributes of the output image are changed according to the parameter values input to the synthetic data encoder. The output image is stored and/or displayed <NUM>. In this way a highly realistic output image is created in an efficient manner whilst being able to control individual attributes of the output image. It is possible to animate the object depicted in the real image.

<FIG> is an example of a method of generating a new image from the neural renderer without the need to input an image as part of the test time operation. <FIG> also illustrates a method of generating a new image from an existing real or synthetic image by using a sample generated from the latent space of the neural renderer.

Samples of the latent space are used to generate novel images or to sample individual factor zi. The sampled zi are used to generate additional variations of an existing image that was embedded in z. A latent generative adversarial network (GAN) is used. The latent GAN is trained to map between its input ω ~ N(<NUM>, I) and the latent space z. This approach allows for sampling the latent space without the constraints on z imposed by variational auto encoders that lead to reduced quality. The latent GAN is trained with the GAN losses described with reference to <FIG> below. Both the discriminator and generator Glat are <NUM>-layer multi-layer perceptrons.

The neural renderer computes <NUM> a sample from the latent space. The sample is an embedding in the multi-dimensional space of the encoders. The neural renderer checks <NUM> whether it is desired to generate a variation of an existing real or synthetic image. If not, it sends the sample to the decoder and the decoder decodes <NUM> the sample to generate an output image. The output image depicts an object of the type of objects that the neural renderer has been trained to deal with. The attributes of the object are as specified in the sample; that is, a user or other process has not needed to input values of parameters to specify the attributes.

If it is desired to generate a variation of an existing real of synthetic image then a swap is done at operation <NUM>. One or more parts of the sample are swapped <NUM> with factors from an embedding of a real or synthetic image. Once the swap has been done the modified sample is input to the decoder. The decoder decodes <NUM> the modified sample to produce an output image. The output image depicts an object of the type of objects that the neural renderer has been trained to deal with but where one or more attributes of the object are as in the real or synthetic image used to obtain the embedding at operation <NUM>.

The output image is stored and/or displayed <NUM>.

<FIG> is a flow diagram of a method of operation at a neural renderer to achieve finer grained control of attributes of output images than achieved using the method of <FIG> or <FIG>. If face expression is an attribute then fine grained control means being able to control a single aspect of face expression such as intensity of smile whilst leaving other aspects of face expression such as eyebrow pose invariant. If illumination is an attribute then fine grained control means being able to control a single aspect of illumination such as brightness whilst leaving other aspect such as contrast static. Thus attributes of images have one or more aspects which are individually controllable by using the method of <FIG>.

The neural renderer accesses <NUM> an image which is either a real image or a synthetic image. The neural renderer computes <NUM> a factorized embedding of the image by using one of the encoders. The neural renderer checks <NUM> whether a parameter vector is available where the parameter vector specifies values of parameters used to generate the image using the synthetic rendering apparatus. If the image accessed at operation <NUM> is a real image there is no parameter vector available. In this case the parameter vector is estimated <NUM>. To estimate the parameter vector an optimization is computed to find an estimated parameter vector which when given to the synthetic encoder will produce an embedding which is similar to the embedding computed at operation <NUM>.

If the image accessed at operation <NUM> is a synthetic image there is a parameter vector already available since the values of the parameters used to generate the synthetic image are known.

The neural renderer modifies <NUM> part of the parameter vector (which is either the estimated parameter vector or the actual parameter vector) for fine grained control. In an example, to control brightness the neural renderer finds the part of the parameter vector which controls brightness and modifies it appropriately. It is known how to modify the parameter vector since the parameter vector is input to the synthetic renderer.

The modified parameter vector is encoded <NUM> to produce a factorized embedding. The factorized embedding is decoded <NUM> by the decoder to produce an output image which depicts the object in the image accessed at operation <NUM> and with fine grained control of the aspect of the attribute. The output image is stored and/or displayed at operation <NUM>.

Given an existing face image embedded into z, it is possible to swap any part, zi, of its embedding with one that is obtained from ES or ER. However, sometimes a finer level of control is desired such as to only modify a single aspect of zi while leaving the rest the same. If zi is a face expression, its single aspect might be the intensity of smile, if zi is illumination, the brightness might be one aspect. These aspects are controlled by individual elements of the corresponding θi vector. However θi is unknown if the z was generated by ER or Glat.

Compute an approximation θ̃i obtained by solving the minimization problem min θ̃i |zi - Esi(θ̃i)|<NUM> with gradient descent, where Esi is the part of ES that corresponds to θi. Optionally incorporate constraints on θi into the optimization algorithm. For example, expression parameters lie in the convex set [<NUM>,<NUM>] and use projected gradient descent to incorporate the constraint into the minimization algorithm. Given θ̃i, e.g. a face expression vector, modify the part of the vector responsible for an individual expression and use ES to obtain a new latent code zi that generates images where only this individual expression is modified.

<FIG> shows a two stage training process which is found to improve controllability and image quality. It is not essential to use the two stage training process.

With reference to <FIG> a first stage <NUM> involves omitting the real data encoder <NUM> and randomly generating <NUM> embeddings of real images. During the first stage the synthetic data encoder and the decoder are trained using backpropagation <NUM> and using synthetic images.

The first stage ends when there is little or no change in the synthetic data encoder and the decoder; or the first stage ends when a specified amount of synthetic training images have been used.

In the second stage <NUM> the real data encoder is included <NUM>. The autoencoder is trained using backpropagation <NUM> and using batches of training data comprising both real and synthetic images. The second stage ends when there is little or no change in the encoders and the decoder; or the second stage ends when a specified amount of training images have been used.

In examples where two stage training is not used there is a single training stage which is the same as the second stage <NUM> of <FIG>.

In the first stage <NUM>: train all the sub-networks except ER <NUM>, sampling zR ~ N (<NUM>, I) as there is no encoder for real data at this stage. At this stage ES and G <NUM> are trained with the following loss: <MAT>
where zS = ES (θ) and λ are the weights assigned to the corresponding losses. The domain discriminator DDA acts on ES to bring the distribution of its outputs closer to N(<NUM>, I) and so ES effectively maps the distribution of each θi to N(<NUM>, I).

In the second stage <NUM>: add the real data encoder ER so that zR = ER(IR). The loss used for training ES and G is then: <MAT>
where the goal of log - DDA(zR) is to bring the output distribution of ER closer to that of ES. In the second stage increase the weight of λperc, in the first stage it is set to a lower value as otherwise total loss for synthetic data would overpower that for real data. In the second stage both real and synthetic data use the perceptual loss and increase its weight. Experiments show that this two-stage training improves controllability and image quality.

One-shot learning by fine tuning is used in some examples. It is not essential to use one-shot learning by fine tuning. One-shot learning by fine tuning comprises pre-training the encoder and the decoders (using the first and second stages of <FIG> or only the second stage of <FIG>) and then training again using real images and with a loss function that encourages the neural renderer to reduce an identity gap between a face depicted in the real image and in the output image. It is unexpectedly found that one-shot learning by fine tuning is effective. One-shot learning modifies the embeddings and the whole decoder and it is surprising that control of the output image is still possible through the factorized embeddings even after one-shot learning by fine tuning has been done.

The neural renderer architecture allows for embedding face images into z using the real data encoder ER, and therefore individual factors zi can be modified to modify the corresponding output image. To reduce any identity gap between the face in IR and in the generated image it is possible to fine-tune the generator on IR by minimizing the following loss: <MAT>
where Lface is a perceptual loss with a <NUM> layer convolutional neural network as the pre-trained network. Optimize over the weights of G as well as z^R which is initialized with (ER(IR)). The addition of a Lface improves the perceptual quality of the generated face images.

A detailed example is now given for the case where the images depict faces. This example also works for the case where the images depict another type of object, such as automobiles, human body, laptop computer, cat, dog, or other type of object.

In this example, the architecture of the decoder G <NUM> is configured to decouple object rotation from the latent space and allow for specifying object rotation with any parametrization (Euler angles or quaternions). Object pose such as head pose is thus obtained in a parameterization which is suitable for input to the decoder without requiring an encoder.

The remaining k - <NUM> parts of θ are encoded with separate multi -layer perceptrons (MLPs <NUM>) Esi, each of which consists of <NUM> layers with a number of hidden units equal to the dimensionality of the corresponding θi. The real image encoder ER is a ResNet-<NUM> pre-trained on ImageNet. The domain discriminator DDA is a <NUM>-layer MLP. The two image discriminators DR and DS share the same basic convolutional architecture.

For the perceptual loss use layers conv_1_2, conv_2_2, conv_3_4, conv_4_4 of the <NUM> layer convolutional neural network. Regularize the discriminators with an R<NUM> gradient penalty. In the image discriminators, use a style discriminator loss Lstyle , while in the generator add an identity loss Lidentity. Use a separate network that has the same architecture as the image discriminators because neither of the discriminators is trained to work with both real and synthetic data. Set the loss weights as follows: eye loss weight λeye = <NUM>, domain adversarial loss weight λDA = <NUM>, identity loss weight λidentity = <NUM>, gradient penalty loss weight λR1 = <NUM>, perceptual loss weight in 1st stage λperc = <NUM>, perceptual loss weight in 2nd stage λperc = <NUM>. The adversarial losses on the images and style discriminator losses have weight <NUM>.

In the first training stage sample z ~ N (<NUM>, I) and rR ~ U(-rlim, rlim), where rR is the rotation sample for real data and rlim is a pre-determined, per axis rotation limit. In the experiments set rlim to be identical to the rotation limits used in synthetic data generation as described in the dataset section. In the second stage the ER output corresponding to rR is constrained to the range specified in rlim by using a tanh activation and multiplying the output by rlim.

The architecture of the generator network G (also referred to as the decoder <NUM>) is given in the table below.

The table below shows the architecture of the image discriminators DR, DS. Most of the convolutional layers of the discriminator use instance normalization. The latent GAN generator Glat and discriminator share the same <NUM>-layer MLP architecture.

The networks are optimized using the Adam algorithm with a learning rate of 4e-<NUM>. Perform the first stage of training for <NUM> iterations and then the second stage for <NUM> iterations. The latent GAN is also trained for <NUM> iterations. In both the latent GAN and decoder G, keep an exponential running mean of the weights during training and use those smoothed weights to generate results.

In the present example the training data included <NUM>,<NUM> real images each of size <NUM> Mpix and <NUM>,<NUM> synthetic images each of size <NUM> Mpix. The real and synthetic images are of faces and where aligned to a standard reference frame using landmarks and were reduced in resolution to 256x256 pixels.

The validation data included <NUM>,<NUM> real images.

The synthetic images were generated using a synthetic image renderer and setting rotation limits for yaw and pitch to ±<NUM>° and ±<NUM>° to cover a typical range of poses in face images. For the synthetic images generated, θ has m = <NUM> dimensions, while z has n = <NUM> dimensions, and is divided into k = <NUM> factors.

The following table shows the dimensionalities of latent space factors zi and corresponding synthetic data parameters θi. The dimensionalities of each zi were chosen based on perceived complexity of the feature, for example allocate more dimensions to expression than to hair colour. The expression parameters consist of <NUM> expression blendshapes and one additional dimension for the rotation of the jaw bone that leads to mouth opening.

A user study was carried out with <NUM> users. The users evaluated the presence of an attribute in a total of <NUM> image pairs. Each image pair was made up of an image of a face with the attribute and an image of the same face with the opposite attribute. An example of an image pair is an image of a person with blond hair and an image of the same person with black hair. Another example of an image pair is an image of a person with eyes shut and an image of the same person with eyes open. The images in the image pairs were generated using the neural renderer of the detailed example and by controlling individual attributes as described herein. The users also had to indicate whether the images in a pair depicted the same person or not. The results of the user study found that the neural renderer was able to generate images which were perceived by the human subjects to have attributes controlled as expected. The results of the user study found that the neural renderer was able to control the images without influencing whether the face in the images of a pair were perceived by the human subjects as depicting the same person or not.

<FIG> illustrates various components of an exemplary computing-based device <NUM> which are implemented as any form of a computing and/or electronic device, and in which embodiments of a neural renderer are implemented in some examples.

Computing-based device <NUM> comprises one or more processors <NUM> which are microprocessors, controllers or any other suitable type of processors for processing computer executable instructions to control the operation of the device in order to generate images, edit existing images and optionally train a neural renderer. In some examples, for example where a system on a chip architecture is used, the processors <NUM> include one or more fixed function blocks (also referred to as accelerators) which implement a part of the method of operating or training a neural renderer in hardware (rather than software or firmware). Platform software comprising an operating system <NUM> or any other suitable platform software is provided at the computing-based device to enable application software to be executed on the device. A neural renderer <NUM> is at the computing-based device as well as data store <NUM>. Data store <NUM> stores parameter values, images and other data.

The computer executable instructions are provided using any computer-readable media that is accessible by computing based device <NUM>. Computer-readable media includes, for example, computer storage media such as memory <NUM> and communications media. Computer storage media, such as memory <NUM>, includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or the like. Computer storage media includes, but is not limited to, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM), electronic erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that is used to store information for access by a computing device. In contrast, communication media embody computer readable instructions, data structures, program modules, or the like in a modulated data signal, such as a carrier wave, or other transport mechanism. As defined herein, computer storage media does not include communication media. Therefore, a computer storage medium should not be interpreted to be a propagating signal per se. Although the computer storage media (memory <NUM>) is shown within the computing-based device <NUM> it will be appreciated that the storage is, in some examples, distributed or located remotely and accessed via a network or other communication link (e.g. using communication interface <NUM>).

The computing-based device <NUM> also comprises an output interface <NUM> arranged to output display information, such as output images and a graphical user interface to enable a user to enter parameter values, to a display device <NUM> which is separate from or integral to the computing-based device <NUM>. An input interface <NUM> is arranged to receive and process input from a capture device <NUM> such as a camera. The input interface <NUM> receives and processes input from one or more user input devices such as game controller <NUM>, keyboard <NUM>, mouse <NUM> or other user input device. In some examples one or more of the user input devices detects voice input, user gestures or other user actions and provides a natural user interface (NUI). This user input may be used to view output images, specify input images and specify parameter values for input to the neural renderer. In an embodiment the display device <NUM> also acts as a user input device if it is a touch sensitive display device.

Claim 1:
A computer-implemented method of image processing comprising:
storing a real image (<NUM>) of an object in memory, the object being a specified type of object;
computing, using a first encoder (<NUM>), a factorized embedding of the real image (<NUM>);
receiving a value of at least one parameter of a synthetic object rendering apparatus for rendering synthetic images of objects of the specified type, the parameter controlling an attribute of synthetic images of objects rendered by the rendering apparatus;
computing an embedding factor of the received value using a second encoder (<NUM>);
modifying the factorized embedding with the computed embedding factor; and
computing, using a decoder (<NUM>) with the modified embedding as input, an output image (<NUM>) of the object which is substantially the same as the real image (<NUM>) except for the attribute controlled by the parameter.