Patent Description:
Face reenactment can be used in many applications, such as entertainment shows, social media networks, computer games, video conversations, virtual reality, augmented reality, and the like. Face reenactment can be carried out by animating a single image of a person. Deep learning methods may allow obtaining photorealistic results. However, currently existing deep learning methods are time-consuming and are not suitable to perform a real-time face reenactment on standard mobile devices.

The scientific paper "<NPL>et al. discloses a method which takes any text as input and generates both the corresponding speech and synchronized photo-realistic lip-sync videos. The method uses trainable neural modules. In particular, a text-to-speech network based on Char2Wav, a time-delayed LSTM to generate mouth-keypoints synced to the audio, and a network based on Pix2Pix is used to generate the video frames. The model is trained on any set of close shot videos of a person speaking, along with the corresponding transcript. To compute the mouth-shape representation, the method uses mouth keypoints extracted from the face, and normalizes the points to be invariant to image size, face location, face rotation and face size. A video is generated by translating an input face image with cropped mouth area to an output image with in-painted mouth area, conditioned on the mouth shape representation.

Further, the scientific paper "<NPL>et al. discloses a method based on training with video footage of the target person. First audio input is converted to a time-varying sparse mouth shape. Based on this mouth shape, photo-realistic mouth texture is generated, that is composited into the mouth region of a target video. Before the final composite, the mouth texture sequence and the target video are matched and re-timed so that the head motion appears natural and fits the input speech.

In the scientific paper "<NPL>et al. , a method for analyzing scene dynamics is disclosed. The method is based on large amounts of unlabeled video in order to learn a model of scene dynamics for both video recognition tasks (e.g., action classification) and video generation tasks (e.g., future prediction). The method uses a generative adversarial network for video with a spatio-temporal convolutional architecture that untangles the scene's foreground from the background.

This section is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description section.

A method for text and audio-based real-time face reenactment is disclosed. The method includes receiving, by a computing device, an input text and a target image. The target image includes a target face. The method further includes generating, by the computing device and based on the input text, a sequence of sets of acoustic features representing the input text. The method may include generating, by the computing device and based on the sequence of sets of acoustic features, a sequence of sets of scenario data. The sets of scenario data may indicate modifications of the target face for uttering the input text. The method may include generating, by the computing device and based on the sequence of sets of scenario data, a sequence of frames. Each of the frames may include the target face modified based on at least one set of scenario data of the sequence of sets of scenario data. The method may include generating, by the computing device and based on the sequence of frames, an output video.

The method may further include synthesizing, by the computing device and based on the sequence of sets of acoustic features, an audio data representing the input text and adding the audio data to the output video. The acoustic features may include Mel-frequency cepstral coefficients. The sequence of sets of acoustic features can be generated by a neural network.

The generating the sequence of sets of scenario data may include generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points and generating, based on the sequence of sets of mouth key points, a sequence of sets of facial key points. The generating the sequence of frames may include determining, based on a sequence of sets of facial key points, a sequence of sets of two-dimensional (2D) deformations and applying each set of 2D deformations of the sequence of the sets of 2D deformations to the target input.

The sequence of sets of mouth key points can be generated by a neural network. A set of the sequence of sets of mouth key points can be generated based on a pre-determined number of sets preceding the set in the sequence of sets of mouth key points. The set of the sequence of sets of mouth key points may correspond to at least one set (S) of the sequence of sets of acoustic features. The set of the sequence of sets of mouth key points can be generated based on a first pre-determined number of sets of acoustic features preceding the S in the sequence of sets of acoustic features and a second pre-determined number of sets of acoustic features succeeding the S in the sequence of sets of acoustic features.

The sequence of sets of facial key points can be generated by a neural network. A set of the sequence of sets of facial key points can be generated based on a pre-determined number of sets preceding the set in the sequence of sets of facial key points.

The method may further include generating, by the computing device and based on the sequence of sets of mouth key points, a sequence of mouth texture images. The method may further include inserting, by the computing device, each of the sequence of mouth texture images in a corresponding frame of the sequence of the frames. Each mouth texture image of the sequence of mouth texture images can be generated by a neural network based on a first pre-determined number of mouth texture images preceding the mouth region image in the sequence of mouth region images.

A system for text and audio-based real-time face reenactment is also disclosed. The system may include at least one processor and a memory storing processor-executable codes, wherein the at least one processor can be configured to implement operations of the above-mentioned method for text and audio-based real-time face reenactment upon execution of the processor-executable codes.

According to yet another aspect of the disclosure, there is provided a non-transitory processor-readable medium, which stores processor-readable instructions. When the processor-readable instructions are executed by a processor, they cause the processor to implement the above-mentioned method for text and audio-based real-time face reenactment.

Additional objects, advantages, and novel features of the examples will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following description and the accompanying drawings or may be learned by production or operation of the examples. The objects and advantages of the concepts may be realized and attained by means of the methodologies, instrumentalities and combinations particularly pointed out in the appended claims.

Embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements.

The present disclosure can be implemented using a variety of technologies. For example, methods described herein can be implemented by software running on a computer system and/or by hardware utilizing either a combination of microprocessors or other specifically designed application-specific integrated circuits (ASICs), programmable logic devices, or any combinations thereof. In particular, the methods described herein can be implemented by a series of computer-executable instructions residing on a non-transitory storage medium such as a disk drive or computer-readable medium. The present disclosure can be designed to work on mobile devices, such as smartphones, tablet computers, or mobile phones, in real-time and without connection to the Internet or the need for use of server-side computational resources, although the embodiments can be extended to approaches involving web service or a cloud-based resources.

This disclosure relates to methods and systems for text and audio-based real-time face reenactment. Some embodiments of the disclosure may allow animation of a face of a person in a target image based on an input text. Embodiments of the disclosure may allow generating a video in which the person's face on the target image becomes "alive" and pronounces the input text. One example embodiment may include a mobile application. The mobile application may allow a user to change an audio in an existing video (for example, a video reel). The mobile application may allow changing visuals and audio of the existing video based on the input text. The user can record or select a new audio for the video or, preferably, enter a new text. The mobile application may synthesize new audio based on the new text and animate a face in the video in accordance with the new audio and new text. Some embodiments may allow transferring an audio style of initial video reel into the synthesized audio.

According to one embodiment of the disclosure, an example method for text and audio-based real-time face reenactment may include receiving, by a computing device, an input text and a target image. The target image may include a target face. The method may include generating, by the computing device and based on the input text, a sequence of sets of acoustic features representing the input text. The method may include generating, by the computing device and based on the sequence of sets of acoustic features, a sequence of sets of scenario data. The sets of scenario data may indicate modifications of the target face for pronouncing the input text. The method may include generating, by the computing device and based on the sequence of sets of scenario data, a sequence of frames. Each of the frames includes the target face modified based on at least one set of scenario data of the sequence of sets of scenario data. The method may include generating, by the computing device and based on the sequence of frames, an output video. The method may further include synthesizing, by the computing device and based on the sequence of sets of acoustic features, an audio data representing the input text. The method may include adding, by the computing device, the audio data to the output video.

Referring now to the drawings, exemplary embodiments are described. The drawings are schematic illustrations of idealized example embodiments. Thus, the example embodiments discussed herein should not be understood as limited to the particular illustrations presented herein, rather these example embodiments can include deviations and differ from the illustrations presented herein as shall be evident to those skilled in the art.

<FIG> shows an example environment <NUM>, wherein a method for text and audio-based real-time face reenactment can be practiced. The environment <NUM> may include a computing device <NUM> and a user <NUM>. The computing device <NUM> may include a graphical display system <NUM>. The computing device <NUM> can refer to a mobile device such as a mobile phone, smartphone, or tablet computer. In further embodiments, however, the computing device <NUM> can refer to a personal computer, laptop computer, netbook, set top box, television device, multimedia device, personal digital assistant, game console, entertainment system, infotainment system, vehicle computer, or any other computing device.

In some embodiments of the disclosure, the computing device <NUM> can be configured to display a target image <NUM>. The target image <NUM> may include at least a target face <NUM> and a background <NUM>. The target face <NUM> may belong to the user <NUM> or a person other than the user <NUM>. In some embodiments, the target image <NUM> can be stored in a memory storage of the computing device <NUM> or in a cloud-based computing resource to which the computing device <NUM> is communicatively connected.

In yet other embodiments, the target image <NUM> can be pre-recorded and stored in the memory of the computing device <NUM> or in the cloud-based computing resource. A user <NUM> may select the target image to be animated.

In some embodiments, the computing device <NUM> can be configured to receive an input text <NUM>. The computing device <NUM> can be configured to analyze the input text to generate a sequence of sets of scenario data indicating modifications of the target face for pronouncing the input text <NUM>. The computing device <NUM> can be further configured to modify the target image <NUM> based on the scenario data to generate a sequence of frames. Each frame for the sequence of frames may include the target face modified based on one of the sets of scenario data. The computing device <NUM> may further generate a video based on the sequence of frames. The computing device <NUM> may further synthesize an audio data based on the input text and add the audio data to the video. The computing device <NUM> may play the video via the graphical display system <NUM>. The video may feature a person with the target face, wherein the person pronounces the input text.

In the example shown in <FIG>, the computing device <NUM> may include both hardware components and software components. Particularly, the computing device <NUM> may include a camera <NUM> or any other image-capturing device or scanner to acquire digital images. The computing device <NUM> can further include a processor module <NUM> and a storage module <NUM> for storing software components and processor-readable (machine-readable) instructions or codes, which when performed by the processor module <NUM> cause the computing device <NUM> to perform at least some steps of methods for text and audio-based real-time face reenactment as described herein.

The computing device <NUM> can further include a face reenactment system <NUM>, which, in turn, can include hardware components (e.g., a separate processing module and memory), software components, or a combination thereof.

As shown in <FIG>, the face reenactment system <NUM> can be configured to receive, as an input, a target image <NUM> and an input text <NUM>. The target image <NUM> may include a target face <NUM>. The face reenactment system <NUM> can be configured to analyze the text input to determine a sequence of sets of acoustic features representing the input text. Each of the sets of acoustic features may correspond to a timestamp which can be assigned to one of the frames <NUM> of output video <NUM>.

Based on the sequence of sets of acoustic features, the face reenactment system <NUM> may generate a sequence of sets of scenario data. Each set of the scenario data may indicate modifications of the target face <NUM> in at least one of the frames <NUM>. Each of the sets of scenario data may include a set of mouth key points, a mouth texture image, and a set of facial key points. The mouth key points may represent facial landmarks located substantially around a mouth region of a model face. The facial key points may represent facial landmarks located at least around face shape, in regions of nose, eyes, and brows of the model face. Thus, each of the sets of the scenario data may represent a facial expression and head movement of the model face.

<FIG> shows mouth key points and facial key points of a model face <NUM>, according to an example embodiment. In example <FIG>, a set of mouth key points includes <NUM> key points enumerated from <NUM> to <NUM>. The mouth key points <NUM>-<NUM> are located substantially around a mouth region of the model face <NUM>. In some embodiments, the facial key points may include the <NUM> mouth key points and additional facial landmarks around face shape, in regions of nose, eyes, and brows. In example of <FIG>, the number of facial key points is <NUM>. The facial key points are enumerated from <NUM> to <NUM>. The facial key points and mouth key points can be represented as points in a 2D space. The facial key points and mouth key points may correspond to particular facial landmarks (for example, a corner of a brow, a corner of an eye, a corner of mouth, a bottom of a chin, and so forth).

Referring back to <FIG>, the face reenactment system <NUM> may further transfer the facial expression and head movement of the model face to the target image <NUM>. The output video <NUM> may include the same number of frames as the number of sets in the sequence of sets of scenario data. As result, the output video <NUM> may represent an animation of the target image <NUM>. In some embodiments, the animation can be carried out by performing 2D deformations of the target image <NUM>, wherein the 2D deformations imitate the facial expression and the head movements of the model face. In some embodiments, the 2D deformations can be followed by generation of hidden regions and fine-scale details to achieve photorealistic results. The hidden regions may include a mouth region of the target face <NUM>.

The face reenactment system <NUM> may further generate, based on the sequence of sets of acoustic features, an audio data representing the input text and add the audio data to the output video <NUM>.

<FIG> is a block diagram of a face reenactment system <NUM>, according to one example embodiment. The face reenactment system <NUM> can include a three-dimensional (3D) face model <NUM>, a sparse correspondence module <NUM>, a scenario data generation module <NUM>, a target image pre-processing module <NUM>, image segmentation and background prediction module <NUM>, and an image animation and refinement module <NUM>. The modules <NUM>-<NUM> can be implemented as software components for use with hardware devices such as computing device <NUM>, a server, and the like.

In some embodiments of the disclosure, the 3D face model <NUM> can be pre-generated based on images of a pre-defined number of individuals of different ages, genders, and ethnic backgrounds. For each of the individuals, the images may include an image of the individual having a neutral facial expression and one or more images of the individual having different facial expressions. The facial expression may include mouth-open, smiling, angry, astonished, and so forth.

The 3D face model <NUM> may include a template mesh with a pre-determined number of vertices. The template mesh may be represented as 3D triangulation defining a shape of a head. Each of the individuals can be associated with an individual-specific blend shape. The individual-specific blend shape can be adjusted to the template mesh. The individual-specific blend shape may correspond to specific coordinates of vertices in the template mesh. Thus, different images of individuals may correspond to the template mesh of the same structure. However, coordinates of vertices in the template mesh are different for the different images.

In some embodiments of the disclosure, the 3D face model <NUM> may include a bilinear face model depending on two parameters: facial identity and facial expression. The bilinear face model can be built based on blend shapes corresponding to the images of individuals. Thus, the 3D face model includes the template mesh of a pre-determined structure, wherein the coordinates of vertices depend on facial identity and facial expression. The facial identity may represent a geometrical shape of a head.

In some embodiments of the disclosure, the scenario data generation module <NUM> can be configured to generate, based on an input text, a sequence of sets of scenario data and an audio data. Each set of the scenario data may include a set of facial key points which represent 2D facial landmarks of a model face. The scenario data generation module <NUM> can be configured to fit the 3D face model <NUM> to the 2D facial landmarks to find parameters of 3D face model <NUM> for the model face. The scenario data generation module <NUM> can be configured to determine location of the 2D facial landmarks on the template mesh of the 3D face model. It can be assumed that the facial identity is the same for all frames of the scenario video <NUM>. The module <NUM> can be further configured to approximate the resulting changes of 3D face parameters for each set of the sequence of the scenario data. The scenario data generation module <NUM> can be also configured to generate, based on an input text, a sequence of mouth texture images. Further details of the scenario data generation module <NUM> are described further with connection to <FIG>.

In some embodiments, the sparse correspondence module <NUM> can be configured to determine a sparse correspondence between facial key points of each set in the sequence of sets of scenario data and facial key points of the target image <NUM>. The sparse correspondence module <NUM> can be configured to obtain a set of control points (facial landmarks), which can be robustly tracked through the scenario video. Facial landmarks and additional control points can be tracked using state-of-the-art tracking methods, such as optical flow. The sparse correspondence module <NUM> can be configured to determine an affine transformation that approximately aligns the facial key points in the first set of the sequence of sets of scenario data and facial key points in the target image <NUM>. The affine transformation can be further used to predict the location of additional control points in the target image <NUM>. The sparse correspondence module <NUM> can be further configured to build a triangulation of the control points.

In some embodiments, the target image preprocessing module <NUM> may be configured to detect 2D facial landmarks and visible parts of the head in the target image <NUM> and fit the 3D facial model to the 2D facial landmarks and visible parts of the head in the target image <NUM> and target face <NUM>. The target face <NUM> may not have a neutral facial expression, or have eyes closed or mouth open, and a person depicted on the target image may be of a different age than the person depicted in the scenario video <NUM>. The module <NUM> can be configured to normalize the target face (for example, rotate the head to a neutral state, close the mouth or open the eyes of the target face). Facial landmark detection and 3D face model fitting can be carried out using an iterative process. In some embodiments, the iterative process can be optimized for central processing unit (CPU) and graphics processing unit (GPU) of a mobile device which may allow to significantly decrease the time needed for pre-processing of the target image <NUM>.

In some embodiments, the target image preprocessing module <NUM> can be further configured to apply beauty effects and/or change the appearance of person depicted on the target image <NUM> (for example, change the hair color or hairstyle of the person or make the person look older or younger).

In some embodiments, the image segmentation and background separation module <NUM> can be configured to perform segmentation of a head of a person from an image of the person. The segmentation of the head can be performed for the target image to obtain an image of the head and/or the target face <NUM>. Animation can be further carried out on the image of the head or the target face <NUM>. The animated head and/or target face <NUM> can be further inserted back into the background <NUM>. Animating only the image of the head and/or face target <NUM> by applying 2D deformation may help to avoid unnecessary changes in background <NUM> which may be caused by the 2D deformations. Since the animation may include changes in head pose, some parts of a background which are previously invisible may become visible leading to gaps in the resulting image. To fill the gaps, the part of the background which is covered by the head can be predicted. In some embodiments, a deep learning model can be trained to perform the segmentation of a person's head from an image. Similarly, deep learning techniques can be used for predictions of background.

In some embodiments, the image animation and refinement module <NUM> can be configured to animate a target image frame by frame. For each set of facial key points in the sequence of sets of the scenario data, changes in positions of the control points can be determined. The changes in position of control points can be projected onto the target image <NUM>. The module <NUM> can be further configured to build a warp field. The warp field can include a set of piecewise linear transformations induced by changes of each triangle in triangulation of the control points. The module <NUM> can be further configured to apply the warp field to the target image <NUM> and by doing so produce a frame of output video <NUM>. Application of the warp field to an image can be performed relatively fast, which may allow the animation to be performed in real time.

In some embodiments, the image animation and refinement module <NUM> can be further configured to generate hidden regions (for example, the inner mouth region). Several approaches can be used to generate the hidden region. One approach may include inserting the mouth texture images generated by the scenario data generation module <NUM> based on the input text. The mouth texture images can be inserted to the mouth interior of a person in the target image. Another approach may include generating hidden regions with the use of a 3D mouth model. The 3D mouth model can match the geometry of the 3D face model.

In some embodiments, the module <NUM> can be configured to generate partly occluded regions (like mouth, iris, or eyelids) and fine-scale details. Generative adversarial networks could be used to synthesize realistic textures and realistic eye images. The module <NUM> may be further configured to replace eyes in the hidden regions of the target image with the realistic eye images generated using the generative adversarial networks. The module <NUM> can be configured to generate the photorealistic textures and fine-scale details for the target image based on the target image and original and current parameters of a 3D face model. The module <NUM> may further refine the target image by replacing hidden regions with generated photorealistic textures and applying the fine-scale details to the whole target image. Applying the fine-scale details may include applying shadow masks to each frame of the target image.

In some embodiments, the module <NUM> can be further configured to apply other effects (for example, color correction and light correction) on target images that are required to make the animation look realistic.

<FIG> is a block diagram showing an example scenario data generation module <NUM>, according to one example embodiment. The scenario data generation module may include a character embedding module <NUM>, a deep neural network (DNN) <NUM>, a mouth key points generation module <NUM>, a mouth texture generation module <NUM>, a facial key points generation module <NUM>, and a vocoder <NUM>.

In some embodiments, the character embedding module <NUM> may generate, based on an input text <NUM>, linguistic features <NUM>. The module <NUM> may pre-process the input text <NUM>. The pre-processing may include replacing abbreviations in the input text with words, converting the input text to lower case, removing unnecessary whitespaces, converting numbers to words, and so forth. The module <NUM> may further convert the words to phonemes using a pronouncing dictionary. The module <NUM> may further convert the phonemes to a sequence of linguistic features <NUM>. The linguistic features <NUM> can be represented as padded sequences of one-hot vectors.

In some embodiments, the DNN <NUM> may convert the sequence of linguistic numerical features to a sequence of sets of acoustic (numerical) features <NUM>. Each of the sets of acoustic (numerical) features <NUM> may be assigned a timestamp. Each of the sets of acoustic features may correspond to one of frames <NUM> in output video <NUM>. In some embodiments, the output video <NUM> can be generated at a rate of an N frames per second (for example, <NUM> frames per second). Thus, the number of sets in the sequence of the sets of acoustic features can be determined based on a desired frame rate of the output video. Additionally, the number of sets can be also determined based on the desired duration of an audio representing the input text.

The acoustic features may be presented as a Mel-frequency cepstrum or a set of speech parameters including fundamental frequency (F0), a spectral envelope (SP), and aperiodicity (AP). Generation of acoustic numerical features can be conditioned based on speaker identification data or speaker attributes. The speaker attributes may include "male," female," "child," "accent," and so forth. In these embodiments, DNN <NUM> may be configured to receive, as an input, an additional one-hot encoded vector representing speaker identification data or speaker attributes. In further embodiments, DNN <NUM> may receive an audio sample. The DNN <NUM> may include fine tuning an embedding layer to transfer a style of the audio sample to the acoustic features. In some embodiments, DNN <NUM> can be configured to synthesize singing. For singing synthesis, DNN <NUM> can be conditioned on continuous pitch parameter and linguistic features as control input.

In some embodiments, the vocoder <NUM> may apply a deterministic algorithm that decode set of speech parameters (a fundamental frequency (F0), a spectral envelope (SP), and aperiodicity (AP)) to produce an audio data (for example, a speech waveform as output). In other embodiments, the vocoder <NUM> may include a neural vocoder based on a neural network. The neural vocoder may decode a Mel-frequency cepstrum and produce the speech waveform.

In some embodiments, the mouth key points generation module <NUM> may generate, based on the sequence of the sets of acoustic features, a sequence of sets of mouth key points <NUM>. Each of the sets of mouth key points can correspond to one of frames <NUM> of output video <NUM> (shown in <FIG>) and be assigned a timestamp of the corresponding frame. The sets of mouth key points <NUM> can be generated sequentially by a neural network based on sets of acoustic features <NUM>. The mouth key points can be represented by a list of x-coordinates and y-coordinates for each frame.

At each step, the neural network can use, as an input, a pre-determined number of sets of mouth key points generated for previous frames by the same neural network. To obtain a temporal coherence, the neural network can also use, as an input, sets of acoustic features corresponding to a fixed-length time window. The time window may cover a pre-determined time before a current timestamp and a pre-determined time after the current timestamp. The acoustic features can be presented in the form of Mel Frequency Cepstral Coefficients. The neural network may apply convolutions to the acoustic features and the key points to extract latent features and then to concatenate the latent features. The neural network may further apply one or more fully connected layers to concatenated latent features to obtain final prediction for the key points at current timestamp. The neural network can be trained on a set of videos recorded in a controlled environment and featuring one or more actors speaking different predefined sentences.

During the training the neural network, the mouth key points of training sets can be normalized on each frame independently using affine transformation. The normalization may include determining coordinates of the mouth key points in a Cartesian coordinate system. Affine transformation is applied to the mouth key points in such a way that points corresponding to mouth corners (the key points numbers <NUM> and <NUM> in <FIG>) are moved to points (⅓, ½) and ( <MAT>, ½), respectively. The affine transformation may include combination of rotation, translation and scaling.

In some embodiments, during the training the neural network, a principal component analysis (PCA) can be further performed on the mouth key points of the training sets. In these embodiments, the neural network can be trained to predict PCA coefficients instead of raw coordinates of the mouth key points. Configuring the neural network to predict PCA coefficients may allow to decorrelate the input features, decorrelate output values, and reduce sizes of input and output of the neural network. The output of the neural network in the form of the PCA coefficients can be more interpretable than the output in the form of raw coordinates.

In some embodiments, the mouth texture generation module <NUM> can be configured to generate a sequence of mouth texture images <NUM>. Each of the mouth texture images <NUM> can correspond to one frame <NUM> of output video <NUM>. The sequence of the mouth texture images can be generated by a convolutional neural network (CNN) based on the sequence of the sets of mouth key points <NUM> and, optionally, the sequence of the sets of acoustic features <NUM>. The CNN may sequentially generate each of the mouth texture images based on a pre-determined number of previously generated mouth texture images. The CNN may also use, as an input, a pre-determined number of masks generated based on the sequence of mouth key points from a time window around a current timestamp. The current timestamp corresponds to the frame for which the mouth texture image is generated.

<FIG> shows an example mask <NUM> generated for a set of mouth key points, according to an example embodiment. The mask may include lines connecting internal mouth key points and lines connecting external mouth key points.

<FIG> shows an example mouth texture image <NUM>.

Referring back to the <FIG>, the CNN used in the mouth texture generation module <NUM> can be trained on a training set generated based on real videos recorded in a controlled environment with a single actor speaking different predefined sentences. The single actor can be the same in all the videos.

To achieve time-consistency, the training neutral network may include the following iterations:.

A fixed number of the iterations can be performed to obtain consistency between consecutive images in the sequence of the mouth texture images.

To achieve photo-realism of the mouth texture images, a "discriminator" neural network, such as Generative Adversarial Network (GAN), can be used with the CNN ("generator") to generate the mouth texture images. The discriminator is used to classify images between real images (ground-truth images taken from real videos) and fake images (images generated by a generator). At each iteration of the training, the generator may "fool" the discriminator to maximize a probability of the generated image being "real" according to the discriminator.

Additionally, a multiscale U-net-like architecture for GAN, a feature matching loss (in discriminator), and a perceptual loss can be used to improve photo-realism and resolution of generated mouth texture images. A basic GAN loss function can be replaced by a more advanced GAN loss function (for example, a loss function from Least-Square GAN (LSGAN)). The LSGAN loss function can be described by formulas: <MAT> <MAT>
wherein Gloss is a generator loss and Dloss is a discriminator loss, x is a generated image, and c are facial landmarks of an image on which the generated image is conditioned.

During training the CNN, mouth key points, corresponding mouth images, and key points mask images can be normalized as described above with connection to the mouth key points generation module <NUM>.

In some embodiments, the facial key points generation module <NUM> may generate, based on the sequence of sets of mouth key points <NUM>, a sequence of sets of facial key points <NUM>. An example set of facial key points and an example set of mouth key points were described above in connection with <FIG>. The sequence of sets of facial key points <NUM> can be generated by a neural network. The neural network can sequentially generate each of the sets of face key points for each frame based on a fixed number of sets of face key points previously generated by the same neural network. For generation of each of the sets of face key points, the neural network may also receive, as an input, a pre-determined number of sets of mouth key points generated for frames of a fixed-length time window before current timestamp.

The neural network for generating the sets of facial key points can be trained on a set of real videos recorded in a controlled environment and featuring a single actor speaking different predefined sentences. The single actor can be the same in all the videos.

During the training, the mouth key points can be normalized as described above with connection of the mouth key points generation module <NUM>. Face key points can be normalized with an affine transformation determined only on the first frame of the video. The affine transformation can then be applied to each next frame of the video. The normalization includes determine coordinates of a pre-determined number of facial key points in a Cartesian coordinate system. The affine transformation can be determined based on restrictions that a left eye center is moved to point (⅓, ⅓), a right eye center is moved to point ( <MAT>, ⅓), and a mouth center is moved to the point (½, <MAT>).

<FIG> shows screenshots <NUM>, <NUM>, and <NUM> of an example mobile application for a computing device <NUM>, according to an example embodiment. The mobile application may include the face reenactment system <NUM>.

As shown in screenshot <NUM>, the mobile application may display a target image <NUM> with a target face <NUM> to be animated and an input text <NUM>. The mobile application may generate, using the face reenactment system <NUM>, audio data (sound waveform) for the input text <NUM> and a video animating the target face <NUM> in the target image <NUM>.

As shown in screenshot <NUM>, the mobile application may include a user interface to modify the input text <NUM> using a keyboard input. In further embodiments, the mobile application may allow changing text style, font, size, position, and color, selection of text appearance time and animation, visual effects, and an audio filter. The audio filter can be based on a preset style. The mobile application may start generating, based on the modified input text, a sequence of sets of scenario data and playing the animated target image <NUM> when user taps a "Done" button.

As shown in screenshot <NUM>, original input text <NUM> is replaced by the text modified by the user. A vocoder may generate an audio data to match the input text <NUM>. The face reenactment system <NUM> may generate an animation of the target face <NUM> to match the audio data. In further embodiments, the mobile device may allow to enter an input text and/or audio via an audio or video recording.

<FIG> is a flow chart showing a method <NUM> for text and audio-based real-time face reenactment, according to some example embodiments. The method <NUM> can be performed by computing device <NUM> and the face re-enactment system <NUM>.

The method <NUM> may commence, in block <NUM>, with receiving, by the computing device, an input text and a target image. The target image may include a target face.

In block <NUM>, the method <NUM> may include generating, by the computing device and based on the input text, a sequence of sets of acoustic features representing the input text.

In block <NUM>, the method <NUM> may include generating, by the computing device and based on the sequence of sets of acoustic features, a sequence of sets of scenario data. The sets of scenario data may indicate modifications of the target face for pronouncing the input text.

In block <NUM>, the method <NUM> may include generating, by the computing device and based on the sequence of sets of scenario data, a sequence of frames. Each of the frames may include the target face modified based on at least one set of scenario data of the sequence of sets of scenario data.

In block <NUM>, the method <NUM> may include generating, by the computing device and based on the sequence of frames, an output video.

In block <NUM>, the method <NUM> may include synthesizing, by the computing device and based on the sequence of sets of acoustic features, an audio data representing the input text.

In block <NUM>, the method <NUM> may include adding, by the computing device, the audio data to the output video.

<FIG> illustrates an example computing system <NUM> that may be used to implement methods described herein. The computing system <NUM> may be implemented in the contexts of the likes of the computing device <NUM>, the face reenactment system <NUM>, the 3D face model <NUM>, the sparse correspondence module <NUM>, the scenario data generation module <NUM>, the target image preprocessing module <NUM>, and the image animation and refinement module <NUM>.

As shown in <FIG>, the hardware components of the computing system <NUM> may include one or more processors <NUM> and memory <NUM>. Memory <NUM> stores, in part, instructions and data for execution by processor <NUM>. Memory <NUM> can store the executable code when the system <NUM> is in operation. The system <NUM> may further include an optional mass storage device <NUM>, optional portable storage medium drive(s) <NUM>, one or more optional output devices <NUM>, one or more optional input devices <NUM>, an optional network interface <NUM>, and one or more optional peripheral devices <NUM>. The computing system <NUM> can also include one or more software components <NUM> (e.g., ones that can implement the method for portrait animation as described herein).

The components shown in <FIG> are depicted as being connected via a single bus <NUM>. The components may be connected through one or more data transport means or a data network. The processor <NUM> and memory <NUM> may be connected via a local microprocessor bus, and the mass storage device <NUM>, peripheral device(s) <NUM>, portable storage device <NUM>, and network interface <NUM> may be connected via one or more input/output (I/O) buses.

The mass storage device <NUM>, which may be implemented with a magnetic disk drive, solid-state disk drive, or an optical disk drive, is a non-volatile storage device for storing data and instructions for use by the processor <NUM>. Mass storage device <NUM> can store the system software (e.g., software components <NUM>) for implementing embodiments described herein.

Portable storage medium drive(s) <NUM> operates in conjunction with a portable non-volatile storage medium, such as a compact disk (CD) or digital video disc (DVD), to input and output data and code to and from the computing system <NUM>. The system software (e.g., software components <NUM>) for implementing embodiments described herein may be stored on such a portable medium and input to the computing system <NUM> via the portable storage medium drive(s) <NUM>.

The optional input devices <NUM> provide a portion of a user interface. The input devices <NUM> may include an alphanumeric keypad, such as a keyboard, for inputting alphanumeric and other information, or a pointing device, such as a mouse, a trackball, a stylus, or cursor direction keys. The input devices <NUM> can also include a camera or scanner. Additionally, the system <NUM> as shown in <FIG> includes optional output devices <NUM>. Suitable output devices include speakers, printers, network interfaces, and monitors.

The network interface <NUM> can be utilized to communicate with external devices, external computing devices, servers, and networked systems via one or more communications networks such as one or more wired, wireless, or optical networks including, for example, the Internet, intranet, LAN, WAN, cellular phone networks, Bluetooth radio, and an IEEE <NUM>-based radio frequency network, among others. The network interface <NUM> may be a network interface card, such as an Ethernet card, optical transceiver, radio frequency transceiver, or any other type of device that can send and receive information. The optional peripherals <NUM> may include any type of computer support device to add additional functionality to the computer system.

The components contained in the computing system <NUM> are intended to represent a broad category of computer components. Thus, the computing system <NUM> can be a server, personal computer, hand-held computing device, telephone, mobile computing device, workstation, minicomputer, mainframe computer, network node, or any other computing device. The computing system <NUM> can also include different bus configurations, networked platforms, multi-processor platforms, and so forth. Various operating systems (OS) can be used including UNIX, Linux, Windows, Macintosh OS, Palm OS, and other suitable operating systems.

Some of the above-described functions may be composed of instructions that are stored on storage media (e.g., computer-readable medium or processor-readable medium). The instructions may be retrieved and executed by the processor. Some examples of storage media are memory devices, tapes, disks, and the like. The instructions are operational when executed by the processor to direct the processor to operate in accord with the invention. Those skilled in the art are familiar with instructions, processor(s), and storage media.

It is noteworthy that any hardware platform suitable for performing the processing described herein is suitable for use with the invention. The terms "computer-readable storage medium" and "computer-readable storage media" as used herein refer to any medium or media that participate in providing instructions to a processor for execution. Such media can take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks, such as a fixed disk. Volatile media include dynamic memory, such as system random access memory (RAM). Transmission media include coaxial cables, copper wire, and fiber optics, among others, including the wires that include one embodiment of a bus. Transmission media can also take the form of acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-read-only memory (ROM) disk, DVD, any other optical medium, any other physical medium with patterns of marks or holes, a RAM, a PROM, an EPROM, an EEPROM, any other memory chip or cartridge, a carrier wave, or any other medium from which a computer can read.

Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution. A bus carries the data to system RAM, from which a processor retrieves and executes the instructions. The instructions received by the system processor can optionally be stored on a fixed disk either before or after execution by a processor.

Claim 1:
A computer-implemented method comprising:
receiving (<NUM>) an input text (<NUM>, <NUM>) and a target image (<NUM>), the target image (<NUM>) including a target face (<NUM>);
generating (<NUM>), based on the input text (<NUM>, <NUM>), a sequence of sets of acoustic features (<NUM>) corresponding to the input text (<NUM>, <NUM>);
generating (<NUM>), based on the sequence of sets of acoustic features (<NUM>), a sequence of sets of mouth key points (<NUM>) representing facial landmarks located substantially around a mouth region of a model face;
generating, based on the sequence of sets of mouth key points (<NUM>), a sequence of sets of facial key points (<NUM>) representing facial landmarks located at least around face shape, in regions of nose, eyes, and brows of the model face;
determining, based on the sequence of sets of facial key points (<NUM>), a sequence of sets of two-dimensional, 2D, deformations of the target face (<NUM>); and
applying the sequence of deformations to the target image (<NUM>), thereby generating (<NUM>), a sequence of frames (<NUM>) of an output video (<NUM>).