Patent ID: 12236558

DETAILED DESCRIPTION

Hereinafter, specific embodiments of the present invention will be described with reference to the accompanying drawings. The following detailed description is provided to assist in a comprehensive understanding of the methods, devices and/or systems described herein. However, the detailed description is only illustrative, and the present invention is not limited thereto.

In describing embodiments of the present invention, when a specific description of known technology associated with the present invention is deemed to make the gist of the present invention unnecessarily vague, the detailed description thereof will be omitted. The terms used below are defined in consideration of functions in the present invention, but may vary in accordance with the customary practice or the intention of a user or an operator. Therefore, the terms should be defined based on whole content throughout the present specification. The terms used herein are only for describing the embodiments of the present invention, and should not be construed as limited. A singular expression includes a plural meaning unless clearly used otherwise. In the present description, expressions such as “include” or “have” are for referring to certain characteristics, numbers, steps, operations, components, and some or combinations thereof, and should not be construed as excluding the presence or possibility of one or more other characteristics, numbers, steps, operations, components, and some or combinations thereof besides those described.

In the following description, the terms “transmission,” “communication,” and “reception” of a signal or information and other similar terms may include a meaning in which the signal or information is directly transmitted from one element to another element and transmitted from one element to another element through an intervening element. In particular, “transmission” or “sending” of the signal or information to one element may indicate a final destination of the signal or information and may not imply a direct destination. The same applies to “reception” of the signal or information. In addition, in the present specification, a meaning in which two or more pieces of data or information are “related” indicates that when any one piece of data (or information) is obtained, at least a portion of other data (or information) may be obtained based thereon.

Furthermore, the terms “first,” “second,” and the like may be used for describing various elements, but the elements should not be construed as being limited by the terms. These terms may be used for distinguishing one element from another element. For example, a first element could be termed a second element and vice versa without departing from the scope of the present invention.

FIG.1is a view showing the configuration of an image synthesis device capable of improving image quality according to an embodiment of the present invention.

Referring toFIG.1, an image synthesis device100may include a first artificial neural network model102and a second artificial neural network model104.

The first artificial neural network model102may be a model trained to use a damaged image as an input to output a restored image. Here, the damaged image may be an image in which a portion of an original image is damaged, deformed, or removed. In order to improve the image quality of the restored image, the first artificial neural network model102may additionally perform a task of using the original image as an input to output a reconstructed image (i.e., an image in which the original image is reconstructed) other than a task of using the damaged image as an input to output the restored image.

Here, the image quality may include one or more of the definition and resolution of an image. That is, improving the image quality of the restored image may include all of improvement in the definition of the restored image, improvement in the resolution of the restored image, and improvement in the definition and resolution of the restored image.

In the first artificial neural network model102, the task of using a damaged image as an input to output a restored image may be referred to as a first task, and the task of using an original image as an input to output a reconstructed image may be referred to as a second task. In an example embodiment, the first artificial neural network model102may be implemented by a convolutional neural network (CNN)-based machine learning technology, but the machine learning technology is not limited thereto, and other various machine learning technologies may be applied.

The second artificial neural network model104may be provided to improve the image quality of the reconstructed image output from the first artificial neural network model102. That is, the second artificial neural network model104may be provided to improve one or more of the definition and resolution of the reconstructed output from the first artificial neural network model102.

Specifically, the second artificial neural network model104may be trained to use the reconstructed image output from the first artificial neural network model102as an input to improve the image quality of the reconstructed image. That is, the second artificial neural network model104may be trained in conjunction with the second task of the first artificial neural network model102.

When the second artificial neural network model104is trained, when the restored image, which is a product of the first task of the first artificial neural network model102, is input to the second artificial neural network model104, the restored image with improved image quality may be obtained through the second artificial neural network model104. In an example embodiment, the second artificial neural network model104may be implemented by a convolutional neural network (CNN)-based machine learning technology, but the machine learning technology is not limited thereto, and other various machine learning technologies may be applied.

FIG.2is a view schematically showing the training process of the first artificial neural network model102and the second artificial neural network model104in an embodiment of the present invention.

Referring toFIG.2, the first artificial neural network model102may learn the first task to output a restored image using a damaged image as an input. Here, the first artificial neural network model102for learning the first task may be represented by Equation 1 below.
{circumflex over (X)}synth=F(X′;θ)  (Equation 1)

{circumflex over (X)}synth: Restored image output from first artificial neural network model102

F: Image constituting first artificial neural network model102

X′: Damaged image in which portion of original image X is damaged, deformed, or removed

θ: Parameter of neural network F

In an example embodiment, the damaged image X′ may be an image in which a mask is synthesized in the original image X. Here, the mask may be a binary mask which has the same size as the original image, and in which a damaged pixel (e.g., a pixel damaged, deformed, or removed from an original image) is set to 0, and a normal pixel is set to 1. However, the embodiment of the present invention is not limited thereto, and various types of masks may be used.

At this time, an objective function Lsynthfor learning the first task in the first artificial neural network model102may be represented by Equation 2 below.
Lsynth=∥X−{circumflex over (X)}synth∥  (Equation 2)

In Equation 2, X is an original image, the ∥A−B∥ function represents a function for obtaining the difference between A and B (e.g., a function obtaining the Euclidean distance (L2 distance) or the Manhattan distance (L1 distance) of A and B). That is, the first artificial neural network model102may be trained to minimize the difference between the original image and the restored image when learning the first task.

In addition, the first artificial neural network model102may learn the second task of using an original image as an input to output a reconstructed image. Here, the first artificial neural network model102for learning the second task may be represented by Equation 3 below.
{circumflex over (X)}recon=F(X;θ)  (Equation 3)

{circumflex over (X)}recon: Reconstructed image output from first artificial neural network model102

At this time, an objective function Lreconfor learning the second task in the first artificial neural network model102may be represented by Equation 4 below. That is, the first artificial neural network model102may be trained to minimize the difference between the original image and the reconstructed image when learning the second task.
Lrecon=∥X−{circumflex over (X)}recon∥  (Equation 4)

The first artificial neural network model102performs both the first task and the second task, and an objective function Lsynth-reconof the first artificial neural network model102for performing the first task and the second task may be represented by Equation 5 below.
Lsynth-recon=LsynthλLrecon(Equation 5)

λ: Weight between objective function Lsynthfor learning first task and objective function Lreconfor learning second task

In addition, an optimized parameter θ*of the first artificial neural network model102for performing the first task and the second task may be represented by Equation 6 below.
θ*=argminθ(Lsynth-recon)  (Equation 1)

Here, argminθrepresents a function that obtains θ for minimizing the Lsynth-recon. That is, according to Equation 5 and Equation 6, the first artificial neural network model102may be trained to o minimize the objective function Lsynthand the objective function Lrecon, simultaneously.

Since the reconstructed image output from the first artificial neural network model102according to the training of the second task uses the original image as an input image, the shapes of objects included in the original image are the same as those in the original image. However, since the first task is also trained in the first artificial neural network model102, there is a constraint in that parameters shared for the first task and the second tasks should be used, and as a result, the reconstructed image is output with image quality similar to that of the restored image. That is, the image quality of a reconstructed image when both the first task and the second task are trained is degraded compared to the image quality of a reconstructed image when only the second task is trained in the first artificial neural network model102.

Therefore, the reconstructed image output by the first artificial neural network model102by learning the second task may be input to the second artificial neural network model104to improve the image quality. The second artificial neural network model104may be trained to use the reconstructed image output from the first artificial neural network model102as an input to improve the image quality of the reconstructed image.

The second artificial neural network model104may be trained to improve the image quality of the reconstructed image using a known artificial intelligence-based definition and/or resolution improvement techniques. Here, the second artificial neural network model104for improving the image quality of the reconstructed image may be represented by Equation 7 below.
{circumflex over (X)}sr=G(Xrecon;γ)  (Equation 7)

{circumflex over (X)}sr: Image output from second artificial neural network model104

G: Neural network constituting second artificial neural network model104

γ: Parameter of neural network G

In addition, an objective function Lsrof the second artificial neural network model104may be represented by Equation 8 below.
Lsr=∥X−{circumflex over (X)}sr∥  (Equation 8)

That is, the second artificial neural network model104may be trained to minimize the difference between an image {circumflex over (X)}sroutput from second artificial neural network model104and the original image X. In this case, one or more of the definition and resolution of the image {circumflex over (X)}sroutput from second artificial neural network model104are improved, so that the difference with the original image is minimized.

When the second artificial neural network model104is trained as described above, the restored image, which is a product of the first task of the first artificial neural network model102, may be input to the second artificial neural network model104. Then, the second artificial neural network model104improves one or more of the definition and resolution of the restored image to output the restored image with improved image quality.

According to a disclosed embodiment, in the first artificial neural network model102, training is performed to perform both a first task of using a damaged image as an input to output a restored image and a second task of using an original image as an input to output a reconstructed image, and in the second artificial neural network model104, training is performed to improve the image quality of the reconstructed image, which is a product of the second task, so that it is possible to input the restored image, which is a product of the first task of the first artificial neural network model102, to the second artificial neural network model104to improve image quality. At this time, in the second artificial neural network model104, training is performed by using the reconstructed image, which is a product of the second task, as an input, so that it is possible to minimize the shape deformation of an object in an image while improving image quality.

Meanwhile, herein, the first artificial neural network model102has been described to perform both the first task and the second task in one artificial neural network model, but is not limited thereto, and as illustrated inFIG.3, the first artificial neural network model102may include a 1-1 artificial neural network model102-1and a 1-2 artificial neural network model102-2.

The 1-1 artificial neural network model102-1may be a model for learning a first task of using a damaged image as an input to output a restored image. The 1-2 artificial neural network model102-2may be a model for learning a second task of using an original image as an input to output a reconstructed image. Here, the 1-1 artificial neural network model102-1and the 1-2 artificial neural network model102-2may share neural network parameters with each other.

In addition, herein, the first artificial neural network model102and the second artificial neural network model104have been described as separate artificial neural network models, but are not limited thereto, and may be implemented as one artificial neural network model. That is, the neural network of the second artificial neural network model104may be added to the neural network of the first artificial neural network model102to be implemented as one artificial neural network model.

FIG.4andFIG.5are views schematically showing an embodiment in which the image synthesis device100of the present invention is implemented as one artificial neural network model. Referring toFIG.4, the image synthesis device100may include a first artificial neural network111and a second artificial neural network113.

The first artificial neural network111may be a neural network for learning the first task of using a damaged image as an input to output a restored image and the second task of using an original image as an input to output a reconstructed image. The first artificial neural network111may include a first encoder111aand a first decoder111b. The first encoder111amay use a damaged image or an original image as an input to extract a first image feature vector or a second image feature vector, respectively. The first decoder111bmay use the first image feature vector or the second image feature vector as an input to output a restored image or a reconstructed image, respectively.

The second artificial neural network113may be a neural network trained to use the reconstructed image output from the first artificial neural network111as an input to improve the image quality of the reconstructed image. The second artificial neural network113may be connected to an output layer of the first artificial neural network111. The second artificial neural network113may include a second encoder113aand a second decoder113b.

The second encoder113amay be connected to the first decoder111b. The second encoder113amay use the restored image or the reconstructed image output from the first decoder111bas an input to output a third image feature vector or a fourth image feature vector, respectively. The second decoder113bmay use the third image feature vector or the fourth image feature vector as an input to output the restored image or the reconstructed image with improved image quality, respectively.

In a training process, as illustrated inFIG.4, a damaged image may be input to the first artificial neural network111during learning of the first task to output a restored image. That is, during the learning the first task, it is possible to bypass without using the second artificial neural network113. During the learning of the second task, an original image may be input to the first artificial neural network111to output a reconstructed image, and the output reconstructed image may be input to the second artificial neural network113to output an image with improved image quality.

In a prediction process, as illustrated inFIG.5, a damaged image may be input to the first artificial neural network111to output a restored image, and the output restored image may be input to the second artificial neural network113to output the restored image with improved image quality.

Here, the second artificial neural network113for improving image quality is described as being connected to the output layer of the first artificial neural network111, but is not limited thereto, and the neural network of the second artificial neural network113may be included in the first artificial neural network111. For example, the neural network of the second artificial neural network113may be included in the first decoder111bof the first artificial neural network111.

FIG.6is a view schematically showing another embodiment in which the image synthesis device100of the present invention is implemented as one artificial neural network model. Referring toFIG.6, the image synthesis device100may include a first artificial neural network111and a second artificial neural network113. Here, the first artificial neural network111may include an encoder111aand a decoder111b. At this time, the second artificial neural network113may be added between neural network layers constituting the decoder111b. A plurality of the second artificial neural network113may be divided and inserted between the neural network layers constituting the decoder111b.

In the case of learning the first task, a damaged image may be inserted into the encoder111ato extract a feature, and then restored through only the decoder113b, bypassing the second artificial neural network113. In the case of learning the second task, an original image may be inserted into the encoder111ato extract a feature, and then reconstructed through the neural network constituting the second artificial neural network113, and the decoder113b.

FIG.7is a view showing the configuration of an image synthesis device according to another embodiment of the present invention. Here, an example is shown in which the image synthesis device synthesizes a lip sync image.

Referring toFIG.7, an image synthesis device200may include a first artificial neural network model202and a second artificial neural network model204. The first artificial neural network model202may include a 1-1 artificial neural network model202-1and a 1-2 artificial neural network model202-2.

The 1-1 artificial neural network model202-1may be a model for learning a first task of using a person background image as an input to output a lip sync image.

The 1-1 artificial neural network model202-1may include a first encoder211, a second encoder213, a combiner215, and a first decoder217.

The first encoder211may be trained to use a person background image as an input to extract an image feature vector. Hereinafter, the term “vector” may also be used to refer to a “tensor.”

Here, the person background image input to the first encoder211is an image in which a person utters (speaks). The person background image may be an image including a face and upper body of a person. That is, the person background image may be an image including not only the face but also the upper body of a person who utters so as to show movements of the face, neck, shoulder, and the like of the corresponding person, but is not limited thereto, and may be an image including the face of the person.

A portion associated with an utterance in the person background image input to the first encoder211may be masked. That is, the portion (e.g., a mouth and portions around the mouth) associated with the utterance in the person background image may be covered by a mask M. In addition, during a masking process, portions associated with facial movement, neck movement, shoulder movement, and the like as a result of the utterance of the person in the person background image may not be masked. Then, the first encoder211extracts an image feature vector of a portion excluding the portion associated with the utterance in the person background image.

In an example embodiment, the first encoder211may include at least one convolutional layer and at least one pooling layer. The convolutional layer, while moving a filter of a preset size (e.g., 3×3 pixel size) at regular intervals in the input person background image, may extract a feature value of pixels corresponding to the filter. The pooling layer may receive an output from the convolutional layer as an input to perform down sampling.

The second encoder213may be trained to use the utterance audio signal as an input to extract a voice feature vector. Here, the utterance audio signal corresponds to an audio portion in the person background image (i.e., an image in which a person utters) input to the first encoder211. In other words, a video portion in a video in which a person utters may be input to the first encoder211, and an audio portion may be input to the second encoder213. The second encoder213may include at least one convolutional layer and at least one pooling layer, but a neural network structure of the second encoder213is not limited thereto.

The person background image input to the first encoder211and the utterance audio signal input to the second encoder213may be synchronized in time. That is, in a section of the same time band in a video in which a person utters, video may be input to the first encoder211, and audio may be input to the second encoder213. For example, when the person background image is an image for time t from a specific point of time, the utterance audio signal may be a voice for the time t from the same point of time. At this time, the person background image and the utterance audio signal may be input to the first encoder211and the second encoder213every preset unit time (e.g., one frame or a plurality of successive frames).

The combiner215may combine the image feature vector output from the first encoder211and the voice feature vector output from the second encoder213to generate a combined vector. In an example embodiment, the combiner215may concatenate the image feature vector and the voice feature vector to generate a combined vector, but is not limited thereto.

The first decoder217may use the combined vector output from the combiner215as an input to generate a lip sync image. Specifically, the first decoder217may be trained to restore the portion (i.e., the portion associated with the utterance) covered by the mask M of the image feature vector (i.e., a video portion in a video in which a person utters, a feature of a portion in which a portion associated with an utterance is covered by a mask) output from the first encoder211, on the basis of the voice feature vector (i.e., a feature of an audio portion in the video in which a person utters) output from the second encoder113.

That is, when a portion associated with an utterance is masked in the person background image, the first decoder217may be a model trained to restore the masked region using the utterance audio signal. The first decoder217may compare a generated lip sync image with an original utterance image (i.e., a correct value), and may adjust a training parameter (e.g., a loss function, a softmax function, etc.) such that the generated lip sync image approximates the original utterance image (i.e., to minimize the difference with the original utterance image).

The 1-2 artificial neural network model202-2may be a model for learning a second task of using an original utterance image as an input to output a reconstructed image. The 1-2 artificial neural network model202-2may share neural network parameters with the 1-1 artificial neural network model202-1. The 1-2 artificial neural network model202-2may include a third encoder221and a second decoder223.

The third encoder221may be trained to use the original utterance image as an input to extract an image feature vector. Here, the original utterance image may be an image in which the mask M is removed from a person background image. That is, the person background image may be an image in which a portion associated with an utterance in the original utterance image is covered by the mask M.

The second decoder223may be trained to output a reconstructed image (i.e., an image in which the original utterance image is reconstructed) on the basis of the image feature vector output from the third encoder221. The second decoder223may adjust a training parameter (e.g., a loss function, a softmax function, etc.) to minimize the difference between the reconstructed image and the original utterance image.

The second artificial neural network model204may be a model trained to improve the image quality of a reconstructed image output from the 1-2 artificial neural network model202-2and output the reconstructed image with improved image quality. The second artificial neural network model204may include a fourth encoder231and a third decoder233.

The fourth encoder231may be trained to use the reconstructed image output from the 1-2 artificial neural network model202-2as an input to extract an image feature vector. The third decoder233may be trained to output a reconstructed image with improved image quality on the basis of the image feature vector output from the fourth encoder231.

When the second artificial neural network model204is trained, a lip sync image output from the 1-1 artificial neural network model202-1may be input to the fourth encoder231. Then, the fourth encoder231may extract an image feature vector from the lip sync image. The third decoder233may output a lip sync image with improved image quality on the basis of the image feature vector output from the fourth encoder231.

FIG.8is a block diagram illustrating a computing environment10that includes a computing device suitable for use in example embodiments. In the illustrated embodiment, each component may have different functions and capabilities in addition to those described below, and additional components may be included in addition to those described below.

The illustrated computing environment10includes a computing device12. In an embodiment, the computing device12may be the image synthesis device100or200.

The computing device12includes at least one processor14, a computer-readable storage medium16, and a communication bus18. The processor14may allow the computing device12to operate according to the example embodiment mentioned above. For example, the processor14may execute one or more programs stored in the computer-readable storage medium16. The one or more programs may include one or more computer-executable commands, and when executed by the processor14, the computer-executable command may be configured to allow the computing device12to perform operations according to the example embodiment.

The computer-readable storage medium16is configured to store computer-executable commands or program codes, program data, and/or other suitable types of information. A program20stored in the computer-readable storage medium16includes a set of commands executable by the processor14. In one embodiment, the computer-readable storage medium16may be a memory (a volatile memory such as a random access memory, a non-volatile memory, or any suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, other types of storage media accessible by the computing device12and capable of storing desired information, or any suitable combination thereof.

The communication bus18includes the processor14and the computer-readable storage medium16to interconnect various other components of the computing device12.

The computing device12may also include one or more input/output interfaces22which provide an interface for one or more input/output devices24, and one or more network communication interfaces26. The input/output interface22and the network communication interface26are connected to the communication bus18. The input/output device24may be connected to other components of the computing device12through the input/output interface22. The exemplary input/output device24may include a pointing device (such as a mouse or track pad), a keyboard, a touch input device (such as a touch pad or touch screen), a voice or sound input device, an input device such as various types of sensor devices and/or photographing devices, and/or an output device such as a display device, a printer, a speaker, and/or a network card. The exemplary input/output device24may be included inside the computing device12as one component constituting the computing device12, or may be connected to the computing device12as a separate device distinct from the computing device12.

Although the example embodiment of the present invention has been described in detail as above, those skilled in the art to which the present invention pertains will understand that various modifications may be made thereto within the limit that do not depart from the scope of the present invention. Therefore, the scope of rights of the present invention should not be limited to the described embodiments, but should be defined not only by claims set forth below but also by equivalents of the claims.