Patent ID: 11922723
Assignee: LIONROCKET INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 4:
5. A mouth shape synthesis device using artificial neural network, comprising:
an original video encoder that encodes original video data which is a target of a mouth shape synthesis as a video including a face of a synthesis target to output an original video embedding vector;
an audio encoder that encodes audio data that is a basis for the mouth shape synthesis and outputs an audio embedding vector; and
a synthesized video decoder that uses the original video embedding vector and the audio embedding vector as input data, and outputs synthesized video data in which a mouth shape corresponding to the audio data is synthesized on the synthesis target face,
wherein the audio encoder is configured to output an emotion vector for distinguishing a plurality of emotion state classes together with the audio embedding vector as output data,
wherein in a learning session of the original video encoder, the audio encoder, and the synthesized video decoder, the learning session is configured to be updated in a direction in which a sum of a reconstruction loss configured of a difference between the synthesized video data and the original video data, the synthesized video loss, the sync loss, and an emotion loss configured of a difference between the emotion vector and the labeled emotion state class is reduced,
wherein the synthesized video loss is a loss output from a synthesized video discriminator, which is an artificial neural network module, using the synthesized video data as input data and using a synthesized video discrimination vector, which distinguishes whether the synthesized video data is generated from the synthesized video decoder, as output data, and means a loss configured based on the synthesized video discrimination vector,
wherein the sync loss is a loss output from a sync discriminator including an audio embedding module which is a pre-learned artificial neural network module that receives the audio data and outputs an audio sync vector; and a synthesized video embedding module which is a pre-learned artificial neural network module that receives the synthesized video data and outputs a synthesized video sync vector, and means a loss configured of a difference between the audio sync vector and the synthesized video sync vector, and
wherein the audio embedding module and the synthesized video embedding module of the sync discriminator are pre-learned such that the higher a sync rate between the audio data and mouth shapes of the synthesized video data, the lower a sync loss is output.