Patent ID: 11922723
Assignee: LIONROCKET INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A mouth shape synthesis device using artificial neural network, comprising:
an original video encoder that encodes original video data which is a target of a mouth shape synthesis as a video including a face of a synthesis target to output an original video embedding vector;
an audio encoder that encodes audio data that is a basis for the mouth shape synthesis and outputs an audio embedding vector;
a synthesized video decoder that uses the original video embedding vector and the audio embedding vector as input data, and outputs synthesized video data in which a mouth shape corresponding to the audio data is synthesized on the synthesis target face;
a synthesized video discriminator that is an artificial neural network module which uses the synthesized video data as input data and uses a synthesized video discrimination vector, which distinguishes whether the synthesized video data is generated from the synthesized video decoder as output data, and outputs a synthesized video loss configured based on the synthesized video discrimination vector;
an audio embedding module that is a pre-learned artificial neural network module which receives the audio data and outputs an audio sync vector;
a synthesized video embedding module that is a pre-learned artificial neural network module that which receives the synthesized video data and outputs a synthesized video sync vector; and
a sync discriminator that outputs a sync loss configured of a difference between the audio sync vector and the synthesized video sync vector,
wherein the audio embedding module and the synthesized video embedding module of the sync discriminator are pre-learned such that the higher a sync rate between the audio data and mouth shapes of the synthesized video data, the lower a sync loss is output,
wherein the audio encoder is configured to output an emotion vector for distinguishing a plurality of emotion state classes together with the audio embedding vector as output data,
wherein the synthesized video discriminator is configured to output the emotion vector together with the synthesized video discrimination vector as output data, and
wherein in a learning session of the original video encoder, the audio encoder, and the synthesized video decoder, the learning session is configured to be updated in a direction in which a sum of a reconstruction loss configured of a difference between the synthesized video data and the original video data, the synthesized video loss, the sync loss, and an emotion loss configured of a difference between the emotion vector and the labeled emotion state class is reduced.