Patent ID: 11972516
Assignee: DEEPBRAIN AI INC.
Field: Audio-visual technology (Electrical engineering)
Classification: CPC G  H | IPC G

Claim 0:
1. A device for generating a speech video, the device having one or more processor and a memory storing one or more programs executable by the one or more processors, the device comprising:
a video part generator configured to receive a person background image of a person and generate a video part of a speech video of the person; and
an audio part generator configured to receive text, generate an audio part of the speech video of the person, and provide speech-related information occurring during the generation of the audio part to the video part generator,
wherein the person background image is an image in which a portion of related to speech of the person is covered with a mask,
wherein the video part generator comprises:
a first encoder configured to receive the person background image and extract an image feature vector from the person background image;
a combiner configured to generate a combined vector by combining the image feature vector output from the first encoder and the speech-related information input from the audio part generator;
a first decoder configured to reconstruct the video part of the speech video using the combined vector as an input;
an attention unit configured to receive the video part output from the first decoder and generate an attention map by determining an attention weight for each pixel of the video part;
a speech-related portion extractor configured to receive the video part output from the first decoder and output a speech-related image by extracting a speech-related portion from the video part; and
a reconstruction outputter configured to receive the background image input to the first encoder, the attention map, and the speech-related image and output a final video part of the person.