Patent ID: 11972516
Assignee: DEEPBRAIN AI INC.
Field: Audio-visual technology (Electrical engineering)
Classification: CPC G  H | IPC G

Claim 9:
10. A method for generating a speech video, the method executed by a computing device having one or more processors and a memory storing one or more programs executable by the one or more processors, the method comprising:
receiving, by a video part generator of the computing device, a person background image of a person;
generating, by the video part generator, a video part of a speech video of the person; and
receiving, by an audio part generator of the computing device, text;
generating, by the audio part generator, an audio part of the speech video of the person; and
providing, by the audio part generator, speech-related information occurring during the generation of the audio part to the video part generator,
wherein the person background image is an image in which a portion of related to speech of the person is covered with a mask,
wherein the generating method, by the video part generator, comprising:
receiving, by a first encoder, the person background image;
extracting, by the first encoder, an image feature vector from the person background image;
generating, by a combiner, a combined vector by combining the image feature vector output from the first encoder and the speech-related information input from the audio part generator;
reconstructing, by a first decoder, the video part of the speech video using the combined vector as an input;
receiving, by an attention unit, the video part output from the first decoder;
generating, by the attention unit, an attention map by determining an attention weight for each pixel of the video part;
receiving, by a speech-related portion extractor, the video part output from the first decoder;
outputting, by the speech-related portion extractor, a speech-related image by extracting a speech-related portion from the video part;
receiving, by a reconstruction outputter, the background image input to the first encoder, the attention map, and the speech-related image; and
outputting, by the reconstruction outputter, a final video part of the person.