Patent ID: 11928767
Assignee: NANJING SILICON INTELLIGENCE TECHNOLOGY CO., LTD.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 7:
8. A method for audio-driven character lip sync, applicable to a model for audio-driven character lip sync that is obtained by a training method for a model for audio-driven character lip sync, and comprising:
acquiring target speech and a target image for driving a target lip sync action, wherein the target speech indicates speech for a target dynamic image generated for a target character, the target image indicates a lip sync image for the target dynamic image generated for the target character, and the target dynamic image indicates a video image when the target image performs lip sync actions corresponding to the target speech;
extracting an audio feature of the target speech and an image feature of the target image;
encoding the audio feature and the image feature to obtain audio encoded data and image encoded data, respectively;
stitching the audio encoded data and the image encoded data to obtain image-audio data;
performing synthesis processing on the image-audio data to obtain a dynamic image encoding result; and
decoding the dynamic image encoding result to obtain the target dynamic image,
wherein the training method for a model for audio-driven character lip sync comprises:
acquiring sample data comprising a sample image and auxiliary data, wherein the sample image indicates a lip sync image for a target character, the auxiliary data is obtained by pre-processing an auxiliary video, and the auxiliary video comprises a non-target lip sync action generated through speaking by a non-target character and non-target speech corresponding to the non-target lip sync action;
generating a silent video with a preset duration based on the sample image, and processing the silent video as sample image data, wherein audio in the silent video is blank audio;
inputting the sample image data and the auxiliary data in the sample data into a to-be-trained model according to a preset ratio, to obtain an image encoding result, and training the to-be-trained model based on the image encoding result and a label of the sample data, to obtain a training result, wherein the to-be-trained model is a model on cloud for audio-driven character lip sync, and the label of the sample data comprises the blank audio corresponding to the sample image data in the silent video, and the non-target speech corresponding to the non-target lip sync action in the auxiliary video; and
obtaining a trained model when the training result meets a preset result, wherein the preset result indicates that a loss function of the to-be-trained model is trained to converge,
wherein after the input of the sample data into the model on cloud for audio-driven character lip sync,
the sample data is preprocessed to obtain the sample image data and the auxiliary data;
an audio feature of target speech and an image feature of a target image are extracted from the sample data, wherein the target speech indicates speech for a target dynamic image generated for a target character, the target image indicates a lip sync image for the target dynamic image generated for the target character, and the target dynamic image indicates a video image when the target image performs lip sync actions corresponding to the target speech;
the audio feature and the image feature are encoded to obtain audio encoded data and image encoded data, respectively;
the audio encoded data and the image encoded data are stitched to obtain image-audio data;
synthesis processing is performed on the image-audio data to obtain a dynamic image encoding result;
the dynamic image encoding result is decoded to obtain the target dynamic image.