Patent ID: 11900518
Assignee: VIRTTARI LIMITED
Field: Computer technology (Electrical engineering)
Classification: CPC G  H | IPC G

Claim 16:
17. A system for producing an avatar video, a method comprising the steps of:
an image processing model for receiving a reference image of a person's face and for extracting a plurality of characteristic features representative of an initial facial model XO of the person's face, wherein the plurality of characteristic features of the initial facial model XO comprise at least one landmark and at least one latent descriptor representing an abstract appearance feature, the plurality of characteristic features defining facial position and facial pose dependent on the person speaking;
a speech processing module for extracting a target phrase to be rendered over a predetermined time period during the avatar video and for providing a plurality of time intervals t within the predetermined time period;
the speech processing module configured to generate, for each of said plurality of time intervals t, speech features from the target phrase, to provide a sequence of speech features, the speech features representing abstract quantifiers of audio and linguisic information; and
an avatar rendering module configured to use a recursive model comprising a sequence-to-sequence encoder decoder method to generate, from the initial facial model XO and the sequence of speech features, a sequence of expected facial models for each of said plurality of time intervals t, wherein physical spatio-temporal dynamics of a facial model at each of said plurality of time intervals t are generated by solving a system of ordinary differential equations, ODEs, an expected facial position being derived from a recursive transformation of the speech features and the facial position of the facial model at a current time interval of said plurality of time intervals, which is sampled and being combined with the plurality of characteristic features of the facial model Xt at the current time interval to obtain the plurality of characteristic features of a next facial model Xt+1 in the sequence of expected facial models;
wherein the avatar rendering module comprises a frame generator configured to combine and decode a sequence of facial models Xt with the initial facial model X0 to generate a sequence of face images to produce the avatar video.