Patent ID: 11900518
Assignee: VIRTTARI LIMITED
Field: Computer technology (Electrical engineering)
Classification: CPC G  H | IPC G

Claim 0:
1. A method of producing an avatar video, the method comprising the steps of:
providing a reference image of a person's face;
using the reference image to provide a plurality of characteristic features representative of an initial facial model X0 of the person's face, wherein the plurality of characteristic features of the initial facial model XO comprise at least one set of landmarks and at least one latent descriptor representing an abstract appearance feature, the plurality of characteristic features defining facial position and facial pose dependent on a person speaking;
providing a target phrase to be rendered over a predetermined time period during the avatar video and providing a plurality of time intervals t within the predetermined time period;
generating, for each of said plurality of time intervals t, speech features from the target phrase, to provide a sequence of speech features, the speech features representing abstract quantifiers of audio and linguistic information; and
using a recursive model comprising a sequence-to-sequence encoder decoder method to generate, from the initial facial model X0 and the sequence of speech features, a sequence of expected facial models for each of said plurality of time intervals t,
wherein physical spatio-temporal dynamics of a facial model at each of said plurality of time intervals t are generated by solving a system of ordinary differential equations, ODEs, an expected facial position being derived from a recursive transformation of the speech features and the expected facial position of the facial model at a current time interval of said plurality of time intervals, which is sampled and being combined with the plurality of characteristic features of a facial model Xt at the current time interval to obtain the plurality of characteristic features of a next facial model Xt+1 in the sequence of expected facial models; and
combining and decoding a sequence of facial models Xt with the initial facial model X0 to generate a sequence of face images to produce the avatar video.