Patent ID: 11967092
Assignee: SONY GROUP CORPORATION
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 20:
21. A non-transitory computer-readable medium having stored thereon computer-readable instructions, which when executed by a system, cause the system to execute operations, the operations comprising:
receiving an input human-dynamics sequence comprising geometry information of a human object and an RGB video of the human object;
inputting the RGB video to a neural network;
estimating a pose of the human object in each frame of the RGB video based on an output of the neural network for the input, wherein the estimated pose comprises features associated with human joints and face landmarks of the human object;
selecting, from the input human-dynamics sequence, a key-frame for which the estimated pose is closest to a reference human pose;
generating a tracking sequence for a 3D human mesh, from the selected key-frame and up to a number of frames of the input human-dynamics sequence, wherein the generated tracking sequence comprises final values of parameters of articulate motion and non-rigid motion of a set of 3D points on the 3D human mesh; and
generating a free-viewpoint video (FVV) based on the generated tracking sequence for the 3D human mesh.