Patent ID: 11967092
Assignee: SONY GROUP CORPORATION
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A system, comprising:
circuitry configured to:
receive an input human-dynamics sequence comprising geometry information of a human object and an RGB video of the human object;
input the RGB video to a neural network;
estimate a pose of the human object in each frame of the RGB video based on an output of the neural network for the input, wherein the estimated pose comprises features associated with human joints and face landmarks of the human object;
select, from the input human-dynamics sequence, a key-frame for which the estimated pose is closest to a reference human pose;
generate a tracking sequence for a 3D human mesh, from the selected key-frame and up to a number of frames of the input human-dynamics sequence, wherein the generated tracking sequence comprises final values of parameters of articulate motion and non-rigid motion of a set of 3D points on the 3D human mesh; and
generate a free-viewpoint video (FVV) based on the generated tracking sequence for the 3D human mesh.