Patent ID: 11880935
Assignee: SHANGHAITECH UNIVERSITY
Field: Audio-visual technology (Electrical engineering)
Classification: CPC G  H | IPC G  H

Claim 0:
1. An image-based method of modeling and rendering a three-dimensional model of an object, the method comprising:
obtaining a three-dimensional point cloud at each frame of a synchronized, multi-view video of an object, wherein the video comprises a plurality of frames;
extracting, using a feature extracting (FE) network, a feature descriptor for each point in the point cloud for the plurality of frames;
producing a two-dimensional feature map for a target camera based on the feature descriptors and parameters of the target camera, wherein the producing comprises:
rasterizing points in the point clouds into pixel squares covering a subset of pixels in the two-dimensional feature map based on the feature descriptors, and
assigning a default feature vector to each of the rest of the pixels in the two-dimensional feature map;

generating a depth map corresponding to the two-dimensional feature map; and
inputting the two-dimensional feature map and the depth map into a rendering convolutional neural (RE) network to decode the feature map into an image and a foreground mask,
the method further comprises:
training the FE network and the RE network based on training data captured by a multi-camera dome system, wherein the training comprises a shared training followed by an individual training, wherein:
the shared training trains on a plurality of objects to obtain a shared RE network and different FE networks for the plurality of objects, and
the individual training trains on a specific object for fine tuning the corresponding FE network.