Patent ID: 11908067
Assignee: TSINGHUA UNIVERSITY
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A method for gigapixel-level light field intelligent reconstruction of a large-scale scene, comprising:
obtaining a coarse three-dimensional geometric model based on a multi-view three-dimensional reconstruction system;
constructing an implicit representation of a meta-deformed manifold on the coarse three-dimensional geometric model, wherein the meta-deformed manifold is at least one of a manifold that is deformed through a meta deformation network or a manifold that is output from the meta deformation network; and
optimizing the implicit representation of the meta-deformed manifold to obtain a light field reconstruction in the form of a free viewpoint rendering of the large-scale scene;
wherein constructing the implicit representation of the meta-deformed manifold on the coarse three-dimensional geometric model comprises:
normalizing a two-dimensional parameter grid of local patches into a rectangle, inputting into the meta deformation network a feature vector obtained by interpolating a rectangular sampling point on the two-dimensional parameter grid, and outputting a deformation feature of the rectangular sampling point in a high-dimensional space;
inputting a high-dimensional feature vector obtained based on the deformation feature into a feature mapping network, and outputting a normal vector, a diffuse reflection color value, a non-Lambert material property and a geometric feature vector of the rectangular sampling point;
inputting a view direction vector of the rectangular sampling point, the normal vector of the rectangular sampling point, the non-Lambert material property and the geometric feature vector into a neural renderer, outputting a specular color value in a current viewing direction, and obtaining a color prediction result of the rectangular sampling point according to the specular color and the diffuse color;
wherein optimizing the implicit representation of the meta-deformed manifold to obtain the light field reconstruction in the form of free viewpoint rendering of the large-scale scene comprises:
constructing a multi-layer perceptron network model;
inputting coordinates of the rectangular sampling point, the feature vector obtained by interpolating and the view direction vector into the multi-layer perceptron network model, and outputting a color red green blue (RGB) value of the rectangular sampling point;
wherein the method further comprises:
acquiring sparse multi-view images of the large-scale scene;
randomly selecting a plurality of pixels on the sparse multi-view images to perform back projection, each pixel emitting a ray, calculating points on a coarse three-dimensional surface through which the ray of each pixel passes, and calculating a color value rendered at each point.