Patent ID: 11887248
Assignee: MASSACHUSETTS INSTITUTE OF TECHNOLOGY
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 12:
13. A method of reconstructing a scene in three dimensions from a two-dimensional image, the method comprising:
processing an image using a detection transformer to detect an object in the scene and to generate a first latent vector for the object, a Normalized Object Coordinate Space (NOCS) map of the object, and a depth map for a background portion of the scene;
processing the first latent vector using one or more multilayer perceptrons (MLPs) to produce a second latent vector for the object that represents the object in a differentiable database of object priors, wherein the differentiable database of object priors encodes geometry of the object priors using signed distance fields (SDFs) and appearance of the object priors using luminance fields (LFs);
recovering, from the NOCS map of the object, a partial three-dimensional (3D) shape of the object;
estimating an initial pose of the object;
fitting an object prior in the differentiable database of object priors to align in geometry and appearance with the partial 3D shape of the object to produce a complete shape of the object and refining the initial pose of the object using a surfel-based differentiable renderer to produce a refined estimated pose of the object;
generating an editable and re-renderable 3D reconstruction of the scene based, at least in part, on the complete shape of the object, the refined estimated pose of the object, and the depth map for the background portion of the scene; and
controlling operation of a robot based, at least in part, on the editable and re-renderable 3D reconstruction of the scene.