Patent ID: 11948320
Assignee: MAGIC LEAP, INC.
Field: Audio-visual technology (Electrical engineering)
Classification: CPC G  H | IPC G  H

Claim 8:
9. A cross reality system, comprising:
a head-mounted display device having a display system;
a computing system in operable communication with the head-mounted display;
a plurality of camera sensors in operable communication with the computing system;
wherein the computing system is configured to estimate depths of features in a scene from a plurality of multi-view images captured by the camera sensors by a process comprising:
obtaining a multi-view images, including an anchor image of the scene and a set of reference images of a scene within a field of view of the camera sensors from the camera sensors;
passing the anchor image and reference images through a shared RGB encoder and descriptor decoder which (1) outputs a respective descriptor field of descriptors for the anchor image and each reference image, (ii) detects sparse interest points in the anchor image in conjunction with relative poses to determine a search space in the reference images from alternate view-points, and (iii) outputs intermediate feature maps, wherein in the descriptor decoder utilizes a learning architecture configured to train descriptors;
sampling the respective descriptors in the search space of each reference image to determine descriptors in the search space and matching the identified descriptors with descriptors for the sparse interest points in the anchor image, such matched descriptors referred to as matched keypoints;
triangulating the matched keypoints using singular value decomposition (SVD) to output 3D points;
passing the 3D points through a sparse depth encoder to create a sparse depth image from the 3D points and output feature maps; and
a depth decoder generating a dense depth image based on the output feature maps for the sparse depth encoder and the intermediate feature maps from the RGB encoder.