System and method for three-dimensional scene reconstruction and understanding in extended reality (XR) applications

A method includes receiving depth data of a real-world scene from a depth sensor, receiving image data of the scene from an image sensor, receiving movement data of the depth and image sensors from an IMU, and determining an initial 6DOF pose of an apparatus based on the depth data, image data, and/or movement data. The method also includes passing the 6DOF pose to a back end to obtain an optimized pose and generating, based on the optimized pose, image data, and depth data, a three-dimensional reconstruction of the scene. The reconstruction includes a dense depth map, a dense surface mesh, and/or one or more semantically segmented objects. The method further includes passing the reconstruction to a front end and rendering, at the front end, an XR frame. The XR frame includes a three-dimensional XR object projected on one or more surfaces of the scene.

TECHNICAL FIELD

This disclosure relates to systems and methods for generating extended reality (XR) displays that combine image data of real-world objects of a user's current operating environment (such as walls, floors, or furniture) with virtual objects presented to appear as elements of the real-world operating environment. More specifically, this disclosure relates to a system and method for performing three-dimensional scene reconstruction and understanding for XR applications.

BACKGROUND

Smartphones, tablets, and other readily portable, battery-powered devices that combine sensors for tracking one or more of the device's motion or a user's position relative to the device have become the dominant computing platform for many users. The integration of processing power, motion, and visual sensors in a compact, battery-powered apparatus presents new and exciting opportunities for extending the functionality of smartphones and tablets—including, without limitation, motion or viewpoint-adjusted projection of extended reality (XR) displays that provide natural-looking projections of virtual objects on image data of real-world objects in a user's operating environment.

Unfortunately, extending the functionality of (typically) battery-powered devices to support providing XR displays present new, as-yet unsolved technical challenges. Specifically, developing a spatial and semantic understanding of a real-world operating environment of an XR platform can involve computationally-intensive operations using machine learning models for object recognition, as well as computationally intensive operations to generate detailed depth maps from sparse data or by generating disparity maps. Additionally, generating detailed depth maps and accurate, machine-level understandings of a real-world environment facilitates the incorporation of a wide range of reprojection and cloning effects within an XR display. Examples of such effects include, without limitation, “freeze motion” effects and creating additional, virtual instances of real-world objects. While portable, battery-powered, front-end devices may be able to perform such computationally intensive operations in short bursts or limited quantities, doing so stresses battery and processor resources, exhausting batteries and generating heat. From an overall system perspective, this can be undesirable.

SUMMARY

This disclosure provides a system and method for three-dimensional scene reconstruction and understanding in extended reality (XR) applications.

In a first embodiment, an apparatus includes a depth sensor, an image sensor, an inertial measurement unit (IMU), and a controller. The controller is configured to receive depth data of a real-world scene from the depth sensor, receive image data of the real-world scene from the image sensor, receive movement data of the depth sensor and the image sensor from the IMU, and determine an initial six-degree-of-freedom (6DOF) pose of the apparatus based on at least one of the depth data, the image data, and the movement data. The controller is also configured to pass the initial 6DOF pose of the apparatus to a back end to obtain an optimized pose and generate, based on the optimized pose, the image data, and the depth data, a three-dimensional reconstruction of the real-world scene, where the three-dimensional reconstruction includes at least one of a dense depth map of the real-world scene, a dense surface mesh of the real-world scene, and one or more semantically segmented objects. The controller is further configured to pass the three-dimensional reconstruction of the real-world scene to a front end and render, at the front end, an XR frame, where the XR frame includes a three-dimensional XR object projected on one or more surfaces of the real-world scene.

In a second embodiment, a method for performing three-dimensional scene reconstruction and understanding for XR applications includes receiving depth data of a real-world scene from a depth sensor of an apparatus, receiving image data of the real-world scene from an image sensor of the apparatus, receiving movement data of the depth sensor and the image sensor from an IMU of the apparatus, and determining an initial 6DOF pose of the apparatus based on at least one of the depth data, the image data, and the movement data. The method also includes passing the initial 6DOF pose of the apparatus to a back end to obtain an optimized pose and generating, based on the optimized pose, the image data, and the depth data, a three-dimensional reconstruction of the real-world scene, where the three-dimensional reconstruction includes at least one of a dense depth map of the real-world scene, a dense surface mesh of the real-world scene, and one or more semantically segmented objects. The method further includes passing the three-dimensional reconstruction of the real-world scene to a front end and rendering, at the front end, an XR frame, where the XR frame includes a three-dimensional XR object projected on one or more surfaces of the real-world scene.

In a third embodiment, a non-transitory computer-readable medium contains instructions that, when executed by at least one processor of an apparatus including a depth sensor, an image sensor, and an IMU, causes the apparatus to receive depth data of a real-world scene from the depth sensor, receive image data of the real-world scene from the image sensor, receive movement data of the depth sensor and the image sensor from the IMU, and determine an initial 6DOF pose of the apparatus based on at least one of the depth data, the image data, and the movement data. The medium also contains instructions that, when executed by the at least one processor, causes the apparatus to pass the initial 6DOF pose of the apparatus to a back end to obtain an optimized pose and generate, based on the optimized pose, the image data, and the depth data, a three-dimensional reconstruction of the real-world scene, where the three-dimensional reconstruction includes at least one of a dense depth map of the real-world scene, a dense surface mesh of the real-world scene, and one or more semantically segmented objects. The medium further contains instructions that, when executed by the at least one processor, causes the apparatus to pass the three-dimensional reconstruction of the real-world scene to a front end and render, at the front end, an XR frame, where the XR frame includes a three-dimensional XR object projected on one or more surfaces of the real-world scene.

DETAILED DESCRIPTION

FIG.1illustrates a non-limiting example of a device100for performing three-dimensional scene reconstruction and understanding in extended reality (XR) applications according to some embodiments of this disclosure. The embodiment of the device100shown inFIG.1is for illustration only, and other configurations are possible. Suitable devices come in a wide variety of configurations, andFIG.1does not limit the scope of this disclosure to any particular implementation of a device. For example, the device100may be implemented as a head mounted display (HMD) or as a separate device (such as a smartphone) controlling an augmented reality (AR) display presented at a connected HMD (such as through a BLUETOOTH or ZIGBEE connection).

As shown in the non-limiting example ofFIG.1, the device100includes a communication unit110that may include, for example, a radio frequency (RF) transceiver, a BLUETOOTH transceiver, or a WI-FI transceiver, etc. The device100also includes transmit (TX) processing circuitry115, a microphone120, and receive (RX) processing circuitry125. The device100further includes a speaker130, a main processor140, an input/output (I/O) interface (IF)145, I/O device(s)150, and a memory160. The memory160includes an operating system (OS) program161and one or more applications162.

Applications162can include games, social media applications, applications for geotagging photographs and other items of digital content, virtual reality (VR) applications, augmented reality (AR) applications, extended reality (XR) applications, operating systems, device security (such as anti-theft and device tracking) applications, or any other applications that access resources of the device100. The resources of the device100may include, without limitation, the speaker130, microphone120, I/O devices150, and additional resources180. According to some embodiments, applications162include XR applications that can project, on a display device, an XR display that combines elements of a view of a real-world operating environment of the device100in combination with one or more virtual objects, where each virtual object's position or dynamics embody a physical interaction (such as appearing to sit on a real-world table or bouncing off of a wall of a room) with a physical object of the real-world operating environment.

The communication unit110may receive an incoming RF signal, such as a near field communication signal like a BLUETOOTH or WI-FI signal. The communication unit110can down-convert the incoming RF signal to generate an intermediate frequency (IF) or baseband signal. The IF or baseband signal is sent to the RX processing circuitry125, which generates a processed baseband signal by filtering, decoding, or digitizing the baseband or IF signal. The RX processing circuitry125transmits the processed baseband signal to the speaker130(such as for voice data) or to the main processor140for further processing (such as for web browsing data, online gameplay data, notification data, or other message data). Additionally, the communication unit110may contain a network interface, such as a network card, or a network interface implemented through software.

The TX processing circuitry115receives analog or digital voice data from the microphone120or other outgoing baseband data (such as web data, e-mail, or interactive video game data) from the main processor140. The TX processing circuitry115encodes, multiplexes, or digitizes the outgoing baseband data to generate a processed baseband or IF signal. The communication unit110receives the outgoing processed baseband or IF signal from the TX processing circuitry115and up-converts the baseband or IF signal to an RF signal for transmission.

The main processor140can include one or more processors or other processing devices and execute the OS program161stored in the memory160in order to control the overall operation of the device100. For example, the main processor140could control the reception of forward channel signals and the transmission of reverse channel signals by the communication unit110, the RX processing circuitry125, and the TX processing circuitry115in accordance with well-known principles. In some embodiments, the main processor140includes at least one microprocessor or microcontroller.

The main processor140is also capable of executing other processes and programs resident in the memory160. The main processor140can move data into or out of the memory160as required by an executing process. In some embodiments, the main processor140is configured to execute the applications162based on the OS program161or in response to inputs from a user or applications162. Applications162can include applications specifically developed for the platform of device100or legacy applications developed for earlier platforms. Additionally, the main processor140can be manufactured to include program logic for implementing techniques for monitoring suspicious application access according to some embodiments of this disclosure. The main processor140is also coupled to the I/O interface145, which provides the device100with the ability to connect to other devices such as laptop computers and handheld computers. The I/O interface145is the communication path between these accessories and the main processor140.

The main processor140is also coupled to the I/O device(s)150. The operator of the device100can use the I/O device(s)150to enter data into the device100. The I/O device(s)150can include a keyboard, HMD, touchscreen, mouse, track ball, or other device(s) capable of acting as a user interface to allow a user to interact with the device100. In some embodiments, the I/O device(s)150can include a touch panel, a (digital) pen sensor, a key, or an ultrasonic input device. The I/O device(s)150can include one or more screens, which can be a liquid crystal display, a light-emitting diode (LED) display, an optical LED (OLED), an active-matrix OLED (AMOLED), or other screen(s) capable of rendering graphics.

The memory160is coupled to the main processor140. According to some embodiments, part of the memory160includes a random-access memory (RAM), and another part of the memory160includes a Flash memory or other read-only memory (ROM).

According to some embodiments, the device100can further include a separate graphics processing unit (GPU)170. Also, according to some embodiments, the device100may further include a variety of additional resources180that can, if permitted, be accessed by the applications162. According to particular embodiments, the additional resources180may include an accelerometer or inertial motion unit182, which can detect movements of the device100along one or more degrees of freedom. As another example, according to particular embodiments, the additional resources180may include a dynamic vision sensor (DVS)184or one or more cameras186.

AlthoughFIG.1illustrates one example of a device100for generating a three-dimensional reconstruction of a real-world scene and developing a scene understanding for an XR application, various changes may be made toFIG.1. For example, the device100could include any number of components in any suitable arrangement. In general, devices including computing and communication systems come in a wide variety of configurations, andFIG.1does not limit the scope of this disclosure to any particular configuration. WhileFIG.1illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

FIGS.2A-2Fillustrate example aspects of scene reconstruction and reprojection in XR applications according to some embodiments of this disclosure. For convenience, elements common to more than one ofFIGS.2A-2Fare numbered similarly.

Referring to the illustrative example ofFIG.2A, a frame200of image data of a real-world operating environment of an apparatus (such as an HMD embodying the componentry and architecture of the device100inFIG.1) is shown. In this example, the frame200includes a raster of pixels representing color and luminance values of points within the operating environment as observed at one or more cameras having a specific pose. As used in this description, the expressions “pose” and “camera pose” encompass an expression of the location and viewing angle of a vantage point from which a frame of image data (such as the frame200) is collected. The frame200includes objects that human viewers innately recognize as instances of known objects having known structural properties (such as objects that can support other objects or need to stand on three or more legs) and innately-identifiable spatial relationships to each other. For example, a first object201includes a collection of rigid planar sections of material that human viewers recognize as a coffee table. Upon the coffee table is positioned a second object203, which human viewers recognize as a plush bunny. The second object203sits in front of a third object205, which human viewers recognize as a sectional sofa. Additionally, the operating environment is bounded on two sides by first and second planes207A and207B, which human viewers recognize as first and second walls. Further, the real-world operating environment shown in the frame200is bounded by a third plane207C, which human viewers understand to be the floor.

In the illustrative example described with reference toFIGS.2A-2F, an XR display extending the real-world scene represented by the frame200is generated. Specifically, in this example, the second object203(the plush bunny sitting on the coffee table) is duplicated, and virtual bunnies appearing to interact with the physical objects of the real-world operating environment are shown in the frame200. Additionally, elements of the scene shown in the frame200are reprojected onto a different set of real-world objects and presented from a different camera angle. As the expression “extended reality” suggests, in the explanatory example ofFIGS.2A-2F, image data of the constituent components of the real-world operating environment, such as the furniture, planes of the walls and floors, and recognizable objects, is used to create an image that includes an extension of the objects present in the original image data.

Humans can typically visualize multiple aspects of what the room shown in the frame200might look like from a different vantage point or aspects of how additional bunnies might appear in the scene (such as the bunnies' faces and regions in the operating environment in which they are likely or not likely to be found). For example, as non-flying mammals, additional bunnies would be found on surfaces of the environment rather than levitating in the air. At least initially, a computing platform generating an XR display may have no such innate understanding of how a virtual object could plausibly interact with the objects shown in the frame200. Rather, the computing platform may initially understand the frame200as including a plurality of regions of pixels with similar color and/or luminance values. In order to generate an effective XR display, the computing platform can obtain both a spatial understanding of the scene represented in the frame200(such as an awareness of sizes, shapes, and relative depths), as well as a semantic understanding of the scene (such as an understanding that the third plane207C is a floor, which is a plane that cannot be broken through and towards which objects in the frame200are drawn).

Referring to the illustrative example ofFIG.2B, in order to provide an XR display in which virtual objects interact with and adhere (at least in part) to the spatial or physical constraints of object in a real-world, some embodiments need to generate a three-dimensional reconstruction220of the scene shown in the frame200. As shown inFIG.2B, in some embodiments, the three-dimensional reconstruction220includes a surface mesh, such as an incremental or dense surface mesh. As used in this disclosure, the expression “mesh” encompasses a map of depth point values in which adjacent depth points may be connected by edges to define planar regions. From the three-dimensional reconstruction, the location of one or more fundamental planes (such as the ground plane and walls) and major objects (such as the couch and coffee table) can be recognized. Additionally, dense depth maps of objects that can be reprojected are obtained.

Referring to the explanatory example ofFIG.2C, having decomposed the scene based on the three-dimensional reconstruction220, and (in some embodiments) having performed object recognition on the image data in the frame200to obtain classifications of instances of recognized objects in the scene, the computing platform supporting one or more XR applications begins utilizing the understanding of the real-world operating environment encoded in the object classifications and three-dimensional reconstruction to build an XR display. The XR display can extend the objects and underlying structure of the real-world environment in some manner. In the illustrative example ofFIG.2C, first and second additional instances225A and225B of the second object203are created. As shown inFIG.2C, the first and second additional instances225A and225B are not simply copied and pasted instances of the second object203but rather present different poses. To accomplish this, some embodiments of this disclosure recognize that the second object203is a bunny and draw upon stored knowledge of the general shape of a bunny to create the first and second additional instances225A and225B.

Referring to the explanatory example ofFIG.2D, an example of an XR display299is shown in the figure. As shown inFIG.2D, elements of the scene shown inFIG.2Ahave been reprojected onto different real-world objects. Also, as discussed with reference toFIG.2C, the XR display299also includes the first and second additional instances225A and225B of the second object203. In this way, the image data of the frame200has been “extended” to create the new, photorealistic scene shown in the XR display299.

Significant factors affecting the performance with which a computing platform can generate XR displays, such as the XR display299, include without limitation, the accuracy and efficiency with which the processing platform can perform scene reconstruction and scene comprehension. Without these, reprojection and creating additional, re-posed instances of scene objects can be difficult if not impossible.FIGS.2E and2Fillustrate aspects of the effects of inaccurate scene reconstruction on generating an XR display in some embodiments As used in this disclosure, the expression “scene reconstruction” encompasses developing a dense depth map and representation of a plane structure of an operating environment from source data received at a computing platform, where the source data includes image data of the scene.

Referring to the illustrative example ofFIG.2E, an example of a first reprojection291of an object in a scene is shown in the figure. According to some embodiments, the first reprojection291is obtained by generating, from a dense depth map, a representation of the three-dimensional structure of an object in the scene (in this case, a rabbit). The representation of the three-dimensional structure may include a linked set of data points (such as an incremental mesh) corresponding to linked points on the surface of the object. From the representation of the three-dimensional structure, a view of the object from a different perspective (such as a reprojection) can be generated. However, the quality of the reprojection can be heavily dependent on the quality of the representation of the three-dimensional structure of the object, and errors in the depth map of the object or location of the boundaries of the object can be conspicuously propagated in the reprojection. As shown in the illustrative example ofFIG.2E, defects in the depth map or three-dimensional representation of a source object can produce artifacts (such as an artifact293) in a reprojection, creating an unnatural or ragged appearance of objects in an XR display. From a performance standpoint, such artifacts make virtual objects look unrealistic in an XR display, which is typically undesirable.

Referring to the illustrative example ofFIG.2F, an example of a second reprojection295of the object is provided. In this example, the three-dimensional representation of the source object is mapped accurately and with sufficient detail to produce a reprojection of the object without the unwanted artifacts shown inFIG.2E. As discussed in greater detail below, some embodiments of this disclosure provide, without limitation, the performance benefits of accurate, detailed scene reconstruction for XR displays in a way that is computationally efficient and that conserves processing and power resources at the processing platform. In embodiments in which the processing platform is battery-powered or worn on a user's head, these gains in computational efficiency can translate into highly beneficial gains in extended battery life and reduction of heat buildup from the processor(s) of the processing platform.

AlthoughFIGS.2A-2Fillustrate examples of aspects of scene reconstruction and reprojection in XR applications, various changes may be made toFIGS.2A-2F. For example, operational environments for devices can vary widely, and the contents of the images shown here are for illustration and explanation only.

FIG.3illustrates an example architecture300for performing three-dimensional scene reconstruction and scene comprehension for XR applications according to some embodiments of this disclosure. For ease of explanation, the architecture300is described as being used in the device100ofFIG.1. However, the architecture300may be implemented using any suitable device(s) and in any suitable system(s).

Certain existing approaches for tuning and improving the performance of scene reconstruction and scene comprehension for XR applications seek to locally optimize stages (such as data fusion depth reconstruction or volume based-reconstruction) of a processing pipeline. Some embodiments according to this disclosure adopt a holistic approach to achieving performance gains through the use of both refined processing stages and the overall architecture of the processing pipeline. Accordingly, the holistic approach embodied by some embodiments according to this disclosure can provide significant performance gains beyond those achievable with incremental or localized approaches.

Referring to the illustrative example ofFIG.3, the architecture300includes four main stages or modules, at least some of which include multiple sub-modules or processing stages. In this example, the four modules include a sensor module301, an XR display device309, a front end397, and a back end395. Depending on the embodiment, two or more of the sensor module301, XR display device309, front end397, and back end395may be implemented using separate processing platforms (such as an HMD and a smartphone). However, in many embodiments, the sensor module301and XR display device309may represent a common piece of hardware, given that XR displays are often reprojected in response to a user's pose and it is often useful for the image sensor to be trained on a field of view significantly overlapping with the field of view of the user's own eyes.

According to various embodiments, the sensor module301includes a suite of sensors disposed on an apparatus, such as near a user's head (like in an HMD), for collecting image and depth data of the user's real-world operating environment, as well as data indicating a pose associated with the image and depth data. Here, the pose may be expressed as a first set of values quantifying a sensor's location within an operating environment and a second set of values quantifying the direction in which a sensor is pointed. For example, a six-degree-of-freedom (“6DOF”) pose measurement may capture a sensor's physical location using three coordinates in a Cartesian coordinate system and express the direction in which the sensor is pointed using a three-dimensional vector.

In this example, the sensor module301includes a depth sensor303, which is configured to obtain measurements of the depths or distances between real world objects in a field of view and the depth sensor303. Examples of sensors that may be used as the depth sensor303include, without limitation, time-of-flight sensors, structured light sensors (which project a known pattern onto a field of view and determine depths based on the distortions of the pattern as projected), and visual sensors (which may include a stereoscopic pair of image sensors from which depth values may be obtained based on differences between the two sensors' images of the same scene). As shown inFIG.3, the sensor module301also includes at least one image sensor305, which can be configured to obtain image data of a real-world operating environment from which object recognition (and, where appropriate, scene reconstruction) can be performed. In some embodiments, the depth sensor303and the image sensor305can have overlapping fields of view so that image data can be used to assist in generating dense depth maps and depth data may be used to augment image data for object recognition. Examples of sensors that can be used as the image sensor305include, without limitation, digital cameras (such as cameras with CMOS sensors) or dynamic vision sensors (which, unlike CMOS sensors, do not output frames of image data across a full field of view but rather output a stream of events corresponding to changes in measured luminance at points on a raster grid). Depending on the application, dynamic vision sensors can be particularly advantageous in that they consume less energy than CMOS sensors and can record rapid motion with less blur. For other applications, such as where the real-world operating environment has low light or limited dynamism (meaning little change over time), CMOS sensors may be advantageous. According to some embodiments, the sensor module301further includes one or more inertial measurement units (IMU)307, which are configured to capture changes in a user's location and the directionality of the depth sensor303and the image sensor305. Examples of sensors suitable for use as the IMU307include, without limitation, accelerometers and digital gyroscopes.

Referring to the explanatory example ofFIG.3, the XR display device309includes a head-mounted display, smart glasses, or other apparatus having a screen through which an XR display (such as the XR display299inFIG.2D) is presented to a user.

According to various embodiments, the architecture300splits up the operations of generating an XR display such that the often computationally-expensive tasks associated with generating a scene reconstruction and scene comprehension are performed by the back end395and obtaining initial pose data and rendering virtual objects of a frame of an XR display are performed by the front end397. This bifurcation between the back end395and the front end397facilitates the reapportionment of the processing tasks associated with generating an XR display. As a result, the computationally-intensive tasks performed by the back end395may, in certain embodiments, be performed using multicore processor architectures (such as one where one set of processing cores is designed for energy efficiency and a second set of processing cores is designed for performance) or chips (such as neural processing units (NPUs) specifically intended for implementing neural networks and machine learning algorithms). Alternatively or additionally, in some embodiments, the back end395may be implemented at an accessory device that is less subject to the battery or processing power constraints commonly associated with HMDs or other user-worn or user-carried devices.

In the example architecture300, the front end397is generally responsible for performing an initial determination of the pose of the apparatus including the sensor module301and rendering an XR frame utilizing a representation of the three-dimensional structure of the real-world operating environment. In this illustrative example, the front end397includes a pose module311and a display module317. According to various embodiments, the pose module311receives, as inputs, image data from the image sensor(s)305and data from the IMU307and processes this data to obtain a 6DOF pose315associated with the sensor module301. Once the pose of the sensor module301is determined, one or more object poses for one or more virtual objects rendered to appear on one or more surfaces of the real-world operating environment can be determined. Accordingly, using the 6DOF pose315and the scene reconstruction data output by the back end395, the display module317can render an XR frame including a view of objects of a real-world operating environment and extended to include one or more virtual objects interacting with one or more of the detected surfaces or objects of the real-world operating environment.

Referring to the illustrative example ofFIG.3, the back end395may be implemented on the same processing platform at the front end397or the sensor module301. However, one technical benefit of the architecture300and the division of the macro components of a processing pipeline for performing scene reconstruction and scene understanding between the discrete modules shown inFIG.3is that, in some embodiments, the back end395(which includes more computationally-intensive phases of the pipeline) may be implemented on a separate more-powerful processing platform than the front end397or the sensor module301. In this example, the back end395includes a pose refining module319. According to some embodiments, while often necessary for computing a 6DOF pose315, the outputs of the sensors of the sensor module301can be noisy or contain artifacts that diminish the accuracy of the determined pose of the sensor module301. In some embodiments, further processing can refine the accuracy of a 6DOF determination.

As shown inFIG.3, the further processing operations undertaken by the pose refining module319may include an operation321, where a global pose is determined from registration of image data and bundle adjustment. A global pose can refer to the location of the sensor module301in a world coordinate system, meaning a coordinate system also used to express the locations of objects within a real-world operating environment. According to various embodiments, bundle adjustment includes refining an estimate of the global pose based on multiple frames of image data obtained at different poses. As rays of light (also known as bundles) radiate light at a camera along predictable linear paths, the value of the global pose of the sensor module301may be adjusted to provide the best fit for the image data including multiple instances of the same object(s).

Referring to the illustrative example ofFIG.3, an operation323is used to further refine the estimation of the 6DOF pose of the sensor module301based on sparse point cloud data (such as depth data directly from the depth sensor303) obtained across a plurality of different pose points. As with light rays from a particular object, the distance between the sensor module301may (excluding noise from the depth sensor303) vary predictably with pose. During the operation323, the initial 6DOF pose315may be further refined based on the sparse point cloud data, where the determined 6DOF pose value is adjusted to best fit the sparse cloud data obtained across multiple frames of point cloud data.

Recognizing that the source data from the sensor module301can be noisy and contain artifacts and that, from a pose estimation perspective, certain frames of sensor data may simply be better for pose estimation than others, an operation325can generate keyframes of image and depth sensor data and determine revised values of 6DOF pose values associated with the generated keyframes. A keyframe may represent a frame (such as a set of image and depth data) associated with a common capture time and pose. In some embodiments according to this disclosure, rather than tracking the changes in the pose of the sensor module301over time based on just a data feed from the sensor module301, a series of keyframes can be determined, and the changes in view and pose at any instant in time can be determined based on intermediate values between keyframes. In this way, any noisiness in the originally-obtained sensor data from the sensor module301may be smoothed out and an improved, less jumpy XR experience may be provided through the XR display device309. Additionally, depending on the embodiment, the determination of keyframes at the operation325may also conserve memory and storage resources at the processing platform implementing the back end395.

Referring to the illustrative example ofFIG.3, the back end395further includes a three-dimensional scene shape generating module327. According to various embodiments, the three-dimensional scene shape generating module327implements a processing pipeline that receives, as inputs, the refined pose determinations from the pose refining module319and depth sensor outputs from the sensor module301. The three-dimensional scene shape generating module327outputs a digital representation343of the three-dimensionality of portions of the real-world operating environment captured in the sensor data (such as a surface mesh) and performs anchor configuration and tracking341for the real-world operating environment represented by the sensor mesh. The anchor configuration and tracking may include determining a consistent reference point (also known as an “anchor”) for a coordinate system for defining the positions of real-world objects and positioning virtual objects.

Referring to the illustrative example ofFIG.3, during an operation329, the three-dimensional scene shape generating module327performs data fusion for sparse points from depth data obtained from the depth sensor303and depth values obtained from parallax analysis of data from the image sensor305. According to various embodiments, operation329includes integrating one or more point clouds of depth data of the real-world operating environment captured by the depth sensor(s)303with one or more point clouds of depth data of the operating environment determined from comparing parallax changes between two or more frames of image data obtained from the image sensor(s)305to create a single cloud of depth data points. Also, in some embodiments, one or more integration filters may be created and applied to remove noises in the source data and assign confidence weightings to the depth points of the integrated depth map.

As shown inFIG.3, at an operation331, a dense depth map is generated. The dense depth map covers, at a minimum, a portion of the real-world operating environment represented in the fused depth map generated by the operation229. A dense depth map can represent a depth map (such as a mapping of depth values relative to a measurement point to locations in a coordinate system) generated from a source depth map with a lower depth point resolution. For example, a dense depth map may be obtained through guided interpolation and propagating additional depth point values to a sparser depth map based on one or more of spatial, color, and pose information, from which planes and regions of equivalent or reliably predictable depth in the neighborhood of existing depth values can be identified.

According to various embodiments, at an operation335, the three-dimensional scene shape generating module327identifies shapes (such as primitives like cubes, pyramids, prisms, and combinations thereof) from the dense depth maps and, for each surface of each shape identified in the dense depth map, determines a vector defining the location of a surface plane and its normal. In some embodiments, the operation335may be performed using volumetric approaches, where the dense depth map is provided to a convolutional neural network trained to apply classifications (such as shapes) to point clouds of depth data. Also, in some embodiments, the operation335may be performed using a truncated signed distance field (TSDF), which utilizes multiple views of an object or operating environment and determines distances between each object and its distance to the nearest surface and can be less sensitive to artifacts or irregularities in a depth map.

As shown in the explanatory example ofFIG.3, at an operation333, the three-dimensional scene shape generating module327performs semantic segmentation, identifying objects and boundaries of object regions from the data obtained from the image sensor(s)305and the depth sensor(s)303. According to some embodiments, the operation333may be performed by applying one or more neural network-based object recognition algorithms executed by a processor. Alternatively or additionally, in some embodiments, the operation333may be performed by a purpose-specific neural processor trained to identify a predefined set of objects in image data.

Referring to the illustrative example ofFIG.3, at an operation337, the three-dimensional scene shape generating module327performs plane detection from the dense depth map generated by the operation331. According to various embodiments, the operation337includes identifying the foundational planes of the real-world operating environment. This typically includes identifying the floor plane and optionally one or more planes defining the walls and ceiling (if any) of the real-world operating environment. In some embodiments, the operation337may be performed using one or more of real-time consistent plane detection (RCPD) algorithms and/or random sample consensus (RANSAC) techniques.

As shown inFIG.3, at an operation339, the three-dimensional scene shape generating module327performs scene reconstruction. In some embodiments, this includes generating one or more incremental meshes representing the three-dimensional features of the real-world operating environment based on the outputs of the operations333and335. According to various embodiments, at the operation339, an initial coarse mesh is generated and may include planar cells defined by depth points and edges connecting depth points of part or all of the space to be described by a surface mesh. In some embodiments, the initial coarse mesh covers only the outer boundary of the space to be described. In other embodiments, the initial mesh covers the full area of the space to be represented through the mesh. Starting from the initial coarse mesh, additional new depth points at points near the existing coarse mesh are calculated and new elements are incrementally added to the mesh, hence the name “incremental mesh.” Techniques suitable for incrementing and deepening a mesh of the three-dimensional forms obtained from depth data according to this disclosure include, without limitation, advancing front techniques and automatic hex-mesh generation.

According to various embodiments, the three-dimensional scene shape generating module327returns a digital representation343of the three-dimensionality of at least a portion of the scene. In some embodiments, the digital representation343includes a dense surface map for rendering and reprojecting three-dimensional objects in the scene (such as in the reprojection295inFIG.2F). Additionally, the three-dimensional scene shape generating module327returns, based on the plane detection obtained at the operation337, an anchor configuration for the major tracked planes (such as the floor and ceiling) of the real-world operating environment.

AlthoughFIG.3illustrates one example of an architecture300for performing three-dimensional scene reconstruction and scene comprehension for XR applications, various changes may be made toFIG.3. For example, various components may be combined, further subdivided, replicated, omitted, or rearranged and additional components may be added according to particular needs.

FIG.4illustrates an example of an architecture400for performing depth reconstruction, segmentation, and three-dimensional reconstruction according to some embodiments of this disclosure. For ease of explanation, the architecture400is described as being used in the device100ofFIG.1. Also, the operations described with reference toFIG.4may be performed at one or more processing platforms implementing a back end (such as the back end395inFIG.3). In some embodiments, the operations described with reference to the example ofFIG.4may be performed as sub-parts of the processing operations described above (such as operations331,333and339inFIG.3). However, the architecture400may be implemented using any suitable device(s) and in any suitable system(s).

Referring to the illustrative example ofFIG.4, a depth reconstruction stage401outputs a reconstructed high-resolution depth map. The reconstructed high-resolution depth map can be generated using various inputs. In this example, the inputs include depth sensor data405(such as data provided by the depth sensor(s)303inFIG.3); sparse depth values407obtained from a sensor fusion (such as fused depth maps obtained at the operation329inFIG.3); one or more keyframes409of image data (such as keyframes obtained at the operation325inFIG.3); calibration data411between one or more image sensors305and one or more depth sensors303; and 6DOF pose information417(such as the 6DOF pose315inFIG.3) associated with keyframes409of image data and depth map data. While it may be possible to construct a camera (such as a camera with two sensors and a shared aperture and lens) so that a depth sensor and a camera/DVS or other image sensor have a common field of view and camera model, such a combined camera/depth sensor in some real-world embodiments is often not a viable design option. Rather, in some embodiments, image data and depth data may be obtained by physically-separate sensors mounted at different locations on an apparatus (such as an HMD), where the image and depth data project light onto a sensor according to different camera models (such as when a depth sensor may be modeled as a pinhole camera while an image sensor has a variable aperture and focal length that is not solely a function of aperture width). According to various embodiments, the calibration data411includes a set of values (or, depending on the embodiment, a matrix of values) for projecting data from an image sensor and a depth sensor to a common coordinate system.

Referring to the illustrative example ofFIG.4, at an operation413, the depth sensor data405and the image data in a keyframe409are corrected and synchronized such that common locations in the depth and image data are mapped to a common coordinate system. This can be done to help ensure that the same point in physical space represented in both sets of data is assigned to a single common coordinate. In some embodiments, the operation413includes correcting for parallax effects associated with a physical distance between the respective locations of the image and depth sensors (such as where an image sensor is closer to a left eye and the depth sensor is closer to the right eye) and reprojecting the image data and depth data based on the calibration data411to a common coordinate system.

According to various embodiments, at an operation415, the depth sensor data405is mapped and fused to the sparse depth values407obtained from a sensor fusion to obtain a fused high-resolution sparse depth map419. In some embodiments, mapping and fusing the depth sensor data405and the sparse depth values407may be performed using a depth fusion filter, such as one configured according to Equation 1 below.

In some embodiments according to this disclosure, the fused sparse depth map419can be densified by adding additional depth points in the spaces surrounding existing data points, such as by using a depth reconstruction filter. This propagates depth value-based weightings determined from pose information, intensity information, and spatial information. One example approach for performing this function is described in U.S. Patent Application Publication No. 2021/0358158, the contents of which are hereby incorporated by reference. For example, where intensity information (such as values of one or more channels of a color channel of a color model) shows consistency in a given area in the sparse depth map, the depth value in the sparse depth map may be propagated across the area. In this way, propagation of depth values is based on normalized differences in pose difference, color texture, and spatial information in the neighborhood surrounding a point of the sparse depth map. In some embodiments, these localized variations in pose, color, and spatial information may be expressed as weights computed based on a Gaussian distribution, such as the one shown by Equation 2 below.
wpose=G(diffpose, μpose, σpose),
wcolor=G(diffcolor, μcolor, σcolor),
wspatial=G(diffspatial, μspatial, σspatial)  Equation 2:

As shown inFIG.4, once Gaussian distribution weights for pose differentials, color differentials, and spatial differentials in the neighborhood of a point p are computed at the operation421, a depth reconstruction filter T can propagate depth points in the neighborhood N of point p at the operation423to obtain a reconstruct dense depth map data425. In some cases, this may occur according to Equation 3 shown below.
d(p)=q∈N(p)(wpose, wcolor, wspatial, d, q)  Equation 3:

At a general level, depth reconstruction and three-dimensional scene reconstruction of a real-world operating environment may be improved through generating a semantic understanding of the operating environment by performing semantic and instance segmentation. As used in this disclosure, semantic understanding may include identifying instances of recognized objects in image data of a real-world operating environment. Once certain shapes in an operating environment are recognized as instances of objects, certain aspects of depth and scene reconstruction may be made easier since information on the objects may be used to assist in the generation of a dense depth map.

Referring to the illustrative example ofFIG.4, a segmentation stage420begins at an operation427, where one or more frames of image data (such as one or more keyframes409) are analyzed to identify one or more objects of interest in the image data. Object detection may be performed using any suitable image recognition technique, which can recognize objects from a training set of image data. Examples of this may include, without limitation, a pretrained neural network executed by the processor implementing a back end of the architecture according to this disclosure or a suitably configured separate NPU chip. Having detected one or more forms in the image data corresponding to one or more objects in the training set, one or more regions of interest (ROIs) representing one or more defined regions within the image data can be set at the operation427.FIG.5Aprovides an example visualization of an output of an object ROI detection and extraction operation according to this disclosure.

At an operation429, for each ROI identified at the operation427, a feature map of features of interest in each ROI is generated for each frame of image data. According to some embodiments, the efficiency and computational load associated with segmentation and three-dimensional depth reconstruction can be reduced by confining three-dimensional depth reconstruction and scene understanding calculations to the vicinity of a set of one or more predefined objects within the real-world operating environment. By way of illustrative example, if the only virtual object to be added to an XR display includes a digitally-generated chess board, three-dimensional reconstruction of objects where a chess board might be placed (such as on a table) may be needed in some embodiments. In such cases, detailed reconstruction of other recognized objects or the entire real-world operating environment may be unnecessary since an XR-generated chessboard is unlikely to be positioned on the ceiling or on a person in the scene.

At an operation431, for each ROI identified at the operation431, a dense depth map for the ROI is determined. In some embodiments, the depth reconstruction stage401may be performed globally across the full area of each frame of image data. However, in other embodiments, the depth reconstruction stage401may be performed locally, such as only within the vicinity of one or more objects recognized as a foreseeable target or anchor for an item of digital content.

Referring to the illustrative example ofFIG.4, at an operation433, the one or more ROIs generated at operation427, as well as the depth and feature maps of the ROIs generated at the operations429and431, are aligned and mapped to a common size scale. This can facilitate semantic segmentation and instance segmentation since the neural networks for performing semantic and instance segmentation may use a single patch size. According to various embodiments, at an operation435, semantic and instance segmentations are performed on each ROI obtained at the operation427. Semantic segmentation may include parsing a frame of image data (such as a keyframe409) to associate regions of pixels within the frame with identified objects.FIG.5Bprovides an example visualization of the output of a semantic segmentation process according to some embodiments of this disclosure, where regions within the pixel grid associated with objects identified as people are defined and contrasted with regions associated with other surfaces and objects.

At an operation435, the processing platform implementing the segmentation stage420also performs instance segmentation. Instance segmentation may include parsing image data to divide regions identified by semantic segmentation as being associated with identified objects into sub-regions of pixels associated with specific individual instances of the recognized object(s).FIG.5Cprovides an example visualization of the output of instance segmentation according to some embodiments of this disclosure. The regions of an image frame identified as being associated with identified objects (such as people) may be further subdivided to define sub-regions of the image frame associated with individual instances of object data recognized as people. At an operation437, the processing platform implementing the back end processes obtains boundaries for each identified object in the image frame.

Referring to the illustrative example ofFIG.4, a three-dimensional scene reconstruction stage450includes generating a machine representation of the three-dimensional form of one or more of the ROIs determined at the operation427. In some embodiments, instead of confining the three-dimensional scene reconstruction stage450to one or more ROIs, three-dimensional scene reconstruction is performed for all of the areas of the real-world operating environment for which image data and depth data are available.

According to some embodiments, at an operation453, the processing platform implementing the back end according to some embodiments of this disclosure performs plane detection, size measurement, and plane reconstruction. For example, in some embodiments, the three-dimensional scene reconstruction stage450takes as its inputs the 6DOF pose information417, dense depth map data425, frames of image data (such as one or more keyframes409), and object segmentation and object boundary data. Using the pose information417to correct for projection angles, the processing platform identifies, in the dense depth map data, regions within the real-world operating environment presenting candidate planes. Using the image data and object boundary data, the processing platform also determines the size and boundary of each of one or more detected planes.

As shown in the illustrative example ofFIG.4, at an operation441, the processing platform performing three-dimensional scene reconstruction sets one or more anchor points to define and track the position of the detected plane in response to changes in pose of the sensors providing sensor data for scene reconstruction and scene understanding (such as the sensor module301inFIG.3). According to various embodiments, at an operation443, the processing platform determines, for each detected and tracked plane, a vector representing the normal of the detected plane and coordinates defining the known extents of the detected plane.

Referring to the illustrative example ofFIG.4, at an operation445, values of a TSDF are computed based on the surface normal and dense depth map to generate one or more voxel grids for the one or more areas representing one or more planes detected and tracked at the operations439and441. A voxel grid may include a grid or raster of coordinate regions (such as pixel-like regions), each of which may be associated with a depth value (rather than a value in a color space as is the case for pixels of a digital image). By calculating the TSDF at the operation445, the processing platform expands the dense depth cloud data, which only defines a depth value at discrete points of zero area within the image frame, to define regions of non-zero area associated with depth values.

At an operation447, the processing platform performs volume reconstruction based on the one or more voxel grids obtained through computation of the TSDF for one or more regions of the real-world operating environment to obtain a three-dimensional scene reconstruction449. Specifically, at the operation447, image data is used to transform the voxel grid (which in many embodiments includes a grid or matrix of square or hexagonal voxels that may or may not accurately represent object boundaries or edges of objects in the real-world operating environment) to a three-dimensional mesh of depth points connected by lines to define planar regions (typically triangles) within part or all of the real-world operating environment described by the one or more voxel grids. According to various embodiments, the three-dimensional mesh may be generated incrementally from the boundaries of planes or objects, although other meshing algorithms may be used in other embodiments according to this disclosure.

As shown inFIG.4, at an operation451, one or more virtual objects are rendered at one or more locations in order to appear as if they interact with the reconstructed three-dimensional scene, where the one or more locations at which the one or more virtual objects are rendered are based (at least in part) on at least one generated three-dimensional mesh of at least one surface in the real-world operating environment and semantic segmentation of the scene. For example, a virtual object chessboard may appear to sit directly on the surface of a table like a real chessboard when the table has been semantically recognized as an instance of a table (and thus a suitable surface for placing a virtual chessboard) and because the table, in particular, has a top that has been reconstructed from image and depth data.

Depending on the parameters of the XR display and, in particular, the size of an object, plane reconstruction at the operation439may be performed on an as-needed basis, where the computational need is based on one or dimensions of a virtual object to be positioned on a reconstructed plane. For example, when an XR application specifies a chessboard as a virtual object to appear on the surface of a table or other real-world component, the operation439may only be performed to the extent necessary to reconstruct enough of a plane upon which to position the virtual chessboard. The operation439may stop when a flat area of sufficient size has been determined.

AlthoughFIG.4illustrates one example of an architecture400for performing depth reconstruction, segmentation, and three-dimensional reconstruction, various changes may be made toFIG.4. For example, various components may be combined, further subdivided, replicated, omitted, or rearranged and additional components may be added according to particular needs.

FIGS.5A-5Cillustrate visual aspects of object detection, semantic segmentation, and instance segmentation according to some embodiments of this disclosure. Referring to the illustrative example ofFIG.5A, a frame of image data500of a scene is shown. The scene includes a person and six sculptures having human forms. In the example ofFIG.5A, object detection and extraction of regions of interest (ROIs) according to some embodiments of this disclosure have been performed. As shown inFIG.5A, ROIs corresponding to objects for which a neural network trained for object recognition has been detected have been defined. In this example, the ROIs include boxes around the human forms labeled “Person1” through “Person7.” In this example, while ROIs containing recognized objects have been found, the specific boundaries between the detected objects and the rest of the scene in the image data500have not yet been determined, nor have the boundaries between overlapping instances of the same object (such as the regions designated “Person6” and “Person7”) been determined.

FIG.5Bprovides an illustrative visualization of a semantic segmentation505of frame of image data500fromFIG.5A. Referring to the explanatory example ofFIG.5B, each the constituent pixels of the image data500has been classified (such as by using a machine learning tool trained for semantic segmentation, like DeepLab) and colored according to their classification. In this example, pixels associated with human forms (labeled “person pixels” in the figure) have been colored white, while background components of the scene have been colored in shades of dark grey.

FIG.5Cprovides an illustrative visualization of an instance segmentation510as applied to the semantic segmentation505inFIG.5B. As described elsewhere in this disclosure, instance segmentation includes further subdividing a semantic segmentation to recognize the regions of pixels including individual instances of a recognized object. In this illustrative example, the pixels labeled as “person pixels” inFIG.5Bhave been separately identified as instances of people and separately labeled “Person1” through “Person7.” Additionally, an object boundary515between “Person6” and “Person7” has been determined from what was, inFIG.5B, an undifferentiated region of pixels generally associated with image content recognized as having human forms.

AlthoughFIGS.5A-5Cillustrate visual aspects of object detection, semantic segmentation, and instance segmentation, various changes may be made toFIGS.5A-5C. For example, the contents of actual images that are obtained and processed can vary widely, andFIGS.5A-5Cdo not limit this disclosure to any specific type of scene or image.

FIG.6illustrates an example of plane detection and reconstruction according to some embodiments of this disclosure. Referring to the illustrative example ofFIG.6, a frame600of image data is shown. The frame600of image data includes a sensor view of a real-world operating environment of an XR device (such as an HMD). The scene data may be obtained for an XR application configured to render a virtual chessboard on a suitable surface represented in the frame600of image data. Thus, the technical challenges associated with providing the above-described XR display of a virtual chessboard in this real-world operating environment include finding a planar surface that is dimensionally suitable (such as occupying sufficient area within the frame600to appear to support the virtual chessboard) and, in some embodiments, that is semantically suitable (such as belonging to a class of recognized objects the XR application permits virtual chessboards to be placed upon).

In this explanatory example, using the results of a previously-performed segmentation of objects in the frame600, a table603is identified as semantically suitable for positioning the virtual object. According to various embodiments, the output of an instance segmentation (such as the instance segmentation510) showing the boundaries of the table603is compared against data showing a rendered size605of the virtual chessboard to determine whether the table603is large enough to position the virtual chessboard. Responsive to determining that the table603is large enough to position the virtual chessboard, a three-dimensional reconstruction (such as an incremental mesh607) of the table603is generated.

AlthoughFIG.6illustrates one example of plane detection and reconstruction, various changes may be made toFIG.6. For example, the contents of actual images that are obtained and processed can vary widely, andFIG.6does not limit this disclosure to any specific type of scene or image.

FIG.7illustrates operations of an example method700for performing three-dimensional scene reconstruction and scene comprehension for XR applications according to some embodiments of this disclosure. The operations of the method700may be performed at one or more processing platforms implementing an architecture (such as the architecture300inFIG.3) including a front end and a back end according to some embodiments of this disclosure.

Referring to the illustrative example ofFIG.7, at an operation705, a processing platform (such as the device100inFIG.1) implementing a front end (such as the front end397inFIG.3) receives depth, image, and movement data from a sensor module (such as the sensor module301inFIG.3) trained upon a portion of a real-world operating environment. According to some embodiments, the depth data includes data from a depth sensor (such as the depth sensor303), and the image data includes image frames from an image sensor (such as the image sensor305). Motion data may include sensor outputs recording changes in location of a sensor module, as well as data recording changes in the viewing angle of the sensor module.

As shown inFIG.7, at an operation710, the processing platform implementing the front end of the architecture determines an initial six-degree-of-freedom pose (such as the 6DOF pose315inFIG.3) based on one or more of the depth data, the image data, and the movement data. At an operation715, the initial pose data is passed to the back end of the architecture (such as the back end395inFIG.3) to determine one or more of a keyframe or optimized pose. In some embodiments, the sensors (such as the IMU307) in the sensor module301may be susceptible to drift, external effects, or other factors that add noise or otherwise degrade the accuracy of the determined pose. In embodiments in which three-dimensional reconstruction involves calculating projections of rays from one or more specified poses (such as in TSDF-based approaches), errors or inconsistencies in pose data can be readily propagated into the three-dimensional reconstruction of the scene. According to various embodiments, at an operation725, the back end generates a three-dimensional reconstruction of at least part of the real-world scene based on the image data, the depth data, and (in some embodiments) the optimized pose data. The three-dimensional reconstruction may include one or more of a dense depth map, a dense surface mesh describing a portion of the real-world scene, and one or more semantically segmented objects within the scene.

At an operation725, the three-dimensional reconstruction of the real-world scene generated at the operation720is passed from the back end of the processing architecture to the front end. At an operation730, the front end renders a frame of XR content including a virtual three-dimensional object positioned on a surface or otherwise interacting with a surface of the real-world scene.

AlthoughFIG.7illustrates operations of one example method700for performing three-dimensional scene reconstruction and scene comprehension for XR applications, various changes may be made toFIG.7. For example, while shown as a series of steps, various steps inFIG.7may overlap, occur in parallel, occur in a different order, or occur any number of times. Also, various steps may be omitted or replaced by other steps.