Reconstruction of essential visual cues in mixed reality applications

A mixed reality (MR) simulation system includes a console and a head mounted device (HMD). The MR system captures stereoscopic images from a real-world environment using outward-facing stereoscopic cameras mounted to the HMD. The MR system preprocesses the stereoscopic images to maximize contrast and then extracts a set of features from those images, including edges or corners, among others. For each feature, the MR system generates one or more two-dimensional (2D) polylines. Then, the MR system triangulates between 2D polylines found in right side images and corresponding 2D polylines found in left side images to generate a set of 3D polylines. The MR system interpolates between 3D vertices included in the 3D polylines or extrapolates additional 3D vertices, thereby generating a geometric reconstruction of the real-world environment. The MR system may map textures derived from the real-world environment onto the geometric representation faster than the geometric reconstruction is updated.

BACKGROUND

Field of the Various Embodiments

Various embodiments relate generally to virtual reality and augmented reality, and, more specifically, to reconstruction of essential visual cues in mixed reality applications.

Description of the Related Art

Artificial reality systems simulate virtual environments or add virtual elements to real environments in order to provide virtual reality (VR), augmented reality (AR), and/or mixed reality (MR) content to a user. The artificial reality system includes a console and a headset. The console generates and renders graphical elements, while the headset outputs these graphical elements to a user. A headset includes a stereoscopic display that outputs slightly different images to each eye of the user, thereby causing the user to perceive a 3D spatial environment. Headsets may also include outward facing cameras configured to capture stereoscopic imagery in order to generate a mixed reality simulation.

The term “mixed reality” generally refers to any type of simulation where simulated graphical elements are displayed to a user in conjunction with real-world elements and/or simulated versions of real-world elements. For example, a console could capture stereoscopic images representing the environment surrounding the user via the outward facing cameras on the headset. The console would then generate three-dimensional (3D) graphical elements associated with a video game or other type of computer simulation. The console would then incorporate the generated 3D graphical elements into the stereoscopic images and output the result to the user via the headset.

Outward-facing cameras mounted on headsets are usually separated from one another with a camera disparity that differs from the interpupillary distance of most users. Consequently, stereoscopic imagery that is captured and subsequently displayed to a user may appear spatially distorted, which can lead to a poor user experience. Further, this spatial distortion can diminish the user's spatial awareness of the real-world environment, which can be dangerous in some cases.

Artificial reality systems typically lack the processing power to generate a mixed reality simulation that addresses this spatial distortion while also maintaining a high quality user experience. Consequently, these systems output mixed reality simulations with very low frame rates, and, therefore, a noticeable latency. This latency is distracting and leads to a poor user experience. Large latencies can also be dangerous because users may unknowingly encounter (and potentially collide with) real-world objects before the simulation can be updated to represent those objects. Furthermore, without addressing the spatial distortion mentioned above, the mixed reality simulations generated by conventional artificial reality systems can cause the user to experience motion sickness and disorientation, resulting in a poor user experience.

SUMMARY

Various embodiments include a computer-implemented method, including identifying a set of features associated with one or more objects in a real-world scene, generating two-dimensional (2D) geometry based on the set of features, generating three-dimensional (3D) geometry based on the 2D geometry, the 3D geometry comprising a geometric reconstruction of the real-world scene that only partially represents the one or more objects, and rendering, based on the 3D geometry, a first graphical representation of the real-world scene for display.

At least one advantage of the disclosed techniques is that the MR system can produce an immersive and spatially realistic mixed reality experience with little or no latency. Because the MR system generates the geometric reconstruction based only on essential geometric features of the real-world environment, the MR system can update that reconstruction much faster compared to conventional systems that must generate and update highly complex geometric reconstructions. In addition, since the MR system maps textures derived from the real-world environment onto the geometric reconstruction in real time, the MR system can output mixed reality simulations with negligible latency. Accordingly, the techniques disclosed herein represent a significant technological advancement compared to conventional approaches.

DETAILED DESCRIPTION

Some mixed reality systems capture stereoscopic images from the real-world and then render a geometrical reconstruction of the real-world to users. Because this approach requires highly complex geometry to be updated in real time, such simulations cannot operate with a sufficiently high frame rate to approximate reality. Consequently, these types of simulations suffer from noticeable latencies, which can diminish the user experience and potentially expose the user to safety risks. These issues are especially problematic in mobile platforms that have limited compute power and limited memory.

To address these issues, various embodiments include a mixed reality (MR) system configured to update a geometric reconstruction of a real-world environment with a given latency and render portions of the geometric reconstruction for display to the user with a lower latency. The MR system may additionally map textures derived from the real-world environment onto the geometric representation with a lower latency.

In operation, the MR system captures stereoscopic images from the real-world environment using a set of outward-facing stereoscopic cameras mounted to a head mounted device (HMD). The MR system preprocesses the stereoscopic images to maximize contrast and then extracts a set of features from those images, including lines, edges, or corners, among others. In other embodiments, the MR system extracts the set of features without performing contrast enhancement operations. For each feature, the MR system generates one or more two-dimensional (2D) polylines. The MR system triangulates between 2D polylines found in right side images of the stereoscopic images and corresponding 2D polylines found in left side images of the stereoscopic images to generate a set of 3D polylines. The MR system interpolates between 3D vertices included in the 3D polylines or extrapolates additional 3D vertices, thereby generating an approximate geometric reconstruction of the real-world environment.

The MR system may update the geometric reconstruction in real-time or slower than real time, depending on hardware capabilities. The MR system may also generate and render 3D graphics for display to the user in real time and with a high frame rate. When rendering the graphics, the MR system is configured to remap textures derived from the raw stereoscopic images of the real-world environment back onto the geometric reconstruction in order to enhance the realism of those 3D graphics.

At least one advantage of the techniques described herein is that the MR system can produce an immersive and spatially realistic mixed reality experience with little or no latency. Because the MR system generates the geometric reconstruction based on essential geometric features of the real-world environment, the MR system can update that reconstruction much faster compared to conventional systems that generate and update highly complex geometric reconstructions. In addition, since the MR system maps textures derived from the real-world environment onto the geometric reconstruction in real time, the MR system can output realistic mixed reality simulations with negligible latency. Accordingly, the techniques disclosed herein represent a significant technological advancement compared to conventional approaches.

System Overview

FIG. 1illustrates a system configured to implement one or more aspects of the various embodiments. As shown, MR system100includes a head mounted device (HMD)110and a console120. HMD110is a wearable device configured to be worn by a user150. HMD110is configured to immerse user150into a mixed reality simulation generated by console120or HMD110. That simulation is based, at least in part, on a real-world scene160associated with the environment where user150resides. In one embodiment, HMD110may operate as a pass-through device to relay images of the environment to user150. Those images may be either unmodified or augmented by HMD110.

HMD110includes a display112, optics114, eye tracking module116, and head tracking module118. Display112is a stereoscopic output device configured to output stereoscopic images to user150. Those images may represent real-world scene160as viewed from slightly different perspectives, thereby causing user150to perceive a three-dimensional (3D) spatial environment. Optics114includes an assembly of optical devices, including one or more outward facing cameras. In one embodiment, optics114performs various optical adjustments with light that is captured and/or emitted via HMD110.

Eye tracking module116is configured to track the eye gaze direction of user150. In one embodiment, eye tracking module116measures various attributes of user150, including the interpupillary distance of user150, and then configures optics114to perform various optical adjustments based on those measurements. In another embodiment, eye tracking module116may be omitted. Head tracking module118is configured to track the head position and/or orientation of user150. In one embodiment, one or more perspectives associated with simulations output via HMD110are modified based on head tracking data gathered via head tracking module118.

Console120is a computing device configured to execute a simulation application130in order to generate the mixed reality simulation output to user150via HMD110. In one embodiment, HMD110executes simulation application130to generate the mixed reality simulation. An exemplary computing device that may be configured to implement either of HMD110and console120is illustrated inFIG. 9. Simulation application130includes a preprocessor132, a feature identifier134, a geometry modeler136, a depth estimator138, a geometry projector140, and a texture mapper142. Each of these blocks represents a different stage in a processing pipeline configured to generate the mixed reality simulation of real-world scene160mentioned above. As is shown, real-world scene160includes walls162and164and floor166. A frame170is mounted to wall162and holds a painting172. A box180rests on floor166, and a placemat182rests on box180.

In operation, preprocessor132receives left side and right side images of real-world scene160captured stereoscopically via optics114. In one mode of operation, preprocessor132preprocesses those images to increase contrast, as described in greater detail below in conjunction withFIG. 2. Feature identifier134analyzes the preprocessed images to identify one or more features, as described in greater detail below in conjunction withFIG. 3. Geometry modeler136generates, based on the identified features, feature geometry representing real-world scene160, as described in greater detail below in conjunction withFIG. 4. Depth estimator138interpolates and/or extrapolates depth information to enhance the feature geometry and generate a geometric reconstruction, as described in greater detail below in conjunction withFIG. 5. Geometry projector140renders stereoscopic images based on the geometric reconstruction, as described in greater detail below in conjunction withFIG. 6. Additionally, texture mapper142may extract textures from the original raw images and map these textures onto the geometric reconstruction to provide more detailed renderings of the geometric reconstruction, as described in greater detail below in conjunction withFIG. 7.

The approach described herein confers several technological advantages. In particular, simulation application130generates a simplified geometrical reconstruction that may be rendered in real time with sufficiently powerful hardware. However, if such hardware is not available and the reconstruction cannot be updated in real time, then simulation application130can nonetheless project that geometry for stereoscopic rendering in real time and with minimal latency. In either case, simulation application130is typically capable of mapping textures onto the geometrical reconstruction in real time, thereby enhancing the appearance and realism of the mixed reality simulation. These techniques represent a significant advancement over conventional mixed reality simulations in general, and 3D passthrough for mixed reality applications specifically, which can appear either physically distorted or slow due to excessive latency.

Exemplary Generation of a Mixed Reality Simulation

FIGS. 2-7illustrate how simulation application130implements the various processing stages discussed briefly above. Persons skilled in the art will appreciate that these Figures and corresponding descriptions are provided for exemplary purposes and not meant to be limiting in scope.

FIG. 2illustrates how the simulation application ofFIG. 1preprocesses raw images associated with a real-world scene, according to various embodiments. As shown, preprocessor132within simulation application130generates preprocessed scene200with an elevated overall contrast level to highlight sets of edges202,204,206,210, and220. Preprocessor132analyzes raw stereoscopic images depicting real-world scene160and then boosts contrast in some or all regions of those images. In one embodiment, preprocessor132may adjust image contrast differently across different image regions. Preprocessor132can increase image contrast in this manner to facilitate subsequent detection of features within preprocessed images, although in some embodiments feature detection may be performed without first adjusting image contrast. The detection of features is described in greater detail below in conjunction withFIG. 3.

FIG. 3illustrates how the simulation application ofFIG. 1identifies features within preprocessed images, according to various embodiments. As shown, feature identifier134within simulation application130detects features of the real-world scene160including, for example, specific scene features300, including edges302,304, and306of the room where user150resides, edges310,312,314, and316of frame170, and edges320,322,324, and326(among others) of box180.

When identifying features in this manner, feature identifier134implements computer vision techniques to analyze left side and right side images. Feature identifier134may identify the presence of lines, edges, corners, surfaces, and any other technically feasible feature that can be identified based on a set of pixels. In one embodiment, the essential features identified via feature identifier134may include any type of visual cue or visual reference that user150may recognize for the purposes of localization and/or orientation. Such features may be referred to as “essential” in this disclosure because human spatial perception relies, at least to some degree, on recognizing these features for precise navigation of a physical space. As described in greater detail below, MR system100reproduces these essential features to assist user150with safe navigation of real-world environments during immersion in the mixed-reality simulation. In identifying features, feature identifier134may implement a Canny edge detection algorithm to identify specific groups of pixels that belong to specific edges or any other computer vision technique for identifying recognizable features. Feature identifier134may tag pixels or groups of pixels within preprocessed images with specific classifications and then output tagged, preprocessed images to geometry modeler136, as described in greater detail below in conjunction withFIG. 4.

FIG. 4illustrates how the simulation application ofFIG. 1generates feature geometry based on a set of identified features, according to various embodiments. As shown, geometry modeler136within simulation application130generates feature geometry400to represent specific geometric attributes of real-world scene160. Feature geometry400includes polylines402,404, and406representing edges of the room where user150resides, polylines410representing frame170, and polylines420representing box180. Each polyline includes a set of vertices. For example, polyline402includes vertices402(0) through402(4). In various embodiments, geometry modeler136may generate any technically feasible geometric primitive instead of 2D polylines. For example, geometry modeler136could generate 2D feature points or 2D patches, among other types of geometric primitives. Persons skilled in the art will understand how the techniques described herein can be performed with any type of geometric primitive instead of 2D polylines.

To generate feature geometry400, geometry modeler136generates a different 2D polyline to represent each feature identified by feature identifier143. For example, geometry modeler136could overlay a line segment across a given edge detected by feature identifier143and generate a set of vertices defining that line segment. Geometry modeler136performs this operation for both left side and right side images. Then, geometry modeler136matches some or all 2D polylines included in the left side image to corresponding 2D polylines included in the associated right side image.

Geometry modeler136may implement any technically feasible approach to matching 2D polylines, vertices of those polylines, and individual pixels associated with those polylines between different images. In one embodiment, geometry modeler136implements an image-space patch matching approach to correlate polyline vertices between left and right side images. Geometry modeler136then triangulates between matched 2D polylines in order to infer depth information for each vertex of those polylines, thereby generating 3D polylines. Each 3D polyline includes a set of 3D vertices. Geometry modeler136may also discard specific 2D polylines that are too far away from one another to permit the accurate computation of depth information. Geometry modeler136may implement any technically feasible approach for triangulating elements of stereoscopic image pairs to infer the depth information described herein. The generated 3D polylines form the basis for a geometric reconstruction of real-world scene160and are processed further via depth estimator138, as described in greater detail below in conjunction withFIG. 5.

FIG. 5illustrates how the simulation application ofFIG. 1generates a geometric reconstruction based on feature geometry, according to various embodiments. As shown, depth estimator138within simulation application130generates geometric reconstruction500based on feature geometry400discussed above in conjunction withFIG. 4. Geometric reconstruction500includes 3D vertices502interpolated between polylines410, 3D vertices504interpolated between polylines404and406, 3D vertices506interpolated between polylines420, and 3D vertices508interpolated between 402 and 406. Geometric reconstruction500includes additional depth information that is interpolated and/or extrapolated based on the 3D vertices of the 3D polylines generated via geometry modeler136. In one embodiment, depth estimator138also filters outliers prior to interpolation/extrapolation in order to avoid generating 3D vertices based on polylines that are exceedingly far away from one another.

Because geometric reconstruction500is generated based on essential features of real-world scene160, such as lines, edges, and so forth, geometric reconstruction500may only partially represent objects within that scene. Accordingly, geometric reconstruction500may be considered a coarse approximation of real-world scene160. Nonetheless, geometric reconstruction500may appear highly accurate from a geometrical standpoint due to the computational emphasis place on reconstructing essential cues and other features, including object contours and silhouettes.

In one embodiment, depth estimator138may implement a Delaunay triangulation based on the 3D polylines generated via geometry modeler136in order to generate a triangulated mesh for approximating additional 3D vertices. This mesh traverses between 3D polylines and forms a surface that approximates the geometry of real-world scene160at a more global level than previous interpolation/extrapolation operations. Estimating depth for 3D vertices in this manner may be computationally efficient because the actual geometry between any two edges or corners in a typical real-world structure is often planar. For example, floor166is a flat plane, and so numerous 3D vertices across that plane can be approximated with high precision based on polylines402and406bordering that plane. An advantage of this approach is that visual artifacts commonly associated with the faithful reconstruction of certain real-world features, such as flat surfaces, for example, can be reduced because the geometry of those features can be estimated instead of reconstructed. Stylizing those features may further reduce artifacts, as well. Once depth estimator138generates the additional 3D vertices in the manner discussed above, geometry projector140may render 3D graphics that can be output to user150, as described in greater detail below in conjunction withFIG. 6.

FIG. 6illustrates how the simulation application ofFIG. 1renders a reconstructed version of a real-world scene, according to various embodiments. As shown, geometry projector140within simulation application130generates rendered scene600for display to user150based on geometric reconstruction500. Geometry projector140may generate stereoscopic images representing those 3D graphics based on geometrical features of user150, including interpupillary distance, among others. Accordingly, rendered scene600may appear spatially realistic to user150and lack significant spatial distortion. In one embodiment, geometry projector140may implement a stylized rendering approach when generating rendered scene600. For example, geometry projector140may render individual lines to appear as hand-sketched lines. This approach may reduce the noticeability of any visual artifacts that may be introduced via the reconstruction process described above.

Rendered scene600represents the essential geometry of real-world scene160that is needed to permit the localization and orientation of user150within the environment. Due to the simplicity of rendered scene600, that scene can be rendered for display in real time. However, geometric reconstruction500underlying that scene need not be updated at an equally high rate. Instead, geometric reconstruction500can be updated based on captured stereoscopic imagery as fast as hardware permits, yet still used for real-time rendering purposes based on the position and/or orientation of HMD110. To further improve realism, textures derived from real-world scene160can also be mapped onto geometric reconstruction500prior to rendering, as described in greater detail below in conjunction withFIG. 7.

FIG. 7illustrates how the simulation application ofFIG. 1maps textures from a real-world scene onto a geometric reconstruction, according to various embodiments. As shown, texture mapper142within simulation application130maps various textures onto geometric reconstruction500and then regenerates rendered scene600to produce a textured scene700. Textured scene700includes reconstructed frame710with painting texture712mapped onto that frame, and reconstructed box720with placemat texture722mapped thereto. Painting texture712is derived from raw images of painting172, while placemat texture722is derived from raw images of placement182.

In one embodiment, texture mapper142analyzes raw images captured via HMD110to identify textures residing between the 3D polylines generated via geometry modeler136. Texture mapper142then captures stereoscopic clips of these textures and remaps those clips onto the corresponding portion of geometric reconstruction600. For example, texture mapper142could analyze raw images depicting painting172and then, based on polylines410and/or 3D vertices502, capture a stereoscopic clip representing painting172. Subsequently, texture mapper142could map that clip onto rendered scene600at a location associated with polylines410. Texture mapper142may perform these steps in real-time or as fast as hardware allows. This approach may increase the realism of the mixed reality simulation by adding images to spaces that may otherwise remain blank within geometric reconstruction500. In another embodiment, geometry projector140may implement a technique known in the art as “superpixels” to render large regions of geometric reconstruction500with similar color values and with edges aligned to edges within geometric reconstruction500. In other embodiments, geometric reconstruction500can be tessellated uniformly or non-uniformly to fill in otherwise blank areas.

Referring generally toFIGS. 1-7, each of the different processing stages discussed thus far may be performed at a different rate depending on the hardware capabilities of HMD110and/or console120. In some cases, all of these steps can be performed in real-time. In other cases, preprocessor132, feature identifier134, geometry modeler136, and depth estimator138operate slower than real time, while geometry projector140and texture mapper142operate in real time. In any case, simulation application130is configured to immersing user150into a mixed reality simulation that is spatially realistic and, further, appears to update and render in real time.

Procedure for Generating a Mixed Reality Simulation

FIG. 8is a flow diagram of method steps for generating a low-latency simulation of a real-world scene, according to various embodiments. Although the method steps are described in conjunction with the systems ofFIGS. 1-7, persons skilled in the art will understand that any system may be configured to perform the method steps in any order.

As shown, a method800begins at step802, where preprocessor132within simulation application130preprocesses raw stereoscopic images captured via HMD110to increase contrast in specific regions. In one embodiment, preprocessor132may adjust image contrast differently across different regions of the raw images. Preprocessor132generally increases image contrast in this manner to facilitate subsequent detection of features within preprocessed images, including edges, among other features. An example of how preprocessor132preprocessed an image is discussed above in conjunction withFIG. 2.

At step804, feature identifier134within simulation application130analyzes preprocessed images to extract and tag features. Feature identifier134may implement any technically feasible feature identification technique to identify features within left and right side images captured via HMD110. Those features may include lines, edges, corners, surfaces, and so forth. In one embodiment, feature identifier134implements a Canny edge detection algorithm to identify specific groups of pixels that belong to specific edges. Feature identifier134then tags pixels or groups of pixels within preprocessed images with specific feature classifications. An example of how feature identifier134identifies features is discussed above in conjunction withFIG. 3.

At step806, geometry modeler136within simulation application130generates feature geometry by triangulating between 2D polylines generated for tagged features. In doing so, geometry modeler136generates one or more 2D polylines for each feature tagged via step804. For example, geometry modeler136could trace a detected edge and then generate a set of 2D vertices along that edge. Those 2D vertices would form a 2D polyline aligned with the edge. Then, geometry modeler136triangulates between matched pairs of the 2D polylines generated at step806to estimate depth values for each vertex of those 2D polylines, thereby generating 3D polylines. An example of how geometry modeler136generates 3D polylines is discussed above in conjunction withFIG. 4.

At step808, depth estimator138within simulation application130generates a geometric reconstruction by interpolating or extrapolating the 3D polylines generated at step806to produce additional 3D vertices. Those additional 3D vertices reside in regions of the feature geometry for which little to no geometrical information is available. However, because many spatial environments include numerous planar surfaces, 3D vertices can be generated to represent these regions based on detected features bordering those regions. In one embodiment, depth estimator138implements a Delaunay triangulation based on the 3D polylines generated at step806in order to generate a triangulated mesh for approximating the additional 3D vertices. An example of how depth estimator138generates a geometric reconstruction is discussed above in conjunction withFIG. 5.

At step810, geometry projector140renders 3D graphics based on the geometric reconstruction generated at step808. Geometry projector140may implement any technically feasible approach to rendering graphics based on a 3D model, and may also generate stereoscopic images based on camera angles corresponding to geometrical attributes of user150. In one embodiment, geometry projector140may implement a stylized rendering approach when generating rendered scene600. For example, geometry projector140may render individual lines to appear as hand-sketched lines. This approach may reduce the noticeability of any visual artifacts potentially introduced via the reconstruction process described above. An example of how geometry projector140renders graphics is discussed above in conjunction withFIG. 6.

At step812, texture mapper142maps textures derived from raw images captured via HMD110onto corresponding portions of the reconstructed geometry. For example, texture mapper142could analyze raw images captured prior to step802, and then clip specific portions of those images corresponding to particular regions of the geometric reconstruction. Subsequently, texture mapper142could map the clipped portions onto the geometric reconstruction at appropriate locations. Texture mapper142may perform these steps in real-time or as fast as hardware allows. In one embodiment, texture mapper142performs step812prior to geometry projector810rendering the 3D graphics. In another embodiment, the method800skips the method812altogether. An example of how texture mapper142maps textures is discussed above in conjunction withFIG. 7.

Referring generally toFIGS. 1-8, person skilled in the art will understand that any technically feasible form of computer hardware and/or or software may be configured to perform any of the techniques discussed thus far. An exemplary computing device is described in greater detail below in conjunction withFIG. 9.

Example Computing Device

FIG. 9illustrates a computing device included in the system ofFIG. 1, according to various embodiments. As shown, computing device900includes a processor910, input/output (I/O) devices920, and memory930. Memory930includes a software application932and a database934. Processor910may include any hardware configured to process data and execute software applications. I/O devices920include devices configured to receive input, devices configured to provide output, and devices configured to both receive input and provide output. Memory930may be implemented by any technically feasible storage medium. Software application932includes program code that, when executed by processor910, performs any of the functionality described herein, including that associated with simulation application130. Software application932may access data stored in database934. Those skilled in the art will understand that computing device900is provided for example purposes only and not meant to limit the scope of the present embodiments.

In sum, a mixed reality (MR) simulation system includes a console and a head mounted device (HMD). The MR system captures stereoscopic images from a real-world environment using a set of outward-facing stereoscopic cameras mounted to the HMD. The MR system preprocesses the stereoscopic images to maximize contrast and then extracts a set of features from those images, including edges or corners, among others. For each feature, the MR system generates one or more two-dimensional (2D) polylines. Then, the MR system triangulates between 2D polylines found in right side images and corresponding 2D polylines found in left side images to generate a set of 3D polylines. The MR system interpolates between 3D vertices included in the 3D polylines or extrapolates additional 3D vertices, thereby generating a geometric reconstruction of the real-world environment. The MR system may map textures derived from the real-world environment onto the geometric representation faster than the geometric reconstruction is updated.

At least one advantage of the techniques described herein is that the MR system can produce an immersive and spatially realistic mixed reality experience with little or no latency. Because the MR system generates the geometric reconstruction based only on essential geometric features of the real-world environment, the MR system can update that reconstruction much faster compared to conventional systems that must generate and update highly complex geometric reconstructions. In addition, since the MR system maps textures derived from the real-world environment onto the geometric reconstruction in real time, the MR system can output mixed reality simulations with negligible latency. Accordingly, the techniques disclosed herein represent a significant technological advancement compared to conventional approaches.

1. In some embodiments, a computer-implemented method comprises identifying a set of features associated with one or more objects in a real-world scene, generating two-dimensional (2D) geometry based on the set of features, generating three-dimensional (3D) geometry based on the 2D geometry, the 3D geometry comprising a coarse geometric reconstruction of the real-world scene, and rendering, based on the 3D geometry, a first graphical representation of the real-world scene for display.

2. The computer-implemented method of clause 1, wherein the coarse geometric reconstruction includes one or more visual cues for navigating the real-world scene.

3. The computer-implemented method of clause 1 or 2, wherein the coarse geometric reconstruction comprises a representation of the real-world scene at a first point in time, and further comprising rendering, based on the 3D geometry and at a second point in time later than the first point in time, a second graphical representation of the real-world scene.

4. The computer-implemented method of any of clauses 1-3, wherein the first graphical representation of the real-world scene is displayed via a mixed reality system.

5. The computer-implemented method of any of clauses 1-4, wherein identifying the set of features comprises identifying at least one of an edge, a corner, and a surface within a set of images depicting the real-world scene.

6. The computer-implemented method of any of clauses 1-5, wherein the set of images comprises a stereoscopic pair of images that includes a left camera image and a right camera image.

7. The computer-implemented method of any of clauses 1-6, wherein generating the 2D geometry comprises generating at least one polyline for each feature included in the set of features.

8. The computer-implemented method of any of clauses 1-7, wherein generating the 3D geometry comprises identifying a first polyline included in the 2D geometry, identifying a second polyline included in the 2D geometry, triangulating between the first polyline and the second polyline to generate a plurality of depth values, and generating a 3D polyline based on the first polyline, the second polyline, and the plurality of depth values.

9. The computer-implemented method of any of clauses 1-8, wherein generating the 3D geometry comprises identifying a first polyline included in the 2D geometry, identifying a second polyline included in the 2D geometry, and generating a triangulated mesh based on the first polyline and the second polyline to produce a plurality of 3D vertices for the 3D geometry.

10. The computer-implemented method of any of clauses 1-9, wherein the first polyline is generated based on a first image depicting the real-world scene from a first perspective, and wherein the second polyline is generated based on a second image depicting the real-world scene from a second perspective.

11. The computer-implemented method of any of clauses 1-10, wherein the graphics are rendered for display based further on at least one attribute of a user.

12. The computer-implemented method of any of clauses 1-11, wherein the at least one attribute of the user comprises an interpupillary distance associated with the user.

13. The computer-implemented method of any of clauses 1-12, further comprising increasing a locally-adaptive contrast value associated with a set of images depicting the real-world scene to generate a set of preprocessed images.

14. The computer-implemented method of any of clauses 1-13, wherein the set of features are identified within the set of preprocessed images.

15. The computer-implemented method of any of clauses 1-14, wherein rendering the first graphical representation of the real-world scene comprises extracting a texture from a region within an image depicting the real-world scene, and mapping the texture onto a portion of the coarse geometric reconstruction corresponding to the region.

16. In some embodiments, a non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the steps of identifying a set of features associated with one or more objects in a real-world scene, generating two-dimensional (2D) geometry based on the set of features, generating three-dimensional (3D) geometry based on the 2D geometry, the 3D geometry comprising a coarse geometric reconstruction of the real-world scene, and rendering, based on the 3D geometry, a first graphical representation of the real-world scene for display.

17. The non-transitory computer-readable medium of clause 16, further comprising the steps of updating the 3D geometry with a first latency, and mapping the first texture onto the portion of the 3D geometry with a second latency, the first latency exceeding the second latency.

18. The non-transitory computer-readable medium of clause 16 or 17, wherein the first graphical representation is rendered for display with the second latency.

19. In some embodiments, a system comprises a memory storing program instructions, and a processor that executes the instructions to perform the steps of identifying a set of features associated with one or more objects in a real-world scene, generating two-dimensional (2D) geometry based on the set of features, generating three-dimensional (3D) geometry based on the 2D geometry, the 3D geometry comprising a coarse geometric reconstruction of the real-world scene, and rendering, based on the 3D geometry, a first graphical representation of the real-world scene for display.

20. The system of clause 19, wherein the processor performs the additional steps of updating the 3D geometry with a first latency, and mapping the first texture onto the portion of the 3D geometry with a second latency, the first latency exceeding the second latency.