FEATURE CORRELATION

Systems and techniques are described herein for improved feature correlation. For instance, an apparatus for improved featured correlation is provided. The method may include a projector configured to project a pattern into a scene for feature correlation by an imaging device that captures images of the pattern as projected into the scene; wherein the apparatus is separate from the imaging device.

TECHNICAL FIELD

The present disclosure generally relates to feature correlation. For example, aspects of the present disclosure include systems and techniques for improving the capability of devices to correlate features captured in images.

BACKGROUND

A passive stereo-vision system may capture stereoscopically-paired images of a scene using two cameras that are a predetermined distance apart. The passive stereo-vision system may correlate features within the images and determine respective depths to points in the scene represented by the features. For example, the passive stereo-vision system may determine distances to points in the scene based on where the unique features appear in each of the images and the predetermined distance between the cameras.

A six-degree-of-freedom (6DoF) system, according to a visual simultaneous localization and mapping (VSLAM or SLAM) technique, may capture successive images of a scene and track positions of unique features between the successive images. The 6DoF system may assume that the unique features are stationary and may assume that any change in the position of the unique features between the successive images is based on movement or reorientation of the 6DoF system. The 6DoF system may calculate a change in pose of the 6DoF system based on changes in positions of the unique features between the successive images.

SUMMARY

Systems and techniques are described for improved feature correlation. According to at least one example, an apparatus for improved feature correlation is provided. The apparatus includes: a projector configured to project a pattern into a scene for feature correlation by an imaging device that captures images of the pattern as projected into the scene; wherein the apparatus is separate from the imaging device.

In another example, a method is provided for improved feature correlation. The method includes: determining to project a pattern into a scene for feature correlation by an imaging device that captures images of the pattern as projected into the scene; and projecting the pattern into the scene from a projector that is separate from the imaging device.

In another example, an apparatus for improved feature correlation is provided that includes at least one memory and at least one processor (e.g., configured in circuitry) coupled to the at least one memory. The at least one processor configured to: determine to project a pattern into a scene for feature correlation by an imaging device that captures images of the pattern as projected into the scene; and project the pattern into the scene from a projector that is separate from the imaging device.

In another example, a non-transitory computer-readable medium is provided that has stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: determine to project a pattern into a scene for feature correlation by an imaging device that captures images of the pattern as projected into the scene; and project the pattern into the scene from a projector that is separate from the imaging device.

In another example, an apparatus for improved feature correlation is provided. The apparatus includes: means for determining to project a pattern into a scene for feature correlation by an imaging device that captures images of the pattern as projected into the scene; and means for projecting the pattern into the scene from a projector that is separate from the imaging device.

In some aspects, one or more of the apparatuses described herein is, can be part of, or can include a mobile device (e.g., a mobile telephone or so-called “smart phone”, a tablet computer, or other type of mobile device), an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a vehicle (or a computing device or system of a vehicle), a smart or connected device (e.g., an Internet-of-Things (IoT) device), a wearable device, a personal computer, a laptop computer, a video server, a television (e.g., a network-connected television), a robotics device or system, or other device. In some aspects, each apparatus can include an image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, each apparatus can include one or more displays for displaying one or more images, notifications, and/or other displayable data. In some aspects, each apparatus can include one or more speakers, one or more light-emitting devices, and/or one or more microphones. In some aspects, each apparatus can include one or more sensors. In some cases, the one or more sensors can be used for determining a location of the apparatuses, a state of the apparatuses (e.g., a tracking state, an operating state, a temperature, a humidity level, and/or other state), and/or for other purposes.

DETAILED DESCRIPTION

Certain aspects of this disclosure are provided below. Some of these aspects may be applied independently and some of them may be applied in combination as would be apparent to those of skill in the art. In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of aspects of the application. However, it will be apparent that various aspects may be practiced without these specific details. The figures and description are not intended to be restrictive.

As described above, a passive stereo-vision system may correlate unique features within stereoscopically-paired images of a scene and determine respective depths to points in the scene represented by the unique features based on the positions of the unique features in the stereoscopically-paired images and a distance between the cameras that captured the stereoscopically-paired images. For a passive stereo-vision system to determine the depths to the points in the scene, the points should be represented by visually unique features so that the passive stereo-vision system may correlate the features between the stereoscopically-paired images. A passive stereo-vision system may have difficulty determining a distance between the passive stereo-vision system and an object that lacks visually distinct features. For example, the passive stereo-vision system may have difficulty determining distances between the passive stereo-vision system and points of a blank wall. The blank wall may be visually uniform and may thus lack visually unique features. The passive stereo-vision system may be unable to correlate features of the blank wall between stereoscopically-paired images of the blank wall. Because the passive stereo-vision system is unable to correlate features, the passive stereo-vision system may be unable to determine distances between the passive stereo-vision system and the blank wall.

Visual simultaneous localization and mapping (VSLAM or SLAM) is a computational geometry technique used in devices with cameras, such as robots, extended reality (XR) devices (e.g., head-mounted displays (HMDs)), mobile handsets, autonomous vehicles, among others. In VSLAM, a device can construct and update a map of an unknown environment based on images captured by the device's camera. The device can keep track of the device's pose within the environment (e.g., location and/or orientation) as the device updates the map. For example, the device can be activated in a particular room of a building and can move throughout the interior of the building, capturing images. The device can map the environment, and keep track of its location in the environment, based on tracking where different objects in the environment appear in different images.

Degrees of freedom (DoF) refer to the number of basic ways a rigid object can move through three-dimensional (3D) space. In some cases, six different DoF can be tracked. The six degrees of freedom include three translational degrees of freedom corresponding to translational movement along three perpendicular axes. The three axes can be referred to as x, y, and z axes. The six degrees of freedom include three rotational degrees of freedom corresponding to rotational movement around the three axes, which can be referred to as pitch, yaw, and roll. In the present disclosure, the term “pose” may refer to position (e.g., described with regard to the three translational degrees of freedom) and orientation (e.g., as described with regard to the three rotational degrees of freedom). Thus a pose of an object may refer to the position and orientation of the object according to six degrees of freedom.

In the context of systems that track movement through an environment, such as XR systems and/or VSLAM systems, degrees of freedom can refer to which of the six degrees of freedom the system is capable of tracking. 3DoF systems generally track the three rotational DoFpitch, yaw, and roll. A 3DoF headset, for instance, can track the user of the headset turning their head left or right, tilting their head up or down, and/or tilting their head to the left or right. 6DoF systems can track the three translational DoF as well as the three rotational DoF. Thus, a 6DoF system can track the user moving forward, backward, laterally, and/or vertically in addition to tracking the three rotational DoF.

A 6DoF system, using a VSLAM technique, may calculate a change in pose of the 6DoF system based on changes in positions of unique features as captured in successive images of a scene. For the 6DoF system to determine changes in its pose, the unique features should be stationary in the scene and should be visually distinct (e.g., such that the 6DoF system may easily recognize the unique features in the successive images despite the unique features changing position between the subsequent images). Similar to the passive stereo-vision system, a 6DoF system may have difficulty determining its pose when a camera of the 6DoF system is facing an object without distinct features (e.g., a blank wall). For example, the blank wall may be visually uniform and may thus lack visually unique features. The 6DoF system may be unable to identify and/or correlate unique features between the successive images. Because the 6DoF system is unable to correlate unique features between successive images, the 6DoF system may be unable to determine its pose based on images of the blank wall.

Systems, apparatuses, methods (also referred to as processes), and computer-readable media (collectively referred to herein as “systems and techniques”) are described herein for enabling pose and/or distance determinations. The systems and techniques described herein may project a pattern (e.g., a unique pattern) into a scene to enable a passive stereo-vision system to correlate the pattern in stereoscopically-paired images of the scene to determine distances between the passive stereo-vision system and points in the scene. Additionally or alternatively, the systems and techniques may project the pattern into a scene to enable a 6DoF system to track the pattern in images of the scene to determine a pose of the 6DoF system relative to the scene.

The systems and techniques may include a projector. The projector may be independent of a passive stereo-vision system and independent of a 6DoF system. For example, the projector may be separate from the passive stereo-vision system and separate from the 6DoF system. Additionally, the projector may project the pattern independent of the passive stereo-vision system and independent of the 6DoF system. For example, the projector may project the pattern into the scene whether a passive stereo-vision system is in the scene or not and/or independent of whether a passive stereo-vision system is capturing stereoscopically-paired images of the pattern or not. Additionally, the projector may project the pattern into the scene whether a 6DoF system is in the scene or not and/or independent of whether a 6DoF system is capturing successive images of the pattern or not.

A passive stereo-vision system may determine distances between points in a scene and the passive stereo-vision system independent of the projector. For example, the passive stereo-vision system may determine distances between the points in the scene and the passive stereo-vision system independent of whether a projector is projecting a pattern into the scene or not. The projector projecting the pattern into the scene may enable the passive stereo-vision system to determine distances between the passive stereo-vision system and points in the scene onto which the pattern is projected more accurately. For example, the projector may project the pattern onto visually indistinct portions of the scene. The passive stereo-vision system may be better able to correlate features in the stereoscopically-paired images of the visually indistinct portions of the scene with the pattern projected thereonto than the passive stereo-vision system would be if the projector did not project the pattern onto the visually indistinct portions of the scene. Further, because the passive stereo-vision system operates independent from the projector, the passive stereo-vision system may determine distances between the points in the scene and the passive stereo-vision system without a priori information regarding the projector and/or the pattern.

Additionally or alternatively, a 6DoF system may determine a pose of the 6DoF system independent of the projector. For example, the 6DoF system may determine the pose of the 6DoF system independent of whether a projector is projecting a pattern into the scene or not. The projector projecting the pattern into the scene may enable the 6DoF system to determine a pose of the 6DoF system more accurately. For example, the projected may project the pattern onto visually indistinct portions of the scene. The 6DoF system may be better able to correlate features in the successive images of the visually indistinct portions of the scene with the pattern projected thereonto than the 6DoF system would be if the projector did not project the pattern onto the visually indistinct portions of the scene. Further, because the 6DoF system operates independent from the projector, the 6DoF system may determine the pose of the 6DoF system without a priori information regarding the projector and/or the pattern.

Further, the projector may be independently steerable relative to a passive stereo-vision system and independently steerable relative to a 6DoF system. For example, the projector may be steered (e.g., pointed) at a point or surface of the scene independent of where the passive stereo-vision system is steered and independent of where the 6DoF system is steered. For example, the projector may be pointed at a visually indistinct portion of a scene (e.g., a blank wall) to project the pattern onto the visually indistinct portion of the scene independent of where the passive stereo-vision system is pointed and/or independent of where the 6DoF system is pointed.

In some aspects, the projector may be stationary in the scene. For example, the projector may be positioned within the scene to remain pointing to a visually indistinct portion of the scene (e.g., a blank wall) to project a pattern onto the visually indistinct portion of the scene. A passive stereo-vision system may move within the scene and determine distances between the passive stereo-vision system and the visually indistinct portion of the scene based on stereoscopically-paired images of the pattern as projected onto the otherwise visually indistinct portion of the scene. Additionally or alternatively, a 6DoF system may move within the scene and determine poses of the 6DoF system based on successive images of the pattern as projected onto the otherwise visually indistinct portion of the scene.

In other aspects, the projector may move within the scene. For example, the projector may move within the scene and may point to one or more visually indistinct portions of the scene (e.g., one or more blank walls) to project a pattern onto the one or more visually indistinct portions of the scene as the projector moves through the scene. A passive stereo-vision system may move within the scene and determine distances between the passive stereo-vision system and the one or more visually indistinct portions of the scene based on stereoscopically-paired images of the pattern as projected onto the otherwise visually indistinct portions of the scene. Additionally or alternatively, a 6DoF system may move within the scene and determine poses of the 6DoF system based on successive images of the pattern as projected onto the otherwise visually indistinct portions of the scene.

Various aspects of the application will be described with respect to the figures below. For example,FIG.1AthroughFIG.4(and the corresponding text) provide examples of 6DoF systems and techniques.FIG.5andFIG.6(and the corresponding text) provide examples of passive stereo-vision techniques.FIG.7throughFIG.13provide examples of systems and techniques for enabling pose and/or distance determinations, according to various aspects of the present disclosure.

In particular,FIG.1Ais a perspective diagram illustrating head-mounted display (HMD)100, according to various aspects of the present disclosure. HMD100may be, for example, an augmented reality (AR) headset, a virtual reality (VR) headset, a mixed reality (MR) headset, an extended reality (XR) headset, or some combination thereof. HMD100may be an example of, or implement, XR system300ofFIG.3, SLAM system400ofFIG.4, or a combination thereof. HMD100includes a first camera102and a second camera104along a front portion of HMD100. First camera102and second camera104may be two of the one or more camera(s)404of SLAM system400ofFIG.4. In some examples, HMD100may only have a single camera. In some examples, HMD100may include one or more additional cameras in addition to first camera102and second camera104. In some aspects, HMD100may include one or more additional sensors, such as, for example, inertial measurement units (IMUs).

FIG.1Bis a perspective diagram illustrating the head-mounted display (HMD)100ofFIG.1Abeing worn by a user106, according to various aspects of the present disclosure. User106wears HMD100on the head of on user106over the eyes of user106. HMD100may capture images with first camera102and second camera104. In some examples, HMD100may display one or more display images toward the eyes of user106. The display images may be based on the images captured by first camera102and/or second camera104. The display images may provide a stereoscopic view of the environment, in some cases with information overlaid and/or with other modifications. For example, HMD100may display a first display image to the left eye of user106, the first display image based on an image captured by first camera102. HMD100may display a second display image to the right eye of user106, the second display image based on an image captured by second camera104. For instance, HMD100may provide overlaid information in the display images overlaid over the images captured by first camera102and/or second camera104.

HMD100may determine a pose of HMD100and is therefore provided as an example of a 6DoF system. HMD100may determine a pose of HMD100using visual simultaneous localization and mapping (VSLAM or SLAM) techniques (e.g., based on successive images captured by first camera102and/or second camera104).

FIG.2is a diagram illustrating an example of an extended reality (XR) system200, according to aspects of the disclosure. XR system200may be, for example, an augmented reality (AR) headset, a virtual reality (VR) headset, a mixed reality (MR) headset, an extended reality (XR) headset, or some combination thereof. XR system200may be an example of, or implement, XR system300ofFIG.3, SLAM system400ofFIG.4, or a combination thereof.

As shown, XR system200includes an XR device202, a companion device204, and a communication link206between XR device202and companion device204. In some cases, XR device202may generally implement display, image-capture, and/or view-tracking aspects of extended reality, including virtual reality (VR), augmented reality (AR), mixed reality (MR), etc. In some cases, companion device204may generally implement computing aspects of extended reality. For example, XR device202may capture images of an environment of a user208and provide the images to companion device204(e.g., via communication link206). Companion device204may render virtual content (e.g., related to the captured images of the environment) and provide the virtual content to XR device202(e.g., via communication link206). XR device202may display the virtual content to a user208(e.g., within a field of view210of user208).

Generally, XR device202may display virtual content to be viewed by a user208in field of view210. In some examples, XR device202may include a transparent surface (e.g., optical glass) such that virtual objects may be displayed on (e.g., by being projected onto) the transparent surface to overlay virtual content on real-word objects viewed through the transparent surface (e.g., in a see-through configuration). In some cases, XR device202may include a camera and may display both real-world objects (e.g., as frames or images captured by the camera) and virtual objects overlaid on the displayed real-world objects (e.g., in a pass-through configuration). In various examples, XR device202may include aspects of a virtual reality headset, smart glasses, a live feed video camera, a GPU, one or more sensors (e.g., such as one or more inertial measurement units (IMUs), image sensors, microphones, etc.), one or more output devices (e.g., such as speakers, display, smart glass, etc.), etc.

Companion device204may render the virtual content to be displayed by companion device204. In some examples, companion device204may be, or may include, a smartphone, laptop, tablet computer, personal computer, gaming system, a server computer or server device (e.g., an edge or cloud-based server, a personal computer acting as a server device, or a mobile device acting as a server device), any other computing device and/or a combination thereof.

Communication link206may be a wireless connection according to any suitable wireless protocol, such as, for example, Institute of Electrical and Electronics Engineers (IEEE) 802.11 (Wi-Fi), IEEE 802.15, or Bluetooth™. In some cases, communication link206may be a direct wireless connection between XR device202and companion device204. In other cases, communication link206may be through one or more intermediary devices, such as, for example, routers or switches and/or across a network.

Similar to HMD100, XR system200(or companion device204of XR system200) may determine a pose of user208according to 6 degrees of freedom. Thus, XR system200is provided as an example of a 6DoF system. XR system200(or companion device204of XR system200) may determine a pose of user208using visual simultaneous localization and mapping (VSLAM or SLAM) techniques (e.g., based on successive images captured by one or more cameras of XR device202).

FIG.3is a diagram illustrating an architecture of an example extended reality (XR) system300, in accordance with some aspects of the disclosure. XR system300may execute XR applications and implement XR operations. XR system300may be an example of, or may be implemented in, HMD100ofFIG.1AandFIG.1Band/or XR system200ofFIG.2.

In this illustrative example, XR system300includes one or more image sensors302, an accelerometer304, a gyroscope306, storage308, an input device310, a display312, Compute components314, an XR engine324, an image processing engine326, a rendering engine328, and a communications engine330. It should be noted that the components302-330shown inFIG.3are non-limiting examples provided for illustrative and explanation purposes, and other examples may include more, fewer, or different components than those shown inFIG.3. For example, in some cases, XR system300may include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radars, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, audio sensors, etc.), one or more display devices, one more other processing engines, one or more other hardware components, and/or one or more other software and/or hardware components that are not shown inFIG.3. While various components of XR system300, such as image sensor302, may be referenced in the singular form herein, it should be understood that XR system300may include multiple of any component discussed herein (e.g., multiple image sensors302).

Display312may be, or may include, a glass, a screen, a lens, a projector, and/or other display mechanism that allows a user to see the real-world environment and also allows XR content to be overlaid, overlapped, blended with, or otherwise displayed thereon.

XR system300may include, or may be in communication with, (wired or wirelessly) an input device310. Input device310may include any suitable input device, such as a touchscreen, a pen or other pointer device, a keyboard, a mouse a button or key, a microphone for receiving voice commands, a gesture input device for receiving gesture commands, a video game controller, a steering wheel, a joystick, a set of buttons, a trackball, a remote control, any other input device discussed herein, or any combination thereof. In some cases, image sensor302may capture images that may be processed for interpreting gesture commands.

XR system300may also communicate with one or more other electronic devices (wired or wirelessly). For example, communications engine330may be configured to manage connections and communicate with one or more electronic devices. In some cases, communications engine330may correspond to communication interface1326ofFIG.13.

In some implementations, image sensors302, accelerometer304, gyroscope306, storage308, display312, compute components314, XR engine324, image processing engine326, and rendering engine328may be part of the same computing device (such as HMD100ofFIG.1AandFIG.1B). For example, in some cases, image sensors302, accelerometer304, gyroscope306, storage308, display312, compute components314, XR engine324, image processing engine326, and rendering engine328may be integrated into an HMD, extended reality glasses, smartphone, laptop, tablet computer, gaming system, and/or any other computing device.

In other implementations, image sensors302, accelerometer304, gyroscope306, storage308, display312, compute components314, XR engine324, image processing engine326, and rendering engine328may be part of two or more separate computing devices. For instance, in some cases, some of the components302-330may be part of, or implemented by, one computing device and the remaining components may be part of, or implemented by, one or more other computing devices. For example, such as in a split perception XR system, XR system300may include a first device (such as XR device202ofFIG.2) including display312, image sensor302, accelerometer304, gyroscope306, and/or one or more compute components314. XR system300may also include a second device (such as companion device204ofFIG.2) including additional compute components314(e.g., implementing XR engine324, image processing engine326, rendering engine328, and/or communications engine330). In such an example, the second device may generate virtual content based on information or data (e.g., images, sensor data such as measurements from accelerometer304and gyroscope306) and may provide the virtual content to the first device for display at the first device. The second device may be, or may include, a smartphone, laptop, tablet computer, personal computer, gaming system, a server computer or server device (e.g., an edge or cloud-based server, a personal computer acting as a server device, or a mobile device acting as a server device), any other computing device and/or a combination thereof.

Storage308may be any storage device(s) for storing data. Moreover, storage308may store data from any of the components of XR system300. For example, storage308may store data from image sensor302(e.g., image or video data), data from accelerometer304(e.g., measurements), data from gyroscope306(e.g., measurements), data from compute components314(e.g., processing parameters, preferences, virtual content, rendering content, scene maps, tracking and localization data, object detection data, privacy data, XR application data, face recognition data, occlusion data, etc.), data from XR engine324, data from image processing engine326, and/or data from rendering engine328(e.g., output frames). In some examples, storage308may include a buffer for storing frames for processing by compute components314.

Compute components314may be, or may include, a central processing unit (CPU)316, a graphics processing unit (GPU)318, a digital signal processor (DSP)320, an image signal processor (ISP)322, and/or other processor (e.g., a neural processing unit (NPU) implementing one or more trained neural networks). Compute components314may perform various operations such as image enhancement, computer vision, graphics rendering, extended reality operations (e.g., tracking, localization, pose estimation, mapping, content anchoring, content rendering, predicting, etc.), image and/or video processing, sensor processing, recognition (e.g., text recognition, facial recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), trained machine-learning operations, filtering, and/or any of the various operations described herein. In some examples, compute components314may implement (e.g., control, operate, etc.) XR engine324, image processing engine326, and rendering engine328. In other examples, compute components314may also implement one or more other processing engines.

Image sensor302may include any image and/or video sensors or capturing devices. In some examples, image sensor302may be part of a multiple-camera assembly, such as a dual-camera assembly. Image sensor302may capture image and/or video content (e.g., raw image and/or video data), which may then be processed by compute components314, XR engine324, image processing engine326, and/or rendering engine328as described herein.

In some examples, image sensor302may capture image data and may generate images (also referred to as frames) based on the image data and/or may provide the image data or frames to XR engine324, image processing engine326, and/or rendering engine328for processing. An image or frame may include a video frame of a video sequence or a still image. An image or frame may include a pixel array representing a scene. For example, an image may be a red-green-blue (RGB) image having red, green, and blue color components per pixel; a luma, chroma-red, chroma-blue (YCbCr) image having a luma component and two chroma (color) components (chroma-red and chroma-blue) per pixel; or any other suitable type of color or monochrome image.

In some cases, image sensor302(and/or other camera of XR system300) may be configured to also capture depth information. For example, in some implementations, image sensor302(and/or other camera) may include an RGB-depth (RGB-D) camera. In some cases, XR system300may include one or more depth sensors (not shown) that are separate from image sensor302(and/or other camera) and that may capture depth information. For instance, such a depth sensor may obtain depth information independently from image sensor302. In some examples, a depth sensor may be physically installed in the same general location or position as image sensor302but may operate at a different frequency or frame rate from image sensor302. In some examples, a depth sensor may take the form of a light source that may project a structured or textured light pattern, which may include one or more narrow bands of light, onto one or more objects in a scene. Depth information may then be obtained by exploiting geometrical distortions of the projected pattern caused by the surface shape of the object. In one example, depth information may be obtained from stereo sensors such as a combination of an infra-red structured light projector and an infra-red camera registered to a camera (e.g., an RGB camera).

XR system300may also include other sensors in its one or more sensors. The one or more sensors may include one or more accelerometers (e.g., accelerometer304), one or more gyroscopes (e.g., gyroscope306), and/or other sensors. The one or more sensors may provide velocity, orientation, and/or other position-related information to compute components314. For example, accelerometer304may detect acceleration by XR system300and may generate acceleration measurements based on the detected acceleration. In some cases, accelerometer304may provide one or more translational vectors (e.g., up/down, left/right, forward/back) that may be used for determining a position or pose of XR system300. Gyroscope306may detect and measure the orientation and angular velocity of XR system300. For example, gyroscope306may be used to measure the pitch, roll, and yaw of XR system300. In some cases, gyroscope306may provide one or more rotational vectors (e.g., pitch, yaw, roll). In some examples, image sensor302and/or XR engine324may use measurements obtained by accelerometer304(e.g., one or more translational vectors) and/or gyroscope306(e.g., one or more rotational vectors) to calculate the pose of XR system300. As previously noted, in other examples, XR system300may also include other sensors, such as an inertial measurement unit (IMU), a magnetometer, a gaze and/or eye tracking sensor, a machine vision sensor, a smart scene sensor, a speech recognition sensor, an impact sensor, a shock sensor, a position sensor, a tilt sensor, etc.

As noted above, in some cases, the one or more sensors may include at least one IMU. An IMU is an electronic device that measures the specific force, angular rate, and/or the orientation of XR system300, using a combination of one or more accelerometers, one or more gyroscopes, and/or one or more magnetometers. In some examples, the one or more sensors may output measured information associated with the capture of an image captured by image sensor302(and/or other camera of XR system300) and/or depth information obtained using one or more depth sensors of XR system300.

The output of one or more sensors (e.g., accelerometer304, gyroscope306, one or more IMUs, and/or other sensors) can be used by XR engine324to determine a pose of XR system300(also referred to as the head pose) and/or the pose of image sensor302(or other camera of XR system300). In some cases, the pose of XR system300and the pose of image sensor302(or other camera) can be the same. The pose of image sensor302refers to the position and orientation of image sensor302relative to a frame of reference (e.g., with respect to a field of view210ofFIG.2). In some implementations, the camera pose can be determined for 6-Degrees of Freedom (6DoF), which refers to three translational components (e.g., which can be given by X (horizontal), Y (vertical), and Z (depth) coordinates relative to a frame of reference, such as the image plane) and three angular components (e.g. roll, pitch, and yaw relative to the same frame of reference). In some implementations, the camera pose can be determined for 3-Degrees of Freedom (3DoF), which refers to the three angular components (e.g. roll, pitch, and yaw).

In some cases, a device tracker (not shown) can use the measurements from the one or more sensors and image data from image sensor302to track a pose (e.g., a 6DoF pose) of XR system300. For example, the device tracker can fuse visual data (e.g., using a visual tracking solution) from the image data with inertial data from the measurements to determine a position and motion of XR system300relative to the physical world (e.g., the scene) and a map of the physical world. As described below, in some examples, when tracking the pose of XR system300, the device tracker can generate a three-dimensional (3D) map of the scene (e.g., the real world) and/or generate updates for a 3D map of the scene. The 3D map updates can include, for example and without limitation, new or updated features and/or feature or landmark points associated with the scene and/or the 3D map of the scene, localization updates identifying or updating a position of XR system200within the scene and the 3D map of the scene, etc. The 3D map can provide a digital representation of a scene in the real/physical world. In some examples, the 3D map can anchor position-based objects and/or content to real-world coordinates and/or objects. XR system200can use a mapped scene (e.g., a scene in the physical world represented by, and/or associated with, a 3D map) to merge the physical and virtual worlds and/or merge virtual content or objects with the physical environment.

In some aspects, the pose of image sensor302and/or XR system300as a whole can be determined and/or tracked by compute components314using a visual tracking solution based on images captured by image sensor302(and/or other camera of XR system300). For instance, in some examples, compute components314can perform tracking using computer vision-based tracking, model-based tracking, and/or simultaneous localization and mapping (SLAM) techniques. For instance, compute components314can perform SLAM or can be in communication (wired or wireless) with a SLAM system (not shown). SLAM refers to a class of techniques where a map of an environment (e.g., a map of an environment being modeled by XR system300) is created while simultaneously tracking the pose of a camera (e.g., image sensor302) and/or XR system300relative to that map. The map can be referred to as a SLAM map and can be three-dimensional (3D). The SLAM techniques can be performed using color or grayscale image data captured by image sensor302(and/or other camera of XR system300) and can be used to generate estimates of 6DoF pose measurements of image sensor302and/or XR system300. Such a SLAM technique configured to perform 6DoF tracking can be referred to as 6DoF SLAM. In some cases, the output of the one or more sensors (e.g., accelerometer304, gyroscope306, one or more IMUs, and/or other sensors) can be used to estimate, correct, and/or otherwise adjust the estimated pose.

In some cases, the 6DoF SLAM (e.g., 6DoF tracking) can associate features observed from certain input images from the image sensor302(and/or other camera) to the SLAM map. For example, 6DoF SLAM can use feature point associations from an input image to determine the pose (position and orientation) of the image sensor302and/or XR system300for the input image. 6DoF mapping can also be performed to update the SLAM map. In some cases, the SLAM map maintained using the 6DoF SLAM can contain 3D feature points triangulated from two or more images. For example, key frames can be selected from input images or a video stream to represent an observed scene. For every key frame, a respective 6DoF camera pose associated with the image can be determined. The pose of the image sensor302and/or the XR system300can be determined by projecting features from the 3D SLAM map into an image or video frame and updating the camera pose from verified 2D-3D correspondences.

In one illustrative example, the compute components314can extract feature points from certain input images (e.g., every input image, a subset of the input images, etc.) or from each key frame. A feature point (also referred to as a registration point) as used herein is a distinctive or identifiable part of an image, such as a part of a hand, an edge of a table, among others. Features extracted from a captured image can represent distinct feature points along three-dimensional space (e.g., coordinates on X, Y, and Z-axes), and every feature point can have an associated feature location. The feature points in key frames either match (are the same or correspond to) or fail to match the feature points of previously-captured input images or key frames. Feature detection can be used to detect the feature points. Feature detection can include an image processing operation used to examine one or more pixels of an image to determine whether a feature exists at a particular pixel. Feature detection can be used to process an entire captured image or certain portions of an image. For each image or key frame, once features have been detected, a local image patch around the feature can be extracted. Features may be extracted using any suitable technique, such as Scale Invariant Feature Transform (SIFT) (which localizes features and generates their descriptions), Learned Invariant Feature Transform (LIFT), Speed Up Robust Features (SURF), Gradient Location-Orientation histogram (GLOH), Oriented Fast and Rotated Brief (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retina Keypoint (FREAK), KAZE, Accelerated KAZE (AKAZE), Normalized Cross Correlation (NCC), descriptor matching, another suitable technique, or a combination thereof.

As one illustrative example, the compute components314can extract feature points corresponding to a mobile device, or the like. In some cases, feature points corresponding to the mobile device can be tracked to determine a pose of the mobile device. As described in more detail below, the pose of the mobile device can be used to determine a location for projection of AR media content that can enhance media content displayed on a display of the mobile device.

In some cases, the XR system300can also track the hand and/or fingers of the user to allow the user to interact with and/or control virtual content in a virtual environment. For example, the XR system300can track a pose and/or movement of the hand and/or fingertips of the user to identify or translate user interactions with the virtual environment. The user interactions can include, for example and without limitation, moving an item of virtual content, resizing the item of virtual content, selecting an input interface element in a virtual user interface (e.g., a virtual representation of a mobile phone, a virtual keyboard, and/or other virtual interface), providing an input through a virtual user interface, etc.

FIG.4is a block diagram illustrating an architecture of a simultaneous localization and mapping (SLAM) system400. In some examples, the SLAM system400can be, or can include, an extended reality (XR) system, such as the XR system200ofFIG.2. In some examples, the SLAM system400can be a wireless communication device, a mobile device or handset (e.g., a mobile telephone or so-called “smart phone” or other mobile device), a wearable device, a personal computer, a laptop computer, a server computer, a portable video game console, a portable media player, a camera device, a manned or unmanned ground vehicle, a manned or unmanned aerial vehicle, a manned or unmanned aquatic vehicle, a manned or unmanned underwater vehicle, a manned or unmanned vehicle, an autonomous vehicle, a vehicle, a computing system of a vehicle, a robot, another device, or any combination thereof.

The SLAM system400ofFIG.4includes, or is coupled to, each of one or more sensor(s)402. The sensor(s)402can include one or more camera(s)404. Each of the camera(s)404may include an image capture device, an image processing device, an image capture and processing system, another type of camera, or a combination thereof. Each of the camera(s)404may be responsive to light from a particular spectrum of light. The spectrum of light may be a subset of the electromagnetic (EM) spectrum. For example, each of the camera(s)404may be a visible light (VL) camera responsive to a VL spectrum, an infrared (IR) camera responsive to an IR spectrum, an ultraviolet (UV) camera responsive to a UV spectrum, a camera responsive to light from another spectrum of light from another portion of the electromagnetic spectrum, or some combination thereof.

The sensor(s)402can include one or more other types of sensors other than camera(s)404, such as one or more of each of: accelerometers, gyroscopes, magnetometers, inertial measurement units (IMUs), altimeters, barometers, thermometers, radio detection and ranging (RADAR) sensors, light detection and ranging (LIDAR) sensors, sound navigation and ranging (SONAR) sensors, sound detection and ranging (SODAR) sensors, global navigation satellite system (GNSS) receivers, global positioning system (GPS) receivers, BeiDou navigation satellite system (BDS) receivers, Galileo receivers, Globalnaya Navigazionnaya Sputnikovaya Sistema (GLONASS) receivers, Navigation Indian Constellation (NavIC) receivers, Quasi-Zenith Satellite System (QZSS) receivers, Wi-Fi positioning system (WPS) receivers, cellular network positioning system receivers, Bluetooth® beacon positioning receivers, short-range wireless beacon positioning receivers, personal area network (PAN) positioning receivers, wide area network (WAN) positioning receivers, wireless local area network (WLAN) positioning receivers, other types of positioning receivers, other types of sensors discussed herein, or combinations thereof. In some examples, the sensor(s)402can include any combination of sensors of the XR system200ofFIG.2.

Upon receipt of the sensor data426from the sensor(s)402, the VIO tracker406performs feature detection, extraction, and/or tracking using a feature tracking engine408of the VIO tracker406. For instance, where the sensor data426includes one or more images captured by the camera(s)404of the SLAM system400, the VIO tracker406can identify, detect, and/or extract features in each image. Features may include visually distinctive points in an image, such as portions of the image depicting edges and/or corners. The VIO tracker406can receive sensor data426periodically and/or continually from the sensor(s)402, for instance by continuing to receive more images from the camera(s)404as the camera(s)404capture a video, where the images are video frames of the video. The VIO tracker406can generate descriptors for the features. Feature descriptors can be generated at least in part by generating a description of the feature as depicted in a local image patch extracted around the feature. In some examples, a feature descriptor can describe a feature as a collection of one or more feature vectors. The VIO tracker406, in some cases with the mapping engine412and/or the relocalization engine422, can associate the plurality of features with a map of the environment based on such feature descriptors. The feature tracking engine408of the VIO tracker406can perform feature tracking by recognizing features in each image that the VIO tracker406already previously recognized in one or more previous images, in some cases based on identifying features with matching feature descriptors in different images. The feature tracking engine408can track changes in one or more positions at which the feature is depicted in each of the different images. For example, the feature extraction engine can detect a particular corner of a room depicted in a left side of a first image captured by a first camera of the camera(s)404. The feature extraction engine can detect the same feature (e.g., the same particular corner of the same room) depicted in a right side of a second image captured by the first camera. The feature tracking engine408can recognize that the features detected in the first image and the second image are two depictions of the same feature (e.g., the same particular corner of the same room), and that the feature appears in two different positions in the two images. The VIO tracker406can determine, based on the same feature appearing on the left side of the first image and on the right side of the second image that the first camera has moved, for example if the feature (e.g., the particular corner of the room) depicts a static portion of the environment.

The VIO tracker406can include a sensor integration engine410. The sensor integration engine410can use sensor data from other types of sensor(s)402(other than the camera(s)404) to determine information that can be used by the feature tracking engine408when performing the feature tracking. For example, the sensor integration engine410can receive IMU data (e.g., which can be included as part of the sensor data426) from an IMU of the sensor(s)402. The sensor integration engine410can determine, based on the IMU data in the sensor data426, that the SLAM system400has rotated 15 degrees in a clockwise direction from acquisition or capture of a first image and capture to acquisition or capture of the second image by a first camera of the camera(s)404. Based on this determination, the sensor integration engine410can identify that a feature depicted at a first position in the first image is expected to appear at a second position in the second image, and that the second position is expected to be located to the left of the first position by a predetermined distance (e.g., a predetermined number of pixels, inches, centimeters, millimeters, or another distance metric). The feature tracking engine408can take this expectation into consideration in tracking features between the first image and the second image.

Based on the feature tracking by the feature tracking engine408and/or the sensor integration by the sensor integration engine410, the VIO tracker406can determine a 3D feature positions428of a particular feature. The 3D feature positions428can include one or more 3D feature positions and can also be referred to as 3D feature points. The 3D feature positions428can be a set of coordinates along three different axes that are perpendicular to one another, such as an X coordinate along an X axis (e.g., in a horizontal direction), a Y coordinate along a Y axis (e.g., in a vertical direction) that is perpendicular to the X axis, and a Z coordinate along a Z axis (e.g., in a depth direction) that is perpendicular to both the X axis and the Y axis. The VIO tracker406can also determine one or more keyframes430(referred to hereinafter as keyframes430) corresponding to the particular feature. A keyframe (from one or more keyframes430) corresponding to a particular feature may be an image in which the particular feature is clearly depicted. In some examples, a keyframe (from the one or more keyframes430) corresponding to a particular feature may be an image in which the particular feature is clearly depicted. In some examples, a keyframe corresponding to a particular feature may be an image that reduces uncertainty in the 3D feature positions428of the particular feature when considered by the feature tracking engine408and/or the sensor integration engine410for determination of the 3D feature positions428. In some examples, a keyframe corresponding to a particular feature also includes data associated with the pose436of the SLAM system400and/or the camera(s)404during capture of the keyframe. In some examples, the VIO tracker406can send 3D feature positions428and/or keyframes430corresponding to one or more features to the mapping engine412. In some examples, the VIO tracker406can receive map slices432from the mapping engine412. The VIO tracker406can feature information within the map slices432for feature tracking using the feature tracking engine408.

Based on the feature tracking by the feature tracking engine408and/or the sensor integration by the sensor integration engine410, the VIO tracker406can determine a pose436of the SLAM system400and/or of the camera(s)404during capture of each of the images in the sensor data426. The pose436can include a location of the SLAM system400and/or of the camera(s)404in 3D space, such as a set of coordinates along three different axes that are perpendicular to one another (e.g., an X coordinate, a Y coordinate, and a Z coordinate). The pose436can include an orientation of the SLAM system400and/or of the camera(s)404in 3D space, such as pitch, roll, yaw, or some combination thereof. In some examples, the VIO tracker406can send the pose436to the relocalization engine422. In some examples, the VIO tracker406can receive the pose436from the relocalization engine422.

The SLAM system400also includes a mapping engine412. The mapping engine412generates a 3D map of the environment based on the 3D feature positions428and/or the keyframes430received from the VIO tracker406. The mapping engine412can include a map densification engine414, a keyframe remover416, a bundle adjuster418, and/or a loop closure detector420. The map densification engine414can perform map densification, in some examples, increase the quantity and/or density of 3D coordinates describing the map geometry. The keyframe remover416can remove keyframes, and/or in some cases add keyframes. In some examples, the keyframe remover416can remove keyframes430corresponding to a region of the map that is to be updated and/or whose corresponding confidence values are low. The bundle adjuster418can, in some examples, refine the 3D coordinates describing the scene geometry, parameters of relative motion, and/or optical characteristics of the image sensor used to generate the frames, according to an optimality criterion involving the corresponding image projections of all points. The loop closure detector420can recognize when the SLAM system400has returned to a previously mapped region and can use such information to update a map slice and/or reduce the uncertainty in certain 3D feature points or other points in the map geometry. The mapping engine412can output map slices432to the VIO tracker406. The map slices432can represent 3D portions or subsets of the map. The map slices432can include map slices432that represent new, previously-unmapped areas of the map. The map slices432can include map slices432that represent updates (or modifications or revisions) to previously-mapped areas of the map. The mapping engine412can output map information434to the relocalization engine422. The map information434can include at least a portion of the map generated by the mapping engine412. The map information434can include one or more 3D points making up the geometry of the map, such as one or more 3D feature positions428. The map information434can include one or more keyframes430corresponding to certain features and certain 3D feature positions428.

The SLAM system400also includes a relocalization engine422. The relocalization engine422can perform relocalization, for instance when the VIO tracker406fail to recognize more than a threshold number of features in an image, and/or the VIO tracker406loses track of the pose436of the SLAM system400within the map generated by the mapping engine412. The relocalization engine422can perform relocalization by performing extraction and matching using an extraction and matching engine424. For instance, the extraction and matching engine424can by extract features from an image captured by the camera(s)404of the SLAM system400while the SLAM system400is at a current pose436and can match the extracted features to features depicted in different keyframes430, identified by 3D feature positions428, and/or identified in the map information434. By matching these extracted features to the previously-identified features, the relocalization engine422can identify that the pose436of the SLAM system400is a pose436at which the previously-identified features are visible to the camera(s)404of the SLAM system400and is therefore similar to one or more previous poses436at which the previously-identified features were visible to the camera(s)404. In some cases, the relocalization engine422can perform relocalization based on wide baseline mapping, or a distance between a current camera position and camera position at which feature was originally captured. The relocalization engine422can receive information for the pose436from the VIO tracker406, for instance regarding one or more recent poses of the SLAM system400and/or camera(s)404, which the relocalization engine422can base its relocalization determination on. Once the relocalization engine422relocates the SLAM system400and/or camera(s)404and thus determines the pose436, the relocalization engine422can output the pose436to the VIO tracker406.

In some examples, the VIO tracker406can modify the image in the sensor data426before performing feature detection, extraction, and/or tracking on the modified image. For example, the VIO tracker406can rescale and/or resample the image. In some examples, rescaling and/or resampling the image can include downscaling, downsampling, subscaling, and/or subsampling the image one or more times. In some examples, the VIO tracker406modifying the image can include converting the image from color to greyscale, or from color to black and white, for instance by desaturating color in the image, stripping out certain color channel(s), decreasing color depth in the image, replacing colors in the image, or a combination thereof. In some examples, the VIO tracker406modifying the image can include the VIO tracker406masking certain regions of the image. Dynamic objects can include objects that can have a changed appearance between one image and another. For example, dynamic objects can be objects that move within the environment, such as people, vehicles, or animals. A dynamic objects can be an object that have a changing appearance at different times, such as a display screen that may display different things at different times. A dynamic object can be an object that has a changing appearance based on the pose of the camera(s)404, such as a reflective surface, a prism, or a specular surface that reflects, refracts, and/or scatters light in different ways depending on the position of the camera(s)404relative to the dynamic object. The VIO tracker406can detect the dynamic objects using facial detection, facial recognition, facial tracking, object detection, object recognition, object tracking, or a combination thereof. The VIO tracker406can detect the dynamic objects using one or more artificial intelligence algorithms, one or more trained machine learning models, one or more trained neural networks, or a combination thereof. The VIO tracker406can mask one or more dynamic objects in the image by overlaying a mask over an area of the image that includes depiction(s) of the one or more dynamic objects. The mask can be an opaque color, such as black. The area can be a bounding box having a rectangular or other polygonal shape. The area can be determined on a pixel-by-pixel basis.

As noted previously, a passive stereo-vision system may capture stereoscopically-paired images of a scene using two cameras that are a predetermined distance apart. For example, in a passive stereo-vision system, the two cameras may be positioned with different perspectives of the same scene, where each camera may capture an image of the scene at substantially the same time. A system may determine depth information for the scene (e.g., a depth map of scene) based on the images captured by the two cameras, which can be referred to as stereoscopically-paired images. The depth information may include depths of objects in the scene (e.g., distances between the cameras (or a point relative to the cameras) and the objects).

For example, if a scene captured in stereoscopically-paired images includes an object, a pixel in the image from one camera, which represents a point on the object, may have a corresponding pixel in the image from the second camera that represents the same point on the same object. However, because the images are taken by cameras with different perspectives of the same scene, a position of the pixel corresponding to the point on the object in the first image may be different from a position of the pixel corresponding to the same point on the object in the second image. By matching corresponding pixels in the two images and calculating the distance between these corresponding pixels, it is possible to determine a relative depth of the point of objects within the scene. For example, in some cases, the nearer an object is to the cameras, the greater the distance between corresponding pixels within the images.

FIG.5illustrates two images, image506and image508(also denoted inFIG.5as image ILand image IR), of a single scene502captured from different camera positions, according to various aspects of the present disclosure. The different camera positions are marked as left and right “origin” points, OLand OR, which are offset by a distance Tx. Because of the offset Tx, the same point P of object504appears at different pixel locations pLand pRwithin the two images506(IL) and508(IR). As can be seen, the x-axis coordinate xRin image508(IR), corresponding to point PRin image508(IR), is offset along epi-polar line510by disparity d from a coordinate xL, where the coordinate xL, corresponds to the position of the point P in the image506(IL). This disparity in pixel locations (also referred to as discrepancy) may be used to determine an approximate distance from the cameras to the point P on object504in scene502. By knowing the stereo camera geometry and applying such an analysis to each point in the images, a depth map of the scene may be generated.

In order to determine the disparity d, a system may determine that the pixel location pRin the image508(IR) corresponds to the pixel location pLin the image506(IL), for example, by comparing a window of pixels including pixels at, and around, the pixel location pLto a number of windows of pixels in image508(IR). An example of such a window-based comparison technique is described with respect toFIG.6. For example, a passive stereo-vision system may determine epi-polar line510in the image508(IR). Epi-polar line510may be a defined by a ray projected from origin point OLto the point P as viewed in in the image506(IR). The passive stereo-vision system may compare the window of pixels including pixels at, and around, the pixel location pLto similarly-sized windows along epi-polar line510.

FIG.6illustrates two images, including image602(which may be a “right image” or a “reference image”) and image604(which may be a “left image”), and an associated cost function614, according to various aspects of the present disclosure. To compare windows between image602and image604, a window606of pixels from the image602may be selected. Window606of pixels from image602may be compared to one or more windows of pixels from image604. In some cases, window606may be compared to similarly-sized windows (e.g., all similarly-sized windows) along an epi-polar line612of image604.

The cost function614shown inFIG.6is representative of a similarity between window606and similarly-sized windows along epi-polar line612of image604as a function of disparity. The similarity between windows may be based on similarities between respective red, green, blue, and/or intensity (or brightness or luminance) values of pixels included in the respective windows. The lower the value of cost function614for a particular disparity, the higher the degree of similarity is between window606and a window of image602at the corresponding disparity. For example, cost function614includes two minima, c1and c2. The minima c1corresponds to a disparity d1, which corresponds to a comparison between window606and candidate window608of image604. The minima c2corresponds to a disparity d2which corresponds to a comparison between window606and candidate window610of image604.

A disparity map may be a two-dimensional map of disparities. The two-dimensional map may relate to an image (e.g., image506ofFIG.5). For instance, a two-dimensional disparity map may include a resolution that is the same (or substantially the same in some cases) as a corresponding image, with a respective disparity value for each pixel of the image. In one illustrative example, a disparity map may be generated by determining a respective disparity for each pixel of a number of pixels (e.g., all, or most, of the pixels) of an image (e.g., by scanning windows across epi-polar lines of a stereoscopically-paired image and determining a disparity for each of the number of pixels). Each value of the disparity map may represent a disparity (e.g., disparity d ofFIG.5). A depth map may be derived from a disparity map based on the three-dimensional geometry of a scene (e.g., scene502ofFIG.5) including a distance between the cameras which captured the images (e.g., the distance TXofFIG.5).

A depth map may be a representation of three-dimensional information (e.g., depth information). For example, a depth map may be a two-dimensional map of values (e.g., pixel values) representing depths. The values of the depth map may correspond to pixels in a corresponding image (e.g., image506ofFIG.5). For instance, the depth map may have a resolution that is the same or substantially the same as the corresponding image, with each depth value of the depth map representing a depth, or distance, between an origin point (e.g., origin point OLofFIG.5) and points (e.g., point P ofFIG.5). In some cases, each pixel in the depth map may have one depth value. Because a depth map is based on a disparity map, in some cases, each pixel of a disparity may have one disparity.

A system or device including two camera a known distance apart (e.g., TXofFIG.5) may implement passive stereo-vision techniques and be a passive stereo-vision system. For example, HMD100ofFIG.1AandFIG.1Bincludes first camera102and user106and may be an example of a passive stereo-vision system. Other examples of passive stereo-vision systems include, robots, vehicles, and cameras.

FIG.7is a diagram of an example environment700in which systems and techniques may enable pose and/or distance determinations, according to various aspects of the present disclosure. For example, a projector702may project a pattern704into a scene708(including onto a surface706of scene708). A 6DoF system710may operate in environment700and may capture successive images of scene708. 6DoF system710may determine a pose of 6DoF system710based on the successive images of scene708. 6DoF system710may be enabled to determine a pose of 6DoF system710based on pattern704being projected into scene708. For example, 6DoF system710may be able to more accurately and/or more quickly determine the pose of 6DoF system710based on successive images of scene708including pattern704than if the successive images did not include pattern704. Additionally or alternatively, a passive stereo-vision system712may operate in environment700and may capture stereoscopically-paired images of scene708. Passive stereo-vision system712may determine distances between passive stereo-vision system712and points in scene708based on the stereoscopically-paired images. Passive stereo-vision system712may be enabled to determine the distances based on pattern704being projected into scene708. For example, passive stereo-vision system712may be able to more accurately and/or more quickly determine the distances between passive stereo-vision system712and various points of scene708based on stereoscopically-paired images of scene708including pattern704than the stereoscopically-paired images did not include pattern704.

6DoF system710may be any suitable 6DoF system capable of determine a pose of 6DoF system710based on successive images of scene708. 6DoF system710may be, or may be included in, as examples, a head-mounted display (e.g., HMD100ofFIG.1AandFIG.1B), an XR system (e.g., XR system200ofFIG.2), or a robot.

Passive stereo-vision system712may be any suitable passive stereo-vision system capable of determining distances between passive stereo-vision system712and respective points of scene708based on stereoscopically-paired images of scene708. Passive stereo-vision system712may be, or may be included in, as examples, a head-mounted display (e.g., HMD100ofFIG.1AandFIG.1B), an XR system (e.g., XR system200ofFIG.2), a robot, a vehicle, or a camera.

Projector702may be a projector or transmitter capable of projecting or transmitting electromagnetic radiation into scene708to generate pattern704. The electromagnetic radiation may be of any suitable wavelength including, as examples, visible light (of any color), near-infrared light, infrared light, or any combination thereof. In some aspects, projector702may pattern electromagnetic radiation to generate pattern704, for example, by projecting light through one or more digital micromirror devices (DMDs) or liquid crystal displays (LCDs). Additionally or alternatively, projector702may generate pattern704using one or more lasers and/or mirrors. Projector702may be, or may include, a digital light processing (DLP) projector, an LCD projector, a light emitting diode (LED) projector, a liquid crystal on silicon (LCOS) projector, and/or a laser projector. In some cases, projector702may include a number of projectors, for example, a bundle of projectors. The projectors may be in one location and pointed in the same, or in different directions (e.g., to cover a wider field of view). Additionally or alternatively, the projectors may be in different locations throughout an environment.

Projector702may include a pattern generator (e.g., pattern generator906ofFIG.9) that may generate pattern704and a projection module (e.g., projection module904ofFIG.9) that may project pattern704into scene708. Pattern704may include dots or shapes in a unique arrangement. In the present disclosure, the term “unique” may refer to a pattern (or portion of the pattern) being visually unlike other patterns in the scene. “For example, a “unique pattern” may include unique and/or distinctive elements or shapes (such as dots, lines, etc.) that alone or together with the scene content form patches that can be detected and/or matched when compared between images captured by two cameras. Thus, pattern704may be unique relative to scene708. Additionally or alternatively, pattern704may include unique portions at various points of scene708. For example, as illustrated inFIG.7, pattern704may include different unique portions at different points on surface706. In some aspects, pattern704may be composed of dots arranged in one or more patterns. The dots may be of any shape (e.g., circles, squares, triangles, or stars). The dots may all be of the same shape or the dots of pattern704may have different shapes. Additionally or alternatively, pattern704may include lines (e.g., lines extending across surface706).

Pattern704may be of a uniform color (or wavelength or combination of wavelengths). Alternatively, dots, or portions, of pattern704, may have different colors (or wavelength or combinations of wavelengths) than other dots or portions. For example, dots on a first side of surface706may be of a first color and does on a second side surface706may be of a second, different color. Additionally or alternatively, dots on a top side of each group of dots may be of a third color and dots on a bottom side of each group of dots may be of a fourth color.

Pattern704may cause surface706to appear visually distinct. For example, absent pattern704, surface706may be substantially visually uniform. For example, if 6DoF system710were to capture successive images of surface706absent pattern704, 6DoF system710may not be able to correctly correlate features of the successive images and 6DoF system710may be unable to accurately perform visual simultaneous localization and mapping (VSLAM or SLAM) techniques. However, if 6DoF system710were to capture successive images of surface706with pattern704projected thereonto, 6DoF system710may be able to correlate points of the successive images to perform VSLAM techniques. Similarly, if passive stereo-vision system712were to capture stereoscopically-paired images of surface706absent pattern704, passive stereo-vision system712may not be able to correctly correlate features of the stereoscopically-paired images and passive stereo-vision system712may be unable to accurately determine distances to the points of surface706. However, if passive stereo-vision system712were to capture stereoscopically-paired images of surface706with pattern704projected thereonto, passive stereo-vision system712may be able to correlate features of the stereoscopically-paired images to determine depths of the points.

Projector702may generate and project pattern704onto surface706in such a way that pattern704is stationary relative to surface706. Pattern704(as projected onto surface706) may remain constant. The consistency of pattern704relative to surface706may enable 6DoF system710to determine the pose of 6DoF system710based on successive images of pattern704on surface706. Additionally or alternatively, the consistency of pattern704relative to surface706may enable passive stereo-vision system712to determine a distance between passive stereo-vision system712and points of surface706.

In some cases, a user may place projector702in scene708relative to surface706. For example, the user may position projector702to project pattern704onto surface706. Further, the user may adjust pattern704based on scene708and/or surface706. For example, the user may adjust an intensity of light of pattern704, a wavelength of light of704, a sparsity of dots of pattern704, sizes of dots of pattern704, and/or shapes of dots of pattern704.

For example,FIG.8illustrates four scenarios in which projector702may project adjusted patterns704into scene708, according to various aspects of the present disclosure. For example, in a first scenario802, projector702may be positioned on a floor810of scene708and may project a sparse pattern812onto surface706. In a second scenario804, projector702may be positioned on floor810of scene708and may project a dense pattern814onto surface706. In a third scenario806, projector702may be positioned on a wall (e.g., surface706) of scene708and may project dense pattern814onto surface706(albeit from a different angle than the angle from which projector702projects dense pattern814onto surface706in second scenario804). In a fourth scenario808, projector702may be positioned on a ceiling816of scene708and may project a very dense pattern818onto surface706and/or other surfaces of scene708.

Returning toFIG.7, in other cases, projector702may determine to project pattern704into scene708and/or onto surface706. For example, projector702may include a camera (e.g., camera908ofFIG.9) and an image analyzer (e.g., image analyzer910ofFIG.9). Projector702may use camera908to capture one or more images of scene708and determine that surface706includes visually indistinct portions. Projector702may determine to project pattern704onto surface706to cause the visually indistinct portions to be visually distinct. For example, projector702may capture an image of surface706(which may be a blank wall). Projector702may identify surface706as being visually indistinct within the scene. Projector702may determine pattern704and may determine to project pattern704onto surface706.

Further, projector702may take an image of surface706with pattern704projected thereonto and determine if pattern704causes the visually indistinct portions of surface706to be visually distinct. For example, projector702may determine if surface706, with pattern704projected thereonto, is visually distinct enough for 6DoF system710to determine a pose of 6DoF system710and/or for passive stereo-vision system712to determine a distance between passive stereo-vision system712and surface706. In some aspects, projector702may compare portions (e.g., windows or features) of an image of surface706(with pattern704projected thereonto) with other portions of the image to determine whether surface706(with pattern704projected thereonto) is visually distinct enough. If surface706(with pattern704projected thereonto) is not visually distinct enough, projector702may adjust pattern704and project the adjusted pattern onto surface706. Projector702may adjust an intensity of light of pattern704, a wavelength of light of704, a sparsity of dots of pattern704, sizes of dots of pattern704, and/or shapes of dots of pattern704.

Additionally or alternatively, projector702may receive an indication of surface706(e.g., an indication that surface706is visually indistinct or includes visually indistinct portions) from another system or devices (e.g., using a communication module, such as communication module912ofFIG.9). For example, 6DoF system710may determine that 6DoF system710is having difficulty determining the pose of 6DoF system710based on surface706and may transmit an indication of such difficulty to projector702. Additionally or alternatively, passive stereo-vision system712may determine that passive stereo-vision system712is having difficulty determining distances between passive stereo-vision system712and surface706and may transmit an indication of such difficulty to projector702. Projector702may project pattern704onto surface706and/or adjust pattern704responsive to such indications.

In some aspects, projector702may generate pattern704to encode information, such as position information (e.g., coordinates, such as latitude and longitude or local coordinates), time information (e.g., a time of day), or a message (e.g., labels, instructions, and/or warnings). The information may be decoded by image-processing systems. For example, 6DoF system710and/or passive stereo-vision system712may decode the information. 6DoF system710and/or passive stereo-vision system712may use the information. For example, a robot may capture an image of surface706, identify pattern704, and decode the information encode by pattern704. The information may include instructions, for example, regarding how to navigate within environment700. The robot may navigate according to the instructions.

Additionally or alternatively, projector702may change the information. For example, if the information is time information, projector702may update the time information over time. As another example, if the information is a message, projector702may change the message responsive to a user providing a different message.

FIG.9is a block diagram illustrating an example architecture of an example projector902, according to various aspects of the present disclosure. Projector902may be an example of projector702ofFIG.7and/orFIG.8.

Projector902includes a projection module904that may project a pattern (e.g., a unique pattern into an environment (e.g., onto a surface of the environment). Projector902may include one or more light sources (e.g., lamps, bulbs, or lasers), and/or one or more patterning modules (e.g., mirrors or LCDs).

In some aspects, projector902may include a pattern generator906that may generate the pattern. Pattern generator906may be implemented by one or more processors. In other aspects, projector902may receive the pattern from another source (e.g., via communication module912).

In some aspects, projector902may include a camera908that may capture one or more images of an environment of projector902. In some cases, projector902may be configured to scan the environment with camera908. In other aspects, projector902may not include camera908.

In some aspects, projector902may include an image analyzer910that may analyze images captured by camera908. Image analyzer910may be implemented by one or more processors (e.g., the one or more processors that implemented pattern generator906). Image analyzer910may analyze the images to determine whether the environment includes visually indistinct portions. In other aspects, projector902may not include image analyzer910.

In some aspects, projector902may include a communication module912that may receive an indication of one or more visually indistinct portions of the environment. For example, a 6DoF system or a passive stereo-vision system may transmit an indication of one or more visually indistinct portions of the environment to projector902via communication module912. In other aspects, projector902may not include communication module912.

In cases in which image analyzer910determine a visually indistinct portion of the environment and/or in cases in which projector902received an indication of the visually indistinct portion of the environment via communication module912, projector902may determine to project the pattern onto the visually indistinct portion. Additionally or alternatively, projector902may determine to adjust a projected pattern based on the determined visually indistinct portion and/or the visually indistinct portion indicated by the received indication.

FIG.10is a diagram of an example environment1000in which systems and techniques may enable pose and/or distance determinations, according to various aspects of the present disclosure. For example, a projector1002may project a pattern1004into a scene1008(including onto a surface1006of scene1008). A 6DoF system710may capture successive images of scene1008(including of pattern1004projected onto surface1006) and may determine a pose of 6DoF system710based on the captured successive images. Additionally or alternatively, a passive stereo-vision system712may capture stereoscopically-paired images of scene1008(including pattern1004projected onto surface1006) and may determine a distance between passive stereo-vision system712and points of surface1006based on the captured stereoscopically-paired images.

Projector1002ofFIG.10may be the same as, may be substantially similar to, and/or may perform the same, or substantially the same, operations as projector702ofFIG.7. However, whereas projector702is described as being stationary relative to scene708, projector1002may move, or be moved, relative to scene1008. For example, projector1002may be positioned on a moving object (e.g., a robot or drone). Projector1002may be moved in environment1000. Projector1002may project pattern1004onto surface1006despite1002moving. In some aspects, projector1002may project pattern1004such that pattern1004is constant with regard to surface1006despite projector1002moving within environment1000.

FIG.11is a flow diagram illustrating a process1100for enabling pose and/or distance determinations, in accordance with aspects of the present disclosure. One or more operations of process1100may be performed by a computing device (or apparatus) or a component (e.g., a chipset, codec, etc.) of the computing device. The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or augmented reality (AR) device, a vehicle or component or system of a vehicle, a desktop computing device, a tablet computing device, a server computer, a robotic device, and/or any other computing device with the resource capabilities to perform the process1100. The one or more operations of process1100may be implemented as software components that are executed and run on one or more processors.

At a block1102, a computing device (or one or more components thereof) may cause a projector to project a pattern into a scene for feature correlation by an imaging device that captures images of the pattern as projected into the scene. The projector may be separate from the imaging device. For example, projector702may project pattern704into scene708for an imaging device (e.g., 6DoF system710or passive stereo-vision system712) to perform feature correlation based on images of pattern704as projected into scene708.

In some aspects, the imaging device may be configured to determine distances between the imaging device and points in the scene based on the images of the pattern as projected into the scene. For example, passive stereo-vision system712may be configured to determine distances between passive stereo-vision system712and points in scene708based on images captured by passive stereo-vision system712of pattern704as pattern704appears in scene708.

In some aspects, the imaging device may be configured to determine distances between the imaging device and the points in the scene whether the projector projects the pattern or not. For example, the imaging device may be configured to determine distances between the imaging device and objects or surfaces of the scene even if the projector is not projecting the pattern into the scene. As another example, the imaging device may be configured to determine the distances even if the imaging device is capturing images of other portions of the scene, for example, portions that do not include the projected pattern. For example, passive stereo-vision system712may be configured to determine distances between passive stereo-vision system712and points in scene708whether projector702projects pattern704into scene708or not.

In some aspects, the imaging device may be configured to determine distances between the imaging device and the points in the scene without a priori information regarding the pattern. For example, passive stereo-vision system712may determine distances between passive stereo-vision system712and points in scene708without a priori information regarding pattern704. For example, passive stereo-vision system712may match features of pattern704, as projected into scene708, between images captured by passive stereo-vision system712without passive stereo-vision system712having a priori information regarding pattern704. For example, passive stereo-vision system712may not have information regarding pattern704(e.g., shape or pattern of pattern704) or even information regarding whether pattern704is being projected into scene708.

In some aspects, the imaging device may be, or may include, a passive stereo-vision system configured to correlate features of the pattern in stereoscopically-paired images of the scene to determine distances between the passive stereo-vision system and points in the scene. For example, the imaging device may be passive stereo-vision system712and passive stereo-vision system712may capture stereoscopically-paired images of scene708and correlate features between the stereoscopically paired images to determine distances between passive stereo-vision system712and points in scene708.

In some aspects, the imaging device may be configured to determine a pose of the imaging device relative to the scene based on the images of the pattern as projected into the scene. For example, 6DoF system710may determine a pose of 6DoF system710relative to scene708based on image captured by 6DoF system710of pattern704as projected into scene708.

In some aspects, the imaging device may be configured to determine the pose of the imaging device whether the projector projects the pattern or not. For example, the imaging device may be configured to determine a pose of the imaging device relative to the scene even if the projector is not projecting the pattern into the scene. As another example, the imaging device may be configured to determine the pose even if the imaging device is capturing images of other portions of the scene, for example, portions that do not include the projected pattern. For example, 6DoF system710may be configured to determine a pose of 6DoF system710whether projector702projects pattern704into scene708or not.

In some aspects, the imaging device may be configured to determine the pose of the imaging device without a priori information regarding the pattern. For example, 6DoF system710may determine a pose of 6DoF system710without a priori information regarding pattern704. For example, 6DoF system710may match features of pattern704, as projected into scene708, between images captured by 6DoF system710without 6DoF system710having a priori information regarding pattern704. For example, 6DoF system710may not have information regarding pattern704(e.g., shape or pattern of pattern704) or even information regarding whether pattern704is being projected into scene708.

In some aspects, the imaging device may be, or may include, a six-degree-of-freedom (6DoF) system configured to correlate features of the pattern in sequential images of the scene to determine a pose of the 6DoF system relative to the scene. For example, the imaging device may be 6DoF system710and 6DoF system710may be configured to capture sequential images of scene708and correlate features of pattern704as projected into scene708to determine a pose of 6DoF system710relative to scene708.

In some aspects, the projector may project the pattern into the scene without receiving a communication from the imaging device. For example, projector702may project pattern704into scene708without receiving any communication (e.g., any instruction, request, etc.) from any of 6DoF system6710or passive stereo-vision system712.

In some aspects, the projector may project the pattern into the scene whether the imaging device captures the images of the pattern as projected into the scene or not. For example, projector702may project pattern704into scene708whether 6DoF system710or passive stereo-vision system712captures images of scene708or not. Further, projector702may not have any information regarding whether 6DoF system710and/or passive stereo-vision system712are present in the environment of scene708or whether or not 6DoF system710and/or passive stereo-vision system712are capturing images of scene708or not.

In some aspects, the projector may project the pattern into the scene independent of the imaging device. For example, projector702may project pattern704into scene708independent of 6DoF system710and/or passive stereo-vision system712. For example, projector702may project pattern704into scene708regardless of whether 6DoF system710and/or passive stereo-vision system712are present in an environment of projector702, regardless of whether 6DoF system710and/or passive stereo-vision system712are operating in an environment of projector702, regardless of whether 6DoF system710and/or passive stereo-vision system712are capturing images of scene708, without any communication from 6DoF system710and/or passive stereo-vision system712.

In some aspects, the projector may be movable relative to the imaging device. For example, the projector702may be moveable (and/or steerable) relative to 6DoF system710and/or passive stereo-vision system712. For example, projector702may move (and/or reorient) separately from 6DoF system710and/or passive stereo-vision system712. For example, projector702may independently moveable.

In some aspects, the projector may project the pattern into the scene at a first time, wherein the imaging device has a first pose relative to the projector at the first time; and project the pattern into the scene at a second time, wherein the imaging device has a second pose relative to the projector at the second time. For example, projector702may project pattern704at scene708at a first time. At the first time, 6DoF system710and/or passive stereo-vision system712may have a first pose relative to projector702. Projector702may project pattern704at scene708at a second time. At the second time, 6DoF system710and/or passive stereo-vision system712may have a first pose relative to projector702. For example, the pose of 6DoF system710and/or passive stereo-vision system712relative to projector702may change over time.

In some aspects, the projector may project the pattern such that the pattern is stationary relative to the scene. For example, projector702may project pattern704such that pattern704remains stationary relative to scene708.

In some aspects, the projector may configured to be stationary relative to the scene and the imaging device may be configured to move relative to the scene. For example, projector702may be stationary relative to scene708. For example, projector702may include legs and/or a stand or may be attachable to a wall or the ceiling. 6DoF system710and/or passive stereo-vision system712may be configured to be moved. for example, may be wearable or attachable to a moving system (e.g., a robot).

In some aspects, the projector may be configured to be moved relative to the scene separately from the imaging device. For example, projector1002may be configured to move relative to scene1008. In such cases, the projector may project the pattern such that the pattern is stationary relative to the scene. For example, projector1002may project pattern1004such that pattern1004remains stationary relative to scene708despite1002moving relative to scene1008.

In some aspects, the computing device (or one or more components thereof) may cause a camera to capture an image of the scene. The computing device (or one or more components thereof) may determine a visually indistinct portion of the scene based on the image; and cause the projector to project the pattern onto the visually indistinct portion of the scene. For example, camera908of projector902may capture an image of scene708. Image analyzer910of projector902may analyze the image to determine a visually indistinct portion of scene708. Projector902may project pattern704onto the visually indistinct portion of scene708.

In some aspects, the computing device (or one or more components thereof) may cause a camera to capture an image of the scene. The computing device (or one or more components thereof) may analyze the pattern as projected into the scene as represented in the image of the scene, and cause the projector to, responsive to analyzing the pattern, adjust an intensity used to project the pattern; adjust a wavelength used to project the pattern; adjust a sparsity of dots of the pattern; adjust sizes of the dots of the pattern; and/or adjust shapes of the dots of the pattern. For example, camera908of projector902may capture an image of scene708. Image analyzer910of projector902may analyze the image. Image analyzer910may determine how pattern704, as projected into scene708appears. Projector902may adjust pattern704or how pattern704is projected into scene708bases on the analysis, for example, to make pattern704cause scene708to include more visually distinct portions or to make visually indistinct portions of scene708appear more visually distinct.

In some aspects, the computing device (or one or more components thereof) may generate the pattern. For example, projector902may include pattern generator906that may generates pattern704. In some aspects, the projector may change the pattern. For example, pattern generator906of projector902may change pattern704.

In some aspects, the pattern may encode information. For example, pattern704may encode information. In some aspects, the information may be, or may include, position information; time information; or a message. For example, pattern704may encode position information (e.g., relative to scene708and/or relative to a world coordinate system). As another example, pattern704may encode time information (e.g., a time of day and/or a date). As another example, pattern704may encode a message (e.g., an instruction or a warning).

In some aspects, the pattern may encode first information and the projector may be further configured to change the pattern to encode second information. For example, at a first time, projector702may project pattern704that encodes first information. Projector702may change pattern704to encode second information. Then, projector702may project the changed pattern704that encodes the second information.

FIG.12is a flow diagram illustrating a process1200for enabling pose and/or distance determinations, in accordance with aspects of the present disclosure. One or more operations of process1200may be performed by a computing device (or apparatus) or a component (e.g., a chipset, codec, etc.) of the computing device. The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or augmented reality (AR) device, a vehicle or component or system of a vehicle, a desktop computing device, a tablet computing device, a server computer, a robotic device, and/or any other computing device with the resource capabilities to perform the process1200. The one or more operations of process1200may be implemented as software components that are executed and run on one or more processors.

At a block1202, a computing device (or one or more components thereof) may determine to project a pattern into a scene for feature correlation by an imaging device that captures images of the pattern as projected into the scene. For example, projector702may determine to project pattern704into scene708so that 6DoF system710and/or passive stereo-vision system712may capture images of pattern704in scene708and correlate features of pattern704in the images.

At a block1204, the computing device (or one or more components thereof) may cause a projector to project the pattern into the scene. The projector may be separate from the imaging device. For example, projector702may project pattern704into scene708. Projector702may be separate from 6DoF system710and/or passive stereo-vision system712.

In some aspects, the computing device (or one or more components thereof) may cause a camera to capturing an image of the scene and determining a visually indistinct portion of the scene based on the image. Projecting the pattern into the scene may be, or may include, projecting the pattern at the visually indistinct portion of the scene. For example, camera908may capture an image of scene708. Image analyzer910may determine a visually indistinct portion of scene708. Projector902may project pattern704onto the visually indistinct portion of scene708.

In some aspects, the computing device (or one or more components thereof) may cause a camera to capture an image of the scene. The computing device (or one or more components thereof) may analyze the pattern as projected into the scene as represented in the image of the scene, and cause the projector to, responsive to analyzing the pattern, adjust an intensity used to project the pattern; adjust a wavelength used to project the pattern; adjust a sparsity of dots of the pattern; adjust sizes of the dots of the pattern; and/or adjust shapes of the dots of the pattern. For example, camera908of projector902may capture an image of scene708. Image analyzer910of projector902may analyze the image. Image analyzer910may determine how pattern704, as projected into scene708appears. Projector902may adjust pattern704or how pattern704is projected into scene708bases on the analysis, for example, to make pattern704cause scene708to include more visually distinct portions or to make visually indistinct portions of scene708appear more visually distinct.

In some examples, as noted previously, the methods described herein (e.g., process1100ofFIG.11, process1200ofFIG.12, and/or other methods described herein) can be performed, in whole or in part, by a computing device or apparatus. In one example, one or more of the methods can be performed by projector702ofFIG.7and/orFIG.8, projector902ofFIG.9, projector1002ofFIG.10, or by another system or device. In another example, one or more of the methods (e.g., process1100ofFIG.11, process1200ofFIG.12, and/or other methods described herein) can be performed, in whole or in part, by the computing-device architecture1300shown inFIG.13. For instance, a computing device with the computing-device architecture1300shown inFIG.13can include, or be included in, the components of the projector702, projector902, and/or projector1002and can implement the operations of process1100, process1200and/or other process described herein. In some cases, the computing device or apparatus can include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and/or other component(s) that are configured to carry out the steps of processes described herein. In some examples, the computing device can include a display, a network interface configured to communicate and/or receive the data, any combination thereof, and/or other component(s). The network interface can be configured to communicate and/or receive Internet Protocol (IP) based data or other type of data.

Additionally, process1100, process1200and/or other process described herein can be performed under the control of one or more computer systems configured with executable instructions and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware, or combinations thereof. As noted above, the code can be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

FIG.13illustrates an example computing-device architecture1300of an example computing device which can implement the various techniques described herein. In some examples, the computing device can include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or computing device of a vehicle), or other device. For example, the computing-device architecture1300may include, implement, or be included in any or all of projector702ofFIG.7and/orFIG.8, projector902ofFIG.9, and/or projector1002ofFIG.10. Additionally or alternatively, computing-device architecture1300may be configured to perform process1100, process1200, and/or other process described herein.

The components of computing-device architecture1300are shown in electrical communication with each other using connection1312, such as a bus. The example computing-device architecture1300includes a processing unit (CPU or processor)1302and computing device connection1312that couples various computing device components including computing device memory1310, such as read only memory (ROM)1308and random-access memory (RAM)1306, to processor1302.

Computing-device architecture1300can include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of processor1302. Computing-device architecture1300can copy data from memory1310and/or the storage device1314to cache1304for quick access by processor1302. In this way, the cache can provide a performance boost that avoids processor1302delays while waiting for data. These and other modules can control or be configured to control processor1302to perform various actions. Other computing device memory1310may be available for use as well. Memory1310can include multiple different types of memory with different performance characteristics. Processor1302can include any general-purpose processor and a hardware or software service, such as service11316, service21318, and service31320stored in storage device1314, configured to control processor1302as well as a special-purpose processor where software instructions are incorporated into the processor design. Processor1302may be a self-contained system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

To enable user interaction with the computing-device architecture1300, input device1322can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. Output device1324can also be one or more of a number of output mechanisms known to those of skill in the art, such as a display, projector, television, speaker device, etc. In some instances, multimodal computing devices can enable a user to provide multiple types of input to communicate with computing-device architecture1300. Communication interface1326can generally govern and manage the user input and computing device output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

Storage device1314is a non-volatile memory and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random-access memories (RAMs)1306, read only memory (ROM)1308, and hybrids thereof. Storage device1314can include services1316,1318, and1320for controlling processor1302. Other hardware or software modules are contemplated. Storage device1314can be connected to the computing device connection1312. In one aspect, a hardware module that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor1302, connection1312, output device1324, and so forth, to carry out the function.

Aspects of the present disclosure are applicable to any suitable electronic device (such as security systems, smartphones, tablets, laptop computers, vehicles, drones, or other devices) including or coupled to one or more active depth sensing systems. While described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and are therefore not limited to specific devices.

The term “device” is not limited to one or a specific number of physical objects (such as one smartphone, one controller, one processing system and so on). As used here in, a device may be any electronic device with one or more parts that may implement at least some portions of this disclosure. While the below description and examples use the term “device” to describe various aspects of this disclosure, the term “device” is not limited to a specific configuration, type, or number of objects. Additionally, the term “system” is not limited to multiple components or specific aspects. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. While the below description and examples use the term “system” to describe various aspects of this disclosure, the term “system” is not limited to a specific configuration, type, or number of objects.

Processes and methods according to the above-described examples can be implemented using computer-executable instructions that are stored or otherwise available from computer-readable media. Such instructions can include, for example, instructions and data which cause or otherwise configure a general-purpose computer, special purpose computer, or a processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc.

Illustrative aspects of the disclosure include:

Aspect 1. An apparatus comprising: a projector configured to project a pattern into a scene for feature correlation by an imaging device that captures images of the pattern as projected into the scene; wherein the apparatus is separate from the imaging device.

Aspect 2. The apparatus of aspect 1, wherein the imaging device is configured to determine distances between the imaging device and points in the scene based on the images of the pattern as projected into the scene.

Aspect 3. The apparatus of any one of aspects 1 or 2, wherein the imaging device is configured to determine distances between the imaging device and the points in the scene whether the projector projects the pattern or not.

Aspect 4. The apparatus of any one of aspects 2 or 3, wherein the imaging device is configured to determine distances between the imaging device and the points in the scene without a priori information regarding the pattern.

Aspect 5. The apparatus of aspect 1, wherein the imaging device comprises a passive stereo-vision system configured to correlate features of the pattern in stereoscopically-paired images of the scene to determine distances between the passive stereo-vision system and points in the scene.

Aspect 6. The apparatus of aspect 1, wherein the imaging device is configured to determine a pose of the imaging device relative to the scene based on the images of the pattern as projected into the scene.

Aspect 7. The apparatus of aspect 6, wherein the imaging device is configured to determine the pose of the imaging device whether the projector projects the pattern or not.

Aspect 8. The apparatus of any one of aspects 6 or 7, wherein the imaging device is configured to determine the pose of the imaging device without a priori information regarding the pattern.

Aspect 9. The apparatus of aspect 1, wherein the imaging device comprises a six-degree-of-freedom (6DoF) system configured to correlate features of the pattern in sequential images of the scene to determine a pose of the 6DoF system relative to the scene.

Aspect 10. The apparatus of any one of aspects 1 to 9, wherein the projector is configured to project the pattern into the scene without receiving a communication from the imaging device.

Aspect 11. The apparatus of any one of aspects 1 to 10, wherein the projector is configured to project the pattern into the scene whether the imaging device captures the images of the pattern as projected into the scene or not.

Aspect 12. The apparatus of any one of aspects 1 to 11, wherein the projector is configured to project the pattern into the scene independent of the imaging device.

Aspect 13. The apparatus of any one of aspects 1 to 12, wherein the projector is movable relative to the imaging device.

Aspect 14. The apparatus of any one of aspects 1 to 13, wherein the projector is configured to: project the pattern into the scene at a first time, wherein the imaging device has a first pose relative to the projector at the first time; and project the pattern into the scene at a second time, wherein the imaging device has a second pose relative to the projector at the second time.

Aspect 15. The apparatus of any one of aspects 1 to 14, wherein the projector is configured to project the pattern such that the pattern is stationary relative to the scene.

Aspect 16. The apparatus of any one of aspects 1 to 15, wherein the projector is configured to be stationary relative to the scene and wherein the imaging device is configured to move relative to the scene.

Aspect 17. The apparatus of any one of aspects 1 to 16, wherein the projector is configured to be moved relative to the scene separately from the imaging de vice.

Aspect 18. The apparatus of any one of aspects 1 to 17, further comprising: a camera configured to capture an image of the scene; and at least one processor configured to: determine a visually indistinct portion of the scene based on the image; and cause the projector to project the pattern onto the visually indistinct portion of the scene.

Aspect 19. The apparatus of any one of aspects 1 to 18, further comprising: a camera configured to capture an image of the scene; and at least one processor configured to analyze the pattern as projected into the scene as represented in the image of the scene, wherein, responsive to analyzing the pattern, the projector is configured to at least one of: adjust an intensity used to project the pattern; adjust a wavelength used to project the pattern; adjust a sparsity of dots of the pattern; adjust sizes of the dots of the pattern; or adjust shapes of the dots of the pattern.

Aspect 20. The apparatus of any one of aspects 1 to 19, further comprising a pattern generator configured to generate the pattern.

Aspect 21. The apparatus of any one of aspects 1 to 20, wherein the pattern encodes information.

Aspect 22. The apparatus of aspect 21, wherein the information comprises at least one of: position information; time information; or a message.

Aspect 23. The apparatus of any one of aspects 1 to 22, wherein the projector is further configured to change the pattern.

Aspect 24. The apparatus of any one of aspects 1 to 23, wherein the pattern encodes first information and wherein the projector is further configured to change the pattern to encode second information.

Aspect 25. A method comprising: determining to project a pattern into a scene for feature correlation by an imaging device that captures images of the pattern as projected into the scene; and projecting the pattern into the scene from a projector that is separate from the imaging device.

Aspect 26. The method of aspect 25, further comprising: capturing an image of the scene; and determining a visually indistinct portion of the scene based on the image; wherein projecting the pattern into the scene comprises projecting the pattern at the visually indistinct portion of the scene.

Aspect 27. The method of any one of aspects 25 or 26, further comprising: capturing an image of the scene; analyzing the pattern as projected into the scene as represented in the image of the scene; and responsive to analyzing the pattern, at least one of: adjusting an intensity used to project the pattern; adjusting a wavelength used to project the pattern; adjusting a sparsity of dots of the pattern; adjusting sizes of the dots of the pattern; or adjusting shapes of the dots of the pattern.

Aspect 28. The method of any one of aspects 25 to 27, further comprising generating the pattern.

Aspect 29. The method of any one of aspects 25 to 28, wherein the pattern encodes information.

Aspect 30. The method of aspect 29, wherein the information comprises at least one of: position information; time information; or a message.

Aspect 31. The method of any one of aspects 25 to 30, further comprising changing the pattern.

Aspect 32. The method of any one of aspects 25 to 31, wherein the pattern encodes first information, and further comprising changing the pattern to encode second information.

Aspect 33. An imaging device comprising: two cameras configured to capture stereoscopically-paired images of a scene; and at least one processor configured to correlate features of the stereoscopically-paired images to determine distances between the two cameras and points in the scene corresponding to the features; wherein the points in the scene are illuminated by a pattern and wherein the pattern is projected by a projector that is separate from the imaging device.

Aspect 34. An imaging device comprising: a cameras configured to capture sequential images of a scene; and at least one processor configured to track features across the sequential images to determine a pose of the imaging device relative to the scene; wherein the features correspond to points in the scene that are illuminated by a pattern and wherein the pattern is projected by a projector that is separate from the imaging device.

Aspect 35. The apparatus of aspect 1, wherein the imaging device comprises a three-degree-of-freedom (3DoF) system configured to correlate features of the pattern in sequential images of the scene to determine a position of the 3DoF system relative to the scene.