Patent Description:
<CIT> discloses a method for depth sensor aided estimation of virtual reality environment boundaries includes generating depth data at a depth sensor of an electronic device based on a local environment proximate the electronic device. A set of initial boundary data is estimated based on the depth data, wherein the set of initial boundary data defines an exterior boundary of a virtual bounded floor plan. The virtual bounded floor plan is generated based at least in part on the set of initial boundary data. Additionally, a relative pose of the electronic device within the virtual bounded floor plan is determined and a collision warning is displayed on a display of the electronic device based on the relative pose.

<CIT> discloses systems and methods for interacting with virtual reality (VR) space using a head mounted display, including detecting a real-world object in a real-world space in which a user is interacting the VR space. An image of the real-world object is presented in the VR space. The image of the real-world object is used to identify presence of the real-world object while interacting with the VR space. Interaction by the user with the real-world object is detected. The VR space is configured to generate a simulated view of the user interacting with a VR object that is mapped to the interaction with the real-world object. The simulated view is presented in the HMD.

<CIT> discloses technology for displaying a collision between objects by an augmented reality display device system. A collision between a real object and a virtual object is identified based on three-dimensional space position data of the objects. At least one effect on at least one physical property of the real object is determined based on physical properties of the real object, like a change in surface shape, and physical interaction characteristics of the collision. Simulation image data is generated and displayed simulating the effect on the real object by the augmented reality display. Virtual objects under control of different executing applications can also interact with one another in collisions.

The same, or like, reference numbers in different figures indicate similar or identical items.

Head-mounted displays are worn by users to view and interact with content in virtual-reality environments. To provide an immersive experience, head-mounted displays may cover a large portion or even all of the user's field-of-view. As such, head-mounted displays may block the user's vision from the real-world, which may result in the user tripping over objects, running into furniture, failing to notice individuals within the real-world, and so forth. Removing the head-mounted display allows the user to see the real-world but requires the user to re-orient themselves between virtual-reality environments and the real-world, respectively. Some head-mounted displays may include, for example a visor that opens to allow the user to look into the real-world. Such solution, however, interrupts an immersion of the user within the virtual-reality environment.

In an effort to overcome these deficiencies, some head-mounted displays may enable pass-through imaging that allow respective users to view the real-world without removing their respective head-mounted display. However, existing pass-through imaging tends to exhibit fairly crude response times, may fail to depict the real-world from a perspective or point-of-view of the user, and/or may be distorted. As a result, users may become unbalanced, dizzy, disoriented, or even ill.

This application describes, in part, a head-mounted display (HMD) for use in virtual-reality (VR) environments. The systems and methods described herein may determine information about a real-world environment surrounding the user, a location of the user within the real-world environment, and/or a pose of the user within the real-world environment. Such information may allow the HMD to display images of the real-world environment in a pass-through manner and without detracting the user from the VR environment. In some instances, the HMD may pass-through images of the real-world environment based on one or more triggering events. For example, while the HMD is worn and while the user is immersed within the VR environment, the user may actuate a button that allows the user to look around within the real-world environment. As an additional example, if the user hears something suspicious or of interest, if the user wants to locate an item in the real-world environment (e.g., controller), and/or if a visitor enters a vicinity of the user, the HMD may display content associated with the real-world environment. In some instances, the content may be provided to the user in an aesthetic way to limit detracting the user from the VR environment. For example, the content may be provided as an overlay to virtual content associated with the VR environment. In such instances, the user may continue to wear the HMD and remain immersed within the VR environment. Accordingly, the HMD according to the instant application may increase user experiences when transitioning or displaying content between the real-world environment and the VR environment.

The HMD may include a front having a display worn on the face, adjacent to the eyes, of the user. The display may output images (or other content) for viewing by the user. As an example, the user may wear the HMD to play a game or view media content (e.g., movie).

The HMD may include cameras that capture images of the real-world environment. In some instances, the cameras may mount to the display and/or may be incorporated within the front of the HMD. Additionally, or alternatively, the cameras may be forward facing to capture images external to the HMD and in front of the user. Moreover, in some instances, the cameras may be separate from the HMD and placed throughout the environment or on other portions of the user (e.g., waist).

In some instances, the cameras may be spatially separated such that the optical axis of the cameras are parallel and separated by a known distance. The cameras may therefore capture images of the real-world environment from a slightly different viewpoint. The diversity of information between the viewpoints may be used to calculate depth information (i.e., stereo camera imaging) of the real-world environment. For example, the HMD and/or a communicatively coupled computing device (e.g., gaming console, personal computer, etc.) may use the image data captured by the cameras to generate depth information associated with the real-world environment.

For example, the cameras may include a first camera and a second camera displaced horizontally and/or vertically from one another on the front of the HMD. In some instances, the first camera may be located on the front at a first side of the HMD, while the second camera may be located on the front at a second side of the HMD. However, as noted above, the first camera and the second cameras may be located elsewhere within the environment and/or on other portions of the user.

The image data captured by the first camera and/or the second camera may represent different views of the real-world environment (e.g., room). By comparing the images (or image data) captured by the first camera and/or the second camera, the HMD (and/or another communicatively coupled computing device) may determine differences or disparities (e.g., using a disparity mapping algorithm). The disparities may represent a difference in coordinates of corresponding image points in the two images. As the disparities (or disparity values) are inversely proportional to depths within the real-world environment, the HMD and/or another communicatively coupled computing device, such as a gaming console, may determine depth information associated with the real-world environment (or a portion thereof). In some instances, the depth information may be from the perspective of the user (i.e., the user's gaze).

Using the depth information, the HMD and/or another communicatively coupled computing device such as a gaming console may generate a depth map or a three-dimension (3D) mesh of the real-world environment (or a portion thereof). For example, the depth map may represent distances between the user and objects within the real-world environment (e.g., walls of a room, furniture, etc.). Additionally, or alternatively, the HMD may include other sensors utilized to generate the depth map and/or 3D. For example, the HMD may include a depth sensor to determine distances between the user and objects in the real-world environment and may determine that the user is proximate (e.g., predetermined proximity or threshold proximity) to objects, or a boundary, of the real-world environment. However, the HMD or the gaming console may additionally, or alternatively, use light detection and ranging (LIDAR), ultrasonic ranging, stereoscopic ranging, structured light analysis, dot projection, particle projection, time-of-flight observations, and so forth for use in generating the depth map and/or 3D mesh.

Upon generating the depth map and/or 3D mesh of the real-world environment, the HMD and/or another communicatively coupled computing device, such as a gaming console, may project image data onto the depth map and/or 3D mesh. In this sense, the image data may be first utilized to generate the depth map and/or 3D mesh, and secondarily may be superimposed, overlaid, or projected onto the depth map and/or 3D mesh. In doing so, the HMD may display content to depict the real-world environment to the user.

In some instances, the depth map, the 3D mesh, and/or the images captured by the first camera and/or the second camera may be modified to account for a pose or point-of-view the user. For example, as the cameras may not align (e.g., horizontally, vertically, and depth wise) with the eyes of the user (i.e., the cameras are not in the exact position of the eyes of the user), the depth map and/or 3D mesh may account for this discrepancy. In other words, the image data captured by the first camera and the second camera may represent a point-of-view that is different than the point-of-view of the user. Failing to account for this discrepancy may illustrate an imperfect real-world environment and the user may find it difficult to pick up objects as the depth values or image data is not from the point-of-view of the user. For example, because the cameras capture image data from a perspective and/or depth that is different than the perspective and/or depth of the eyes of the user (i.e., the cameras are not in the same position horizontally, vertically, and or depth wise as the eyes of the user), the 3D mesh may account for this offset to accurately portray and present an undistorted view of the real-world environment to the user. That is, the depth map, the 3D mesh, and/or the image data may be modified based at least in part on a difference (or offset) in coordinate positions (or location) of the first camera and the second camera and the eyes (e.g., a first eye and/or a second eye) or point-of-view of the user. Accordingly, the images of the real-world environment displayed to the user may accurately represent objects in the real-world environment from the perspective of the user.

Additionally, or alternatively, in some instances, the HMD and/or the real-world environment may include sensors that track a gaze, point-of-view, and/or field-of-view of the user. For example, the HMD may include a interpupillary distance (IPD) sensor to measure the distance between pupils of the eyes of the user and/or other sensors that detect an eye gaze direction of the user. Such sensors may be utilized to determine the point-of-view of the user to accurately portray images of the real-world environment to the user.

Pass-through imaging may allow the user to interact with and view objects in the real-world environment, such as co-workers, computer screens, mobile devices, etc. In some instances, the user may toggle or switch between the VR environment and the real-world environment or the HMD may automatically switch between the VR environment and the real-world environment in response to one or more triggering events. That is, the HMD may include one or more modes, such as a pass-through mode where the real-world environment is presented to the user and/or a virtual-reality mode where the VR environment is presented to the user. As an example, while wearing the HMD, the user may want to take a drink of water. Rather than removing the HMD, the user may actuate (e.g., double press) a button on the HMD that causes display of the real-world environment. In turn, the display may present images of the real-world environment to allow user to locate his or her glass of water and without taking off the HMD. Thereafter, after locating the glass of water, the user may actuate the button (e.g., single press) to cause the display to present virtual content of the VR environment. As such the passed-through images representing the real-word environment may permit users to move about the real-world environment to locate objects and without bumping into objects (e.g., furniture). As an additional example, the pass-through images may represent another individual that comes into a real-world environment of the user. Here, the HMD may detect the other individual and may present images on the display such that the user may recognize or be made aware of the other individual.

In some instances, content associated with the real-word environment displayed to the user may be partially transparent to maintain the user's sense of being in the VR environment. In some instances, content associated with the real-word environment may be combined with the virtual content or only the content associated with the real-world environment may be presented (e.g., <NUM> percent pass-through imaging). Additionally, or alternatively, in some instances, the images captured by the cameras may be presented on an entirety of the display or may be presented within a specific portion. Moreover, in some instances, content associated with the real-world environment may be displayed with dotted lines to indicate to the user which content is part of the real-world environment and which content is part of the VR environment. Such presentation may allow the user to see approaching individuals, commotion, and/or objects surrounding the user. Regardless of the specific implementation or configuration, the HMD may to function to display content associated with the real-world environment to warn, detect, or otherwise recognize objects that come within a field-of-view of the user.

In some instances, the HMD and/or other computing devices associated with the VR environment, such as the gaming console, may operate in conjunction with a tracking system within the real-world environment. The tracking system may include sensors that track a position of the user within the real-world environment. Such tracking may be used to determine information about the real-world environment surrounding the user while the user is immersed in the VR environment, such as a location of the user within the environment and/or a pose or point-of-view of the user. Within the real-world environment, the tracking system may determine the location and/or pose of the user. In some instances, the tracking system may determine a location and/or a pose of the user relative to a center of the real-world environment.

In some instances, the tracking system may include lighting elements that emit light (e.g., visible or non-visible) into the real-world environment and sensors which detect incident light. In some instances, to detect the location and/or pose of the user the HMD may include markers. Upon projecting light into the real-world environment, the markers may reflect the light and the sensors may capture incident light reflected by the markers. The captured incident light may be used to track and/or determine the locations of the markers within the environment, which may be used to determine the location and/or pose of the user.

In some instances, the location and/or pose of the user within the real-world environment may be utilized to present warnings, indications, or content to the user. For example, if the user is approaching a wall of the real-world environment, knowing the location of the user and the wall (via the tracking system), the HMD may display images representing the wall within the real-world environment. That is, in addition to, or alternative from, using the images captured by the cameras to determine the user is approaching a wall (i.e., via the depth values), the tracking system may determine the relative location of the user within the real-world environment. Such tracking may assist in presenting images accurately corresponding to a depth (or placement of) objects within the real-world environment.

Moreover, images or data obtained from the tracking system may be used to generate a 3D model (or mesh) of the real-world environment. For example, knowing the location and/or pose of the user, the HMD, the tracking system, the gaming console, and/or another communicatively coupled computing device may determine a relative location of the user within the real-world environment. This location, and/or pose of the user, may be utilized to determine a corresponding portion of the 3D model of the real-world environment were the user is looking (i.e., the field-of-view of the user). For example, as the cameras capture images that are not associated with a point-of-view of the user, the tracking system may determine the gaze, pose, or point-of-view of the user. Such information may be used to determine where the user is looking within the real-world environment. Knowing where the user is looking in the real-world environment may be used to modify the images captured by the cameras. In doing so, the HMD may accurately display the real-world environment.

The HMD, the tracking system, the gaming console, and/or another communicatively coupled computing device may also compare the depth map and/or 3D mesh generated using the image data of the cameras with the 3D model to determine a relative location of the user within the real-world environment. Regardless of the specific implementation, knowing the location and/or pose of the user within the real-world environment, the HMD and/or another communicatively coupled computing device may transform the points of depth map and/or the points of the 3D mesh onto the 3D model of the real-world environment. In turn, the images captured by the cameras may be projected onto the 3D model corresponding to the point-of-view of the user.

Accordingly, in light of the above, this application discusses a HMD that provides pass-through imaging to enhance VR experiences. The pass-through imaging may provide a relatively seamless experience when displaying content of the VR environment and content associated with the real-world environment. Such pass-through imaging provides for a less intrusive and disturbing solution to view content associated with the real-world environment. In some instances, information or content associated with the real-world environment may be selectively provided to the user in response to triggers including, but not limited to motions, sounds, gestures, preconfigured events, user movement changes, etc. Moreover, in some instances, the HMD according to the instant application may take many forms, including helmets, visors, goggles, masks, glasses, and other head or eye wear worn on the head of the user.

The present disclosure provides an overall understanding of the principles of the structure, function, device, and system disclosed herein. One or more examples of the present disclosure are illustrated in the accompanying drawings. Those of ordinary skill in the art will understand that the devices and/or the systems specifically described herein and illustrated in the accompanying drawings are non-limiting embodiments. The features illustrated or described in connection with one embodiment may be combined with the features of other embodiments, including as between systems and methods. Such modifications and variations are intended to be included within the scope of the appended claims.

<FIG> illustrates a user <NUM> residing within an environment <NUM> and wearing a HMD <NUM>. In some instances, the user <NUM> may wear the HMD <NUM> to immerse the user <NUM> within a VR environment. In some instances, the user <NUM> may interact within VR environment using one or more controllers <NUM>. The HMD <NUM> includes a display <NUM> for providing virtual content and/or images to the user <NUM> and in some instances, images capturing devices, such as a first camera <NUM> and/or a second camera <NUM>.

The first camera <NUM> and/or the second camera <NUM> may capture images of the environment <NUM> and pass-through images of the environment <NUM> to the user <NUM> for viewing on the display <NUM>. That is, and as discussed in detail herein, images captured by the first camera <NUM> and/or the second camera <NUM> may be presented to the user <NUM> in a pass-through manner to allow the user <NUM> to view the environment <NUM> without having to disengage from the VR environment and/or remove the HMD <NUM>.

In some instances, the first camera <NUM> and/or the second camera <NUM> may be disposed within or near a front of the HMD <NUM>. In some instances, the first camera <NUM> and/or the second camera <NUM> may represent stereo cameras, infrared (IR) cameras, depth cameras, and/or any combinations thereof. Images captured by the first camera <NUM> and/or the second camera <NUM> may represent the environment <NUM> surrounding the user <NUM>. In some instances, the first camera <NUM> and/or the second camera <NUM> may be forward facing to capture images of the environment <NUM> in front of the user <NUM>. In some instances, the first camera <NUM> and/or the second camera <NUM> may be spatially separated such that their optical axes are parallel. Images captured by the first camera <NUM> and/or the second camera <NUM> may therefore represent the environment <NUM> from different viewpoints and may be used to determine depth information associated with the environment <NUM> (i.e., stereo camera imaging). However, in some instances, the first camera <NUM> and/or the second camera <NUM> may be located elsewhere within the environment <NUM>. For example, the first camera <NUM> and/or the second camera <NUM> may be located on the floor of the environment <NUM>, may be located on a desk within the environment <NUM>, etc..

As illustrated, the HMD <NUM> may include processor(s) <NUM> that carry out or otherwise perform operations associated with the HMD <NUM>. For example, the processor(s) <NUM> cause the first camera <NUM> and/or the second camera <NUM> to capture images, and subsequently, may receive images captured by the first camera <NUM> and/or the second camera <NUM>, compare the images (or image data), and determine differences therebetween. As the differences are inversely proportional to depths of objects within the environment <NUM>, the processor(s) <NUM> may determine depth information associated with the environment <NUM> (e.g., walls, furniture, TVs, etc.).

In some instances, using the depth information, the processor(s) <NUM> may generate a depth map or 3D mesh of the environment <NUM>. For example, as illustrated, the HMD <NUM> includes memory <NUM> that stores or otherwise has access to a depth map <NUM> of the environment <NUM> and/or a 3D mesh <NUM> of the environment <NUM>. As the image data captured by the first camera <NUM> and/or the second camera <NUM> represents a portion of the environment <NUM>, the depth map <NUM> and/or the 3D mesh <NUM> may correspondingly represent a portion of the environment <NUM>. In some instances, upon generating the depth map <NUM> and/or the 3D mesh <NUM>, the processor(s) <NUM> may store the depth map <NUM> and/or the 3D mesh <NUM> within the memory <NUM>.

As the image data captured by the first camera <NUM> and/or the second camera <NUM>, the depth map, and/or the 3D mesh is not from the perspective of the user <NUM> (i.e., the point-of-view of the user <NUM>), the HMD <NUM> (and/or another communicatively coupled computing device) may account the placement of the first camera <NUM> and/or the second camera <NUM> relative to the perspective of the user <NUM> (e.g., relative to a first eye and/or a second eye, respectively). That is, whether located on the HMD <NUM> or elsewhere within the environment, the first camera <NUM> and/or the second camera <NUM> do not capture images corresponding to the point-of-view of the user <NUM> and/or the perspective of the user <NUM>. Accordingly, the image data, or points within the depth map and/or 3D mesh, may be modified or offset to account for this displacement.

In some instances, the HMD <NUM> may operate in conjunction with a tracking system <NUM>. In some instances, the HMD <NUM> may communicatively couple to the tracking system <NUM> over a network <NUM>. For example, the HMD <NUM> and the tracking system <NUM> may include one or more interfaces, such as network interfaces <NUM> and/or network interfaces <NUM>, respectively, to facilitate the wireless connection to the network <NUM>. The network <NUM> is representative of any type of communication network, including data and/or voice network, and may be implemented using wired infrastructure (e.g., cable, CAT5, fiber optic cable, etc.), a wireless infrastructure (e.g., RF, cellular, microwave, satellite, Bluetooth, etc.), and/or other connection technologies.

The tracking system <NUM> may include components that determine or track a pose of the user <NUM>, the HMD <NUM>, the first camera <NUM>, and/or the second camera <NUM> within the environment <NUM>. In this sense, the tracking system <NUM> may determine the location, orientation, and/or pose of the user <NUM>, the HMD <NUM>, the first camera <NUM>, and/or the second camera <NUM> at a time in which the first camera <NUM> and/or the second camera <NUM> captured images of the environment <NUM> for passing-through to the user <NUM>. For example, the tracking system <NUM> (and/or another computing device) may analyze and parse images captured by the tracking system <NUM> to identify the user <NUM> within the environment <NUM> and/or the pose of the user <NUM>. For example, in some instances, the tracking system <NUM> may include projector(s) <NUM> and/or sensor(s) <NUM> that operate to determine the location, orientation, and/or pose of the user <NUM>. As shown, and in some instances, the tracking system <NUM> may mount to walls of the environment <NUM>. Additionally, or alternatively, the tracking system <NUM> may mount elsewhere within the environment <NUM> (e.g., ceiling, floor, etc.).

The projector(s) <NUM> are configured to generate and project light and/or images into the environment <NUM>. In some instances, the images may include visible light images perceptible to the user <NUM>, visible light images imperceptible to the user <NUM>, images with non-visible light, or a combination thereof. The projector(s) <NUM> may be implemented with any number of technologies capable of generating the images and projecting within/into the environment <NUM>. Suitable technologies include a digital micromirror device (DMD), liquid crystal on silicon display (LCOS), liquid crystal display, 3LCD, and so forth.

The sensor(s) <NUM> may include high resolution cameras, infrared (IR) detectors, sensors, 3D cameras, IR cameras, RGB cameras, and so forth. The sensor(s) <NUM> are configured to image the environment <NUM> in visible light wavelengths, non-visible light wavelengths, or both. The sensor(s) <NUM> may be configured to capture information for detecting depth, location, orientation, and/or pose of objects within the environment <NUM>. For example, as the user <NUM> maneuvers about the environment <NUM>, the sensor(s) <NUM> may detect positions, orientations, and/or poses of the user <NUM>. In some instances, the sensor(s) <NUM> may capture some or all angles and positions within the environment <NUM>. Alternatively, the sensor(s) <NUM> may focus on or capture images within a predefined area of the environment <NUM>.

The projector(s) <NUM> and/or the sensor(s) <NUM> may operate in conjunction with marker(s) <NUM> of the HMD <NUM>. For example, the tracking system <NUM>, via the projector(s) <NUM>, may project light into the environment <NUM> and the sensor(s) <NUM> may capture images of the reflections of the marker(s) <NUM>. Using the captured images, the tracking system <NUM>, such as processor(s) <NUM> of the tracking system <NUM>, may determine distance information to the marker(s) <NUM>. Additionally, or alternatively, the tracking system <NUM> may detect a pose (e.g., orientation) of the user <NUM> within the environment <NUM>. In some instances, the marker(s) <NUM> may be used to determine a point-of-view of the user <NUM>. For example, a distance between the marker(s) <NUM> and the eyes of the user <NUM> may be known. In capturing image data of the marker(s) <NUM>, the tracking system <NUM> (and/or other communicatively coupled computing device) may determine the relative point-of-view of the user <NUM>. Accordingly, the tracking system <NUM> may utilize the marker(s) <NUM> of the HMD <NUM> to determine a relative location and/or pose of the user <NUM> within the environment <NUM>.

To define or determine characteristics about the environment <NUM>, upon starting a gaming application, the HMD <NUM> may request the user <NUM> define a boundary, perimeter, or area of the environment <NUM> in which the user <NUM> may maneuver while being immersed in the VR environment. As an example, the processor(s) <NUM> may cause the display <NUM> to present instructions to the user <NUM> to walk around the environment <NUM> and define a boundary of the environment <NUM> (or the area in which the user <NUM> will maneuver while immersed in a VR environment). As the user <NUM> walks around the environment <NUM>, the HMD <NUM> may capture images of the environment <NUM> via the first camera <NUM> and/or the second camera <NUM> and the tracking system <NUM> may track the user <NUM>. Therein, upon determining the boundaries of the environment <NUM>, the tracking system <NUM> may determine a central location (e.g., origin) of the area. Knowing the central location of the area may allow for the HMD <NUM> to properly display relative locations of objects or scenes within the environment <NUM>. In some instances, the central location may be represented as (<NUM>, <NUM>, <NUM>) in a (X, Y, Z) Cartesian Coordinate System.

In some instances, the tracking system <NUM> may transmit the boundary and/or central location to the HMD <NUM>. For example, the processor(s) <NUM> of the HMD <NUM> may store may store the boundary and/or the central origin in the memory <NUM>, as indicated by boundary <NUM>. Additionally, or alternatively, in some instances, the images captured by the first camera <NUM> and/or the second camera <NUM> may be associated with images captured by the tracking system <NUM>. For example, using the images captured by the first camera <NUM> and/or the second camera <NUM> while defining the area, depth maps of the environment <NUM> may be generated. These depth maps may correspondingly be associated with certain locations/poses within the environment <NUM>, as determined through tracking the user <NUM> throughout the environment <NUM>. In some instances, these depth maps may be combined or otherwise used to generate a 3D model or mesh of the environment <NUM>. In receiving subsequent image data from the HMD <NUM> and/or the tracking system <NUM>, the location of the user <NUM> within the environment <NUM> may be determined, which may assist in determining depth information within the environment <NUM> and from the perspective, location, or pose of the user. For example, as the image data captured by the first camera <NUM> and/or the second camera <NUM> does not correspond to the point-of-view of the user <NUM>, the tracking system <NUM> may determine a point-of-view of the user <NUM> via images captured from the marker(s) <NUM>. Using this point-of-view, the image data captured by the first camera <NUM> and/or the second camera <NUM> may be modified to represent the point-of-view of the user <NUM>.

For example, as the user <NUM> engages in the VR environment and maneuvers about the environment <NUM>, the tracking system <NUM> may determine a relative location of the HMD <NUM> within the environment <NUM> by comparing the reflected light from the marker(s) <NUM> with the central location. Using this information, the HMD <NUM>, the tracking system <NUM>, and/or another communicatively coupled computing device (e.g., gaming console) may determine a distance of the HMD <NUM> from the central location, or a location of the HMD <NUM> relative to the central location. Additionally, the HMD <NUM>, the tracking system <NUM>, and/or another communicatively coupled computing device may determine a pose, such as a point-of-view of the user <NUM> within the environment <NUM>. For example, the tracking system <NUM> may determine the user <NUM> is looking towards a ceiling, wall, or floor of the environment <NUM>. Such information may be used to modified or otherwise account for a position of the first camera <NUM> and/or the second camera <NUM> relative to the eyes or point-of-view of the user.

In some instances, the tracking system <NUM> may couple to a chassis with a fixed orientation, or the chassis may couple to actuator(s) <NUM> such that the chassis may move. The actuators <NUM> may include piezoelectric actuators, motors, linear actuators, and other devices configured to displace or move the chassis or components of the tracking system <NUM>, such as the projector(s) <NUM> and/or the sensor(s) <NUM>.

The HMD <NUM> may additionally, or alternatively, operate in conjunction with remote computing resources <NUM>. The tracking system <NUM> may also communicatively couple to the remote computing resources <NUM>. In some examples, the HMD <NUM> and/or the tracking system <NUM> may communicatively couple to the remote computing resources <NUM> given that the remote computing resources <NUM> may have a computational capacity that far exceeds the computational capacity of the HMD <NUM> and/or the tracking system <NUM>. The HMD <NUM> and/or the tracking system <NUM> may therefore utilize the remote computing resources <NUM> for performing relatively complex analysis and/or generating image data models (or meshes) of the environment <NUM>. For example, the first camera <NUM> and/or the second camera <NUM> may capture image data that and the HMD <NUM> may provide to the remote computing resources <NUM> over the network <NUM> for analysis and processing. In some instances, the remote computing resources <NUM> may transmit content to the HMD <NUM> for display. For example, in response to a triggering event (e.g., button press) that configures the HMD <NUM> to pass-through images captured by the first camera <NUM> and/or the second camera <NUM>, the remote computing resources <NUM> may transmit a depth map, 3D mesh, and/or 3D model onto which the HMD <NUM> is to project the images captured by the first camera <NUM> and/or the second camera <NUM>. As such, images captured by the first camera <NUM> and/or the second camera <NUM> for passing onto the user <NUM> may be transmitted to the remote computing resources <NUM> for processing and the HMD <NUM> may receive content to be displayed on the display <NUM>.

As illustrated, the remote computing resources <NUM> include processor(s) <NUM> and memory <NUM>, which may store or otherwise have access to some or all of the components described with reference to the memory <NUM> of the HMD <NUM>. For example, the memory <NUM> may have access to and utilize the depth map <NUM>, the 3D mesh <NUM>, and/or the boundary <NUM>. The remote computing resources <NUM> may additionally, or alternatively, store or otherwise have access to memory <NUM> of the tracking system <NUM>.

In some instances, the remote computing resources <NUM> may be remote from the environment <NUM> and the HMD <NUM> may communicatively couple to the remote computing resources <NUM> via the network interfaces <NUM> and over the network <NUM>. The remote computing resources <NUM> may be implemented as one or more servers and may, in some instances, form a portion of a network-accessible computing platform implemented as a computing infrastructure of processors, storage, software, data access, and so forth that is maintained and accessible via a network such as the Internet. The remote computing resources <NUM> do not require end-user knowledge of the physical location and configuration of the system that delivers the services. Common expressions associated with these remote computing resources <NUM> may include "on-demand computing," "software as a service (SaaS)," "platform computing," "network-accessible platform," "cloud services," "data centers," and so forth.

In some instances, the environment <NUM> may include a computer <NUM> (or gaming application, gaming console, gaming system) that communicatively couples to the HMD <NUM>, the controllers <NUM>, the tracking system <NUM>, and/or the remote computing resources <NUM> via the network <NUM> and/or wired technologies. In some instances, the computer <NUM> may perform some of or all of the processes described herein, such as those performable by the HMD <NUM>, the controllers <NUM>, the tracking system <NUM>, and/or the remote computing resources <NUM>. For example, the computer <NUM> may modify image data received from the first camera <NUM> and/or the second camera <NUM>, or a depth map pertaining thereto, to account for a point-of-view of the user <NUM>. In such instances, the computer <NUM> may determine or receive an indication from the tracking system <NUM> regarding the point-of-view of the user <NUM> and/or an origin of the area where the user is located (e.g., an origin of the real world environment) and/or an origin of a virtual world. Accordingly, the computer may modify the image data received from the first camera <NUM> and/or the second camera <NUM>, or the depth map pertaining thereto based on the indication received from the tracking system <NUM>. In some embodiments, a time(s) (e.g., time stamp(s)) at which the image data was captured by one or both of the cameras <NUM>, <NUM> may be used to determine a time difference between a time of capturing image data and a time of displaying the captured image data. For example, the image data that is ultimately displayed to the user may be delayed by tens of milliseconds after the image data was captured by the camera(s) <NUM> and/or <NUM>. An adjustment may be made to the image data to account for this temporal disparity or disagreement, such as by modifying the pixel data (e.g. through rotational adjustments, and/or translational adjustments) in order to present images as they would appear in the physical world at the time when the imagery is displayed on the HMD, which avoids presenting imagery that appears to lag behind head motion of the user. In some embodiments, the camera(s) <NUM> and/or <NUM> may be tracked to maintain time stamp information so that the pose of the camera(s) at a time of capturing the image(s) is known, and so that the pose of the camera(s) at the time the resultant image data is received and processed can be used to accurately represent the imagery spatially, instead of being dependent on camera motion. Moreover, the computer <NUM> may store or otherwise have access to some or all of the components described with reference to the memory <NUM>, <NUM>, and/or <NUM>.

As used herein, a processor, such as processor(s) <NUM>, <NUM>, <NUM>, and/or processors of the computer may include multiple processors and/or a processor having multiple cores. Further, the processor(s) may comprise one or more cores of different types. For example, the processor(s) may include application processor units, graphic processing units, and so forth. In one implementation, the processor(s)may comprise a microcontroller and/or a microprocessor. The processor(s) may include a graphics processing unit (GPU), a microprocessor, a digital signal processor or other processing units or components known in the art. Alternatively, or in addition, the functionally described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc. Additionally, each of the processor(s) may possess its own local memory, which also may store program components, program data, and/or one or more operating systems.

As used herein, memory, such as the memory <NUM>, <NUM>, <NUM>, and/or memory of the computer <NUM> may include volatile and nonvolatile memory, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program component, or other data. Such memory may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium which can be used to store the desired information and which can be accessed by a computing device. The memory may be implemented as computer-readable storage media ("CRSM"), which may be any available physical media accessible by the processor(s) to execute instructions stored on the memory. In one basic implementation, CRSM may include random access memory ("RAM") and Flash memory. In other implementations, CRSM may include, but is not limited to, read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), or any other tangible medium which can be used to store the desired information and which can be accessed by the processor(s).

<FIG> illustrates a perspective view of the HMD <NUM>. The HMD <NUM> may include a front <NUM> and a back <NUM> that secure to the head of the user <NUM>. For instance, the HMD <NUM> may include strands, cords, sections, straps, bands, or other members that operably couple the front <NUM> of the HMD <NUM> and the back <NUM> of the HMD <NUM>. The front <NUM> includes the display <NUM> positioned in front or over the eyes of the user <NUM> to render images output by an application (e.g., a video game). Discussed above, the display <NUM> may output images (frames) viewed by the user <NUM> to make the user <NUM> perceive the images as if immersed in a VR environment.

In some instances, the front <NUM> of the HMD <NUM> may include the first camera <NUM> and/or the second camera <NUM>. The first camera <NUM> and/or the second camera <NUM> may capture images external to the HMD <NUM> (i.e., of the environment <NUM>) for viewing by the user <NUM> on the display <NUM>. Noted above, the optical axes of the first camera <NUM> and the second camera <NUM> may be parallel and separated by a predetermined distance to generate depth information of at least a portion of the environment <NUM> using images captured by the first camera <NUM> and/or the second camera <NUM>. However, in some instances and as discussed above, the HMD <NUM> may not include the first camera <NUM> and/or the second camera <NUM>. For example, the first camera <NUM> and/or the second camera <NUM> may be located elsewhere within the environment <NUM>.

In either scenario, the first camera <NUM> and/or the second camera <NUM> capture images that may not correspond to the point-of-view of the user <NUM> (i.e., as the first camera <NUM> and/or the second camera <NUM> are not in the actual position of the eyes of the user <NUM>). Using the techniques described herein, however, the image data captured by the first camera <NUM> and/or the second camera <NUM> may be modified to account for the displacement of the first camera <NUM> and/or the second camera <NUM> relative to the point-of-view of the user <NUM>.

For example, to determine the point-of-view of the user <NUM>, the HMD <NUM> may include the marker(s) <NUM>. As shown in <FIG>, in some instances, the marker(s) <NUM> may include a first marker <NUM>(<NUM>), a second marker <NUM>(<NUM>), a third marker <NUM>(<NUM>), and/or a fourth marker <NUM>(<NUM>) disposed at corners, edges, or along a periphery of the front <NUM>. However, the marker(s) <NUM> may be located elsewhere on the HMD <NUM>, such along the top, sides, or the back <NUM>. In some instances, the marker(s) <NUM> may include infrared elements, reflectors, digital watermarks, and/or images that are responsive to electromagnetic radiation (e.g., infrared light) emitted by the projector(s) <NUM> of the tracking system <NUM>. Additionally, or alternatively, the marker(s) <NUM> may include tracking beacons that emit electromagnetic radiation (e.g., infrared light) that is captured by the sensor(s) <NUM> of the tracking system <NUM>. That is, the projector(s) <NUM> project light into the environment <NUM> and the marker(s) <NUM> may reflect light. The sensor(s) <NUM> may capture incident light reflected by the marker(s) <NUM> and the tracking system <NUM>, or another communicatively coupled computing device, such as the remote computing resources <NUM>, may track and plot the locations of the marker(s) <NUM> within the environment <NUM> to determine movements, positions, poses and/or orientations of the user <NUM>. The marker(s) <NUM> may therefore be used to indicate the point-of-view of the user <NUM> for use in modifying, adjusting, or otherwise adapting the image data from the first camera <NUM> and/or the second camera <NUM> before being displayed to the user <NUM>.

<FIG> illustrates the user <NUM> wearing the HMD <NUM> within an environment <NUM>. <FIG> illustrates the display <NUM> of the HMD <NUM> displaying virtual content <NUM> while the user <NUM> is wearing the HMD <NUM> and interacting within a VR environment. As discussed above, the HMD <NUM> may display the virtual content <NUM> as the user <NUM> moves about the environment <NUM>.

The tracking system <NUM> may be positioned within the environment <NUM> to track the user <NUM> from one location to another and determine a point-of-view of the user <NUM> within the environment. For example, the tracking system <NUM> may utilize marker(s) <NUM> on the HMD <NUM> to determine a pose (e.g., location and orientation) of the user <NUM>. In some instances, the pose of the user <NUM> may be relative to a central location of the environment <NUM>. For example, the central location may have coordinates (<NUM>, <NUM>, <NUM>), and using reflected light from the marker(s) <NUM>, the tracking system <NUM> (or the remote computing resources <NUM>) may determine the pose of the user <NUM> in coordinate space (e.g., (X, Y, Z)). The tracking system <NUM> may also determine depths of objects within the environment <NUM> from the perspective or point-of-view of the user <NUM>. Image data received from the first camera <NUM> and/or the second camera <NUM> may additionally, or alternatively, be used to determine depth within the environment <NUM>.

The HMD <NUM> may coordinate display of the virtual content <NUM> with objects (e.g., furniture, walls, etc.) in the environment <NUM> and/or within the point-of-view of the user <NUM> and/or in front of the user <NUM>. In other words, the objects of the environment <NUM> (or the real-world) displayed to the user <NUM> may correspond to the point-of-view of the user <NUM> in the environment <NUM>. For example, the HMD <NUM> may display objects within the environment <NUM> using image data captured by the first camera <NUM> and/or the second camera <NUM> (e.g., pass-through images) based at least in part on one or more triggering events. That is, the HMD <NUM> may be configured to display images at a location on the display <NUM> corresponding to the actual placement of objects in the environment <NUM>. Utilizing the depth map generated from the image data of the first camera <NUM> and/or the second camera <NUM> as well as a 3D model or mesh of the environment <NUM>, the pass-through images presented to the user <NUM> may illustrate objects located at their actual placement within the environment <NUM> (i.e., proper depth value) and the from point-of-view of the user <NUM>, thereby allowing the user <NUM> to pick up objects (e.g., controller, glass of water, and so forth). However, in some instances, image data from the first camera <NUM> and/or the second camera <NUM> may be continuously captured, and upon detecting of a triggering event, the HMD <NUM> may display content associated with the environment <NUM> to the user <NUM>.

As shown in <FIG>, the triggering event may include determining that the user <NUM> is approaching or nearing a boundary of the environment <NUM>. For example, the user <NUM> may approach a corner <NUM> between two walls <NUM>, <NUM> of the environment <NUM>. Knowing the location of the user <NUM> via the tracking system <NUM>, the HMD <NUM> may display content on the display <NUM> to indicate that the user <NUM> is nearing the corner <NUM>. For example, as shown, indications of the corner <NUM> and the walls <NUM>, <NUM> may be displayed as dashed or dotted lines on the display <NUM>. In doing so, the user <NUM> is presented with an indication that the user <NUM> is about to walk into the walls. In some instances, and as shown in <FIG>, indications, or content pertaining to the corner <NUM> and walls <NUM>, <NUM> may be overlaid (combined with virtual content displayed on the display <NUM> of the HMD <NUM>) or presented on (or in combination with) the virtual content <NUM>.

Noted above, the image data captured by the first camera <NUM> and/or the second camera <NUM> may be modified to account for the point-of-view of the user <NUM>. Additionally, or alternatively, the depth map, the 3D mesh, or a model of the environment may take into account, or factor into consideration, a location of the first camera <NUM> and/or the second camera <NUM> relative to the eyes or point-of-view of the user <NUM>. As such, as the first camera <NUM> and/or the second camera <NUM> have a point-of-view that may be different than the user <NUM>, the image data may be modified to adjust to the point-of-view of the user prior to presentation on the HMD <NUM>.

While <FIG> illustrates a particular implementation of displaying the corner <NUM> and/or the walls <NUM>, <NUM> on the display <NUM>, content may be presented differently such as, without modifications, modified for display, embedded, merged, stacked, split, rerendered, or otherwise manipulated to be appropriately provided to the user <NUM> without interrupting the immersive virtual experience. For example, in some instances, the content may be displayed in a particular region of the display <NUM> (e.g., upper right-hand corner). Additionally, or alternatively, the display <NUM> may only present content associated with the corner <NUM> and walls <NUM>, <NUM> so as to have <NUM> percent pass-through. In some instances, the HMD <NUM> may fade in content associated with the environment <NUM> on the display <NUM>.

<FIG> illustrates the user <NUM> wearing the HMD <NUM> within an environment <NUM>. In some instances, the HMD <NUM> may display pass-through images based at least in part on one or more triggering events. In some instances, the triggering event may include determining that a visitor <NUM> has entered the environment <NUM> and/or a predefined area within the environment <NUM>. As shown in <FIG>, the display <NUM> depicts the visitor <NUM> approaching the user <NUM> wearing the HMD <NUM>. Here, the visitor <NUM> appears to be approaching the user <NUM> and as such, the display <NUM> may display images captured by the first camera <NUM> and/or the second camera <NUM>. The images, however, however, may be first modified to account for a different in a point-of-view between the first camera <NUM> and/or the second camera <NUM> and the point-of-view of the user <NUM>. In some instances, the HMD <NUM> may display the visitor <NUM> based at least in part on the visitor <NUM> being within the point-of-view of the user <NUM> and/or in front of the user <NUM>, as determined using the marker(s) <NUM>, while wearing the HMD <NUM> or coming within a threshold distance of the user <NUM>. In some instances, the visitor <NUM> may be detected via a motion sensor, analyzing image data from the first camera <NUM> and/or the second camera <NUM>, via the tracking system <NUM>, etc..

<FIG> and <FIG> illustrates processes according to the embodiments of the instant application. The processes described herein are illustrated as collections of blocks in logical flow diagrams, which represent a sequence of operations, some or all of which may be implemented in hardware, software, or a combination thereof. In the context of software, the blocks may represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, program the processors to perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures and the like that perform particular functions or implement particular data types. The order in which the blocks are described should not be construed as a limitation, unless specifically noted. Any number of the described blocks may be combined in any order and/or in parallel to implement the process, or alternative processes, and not all of the blocks need be executed. For discussion purposes, the processes are described with reference to the environments, architectures and systems described in the examples herein, such as, for example those described with respect to <FIG>, although the processes may be implemented in a wide variety of other environments, architectures and systems.

<FIG> illustrates an example process <NUM> for generating a 3D model, depth map, or mesh of the environment <NUM>. In some instances, the process <NUM> may be performed by the remote computing resources <NUM>. However, the HMD <NUM>, the tracking system <NUM>, and/or the computer <NUM> may additionally perform some of or all of the process <NUM>.

At <NUM>, the process <NUM> may transmit a request to define an area within the environment <NUM>. For example, the remote computing resources <NUM> may transmit a request to the HMD <NUM> requesting that the user <NUM> define an area of the environment <NUM>. In some instances, the area may represent an area in which the user <NUM> intends to move about while immersed in a VR environment. In response to receiving the request, the processor(s) <NUM> of the HMD <NUM> may present the request, or information associated with the request, on the display <NUM>. For example, the request may inform the user <NUM> to wear the HMD <NUM> and walk around the environment <NUM> and define the area in which the user intends move about.

At <NUM>, the process <NUM> may transmit a request to track the HMD <NUM> within the environment <NUM> and as the user <NUM> wears the HMD <NUM> to define the area. For example, the remote computing resources <NUM> may transmit a request to the tracking system <NUM> to track the user <NUM> throughout the environment <NUM> while the user <NUM> defines the area. In some instances, the tracking system <NUM> may track the user <NUM> via the projector(s) <NUM> projecting images into the environment <NUM> and the sensor(s) <NUM> capturing images of light reflected via the marker(s) <NUM> of the HMD <NUM>. In some instances, the remote computing resources <NUM> may transmit the request at <NUM> and the request at <NUM> at the same or substantially the same time.

At <NUM>, the process <NUM> may receive first image data from the tracking system <NUM>. For example, the remote computing resources <NUM> may receive first image data from the tracking system <NUM>, where the first image data represents the locations of the HMD <NUM> and/or the user <NUM> as the user <NUM> walks about the environment <NUM> to define the area. For instances, the first image data may represent the locations and/or poses of the marker(s) <NUM> within the environment <NUM>. The first image data received by the remote computing resources <NUM> may include a time stamp at which the images were captured by the tracking system <NUM> (or the sensor(s) <NUM>).

At <NUM>, the process <NUM> may determine characteristics of the area. For example, based at least in part on receiving the first image data, the remote computing resources <NUM> may determine a boundary or perimeter of the area. Additionally, or alternatively, the remote computing resources <NUM> may determine, based at least in part on the first image data, a center or origin of the area. In some instances, the origin may be defined in 3D space having values (<NUM>, <NUM>, <NUM>) that correspond to X, Y, and Z coordinates, respectively, in a Cartesian Coordinate System. Accordingly, in receiving subsequent image data from the tracking system <NUM> and/or the remote computing resources <NUM> may determine a relative location of the user <NUM> (and/or the HMD <NUM>) to the origin of the area.

At <NUM>, the process <NUM> may receive second image data from the HMD <NUM>. For example, the remote computing resources <NUM> may receive second image data from the HMD <NUM>, where the second image data represents images captured by the first camera <NUM> and/or the second camera <NUM> while the user <NUM> defines the area of the environment <NUM>. In some instances, the second images data received by the remote computing resources <NUM> may include a time stamp at which the images were captured by the first camera <NUM> and/or the second camera <NUM>. That is, while the first camera <NUM> and/or the second camera <NUM> capture the images, another communicatively coupled computing device may determine a pose of the user <NUM> within the environment <NUM>. For example, mentioned above, while the first camera <NUM> and/or the second camera <NUM> capture the images, the processor(s) <NUM> of the tracking system <NUM> may cause the projector(s) <NUM> to project the images into the environment <NUM>. The marker(s) <NUM> of the HMD <NUM> may reflect light associated with the images and the sensor(s) <NUM> may capture images of the reflections of the marker(s) <NUM>. Such images may be used to determine depth, location, orientation, and/or pose of the user <NUM> (and/or the HMD <NUM>) within the environment <NUM>, and may be associated with the image data captured by the first camera <NUM> and/or the second camera <NUM>.

At <NUM>, the process <NUM> may generate a 3D model (or mesh) of the environment <NUM>. For example, the remote computing resources <NUM> may generate a 3D model of the environment <NUM> based at least in part on the first image data and/or the second image data. For instance, using the second image data, the remote computing resources <NUM> may compare images captured by the first camera <NUM> and/or the second camera <NUM>, respectively, to determine disparities. In turn, the remote computing resources <NUM> may determine depth information of the environment <NUM> for use in generating the 3D model of the environment <NUM>. Additionally, the remote computing resources <NUM> may utilize the first image data received from the tracking system <NUM> to associate the depth map, or depth values of the environment <NUM>, with certain locations within the area and/or the environment <NUM>. In some instances, the remote computing resources <NUM> may compare or associate the first image data and the second image data (or depth map generated therefrom) using the time stamps at which the first image data and/or the second image data was captured. In some instances, generating the 3D model of the environment <NUM> may include portraying objects within the environment. For example, the remote computing resources <NUM> may augment the VR environment with objects (or a volume) in the environment <NUM>, such as a chair. In some instances, the 3D model may be transmitted to the HMD <NUM> and/or the tracking system <NUM>, and/or may be stored in memory of the HMD <NUM>, the tracking system <NUM>, and/or the remote computing resources <NUM>.

<FIG> illustrates an example process <NUM> for passing-through images on a HMD, such as the HMD <NUM>. In some instances, the process <NUM> may continue from <NUM> of the process <NUM>, after the 3D model of the environment <NUM> is generated. In other words, in some instances, after the 3D model is generated, the HMD <NUM> may display pass-through images to enable the user <NUM> to switch between a VR environment and real-world environment (i.e., the environment <NUM>). In some instances, the process <NUM> may be performed by the remote computing resources <NUM>. However, the HMD <NUM>, the tracking system <NUM>, and/or the computer <NUM> may additionally perform some of or all of the process <NUM>.

At <NUM>, the process <NUM> may receive first image data from the HMD <NUM>. For example, the remote computing resources <NUM> may receive first image data from the HMD <NUM>, where the first image data represents images captured by the first camera <NUM> and/or the second camera <NUM>. In some instances, the remote computing resources <NUM> may receive the first image data based at least in part on the HMD <NUM> detecting a trigger event, such as a press of a button by the user <NUM>, a verbal command issued by the user <NUM>, motion being detected within the environment <NUM> (e.g., visitor approaching the user <NUM>), and/or the user <NUM> nearing or coming within a threshold distance of a boundary of the environment <NUM>. In other instances, the remote computing resources <NUM> may continuously receive the first image data and may be configured to pass-through these images based at least in part on the triggering event. For example, as the user <NUM> may be immersed within the VR environment, the user <NUM> may press a button of the HMD <NUM> to display content external to the HMD <NUM>. In this sense, the HMD <NUM> may include a pass-through mode that displays images captured by the first camera <NUM> and/or the second camera <NUM>. Accordingly, based at least in part on detecting the trigger expression, the processor(s) <NUM> of the HMD <NUM> may cause the first camera <NUM> and/or the second camera <NUM> to capture images of the environment <NUM>.

At <NUM>, the process <NUM> may generate a depth map and/or a 3D mesh based at least in part on the first image data. For example, the remote computing resources <NUM> may generate a depth map and/or the 3D mesh based at least in part on the first image data being received from the HMD <NUM> (i.e., using stereo camera imaging). As the first image data represents a portion of the environment <NUM>, the depth map and/or the 3D mesh may also correspond to a depth map and/or a 3D mesh of the portion of the environment <NUM>. Additionally, or alternatively, the HMD <NUM> may generate the depth map and/or 3D mesh. For example, upon receiving the image data from the first camera <NUM> and/or the second camera <NUM>, the processor(s) <NUM> may utilize stereoscopic camera imaging to generate the depth map. In some instances, using the depth map, the processor(s) <NUM> may generate a 3D mesh of the environment <NUM>. Additionally, in some instances, the processor(s) <NUM> may store the depth map and/or 3D mesh, such as the depth map <NUM> and/or the 3D mesh <NUM> within the memory <NUM>.

In some instances, the HMD <NUM>, the remote computing resources <NUM>, and/or the computer <NUM> may modify the first image data received from the first camera <NUM> and/or the second camera <NUM> prior to generating the depth map and/or the 3D mesh. For example, as the first image data may not represent a point-of-view of the user <NUM>, the first image data may be modified (e.g., translated, transformed, skewed, etc.) to account for a different between the point-of-view of the first camera <NUM> and the second camera <NUM>, and the point-of-view of the user <NUM>. In some instances, the point-of-view of the user <NUM> may be determined using the tracking system <NUM> and the position of the marker(s) <NUM> within the environment <NUM>. Additionally, or alternatively, after generating the depth map and/or the 3D mesh, the depth map and/or the 3D mesh may be modified according to or based at least in part on the point-of-view of the user <NUM>.

At <NUM>, the process <NUM> may receive second image data form the tracking system <NUM> representing a pose of the HMD <NUM>. For example, the remote computing resources <NUM> may receive, from the tracking system <NUM>, second image data corresponding to the HMD <NUM> within the environment <NUM>. As discussed above, the second image data may be captured by sensor(s) <NUM> of the tracking system <NUM> which detect light reflected by the marker(s) <NUM> of the HMD <NUM> in response to the images being projected by the projector(s) <NUM>.

At <NUM>, the process <NUM> may determine a pose of the HMD <NUM>. For example, based at least in part on receiving the second image data, the remote computing resources <NUM> may determine a pose of the HMD <NUM> within the environment <NUM>. In some instances, the pose may represent a location of the HMD <NUM> within the environment <NUM> and/or an orientation of the HMD <NUM> within the environment <NUM>. That is, the remote computing resources <NUM> may analyze the first image data and/or the second image data to determine the location of the user <NUM> within the environment <NUM> in relation to the center of the environment <NUM>. Such analysis may determine the relative location, gaze, and/or point-of-view of the user <NUM> with respect to the center of the environment <NUM>, or a particular area within the environment <NUM>.

At <NUM>, the process <NUM> may transform the depth map and/or 3D mesh into points associated with the 3D model of the environment. For example, the remote computing resources <NUM> may utilize the pose (e.g., location and orientation) to determine a location of the user <NUM> within the environment <NUM> and/or the point-of-view of the user <NUM> within the environment <NUM>. The remote computing resources <NUM> may transform the points of the depth map and/or 3D mesh into points associated with the 3D model of the environment <NUM>, thereby accounting for the point-of-view of the user <NUM>. In other words, the processor(s) <NUM> of the remote computing resources <NUM> may locate, find, or determine the depth values of points in the 3D model of the environment <NUM> and by transforming the points of the depth map and/or 3D mesh onto points associated with the 3D model of the environment <NUM>. Using the pose, the remote computing resources <NUM> may translate the depth map and/or 3D mesh generated at <NUM> onto the 3D model of the environment <NUM>. Such translating may assist in accurately depicting the environment <NUM> (e.g., proper depth values) to the user <NUM>. That is, the first image data, together with the second image data and/or the 3D model of the environment <NUM>, may provide an absolute position and/or gaze of the HMD <NUM>, which may assist in depicting the point-of-view of the user <NUM> at proper depth values or depth perception.

At <NUM>, the process <NUM> may project the first image data onto a portion of the 3D model to generate third image data. For example, knowing the pose of the user <NUM>, the remote computing resources <NUM> may project or overlay the first image data onto the portion of the 3D model of the environment <NUM> to generate the third image data. In some instances, generating the third image data may include cross-blending or filling in depth values and/or color values for certain pixels of the third image data. For example, as the first image data may not represent a point-of-view of the user <NUM>, when generating the third image data depicting the point-of-view of the user <NUM>, the third image data may have undefined depth and/or color values for certain pixels. Here, pixels without color values may be assigned a color value from neighboring pixels, or an average thereof. Additionally, or alternatively, the third image data may be generated using previous depth maps or 3D meshes of the environment <NUM>, particle systems, and so forth.

At <NUM>, the process <NUM> may transmit the third image data to the HMD <NUM>. For example, after projecting the first image data onto the portion of the 3D model, the remote computing resources <NUM> may transmit the third image data to the HMD <NUM>, where the third image data represents the first image data as projected onto the portion of the 3D model.

From <NUM>, the process <NUM> may loop to <NUM> to receive subsequent image data. As a result, in response to continuous trigger events (e.g., press of a button, voice command, motion detection, etc.), the HMD <NUM> may transition to a pass-through mode, to provide a convenient way for the user <NUM> to check out the environment <NUM> and without having to take off the HMD <NUM>. For example, the remote computing resources <NUM> may receive an indication, from the tracking system <NUM>, that the user <NUM> is approaching the boundary of the area and/or is about to run into a wall of the environment <NUM>. Upon receiving this indication, the remote computing resources <NUM> may receive image data from the HMD <NUM> representing the point-of-view of the user (i.e., the images captured by the first camera <NUM> and/or the second camera <NUM>). Upon determining the pose of the user <NUM>, the remote computing resources <NUM> may project the image data onto the 3D model of the environment <NUM> and cause the image data to be displayed on the HMD <NUM>. In this sense, the images may be automatically "passed-through" onto the user <NUM> to allow the user to see the real-world environment without having to break immersion.

Claim 1:
A system comprising:
a head-mounted display (<NUM>);
a first camera (<NUM>);
a second camera (<NUM>);
one or more processors (<NUM>, <NUM>); and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors (<NUM>, <NUM>), cause the one or more processors (<NUM>, <NUM>) to perform acts comprising:
receiving, from the first camera (<NUM>), first image data representing a first portion of an environment (<NUM>, <NUM>, <NUM>);
receiving, from the second camera (<NUM>), second image data representing a second portion of the environment (<NUM>, <NUM>, <NUM>);
generating, based at least in part on the first image data and the second image data, a first depth map of a portion of the environment (<NUM>, <NUM>, <NUM>);
receiving, from a tracking system (<NUM>), data corresponding to a point-of-view of a user (<NUM>) within the environment (<NUM>, <NUM>, <NUM>); characterised by further performing acts comprising:
generating a second depth map based at least in part on the first depth map and the data;
generating third image data based at least in part on projecting the first image data and the second image data onto the second depth map; and
transmitting the third image data to the head-mounted display (<NUM>).