Co-located pose estimation in a shared artificial reality environment

Artificial reality (AR) systems track pose and skeletal positioning for multiple co-located participants, each having a head mounted display (HMD). Participants can join a shared artificial reality event or experience with others in the same location. Each participant's HMD can independently render AR content for the participant based on the participant's pose and pose information obtained from other participants' HMDs. A participating HMD may broadcast tracking estimates for skeletal points of interest (e.g., joints, finger tips, knees, ankle points, etc.) that are within the field-of-view of the HMD's cameras and/or sensors. A participating HMD may receive skeletal position information determined by other HMDs, and aggregate the received tracking information along with internal tracking information to construct an accurate, full estimate of its own pose and skeletal positioning information for its corresponding participant.

TECHNICAL FIELD

This disclosure generally relates to artificial reality systems, such as virtual reality, mixed reality, and/or augmented reality systems, and more particularly, pose estimation performed by artificial reality systems.

BACKGROUND

Artificial reality systems are becoming increasingly ubiquitous with applications in many fields such as computer gaming, health and safety, industrial, and education. As a few examples, artificial reality systems are being incorporated into mobile devices, gaming consoles, personal computers, movie theaters, and theme parks. In general, artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof.

Typical artificial reality systems include one or more devices for rendering and displaying content to users. As one example, an artificial reality system may incorporate a head mounted display (HMD) worn by a user and configured to output artificial reality content to the user. The artificial reality content may include completely-generated content or generated content combined with captured content (e.g., real-world video and/or images). Multiple users, each having their own HMD, may participate in a shared artificial reality experience.

SUMMARY

In general, this disclosure describes artificial reality systems and, more specifically, a pose tracking system that tracks pose and body positioning for multiple co-located participants, each having an artificial reality system that includes a head mounted display (HMD). Cameras or other sensors on the participants' HMDs cooperate to provide an accurate estimation of the pose and body position of each of the participants. The system does not require any markers to be to be placed on participants in order to determine pose or body position. Further, the system does not require any external cameras or sensors to determine a participant's pose and body position. Thus, the techniques described in the application can provide a “sandbox” AR/VR system that can be simpler and less costly to set up than previous systems.

Participants can join a shared artificial reality event or experience with others in the same location. Each participant's HMD can independently render artificial reality content for the participant based on the participant's pose and body position information obtained from other participants' HMDs. An HMD of a participant in the shared artificial reality event or experience can be referred to as a “participating HMD.” The estimated pose and body position information for each participating HMD can be updated when a new frame is generated, or when there is a change in the pose or body position of a participant. The participants' HMDs can perform various operations to update the pose and body position information of the participants within the artificial reality content. For example, a participating HMD may calibrate positions of other participating HMDs into a joint artificial reality space (e.g., a shared map). Calibration of poses and body position information may improve over time as more pose and body position information is obtained.

A participating HMD may broadcast tracking estimates for body position information of co-located participants that are within the field-of-view of the HMD's cameras and/or sensors. The body position of a co-located participant may or may not be within the field-of-view of other participating HMDs. A participating HMD may receive body position information determined by other HMDs, and aggregate the received tracking information along with internal tracking information to construct an accurate, full estimate of its own pose and body positioning information for its corresponding participant. In this way, an HMD receives the two dimensional (2D) or three dimensional (3D) pose information and body position information from other participating HMDs. In some aspects, a participating HMD may use such 2D or 3D pose information and body position information to refine pose estimates and body position information for itself. In some aspects, a participating HMD can use the 2D or 3D pose information and body position information to “fill in” missing information with respect to the body position of its corresponding participant. For example, a user's lower body (e.g., lower torso, legs and feet) may not be within the field-of-view of the user's own HMD, but the user's lower body may be within the field-of-view of one or more other users' HMDs. The 2D or 3D pose and body position information received from these other HMDs can be used to fill in details regarding the body positioning of the user's lower body. Additionally, an HMD can use pose and body position information to refine previous estimates of pose and body position information. In some aspects, a participating HMD can use the 2D or 3D pose information and body position information to locate itself and other HMDs within the shared map.

Each participating HMD may broadcast its known 2D or 3D pose and body position information determined by the HMD to the other HMDs for use in constructing or updating the shared map. Each participating HMD may render artificial reality content using its copy of the shared map and the refined 2D or 3D pose and body position information determined by itself and as received from other HMDs, as described above.

A technical problem with conventional artificial reality systems is that markers or other indicia are typically used to determine the body position of users. In such conventional systems, markers or other indicia are placed at body positions of interest. Placement of markers can be cumbersome and time consuming thus leading to user dissatisfaction. Additionally, some conventional systems utilize external cameras (i.e., cameras not integrated with an HMD) to determine body position of users. This adds complexity and expense to an artificial reality system. The techniques disclosed herein provide a technical solution to the aforementioned technical problems. An HMD can receive pose and body position information from other HMDs participating in a multi-user artificial reality application. The receiving HMD can use the received pose and body position information to fill in missing information and refine existing estimates of body position information for co-located participants.

The aspects described above and further aspects described below can provide a technical improvement over conventional artificial reality system implementations, and can provide one or more practical applications, such as enabling an artificial reality system to accurately determine pose and body position information without the use of external image capture devices and without requiring the use of markers placed on the users body to indicate body position.

In one or more further example aspects, a method includes obtaining, from an image capture device of a first head mounted display (HMD), first image data representative of a physical environment; determining a first HMD pose representing a position and orientation of the first HMD; determining, from the first image data, first body position information of the users of one or more second HMDs within the physical environment; transmitting, by the first HMD, the first HMD pose and the first body position information for use by the one or more second HMDs; receiving, by the first HMD from each second HMD of the one or more second HMDs, a second HMD pose of the respective second HMD and second body position information determined by the second HMD from second image data obtained by the second HMD; integrating, by the first HMD, the first body position information with the second body position information to create first solved body position information of the user of the first HMD; and rendering, for display at the first HMD, artificial reality content in accordance with the first pose and the first solved body position information.

In one or more example aspects, an artificial reality system includes an image capture device configured to capture first image data representative of a physical environment; a first head mounted display (HMD) configured to output artificial reality content; a pose tracker configured to: determine a first HMD pose representing a position and orientation of the first HMD, determine, from the first image data, first body position information of the users of one or more second HMDs within the physical environment, transmit the first HMD pose and the first body position information for use by the one or more second HMDs, receive, from each second HMD of the one or more second HMDs, a second HMD pose of the respective second HMD and second body position information determined by the second HMD from second image data obtained by the second HMD, and integrate the first body position information with the second body position information to create first solved body position information of the user of the first HMD; and a rendering engine configured to render, for display at the first HMD, artificial reality content in accordance with the first HMD pose and the first solved body position information.

In one or more additional example aspects, a non-transitory, computer-readable medium comprises instructions that, when executed, cause one or more processors of an artificial reality system to obtain, from an image capture device of a first head mounted display (HMD), first image data representative of a physical environment; determine a first HMD pose representing a position and orientation of the first HMD; determining, from the first image data, first body position information of the users of one or more second HMDs; transmit, by the first HMD, the first HMD pose and the first body position information for use by the one or more second HMDs; receive, by the first HMD from each second HMD of the one or more second HMDs, a second HMD pose and second body position information determined by the second HMD from second image data obtained by the second HMD; integrate, by the first HMD, the first body position information with the second body position information to create first solved body position information of the user of the first HMD; and render, for display at the first HMD, artificial reality content in accordance with the first pose and the first solved body position information.

DETAILED DESCRIPTION

FIG. 1Ais an illustration depicting an example artificial reality system100that performs pose tracking and body position tracking for one or more co-located users102, in accordance with the techniques of the disclosure. In the example illustrated inFIG. 1A, artificial reality system100includes users102A-102C (collectively, “users102”) wearing head-mounted displays (HMDs)112A-112C (collectively, “HMDs112), respectively.

Each of HMDs112is worn by one of users102and includes an electronic display and optical assembly for presenting artificial reality content to user102. In addition, HMD112may include one or more motion sensors (e.g., accelerometers) for tracking motion of the HMD112and may include one or more image capture devices138, e.g., cameras, infrared (IR) detectors, Doppler radar, line scanners and the like, for capturing image data of the surrounding physical environment120. For example, user102A wears HMD112A having image-capture device138A. Image-capture device138A defines a field-of-view116A. A user102can be referred to as a co-located user when the user102is within the same physical environment as at least one other user with an HMD, and may therefore be within the field-of-view of an image capture device or within a sensor range of the at least one other user's HMD.

In some example implementations, artificial reality system100generates and renders artificial reality content to a user102based on one or more detected poses of an HMD112worn by user102and on body position information of users102that are within the field of view138of the user's HMD112. In the example implementation illustrated inFIG. 1A, each of HMDs112operates as a stand-alone, mobile artificial reality system. The HMDs112participating in a shared artificial reality experience may be communicably coupled via a network104, which may be a wired or wireless network, such as WiFi, a mesh network or a short-range wireless communication medium. For example, the colocation of the users102in the same physical environment can facilitate the use of Bluetooth or other short range local or personal area network technology.

In general, artificial reality system100uses information captured from a real-world, 3D physical environment120to render artificial reality content for display to user102. In the example ofFIG. 1A, each user102A,102B and102C views the artificial reality content constructed and rendered by an artificial reality application executing on the user's respective HMD112(e.g.,112A,112B and112C). In some examples, the artificial reality content may comprise a mixture of real-world imagery and virtual objects, e.g., mixed reality and/or augmented reality. In other examples, artificial reality content may be, e.g., a video conferencing application, a navigation application, an educational application, simulation, gaming applications, or other types of applications that implement artificial reality.

During operation, the artificial reality application constructs artificial reality content for display to user102by tracking and computing pose information for a frame of reference, typically a viewing perspective of HMD112. Additionally, the artificial reality application may track body position information for the user of HMD112and for other users visible to user102via the user's HMD112. Using HMD112as a frame of reference, and based on a current field of view130as determined by current estimated pose of HMD112, the artificial reality application renders 3D artificial reality content which, in some examples, may be overlaid, at least in part, upon the real-world, 3D physical environment120of a user102. During this process, the artificial reality application uses sensed data received from HMD112, such as movement information and user commands to capture 3D information within the real world, physical environment, such as motion by user102and/or motion of one or more hand-held controllers. Based on the sensed data, the artificial reality application determines a current pose for the frame of reference of HMD112, body position information for the user and other users within the field of view of the user's HMD112. Further, the artificial reality application can receive information from other user's HMDs112such as 2D or 3D pose and body position information sensed and/or determined by the other HMDs112. The information determined by the user's HMD112, and the information received from the other user's HMDs112may be partial or incomplete. For example, the information may lack pose or body position information with respect to portions of collocated users' bodies that are not within the field-of-view of a user's own HMD112. Other users' HMDs112may be able to supply some or all of the missing information, and different HMDs112may supply different portions of body position information depending on the portions that are within the field-of-view of the respective HMDs. Thus, even if one user's HMD112is not able to determine the other co-located users' 3D poses or body position information using its own data, the HMD112may utilize 2D or 3D pose and body position information received from other co-located HMDs to fill in and augment its own information and use such additional information to more accurately solve 3D pose and body position information for both itself and other co-located users.

Additionally, in some aspects, the artificial reality system100may maintain a shared map indicating the positioning of the user112with respect to other users of the artificial reality system100. The shared map may be generated and maintained based on information received from each participating user's HMD112. In accordance with the shared map, current poses and skeletal positioning information, the artificial reality application renders the artificial reality content.

FIG. 1Bis an illustration depicting another example artificial reality system130that performs pose tracking and body position tracking for one or more co-located users102, in accordance with the techniques of the disclosure. The example illustrated inFIG. 1Bincludes HMDs112that may be similarly configured to those discussed above with reference toFIG. 1A. In the example illustrated inFIG. 1B, the artificial reality system130includes a console106, and may optionally include external cameras such as cameras102A and102B. Additionally, artificial reality system130may optionally include external sensors90.

In the example illustrated inFIG. 1B, console106is shown as a single computing device, such as a gaming console, workstation, a desktop computer, or a laptop. In other examples, console106may be distributed across a plurality of computing devices, such as a distributed computing network, a data center, or a cloud computing system. Console106, HMDs112, cameras102, and sensors90may, as shown in this example, be communicatively coupled via network104, which as discussed above, may be a wired or wireless network, such as WiFi, a mesh network or a short-range wireless communication medium.

In the example illustrated inFIG. 1B, some or all of the functions described as being performed by the HMDs112ofFIG. 1Amay be offloaded to console106. For example, console106may receive image data from cameras on HMD112and optionally external cameras102, sensor data from sensors90, and pose information from each HMD112. Console106may use the received data to render artificial reality content for display to each of users102via their respective HMDs112. As discussed above, in some examples, the artificial reality content may comprise a mixture of real-world imagery and virtual objects, e.g., mixed reality and/or augmented reality. In other examples, artificial reality content may be, e.g., a video conferencing application, a navigation application, an educational application, simulation, gaming applications, or other types of applications that implement artificial reality.

Additionally, console106may maintain a shared map indicating the positioning of each of the users102with respect to other users of the artificial reality system100. The shared map may be generated and maintained based on information received from each participating user's HMD112. In accordance with the shared map, current poses and body position information, the artificial reality application renders the artificial reality content.

The example artificial reality systems100,130illustrated inFIGS. 1A and 1Brepresent a use case in which a physical environment120includes users102A,102B and102C that are participating in a training exercise for first responders. The physical environment120for this example includes an accident victim126. Example artificial reality content corresponding to this training example and the physical environment120depicted inFIGS. 1A and 1Bare presented with respect toFIG. 2discussed below. Artificial reality content for a multi-player game can also be generated. Other example use cases are possible and within the scope of the disclosure.

FIG. 1Cillustrates further aspects of the artificial reality system100,130described inFIGS. 1A and 1B. As discussed above, artificial reality system100receives body position information from a user's HMD112. In some aspects, the body position information can be skeletal positioning information. The skeletal positioning information can include 2D or 3D positioning information for skeletal points within a field-of-view of an HMD112. The skeletal positioning information can include information for both the user of the HMD112and for other users in the physical environment120. In the example illustrated inFIG. 1C, HMD112A of user102A has an image sensor138A having a field-of-view indicated by dashed lines116A. Various skeletal points118on user102B are within the field-of-view116A of image sensors138A of HMD112A. As an example, skeletal points118for user102B that are detected by image sensor138A of HMD112A may include the shoulders, elbows, wrist joints, finger joints, fingertips, etc. of user102B.

In some aspects, the body position information can be body segment position information. The body segment information can include 2D or 3D positioning information for body segments within a field-of-view of an HMD112. A body segment can be a region of the body, for example, a head, trunk, arms, forearms, hands, thighs, legs, and feet of a participant. The body segment position information can include information for both the user of the HMD112and for other users in the physical environment120.

In some aspects, the body position information may include both skeletal position information and body segment position information.

The body position information may be shared by HMD112A with other co-located participating HMDs112(e.g., HMDs112B and112C) for use by artificial reality applications in creating artificial reality content for HMDs112B and112C. As an example, the artificial reality content displayed to user102B via HMD112B can be generated using information on skeletal points118and/or body segments122shared by HMD112A, some of which may not be detectable by image capture devices138B or other sensors of HMD112B (e.g., due to occlusion by user112B's body portions or other objects).

The pose and body position information provided by other users' HMDs can be used to fill in and refine the skeletal position information determined by the first HMD. For example, the body position information received from HMDs112B and112C can be used by HMD112A to both fill in and refine the body position information determined on HMD112A from the image data from HMD112A. Each HMD112can independently determine pose and body position information based on 2D or 3D body position information received from other HMDs of co-located participants. For example, HMD112A may not be able to determine the pose and body position information of other participants based solely on the data acquired by HMD112A's own image capture devices. However, using information received from other co-located HMDs (e.g., HMDs112B and112C), HMD112A can determine pose and body position information for itself and other co-located users. The body position information determined by HMD112A and the body position information received from other co-located HMDs need not be complete in order for HMD112A to determine pose and body position information for itself and co-located participants. Instead, HMD112A can use 2D or 3D full or partial body position information determined by itself combined with 2D or 3D full or partial body position information received from other HMDs to accurately determine 3D pose and body position information for itself and co-located users. HMDs112B and112C can perform similar operations to use 2D or 3D full or partial body position information received from other participating HMDs to accurately determine 3D pose and body position information for themselves and other co-located participants. Accordingly, the techniques of the disclosure provide specific technical improvements to the computer-related field of rendering and displaying content by an artificial reality system. For example, artificial reality systems as described herein may provide a high-quality artificial reality experience to a user, such as user102, of the artificial reality application by generating and rendering accurate pose and positioning information for a user102even when some pose and/or skeletal positioning information is not locally available to an HMD112of the user102.

FIG. 2illustrates example artificial reality content200that may correspond to the example physical environment120ofFIGS. 1A and 1B. Reference will be made to elements ofFIGS. 1A and 1Bin order to assist in the description of aspects illustrated inFIG. 2. As discussed above, artificial reality system100,130is configured to generate artificial reality content200based at least in part on one or more physical objects within physical environment120. Each of HMDs112is configured to render and output artificial reality content from the point-of-view of the respective HMD112. For example, artificial reality content200ofFIG. 2is generated and rendered from the point-of-view of HMD112C of user102C fromFIGS. 1A and 1B, an observer of a training exercise. Thus, artificial reality content200may include one or more graphical or virtual objects, some or all of which may correspond to physical objects within physical environment120. In the example illustrated inFIG. 2, artificial reality content200may include avatars202A,202B, and202C (collectively, “avatars202”) that correspond to the physical bodies of users102A,102B and102C, respectively, fromFIGS. 1A and 1B. Artificial reality system100,130may be configured to generate and render each of avatars202to have a same or similar pose or orientation as the pose of the physical body of its corresponding user102. For example, as shown inFIG. 2, avatars202A and202B are depicted as kneeling on one knee, corresponding to users102A and102B, respectively, who are also each kneeling on one knee.

For various reasons, any one of HMDs112may not be able to accurately render one or more virtual objects or avatars202from its own point of view. In one example, the image frame rendered by an artificial reality application on a display screen of a user's HMD may contain more image data than what is within the field-of-view116of a particular image-capture device138. Accordingly, HMD112may fail to identify some or all of a physical body of a user102that is not captured by its respective image-capture device138. For example, as shown inFIG. 1A, the right knee110A and right hand114A of user102A do not fall within the field of view116A of image-capture device138A, however, these physical elements may still fall within the image displayed on user102A's HMD112A. Accordingly, HMD112A may be unable to accurately render virtual representations of these physical body parts for display on the display screen.

In other examples, even if a physical object falls within the field-of-view116of a particular image-capture device138, artificial reality system100,130may experience difficulty identifying the physical object, thereby preventing artificial reality system100from rendering and outputting a corresponding virtual object. For example, even if right knee110A of user102A was within the field-of-view116A of image-capture device138A, right knee110A would appear in the captured 2D imagery as a relatively large, rounded object with no identifying features that would enable image-recognition software to identify it as a knee of user102. Accordingly, HMD112A may unable to accurately render a virtual representation of this body part for display on the display screen.

In other examples, part or all of the body of a user102may be sufficiently occluded from a specific image capture device138, such as by clothing (particularly loose or baggy clothing) another body part, or other interfering physical object, such that an HMD112or console106of artificial reality system100may be unable to identify the relative pose of the respective body part, or even the presence of the respective body part itself, and accordingly is unable to render a corresponding avatar202having the same pose.

In some examples in accordance with this disclosure, artificial reality system100,130is configured to perform pose tracking and body position tracking for one or more of users102, where some or all of an individual user's pose or body position is not trackable by a first user's own HMD. Other users' HMDs can provide pose and body position information to fill in the missing information and further refine existing information thereby facilitating accurate rendering of the pose and body positioning of co-located users. Thus, artificial reality system100can accurately generate artificial reality content200having avatars that are virtual representations202of users102in the same or similar poses as the corresponding users' physical bodies.

An HMD112or console106of artificial reality system100,130is configured to receive 2D or 3D image data from image-capture devices138within artificial reality system100,130. Based on the received 2D or 3D image data, the HMDs112or console106of artificial reality system100,130can be configured to determine (e.g., estimate) the relative positions of each of image-capture devices138via inside-out tracking. Inside-out tracking refers to tracking objects outside an HMD112using cameras or other sensors inside or on the HMD112. For example, artificial reality system100is configured to identify two or more objects within the received image data, and based on the relative orientations of the two or more objects with respect to one another in the various images, estimate the relative distance between and/or orientations of image-capture devices138with respect to one another.

In the example depicted inFIG. 1A, HMD112A or console106of artificial reality system100can be configured to receive pose and body position information from HMDs112B and112C respectively. HMD112A (or console106), can use the received pose and body position information, along with pose and body information that is locally available to HMD112A, to determine pose and body position information of limbs or other body parts of their own bodies and the physical body of user102A. In the example illustrated inFIG. 1B, console106of artificial reality system130can be configured to receive image data from image capture devices138A,138B and138C, and, optionally, external cameras102and identify from the images, pose information and body position information of limbs or other body parts of users102A,102B and102C.

Based on the identified physical body parts of user102A in the image data, artificial reality system100,130is further configured to determine a three-dimensional (3D) pose and body positioning of user102A. For example, based on a relative orientation of one or more identified limbs of user102A, artificial reality system100,130may be configured to construct an avatar202A having limbs in the same relative orientation as the limbs of user102A.

Artificial reality system100,130may repeat this process for each of users102within physical environment120. For example, in the scenario depicted inFIGS. 1A and 1B, HMD112B of artificial reality system100,130may receive pose and body position information determined from image data provided by image-capture devices138A and138C depicting user102B. Similarly, HMD132C of artificial reality system100may receive pose and body position information determined from image data provided by image capture devices138A and138B depicting user102C. Additionally, HMD112A of artificial reality system100may determine or identify pose and body position information for users102B and102C based on the received image data from image capture devices132A.

Once an HMD112of artificial reality system100has identified pose and body position information for each of users102, HMD112may transfer (e.g., wirelessly broadcast) the pose data and body position information to each of the other co-located HMDs112in the artificial reality system100. Based on the received pose data and body position information, each HMD112may generate and display artificial reality content200depicting a virtual or augmented reality environment from the point-of-view of the respective HMD112. Artificial reality content200may include avatars202corresponding to the users' physical bodies102that are otherwise obscured, occluded, unidentifiable or out-of-view of the image-capture device138of the user's respective HMD112. For example, as shown inFIG. 2, artificial reality content200includes a virtual knee210and virtual hand214, even though these virtual objects' respective physical counterparts110A,114A are out-of-view of image-capture device138A (FIGS. 1A and 1B).

FIG. 3Ais an illustration depicting an example HMD112configured to operate in accordance with the techniques of the disclosure. HMD112ofFIG. 3Amay be an example of any of HMDs112ofFIGS. 1A and 1B. HMD112may operate as a stand-alone, mobile artificial realty system configured to implement the techniques described herein or may be part of an artificial reality system, such as artificial reality systems100,130ofFIGS. 1A, 1B.

In this example, HMD112includes a front rigid body and a band to secure HMD112to a user. In addition, HMD112includes an interior-facing electronic display303configured to present artificial reality content to the user. Electronic display303may be any suitable display technology, such as liquid crystal displays (LCD), quantum dot display, dot matrix displays, light emitting diode (LED) displays, organic light-emitting diode (OLED) displays, cathode ray tube (CRT) displays, e-ink, or monochrome, color, or any other type of display capable of generating visual output. In some examples, the electronic display is a stereoscopic display for providing separate images to each eye of the user. In some examples, the known orientation and position of display303relative to the front rigid body of HMD112is used as a frame of reference, also referred to as a local origin, when tracking the position and orientation of HMD112for rendering artificial reality content according to a current viewing perspective of HMD112and the user. In other examples, HMD112may take the form of other wearable head mounted displays, such as glasses or goggles.

As further shown inFIG. 3A, in this example, HMD112further includes one or more motion sensors306, such as one or more accelerometers (also referred to as inertial measurement units or “IMUs”) that output data indicative of current acceleration of HMD112, GPS sensors that output data indicative of a location of HMD112, radar or sonar that output data indicative of distances of HMD112from various objects, or other sensors that provide indications of a location or orientation of HMD112or other objects within a physical environment. Moreover, HMD112may include integrated image capture devices138A and138B (collectively, “image capture devices138”), such as video cameras, still cameras, IR scanners, UV scanners, laser scanners, Doppler radar scanners, depth scanners, or the like, configured to output image data representative of the physical environment. In some aspects, the image capture devices138can capture image data from a visible spectrum and an invisible spectrum of the electromagnetic spectrum (e.g., IR light). The image capture devices138may include one or more image capture devices that capture image data from the visible spectrum and one or more separate image capture devices that capture image data from the invisible spectrum, or these may be combined in the same one or more image capture devices. More specifically, image capture devices138capture image data representative of objects in the physical environment that are within a field of view130A,130B of image capture devices138, which typically corresponds with the viewing perspective of HMD112. HMD112includes an internal control unit310, which may include an internal power source and one or more printed-circuit boards having one or more processors, memory, and hardware to provide an operating environment for executing programmable operations to process sensed data and present artificial reality content on display303.

In one example, in accordance with the techniques described herein, control unit310is configured to, based on the sensed image data, determine body position information for a user of HMD112and for other co-located users within a physical environment120(FIGS. 1A-1C). When within the field of view of the image capture devices138, the control unit310can detect body position information within the image data and use the body position information determined locally along with body position information received from other HMDs112and other sensor information to determine pose and body position information for the user of HMD112and other co-located users.

FIG. 3Bis an illustration depicting an example HMD112, in accordance with techniques of the disclosure. As shown inFIG. 3B, HMD112may take the form of glasses. HMD112ofFIG. 3Bmay be an example of any of HMDs112ofFIGS. 1A and 1B. HMD112may be part of an artificial reality system, such as artificial reality systems100,130ofFIG. 1A, 1B, or may operate as a stand-alone, mobile artificial realty system configured to implement the techniques described herein.

In this example, HMD112are glasses comprising a front frame including a bridge to allow the HMD112to rest on a user's nose and temples (or “arms”) that extend over the user's ears to secure HMD112to the user. In addition, HMD112ofFIG. 3Bincludes interior-facing electronic displays303A and303B (collectively, “electronic displays303”) configured to present artificial reality content to the user. Electronic displays303may be any suitable display technology, such as liquid crystal displays (LCD), quantum dot display, dot matrix displays, light emitting diode (LED) displays, organic light-emitting diode (OLED) displays, cathode ray tube (CRT) displays, e-ink, or monochrome, color, or any other type of display capable of generating visual output. In the example shown inFIG. 3B, electronic displays303form a stereoscopic display for providing separate images to each eye of the user. In some examples, the known orientation and position of display303relative to the front frame of HMD112is used as a frame of reference, also referred to as a local origin, when tracking the position and orientation of HMD112for rendering artificial reality content according to a current viewing perspective of HMD112and the user.

As further shown in the example illustrated inFIG. 3B, HMD112further includes one or more motion sensors306, such as one or more accelerometers (also referred to as inertial measurement units or “IMUS”) that output data indicative of current acceleration of HMD112, GPS sensors that output data indicative of a location of HMD112, radar or sonar that output data indicative of distances of HMD112from various objects, or other sensors that provide indications of a location or orientation of HMD112or other objects within a physical environment. Moreover, HMD112may include integrated image capture devices138A and138B (collectively, “image capture devices138”), such as video cameras, laser scanners, Doppler radar scanners, depth scanners, or the like, configured to output image data representative of the physical environment. HMD112includes an internal control unit310, which may include an internal power source and one or more printed-circuit boards having one or more processors, memory, and hardware to provide an operating environment for executing programmable operations to process sensed data and present artificial reality content on display303.

FIG. 4is a block diagram depicting an example in which pose tracking including body position tracking for co-located users is performed by an example instance of the HMD112A of the artificial reality system100ofFIG. 1Ain accordance with the techniques of the disclosure. In the example ofFIG. 4, HMD112A performs pose tracking including body position tracking for HMD112A in accordance with the techniques described herein based on sensed data, such as motion data and image data received from an HMD112A of the user and from HMDs112B and112C of other co-located users.

In this example, HMD112A includes one or more processors402and memory404that, in some examples, provide a computer platform for executing an operating system405, which may be an embedded, real-time multitasking operating system, for instance, or other type of operating system. In turn, operating system405provides a multitasking operating environment for executing one or more software components417. Processors402are coupled to one or more I/O interfaces415, which provide I/O interfaces for communicating with devices such as a keyboard, game controllers, display devices, image capture devices, other HMDs, and the like. Moreover, the one or more I/O interfaces415may include one or more wired or wireless network interface controllers (NICs) for communicating with a network, such as network104. Additionally, processor(s)402are coupled to electronic display303, motion sensors306, and image capture devices138. In some examples, processors402and memory404may be separate, discrete components. In other examples, memory404may be on-chip memory co-located with processors402within a single integrated circuit.

Software applications417of HMD112A operate to provide an overall artificial reality application. In this example, software applications417include application engine440, rendering engine422, pose tracker426, and mapping engine446.

In general, application engine440includes functionality to provide and present an artificial reality application, e.g., a teleconference application, a gaming application, a navigation application, an educational application, training or simulation applications, and the like. Application engine440may include, for example, one or more software packages, software libraries, hardware drivers, and/or Application Programming Interfaces (APIs) for implementing an artificial reality application on HMD112A. Responsive to control by application engine440, rendering engine422generates 3D artificial reality content for display to the user by application engine440of HMD112A.

Application engine440and rendering engine422construct the artificial content for display to user102in accordance with current pose of HMD112A and available body position information within a frame of reference, typically a viewing perspective of HMD112A, as determined by pose tracker426. Based on the current viewing perspective and on pose information and body positioning information received from other users' HMDs112B and112C, rendering engine422constructs the 3D, artificial reality content which may in some cases be overlaid, at least in part, upon the real-world 3D environment of user102. During this process, pose tracker426operates on data received from HMD112A such as image data from image capture devices138and motion sensors106, such as movement information and user commands to capture 3D information within the real world environment, such as motion by user102, and/or feature tracking information with respect to user102. Pose tracker426also operates on similar data received from other co-located users' similarly configured HMDs112B and112C that capture 2D and 3D information with the image capture devices and sensors on the other co-located users' HMDs112B and112C. Based on both the data gathered on the local HMD112A and the data received from other co-located users' HMDs112B and112C, pose tracker426determines a current pose and body position for the user of HMD112A within the frame of reference of HMD112A and, in accordance with the current pose and skeletal positioning information, constructs the artificial reality content for display to user102.

Pose tracker426includes a body position tracker452and a pose integrator442. Body position tracker452analyzes image data received via image capture devices138and other sensor data to determine body positioning of co-located users that appear in the image data. In some aspects, body position tracker452can process the image data through a machine learning model that has been trained to recognize various body positions. For example, in some aspects the machine learning model may be trained to recognize skeletal positions, including joints such as knees, wrists, shoulders etc. In some aspects, the machine learning model may be trained to recognize body segments such as head, neck, torso, arm, forearm, thigh, leg and foot segments. In some aspects, the machine learning model may be trained to recognized both skeletal points and body segments. The body position tracker452can analyze the image data to determine body position information for both the user of HMD112A and for other co-located users that appear in the image data. The body position information can include a confidence level, perhaps assigned by the machine learning model, for the body positions detected by the body position tracker452.

Pose integrator442integrates pose and body position information as determined on HMD112A with pose and body position information as determined on other co-located HMDs (e.g., HMD112B and112C). Pose integrator542can receive 2D or 3D pose and body position information from co-located HMDs via network104. The 2D or 3D pose and body position information may be partial information, i.e., incomplete information regarding pose and body positions of the co-located users. Pose integrator442can then integrate the locally determined pose and body position information along with pose and body position information received from co-located HMDs to solve accurate pose and body position information for all co-located users within a field of view of HMD112A. In some aspects, the accuracy of the pose and body position may be increased due to the availability of body position information determined from image data taken from different angles of the various co-located users. Further, the accuracy of the pose and body position information may be enhanced due to the availability of body position information that can be used to fill-in or augment body position information that is not locally available to HMD112A. For example, HMD112A can use pose and body position information received from other HMDs (e.g., HMD112B and/or HMD112C) to fill in or augment pose and body position information determined from image data and other data local to HMD112A.

The pose integrator442can use various data to solve the pose and body positions of the co-located users. For example, in some aspects, the pose integrator442may use the confidence levels associated with the body position information in order to determine which body position data to use, or to weight the influence of the body position data.

In some aspects, HMD112A may include a mapping engine446configured to generate mapping data448of a physical 3D environment using mapping information received from co-located HMDs112. Mapping engine446may receive the mapping information in the form of images captured by image capture devices138at local poses of HMDs112and/or tracking information for HMDs112, for example. Mapping engine446may process the image data to identify map points for determining topographies of the scenes in the images and use the map points to generate map data448that is descriptive of an area of the physical 3D environment in which HMD112A is operating. Mapping engine446may progressively generate a map for an area in which co-located HMDs112are operating over time.

Further details on the operation of pose tracker426, including skeletal position tracker452and pose integrator442are provided below with respect toFIG. 6.

FIG. 5is a block diagram showing example implementations in which pose tracking and body position tracking for co-located users is performed by example instances of the console and the HMD of the artificial reality systems ofFIG. 1B. In the example ofFIG. 5, console106performs pose tracking and rendering for HMDs112A,112B and112C in accordance with the techniques described herein based on sensed data, such as image and motion data received from HMDs112A,112B and112C.

In this example, similar toFIG. 4, HMD112A includes one or more processors402and memory404that, in some examples, provide a computer platform for executing an operating system405, which may be an embedded, real-time multitasking operating system, for instance, or other type of operating system. In turn, operating system405provides a multitasking operating environment for executing one or more software components527. Moreover, processor(s)402are coupled to electronic display303, motion sensors306, and image capture devices138. HMDs112B and112C may be configured similarly to HMD112A.

In general, console106is a computing device that processes image and skeletal position information received from cameras102(FIG. 1B) and/or HMDs112to perform pose tracking, and content rendering for HMDs112. In some examples, console106is a single computing device, such as a workstation, a desktop computer, a laptop, or gaming system. In some examples, at least a portion of console106, such as processors512and/or memory514, may be distributed across a cloud computing system, a data center, or across a network, such as the Internet, another public or private communications network, for instance, broadband, cellular, Wi-Fi, and/or other types of communication networks for transmitting data between computing systems, servers, and computing devices.

In the example ofFIG. 5, console106includes one or more processors512and memory514that, in some examples, provide a computer platform for executing an operating system516, which may be an embedded, real-time multitasking operating system, for instance, or other type of operating system. In turn, operating system516provides a multitasking operating environment for executing one or more software components517. Processors512are coupled to one or more I/O interfaces515, which provide I/O interfaces for communicating with external devices, such as a keyboard, game controllers, display devices, image capture devices, HMDs, and the like. Moreover, the one or more I/O interfaces515may include one or more wired or wireless network interface controllers (NICs) for communicating with a network, such as network104. Each of processors402,512may comprise any one or more of a multi-core processor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or equivalent discrete or integrated logic circuitry. Memory404,514may comprise any form of memory for storing data and executable software instructions, such as random-access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), and flash memory.

Software applications517of console106operate to provide an overall artificial reality application. In this example, software applications517include application engine520, rendering engine522, pose tracker526, and mapping engine546.

In general, application engine520includes functionality to provide and present an artificial reality application, e.g., a teleconference application, a gaming application, a navigation application, an educational application, training or simulation applications, and the like. Application engine520may include, for example, one or more software packages, software libraries, hardware drivers, and/or Application Program Interfaces (APIs) for implementing an artificial reality application on console106. Responsive to control by application engine520, rendering engine522generates 3D artificial reality content for display to the user by application engines440of HMDs112(e.g., HMD112A,112B and112C).

Application engine520and rendering engine522construct the artificial content for display to each of users102in accordance with current pose information for the user's respective HMD112and body position information within a frame of reference, typically a viewing perspective of the respective co-located HMD112, as determined by pose tracker526. Based on the current viewing perspective, rendering engine522constructs the 3D, artificial reality content which may in some cases be overlaid, at least in part, upon the real-world 3D environment of users102. During this process, pose tracker526operates on sensed data received from each respective co-located HMD112, such as image data from sensors on an HMD112, and, in some examples, data from any external sensors90(FIG. 1B), such as external cameras, to capture 3D information within the real world environment, such as motion by user102, and/or body position tracking information with respect to a user102. Based on the sensed data, pose tracker526determines a current pose for each respective co-located HMD112within the frame of reference of the respective HMD112and, in accordance with the current poses and body position information, constructs the artificial reality content for communication, via the one or more I/O interfaces415,515, to HMDs112for display to users102.

Similar to pose tracker426described above with respect toFIG. 4, pose tracker526includes body position tracker552and a pose integrator542. Body position tracker552analyzes image data received via image capture devices and other sensor data to determine body positioning of co-located users that appear in the image data. In some aspects, body position tracker552can process the image data through a machine learning model that has been trained to recognize various body positions, including skeletal positions and body segment positions. The body position tracker552can analyze the image data received from the respective co-located HMDs to determine body position data for co-located users102that appear in the image data. The body position data can include a confidence level, perhaps assigned by the machine learning model, for the body positions detected by the body position tracker552.

Pose integrator542integrates the pose and body position information position information as determined by the pose tracker526and body position tracker552. Pose integrator542can then integrate the pose and body position information for each of the co-located HMDs to solve accurate pose and body position information for all co-located users within a field of view of each respective HMD112. In some aspects, the accuracy of the pose and body position may be increased due to the availability of image data taken from different angles of the various co-located users.

The pose integrator542can use various data to solve the pose and skeletal positions of the co-located users. For example, in some aspects, the pose integrator542may use the confidence levels associated with the body positions in order to determine which skeletal position data to use, or to weight the influence of the skeletal position data.

In some aspects, console106may include a mapping engine546configured to generate mapping data448of a physical 3D environment using mapping information received from co-located HMDs112. Mapping engine546may receive the mapping information in the form of images captured by image capture devices138of HMDs112and/or tracking information for HMDs112, for example. Mapping engine546may process the images to identify map points for determining topographies of the scenes in the images and use the map points to generate map data448that is descriptive of an area of the physical 3D environment in which the co-located HMDs112are operating. Mapping engine546may progressively generate a map for an area in which co-located HMDs112are operating over time.

Further details on the operation of pose tracker526, including skeletal position tracker552and pose integrator542are provided below with respect toFIG. 7.

FIG. 6is a flowchart600illustrating example operations of a method for determining pose and body positions for co-located users of HMDs in accordance with aspects of the disclosure. Some of the example operations described in flowchart600can be performed periodically or in response to an event. For example, the example operations can be performed as part of a response to a display frame generation event where the event causes an artificial reality system to render a display frame for presentation on HMD112. The operations described below can be performed by each HMD112of co-located users that are participating in a multi-user artificial reality experience that is provided, at least in part, by an application engine on each co-located user's respective HMD112. The operations below are described from the perspective of the HMD112of one of the co-located users.

An application engine on an HMD112can initialize a multi-user artificial reality application on an HMD112(602). In some aspects, if the HMD112is the first HMD to execute the application, the application may establish a wireless network. For example, the application may establish an ad hoc wireless network for use in communicating with the HMDs112of other co-located users that are participating in the application. If the HMD112is not the first HMD to execute the application, the HMD may join an already established network. In some aspects, application initialization procedures can also include obtaining permission from the user to share pose and body position information with the HMDs of other co-located users of the application. Further initialization procedures can include a mapping engine446initializing map data448with an estimate of the user's position and the position of other co-located users.

A pose tracker426of the HMD112can determine 2D or 3D pose and body position information for the user and co-located users that appear in the image data and other sensor data received from one or more on-board image capture devices138and perhaps other sensors of the HMD112(604). In some aspects, the body positions of the users appearing in the image data can be determined by passing the image data through a machine learning model that has been trained to recognize various body positions such as skeletal positions (e.g., joints such as elbows, wrists, shoulders, knees, hips etc.) and/or body segments (e.g., head, neck, trunk, arm, forearm, thigh, leg etc.). The body position information may be associated with confidence values for each joint, segment or other position that indicates the machine learning model's confidence in the position estimates. Other methods of body position recognition are possible and within the scope of the disclosure. In some aspects, the body positions are determined without the aid of markers or other devices placed on the joints, segments or other body parts used by conventional systems to identify body positions.

The HMD112can then determine the user's position in a joint mapping space with other co-located users (606). For example, a mapping engine446may use image data and other sensor data to determine relative positions of the user and other co-located users. The HMD112can then calibrate the user's position in the shared map with respect to the position of other co-located users.

The HMD112can transmit the pose and body position information as determined by HMD112for the users that are within the field-of-view of HMD112(and associated confidence values) over the network for use by other co-located HMDs participating in the artificial reality application (608). The transmitted pose and body position information may be 2D or 3D information. Further, the transmitted pose and body position information may be incomplete information. For example, body position information may be provided for joints or segments that are within the field-of-view of a respective HMD112, and may omit data with respect to joins or segments that are not within the field-of-view of the respective HMD112. The pose and body position information may be provided relative to a shared map. In some aspects, the pose and body position information may be a full set of information comprising relative or absolute positions within the shared map. In some aspects, only information that has changed from a previous frame is transmitted.

Similarly, the HMD112can receive pose and body position information from other co-located HMDs participating in the artificial reality application (610). The received pose and body position information may include pose and body position information for the user of HMD112as determined by the other co-located HMDs, and may include associated confidence levels representing the confidence that the co-located HMDs have in the values they provide. The body position information may be incomplete. For example, the body information received from an HMD112may only include information for joints and/or body segments that are within the field of view of the corresponding HMD112. The received pose and body position information can be 2D or 3D information.

A pose integrator442on the user's HMD112can integrate the locally determined pose and body position information with the pose and body position information received from other co-located users' HMDs112to determine solved pose and body position information for the co-located users (612). The integration of the pose and body position information can be performed in various ways. For example, in some aspects, a confidence level of the pose and body position information received from other co-located HMDs112can be used to determine the solved pose and body position information for the user of HMD112performing the integration. For example, HMD112A may determine a solved body position information for user102A of HMD112A. In some aspects, the body position information with the highest confidence level can be used as the solved body position. In some aspects, the solved body position information can be based on a combination of the body position information received from the HMDs112of co-located users. For example, the confidence level of each HMD's112body position information can be used to weight the respective HMD's contribution to a solved body position. In some aspects, preference or increased weight may be given to body position information that is closer to the user's HMD112. In some aspects, HMD112A may also determine solved body position information for the other co-located users (e.g., users of HMDs112B and112C). HMD112A may use any of the aforementioned techniques to determine the solved body position information for other users.

Various techniques can be used to avoid overtaxing processing resources, especially in the case that there are many users participating in a multi-user artificial reality application. In some aspects, some data may be filtered out during the pose integration process. For example, pose and body position information received from users that are positioned past a predetermined or configurable threshold may be ignored. This may be desirable because data obtained from HMDs that are close may be more reliable than data from HMDs that are farther away. Similarly, in some aspects, pose and body position information received from the top n closest users may be used while information from users that are not in the top n closest users may be ignored. The value for n may vary depending on the number of co-located users present. A further technique can be to perform the integration operations only when a change in data values exceeds a predetermined or configurable threshold.

The HMD112then transmits the solved 3D pose and body position information for the user and co-located users to the other co-located HMDs via network104(614). The HMD112may also receive solved 3D pose and body position information from other co-located HMDs (616).

A rendering engine422on HMD112can render artificial reality content based on the solved pose and body position information (618). For example, the rendering engine can map avatar characteristics onto the solved body position information for the associated user so that the user's avatar is posed in the same position as determined by the solved body position information associated with the user. The avatar characteristics can include clothing, body shape, skin tone and other characteristics.

FIG. 7is a flowchart700illustrating example operations of a method for determining pose and skeletal positions for co-located users of HMDs in accordance with further aspects of the disclosure. As with the flowchart600ofFIG. 6, some of the example operations described in flowchart600can be performed periodically or in response to an event. For example, the example operations can be performed as part of a response to a display frame generation event where the event causes an artificial reality system to render a display frame for presentation on HMD112. Some or all of the operations described below can be performed by a console106that communicates with HMDs112of co-located users that are participating in a multi-user artificial reality experience that is provided, at least in part, by an application engine on the console106and each co-located user's respective HMD112.

An application engine on console106can initialize a multi-user artificial reality application (702). In some aspects, the console106may establish a wireless network. For example, the application may establish an ad hoc wireless network for use in communicating with the HMDs112of other co-located users that are participating in the application. In some aspects, application initialization procedures can also include obtaining permission from the users to share pose and skeletal position information with the HMDs of other co-located users of the application. Further initialization procedures can include a mapping engine546initializing map data448with current estimates of the positions of participating co-located users.

A pose tracker526of console106can determine pose and body position information for co-located users (704). In some aspects, the console106may receive image data from each of the HMDs112of the co-located users and use the image data along with other data received from the HMDs112(if any) to determine poses and body position information for the co-located users that appear in the image data and other sensor data. In some aspects, the body positions of the users appearing in the image data can be determined by passing the image data through a machine learning model that has been trained to recognize various body positions such as skeletal points (e.g., joints such as elbows, wrists, shoulders, knees, hips etc.) and/or body segments (e.g., head, neck, torso, arm, forearm, hand, thigh, leg, foot etc.). The body position information may be associated with confidence values for each joint, body segment or other position that indicates the machine learning model's confidence in the position estimates. Other methods of body position recognition are possible and within the scope of the disclosure. In some aspects, the body position information is determined by pose tracker526without the aid of markers or other devices placed on the joints or other body parts used by conventional systems to identify body positions.

In some aspects, some or all of the HMDs112of the co-located users may determine their own pose and body position information. In such aspects, the console may receive the pose and body position information from the respective HMDs112and may not receive image data from some or all of the co-located HMDs112. The pose and body position information may be 2D or 3D information.

The console106can determine the co-located users' positions in a joint mapping space (706). For example, a mapping engine546may use image data, pose data and/or other sensor data received from HMDs112to determine relative positions of the co-located users. The console106can then calibrate the user's position in the shared map with respect to the position of other co-located users.

A pose integrator542on console106can integrate the pose and body position information determined based on the image data, pose information, body positioning information and/or sensor data from the co-located users' HMDs112to determine solved pose and body position information for the co-located users (708). The integration of the pose and body position information can be performed as discussed above with respect toFIG. 6. For example, in some aspects, the confidence level of the pose and body position information can be used to determine the solved pose and body position information. In some aspects, the body position information with the highest confidence level can be used as the solved body position information. In some aspects, the body position information received from co-located users' HMDs112can be combined using the confidence level to weight the information from the respective HMD112used to determine the solved body position information.

A rendering engine522on console106can render AR content based on the solved pose and body position information (710). For example, the rendering engine522can map avatar characteristics onto the solved body position information for the associated user so that the user's avatar is posed in the same position as determined by the solved body position information associated with the user. The avatar characteristics can include clothing, body shape, skin tone and other characteristics.