Late-Stage Warp of Dynamic-Anchored Content

In one implementation, a method of displaying an image is performed by a device including an image sensor, a display, one or more processors, and non-transitory memory. The method includes capturing, using the image sensor, an image of an object in a physical environment. The method includes obtaining, based on the image, a first predicted object pose of the object in the physical environment at a display time. The method includes rendering virtual content based on the first predicted object pose. The method includes obtaining a second predicted object pose of the object in the physical environment at the display time. The method includes warping the virtual content based on the second predicted object pose. The method includes displaying, on the display at the display time, the warped virtual content.

TECHNICAL FIELD

The present disclosure generally relates to systems, methods, and devices of rendering virtual content anchored to a real-world object.

BACKGROUND

In various implementations, an extended reality (XR) environment presented by an electronic device including a display includes world-locked content (which moves on the display when the pose of the electronic device changes) and display-locked content (which maintains its location on the display when the pose of the electronic device changes). Further, some world-locked content is anchored to a static object, such as a virtual cylinder displayed on a real table and some world-locked content is anchored to a dynamic object, such as a virtual mask displayed on the face of a person.

SUMMARY

Various implementations disclosed herein include devices, systems, and methods for displaying an image. In various implementations, the method is performed by a device including an image sensor, a display, one or more processors, and non-transitory memory. The method includes capturing, using the image sensor, an image of an object in a physical environment. The method includes obtaining, based on the image, a first predicted object pose of the object in the physical environment at a display time. The method includes rendering virtual content based on the first predicted object pose. The method includes obtaining a second predicted object pose of the object in the physical environment at the display time. The method includes warping the virtual content based on the second predicted object pose. The method includes displaying, on the display at the display time, the warped virtual content.

Various implementations disclosed herein include devices, systems, and methods for displaying an image. In various implementations, the method is performed by a device including a display, one or more processors, and non-transitory memory. The method includes obtaining a first predicted device pose of the device at a display time and a first predicted object pose of an object at the display time. The method includes rendering virtual content based on the first predicted device pose and the first predicted object pose. The method includes obtaining a second predicted device pose of the device at the display time and a second predicted object pose of the object at the display time. The method includes warping the virtual content based on the second predicted device pose and the second predicted object pose. The method includes displaying, on the display at the display time, the warped virtual content.

Various implementations disclosed herein include devices, systems, and methods for extrapolating an image. In various implementations, the method is performed by a device including an image sensor, a display, one or more processors, and non-transitory memory. The method includes capturing, using the image sensor, an image of an object in a physical environment. The method includes obtaining, based on the image, a first predicted object pose of the object at a first display time. The method includes rendering virtual content based on the first predicted object pose. The method includes displaying, at the first display time, the virtual content. The method includes obtaining a second predicted object pose of the object at a second display time subsequent to the first display time. The method includes warping the virtual content based on the second predicted object pose. The method includes displaying, at the second display time, the warped virtual content.

Various implementations disclosed herein include devices, systems, and methods for extrapolating an image. In various implementations, the method is performed by a device including a display, one or more processors, and non-transitory memory. The method includes obtaining a first predicted device pose of the device at a first display time and a first predicted object pose of an object at the first display time. The method includes rendering virtual content based on the first predicted device pose and the first predicted object pose. The method includes displaying, on the display at the first display time, the virtual content. The method includes obtaining a second predicted device pose of the device at a second display time and a second predicted object pose of the object at the second display time. The method includes warping the virtual content based on the second predicted device pose and the second predicted object pose. The method includes displaying, on the display at the second display time, the warped virtual content.

DESCRIPTION

As noted above, in various implementations, some world-locked content is anchored to a dynamic object, such as a virtual mask displayed on the face of a person. Determining the location on the display to display the dynamic-anchored world-locked content is based on the pose of the dynamic object. Accurately predicting this pose is important to registration of the virtual content. After rendering has begun, a more accurate pose may be available. Accordingly, in various implementations, the rendered content is shifted (or otherwise warped) based on the updated pose.

FIG.1is a block diagram of an example operating environment100in accordance with some implementations. While pertinent features are shown, those of ordinary skill in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. To that end, as a non-limiting example, the operating environment100includes a controller110and an electronic device120.

In some implementations, the controller110is configured to manage and coordinate an XR experience for the user. In some implementations, the controller110includes a suitable combination of software, firmware, and/or hardware. The controller110is described in greater detail below with respect toFIG.8. In some implementations, the controller110is a computing device that is local or remote relative to the physical environment105. For example, the controller110is a local server located within the physical environment105. In another example, the controller110is a remote server located outside of the physical environment105(e.g., a cloud server, central server, etc.). In some implementations, the controller110is communicatively coupled with the electronic device120via one or more wired or wireless communication channels144(e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller110is included within the enclosure of the electronic device120. In some implementations, the functionalities of the controller110are provided by and/or combined with the electronic device120.

In some implementations, the electronic device120is configured to provide the XR experience to the user. In some implementations, the electronic device120includes a suitable combination of software, firmware, and/or hardware. According to some implementations, the electronic device120presents, via a display122, XR content to the user while the user is physically present within the physical environment105that includes a table107within the field-of-view111of the electronic device120. As such, in some implementations, the user holds the electronic device120in his/her hand(s). In some implementations, while providing XR content, the electronic device120is configured to display an XR object (e.g., an XR cylinder109) and to enable video pass-through of the physical environment105(e.g., including a representation117of the table107) on a display122. The electronic device120is described in greater detail below with respect toFIG.9.

According to some implementations, the electronic device120provides an XR experience to the user while the user is virtually and/or physically present within the physical environment105.

In some implementations, the user wears the electronic device120on his/her head. For example, in some implementations, the electronic device includes a head-mounted system (HMS), head-mounted device (HMD), or head-mounted enclosure (HME). As such, the electronic device120includes one or more XR displays provided to display the XR content. For example, in various implementations, the electronic device120encloses the field-of-view of the user. In some implementations, the electronic device120is a handheld device (such as a smartphone or tablet) configured to present XR content, and rather than wearing the electronic device120, the user holds the device with a display directed towards the field-of-view of the user and a camera directed towards the physical environment105. In some implementations, the handheld device can be placed within an enclosure that can be worn on the head of the user. In some implementations, the electronic device120is replaced with an XR chamber, enclosure, or room configured to present XR content in which the user does not wear or hold the electronic device120.

FIGS.2A-2Eillustrate an XR environment200from the perspective of a user of an electronic device displayed, at least in part, by a display of the electronic device. In various implementations, the perspective of the user is from a location of an image sensor of the electronic device. For example, in various implementations, the electronic device is a handheld electronic device and the perspective of the user is from a location of the image sensor of the handheld electronic device directed towards the physical environment. In various implementations, the perspective of the user is from the location of a user of the electronic device. For example, in various implementations, the electronic device is a head-mounted electronic device and the perspective of the user is from a location of the user directed towards the physical environment, generally approximating the field-of-view of the user were the head-mounted electronic device not present. In various implementations, the perspective of the user is from the location of an avatar of the user. For example, in various implementations, the XR environment200is a virtual environment and the perspective of the user is from the location of an avatar or other representation of the user directed towards the virtual environment.

In a particular implementation, the electronic device is a head-mounted device with a transparent display. Thus, the XR environment includes a real environment (which the user views through the transparent display) with virtual objects (displayed by the transparent display) superimposed over the real environment. Further, being head-mounted, the pose of the electronic device (e.g., its position and/or orientation) is changed as the user changes the pose of the user's head.

FIGS.2A-2Eillustrate the XR environment200during a series of time periods. In various implementations, each time period is an instant, a fraction of a second, a few seconds, a few hours, a few days, or any length of time.

The XR environment200includes a plurality of objects, including one or more real objects (e.g., a hand211of the user, a table212, a person213other than the user, and a smartwatch214worn by the user) and one or more virtual objects (e.g., a virtual ring221, virtual flowers222, virtual glasses223, and a virtual clock224). In various implementations, certain objects (such as the virtual ring221, the virtual flowers222, and the virtual glasses223) are displayed at a location in the XR environment200, e.g., at a location defined by three coordinates in a three-dimensional (3D) XR coordinate system. Accordingly, when the electronic device moves in the XR environment200(e.g., changes either position and/or orientation), the objects are moved on the display of the electronic device, but retain their (possibly time-dependent) location in the XR environment200. Such virtual objects that, in response to motion of the electronic device, move on the display, but retain their position in the XR environment200are referred to as world-locked objects. In various implementations, certain virtual objects (such as the virtual clock224) are displayed at locations on the display such that when the electronic device moves in the XR environment200, the objects are stationary on the display on the electronic device. Such virtual objects that, in response to motion of the electronic device, retain their location on the display are referred to as head-locked objects or display-locked objects.

Some world-locked objects are anchored to stationary objects and may be referred to as static-anchored world-locked objects. For example, the virtual flowers222are displayed as being on the table212. However, some world-locked objects are anchored to dynamic (or moving) objects and may be referred to as dynamic-anchored world-locked objects. For example, the virtual ring221is displayed as being worn on the hand211of the user. Thus, as the hand211of the user moves in the XR environment200, the virtual ring221moves on the display. As another example, the virtual glasses223are displayed as being worn on the face of the person213. Thus, as the person213moves in the XR environment, the virtual glasses223move on the display.

FIG.2Aillustrates the XR environment200during a first time period. During the first time period, the electronic device displays the virtual ring221at a first ring location on the display corresponding to, for a first pose of the electronic device and a first pose of the hand211, a first ring location in the XR environment200, e.g., a location on the hand211while the hand211has the first pose. During the first time period, the electronic device displays the virtual flowers222at a first flowers location on the display corresponding to, for the first pose of the electronic device, a flowers location in the XR environment200, e.g., a location on the table212. During the first time period, the electronic device displays the virtual glasses223at a first glasses location on the display corresponding to, for the first pose of the electronic device and a first pose of the person213, a first glasses location in the XR environment200, e.g., a location on the person213while the person213has the first pose. During the first time period, the electronic device displays the virtual clock224and a fixed clock location on the display.

FIG.2Billustrates the XR environment200during a second time period subsequent to the first time period. During the second time period, as compared to the first time period, the pose of the electronic device has changed from the first pose to a second pose. In particular, the electronic device has moved to the left. During the second time period, as compared to the first time period, the pose of the hand211and the pose of the person213has not changed.

During the second time period, the electronic device displays the virtual ring221at a second ring location on the display corresponding to, for the second pose of the electronic device and the first pose of the hand211, the first hand location in the XR environment200. During the second time period, the electronic device displays the virtual flowers222at a second flowers location on the display corresponding to, for the second pose of the electronic device, the flowers location in the XR environment200. During the second time period, the electronic device displays the virtual glasses223at a second glasses location on the display corresponding to, for the second pose of the electronic device and the first pose of the person213, the first person location in the XR environment200. During the second time period, the electronic device displays the virtual clock224at the fixed clock location on the display.

Thus, during the second time period as compared to the first time period, the world-locked objects (e.g., the virtual ring221, the virtual flowers222, and the virtual glasses223) have moved to the right on the display due to the electronic device moving to the left in the XR environment200.

FIG.2Cillustrates the XR environment200during a third time period subsequent to the second time period. During the third time period, as compared to the second time period, the pose of the electronic device and the pose of the hand211has not changed. During the third time period, as compared to the second time period, the pose of the person213has changed from the first pose to a second pose. In particular, the person213has moved closer to the electronic device.

During the third time period, the electronic device displays the virtual ring221at a second ring location on the display corresponding to, for the second pose of the electronic device and the first pose of the hand211, the first hand location in the XR environment200. During the third time period, the electronic device displays the virtual flowers222at the second flowers location on the display corresponding to, for the second pose of the electronic device, the flowers location in the XR environment200. During the third time period, the electronic device displays the virtual glasses223at a third glasses location on the display corresponding to, for the second pose of the electronic device and the second pose of the person213, the second person location in the XR environment200. During the third time period, the electronic device displays the virtual clock224at the fixed clock location on the display.

Thus, during the third time period as compared to the second time period, the virtual glasses223have enlarged and moved down on the display due to the person213moving closer to the electronic device in the XR environment200.

FIG.2Dillustrates the XR environment200during a fourth time period subsequent to the third time period. During the fourth time period, as compared to the third time period, the pose of the electronic device and the pose of the person213has not changed. During the fourth time period, as compared to the third time period, the pose of the hand211has changed from the first pose to a second pose. In particular, the hand211has moved to the left in the XR environment200.

During the fourth time period, the electronic device displays the virtual ring221at a third ring location on the display corresponding to, for the second pose of the electronic device and the second pose of the hand211, the second hand location in the XR environment200. During the fourth time period, the electronic device displays the virtual flowers222at the second flowers location on the display corresponding to, for the second pose of the electronic device, the flowers location in the XR environment200. During the fourth time period, the electronic device displays the virtual glasses223at the third glasses location on the display corresponding to, for the second pose of the electronic device and the second pose of the person213, the second person location in the XR environment200. During the fourth time period, the electronic device displays the virtual clock224at the fixed clock location on the display.

Thus, during the fourth time period as compared to the third time period, the virtual ring221has moved left on the display due to the hand211moving left in the XR environment200.

FIG.2Eillustrates the XR environment200during a fifth time period subsequent to the fourth time period. During the fifth time period, as compared to the fourth time period, the pose of the person213has not changed. During the fifth time period, as compared to the fourth time period, the pose of the hand211has changed from the second pose to a third pose. In particular, the hand211has moved to the right in the XR environment200. Further, during the fifth time period, as compared to the fourth time period, the pose of the electronic device has changed from the second pose to a third pose. In particular, the electronic device has moved to the right in the XR environment200.

During the fifth time period, the electronic device displays the virtual ring221at a fourth ring location on the display corresponding to, for the third pose of the electronic device and the third pose of the hand211, the third hand location in the XR environment200. During the fifth time period, the electronic device displays the virtual flowers222at a third flowers location on the display corresponding to, for the third pose of the electronic device, the flowers location in the XR environment200. During the fifth time period, the electronic device displays the virtual glasses223at a fourth glasses location on the display corresponding to, for the third pose of the electronic device and the second pose of the person213, the second person location in the XR environment200. During the fifth time period, the electronic device displays the virtual clock224at the fixed clock location on the display.

Thus, during the fifth time period as compared to the fourth time period, the world-locked objects (e.g., the virtual ring221, the virtual flowers222, and the virtual glasses223) have moved to the left on the display due to the electronic device moving to the right in the XR environment200. However, the virtual ring221is not moved to the left as much as the other world-locked objects due to the hand211also moving to the right in the XR environment200.

FIG.3illustrates an electronic device300in accordance with some implementations. The electronic device300is present in a physical environment. The electronic device300includes a pose estimation module301that determines a current pose of the electronic device in the physical environment and determines a predicted pose of the electronic device at future times. Further, the pose estimation module201determines a current pose of one or more dynamic objects in the physical environment and determines a predicted pose of the one or more dynamic objects at future times. For example, in various implementations, the pose estimation module301determines a current pose and a predicted pose of a hand of a user. In various implementations, the pose estimation module301determines a current pose and predicted pose of the face of a person (other than the user). In various implementations, the pose estimation module301determines the current pose of the electronic device300based on camera-based pose tracking and/or IMU (inertial measurement unit) tracking. In various implementations, the pose estimation module301determines the predicted pose of the electronic device300by extrapolating previous motion of the electronic device300, e.g., current speed and/or acceleration of the electronic device300.

In various implementations, the pose estimation module301determines the current pose of the one or more objects based on an image of the physical environment. For example, in various implementations, the pose estimation module301determines the current pose of the hand of the user based on a hand tracking algorithm applied to an image of the physical environment. In various implementations, the pose estimation module301determines the current pose of the face of the person based on a face detection algorithm applied to an image of the physical environment. In various implementations, the pose estimation module301determines the current pose of the one or more objects based on an IMU attached to the object. For example, in various implementations, the pose estimation module301determines the current pose of the hand of the user based on IMU data received from a watch worn by the user. As another example, in various implementations, the pose estimation module301determines the current pose of the hand of the user based on IMU data received from a stylus held by the user. As another example, in various implementations, the pose estimation module301determines the current pose of the face of the person based on IMU data received from a head-mounted device worn by the person. In various implementations, the pose estimation module301determines the predicted pose of the one or more objects by extrapolating previous motion of the one or more objects, e.g., current speed and/or acceleration of the one or more objects.

The electronic device300includes a content module310that generates images at a display frame rate based on the pose information from the pose estimation module301and further includes a display module320that displays the images at the display frame rate.

The content module310includes a rendering module311that generates rendered images at a rendering frame rate based on pose information from the pose estimation module301. In various implementations, the rendering frame rate is approximately equal to the display frame rate. In various implementations, the rendering frame rate is less than the display frame rate. For example, in various implementations, the rendering frame rate is half the display frame rate.

In various implementations, the rendered images each include a display-locked content layer including display-locked content that is rendered independent of the pose information from the pose estimation module301and one or more world-locked content layers including world-locked content that is rendered based on the pose information from the pose estimation module301. In various implementations, the world-locked content layers include a static-anchored world-locked content layer that is rendered based on device pose information of the electronic device300and independent of any other pose information and a layer for each dynamic object being tracked by the pose prediction module301.

As an example, the rendering module311generates, at a first time period prior to a first display time period of a first display image, a first rendered image based on a first predicted pose of the electronic device300during the first display time period and a first predicted pose of the one or more objects from the pose estimation module301. The first rendered image includes a display-locked content layer, a static-anchored world-locked content layer, and one or more dynamic-anchored world-locked content layers.

The content module310includes a reprojection module312that generates composite images at the display frame rate based on updated pose information from the pose estimation module301. In various implementations, the reprojection module312generates the composite images by transforming the world-locked content layers of rendered images based on pose information from the pose estimation module301and flattening the rendered images into a single layer. In various implementations, the reprojection module312transforms each world-locked content layer by applying a homographic transformation to the world-locked content layer. In various implementations, the reprojection module312transforms each world-locked content layer using one or more other perspective transformations.

As an example, the reprojection module312generates, at a second time period after the first time period and prior to the first display time period of the first display image, a first composite image based on a second predicted pose of the electronic device300during the first display time period and a second predicted pose of the one or more objects from the pose estimation module301. In various implementations, the reprojection module312generates the first composite image by transforming the static-anchored world-locked content layer of the first rendered image based on the second predicted pose of the electronic device300and transforming the one or more dynamic-anchored world-locked content layers of the first rendered image based on the second predicted pose of the electronic device and the second predicted pose of the one or more objects. The pose estimation module301generates the second predicted pose after generating the first predicted pose. The second predicted pose, being generated closer in time to the first display time period than the first predicted pose, is more accurate than the first predicted pose.

The reprojection module312further generates, for each composite image, a stencil indicating how portions of the composite image are to be warped by the late-stage warp (LSW) module321described further below.

In various implementations, the stencil includes a matrix of the same resolution of the composite image, each element of the matrix corresponding to a pixel of the composite image. In various implementations, each element of the matrix has a value of ‘0’ if the corresponding pixel of the composite image is not to be warped by the LSW module321, a value of ‘1’ if the corresponding pixel of the composite image is to be warped by the LSW module321based only on the pose of the electronic device300, a value of ‘2’ if the corresponding pixel of the composite image is to be warped by the LSW module321based on the pose of the electronic device300and the pose of a first dynamic object, a value of ‘3’ if the corresponding pixel of the composite image is to be warped by the LSW module321based on the pose of the electronic device300and the pose of a second dynamic object, etc.

For example, pixels in the composite image corresponding to display-locked content have a value of ‘0’, pixels in the composite image corresponding to static-anchored world-locked content have a value of ‘1’, pixels in the composite image corresponding to dynamic-anchored world-locked content anchored to, e.g., a hand of a user, have a value of ‘2’, and pixels in the composite image corresponding to dynamic-anchored world-locked content anchored to, e.g., a face of a person, have a value of ‘3’.

The stencil may be defined in ways other than a matrix. For example, in various implementations, the stencil includes pixel coordinates indicating regions that are to be shifted by the LSW module321, such as the corners of a rectangle surrounding a region to be shifted by the LSW module321.

The display module320includes the LSW module321that generates display images at the display frame rate based on updated pose information from the pose estimation module301. In various implementations, the LSW module321performs a one-dimensional or two-dimensional pixel shift of portions of the composite images indicated by the corresponding stencil based on updated pose information from the pose estimation module301.

As an example, the LSW module321generates, at a third time period after the second time period and prior to the first display time period of the first display image, the first display image based on a third predicted pose of the electronic device300during the first display time period and a third predicted pose of the one or more objects from the pose estimation module301. In various implementations, the LSW module321generates the first display image by shifting portions of the first composite image indicated by the stencil as corresponding to static-anchored world-locked content based on the third predicted pose of the electronic device300and shifting portions of the first composite image indicated by the stencil as corresponding to dynamic-anchored world-locked content based on the third predicted pose of the electronic device300and the third predicted pose of the one or more objects. The pose estimation module301generates the third predicted pose after generating the second predicted pose. The third predicted pose, being generated closer in time to the first display time period than the second predicted pose, is more accurate than the second predicted pose.

The display322displays the display images at the display frame rate. As an example, the display322displays the first display image at the first display time period.

As noted above, in various implementations, the display frame rate is greater than the rendering frame rate. To achieve this, in various implementations, the reprojection module312performs frame rate extrapolation.

As an example, the reprojection module312generates, at a first time period prior to a second display time period of a second display image, a second composite image based on the first rendered image, a first predicted pose of the electronic device300during the second display time period from the pose estimation module301, and a first predicted pose of the one or more objects from the pose estimation module301. In various implementations, the reprojection module312generates the second composite image by transforming the world-locked content layers of the first rendered image based on the first predicted pose of the electronic device and the first predicted pose of the one or more objects and compositing the transformed world-locked content layers and the display-locked content layer into a single layer. The LSW module321generates, at a second time period after the first time period and prior to the second display time period of the second display image, the second display image based on a second predicted pose of the electronic device300during the second display time period and a second predicted pose of the one or more objects from the pose estimation module301. In various implementations, the LSW module321generates the second display image by shifting portions of the second composite image indicated by the stencil as corresponding to static-anchored world-locked content based on the second predicted pose of the electronic device300and shifting portions of the second composite image indicated by the stencil as corresponding to dynamic-anchored world-locked content based on the third predicted pose of the electronic device300and the third predicted pose of the one or more objects. The display322displays the second display image at the second display time period.

In various implementations, the rendering frame rate is substantially equal to the display frame rate. Thus, in various implementations, the electronic device300does not include a reprojection module312and the rendering module311outputs a composite image.

As an example, referring toFIG.2A, at a first time prior to the first time period ofFIG.2A, the rendering module311renders a display-locked content layer including the virtual clock224. The rendering module311renders a static-anchored world-locked content layer including the virtual flowers222based on a first prediction of the first pose of the electronic device300. The rendering module311renders a first dynamic-anchored world-locked content layer including the virtual ring221based on the first prediction of the first pose of the electronic device300and a first prediction of the first pose of the hand211. The rendering module311renders a second dynamic-anchored world-locked content layer including the virtual glasses223based on the first prediction of the first pose of the electronic device300and a first prediction of the first pose of the person.

At a second time subsequent to the first time but before the first time period ofFIG.2A, the reprojection module312transforms the static-anchored world-locked content layer based on a second prediction of the first pose of the electronic device300. The reprojection module312transforms the first dynamic-anchored world-locked content layer based on the second prediction of the first pose of the electronic device300and a second prediction of the first pose of the hand211. The reprojection module312transforms the second dynamic-anchored world-locked content layer based on the second prediction of the first pose of the electronic device300and a second prediction of the pose of the person213.

The reprojection module312flattens the transformed world-locked content layers and the display-locked content layer into a composite image. Further, the reprojection module312generates a stencil that includes a value of ‘0’ at the pixel locations of the virtual clock224, a value of ‘1’ at the pixel locations of the virtual flowers222, a value of ‘2’ at the pixel locations of the virtual ring221, and a value of ‘3’ at the pixel locations of the virtual glasses223.

At a third time subsequent to the second time but before the first time period ofFIG.2A, the LSW module321warps the pixels of the composite image having corresponding stencil values of ‘1’ based on a third prediction of the first pose of electronic device300. The LSW module321warps the pixels of the composite image having corresponding stencil values of ‘2’ based on the third prediction of the first pose of the electronic device300and a third prediction of the first pose of the hand211. The LSW module321warps the pixels of the composite image having corresponding stencil values of ‘3’ based on the third prediction of the first pose of the electronic device300and a third prediction of the first pose of the person213.

During the first time period ofFIG.2A, the display322displays the warped composite image, e.g., the display image.

As an example excluding use of the reprojection module312, referring toFIG.2A, at a first time prior to the first time period ofFIG.2A, the rendering module311renders a display-locked content layer including the virtual clock224. The rendering module311renders a static-anchored world-locked content layer including the virtual flowers222based on a first prediction of the first pose of the electronic device300. The rendering module311renders a first dynamic-anchored world-locked content layer including the virtual ring221based on the first prediction of the first pose of the electronic device300and a first prediction of the first pose of the hand211. The rendering module311renders a second dynamic-anchored world-locked content layer including the virtual glasses223based on the first prediction of the first pose of the electronic device300and a first prediction of the first pose of the person.

The rendering module311flattens the world-locked content layers and the display-locked content layer into a composite image and generates a stencil that includes a value of ‘0’ at the pixel locations of the virtual clock224, a value of ‘1’ at the pixel locations of the virtual flowers222, a value of ‘2’ at the pixel locations of the virtual ring221, and a value of ‘3’ at the pixel locations of the virtual glasses223.

At a second time subsequent to the first time but before the first time period ofFIG.2A, the LSW module321warps the pixels of the composite image having corresponding stencil values of ‘1’ based on a second prediction of the first pose of electronic device300. The LSW module321warps the pixels of the composite image having corresponding stencil values of ‘2’ based on the second prediction of the first pose of the electronic device300and a second prediction of the first pose of the hand211. The LSW module321warps the pixels of the composite image having corresponding stencil values of ‘3’ based on the second prediction of the first pose of the electronic device300and a second prediction of the first pose of the person213.

During the first time period ofFIG.2A, the display322displays the warped composite image, e.g., the display image.

FIG.4is a flowchart representation of a method400of displaying an image in accordance with some implementations. In various implementations, the method400is performed by an electronic device, such as the electronic device120ofFIG.1or the electronic device300ofFIG.3. In various implementations, the method400is performed by a device with an image sensor, a display, one or more processors, and non-transitory memory. In some implementations, the method400is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method400is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).

The method400begins, in block410, with the device capturing, using the image sensor, an image of an object in a physical environment. In various implementations, the object is a dynamic object in the physical environment. In various implementations, the object is a hand of a user of the device. In various implementations, the object is the face of a person (other than the user of the device).

The method400continues, in block420, with the device obtaining, based on the image, a first predicted object pose of the object in the physical environment at a display time. In various implementations, obtaining the first predicted object pose includes performing a hand tracking algorithm on the image. In various implementations, obtaining the first predicted object pose includes performing a face detection algorithm on the image. In various implementations, obtaining the first predicted object pose is based on the image and previously obtained images. Thus, in various implementations, obtaining the first predicted object pose includes determining a current object pose at various times based on the previously obtained images and the image and extrapolating the current object poses to predict the first predicted object pose at the display time.

The method400continues, in block430, with the device rendering virtual content based on the first predicted object pose. In various implementations, the method400further includes obtaining a first predicted device pose of the device in the physical environment at the first display time. In various implementations, rendering the virtual content is further based on the first predicted device pose.

The method400continues, in block440, with the device obtaining a second predicted object pose of the object in the physical environment at the display time. In various implementations, obtaining the second predicted object pose is performed after rendering the virtual content or at least after rendering the virtual content has begun.

In various implementations, obtaining the second predicted object pose is based on an additional image. For example, in various implementations, obtaining the second predicted object pose includes determining a current object pose at various times based on the previously obtained images, the image, and the additional image and extrapolating the current object poses to predict the second predicted object pose at the display time.

In various implementations, the image is captured at a first capture rate and the additional image is captured at a second capture rate higher than the first capture rate. In various implementations, the additional image has a lower resolution than the image.

In various implementations, obtaining the second predicted object pose is based on data received from an inertial measurement unit (IMU). For example, in various implementations, the data is received from the IMU of a watch worn by the user. As another example, in various implementations, the data is received from the IMU of a stylus held by the user. As another example, in various implementations, the data is received from the IMU of a head-mounted device worn by the person (other than the user). In various implementations, the image is captured at a capture rate and the data is received at a reception rate higher than the capture rate.

The method400continues, in block450, with the device warping the virtual content based on the second predicted object pose. In various implementations, the method400includes obtaining a second predicted device pose of the device at the display time and warping the virtual content is further based on the second predicted device pose. In various implementations, obtaining the second predicted device pose is based on data received from an inertial measurement unit (IMU) of the device. Thus, in various implementations, determining the second predicted device pose is based on data received from an IMU of the device and determining the second predicted object pose is based on data received from a different IMU of a different device.

In various implementations, warping the virtual content is based on a difference between the first predicted object pose and the second predicted object pose. In various implementations, warping the virtual content is further based on a difference between the first predicted device pose and the second device object pose.

In various implementations, warping the virtual content includes performing a homographic transformation of the virtual content. In various implementations, warping the virtual content includes shifting the virtual content. In various implementations, warping the virtual content includes scaling the virtual content.

In various implementations, the method400further includes generating a stencil indicating portions of the virtual content to be warped based on the second predicted object pose and warping the virtual content is further based on the stencil. In various implementations, the stencil indicates portions of the virtual content to be warped based on the second predicted device pose independent of the second predicted object pose. In various implementations, the stencil further indicates portions of the virtual content that are not to be warped.

The method400continues, in block460, with the device displaying, on the display at the display time, the warped virtual content.

FIG.5is a flowchart representation of another method500of displaying an image in accordance with some implementations. In various implementations, the method500is performed by an electronic device, such as the electronic device120ofFIG.1or the electronic device300ofFIG.3. In various implementations, the method500is performed by a device with an image sensor, a display, one or more processors, and non-transitory memory. In some implementations, the method500is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method500is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).

The method500begins, in block510, with the device obtaining a first predicted device pose of the device at a display time and a first predicted object pose of an object at the display time. In various implementations, the object is a dynamic object in the physical environment. In various implementations, the object is a hand of a user of the device. In various implementations, the object is the face of a person (other than the user of the device).

In various implementations, obtaining the first predicted device pose is based on data received from an inertial measurement unit (IMU) of the device. In various implementations, obtaining the first predicted object pose is based on an image of the object. In various implementations, obtaining the first predicted object pose includes performing a hand tracking algorithm on the image. In various implementations, obtaining the first predicted object pose includes performing a face detection algorithm on the image. In various implementations, obtaining the first predicted object pose is based on the image and previously obtained images. Thus, in various implementations, obtaining the first predicted object pose includes determining a current object pose at various times based on the previously obtained images and the image and extrapolating the current object poses to predict the first predicted object pose at the display time.

The method500continues, in block520, with the device rendering virtual content based on the first predicted device pose and the first predicted object pose.

The method500continues, in block530, with the device obtaining a second predicted device pose of the device at the display time and a second predicted object pose of the object at the display time. In various implementations, obtaining the second predicted device pose and the second predicted object pose is performed after rendering the virtual content or at least after rendering the virtual content has begun.

In various implementations, obtaining the second predicted object pose is based on an additional image. For example, in various implementations, obtaining the second predicted object pose includes determining a current object pose at various times based on the previously obtained images, the image, and the additional image and extrapolating the current object poses to predict the second predicted object pose at the display time.

In various implementations, the image is captured at a first capture rate and the additional image is captured at a second capture rate higher than the first capture rate. In various implementations, the additional image has a lower resolution than the image.

In various implementations, obtaining the second predicted device pose is based on data received from an inertial measurement unit (IMU) of the device. In various implementations, obtaining the second predicted object pose is based on data received from a different inertial measurement unit (IMU) of a different device. For example, in various implementations, the data is received from the IMU of a watch worn by the user. As another example, in various implementations, the data is received from the IMU of a stylus held by the user. As another example, in various implementations, the data is received from the IMU of a head-mounted device worn by the person (other than the user). In various implementations, the image is captured at a capture rate and the data is received at a reception rate higher than the capture rate.

The method500continues, in block540, with the device warping the virtual content based on the second predicted device pose and the second predicted object pose. In various implementations, warping the virtual content is based on a difference between the first predicted device pose and the second device object pose and a difference between the first predicted object pose and the second predicted object pose.

In various implementations, warping the virtual content includes performing a homographic transformation of the virtual content. In various implementations, warping the virtual content includes shifting the virtual content. In various implementations, warping the virtual content includes scaling the virtual content.

In various implementations, the method500further includes generating a stencil indicating portions of the virtual content to be warped based on the second predicted device pose and the second predicted object pose and warping the virtual content is further based on the stencil. In various implementations, the stencil indicates portions of the virtual content to be warped based on the second predicted device pose independent of the second predicted object pose. In various implementations, the stencil further indicates portions of the virtual content that are not to be warped.

Thus, in various implementations, warping the virtual content includes warping a first portion of the virtual content based on the second predicted device pose and the second predicted object pose and warping a second portion of the virtual content based on the second predicted device pose independent of the second predicted object pose.

The method500continues, in block550, with the device displaying, on the display at the display time, the warped virtual content.

FIG.6is a flowchart representation of a method600of extrapolating an image in accordance with some implementations. In various implementations, the method600is performed by an electronic device, such as the electronic device120ofFIG.1or the electronic device300ofFIG.3. In various implementations, the method600is performed by a device with an image sensor, a display, one or more processors, and non-transitory memory. In some implementations, the method600is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method400is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).

The method600begins, in block610, with the device capturing, using the image sensor, an image of an object in a physical environment. In various implementations, the object is a dynamic object in the physical environment. In various implementations, the object is a hand of a user of the device. In various implementations, the object is the face of a person (other than the user of the device).

The method600continues, in block620, with the device obtaining, based on the image, a first predicted object pose of the object in the physical environment at a first display time. In various implementations, obtaining the first predicted object pose includes performing a hand tracking algorithm on the image. In various implementations, obtaining the first predicted object pose includes performing a face detection algorithm on the image. In various implementations, obtaining the first predicted object pose is based on the image and previously obtained images. Thus, in various implementations, obtaining the first predicted object pose includes determining a current object pose at various times based on the previously obtained images and the image and extrapolating the current object poses to predict the first predicted object pose at the display time.

The method600continues, in block630, with the device rendering virtual content based on the first predicted object pose. In various implementations, the method600further includes obtaining a first predicted device pose of the device in the physical environment at the first display time. In various implementations, rendering the virtual content is further based on the first predicted device pose.

The method600continues, in block640, with the device displaying, on the display at the first display time, the virtual content. In various implementations, the virtual content is warped based on an updated predicted object pose of the object at the first display time and/or an updated predicted device pose of the device of the first display time as described above with respect toFIGS.4and5.

The method600continues, in block650, with the device obtaining a second predicted object pose of the object in the physical environment at a second display time. In various implementations, obtaining the second predicted object pose is performed after rendering the virtual content or at least after rendering the virtual content has begun.

In various implementations, obtaining the second predicted object pose is based on an additional image. For example, in various implementations, obtaining the second predicted object pose includes determining a current object pose at various times based on the previously obtained images, the image, and the additional image and extrapolating the current object poses to predict the second predicted object pose at the display time.

In various implementations, the image is captured at a first capture rate and the additional image is captured at a second capture rate higher than the first capture rate. In various implementations, the additional image has a lower resolution than the image.

In various implementations, obtaining the second predicted object pose is based on data received from an inertial measurement unit (IMU). For example, in various implementations, the data is received from the IMU of a watch worn by the user. As another example, in various implementations, the data is received from the IMU of a stylus held by the user. As another example, in various implementations, the data is received from the IMU of a head-mounted device worn by the person (other than the user). In various implementations, the image is captured at a capture rate and the data is received at a reception rate higher than the capture rate.

The method600continues, in block660, with the device warping the virtual content based on the second predicted object pose. In various implementations, the method600includes obtaining a second predicted device pose of the device at the second display time and the virtual content is further based on the second predicted device pose. In various implementations, obtaining the second predicted device pose is based on data received from an inertial measurement unit (IMU) of the device. Thus, in various implementations, determining the second predicted device pose is based on data received from an IMU of the device and determining the second predicted object pose is based on data received from a different IMU of a different device.

In various implementations, warping the virtual content is based on a difference between the first predicted object pose and the second predicted object pose. In various implementations, warping the virtual content is further based on a difference between the first predicted device pose and the second device object pose.

In various implementations, warping the virtual content includes performing a homographic transformation of the virtual content. In various implementations, warping the virtual content includes shifting the virtual content. In various implementations, warping the virtual content includes scaling the virtual content.

In various implementations, the method600further includes generating a stencil indicating portions of the virtual content to be warped based on the second predicted object pose and warping the virtual content is further based on the stencil. In various implementations, the stencil indicates portions of the virtual content to be warped based on the second predicted device pose independent of the second predicted object pose. In various implementations, the stencil further indicates portions of the virtual content that are not to be warped.

The method600continues, in block670, with the device displaying, on the display at the second display time, the warped virtual content.

FIG.7is a flowchart representation of another method700of extrapolating an image in accordance with some implementations. In various implementations, the method700is performed by an electronic device, such as the electronic device120ofFIG.1or the electronic device300ofFIG.3. In various implementations, the method700is performed by a device with an image sensor, a display, one or more processors, and non-transitory memory. In some implementations, the method700is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method700is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).

The method700begins, in block710, with the device obtaining a first predicted device pose of the device at a first display time and a first predicted object pose of an object at the first display time. In various implementations, the object is a dynamic object in the physical environment. In various implementations, the object is a hand of a user of the device. In various implementations, the object is the face of a person (other than the user of the device).

In various implementations, obtaining the first predicted device pose is based on data received from an inertial measurement unit (IMU) of the device. In various implementations, obtaining the first predicted object pose is based on an image of the object. In various implementations, obtaining the first predicted object pose includes performing a hand tracking algorithm on the image. In various implementations, obtaining the first predicted object pose includes performing a face detection algorithm on the image. In various implementations, obtaining the first predicted object pose is based on the image and previously obtained images. Thus, in various implementations, obtaining the first predicted object pose includes determining a current object pose at various times based on the previously obtained images and the image and extrapolating the current object poses to predict the first predicted object pose at the display time.

The method700continues, in block720, with the device rendering virtual content based on the first predicted device pose and the first predicted object pose.

The method700continues, in block730, with the device displaying, on the display at the first display time, the virtual content. In various implementations, the virtual content is warped based on an updated predicted object pose of the object at the first display time and/or an updated predicted device pose of the device of the first display time as described above with respect toFIGS.4and5.

The method700continues, in block740, with the device obtaining a second predicted device pose of the device at a second display time and a second predicted object pose of the object at the second display time. In various implementations, obtaining the second predicted device pose and the second predicted object pose is performed after rendering the virtual content or at least after rendering the virtual content has begun.

In various implementations, obtaining the second predicted object pose is based on an additional image. For example, in various implementations, obtaining the second predicted object pose includes determining a current object pose at various times based on the previously obtained images, the image, and the additional image and extrapolating the current object poses to predict the second predicted object pose at the display time.

In various implementations, the image is captured at a first capture rate and the additional image is captured at a second capture rate higher than the first capture rate. In various implementations, the additional image has a lower resolution than the image.

In various implementations, obtaining the second predicted device pose is based on data received from an inertial measurement unit (IMU) of the device. In various implementations, obtaining the second predicted object pose is based on data received from a different inertial measurement unit (IMU) of a different device. For example, in various implementations, the data is received from the IMU of a watch worn by the user. As another example, in various implementations, the data is received from the IMU of a stylus held by the user. As another example, in various implementations, the data is received from the IMU of a head-mounted device worn by the person (other than the user). In various implementations, the image is captured at a capture rate and the data is received at a reception rate higher than the capture rate.

The method700continues, in block750, with the device warping the virtual content based on the second predicted device pose and the second predicted object pose. In various implementations, warping the virtual content is based on a difference between the first predicted device pose and the second device object pose and a difference between the first predicted object pose and the second predicted object pose.

In various implementations, warping the virtual content includes performing a homographic transformation of the virtual content. In various implementations, warping the virtual content includes shifting the virtual content. In various implementations, warping the virtual content includes scaling the virtual content.

In various implementations, the method700further includes generating a stencil indicating portions of the virtual content to be warped based on the second predicted device pose and the second predicted object pose and warping the virtual content is further based on the stencil. In various implementations, the stencil indicates portions of the virtual content to be warped based on the second predicted device pose independent of the second predicted object pose. In various implementations, the stencil further indicates portions of the virtual content that are not to be warped.

Thus, in various implementations, warping the virtual content includes warping a first portion of the virtual content based on the second predicted device pose and the second predicted object pose and warping a second portion of the virtual content based on the second predicted device pose independent of the second predicted object pose.

The method700continues, in block760, with the device displaying, on the display at the second display time, the warped virtual content.

In some implementations, the one or more communication buses804include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices806include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and/or the like.

The memory820includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDR RAM), or other random-access solid-state memory devices. In some implementations, the memory820includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory820optionally includes one or more storage devices remotely located from the one or more processing units802. The memory820comprises a non-transitory computer readable storage medium. In some implementations, the memory820or the non-transitory computer readable storage medium of the memory820stores the following programs, modules and data structures, or a subset thereof including an optional operating system830and an XR experience module840.

The operating system830includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the XR experience module840is configured to manage and coordinate one or more XR experiences for one or more users (e.g., a single XR experience for one or more users, or multiple XR experiences for respective groups of one or more users). To that end, in various implementations, the XR experience module840includes a data obtaining unit842, a tracking unit844, a coordination unit846, and a data transmitting unit848.

In some implementations, the data obtaining unit842is configured to obtain data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the electronic device120ofFIG.1. To that end, in various implementations, the data obtaining unit842includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the tracking unit844is configured to map the physical environment105and to track the position/location of at least the electronic device120with respect to the physical environment105ofFIG.1. To that end, in various implementations, the tracking unit844includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the coordination unit846is configured to manage and coordinate the XR experience presented to the user by the electronic device120. To that end, in various implementations, the coordination unit846includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the data transmitting unit848is configured to transmit data (e.g., presentation data, location data, etc.) to at least the electronic device120. To that end, in various implementations, the data transmitting unit848includes instructions and/or logic therefor, and heuristics and metadata therefor.

Although the data obtaining unit842, the tracking unit844, the coordination unit846, and the data transmitting unit848are shown as residing on a single device (e.g., the controller110), it should be understood that in other implementations, any combination of the data obtaining unit842, the tracking unit844, the coordination unit846, and the data transmitting unit848may be located in separate computing devices.

FIG.9is a block diagram of an example of the electronic device120in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations the electronic device120includes one or more processing units902(e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, and/or the like), one or more input/output (I/O) devices and sensors906, one or more communication interfaces908(e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and/or the like type interface), one or more programming (e.g., I/O) interfaces910, one or more XR displays912, one or more optional interior- and/or exterior-facing image sensors914, a memory920, and one or more communication buses904for interconnecting these and various other components.

In some implementations, the one or more XR displays912are configured to provide the XR experience to the user. In some implementations, the one or more XR displays912correspond to holographic, digital light processing (DLP), liquid-crystal display (LCD), liquid-crystal on silicon (LCoS), organic light-emitting field-effect transitory (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field-emission display (FED), quantum-dot light-emitting diode (QD-LED), micro-electro-mechanical system (MEMS), and/or the like display types. In some implementations, the one or more XR displays912correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. For example, the electronic device120includes a single XR display. In another example, the electronic device includes an XR display for each eye of the user. In some implementations, the one or more XR displays912are capable of presenting MR and VR content.

In some implementations, the one or more image sensors914are configured to obtain image data that corresponds to at least a portion of the face of the user that includes the eyes of the user (any may be referred to as an eye-tracking camera). In some implementations, the one or more image sensors914are configured to be forward-facing so as to obtain image data that corresponds to the physical environment as would be viewed by the user if the electronic device120was not present (and may be referred to as a scene camera). The one or more optional image sensors914can include one or more RGB cameras (e.g., with a complimentary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and/or the like.

The memory920includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, the memory920includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory920optionally includes one or more storage devices remotely located from the one or more processing units902. The memory920comprises a non-transitory computer readable storage medium. In some implementations, the memory920or the non-transitory computer readable storage medium of the memory920stores the following programs, modules and data structures, or a subset thereof including an optional operating system930and an XR presentation module940.

The operating system930includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the XR presentation module940is configured to present XR content to the user via the one or more XR displays912. To that end, in various implementations, the XR presentation module940includes a data obtaining unit942, a warping unit944, an XR presenting unit946, and a data transmitting unit948.

In some implementations, the data obtaining unit942is configured to obtain data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller110ofFIG.1. To that end, in various implementations, the data obtaining unit942includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the warping unit944is configured to warp content based on an updated predicted pose of an object. To that end, in various implementations, the warping unit944includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the XR presenting unit946is configured to display the warped image via the one or more XR displays912. To that end, in various implementations, the XR presenting unit946includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the data transmitting unit948is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller110. In some implementations, the data transmitting unit948is configured to transmit authentication credentials to the electronic device. To that end, in various implementations, the data transmitting unit948includes instructions and/or logic therefor, and heuristics and metadata therefor.

Although the data obtaining unit942, the warping unit944, the XR presenting unit946, and the data transmitting unit948are shown as residing on a single device (e.g., the electronic device120), it should be understood that in other implementations, any combination of the data obtaining unit942, the warping unit944, the XR presenting unit946, and the data transmitting unit948may be located in separate computing devices.

Therefore, although the present disclosure broadly covers use of personal information data to implement one or more various disclosed embodiments, the present disclosure also contemplates that the various embodiments can also be implemented without the need for accessing such personal information data. That is, the various embodiments of the present technology are not rendered inoperable due to the lack of all or a portion of such personal information data. For example, virtual content can be displayed over a person's face and/or hand based on non-personal information data or a bare minimum amount of personal information, such as the content being requested by the device associated with a user, other non-personal information available to the electronic device, or publicly available information.