Patent Description:
Content may be inserted into an image or a user's field of view of the physical environment or another three-dimensional (3D) environment. For example, an augmented reality (AR) system may generate an immersive augmented environment for a user by inserting content. The immersive augmented environment can be generated by superimposing computer-generated content on a user's field of view of the real world. For example, the computer-generated content can include labels, textual information, images, sprites, and three-dimensional entities. These images may be displayed at a position in the user's field of view so as to appear to overlay an object in the real world and be spatially retained relative to the real word even when outside of the users' field of view. Similarly, the computer-generated content may be overlaid on a displayed image. In certain applications, such as educational applications, it may be helpful to draw a user's attention to a particular point of interest on the inserted content. Document "Augmented reality presents storytelling opportunities in weather" (XP002788066), discloses an Augmented Reality environment to present weather forecasts, in which computer generated graphics are overlaid on a real image of a TV studio.

This disclosure describes systems and methods for directing user attention to points of interest on inserted content, such as augmented reality content, that a user is viewing. For example, a portion of the inserted content that is identified based on a point of interest may be shaded differently than the rest of the content so as to draw the user's attention. Additionally, a pointing entity, such as a conical structure, may be rendered so as to be visible from the user's viewing position and to indicate an appropriate position/direction from which to view the point of interest on the inserted content.

In one aspect, a method of directing user attention is disclosed in claim <NUM>.

In another aspect, a non-transitory computer-readable storage medium is described according to claim <NUM>.

In yet another aspect, a system of directing user attention is disclosed in claim <NUM>.

Reference will now be made in detail to non-limiting examples of this disclosure, examples of which are illustrated in the accompanying drawings. The examples are described below by referring to the drawings, wherein like reference numerals refer to like elements. When like reference numerals are shown, corresponding description(s) are not repeated and the interested reader is referred to the previously discussed figure(s) for a description of the like element(s).

Augmented reality (AR) systems include systems that insert computer-generated content into a user's perception of the physical space surrounding the user. For example, the inserted content may include three-dimensional structures. In some situations, it may be useful to draw attention to a specific portion (i.e., a point of interest (POI)) of the inserted content. The point of interest may be located anywhere on an inserted three-dimensional structure, including on a sider of the structure that is facing away from a user of the AR system. These POIs may not be visible until the user moves to a new position within the AR environment. Additionally, even when the user is in a position to view the POI, it may be difficult for the user to identify the POI without modifying or interfering with viewing of the POI. Thus, technical problems exist in AR systems when it comes to directing user attention to specific points of interest within AR content. Embodiments disclosed herein provide technical solutions to this technical problem by causing the AR environment to be displayed in a manner that draws attention to a specific POI. For example, the POI or a region of the AR content surrounding the POI may be rendered using shading parameters that are different than the shading parameters for rest of the AR environment. In some implementations, the POI may be rendered using ordinary shading while the rest of the AR environment is rendered using dimmer shading. Beneficially, the different shading parameters can draw attention to the POI without altering the appearance of the POI. Some implementations also cause a pointing entity to be displayed that points to the POI. The pointing entity may be generated based on the position of a user such that from the users point-of-view the pointing entity directs the user to the POI. Beneficially, the pointing entity may help a user identify and locate POIs that are on a side of the AR content that does not face the user.

<FIG> is a block diagram illustrating a system <NUM> according to an example implementation. The system <NUM> generates an augmented reality (AR) environment for a user of the system <NUM>. In some implementations, the system <NUM> includes a computing device <NUM>, a head-mounted display device (HMD) <NUM>, and an AR content source <NUM>. Also shown is a network <NUM> over which the computing device <NUM> may communicate with the AR content source <NUM>.

The computing device <NUM> may include a memory <NUM>, a processor assembly <NUM>, a communication module <NUM>, a sensor system <NUM>, and a display device <NUM>. The memory <NUM> may include an AR application <NUM>, AR content <NUM>, an image buffer <NUM>, a location engine <NUM>, an attention engine <NUM>, and a content display engine <NUM>.

The AR application <NUM> may insert computer-generated content into a user's perception of the physical space surrounding the user. The computer-generated content may include labels, textual information, images, sprites, and three-dimensional entities. In some implementations, the content is inserted for entertainment, educational, or informational purposes. As an example, an augmented reality system may allow students to take a virtual field trip. During the virtual field trip, students may be able to view content representing artifacts, animals, or other entities using augmented reality.

The AR application <NUM> may display the content in a manner that draws attention to a POI. For example, the system may generate a spotlight effect for the POI. In some implementations, the spotlight effect includes using different shading parameters to render a region of the content surrounding the POI and rendering a pointing entity that is similar in appearance to a cone of light that would emanate from a spotlight.

As discussed above, the AR application <NUM> may display the content using different shading parameters so as to distinguish the point of interest from the other parts of the content. For example, the point of interest may be rendered with shading parameters that are lighter than the other parts of the content. In this manner, the point of interest may appear to have a spotlight directed at it. For example, pixels that correspond to portions of the content that are within a predetermined threshold distance from a three-dimensional coordinate associated with the POI may be rendered using lighter shading parameters than the other parts of the content. In some implementations, pixels that are outside of the predetermined threshold are rendered with shading parameters that reduce the lightness of the pixels (e.g., the color values of the pixels are scaled by a multiplier that is less than <NUM>, such as <NUM>) surrounding the point of interest. The use of differential shading parameters may have several benefits over using a directional light source when rendering the content to draw attention to the POI. First, the differential shading parameters allow a curved (e.g., roughly circular, spherical region) to be identified without regard to the location of the directional light source. In contrast, a directional light source would generally result in an oval-shaped region being identified based on distortion due to the angle of incidence from the light source. Second, the differential lighting allows for the entirety of a complex geometrical shape (e.g., a tree branch with leaves) to be identified without regard for shadows (e.g., caused by other parts of the complex geometry). Additionally, using the differential lighting may use fewer processing cycles than using a directional light source as the directional light source may impact surfaces beyond the POI and require additional rendering resources.

Some implementations of the AR application <NUM> can generate a pointing entity to draw the user's attention to the POI. The pointing entity may help a user find the POI from a distance and determine a viewing angle or position from which to observe the POI. The pointing entity may be especially beneficial in situations where the POI is on a surface of the content that does not face toward the user and thus the differential shading of the POI cannot be seen by the user. In these situations, the pointing entity may still be visible to the user and may help the user identify a position relative to the content from which the POI may be observed. In some implementations, the pointing entity is generated as a conical structure having a conical vertex (or point) disposed at a vertical offset above or below the content. In this manner, at least a portion of the pointing entity will not be occluded by the content. In some implementations, the offset is below the content when the normal vector of the surface of the content is pointed downward and the offset is above the content when the normal vector of the surface of the content is pointed upward.

The AR application <NUM> may also offset the conical vertex laterally from the content so that the conical structure of the pointing entity appears to point laterally in toward the POI. In some implementations, the position of the conical vertex of the conical structure is selected, at least in part, so as to have a component that is perpendicular to the user's view axis. As the user moves, the position of the conical vertex may be moved too. In some implementations, the conical structure (or portions of it) may be rendered as partially transparent. For example, a transparency gradient may be applied to the conical structure where the conical vertex is rendered with low or no transparency and the conical base is rendered as fully transparent. The transparency level may increase gradually along the cone from the conical vertex to the base. In this manner, the conical base of the conical structure will not occlude the user's view of the POI while the conical vertex will be visible to guide the user to the POI.

Additionally, some implementations of the AR application <NUM> draw a user's attention to a POI when the user is not facing the content. In these implementations, an orb light source may be generated in the user's peripheral field of view so as to indicate a direction the user should turn to face. Additionally, a glowing orb or shimmer may move through an AR or virtual reality (VR) environment to guide a user to inserted content and/or a POI associated with that content. In some implementations, a pointing entity may grow when the user is orientated so that the pointing entity is in the periphery of a user's field of view. The pointing entity may then shrink slowly as the user turns toward the pointing entity. In this way, as the user's orientation aligns toward the pointing entity, the pointing entity gets smaller and points more precisely to the desired target POI.

The AR application <NUM> may allow a teacher or guide can facilitate the virtual field trip and may, for example, select, switch, or otherwise manage the content shown to a group of students or visitors during the virtual field trip. Alternatively, a user may independently explore the content of the virtual field trip without a teacher or guide. In some implementations, as the virtual field trip progresses the content shown at a particular physical location changes. For example, a set of content may be displayed sequentially at a particular location, and the content displayed at a particular location may change in response to a user input or based on a particular amount of time elapsing. In some implementations, the teacher or guide can view a 2D map of the content and POIs associated with the content. The teacher or guide may then select the POI on the 2D map to activate a pointing entity for that POI.

In some implementations, the AR application <NUM> may cause content to be displayed that includes multiple POIs, and the AR application <NUM> may be configured to sequentially draw the user's attention to those POIs. For example, a first POI may be selected (e.g., based on an order determined in a tour, based on proximity to the user, or based on an input from a teacher or tour guide user). After the user has reached and viewed the first POI, the differential shading and pointing entity associated with that POI may be removed. Thereafter, differential shading may be applied to the second POI and a pointing entity may be generated for the second entity, and so on. In some implementations, the pointing entity may move from the first POI to a second POI (e.g., the pointing entity may float through the AR environment from the first POI to the second POI).

The AR application <NUM> may cause the computing device <NUM> to captures images of the physical space surrounding a user. The AR application <NUM> may then determine a physical location at which to insert content within the captured image/s. For example, the AR application <NUM> may identify a physical marker such as a QR code, picture, sticker, or other type of visual indicator within the physical space. The sticker may be formed from a paper of vinyl material and an adhesive, which may be used to permanently or temporarily attach the sticker to a surface in the physical space. The stickers may be configured to allow for removal and reattachment within the physical space.

In some implementations, the AR application <NUM> may determine the physical location based on a coordinate system being mapped to the captured image/images based on determining a location of the computing device <NUM> based on using, for example, a visual positioning system or global positioning system. Content may then be identified to insert at the physical location. The content may include one or more POIs and when the content is inserted, the content may be displayed in a manner to draw a user's attention to that POI, such as by emulating a spotlight directed at the POI.

The HMD <NUM> may include a display device that is positioned in front of a user's eyes. For example, the HMD <NUM> may occlude the user's entire field of view so that the user can only see the content displayed by the display device. In some examples, the display device is configured to display two different images, one that is viewable by each of the user's eyes. For example, at least some of the content in one of the images may be slightly offset relative to the same content in the other image so as to generate the perception of a three-dimensional scene due to parallax. In some implementations, the HMD <NUM> includes a chamber in which the computing device <NUM> (e.g., a portable electronic device, such as a smartphone) may be placed so as to permit viewing of the display device of the computing device <NUM> through the HMD <NUM>. In some implementations, the HMD <NUM> may be configured to generate a VR environment too.

As another example, the HMD <NUM> may permit a user to see the physical space while the HMD is being worn. The HMD <NUM> may include a micro-display device that displays computer-generated content that is overlaid on the user's field of view. For example, the HMD <NUM> may include an at least partially transparent visor that includes a combiner that permits light from the physical space to reach the user's eye while also reflecting images displayed by the micro-display device toward the user's eye.

Some implementations may not include an HMD <NUM>. In at least some of these implementations, the computing device <NUM> is a portable electronic device, such as a smartphone, that includes a camera and a display device. The AR application <NUM> may cause the portable electronic device may capture images using the camera and show AR images on the display device that include computer-generated content overlaid upon the images captured by the camera.

Although many examples described herein relate to an AR application, such as the AR application <NUM>, directing user attention to POIs on inserted content, the techniques described herein may be incorporated in other types of systems too. For example, the techniques described herein may be used to draw user attention to POIs in a VR environment or within an image or video.

The sensor system <NUM> may include various sensors, such as a camera assembly <NUM>. Implementations of the sensor system <NUM> may also include other sensors, including, for example, an inertial motion unit (IMU) <NUM>, a light sensor, an audio sensor, an image sensor, a distance and/or proximity sensor, a contact sensor such as a capacitive sensor, a timer, and/or other sensors and/or different combination(s) of sensors.

The IMU <NUM> detects motion, movement, and/or acceleration of the computing device <NUM> and/or the HMD <NUM>. The IMU <NUM> may include various different types of sensors such as, for example, an accelerometer, a gyroscope, a magnetometer, and other such sensors. A position and orientation of the HMD <NUM> may be detected and tracked based on data provided by the sensors included in the IMU <NUM>. The detected position and orientation of the HMD <NUM> may allow the system to detect and track the user's gaze direction and head movement.

In some implementations, the AR application may use the sensor system <NUM> to determine a location and orientation of a user within a physical space and/or to recognize features or objects within the physical space.

The AR application <NUM> may present or provide AR content to a user via the HMD and/or one or more output devices of the computing device <NUM> such as the display device <NUM>, speakers, and/or other output devices. In some implementations, the AR application <NUM> includes instructions stored in the memory <NUM> that, when executed by the processor assembly <NUM>, cause the processor assembly <NUM> to perform the operations described herein. For example, the AR application <NUM> may generate and present an AR environment to the user based on, for example, AR content, such as the AR content <NUM> and/or AR content received from the AR content source <NUM>. The AR content <NUM> may include content such as images or videos that may be displayed on a portion of the user's field of view in the HMD <NUM>. For example, the AR application <NUM> may generate content corresponding to a virtual field trip for one or more users (e.g., for example the AR application <NUM> may coordinate display of AR content with other computing devices). The content may include objects that overlay various portions of the physical space. The content may be rendered as flat images or as three-dimensional (3D) objects. The 3D objects may include one or more objects represented as polygonal meshes. The polygonal meshes may be associated with various surface textures, such as colors and images.

The AR application <NUM> may use the image buffer <NUM>, location engine <NUM>, attention engine <NUM>, and content display engine <NUM> to generate images for display via the HMD <NUM> based on the AR content <NUM>. The AR content may be associated with (or include) one or more POIs. For example, one or more images captured by the camera assembly <NUM> may be stored in the image buffer <NUM>. The AR application <NUM> may use the location engine <NUM> to determine one or more physical locations within the images to insert content. For example, the location engine <NUM> may analyze the images to identify features within the image so that the images can be mapped to a coordinate system associated with physical locations for displaying content. Additionally, the location engine <NUM> may analyze the images to identify markers associated with physical locations for displaying content. The markers and/or coordinates of physical locations may be defined by a user during a setup process.

Once a physical location has been identified, the content display engine <NUM> can then display content at the identified physical location. In some implementations, the attention engine <NUM> may direct the content display engine <NUM> to use various parameters for displaying the content, such as differential shading parameters, to direct the user's attention to a POI on the content. Additionally, the attention engine <NUM> may generate a pointing entity, such as a conical structure, to point toward the POI and/or a light source to guide the user toward the POI. The AR application <NUM> may for example determine which content to display at any given time, when to update/change the content, which POI to draw the user's attention to, and when to switch to another POI. In some implementations, the AR application <NUM> may simultaneously display different content at multiple different physical locations identified by the location engine <NUM>.

In some implementations, the image buffer <NUM> is a region of the memory <NUM> that is configured to store one or more images. In some implementations, the computing device <NUM> stores images captured by the camera assembly <NUM> as a texture within the image buffer <NUM>. The image buffer may also include a memory location that is integral with the processor assembly <NUM>, such as dedicated random access memory (RAM) on a GPU.

In some implementations, the location engine <NUM> and content display engine <NUM> may include instructions stored in the memory <NUM> that, when executed by the processor assembly <NUM>, cause the processor assembly <NUM> to perform operations described herein to generate an image or series images that are displayed to the user (e.g., via the HMD <NUM>).

The AR application <NUM> may update the AR environment based on input received from the camera assembly <NUM>, the IMU <NUM>, and/or other components of the sensor system <NUM>. For example, the IMU <NUM> may detect motion, movement, and/or acceleration of the computing device <NUM> and/or the HMD <NUM>. The IMU <NUM> may include various different types of sensors such as, for example, an accelerometer, a gyroscope, a magnetometer, and other such sensors. A position and orientation of the HMD <NUM> may be detected and tracked based on data provided by the sensors included in the IMU <NUM>. The detected position and orientation of the HMD <NUM> may allow the system to detect and track the user's position and orientation within a physical space. Based on the detected position and orientation, the AR application <NUM> may update the AR environment to reflect a changed orientation and/or position of the user within the environment.

The memory <NUM> can include one or more non-transitory computer-readable storage media. The memory <NUM> may store instructions and data that are usable to generate an AR environment for a user.

The processor assembly <NUM> includes one or more devices that are capable of executing instructions, such as instructions stored by the memory <NUM>, to perform various tasks associated with generating an AR environment. For example, the processor assembly <NUM> may include a central processing unit (CPU) and/or a graphics processor unit (GPU). For example, if a GPU is present, some image/video rendering tasks, such as adjusting and rendering content and pointing entities, may be offloaded from the CPU to the GPU.

The communication module <NUM> includes one or more devices for communicating with other computing devices, such as the AR content source <NUM>. The communication module <NUM> may communicate via wireless or wired networks, such as the network <NUM>.

The camera assembly <NUM> captures images and/or videos of the physical space around the computing device <NUM>. The camera assembly <NUM> may include one or more cameras. The camera assembly <NUM> may also include an infrared camera. Images captured with the camera assembly <NUM> may be used to determine a location and orientation of the computing device <NUM> within a physical space, such as an interior space. For example, the computing device <NUM> may include a visual positioning system that compares images captured by the camera assembly <NUM> (or features extracted from those images) to a known arrangement of features within a physical space to determine the location of the computing device <NUM> within the space.

The computing device <NUM> may also include various user input components (not shown) such as a controller that communicates with the computing device <NUM> using a wireless communications protocol. In some implementations, the computing device <NUM> is a mobile device (e.g., a smartphone) which may be configured to provide or output AR content to a user via the HMD <NUM>. For example, the computing device <NUM> and the HMD <NUM> may communicate via a wired connection (e.g., a Universal Serial Bus (USB) cable) or via a wireless communication protocol (e.g., any WiFi protocol, any BlueTooth protocol, Zigbee, etc.). In some implementations, the computing device <NUM> is a component of the HMD <NUM> and may be contained within a housing of the HMD <NUM>.

Although the computing device <NUM> and the HMD <NUM> are shown as separate devices in <FIG>, in some implementations, the computing device <NUM> may include the HMD <NUM>. In some implementations, the computing device <NUM> communicates with the HMD <NUM> via a cable, as shown in <FIG>. For example, the computing device <NUM> may transmit video signals and/or audio signals to the HMD <NUM> for display for the user, and the HMD <NUM> may transmit motion, position, and/or orientation information to the computing device <NUM>.

The AR content source <NUM> may generate and output AR content, which may be distributed or sent to one or more computing devices, such as the computing device <NUM>, via the network <NUM>. In an example implementation, the AR content includes three-dimensional scenes and/or images. Additionally, the AR content may include audio/video signals that are streamed or distributed to one or more computing devices. The AR content may also include an AR application that runs on the computing device <NUM> to generate 3D scenes, audio signals, and/or video signals.

The network <NUM> may be the Internet, a local area network (LAN), a wireless local area network (WLAN), and/or any other network. A computing device <NUM>, for example, may receive the audio/video signals, which may be provided as part of AR content in an illustrative example implementation, via the network.

<FIG> is a third person view of an example physical space <NUM>, in which a user is experiencing an AR environment <NUM> through the example HMD <NUM>. The AR environment <NUM> is generated by the AR application <NUM> of the computing device <NUM> and displayed to the user through the HMD <NUM>.

The physical space <NUM> includes a physical location <NUM> that is identified by a marker <NUM>. In this example, the marker <NUM> includes a barcode disposed on a circular object. The location engine <NUM> may identify the middle of the circular object to identify a physical location, and the AR application <NUM> may identify content to display at that physical location in the AR environment <NUM> based, for example, on the barcode from the marker <NUM>. Additionally or alternatively, the marker <NUM> may include a QR code, an image, or a sticker.

For example, the marker <NUM> may have been placed in the physical space <NUM> by a teacher or guide. Although this example includes a marker <NUM>, some implementations identify the physical location <NUM> without a marker (e.g., based on the position and/or orientation of the computing device <NUM> as determined using a global positioning system (GPS), visual positioning system, other location determining technology, and/or sensors of the IMU). Further, the marker <NUM> does not necessarily include an identifier (e.g., barcode, QR code, etc.) in some implementations.

The AR environment <NUM> includes inserted content <NUM> that includes a point of interest (POI) <NUM> and a pointing entity <NUM> that is displayed over an image of the physical space <NUM>. In this example, the content <NUM> is a building and the POI <NUM> is a feature on a side of the building. The pointing entity <NUM> has a conical structure around an axis directed at the POI <NUM> in this example. The conical vertex of the pointing entity <NUM> is further away from the POI <NUM> than the conical base of the pointing entity <NUM>. Additionally, the pointing entity <NUM> is displayed with a transparency gradient in which the conical structure is fully opaque near the conical vertex and becomes fully transparent near the base. In <FIG>, the higher degree of transparency is shown with darker shading and the higher degree of opacity is shown with lighter shading. As can also be seen, the conical base of the pointing entity <NUM> is offset above the POI <NUM>.

In some implementations, the AR environment <NUM> is provided to the user as a single image or a pair of stereoscopic images that occupy substantially all of the user's field of view and are displayed to the user via the HMD <NUM>. In other implementations, the AR environment is provided to the user by displaying/projecting the inserted content <NUM> on an at least partly transparent combiner that occupies at least a portion of the user's field of view. For example, portions of the HMD <NUM> may be transparent, and the user may be able to see the physical space <NUM> through those portions while the HMD <NUM> is being worn.

<FIG> are perspective views of an example HMD <NUM>, such as, for example, the HMD <NUM> worn by the user in <FIG>, and <FIG> illustrates an example handheld electronic device <NUM> for controlling and/or interacting with the HMD <NUM>.

The handheld electronic device <NUM> may include a housing <NUM> in which internal components of the device <NUM> are received, and a user interface <NUM> on an outside of the housing <NUM>, accessible to the user. The user interface <NUM> may include a touch sensitive surface <NUM> configured to receive user touch inputs. The user interface <NUM> may also include other components for manipulation by the user such as, for example, actuation buttons, knobs, joysticks and the like. In some implementations, at least a portion of the user interface <NUM> may be configured as a touchscreen, with that portion of the user interface <NUM> being configured to display user interface items to the user, and also to receive touch inputs from the user on the touch sensitive surface <NUM>. The handheld electronic device <NUM> may also include a light source <NUM> configured to selectively emit light, for example, a beam or ray, through a port in the housing <NUM>, for example, in response to a user input received at the user interface <NUM>.

The HMD <NUM> may include a housing <NUM> coupled to a frame <NUM>, with an audio output device <NUM> including, for example, speakers mounted in headphones, also being coupled to the frame <NUM>. In <FIG>, a front portion 310a of the housing <NUM> is rotated away from a conical base portion 310b of the housing <NUM> so that some of the components received in the housing <NUM> are visible. A display <NUM> may be mounted on an interior facing side of the front portion 310a of the housing <NUM>. Lenses <NUM> may be mounted in the housing <NUM>, between the user's eyes and the display <NUM> when the front portion 310a is in the closed position against the conical base portion 310b of the housing <NUM>. In some implementations, the HMD <NUM> may include a sensing system <NUM> including various sensors and a control system <NUM> including a processor <NUM> and various control system devices to facilitate operation of the HMD <NUM>.

In some implementations, the HMD <NUM> may include a camera <NUM> to capture still and moving images. The images captured by the camera <NUM> may be used to help track a physical position of the user and/or the handheld electronic device <NUM> in the real world, or physical space relative to the augmented environment, and/or may be displayed to the user on the display <NUM> in a pass through mode, allowing the user to temporarily leave the augmented environment and return to the physical environment without removing the HMD <NUM> or otherwise changing the configuration of the HMD <NUM> to move the housing <NUM> out of the line of sight of the user.

For example, in some implementations, the sensing system <NUM> may include an inertial measurement unit (IMU) <NUM> including various different types of sensors such as, for example, an accelerometer, a gyroscope, a magnetometer, and other such sensors. A position and orientation of the HMD <NUM> may be detected and tracked based on data provided by the sensors included in the IMU <NUM>. The detected position and orientation of the HMD <NUM> may allow the system to detect and track the user's head gaze direction and movement.

In some implementations, the HMD <NUM> may include a gaze tracking device <NUM> to detect and track an eye gaze of the user. The gaze tracking device <NUM> may include, for example, an image sensor 365A, or multiple image sensors 365A, to capture images of the user's eyes, for example, a particular portion of the user's eyes, such as, for example, the pupil, to detect, and track direction and movement of, the user's gaze. In some implementations, the HMD <NUM> may be configured so that the detected gaze is processed as a user input to be translated into a corresponding interaction in the AR or VR environment.

In some implementations, the HMD <NUM> includes a portable electronic device, such as a smartphone, that is removably disposed within a chamber of the housing <NUM>. For example, the display <NUM> and the camera <NUM> may be provided by the portable electronic device. When the chamber is closed (as shown in <FIG>), the display <NUM> is aligned with the lenses <NUM> so that a user can view at least a portion of the display <NUM> (provided by the portable electronic device) through each eye. The camera <NUM> may align with an aperture in the housing <NUM> so that the portable electronic device of the HMD <NUM> can capture images while disposed in the housing <NUM>.

<FIG> is a schematic view of a user experiencing the AR environment <NUM> via an example portable electronic device <NUM>. The portable electronic device <NUM> is an example of the computing device <NUM>. The portable electronic device <NUM> may be a smartphone, a tablet, or another type of portable computing device. In this example, the user is experiencing the AR environment through a display device <NUM> of the portable electronic device. For example, the display device <NUM> may include a screen that can show images and/or videos.

<FIG> is a diagram of an example method <NUM> of inserting content and drawing attention to a POI, in accordance with implementations described herein. This method <NUM> may for example be performed by the computing device <NUM> to provide a virtual field trip or tour experience using an AR environment for a user.

At operation <NUM>, an image is received. For example, the image may be captured by a camera assembly of a computing device, such as the computing device <NUM>. The captured image may be stored as a texture in the image buffer <NUM>. In some implementations, the image may be received from a storage location such as a computer-readable memory device or from another computing device via a network.

At operation <NUM>, content to display is identified. The content may be identified based on a predefined virtual tour of field trip. For example, the virtual tour or field trip may define a sequence of content to display. Initially, the first content from the sequence is identified for display. Thereafter, the content identified for display may be updated to the next content in the sequence in response to a user input (e.g., from a teacher or guide) or based on a predetermined time period elapsing.

At operation <NUM>, a location to display the content is identified within the image. As described previously, the physical location may be identified based on a marker or based on a physical coordinate determined with respect to the location of the computing device <NUM>. Based on the identified physical location, a three-dimensional coordinate in the AR environment may be identified. The identified content may be displayed at the identified location in a particular orientation, which may be configured by, for example, a teacher, tour guide, or user who has setup a virtual tour.

At operation <NUM>, a POI within the content to display is identified. The POI can be anything a teacher, tour guide, or anyone else would like to draw attention to. For example, the POI may be an architectural feature of the building shown in <FIG>. The POI may be positioned anywhere on the content. Depending on the user's point of view and the orientation of the content, the POI may be disposed on a portion of the content that is not facing toward the user. Additionally, in some implementations, multiple POIs are associated with the content. In these implementations, one of those POIs may be identified initially.

At operation <NUM>, the content is displayed over the image in a manner that draws attention to the identified POI. For example, a light sphere effect (e.g., using differential shading parameters) may be applied to the content when it is rendered so that a portion of the content associated with (or near) the POI is displayed using brighter lighting settings than the rest of the content. If the POI is on a side of the content the user can see, the user would then see the POI as a bright spot on the content. Additionally, in some implementations, a pointing entity is rendered to draw the user's attention to a POI. For example, the pointing entity may help the user find a POI that is located on a side of the content that cannot be seen from the user's current position. In some implementations, the light sphere and pointing entity generate a spotlight-like effect.

<FIG> is a diagram of an example method <NUM> of inserting content and drawing attention to a POI, in accordance with implementations described herein. This method <NUM> may, for example, be performed by the computing device <NUM> to provide a virtual field trip or tour experience using an AR environment for a user.

At operation <NUM>, content to display over an image and a POI within the content are identified. Examples of identifying content and POIs are described with respect to <FIG>. An illustration 700A of a side view of an example of identified content <NUM> and a POI <NUM> on the content <NUM> are shown in <FIG>. The content <NUM> and POI <NUM> are also shown in <FIG>, which provides an overhead view.

At operation <NUM>, a portion of the content is identified based on the POI. For example, the portion may be identified based on distance to a three-dimensional coordinate associated with the POI. In some implementations, the portion is identified based on a sphere surrounding the POI (e.g., all parts of the content that are within a pre-determined threshold distance from the three-dimensional coordinate are identified as the portion). In <FIG>, an example identified portion <NUM> is shown. The example identified portion <NUM> is identified based on being contained within a sphere <NUM> that is centered at the POI <NUM>.

At operation <NUM>, a pointing origin is identified based on the user's position and the point of interest. An example pointing entity <NUM> having a pointing origin <NUM> is shown in <FIG>. The pointing origin <NUM> may be the location at which the pointing entity <NUM> appears to originate. The pointing entity <NUM> may be aligned with an axis running between the pointing origin <NUM> and the POI <NUM>. For example, <FIG> shows an axis <NUM> around which the pointing entity <NUM> is disposed.

In some implementations, the pointing origin <NUM> is vertically offset from the content <NUM> so that the pointing entity <NUM> appears to originate above or below the content <NUM>. In some implementations, the vertical offset is a static value. In other implementations, the vertical offset is determined based on the vertical dimension of the content and the user's position with respect to the content. For example, a vertical offset may be determined so that the pointing origin <NUM> is not occluded in the user's field of view by content <NUM>. Thus, a larger vertical offset may be selected when the user is nearer to the content <NUM> and/or when the content has a larger vertical dimension. Additionally, the direction of the vertical offset may be determined based on the normal vector of the surface of the content at the POI. For example, the vertical offset may be negative (below) when the normal vector of the surface has a component that is directed downward and the vertical offset may be positive (above) when the normal vector of the surface has a component that is directed upward.

In some implementations, the pointing origin <NUM> is also laterally offset from the content <NUM>. For example, <FIG> illustrates one technique for laterally offsetting the pointing origin <NUM>. A circle <NUM> is disposed around the content <NUM>. In some implementations, the circle <NUM> may be centered at a midpoint of the content <NUM>. Additionally, the circle may be sized to fully contain the content. In some implementations, the circle is further scaled by a pre-determined factor that is greater than <NUM> (e.g., <NUM>, <NUM>). In this example, the circle <NUM> is approximately <NUM>% greater than a circle that would fully contain the content <NUM> (i.e., a scaling factor of approximately <NUM> is used). In other implementations, different scaling factors can be used. A line is projected from the user U through the POI <NUM> to the circle <NUM> to generate an initial pointing origin <NUM>. The initial pointing origin <NUM> can be used to calculate an initial pointing entity <NUM> around an initial pointing axis <NUM>.

In some implementations, an angle between the initial pointing axis <NUM> and the line <NUM> is determined. In this example, the initial point axis <NUM> is colinear with the line <NUM> so that angle is zero. If the angle is less than a predetermined factor, the initial pointing origin <NUM> may be shifted along the perimeter of the circle <NUM>. In this example, the initial pointing origin <NUM> is shifted to the pointing origin <NUM>.

For example, the pointing origin <NUM> may be shifted by a pre-determined factor (e.g., <NUM> degrees) to one side or another of the user. In some implementations, the side is selected to indicate the direction the user should walk to get to the POI (e.g., the path around the content on the right side may be shorter than on the left side). As can be seen in <FIG>, after shifting, the pointing axis <NUM> would not be colinear with a line between the user and the POI <NUM>. In this manner, when the pointing entity <NUM> is displayed, it will not be pointing directly at the user. Instead, the user will see an angled/side view of the pointing entity <NUM>, which may be more helpful in determining where to move to see the POI <NUM>. In some implementations, as the user moves, the location of the pointing origin <NUM> is updated.

In some implementations, the pointing origin <NUM> may be determined at least in part based on the normal direction of the surface of the content at the POI (or the average normal of the surface of the content surrounding the POI). The pointing origin <NUM> may also be shifted or moved so that it does not pass through the user. In some implementations, when the user is in the path of the pointing entity, the pointing entity may not be rendered anymore. In some implementations, the pointing origin may also be determined based on properties of the actual physical location. For example, the pointing origin may be aligned with a known light source in the room or another common location in the physical location (e.g., such as a teacher's desk or a physical location of a teacher or guide). In some implementations, the content may be rotated to align the POI with lighting of the room (e.g., if there is a single light source in the physical room). The pointing entity and pointing origin may also be associated with a physical device (e.g., the teacher or guide's smartphone) and may move around as the device moves. Additionally, the pointing entity may be connected to a flashlight function of the smartphone (e.g., when the teacher enables a flashlight on a smartphone, the pointing entity is activated).

In some implementations, users may add POIs, which may then be shared with other users in the AR or VR environment. For example, the user may touch a portion of a screen corresponding to a desired POI on the screen of a smartphone. As another example, a teacher or guide may view a representation of all of the users in the VR or AR environment with pointing entities emanating from each of the user toward the area that user is focusing on in the AR or VR environment. In this manner, the teacher or guide may learn about or observe what participants are interested in and how the participants interact with the AR or VR environment.

At operation <NUM>, a pointing entity is rendered over the image based on the pointing origin and the POI. As described above, in some implementations, the pointing entity is aligned with a pointing axis (i.e., a line from the pointing origin to the POI). The pointing axis may be determined based on the surface normal of the content at (or near) the POI. For example, the pointing entity may be oriented based on the surface normal. The pointing axis may also be determined, at least in part, based on the position of the user. For example, the pointing axis may be determined so as to increase the projection of the pointing axis onto a view plane of the user.

The pointing entity may have a conical shape that is centered around the pointing axis. The conical-shaped pointing entity may have a conical vertex at the pointing origin that expands to a conical base in the direction of the POI. In some implementations, the pointing entity may be rendered with a gradually transitioning transparency. For example, the conical vertex may have a lower transparency value (e.g., the conical vertex may be completely opaque) and the conical base may have a higher transparency value (e.g., the conical base may be completely transparent).

<FIG> shows an example of the conical shape of the example pointing entity <NUM> and an alpha (transparency) gradient that is applied to the pointing entity <NUM>. In some implementations, the pointing entity is not rendered at all when the user is facing the POI and can see the POI as the pointing entity may no longer be necessary to guide the user to the POI.

At operation <NUM>, the content is rendered over the image using different shading parameter for the identified portion relative to the rest of the content. In some implementations, the identified portion is rendered using a first shading factor, while the rest of the content is rendered using a second shading factor. The shading factors may be multiplied by the color values of the pixels that are being rendered. A higher shading factor will result in a pixel that appears brighter. For example, the first shading factor may be greater than the second shading factor so that the pixels within the identified portion are brighter than the rest of the content. This may have the effect of causing the POI to appear within a sphere of light. In some implementations, the first brightness factor is <NUM> and the second brightness factor is <NUM>. Because the first brightness factor is <NUM>, the colors within the identified portion are not modified, but the portion still appears brighter in comparison to the rest of the content. Beneficially, this may preserve the original coloring of the POI and prevent wash-out of the colors due to bright lighting. <FIG> and <FIG> show an example of the sphere <NUM> used to identify the portion <NUM>, which appears more brightly than the rest of the content <NUM>.

Although the operations of the methods <NUM> and <NUM> are described sequentially, at least some of the operations may be performed in different orders or simultaneously.

<NUM> shows an example of a computer device <NUM> and a mobile computer device <NUM>, which may be used with the techniques described here.

Expansion memory <NUM> may also be provided and connected to device <NUM> through expansion interface <NUM>, which may include, for example, a SIMM (Single In-Line Memory Module) card interface. Thus, for example, expansion memory <NUM> may be provided as a security module for device <NUM>, and may be programmed with instructions that permit secure use of device <NUM>.

In addition, short-range communication may occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown).

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (a LED (light-emitting diode), or OLED (organic LED), or LCD (liquid crystal display) monitor/screen) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer.

In some implementations, the computing devices depicted in FIG. <NUM> can include sensors that interface with an AR headset/HMD device <NUM> to generate an augmented environment for viewing inserted content within the physical space. For example, one or more sensors included on a computing device <NUM> or other computing device depicted in FIG. <NUM>, can provide input to the AR headset <NUM> or in general, provide input to an AR space. The sensors can include, but are not limited to, a touchscreen, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device <NUM> can use the sensors to determine an absolute position and/or a detected rotation of the computing device in the AR space that can then be used as input to the AR space. For example, the computing device <NUM> may be incorporated into the AR space as a virtual object, such as a controller, a laser pointer, a keyboard, a weapon, etc. Positioning of the computing device/virtual object by the user when incorporated into the AR space can allow the user to position the computing device so as to view the virtual object in certain manners in the AR space. For example, if the virtual object represents a laser pointer, the user can manipulate the computing device as if it were an actual laser pointer. The user can move the computing device left and right, up and down, in a circle, etc., and use the device in a similar fashion to using a laser pointer.

In some implementations, one or more input devices included on, or connect to, the computing device <NUM> can be used as input to the AR space. The input devices can include, but are not limited to, a touchscreen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, earphones or buds with input functionality, a gaming controller, or other connectable input device. A user interacting with an input device included on the computing device <NUM> when the computing device is incorporated into the AR space can cause a particular action to occur in the AR space.

In some implementations, a touchscreen of the computing device <NUM> can be rendered as a touchpad in AR space. A user can interact with the touchscreen of the computing device <NUM>. The interactions are rendered, in AR headset <NUM> for example, as movements on the rendered touchpad in the AR space. The rendered movements can control virtual objects in the AR space.

In some implementations, one or more output devices included on the computing device <NUM> can provide output and/or feedback to a user of the AR headset <NUM> in the AR space. The output and feedback can be visual, tactical, or audio. The output and/or feedback can include, but is not limited to, vibrations, turning on and off or blinking and/or flashing of one or more lights or strobes, sounding an alarm, playing a chime, playing a song, and playing of an audio file. The output devices can include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), strobes, and speakers.

In some implementations, the computing device <NUM> may appear as another object in a computer-generated, 3D environment. Interactions by the user with the computing device <NUM> (e.g., rotating, shaking, touching a touchscreen, swiping a finger across a touch screen) can be interpreted as interactions with the object in the AR space. The actions could alter the display of content for a single user or for multiple users (e.g., a teacher or guide's actions may alter the display of content for all users participating in the virtual tour or field trip). In the example of the laser pointer in a AR space, the computing device <NUM> appears as a virtual laser pointer in the computer-generated, 3D environment. As the user manipulates the computing device <NUM>, the user in the AR space sees movement of the laser pointer. The user receives feedback from interactions with the computing device <NUM> in the AR environment on the computing device <NUM> or on the AR headset <NUM>.

In some implementations, a computing device <NUM> may include a touchscreen. For example, a user can interact with the touchscreen to interact with the AR environment. For example, the touchscreen may include user interface elements such as sliders that can control properties of the AR environment, such as selecting a POI or adjusting the brightness with which a POI is displayed. The user interface elements may also allow a user to switch how content is scaled (e.g., switching from scaling based on a relative scale to a real-world scale). Some implementations also respond to 2D touch screen gestures such as pinches or swipes to change the content (e.g., to scale, rotate, or switch content or display properties). For example, a user can interact with the touchscreen in a particular manner that can mimic what happens on the touchscreen with what happens in the AR space. For example, a user may use a pinching-type motion to zoom/scale content displayed on the touchscreen. This pinching-type motion on the touchscreen can cause information provided in the AR space to be zoomed/scaled. Additionally, in some implementations, the computing device <NUM> may support 3D gestures in the physical world. For example, reaching the computing device <NUM> out into displayed content may be interpreted as a particular input by the computing device <NUM>. Actuating a physical button, touchscreen virtual button, or other user actuatable input (e.g., a squeeze or shake motion) of the computing device may be recognized as another input by the computing device <NUM>. Some implementations treat combinations of these actions as gestures that correspond to commands within the AR environment. For example, some implementations may scale content in response to a user reaching out the computing device into a defined volume, squeezing the computing device, and pulling the computing device back. As another example, a casting motion while holding the computing device <NUM> may be recognized as a command to grab an item.

In some implementations, one or more input devices in addition to the computing device (e.g., a mouse, a keyboard) can be rendered in a computer-generated, 3D environment. The rendered input devices (e.g., the rendered mouse, the rendered keyboard) can be used as rendered in the AR space to control objects in the AR space.

Computing device <NUM> is intended to represent various forms of digital computers and devices, including, but not limited to laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the specification.

In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other embodiments are within the scope of the following claims.

Claim 1:
A computer-implemented method comprising:
receiving an image;
identifying content to display over the image for a user;
identifying a location within the image to display the content;
identifying a point of interest of the content;
triggering display of the content overlaid on the image by:
identifying a portion of the content based on the point of interest,
rendering the portion of the content using first shading parameters, and
rendering the content other than the portion using second shading parameters;
generating a pointing entity directed at the point of interest the pointing entity having a pointing origin identified based on a position of the user in a physical space and the point of interest and indicating a position and/or direction for the user from which to view the point of interest; and
triggering display of the pointing entity overlaid on the image.