Extended reality information for identified objects

One embodiment provides a method, including: receiving, at an information handling device, an indication to display extended reality content; determining, using a processor, a user's geographic position in an area; accessing, from an accessible storage location, map data associated with the area; identifying, using at least one sensor associated with the information handling device, a user's line of sight; determining, based on the user's geographic position and the map data, an object associated with the user's line of sight; determining, using a processor, an identity of the object; and displaying the extended-reality content for the identified object in a field of view of the information handling device. Other aspects are described and claimed.

BACKGROUND

Advances in technology have led to the development of information handling devices (“devices”), for example smart phones, tablet devices, head-mounted wearable headsets, and the like, capable of displaying a variety of different types of extended reality (“XR”) content such as augmented reality (“AR”) content, mixed reality (“MR”) content, virtual reality (“VR”) content, a combination thereof, and the like. These XR-enabled devices may provide an overlay of virtual elements onto a live view of a physical, real-world environment. The virtual elements may be spatially registered with the physical world so that the virtual elements may be perceived as an immersive aspect of the physical world.

BRIEF SUMMARY

In summary, one aspect provides a method, comprising: receiving, at an information handling device, an indication to display extended reality content; determining, using a processor, a user's geographic position in an area; accessing, from an accessible storage location, map data associated with the area; identifying, using at least one sensor associated with the information handling device, a user's line of sight; determining, based on the user's geographic position and the map data, an object associated with the user's line of sight; determining, using a processor, an identity of the object; and displaying the extended-reality content for the identified object in a field of view of the information handling device.

Another aspect provides an information handling device, comprising: at least one sensor; a processor; a memory device that stores instructions executable by the processor to: receive an indication to display extended reality content; determine a user's geographic position in an area; access, from an accessible storage database, map data associated with the area; identify a user's line of sight; determine, based on the user's geographic position and the map data, an object associated with the user's line of sight; determine an identity of the object; and display the extended content for the identified object in a field of view of the information handling device.

A further aspect provides a product, comprising: a storage device that stores code, the code being executable by a processor and comprising: code that receives an indication to display extended reality content; code that determines a user's geographic position in an area; code that accesses map data associated with the area; code that identifies a user's line of sight; code that determines, based on the user's geographic position and the map data, an object associated with the user's line of sight; code that determines an identity of the object; and code that displays the extended reality content for the identified object.

DETAILED DESCRIPTION

Individuals may desire to obtain additional information regarding certain objects in their surroundings. For example, a tourist visiting a city for the first time may wish to obtain additional information (e.g., historical information, structural information, etc.) about buildings, monuments, statues, and the like that they see in the city. Advances in technology have enabled users to search for this information on their mobile device (e.g., using a search engine on their smart phone, tablet, other electronic device, etc.). However, such a method requires a user to provide explicit search input to the device, which requires additional effort from the user and may be time-consuming. Additionally, if the user does not know the name of the object, or cannot describe a defining characteristic of the object, conducting a search for information about the object may be difficult and time-consuming.

An alternative method of receiving information about an object is to utilize a device that can automatically identify objects proximate to a user and thereafter provide informative output associated with those objects. For example, a head-mounted display (HMD) may be worn by a user that can display information about proximate objects in mixed or augmented reality. Such a device may utilize global positioning system (GPS) sensors integrated into the HMD to identify a user's geographic position. The device may also access map data, or GPS data associated with other objects, to identify the geographic positions of surrounding objects. The device may then utilize this information to determine whether a user is within a threshold distance from an object. Responsive to determining that the user is within a threshold distance from an object, the device may automatically provide information about that object.

Although the foregoing method has its advantages over manual user search attempts, it is still wrought with a variety of issues. For instance, in a situation where an object-of-interest is far away from the user, geographic position sensors alone would not be able to accurately identify the distant object, especially if the object-of-interest is positioned among a plurality of other objects. In another situation, if a user is standing proximate to a plurality of objects, the device may not know which object to provide information on because a user may be within a threshold distance from a plurality of objects.

Accordingly, an embodiment provides a method for displaying information about an object a user is looking at in their device's field of view. In an embodiment, an indication may be received by an extended reality-capable device to display content associated with an object. The indication may result from a user-provided command to display information (e.g., a gesture command, a verbal command, etc.). Alternatively, the indication may be generated responsive to positively identifying an object the user is looking at. In order to accurately identify the object, an embodiment may collect various types of information associated with the user's contextual situation. For example, an embodiment may determine a user's geographic position in an area (e.g., using GPS data, etc.), an embodiment may access map data associated with the area, and an embodiment may identify a user's line of sight by using one or more sensors. An embodiment may thereafter utilize this combined information to accurately determine the object the user is looking. Responsive to making this determination, an embodiment may display extended reality content about the object in a field of view of the device. For example, an embodiment may display historical information about a building in a space in the device's field of view that overlaps the building. Such a method may more accurately identify objects users are interested in, enabling the user to receive relevant information about those objects.

System100typically includes one or more of a WWAN transceiver150and a WLAN transceiver160for connecting to various networks, such as telecommunications networks and wireless Internet devices, e.g., access points. Additionally, devices120are commonly included, e.g., an image sensor such as a camera, audio capture device such as a microphone, motion sensor such as an accelerometer or gyroscope, a thermal sensor, etc. System100often includes one or more touch screens170for data input and display/rendering. System100also typically includes various memory devices, for example flash memory180and SDRAM190.

Information handling device circuitry, as for example outlined inFIG. 1orFIG. 2, may be used in devices such as smart phones, tablets, AR or MR capable HMDs, personal computer devices generally, and/or electronic devices that are capable of capturing worldview image data and displaying AR or MR content on one or more display screens of the device. For example, the circuitry outlined inFIG. 1may be implemented in a tablet or smart phone embodiment, whereas the circuitry outlined inFIG. 2may be implemented in an HMD embodiment.

Referring now toFIG. 3, an embodiment may accurately identify an object a user is looking at and thereafter provide information about that object on an extended reality capable display. At301, an embodiment may receive an indication to display extended reality (“XR”) content related to an object in a user's field of view. In the context of this application, XR-content may refer to augmented reality (“AR”) content, mixed reality (“MR”) content, a combination thereof, and the like. The content may be displayed on one or more displays of an XR-capable device (e.g., a smart phone or tablet capable of displaying XR content on a display screen of a device, a head-mounted display (HMD) capable of displaying content on a transparent, or semi-transparent, visor or display lens, etc.). In the context of this application, the user's field of view may correspond to the observable area a user may be able to see through their eyes or through a transparent, or semi-transparent, screen/lens associated with a device. For simplicity purposes, the majority of the discussion herein will involve AR content displayed on an AR-HMD. However, it should be understood that such a generalization is not limiting and any XR-capable device may be utilized to execute the processes of the underlying application.

In an embodiment, the object may be virtually any outdoor object that location data can be obtained for such as a building, statue, monument, artistic structure, park, and the like. In an embodiment, the content may comprise information about an object. For example, the information may be historical information, structural information, trivia information, etc. For instance, an embodiment may be able to identify when a structure was built, by who it was built, the type of structure it is, a major event that occurred at the structure, other information about the structure, etc.

In an embodiment, the indication may be associated with a user-provided command to display content. The command may be detected at an input device (e.g., an audio capture device, an image and/or video capture device, physical or digital keyboard, a touch screen, etc.) and may be provided by any known method of providing a command (e.g., touch input, voice input, gesture input, etc.). As a non-limiting example, a user may perform a pointing gesture at an object they are looking. Recognition of this gesture may provide an indication to the device to begin the content display process. In another non-limiting example, a user may provide a verbal expression comprising one or more words or phrases that a system may recognize as being associated with content display. For instance, a user may say “what's that?”, “tell me about that”, “I don't know what that is”, etc. In another embodiment, the indication may be created responsive to the device identifying an object. For example, a user may be running an object identifying application that may be configured to automatically output content associated with the object responsive to positively identifying the object.

In order to display information about an object, an embodiment must first identify the object a user is interested in receiving information about and thereafter identify what the object is. An embodiment may accomplish this by acquiring three different types of information: a user's geographic position, map data associated with a user's geographic area, and line-of-sight data identifying a gaze direction of the user.

At302, an embodiment may determine a user's geographic position in an area. This determination may be accomplished, for example, by receiving Global Positioning System (GPS) data from one or more GPS receivers integrated into the user's device. Alternatively, GPS data may be received from other devices that are associated with a user. For example, an embodiment may receive GPS data from a user's smart watch, smart phone, other wearable electronic device, a combination thereof, etc. In another embodiment, a user's geographic position may be identified based upon data received from other, non-user devices identified as being associated with the user's position. For example, an embodiment may access social media data to identify that another user has just tagged the user as being present in a particular location. Other conventional location identification methods may also be utilized such as geolocation, radiolocation, and other conventional types of position tracking methods utilized by various other positioning systems. The aforementioned determination methods may be used alone or in combination. For example, an embodiment may utilize more than one determination method in order to obtain confirmation, or higher confidence, in their location identification determination.

At303, an embodiment may access map data associated with the area. In an embodiment, the map data may be three-dimensional (3D) map data that may comprise accurate location information for all objects encompassed in the area. In an embodiment, the map data that is accessed and/or utilized may correspond to map data a predetermined distance around a user's geographic position. For example, an embodiment may access map data for all objects located: 500 feet around a user, 1 square mile around a user, in the city a user is identified as being located in, etc. The amount of map data accessed/utilized may be set by a manufacturer and/or adjusted by a user. In an embodiment, the map data may be accessed from an accessible storage location. The accessible storage location may be stored locally, on the device, or may be stored remotely (e.g., on another device, on another server, etc.) and accessed by the device using a wireless connection.

At304, an embodiment may identify a user's line of sight. In this regard, an embodiment may identify a gaze direction and a gaze point of the user. In an embodiment, one or more sensors may be integrated into the device that may be able to identify both of these data points. For instance, with respect to the gaze direction, an embodiment may comprise a gyroscopic compass that may be able to identify a direction that a user is facing. Additionally or alternatively, an embodiment may obtain information from a navigation and/or a map application that is able to make this determination. With respect to the gaze point, an embodiment may utilize conventional eye-tracking processes to identify a point in the user's field of view they are looking at. With these two combined pieces of information, an embodiment may be able to identify the direction a user is oriented and a point in that direction the user is looking at.

At305, an embodiment may determine an object associated with the user's gaze. An embodiment may make this determination, for example, by first utilizing the location data to identify a user's position in geographic space. An embodiment may then access map data associated with the user's position to identify all of the objects surrounding a user. An embodiment may thereafter determine a user's line of sight by identifying a user's gaze direction and gaze point. With all data points obtained, an embodiment may be able to identify the object a user is looking at. For example, an embodiment may be able to identify where a line representative of a user's line of sight intersects with an object in the map data. It is important to note that although the obtainment of map data was described first, this step does not always need to occur prior to the determination of a user's line of sight. Stated differently, an embodiment may determine a user's line of sight prior to obtaining map data. Alternatively, an embodiment may procure both pieces of information substantially simultaneously.

Responsive to not determining, at305, the object associated with the gaze direction, an embodiment may, at306, do nothing. Alternatively, an embodiment may output a notification to the user (e.g., an audible notification, a visual notification, etc.) that an object could not be accurately identified. Conversely, responsive to determining the object associated with the user's line of sight, an embodiment may, at307, identify what the object actually is. In an embodiment, map data may comprise object identity information for all objects in the user's area. Accordingly, an embodiment may determine an object's identity by identifying the object associated with the user's line of sight and thereafter obtaining the name for that object from the map data. In another embodiment, the user's device may capture an image of the object that may subsequently be provided into an image-based search engine that may be able to determine the object's identity. Responsive to determining the object's identity, an embodiment may also be able to access additional information about the object by referring to a data store accessible to the device (e.g., stored locally, available on a website online, etc.).

At308, an embodiment may display content associated with the object in a field of view of the device. In an embodiment, the content may be virtually any type of content that is able to be visualized by a user. For example, the content may comprise text, colors, graphics, animations, a combination of the foregoing, and the like. In an embodiment, content may be displayed on one or more portions of the display such as a central portion, a peripheral portion, other portions, etc. For example, in a situation where historical information was being provided to the user about a building, an embodiment may display text on portions of the display that do not obstruct a user's view of the building. Alternatively, in a similar situation, an embodiment may be able to display content overtop a display portion of the object. For example, an embodiment may be able to identify the boundaries of a building and thereafter adjust the font size, borders, and position of an augmented text block to conform to the dimensions and display position of the building. In a situation where an object occupies all, or a majority of, the display field of view (e.g., because it is large and close to a user, etc.), an embodiment may display content on a predetermined portion of the display according to a policy. For example, an embodiment may display content on a periphery of the display so as not to interfere with a user's field of view. An embodiment may also be able to visually distinguish a portion of the object. For example, if an important event happened on the 22ndfloor of a 50 story building, an embodiment may highlight (e.g. in a predetermined color, etc.) the entire floor, provide a visual indicator of the floor (e.g., an arrow pointing to the floor-of-interest, etc.).

The various embodiments described herein thus represent a technical improvement to conventional object-of-interest identification techniques. Using the techniques described herein, an embodiment may receive an indication to display extended reality content in a user's field of view. An embodiment may then determine an object associated with a user's line of sight by: identifying a user's geographic position in an area, accessing high quality map data, and identifying a user's line of sight. Responsive to determining the object associated with the user's line of sight, an embodiment may determine an identity of the object and thereafter display content associated with that object in a field of view of the user's device. Such a method may more accurately identify an object a user is interested in receiving additional information about.