Marker positioning for augmented reality overlays

Methods, systems, and computer programs for nesting augmented reality markers within augmented reality overlays to create nested menu options in augmented reality applications are provided. An input image is received from an augmented reality input buffer. The input buffer is a digital representation of information taken from a camera. An output image including an overlay is received from an augmented reality output buffer. The output buffer is scanned for one or more markers, where the markers are associated with the overlay. A first user input is received, indicating a user selection of a first marker. A first marker overlay is displayed. The first marker overlay corresponds to the first marker.

BACKGROUND OF THE INVENTION

The present invention relates generally to the field of augmented reality, and more particularly to the use of nested augmented reality markers within augmented reality overlays to create nested menu options in augmented reality applications.

Augmented reality (AR) is a view of a physical, real-world environment with elements augmented by a computer-generated sensory input. Computer-generated sensory inputs can include sound, video, graphics, or global positioning system (GPS) data. Generally, augmentation is in real-time and in semantic context with environmental elements. Often, AR display devices can align virtual information with the physical world. In some examples, virtual information is layered over the physical world view on a computing device. The virtual information layer is an overlay.

SUMMARY

According to one embodiment of the present invention, a method for nesting augmented reality overlays is provided. The method includes receiving, by one or more processors, an input image from an augmented reality input buffer, wherein the input buffer is a digital representation of information taken from a camera; receiving, by one or more processors, an output image from an augmented reality output buffer, wherein the output buffer comprises an overlay; scanning, by one or more processors, the output buffer image for one or more markers, wherein the one or more markers are associated with one or more marker overlays; receiving, by one or more processors, a first user input, wherein the first user input indicates a user selection of a first marker of the one or more markers; and displaying, by one or more processors, a first marker overlay, wherein the first marker overlay is associated with the first marker.

According to another embodiment of the present invention, a computer program product for nesting augmented reality overlays is provided. The computer program product comprises a computer readable storage medium and program instructions stored on the computer readable storage medium. The program instructions include program instructions to receive an input image from an augmented reality input buffer, wherein the input buffer is a digital representation of information taken from a camera; program instructions to receive an output image from an augmented reality output buffer, wherein the output buffer comprises an overlay; program instructions to scan the output buffer image for one or more markers, wherein the one or more markers are associated with one or more marker overlays; program instructions to receive a first user input, wherein the first user input indicates a user selection of a first marker of the one or more markers; and program instructions to display a first marker overlay, wherein the first marker overlay is associated with the first marker.

According to another embodiment of the present invention, a computer system for nesting augmented reality overlays is provided. The computer system includes one or more computer processors, one or more computer readable storage media, and program instructions stored on the computer readable storage media for execution by at least one of the one or more processors. The program instructions include program instructions to program instructions to receive an input image from an augmented reality input buffer, wherein the input buffer is a digital representation of information taken from a camera; program instructions to receive an output image from an augmented reality output buffer, wherein the output buffer comprises an overlay; program instructions to scan the output buffer image for one or more markers, wherein the one or more markers are associated with one or more marker overlays; program instructions to receive a first user input, wherein the first user input indicates a user selection of a first marker of the one or more markers; and program instructions to display a first marker overlay, wherein the first marker overlay is associated with the first marker.

DETAILED DESCRIPTION

Augmented reality (AR) displays enable a user to merge real world experience with a virtual world via a visual overlay to supplement what the user views. Connection to various databases, via a computer network, allows AR displays to add information to the user's view through an overlay. For example, if a user's view includes a landmark, the AR overlay may provide historical facts, visitor center information, etc. An embodiment of the present invention recognizes that AR overlays can enhance user experiences. AR overlays are produced based on markers associated with physical indicators in the real world. In some embodiments, the marker is a computer generated user interface element shown in the AR display. Generally, information associated with a marker is displayed on the AR display when a user interacts with the marker (e.g., the user focuses on the marker for a predetermined time). However, an embodiment of the present invention recognizes that all information associated with a marker is generally displayed simultaneously. Some embodiments recognize that the information associated with a marker can take up a large portion of the AR display. Further recognized, downloading information associated with a marker consumes computing resources, which is wasteful if the user is interested in only a sub-set of the information.

An embodiment of the present invention provides nested markers within AR overlays. The nested markers are used to create multi-level AR overlays. For example, multi-level overlays are used to create nested menus. Nested marker overlays allow information associated with a physical object to be shown in smaller increments. For example, where a user views a landmark, a marker associated with the landmark provides a menu. The menu may have multiple markers, each associated with a menu option. Nested markers allows the user to choose what information is shown. In some embodiments, nested markers conserves resources, where the user does not want to download all of the information. In some embodiments, nested markers can conserve space on the AR display. For example, a small portion of the information, pertaining to one of the menu options, is displayed on the AR display.

The present invention will now be described in detail with reference to the Figures.FIG. 1is a functional block diagram illustrating a computing environment, in accordance with an embodiment of the present invention. For example,FIG. 1is a functional block diagram illustrating augmented reality computing environment100. Augmented reality computing environment100includes computing device102and client device110connected over network120. Computing device102includes database104.

In various embodiments, computing device102is a computing device that can be a standalone device, a server, a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), or a desktop computer. In another embodiment, computing device102represents a computing system utilizing clustered computers and components to act as a single pool of seamless resources. In general, computing device102can be any computing device or a combination of devices with access to client device110, and with access to and/or capable of executing database104. Computing device102may include internal and external hardware components, as depicted and described in further detail with respect toFIG. 4.

In this exemplary embodiment, database104stored on computing device102. In other embodiments, database104may reside on another computing device, provided that database104can access and is accessible by each of augmented reality (AR) marker program112and camera116. In yet other embodiments, database104may be stored externally and accessed through a communication network, such as network120. Network120can be, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the two, and may include wired, wireless, fiber optic or any other connection known in the art. In general, network120can be any combination of connections and protocols that will support communications between computing device102and client device110, in accordance with a desired embodiment of the present invention.

In one embodiment, database104resides on computing device102. In another embodiment, database104resides on client device110, or on another device or component (not shown) within augmented reality computing environment100accessible via network120. Database104is a data repository that may be written to and read by AR marker program112and camera116. Database104can be implemented with any type of storage device capable of storing data that may be accessed and utilized by computing device102, such as a database server, a hard disk drive, or a flash memory. In other embodiments, database104can represent multiple storage devices within computing device102. Database104stores data regarding AR overlays, markers, and information related to objects and locations that a user of client device110may access or view. Database104may receive updates, via network120, regarding new objects and locations, as well as additional information related to objects and locations that are currently stored.

In various embodiments of the present invention, client device110can be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, or any programmable electronic device capable of communicating with computing device102via network120. Client device110can be a wearable computer. Wearable computers are electronic devices worn by the user (e.g., as glasses, hats, clothing, accessories, etc.). In general, client device110represents any programmable electronic device or combination of programmable electronic devices capable of executing machine readable program instructions and communicating with other computing devices via a network, such as network120. Client device110includes AR marker program112, user interface114, and camera116. Client device110may include internal and external hardware components, as depicted and described in further detail with respect toFIG. 4.

AR marker program112operates identify markers in AR overlays. The AR display is comprised of two components: an input buffer and an output buffer. The input buffer is the digital representation of information taken from the device's camera. For example, the input buffer includes a user's view of the real world created by camera116. In some embodiments, a marker is associated with elements of the input buffer. When AR marker program112identifies a marker in the input buffer, the overlay for the marker is generated. The output buffer is the digital representation of the computer generated image that the user sees through the device. For example, the overlays are elements of the output buffer. In some embodiments, markers are associated with areas or elements of the overlay. In some embodiments, AR marker program112identifies markers in the overlay and generates a new overlay.

Client device110includes a user interface (UI)114, which includes software, hardware, or a combination thereof. Software of user interface114executes locally on client device110and operates to provide a UI to a user of client device110. User interface114further operates to receive user input from a user via the provided user interface, thereby enabling the user to interact with client device110. In one embodiment, user interface114provides a user interface that enables a user of client device110to interact with AR marker program112of client device110. In various examples, the user interacts with AR marker program112in order configure markers within an AR overlay. In one embodiment, user interface114includes software stored on client device110. In other embodiments, user interface114includes software stored on another computing device (e.g., computing device102), provided that software of user interface114can access and is accessible by at least AR marker program112.

In some embodiments, user interface114is a graphical user interface used to display augmented reality visuals to a user. For example, in some embodiments, one or both of the input buffer and output buffer are displayed on user interface114In other embodiments, user interface includes one or more interface devices used to enable user interaction with client device110. In various embodiments, user interface114includes one or more input/output devices, human interface devices, pointing devices, sensors, or any combination thereof. For example, in some embodiments, user interface114includes a sensor that tracks a user's eye movement. The sensor can allow the user to interact with client device110through eye movement. In another embodiment, user interface114includes a microphone to enable the user to interact with client device110through voice commands.

Client device110includes camera116. In some embodiments, camera116resides on another device or component, so long as camera116can access and is accessible by AR marker program112. Camera116captures the user's real world view. Images captured by camera116are converted to a digital representation that is the input buffer.

FIG. 2is a flowchart depicting operations for positioning markers in augmented reality overlays, on a computing device within the computing environment ofFIG. 1, in accordance with an embodiment of the present invention. For example,FIG. 2is a flowchart depicting operations200of AR marker program112, on computing device102within computing environment100.

In step202, AR marker program112receives an image from an AR input buffer. The AR input buffer is the digital representation of the information taken from the device's camera. In some embodiments, the input buffer contains an element of an image associated with one or more markers. AR marker program112identifies an element in the input buffer image associated with a marker. A user is able to interact with the marker in response to a user action. For example, where client device110includes an eye tracking sensor, the user interaction may be to look at the marker. In another example, the user interaction may be a voice command. In yet another example, the user interaction may be a user selection with a computer pointing device (e.g., a computer mouse, a touchscreen, etc.).

In step204, AR marker program112receives an image from an AR output buffer. The AR output buffer is a digital representation of computer generated content that the user sees through the device. For example, an AR overlay is a component of the AR output buffer. In some embodiments, AR marker program112receives the AR output buffer image in response to a user interaction with the AR input buffer image. In some embodiments, the AR output buffer image has one or more embedded markers.

In step206, AR marker program112scans the AR output buffer image for markers. AR marker program112identifies the number of markers embedded in the AR output buffer image and the locations of each marker. Each marker embedded in the AR output buffer image is associated with a marker overlay.

In decision208, AR marker program112determines whether to display a marker overlay. If AR marker program112determines that a marker overlay should be displayed (decision208, YES branch), then AR marker program112displays the marker overlay. In some embodiments, digital markers are placed in the output buffer. In one embodiment, where the output buffer is a menu, a marker is placed next to each of menu options. In another embodiment, one or more markers are placed in the output buffer, where each marker corresponds to an element of the output buffer. In one embodiments, AR marker program112determines whether to display a marker overlay based on a user input. In some embodiments, the user input indicates a user selection of a marker in the AR output buffer image. In some embodiments, the user indicates a selection via user interface114. For example, where user interface114includes a sensor for tracking eye movement, a user selection can be indicated by the user maintaining eye contact with a marker for a predetermined period of time. In another example, where user interface114includes a microphone, a user selection can be indicated by voice command. In still another example, a user selection can be indicated by a point device (e.g., a touch screen, a computer mouse, etc.). If AR marker program112determines that a marker overlay should not be displayed (decision208, NO branch), then a marker overlay is not generated in output buffer. Where a marker overlay is not displayed, operations200end.

In step210, AR marker program112displays a marker overlay. In response to a user input indicating a selection, a marker overlay is displayed on a user interface. In some embodiments, the marker overlay includes one or more markers. In these embodiments, the user input can be used to select a marker to generate another marker overlay. In some embodiments, the marker overlay includes a marker that allows the user to close the marker overlay or return to the previous overlay.

FIG. 3is an example augmented reality overlay presenting nested markers, in accordance with an embodiment of the present invention. For example,FIG. 3is an example AR overlay depicting operations200of AR marker program112, on computing device102within computing environment100.FIG. 3includes AR input buffer image310, AR overlay302and AR marker overlay308.

AR input buffer image310is a sample of the image received by the AR input buffer. AR input buffer image310is the digital representation of the user's real world view. AR input buffer image310is retrieved using camera116on client device110. AR input buffer image310includes marker304aand indicator306a. In some embodiments, marker304ais associated with an object in AR input buffer image310. Indicator306aindicates a location of a first user interaction with AR input buffer image310. In some embodiments, the location of indicator306ais representative of a user's interaction (e.g., user focus), based on data from an eye tracking sensor at the time of the first user interaction. In other embodiments, the location of indicator306ais based on a user interaction with a pointing device.

AR overlay302is a sample overlay produced by the AR output buffer. AR overlay302is generated by AR marker program112in response to a user interaction with a marker in AR input buffer image310(e.g., marker304a). AR overlay302includes one or more markers304b. Each marker304bis associated with an AR marker overlay. A user interaction with marker304bcauses an AR marker overlay to be generated. Indicator306bindicates a location of a second user interaction with AR overlay302. In some embodiments, the location of indicator306bis representative of a user's focus, based on data from an eye tracking sensor at the time of the second user interaction. In other embodiments, the location of indicator306bis based on a user interaction with a pointing device.

AR marker overlay308is a sample marker overlay produced by the AR output buffer. AR marker overlay308is generated by AR marker program112in response to a user interaction with a marker in AR overlay302(e.g., marker304b“Option 2”). In some embodiments, AR marker overlay308includes one or more markers304c. Each marker304cis associated with an AR marker overlay. In some embodiments, marker304cis an option to close AR marker overlay308and return to AR overlay302.

FIG. 4is a block diagram of components of a computing device, generally designated400, in accordance with an embodiment of the present invention. In one embodiment, computing device400is representative of client device110. For example,FIG. 4is a block diagram of computing device102within computing environment100executing operations of AR marker program112.

It should be appreciated thatFIG. 4provides only an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.

Computing device400includes communications fabric408, which provides communications between computer processor(s)402, memory404, cache406, persistent storage410, communications unit414, and input/output (I/O) interface(s)412. Communications fabric408can be implemented with any architecture designed for passing data and/or control information between processors (such as microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within a system. For example, communications fabric408can be implemented with one or more buses.

Memory404and persistent storage410are computer-readable storage media. In this embodiment, memory404includes random access memory (RAM). In general, memory404can include any suitable volatile or non-volatile computer readable storage media. Cache406is a fast memory that enhances the performance of processors402by holding recently accessed data, and data near recently accessed data, from memory404.

Communications unit414, in these examples, provides for communications with other data processing systems or devices, including resources of network120. In these examples, communications unit414includes one or more network interface cards. Communications unit414may provide communications through the use of either or both physical and wireless communications links. Program instructions and data used to practice embodiments of the present invention may be downloaded to persistent storage410through communications unit414.

I/O interface(s)412allows for input and output of data with other devices that may be connected to computing device400. For example, I/O interface412may provide a connection to external devices416such as a keyboard, keypad, a touch screen, and/or some other suitable input device. External devices416can also include portable computer-readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the present invention (e.g., software and data) can be stored on such portable computer-readable storage media and can be loaded onto persistent storage410via I/O interface(s)412. I/O interface(s)412also connect to a display418.