Augmented reality image offset and overlay

A method is provided to implement augmented reality using markers. An image is captured of an environment. An image of a marker is detected in the image of the environment. A virtual image is displayed overlaid on the image of the environment at an offset from the marker, wherein the virtual image is based on the marker.

CROSS-REFERENCE TO RELATED APPLICATIONS

Not applicable.

REFERENCE TO A MICROFICHE APPENDIX

Not applicable.

BACKGROUND

Augmented reality (AR) is a field of computer applications which deals with the combination of real world images and computer generated data. Many augmented reality applications are concerned with the use of live video imagery which is digitally processed and augmented by the addition of computer generated graphics. For instance, an augmented reality user may wear translucent goggles through which the user may see the real world as well as computer-generated images projected on top of that real world.

For example, virtual chess pieces may be overlaid on top of an empty chess board that is depicted as a camera image. The computer that overlays the virtual chess pieces must know where the camera is relative to the chess board in order to know how to augment the image correctly. One way to achieve this is through the use of markers, which may be a two dimensional bar code such as the circular ShotCode, or any asymmetrical pattern. Markers are put in the scene so that when these markers are recognized in the camera image, the computer rendering the virtual chess pieces knows the position and angle to render virtual chess pieces and draw them over the camera image so that the virtual chess pieces appear natural to the viewer looking at the camera image.

SUMMARY

In some embodiments, a method is provided to implement augmented reality using markers. An image is captured of an environment. An image of a marker is detected in the image of the environment. A virtual image is displayed overlaid on the image of the environment at an offset from the marker, wherein the virtual image is based on the marker.

In other embodiments, a system is provided to implement augmented reality using markers. The system includes at least one processor, a camera associated with a mobile device, a plurality of instructions in a memory, and a user interface associated with the mobile device. The camera captures an image of an environment. The plurality of instructions, when executed by the at least one processor, detect an image of a marker in the image of the environment. The user interface displays a virtual image overlaid on the image of the environment, wherein a plane of the virtual image is displayed parallel to a plane of a display associated with the user interface.

In still other embodiments, a method is provided to implement augmented reality using markers. An image is captured of an environment. An image of a marker is detected in the image of the environment. A virtual image is displayed overlaid on the image of the environment, wherein the virtual image is based on the marker, and wherein the virtual image is displayed at least partially over the marker and over another portion of the image of the environment.

DETAILED DESCRIPTION

In embodiments of the present disclosure, methods and systems are provided for markers to implement augmented reality. A mobile device display may overlay a virtual image on an image of a marker in an environment that is visually cluttered, such as a virtual image of a price discount overlaid on a marker on a box of children's cereal that is located on a shelf that is crowded with many colorful boxes of children's cereal. Alternatively, the mobile device may overlay the virtual image at an offset from the image of the marker, such as overlaying the virtual image of the price discount offset above the top of the corresponding children's cereal box, where the mobile device user may easily see the virtual image of the price discount uncluttered by the background of its environment. If a mobile device user is walking down an aisle of items that have markers, the mobile device user may not be able to read the virtual images that are overlaid on the images of the markers due to the sharp viewing angle between the perspective of the mobile device and the markers. By overlaying the virtual images at an offset from the images of the markers, the mobile device may display the virtual images in a plane that is parallel to the plane of the mobile device display, which makes viewing and reading the virtual image much easier for the mobile device user. The mobile device may also overlay virtual images of portions of items that are not visible at an offset from the images of the markers on the items. For example, if only a portion of a boiler is visible above a factory floor, the mobile device may overlay an image of the portion of the boiler that is below the factory floor at an offset from a marker on the boiler, where the offset results in the virtual image of the portion of the boiler that is below the factory floor lining up with the visible portion of the boiler that is above the factory floor. The mobile device displaying both the visible portion of the boiler and a virtual image of the portion of the boiler that is not visible may enable a factory worker to completely trace boiler pipes that are only partially visible to solve a problem with the boiler. The virtual image may be overlaid partially over the marker and another portion of the image of the environment. For example, the mobile device may overlay a virtual image of a transparent green cover on an image of a refrigerator and its marker to indicate that a database of information associated with the refrigerator's marker indicates that the refrigerator matches user selected refrigerator requirements.

FIG. 1shows a wireless communications system100that illustrates an embodiment of the present disclosure's systems and methods. The wireless communication system100includes an illustrative mobile device102. Though illustrated as a mobile phone, the mobile device102may take various forms including a personal digital assistant (PDA), a mobile handset, a wireless mobile device, a pager, a mobile computer, a portable computer, a tablet computer, a laptop computer, a digital camera, a mobile phone or mobile telecommunication device, a handheld gaming device, a digital calculator, a digital music player, and a headset with goggles, lenses, glasses, or similar viewing devices. Many suitable mobile devices combine some or all of these functions.

The mobile device102includes a display104and a touch-sensitive surface or keys106with which a user can interact. The mobile device102may present options for the user to select, controls for the user to actuate, and/or cursors or other indicators for the user to direct. The mobile device102may further accept data entry from the user, including numbers to dial or various parameter values for configuring the operation of the mobile device102. The mobile device102may further execute one or more software or firmware applications in response to user commands. These applications may configure the mobile device102to perform various customized functions in response to user interaction, such as selecting from virtual image options. In some embodiments, the mobile device102may take various forms with a screen size reduced to a maximum of four inches by six inches, including a mobile telecommunication device, a mobile handset, a personal digital assistant, a handheld gaming device, a wireless mobile device, a pager, a digital camera, a digital music player, a digital calculator, a mobile phone, and a headset with at least one of goggles, lenses, and glasses.

The mobile device102communicates through a base transceiver station108and a wired or wireless network110to access information on various servers, such as a communication server112and a content server114. The content server114may provide content that may be shown on the display104, such as virtual images. The content server114may access a database116of virtual images that may be shown on the display104. The database116may reference markers and virtual images, wherein each of the virtual images is associated with at least one of the markers. In some embodiments, the function of the content server114and the database116is incorporated in the mobile device102instead of in the content server114and the database116that communicate with the mobile device102. While one base transceiver station108is shown inFIG. 1, other base transceiver stations108could be present.

The mobile device102includes a camera118that can capture images of an environment120, whereby an image122of the environment120may be viewed through the display104. InFIG. 1, the camera118is depicted on the side of the mobile device102, but the camera118may be located anywhere on the mobile device102, such as on the opposite side of the mobile device102from the display104. When the environment120includes an item124associated with a marker126, the image122of the environment120may include an image128of the item124and an image130of the marker126. The marker126may be any of an asymmetrical pattern, a moving marker, a two dimensional bar code, a ShotCode, a SEMACODE, a color light pattern, a primary marker assigned to a marker hierarchy, a secondary marker assigned to the marker hierarchy, and/or a dependent marker assigned to the marker hierarchy. The color light pattern may be a combination of different colors of light that are arranged in a manner that uniquely identifies a marker. A marker hierarchy is a designated sequence in which virtual images are overlaid on markers, such as the hierarchy for markers described below in reference to box608(FIG. 6). If the marker126is displayed by an updateable digital marker display, the updateable digital marker display may communicate with the content server114and the database116via the network110. Either the mobile device102, and/or the content server114in conjunction with the database116, may overlay a virtual image132on the image122of the environment120at an offset134from the image130of the marker126based on the image130of the marker126. Overlaying the virtual image132at the offset134from the image130of the marker126may result in displaying the virtual image132at least partially over the image130of the marker126and over the image122of the environment120while leaving at least part of the image130of the marker126not overlaid.

FIG. 2shows a view200of a graphic user interface for markers to implement augmented reality according to some embodiments of the present disclosure. The view200includes the display104, the environment120, an image122of the environment120, an image of a shelf202, an image of a first item204, an image of a second item206, an image of a third item208, an image of a first marker210, an image of a second marker212, an image of a third marker214, a first virtual image216, a second virtual image218, a third virtual image220, a first offset222, and a second offset224. Either the mobile device102, and/or the content server114in conjunction with the database116, may overlay the virtual images216-220on the image122of the environment120based on the location of the mobile device102, more specifically the camera118for the mobile device102, in relation to the location of the images of the markers210-214in the environment120. Each of the virtual images216-220may be an image, navigational information, an advertisement, information about the items206-208in the image122of the environment120, an outline of an item associated with a corresponding marker, or colors that indicate a level of matching.

For example, the mobile device102is located such that the camera118captures the image of the first item204and the image of the first marker210, which may be a ShotCode. The mobile device102reads data-bits from the ShotCode, which is a dartboard-like circle, with a bullseye in the center and data-circles surrounding it, by measuring the angle and distance from the bullseye for each data-circle. The mobile device102communicates the data from the image of the first marker210to the content server114via the network110, and the content server114identifies a virtual image in the database116that corresponds to the image of the first marker210. The content server114communicates the corresponding virtual image to the mobile device102via the network110, and the mobile device102overlays the first virtual image216on the image122of the environment120over the image of the first marker210. In this example, if the first item204is a box of candy and the image of the first marker210is a trademarked pattern that is specific to the particular box of candy, the mobile device102overlays the first virtual image216, which may be a price discount image for the candy, over the image of the first marker210. The image of the first marker210may be a SEMACODE, which is a machine-readable dot matrix symbol that encodes an internet uniform resource locator (URL). An image of the SEMACODE may be captured by the camera118and decoded to obtain a web site address. The mobile device102may access this address via the mobile device's web browser to obtain the virtual image that corresponds to the image of the first marker210.

In another example, when the mobile device102is located such that the camera118captures the image of the second item206and the image of the second marker212, the mobile device102overlays the second virtual image218on the image122of the environment120at the first offset222from the image of the second marker212. In this example, the second item206is a box of children's cereal and the image of the second marker212is a pattern specifically designed for the second virtual image218that corresponds to the box of children's cereal. The mobile device102overlays the second virtual image218, which may be an advertisement for the children's cereal, at the first offset222from the image of the second marker212.

In yet another example, when the mobile device102is located such that the camera118captures the image of the third item208and the image of the third marker214, the mobile device102overlays the third virtual image220on the image122of the environment120at the second offset224from the third marker214. In this example, the image of the third marker214is not detected on the image of the third item208, but detected on the shelf202below the image of the third item208, such as on a promotional display area. If the third item208is a box of health-oriented cereal and the image of the third marker214displays a pattern that is specific to the particular box of health-oriented cereal, the mobile device102overlays the third virtual image220, which may be information about the health-oriented cereal, above the shelf202and above the image of the third item208at a the second offset224from the image of the third marker214.

FIG. 3shows a view300of another graphic user interface for markers to implement augmented reality according to some embodiments of the present disclosure. The view300includes the display104, the environment120, the image122of the environment120, an image of a first item302, an image of a second item304, an image of a first marker306, an image of a second marker308, a first virtual image310, a second virtual image312, a first offset314, and a second offset316. When the mobile device102is located such that the camera118captures an image of the first marker306, the mobile device102may overlay the first virtual image310on the image122of the environment120at the first offset314from the image of the first marker306. The first item302, for example a large screen television, and the first marker306are situated in a warehouse aisle at an angle from the camera118that may make correctly capturing the images of the first item302and the first marker306challenging. To compensate for the angle of the items302-304in the aisle from the camera118, the mobile device102overlays the first virtual image310, specification information for the large screen television, in a plane that is parallel to the plane of the display104for the mobile device102, which enables a user of the mobile device102to identify the first virtual image310more easily. If the first virtual image310was overlaid in a plane that is parallel to the plane for the first item302and the first marker306, the user of the mobile device102may be challenged in identifying the first virtual image310, which may make reading the specification information for the large screen television somewhat difficult. The mobile device102may overlay the second virtual image312on the image of the second marker308or at the second offset316from the image of the second marker308.

FIG. 4shows a view400of another graphic user interface for markers to implement augmented reality according to some embodiments of the present disclosure. The view400includes the display104, the environment120, the image122of the environment120, an image of a first item402, an image of a second item404, an image of a third item406, an image of a first marker408, an image of a second marker410, an image of a third marker412, a first virtual image414, a second virtual image416, and an offset418.

For example, when the mobile device102is located such that the camera118captures the image of the second marker410on the second item404, which is a refrigerator, the mobile device102overlays the first virtual image414, which may be a green outline or a transparent green cover, on the image of the second marker410and the image of the second item404. The first virtual image414may be a green outline or a transparent green cover that is overlaid on the image of the first item402to indicate that the first item402matches specifications for the items402-406entered by the user of the mobile device102via the keys106. In addition to the first virtual image414, the mobile device102may overlay the second virtual image416, which may be detailed product information, at the offset418from the image of the second marker410.

FIG. 5shows a view500of an additional graphic user interface for markers to implement augmented reality according to some embodiments of the present disclosure. The view500includes the display104, the environment120, the image122of the environment120, an image of a first item502, an image of a second item504, an image of a first marker506, an image of a second marker508, a floor510, an above-floor portion512of the environment120, a below-floor portion514of the environment120, a virtual image516, and an offset518.

For example, when the mobile device102is located such that the camera118captures an image of the first marker506on the image of the first item502, which is a boiler in a factory, the mobile device102overlays the virtual image516, which may be current operating information for the boiler, on the image122of the environment120at the offset518from the image of the first marker506. For this example, the virtual image516may specify the operational status of the boiler, such information that identifies whether the boiler's temperature is excessively hot, whether the boiler needs maintenance, and whether the boiler needs a re-supply of a consumable material, such as the boiler's fuel. The virtual image516may also include a history associated with the first item502, such as the maintenance history for the boiler. Additionally, the virtual image516may include instructions, such as procedures for providing boiler maintenance and a troubleshooting chart that lists actions to take in response to the most common problems. The mobile device102may overlay the virtual image516on the image of the first marker506or overlay the virtual image516at the offset518from the image of the first marker506.

It will be readily appreciated that the disclosed concept could be extended to other devices besides boilers, for example to electric motors, to hydraulic pumps, to injection plastic mold machines, to computer numerical control (CNC) machines, to assembly line robotic devices, to conveyor systems, to weaving machinery, to food preparation and processing machinery, to beverage processing machinery, to printing presses, to motor vehicles, to chemical reactor vessels, to refinery equipment, to steam turbine generators, to hydroelectric generators and pumps, and to many other devices, all of which are contemplated by the present disclosure.

In response to detecting the images of the markers506and508, the mobile device102may overlay a virtual image of the below-flow portion514of the environment120on the image122of the environment120based on the location of the mobile device102in relation to the location of the image of the markers506-508in the environment120. The virtual image of the below-flow portion514may be a schematic drawing that shows the plumbing beneath the floor of the factory, or an electrical schematic of wires and cables beneath the factory floor. In other examples, virtual images may show an electrical schematic of the wires and cables inside equipment and/or inside walls. Overlaying the virtual image of the below-flow portion514of the environment120may be a user-selected response to detecting the images of the markers506and508.

FIG. 6is a flowchart of a method for markers to implement augmented reality according to some embodiments of the present disclosure. The mobile device102can use the method to display a virtual image at an offset from an image of a marker.

In box602, an image is captured of an environment. For example, the camera118captures the image122of the environment120, which includes the image of the shelf202, the images of the items204-208and the images of the markers210-214.

In box604, an image of a marker is detected in the image of the environment. For example, the mobile device102detects the image of the second marker212on the image of the second item206.

In box606, a virtual image is displayed overlaid on the image of the environment at an offset from the image of the marker, wherein the virtual image is based on the marker. For example, the display104displays the second virtual image218of an animated cartoon character that is associated with the second item206, which is children's cereal, at the first offset222from the second marker212. An amount of the offset may be based on a marker. For example, the second marker212indicates for the mobile device102to display the second virtual image218at the first offset220above the image of the second marker212, where the first offset222is equal to 120% of the height of the second item206in the image122of the environment120. The second virtual image218may be displayed at least partially over the second marker212and over another portion of the image122of the environment120. For example, the display104displays the second virtual image218of an animated cartoon character at least partially over the second marker212, where at least part, or possibly all, of the second marker212is not overlaid. Displaying the second virtual image218at the first offset222from the second marker212results in the animated cartoon character being displayed sufficiently high above the children's cereal box for the user of the mobile device102to clearly see the second virtual image218.

In box608, another virtual image is displayed overlaid on the image of the environment in response to a user selection, wherein the other virtual image is based on the marker. For example, the display104displays the virtual image of the below-floor portion514of the environment120in response to a user selection, wherein the virtual image of the below-floor portion of the boilers502-504is based on the combination of the images of the markers506-508. The image of the marker506for the first boiler502may serve as a primary marker for the below-floor portion of the boilers502-504, while the image of the marker508for the second boiler504may serve as a secondary marker for the below-floor portion of the boilers502-504. When the user views the image122of the environment120in the factory, the mobile device102detects the primary and the secondary markers506-508, but does not display virtual images associated with the combination of the primary and the secondary markers506-508when the mobile device102is configured to display only the virtual images based on the primary marker506and the secondary marker508individually. However, the mobile device102offers options via the display104for the user to select to also display virtual images corresponding to combinations of markers, such as the markers506-508.

The mobile device102may also detect secondary or dependent markers in the image122of the environment120that may be selected to display a sub-portion of a virtual image associated with a secondary marker. For example, when the mobile device102displays a virtual image of the below-floor portion of the boilers502-504, the mobile device102may also detect secondary markers and offer to let the user select between displaying only plumbing pipes that input water into the boilers502-504or only plumbing pipes that exhaust steam from the boilers502-504, with each sub-section independently displayed based on a different user selection.

Displaying the virtual image may be based on a progressive algorithm for the marker, wherein a first virtual image is based on a first marker, a second virtual image is based on a second marker, and a third virtual image is based on a combination of the first marker and the second marker. For example, the first virtual image516is based on the first marker506, a second virtual image is based on the second marker508, and a third virtual image of the below-floor portion of the boilers502-504is based on a combination of the first marker506and the second marker508.

In box610, another marker is detected that is indirectly associated with the marker based on at least one of compatibility with an item directly associated with the marker and customers purchasing pattern. For example, the mobile device102detects the image of a marker for a children's breakfast drink that is indirectly associated with the second marker212for the second item206, which is the children's cereal. Alternatively, the other marker may be for a children's snack indirectly associated with the second marker212for the second item206based on customers purchasing patterns, which indicate that an unusually high percentage of customers who purchased the children's breakfast cereal also purchased the children's snack.

In box612, another virtual image associated with the other marker is displayed. For example, the mobile device102displays another virtual image, which is an advertisement that encourages customers to purchase either the children's breakfast drink or the children's snack in addition to purchasing the children's cereal.

FIG. 7is another flowchart of a method for markers to implement augmented reality according to some embodiments of the present disclosure. The mobile device can use the method to display virtual images clearly even when the corresponding markers are at a sharp angle from the camera.

In box702, an image of an environment is captured. For example, the camera118captures the image122of the environment120that includes the markers306-308on the images of the large screen televisions302-304while the user of the mobile device102walks down an aisle in a warehouse.

In box704, an image of a marker is detected in the image of the environment. For example, the mobile device102detects the markers306-308even though the markers306-308are at sharp angles from the camera118on the mobile device102.

In box706, a virtual image is displayed overlaid on the image of the environment, wherein a plane of the virtual image is displayed parallel to a plane of a display associated with the user interface. For example, the display104displays the virtual images310-312, which include detailed product information about the large screen televisions302-304, in planes that are parallel to the plane of the display104for the mobile device102. While the user of the mobile device102may have a difficult time reading product information on the large screen televisions302-304due to the sharp angles that the large screen televisions302-304are situated from the user, the virtual images310-312are overlaid in planes that are easy for the user to read.

FIG. 8is yet another flowchart of a method for markers to implement augmented reality according to some embodiments of the present disclosure. The mobile device102can use the method to display virtual images at least partially over an image of a marker while leaving at least part of the image of the marker not overlaid by the virtual images and over another portion of the image of the environment.

In box802, an image is captured of an environment. For example, the camera118captures the image122of the environment120that includes the images of the items402-406, which are refrigerators, and the images of the markers408-412.

In box804, a marker is detected in the image of the environment. For example, the mobile device102detects the image of the second marker410, which is on the second refrigerator404.

In box806, information about an item that is associated with the marker is optionally accessed. For example, the mobile device102accesses detailed product information, including height, width, and depth, about the second refrigerator404, based on the image of the second marker410.

In box808, a size of the item is optionally determined based on the information. For example, the mobile device102determines a height and a width of the second refrigerator404based on the information accessed.

In box810, the image of the environment is optionally evaluated based on a size of the marker in the image of the environment. For example, the mobile device102determines that the image122of environment120is one-thirty-sixth the size of the environment120because the height of the image of the second marker410is one-twelfth of an inch while the database116indicates that the height of the second marker410is three inches.

In box812, a relative position for the virtual image is optionally determined based on the size of the item in relation to the image of the environment. For example, the mobile device102determines that the first virtual image414will be one-thirty-sixth the size of the second refrigerator404, which is six feet tall. Because one-thirty-sixth of six feet is two inches, the mobile device102determines that the first virtual image414will be two inches high when overlaid on the image122of the environment120via the display104.

In box814, a distance from a reference point in the image of the environment to the image of the marker is optionally measured to determine a relative position for the virtual image in the image of the environment. For example, the mobile device102measures from the top of the image of the second refrigerator404to the image of the second marker410to determine that one quarter of the image of the second refrigerator404is visible above the image of the second marker410. This measurement will result in one quarter of the first virtual image414being overlaid above the image of the second marker410.

In box816, whether the virtual image matches the size of the item in the image of the environment is optionally determined. For example, prior to displaying the first virtual image414on the image of the second refrigerator404via the display104, the mobile device102determines whether the first virtual image414matches the image of the second refrigerator404. If the database116provides the incorrect height for the second refrigerator404, the size of the first virtual image414may be larger or smaller than the image of the second refrigerator404. In other examples, determining whether a virtual image matches a size of an item may be part of a validation process for the item. For this example, the display104may project a virtual image of a piece of equipment on the image of the piece of equipment to assist the mobile device user in validating that the correct piece of equipment is about the be deployed for use.

In box818, an error message is optionally output in response to a determination that the virtual image does not match the size of the item in the image of the environment. For example, the mobile device102outputs an error message in response to a determination that the two inch high first virtual image414is ten percent larger than the image of the second refrigerator404. A mismatch for the first virtual image414may indicate that the database116provided some incorrect information about the second refrigerator404, which may call into question the accuracy of the rest of the information provided by the first virtual image414that is overlaid on the image of the second refrigerator404.

In box820, the displayed virtual image is optionally analyzed. For example, the mobile device102analyzes the first virtual image414to insure that it is properly overlaid on the image of the second refrigerator404. If the user of the mobile device102is not holding the mobile device102steady, the display104location calculated for overlaying the first virtual image414may not match the image of the second refrigerator404.

In box822, the display of the virtual image overlaid on the image of the environment is optionally adjusted. For example, the mobile device102uses a digital compass and an accelerometer to adjust the display of the first virtual image414on the image of the second refrigerator404. Adjusting the display of virtual images may be helpful when the mobile device102is moved in relation to its initial orientation. For example, the display104of the mobile device102may overlay the detailed product information as the second virtual image416that appears to be fifteen inches high and depicted six inches above the six foot high second refrigerator404. When the mobile device102is ten feet away from the second refrigerator404, the display104may offer a perspective that enables the mobile device user to read the second virtual image416that includes the detailed product information.

However, when the mobile device user moves closer to the second refrigerator404, the display104may offer a perspective that depicts only the second refrigerator404and a small amount of room above the second refrigerator404. In this situation, the display104does not have sufficient room in its current perspective to display both the second refrigerator404and the detailed product information in their original proportions. In response, the mobile device102may adjust the display of the detailed product information to overlay the second refrigerator404so that the mobile device user may still read all of the detailed product information. Alternatively, the mobile device102may reduce the relative size of the detailed product information that is depicted above the second refrigerator404to fit the available free space above the second refrigerator404, but not reduced so much so that the mobile device user may not read all of the detailed product information.

In box824, a virtual image is displayed overlaid on the image of the environment, wherein the virtual image is based on the marker, and wherein the virtual image is displayed at least partially over an image of the marker and over another portion of the image of the environment. For example, the mobile device102displays the first virtual image414, which is a transparent green cover, overlaid on the image of the second marker410and the image of the second refrigerator404.

Virtual images overlaid on the image122of the environment120may include colors that indicate a level of matching. For example, if a user inputs desired features, including a desired price, for a refrigerator via the keys106of the mobile device102, the communication server112communicates with the content server114and the database116to determine which refrigerator models most closely match the desired features. The content server114associates virtual images of different overlay colors with the image of each of the refrigerators402-406depending upon the level of matching. For example, the mobile device102overlays the color red over the image of the first refrigerator402that matches few of the desired features, overlays the color yellow over the image of a third refrigerator406that matches some of the desired features, and overlays the color green over the image of the second refrigerator404that matches most of the desired features. The user may briefly scan the image122of the environment120that includes the images of the three refrigerators402-406and quickly identify the second refrigerator404that matches most of the desired features and price by simply observing the colors overlaid as virtual images on the images of the refrigerators402-406.

While most examples depict the mobile device102as a mobile phone with the display104displaying virtual images, one of skill in the art would recognize that the mobile device102may also use a headset with goggles, lenses, glasses, or similar viewing devices as discussed, or other coupled display devices. In some instances, the mobile device102may also use a projector to project the virtual images onto the environment120, such as a projector in the mobile device102or a projector coupled with the mobile device102. This projection would also “display” the virtual images as described and claimed herein.

FIG. 9is a block diagram of the mobile device102. While a variety of known components of mobile devices102are depicted, in an embodiment a subset of the listed components and/or additional components not listed may be included in the mobile device102. The mobile device102includes a digital signal processor (DSP)902and a memory904. As shown, the mobile device102may further include an antenna and front end unit906, a radio frequency (RF) transceiver908, an analog baseband processing unit910, a microphone912, an earpiece speaker914, a headset port916, an input/output interface918, a removable memory card920, a universal serial bus (USB) port922, an infrared port924, a vibrator926, a keypad928, a touch screen liquid crystal display (LCD) with a touch sensitive surface930, a touch screen/LCD controller932, a charge-coupled device (CCD) camera934, a camera controller936, and a global positioning system (GPS) sensor938. In an embodiment, the mobile device102may include another kind of display that does not provide a touch sensitive screen. In an embodiment, the DSP902may communicate directly with the memory904without passing through the input/output interface918.

The DSP902or some other form of controller or central processing unit operates to control the various components of the mobile device102in accordance with embedded software or firmware stored in memory904or stored in memory contained within the DSP902itself. In addition to the embedded software or firmware, the DSP902may execute other applications stored in the memory904or made available via information carrier media such as portable data storage media like the removable memory card920or via wired or wireless network communications. The application software may comprise a compiled set of machine-readable instructions that configure the DSP902to provide the desired functionality, or the application software may be high-level software instructions to be processed by an interpreter or compiler to indirectly configure the DSP902.

The antenna and front end unit906may be provided to convert between wireless signals and electrical signals, enabling the mobile device102to send and receive information from a cellular network or some other available wireless communications network or from a peer mobile device102. In an embodiment, the antenna and front end unit906may include multiple antennas to support beam forming and/or multiple input multiple output (MIMO) operations. As is known to those skilled in the art, MIMO operations may provide spatial diversity which can be used to overcome difficult channel conditions and/or increase channel throughput. The antenna and front end unit906may include antenna tuning and/or impedance matching components, RF power amplifiers, and/or low noise amplifiers.

The RF transceiver908provides frequency shifting, converting received RF signals to baseband and converting baseband transmit signals to RF. In some descriptions a radio transceiver or RF transceiver may be understood to include other signal processing functionality such as modulation/demodulation, coding/decoding, interleaving/deinterleaving, spreading/despreading, inverse fast fourier transforming (IFFT)/fast fourier transforming (FFT), cyclic prefix appending/removal, and other signal processing functions. For the purposes of clarity, the description here separates the description of this signal processing from the RF and/or radio stage and conceptually allocates that signal processing to the analog baseband processing unit910and/or the DSP902or other central processing unit. In some embodiments, the RF Transceiver908, portions of the antenna and front end906, and the analog baseband processing unit910may be combined in one or more processing units and/or application specific integrated circuits (ASICs).

The analog baseband processing unit910may provide various analog processing of inputs and outputs, for example analog processing of inputs from the microphone912and the headset port916and outputs to the earpiece speaker914and the headset port916. To that end, the analog baseband processing unit910may have ports for connecting to the built-in microphone912and the earpiece speaker914that enable the mobile device102to be used as a cell phone. The analog baseband processing unit910may further include a port for connecting to a headset or other hands-free microphone and speaker configuration. The analog baseband processing unit910may provide digital-to-analog conversion in one signal direction and analog-to-digital conversion in the opposing signal direction. In some embodiments, at least some of the functionality of the analog baseband processing unit910may be provided by digital processing components, for example by the DSP902or by other central processing units.

The DSP902may communicate with a wireless network via the analog baseband processing unit910. In some embodiments, the communication may provide Internet connectivity, enabling a user to gain access to content on the Internet and to send and receive e-mail or text messages. The input/output interface918interconnects the DSP902and various memories and interfaces. The memory904and the removable memory card920may provide software and data to configure the operation of the DSP902. Among the interfaces may be the USB port922and the infrared port924. The USB port922may enable the mobile device102to function as a peripheral device to exchange information with a personal computer or other computer system. The infrared port924and other optional ports such as a Bluetooth interface or an IEEE 802.11 compliant wireless interface may enable the mobile device102to communicate wirelessly with other nearby mobile devices and/or wireless base stations.

The input/output interface918may further connect the DSP902to the vibrator926that, when triggered, causes the mobile device102to vibrate. The vibrator926may serve as a mechanism for silently alerting the user to any of various events such as an incoming call, a new text message, and an appointment reminder.

The keypad928couples to the DSP902via the I/O interface918to provide one mechanism for the user to make selections, enter information, and otherwise provide input to the mobile device102. Another input mechanism may be the touch screen LCD930, which may also display text and/or graphics to the user. The touch screen LCD controller932couples the DSP902to the touch screen LCD930.

The CCD camera934enables the mobile device102to take digital pictures. The DSP902communicates with the CCD camera934via the camera controller936. The GPS sensor938is coupled to the DSP902to decode global positioning system signals, thereby enabling the mobile device102to determine its position. In another embodiment, a camera operating according to a technology other than Charge Coupled Device cameras may be employed. Various other peripherals may also be included to provide additional functions, e.g., radio and television reception.

FIG. 10illustrates a software environment1002that may be implemented by the DSP902. The DSP902executes operating system software1004that provides a platform from which the rest of the software operates. The operating system generally provides processing resource management support. The operating system software1004provides drivers for the mobile device hardware to make standardized interfaces available to application software. The operating system software1004may transfer control between applications running on the mobile device102. Alternatively, an application management services1006component may transfer control between applications running on the handset mobile device102. Also shown inFIG. 10are a web browser1008application, a media player1010, and application JAVA applets1012. The web browser1006application configures the mobile device102to operate as a web browser, allowing the user to enter information into forms and select links to retrieve and view web pages. The media player1010application configures the mobile device102to retrieve and play audio or audiovisual media on its own output components. The JAVA applets1012may configure the mobile device102to provide games, utilities, and other functionality on the mobile device102.

The system described above may be implemented on any general-purpose computer with sufficient processing power, memory resources, and network throughput capability to handle the necessary workload placed upon it.FIG. 11illustrates a typical, general-purpose computer system, suitable for implementing one or more embodiments disclosed herein. The computer system1080includes a processor1082(which may be referred to as a central processor unit or CPU) that is in communication with memory devices including secondary storage1084, read only memory (ROM)1086, random access memory (RAM)1088, input/output (I/O) devices1090, and network connectivity devices1092. The processor may be implemented as one or more CPU chips.

The secondary storage1084is typically comprised of one or more disk drives or tape drives and is used for non-volatile storage of data and as an over-flow data storage device if RAM1088is not large enough to hold all working data. Secondary storage1084may be used to store programs which are loaded into RAM1088when such programs are selected for execution. The ROM1086is used to store instructions and perhaps data which are read during program execution. ROM1086is a non-volatile memory device which typically has a small memory capacity relative to the larger memory capacity of secondary storage. The RAM1088is used to store volatile data and perhaps to store instructions. Access to both ROM1086and RAM1088is typically faster than to secondary storage1084.

I/O devices1090may include printers, video monitors, liquid crystal displays (LCDs), touch screen displays, keyboards, keypads, switches, dials, mice, track balls, voice recognizers, card readers, paper tape readers, or other well-known input devices. The network connectivity devices1092may take the form of modems, modem banks, Ethernet cards, universal serial bus (USB) interface cards, serial interfaces, token ring cards, fiber distributed data interface (FDDI) cards, wireless local area network (WLAN) cards, radio transceiver cards such as code division multiple access (COMA) and/or global system for mobile communications (GSM), and worldwide interoperability for microwave access (WiMAX) radio transceiver cards, cards, and other well-known network devices. These network connectivity devices1092may enable the processor1082to communicate with an Internet or one or more intranets. With such a network connection, it is contemplated that the processor1082might receive information from the network, or might output information to the network in the course of performing the above-described method steps. Such information, which is often represented as a sequence of instructions to be executed using processor1082, may be received from and outputted to the network, for example, in the form of a computer data signal embodied in a carrier wave

Such information, which may include data or instructions to be executed using processor1082for example, may be received from and outputted to the network, for example, in the form of a computer data baseband signal or signal embodied in a carrier wave. The baseband signal or signal embodied in the carrier wave generated by the network connectivity devices1092may propagate in or on the surface of electrical conductors, in coaxial cables, in waveguides, in optical media, for example optical fiber, or in the air or free space. The information contained in the baseband signal or signal embedded in the carrier wave may be ordered according to different sequences, as may be desirable for either processing or generating the information or transmitting or receiving the information. The baseband signal or signal embedded in the carrier wave, or other types of signals currently used or hereafter developed, referred to herein as the transmission medium, may be generated according to several methods well known to one skilled in the art.

The processor1082executes instructions, codes, computer programs, scripts which it accesses from hard disk, floppy disk, optical disk (these various disk based systems may all be considered secondary storage1084), ROM1086, RAM1088, or the network connectivity devices1092. While only one processor1082is shown, multiple processors may be present. Thus, while instructions may be discussed as executed by a processor, the instructions may be executed simultaneously, serially, or otherwise executed by one or multiple processors.

While several embodiments have been provided in the present disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure.

The present examples are to be considered as illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted, or not implemented.