Augmented reality experience for shopping

A retail store including a server having a processor and a memory; a communication network; and a database are disclosed. The server includes an electronic product recognizer that receives a video stream including a plurality of frames from a camera of an electronic mobile device. At least one of the plurality of frames is selected as a captured image. A plurality of products in the captured image is segmented into a plurality of segments. Image recognition is performed using each of the plurality of segments to identify the product in each of the plurality of segments. One or more recognized products identified in the image recognition are output. The one or more recognized products identified in the image recognition are configured to be sent to a user device communicable with the server via the communication network, the server configured to cause one or more stickers to be displayed on the user device.

FIELD

This disclosure relates generally to augmented reality systems. More specifically, this disclosure relates to augmented reality systems for use in a retail environment such as a retail store.

BACKGROUND

Retail shopping continues to evolve as electronic mobile devices develop additional functionality. Consumers can now use a camera on the electronic mobile device to, for example, scan a barcode on a product to learn more information about that product. Augmented reality can be utilized to superimpose information onto a display of the consumer's electronic mobile device to enable virtual interactions with the real-world environment.

Improved ways of identifying the products captured from the consumer's electronic device are desirable.

SUMMARY

This disclosure relates generally to augmented reality systems. More specifically, this disclosure relates to augmented reality systems for use in a retail environment such as a retail store.

In an embodiment, a product can be recognized within at or about two seconds and with confidence of at or about 95% or greater.

In an embodiment, the electronic product recognition can be performed in real-time or substantially real-time.

A system is disclosed. The system includes a retail store including a server having a processor and a memory; a communication network; and a database in electronic communication with the server via the communication network. The server includes an electronic product recognizer that receives a video stream from a camera of an electronic mobile device, the video stream including a plurality of frames. The server selects one of the plurality of frames from the video stream, the one of the plurality of frames from the video stream being selected is a captured image. The server segments a plurality of products in the electronic image into a plurality of segments. The server performs an image recognition using each of the plurality of segments to identify the product in each of the plurality of segments. The server outputs one or more recognized products identified in the image recognition. The one or more recognized products identified in the image recognition are sent to a user device communicable with the server via the communication network to cause one or more stickers to be displayed on the user device.

A computer-implemented method augmenting a user's shopping experience via an electronic mobile device including a mobile application that is loadable onto the electronic mobile device having a display screen and a camera device is disclosed. The mobile application permits the user to electronically capture one or more images using the camera device and have the one or more images displayed on the display of the user device. The method includes receiving, by a server, a video stream from a camera of the electronic mobile device, the video stream including a plurality of frames. The server selects one of the plurality of frames from the video stream, the one of the plurality of frames from the video stream being selected is a captured image. The server segments a plurality of products in the captured image into a plurality of segments. The server performs an image recognition using each of the plurality of segments to identify the product in each of the plurality of segments. The server outputs one or more recognized products identified in the image recognition. A sticker is overlaid on one or more images electronically captured using the camera device and displayed on the display screen.

A system is disclosed. The system includes retail store including a server having a processor and a memory; a communication network; and a database in electronic communication with the server via the communication network. The server includes an electronic product recognizer that receives a segmented image from a camera of an electronic mobile device, the segmented image including a plurality of segments. The server performs an image recognition using each of the plurality of segments to identify the product in each of the plurality of segments. The server outputs one or more recognized products identified in the image recognition. The server outputs one or more recognized products identified in the image recognition. The one or more recognized products identified in the image recognition are sent to a user device communicable with the server via the communication network to cause one or more stickers to be displayed on the user device.

A computer-implemented method augmenting a user's shopping experience via an electronic mobile device including a mobile application that is loadable onto the electronic mobile device having a display screen and a camera device is disclosed. The mobile application permits the user to electronically capture one or more images using the camera device and have the one or more images displayed on the display of the user device. The method includes receiving, by a server, a segmented image from a camera of the electronic mobile device, the video stream including a plurality of segments. The method further includes performing an image recognition using each of the plurality of segments to identify the product in each of the plurality of segments. The server outputs one or more recognized products identified in the image recognition.

Like reference numbers represent like parts throughout.

DETAILED DESCRIPTION

This disclosure relates generally to augmented reality systems. More specifically, this disclosure relates to augmented reality systems for use in a retail environment such as a retail store.

Augmented reality generally refers to technology in which a computer-generated image is superimposed on a display device of a consumer's electronic device (e.g., an electronic mobile device, etc.) to provide a composite image that includes the consumer's view of the real world as well as the superimposed image. In today's retail environment, augmented reality is being utilized to superimpose information about one or more products sold in a retailer's store to provide the consumer with additional information about the products. Success of the augmented reality application is based at least in part on the ability of the retailer to identify what products are in the consumer's view of the real world (through the electronic mobile device).

Image recognition currently may be limited to recognizing a single type of object at a time. For example, current image recognition typically is applied in recognizing a single product. However, retail stores may have thousands of products arranged on a shelf including a number of similar products. For a consumer to utilize current image recognition, the consumer must position the product and wait for the image recognition to complete. This can be slow and inconvenient for the consumer, especially in instances where the consumer would like to recognize more than one product. Improved methods of recognizing a plurality of products accurately and quickly are desirable.

In an embodiment, a retailer can limit a scope of the image recognition based on the products that the retailer sells. Limiting the scope of the image recognition can reduce an overall computational load in completing the image recognition and increase a speed at which the image recognition is completed.

Disclosed herein are augmented reality systems and methods including electronic product recognition methods and systems in which an electronic image or images are received from an electronic mobile device and the electronic images are utilized to identify the product or products included in the electronic image and provide product information relevant to the product or products identified in the electronic image(s).

An electronic product recognizer can receive the electronic images from a camera on an electronic mobile device and generate a list of recognized products. The list of recognized products permits the consumer to interact with the recognized products via the electronic mobile device.

As described in further detail below, in one embodiment, a consumer uses a camera on an electronic mobile device, such as a smartphone, tablet device, wearable device, or the like, to capture an electronic image of a product or a plurality of products using an application that resides on the electronic mobile device. The product or products can then be recognized. A list of products as recognized (e.g., recognized products) can be generated and a sticker or other window may be displayed on the display screen of the electronic mobile device to enable the consumer to, for example, learn more about the products, shop the products and, if desired, purchase one or more of the products. The list of products from the image may generally be referred to as recognized products herein.

The product or products can be any type of product that the retailer sells. The products are products that are available for sale within a retail store. The products may be available for sale online (e.g., on a website of the retail store, etc.) or that are available for sale via another sales channel. In an embodiment, products in the electronic images that are not for sale may be excluded from the image recognition. For example, products for promotional or other purposes that are intended to permit consumers to learn more about a product, but not necessarily to permit the consumer to be able to purchase the product, may be excluded from the image recognition.

As used herein, an electronic mobile device is any device that can be used to electronically capture an image and that can display a list of recognized products on a display screen that is connected to or associated with the electronic mobile device.

Examples of electronic mobile devices include, but are not limited to, mobile phones, smartphones, tablet-style devices, wearable devices, laptop computers, and the like. In one embodiment, the electronic mobile device includes at least a camera and some means to control the camera. The electronic mobile device also includes a display screen and some means to select products displayed on the display screen, and some means to send and receive data communications as described further below.

Cameras on most current electronic mobile devices are capable of capturing photographs and/or video. In an embodiment, the camera may be capable of capturing an image without the user performing an action (e.g., pressing a shutter button) to cause an image to be taken. Any type of image capture technology on an electronic mobile device that is capable of capturing an image of one or more products can be used.

As used herein, to capture or capturing an image refers to the act of obtaining an image of the products using the camera or other image capture technology of the electronic mobile device. Obtaining the image of the products may also be referred to as imaging the products. A captured image is an image of the products that has been captured by the electronic mobile device. The products from which a captured image has been obtained may be referred to as the imaged products. It is to be appreciated that capturing an image of the products includes capturing an image of the entire product as well as capturing an image of a portion of the product.

As used herein, a recognized product, or a list of recognized products includes those products identified from a captured image.

As used herein, optical character recognition (OCR) includes an electronic conversion of an image of text into machine-readable text.

As used herein, histogram of oriented gradients includes a technique in which occurrences of gradient orientation in localized portions of an image are counted.

As used herein, edge detection includes identifying locations within an electronic image in which image brightness abruptly changes.

As used herein, a marker includes an electronic visual cue which triggers display of electronic information.

As used herein, a planogram includes an electronic representation of products within a retail store.

In an embodiment, a product can be recognized within at or about two seconds and with confidence of at or about 95% or greater.

In an embodiment, the product recognition occurs in real-time or substantially real-time.

In an embodiment, the one or more products can be recognized regardless of location. For example, the products can be arranged on a shelf, in a cart, in a consumer's hand, or the like.

FIG. 1is a schematic diagram of a system10for implementing the electronic product recognition systems and methods described in this specification, according to an embodiment. The system10can identify one or more products captured in a real world scene from a user device15. In an embodiment, the system10can be utilized in a retail store environment to enable a consumer to capture an image of one or more products located in the retail store while, for example, the consumer is browsing the retail store.

The system10includes a server25in electronic communication with a plurality of user devices15via a network20. The server25includes an electronic product recognizer60that can receive a captured image from the user devices15. The electronic product recognizer60can identify one or more products included in the captured image received from the user devices15. The electronic product recognizer60can make the product information available to the system10, as described in this Specification available, to the user devices15via the network20. In an embodiment, the electronic product recognizer60can be configured to generate one or more messages to a user that includes the product information or a message with details about the product.

The user devices15can include a user application45. In an embodiment, the user application45can be an augmented reality application45in accordance with the embodiments described in this Specification. It will be appreciated that the user application45can be installed on a memory (e.g., memory520inFIG. 8) and executable using a processor (e.g., processor510) on the user device15. When executed, the user application45provides a user interface via which the user can have her shopping experience enhanced via augmented reality. For example, the user may be able to execute the user application45such that one or more products in a retail store are electronically recognized via the electronic product recognizer60. When the one or more products are electronically recognized by the electronic product recognizer60, one or more interactive features (e.g., price information, product detail information, customer reviews, similar products, etc.) can be displayed on the real-time image displayed on the user device15.

The electronic product recognizer60can include a plurality of modules utilized in the identification of the products within the captured image. In the illustrated embodiment, three modules65-75are shown. It will be appreciated that the number of modules can vary within the scope of this disclosure.

In the embodiment shown inFIG. 1, the electronic product recognizer60includes a marker detector65, an optical character recognizer70, and a logo recognizer75. In an embodiment, the one or more additional modules can include an edge detector, a planogram identifier, a histogram of oriented gradient detector, a global positioning system (GPS) recognizer, an in-store positioning system (IPS), a color recognizer, an artificial neural network, or the like.

The marker detector65can identify one or more known markers in a captured image. A marker generally includes an electronic visual cue which triggers display of electronic information. For example, a marker can include a barcode on a shelf, a tag or other aspect of a display which is included in the captured image, an aspect of product packaging, or the like.

The optical character recognizer70can identify one or more text characters in the captured image. Optical character recognition (OCR) systems are generally well-known and the optical character recognizer70functions according to these generally known principles.

The logo recognizer75can identify a logo or brand identifier present on the one or more products included in the captured image.

In an embodiment, the network20may be representative of the Internet. In an embodiment, the network20can include a local area network (LAN), a wide area network (WAN), a wireless network, a cellular data network, suitable combinations thereof, or the like. Aspects of the network20can be the same as or similar to aspects of the network540as shown and described in accordance withFIG. 8below.

In the illustrated embodiment, a dashed rectangle50is shown. The rectangle50is representative of a particular location for the server25and the database30. For example, in an embodiment, the rectangle50represents that, for example, a single store of a retailer can include the server25and the database30on premises. In an embodiment, the user devices15may be located on premises of a retail store of the retailer and can electronically connect to the server25and database30located on premises as well. In an embodiment, one or more components of the server25or the database30may be located off premises. In an embodiment, if the user device15is used off premises, the server25and the database30may be, for example, at a centralized location of the retailer (e.g., a headquarters, or the like) or a cloud-based system.

Examples of the user devices15include, but are not limited to, an electronic mobile device (e.g., a smartphone, a personal digital assistant (PDA), a tablet-style device, etc.), a wearable mobile device (e.g., a smart watch, a head wearable device, etc.), or the like. The user devices15generally include a display device and an input device. Examples of the display devices for the user devices15include, but are not limited to, an electronic mobile device screen, a tablet screen, a wearable mobile device screen, or the like. Examples of the input devices for the user devices15include, but are not limited to, a keyboard, a button, a voice command, a proximity sensor, a touch sensor, an ocular sensing device for determining an input based on eye movements (e.g., scrolling based on an eye movement), suitable combinations thereof, or the like. Aspects of the user devices15can be the same as or similar to aspects of the user devices501as shown and described in accordance withFIG. 8below.

The server25is in electronic communication with database30. The database30can include a variety of data related to the products being recognized by the electronic product recognizer60.

In an embodiment, the database30can include one or more databases. For example the database30shown inFIG. 1includes a product database35and a marker database40.

The product database35can generally store information related to all of the products available from the retailer. In an embodiment in which the server25and the database30are stored on premises of one of the retailer's stores, the product database35can include information that is unique to the particular store. For example, a first retail store of the retailer may sell 10,000 products and a second retail store may sell 10,000 products where at least one of the products sold at the first retail store is not available at the second retail store, or vice versa. In such a scenario, the product database35on premises of the first retail store may include different information than the product database35on premises of the second retail store. It will be appreciated that in an embodiment, even if the retailer sells different products at different stores, the product database35at each of the retail stores may include the same information (including items not sold at a particular store) for simplicity of managing the data in the product database35.

The product database35may be connected to an external network so that updates can be made on a periodic basis to the information stored in the product database35. For example, on a daily, weekly, etc., basis, the product database35may be updated to remove product information that is associated with a product the retailer no longer sells, to add product information for a new product that the retailer is beginning to sell, to modify information about products the retailer currently sells, or the like. It will be appreciated that updating the product database35can also be updated as needed (e.g., on an irregular basis).

In general, the product database35may include any information that is descriptive of the product. For example, the product database35can include a unique identifier for the product, the product name, product type, price, etc. Additionally, the product database35can include information relevant to the identification of the product in a captured image. For example, the product database35can include color information, label text, product location in the store, nearby products, or the like.

The marker database40includes markers associated with products available in a retailer's store.

It will be appreciated that the above databases are examples. Additional databases can be included. One or more of the databases can be combined into a single database. For example, in an embodiment, the product database35and the marker database40can be combined into a single database (not shown).

It is to be appreciated that various roles of user devices15, server25, and database30may be distributed among the devices in the system10. For example, the electronic product recognizer60can be partially or entirely included on the user devices15in, for example, an application or the like. Similarly, the database30can be maintained on the server25.

In general, the system10is disposed physically at the retail store of the retailer. That is, eliminating communication between the user devices15, server25, and database30occurring via a network remote from the retail store may increase a performance and response time of the system10. In an embodiment, the server25and database30may be physically located at the retail store, but may be connected in communication with an external network so that, for example, updates may be provided on an ongoing basis (e.g., nightly, weekly, etc.).

FIGS. 2A and 2Billustrate methods for recognizing a plurality of products within a captured image, according to an embodiment.

FIG. 2Ais a flowchart of a method100for recognizing a plurality of products within a captured image, according to an embodiment. The method100generally includes receiving a captured image from a user device and determining what products are visible in the captured image. The determination of what products are visible in the captured image can be performed with greater than 95% confidence. In an embodiment, the determination can be performed in less than at or about two seconds. It will be appreciated that the timing is an example, and the actual target for timing can vary beyond this number. If the method100does not identify the products visible in the captured image with greater than 95% confidence, the method100may not provide recognized products to the user devices15. In an embodiment, if the method100does not identify the products visible in the captured image with greater than 95% confidence, the server may initiate a message to the user device that enables the user to identify the product in the captured image.

At105, a server (e.g., the server25inFIG. 1) receives a video stream from a user device (e.g., the user devices15inFIG. 1). The video stream may include a plurality of still images (frames) played in succession. In general, the electronic mobile device may have a frame rate of at or about 20 to at or about 30 frames per second (FPS). It will be appreciated that these frame rates are examples and can vary according to, for example, the particular electronic mobile device being used. The frame rate of the electronic mobile device may determine how may frames are provided to the server25.

At110, the server25selects a frame of the video stream. The frame serves as the captured image. In this manner, the consumer is able to view the one or more products via the camera device on the electronic mobile device without taking action to capture the image. Also at110, the server segments the products included in the captured image (e.g., the selected frame) into discrete product images. The segmenting can include, for example, breaking the captured image into a series of discrete segments. In an embodiment, the segments can be, for example, rectangles or the like. In an embodiment, the selected frame can include a single frame from the video stream. In an embodiment, the selected frame can include a plurality of frames from the video stream.

An example of a captured image that has been segmented is shown inFIG. 3. InFIG. 3, the captured image150shows a plurality of bottles155on a plurality of shelves160. Each of the bottles155is shown with a rectangle (segment)165drawn around the bottles155. Referring back toFIG. 2, the segmenting at110can be performed using various approaches to complete the segmenting. At110, the server25may use an optical character recognition (OCR), an edge detection, a histogram of oriented gradients, a marker, an artificial neural network, or combinations thereof, to identify where the segments165should be placed. Following the segmentation at110, image recognition can be performed on each of the plurality of segments165(e.g., different product images) to identify which product is in each segment.

At115the server determines matching products for each of the segmented images for the plurality of products in the captured image using the electronic product recognizer60. In an embodiment, one manner of increasing the performance of the electronic product recognizer60is to limit the pool of possible matching products to those products sold by the retailer (e.g., only those products identified in the product database35). Accordingly, if the consumer is imaging a product or products that are not for sale by the retailer, then the electronic product recognizer60will not identify the product or products as a recognized product.

Additionally,115is performed for each of the segmented images from110. The recognition for the various images can be performed concurrently to meet the processing time requirements of identifying products in less than 2 seconds. Furthermore, as will be described in accordance withFIG. 4below,115includes a plurality of recognition processes that are concurrently executed. As a result, the recognition process may be a “race to the finish” in which the first recognition process to identify the product with greater than 95% confidence may be used as the recognized product.

At120, after the recognition of the matching products in the captured image, product information from the product database35can be output by the server25to the user device15.

FIG. 2Bis a flowchart of a method130for recognizing a plurality of products within a captured image, according to an embodiment.

The method130generally includes receiving a segmented image from a user device and determining what products are visible in the segmented image. The determination of what products are visible in the captured image can be performed with greater than 95% confidence. In an embodiment, the determination can be performed in less than at or about two seconds. It will be appreciated that the timing is an example, and the actual target for timing can vary beyond this number. If the method130does not identify the products visible in the captured image with greater than 95% confidence, the method130may not provide recognized products to the user devices15. In an embodiment, if the method130does not identify the products visible in the captured image with greater than 95% confidence, the server may initiate a message to the user device that enables the user to identify the product in the captured image.

At135, a server (e.g., the server25inFIG. 1) receives a segmented image of a selected frame of a video stream from a user device (e.g., the user devices15inFIG. 1). In an embodiment, the segmenting is performed on the user device15. As a result, the method130may consume less transmission bandwidth than the method100inFIG. 2A. In this manner, the consumer is able to view the one or more products via the camera device on the electronic mobile device without taking action to capture the image. The segmenting can include, for example, breaking the captured image into a series of discrete segments. In an embodiment, the segments can be, for example, rectangles or the like.

An example of a segmented image is shown inFIG. 3. InFIG. 3, the segmented image150shows a plurality of bottles155on a plurality of shelves160. Each of the bottles155is shown with a rectangle (segment)165drawn around the bottles155. Referring back toFIG. 2B, the segmenting can be performed using various approaches to complete the segmenting such as, but not limited to, an optical character recognition (OCR), an edge detection, a histogram of oriented gradients, a marker, an artificial neural network, or combinations thereof, to identify where the segments165should be placed. Image recognition can be performed on each of the plurality of segments165(e.g., different product images) to identify which product is in each segment.

At140the server determines matching products for each of the segmented images for the plurality of products in the segmented image using the electronic product recognizer60. In an embodiment, one manner of increasing the performance of the electronic product recognizer60is to limit the pool of possible matching products to those products sold by the retailer (e.g., only those products identified in the product database35). Accordingly, if the consumer is imaging a product or products that are not for sale by the retailer, then the electronic product recognizer60will not identify the product or products as a recognized product.

Additionally,140is performed for each of the segmented images received at135. The recognition for the various images can be performed concurrently to meet the processing time requirements of identifying products in less than 2 seconds. Furthermore, as will be described in accordance withFIG. 4below,140includes a plurality of recognition processes that are concurrently executed. As a result, the recognition process may be a “race to the finish” in which the first recognition process to identify the product with greater than 95% confidence may be used as the recognized product.

At145, after the recognition of the matching products in the captured image, product information from the product database35can be output by the server25to the user device15.

FIG. 4is a flowchart of a method200for performing image recognition, according to an embodiment. The method200generally corresponds to115in the method100ofFIG. 2A or 140inFIG. 2B. That is, when the method100is executed, at115the method200can be performed, and when the method130is executed, at140, the method200can be performed.

The method200generally includes a plurality of recognition methods that are concurrently performed to result in a recognized product. In an embodiment, the method200can be utilized to recognize a product at greater than 95% confidence. In an embodiment, the determination can be performed in less than at or about two seconds. It will be appreciated that the timing is an example, and the actual target for timing can vary beyond this number. In an embodiment, the method200can include one or more processes that are performed sequentially.

At205, a captured image of a product that was received by the electronic product recognizer60is filtered for possible matching products using a gross filtering approach. The gross filtering approach can, for example, reduce a number of possible products to less than 10% of the initial possible products. For example, if the product database35includes 10,000 products, the gross filtering approach at205can reduce a number of possible products to less than 1,000 products. It will be appreciated that these numbers are examples, and the number of possible products and products following the gross filtering approach at205can vary beyond the stated values in accordance with the principles in this disclosure.

The gross filtering approach can rely on a plurality of image recognition methods to reduce a number of possible products in the captured image. For example, in an embodiment, the gross filtering approach can include a location determination for the electronic mobile device. The location determination can include several considerations. In an embodiment, the location determination can take into account coordinates from a global positioning system (GPS) sensor on the electronic mobile device. The current GPS information can be included in the determination, as well as GPS information identifying a path by which the consumer arrived at the current location. The current GPS information, the path of the consumer, and a planogram of the retail store can be used to make an estimated determination of what the consumer should have in her field of view. In an embodiment, the GPS information can be combined with, or alternatively replaced by, location information from an in-store positioning system (IPS). The gross filtering can also include a consideration of whether a logo or brand identifier on the product image can be identified. Utilizing these approaches in combination, the possible products can be reduced to a more manageable number of possible products.

At210, the electronic product recognizer60performs a fine filtering on the possible products identified from the gross filtering at205. The fine filtering approach can, for example, reduce a number of possible products to less than 2% of the initial possible products. Continuing with the example above, if the product database35includes 10,000 products, the fine filtering approach at210can reduce a number of possible products to less than 200 products. It will be appreciated that these numbers are examples, and the number of possible products and products following the fine filtering approach at210can vary beyond the stated values in accordance with the principles in this disclosure.

The fine filtering approach can rely on a variety of approaches for reducing a number of possible products. In the fine filtering approach, the electronic product recognizer60can additionally utilize a marker associated with a brand, can determine whether there are any identifying features on an edge of the shelf (e.g., product identifiers, etc.), a color of the product, OCR, and combinations thereof. The user path may be utilized in210in addition to205to further reduce the possible product images.

At215the electronic product recognizer60selects a match from the pool of products identified in the fine filtering approach at210. If a match cannot be determined from the pool of products identified in the fine filtering approach, then the method200may end without selecting a match. At215, the determination can be based on the color of the product in the image, OCR for text in the image, and markers associated with the item, along with suitable combinations thereof. In an embodiment, a tie can be broken randomly. In another embodiment, a tie can be broken using a selected fine filtering approach.

FIG. 5is a schematic diagram of an augmented reality user interface250, according to an embodiment.

FIG. 5includes user device15that is executing the user application45. As shown inFIG. 5, the user device15includes display255. Additional details about a suitable display for the display255are discussed with respect toFIG. 8below.

The display255shows a user's view when using the augmented reality system described in this Specification. In particular, the display shows an example of a user's view within, for example, a retail store. The user's view shown includes the captured image shown inFIG. 3. As such, the display255shows a plurality of bottles155on a plurality of shelves160. UnlikeFIG. 3, each of the bottles155is not shown with a rectangle (e.g., segment165) drawn around the bottles155. The segmenting described above is not shown in the display255to the user. The user interface250may be representative of a view the user sees when first using the user application45. That is, the user interface250may be representative of the user's view prior to electronic product recognition being completed. Once one or more products are electronically recognized, the user interface250may be modified and the display may be similar to that shown inFIG. 6.

FIG. 6is a schematic diagram of an augmented reality user interface300once a product has been electronically recognized, according to an embodiment.

FIG. 6includes user device15that is executing the user application45. As shown inFIG. 6, the user device15includes display255. Additional details about a suitable display for the display255are discussed with respect toFIG. 8below.

The display255shows a user's view when using the augmented reality system described in this Specification. In particular, the display shows an example of a user's view within, for example, a retail store. The user's view shown includes the captured image shown inFIG. 3. As such, the display255shows a plurality of bottles155on a plurality of shelves160. UnlikeFIG. 3, each of the bottles155is not shown with a rectangle (e.g., segment165) drawn around the bottles155. The segmenting described above is not shown in the display255to the user.

The user interface300may be representative of a view the user sees after one or more products within the captured image are electronically recognized using the systems and methods described in this Specification. That is, the user interface300may be representative of the user's view after electronic product recognition is completed.

As shown inFIG. 6, once one or more products are electronically recognized, the user interface300can include a plurality of stickers305,310. In the illustrated embodiment, two stickers305,310are shown. It will be appreciated that the number of stickers305,310is illustrative and can vary according to the principles described in this Specification. For example, a single sticker (e.g., sticker305or sticker310) can be shown. Alternatively, more than two stickers305,310can be shown on the display255.

The stickers305,310are inserted with a connection to the product that was electronically recognized by the electronic product recognizer60. In an embodiment, the stickers305,310will follow the connected product so long as the connected product remains within the field of view shown in the display255. That is, even if the user moves bottle155connected to sticker305, the sticker305will track the movement of the bottle155while the bottle155remains in the field of view of the user device15. Accordingly, the product can be located in places other than on shelves160and still include the additional information included in the stickers305,310.

It will be appreciated that the stickers305,310are examples and that the size, placement, aesthetic, and text are representative. Accordingly, the stickers305,310and their content can be modified within the context of this Specification.

Sticker305includes product information including, for example, a product name, price, description, and product reviews. The text, the sticker305, or a combination thereof, may be selectable so that the user can browse additional information about the product. For example, the user may be able to review nutrition information if the product is a grocery product, size information if the product is clothing, or the like.

Sticker310includes product information including, for example, a product name, price, description, and product reviews. The text, the sticker310, or a combination thereof, may be selectable so that the user can browse additional information about the product. For example, the user may be able to review nutrition information if the product is a grocery product, size information if the product is clothing, or the like.

The user interfaces250(FIG. 5) and300(FIG. 6), are representative of one example application of the augmented reality system described in this Specification. It will be appreciated that the augmented reality system does not require the one or more products be arranged on a shelf. For example, the augmented reality system can be used to electronically recognize one or more products that are on a hanging rack, in a cart, on a floor display, or the like. In an embodiment, the augmented reality system may be usable when connected to a network of a retail store and the user device15is disposed on premises of the retail store so that the user device15can receive information from a server (e.g., server25inFIG. 1) and a database (e.g., database30inFIG. 1) that are also located on premises of the retail store. In another embodiment, the augmented reality system may be usable when the user is connected to a network (e.g., a wireless network, a cellular network, etc.) even when the user is not physically located on premises of the retail store.

FIG. 7is a flowchart of a method350for using an augmented reality system, according to an embodiment.

The method350can generally be performed using the system10inFIG. 1. The method350generally enables a user to use an electronic mobile device (e.g., user device15inFIG. 1) to utilize a camera of the electronic mobile device to display a real-time or substantially real-time view of one or more products and receive product information based on an electronic product recognition that is overlaid onto the real-time or substantially real-time view of the one or more products. In an embodiment, the method350can be performed by running an application (e.g., user application45inFIG. 1) that is stored in a memory of the user's mobile device. In an embodiment, the method350may be capable of being performed when the user is physically located on premises of a retail store. In an embodiment, the method350may be capable of being performed even when the user is physically remote from the retail store.

At355, a user can download and install a user application (e.g., the user application45inFIG. 1) onto a user device (e.g., the user device15inFIG. 1). It will be appreciated that355may be executed at a different time from the remainder of the method350. Accordingly,355is identified as being optional. It is to be appreciated that355and the installation of the user application45is to be performed prior to executing the remainder of the method350.

At360, the user application45enables the camera device on the user device15. At365, image information is received by the user application45from the camera device on the user device15. At370, the method300can include segmenting the received image to identify discrete products within the received image as described above, and performing the method130shown and described with respect toFIG. 2B, according to an embodiment. Alternatively, in an embodiment, at370the method300can include performing the method100shown and described with respect toFIG. 2A.

At375, once the product information is received by the user device15from the method100or the method130, a sticker (e.g., stickers305,310) is overlaid on the display of the user device15.

FIG. 8is a schematic diagram of architecture for a computer device500, according to an embodiment. The computer device500and any of the individual components thereof can be used for any of the operations described in accordance with any of the computer-implemented methods described herein.

The computer device500generally includes a processor510, memory520, a network input/output (I/O)525, storage530, and an interconnect550. The computer device500can optionally include a user I/O515, according to some embodiments. The computer device500can be in communication with one or more additional computer devices500through a network540.

The computer device500is generally representative of hardware aspects of a variety of user devices501and a server device535. The illustrated user devices501are examples and are not intended to be limiting. Examples of the user devices501include, but are not limited to, a desktop computer502, a cellular/mobile phone503, a tablet device504, and a laptop computer505. It is to be appreciated that the user devices501can include other devices such as, but not limited to, a wearable device, a personal digital assistant (PDA), a video game console, a television, or the like. In an embodiment, the user devices501can alternatively be referred to as client devices501. In such an embodiment, the client devices501can be in communication with the server device535through the network540. One or more of the client devices501can be in communication with another of the client devices501through the network540in an embodiment.

The processor510can retrieve and execute programming instructions stored in the memory520and/or the storage530. The processor510can also store and retrieve application data residing in the memory520. The interconnect550is used to transmit programming instructions and/or application data between the processor510, the user I/O515, the memory520, the storage530, and the network I/O540. The interconnect550can be, for example, one or more busses or the like. The processor510can be a single processor, multiple processors, or a single processor having multiple processing cores. In some embodiments, the processor510can be a single-threaded processor. In an embodiment, the processor510can be a multi-threaded processor.

The user I/O515can include a display516and/or an input517, according to an embodiment. It is to be appreciated that the user I/O515can be one or more devices connected in communication with the computer device500that are physically separate from the computer device500. For example, the display516and input517for the desktop computer502can be connected in communication but be physically separate from the computer device500. In some embodiments, the display516and input517can be physically included with the computer device500for the desktop computer502. In an embodiment, the user I/O515can physically be part of the user device501. For example, the cellular/mobile phone503, the tablet device504, and the laptop505include the display516and input517that are part of the computer device500. The server device535generally may not include the user I/O515. In an embodiment, the server device535can be connected to the display516and input517.

The display516can include any of a variety of display devices suitable for displaying information to the user. Examples of devices suitable for the display516include, but are not limited to, a cathode ray tube (CRT) monitor, a liquid crystal display (LCD) monitor, a light emitting diode (LED) monitor, or the like.

The input517can include any of a variety of input devices or input means suitable for receiving an input from the user. Examples of devices suitable for the input517include, but are not limited to, a keyboard, a mouse, a trackball, a button, a voice command, a proximity sensor, an ocular sensing device for determining an input based on eye movements (e.g., scrolling based on an eye movement), or the like. It is to be appreciated that combinations of the foregoing inputs517can be included for the user devices501. In some embodiments the input517can be integrated with the display516such that both input and output are performed by the display516.

The memory520is generally included to be representative of a random access memory such as, but not limited to, Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), or Flash. In some embodiments, the memory520can be a volatile memory. In some embodiments, the memory520can be a non-volatile memory. In some embodiments, at least a portion of the memory can be virtual memory.

The storage530is generally included to be representative of a non-volatile memory such as, but not limited to, a hard disk drive, a solid state device, removable memory cards, optical storage, flash memory devices, network attached storage (NAS), or connections to storage area network (SAN) devices, or other similar devices that may store non-volatile data. In some embodiments, the storage530is a computer readable medium. In some embodiments, the storage530can include storage that is external to the computer device500, such as in a cloud.

The network I/O525is configured to transmit data via a network540. The network540may alternatively be referred to as the communications network540. Examples of the network540include, but are not limited to, a local area network (LAN), a wide area network (WAN), the Internet, or the like. In some embodiments, the network I/O525can transmit data via the network540through a wireless connection using Wi-Fi, Bluetooth, or other similar wireless communication protocols. In some embodiments, the computer device500can transmit data via the network540through a cellular, 3G, 4G, or other wireless protocol. In some embodiments, the network I/O525can transmit data via a wire line, an optical fiber cable, or the like. It is to be appreciated that the network I/O525can communicate through the network540through suitable combinations of the preceding wired and wireless communication methods.

The server device535is generally representative of a computer device500that can, for example, respond to requests received via the network540to provide, for example, data for rendering a website on the user devices501. The server535can be representative of a data server, an application server, an Internet server, or the like.

Aspects described herein can be embodied as a system, method, or a computer readable medium. In some embodiments, the aspects described can be implemented in hardware, software (including firmware or the like), or combinations thereof. Some aspects can be implemented in a non-transitory, tangible computer readable medium, including computer readable instructions for execution by a processor. Any combination of one or more computer readable medium(s) can be used.

The computer readable medium can include a computer readable signal medium and/or a computer readable storage medium. A computer readable storage medium can include any tangible medium capable of storing a computer program for use by a programmable processor to perform functions described herein by operating on input data and generating an output. A computer program is a set of instructions that can be used, directly or indirectly, in a computer system to perform a certain function or determine a certain result. Examples of computer readable storage media include, but are not limited to, a floppy disk; a hard disk; a random access memory (RAM); a read-only memory (ROM); a semiconductor memory device such as, but not limited to, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), Flash memory, or the like; a portable compact disk read-only memory (CD-ROM); an optical storage device; a magnetic storage device; other similar device; or suitable combinations of the foregoing. A computer readable signal medium can include a propagated data signal having computer readable instructions. Examples of propagated signals include, but are not limited to, an optical propagated signal, an electro-magnetic propagated signal, or the like. A computer readable signal medium can include any computer readable medium that is not a computer readable storage medium that can propagate a computer program for use by a programmable processor to perform functions described herein by operating on input data and generating an output.

An embodiment can be provided to an end-user through a cloud-computing infrastructure. Cloud computing generally includes the provision of scalable computing resources as a service over a network (e.g., the Internet or the like).