Real-time shelf inventory and notification

Cameras capture images of shelves/displays having items. The images are processed to identify real-time item counts on the shelves and/or empty or partially empty shelves with less than a configured number of items. Notifications or messages regarding item counts and/or empty shelving conditions are sent through an Application Programming Interface (API) to consuming services in accordance with custom defined rules. In an embodiment, real-time item counts and/or empty shelving conditions are dynamically reported through the API based on on-demand requests from the consuming services.

BACKGROUND

Recent advancements in image processing has permitted retailers to provide a frictionless shopping experience to its consumers. In a frictionless store, an individual can check in with a store electronically, browse for items to purchase, place items in bags/carts, and exit the store without any interaction with a store agent or with a checkout station. The items that the individual possesses when leaving the store are recognized through image processing and an account associated with the individual is automatically charged for the price of the items. This allows for quick and easy shopping and is referred to as a frictionless store or frictionless shopping because the consumer does not have to interact with any agent of the store or any terminal of the store to purchase items.

However, even with these advancements item inventory on store shelving is still problematic for the industry. Typically, restocking is based on an onsite inventory management system combined with a manager's physical inspection of the store shelves. Inventory management and re-ordering of stocked goods are accurate and efficient on weekly or on monthly bases but not on daily or during a single day. Existing techniques do not address inventory shortages or shelf stocking issues in real time. Many events can occur within the store that result in deficiencies of items on the shelves of the store, which inhibit sales and customer satisfaction.

Some of these events include, item shelf inventory being depleted because items are in carts of customers within the store, customers have purchased the items recently, customers picked the items up and moved them to other locations within the store, and others. Moreover, as more and more third-party shopping services are offered to customers, this lack of real-time item shelf-inventory can result in customers ordering and paying for items only to discover when the third-party picker goes to the story shelf there are no corresponding items to fulfill a given customer's order.

Thus, there is a need for real-time item-shelf inventory and notification.

SUMMARY

In various embodiments, methods and a system for real-time shelf inventory and notification are presented.

According to an aspect, a method for real-time shelf inventory and notification is presented. A shelf of items is identified from an image. A determination is made as to whether the shelf is partially empty of the items for an item type from the image. A notification message is provided for the shelf and the item type to a service.

DETAILED DESCRIPTION

FIG.1is a diagram of a system100for real-time shelf inventory and notification, according to an example embodiment, according to an example embodiment. It is to be noted that the components are shown schematically in greatly simplified form, with only those components relevant to understanding of the embodiments being illustrated.

Furthermore, the various components (that are identified in theFIG.1) are illustrated and the arrangement of the components is presented for purposes of illustration only. It is to be noted that other arrangements with more or less components are possible without departing from the teachings of real-time shelf inventory and notification, presented herein and below.

The system100includes a shelves/displays with items110within a store, cameras120that capture images items on the shelves/display. The system100also includes an edge server130local or physically on-site of the store. The server130includes a stream data store131, an item identifier132, and a shelf-item monitor133. The system100further includes an integration Application Programming Interface (API)140, planogram services150, inventory services160, transaction services170, notification services180, and third-party services190.

Server130includes one or more processors and non-transitory computer-readable storage medium having executable instructions representing interfaces for accessing the stream data store131, item identifier132, and shelf-item monitor133. The executable instructions when executed by the one or more processors of server130perform the processing discussed herein and below with respect to interfaces to data store131, item identifier132, and shelf-item monitor133.

Integration API140represents executable instructions that may be executed by processors of server130, an independent server, and/or any server associated with services150-190.

Each of the services150-190(may also be referred to as (systems150-190herein) represent sets of executable instructions that are executed by processors of different servers to cause the processors to perform the processing discussed herein and below with respect to systems150-190.

The system100operates from within a given retail store that includes shelves/displays110having items for purchase by customers. Some items may be purchased by on-line customers through third-party pickers/shoppers utilizing third-party services190. In an embodiment, the store is a frictionless store or at least a portion of the store is frictionless. In an embodiment, the store does not include frictionless shopping systems or capabilities.

It is noted cameras120are configured and situated proximate to, overhead of, or on shelves/displays110. Moreover, other cameras may be directed capture images of store aisles, entry and exits of the store, checkout areas for Self-Service checkouts, entrance to restrooms, and the stocking room.

The cameras120are preconfigured to capture images of the defined areas on shelves/displays110where items are located based on the field-of-view of the lenses of the cameras120. Some of the cameras120may capture images representing portions of a different area that a different one of the cameras120captures images for. That is, each image can include pixel values that overlap multiple ones of the defined areas on shelves/displays110, such that a single image includes pixel values representing different items on shelves/displays110.

Thus, each camera lens is configured to cover one or more predefined areas of the shelves/displays110.

Metadata is assigned to each camera120to include a unique camera identifier, a location identifier (representing the physical location that the camera120is situated within store, and one or more area identifiers (that map to the predefined areas on particular shelves/displays110that the lens of the camera120captures in the images. Each camera120provides time stamp and frame stamped images to unique identify a date and time of day and particular image frame within a stream of image frames.

These images are streamed as captured over a wired or wireless connection between the cameras120and the server130to a stream data store131on the server130that is accessible to the item identifier132and/or the shelf-item monitor133. In an embodiment, some of the images when streamed from the cameras120can be buffered or cached in memory of cache and made accessible from the memory or cache directly to item identifier132and/or shelf-item monitor133.

Each accessible image includes its metadata (minimally including what was discussed above) with its pixels accessible from server130.

Again, lenses of cameras120have their field-of-views directed to items situated on shelves/displays110in accordance with a store planogram. Integration API140permits shelf-item monitor133to make API calls to planogram services150based on a store identifier. Planogram services150return a physical layout of the given requested store represented in a data structure, such as table. The layout includes area identifiers, shelf identifiers associated with area identifiers, and item identifiers for items on given shelve identifiers. The area identifiers also map to camera identifiers. It is noted that other types of information may also be included in the layout from planogram service150.

Shelf-item monitor133also uses the store identifier and API140to obtain item inventory for items at a given store through inventory system/service160.

Item identifier132and shelf-item monitor133are configured with the store's layout (planogram) to identify specific shelves110having specific items, camera identifiers, and area identifiers that each camera120produces images for. In some cases, item identifiers for given items may also be linked to another data store that includes image templates for a given item image, the templates identify image features for the given item image (such as lines, shape, texture, size, dimension, colors, text, package design, etc.). Shelf-item monitor133also retains item inventory for processing during operation of the system100.

During operation of system100, cameras120capture image frames with the metadata and stream to data store131. Item identifier132processes receives each frame to identify item counts on the shelves110, empty spaces of missing items on shelves110, and/or item identification or verification of items on shelves110(to identify items placed on wrong shelves110).

Item identifier132can perform pixel analysis of each frame utilizing a number of techniques. For example, known background pixels associated with a background having no shelf or items can be subtracted from each frame. Edge detection and shape processing to separate each individual shelves110from one another and to separate each item from other items of the same type or a different type present within each shelf110. Adjustments to the frames for lightening known to be present in the area of the store associated with the cameras120(this may also be based on day of year and time of day based on location of the store to account for natural lightening within the store). Frame averaging and state majority voting can be added to stabilize the filtering of each frame.

Item identifier132produces as output an image of a shelf110for the frame (or multiple frames when multiple cameras120cover a single shelf120) for a given shelf110representing a total item count for the shelf and positions on the shelf110that are empty (no pixel values for a given item) and therefore not on the shelf110.

Item identifier132maintains a previous processed output image for comparison with each current processed output. In this way, item identifier132can alert shelf-item monitor133when shelf item counts decrease and/or when a given shelf110is empty entirely. Shelf identifiers and item counts are provided from item identifier132to shelf-item monitor133. Shelf-item monitor133uses configurable rules to determine when to send a notification and to whom to provide a given notification. For example, when a soup count for a soup shelf falls below X amount or when soup shelf is Y % empty send a message or alert to notification service180using API140. Notification service180knows how to handle the notification and relays it to device's operated by store personnel associated with restocking the soup shelf with soup. In another instance, shelf-item monitor133receives an item count for soup and compares against a reported item inventory for soup at the store (through inventory service160). When the difference falls below a threshold of a given rule for soup, shelf-item monitor133uses API and sends a message to inventory services160and/or third-party services190. Should an online customer of third-party service190orders a quantity of soup that exceeds the remaining soups in total inventory for the store. Third-party service190may notify the online customer before the customer is able to place and pay for the order. Third-party service190may also display a warning message to the customer when soup is ordered if the total inventory of soup is within a threshold amount, since soup may be purchased and be out of stock before the third-party shopper is able to obtain soup for the customer. In such a case, third-party service190may allow the customer to place the order for soup but also provide backup instructions should soup not be there when the shopper arrives (provide a credit for the price of the soup to customer account or obtain a different item in the store). Upon receiving the message regarding soup inventor from shelf-item monitor133, inventory services160may generate a soup purchase order, place the order with a soup vendor, and record a scheduled date that the soup order is to arrive at the store.

In some cases, real-time item inventories based on the shelf inventory may be continually sent from shelf-item monitor as messages using API140to inventory services160, transaction services170, notification services180, and/or third-party services190. Any of these services160-190may include a user-facing dashboard service where item counts of specific items within the store and on specific shelves110are streamed in real time. This may be useful to re-stocking staff, inventory staff, third-party services' staff, and/or cashiers within the store.

The above-noted examples are only illustrative and are not intended to limit other embodiments of system100. That is, system100provides a platform for identifying and sending notifications regarding real-time item-shelf inventory for a given store. Consuming services160-190can subscribed through integration API140for receiving specific types of notifications based on specific rules regarding item shelf inventory at a given store. System100can configure item identifier132and shelf-item monitor based on a given planogram from a given store provided through planogram service150.

In an embodiment, third-party service190aggregates real-time item shelf inventory reported by two or more stores to automatically decide where to route shoppers/pickers that fulfill online customer orders.

In an embodiment, transaction services170aggregates real-time item shelf-inventory reported by two or more stores of a same retailer to automatically instruct retailer staff of one store that a given item is available in store at a different store, such that transaction service's interfaces can be used to have the different store hold the item for a customer at the store where such item is unavailable.

In an embodiment, item identifier132uses template information for images of a given item type to identify items located on an incorrect shelf110according to a given store layout (planogram). The location of these miss-shelved items within the store can be provided in a report generated by shelf-item monitory133at configured intervals of time based on custom rules (such as once an hour, once a day, once every two days, etc.). The list can be sent as a notification message to notification services180. Notification service180may send the list as a report to restock or re-shelving staff within the store. This allows for more efficient use of time of staff; rather than throwing all miss-shelved items in a cart as identified by walking the store and then re-walking the store to re-shelve the items.

In an embodiment, shelf-item monitor133may push notifications to services160-190based on rules defined through an interface accessible through API140.

In an embodiment, shelf-item monitory133may have notifications pulled by services160-190in real time and as requested on demand by services160-190through API140. For example, a manager of a given store may access interfaces of notification services180use API140are request a current item shelf count for a given item within the store because the manager is expecting a rush on such items because of some external event.

Again, a variety of beneficial scenarios can be processed and provided through system100in an easy to integrate manner with a plurality of systems150-190with integration API140.

These and other embodiments are now discussed with reference toFIGS.2-4.

FIG.2is a diagram of a method200for real-time shelf inventory and notification, according to an example embodiment. The software module(s) that implements the method200is referred to as an “item shelf inventory manager.” The item shelf inventory manager is implemented as executable instructions programmed and residing within memory and/or a non-transitory computer-readable (processor-readable) storage medium and executed by one or more processors of a device. The processor(s) of the device that executes the item shelf inventory manager are specifically configured and programmed to process the item shelf inventory manager. The item shelf inventory manager has access to one or more network connections during its processing. The network connections can be wired, wireless, or a combination of wired and wireless.

In an embodiment, the device that executes the item shelf inventory manager is the server130. In an embodiment, the server130is located proximate to or on-site of a retail store.

In an embodiment, device that executes the item shelf inventory manager is a cloud-based processing environment associated with a collection of servers. In this embodiment, images captured of shelves and items on shelves are streamed to the cloud-processing environment or a local storage location local to a given store that provides access over a network connection to the could-based processing environment.

In an embodiment, the item shelf inventory manager is all or some combination of: the item identifier132, shelf-item monitor133, and/or integration API140.

At210, the item shelf inventory manager identifies a shelf of items from an image.

In an embodiment, at211, the item shelf inventory manager identifies the shelf from a plurality of shelves based on a planogram for a store where the shelf is located. In an embodiment, the planogram is obtained from the planogram service150using API140.

In an embodiment of211and at212, the item shelf inventory manager identifies the item type for the items on the shelf based on the planogram.

At220, the item shelf inventory manager at least determines whether the shelf is partially empty of items for an item type from the image. A percentage of emptiness or fullness of the shelf for the given type of item may be calculated or derived from the image.

In an embodiment of212and220, at221, the item shelf inventory manager counts a total number of items present on the shelf from the image.

In an embodiment of212and220, at222, the item shelf inventory manager identifies other items that do not belong on the shelf from the image based on the item type and the planogram (miss-placed items within the store).

In an embodiment, at223, the item shelf inventory manager identifies edges, a shape of the shelf, and each individual item from the image.

In an embodiment of223and at224, the item shelf inventory manager remotes portions of the image associated with a known background for the shelf.

In an embodiment, at225, the item shelf inventory manager compares a previous image of the shelf against the image and determines the shelf is partially empty based on the comparing at220.

In an embodiment of225and at226, the item shelf inventory manager determines a total item count for the item based on the comparing at220.

At230, the item shelf inventory manager provides a notification message for the shelf and the item type to a service.

In an embodiment of225and230, at231, the item shelf inventory manager identifies a service identifier for the service from a plurality of service identifiers associated with a plurality of services based on a rule associated with the item type and the total item count for the items.

FIG.3is a diagram of another method300for real-time shelf inventory and notification, according to an example embodiment. The software module(s) that implements the method300is referred to as a “real-time item inventory manager.” The real-time item inventory manager is implemented as executable instructions programmed and residing within memory and/or a non-transitory computer-readable (processor-readable) storage medium and executed by one or more processors of a device. The processors that execute the real-time item inventory manager are specifically configured and programmed to process the real-time item inventory manager. real-time item inventory manager has access to one or more network connections during its processing. The network connections can be wired, wireless, or a combination of wired and wireless.

In an embodiment, the device that executes the real-time item inventory manager is the server130. In an embodiment, the server130is a located proximate to or on premises of a given retail store.

In an embodiment, the device that executes the real-time item inventory manager is a cloud-based processing environment associated with a collection of servers. In this embodiment, images captured of shelves and items on shelves are streamed to the cloud-processing environment or a local storage location local to a given store that provides access over a network connection to the could-based processing environment.

In an embodiment, the real-time item inventory manager is all of or some combination of: the item identifier131, shelf-item monitor133, and/or the method200.

The real-time item inventory manager presents another and, in some ways, enhanced processing perspective of the method200discussed above.

At310, the real-time item inventory manager determines item types for items on shelves of a store based on a planogram.

In an embodiment, at311, the real-time item inventory manager obtains the planogram from planogram service150using API140and a store identifier for the store.

At320, the real-time item inventory manager compares first images taken for the shelves at a first time against second images taken for the shelves at a second time that is subsequent to the first time.

In an embodiment, at321, the real-time item inventory manager isolates background pixels for a background, shelf pixels for a given shelf, and item pixels for a given item type in both the first images and the second images for each shelf before performing the comparing at320.

At330, the real-time item inventory manager identify based on320when at least one shelf requires restocking of a corresponding item having a corresponding item type.

In an embodiment of321and330, at331, the real-time item inventory manager calculates a total item count for the corresponding item of the shelf (identified at330) based on the comparing at320.

In an embodiment of331and at332, the real-time item inventory manager identifies a specific item of a specific type on the shelf identified at330that does not belong on that shelf based on the comparing at320.

In an embodiment, at333, the real-time item inventory manager obtains an item inventory for the corresponding item type from item inventory system160of the store using API140.

At340, the real-time item inventory manager sends a notification to a service identified in a rule based on the identifying at330of the shelf requiring restocking.

In an embodiment of333and340, at341, the real-time item inventory manager sends the notification to an in-store personnel notification system (notification system180) based on conditions in the rule indicating that a difference between a total item count for the corresponding item type and the item inventory falls below a threshold value and a service identifier in the rule that is set to the in-store personnel notification system (notification system180).

In an embodiment, at350, the real-time item inventory manager sends a total item count for the corresponding item type to an inventory dashboard service of an inventory management system160or a transaction service/system170.

FIG.4is a diagram of another system400for real-time shelf inventory and notification, according to an example embodiment. The system400includes a variety of hardware components and software components. The software components of the system400are programmed and reside within memory and/or a non-transitory computer-readable medium and execute on one or more processors of the system400. The system400communicates over one or more networks, which can be wired, wireless, or a combination of wired and wireless.

In an embodiment, the system400implements, inter alia, the processing described above with theFIGS.1-3.

The system400includes a plurality of cameras401, a server402, and a plurality of services406. The server402includes at least one hardware processor403and configured to execute executable instructions from a non-transitory computer-readable storage medium404as an item shelf inventory manager405.

The item shelf inventory manager405when executed from the non-transitory computer-readable storage medium404on the processor403is configured to cause the processor to: 1) determine item inventories for the items present on the shelves from the images based on a planogram for the store; 2) process rules for the item inventories and the shelves; and 3) send notification messages to services based on the rules.

In embodiment, the item shelf inventory manager405when executed by the processor403from the non-transitory computer-readable storage medium404further causes the processor403to: 4) determine when a given item that is located on a given shelf is in an incorrect shelf location within the store based on the planogram and the images.

In an embodiment, the item shelf inventory manager405is all or some combination of the: item identifier132, the shelf-item monitor133, API140, the method200, and/or the method300.

In an embodiment, the server402is server130. In an embodiment server130, is proximate to or located on premises of a retail store.

In an embodiment, server402is one of several servers that cooperate to form a cloud processing environment. The images captured by cameras401are accessible to the cloud processing environment local or proximate to the store where the cameras120are located.

In an embodiment, the services406are: planogram service(s)150, inventory service(s)160, transaction service(s)170, notification service(s)180, and one or more third-party services190.

In an embodiment, the system400is deployed as a portion of a frictionless store implementation where customers (individuals) shop through computer-vision and image processing and items and individuals are associated with one another with a shopping cart maintained for each individual. Each individual can checkout and pay for his/her shopping cart items using any traditional terminals or mobile devices.

In an embodiment, the system400is deployed in a traditional and non-frictionless store to provide custom item shelf inventory notification and reporting services in the manners discussed herein and above.