Patent ID: 12254671

DETAILED DESCRIPTION OF THE INVENTION

The present disclosure, in some embodiments, concerns systems and methods for inventory management, and more specifically, but not exclusively, to a system applying simultaneous localization and mapping in three dimensions (SLAM 3D) to optimize the training and use of deep neural networks for the identification of products.

The examples depicted below illustrate applications of the systems and methods to tracking inventory of products in a retail establishment, such as dry foods and toiletries. However, the process depicted herein is potentially applicable to tracking of any objects that remain substantially stationary for long periods of time, but whose extent or position may change gradually over time. Examples of such applications include, but are not limited to, tracking inventory materials for a factory stored in a warehouse, or tracking progress of crops growing in a greenhouse.

Referring now toFIG.1, system10for inventory management includes at least one mobile device12. Mobile device12may be a handheld electronic device such as a tablet computer or a mobile phone. Mobile device12may be, in alternative embodiments, AR glasses, a drone, a robot, or, in general, any device that is capable of moving or being moved around a 3D scene.

Mobile device12is also referred to herein as an edge computing device. The term “edge computing” refers to a distributed computing paradigm that brings computation and data storage closer to a location in which it is needed. Unlike cloud computing, edge computing runs most operations on a local device without a remote connection to a central server. As a result, in general, edge computing is able to produce much quicker computing results than cloud computing.

Mobile device12is equipped with an image sensor14, a processor16, a memory18, and, optionally, a display20. Generally, these components of mobile device12have conventional characteristics. Image sensor14is any image sensor suitable for capturing images of a scene for purposes of object detection. The image sensor14may be, for example, an RGB image sensor, an infrared image sensor, a CCD image sensor, or a CMOS image sensor. Processor16includes circuitry for executing computer readable program instructions stored on the memory. Memory18is a non-transitory storage medium having stored thereon code instructions that, when executed by processor16, causes performance of various steps. The storage medium may be, for example, an electronic storage device, a magnetic storage device, an optical storage device, a semiconductor storage device, or any suitable combination of the foregoing. In particular, the functions described herein may be performed on an application, or an “app,” installed on the mobile device12.

Display20may be any standard screen or display suitable for implementation in a mobile computing device, such as LCD, OLED, AMOLED, Super AMOLED, TFT, or IPS. In addition or alternatively, display20may be or include a wearable holographic projection headset. An example of a suitable headset is the Hololens® mixed reality headset manufactured by Microsoft.

The display20and processor16must have the technical capacity to support all of the technical functions described below, including, for example, evaluation of a three-dimensional scene, holographic projection, augmented reality, and ray-casting. These capabilities are typically found in tablet computers and mobile phones that include suitable hardware. In some cases, the image sensor14may also include depth-sensing capabilities, e.g. using Lidar, which enables improved 3D capture of a scene, leading to faster and more accurate SLAM tracking needed for augmented reality.

Mobile device12is further equipped with a communications module for wirelessly communicating with the cloud, for example, via a Bluetooth or wireless internet connection. This wireless connection is used, inter alia, for downloading software updates, including updates to the deep neural networks disclosed herein.

Memory18includes a program for generating and tracking SLAM 3D location markers for objects, and running a deep neural network for object identification from images. The deep neural network may have any architecture that is known or that may become known for performing object recognition, such as a fast-CNN architecture, or a YOLO architecture.

Optionally, the deep neural network is a hierarchical system of multiple neural networks. The hierarchical system uses hierarchical choice algorithms, based on inference results and spatial awareness information. The hierarchical system is used to reduce computing resources required to identify objects, by narrowing the number of items in a class. Thus, for example, instead of identifying a product from among every product in a store, which requires selection of one object out of thousands or tens of thousands, the deep neural network first selects the shelf or type of product that is being viewed (for example, toiletries or shampoos). After the shelf is selected, a second deep neural network selects the brand of product (for example, Pantene®), coupling its selection to the same product instance based on the SLAM 3D tracking of it in the scene (as will be discussed further herein). After the brand of product is selected, a final deep neural network selects the particular product that is depicted (for example, Pantene® Pro-v Classic Clean shampoo). In addition to identifying products based on visual properties such as shape, color, and text of packaging, the deep neural network may also be trained to read and identify bar codes, in a manner known to those of skill in the art.

In exemplary embodiments, there are hundreds of micro-deep neural networks stored on each mobile device. This configuration of the deep neural networks as micro-networks optimizes the speed of the offline training and real-time running of each model, the ability to adapt each model to a specific store, and the memory required to store each deep neural network on the mobile device12.

Typically, all of the neural networks relevant to a given location are stored on the memory of each mobile device10. In exemplary embodiments, deep learning networks able to identify50,000products are stored in 2 GB of a mobile device's memory. Because the mobile device is preloaded with all neural networks relevant to a location, it is able to function completely autonomously, as an edge computing device. Nevertheless, because deep-neural networks are memory-intensive, and in view of current memory limitations of mobile computing devices, it may not be possible to store deep neural networks relevant to all products in active short-term memory (i.e., RAM) of the mobile device. Accordingly, in exemplary embodiments, a select number of deep neural networks relevant to particular shelves are uploaded onto the mobile device's short-term memory at any given time. For example, if the user wishes to image shelves containing shampoos, the user pre-fetches, in advance, all deep neural networks relevant to those products from the mobile device's long-term memory onto the short term memory of mobile device12.

Memory18may also include a floor plan. The floor plan may include a representation of layout of shelves on a floor of a retail establishment, and a general description of the contents of the shelves (for example, shampoos, potato chips, or cereals). The floor plan24may optionally include a planogram, namely a three-dimensional representation of the layout of products on each of the shelves.

System10may also include a central management system22. In exemplary embodiments, central management system22is a cloud-based product, such as Google® Cloud Platform, running on a virtualized machine. Alternatively, central management system22is a physical computer. The same deep neural networks and floor plan that are stored on the memory18of mobile devices10may also be stored on a memory26of the central management system22.

Central management system22further includes a dashboard-style computer program24for aggregating and managing the results of the inventory tracking performed by each of the mobile devices12at each shelf. In exemplary embodiments, when a mobile device12is used to scan shelves, the mobile device12stores results and data of the scan on memory18of the mobile device. When mobile device12is connected to the internet, the mobile device12transfers the results and data to a cloud-based virtual machine on which the dashboard-style program24is stored.

FIG.2lists steps in a method for recognizing and tracking products in an inventory, andFIGS.3A-6Cdepict illustrations of various steps in this method.

At step101, the deep neural networks are loaded onto the mobile device12. As discussed above, the deep neural networks are stored on the mobile device, so that the mobile device is able to perform the subsequent steps as an edge computing process. This loading may be done well in advance of scanning any particular shelf. In addition, prior to scanning any particular shelf, the deep neural networks relevant to the products on that shelf are loaded into short term (RAM) memory of the mobile device12. This is typically performed in close proximity to the subsequent steps.

At step102, the processor marks products on the shelf with simultaneous localization and mapping in 3 dimensions (SLAM-3D) markers. This SLAM 3D marking may be performed using conventional methods. For example, referring toFIG.3A, the processor may receive multiple images of shelf unit200. Shelf unit200has shelves210,220, and230, with different products212,214,216,222,224,226,232,234on the shelves210,220,230. The processor executes a SLAM 3D program continuously on all the images, along with motion sensor data concurrently recording with each image. Examples of SLAM 3D programs suitable for carrying out the identification process described herein include ARKit® for Apple® operating systems, ARCore® for Android® operating systems, and ARFoundation as an abstraction layer for running ARKit or ARCore within Unity engine apps. The SLAM 3D program generates a plurality of bounding boxes. Each bounding box represents a product from the inventory whose three-dimensional location is to be tagged and tracked. For example, atFIG.3B, bounding boxes242,244,246, and248are drawn around certain of the products. The bounding boxes may be two-dimensional, as shown, or three-dimensional. Optionally, the bounding boxes may be displayed to the user. However, this is not necessary, and it is sufficient that the bounding boxes be stored as data points in three-dimensional space.

Notably, the bounding boxes are derived from the received images of the shelf, rather than from a 3D depth sensor. The algorithm for determining the bounding boxes may use an approach similar to that used by the human brain to derive depth from a monocular view. In particular, the algorithm may rely on 1) the parallax between objects at different depths when they are imaged from different perspectives in three-dimensions; and 2) the location of the products and points of contact thereof with the shelves, whose height and depth are generally known and may be incorporated into software for the 3D mapping. The 3D mapping may also incorporate deep learning models able to determine the depth of an item based on its appearance in images.

The bounding boxes are used for aggregating identifications of products from deep neural networks applied on two-dimensional images, as will be discussed further herein. The bounding boxes are also used for persistent capturing and reloading of 3D positions, for example, upon return visits to the same shelf, as will be discussed further herein.

Still referring to step102, and as shown inFIG.3C, the SLAM 3D program also assigns a persistent marker240to each product. The persistent marker is also referred to herein as a spatial anchor. The persistent marker may be portrayed to the user as a holographic image overlaying the product on the screen, as shown in subsequent Figures. Alternatively, the persistent marker may simply be a data point without an accompanying graphical representation.

At step103, the processor identifies various objects in the 2D images. As part of this process, and as depicted inFIG.4A, the processor defines bounding boxes, such as bounding boxes252,254,256,258, around each of the products on the shelf. This process is also referred to herein as object localization.

The processor then identifies a product contained within each localized object. This identification may proceed, for example, through detecting a barcode on the product, or an SKU number. Typically, the identification is performed through application of a hierarchical deep neural network. In some embodiments, the first level of the hierarchical deep neural network is identification of the shelf or type of product, from among a selection of shelves on a floor plan. In such cases, the entire floor plan24is uploaded onto the active memory of mobile device12in advance of operation. In other embodiments, the type of shelf or product is pre-selected, so that the deep neural network proceeds with identification of brand and specific products.

Because the images used to identify the different products are taken from different angles, the perspective of the image may be used as part of the product identification process. For example, the processor may use estimated 3D shelf angle, distance to the shelf, and illumination—which are derivable from the 3D mapping—as an additional signal for the deep neural network, or in order to align the input images for the deep neural network. Such alignment may be performed using image processing, geometric manipulations, and transformations, also based on generative adversarial neural networks (GAN).

Following completion of step103, every product on the shelf is identified. The processor assigns product identities to each of the identified products in the image. The product identities are typically a textual identification including the SKU code, brand name and product name. For example, as seen inFIG.4A, the product in bounding box252is assigned label251a, identifying it as XYZ shampoo1, with “XYZ” being the brand and “shampoo1” being the product type. The product in bounding box254is assigned label253a, identifying it as XYZ shampoo2; the product in bounding box256is assigned label255a, identifying it as ABC can1, and the product in bounding box258is assigned label257a, identifying it as DEF box1. Although not illustrated inFIG.4A, it is understood that each identical product is assigned the same textual label. For example, each of the four products232is assigned a textual label “DEF box1,” while having a different persistent 3D marker.

At step104, the processor associates each product identity from each two-dimensional image with the corresponding bounding box of the SLAM-3D mapping. As a result, following this step, the mobile device has stored therein the 3D locations of objects, whose contours are delineated with bounding boxes, and which have assigned thereto textual identifiers obtained through image processing. This association is schematically illustrated inFIG.4C, which shows the products on the shelf having both the persistent markers previously shown inFIG.3C, as well as the textual identifiers previously shown inFIG.4A.

Referring now to step105, theoretically, and as described above, it is possible to complete the identification of products based on evaluation of a single image. However, it is particularly advantageous, to perform the identification process simultaneously on multiple images of the same shelf, taken from different angles. In such embodiments, the processor applies the deep neural network on multiple images of the shelf captured from different angles. As a result, each SLAM-3D bounding box is associated with identifications from multiple corresponding images. These product identities are aggregated, and a voting analysis (e.g. even a simple majority-vote) is then applied to determine the correct identifier for the product.

This process is schematically illustrated by comparingFIG.4BtoFIG.4A.FIG.4Bschematically illustrates operation of the product identification on a second view of the shelf unit200. Sometimes the deep neural networks analyzing different images produce the same results, and sometimes they produce different results. For example, in the analysis of the scene illustrated inFIG.4B, object232is identified with identifier257bas DEF box1, just like identifier257a. However, other identifications are different. Identifier251bindicates XYZ shampoo2, as opposed to XYZ shampoo1, and identifier255bindicates ABC can2instead of ABC can1. There is no identifier corresponding to identifier257a; the perspective of the image did not provide a clear-enough view of that product to supply the basis for an identification.

By aggregating the different identifications for “n” views of the shelf, and applying a decision analysis to the aggregated identifications, the system is able to produce highly accurate results. Consider the aggregation of n identifications of the same product, in which each identification is estimated to have a reliability r of 80%. Assume further that a simple majority voting mechanism is applied, so that the product identification that is the most commonly selected is chosen, and the textual identifier corresponding to that product is output. As the number of iterations n increases, the probability p of reaching ground truth converges to 1, in the spirit of the Condorcet Jury Theorem. For the values given, using one iteration, the probability of the network being correct is 80%; for 3 iterations, ˜90%; for 5 iterations, ˜95%, and for 9 iterations, ˜99%. Notably, these highly accurate results are achievable even with deep neural networks that are not particularly robust and that do not necessarily have high accuracy when operating on their own. For example, a deep neural network with only 70% accuracy for a single image reaches ˜90% accuracy after just 10 iterations. In practice, many more images are captured and processed in real-time than 5-10, and thus the system is virtually guaranteed to select the correct identification.

The result of the “voting” analysis is illustrated schematically inFIG.4C. As shown inFIG.4C, the system has output textual identifiers251,253,255,257, respectively, each of which is indicated to be final.

Optionally, the system is configured to update the voting analysis continuously, as different images of the shelf are captured from different angles. The system may further be configured to display the continuously updating results of the voting analysis while the voting analysis is ongoing, although this is not required.

In the foregoing examples, the memory18is pre-loaded with information about all the products on the shelf, so that as soon as a product is identified, the memory is able to select the pre-stored textual identifier to match that product. In an alternative scenario, the processor is configured to identify a new product.

For example, the processor may identify a new product based on its barcode. The barcode may be read with a conventional barcode scanner connected to the mobile device12, or with an image-processing implementation running on the mobile device12. The mobile device may receive further information about the product associated with the barcode via a wireless network connection, for example from an enterprise resource planning (ERP) or inventory management system. The mobile device12then captures images of the new product, and associates the new images with the scanned barcode. These images form the basis for a classifier for identifying subsequent instances of the new product.

In addition to identifying new products, the processor is also capable of assigning new identifications of products to images. In particular, the processor is configured to apply a textual identification onto a corresponding image of the product associated with a 3D bounding box, in which the deep neural network was unable to identify the product from analysis of the image. Consider againFIGS.4A-4C. The deep neural network, operating on the view ofFIG.4A, is able to identify object216as XYZ shampoo2. From the view ofFIG.4B, however, the deep neural network was unable to identify the same object. Once the object is definitively identified as XYZ shampoo2, based on the aggregated voting analysis, the processor “ray-casts” that identification onto all examples of that object in all the images. As a result, the system now learns that an object appearing like product216, from the vantage point of the view ofFIG.4B, is XYZ shampoo2.

Furthermore, the deep neural network may also recognize an object as an unidentified object. The SLAM 3D tracking may associate multiple images of the same unknown product instance on the shelf and annotate or “tag” them with the same textual identifier, for a later re-training of the deep neural network, which thus learns to identify the new product in later sessions. This identifier may be selected on the basis of the barcode scanning, as discussed above. Optionally, the identifier may be user-generated. For example, a user may assign a product identifier to a single product within a single image, from among the group of annotated unknown products. The processor then applies the user-generated identification to instances of said product in additional two-dimensional images.

FIGS.5A-5Cdepict views of a mobile device performing steps101-106described above.FIG.5Adepicts a mobile device310scanning a shelf400full of shampoo products. The shampoo products are viewable on the display318of the mobile device, without any additions or enhancements. AtFIG.5B, the mobile device310has completed its 3D scanning of the shelf400, and has applied persistent 3D markers440to each of the shampoo bottles on the shelf400. The persistent 3D markers440are shown as holographic images overlaying the images of the shampoo bottles in the display. AtFIG.5C, in addition to the persistent markers440aand440b, each product has a textual identifier, for example textual identifier441that is projected on the screen. Also visible inFIG.5Cis an action box449, through which the user may perform actions associated with updating the products on the inventory, as will be discussed further herein.

In the embodiments described above, the persistent 3D markers are overlaid onto the image of the shelf in the form of dots, with a single dot in front of each product. In alternative embodiments, the processor uses augmented reality technology to project continuously a portion of the shelf in front of the shelf, as will be described further herein.

Typically, steps101-106are completed within no more than 20 seconds after commencement of operation. This short time frame is a dramatic improvement from known systems, which typically require at least 10 minutes and up to 24 hours to perform similar work. One reason that the disclosed embodiments are able to achieve such improvements in speed is that all SLAM 3D marking and image processing is performed as an edge computing process on the mobile device. In addition, testing of systems10implemented according to the present disclosure have shown a 95-97% accuracy rate for identifying products within this short time frame of approximately 10-20 seconds.

FIGS.6A-6Cillustrate, in three-dimensions, the generation of two-dimensional images from different vantage points, the display of these images, and the use of the SLAM-3D markers and bounding boxes to maintain persistent identification of the products regardless of the location of the mobile device relative to the shelf.

FIG.6Aillustrates this projection of the bounding boxes when the mobile device is moved between different vantage points.FIG.6Ashows an augmented reality display shown by a mobile device of a product502on shelf500. When viewing the shelf from the left, the mobile device displays holographic projection560ain front of the shelf. Holographic projection560ais a representation of a 2D image of the shelf taken from the left. The projection may include a bounding box562aaround an image of the product502. The holographic projection560ais placed such that, when extended to the shelf along vector564a, the bounding box would demarcate the boundaries of product502. When viewing the shelf from the right, the mobile device displays holographic projection560b. Holographic projection560bincludes bounding box562b, which, again, is placed so that when extended to the shelf along vector564b, the bounding box562bwould delineate the boundaries of the product502. Vectors564aand564bmay be “ray-cast” onto the display as well, for ease of identification of the correlation by a user. The holographic projection thus remains visible on an image of product502, regardless of the angle at which product502is viewed.

Thus, the holographic projections560aand560bare visible manifestations of the processes described above, and specifically, the association of SLAM 3D markers to corresponding sections of multiple images. This association enables the voting analysis used for highly accurate identification, as well as annotation of new views of images for which the deep neural networks were unable to provide a correct identification, as discussed above.

FIG.6Billustrates a composite view of a shelf with multiple holographic projections560a,560b, and560c. Each holographic projection depicts the products on the shelf as viewed by the mobile device at a corresponding vantage point. For example, in projection560b, a white container is visible to the right; this container is not visible in holographic projection560a.FIG.6Cis a close-up view of the portion ofFIG.6Bthat is within the dashed rectangle. As can be seen in this view, projection560aincludes a bounding box512,514,516around each product. Projection560aalso includes a textual identifier511,513,515for each product. Each textual identifier is ray-cast onto the corresponding product in the 3D scene. When a product is captured in more than one view, a 2D image of that product is shown with multiple textual identifiers ray cast therein. For example, container505ahas the text “pringles ranch” ray cast three times onto it, one from each of the holographic views depicted inFIG.6B.

The processor uses the new product identifications from the two-dimensional images to optimize the deep neural networks. In contrast to a conventional deep neural network, whose training is completed on images having known ground truths before it identifies unknown images, the deep neural networks of the disclosed embodiments are continually exposed to new images suitable for training. This is because, due to the persistent identification of objects in subsequent 2D images through association with the 3D markers, each product in a subsequent 2D image is identified. Thus, no matter the angle from which the object is viewed, or the percentage of the product that is visible from that angle, the product is always identifiable. The new views of the product may thus be saved and incorporated into a classifier used for the training of the deep learning network. This capacity of the deep neural network for autonomous self-learned image annotation eliminates the need for “human in the loop” annotation of new images for the deep learning network. In addition, because the deep neural network is capable of self-learning, it is possible to train the deep neural network with just a few images per product, and then to supplement the training with the newly-generated views.

Optionally, this updating of the training of the deep neural network may take place on a central management system rather than the mobile devices themselves. That is, the additional images and identifiers are uploaded to the central management system22, and the central management system22updates the relevant deep neural networks, and transmits updated versions of the deep neural networks back to the mobile devices. This updating may occur during a period when the mobile device is not otherwise in use, for example overnight. Due to this capacity for revising the training of the network, subsequent applications of the network for object identification (e.g., to identify products on previously unseen shelves) are more accurate than previous applications.

When a floor plan24is uploaded into the mobile device12, in addition to being used for persistent identification and improvement of the deep neural network, the 3D markers may be used for navigation. Specifically, as discussed above, as part of the first step of product identification, the mobile device identifies the shelf in front of which it is positioned. When this identification is overlaid with the floor plan, the mobile device12is able to provide directions to any other location on the floor plan. These directions may likewise be projected onto the screen20in augmented reality, for example, over the images of aisles and shelves.

Referring now toFIG.7, the system and method described above may be used in a method600for monitoring and controlling inventory.FIGS.8A-8C and9A-9Bdepict examples of graphic user interfaces for carrying out the method described inFIG.7.

At step601a, the processor16derives a shelf plan from a first scan, or imaging session, of a shelf. This first scan is used as a baseline against which subsequent statuses of the shelf are to be compared. Thus, in certain respects, the first scan of the shelf is akin to a planogram. However, unlike a conventional planogram, which is prepared “offline,” in a theoretical manner, the baseline generated in step601ais based on actual arrangement of a shelf. In addition, the shelf plan derived in step601ais derived as part of an edge computing process. It is not necessary for the mobile device12to be connected to a network at all when generating the shelf plan.

Alternatively, at step601b, the processor16receives a shelf plan from a planogram.

At step602, the user obtains a subsequent image of a shelf. For example, a user may return to the shelf a day after running the first scan or receiving the planogram and generating the shelf plan, to evaluate the status of inventory on the shelf.

At step603, the processor identifies absent or misplaced products. The processor does this by comparing the “realogram”, i.e. the actual final 3D locations of the products, as inferred from the current scan of the shelf, with the “planogram”, i.e. the 3D marker positions showing expected locations of products on the shelf. The processor then modifies its display of the holographic markers corresponding to each of the products, depending on the status of that product.

Advantageously, this comparison and identification is performed as an edge-computing process, effectively in real time, as the mobile device scans the shelves. Known systems for comparing actual shelf setup to prior setups require sending images to be processed to a central computer via the cloud. This process may take several minutes. By contrast, the comparison and identification according to the described embodiments is completed in just a few seconds, as an edge computing process.

FIG.8Adepicts an exemplary graphic user interface720showing results of the identification performed at step603. Each of the products on the shelf is given one of four holographic identifiers: “check” mark702, indicating that the product was scanned and found to be present in the right location and the right orientation; “null” mark704, indicating that the product is absent; “eye” mark706, indicating that the product is misplaced; or “rotation” mark708, meaning that the product is misaligned. Each of these markers may be in a different color, for ease of reference. In addition to marking each product location on the shelf, the graphic user interface720includes an aggregate shelf compliance summary710. The shelf compliance summary includes, inter alia, a shelf identifier (such as “D321”, in the present example), and a shelf “score” representing the percentage of products that are present and correctly situated (81%, in the present example). The graphic user interface720also includes a comparison region712, showing images from previous scans of the same shelf, marked with the dates and times of those scans.

Optionally, the graphic user interface may also display a calculation of how much time would be required to restore the shelf to its initial state. The system may further include an algorithm for calculating the minimal number of steps necessary to convert the present state of the shelf (realogram) to the original or desired state of the shelf (planogram), and correspondingly an output to the user of the recommended sequence of steps to take to achieve this conversion. In exemplary embodiments, the recommendation of how to achieve the conversion is determined with a sequence alignment algorithm. Sequence alignment algorithms are used in bioinformatics to evaluate differences between long strands of DNA. The order of products on a shelf may be analogized to an order of nucleotides, for purposes of applying the algorithm. For example, suppose that a group of four products is placed in the correct order on a shelf, but is displaced. In theory, this displacement results in four consecutive errors. However, because the algorithm recognizes the sequence, it is able to determine that it would be more efficient to simply move the sequence of products down the shelf, as opposed to replacing each product.

At step604, when there is a discrepancy between the current state of the shelf and the initial state, the user may order new inventory, directly from the graphic user interface depicting the comparison. To enable ordering the new inventory, the mobile device is integrated with an inventory management or enterprise resource planning (ERP) system used by the retail establishment. For example, referring toFIG.8B, selection of one of the markers (e.g., marker704) calls up a prompt714. The prompt714may include, for example, a name of a product (“Kleenex®”), an image of the product, a description of the product (“Tissue handkerchiefs for daily use”) and a series of buttons for selecting a number of boxes to order and executing the order. Optionally, the prompt may include a specific recommendation of how many boxes to order. AtFIG.8C, following execution of the order, the graphic user interface shows a confirmation screen716indicating that the order was sent. Of course, instead of using the mobile device12to order the new inventory, the user may also address the inventory needs in a low-tech fashion, for example by manually fetching and shelving the new products.

The augmented reality prompts may also be used for other purposes, in addition to ordering new products. For example, augmented reality communications may be sent to employees, including, for example, product-specific notes, instructions, and real-time remote assistance.

At step605, the mobile device12communicates information regarding each shelf to the central computer22. At step606, the user monitors aggregate shelf performance at the central computer22.

FIGS.9A and9Billustrate exemplary graphic user interfaces for aggregate monitoring of a store floor plan in a “dashboard” view, according to embodiments of the present disclosure. Referring toFIG.9A, graphic user interface820includes a schematic view of floor plan800. Within floor plan800are individual shelves802,804,806,808,810. On top of each shelf are three indicators, corresponding to “missing” items, “misplaced” items, or “rotated” items. The indicators may be round, indicating that there are such missing, misplaced, or rotated items, or flat, indicating that there are none. The graphic user interface820further includes a summary section812. The summary section812includes the name of the store and the cumulative number of missing, misplaced, and rotated items on all the shelves. The graphic user interface may further include a collection814of the most recent images scanned by the mobile devices. Each shelf is selectable for display of more information about that particular shelf. For example, in the view ofFIG.9A, a user is preparing to select shelf810, as illustrated by the “bulls-eye” appearing on top of the shelf. AtFIG.9B, the user has selected shelf810, to reveal a status box814. Status box814includes an identifier for the shelf (for example, “D325”), a time of the last scan, a cumulative shelf compliance score, the number of missing, misplaced, and rotated products, and a button for calling up a preview image of the shelf.

The aggregate monitoring of the shelves may also include other analyses relevant to store management, such as quantity of sales, effective deployment of sales staff, comparison of the actual shelf setup with a planogram, and recommendations for changing the floor plan or planogram of each shelf in order to maximize sales.

In addition to the dashboard being organized according to the store layout, the dashboard may be organized based on other data, such as employee, product stock keeping unit (SKU), shelf, chain store, etc. The dashboard may also present more elaborate statistical analytics of the collected data as well as predicted trends of various retail-relevant parameters.

Optionally, the aggregate monitoring may also be used to direct subsequent scanning of shelves. For example, because the central computer stores the time of the last scan of each shelf, the central computer may identify priorities regarding which shelves should be scanned next. These priorities may be communicated to the mobile device in the form of a task list, in order to instruct the user (e.g., a store employee) which shelves to scan. When the employee completes a scan on the task list, he or she marks that task completed.

The augmented reality features described herein may be implemented in other aspects of the retail experience. For example, augmented reality markers of products in the store may be integrated with augmented reality headsets worn by shoppers. This, in turn, may support continuous updating of products selected by shoppers, enabling a seamless self-checkout process. The augmented reality headsets or devices used by shoppers may also enhance the shopping experience in other ways. For example, augmented reality prompts may be used to show shoppers information about products and recommendations for product usage. Data from the headsets may also serve as a form of crowd-sourcing of the stock data, and training images collected usually by auditors, as described in the aforementioned embodiments.