Image-based detection of planogram product spaces

This disclosure describes techniques for updating planogram data associated with a facility. The planogram may indicate inventory locations within the facility for various types of items supported by product fixtures. In particular an image of a product fixture is analyzed to identify image segments corresponding to product groups, where each product group consists of instances of the same product and each image segment corresponds to a group of image points. Image data is further analyzed to determine coordinates of the points of each image segment. A product space corresponding to the product group is then defined based on the coordinates of the points of the product group. In some cases, for example, a product space may be defined in terms of the coordinates of the corners of a rectangular bounding box or volume.

BACKGROUND

Retailers, wholesalers, and other product distributors often manage physical stores that utilize cashiers or dedicated self-checkout stands to finalize transactions with customers. During these traditional checkout processes, customers may have to carry and use physical objects for payment or identification, such a credit card or debit card, a driver’s license, a phone, and so forth. In the future, physical stores may utilize several types of sensors to allow users to acquire and pay for items without cashiers or dedicated self-checkout stands. For example, cameras and/or other sensors may be used to detect when a customer picks product items from shelves and to automatically charge an account of the customer when the customer leaves the store. In store such as this, as well as traditional physical stores, it may be desirable to have a planogram of the store indicating the locations of products within the store.

DETAILED DESCRIPTION

Described herein are techniques for creating and maintaining a planogram of a facility. A planogram is a data structure that indicates locations in a facility of product items that are held by different fixtures such as racks, shelves, hooks, counters, etc. Specifically, the planogram indicates the locations of multiple different products as well as which product will be found at any given location.

As an example, envision a facility that includes numerous aisles and fixtures for holding arrays of different product items for acquisition. Fixtures may be of various types, and may include, as examples, shelves, counters, hooks, cases, racks, kiosks, stands, easels, bins, table tops, and other types of supports and holders. Some fixtures, such as refrigerated fixtures, may have glass or other transparent doors. Typically, a fixture supports groups of products, where each group has multiple instances of a given product. As an example, multiple candy bars of a particular brand and flavor may be held by a hook. As another example, a shelf may have lanes that extend rearward, where each lane holds multiple cans of a particular brand and flavor of soft drink. In some cases, a number of contiguous locations, such as adjacent hooks or lanes, may contain instances of the same product. For example, three adjacent lanes of a shelf may hold a particular brand and type of milk.

In embodiments described herein, a planogram specifies coordinates of a product space that contains a group of instances of a single product. For example, such a product space might comprise an area of a countertop or shelf having multiple instances of a particular product. As another example, a product space might comprise a shelf lane that has multiple instances of a particular product. As another example, a product space might comprise a number of adjacent or otherwise contiguous shelf lanes that hold the same product. As yet another example, a product space might correspond to multiple instances of a particular product that are held on a hook or on a number of adjacent hooks.

In some embodiments, a planogram may specify a product space using three-dimensional (3D) coordinates relative to a facility within which the products are located. For example, 3D coordinates of one or more points corresponding to a product area or volume might be specified. In some embodiments, a product space may be specified as 3D coordinates of multiple points that define an area or volume that contain a product group. In some cases, a product space may correspond to an organizational unit of a fixture, such as a lane or set of adjacent lanes, even though one or more of the lanes may have empty space. For example, the product volume may extend to the rear of the shelf, even though product instances do not extend to the rear of the shelf.

In order to determine coordinates of product spaces within a store or other facility, cameras are positioned at multiple locations in the store so that each product group is within the field of view of at least one camera. Images of the entire store may be captured at any time and analyzed to identify product groups, to identify the products of the product groups, and to determine physical coordinates of product spaces corresponding to the product groups.

A process of identifying product spaces within a facility may include segmenting camera images to identify image segments corresponding to visible portions of respective product groups. For each identified image segment, corresponding image data may be analyzed to determine the physical coordinates of visible surfaces of the corresponding product group.

In some cases, coordinates of a product group represented by an image segment may be determined based on depth information provided by a 3D camera such as a time-of-flight camera. In these cases, 3D coordinates of any image point may be determined based on its X-Y coordinates within the image in combination with the corresponding depth, assuming knowledge of camera characteristics such as position, orientation, and focal length.

Physical coordinates of a product group may also be determined by comparing two or more image segments, corresponding to the same product group, that has been obtained from different cameras. For matching points of two such image segments, triangulation can be performed to determine three-dimensional coordinates of the surface point represented by the matching points, based on the X-Y coordinates of the points within the images and the known orientations, positions, and optical characteristics of the cameras.

Having identified physical coordinates of visible surfaces of a product group, further processing may be performed to estimate a corresponding product volume. In some embodiments, a front-facing or outward-facing portion of the visible surface of a product group may be identified and the product volume may be defined as extending rearward from that surface. For a product group on a shelf, for example, the corresponding product volume may be assumed to extend rearward from front-facing surfaces of the visible instances of the product group.

In some embodiments, store-wide analysis of camera images may be performed to identify planes formed by the visible portions of product instances held by different types of fixtures, and product volumes may be assumed to extend from those planes. For example, an analysis such as this may identify a vertical plane that is generally formed by the faces of product instances held by a row of shelf fixtures. For a product group in that row of shelf fixtures, the front of the corresponding product volume can be assumed to be at the identified plane and the product volume is assumed to extend orthogonally rearward from there. As another example, the store-wide analysis may identify a horizontal plane that is generally formed by the visible surfaces of product instances supported on a counter or other horizontal surface. For a product group on such a horizontal surface, the front of the corresponding product volume can be assumed to be at the identified horizontal plane and the product volume is assumed to extend orthogonally downward from the identified horizontal plane.

Camera images may be analyzed using object recognition techniques to obtain product identifications of the products held by each product space. Upon identifying a product space, planogram data is updated to indicate the product held by that product space and the three-dimensional coordinates of the product space. For example, planogram data may be updated to reflect the product space coordinates of a new location of a product and/or to reflect that a different product has been placed in a particular product space.

The techniques described herein allow product spaces to be identified and updated without human involvement. Based on this, a planogram can be created and updated to reflect the position, area, and/or volume occupied by any recognized product.

In some embodiments, a planogram may be updated using this process every morning before a store opens to obtain accurate planogram data for the entire store. The process may also be performed multiple times throughout a day, even while the store is open for business. In some cases, the process may be performed in response to detected events, such as when products are moved from one location to another or when store personnel indicate that a planogram update is needed.

Planogram data may be used for inventory, for directing customers to product locations, and/or for other management functions. In automated-checkout stores, where sensors are used to detect when customers pick product items from shelves, planogram data may be used to identify the product items that are being picked by a customer so that a virtual shopping cart of that customer may be updated to indicate the purchase of the product items.

Although certain techniques are described herein in the context of a retail store or other materials handling facility, the techniques are generally applicable to any other environment. Other examples may include inventory management systems, automating the intake of new shipments of item inventory, libraries for processing returned books and/or identifying books obtained by a user during a library session, video-rental facilities for processing and sorting returned movies, and so forth.

The facility described herein may include, but is not limited to, warehouses, distribution centers, cross-docking facilities, order fulfillment facilities, packaging facilities, shipping facilities, rental facilities, libraries, retail stores, wholesale stores, museums, or other facilities or combinations of facilities for performing one or more functions of materials (inventory) handling. In other implementations, the techniques described herein may be implemented in other facilities or situations.

FIG.1illustrates an example environment100in which the described techniques may be performed. The environment100might be part of a physical store, for example, that has multiple fixtures holding product instances to be picked up and taken by a customer or other person. In some embodiments, the environment100may be part of an automated checkout store. The described techniques may also be used in other situations and facilities.

FIG.1shows examples of different types of store fixtures that may be used within a facility to hold product instances. In this example, the fixtures are positioned end-to-end, in an arrangement that might be found in a grocery store, for example, where the fixtures might be lined up along one side of an aisle or along a wall.

The fixtures include several shelf fixtures102having multiple shelves104upon which products may be placed. The shelves are used to support product instances, several of which have been designated inFIG.1by the reference numeral106. In this example, the product instances106are arranged in single file lanes that extend from the front of the shelves104to the rear of the shelves104. Each lane is defined by a depthwise column of product instances.

The fixtures also include a shelf fixture108in which product groups extend sideways on shelves, rather than extending rearward. In some cases, product instances may also be stacked vertically on a shelf. In this example, the stack fixture108supports product packages110such as boxes or cases of soda, as examples.

The fixtures further include a hook fixture112, which is positioned as an endcap relative to the shelf fixtures102and108. The hook fixture112has an array of multiple hooks or rods114upon which product instances116may be hung. Each hook114may support a number of product instances116. Each hook114typically contains instances of a single product.

The fixtures further include a table fixture118having an upper surface or counter upon which stacks120of product instances may be placed. In this example, each stack120has instances of a single product.

FIG.1shows a first camera122(a) and a second camera122(b), which are representative of multiple cameras (referred to collectively as cameras122) are positioned throughout the store and oriented to capture and provide images of the various fixtures and the product instances supported or held by the fixtures. For example, the facility may include overhead cameras, in-shelf cameras, or other cameras. In some embodiments, it may be desirable for the cameras122to be above the fixtures so that tops of the product instances are visible to the cameras. Cameras may also be positioned at lower heights for better views of the front horizontal surfaces of product instances. In some embodiments, cameras might be distributed throughout the facility so that every fixture is within the field of view of at least one of the cameras122. In some embodiments, each product instance may be within the field of view of at least two of the cameras122so that triangulation can be used to determine physical coordinates of surfaces that are visible to the cameras.

In the embodiment ofFIG.1, the cameras122are configured to capture still images of the fixtures and to provide the still images to one or more computer systems for processing. The computer systems may use the images for performing tasks related to inventory, checkout, payroll, time scheduling, and/or other aspects of store management. In addition, the computer systems may use the images to create and maintain a planogram that indicates the locations of different products and/or product spaces within the store.

A planogram associated with a store may associate a particular product with a location at which instances of the product are located. The location may be specified and referred to as a product space or item space, which may correspond to an area or volume within which a group of products or other items having the same product or item identification are contained. A product space may be specified as a set of three-dimensional (3D) coordinates. As described above, instances within a product space may be supported by any of various types of fixtures, including shelves, counters, hooks, cases, racks, kiosks, stands, easels, bins, table tops, and other types of supports and holders. A product space might correspond to a shelf lane, multiple adjacent shelf lanes, a horizontal row of product instances, a vertical stack of product instances, multiple adjacent vertical stacks of product instances, a line of product instances supported by a hook, multiple lines of product instances supported by adj acent hooks, etc.

In an automated checkout environment, a planogram may be used to identify products that a customer is picking up. In other environments, planograms may be used as a reference for store personnel and/or customers. For example, store personnel may use a planogram when restocking, in order to find locations for particular products. Similarly, customers may be given access to a map or other information based on a planogram to guide the customers to desired products. As another example, a shopping service may use a planogram as a guide or map when picking products on behalf of a customer. In some cases, a planogram may be used to calculate an optimum sequence or route for picking a list of items. In some environments, smartphone applications or other computer applications may be provided to shoppers and store personnel to assist in finding specified products based on a planogram.

Although a particular fixture and camera configuration is illustrated inFIG.1, in practice the described techniques may be implemented in environments having various layouts, which may include different types of fixtures and supports.

Note that as used herein, the term “product” corresponds to a product identity, such as might be defined by a brand/model combination and/or a unique product identifier such as a UPC (Universal Product Code). The terms “product instance,” “item,” and “product item” are used when referring to a single article of a product.

FIG.2illustrates an example process200for identifying the locations of products and/or product spaces within a facility in which cameras are positioned to capture images of product instances that are supported by one or more fixtures. The process200may be performed by computer systems and/or computer devices associated with a business or facility.FIG.2will be described in conjunction withFIGS.3-5, which are used to illustrate certain of the actions of shown byFIG.2.

An action202comprises receiving image data from one or more cameras within the facility. The image data represents images of the product instances captured by the cameras. The image data may be received from one or more cameras that are positioned throughout the facility, such as one of the cameras122ofFIG.1. Each camera image may be of a fixture or a portion of a fixture having multiple product groups, each of which may have one or more instances of the same product.

FIG.3is an example of a camera image302that might be received in the action202. The image302is of a fixture and its product instances. More specifically, in this example the camera image302is of several shelf fixtures, each having multiple shelves that support multiple product instances. Note, however, that the described techniques may be performed to determine product spaces for various different types of fixtures, not limited to shelf fixtures.

An action204comprises analyzing the image data to detect product groups represented by the camera image302, wherein a product group comprises a contiguous group of product instances that have a common product identification, such as a common UPC. Product instances within a shelf lane are an example of a product group. A product group may also comprise product instances of the same product that are in adjacent shelf lanes. As a more specific example, a product group may be one or more contiguously adjacent cans of a particular brand and flavor of soft drink, regardless of the type of fixture they are on. A different product group may comprise one or more contiguously adjacent boxes or cases of a particular brand and flavor of soft drink.

In some embodiments, the action204may comprise segmenting the camera image302to detect segments of the image302that corresponds to respective product groups. Each segment comprises a group of image points that together represent a group of product instances of a single product. A group of image points such as this may be referred to herein as a product mask, item mask, or point group.

FIG.4shows a result of segmentation as might be achieved in an example environment.FIG.4shows an image portion402that contains multiple product instances arranged in lanes as already described. Within the image portion402, a segment404has been identified. The segment404is illustrated as a cross-hatched mask corresponding to a group of product instances. The image segment404comprises the image points underlying the illustrated mask. In the described example, the image segment404includes all contiguous instances of a single product. Although only a single image segment404is illustrated inFIG.4, in practice each group of product instances is represented as a distinct segment.

FIG.5illustrates segments502that have been identified in the image302ofFIG.3. InFIG.5, each segment502corresponds to a group of product instances and is illustrated as a separately hatched region. For purposes of illustration, only four of the segments502are labeled inFIG.5.

The action204may use a trained classifier to identify image segments corresponding to product instances and/or to groups of product instances. The classifier can be trained using supervised learning, based on training images that have been manually annotated to show image segments corresponding to product instances or groups. The classifier, as well as each classifier described throughout, may comprise a convolutional neural network (CNN), a support vector machine (SVM), or any other type of computer-vision-based classifier. Some implementations may use the Mask R-CNN (Regional Convolutional Neural Network) framework for object instance segmentation.

Returning again toFIG.2, an action206comprises further analyzing the image data to determine product identifications of the product instances of the product groups, as represented by the corresponding image segments. In some embodiments, this may comprise performing image recognition to obtain identifications of the products represented by the image segments of the camera image302. Specifically, for each identified image segment, image recognition is performed to identify the product instances shown by the image segment. Product identification may utilize various types of object recognition and identification, such as techniques that compare detected features of a segment to the known features of different products. In some cases, classifiers may be trained on manually annotated images to identify different items.

An action208comprises further analyzing the image data to determine locations of the product groups. More specifically, the action208may comprise determining coordinates of a product space corresponding to each product group, where a product space may comprise an area or volume containing the instances of the product group. The coordinates of a product space may be specified as three-dimensional coordinates of the facility within which the products are located. For example, a product space may comprise a cube or rectangular volume specified by coordinates of its corners relative to the facility.

In some embodiments, physical coordinates of a product group may be determined by comparing two different camera images, taken from different viewpoints, each of which includes a representation of the product group. In particular, two such images may be analyzed to identify a product group that is shown in both image, and triangulation can be used to determine coordinates of visible points of the product group based on image coordinates of the points in the two images and on known positions, orientations, and other characteristics of the cameras that produced the images.

In other embodiments, physical coordinates of points represented by an image or image segment may be determined based on depth information included in the 3D camera images, such as might be produced by a time-of-flight, stereoscopic, RGB-D, predictive depth techniques, or other 3D cameras and techniques.

In some embodiments, identifying a product space corresponding to a product group may comprise identifying image points that are on front-facing surfaces of the instances of the product group, and using those points to identify a front-facing rectangle or other area of the product space. The front-facing is then be extended rearward, using an assumed depth, to define a product volume.

An action210may comprise storing, in one or more datastores, planogram data indicating product locations within a facility. The action210may be performed with respect to a previously stored planogram or a newly created planogram. For example, the action210may comprise storing planogram data in a network-accessible database that is being used for the planogram. In particular, the action210may comprise updating the planogram with the product identifications and location coordinates determined by the previous actions ofFIG.2. In some embodiments, each product location may be defined as coordinates of a product space or product volume as described above, in three-dimensional space.

The process200may be performed periodically to update a facility planogram. For example, the process200may be performed at least once a day, such as every morning after nighttime stocking activities and before store opening. Alternatively, the process200may be performed when stocking activity has been detected and/or when shelves and/or items are rearranged within the facility. In some cases, the process200may be performed even during times when items are being accessed by customers or other persons. For example, the process200may be performed multiple times a day, during times that a store is open.

In some cases, additional data may be available and used for determining group locations. For example, facility personnel may at times perform manual barcode scanning of items as they are being stocked, and this information may be used to qualify or augment the process200. As another example, a facility blueprint may be available and may be used to determine fixture locations.

FIG.6illustrates another example process600that may be used in certain embodiments and environments for identifying the locations of products and/or product lanes. The process600may be performed by computer systems and/or computer devices associated with a facility or business within which the products are located.

An action602comprises receiving first and second image data representing product items or other items supported by one or more fixtures within the facility. The first image data represents a first camera image of multiple product items or other items that are stored on shelves of one or more fixtures. The second image data represents a second camera image of the multiple product items from a different camera viewpoint. The two camera images are used in subsequent actions determining positions of product groups using triangulation. Note that althoughFIG.6references first and second cameras, any number of one or more cameras and corresponding camera images may be analyzed, using the same procedures, to identify the locations of products and/or product spaces.

FIG.7shows an environment700in which two cameras702(a) and702(b) are positioned provide respective camera images captured from different viewpoints, such as might be used in the example process600. The cameras702(a) and702(b) are positioned in different locations and in different orientations to view product instances supported by product fixtures704, which includes a center fixture704(a), a left fixture704(b), and a right fixture704(c). The fixtures704have multiple horizontal shelves706that hold multiple product instances, several of which have been designated inFIG.1by the reference numeral708. In this example, the product instances708are arranged in single file lanes710that extend from the front of the shelves706to the rear of the shelves706. AlthoughFIG.1highlights three lanes710for purposes of illustration, the depicted product instances708can be seen to form multiple single file lanes that extend from front to back. Each lane is defined by a depthwise column of product instances.

In this example, each lane710contains multiple instances708of the same product. For example, a particular lane710might contain cheese wheels of a particular size, type, and brand. As another example, a lane710may contain cans of soda of a particular size, brand, and flavor. Generally, the product instances in a lane will all have the same UPC.

The cameras702(a) and702(b) are oriented so that their fields of view encompass at least an overlapping portion of the fixtures704. In this example, it will be assumed that the cameras702(a) and702(b) are at respective first and second locations and that the center fixture704(a) is within the view of both of the cameras702(a) and702(b). In some embodiments, it may be desirable for the cameras702(a) and702(b) to be above the fixtures704so that tops of the product instances708are visible to the cameras. In some situations, the cameras702(a) and702(b) may be at the same height. In other embodiments, the cameras702(a) and702(b) may be at different heights. In some cases, the cameras may be oriented in different directions, as in the example ofFIG.7. Further, whileFIG.7illustrates an example of two cameras having respective fields of view that at least partly overlap, in other instances the techniques may apply to any other number of cameras having partially-overlapping fields of view.

Although a particular fixture and camera configuration is illustrated inFIG.7, in practice the process600may be implemented in environments having various layouts, which may include different types of fixtures and different kinds of product holders, such as the example fixtures shown inFIG.1. For example, fixtures may have hooks or rods instead of shelves, and the hooks or rods may be arranged irregularly rather than in rows or columns. In some cases, products may lie or be stacked on a shelf, table, or countertop. Generally, fixtures may include shelves, counters, hooks, cases, racks, kiosks, stands, easels, bins, table tops, and other types of supports and holders.

FIGS.8A and8Bshow examples of a first image802(a) and a second image802(b) that have been captured by the cameras702(a) and702(b) ofFIG.7. The images802(a) and802(b) are from the different viewpoints of the two cameras. The images802(a) and802(b) include an overlapping portion of the product fixtures704, which in this example include at least the center product fixture704(a).

Returning again toFIG.6, an action604comprises segmenting each of the first and second camera images802(a) and802(b) to detect segments of each image802(a) and802(b) that correspond to individual product instances or product groups. More specifically, image segmentation is performed to detect first image segments, of the first image802(a), that correspond to product instances or product groups. Image segmentation is performed to detect second image segments, of the second image802(b), that correspond to the same product instances or product groups. Each segment comprises a group of image points that together represent a product instance or product group. A group of image points such as this may be referred to herein as a product mask, item mask, lane mask, or point group.FIGS.4and5, discussed above, show example results of image segmentation such as this.

The action604may use a trained classifier to detect image segments corresponding to product instances and/or to lanes of product instances. The classifier can be trained using supervised learning, based on training images that have been manually annotated to show image segments corresponding to product instances or product lanes. The classifier, may comprise a convolutional neural network (CNN), a support vector machine (SVM), or any other type of computer-vision-based classifier. Some implementations may use the Mask R-CNN (Regional Convolutional Neural Network) framework for object instance segmentation.

An action606comprises performing image recognition to obtain identifications of the products represented by the image segments of the first camera image802(a) and the second camera image802(b). Specifically, for each identified image segment, image recognition is performed to identify the product represented by the image segment. Product identification may utilize various types of object recognition and identification, many of which compare detected features of a segment to the known features of different products. In some cases, classifiers may be trained on manually annotated images to identify different items.

An action608comprises comparing segments of the first and second camera images802(a) and802(b) to find correspondences, also referred to herein as mappings, between the segments of the first and second camera images802(a) and802(b). Each mapping associates one of the segments of the first camera image802(a) with a corresponding one of the segments of the second camera image802(b), wherein corresponding segments represent the same product instance or group of product instances.

In some embodiments, the action608may be performed based on the identifications of the products as determined in the action606. More specifically, the action608may comprise, for an image segment of the first camera image802(a) that represents a particular product, identifying a segment of the second camera image802(b) that represents the same product.

In other embodiments, the action608may comprise analyzing and/or comparing the first and second camera images802(a) and802(b) with each other to find the mappings. That is, the action608may include evaluating point similarities between the first camera image802(a) and the second camera image802(b). More specifically, the action608may comprise, for each point group of the first camera image802(a), finding a similar point group of the second camera image802(b), where a point group comprises the points represented by an image segment. A point group mapping associates a segment and corresponding point group of the first camera image802(a) with a respective segment and corresponding point group of the second camera image802(b).

In some embodiments, a homography may be calculated to translate between the coordinate systems of the first and second camera images802(a) and802(b), based on the mappings of the action608, and then used to update the mappings. A technique such as this will be described below with reference toFIG.9.

An action610comprises triangulating between corresponding segments, points, or point groups of the first and second camera images802(a) and802(b) to determine one or more coordinates of the product or product group represented by each segment or point group. The action610may comprise determining geographic point coordinates for all matching points of each pair of matching image segments. For matching points of a pair of camera images, triangulation is based the X-Y coordinates of the points in the two images and on known positions, orientations, and lens characteristics of the cameras.

An action612may comprise determining the coordinates of a product space, based on the point coordinates determined in the action610. The action612may comprise first identifying a front surface of one of the product instances represented by a segment in either the first or second camera image. In some embodiments, this may be performed by first identifying points of an image segment that are shown by both of the first and second camera images802(a) and802(b). These points are further analyzed to identify the coordinates of the front surface of a product item represented by the image segment. In some embodiments, this may be performed by projecting the intersecting points into two dimensions in the X-Y plane (i.e., top-down view). In a top-down view, a vertically aligned product surface will be shown as a straight or curved line, defined by a relatively high concentration of points. These points are identified by applying the Hough Transform to the two-dimensional projection. A vertically aligned, two-dimensional bounding box is then constructed around the identified surface points and the box is extended rearward with an assumed depth to generate the three-dimensional coordinates of an overall product volume.

In some embodiments, the action612may be performed by first conducting a facility-wide analysis, using cameras through the facility, and building a 3D point cloud of surface points visible to the cameras. The 3D point cloud is then analyzed to detect planar product arrangements. For example, the 3D point cloud may be analyzed to detect vertical planes such as might be formed by the front surfaces of products supported by a row of shelf fixtures. Product spaces along a vertical plane such like this are then modeled using horizontal bounding boxes whose front faces are aligned with the vertical plane. The same process can be repeated for horizontal and slanted planes as well as curved surfaces formed by some product layouts.

In order to identify vertical planes formed by vertical arrangements of products, such as presented by a shelf fixture, the 3D point cloud of the entire facility may be transformed into two dimensions in a horizontal (i.e., X-Y) plane, creating a top-down view of the point cloud. In the top-down view, a vertical surface will appear as concentrations of point that form lines. That is, points will be concentrated around lines that correspond to the front faces of vertical product arrangements. Horizontal planes corresponding to horizontal product arrangements may be detected using the same process, except that the 3D point cloud is transformed into two dimensions in a vertical plane. Slanted arrangements may be detected by transforming into slanted two-dimensional planes. Curved surfaces may be detected by detecting curves formed in the two-dimensional planes by concentrations of points.

An action614may comprise storing, in one or more datastores, planogram data indicating product locations within a facility. The action614may be performed with respect to a previously stored planogram or a newly created planogram. Generally, the action614may comprise storing planogram data in one or more datastores to indicate the product locations within the facility. For example, the action614may comprise storing planogram data in a network-accessible database that is being used for the planogram. In particular, the action614may comprise updating the planogram with the product locations determined by the previous actions ofFIG.6. In some embodiments, each product location may be defined as a product area or volume as described above, in three-dimensional space.

The process600may be performed periodically to update a facility planogram. For example, the process600may be performed at least once a day, such as every morning after nighttime stocking activities and before store opening. Alternatively, the process600may be performed when stocking activity has been detected and/or when shelves and/or items are rearranged within the facility. In some cases, the process600may be performed even during times when items are being accessed by customers or other persons. For example, the process600may be performed multiple times a day, during times that a store is open.

FIG.9illustrates an example process900that may be used in some embodiments to match image segments and/or image points between two camera images of the same product instances, such as between the first and second camera images802(a) and802(b).

An action902comprises performing image stereo rectification of the first and second images802(a) and802(b). Image stereo rectification creates projections of the first and second images802(a) and802(b) on a common image plane. The projected images may be referred to as rectified images. In the rectified images, epipolar lines are horizontal. In addition, both of the rectified images have the same vertical scale so that any part of one rectified image will be at the same or approximately the same image height in the other rectified image.

FIGS.10A and10Bshow rectified images1002(a) and1002(b) that might be produced in the action902based on the camera images802(a) and802(b), respectively. In these examples, the original camera images802(a) and802(b) have been warped so that the shelves of the racks are along epipolar lines and therefore appear horizontal in the transformed images. In addition, any point or item of the first rectified image1002(a) is at the same height as the same point or item of the second rectified image1002(b).

More specifically, image stereo rectification may be performed by finding a linear transformation that is subject to the following constraints (a) epipolar lines are parallel to the horizontal axis of the rectified images and (b) corresponding points of the rectified images have the same vertical coordinates. These constraints are satisfied using a linear transform that rotates, skews, and scales the images.

Either calibrated or non-calibrated image stereo rectification may be used to produce the rectified images1002(a) and1002(b) based on the camera images802(a) and802(b). Calibrated rectification is based on known characteristics of the cameras and on known geometric relationships between the cameras. Non-calibrated rectification may be performed based on point correspondences between the two camera images802(a) and802(b). In some cases, non-calibrated calibration may additionally rely on the anticipated presence of shelves and/or other image lines that can be assumed to be horizontal.

Previously identified image segments are also transformed into the coordinate systems of the rectified images, so that the image segments are defined relative to the rectified images.

Returning toFIG.9, an action904comprises comparing the segments of the first and second rectified images1002(a) and1002(b) to find correspondences, also referred to herein as mappings, between the segments of the first and second rectified images. Each mapping associates one of the segments of the first rectified image1002(a) with a corresponding one of the segments of the second rectified image1002(b), wherein corresponding segments are intended to represent the same product instance or lane of product instances.

In some embodiments, the action904may be performed based on previous identifications of products represented by the image segments. More specifically, the action904may comprise, for an image segment of the first rectified image1002(a) that represents a particular product, identifying a segment of the second rectified image1002(b) that represents the same product.

In other embodiments, the action904may comprise analyzing and/or comparing the first and second rectified images1002(a) and1002(b) with each other to find the mappings. That is, the action904may include evaluating point similarities between the first rectified image1002(a) and the second rectified image1002(b). More specifically, the action904may comprise, for each point group of the first rectified image1002(a), finding a similar point group of the second rectified image1002(b), where a point group comprises the points of an image segment. A point group mapping associates a segment and corresponding point group of the first rectified image1002(a) with a respective segment and corresponding point group of the second rectified image1002(b).

Because of the previously performed image rectification, corresponding points, point groups, and image segments will be at approximately the same heights in the first and second rectified images1002(a) and1002(b). Specifically, searching is constrained to points, point groups, or image segments that are at approximately the same height (e.g., along a common epipolar line) in each of the first and second rectified images1002(a) and1002(b). The search for matching segments between images is constrained and simplified by this characteristic of the two rectified images.

Any search for matching points, point groups, and/or image segments may also be constrained by the horizontal ordering of the image segments or point groups in each of the first and second rectified images1002(a) and1002(b). Specifically, it can be assumed that product instances that are along any horizontal, epipolar line will appear in the same horizontal order in the two images. Thus, given a first row of image segments having a horizontal ordering in the first rectified image1002(a), the analyzing of the action904is constrained to mappings that recognize the same horizontal ordering of image segments in the second rectified image902(b).

In some embodiments, the action904may be performed using dynamic programming or other recursion techniques, which are constrained by the horizontal ordering of the point groups in the first and second rectified images1002(a) and1002(b). Furthermore, in some implementations, the mappings found in the action904may be referred to as preliminary mappings, because they may be updated in subsequent actions.

An action906comprises calculating a homography between the first and second rectified images1002(a) and1002(b). A homography may be calculated based on matching points or point groups of the two images, which in this example are identified by the previously determined mappings between point groups of the first and second rectified images1002(a) and1002(b).

The homography is an equation or matrix that maps between points of the first and second images rectified images1002(a) and1002(b). For any given point of the first rectified image1002(a), the homography can be applied to find a point of the second rectified image1002(b) that corresponds in position to the given point of the first rectified image1002(a).

An action908comprises updating or revising the previously determined mappings, based on the homography. For each segment or point group of the first rectified image1002(a), the homography is applied to determine the segment or point group of the second rectified image1002(b) that corresponds in position to the segment or point group of the first rectified image.

In an action910, the mappings are transformed back to the coordinate systems of the original camera images802(a) and802(b). Generally, either or both of the actions906and908may be performed either in the coordinate systems of the original camera images802(a) and802(b) or in the coordinate systems of the rectified camera images1002(a) and1002(b). For example, prior to the action906the preliminary mappings of the action904may be transformed back to the coordinate systems of the original camera images802(a) and802(b), the homography may be calculated between the original camera images802(a) and802(b), and reevaluating the mappings in the action614may be based on the this homography.

FIGS.11and12represent an illustrative material handing environment1102in which the described techniques may be used. However, the following description is merely one illustrative example of an industry and environment in which the techniques described herein may be utilized.

The materials handling facility1102(or “facility”) comprises one or more physical structures or areas within which one or more items1104(1),1104(2), ...,1104(Q) (generally denoted as1104) may be held. As used in this disclosure, letters in parenthesis such as “(Q)” indicate an integer result. The items1104comprise physical goods, such as books, pharmaceuticals, repair parts, electronic gear, groceries, and so forth.

The facility1102may include one or more areas designated for different functions with regard to inventory handling. In this illustration, the facility1102includes a receiving area1106, a storage area1108, and a transition area1110. The receiving area1106may be configured to accept items1104, such as from suppliers, for intake into the facility1102. For example, the receiving area1106may include a loading dock at which trucks or other freight conveyances unload the items1104.

The storage area1108is configured to store the items1104. The storage area1108may be arranged in various physical configurations. In one implementation, the storage area1108may include one or more aisles1112. An aisle1112may be configured with, or defined by, inventory locations1114on one or both sides of the aisle1112. The inventory locations1114may include one or more of shelves, racks, cases, cabinets, bins, floor locations, or other suitable storage mechanisms for holding or storing the items1104. The inventory locations1114may be affixed to the floor or another portion of the facility’s structure, or may be movable such that the arrangements of aisles1112may be reconfigurable. In some implementations, the inventory locations1114may be configured to move independently of an outside operator. For example, the inventory locations1114may comprise a rack with a power source and a motor, operable by a computing device to allow the rack to move from one location within the facility1102to another.

One or more users1116(1),1116(2), ...,1116(U), totes1118(1),1118(2), ...,1118(T) (generally denoted as1116and1118, respectively) or other material handling apparatus may move within the facility1102. For example, the users1116may move about within the facility1102to pick or place the items1104in various inventory locations1114, placing them on totes1118for ease of transport. An individual tote1118is configured to carry or otherwise transport one or more items1104. For example, a tote1118may include a basket, a cart, a bag, and so forth.

In other implementations, other agencies such as robots, forklifts, cranes, aerial drones, and so forth, may move about the facility1102picking, placing, or otherwise moving the items1104.

One or more sensors1120may be configured to acquire information in the facility1102. The sensors1120in the facility1102may include sensors fixed in the environment (e.g., ceiling-mounted cameras) or otherwise, such as sensors in the possession of users (e.g., mobile phones, tablets, etc.). The sensors1120may include, but are not limited to, cameras1120(1), weight sensors, radio frequency (RF) receivers, temperature sensors, humidity sensors, vibration sensors, and so forth. The sensors1120may be stationary or mobile, relative to the facility1102. For example, the inventory locations1114may contain cameras1120(1) configured to acquire images of pick or placement of items1104on shelves, of the users1116(1) and916(2) in the facility1102, and so forth. In another example, the floor of the facility1102may include weight sensors configured to determine a weight of the users1116or other object thereupon.

During operation of the facility1102, the sensors1120may be configured to provide information suitable for tracking how objects move or other occurrences within the facility1102. For example, a series of images acquired by a camera1120(1) may indicate removal of an item1104from a particular inventory location1114by one of the users1116and placement of the item1104on or at least partially within one of the totes1118. Images may also be analyzed as described above to determine locations of products within the facility1102and to update a facility planogram to indicate the locations.

While the storage area1108is depicted as having one or more aisles1112, inventory locations1114storing the items1104, sensors1120, and so forth, it is understood that the receiving area1106, the transition area1110, or other areas of the facility1102may be similarly equipped. Furthermore, the arrangement of the various areas within the facility1102is depicted functionally rather than schematically. For example, multiple different receiving areas1106, storage areas1108, and transition areas1110may be interspersed rather than segregated in the facility1102.

The facility1102may include, or be coupled to, an inventory management system1122. The inventory management system1122may maintain a virtual cart of each user1116within the facility1102. The inventory management system1122may also store an identifier corresponding to an account of each user1116, the location of each of these identifiers, and whether the user1116is eligible to exit the facility1102with one or more items1104without performing a manual checkout of the items1104. The inventory management system1122may also generate and output notification data to the users1116, indicating whether or not they are so eligible. It is to be appreciated that the system may locate the identifier within the facility1102, but that this identifier may be free from information of an identity of a user. That is, the system may locate identifiers associated with accounts, rather than locate identified users within the facility.

As illustrated, the inventory management system1122may reside at the facility1102(e.g., as part of on-premises servers), on the servers1132that are remote from the facility1102, a combination thereof. In each instance, the inventory management system1122is configured to identify interactions and events with and between users1116, devices such as sensors1120, robots, material handling equipment, computing devices, and so forth, in one or more of the receiving area1106, the storage area1108, or the transition area1110. As described above, some interactions may further indicate the existence of one or more events1124—or predefined activities of interest. For example, the events1124may include the entry of the user1116to the facility1102, stocking of items1104at an inventory location1114, picking of an item1104from an inventory location1114, returning of an item1104to an inventory location1114, placement of an item1104within a tote1118, movement of users1116relative to one another, gestures by the users1116, and so forth. Other events1124involving users1116may include the user1116providing authentication information in the facility1102, using a computing device at the facility1102to authenticate identity to the inventory management system1122, and so forth. Some events1124may involve one or more other objects within the facility1102. For example, the event1124may comprise movement within the facility1102of an inventory location1114, such as a counter mounted on wheels. Events1124may involve one or more of the sensors1120. For example, a change in operation of a sensor1120, such as a sensor failure, change in alignment, and so forth, may be designated as an event1124. Continuing the example, movement of a camera1120(1) resulting in a change in the orientation of the field of view1128(such as resulting from someone or something bumping the camera1120(1)) may be designated as an event1124.

As described herein, the inventory management system1122may also analyze images captured within the facility1102to determine locations of products within the facility1102. In some cases, this analysis may be performed in response to detected changes within the facility, such as inventory locations1114being moved and/or items1104being moved.

By determining the occurrence of one or more of the events1124, the inventory management system1122may generate output data1126. The output data1126comprises information about the event1124. For example, where the event1124comprises an item1104being removed from an inventory location1114, the output data1126may comprise an item identifier indicative of the particular item1104that was removed from the inventory location1114and a user identifier of a user that removed the item. Output data may also include planogram data, such as coordinates of product volumes within the facility1102.

The inventory management system1122may use one or more automated systems to generate the output data1126. For example, an artificial neural network, one or more classifiers, or other automated machine learning techniques may be used to process the sensor data from the one or more sensors1120to generate output data1126. For example, the inventory management system may perform techniques for generating and utilizing a classifier for identifying user activity in image data. The automated systems may operate using probabilistic or non-probabilistic techniques. For example, the automated systems may use a Bayesian network. In another example, the automated systems may use support vector machines to generate the output data1126or the tentative results. The automated systems may generate confidence level data that provides information indicative of the accuracy or confidence that the output data1126or the tentative data corresponds to the physical world.

The confidence level data may be generated using a variety of techniques, based at least in part on the type of automated system in use. For example, a probabilistic system using a Bayesian network may use a probability assigned to the output as the confidence level. Continuing the example, the Bayesian network may indicate that the probability that the item depicted in the image data corresponds to an item previously stored in memory is 95%. This probability may be used as the confidence level for that item as depicted in the image data.

In another example, output from non-probabilistic techniques such as support vector machines may have confidence levels based on a distance in a mathematical space within which the image data of the item and the images of previously stored items have been classified. The greater the distance in this space from a reference point such as the previously stored image to the image data acquired during the occurrence, the lower the confidence level.

In yet another example, the image data of an object such as an item1104, user1116, and so forth, may be compared with a set of previously stored images. Differences between the image data and the previously stored images may be assessed. For example, differences in shape, color, relative proportions between features in the images, and so forth. The differences may be expressed in terms of distance with a mathematical space. For example, the color of the object as depicted in the image data and the color of the object as depicted in the previously stored images may be represented as coordinates within a color space.

The confidence level may be determined based at least in part on these differences. For example, the user1116may pick an item1104(1) such as a perfume bottle that is generally cubical in shape from the inventory location1114. Other items1104at nearby inventory locations1114may be predominately spherical. Based on the difference in shape (cube vs. sphere) from the adjacent items, and the correspondence in shape with the previously stored image of the perfume bottle item1104(1) (cubical and cubical), the confidence level that the user1116has picked up the perfume bottle item1104(1) is high.

In some situations, the automated techniques may be unable to generate output data1126with a confidence level above a threshold result. For example, the automated techniques may be unable to distinguish which user1116in a crowd of users1116has picked up the item1104from the inventory location1114. In other situations, it may be desirable to provide human confirmation of the event1124or of the accuracy of the output data1126. For example, some items1104may be deemed age restricted such that they are to be handled only by users1116above a minimum age threshold.

In instances where human confirmation is desired, sensor data associated with an event1124may be processed to generate inquiry data. The inquiry data may include a subset of the sensor data associated with the event1124. The inquiry data may also include one or more of one or more tentative results as determined by the automated techniques, or supplemental data. The subset of the sensor data may be determined using information about the one or more sensors1120. For example, camera data such as the location of the camera1120(1) within the facility1102, the orientation of the camera1120(1), and a field of view1128of the camera1120(1) may be used to determine if a particular location within the facility1102is within the field of view1128. The subset of the sensor data may include images that may show the inventory location1114or that the item1104was stowed. The subset of the sensor data may also omit images from other cameras1120(1) that did not have that inventory location1114in the field of view1128. The field of view1128may comprise a portion of the scene in the facility1102that the sensor1120is able to generate sensor data about.

Continuing the example, the subset of the sensor data may comprise a video clip acquired by one or more cameras1120(1) having a field of view1128that includes the item1104. The tentative results may comprise the “best guess” as to which items1104may have been involved in the event1124. For example, the tentative results may comprise results determined by the automated system that have a confidence level above a minimum threshold.

The facility1102may be configured to receive different kinds of items1104from various suppliers and to store them until a customer orders or retrieves one or more of the items1104. Specifically, the items1104may be received from one or more suppliers, such as manufacturers, distributors, wholesalers, and so forth, at the receiving area1106. In various implementations, the items1104may include merchandise, commodities, perishables, or any suitable type of item1104, depending on the nature of the enterprise that operates the facility1102. The receiving of the items1104may comprise one or more events1124for which the inventory management system1122may generate output data1126.

Upon being received from a supplier at receiving area1106, the items1104may be prepared for storage. For example, items1104may be unpacked or otherwise rearranged. The inventory management system1122may include one or more software applications executing on a computer system to provide inventory management functions based on the events1124associated with the unpacking or rearrangement. These inventory management functions may include maintaining information indicative of the type, quantity, condition, cost, location, weight, or any other suitable parameters with respect to the items1104. The items1104may be stocked, managed, or dispensed in terms of countable, individual units or multiples, such as packages, cartons, crates, pallets, or other suitable aggregations. Alternatively, some items1104, such as bulk products, commodities, and so forth, may be stored in continuous or arbitrarily divisible amounts that may not be inherently organized into countable units. Such items1104may be managed in terms of measurable quantity such as units of length, area, volume, weight, time, duration, or other dimensional properties characterized by units of measurement. Generally speaking, a quantity of an item1104may refer to either a countable number of individual or aggregate units of an item1104or a measurable amount of an item1104, as appropriate.

After arriving through the receiving area1106, items1104may be stored within the storage area1108. In some implementations, like items1104may be stored or displayed together in the inventory locations1114such as in bins, on shelves, hanging from pegboards, and so forth. In this implementation, all items1104of a given kind are stored in one inventory location1114. In other implementations, like items1104may be stored in different inventory locations1114. For example, to optimize retrieval of certain items1104having frequent turnover within a large physical facility1102, those items1104may be stored in several different inventory locations1114to reduce congestion that might occur at a single inventory location1114. Storage of the items1104and their respective inventory locations1114may comprise one or more event1124.

When a customer order specifying one or more items1104is received, or as a user1116progresses through the facility1102, the corresponding items1104may be selected or “picked” from the inventory locations1114containing those items1104. In various implementations, item picking may range from manual to completely automated picking. For example, in one implementation, a user1116may have a list of items1104they desire and may progress through the facility1102picking items1104from inventory locations1114within the storage area1108, and placing those items1104into a tote1118. In other implementations, employees of the facility1102may pick items1104using written or electronic pick lists derived from customer orders. These picked items1104may be placed into the tote1118as the employee progresses through the facility1102. Picking may comprise one or more events1124, such as the user1116in moving to the inventory location1114, retrieval of the item1104from the inventory location1114, and so forth.

After items1104have been picked, they may be processed at a transition area1110. The transition area1110may be any designated area within the facility1102where items1104are transitioned from one location to another or from one entity to another. For example, the transition area1110may be a packing station within the facility1102. When the item1104arrives at the transition area1110, the items1104may be transitioned from the storage area1108to the packing station. The transitioning may comprise one or more events1124. Information about the transition may be maintained by the inventory management system1122using the output data1126associated with those events1124.

In another example, if the items1104are departing the facility1102a list of the items1104may be obtained and used by the inventory management system1122to transition responsibility for, or custody of, the items1104from the facility1102to another entity. For example, a carrier may accept the items1104for transport with that carrier accepting responsibility for the items1104indicated in the list. In another example, a customer may purchase or rent the items1104and remove the items1104from the facility1102. The purchase or rental may comprise one or more events1124.

The inventory management system1122may access or generate sensor data about the facility1102and the contents therein including the items1104, the users1116, the totes1118, and so forth. The sensor data may be acquired by one or more of the sensors1120, data provided by other systems, and so forth. For example, the sensors1120may include cameras1120(1) configured to acquire image data of scenes in the facility1102. The image data may comprise still images, video, or a combination thereof. The image data may be processed by the inventory management system1122to determine a location of the user1116, the tote1118, and so forth.

The inventory management system1122, or systems coupled thereto, may be configured to determine an account identifier corresponding to the user1116to distinguish the user1116from other users located in the environment based on these respective account identifiers. In some cases, for example, the inventory management system122may detect that a person is entering the facility and may assign a unique identifier to that person such that the identifier is located within the facility. This identifier may be associated to that person based on information provided by the person in some instances. Again, it is to be appreciated that this identifier may be generic and free from information outwardly identifying the person, and that this identifier may be located within the facility rather than information identifying the person.

In some instances, the inventory management system may group users within the facility into respective sessions. That is, the inventory management system1122may utilize the sensor data to determine groups of users that are effectively “together” (e.g., shopping together). In some instances, a particular session may include multiple users that entered the facility1102together and, potentially, that navigate the facility together. For example, when a family of two adults and two children enter the facility together, the inventory management system may associate each user with a particular session. Locating sessions in addition to individual users may help in determining the outcome of individual events, given that users within a session may not only individually pick or return or otherwise interact with items, but may also pass the items back and forth amongst each other. For instance, a child in the above example may pick the box of cereal before handing the box to her mother, who may place it in her tote1118. Noting the child and the mother as belonging to the same session may increase the chances of successfully adding the box of cereal to the virtual shopping cart of the mother.

By determining the occurrence of one or more events1124and the output data1126associated therewith, the inventory management system1122is able to provide one or more services to the users1116of the facility1102. By utilizing one or more human associates to process inquiry data and generate response data that may then be used to produce output data1126, overall accuracy of the system may be enhanced. The enhanced accuracy may improve the user experience of the one or more users1116of the facility1102. In some examples, the output data1126may be transmitted over a network1130to one or more servers1132.

FIG.12illustrates a block diagram of the one or more servers1132. The servers1132may be physically present at the facility1102, may be accessible by the network1130, or a combination of both. The servers1132do not require end-user knowledge of the physical location and configuration of the system that delivers the services. Common expressions associated with the servers1132may include “on-demand computing,” “software as a service (SaaS),” “cloud services,” “data centers,” and so forth. Services provided by the servers1132may be distributed across one or more physical or virtual devices.

The servers1132may include one or more hardware processors1202(processors) configured to execute one or more stored instructions. The processors1202may comprise one or more cores. The servers1132may include one or more input/output (I/O) interface(s)1204to allow the processor1202or other portions of the servers1132to communicate with other devices. The I/O interfaces1204may comprise Inter-Integrated Circuit (I2C), Serial Peripheral Interface bus (SPI), Universal Serial Bus (USB) as promulgated by the USB Implementers Forum, and so forth.

The servers1132may also include one or more communication interfaces1206. The communication interfaces1206are configured to provide communications between the servers1132and other devices, such as the sensors1120, the interface devices, routers, and so forth. The communication interfaces1206may include devices configured to couple to personal area networks (PANs), wired and wireless local area networks (LANs), wired and wireless wide area networks (WANs), and so forth. For example, the communication interfaces1206may include devices compatible with Ethernet, Wi-Fi™, and so forth. The servers1132may also include one or more busses or other internal communications hardware or software that allow for the transfer of data between the various modules and components of the servers1132.

The servers1132may also include a power supply1208. The power supply1208is configured to provide electrical power suitable for operating the components in the servers1132.

The servers1132may further include one or more memories1210. The memory1210comprises one or more computer-readable storage media (CRSM). The CRSM may be any one or more of an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium, a mechanical computer storage medium, and so forth. The memory1210provides storage of computer-readable instructions, data structures, program modules, and other data for the operation of the servers1132. A few example functional modules are shown stored in the memory1210, although the same functionality may alternatively be implemented in hardware, firmware, or as a system on a chip (SOC).

The memory1210may include at least one operating system (OS) component1212. The OS component1212is configured to manage hardware resource devices such as the I/O interfaces1204, the communication interfaces1206, and provide various services to applications or components executing on the processors1202. The OS component1212may implement a variant of the FreeBSD™ operating system as promulgated by the FreeBSD Project; other UNIX™ or UNIX-like variants; a variation of the Linux™ operating system as promulgated by Linus Torvalds; the Windows® Server operating system from Microsoft Corporation of Redmond, Washington, USA; and so forth.

One or more of the following components may also be stored in the memory1210. These components may be executed as foreground applications, background tasks, daemons, and so forth. A communication component1214may be configured to establish communications with one or more of the sensors1120, one or more of the devices used by associates, other servers1132, or other devices. The communications may be authenticated, encrypted, and so forth.

The memory1210may store an inventory management system1216. The inventory management system1216is configured to provide the inventory functions as described herein with regard to the inventory management system1122. For example, the inventory management system1216may track movement of items1104in the facility1102, generate user interface data, determine product locations/coordinates, update a planogram, and so forth.

The inventory management system1216may access information stored in one or more data stores1218in the memory1210. The data store1218may use a flat file, database, linked list, tree, executable code, script, or other data structure to store the information. In some implementations, the data store1218or a portion of the data store1218may be distributed across one or more other devices including other servers1132, network attached storage devices, and so forth.

The data store1218may include physical layout data1220. The physical layout data1220provides a mapping of physical locations within the physical layout of devices and objects such as the sensors1120, inventory locations1114, and so forth. The physical layout data1220may indicate the coordinates within the facility1102of an inventory location1114, sensors1120within view of that inventory location1114, and so forth. For example, the physical layout data1220may include camera data comprising one or more of a location within the facility1102of a camera1120(1), orientation of the camera1120(1), the operational status, and so forth. Continuing the example, the physical layout data1220may indicate the coordinates of the camera1120(1), pan and tilt information indicative of a direction that the field of view1128is oriented along, whether the camera1120(1) is operating or malfunctioning, and so forth. The physical-layout data1220may include planogram data indicating the physical coordinates or different product lanes, as described above, relative to the cameras and other devices.

In some implementations, the inventory management system1216may access the physical layout data1220to determine if a location associated with the event1124is within the field of view1128of one or more sensors1120. Continuing the example above, given the location within the facility1102of the event1124and the camera data, the inventory management system1216may determine the cameras1120(1) that may have generated images of the event1124.

The item data1222comprises information associated with the items1104. The information may include information indicative of one or more inventory locations1114at which one or more of the items1104are stored. In some implementation, planogram data may be included in the item data to indicate the locations of the inventory locations1114. The item data1222may also include order data, SKU or other product identifier, price, quantity on hand, weight, expiration date, images of the item1104, detail description information, ratings, ranking, and so forth. The inventory management system1216may store information associated with inventory management functions in the item data1222.

The data store1218may also include sensor data1224. The sensor data1224comprises information acquired from, or based on, the one or more sensors1120. For example, the sensor data1224may comprise 3D information about an object in the facility1102. As described above, the sensors1120may include a camera1120(1), which is configured to acquire one or more images. These images may be stored as the image data1226. The image data1226may comprise information descriptive of a plurality of picture elements or pixels. Non-image data1228may comprise information from other sensors1120, such as input from the microphones, weight sensors, item dispensers, and so forth.

User data1230may also be stored in the data store1218. The user data1230may include identity data, information indicative of a profile, purchase history, location data, images of the user1116, demographic data, and so forth. Individual users1116or groups of users1116may selectively provide user data1230for use by the inventory management system1122. The individual users1116or groups of users1116may also authorize collection of the user data1230during use of the facility1102or access to user data1230obtained from other systems. For example, the user1116may opt-in to collection of the user data1230to receive enhanced services while using the facility1102.

In some implementations, the user data1230may include information designating a user1116for special handling. For example, the user data1230may indicate that a particular user1116has been associated with an increased number of errors with respect to output data1126. The inventory management system1216may be configured to use this information to apply additional scrutiny to the events1124associated with this user1116. For example, events1124that include an item1104having a cost or result above the threshold amount may be provided to the associates for processing regardless of the determined level of confidence in the output data1126as generated by the automated system.

The inventory management system1216may include one or more of a location component1232, identification component1234, event-determination component1236, inquiry component1238, and a planogram component1205, amongst other components1256. The inventory management system1216may include a planogram component1205that is responsible for determining product volumes and for updating planogram data.

The location component1232functions to locate items or users within the environment of the facility to allow the inventory management system1216to assign certain events to the correct users. The location component1232may assign unique identifiers to users as they enter the facility and, with the users’ consent, may locate the users throughout the facility1102over the time they remain in the facility1102. The location component1232may perform this locating using sensor data1224, such as the image data1226. For example, the location component1232may receive the image data1226and may use recognition techniques to identify users from the images. After identifying a particular user within the facility, the location component1232may then locate the user within the images as the user moves throughout the facility1102. Further, should the location component1232temporarily “lose” a particular user, the location component1232may again attempt to identify the users within the facility based on biometric information, such as voice recognition, or the like.

Therefore, upon receiving the indication of the time and location of the event in question, the location component1232may query the data store1218to determine which one or more users were at or within a threshold distance of the location of the event at the particular time of the event. Further, the location component1232may assign different confidence levels to different users, with the confidence levels indicating how likely it is that each corresponding user is the user that is in fact associated with the event of interest.

The location component1232may access the sensor data1224in order to determine this location data of the user and/or items. The location data provides information indicative of a location of an object, such as the item1104, the user1116, the tote1118, and so forth. The location data may include planogram data. A specified location may be absolute with respect to the facility1102or relative to another object or point of reference. Absolute terms may comprise a latitude, longitude, and altitude with respect to a geodetic reference point. Relative terms may include a location of 25.4 meters (m) along an x-axis and 75.2 m along a y-axis as designated by a floor plan of the facility1102, 5.2 m from an inventory location1114along a heading of 169°, and so forth. For example, the location data may indicate that the user1116(1) is 25.2 m along the aisle1112(1) and standing in front of the inventory location1114. In comparison, a relative location may indicate that the user1116(1) is 32 cm from the tote1118at a heading of 73° with respect to the tote1118. The location data may include orientation information, such as which direction the user1116is facing. The orientation may be determined by the relative direction the user’s916body is facing. In some implementations, the orientation may be relative to the interface device. Continuing the example, the location data may indicate that the user1116(1) is oriented with a heading of 0°, or looking north. In another example, the location data may indicate that the user1116is facing towards the interface device.

The identification component1234is configured to identify an object. In one implementation, the identification component1234may be configured to identify an item1104. In another implementation, the identification component1234may be configured to identify the user1116. For example, the identification component1234may use recognition techniques to process the image data1226and determine the identity data of the user1116depicted in the images by comparing the characteristics in the image data1226with previously stored results. The identification component1234may also access data from other sensors1120, such as from an RFID reader, an RF receiver, fingerprint sensors, and so forth.

The event-determination component1236is configured to process the sensor data1224and generate output data1226. The event-determination component1236may access information stored in the data store1218including, but not limited to, event description data1242, confidence levels1244, or threshold values1246. The event-determination component1236may be configured to create and utilize event classifiers for identifying events (e.g., predefined activity) within image data, potentially without use of other sensor data acquired by other sensors in the environment.

The event description data1242comprises information indicative of one or more events1124. For example, the event description data1242may comprise predefined profiles that designate movement of an item1104from an inventory location1114with the event1124of “pick”. The event description data1242may be manually generated or automatically generated. The event description data1242may include data indicative of triggers associated with events occurring in the facility1102. An event may be determined as occurring upon detection of the trigger. For example, sensor data1224such as a change in weight from a weight sensor1120(6) at an inventory location1114may trigger detection of an event of an item1104being added or removed from the inventory location1114. In another example, the trigger may comprise an image of the user1116reaching a hand toward the inventory location1114. In yet another example, the trigger may comprise two or more users1116approaching to within a threshold distance of one another.

The event-determination component1236may process the sensor data1224using one or more techniques including, but not limited to, artificial neural networks, classifiers, decision trees, support vector machines, Bayesian networks, and so forth. For example, the event-determination component1236may use a decision tree to determine occurrence of the “pick” event1124based on sensor data1224. The event-determination component1236may further use the sensor data1224to determine one or more tentative results1248. The one or more tentative results1248comprise data associated with the event1124. For example, where the event1124comprises a disambiguation of users1116, the tentative results1248may comprise a list of possible user identities. In another example, where the event1124comprises a disambiguation between items1104, the tentative results1248may comprise a list of possible item identifiers. In some implementations, the tentative result1248may indicate the possible action. For example, the action may comprise the user1116picking, placing, moving an item1104, damaging an item1104, providing gestural input, and so forth.

In some implementations, the tentative results1248may be generated by other components. For example, the tentative results1248such as one or more possible identities or locations of the user1116involved in the event1124may be generated by the location component1232. In another example, the tentative results1248such as possible items1104that may have been involved in the event1124may be generated by the identification component1234.

The event-determination component1236may be configured to provide a confidence level1244associated with the determination of the tentative results1248. The confidence level1244provides indicia as to the expected level of accuracy of the tentative result1248. For example, a low confidence level1244may indicate that the tentative result1248has a low probability of corresponding to the actual circumstances of the event1124. In comparison, a high confidence level1244may indicate that the tentative result1248has a high probability of corresponding to the actual circumstances of the event1124.

In some implementations, the tentative results1248having confidence levels1244that exceed the threshold may be deemed to be sufficiently accurate and thus may be used as the output data1126. For example, the event-determination component1236may provide tentative results1248indicative of the three possible items1104(1),904(2), and904(3) corresponding to the “pick” event1124. The confidence levels1244associated with the possible items1104(1),904(2), and1104(3) may be 25%, 70%, 92%, respectively. Continuing the example, the threshold result1246may be set such that confidence level1244of 90% are deemed to be sufficiently accurate. As a result, the event-determination component1236may designate the “pick” event1124as involving item1104(3).

The inquiry component1238may be configured to use at least a portion of the sensor data1224associated with the event1124to generate inquiry data1250. In some implementations, the inquiry data1250may include one or more of the tentative results1248or supplemental data1252. The inquiry component1238may be configured to provide inquiry data1250to one or more devices associated with one or more human associates.

An associate user interface is presented on the respective devices of associates. The associate may generate response data1254by selecting a particular tentative result1248, entering new information, indicating that they are unable to answer the inquiry, and so forth.

The supplemental data1252comprises information associated with the event1124or that may be useful in interpreting the sensor data1224. For example, the supplemental data1252may comprise previously stored images of the items1104. In another example, the supplemental data1252may comprise one or more graphical overlays. For example, the graphical overlays may comprise graphical user interface elements such as overlays depicting indicia of an object of interest. These indicia may comprise highlights, bounding boxes, arrows, and so forth, that have been superimposed or placed atop the image data1126during presentation to an associate.

The inquiry component1238processes the response data1254provided by the one or more associates. The processing may include calculating one or more statistical results associated with the response data1254. For example, statistical results may include a count of the number of times associates selected a particular tentative result1248, determination of a percentage of the associates that selected a particular tentative result1248, and so forth.

The inquiry component1238is configured to generate the output data1126based at least in part on the response data1254. For example, given that a majority of the associates returned response data1254indicating that the item1104associated with the “pick” event1124is item1104(5), the output data1126may indicate that the item1104(5) was picked.

The inquiry component1238may be configured to selectively distribute inquiries to particular associates. For example, some associates may be better suited to answering particular types of inquiries. Performance data, such as statistical data about the performance of the associates, may be determined by the inquiry component1238from the response data1254provided by the associates. For example, information indicative of a percentage of different inquiries in which the particular associate selected response data1254that disagreed with the majority of associates may be maintained. In some implementations, test or practice inquiry data1250having a previously known correct answer may be provided to the associate for training or quality assurance purposes. The determination of the set of associates to use may be based at least in part on the performance data.

By usingthe inquiry component1238, the event-determination component1236may be able to provide high reliability output data1126that accurately represents the event1124. The output data1126generated by the inquiry component1238from the response data1254may also be used to further train the automated systems used by the inventory management system1216. For example, the sensor data1224and the output data1126, based on response data1254, may be provided to one or more of the components of the inventory management system1216for training in process improvement. Continuing the example, this information may be provided to an artificial neural network, Bayesian network, and so forth, to further train these systems such that the confidence level1244and the tentative results1248produced in the future for the same or similar input is improved. Finally, asFIG.12illustrates, the servers1132may store and/or utilize other data1258.

The planogram component1205, meanwhile, may perform some or all of the operations described above with reference to the processes200,600, and900ofFIGS.2,6, and9, respectively. For instance, the planogram component1205may be used to determine product volumes associated with different products within the facility1102. The planogram component1205may also be referenced by other components to provide product location information. In some embodiments, the functionality of the planogram component1205may be implemented by other components, such as by the location component and the identification component.