Patent Description:
In current delivery systems, shoppers, or "pickers," fulfill orders at a physical warehouse, such as a retailer, on behalf of customers as part of an online shopping concierge service. The delivery system provides customers with a user interface that displays an inventory catalog listing items that a customer can add to an order. In current delivery systems, the inventory catalog may be generated from information provided to the delivery system by the physical warehouses. However, the provided item information may not include various properties of an item, such as whether an item is organic or vegan, or any number of food certifications, gradings, etc. This information may be important to a consumer's selection of items to add to a delivery order, e.g., due to a customer's dietary restrictions, health concerns, religious practices, etc. If this information is not included in the inventory catalog, the consumer cannot make a fully informed decision on which items to add to the order. However, adding item properties to the inventory catalog can be difficult if they are not provided by the warehouse or available through other data sources. Food properties, such as organic certifications, are often indicated through common visual symbols, or item labels, which appear on item packaging. In conventional delivery systems, adding item properties to an inventory catalog requires manual input by a person visually observing the item who translates between the item label and an item property. <CIT> describes a system for building a product library without requiring a planogram including an image capture unit operated to provide images of items.

Various aspects and embodiments of the invention are set out in the appended claims.

As described herein, a delivery system can generate and use machine-learned models to update item property information in an item catalog or inventory database. The machine-learned models are trained using images of items in a warehouse and images of item labels. A first-level machine-learned model is trained using images of items in a warehouse to determine if input images of items include any of a variety of item labels, such as organic labels, vegan labels, etc. Second-level machine-learned models are trained using images of item labels to classify input images as indicating a particular item property. For example, one classifying machine-learned model can determine whether or not an item is organic, another classifying machine-learned model can determine whether or not an item vegan, etc. Pixels identified by the first machine-learned model as being associated with an item label may be input into the second machine-learned model to confirm the item property indicated by the identified label. The item entry in an item catalog may then be updated with the confirmed item property.

A method for populating an inventory catalog includes receiving an image showing an item having an entry in an inventory catalog. The image includes a plurality of pixels. The inventory catalog stores inventory information, including properties of items wherein each property is a food attribute used to confirm items. The method determines that a property of the item is missing from the inventory catalog.

The method retrieves a machine learned segmentation neural network, which is trained based on a set of i mages and associated properties, to determine location of pixels in an image that are associated with an item label on the item indicating a property. The method determines, using the machine learned segmentation neural network, a subset of pixels associated with the item label in the received image and identifies locations of the subset of pixels of the received image. The method extracts the subset of pixels from the received image. The method retrieves a machine learned classifier, which is trained based on a set of images of the item label, to determine whether an image shows the item label. The method determines, using the machine learned classifier, that the extracted subset of pixels shows the item label, wherein the item label indicates the property of the item that is missing from the inventory catalog. The method updates the entry for the item in the inventory catalog to indicate that the item has the property associated with the item label.

The figures depict embodiments of the present disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles, or benefits touted, of the disclosure described herein.

<FIG> illustrates an environment <NUM> of an online platform, according to one embodiment. The figures use like reference numerals to identify like elements. A letter after a reference numeral, such as "110a," indicates that the text refers specifically to the element having that particular reference numeral. A reference numeral in the text without a following letter, such as "<NUM>," refers to any or all of the elements in the figures bearing that reference numeral. For example, "<NUM>" in the text refers to reference numerals "110a" and/or "110b" in the figures.

The environment <NUM> includes an online concierge system <NUM>. The online concierge system <NUM> is configured to receive orders from one or more customers <NUM> (only one is shown for the sake of simplicity). An order specifies a list of goods (items or products) to be delivered to the customer <NUM>. The order also specifies the location to which the goods are to be delivered, and a time window during which the goods should be delivered. In some embodiments, the order specifies one or more retailers from which the selected items should be purchased. The customer <NUM> may use a customer mobile application (CMA) <NUM> to place the order; the CMA <NUM> is configured to communicate with the online concierge system <NUM>.

The online concierge system <NUM> is configured to transmit orders received from customers <NUM> to one or more pickers <NUM>. A picker <NUM> may be a contractor, employee, or other person (or entity) who is enabled to fulfill orders received by the online concierge system <NUM>. The picker <NUM> travels between a warehouse and a delivery location (e.g., the customer's home or office). A picker <NUM> may travel by car, truck, bicycle, scooter, foot, or other mode of transportation. In some embodiments, the delivery may be partially or fully automated, e.g., using a self-driving car. The environment <NUM> also includes three warehouses 110a, 110b, and 110c (only three are shown for the sake of simplicity; the environment could include hundreds of warehouses). The warehouses <NUM> may be physical retailers, such as grocery stores, discount stores, department stores, etc., or non-public warehouses storing items that can be collected and delivered to customers. Each picker <NUM> fulfills an order received from the online concierge system <NUM> at one or more warehouses <NUM>, delivers the order to the customer <NUM>, or performs both fulfillment and delivery. In one embodiment, pickers <NUM> make use of a picker mobile application (PMA) <NUM> which is configured to interact with the online concierge system <NUM>.

<FIG> is a diagram of an online concierge system <NUM>, according to one embodiment. The online concierge system <NUM> includes an inventory management engine <NUM>, which interacts with inventory systems associated with each warehouse <NUM>. In one embodiment, the inventory management engine <NUM> requests and receives inventory information maintained by the warehouse <NUM>. The inventory of each warehouse <NUM> is unique and may change over time. The inventory management engine <NUM> monitors changes in inventory for each participating warehouse <NUM>. The inventory management engine <NUM> is also configured to store inventory records in an inventory database <NUM>. The inventory database <NUM> may store information in separate records - one for each participating warehouse <NUM> - or may consolidate or combine inventory information into a unified record. Inventory information includes both qualitative and qualitative information about items, including size, color, weight, SKU, serial number, and so on. The inventory database <NUM> also stores information about various item properties, such as vegan, organic, gluten free, etc. The inventory database <NUM> also stores purchasing rules associated with each item, if they exist. For example, age-restricted items such as alcohol and tobacco are flagged accordingly in the inventory database <NUM>. The inventory management engine <NUM> may receive updated item information from a machine-learned segmentation neural network <NUM> and/or a machine-learned label classifier <NUM>, such as a property of an item in an inventory database <NUM>, and adds the updated item information to the inventory database <NUM>. The inventory management engine <NUM> receives updated item property information from the machine-learned segmentation neural network <NUM> and/or the machine-learned label classifier <NUM> in a process described with respect to the item property learning module <NUM> and <FIG>.

The online concierge system <NUM> includes an order fulfillment engine <NUM> which is configured to synthesize and display an ordering interface to each customer <NUM> (for example, via the CMA <NUM>). The order fulfillment engine <NUM> is also configured to access an inventory database <NUM> in order to determine which items are available at which warehouses <NUM>, and to identify properties associated with the items. The order fulfillment engine <NUM> determines a sale price for each item ordered by a customer <NUM>. Prices set by the order fulfillment engine <NUM> may or may not be identical to in-store prices determined by retailers (which is the price that customers <NUM> and pickers <NUM> would pay at retail warehouses). The order fulfillment engine <NUM> also facilitates transactions associated with each order. In one embodiment, the order fulfillment engine <NUM> charges a payment instrument associated with a customer <NUM> when he/she places an order. The order fulfillment engine <NUM> may transmit payment information to an external payment gateway or payment processor. The order fulfillment engine <NUM> stores payment and transactional information associated with each order in a transaction records database <NUM>.

In some embodiments, the order fulfillment engine <NUM> also shares order details with warehouses <NUM>. For example, after successful fulfillment of an order, the order fulfillment engine <NUM> may transmit a summary of the order to the appropriate warehouses <NUM>. The summary may indicate the items purchased, the total value of the items, and in some cases, an identity of the picker <NUM> and customer <NUM> associated with the transaction. In one embodiment, the order fulfillment engine <NUM> pushes transaction and/or order details asynchronously to retailer systems. This may be accomplished via use of webhooks, which enable programmatic or system-driven transmission of information between web applications. In another embodiment, retailer systems may be configured to periodically poll the order fulfillment engine <NUM>, which provides detail of all orders which have been processed since the last request.

The order fulfillment engine <NUM> may interact with a picker management engine <NUM>, which manages communication with and utilization of pickers <NUM>. In one embodiment, the picker management engine <NUM> receives a new order from the order fulfillment engine <NUM>. The picker management engine <NUM> identifies the appropriate warehouse to fulfill the order based on one or more parameters, such as a probability of item availability, the contents of the order, the inventory of the warehouses, and the proximity to the delivery location. The picker management engine <NUM> then identifies one or more appropriate pickers <NUM> to fulfill the order based on one or more parameters, such as the pickers' proximity to the appropriate warehouse <NUM> (and/or to the customer <NUM>), his/her familiarity level with that particular warehouse <NUM>, and so on. For example, the picker management engine <NUM> identifies pickers by comparing the parameters to data retrieved from a picker database <NUM>. The picker database <NUM> stores information describing each picker <NUM>, such as his/her name, gender, rating, previous shopping history, and so on.

As part of fulfilling an order, the order fulfillment engine <NUM> and/or picker management engine <NUM> may also access a customer database <NUM> which stores information describing each customer. This information could include each customer's name, address, gender, shopping preferences, favorite items, stored payment instruments, and so on.

The online concierge system <NUM> includes an item property learning module <NUM> for learning properties of items in the inventory database <NUM> based on images of items. The inventory management engine <NUM> is configured to populate the inventory records in the inventory database <NUM> based on information received from the item property learning module <NUM>. Properties that can be learned by the item property learning module <NUM> include whether or not an item is USDA Organic, International Organic, Soil Association Certified Organic, Certified Naturally Grown, non-GMO, GE-free, gluten-free, Hormone-free, RBGH-free, antibiotic-free, Kosher, Halal Certified, Certified B Corporation, Vegan, American Grassfed, Free-range, Cage-free, Made in America, Country of Origin, Animal Welfare Approved, American Humane Certified, Bird Friendly, Food Alliance Certified, Salmon Safe, Certified Sustainable Seafood, Fair Trade Certified, USDA quality gradings, any other food grading, any certification or other food characteristic. An item property is any food attribute, such as any attribute that is used to compare food items, such as a grading system or certification. Item properties may be any official or un-official grading or certification. The item properties are each associated with one or more graphic labels, which are standard symbols that convey the property to a consumer and are located on packaging of an item and are referred to herein as item labels.

Item properties are often not included in the inventory information provided to the online concierge system <NUM> by the warehouses <NUM>. However, these properties may be important to consumers' selection of items to add to a delivery order, whether due to a consumer's dietary restrictions, health concerns, religious practices, etc. If these item properties are not included in the inventory database <NUM> to be displayed to a user for selecting items in a delivery order, the customer <NUM> may not be able to make a fully informed decision, and may avoid purchasing items, or purchase items and later decide to return food or request replacement items. This reduces the efficiency of the environment <NUM>.

As discussed above, in conventional systems, adding item properties to the inventory database <NUM> was a manual process. While item properties can be indicated through common graphic labels or "item labels," which appear on item packaging, this information is often not included in the inventory information provided by the warehouses <NUM>. To update property information of an item in the inventory database <NUM>, these item labels were typically manually read and coded as item properties in the inventory database <NUM>. While food labels were designed to be easily understandable to humans, they cannot be simply read and coded by computers. Recognizing a visual symbol on an image of an item is complicated by the fact that the available image of the item's packaging, e.g., a photograph taken in a warehouse, may obscure or distort these visual symbols. For example, due to placement of the item label on the item and/or the placement of the item in the warehouse, a property label depicted in an item image may be bent, partially obscured by other items, or otherwise not easily visible. Complicating matters, the item labels used on packaging do not always translate well to a designated item property used within an inventory database, since labels associated with properties may not be standardized, and there may be many different labels associated with a single property. For example, an item may be associated with an "organic" property, but there are several different organic certifications (USDA, International, etc.), which each have different graphics or item labels. From the images alone, it can be difficult for a computer system to (<NUM>) determine whether an image of an item has a label indicating an item property, (<NUM>) identify the label within the image of an item, and (<NUM>) correlate the label to a particular item property.

To enable an automated process of augmenting the inventory database <NUM> with accurate item properties, the item property learning module <NUM> performs a process for converting images of item labels to one or more item properties used by the inventory database <NUM>. In particular, a property identification engine <NUM> uses a machine-learned segmentation neural network <NUM> to identify item labels in images of items corresponding to items included in the inventory database <NUM>. A set of pixels in the image that are associated with the label are isolated from the image of the item. The property identification engine <NUM> then uses a machine-learned label classifier <NUM> to classify the item labels and to identify the item property associated with the label. The pixels associated with item labels are thus input into the machine-learned label classifier <NUM> to identify a property indicated by the label. The property identification engine <NUM> then updates the inventory database <NUM> with the determined property. Using the machine-learned label classifier and the machine-learned segmentation neural network <NUM> allows the online concierge system <NUM> to augment the inventory database <NUM> with item properties without manual intervention. The process of determining item properties is described in further detail below.

The image property learning module <NUM> includes a machine-learned segmentation neural network <NUM>, a segmentation modeling engine <NUM> and training images of items in a warehouse <NUM>. The segmentation modeling engine <NUM> uses the training images of items in a warehouse <NUM> to generate the machine-learned segmentation neural network <NUM>. The machine-learned segmentation neural network <NUM> can learn from the training images of items in a warehouse <NUM>, rather than follow only explicitly programmed instructions. The property identification engine <NUM> uses the machine-learned segmentation neural network <NUM> to determine which pixels in an image of an item are associated with a potential item label, as well as the pixel locations within the image. Thus the machine-learned segmentation neural network <NUM> performs both a classification of the pixels within an image of an item (e.g., if they are item labels or not item labels) as well as a localization of the classified pixels (e.g., where in the image the classified pixels are located).

The machine-learned segmentation neural network <NUM> identifies the presence and location of item labels in images of items in the inventory database <NUM>. The images of items in the inventory database <NUM> may be provided to the online concierge system <NUM> by a picker, such as picker <NUM>, through the PMA <NUM> (e.g., in a photograph taken by the picker <NUM> in a warehouse <NUM>), from a retailer, or from another source or a combination of sources. The training images of items in a warehouse <NUM> may similarly be provided to the online concierge system <NUM> through the PMA <NUM>, from retailers, or from other sources. In some examples, the training images of items in a warehouse <NUM> are stored in the inventory database <NUM> along with item property identifiers. Thus an item in the inventory database <NUM> may have associated images that are also included in the training images of items in a warehouse <NUM>. An image of an item in the training images of items in a warehouse <NUM> may be a photograph comprised of image pixels.

The training images of items in a warehouse <NUM> are tagged with information indicating which segments of the training images indicate an item label, and which segments of the training images are not item labels. For example, each pixel of the training images of items in a warehouse <NUM> is tagged as being an item label or not an item label. In some examples, the training images of items in a warehouse <NUM> are labeled with areas of an image that are item labels, such as a bounding box that surrounds an item label. In some examples, a single image in the training images of items in a warehouse <NUM> contains images of multiple items and multiple item labels. The training images of items in a warehouse <NUM> may be tagged based on inventory information provided to the online concierge system <NUM> by a warehouse, such as warehouses <NUM>. In some examples, the training images of items in a warehouse <NUM> are tagged by a picker <NUM> that provides the training image to the online concierge system <NUM>. For example, a picker <NUM> in a warehouse <NUM> may obtain a photograph of an item in the warehouse <NUM>, indicate in the photograph the location of an item label, and provide the photograph with the location of an item label to the online concierge system <NUM> via the PMA <NUM>. The online concierge system <NUM> then incorporates the tagged photograph of the item into the training images of items in a warehouse <NUM>. In other examples, the training images of items in a warehouse <NUM> are tagged by a third party other than the picker <NUM>. The training images of items in a warehouse <NUM> may visually reflect actual conditions of items and item labels in a warehouse, such as crumpled packaging that may distort a label or lighting that under or over exposes an item label. In some examples, the online concierge system <NUM> may request a picker in a store to provide more training images of items in a warehouse <NUM>. For example, if an item does not have images associated with it in the inventory database <NUM>, the online concierge system <NUM> may request an image. In other examples, if an item has a low confidence score output by a machine-learned segmentation neural network <NUM>, the online concierge system <NUM> may request an additional image.

The segmentation modeling engine <NUM> uses the training images of items in a warehouse <NUM> to generate the machine-learned segmentation neural network <NUM>. The machine-learned segmentation neural network <NUM> contains a set of functions that relate an input image to a location of an item label in the input image. The set of functions may be kernel functions, which act as filters or mappings between layers of the machine-learned segmentation neural network <NUM>. The kernel functions assign different weights to the pixel values of an image input into the machine-learned segmentation neural network <NUM>. The segmentation modeling engine <NUM> trains the machine-learned segmentation neural network <NUM> with the training images of items in a warehouse <NUM> to determine the kernel functions and relative weights between each layer of the machine-learned segmentation neural network <NUM>. The kernel function weights may be randomly initialized, e.g., from a Gaussian distribution prior to training.

In some examples, the segmentation modeling engine <NUM> trains the machine-learned segmentation neural network <NUM> in response to adding new images to the training images of items in a warehouse <NUM>. In some examples, the segmentation modeling engine <NUM> trains the machine-learned segmentation neural network <NUM> in response to a low confidence score output by the machine-learned segmentation neural network <NUM>. The machine-learned segmentation neural network <NUM> may improve confidence scores through re-training by the segmentation modeling engine <NUM> on new or more images in the training images of items in a warehouse <NUM>.

In some embodiments, the machine-learned segmentation neural network <NUM> is a convolutional neural network (CNN), such as a U-Net Convolutional Neural Network. In this example, the machine-learned segmentation neural network <NUM> is structured with a contracting path and an expansive path. The contracting path includes a series of convolutions, whose outputs are then input into an activation function, such as a rectified linear unit (ReLU), in an activation layer. The convolution layer followed by an activation layer may be repeated twice, before the results are then pooled in a pooling layer. In some examples, the results are Max Pooled in the pooling layer, such that maximum value is selected from a cluster of neurons in the activation layer. In some examples, the pooling layer may be a 2x2 max pooling operation. In the contracting path, there may be any number of convolutions, activation and pooling layers. In some examples, there are four repetitions of: a first convolution layer, a first activation layer, a second convolution layer, a second activation layer, and a max pooling layer. In the contracting path, the max pooling layer down samples the previous activation layer, such that the pixel dimensions of an input image are progressively reduced.

The expansive path includes a series of deconvolutions, whose outputs are then input into an activation function, which may be the same activation function as in the contracting path. The deconvolution layer followed by an activation layer may be repeated twice, before the results are then up-sampled. In some examples, the dimensions of the deconvolution layer and the upsampling are the same as the dimensions of the convolution layer and the pooling layer of the contracting path. In the expansive path, there may be any number of deconvolutions, activation and pooling layers. In some examples, there are four repetitions of the following operations: a first deconvolution layer, a first activation layer, a second deconvolution layer, a second activation layer, and an up sampling. In the expansive path, the upsampling increases the pixel dimensions of a previous activation layer, such that the pixel dimensions of an output image are progressively increased. In some examples, the pixel dimensions of the input image are the same pixel dimensions of the output segmentation image. Alternatively, the input image and output segmentation image have different pixel dimensions. The U-Net structure classifies pixels of an input image as being associated with an item label or not associated in an item label, and also localizes the classified pixels within the original input image. Further details regarding the machine-learned segmentation neural network <NUM> are described with reference to <FIG>.

The item property learning module <NUM> includes a machine-learned label classifier <NUM>. The machine-learned label classifier <NUM> classifies item labels identified by the machine-learned segmentation neural network <NUM> to determine an item property represented by an item label. The machine-learned label classifier <NUM> learns from the training images of item labels <NUM>, rather than follow only explicitly programmed instructions. The property identification engine <NUM> uses the machine-learned label classifier <NUM> to determine item properties from the images of item labels output by the machine-learned segmentation neural network <NUM>. Thus the machine-learned label classifier <NUM> classifies the item labels according to various item properties, such as Organic, Kosher, or any other item properties that can be identified from a label.

The machine-learned label classifier <NUM> identifies the properties associated with item labels. In some examples, the machine-learned label classifier <NUM> identifies the properties associated with portions of images of items in the inventory database <NUM> that the segmentation modeling engine <NUM> has determine are item labels. Thus the property identification engine <NUM> inputs pixels classified as "item labels" by the segmentation engine <NUM> into the machine-learned label classifier <NUM> to determine what property is indicated by the item label. The training images of item labels <NUM> that the classifier modeling engine <NUM> uses to train the machine-learned label classifier are provided to the online concierge system <NUM> by a picker through the PMA <NUM>, by a retailer, or from other sources or a combination of sources. In some examples, the training images of item labels <NUM> are stored in the inventory database <NUM> along with item property identifiers. Thus an item in the inventory database <NUM> may have one or more associated images that are also included in the training images of item labels <NUM>. The images in the training images of item labels <NUM> may be portions of photographs of items taken in a warehouse <NUM>.

The training images of item labels <NUM> are tagged with the item property of each item label. The training images of item labels <NUM> may include both positive images of item labels (i.e., show the item label) and negative images of item labels (i.e., do not show item labels). Item labels that indicate the same property may be grouped together in the training images of item labels <NUM>. For example, item labels for International Organic may be grouped with item labels for USDA Organic. The training images of item labels <NUM> may be tagged based on inventory information provided to the online concierge system <NUM> by a warehouse, such as warehouses <NUM>. In some examples, the training images of item labels <NUM> are tagged by a picker <NUM> that provides the training image to the online concierge system <NUM>. For example, a picker <NUM> in a warehouse <NUM> may obtain a photograph of an item in the warehouse <NUM>, and indicate in the photograph the item label and the property associated with the label, and communicate the photograph to the online concierge system <NUM>. In some examples, the training images of item labels <NUM> are tagged by a third party other than the picker <NUM>. The online concierge system <NUM> then incorporates the tagged photograph of the item into the training images of item labels <NUM>. The training images of item labels <NUM> may visually reflect actual conditions of item labels in a warehouse, such as crumpled packaging that may distort a label or lighting that under or over exposes an item label.

The classifier modeling engine <NUM> uses the training images of item labels <NUM> to generate the machine-learned label classifier <NUM>. The machine-learned label classifier <NUM> contains a set of functions that relate an input image of an item label to an item property. The set of functions may be kernel functions, which act as filters or mappings between layers of the machine-learned label classifier <NUM>. The kernel functions assign different weights to the pixel values of an image of an item label input into the machine-learned label classifier <NUM>. The classifier modeling engine <NUM> trains the machine-learned label classifier <NUM> with the training images of item labels <NUM> to determine the kernel functions and relative weights between each layer of the machine-learned label classifier <NUM>. In some examples, the classifier modeling engine <NUM> may train the machine-learned label classifier <NUM> in response to a low confidence score output by the machine-learned label classifier <NUM>. Thus the machine-learned label classifier <NUM> may improve confidence scores through re-training on new or more images in the training images of item labels <NUM>.

The machine-learned label classifier <NUM> may be a CNN. In some examples, there is a single machine-learned label classifier <NUM> which classifies all input images of item labels with all item properties. In other examples, there may be separate machine-learned label classifiers <NUM> for each item property (e.g., one classifier for organic, another classifier for Kosher, etc.) or for each label (e.g., one classifier for USDA organic, another classifier for International Organic, etc.). If the item property learning module <NUM> includes multiple different classifiers, the classifier modeling engine <NUM> may train the separate machine-learned label classifiers <NUM> with different groupings of training images of item labels <NUM> with the same item properties or labels. For example, if a machine-learned label classifier <NUM> classifies item labels as being organic or not organic, then the classifier modeling engine <NUM> may train the machine-learned label classifier <NUM> with grouped training images of organic labels within the training images of item labels <NUM>.

The machine-learned label classifier <NUM> may have any number of convolutional, activation or pool layers. The convolutional layers of the machine-learned label classifier <NUM> may each be followed by an activation layer. A pooling layer, such as a max pooling layer, may follow an activation layer. The output of the machine-learned label classifier <NUM> may be an item property associated with an input item label. In some examples, the machine-learned label classifier <NUM> provides a confidence score associated with the item label classification. If the confidence score is above a threshold level (e.g., <NUM>, or <NUM>), the inventory management engine <NUM> may update the item entry in the inventory database <NUM> with the item property indicated by the machine-learned label classifier <NUM>. The machine-learned label classifier <NUM> is described in more detail with reference to <FIG>.

<FIG> is a diagram of the CMA <NUM>, according to one embodiment. The CMA <NUM> includes an ordering interface <NUM>, which provides an interactive interface with which the customer <NUM> can browse through and select products and place an order. The CMA <NUM> also includes a system communication interface <NUM> which, among other functions, receives inventory information from the online shopping concierge system <NUM> and transmits order information to the system <NUM>. The CMA <NUM> also includes a preferences management interface <NUM> which allows the customer <NUM> to manage basic information associated with his/her account, such as his/her home address and payment instruments. The preferences management interface <NUM> may also allow the user to manage other details such as his/her favorite or preferred warehouses <NUM>, preferred delivery times, special instructions for delivery, and so on.

<FIG> is a diagram of the PMA <NUM>, according to one embodiment. The PMA <NUM> includes a barcode scanning module <NUM> which allows a picker <NUM> to scan an item at a warehouse <NUM> (such as a can of soup on the shelf at a grocery store). The barcode scanning module <NUM> may also include an interface which allows the picker <NUM> to manually enter information describing an item (such as its serial number, SKU, quantity and/or weight) if a barcode is not available to be scanned. PMA <NUM> also includes a basket manager <NUM> which maintains a running record of items collected by the picker <NUM> for purchase at a warehouse <NUM>. This running record of items is commonly known as a "basket. " In one embodiment, the barcode scanning module <NUM> transmits information describing each item (such as its cost, quantity, weight, etc.) to the basket manager <NUM>, which updates its basket accordingly. The PMA <NUM> also includes a system communication interface <NUM> which interacts with the online shopping concierge system <NUM>. For example, the system communication interface <NUM> receives an order from the system <NUM> and transmits the contents of a basket of items to the system <NUM>. The PMA <NUM> also includes an image encoder <NUM> which encodes the contents of a basket into an image. For example, the image encoder <NUM> may encode a basket of goods (with an identification of each item) into a QR code which can then be scanned by an employee of the warehouse <NUM> at check-out.

The PMA <NUM> includes an imaging module <NUM>, which allows a picker <NUM> to collect images of items in a warehouse, including images of item labels. The imaging module <NUM> allows a picker <NUM> to collect images, e.g., by taking a photograph of one or more items in a warehouse. The imaging module <NUM> may also provide an interface for the picker <NUM> to identify the item, or the picker mobile application <NUM> may request that the picker <NUM> take a photograph of a specific item. The imaging module <NUM> sends the photograph and item identifier to the online concierge system <NUM>, which may identify item properties for the item using the item property learning module <NUM>. In some embodiments, the imaging module <NUM> also allows a picker <NUM> to indicate the location and type of an item label in a collected image. For example, a picker <NUM> may draw a bounding box around an item label to indicate that the identified pixels are associated with an item label, and select a property associated with the item label. The imaging module <NUM> sends the resulting tagged image of an item label on an item in a warehouse to the online concierge system <NUM>, which may incorporate it into the training images of items in a warehouse <NUM>. As another example, the picker <NUM> may select an area in an image that shows an item label and input the item property associated with the item label, such as 'organic,' 'vegan,' etc. The imaging module <NUM> sends the tagged image of an item label and its associated property to the online concierge system <NUM>, which may incorporated it into the training images of item labels <NUM>.

<FIG> is a flowchart illustrating a process <NUM> for updating an inventory catalog, according to one embodiment. The process <NUM> uses the machine-learned segmentation neural network <NUM> in combination with the machine-learned label classifier <NUM> to update item information in the inventory database <NUM>. Specifically, the process <NUM> identifies labels in images of items, determines an item property associated with the label, and updates the item information in the inventory database <NUM> with the item property. The process <NUM> may be carried out by the online concierge system <NUM>, e.g., by the property identification engine <NUM> in conjunction with the machine-learned segmentation neural network <NUM> and the machine-learned label classifier <NUM>.

The property identification engine <NUM> receives <NUM> an image showing an item in an inventory catalog. The inventory catalog may be the inventory database <NUM> as described in <FIG>. The online concierge system <NUM> may receive an image showing an item in the inventory catalog from a picker, such as the picker <NUM>. For example, a picker may provide the online concierge system <NUM> with an image of an item in a warehouse. The picker may identify the item, and convey the image with an item identifier to the online concierge system <NUM>. The property identification engine <NUM> may request that a picker <NUM> provide an image of an item in response to the property identification engine <NUM> determining that there is missing item property information for the item in the inventory catalog. For example, an item entry in the inventory catalog may not have an indication regarding if the item is organic or not.

The property identification engine <NUM> retrieves <NUM> a machine-learned segmentation neural network. This may be the machine-learned segmentation neural network <NUM> as described with reference to <FIG>. The machine-learned segmentation neural network is configured to receive as input an image of an item in the inventory catalog, such as the image received at <NUM>.

The property identification engine <NUM> uses the machine-learned segmentation neural network <NUM> to determine <NUM> a subset of pixels associated with an item label. The determination <NUM> may include both a categorization of the pixels in a received image as either being associated with an item label or not associated with an item label, as well as a localization of the item label pixels within the original received image input into the machine-learned segmentation neural network. An example illustration of the subset of pixels associated with an item label is shown in <FIG> and <FIG>. There may be a confidence score associated with the pixels, which indicates how likely the pixels are to indicate an item label.

The property identification engine <NUM> extracts <NUM> the subset of pixels from the image. The property identification engine <NUM> may define a bounding box around a subset of pixels from the image, and crop the bounding box from the original image. In other examples, the online concierge system <NUM> may extract only pixels identified by a machine-learned segmentation neural network as being associated with an item label. An example illustration of the subset of pixels extracted from the image is shown in <FIG>.

The property identification engine <NUM> retrieves <NUM> a machine-learned classifier. This may be the machine-learned label classifier <NUM> as described with reference to <FIG>. The machine-learned label classifier <NUM> is configured to take as input the extracted subset of pixels identified by the machine-learned segmentation neural network.

The property identification engine <NUM> uses the machine-learned label classifier <NUM> to determine <NUM> whether the subset of pixels shows an item label. The determining may include associating a subset of pixels with an item property. For example, the machine-learned label classifier <NUM> may classify the subset of pixels as being an organic label and thus having an organic property. In some examples, a confidence score may be associated with the item property, which indicates a certainty of a machine-learned label classifier that the pixels show the item label. An example illustration of the determining <NUM> is shown in <FIG> and <FIG>.

The property identification engine <NUM> updates <NUM> the item entry in the inventory catalog to indicate that the item has a property associated with the item label. For example, in response to determining that a subset of pixels of an item shows an item label, the online concierge system <NUM> updates the item entry on which the item label is located with a property, such as being organic or being Kosher. In some examples, if a confidence score is above a threshold, then the online concierge system <NUM> updates the item entry in an item catalog, such as the inventory database <NUM>. Thus as a result of the process <NUM>, an item catalog is updated with additional information about an item, such as an item property, which otherwise was missing from the catalog. The item property can then be displayed to a user of the online concierge system <NUM> to help them make better decisions about which items to select for delivery.

<FIG> is an illustration of an image input into a segmentation model, according to one embodiment. The segmentation model may be the machine-learned segmentation neural network <NUM>. The segmentation model input image <NUM> shows an item <NUM>. In some examples, the online concierge system <NUM> may be missing item properties associated with item <NUM>. For example, the item <NUM> may be included in the inventory database <NUM>, but may not have an indication in the inventory database <NUM> as to whether or not the item <NUM> is organic. The segmentation model input image <NUM> is input into the machine-learned segmentation neural network <NUM>. The segmentation model input image <NUM> is composed of a number of pixels. The machine-learned segmentation neural network <NUM> classifies the pixels in the image as being associated with an item label or not associated with an item label, as well as the location of label pixels in the image.

<FIG> is an illustration of an output image of the segmentation model, according to one embodiment. The segmentation model output image <NUM> is an example of an output of the machine-learned segmentation neural network <NUM>. As shown, the machine-learned segmentation neural network <NUM> classifies the pixels of the segmentation model input image <NUM> into two categories: being associated with an item label, or not associated with an item label. Thus the pixels in the output image <NUM> are either the non-label pixels <NUM> or label pixels <NUM>. In addition to this classification, the segmentation model locates the classified pixels within the original image. This allows the online concierge system to extract these pixels for further identification, as discussed with reference to <FIG>.

<FIG> is an illustration of an image output by segmentation model and input into a classification model, according to one embodiment. The classification model may be the machine-learned label classifier <NUM> as discussed in further detail with reference to <FIG>. The classifier input pixels <NUM> are label pixels <NUM> extracted by the segmentation model. In some examples, the classifier input pixels <NUM> include the label pixels <NUM> identified by the segmentation model as well as additional surrounding pixels in a bounding box around the label pixels <NUM>. In some examples, the classifier input pixels <NUM> are extracted from the original image input into the segmentation model, such as the segmentation model input image <NUM>. In some examples, the classifier input pixels <NUM> are downsampled from the original image input into the segmentation model, such that the pixel resolution of an image input into the classifier model is different from a pixel resolution in the original image.

<FIG> is an illustration of the output of the classification model, according to one embodiment. The classifier output <NUM> identifies the item property <NUM> indicated by the input classifier input pixels <NUM>. Thus the item label is classified as the classified label <NUM> and the item property indicated by the label pixels <NUM> is identified as indicating the item property <NUM>. In response to the classifier model identifying the item property <NUM>, the property identification engine <NUM> updates the entry of the item <NUM> in the inventory database <NUM> with the item property <NUM>. The identified item property <NUM> may have an associated confidence score, which indicates a certainty that the input pixels <NUM> indicate the item property <NUM>.

<FIG> is an illustration of an image processed by a segmentation model, according to one embodiment. The received image <NUM> may be the image received <NUM> showing an item in an inventory catalog, as discussed with reference to <FIG>. The received image <NUM> includes an image of item <NUM>. The property identification engine <NUM> inputs the received image <NUM> into the segmentation model <NUM>. The segmentation model <NUM> may be the machine-learned segmentation neural network <NUM>, as discussed with reference to <FIG>. The segmentation model <NUM> identifies first label pixels <NUM> and second label pixel <NUM> located on the item <NUM>. As shown in <FIG>, the item labels on the item <NUM> in the received image <NUM> may be distorted by rumpled packaging, be under or over exposed in the received image <NUM>, be partial obscured by other items, or otherwise reflect real world conditions of items in a warehouse. Because the segmentation model <NUM> is trained using images of items in a warehouse, as described with reference to <FIG>, the segmentation model <NUM> is still able to identify the item labels. The segmentation model <NUM> identifies any number of labels located on the item <NUM>. As shown, item <NUM> includes the first label pixels <NUM> and the second label pixel <NUM>, which may each indicate different item properties.

<FIG> is an illustration of an image processed by a set of classifiers, according to one embodiment. In response to the segmentation model <NUM> identifying the first label pixels <NUM> and the second label pixel <NUM> within the received image <NUM> as being associated with item labels, the property identification engine <NUM> extracts the first label pixels <NUM> and the second label pixel <NUM> from the received image <NUM>. The property identification engine <NUM> retrieves <NUM> one or more machine learned classifiers, including a non-GMO classifier <NUM>, a vegan classifier <NUM>, and an organic classifier <NUM> as illustrated in <FIG>. In some embodiments, separate classifier models may be trained to identify different properties. Thus the non-GMO classifier <NUM> identifies if an input image is or is not a non-GMO label, a separate vegan classifier <NUM> identifies if an input image is or is not a vegan label, and a separate organic classifier <NUM> identifies if an input image is or is not an organic label. In other embodiments, a single classifier may identify a property associated with an input image of an item label. In these embodiments, the non-GMO classifier <NUM>, vegan classifier <NUM> and organic classifier <NUM> may all be incorporated into a single classifier that is able to classify input images of item labels as being non-GMO, vegan or organic.

In <FIG>, the first label pixels <NUM> are input into the non-GMO classifier <NUM>, the vegan classifier <NUM>, and the organic classifier <NUM> to determine if the first label pixels <NUM> indicate that item <NUM> is non-GMO, vegan or organic. The second label pixel <NUM> are also input into the non-GMO classifier <NUM>, the vegan classifier <NUM> and the organic classifier <NUM>. The non-GMO classifier <NUM> determines that the second label pixel <NUM> are a non-GMO label and that the item <NUM> is associated with the non-GMO item property. The vegan classifier <NUM> determines that neither of the first label pixels <NUM> nor the second label pixel <NUM> are vegan labels, and outputs a negative result <NUM>. The organic classifier <NUM> determines that the first label pixels <NUM> are an organic label and that the item <NUM> is associated with the organic item property. As a result of the outputs from the non-GMO classifier <NUM>, the vegan classifier <NUM> and the organic classifier <NUM>, the item <NUM> may be updated in an item catalog, such as the inventory database <NUM>, as having the non-GMO and organic item properties. Each of the outputs of the non-GMO classifier <NUM>, the vegan classifier <NUM> and the organic classifier <NUM> may have an associated confidence score.

The foregoing description of the embodiments of the invention has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.

Some portions of this description describe the invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be implemented in software, firmware, hardware, or any combinations thereof.

There is disclosed, a software module is implemented with a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described.

There is also disclosed an apparatus for performing
the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a tangible computer readable storage medium, which include any type of tangible media suitable for storing electronic instructions, and coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

There is also disclosed a computer data signal
embodied in a carrier wave, where the computer data signal includes any computer program product or other data combination described herein. The computer data signal is a product that is presented in a tangible medium or carrier wave and modulated or otherwise encoded in the carrier wave, which is tangible, and transmitted according to any suitable transmission method.

Claim 1:
A method (<NUM>) for populating an inventory catalog (<NUM>), the method comprising:
receiving (<NUM>) an image (<NUM>, <NUM>) of an item (<NUM>, <NUM>) having an entry in an inventory catalog (<NUM>), the image comprising a plurality of pixels, the inventory catalog storing inventory information including properties of items, wherein each property is a food attribute used to compare items;
determining that a property of the item is missing from the inventory catalog;
retrieving (<NUM>) a machine learned segmentation neural network (<NUM>, <NUM>) trained, based on a set of images and associated properties, to determine locations of pixels in an image that are associated with an item label on the item indicating a property;
determining (<NUM>), using the machine learned segmentation neural network, a subset of pixels (<NUM>, <NUM>, <NUM>) associated with the item label in the received image and identifying locations of the subset of pixels of the received image;
extracting (<NUM>) the subset of pixels from the received image;
retrieving (<NUM>) a machine learned classifier (<NUM>) trained, based on a set of images of the item label, to determine whether an image shows the item label;
determining (<NUM>), using the machine learned classifier, that the extracted subset of pixels shows the item label, wherein the item label indicates the property (<NUM>) of the item that is missing from the inventory catalog; and
updating (<NUM>) the entry for the item in the inventory catalog to indicate that the item has the property associated with the item label.