Abstract:
A produce recognition method which uses hierarchical Bayesian learning and kernel combination, and which offers classification-oriented synergistic data integration from diverse sources. An example method includes providing a classifier having a plurality of inputs, each input being adapted to receive produce data of a different modality; mapping the produce data to the respective input of the classifier by a computer; for each input, independently operating on the data relating to that input to create a feature set by the computer; comparing each feature in the feature set to respective pre-trained data for that feature to produce a similarity description set; combining all similarity description sets using a dedicated weighting function to produce a composite similarity description by the computer; and deriving a plurality of class values from the composite similarity description to create a recognition result for the produce item by the computer.

Description:
BACKGROUND 
       [0001]    One of the most frustrating problems facing grocery retailers is the lack of an automatic, accurate and efficient method to identify fruits and vegetables at the checkout lane. Many manual methods exist, but all are slow and inaccurate. The most common method involves consulting a printed list of all fruits and vegetables sold in a given store, to find their corresponding price codes. 
         [0002]    Since the process is so inefficient, many cashiers simply guess at the price codes or simply memorize and use a small subset of common price codes—especially if there are long lines at the register. This means the retailer loses revenue if the item actually sold is a more expensive item. For this reason, speed and accuracy are both important. In a self-service checkout transaction, customers often guess or sometimes deliberately defraud the system. 
         [0003]    A successful automatic produce recognition system needs to solve two challenges: (1) select appropriate and discriminative features (for example shape, size, color, aroma, and the like) for produce modeling; and (2) build an efficient and robust classifier. 
         [0004]    There have been attempts at solving this problem, including analysis of spectral response of reflected light, analysis of camera images, and analysis of produce aromas but these attempts have proven unreliable at inconsistently identifying some items. 
         [0005]    Since it is unlikely that any one approach will be sufficient to guarantee accurate recognition of all items, recognition methodologies have been combined. However, increasing the number of different kinds of recognition methodologies complicates classifier design. 
         [0006]    For example, a spectral-based methodology may use a distance-measure-based Gaussian density classifier with checkout frequency. Image-based methodologies may use a nearest neighbor classifier. Both methodologies are simple and easy to update, but they treat all features equally and independently. These simplistic assumptions do not reflect reality, where signals are correlated (not independent), and not all features are equally important. Some features are more important and provide better accuracy for produce recognition than others. Finally, these classifiers have inputs that are specifically configured for the type of feature used, and may not be suitable for use with a feature provided by a future sensor or technology. 
         [0007]    It would be desirable to provide a produce recognition method which addresses these concerns. 
       SUMMARY 
       [0008]    A produce recognition method is provided. The produce recognition method uses hierarchical Bayesian learning and kernel combination, and offers classification-oriented synergistic data integration from diverse sources. 
         [0009]    An example method includes providing a classifier having a plurality of inputs, each input being adapted to receive produce data of a different modality; mapping the produce data to the respective input of the classifier by a computer; for each input, independently operating on the data relating to that input to create a feature set by the computer; comparing each feature in the feature set to respective pre-trained data for that feature to produce a similarity description set; combining all similarity description sets using a dedicated weighting function to produce a composite similarity description by the computer; and deriving a plurality of class values from the composite similarity description to create a recognition result for the produce item by the computer. 
         [0010]    The method can be applied in many different environments. For example, the method can be applied in a transaction environment involving recognition and sale of produce items. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0011]      FIG. 1  illustrates an example produce recognition system. 
           [0012]      FIG. 2  is an example decision engine for recognizing a produce item. 
           [0013]      FIG. 3  is an example produce recognition method. 
       
    
    
     DETAILED DESCRIPTION 
       [0014]    Referring now to  FIG. 1 , produce recognition system  10  includes produce data collector  12 . 
         [0015]    Produce data collector  12  collects data about produce item  18 . Produce data collector  12  may represent a single data collector which captures a plurality of different features of produce item  18 , or a plurality of different data collectors which capture the plurality of different features. Produce data collector  12  may collect data using a plurality of different methodologies. 
         [0016]    For example, produce data collector  12  may include a camera for capturing and providing images of produce item  18 . The images may provide information about a plurality of different features, such as shape (2D or 3D), size, color, non-visible electromagnetic reflection and texture. 
         [0017]    As another example, produce data collector  12  may include the camera above combined with an aroma sensor (sometimes referred to as an olfactory sensor) for capturing chemicals given off by produce item  18  and for providing a chemical signature. 
         [0018]    As another example, produce data collector  12  may include the camera above combined with a scale for providing a weight and density of produce item  18 . 
         [0019]    As another example, produce data collection  12  may include the camera above combined with a thermal sensor for capturing information related to the internal structure and composition of produce item  18 . 
         [0020]    Features of produce item  18  may include any of number, weight, density, color, color variation, non-visible electromagnetic reflection, size, shape (2D or 3D), texture, aroma, and internal structure and composition. The features may also include other features captured using collectors and sensor or technologies yet to be discovered. 
         [0021]    Produce data collector  12  may prove useful in a variety of different environments, such as a retail point-of-sale (POS) environment. Produce data collector  12  may identify produce items  18  purchased during a transaction at a checkout system. For this purpose, the checkout system may be an assisted-service checkout system or a self-service checkout system. 
         [0022]    As another example, produce data collector  12  may be incorporated into a produce selection system in a produce section of a grocery store. 
         [0023]    As another example, produce data collector  12  may be incorporated into a produce freshness system in a grocery or grocery supplier, for additionally providing information for indicating freshness. 
         [0024]    Produce data collector  12  may be combined or integrated into other peripherals. For example, produce data collector  12  may be incorporated into a barcode reader. An example checkout device may include one that is disclosed in U.S. Pat. No. 6,457,644. This patent is hereby incorporated by reference. 
         [0025]    Produce recognition system  10  further includes computer  14 , which executes produce recognition software  16 . 
         [0026]    Produce recognition software  16  includes capture module  20 , processing module  22 , decision engine  24 , and training module  26 . 
         [0027]    Capture module  20  receives data about produce item  18  (referred to as “initial produce data”) from produce data collector  12  and obtains produce features from this initial produce data. Capture module  20  then outputs the obtained produce features  30  to processing module  22 . 
         [0028]    Processing module  22  processes the obtained produce features to create feature data  30 . For example, processing module  22  may process a captured image to extract color, shape, size, and texture information. Processing module  22  outputs the feature data to decision engine  24 . 
         [0029]    Decision engine  24  uses a classifier (described in more detail below) to identify produce item  18  using the feature data received from processing module  22 . 
         [0030]    The classifier used by decision engine  24  is a multi-class classifier comprising a hierarchy of kernels, wherein a composite kernel retains the dimensionality of base kernels from which the composite kernel is derived. The statistical models underlying this multi-class classifier have been described by the assignee of the present application in application Ser. No. 11/899,381, filed Sep. 5, 2007, and published Mar. 5, 2009, as publication number 2009/0057395. This published U.S. patent application is hereby incorporated by reference. 
         [0031]    Training module  26  is used to train decision engine  24 , in particular, to configure parameters used by decision engine  24 . 
         [0032]    Produce recognition software  16  may obtain other data from other sources or derive other data to assist with recognition. For example, in a POS environment, other data may include purchase frequency information associated with each of produce items  18 , where likelihood of purchase is used as a predictor of produce identity. 
         [0033]    In some environments, produce recognition software  16  may be combined with other software or complete additional tasks. For example, in a POS environment, produce recognition software  16  may work in combination with POS transaction software to display a candidate identity of produce item  18  for operator verification, to obtain a price, such as a price per unit weight, from a price look-up (PLU) data file, and to obtain payment for produce item  18 . 
         [0034]    Computer  14  includes a processor and memory, executes an operating system such as a Microsoft operating system, and may include a display, an input device, and other peripherals tailored to its environment. For example, in a POS environment, computer  14  may be coupled to one or more of a barcode reader, card reader, printer, cash drawer, cash money dispenser, and cash money acceptor. 
         [0035]    Computer  14  may include a client computer or a server computer. Computer  14  may include one or more computers at the same or different locations. 
         [0036]    Turning now to  FIG. 2 , decision engine  24  uses hierarchical Bayesian learning. Decision engine  24  is a multinomial probit classifier with a composite kernel (composite similarity description), which offers classification-oriented synergistic integration of diverse feature data and which can be applied to any type or source of feature data. 
         [0037]    The theoretical basis and the mathematical formulae for decision engine  24  have been applied to the problem of ATM fraud by the assignee of the present application and are disclosed in application Serial No. 11/899,381, filed Sep. 5, 2007, and published Mar. 5, 2009, as publication number 2009/0057395. This published U.S. patent application is hereby incorporated by reference. 
         [0038]    Training Mode 
         [0039]    Training module  26  is only used when produce recognition software  16  is operating in training mode; that is, not when the produce recognition software  16  is being operated by a customer, cashier, or other operator. 
         [0040]    For each single type (class or identity) of produce item  18  to be recognized by system  10 , training module  26  captures data from multiple items of the same type, with as many variations of each type of produce item as possible. Training optimizes the parameters used by decision engine  24 , such as θ and β, and other parameters. 
         [0041]    For example, if the decision engine  24  is to be trained to recognize bananas, then training module  26  captures variations in numbers of bananas in a bunch, variations in sizes of bananas, variations in colors of bananas (for example, from light yellow to light brown), variations in shapes (curvature) of bananas, and the like. 
         [0042]    Some produce items may need to be recognized not just based upon type, but also variety. For example, a produce type may be “apple”, and a variety may be “Golden Delicious”, “Discovery”, or the like. Training module  26  captures variations in variety. 
         [0043]    Training may also include varying environmental factors around produce item  18 , such as background light, humidity, and the like. This may be useful where the specific equipment in use rely on such environmental factors. For example, the intensity of background light may be important for some imaging techniques when they are applied to produce recognition. 
         [0044]    Operational Mode 
         [0045]    Decision engine  24  receives feature data  30  from processing module  22 . Feature data  30  include a plurality of different types of feature information F 1  . . . F s , such as color and non-visible EM data, size, shape (2D or 3D), internal structure and composition, aroma, weight, and density. Feature information may come from one or more collector sources. In this embodiment, feature data  30  is supplied by produce data collector  12 . 
         [0046]    The different types (or modalities) of feature information (from the feature data  30 ) are organized or separated into different inputs (feature inputs) as feature spaces  32 . 
         [0047]    Each feature input (F i , iε1, . . . ,S) from features spaces  32  is embedded into a corresponding kernel (similarity measurement) space  34  (F i , iε1, . . . ,S) according to a unique set of mapping parameters θ i  (iε1, . . . ,S). In other words, each feature input is operated on by a respective kernel K i  (iε1, . . . ,S) to obtain a similarity measure between the produce item  18  and the possible identities derived from training and used by decision engine  24  in every feature space  32 . 
         [0048]    Each kernel may have a unique weighting parameter β i  (iε1, . . . ,S) determined by the dedicated weighing function. The values of the weighting parameters indicate how important each feature will be in classifying inputs to them. Weighted kernels are combined to create a composite kernel  36 , which is the combined similarity measures between produce item  18  and the possible identities used by decision engine  24 . 
         [0049]    Composite kernel  36  is then operated on by multiclass classifier  38 . Multiclass classifier  38  is a multinomial regression based model, which produces a predictive output recognition result  40  or identity for produce item  18 , disregarding the numbers and types of feature inputs. Multiclass classifier  38  produces recognition results for each possible identity of produce item  18  used by decision engine  24 . 
         [0050]    Multiclass classifier  38  may use a different set of regression parameters to calculate a probability for each possible identity of produce item  18 , where the higher the probability value, the more likely the associated identity prediction is the item. 
         [0051]    The various parameters (such as θ and β) can be obtained from a training set using, for example, Gibbs sampling or a Variational Bayes Approximation, as described in more detail under the “Theoretical Considerations” section in the incorporated by reference application having publication number 2009/0057395. 
         [0052]    With reference to  FIG. 3 , general operation of produce recognition software  16  is illustrated. 
         [0053]    In step  50 , capture module  20  activates produce data collector  12  to initiate capture of a produce data. For example, capture module  20  may activate produce data collector  12  under operator control. As another example, capture module  20  may activate produce data collector  12  when produce item  18  is placed on a scale. As another example, capture module  20  may activate produce data collector  12  in response to operator-initiated commands. 
         [0054]    In step  52 , capture module  20  receives captured produce data from produce data collector  12  and obtains produce features from this initial produce data. 
         [0055]    In step  54 , processing module  22  processes these produce features as necessary to create feature data  32 . For example, processing module  22  may process a captured image to extract color, shape, size, and texture information. 
         [0056]    In step  56 , processing module  22  feeds feature data  30  into decision engine  24 , which is split into a plurality of feature inputs. 
         [0057]    In step  58 , decision engine  24  provides a predictive output recognition result  40  by applying the configuration parameters (established through the training process) to the feature inputs using the statistical model referenced above. 
         [0058]    In step  60 , decision engine  24  further provides an indication according to its environment. For example, decision engine  24  may display an image associated with predictive output recognition result  40 . This may enable a cashier to verify that decision engine  24  has correctly recognized produce item  18 . Decision engine  24  may additionally display information derived from captured information, such as freshness information. 
         [0059]    Although particular reference has been made to certain embodiments, variations and modifications are also envisioned within the spirit and scope of the following claims.