Systems and methods for verifying machine-readable label associated with merchandise

A system for verifying a machine-readable label comprises a scan table processing device comprising a first input for receiving a list of items with machine-readable labels; a second input for receiving a list of stores that have an inventory of the items in the list of items and that have at least one sensing device for capturing images of the items; and an output that includes a plurality of electronic records. The system further comprises a data repository that stores the captured images of the items and that updates the electronic records to include an association to the captured images; a graphical user interface (GUI) processing apparatus that modifies the captured images in preparation for training an artificial intelligence apparatus to identify the items in the images; and a machine language (ML) model processor that determines whether the images training the artificial intelligence apparatus are correctly identified with machine-readable labels associated with the items.

FIELD OF THE DISCLOSURE

The present disclosure relates generally to asset protection, and more specifically to systems and methods for determining whether a merchandise item has a correct machine-readable label.

BACKGROUND

Barcode switching is a form of retail theft, where a barcode, quick response (QR) code, universal product code (UPC), or related machine-readable label is removed from a low cost item and affixed to a more expensive item. When the label is scanned at a checkout counter, the more expensive item is sold at the price of the low cost item, resulting in a loss to the retailer and inaccuracy in inventory information, for example, generated by a perpetual inventory system.

SUMMARY

In one aspect, provided is a system for verifying a machine-readable label, comprising: a scan table processing device comprising: a first input for receiving a list of items with machine-readable labels; a second input for receiving a list of stores that have an inventory of the items in the list of items and that have at least one sensing device for capturing images of the items; and an output that includes a plurality of electronic records that each includes a time and location of a scan operation performed on at least one machine-readable label of the machine-readable labels. The system further comprising a data repository that stores the captured images of the items and that updates the electronic records to include an association to the captured images; a graphical user interface (GUI) processing apparatus that modifies the captured images in preparation for training an artificial intelligence apparatus to identify the items in the images; and a machine language (ML) model processor that determines whether the images training the artificial intelligence apparatus are correctly identified with machine-readable labels associated with the items.

In some embodiments, the list of items includes items of interest that are identified as being at risk of theft.

In some embodiments, a record for each of the list of stores includes a unique facility identifier and information about sensors available for generating images of items of interest of the list of items.

In some embodiments, the output of the scan table processing device includes a table comprising a plurality of data records, which includes at least one of a store identification, a time of the scan operation performed on the at least one machine-readable label, and an identification of a checkout register where the scan operation is performed.

In some embodiments, the output of the scan table processor includes a time stamp that identifies an image from a video feed taken of an item of interest at a store of the list of stores that is of interest with respect to confirming whether a machine-readable label is associated with a correct item.

In some embodiments, the GUI processing apparatus compares the captured images and positively identified images of the items to determine whether the captured images are qualified for input to the artificial intelligence apparatus.

In some embodiments, the artificial intelligence apparatus includes a trained neural network that recognizes a scanning apparatus that performs a scan operation performed on the at least one machine-readable label to distinguish the scanning apparatus from the item at which the at least one machine-readable label is located.

In some embodiments, the ML model processor generates an event in response to a determination that the machine-readable label is associated with an incorrect item at which the at least one machine-readable label is located.

In another aspect, provided is a system for verifying a machine-readable label, comprising: a listing of a plurality of stores that have an item of interest identified by contents of a machine-readable label affixed to the item; a label processing apparatus that compares the machine-readable label affixed to the item and a valid image of the item to train a neural network; and a machine learning (ML) apparatus that identifies the image of the item to which the machine-readable label is affixed.

In some embodiments, the listing includes a time stamp that identifies when the item of interest was scanned, identifies a store of the plurality of stores, an identification of a register at the identified store where the item is scanned, and an index value that provides an electronic storage location of an image generated at a day and time stated in the time stamp.

In some embodiments, a record for each of the stores includes a unique facility identifier and information about sensors available for generating images of items of interest of the listing.

In some embodiments, the system further comprises a graphical user interface (GUI) processing apparatus that modifies an image of the item for training an artificial intelligence apparatus to identify the item in the image.

In some embodiments, the GUI processing apparatus compares the captured images and positively identified images of the items to determine whether the captured images are qualified for input to the artificial intelligence apparatus.

In some embodiments, the artificial intelligence apparatus includes a trained neural network that recognizes a scanning apparatus that performs a scan operation performed on the at least one machine-readable label to distinguish the scanning apparatus from the item at which the at least one machine-readable label is located.

In some embodiments, the ML model processor generates an event in response to a determination that the machine-readable label is associated with an incorrect item at which the at least one machine-readable label is located.

In another aspect, provided is a method for verifying a machine-readable label, comprising: providing a listing of a plurality of stores that have an item of interest identified by contents of a machine-readable label affixed to the item; comparing, by a label processing apparatus, the machine-readable label affixed to the item and a valid image of the item to train a neural network; and identifying, by machine learning (ML) apparatus, the image of the item to which the machine-readable label is affixed.

DETAILED DESCRIPTION

In brief overview, embodiments of the present concept include a system that minimizes the time necessary to identify bar code switching activities, improves the accuracy of detection, and automates the detection and response protocol of each event related to such activities.

FIG. 1is a diagram of an environment10in which embodiments of the present inventive concepts may be practiced.

The environment10may include but not be limited to a graphical user interface (GUI) processing apparatus12, a data repository14, a machine learning (ML) training apparatus18, a neural network20, a sensing device22, and a scanning apparatus24. Some or all of the GUI processing apparatus12, a data repository14, a machine learning (ML) training apparatus18, a neural network20, camera or other sensing device22, and scanning apparatus24can communicate with each other and/or other electronic apparatuses that may include an electronic exchange of data via a network16. The network16may be a local area network (LAN), a wide area network (WAN), and/or other communications network for transmitting electronic data.

FIG. 2is a flow diagram of a process100for determining whether a machine-readable label is associated with a correct item, in accordance with some embodiments. Some or all of the process100may be performed in the environment shown and described with respect toFIG. 1.

The server18may store and process a known UPC list102, which includes a list of items with machine-readable labels, e.g., stock keeping units (SKUs). In some embodiments, the labels are laser printed “peel & stick” barcode labels or the like that are capable of being affixed to a surface of an item of merchandise or its packaging, but capable of being removed in its entirety for affixing to a different item of merchandise or its packaging. In other embodiments, the barcode labels may be printed using a commercially-available printer. The list102may include items of interest at risk of theft, fraud, and so on, and can be collected using historical data, data mining techniques, and so on. The UPC list102may be stored electronically, for example, at a database or the like that includes store-related data, such as inventory details and so on. The UPC list102may include records, fields, or other electronic data that includes associations to previous or similar items, for example, items sold in a previous season, limited time promotions, and so on.

The server18may store and process a store list103including a list of retail establishments that have one or more cameras22positioned over a checkout counter. The store list may include a list of stores that have an inventory of the items in the list of items of the UPC list102. The server18may include one or more different computer processors at different locations but each connected to the network16. The cameras22are preferably high quality video cameras, for example closed-circuit television (CCTV) video devices, but not limited thereto. For example, a camera22may be of high or low resolution, and with or without color22. The cameras22are preferably part of a security monitoring system, for example, including digital video recorders (DVRs), video analysis software, and so on. In some embodiments, the checkout counter is a self-checkout (SCO) counter or the like that includes point of sale (POS) devices for allowing customers to purchase store items with minimal or no assistance from a store associate. Although a camera22is shown and described, other sensing devices such as stereoscopic cameras, infrared, or IR sensors may be used in addition to or as an alternative to a camera, in particular any device capable of providing item unique attributes, such as size, shape, color, temperature, mass, weight, and so on.

The store list103may be generated from existing asset inventory sheets, user-identified CCTV systems, and/or other store servers or data repositories used by retail establishments. The store list103may include but not be limited to a store number or other unique facility identifier, camera or other available device information, device location or position information, channel or address, for example, for identifying each sensing device available for collecting images or other data regarding an item of interest. This information may be generated as a record for each of the list of stores and stored at the data repository14.

Contents of the UPC list102and store list103are input to a scan table processing device104. The scan table processor104is a computer hardware processor, for example, part of the server18, and may include a memory device or otherwise communicate with a storage device such the data repository14or the like to store and retrieve data to generate the scan table104and/or results generated by the scan table processor104. In some embodiments, the scan table104is constructed and arranged into a plurality of rows and columns, where each row includes data regarding an item scanned at a checkout counter. Each column includes data identifying a store having security cameras at its checkout counters. The scan table processor104can output a listing106or table, matrix, or the like that includes one or more stores that have an item of interest identified by the contents of a machine-readable label affixed to the item. The listing106may include a date/time stamp that identifies when the item of interest was scanned, identifies the particular store, includes a register location/number reference and includes an index value that may direct the system to a set of images taken at the identified store at the date/time stamp, for example, a register (reg.) identifying the location and type of register used, for example, a unique identification to determine the location of an item scan operation.

The digital video recorders (DVRs) of the cameras22, or related security monitoring system, records video clips of a checkout location in a digital format to a storage device108. The video clips can be partitioned into still images and placed into an image repository, for example, which stores digital data via the network16accessible for training the neural network20and auditing the images.

At block110, the digital images are shown as stored prior to evaluation/analysis, for example, either for training or for artificial intelligence processing. The images may be categorized by the scan table listing106. For example, the listing106, or output of the scan table processor104, can include a date/time stamp that identifies a particular photograph from a video feed taken at a particular store, which may be of interest with respect to confirming whether a barcode is associated with a correct item.

In another example, a determination is made that an item is scanned at 2:30 p.m. at a given register having GUI processing apparatus12. An image stored at block110may contain the snapshot image of the register scan taken at the scanning apparatus24, indicating that activity that occurred at the identified register at 2:30 p.m. At block112, a training process by the neural network20and/or other artificial intelligence system may be performed. In particular, the image is evaluated and notated at GUI processing apparatus12, and selected as being ingested into an artificial intelligence system for training and/or enhancement purposes. In this example, a data repository at which the listing106is stored serves as the database storing all the reference and relevant materials needed to triangulate which cameras/time combinations need to be collected from storage device108and loaded into the repository at block110.

The stored images in the repository at block110may be analyzed at block112by a label graphical user interface (GUI) processing apparatus12. As shown inFIG. 3, the label GUI processing apparatus at block112may include but not be limited to an SCO image processor202, a stock image processor204, and an item confirmation processor206. The SCO image processor202processes an image of an item of interest from the repository110. A scanner bed of a register203or other sensor apparatuses may process images of the item, to capture relevant features such as various angles, stock product images, and so on to confirm the item's scannable label matches the actual item scanned at the register203. The stock image processor204displays an image of a known item. The images can be viewed by a user11to determine whether the images are of the same item. If so, then the user11can select a display button indicating whether the items are the same or different. Users may also select at the (GUI) processing apparatus12or a display or other computer in communication with the (GUI) processing apparatus12one or more multiple attributes to add additional context to image to further refine and improve detection variables. Attributes may include but not be limited to other contexts such as lighting, hand or unrelated object in area, image error, incorrect image retrieved, blurriness, and so on. Alternatively, the item confirmation processor can perform an electronic analysis of the two images and provide an automatic representation according to an object recognition application or other image analysis software. More specifically, an item confirmation may be provided to the trainer with one or more reference samples for comparison. After the AI system is trained, it may provide an electronic analysis for reinforcement of the AI system/neural network20.

Referring again toFIGS. 1 and 2, as previously described, the GUI processing apparatus12is constructed and arranged to train an artificial neural network20, which is applied at block112to identify the images by processing learning material from the image data. The results, e.g., cropped images, generated by the GUI processing apparatus12are output to a machine learning (ML) apparatus at block114, where the training data is ingested by a machine learning apparatus, which in turn may be incorporated by the neural network20for identifying the item.

A machine language (ML) model116may be implemented that identifies images of items with machine-readable labels. The ML model116receives at another input data interrupted from the image and used that to weigh it against known data for the item to render a decision or score on the scan accuracy. An exception list/triggering event118is generated from a comparison of the two inputs at the ML model116that includes an alert or exception regarding an item to which a scanned machine-readable label is associated that is not recognized. This detection and trigger event will occur in near real time when not training. The alert or exception may be output via the network16or via a local wireless connection such as Bluetooth or the like to a personal computer, a visual and/or audio alarm at the checkout counter, suspension of the current checkout transaction until authorized personnel are able to respond, security personnel notification, and so on.

Referring toFIG. 4, a process for training the neural network20by GUI processing apparatus12requires the receipt (310) by the GUI processing apparatus12of set of images312A-C (generally,312). The GUI processing apparatus12includes a display for visually displaying the images312, where a user and/or computer program may annotate or otherwise modify the images312. For example, the user11may classify (320) the images, for example, by identifying the images. In some embodiments, the images312are classified for each SKU in a store. The classified images322A-C (generally,322), also referred to as training images, are output to the neural network20. In training the neural network20, an iterative learning process is executed in which the classified images322are processed, and the neural network20are trained to recognize the object, e.g., dog, displayed in the images. Although a dog is shown in this example inFIG. 4, the system is constructed and arranged to recognize merchandise such as retail goods for purchase. In some embodiments, thousands of images or more are required to train the neural network20on a single concept, such as the dog shown inFIG. 4, or more relevant products labelled with a SKU. In some embodiments, the system is constructed and arranged to recognize goods for purchase. In other embodiments, the retaining and adjusting of parameters regarding item identification and confirmation permits the system to I recognize other objects of interest not limited to items of interest for establishing whether a fraudulent purchase is being accepted. For example, other applications may equally apply such as store associates wearing an item such as a vest, cap, and so on, which can be used to track and audit staffing coverage.

One of the challenges with training a neural network20is identifying and classifying objects, namely, store items, in images taken at a checkout counter. When such images are captured, other objects such as a human hand or the portions of the scanning apparatus24may be in the field of view of the camera22. Referring toFIG. 5, a plurality of images401A-F (generally,401) may be captured and used to create a neural network to identify the scanning apparatus24. As shown inFIG. 6, an image401F shown inFIG. 5can be cropped or otherwise edited by the GUI processing apparatus12. The image401F can be tagged with a UPC and item description and/or other identifier of the object in the image401F. The number of training images is greater than one, and preferably at least one thousand to ten thousand images for each SKU. Each image may have a similar background, for example, a human hand, scanning apparatus, and so on. Training the neural network20to recognize the background may improve the ability of the neural network20to identify the object. A process is performed multiple times for each SKU. For example, multiple images, angles, lighting backgrounds, skin tones, and so on with respect to the image are collected and processed to gain an accurate reading and identification of the background of the image in order to improve item recognition accuracy. Other sensors such as thermal sensors or the like may also be provided for identifying items of interest. Therefore, a register having a scanner24can provide information on what items were scanned at a particular time, and the camera22can provide images taken at that time around the scanner24. The GUI processing apparatus12can annotate the item of interest, for example, a box of waffles, relative to the regions around the item to train the neural network20with accuracy. In some embodiments, the trained neural network20that recognizes the scanner24and can distinguish it from other objects may provide a boundary or box, for example, shown inFIG. 5around the scanner so that items inside the box are identified. As shown inFIG. 7, the images501D-501F are cropped from images501A-501C respectively, and used to train the neural network20to identify any item of merchandise notwithstanding the presence of the scanning apparatus24.

FIG. 8is an illustrative example of an application of the environment10ofFIG. 1and process100for identifying a fraudulent switching of a label. In this example, a scanning apparatus24at a checkout register scans the UPC of an inexpensive toy31. However, the UPC has been fraudulently removed from the toy31and placed on an expensive television set32. A camera22captures an image of the television set at the scanner at the same time that the UPC is scanned. The neural network20determines that the item scanned does not fit the parameters for the toy, whereby an alert is automatically generated.

FIGS. 9A and 9Bare views of the results of a process for training a neural network on a predetermined number of photographs taken of an item of interest. The results here illustrate that the model may misclassify an item in a greater number of cases than when the images are cropped, for example, shown inFIG. 10. For example, a CCTV image may be cropped so that the item of interest resting on a flatbed checkout scanner is shown. The neural network, when trained, can classify with a higher degree of accuracy, even when the same sampling of training images, for example, 30 or so images as shown inFIGS. 9A and 9B.

The embodiments and examples set forth herein are presented in order to best explain the present invention and its practical application and to thereby enable those of ordinary skill in the art to make and use the invention. The descriptions of the various embodiments of the present disclosure have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Although the example embodiments have been described with reference to the components, modules and units discussed herein, such functional elements may be combined into fewer elements or separated into additional elements. Various combinations of optional features have been described herein, and it will be appreciated that described features may be combined in any suitable combination.