Patent Description:
Self-checkout machines provide a mechanism for customers to process their own purchases from a retailer. They are an alternative to the traditional cashier-staffed checkout. The customer performs the job of the cashier themselves, by scanning and applying payment for the items. In a typical self-checkout system, the customer is required to scan each item against a scanner, and then do the requisite payment.

However, a customer may have little or no training in the operation of a self-service checkout terminal, and may make errors when checking out their items. The customer may unintentionally miss out some items while scanning, and may move out of the store without making requisite payment. Further, shop-lifting is a major disadvantage associated with self-checkout stores. For example, a customer may not scan some items intentionally, and put the unscanned items in their shopping cart, and may move out of the store without making full payment. As a result, the self-check out stores may incur huge losses. Therefore, the current self-check out stores may still require a high number of personnel or store operators for preventing theft and unauthorized picking-up of items.

Therefore, in light of the foregoing discussion, there exists a need for a method and a system that detects scan irregularities in a self-checkout stores, that generates an alert when there is a mismatch between the products present in a shopping basket of the user, and a scanned list of items generated by the scanner, and that overcomes the aforementioned drawbacks associated with existing self-checkout systems.

<CIT> described systems and methods for preventing fraud during retail checkout. A system includes: item identifier acquisition devices to acquire identifiers ( e.g., barcodes) of items to be transacted; cameras or imagers to acquire images of the items to be transacted; an object recognition component to perform visual recognition of the acquired items through comparison of the acquired images of the items with a database and obtain identifiers of items represented in the database that correspond to an acquired image according to a correspondence criterion; a comparison component to com-pare a set of identifiers acquired through the item identifier acquisition devices with a set of identifiers obtained through the object recognition component; and an alerting component to provide an alert in case of discrepancies between the two sets.

According to a first aspect of the present disclosure, there is provided a system for detecting a scan irregularity in scanning of one or more items by a user, during a check-out process at a retail store. The system may include an image receiving module configured to receive a video stream of a scanning zone in real-time from at least one video camera, wherein the scanning zone is a region in a field of view of a scanner of the retail store. The system may further include an image processing module configured to process each image frame of the video stream for detecting one or more visual scan intervals in one or more image frames, wherein the visual scan interval is a time interval during which an item is identified in the scanning zone for scanning by the scanner. The system may further include a decision module configured to process each detected visual scan interval based on a set of pre-defined rules, wherein a processed visual scan interval includes a valid scan action, wherein the valid scan action is a user action performed for scanning an item, detect a scan irregularity in the check-out process, wherein the scan irregularity occurs when an item identified for scanning in a processed visual scan interval is absent in a list of scanned items generated by the scanner during corresponding interval, and provide an alert regarding the scan irregularity at a user computing device.

According to a second aspect of the present disclosure, there is provided a method for detecting a scan irregularity in scanning of one or more items by a user, during check-out process at a retail store. The method may include receiving a video stream of a scanning zone in real-time from at least one video camera, wherein the scanning zone is a region in a field of view of a scanner of the retail store. The method may further include processing each image frame of the video stream for detecting one or more visual scan intervals in one or more image frames, wherein the visual scan interval is a time interval during which an item is identified in the scanning zone for scanning by the scanner. The method may further include processing each detected visual scan interval based on a set of pre-defined rules, wherein a processed visual scan interval includes a valid scan action, wherein the valid scan action is a user action performed for scanning an item. The method may further include detecting a scan irregularity in the check-out process, wherein the scan irregularity occurs when an item identified for scanning in a processed visual scan interval is absent in a list of scanned items generated by the scanner during corresponding interval. The method may further include providing an alert regarding the scan irregularity at a user computing device.

According to a third aspect of the present disclosure, there is provided a computer program product for detecting a scan irregularity in scanning of one or more items by a user, during check-out process at a retail store. The computer program product includes a set of instructions, the set of instructions when executed by a processor causes the processor to receive a video stream of a scanning zone in real-time from at least one video camera, wherein the scanning zone is a region in a field of view of a scanner of the retail store, process each image frame of the video stream for detecting one or more visual scan intervals in one or more image frames, wherein the visual scan interval is a time interval during which an item is identified in the scanning zone for scanning by the scanner, process each detected visual scan interval based on a set of pre-defined rules, wherein a processed visual scan interval includes a valid scan action, wherein the valid scan action is a user action performed for scanning an item, detect a scan irregularity in the check-out process, wherein the scan irregularity occurs when an item identifiedfor scanning in a processed visual scan interval is absent in a list of scanned items generated by the scanner during corresponding interval, and provide an alert regarding the scan irregularity at a user computing device.

Various embodiments of the present disclosure provide a system and method that detects scan irregularities in a self-checkout stores, that generate an alert when there is a mismatch between the products present in a shopping basket of the user, and a scanned list of items generated by the scanner.

Referring to <FIG>, there is shown a retail environment <NUM>, wherein various embodiments of the present disclosure can be practiced. The retail environment <NUM> includes first through third self-checkout terminals 102a to 102c (hereinafter collectively referred to as self-checkout terminals <NUM>), and a central control unit <NUM>, communicatively coupled to each other through a communication network <NUM>.

The communication network <NUM> may be any suitable wired network, wireless network, a combination of these or any other conventional network, without limiting the scope of the present disclosure. Few examples may include a Local Area Network (LAN), wireless LAN connection, an Internet connection, a point-to-point connection, or other network connection and combinations thereof. In an example, the network may include a mobile communication network, for example, <NUM>, <NUM>, <NUM>, or <NUM> mobile communication network. The communication network may be coupled to one or more other networks, thereby providing coupling between a greater number of devices. Such can be the case, for example, when networks are coupled together via the Internet.

Each self-checkout terminal 102a to 102c, for example, the first check-out terminal 102a is equipped with a first scanner <NUM> for enabling a user to scan one or more items themselves, and a first user display <NUM> for enabling a user to make requisite selection and payment of one or more items. In an example, the first scanner <NUM> may be a bar code scanner for scanning bar code of an item, for identifying the item thereof. Preferably, the first scanner <NUM> is a stationary wall or table-mounted scanner, designed for check-out counters of supermarkets, and other retail stores, for scanning items placed in a scanning zone. In the context of the present disclosure, the scanning zone is an area in front of the first scanner <NUM> where the user brings up the items for scanning for the purpose of buying of those items.

Further, each self-checkout terminal 102a to 102c may be surrounded by one or more overhead video cameras for capturing scanning zone of each self-checkout terminal 102a to 102c, for example, the first self-checkout terminal 102a is surrounded by a first video camera <NUM>. The first video camera <NUM> is configured to continuously capture a video of the scanning zone, in order to facilitate detection of a scan irregularity due to a mismatch in the items brought up for scanning by the user, and the actual items scanned by the first scanner <NUM>.

The first user display <NUM> may be a touch-based display configured to receive and display one or more instructions. Few examples may include, but are not limited to, Liquid Crystal Displays (LCD) devices, Light Emitting Diode (LED)-based displays, Organic LED (OLED)-based displays devices, and micro OLED-based display devices.

In an example, the first check-out terminal 102a includes a processor (not shown) communicatively coupled to the first scanner <NUM> and the first user display <NUM>, for recording scanning of one or more items by the first scanner <NUM>, and providing instructions on the first user display <NUM> for payment of one or more scanned items. Throughout the present disclosure, the term 'processor' relates to a computational element that is operable to respond to and processes instructions that drive respective self-checkout terminal 102a to 102c. Optionally, the processor includes, but is not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or any other type of processing circuit. Furthermore, the term "processor" may refer to one or more individual processors, processing devices and various elements associated thereof.

Each of the second and third check-out terminals 102b and 102c are similar to the first check-out terminal 102a in terms of construction and functionality, therefore, they have not been explained herein again for the sake of brevity.

The central control unit <NUM> is communicatively coupled to each self-checkout terminal 102a to 102c for controlling and managing their operations thereof. In an embodiment of the present disclosure, the scanners of each self-checkout terminal 102a to 102c are communicatively coupled to the central control unit <NUM> to record the scanned content in a memory of the central control unit <NUM>, for further processing. Further, in another embodiment of the present disclosure, the video cameras present in the retail store environment <NUM> are communicatively coupled to the central control unit <NUM>.

The central control unit <NUM> may include a system <NUM> for detecting a mismatch in the items brought up for scanning in the scanning zone, and the actual items scanned by corresponding scanner in the retail environment <NUM>. Throughout the present disclosure, the term `system <NUM> relates to a structure and/or module that include programmable and/or non-programmable components configured to store, process and/or share information. Optionally, the system <NUM> includes any arrangement of physical or virtual computational entities capable of enhancing information to perform various computational tasks. In an example, the system <NUM> may include components such as memory, a processor, a network adapter and the like, to store, process and/or share information with other computing components.

<FIG> illustrates the system <NUM> for generating an alert when there is a scan irregularity in visual scan detection in the retail environment <NUM>, in accordance with an embodiment of the present disclosure. In the context of the present disclosure, "the scan irregularity" takes place, when there is a mismatch between items brought for scanning in a scanning zone by a user, and a list of scanned items generated by corresponding scanner. The system <NUM> may be implemented at the control unit <NUM>, or at each self-check out terminal <NUM>, or at both.

The system <NUM> includes an image receiving module <NUM> for receiving images captured by one or more video cameras of the retail environment <NUM>, an image processing module <NUM> for processing the captured images to detect visual scan intervals, and a decision module <NUM> for detecting valid scan intervals, and generating an alert in the event of a mismatch between item brought up for scanning, and actual items scanned by corresponding scanner of the retail environment <NUM>.

Referring back to <FIG>, the `scan action' is referred to as a user action when the user brings up the item for scanning in the scanning zone of the first scanner <NUM>, but it may or may not be successfully scanned by the first scanner <NUM>. In an example, a user may bring up an item in the scanning zone of the first scanner <NUM>, but the user may hold the item in such a way that the bar code of the item may not be visible to the bar code scanner <NUM>. In such case, the user may put the item in their shopping bag after performing the scan action, but in reality, it may not be scanned by the first scanner <NUM>, and the user may not receive a bill for that item. Therefore, detection of scan actions in image frames captured by the video cameras, are crucial in determining one or more scan irregularities in the items scanned by scanners in the retail environment <NUM>.

Referring again to <FIG>, the image processing module <NUM> includes one or more feature extraction modules that are used to extract features from a current image frame received by the image receiving module <NUM>, that indicate the occurrence of the scan action within the current image frame. The features may be designed in accordance with the action to be detected. In the context of the present disclosure, the image processing module <NUM> includes a skin tone detector <NUM>, a motion detector <NUM>, and a key-point detector <NUM>.

The skin tone detector <NUM> is configured to extract the percentage of skin pixels in the current image frame relative to the previous image frame with the intention to determine if there is a hand involved in a scan action. In the context of the present disclosure, an image pixel is labelled as a skin pixel if the image pixel is found to have a color similar to the color of human skin. While scanning a product, usually the hand of the customer in present in the scanning zone, so an increase in percentage of skin pixels in the current image frame may indicate a scan action. The skin tone detector <NUM> receives a current image frame as an input, and generates a binary map indicating whether skin tone is being detected or not in the current image frame. In an embodiment of the present disclosure, the skin pixel percentage with respect to the foreground pixels have to be smaller than a pre-defined skin pixel threshold value in each image frame. This is done in order to limit the false positives due to the passage of empty hands in the scanning area.

The motion detector <NUM> is configured to extract the percentage of motion pixels in the current image frame relative to the previous image frame with the intention to determine if there is motion involved in a scan action. In the context of the present disclosure, a current image pixel is labelled as a motion pixel if a motion has been detected in the current image pixel with respect to a previous image pixel. While scanning a product, usually the hand of the customer moves in the scanning zone, so a detected motion in the video frame, can indicate a scan action. In an embodiment of the present disclosure, the motion detector <NUM> receives a current image frame as an input, and generates a binary map indicating whether motion is being detected or not in the current image frame.

The key-point detector <NUM> is configured to indicate the presence of an object in the scanning zone. While scanning a product, usually when a new object enters into the scanning zone, the number of key-points would increase due to the new geometrical form (the product) that is present in the scene, and due to the textures that cover the product and which can create new corner points. In an embodiment of the present disclosure, the key-point detector <NUM> receives a current image frame as an input and generates a set of key-points as an output. Usually, a high number of key-points is associated with a scan action. A threshold on the temporal evolution of the number of key-points present in the scanning zone provides an estimate of a visual scan interval. In the context of the present disclosure, the visual scan interval of a product is the time interval when the product was present in the scanning zone. In an example, if the product was present from <NUM>:<NUM> am to <NUM>:<NUM>:<NUM> am in the scanning zone, then the visual scan interval of the product is <NUM> seconds.

The system <NUM> further includes a decision module <NUM> that is configured to decide if a detected visual scan interval is valid, i.e. it includes a scan action or not. The decision module <NUM> is further configured to determine whether the detected visual scan interval includes a scan action or not, based on one or more pre-defined rules, in order to regularize the detected scan intervals, cope with certain synchronization delays, and prevent eventual misclassifications.

According to a first pre-defined rule, the decision module <NUM> is configured to set a pre-defined range of scan interval, and discard the visual scan intervals that are too small with respect to the pre-defined threshold size, or are too big with respect to the pre-defined threshold size. In an embodiment of the present disclosure, the pre-defined threshold size may correspond to a normal speed of the human hand.

According to a second pre-defined rule, the decision module <NUM> is configured to set a pre-defined threshold first distance between consecutive visual scan intervals, and merge the visual scan intervals that are close with respect to the pre-defined threshold first distance. In an example, if the pre-defined threshold distance is two seconds, and the first visual scan interval is from <NUM>:<NUM> am to <NUM>:<NUM>:<NUM> am, and second visual scan interval is from <NUM>:<NUM>:<NUM> am to <NUM>:<NUM>:<NUM> am, then both the visual scan intervals may be combined, and a combined visual scan interval is formed from <NUM>:<NUM> am to <NUM>:<NUM>:<NUM> am. According to a third pre-defined rule, the decision module <NUM> is configured to keep an item from the scanned product list at a certain distance from the scan interval, in order to cope with synchronization delays between the scanner and the video camera. The distance is the time between the border of a visual scan interval and the moment given by the timestamp of the scanned item. The decision module <NUM> is configured to correct small desynchronizations that are smaller than the pause between two consecutive scans. It may happen sometimes, that the due to high latency of the network, the information about the scanned items may be delayed. In an example, the item may be actually scanned at <NUM>:<NUM> am but due to high latency, the timestamp of scanning of the item may be recorded as <NUM>:<NUM> am. So, the decision module <NUM> takes into consideration of this small delay to validate/invalidate a detected visual scan interval, so as to maintain synchronization between the scanner and camera.

According to a fourth pre-defined rule, the decision module <NUM> is configured to validate/invalidate a detected visual scan interval based on computation of a corresponding glass motion coverage. In the context of the present disclosure, the glass motion coverage is a ratio between a number of frames depicting the glass area of the scanning zone, and the number of frames having a foreground other than the glass area, in the scanning zone. The glass area is that area in the scanning zone that contains a glass that covers the scanner. The number of frames for which the glass motion coverage is computed could be around <NUM> frames for <NUM> fps video stream. This is done so as to eliminate false positives due to passage of objects in the scanning zone, but without scanning intention. It is possible to have motion in the scanning zone, but outside the glass area (for example, head passing over some part of the scanner area).

According to a fifth pre-defined rule, the decision module <NUM> is configured to validate/invalidate a detected visual scan interval based on the percentage of skin pixels with respect to the foreground pixels. The decision module <NUM> may detect absence of a visual scan in a visual scan interval, when the percentage of skin pixels is larger than a pre-defined skin pixel threshold value, as it indicates the presence of empty hands in the scanning zone. Also, it may indicate that the customer has operated the scanning zone without the intention of scanning. Thus, the decision module <NUM> invalidates corresponding visual scan interval, and does not use it for determining scan irregularity in visual scan detection.

According to a sixth pre-defined rule, the decision module <NUM> is configured to invalidate a detected visual scan interval if an increase in corresponding number of key-points is less than a key-point threshold value, as it indicates increase in number of key-points due to noise, and noise related misdetections. The decision module <NUM> invalidates corresponding visual scan interval, and does not use it for determining scan irregularity in visual scan detection.

The decision module <NUM> is configured to detect scan irregularity for each validated visual scan interval, and generate an alert when the scan irregularity is detected. The decision module <NUM> compares an item of a validated visual scan interval with a list of scanned items generated for that interval. If the item of the validated visual scan interval is not found in the list of items scanned by the scanner in that interval, then the decision module <NUM> implies that there has been an scan irregularity in visual scan detection. In an example, if the video camera detects that the product was present from <NUM>:<NUM> am to <NUM>:<NUM>:<NUM> am in the scanning zone, however it is not present in the list of items scanned by the scanner from <NUM>:<NUM> am to <NUM>:<NUM>:<NUM> am, then the decision module <NUM> infers that there is a scan irregularity in visual scan detection of the product. The decision module <NUM> is then configured to generate an alert. The alert may be sent as an instant message or as email, or as a short message service, on the user computing devices, or at the display of corresponding self-checkout terminals.

<FIG> is a flowchart illustrating a method <NUM> for generating an alert when there is a scan irregularity in visual scan detection in the retail environment <NUM>, in accordance with an embodiment of the present disclosure. The order in which the method <NUM> is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any appropriate order to carry out the method <NUM>.

At step <NUM>, a video stream of a scanning zone in real-time is received from at least one video camera, wherein the scanning zone is a region in a field of view of a scanner of the retail store. At step <NUM>, each image frame of the video stream is processed for detecting one or more visual scan intervals in one or more image frames, wherein the visual scan interval is a time interval during which an item is identified in the scanning zone for scanning by the scanner. In an embodiment of the present disclosure, the image processing includes detecting a scan action in a current image frame based on presence of a human hand in the current image frame, wherein the presence of a human hand is detected based on a percentage of skin pixels in the current image frame relative to a previous image frame. The image processing further includes detecting a scan action in the current image frame based on a movement of the human hand in the current image frame, wherein the motion is detected based on a percentage of motion pixels in the current image frame relative to the previous image frame. The image processing further includes detecting a scan action in the current image frame based on presence of an object in the human hand in the current image frame, wherein the presence of the object is determined based on a number of key-points in the scanning zone, wherein the key-point detector detects a visual scan interval for the current image frame, if a scan action is found in the current image frame.

At step <NUM>, each detected visual scan interval is processed based on a set of pre-defined rules, wherein a processed visual scan interval includes a valid scan action, wherein the valid scan action is a user action performed for scanning an item.

According to a first pre-defined rule, a pre-defined range of a visual scan interval is set, wherein the detected visual scan interval is invalidated that is outside the pre-defined range of the visual scan interval. According to a second pre-defined rule, a pre-defined threshold distance is set, wherein the two consecutive visual scan intervals are merged, if a distance between the two consecutive visual scan intervals is less than the pre-defined threshold distance. According to a third pre-defined rule, a synchronization delay between the video camera and the scanner is taken into account, while processing a visual scan interval. According to a fourth pre-defined rule, a detected visual scan interval is validated based on computation of a glass motion coverage in one or more corresponding image frames. According to a fifth pre-defined rule, a detected visual scan interval is validated based on a percentage of skin pixels with respect to foreground pixels in corresponding image frames. According to a sixth pre-defined rule, a detected visual scan interval is invalidated if a number of key-points in corresponding image frames is less than a key-point threshold value.

At step <NUM>, a scan irregularity is detected in the check-out process, wherein the scan irregularity occurs when an item identified for scanning in a processed visual scan interval is absent in a list of scanned items generated by the scanner during corresponding interval. In an example, a user may bring up an item in the scanning zone of the scanner, but the user may hold the item in such a way that the bar code of the item may not be visible to the bar code scanner. In such case, the user may put the item in their shopping bag after performing the scan action, but in reality, it may not be scanned by the scanner, and the user may not receive a bill for that item. This leads to scan irregularity.

At step <NUM>, an alert is provided regarding the scan irregularity at a user computing device. The alert may be sent as an instant message or as email, or as a short message service, on the user computing devices, or at the display of corresponding self-checkout terminals.

Claim 1:
A method for detecting a scan irregularity in scanning of one or more items by a user, during check-out process at a retail store, the method comprising:
receiving (<NUM>) a video stream of a scanning zone in real-time from at least one video camera, wherein the scanning zone is a region in a field of view of a scanner of the retail store;
processing (<NUM>) each image frame of the video stream to detect one or more visual scan intervals in one or more image frames, wherein the visual scan interval is a time interval during which an item is identified in the scanning zone for scanning by the scanner;
processing (<NUM>) each detected visual scan interval to determine whether the detected visual scan interval is valid, wherein said processing is based on a set of pre-defined rules and wherein a processed visual scan interval includes a scan action, wherein the scan action is a user action performed for scanning an item;
detecting a scan irregularity in the check-out process for each detected valid visual scan interval and providing an alert regarding the scan irregularity at a user computing device wherein detecting the scan irregularity comprises
comparing the item associated with the valid visual scan interval with a list of scanned items generated by the scanner during the time interval and wherein if the item is absent in the list of scanned items, detecting a scan irregularity in the check-out process,
characterized in that
a first pre-defined rule of the set of predefined rules comprises validating a detected visual scan interval based on computation of a glass motion coverage in one or more corresponding image frames, wherein the glass motion coverage is a ratio between a number of frames depicting a glass area of the scanning zone and a number of frames having a foreground other than the glass area, in the scanning zone, wherein the glass area is that area in the scanning zone that contains a glass that covers the scanner.