ABNORMAL SHOPPING BEHAVIOR DETECTION METHOD AND APPARATUS FOR INTELLIGENT SHOPPING CART, AND SHOPPING CART

An abnormal shopping behavior detection method and apparatus for an intelligent shopping cart, and the shopping cart are disclosed in the present disclosure. The method includes: acquiring code scanning data of commodities and video-frame image data of a basket area during a shopping behavior of a user; segmenting the commodities in an image of each frame in the video-frame image data of the basket area, to obtain image data of each target commodity in the basket area; tracking a trajectory of each target commodity; determining a motion direction and a motion distance of the tracked trajectory of each target commodity; determining an indicative state of whether each target commodity is put in or taken out of the shopping cart; and detecting an abnormal shopping behavior of the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Chinese Patent Application No. 202410338478.9, filed on Mar. 22, 2024, which is hereby incorporated by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to the technical field of self-service shopping, and particularly to an abnormal shopping behavior detection method and apparatus for an intelligent shopping cart, and the shopping cart.

BACKGROUND

This section is intended to provide the background or the context for the embodiments of the present disclosure set forth in the claims. The description here is not recognized as the prior art just because it is included in this section.

With the continuous development of the technologies such as Internet of Things, artificial intelligence, big data analysis, mobile payment and intelligent hardware, the industry of supermarket is also constantly transforming to the directions of intellectualization and digitalization, which enables supermarkets to better meet the needs of the customers, improve the shopping experience of the customers, optimize the business operations and increase the sales, thereby further expanding the market influence of supermarkets. In this process, the shopping settlement is a crucial part of the shopping process, and the intelligent shopping cart appears in the supermarket environment under this background, which brings a brand-new shopping experience and operation mode to customers and merchants.

Generally, the intelligent shopping cart has a series of advanced functions, including automatic scanning and settlement, navigation and commodity positioning, commodity recognition and weighing, personalized recommendation and advertising, data analysis, inventory management, etc. For example, in terms of self-service code scanning and settlement, many intelligent shopping carts are equipped with built-in scanners, built-in weighing devices and built-in settlement channels, which allow a customer to scan a barcode of a commodity in real time during shopping, to check a total price in real time during shopping, and to complete a payment through a self-service settlement system of the shopping cart before leaving the store. This self-service shopping process simplifies the necessary steps of the traditional shopping and payment, so that the customer does not need to interact with a cashier at a checkout counter, thereby reducing the probability of mistakes and improving the shopping experience. Meanwhile, for a merchant, the cashiers can be decreased and the operation cost can be reduced.

Meanwhile, it is necessary to detect the users' abnormal shopping behaviors when the intelligent shopping cart is actually used in the supermarket scenario. Since the intelligent shopping cart is in self-service use by the consumers, it is inevitable that there will be some behaviors attempting not to follow the normal procedures, and the intelligent shopping cart should discover, and prompt or prevent these behaviors. The existing solution for the intelligent shopping cart to detect the users' abnormal shopping behaviors is not accurate enough.

SUMMARY

The embodiments of the present disclosure provide an abnormal shopping behavior detection method for an intelligent shopping cart, to improve the accuracy of detecting the abnormal shopping behavior for the intelligent shopping cart, the method including:

The embodiments of the present disclosure further provide an abnormal shopping behavior detection apparatus for an intelligent shopping cart, so as to improve the accuracy of an abnormal shopping behavior detection for the intelligent shopping cart, the apparatus including:

The embodiments of the present disclosure further provide an intelligent shopping cart, to improve the accuracy of an abnormal shopping behavior detection for the intelligent shopping cart, the intelligent shopping cart including:

The embodiments of the present disclosure further provide a computer device, including a memory, a processor and a computer program stored in the memory and executable on the processor, and when executing the computer program, the processor implements the aforementioned abnormal shopping behavior detection method for the intelligent shopping cart.

The embodiments of the present disclosure further provide a computer-readable storage medium, storing a computer program, and when executed by a processor, the computer program implements the aforementioned abnormal shopping behavior detection method for the intelligent shopping cart.

The embodiments of the present disclosure further provide a computer program product, comprising a computer program, and when executed by a processor, the computer program implements the aforementioned abnormal shopping behavior detection method for the intelligent shopping cart.

In the embodiments of the present disclosure, during working, the abnormal shopping behavior detection solution for the intelligent shopping cart includes: acquiring code scanning data of commodities and video-frame image data of a basket area during a shopping behavior of a user; segmenting the commodities in an image of each frame in the video-frame image data of the basket area, to obtain image data of each target commodity in the basket area in the image of each frame; tracking a trajectory of each target commodity according to the image data of each target commodity in the basket area among images of a plurality of frames, to obtain trajectory tracking data of each target commodity; determining a motion direction and a motion distance of the tracked trajectory of each target commodity according to the trajectory tracking data of each target commodity; determining an indicative state of whether each target commodity is put in or taken out of the shopping cart, according to the motion direction and the motion distance of the tracked trajectory of each target commodity and a preset commodity motion distance threshold; and detecting an abnormal shopping behavior of the user, according to the code scanning data, the trajectory tracking data of each target commodity in the basket area, and the indicative state of whether each target commodity is put in or taken out of the shopping cart.

According to the embodiments of the present disclosure, the intelligent shopping cart has the function of abnormal shopping behavior detection in addition to the functions of self-service checkout, intelligent shopping and the like, so that the abnormal shopping behavior of the user can be detected by combining the code scanning data and the collected video-frame image data of the basket area. The abnormal shopping behavior detection solution for the intelligent shopping cart according to the embodiment of the present disclosure has the following advantageous technical effects:

Firstly, the embodiments of the present disclosure segment the commodities in the image of each frame in video-frame image data of a basket area, and track a trajectory of a target commodity, thereby avoiding the problem that the detection result is affected since the commodities are mutually stacked and shielded, and improving the accuracy of the abnormal shopping behavior detection for the intelligent shopping cart.

Secondly, the embodiments of the present disclosure determine a motion direction and a motion distance of each target commodity according to trajectory tracking data of each target commodity; accurately determine an indicative state of whether each target commodity is put in or taken out of the shopping cart according to the direction and the distance, and then accurately detect an abnormal shopping behavior of the user according to the state.

To sum up, the embodiments of the present disclosure can accurately detect the abnormal shopping behavior for the intelligent shopping cart.

DESCRIPTION OF THE EMBODIMENTS

In order that the objectives, technical solutions and advantages of the embodiments of the present disclosure are clearer, the embodiments of the present disclosure will be further illustrated in detail below with reference to the drawings. Here, the exemplary embodiments of the present disclosure and the description thereof are used to illustrate the present disclosure, but are not intended to limit the present disclosure.

The acquisition, storage, use, processing, etc. of data in the technical solutions of the present disclosure comply with the relevant provisions of laws and regulations.

The embodiments of the present disclosure provide an abnormal shopping behavior solution for an intelligent shopping cart, which detects an abnormal shopping behavior based on intelligent hardware. The whole hardware device may be flexibly mounted and detached from the shopping cart, including a code scanner, a calculation and interaction device, a visual camera, a battery module, etc., which are integrated therein. On the basis of the shopping procedures of self-service code scanning, page interaction (user interaction) and self-checkout, it focuses on discovering and preventing some potential abnormal shopping behaviors, such as putting in a commodity without code scanning, putting in commodity B while commodity A is scanned, or interfering with the sensor to put in a commodity. The abnormal shopping behavior solution for the intelligent shopping cart is introduced in detail below.

FIG. 1 illustrates a flowchart of an abnormal shopping behavior detection method for an intelligent shopping cart according to an embodiment of the present disclosure. As illustrated in FIG. 1, the method includes:

In the embodiments of the present disclosure, during working, the abnormal shopping behavior detection method for the intelligent shopping cart includes: acquiring code scanning data of commodities and video-frame image data of a basket area during a shopping behavior of a user; segmenting the commodities in an image of each frame in the video-frame image data of the basket area, to obtain image data of each target commodity in the basket area in the image of each frame; tracking a trajectory of each target commodity according to the image data of each target commodity in the basket area among images of a plurality of frames, to obtain trajectory tracking data of each target commodity; determining a motion direction and a motion distance of the tracked trajectory of each target commodity according to the trajectory tracking data of each target commodity; determining an indicative state of whether each target commodity is put in or taken out of the shopping cart, according to the motion direction and the motion distance of the tracked trajectory of each target commodity and a preset commodity motion distance threshold; and detecting an abnormal shopping behavior of the user, according to the code scanning data, the trajectory tracking data of each target commodity in the basket area, and the indicative state of whether each target commodity is put in or taken out of the shopping cart.

According to the embodiment of the present disclosure, the intelligent shopping cart has the function of abnormal shopping behavior detection in addition to the functions of self-service checkout, intelligent shopping and the like, so that the abnormal shopping behavior of the user can be detected by combining the code scanning data and the collected video-frame image data of the basket area. The abnormal shopping behavior detection method for the intelligent shopping cart according to the embodiment of the present disclosure has the following advantageous technical effects:

Firstly, the embodiment of the present disclosure segments the commodities in the image of each frame in video-frame image data in a basket area, and tracks a trajectory of a target commodity, thereby avoiding the problem that the detection result is affected since the commodities are mutually stacked and shielded, and improving the accuracy of the abnormal shopping behavior detection for the intelligent shopping cart.

Secondly, the embodiment of the present disclosure determines a motion direction and a motion distance of each target commodity according to trajectory tracking data of each target commodity; accurately determines an indicative state of whether each target commodity is put in or taken out of the shopping cart according to the direction and the distance, and then accurately detects an abnormal shopping behavior of the user according to the state.

To sum up, the embodiment of the present disclosure can accurately detect the abnormal shopping behavior for the intelligent shopping cart.

The abnormal shopping behavior detection method for the intelligent shopping cart according to the embodiment of the present disclosure will be described in detail below.

In order to facilitate the understanding of the implementation of the present disclosure, firstly, the overall architecture of the intelligent shopping cart mentioned in the embodiment of the present disclosure is introduced.

As illustrated in FIGS. 4 and 5, the embodiments of the present disclosure provide an integrated hardware device that can be flexibly mounted and detached. The necessary hardware of an intelligent shopping cart, such as a code scanner 10 in FIG. 5, a calculation and interaction device (including a calculation module and an interaction module in FIG. 4) 30, and a battery module in FIG. 4 are included, and on this basis, a plurality of visual cameras 20 (collection devices) are added to capture video-frame images of a shopping behavior process of a shopper. The run software (that is: the abnormal shopping behavior detection method for the intelligent shopping cart according to the embodiments of the present disclosure, and the method be implemented by a calculation module in FIG. 4, which is an abnormal shopping behavior detection apparatus for the intelligent shopping cart) combines the data of the code scanner and the collected image data to analyze whether the shopping process is normal and whether there are some potential abnormal shopping behaviors, and presents the generated relevant prompt information on an interaction device (e.g., the interaction module in FIG. 4). Specifically:

1. The code scanner may be a conventional code scanning gun or code scanner which is capable of scanning and recognizing a barcode on a commodity, being connected with the interaction device, and sending a recognition result (commodity identification, such as a commodity name, barcode information, commodity price information, code scanning moment, etc.) to the calculation and interaction device.

2. As illustrated in FIG. 5, the visual camera includes one or more RGB (Red, Green, Blue) cameras; and the visual camera is mounted above a periphery of a shopping cart basket and above a periphery of a code scanning area, and connected to the calculation and interaction device for data transmission. In this embodiment, the coverage area of the visual cameras is classified into two types: the camera (basket area collection device) mounted above the periphery of the shopping cart basket has a visual range set as a basket area 40; and the camera (code scanning area collection device) mounted above the periphery of the code scanner has a visual range set as a code scanning area 50.

3. The calculation and interaction device is an electronic device with a touch screen, which is a computing and processing unit with certain computility (e.g., the calculation unit in FIG. 4), mountable on a handle of the shopping cart and connectable to the code scanner and the visual camera. The operating system and software may be run on it to present the data on the interaction screen (the “interaction module” in FIG. 4), so as to realize the basic functions such as self-service code scanning, page interaction and self-service checkout. The data transmitted by the code scanner and the visual camera may be processed by it, and the data or the result may be presented on the interaction screen, so as to discover and prevent some potential abnormal shopping behaviors.

4. The battery module supplies power to the whole device. The battery module may be charged when the whole device is detached from the shopping cart.

Next, the abnormal shopping behavior detection method for the intelligent shopping cart according to the embodiments of the present disclosure will be described in detail with reference to FIGS. 4 and 5.

I. Acquisition of Video-Frame Images of a Shopping Process

At the beginning of shopping, the visual camera is started to acquire the video-frame images of the shopping process.

II. Reasoning Operation on the Image Data.

The calculation module in FIG. 4 processes received image data of each frame using a computer vision algorithm, a machine learning algorithm, a deep learning algorithm, etc. according to the received image data.

1. For the code scanning area, detecting and recording the commodity at a moment of code scanning.

At a moment when a barcode is scanned by the code scanner, the commodity in the code scanning area is detected, and the commodity image data, i.e., the video-frame image data of the code scanning area, is saved (the image data may be used in the following step of judging a wrong scanning behavior). As shown in the above step 301, this step acquires the code scanning data of the commodity during the shopping behavior of the user by the scanner.

2. For the basket area, judging whether the target commodity is put in or taken out of the shopping cart, i.e., performing steps 302 to 305.

If the RGB camera is configured, it is judged whether the target commodity is put in or taken out of the shopping cart according to the trajectory of the target commodity, based on a target tracking algorithm. The specific detailed steps are as follows:

For the video-frame image data, commodities in the image are detected or segmented using target detection algorithm model or image segmentation algorithm model, so as to output a target bounding box B1 (first target bounding box) or a target mask A1 (first target mask), which includes the commodity. In this embodiment, the target detection algorithm model and the segmentation algorithm model are models trained by the deep learning algorithm with training data, including but not limited to a convolutional neural network model or a transformer network model. The target detection algorithm model uses algorithms including but not limited to those of one stage or two stages, such as SSD (Single Shot MultiBox Detector) and yolo series algorithms, and the segmentation algorithm model includes but is not limited to yolact, solo, yolo, segmentation anything models, etc.

During implementation, the target bounding box may be a rectangular bounding box, which can completely include one target commodity, and the mask of the commodity is a pixel block that can surround the target commodity. As for when to use the target bounding box and when to use the mask, it depends on the algorithm model for commodity detection. If the target detection model is adopted, the target bounding box is used, and if the segmentation model is adopted, the mask is used.

Different screening and filtering of different levels are carried out for the target commodities, including the following two items:

a. Filtering of an Interferent (Interference) Detected by Mistake

In order to avoid the influence on the subsequent judgment caused by the false detection or false segmentation of any non-commodity target by the algorithm model in step 1), it is necessary to check and filter the detected or segmented target commodities. The specific method is as follows:

Firstly, the interfering target are detected or segmented. For the video-frame image data, target detection or image segmentation algorithm model is used to detect or segment an interferent (e.g., hand, arm, mobile phone, etc.) in the image, so as to output a target bounding box B2 (second target bounding box) or a target mask A2 (second target mask, which is a mask of the interferent and a pixel block being capable of surrounding the interferent), which includes the interferent. Either the mask of the interferent or the target bounding box can completely include a target interferent.

Next, the commodity is checked. The target detection bounding box B1 of the commodity or the target mask A1 of the commodity in step 1) is used to calculate an intersection over union with the target detection bounding box B2 of the interferent or the target mask A2 of the interferent, i.e., a ratio “a” of overlapping areas of two targets. If the ratio “a” of overlapping areas is greater than a preset threshold, the target detection bounding box B1 or the target mask A1 at this time is considered as an interferent and should be eliminated, otherwise, the target detection bounding box B1 or the target mask A1 at this time is considered as a real commodity and recorded in a commodity set S (which records the bounding box information of each commodity or the mask information of each commodity and the image information of each commodity).

During implementation, in step 302, the commodities in the image of each frame of the video-frame image data of the basket area are segmented, to obtain position information and image data of each target commodity in the basket area in the image of each frame. In step 303, the trajectory of each target commodity is tracked according to the position information and the image data of each target commodity in the basket area in the image of a plurality of frames, to obtain trajectory tracking data of each target commodity.

As can be seen from the above, in one embodiment, if a collection device of the video-frame image data of the basket area is an RGB camera, segmenting the commodities in the image of each frame in the video-frame image data of the basket area, to obtain image data of each target commodity in the basket area in the image of each frame includes:

During implementation, by calculating the intersection over union of the target mask of the commodity and the target mask of the interferent, the segmented and detected commodities are filtered, so as to avoid the influence on the judgment of the subsequent abnormal shopping behavior caused by the false segmentation of the interferents, and improve the detection accuracy of the subsequent abnormal shopping behavior.

As can be seen from the above, in one embodiment, if the collection device of the video-frame image data of the basket area is an RGB camera, the commodities in the image of each frame of the video-frame image data of the basket area are subject to target detection, to obtain the position information and the image data of each target commodity in the basket area in the image of each frame, which may further include:

During implementation, by calculating the intersection over union of the target mask of the commodity and the target masks of the interferent, the commodities after the target detection are filtered, so as to avoid the influence on the judgment of the subsequent abnormal shopping behavior caused by the false target detection of the interferents, and improve the detection accuracy of the subsequent abnormal shopping behavior.

As can be seen from the above, in one embodiment, the abnormal shopping behavior detection method for the intelligent shopping cart further includes: determining that the commodity included in the first target mask is a real commodity and remain the commodity in the target commodity set, when the ratio of overlapping areas is less than the preset overlapping ratio threshold.

B. Filtering Out a Commodity in Motion

In order to more accurately track the commodity put in or taken out of the shopping chart and calculate the trajectory thereof, the embodiments of the present disclosure provide a method for screening out a commodity in motion.

As can be seen from the above, in one embodiment, the abnormal shopping behavior detection method for the intelligent shopping cart further includes:

The specific method for screening out the commodity in motion is as follows:

a). Calibrating Pixels in Motion in the Image

For the video frame data, a foreground extraction algorithm or a background subtraction algorithm is adopted to calibrate an area of motion targets, and extracts a binary image mask A3 corresponding to the calibrated area as a foreground mask A3 of the image of each frame. In the embodiment, the foreground extraction algorithm includes, but is not limited to the background subtraction method of a gaussian Mixture model, a frame difference method, an optical flow method, a KNN (K-Nearest Neighbors) algorithm, etc.

b) Selecting the Motion Target

The target mask A1 is used, and a mask A1′ at the same position as A1 is cropped (cropped mask) on the mask A3 (binary image mask) to calculate an intersection over union of A1′ and A1, i.e., a ratio “a” of overlapping areas of the two masks. If the ratio “a” of overlapping areas is greater than the preset threshold, the commodity corresponding to the mask A1 is considered as in motion and incorporated into the commodity set S (which records the bounding box information of each commodity or the mask information of each commodity and the image information of each commodity).

As can be seen from the above, in one embodiment, filtering out the target commodity in motion from the target commodity set and adding the target commodity in motion into the final target commodity set includes:

So far, step 1) performs target detection or segmentation of commodities, and step 2) introduces the screening and filtering of the target commodities in step 302 and the exemplary solution thereof. It is more practical to adopt a machine learning method or a deep learning method to detect, segment and track the commodities in actual application scenarios, which can record and analyze the process in which the commodities are put in or taken out of the shopping cart, and count the number of the commodities put in or taken out, thereby avoiding the situation that the commodities are mutually stacked and shielded in the judgment time difference and the specific number of the commodities put in or taken out of the shopping cart cannot be discovered.

For the commodities in the commodity set S, the information of each commodity is used to track the trajectory of the commodity. The methods used include, but are not limited to, the following:

a. Tracking Using a Multi-Target Tracking Algorithm

For each target commodity in the commodity set S, information of the target bounding box or description data of the key point is taken as input data of multi-target tracking to finally output information of the target tracked trajectory. The target tracking used is a conventional multi-target tracking algorithm, including:

B. Using a Reid Algorithm for Tracking Calculation.

The target commodities are extracted as the characteristic description data, and the characteristic description data of the target commodities in respective frames is compared to determine similarities therebetween, and the characteristic description data with a similarity within a threshold range is matched as a trajectory id (identification) of the same target.

C. Using a Visual Tracking Algorithm

A model trained with training data using a deep learning method is adopted. The algorithm model can process the information of target bounding box or description data of the key point and time sequence information, and directly output the information of target tracked trajectory.

In this embodiment, a different trajectory id is assigned to each commodity in the commodity set S, and the number of commodities put in or taken out of the shopping cart can be determined according to the number of ids.

In this embodiment, the machine learning model includes but is not limited to an XGboost regression model. After being trained, the machine learning model receives the input trajectory point set information and may output information such as the motion direction and a motion difference.

As can be seen from the above, in one embodiment, determining the motion direction and the motion distance of the tracked trajectory of each target commodity according to the trajectory tracking data of each target commodity includes:

During implementation, by determining the motion direction and the motion distance of the tracked trajectory of each target commodity, the embodiment can improve the efficiency and accuracy of the trajectory calculation, and further improve the accuracy and efficiency of the abnormal shopping behavior detection.

Based on the direction and distance of the target trajectory, if the trajectory of a commodity is from outside the shopping cart into the shopping cart and the motion distance reaches a threshold, it is determined that the commodity is put in the shopping cart; and if the trajectory of a commodity is from inside the shopping cart to outside the shopping cart and the motion distance reaches a threshold (preset commodity motion distance threshold), it is determined that the commodity is taken out of the shopping cart.

As can be seen from the above, in one embodiment, determining the indicative state of whether each target commodity is put in or taken out of the shopping cart, according to the motion direction and the motion distance of the tracked trajectory of each target commodity and a preset commodity motion distance threshold includes:

III. A Loss Prevention Logic Module Performs Data Processing, i.e., the Above Step 306.

1. Receiving and Summarizing all the Data

The logic processing module receives detection tracking data output from an image processing module, code scanning data of the code scanner, and an operation instruction of an interaction interface (e.g., an instruction of commodity item deletion or member log-in);

2. Judging Abnormal Behaviors According to the Fused Data

1) Aligning the Time of Code Scanning Data and the Number of Code Scanning, and Judging a Missed Scanning Behavior

For the commodity put in the basket area, the tracking data is detected, and the code scanning data within a threshold time t before a moment of putting in (placement time) is collected. Within the threshold time range, if n new commodities in the detection and recognition data of the basket area (as mentioned above, the number of the commodities is determined according to the number of the tracked trajectories of the commodities) are put in the shopping cart, while the number of the newly received scanning code data is less than n, it is determined that there is a “missed scanning behavior” at this time.

In one embodiment, the trajectory tracking data of each target commodity includes: determining the number of commodities according to the number of trajectories; detecting the abnormal shopping behavior of the user, according to the code scanning data, the trajectory tracking data of each target commodity in the basket area, and the indicative state of whether each target commodity is put in or taken out of the shopping cart includes:

acquiring, for the trajectory tracking data of each target commodity in the basket area with the state of being put in the shopping cart, code scanning data within a preset time threshold range before placement time, and within the preset time threshold range, determining, if the number of commodities of the scanning code data is less than the number of commodities put in the shopping cart among the trajectory tracking data of the basket area, a missed scanning behavior by the user.

2) Judging a Wrong Scanning Behavior

a. Recognizing the Commodities Put in

Each of n commodities put in among the detection tracking data of the basket area is recognized, so as to determine a commodity identifier, such as a commodity name or barcode information, which is corresponding to each commodity.

In this embodiment, the method of commodity recognition includes but is not limited to: using a deep learning algorithm model for classification recognition, using a retrieval calculation method for retrieval recognition, and outputting a result with a highest confidence as an recognition result.

B. Comparing the Code Scanning Results with the Commodity Recognition Results

During implementation, there may be two ways to judge the wrong scanning behavior:

Implementation 1: matching the recognition results of the n commodities put in with n pieces of code scanning data; if the results of matching are consistent, the code scanning data is cancelled with the recognition data one by one (the code scanning data can be in one-to-one correspondence with the commodity information corresponding to the recognition data); if the n pieces of code scanning data are cancelled with n pieces of recognition data one by one, a “normal behavior” is determined; and if it is finally discovered that there is any commodity recognition data that cannot be cancelled, a “wrong scanning behavior” is determined to be existed at this time.

As can be seen from the above, in one embodiment, detecting the abnormal shopping behavior of the user, according to the code scanning data, the trajectory tracking data of each target commodity in the basket area, and the indicative state of whether each target commodity is put in or taken out of the shopping cart includes:

During implementation, based on a process of putting in or taking out the commodities and the number of the commodities, and in combination with the code scanning data of the code scanner and the result data of commodity recognition, the wrong scanning behavior can be determined more accurately. For example, when the code scanning and the putting in the commodity are out of order, the logical method of one-to-one cancellation can show a better judgment result and a friendlier experience.

Implementation 2: in addition to judging the wrong scanning behavior using the method of commodity recognition, the detected commodity image in the code scanning area may also be used. The specific method is as follows: within a time range t, the image data of the commodity in the detection and identification data of the code scanning area is acquired, and meanwhile, the image of the commodity put in the shopping cart captured in the basket area is acquired, so as to calculate a similarity of the commodity images acquired in two different areas, including but not limited to comparisons of: a similarity of commodity appearance characteristics, a color similarity, a texture similarity, a size difference, etc. If there is a large difference therebetween, a “wrong scanning behavior” is determined at this time.

As can be seen from the above, in one embodiment, the abnormal shopping behavior detection method of the intelligent shopping cart further includes: acquiring video-frame image data of the code scanning area during the shopping behavior of the user;

If the shopping cart deletes n (the number of commodities deleted from the interaction screen) commodities from the interaction screen within the threshold time range, but the number of commodities taken out of the shopping chart, which are included in the detection and identification data of the basket, is less than n, a “over-picking behavior” is determined.

As can be seen from the above, in one embodiment, the trajectory tracking data of each target commodity includes determining the number of commodities according to the number of trajectories; detecting the abnormal shopping behavior of the user, according to the code scanning data, the trajectory tracking data of each target commodity in the basket area, and the indicative state of whether each target commodity is put in or taken out of the shopping cart includes:

IV. The Further Exemplary Solutions of the Embodiments of the Present Disclosure Will be Introduced Below.

1. Firstly, it is introduced the judgement of a shielding behavior and an abnormal action behavior of the user through the recognition of an action of the shopper according to an action type recognition result.

Discovering and determining the occurrence of an action in the basket area.

During implementation, in addition to the determination of abnormal shopping behaviors such as the missed scanning behavior, the wrong scanning behavior and the over-picking behavior, it is also possible to detect the abnormal shopping behaviors according to the action recognition result.

During implementation, if the visual image of the basket area is abnormal, the image processing module infers that the camera is shielded by a commodity which is intentionally shielded or stacked high, and a “shielding behavior” is determined at this time to prevent the shopper from putting in the commodity under the condition that the camera is shielded. If there is any abnormal action in the action identification data of the basket area, such as stacking and shielding the commodity, changing the commodity package, etc., a “shopping behavior with an abnormal action on the commodity” is determined.

As can be seen from the above, in one embodiment, the method further includes:

As can be seen from the above, in one embodiment, the abnormal shopping behavior detection method for the intelligent shopping cart further includes:

During implementation, the solution of reporting and prompting the detected abnormal shopping behavior automatically prompts the irregular behavior of the shopper in time, allows the shopper to make a correction, reduces the manual intervention in the supermarket and improves the user's experience.

If the shopper clicks an interaction button and attempts to enter a payment procedure, it is checked whether there is still any abnormal behavior not eliminated in a shopping list:

During implementation, the solution of entering the payment procedure and checking the shopping list ensures the smooth and safe shopping with the shopping cart.

The abnormal shopping behavior method for the intelligent shopping cart according to the embodiment of the present disclosure has the following advantages:

1) An integrated hardware device is provided, which can be flexibly mounted on and detached from the shopping cart, to achieve the function of detecting the abnormal shopping behavior of the user, in addition to functions of self-service checkout, intelligent shopping, etc. In particular, based on the hardware of the conventional intelligent shopping cart, several visual cameras (especially cameras in the code scanning area) are added to cover the basket area of the shopping cart or the code scanning area, to capture the shopping behavior of the shopper. There is provided software executable on the integrated device. The software can combine the data of the code scanner and the collected image data to analyze whether the shopping process is normal and whether there are some potential theft behaviors, and display prompt information on the interaction device.

2) It is more practical to adopt a machine learning method or a deep learning method to detect, segment and track the commodities or recognize the actions in actual application scenarios, which can record and analyze the process in which the commodities are put in or taken out of the shopping cart, and count the number of the commodities put in or taken out, thereby avoiding the situation that the commodities are mutually stacked and shielded in the judgment time difference and the specific number of the commodities put in or taken out of the shopping cart cannot be discovered.

3) Based on the process of putting in or taking out the commodities and the number of the commodities, and in combination with the code scanning data of the code scanner and the result data of commodity recognition, the wrong scanning behavior can be judged more accurately. For example, when the code scanning and the putting in the commodity are out of order, the logical method of one-to-one cancellation can show a better judgment result and a friendlier experience.

To sum up, the abnormal shopping behavior method for the intelligent shopping cart according to the embodiments of the present disclosure achieve the following:

The embodiments of the present disclosure further provide an abnormal shopping behavior detection apparatus for an intelligent shopping cart, as described in the following embodiments. Since the principle of the apparatus to solve the problem is similar to that of the abnormal shopping behavior detection method for the intelligent shopping cart, the implementations of the apparatus may refer to those of the abnormal shopping behavior detection method for the intelligent shopping cart, and the repeated contents are omitted here.

FIG. 2 illustrates a structural diagram of an abnormal shopping behavior detection apparatus for an intelligent shopping cart according to an embodiment of the present disclosure. As illustrated in FIG. 2, the apparatus includes:

In one embodiment, if a collection device for the video-frame image data of the basket area is an RGB camera, the segmentation processing unit is specifically configured to:

In one embodiment, the segmentation processing unit is further configured to determine that the commodity included in the first target mask is a real commodity and remain the commodity in the target commodity set, when the ratio of overlapping areas is less than the preset overlapping ratio threshold.

In one embodiment, the abnormal shopping behavior detection apparatus further includes:

In one embodiment, the screening processing unit is specifically configured to:

In one embodiment, the trajectory processing unit is specifically configured to:

In one embodiment, the state determination unit is specifically configured to:

In one embodiment, the trajectory tracking data of each target commodity includes: determining the number of commodities according to the number of trajectories; the detection unit is specifically configured to:

In one embodiment, the detection unit is specifically configured to:

In one embodiment, the abnormal shopping behavior detection apparatus for the intelligent shopping cart may further include a code scanning area image acquisition unit configured to acquire the video-frame image data of a code scanning area during the shopping behavior of the user;

In one embodiment, the trajectory tracking data of each target commodity includes: determining the number of commodities according to the number of trajectories; the detection unit is specifically configured to:

In one embodiment, the abnormal shopping behavior detection apparatus for the intelligent shopping cart further includes:

In one embodiment, the abnormal shopping behavior detection apparatus further includes a prompt processing unit configured to:

The embodiments of the present disclosure further provide an intelligent shopping cart, as described in the following embodiments. Since the principle of the intelligent shopping cart to solve the problem is similar to that of the abnormal shopping behavior detection method for the intelligent shopping cart, the implementations of the intelligent shopping cart may refer to those of the abnormal shopping behavior detection method for the intelligent shopping cart, and the repeated contents are omitted here.

FIG. 3 illustrates a schematic diagram of a connection structure of an intelligent shopping cart according to an embodiment of the present disclosure. As illustrated in FIG. 3, the intelligent shopping cart includes:

During implementation, as illustrated in FIGS. 4 and 5, the intelligent shopping cart according to the embodiments of the present disclosure may further include a battery module, a visual camera and other components, as detailed in the above introduction to the overall architecture of the intelligent shopping cart.

The embodiments of the present disclosure further provide a computer device, including a memory, a processor and a computer program stored in the memory and executable on the processor, and when executing the computer program, the processor implements the abnormal shopping behavior detection method for the intelligent shopping cart.

The embodiments of the present disclosure further provide a computer-readable storage medium storing a computer program, and when executed by a processor, the computer program implements the abnormal shopping behavior detection method for the intelligent shopping cart.

The embodiments of the present disclosure further provide a computer program product including a computer program, and when executed by a processor, the computer program implements the abnormal shopping behavior detection method of the intelligent shopping cart.

According to the embodiment of the present disclosure, the intelligent shopping cart has the function of abnormal shopping behavior detection in addition to the functions of self-service checkout, intelligent shopping and the like, so that the abnormal shopping behavior of the user can be detected by combining the code scanning data and the collected video-frame image data of the basket area. The abnormal shopping behavior detection solution for the intelligent shopping cart according to the embodiment of the present disclosure has the following advantageous technical effects:

Firstly, the embodiment of the present disclosure segments the commodities in the image of each frame in video-frame image data in a basket area, and tracks a trajectory of a target commodity, thereby avoiding the problem that the detection result is affected since the commodities are mutually stacked and shielded, and improving the accuracy of the abnormal shopping behavior detection for the intelligent shopping cart.

Secondly, the embodiment of the present disclosure determines a motion direction and a motion distance of each target commodity according to trajectory tracking data of each target commodity; accurately determines an indicative state of whether each target commodity is put in or taken out of the shopping cart according to the direction and the distance, and then accurately detects an abnormal shopping behavior of the user according to the state.

To sum up, the embodiment of the present disclosure can accurately detect the abnormal shopping behavior for the intelligent shopping cart.

Those skilled in the art should appreciate that any embodiment of the present disclosure can be provided as a method, a system or a computer program product. Therefore, the present disclosure can take the form of a full hardware embodiment, a full software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer usable storage mediums (including, but not limited to, a magnetic disc memory, CD-ROM, optical storage, etc.) containing therein computer usable program codes.

The present disclosure is described with reference to a flow diagram and/or a block diagram of the method, apparatus (system) and computer program product according to the embodiments of the present disclosure. It shall be appreciated that each flow and/or block in the flow diagram and/or the block diagram and a combination of flows and/or blocks in the flow diagram and/or the block diagram can be realized by computer program instructions. Those computer program instructions can be provided to a general computer, a dedicated computer, an embedded processor or a processor of other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce device for realizing specified functions in one or more flows in the flow diagram and/or one or more blocks in the block diagram.

These computer program instructions may also be stored in a computer readable memory capable of guiding the computer or other programmable data processing devices to work in a particular manner, so that the instructions stored in the computer readable memory can produce manufacture articles including an instructing device which realizes function(s) specified in one or more flows in the flow diagram and/or one or more blocks in the block diagram.

These computer program instructions may also be loaded onto the computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to produce a processing realized by the computer, thus the instructions executed on the computer or other programmable devices provide step(s) for realizing function(s) specified in one or more flows in the flow diagram and/or one or more blocks in the block diagram.

The above specific embodiments further explain the objectives, technical solutions and advantageous effects of the present disclosure in detail. As should be understood, those described above are merely specific embodiments of the present disclosure, and are not intended to limit the protection scope of the present disclosure. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure should be included in the protection scope of the present disclosure.