Information processing method, information processing device, and recording medium

An information processing method according to an aspect of the present disclosure includes: acquiring, from a video, flow line information of a customer; detecting that the customer acquires an item; and storing, in a storage, flow line information of the customer and information on a number of items acquired by the customer, in association with each other.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a national stage application of International Application No. PCT/JP2019/001998 entitled “INFORMATION PROCESSING METHOD, INFORMATION PROCESSING DEVICE, AND RECORDING MEDIUM,” filed on Jan. 23, 2019, which claims the benefit of the priority of Japanese Patent Application No. JP2018-015408 filed on Jan. 31, 2018, the disclosures of each of which are hereby incorporated by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates to an information processing method, an information processing device, and a recording medium, and relates to a technique for processing data of a video.

BACKGROUND ART

In a mass retailer such as a convenience store or a supermarket, a monitoring camera for capturing inside of a store is installed. In general, a shop clerk or a security guard finds a crime and fraud such as shoplifting by monitoring a video on a monitor. However, there is another method for preventing shoplifting.

For example, in PTL 1, a method in which a radio frequency identification (RFID) tag is attached to an item is disclosed. However, the method described in PTL 1 has a problem that a cost increases, since it is required to attach a RFID tag to all items. Also, it is not possible to acquire information as to which customer acquires an item.

Further, there is a method of monitoring a customer by using a flow line analysis technique (for example, PTL 2) for tracking the customer on a video. In PTL 3, information on the number of items acquired by a customer and an image being captured for the customer are managed in association with each other. When a customer pays for an acquired item, the number of items registered in a point of sales (POS) terminal and the number of items acquired by the customer are compared with each other. When those numbers of items do not match, it is determined that there is a possibility of a fraud.

The method disclosed in PTL 3 has an advantageous effect that a shop clerk or a security guard does not need to continuously monitor a video.

CITATION LIST

Patent Literature

SUMMARY OF INVENTION

Technical Problem

However, the method disclosed in PTL 3 has a problem related to privacy, since an image being captured for a customer is managed without permission from the customer.

An object of the present disclosure is to precisely determine an acquisition state of an item for each customer while considering privacy of a customer, without identifying an individual.

Solution to Problem

An information processing method according to one aspect of the present disclosure includes: acquiring, from a video, flow line information of a customer; detecting that the customer acquires an item; and storing, in a storage, the flow line information of the customer and information on the number of items acquired by the customer, in association with each other.

An information processing device according to one aspect of the present disclosure includes: at least one memory configured to store instructions; and at least one processor configured to execute the instructions to: acquire, from a video, flow line information of a customer; detect that the customer acquires an item; and store, in a storage, the flow line information of the customer and information on the number of items acquired by the customer, in association with each other.

A non-transitory recording medium according to one aspect of the present disclosure stores a program for causing a computer to execute: acquiring, from a video, flow line information of a customer; detecting that the customer acquires an item; and storing, in a storage, the flow line information of the customer and information on the number of items acquired by the customer, in association with each other.

Advantageous Effects of Invention

According to one aspect of the present disclosure, an acquisition state of an item for each customer is able to be precisely determined without identifying an individual.

EXAMPLE EMBODIMENT

A configuration and an operation of an in-store monitoring device according to the present example embodiment is described with reference toFIGS.1to4.

FIG.1is a block diagram illustrating a configuration of an in-store monitoring device1according to the present example embodiment. As illustrated inFIG.1, the in-store monitoring device1includes an imaging unit11, a flow line analysis unit12, a shelf-front action detection unit13, an anomaly detection unit14, a notification unit15, and a storage unit16.

The imaging unit11captures inside of a store, and generates data of a video (moving image). The imaging unit11may include one camera or a plurality of cameras for capturing a video. The imaging unit11transmits the data (pixel data forming a video frame) of the video acquired by capturing the inside of the store to the flow line analysis unit12, the shelf-front action detection unit13, and the anomaly detection unit14. The storage unit16stores flow line—acquisition number information.

The flow line—acquisition number information is information in which flow line information of a customer and information on the number of items acquired by the customer are associated with each other. Note that, the number of items acquired by the customer is, herein, the number of items that the customer picks up from a shelf and possesses.

FIG.2is a diagram illustrating one example of a data base (DB) of flow line—acquisition number information stored in the storage unit16. As illustrated inFIG.2, in the DB of the flow line—acquisition number information, for example, an ID (flow line ID) for identifying a flow line of a customer, time-series positional information (time/coordinate) of the customer, and the number of items acquired by the customer may be associated with each other. Note that, inFIG.2, a time-series position of a customer is represented by 3D view coordinates viewed from the camera of the imaging unit11. Note that, the flow line—acquisition number information is not limited to the DB illustrated inFIG.2.

The flow line information includes information on a positional change of a moving object (herein, a customer) over time. A flow line is a trajectory drawn by movement of a moving object.

An operation of the flow line analysis unit12, the shelf-front action detection unit13, the anomaly detection unit14, and the notification unit15will be described later.

Note that, the in-store monitoring device1may not include the imaging unit11and the storage unit16. In this case, the in-store monitoring device1acquires data of a video captured by an imaging device (for example, a camera). Further, the in-store monitoring device1stores the flow line—acquisition number information in a storage device (for example, a memory).

As one example of anomaly detection processing executed by the in-store monitoring device1, a flow of first anomaly detection processing is described with reference toFIG.3.FIG.3is a flowchart illustrating the flow of the first anomaly detection processing.

As illustrated inFIG.3, in the first anomaly detection processing, the flow line analysis unit12tracks a customer by using data of a video received from the imaging unit11(S101). More specifically, the flow line analysis unit12generates and outputs flow line information of the customer by detecting the customer from each frame forming the video at a certain time interval, and analyzing positional change of the detected customer over time. A technique for tracking a moving object is, for example, disclosed in PTL 2. In the present example embodiment, detailed description on a technique for tracking a customer is omitted.

The shelf-front action detection unit13detects an action by the customer taken on a shelf (gondola) and measures the number of items acquired by the customer from the shelf by analyzing the data of the video received from the imaging unit11(S102). A technique for detecting an action (shelf-front action) in which a customer acquires an item from a shelf or returns an item to a shelf is, for example, disclosed in PTL 3. In the present example embodiment, detailed description on the technique for detecting a shelf-front action is omitted.

The shelf-front action detection unit13stores, as flow line—acquisition number information in the storage unit16, the flow line information of the customer generated by the flow line analysis unit12and information on the number of items acquired by the customer from the shelf in association with each other (S103).

The anomaly detection unit14detects that the customer enters a register-front area. The register-front area is an area where a customer lines up to pay for an acquired item at a register. A location and a shape of an area where a customer waiting for payment lines up is different depending on a structure and the like of a store. Therefore, a register-front area for the anomaly detection unit14to detect a person may be set for each store.

When detecting that the customer moves into the register-front area (Yes in S104), the anomaly detection unit14identifies a flow line relates to a customer at a head of a line among customers in the register-front area, by referring to the flow line—acquisition number information stored in the storage unit16, and acquires information on the number of items acquired by the customer at the head of the line (S105). In the following, the customer at the head of the line among customers in the register-front area is referred to as a determination target.

Further, the anomaly detection unit14also acquires information on the number of items registered in a POS terminal after the determination target moves into the register-front area. Then, the anomaly detection unit14compares the number of items registered in the POS terminal with the number of items acquired by the determination target (S106).

In the first anomaly detection processing, the anomaly detection unit14detects, as an anomaly, that the number of items acquired by the determination target and the number of items registered in the POS terminal do not match. When the number of items acquired by the determination target and the number of items registered in the POS terminal do not match (No in S107), the anomaly detection unit14causes the notification unit15to notify the anomaly (S108). The notification unit15may transmit an anomaly signal (alert) to, for example, a portable terminal possessed by a security guard or a terminal in an office.

In one modification example, anomaly detection unit14may cause the notification unit15to notify an anomaly when a difference between the number of items acquired by the determination target and the number of items registered in the POS terminal exceeds a certain number (threshold value) larger than one.

As another example of the anomaly detection processing executed by the in-store monitoring device1, a flow of second anomaly detection processing is described with reference toFIG.4.FIG.4is a flowchart illustrating the flow of the second anomaly detection processing.

As illustrated inFIG.4, in the second anomaly detection processing, the flow line analysis unit12tracks a customer, based on a video received from the imaging unit11(S201).

The shelf-front action detection unit13detects, from a video received from the imaging unit11, an action by a customer taken on a shelf (gondola) and measures the number of items acquired by the customer from the shelf (S202).

The flow line analysis unit12and the shelf-front action detection unit13store, as flow line—acquisition number information in the storage unit16, flow line information of the customer and information on the number of items acquired by the customer from the shelf in association with each other (S203). S201to S203of the second anomaly detection processing is identical to S101to S103of the first anomaly detection processing.

The anomaly detection unit14detects that the customer goes out of a specific area. The specific area is, for example, area where a shelf for displaying an item is arranged, a register-front area, and another area where a customer is allowed to possess an unpaid item. The anomaly detection unit14may detect that the customer crosses a specific line, instead of detecting that the customer goes out of the specific area. The specific line is a border between an area where a customer is allowed to possess an unpaid item and an area where a customer is not allowed to possess an unpaid item.

When detecting that the customer goes out of the specific area (Yes in S204), the anomaly detection unit14identifies a flow line related to the customer who goes out of the specific area, by referring to the flow line—acquisition number information stored in the storage unit16, and acquires, from the storage unit16, the information on the number of items acquired by the customer (S205).

In the second anomaly detection processing, the anomaly detection unit14determines, as an anomaly, that the customer goes out of the specific area while acquiring an unpaid item. When the customer goes out of the specific area while acquiring an unpaid item (Yes in S206), the anomaly detection unit14causes the notification unit15to notify the anomaly (S207). A notification method is not particularly limited. The notification unit15may transmit an anomaly signal (alert) to, for example, a portable terminal possessed by a security guard or a terminal in an office.

MODIFICATION EXAMPLE

The in-store monitoring device1according to one modification example may not perform flow line analysis and shelf-front action detection of a customer, and an external device (for example, a network server) may instead perform the flow line analysis and the shelf-front action detection of the customer. In the present modification example, the in-store monitoring device1receives, from the external device, a result of the flow line analysis and a result of the shelf-front action detection. Alternatively, the external device may generate flow line—acquisition number information and store the flow line—acquisition number information in the storage unit16, and the in-store monitoring device1may acquire the flow line—acquisition number information stored by the external device in the storage unit16.

According to a configuration of the present modification example, the in-store monitoring device1may not include the flow line analysis unit12and the shelf-front action detection unit13. Thus, it is possible to reduce a computer resource required for processing of flow line analysis (S101inFIG.3) and shelf-front action detection (S102inFIG.3) of a customer.

Advantageous Effect of Present Example Embodiment

According to the configuration of the present example embodiment, flow line information of a customer and information on the number of items acquired by the customer are stored in association with each other. By using the stored information, it is possible to accurately measure which customer acquires how many items, with a simple configuration.

Therefore, it is possible to detect an action of a customer who is suspected to be fraudulent, such as that the customer leaves a store while possessing an unpaid item. Further, privacy of a customer can be properly protected, since a video being captured for the customer is not managed.

In addition, when an action suspected to be fraudulent is detected, it is possible to prompt a security guard or a shop clerk to address by notifying an anomaly. Therefore, a security guard or a shop clerk does not need to continuously monitor a video for finding a fraud. As a result, it is possible to reduce a burden on a security guard and a shop clerk, and prevent the security guard and the shop clerk from overlooking a fraud.

An in-store monitoring device according to the present example embodiment is described with reference toFIG.5.

FIG.5is a block diagram illustrating an in-store monitoring device2according to the present example embodiment. As illustrated inFIG.5, the in-store monitoring device2includes an information acquisition unit21, an action detection unit22, and a recording unit23. The in-store monitoring device2acquires data of a video captured by an external imaging device (for example, a camera), although it is not illustrated. Further, the in-store monitoring device2causes an external storage device (for example, a memory) to store flow line—acquisition number information. Alternatively, the in-store monitoring device2may include an imaging device and a storage device.

The information acquisition unit21acquires flow line information of a customer generated by analyzing the data of the video captured by the external imaging device. A flow line analysis technique for tracking a customer on a video is, for example, disclosed in PTL 2.

The action detection unit22detects, by using a shelf-front action detection technique (for example, PTL 3), that the customer acquires an item in a store.

The recording unit23stores, as the above-described flow line—acquisition number information in a storage means (unillustrated), flow line information of the customer and information on the number of items acquired by the customer in association with each other. Note that, the recording unit23according to the present example embodiment corresponds to a part of the flow line analysis unit12and a part of the shelf-front action detection unit13according to the example embodiment 1. The in-store monitoring device2according to one modification example may not perform flow line analysis and shelf-front action detection of the customer, and an external device (for example, a network server) may instead perform the flow line analysis and the shelf-front action detection of the customer. In the present modification example, the recording unit23acquires each result of the flow line analysis and the shelf-front action detection of the customer, being performed by the external device, and stores the acquired result in a storage means (unillustrated).

Advantageous Effect of Present Example Embodiment

According to the configuration of the present example embodiment, flow line information of a customer and information on the number of items acquired by the customer are stored in association with each other in a storage means. Therefore, it is possible to accurately measure which customer acquires how many items, by referring the information (flow line—acquisition number information) stored in the storage means. The information on the number of items measured in such a way can be used, for example, for preventing shoplifting. This is because it can be known that a customer is about to bring an unpaid item out of a store. Further, the flow line information of a customer does not include information related to privacy, unlike a video being captured for the customer. Therefore, privacy of the customer can be properly protected.

An in-store monitoring device according to the present example embodiment is described with reference toFIG.6.

FIG.6is a diagram illustrating a configuration of an in-store monitoring device3according to the present example embodiment. The in-store monitoring device3is achieved as hardware by a computer device. The in-store monitoring device3includes a central processing unit (CPU)31, a random access memory (RAM)32, a storage device33, an input/output device34, and a communication interface35.

A function of the in-store monitoring device3is identical to the in-store monitoring device1according to the example embodiment 1 or the in-store monitoring device2according to the example embodiment 2. In other words, the in-store monitoring device3achieves an operation of a function block included in the in-store monitoring device1according to the example embodiment 1 or the in-store monitoring device2according to the example embodiment 2. The function of the in-store monitoring device3is achieved by the CPU31executing a program read into the RAM32.

The storage device33includes the storage unit16according to the example embodiment 1. The storage device33stores flow line—acquisition number information.

The input/output device34includes the notification unit15according to the example embodiment 1. The input/output device34may include a user interface such as a display.

The communication interface35is used in order to acquire video data from an external imaging device.

Advantageous Effect of Present Example Embodiment

According to the configuration of the present example embodiment, the function of the in-store monitoring device described in the example embodiment 1 or 2 is achieved as hardware by using a computer resource such as a CPU. Therefore, it is possible to accurately measure which customer acquires how many items, while considering privacy of the customer.

This application is based upon and claims the benefit of priority from Japanese patent application No. 2018-015408, filed on Jan. 31, 2018, the disclosure of which is incorporated herein in its entirety by reference.

REFERENCE SIGNS LIST

1,2,3In-store monitoring device

12Flow line analysis unit

13Shelf-front action detection unit

14Anomaly detection unit

21Information acquisition unit

22Action detection unit