Patent Description:
In recent years, a store which does not have a cash register for a customer to pay money has been put into practical use.

In particular, with the development of technology such as machine learning or computer vision using artificial intelligence (Al), a commodity data processing system has been developed for tracking behavior of a customer in a store by a camera, a sensor or the like. The behavior of the customer is analyzed to automatically register commodities to be purchased by the customer and perform a settlement process for the registered commodities.

In this type of commodity data processing system, the commodities taken by the customer are determined when the customer leaves the store. However, the settlement process is not executed immediately after the determination. The settlement process is executed after a predetermined time has passed since the customer left the store. During this predetermined period of time, the customer can confirm the list of the obtained commodities displayed on his or her smartphone or the like. Then, if a commodity that is not obtained by the customer is contained in the list, the customer can manually remove the commodity from the list before the settlement process is started.

However, in such a system, the customer is not able to confirm the list of the commodities until he or she leaves the store. Therefore, there is a need for a commodity data processing system which allows the customer to confirm the list of commodities obtained, before he or she leaves the store. <CIT> refers to a POS (Point Of Sales) terminal apparatus.

To solve such problem, there is provided a commodity data processing system comprising: a reader configured to read an ID of a customer; a camera configured to acquire images of the customer in a store; a display configured to display information about commodities; and a processor configured to identify the customer with the ID read by the reader when the customer enters the store; track movement of the customer in the store based on the images acquired by the camera and identify one or more commodities taken by the customer in the store; acquire commodity data of each of the identified commodities; after the customer has left the store, perform a settlement process for the commodities based on the acquired commodity data; and control the display to display a list of the commodities before the customer leaves the store.

Preferably, the processor is configured to track the movement of the customer and determine whether the customer faces the display, and control the display to display the list of the commodities when determining that the customer faces the display.

Preferably still, the list includes a name of each commodity and a cancellation button for cancelling the commodity.

Preferably yet, the processor is configured to count a total number of cancellations made by the customer via the cancellation button, and accept the cancellation only when the total number of cancellations is equal to or less than a threshold value.

Suitably, the processor is configured to control the display to display an error message when the total number of cancellations exceeds the threshold value.

Suitably still, the processor is configured to control the display to display the total number of cancellations together with the list of the commodities.

Suitably yet, the system further comprises: a plurality of sensors installed on commodity shelves in the store and detecting movement of commodities displayed on the shelves, wherein the processor is configured to identify each commodity in response to a signal transmitted from one of the sensors that has detected the movement of the commodity.

Typically, the reader is installed at an entrance gate of the store.

Typically still, the display is installed in a digital signage terminal in the store.

Typically yet, the processor is configured to identify the customer based on the image acquired by the camera.

The invention also relates to a method for processing commodity data, the method comprising: reading by a reader an ID of a customer who enters a store; identifying the customer with the ID read by the reader; tracking movement of the customer in the store based on images acquired by a camera and identifying one or more commodities taken by the customer in the store; acquiring commodity data of each of the identified commodities; and after the customer has left the store, performing a settlement process for the commodities based on the acquired commodity data, wherein the method further comprises displaying on a display a list of the commodities before the customer leaves the store.

Preferably, the method further comprises: determining whether the customer faces the display based on the movement of the customer, and displaying on the display the list of the commodities when determining that the customer faces the display.

Preferably yet, the list includes a name of each commodity and a cancellation button for cancelling the commodity.

Preferably still, the method further comprises: counting a total number of cancellations made by the customer via the cancellation button, and accepting the cancellation only when the total number of cancellations is equal to or less than a threshold value.

Suitably, the method according to claim <NUM>, further comprising:
displaying on the display an error message when the total number of cancellations exceeds the threshold value.

Suitably still, the method further comprises:
displaying on the display the total number of cancellations together with the list of the commodities.

Suitably yet, the method further comprises: detecting movement of commodities by a plurality of sensors installed on commodity shelves in the store, wherein each commodity is identified in response to a signal transmitted from one of the sensors that has detected the movement of the commodity.

The invention also concerns a commodity data processing system comprising: a first computing device installed at an entrance of a store and comprising a reader configured to read an ID of a customer who enters the store; a second computing device comprising a camera configured to acquire an image of the customer in the store, identify the customer with the ID read by the first computing device, and track movement of the customer in the store based on the images acquired by the camera; a third computing device configured to identify one or more commodities taken by the customer in the store based on the movement tracked by the second computing device; a fourth computing device configured to manage commodity data of each of the commodities identified by the third computing device; a fifth computing device configured to perform, after the customer has left the store, a settlement process for the commodities based on the acquired commodity data; and a sixth computing device configured to display a list of the commodities before the customer leaves the store.

The invention further relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method described above.

The invention further concerns a computer-readable medium having stored thereon the computer program described above.

The above and other objects, features and advantages of the present invention will be made apparent from the following description of the preferred embodiments, given as non-limiting examples, with reference to the accompanying drawings, in which:
A <NUM>.

In one embodiment, the commodity data processing system includes a reader configured to read an ID of a customer, a camera configured to acquire images of the customer in a store, a display configured to display information about commodities, and a processor configured to identify the customer with the ID read by the reader when the customer enters the store, track movement of the customer in the store based on the images acquired by the camera and identify one or more commodities taken by the customer in the store, acquire commodity data of each of the identified commodities, perform a settlement process for the commodities based on the acquired commodity data after the customer has left the store, and control the display to display a list of the commodities before the customer leaves the store.

A commodity data processing system according to an embodiment will be described below with reference to the drawings. In an embodiment, when a customer enters a store from the entrance gate, takes a commodity to be purchased from among commodities displayed in the sales room, and leaves the store from the exit gate with the commodity, an article data processing system can automatically register the commodity and perform a settlement process for the commodity.

In the present system, it is assumed that the customers have completed membership registration in advance. A unique member ID is issued to each member. The member can confirm a code symbol representing the member ID of the user, which is displayed by an application program on his or her portable information terminal such as a smartphone. The code symbol is, for example, a bar code. The code symbol may be, for example, a two-dimensional data code.

<FIG> is a block diagram showing a configuration of a commodity data processing system <NUM> according to one embodiment. The commodity data processing system <NUM> includes a commodity database <NUM>, a member database <NUM>, a gate device <NUM>, a behavior tracking device <NUM>, a commodity identification device <NUM>, a settlement device <NUM>, a virtual cart device <NUM>, and a cart confirmation device <NUM>. The commodity database <NUM> is connected to the commodity identification device <NUM>. The member database <NUM> is connected to the gate device <NUM>, the settlement device <NUM>, and the cart confirmation device <NUM>. The gate device <NUM> is connected to the behavior tracking device <NUM>. The behavior tracking device <NUM> is connected to the commodity identification device <NUM>, the settlement device <NUM>, the virtual cart device <NUM>, and the cart confirmation device <NUM>. The commodity identification device <NUM>, the settlement device <NUM> and the cart confirmation device <NUM> are each connected to the virtual cart device <NUM>.

The cart confirmation device <NUM> is connected to a digital signage terminal <NUM>. The digital signage terminal <NUM> is a kind of display device that displays contents provided from a content providing device <NUM>. An example of the contents is an advertisement image. The digital signage terminal <NUM> is installed in a sales room. A floor is a space in which a commodity shelf is installed. A plurality of kinds of commodities are sorted and displayed for each of a plurality of kinds of commodities in a commodity shelf.

The digital signage terminal <NUM> is installed at a place different from the commodity shelf. The number of the digital signage terminals <NUM> is not limited to <NUM>. A plurality of digital signage terminals <NUM> may be installed in the sales room. When a plurality of digital signage terminals <NUM> are installed in a sales room or a shopping floor of a store, the cart confirmation device <NUM> is provided for each digital signage terminal <NUM> in the digital signage terminal <NUM>.

The commodity database <NUM> stores data relating to each commodity sold in the sales room. The data is generated in a certain format and stored in the commodity database <NUM>. Hereinafter, the data generated in the format and stored in the commodity database <NUM> will be referred to as commodity record 11R.

<FIG> is a schematic diagram showing a data structure of the commodity record 11R. As shown in the figure, the commodity record 11R includes data items such as a commodity code, a commodity name, a price, a shelf code, and an image for recognition. The commodity code is a commodity unique code set for each commodity in order to identify each commodity. The commodity name is a name of the commodity, and the price indicates a unit price of the commodity. The shelf code is a unique code set for each section of the commodity shelf on which the commodity is displayed. The recognition image is data of an image obtained by photographing the commodity. The recognition image may be an image that includes the entire commodity or a part of the commodity.

The member database <NUM> stores data relating to customers who have been registered as members for shopping in the store. The data is generated in a certain format and stored in the member database <NUM>. Hereinafter, the data generated in the format and stored in the member database <NUM> is referred to as member record 12R.

<FIG> is a schematic diagram showing a data structure of the member record 12R. As shown in the figure, the member record 12R includes data items such as a member ID, a member name, a credit card number, an invalid flag, and the like. As described above, the member ID is a member specific code assigned to each member in order to identify the member. The member name is the name of the member, and the credit card number is the number of a credit card used in the settlement by the member. The invalid flag is <NUM>-bit data for setting whether or not to invalidate the member record 12R. In the present embodiment, when the invalid flag is "<NUM>", the member record 12R is invalid. For example, when a member identified by the member ID is invalid and his or her entry to the store is prohibited, the invalid flag of the member record 12R is set to "<NUM>".

<FIG> is a block diagram showing a configuration of the gate device <NUM>. The gate device <NUM> includes an opening/closing mechanism <NUM> for an entrance gate, a reader <NUM>, a search unit <NUM>, a member authentication unit <NUM>, a gate control unit <NUM>, and an entry output unit <NUM>.

The opening and closing mechanism <NUM> opens and closes the entrance gate to the floor. The entrance gate is closed in the default state. The opening/closing mechanism <NUM> temporarily opens the entrance gate by receiving an open signal from the gate control unit <NUM>. The entrance gate may be opened to allow entry of a customer onto the floor.

The reader <NUM> is a device for reading code symbols displayed on a display of an information terminal such as a smartphone. As described above, a code symbol representing the member ID unique to the member is displayed on the information terminal owned by the member. The reader <NUM> is provided outside the entrance gate to read this code symbol.

The search unit <NUM> searches the member database <NUM>. When the code symbol representing the member ID is read by the reader <NUM>, the search unit <NUM> searches the member database <NUM> for the member record 12R by using the member ID as the search keyword. The search unit <NUM> notifies the member authentication unit <NUM> of the search result.

The member authentication unit <NUM> receives the search result from the search unit <NUM> and determines whether or not to approve the entrance of the customer. When the member record 12R including the member ID used as the search keyword is detected from the member database <NUM>, the member authentication unit <NUM> checks the invalid flag of the member record 12R. When the invalid flag is "<NUM>", that is, when the member record 12R is valid, the member authentication unit <NUM> approves the entrance of the customer as a valid member. On the other hand, when the member record 12R cannot be detected from the member database <NUM> or when the invalid flag of the detected member record 12R is "<NUM>", that is, when the member record 12R is invalid, the member authentication unit <NUM> denies the entrance of the customer. The member authentication unit <NUM> notifies the gate control unit <NUM> and the entry output unit <NUM> of the authentication result.

When the authentication result notified by the member authentication unit <NUM> indicates approval of the entrance of the member, the gate control unit <NUM> outputs an open signal to the opening/closing mechanism <NUM>. When the authentication result is the one which denies the entrance of the member, the gate control unit <NUM> does not output an open signal to the opening/closing mechanism <NUM>.

The entry output unit <NUM> generates the entry command COM1 when the authentication result notified by the member authentication unit <NUM> indicates the approval of the entrance. The entrance output unit <NUM> outputs the entry command COM1 to the behavior tracking unit <NUM>. The entry output unit <NUM> does not generate the entry command COM1 when the authentication result notified by the member authentication unit <NUM> indicates the denial of the entrance of the member. Therefore, the entry command COM1 is not output to the behavior tracking device <NUM>.

<FIG> is a schematic diagram showing a data structure of the entry command COM1. As shown in the figure, the entry command includes a command code <NUM> and a member ID. The command code <NUM> is a code for specifying that the command is the entry command COM1. The member ID is the member ID of the member whose entrance is approved by the member authentication unit <NUM>. That is, the member ID obtained from the code symbol read by the reader <NUM> is included in the entry command COM1.

<FIG> is a block diagram showing a configuration of the behavior tracking device <NUM>. The behavior tracking device <NUM> includes a plurality of cameras <NUM>, a plurality of sensors <NUM>, an image processing unit <NUM>, a signal processing unit <NUM>, a behavior analysis unit <NUM>, a command generation unit <NUM>, and a command output unit <NUM>. The plurality of cameras <NUM> are appropriately arranged on a ceiling or the like of the sales room in the store so as to cover the entire area of the sales area as a photographing area.

The plurality of sensors <NUM> are arranged at each section of the commodity shelf. The sensor <NUM> detects that a commodity displayed on the commodity shelf has been taken out. The sensor <NUM> further detects that the commodity has been returned to the commodity shelf. For example, the sensor <NUM> is an infrared sensor, a pressure sensor, a weight sensor, or the like, or a combination thereof.

The image processing unit <NUM> inputs an image captured by each camera <NUM>. The image processing unit <NUM> generates image data necessary for behavior analysis performed by the behavior analysis unit <NUM> from the image captured by each camera <NUM>, and outputs the generated image data to the behavior analysis unit <NUM>.

The signal processing unit <NUM> inputs a detection signal from each sensor <NUM>. The signal processing unit <NUM> selects the detection signal necessary for the behavior analysis performed by the behavior analysis unit <NUM> from the detection signal from each sensor <NUM>, and outputs the selected detection signal to the behavior analysis unit <NUM>.

The behavior analysis unit <NUM> receives the entry command COM1 transmitted from the gate device <NUM> to the behavior tracking device <NUM>. The behavior analysis unit <NUM> captures the image data from the image processing unit <NUM> and fetches the detection signal from the signal processing unit <NUM>. Based on the image data and the detection signal, the behavior analysis unit <NUM> analyzes the behavior of the member in the sales room, who is specified by the member ID included in the entry command COM1. Specifically, the behavior analysis unit <NUM> keeps track of movement within the sales room of the member who has entered the floor from the entrance gate, until the member leaves the store from the exit gate. The behavior analysis unit <NUM> detects the action of the member taking out the commodity from the commodity shelf while tracking his or her movement, the action of the member returning the commodity to the commodity shelf, and the action of the member facing the screen of the digital signage terminal <NUM>. The detection of such an action can be made by using existing techniques such as machine learning or computer vision using an existing AI technologies.

<FIG> is a flowchart of operations performed by the behavior analysis unit <NUM> from when a member enters at the entrance gate until when he or she exits through the exit gate. The behavior analysis unit <NUM> waits for the entry command COM1 as Act <NUM>. When receiving the entry command COM1 from the gate device <NUM>, the behavior analysis unit <NUM> determines YES in Act <NUM>, and proceeds to Act <NUM>. The behavior analysis unit <NUM> acquires the member ID from the entry command COM1 as Act <NUM>.

The behavior analysis unit <NUM> instructs the command generating unit <NUM> to generate a virtual cart creation command COM2 in Act <NUM>. The instruction to generate the virtual cart creation command COM2 includes the member ID acquired from the entry command COM1. The details of the virtual cart creation command COM2 will be described later.

The behavior analysis unit <NUM> which has instructed generating the virtual cart creation command COM2 starts behavior analysis of the member identified by the member ID acquired from the entry command COM1 in Act <NUM>. Then, the behavior analysis unit <NUM> determines whether or not the member's action of taking out the commodity from the commodity shelf is detected, in Act <NUM>. When the member's action of taking out the commodity from the commodity shelf is not detected, the behavior analysis unit <NUM> determines NO in Act <NUM>, and proceeds to Act <NUM>. The behavior analysis unit <NUM> determines whether or not the member's action of returning the commodity to the commodity shelf is detected as Act <NUM>. When the member's action of returning the commodity to the commodity shelf is not detected, the behavior analysis unit <NUM> determines NO in Act <NUM>, and proceeds to Act <NUM>. The behavior analysis unit <NUM> determines whether or not the member faces the screen of the digital signage terminal <NUM> as Act <NUM>. When the member does not face the screen of the digital signage terminal <NUM>, the behavior analysis unit <NUM> determines NO in Act <NUM>, and proceeds to Act <NUM>. The behavior analysis unit <NUM> determines whether or not the member exits through the exit gate in Act <NUM>. When the member does not exit, the behavior analysis unit <NUM> determines NO in Act <NUM>, and the process returns to Act <NUM>.

In this manner, the behavior analysis unit <NUM> keeps track of the behavior of the member in Act <NUM> to Act <NUM>. That is, the behavior analysis unit <NUM> detects an action of the member taking out the commodity from the commodity shelf, returning the commodity to the commodity shelf, and facing the screen of the digital signage terminal <NUM>. The behavior analysis unit <NUM> continues the behavior analysis until the member exits through the exit gate.

When the behavior analysis unit <NUM> detects the action of taking out the commodity from the commodity shelf, the behavior analysis unit <NUM> determines YES in Act <NUM>, and proceeds to Act <NUM>. The behavior analysis unit <NUM> instructs the command generation unit <NUM> to generate a commodity taken command COM3 as Act <NUM>. The instruction to generate the commodity taken command COM3 includes the member ID of the member whose behavior is being tracked and image data and a shelf code when the member's action of taking out the commodity from the commodity shelf is detected. The shelf code is specified from a detection signal when the member's action of taking out the commodity from the commodity shelf is detected. The shelf code identifies a section of a commodity shelf on which a sensor, which is a transmission source of the detection signal, is provided. When Act <NUM> is completed, the process proceeds to Act <NUM>.

When the behavior analysis unit <NUM> detects the action of returning the commodity to the commodity shelf, the behavior analysis unit <NUM> determines YES in Act <NUM>, and proceeds to Act <NUM>. The behavior analysis unit <NUM> instructs the command generation unit <NUM> to generate a commodity return command COM4 as Act <NUM>. The instruction to generate the commodity return command COM4 includes the member ID of the member whose behavior is being tracked and the image data and shelf code when the member's action of returning the commodity to the commodity shelf is detected. When the process of Act <NUM> is completed, the behavior analysis unit <NUM> proceeds to Act <NUM>.

When the behavior analysis unit <NUM> detects that the member faces the screen of the digital signage terminal <NUM>, it determines YES in Act <NUM>, and proceeds to Act <NUM>. The behavior analysis unit <NUM> instructs the command generation unit <NUM> to generate a cart confirmation command COM5 as Act <NUM>. The instruction to generate the cart confirmation command COM5 includes a terminal code together with the member ID of the member being tracked. The terminal code is a code for specifying the digital signage terminal <NUM> which the member is facing. When a plurality of digital signage terminals <NUM> are installed in a sales room, different terminal codes are set to the respective digital signage terminals <NUM>. When the behavior analysis unit <NUM> detects that the member faces the screen of one of the digital signage terminals <NUM>, the behavior analysis unit <NUM> issues the cart confirmation command including the terminal code of the digital signage terminal <NUM>. When the process of Act <NUM> is completed, the behavior analysis unit <NUM> proceeds to Act <NUM>.

When the behavior analysis unit <NUM> detects that the member has exited from the exit gate to the outside of the store, that is, the member has left the store, the behavior analysis unit <NUM> determines YES in Act <NUM>, and the process proceeds to Act <NUM>. The behavior analysis unit <NUM> instructs the command generation unit <NUM> to generate an exit command COM6 as Act <NUM>. The instruction to generate the exit command COM6 includes the member ID of the member whose behavior has been tracked. When the process of Act <NUM> is completed, the behavior analysis unit <NUM> terminates the behavior analysis of the member identified by the member ID as Act <NUM>.

Referring to <FIG>, the details of the command generation unit <NUM> are described below.

When the command generation unit <NUM> receives the instruction to generate the virtual cart creation command COM2 from the behavior analysis unit <NUM>, the command generation unit <NUM> generates the virtual cart creation command COM2. When the command generation unit <NUM> receives the instruction to generate the commodity taken command COM3 from the behavior analysis unit <NUM>, the command generation unit <NUM> generates the commodity taken command COM3. When the command generation unit <NUM> receives the instructions to generate the commodity return command COM4 from the behavior analysis unit <NUM>, the command generation unit <NUM> generates the commodity return command COM4. When receiving the instruction to generate the cart confirmation command COM5 from the behavior analysis unit <NUM>, the command generation unit <NUM> generates the cart confirmation command COM5. When the command generation unit <NUM> receives the instruction to generate the exit command COM6 from the behavior analysis unit <NUM>, the command generation unit <NUM> generates the exit command COM6.

<FIG> is a schematic diagram showing structures of the commands COM2-COM6 generated by the command generation unit <NUM>. As shown in the figure, the cart creation command COM2 includes a command code <NUM> and a member ID. The command code <NUM> is a code for specifying that the command is the cart creation command COM2. The member ID is the one included in the instruction to generate the virtual cart creation command COM2, received from the behavior analysis unit <NUM>.

The commodity taken command COM3 includes a command code <NUM>, a member ID, a shelf code, and image data. The command code <NUM> is a code for specifying that the command is the commodity taken command COM3. The member ID, the shelf code and the image data are the ones included in the instruction to generate the commodity taken command COM3, received from the behavior analysis unit <NUM>.

The commodity return command COM4 includes a command code <NUM>, a member ID, a shelf code, and image data. The command code <NUM> is a code for specifying that the command is the commodity return command COM4. The member ID, the shelf code and the image data are the ones included in the instruction to generating the commodity return command COM4, received from the behavior analysis unit <NUM>.

The cart confirmation command COM5 includes a command code <NUM>, a member ID, and a terminal ID. The command code <NUM> is a code for specifying that the command is the cart confirmation command COM5. The member ID and the terminal code are the ones included in the instructions to generate the cart confirmation command COM5, received from the behavior analysis unit <NUM>.

The exit command COM6 includes a command code <NUM> and a member ID. The command code <NUM> is a code for specifying that the command is the exit command COM6. The member ID is the one included in the instruction to generate the exit command COM6, received from the behavior analysis unit <NUM>.

Referring back to <FIG>, the details of the command output unit <NUM> are described below.

The command output unit <NUM> outputs the commands COM2-COM6 generated by the command generation unit <NUM> to other devices. For example, the command output unit <NUM> outputs the cart creation command COM2 to the virtual cart device <NUM>. The command output unit <NUM> outputs the commodity taken command COM3 and the commodity return command COM4 to the commodity identification device <NUM>. The command output unit <NUM> outputs the cart confirmation command COM5 to the cart confirmation device <NUM>. The command output unit <NUM> outputs the exit command COM6 to the settlement device <NUM>.

<FIG> is a block diagram showing a configuration of the commodity identification device <NUM>. The commodity identification device <NUM> includes a command analysis unit <NUM>, a shelf code acquisition unit <NUM>, an image acquisition unit <NUM>, a search unit <NUM>, a commodity recognition unit <NUM>, a command generation unit <NUM>, and a command output unit <NUM>. The command analysis unit <NUM> analyzes the command received from the behavior tracking device <NUM>. Based on the command code included in the command, the command analysis unit <NUM> identifies whether the command is the commodity taken command COM3 or the commodity return command COM4, and notifies the command generation unit <NUM> of the identification result.

The shelf code acquisition unit <NUM> acquires a shelf code from the commodity taken command COM3 or the commodity return command COM4 analyzed by the command analysis unit <NUM>. The image acquisition unit <NUM> acquires image data included in the commodity taken command COM3 or the commodity return command COM4 analyzed by the command analysis unit <NUM>.

The search unit <NUM> searches the commodity database <NUM> for one or more commodity records 11R including the shelf code acquired by the shelf code acquisition unit <NUM>, by using the acquired shelf code as a search keyword. The search unit <NUM> reads the commodity records 11R including the shelf code, and outputs the commodity records 11R to the commodity recognition unit <NUM>.

The commodity recognition unit <NUM> calculates the similarity between the recognition image included in each commodity record 11R received from the search unit <NUM> and the image acquired by the image acquisition unit <NUM>. When the commodity recognition unit <NUM> identifies one commodity record 11R having the degree of similarity equal to or greater than a predetermined value, the commodity recognition unit <NUM> outputs commodity data such as a commodity ID, a commodity name, a price and the like included in the identified commodity record 11R to the command generation unit <NUM>.

The command generation unit <NUM> generates a commodity purchase command COM7 based on the commodity data received from the commodity recognition unit <NUM> when the identification result notified from the command analysis unit <NUM> indicates the commodity taken command COM3. When the identification result notified from the command analysis unit <NUM> indicates the commodity return command COM4, the command generation unit <NUM> generates a commodity return command COM8 based on the commodity data received from the commodity recognition unit <NUM>.

<FIG> is a schematic diagram showing a data structure of the commodity purchase command COM7 and the commodity return command COM8 generated by the command generation unit <NUM>. As shown in the figure, the commodity purchase command COM7 includes a command code <NUM>, a member ID, and commodity data. The command code <NUM> is a code for specifying that the command is the commodity purchase command COM7. The member ID is the one included in the commodity taken command COM3 analyzed by the command analysis unit <NUM>. The commodity data includes a commodity code, a commodity name, and a price, which are supplied from the commodity recognition unit <NUM>.

The commodity return command COM8 includes a command code <NUM>, a member ID, and commodity data. The command code <NUM> is a code for specifying that the command is the commodity return command COM8. The member ID is the one included in the commodity return command COM4 analyzed by the command analysis unit <NUM>. The commodity data includes a commodity name, a price, and the like given from the commodity recognition unit <NUM>. The commodity data may include a commodity code, a commodity name, a combination of a commodity code and a price, or the like. The commodity data may include a commodity code alone.

Referring back to <FIG>, details of the command output unit <NUM> is described below. The command output unit <NUM> outputs to the virtual cart device <NUM> the commodity purchase command COM7 and the commodity return command COM8, which have been generated by the command generation unit <NUM>.

<FIG> is a block diagram showing a configuration of the settlement device <NUM>. The settlement device <NUM> includes a command analyzing unit <NUM>, a virtual cart reading unit <NUM>, a search unit <NUM>, and a settlement processing unit <NUM>. The command analysis unit <NUM> analyses the exit command COM6 received from the behavior tracking device <NUM>.

The virtual cart reading unit <NUM> generates a virtual cart reading command COM9 with the member ID included in the exit command COM6 analyzed by the command analyzing unit <NUM>. The virtual cart reading unit <NUM> outputs the virtual cart reading command COM9 to the virtual cart device <NUM> to read the data of the virtual cart stored in the virtual cart device <NUM>. The virtual cart stores the data of the commodity taken by the member specified by the member ID in the sales room. The virtual cart reading unit <NUM> transmits the data of the virtual cart to the settlement processing unit <NUM>.

<FIG> is a schematic diagram showing a data structure of the virtual cart read command COM9. As illustrated in the figure, the virtual cart read command COM9 includes a command code <NUM> and a member ID. The command code <NUM> is a code for specifying that the command is the virtual cart read command COM9. The member ID is the one included in the exit command COM6 analyzed by the command analysis unit <NUM>.

Referring back to <FIG>, the search unit <NUM> searches the member database <NUM> for a member record 12R including the member ID specified by the exit command COM6 analyzed by the command analysis unit <NUM>, by using the member ID as a search keyword. The search unit <NUM> reads the member record 12R including the member ID, and provides the data of the member record 12R to the settlement processing unit <NUM>.

The settlement processing unit <NUM> performs a settlement process based on the data of the virtual cart and the data of the member record 12R. The data of the virtual cart contains the data of the commodity taken by the member identified by the member ID in the sales room. The data of the member record 12R includes the card number of the credit card used by the member specified by the member ID. The settlement processing unit <NUM> executes the settlement process by the credit card on the commodity taken by the customer based on the commodity data and the card number of the credit card. Any known technique can be used to perform the settlement process by the credit card. A receipt associated with the settlement process may be issued by an existing electronic receipt system.

<FIG> is a block diagram showing a configuration of the virtual cart device <NUM>. The virtual cart device <NUM> includes a memory <NUM>, a communication unit <NUM>, a command analysis unit <NUM>, a writing unit <NUM>, a clock unit <NUM>, and a reading unit <NUM>. The clock unit <NUM> acquires the current date and time.

The memory <NUM> is capable of storing data for a plurality of virtual carts VC1, VC2, VC3,. The virtual carts VC1-VCn each operate like a work memory having an area for storing data items such as a member ID, an entry date and time, an exit date and time, a commodity list, a total number of commodities, a total amount, and the like. In the area of the member ID, the member ID of the member whose behavior is tracked by the behavior tracking device <NUM> is stored. The date and time when the member has entered the selling room is stored in the area of the entry date and time. The date and time when the member has exited the sales room is stored in the area of the exit date and time. In the area of the commodity list, data relating to commodities taken by the same member in the sales room is stored. In the area of the total number of commodities, the total number of commodities taken by the same member in the sales room is stored. In the area of the total amount, the total amount of the commodities taken by the same member in the sales room is stored.

<FIG> is a schematic diagram showing a data structure of data record D1 stored in the area of the commodity list. The data record D1 is generated every time the member obtains a commodity, and is stored in the area of the commodity list. The data record D1 includes an in-cart flag, a returned flag, and a cancelled flag in addition to the commodity code, commodity name, price, and the like. The in-cart flag is set to "<NUM>" when the commodity of the corresponding commodity data is in an in-cart state. The returned flag is set to "<NUM>" when the commodity of the corresponding commodity data is in a returned state. The cancelled flag is set to "<NUM>" when the commodity of the corresponding commodity data is in a cancelled state. For example, the in-cart state indicates a state in which a member has taken a commodity from a commodity shelf and is holding it in his or her shopping bag or the like. The returned state indicates a state in which the member has returned the commodity taken from the commodity shelf to the original shelf. The cancellation state indicates a state in which the commodity has been removed from the commodity list by the member's cancellation operation.

Referring back to <FIG>, details of the communication unit <NUM> and the command analysis unit <NUM> are described below.

The communication unit <NUM> communicates with the behavior tracking device <NUM>, the commodity identification device <NUM>, the settlement device <NUM>, and the cart confirmation device <NUM>. More specifically, the communication unit <NUM> receives the cart creation command COM2 from the behavior tracking device <NUM>. The communication unit <NUM> receives the commodity purchase command COM7 or the commodity return command COM8 from the commodity identification device <NUM>. The communication unit <NUM> receives the virtual cart reading command COM9 from the settlement device <NUM>. Although details will be described later, the communication unit <NUM> also receives the virtual cart reading command COM9 from the cart confirmation device <NUM>. The communication unit <NUM> transmits the data of one of the virtual carts VC1-VCn specified by the virtual cart reading command COM9 to the settlement device <NUM> or the cart confirmation device <NUM> which is the transmission source of the virtual cart reading command COM9.

The command analysis unit <NUM> analyzes the command received by the communication unit <NUM>. When the cart creation command COM2, the commodity purchase command COM7, and the commodity return command COM8 are received, the command analysis unit <NUM> activates the writing unit <NUM>. When the virtual cart reading command COM9 is received, the command analysis unit <NUM> activates the reading unit <NUM>.

The writing unit <NUM> secures a storage area for a new virtual cart VCn (n: <NUM>,<NUM>,<NUM>,. ) in the memory unit <NUM> when the command analyzed by the command analyzing unit <NUM> is the cart creation command COM2. The writing unit <NUM> stores the member ID included in the cart creation command COM2 in the virtual cart VCn. The writing unit <NUM> also stores the date and time acquired by the clock unit <NUM> as the entry date and time in the virtual cart VCn.

When the command analyzed by the command analysis unit <NUM> is the commodity purchase command COM7, the writing unit <NUM> stores the data record D1 of the commodity list in the virtual cart VCn in which the member ID included in the commodity purchase command COM7 is stored. The data record D1 describes a commodity code, a commodity name, and a price of the commodity data included in the commodity purchase command COM7. Here, the in-cart flag is set to "<NUM>", and the returned flag and the cancelled flag are both set to "<NUM>". Further, the writing unit <NUM> rewrites the data of the total number of commodities of the virtual cart VCn so that the total number is incremented, and rewrites the data of the total amount so that the total amount includes the price of the added commodity.

When the command analyzed by the command analysis unit <NUM> is the commodity return command COM8, the writing unit <NUM> specifies the virtual cart VCn associated with the member ID included in the commodity return command COM8, and selects the data record D1 related to the commodity data included in the commodity return command COM8. Then, the write unit <NUM> sets the returned flag of the selected data record D1 to "<NUM>". Here, the in-cart flag of the data record D1 may be maintained (i.e., "<NUM>") or may be rewritten to "<NUM>". Further, the writing unit <NUM> rewrites the data of the total number of commodities of the virtual cart VCn so that the number of commodities is decremented, and rewrites the data of the total amount so as to subtract the price of the returned commodity.

When the command analyzed by the command analyzing unit <NUM> is the virtual cart reading command COM9, the reading unit <NUM> reads the data of the virtual cart VCn in which the member code included in the virtual cart reading command COM9 is stored, from the memory unit <NUM>. Then, the reading unit <NUM> supplies the data of the virtual cart VCn to the communication unit <NUM>. The communication unit <NUM> transmits the data of the virtual cart VCn to the settlement device <NUM> or the cart confirmation device <NUM> which is the transmission source of the virtual cart read command COM9.

<FIG> is a block diagram showing a configuration of the cart confirmation device <NUM>. The cart confirmation device <NUM> includes a command analysis unit <NUM>, a display control unit <NUM>, an input detection unit <NUM>, a virtual cart processing unit <NUM>, and a cancellation processing unit <NUM>. The command analysis unit <NUM> analyzes the command received from the behavior tracking device <NUM>. When the command is the cart confirmation command COM5, the command analysis unit <NUM> activates the display control unit <NUM>. The command analysis unit <NUM> notifies the virtual cart processing unit <NUM> of the member ID included in the cart confirmation command COM5.

The display control unit <NUM> controls the screen display of the digital signage terminal <NUM>.

<FIG> shows an example of a screen SC1 controlled by the display control unit <NUM>. In the screen SC1 of the digital signage terminal <NUM>, an image of the content provided from the content providing device <NUM> is usually displayed. The display control unit <NUM> displays an image of a cart confirmation button B1 within the screen SC1 showing the content image.

Returning to <FIG>, the input detection unit <NUM> detects an input made on the screen of the digital signage terminal <NUM>. As shown in <FIG>, when the image of the cart confirmation button B1 is displayed on the screen SC1 of the digital signage terminal <NUM>, the input detection unit <NUM> can detect that the cart confirmation button B1 is operated. When detecting that the cart confirmation button B1 is operated, the input detection unit <NUM> instructs the virtual cart processing unit <NUM> to generate the virtual cart reading command COM9.

When receiving the instruction to generate the virtual cart reading command COM9 from the input detection unit <NUM>, the virtual cart processing unit <NUM> generates the virtual cart reading command COM9. The virtual cart read command COM9 includes the member ID notified from the command analysis unit <NUM>. The virtual cart processing unit <NUM> outputs the virtual cart reading command COM9 to the virtual cart device <NUM> to receive the data of the virtual cart VCn from the virtual cart device <NUM>.

The virtual cart processing unit <NUM> processes the data of the virtual cart VCn. Specifically, the virtual cart processing unit <NUM> extracts the data record D1 having the in-cart flag "<NUM>" and the returned flag and the cancelled flag "<NUM>" from the data record D1 described in the commodity list. The virtual cart processing unit <NUM> controls the display control unit <NUM> to display a list image of the commodity code, the commodity name, and the price included in the extracted data record D1. The virtual cart processing unit <NUM> counts the number of data records D1 whose canceled flag is set to "<NUM>" from the data records D1 described in the commodity list. The virtual cart processing unit <NUM> controls the display control unit <NUM> to display the counted number as the number of cancellations.

The display control unit <NUM> controls the digital signage terminal <NUM> to update the screen display under the control of the virtual cart processing unit <NUM>.

<FIG> shows an example of a screen SC2 updated by the display control unit <NUM>. The screen SC2 of the digital signage terminal <NUM> includes an area SA1 representing a content image, an area SA2 representing the content of the virtual cart VCn, and an area SA3 representing the number of cancellations. The content image displayed on the screen SC1 is reduced and displayed in the area SA1. The area SA2 displays a list of the commodity codes, the commodity names, and the prices of the data record D1 extracted by the virtual cart processing unit <NUM>. The image of the cancel button B2 is also displayed in the area SA2 in association with each commodity code and commodity name. The number of cancellations is displayed in the area SA3.

The contents displayed in the area SA2 are a list of commodities that the member facing the screen SC2 of the digital signage terminal <NUM> has already acquired in the selling room. Thus, the member can confirm the list of commodities to be purchased, which is managed by the virtual cart VCn, before leaving the store. Here, if the data of a commodity that has not been actually obtained is displayed in the area SA2, the member can cancel the commodity by touching the image of the cancel button B2 associated with the commodity name.

When the cancel button B2 is displayed on the screen SC2 of the digital signage terminal <NUM>, the input detection unit <NUM> can detect that the cancel button B2 is input. When detecting that the cancel button B2 is input, the input detection unit <NUM> instructs the virtual cart processing unit <NUM> to generate cancellation data. The instruction to generate cancellation data includes data indicating the input of the cancel button B2 associated with the item.

When receiving the instruction to generate data, the virtual cart processing unit <NUM> generates cancellation data including at least the commodity ID of the commodity specified from the data included in the instruction from the input detection unit <NUM>, and outputs the cancellation data to the cancellation processing unit <NUM>.

The cancellation processing unit <NUM> performs cancellation processing based on the cancellation data. More specifically, the cancellation processing unit <NUM> generates a commodity cancellation command COM10, and outputs the commodity cancellation command COM10 to the virtual cart device <NUM>.

<FIG> is a schematic diagram showing a data structure of the commodity cancellation command COM. As shown in the figure, the commodity cancellation command COM10 includes a command code <NUM>, a member ID, and commodity data. The command code <NUM> is a code for specifying that the command is the commodity cancellation command COM10. The member ID is the one included in the cart confirmation command COM5 analyzed by the command analysis unit <NUM>. The commodity data is the one included in the cancellation data generated by the virtual cart processing unit <NUM>.

Returning to <FIG>, in the virtual cart device <NUM> having received the commodity cancellation command COM10 via the communication unit <NUM>, the cancellation processing of the commodity list is executed by the operations of the command analysis unit <NUM> and the writing unit <NUM>. That is, the command analysis unit <NUM> notifies the writing unit <NUM> of the member code and the commodity data included in the commodity cancellation command COM10. The writing unit <NUM> selects the virtual cart VCn with which the member code is associated from the memory unit <NUM>. The writing unit <NUM> rewrites the cancelled flag of the data record D1 in which the commodity data is stored into "<NUM>" among the data records stored in the commodity list of the virtual cart VCn. At this time, the writing unit <NUM> may set the in-cart flag of the data record D1 to "<NUM>" or to "<NUM>".

Returning to <FIG>, after outputting the commodity cancellation command COM10 to the virtual cart device <NUM>, the virtual cart processing unit <NUM> instructs the virtual cart processing unit <NUM> to re-read the virtual cart. In response to receiving the re-read command, the virtual cart processing unit <NUM> again outputs the virtual cart reading command COM9 to the virtual cart device <NUM> and read the data of the virtual cart VCn. Then, the virtual cart processing unit <NUM> processes the data of the virtual cart VCn in the same manner as described above. The display control unit <NUM>, under the control of the virtual cart processing unit <NUM>, updates the screen display of the digital signage terminal <NUM>.

<FIG> shows an example of a screen SC3 after the cancellation processing is executed in response to an input of the cancel button B2 associated with the commodity C shown in the screen SC2 of <FIG>. The data of the commodity C displayed in the area SA2 is deleted as compared with the screen SC2, and the amount of money equivalent to the price of the commodity C is reduced. Further, the number of cancellations displayed in the area SA3 is changed from <NUM> to <NUM>.

Thus, when the customer notices that a commodity that is not actually taken from a commodity shelf is included in the virtual cart VCn, he or she can cancel and remove the commodity from the cart even before leaving the sales room.

However, if a customer can cancel a commodity stored in the virtual cart VCn without any restriction, there is a concern that the loss due to improper cancellation may be increased. Thus, in the present embodiment, the customer is allowed to cancel commodities up to two times in one store visit via the screen of the digital signage terminal <NUM>.

In order to limit the number of cancellations, when the number of cancellations stored in the data record D1 reaches two, the virtual cart processing unit <NUM> does not generate the cancellation data even when the instruction is notified from the input detection unit <NUM>. Instead, when the virtual cart processing unit <NUM> receives the instruction from the input detection unit <NUM> in a state where the number of cancellations reaches two, the virtual cart processing unit <NUM> notifies the display control unit <NUM> of a cancellation prohibition command. In response to receiving the cancellation prohibition command, as shown in <FIG>, the display control unit <NUM> displays on the screen of the digital signage terminal <NUM> a message WM1 that notifies the member that the cancellation is no longer allowed because the number of cancellation operations exceeds the allowable number of times.

In this manner, even if the member confirms the contents of the virtual cart by using the digital signage terminal <NUM> and tries to remove a commodity improperly, it is prevented for the member to cancel three or more commodities so that the loss due to the improper cancellation can be suppressed to a low level.

The output destination of the message WM1 is not limited to the screen of the digital signage terminal <NUM>. For example, when the store manager is present in the store, the message WM1 can be displayed on a display device monitored by the store manager. This makes it possible to further enhance the effect of suppressing the improper cancellation.

As is apparent from the above description, in the commodity data processing system <NUM> of the present embodiment, the gate device <NUM> is configured to specify a customer or a registered member who enters the sales room of the store where commodities are displayed. The behavior tracking device <NUM> and the commodity identification device <NUM> are configured to acquire commodity data related to the commodities obtained by the member by behavior tracking. The virtual cart device <NUM> and the settlement device <NUM> are configured to perform a settlement process on the commodities acquired by the member based on commodity data acquired by the behavior tracking device <NUM> and the commodity identification device <NUM> when the member exits from the sales room. The behavior tracking device <NUM> and the cart confirmation device <NUM> are configured to detect that the member faces the screen of a display device, such as a digital signage terminal <NUM>. The virtual cart device <NUM> and the cart confirmation device <NUM> are configured to visualize a list of the commodities acquired by the member based on the acquired commodity data before the member exits from the sales room.

Also, the virtual cart device <NUM> and the cart confirmation device <NUM> are configured to accept an input of a cancellation instruction of a commodity displayed on the visualized list, remove the commodity data relating to the cancelled commodity from the virtual cart, and reject such a cancellation input when the number of cancellations exceeds a predetermined number.

According to the present embodiment provided with such functions, when a customer enters from an entrance gate a store where automatic registration and settlement processing are available, and takes some commodities from the commodity shelves, a commodity data processing system allows the customer to confirm the commodities to be purchased before he or she exits the store through the exit gate.

Moreover, according to the present embodiment, since the member can confirm a list of the commodities to be purchased by using the digital signage terminal <NUM> installed in the sales room, it is possible to effectively utilize the facilities in the store, thereby reducing the facility cost.

Furthermore, according to the present embodiment, since the digital signage terminal <NUM> displays a screen that visualizes the list of commodities that the member has obtained only when he or she faces the screen, it is not possible for another customer to see the list. Therefore, the privacy of the customers is protected, which contributes to reputation of the store reliability.

Furthermore, according to the present embodiment, if a customer notifies an error in the list, e.g., a commodity that is not taken by the customer is included in the list, the system allows the customer to correct the error by cancelling the commodity by his or her operations on the digital signage terminal <NUM>. On the other hand, it is possible to reduce potential damages to the store due to improper cancellations by limiting the total number of cancellation operations.

While various embodiments of the commodity data processing system in which the list of commodities obtained by the customer can be confirmed before the customer leaves the store have been described above, the embodiments are not limited thereto.

In the embodiment described above, for example, a code symbol reader is provided at an entrance gate, and a code symbol displayed on a display such as a smartphone is read by the reader. In another embodiment, a reading device for reading a recording medium such as an IC card in which a member ID is recorded may be provided at the entrance gate, and the member code recorded on the recording medium owned by the member may be read by the reading device to specify the member. Alternatively, the member face passing through the entrance gate may be photographed by a camera, and the face image of the member of the captured image and the face image of the member registered in advance may be collated with each other to identify the member.

In the embodiment described above, the digital signage terminal <NUM> is exemplified as a display device for visualizing a list of commodities that a customer has obtained. The display device is not limited to the digital signage terminal <NUM>. A dedicated display device for visualizing the list of commodities may be installed in the sales room. In addition, when a shopping basket dedicated to a store is used in the sales room, the list of commodities that are acquired by the customer may be visualized by a display device attached to the shopping basket. Alternatively, by registering information that can be communicated with an information terminal such as a smartphone at the time of membership registration, the list of commodities acquired by the member may be displayed on the display of the information terminal carried by the member.

In the above embodiment, when the member faces the screen of the digital signage terminal <NUM>, the image of the cart confirmation button B1 is displayed on a part of the screen. In this regard, the image of the cart confirmation button B1 may always be displayed on a part of the screen of the digital signage terminal <NUM>. Furthermore, a cart confirmation button B1 may be implemented as by a hardware key in an area different from the screen.

In the above embodiment, the data record D1 stored in the area of the commodity list is described as being added to the commodity data by the in-cart flag, the returned flag, and the cancelled flag. In this regard, instead of the in-cart flag, the returned flag, and the cancelled flag, a status may be added that is different among the state of the in-cart state, the returned state, and the cancellation state of the commodity.

In the commodity data processing system <NUM>, the gate device <NUM> excluding the opening/closing mechanism <NUM> and the reader <NUM>, the behavior tracking device <NUM> excluding the camera <NUM> and the sensor <NUM>, the commodity identification device <NUM>, the settlement device <NUM>, the virtual cart device <NUM>, and the cart confirmation device <NUM> each can be realized by a computer capable of performing the machine learning and the information processing related to the computer vision. That is, the components of the commodity data processing system <NUM> that can be realized by computers can be implemented by a plurality of programs having the functions including: specifying a customer who enters a sales room of a store where a commodity is displayed, acquiring commodity data related to a commodity obtained by the customer by tracking a behavior of the specified customer in the sales room, performing a settlement process on the commodity price of the commodity acquired by the customer based on the acquired commodity data when the identified customer exits the sales room, and visualizing a list of commodities acquired by the customer based on the acquired commodity data before the customer exits from the sales room.

The programs may be stored in the nonvolatile memory of the computer in advance and provided to a user. The programs may be recorded on a removable recording medium or provided via a communication medium such as the Internet and written into a storage device included in the computer by the user's operation. In this case, the recording medium may be a CD-ROM, a memory card, or the like, and the configuration thereof may be any configuration as long as the programs can be read by the apparatus.

Claim 1:
A commodity data processing system (<NUM>) comprising:
a reader (<NUM>) configured to read an ID of a customer;
a behavior tracking device (<NUM>) comprising a camera (<NUM>) configured to acquire images of the customer in a store and a plurality of sensors (<NUM>);
a cart confirmation device (<NUM>), and
a display (<NUM>) configured to display information about commodities;
wherein the plurality of sensors is installed on commodity shelves in the store and configured to detect movement of commodities displayed on the shelves,
the behavior tracking device further comprises a processor configured to:
identify the customer with the ID read by the reader when the customer enters the store;
track movement of the customer in the store based on the images acquired by the camera and identify one or more commodities taken by the customer in the store;
identify each commodity in response to a signal transmitted from one of the sensors that has detected the movement of the commodity;
acquire commodity data of each of the identified commodities;
after the customer has left the store, perform a settlement process for the commodities based on the acquired commodity data;
detect that the customer faces the display; and
upon detecting that the customer faces the display, generate and transmit to the cart confirmation device a cart confirmation command including a terminal code for specifying the display and the ID of the customer; and
the cart confirmation device is configured to, upon receiving the cart confirmation command from the behavior tracking device, control the display to display a list of the commodities before the customer leaves the store, the list including a name of each commodity and a cancellation button for cancelling the commodity.