SYSTEM FOR REGISTERING AN ITEM FOR PURCHASE AT A STORE

A system for registering an item for purchase at a store includes a terminal attachable to a shopping cart and a camera attached to the cart and positioned to capture an image of an item stored therein. The terminal includes: an interface circuit connectable to the camera, and a processor configured to: control the camera to capture an image, execute object recognition processing on the image and determine one or more candidate items using a result of the recognition processing, when only one candidate item is determined by the recognition processing, register the candidate item as an item for purchase, and when two or more candidate items are determined by the recognition processing, execute character recognition processing on the image and determine a character string in the image, and register one of the candidate items as an item for purchase using a result of the character recognition processing.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2023-062902, filed Apr. 7, 2023, the entire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to a system, a method, and a terminal for registering an item for purchase at a store.

BACKGROUND

An item registration system for stores has been proposed in which items placed in a shopping basket are recognized from a captured image and automatically registered for purchase. However, some items are similar in appearance, and it may be difficult to recognize such items based only on their features shown in the image.

In view of such circumstances, it is desirable to be able to accurately identify items shown in an image.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a system, a method, and a terminal capable of accurately identifying an item shown in a captured image.

In one embodiment, a system for registering an item for purchase at a store comprises a terminal attachable to a shopping cart; and a camera attached to the shopping cart and positioned to capture an image of an item stored therein. The terminal includes: an interface circuit connectable to the camera, and a processor configured to: control the camera to capture an image, execute object recognition processing on the image and determine one or more candidate items using a result of the object recognition processing, when only one candidate item is determined by the object recognition processing, register the candidate item as an item for purchase, and when two or more candidate items are determined by the object recognition processing, execute character recognition processing on the image and determine a character string shown in the image, and register one of the two or more candidate items as an item for purchase using a result of the character recognition processing.

DETAILED DESCRIPTION

Hereinafter, embodiments will be described in detail with reference to the drawings. The present invention is not limited to the embodiments described below.

FIG.1is a schematic block diagram illustrating a configuration of a transaction processing system1and a main circuit configuration of a cart terminal100according to an embodiment.

The transaction processing system1includes a cart terminal100, a checkout machine200, a POS (point of sale) server300, and a store clerk terminal400, which can communicate with each other via a communication network2.

As the communication network2, the Internet, VPN (virtual private network), a LAN (local area network), a public communication network, a mobile communication network, or the like can be used singly or in combination as appropriate. As an example of the communication network2, the Internet and a mobile communication network are used in combination.

Any number of cart terminals100, checkout machines200, POS servers300, and store clerk terminals400may be included in the transaction processing system1unlike the example illustrated inFIG.1.

The cart terminal100is an information processing terminal attached to a shopping cart (hereinafter referred to as a cart) provided in a store. The cart terminal100is lent together with the cart to a customer who shops at a store. The cart terminal100is a terminal device that executes a registration processing for registering an item for which a customer is planned to purchase as an item for purchase, in response to an operation by the customer. As will be described later, the cart terminal100executes a process of recognizing an item to be registered as an item for purchase from an image. That is, the cart terminal100has a function as an item recognition apparatus. As will be described later, the cart terminal100executes a process of registering the recognized item as an item for purchase. That is, the cart terminal100also has a function as a registration system.

The checkout machine200is installed in a store and executes a checkout process related to payment of the price of the item for purchase registered by the cart terminal100. The checkout machine200receives an operation by an operator at the time of checkout processing. The operator of the checkout machine200is mainly a customer. In some cases, the store clerk is an operator of the checkout machine200.

The POS server300is an information processing device that executes a management process for managing a sales transaction of the item processed by the registration processing in the cart terminal100and the checkout process in the checkout machine200.

The store clerk terminal400is an information processing terminal operated by a store clerk. The store clerk terminal400is a terminal device for a user interface related to information processing for assisting a store clerk's work related to a transaction processed by the transaction processing system1. The operation of the store clerk is, for example, an operation of monitoring the execution status of the transaction being processed and appropriately assisting the customer.

The cart terminal100includes a processor101, a memory102, an auxiliary storage unit103, a touch panel104, an interface unit105, a wireless communication unit106, a transmission path107, and the like. The processor101, the memory102, the auxiliary storage unit103, the touch panel104, the interface unit105, and the wireless communication unit106can communicate with each other via the transmission path107.

In one embodiment, the processor101, the memory102, and the auxiliary storage unit103connected via the transmission path107form a controller that performs information processing for controlling the cart terminal100.

The processor101is a core component of the controller. The processor101executes information processing for controlling each unit so as to perform various functions as the cart terminal100in accordance with one or more programs including an operating system and application programs.

The memory102includes a read-only memory area and a rewritable memory area. The read-only memory area of the memory102stores one or more programs. In some cases, the read-only memory area and/or the rewritable memory area of the memory102store data necessary for the processor101to execute processing for controlling each unit. The rewritable memory area of the memory102is used as a work area by the processor101.

The auxiliary storage unit103may be, for example, an EEPROM (electric erasable programmable read-only memory), an HDD (hard disc drive), an SSD (solid state drive, or any other known storage device. The auxiliary storage unit103stores data used by the processor101to perform various types of processing and data generated thereby. The auxiliary storage unit103may store one or more programs. In an embodiment, the auxiliary storage unit103stores a cart terminal program PRA. The cart terminal program PRA is an application program for performing the steps of registration processing. A part of the storage area of the auxiliary storage unit103is used as an area for storing a dictionary database DBA and transaction data DAA. The dictionary database DBA is a set of dictionary data which is referred to in order to identify an item shown in a captured image. The transaction data DAA is data representing the content of one transaction.

The interface unit105is connected to an external device such as a camera199. The interface unit105interfaces data exchange with a connected external device. As the interface unit105, a conventional USB (universal serial bus) controller or the like can be used. The camera199captures an image of a shooting range including the entire interior space of a shopping basket placed on a cart. Then, the camera199outputs image data representing the image to the cart terminal100. Note that the camera199may repeat the shooting at particular intervals, or may perform the shooting in response to an instruction from the processor101.

The wireless communication unit106is a network interface circuit that executes communication processing for wirelessly performing data communication via the communication network2. As the wireless communication unit106, for example, a conventional wireless communication device for wireless LAN can be used. In place of or in addition to the wireless communication unit106, a communication device wired to the communication network2may be used.

The transmission path107includes an address bus, a data bus, a control signal line, and the like, and conveys data and control signals transmitted and received between the connected components.

As the hardware of the cart terminal100, for example, a tablet-type information processing apparatus can be used.

FIG.2depicts a data structure of a data record REA included in the dictionary database DBA.

The dictionary database DBA is a set of data records REA each associated with an item sold in a store in which the cart terminal100is used. The data record REA includes fields FAA, FAB, FAC, and FAD. An item code as an identifier of the associated item is set in the field FAA. The item name of the associated item is set in the field FAB. In the field FAC, object information including a plurality of feature points to be referred to in order to recognize the associated item by object recognition processing to be described later is set. In the field FAD, character information including a plurality of character string information each representing a plurality of character strings printed on the associated item is set.

The dictionary database DBA is set up by an appropriate worker in view of items sold at a store, for example, when the transaction processing system1is introduced to the store, and is stored in the auxiliary storage unit103of the cart terminal100used in the store. The dictionary database DBA may be appropriately updated by the operator or the like at the time of an appropriate maintenance operation even after the use of the transaction processing system1is started.

FIG.3is a perspective view illustrating the cart terminal100attached to a cart900. The cart900includes a caster unit910, a handle frame unit920, and a basket receiving unit930. The caster unit910has four wheels911for smoothly moving the cart900on a sales floor. The wheels911are attached to a frame912to be rotatable about a vertical axis.

The handle frame unit920includes a pair of vertical frames921and a handlebar922. The vertical frames921are erected above two wheels of the caster unit910. The handlebar922connects the upper ends of the vertical frames921.

The basket receiving unit930is attached to an intermediate portion of the handle frame unit920. The basket receiving unit930holds a shopping basket SB as a container for storing items. The caster unit910can also hold a shopping basket SB on the frame912.

A customers using the cart900is typically located on the front side of the handle frame unit920inFIG.3. The customer then pushes the cart900while grasping the handlebar922. That is, in this case, the direction in which the basket receiving unit930protrudes with respect to the handle frame unit920is the forward direction of the cart900.

The camera199is attached to an intermediate portion of the handlebar922so that the inside of the shopping basket SB held by the basket receiving unit930can be photographed in a bird's-eye view from above. Further, a pole940is attached to one of the vertical frames921so that the distal end thereof is positioned above the handlebar922. The above-described cart terminal100is attached to the distal end portion of the pole940in a posture in which the touch panel104faces the opposite side in the forward direction.

A battery case950is mounted between the vertical frames921on the lower end side of the handle frame unit920. The battery case950houses a battery used as an external power source of the cart terminal100.

Next, the operation of the transaction processing system1configured as described above will be described. Note that the steps described below are examples, and the order of some of the steps may be changed, some of the steps may be omitted, or other steps may be added as appropriate. For example, in the following description, descriptions of some steps are omitted in order to explain characteristic operations of the present embodiment in an easy-to-understand manner. For example, in a case where some kind of error occurs, steps for coping with the error may be performed, but descriptions for such steps are omitted.

The customer performs a predetermined operation for starting use at the cart terminal100that is in an unused state. When this operation is performed, the processor101starts registration processing based on the cart terminal program PRA.

FIG.4is a flowchart of the registration processing.

In ACT11, the processor101generates new transaction data DAA for a transaction. That is, the processor101determines, for example, a new transaction code different from the transaction codes for identifying the other existing transactions according to a predetermined rule, generates new transaction data DAA including the new transaction code and not including any item information, and stores the new transaction data DAA in the auxiliary storage unit103.

In ACT12, the processor101checks whether it is a retrieval timing. The processor101determines NO if the corresponding event cannot be confirmed, and proceeds to ACT13. Note that the retrieval timing may be, for example, a timing immediately after proceeding from ACT11to ACT12and a timing at every predetermined time interval thereafter, and may be appropriately determined by, for example, a provider of the cart terminal program PRA.

In ACT13, the processor101checks whether a checkout has been requested. The processor101then determines NO if the event cannot be ascertained and returns to ACT12.

Thus, the processor101waits a retrieval timing or a checkout request in ACT12and ACT13.

The processor101determines YES in ACT12at the retrieval timing, and proceeds to ACT14.

In ACT14, the processor101retrieves image data output by the camera199. For example, if the camera199repeats shooting at predetermined intervals, the processor101retrieves image data that is output in accordance with the shooting most recently performed by the camera199. For example, if the camera199captures an image in response to an instruction from the processor101, the processor101instructs the camera199to capture an image, and in response, the camera199captures an image and acquires image data to be output. The processor101stores the retrieved image data in the memory102or the auxiliary storage unit103.

In ACT15, the processor101checks whether an item has been placed in the shopping basket SB. The processor101determines NO if the corresponding event cannot be confirmed, and proceeds to ACT16.

In ACT16, the processor101checks whether an item has been removed from the shopping basket SB. The processor101then determines NO if the event cannot be ascertained and returns to the standby status in ACT12and ACT13.

If the timing immediately after the advance from ACT11to ACT12is the first retrieval timing, an item other than the shopping basket SB is not shown in the image represented by the image data retrieved by the first execution of ACT14. Therefore, when the processor101executes ACT15and ACT16for the first time, it determines that both ACT15and ACT16are NO and returns to the standby status of ACT12and ACT13. Then, the processor101repeats ACT14every retrieval timing. At this time, the processor101leaves the image data retrieved by the execution of ACT14in the past predetermined period and already stored in the memory102or auxiliary storage unit103(hereafter referred to as past data) in the memory102or auxiliary storage unit103as it is.

The customer searches for a transaction target item on the sales floor of the store. Then, the customer takes out an item to be purchased and puts the item into the shopping basket SB.

For example, the processor101checks whether any item is newly placed in the shopping basket SB on the basis of a difference between image data newly acquired in ACT14(hereinafter, referred to as current data) and past data. The processor101may monitor movement of the customer's hand holding some item on the basis of the current data and the plurality of past data, and determine that the item has been placed in the basket based on the result. If the processor101determines that any item is newly placed in the shopping basket SB, it determines YES in ACT15and proceeds to ACT17.

In ACT17, the processor101detects an item that has been put into the shopping basket this time (hereinafter, referred to as a “placed item”). For example, the processor101detects a placed item that is not shown in an image represented by the past data but is shown in an image represented by the current data by comparing those images. As a method of the process, a well-known method such as deep learning using CNN (convolutional neural network), for example, can be used.

In ACT18, the processor101executes item identification processing for identifying the placed item detected in ACT17.

FIG.5is a flowchart of the item identification processing.

In ACT31, the processor101performs object recognition processing on the item detected in ACT17. The object recognition processing selects, as a candidate item, an item that is considered to coincide with the placed item by comparing the features or feature points of the appearance of the placed item shown in the image represented by the current data with the feature points represented by the object information included in the respective data record REA of the dictionary database DBA. In this object recognition processing, one or a plurality of candidate items may be selected. Also, in the object recognition processing, no candidate item may be selected in some cases. As a method of image identification in such object recognition processing, for example, a well-known method such as feature point matching can be used. In addition, as a method of selecting a candidate item according to the outcome of the image-identification, a well-known method such as deep learning using the CNN can be used.

In ACT32, the processor101checks whether there is one candidate item selected in ACT31. Then, if a plurality of candidate items are selected in ACT31or no candidate item can be selected, the processor101determines that the candidate item is not one and determines NO, and proceeds to ACT33.

In ACT33, the processor101checks whether there are a plurality of candidate items selected in ACT31. The processor101determines YES if a plurality of candidate items are selected in ACT31, and proceeds to ACT34.

In ACT34, the processor101performs character recognition processing on the placed item as a target. That is, the processor101recognizes a character string shown in the image of the placed item. As a specific method of the character recognition processing, for example, a well-known method such as deep learning using the CNN can be used.

In ACT35, the processor101narrows down a plurality of candidate items selected in ACT31based on a result of the character recognition processing performed in ACT34. For example, the processor101finds a data record REA associated with each of the plurality of candidate items in the dictionary database DBA. Then, for example, the processor101selects, from the corresponding data record REA, the data record REA in which any one of the character strings represented by the character string included in the character information set in the field FAD matches the character string recognized by the character recognition processing in ACT34, and sets the item associated with the data record REA as a candidate item after narrowing down. If the characters included in the character information set in the field FAD of each data recode REAdo not match the characters recognized by the character recognition processing in ACT34, the processor101may select the item associated with the data record REA having a greatest or greater number of matches as the candidate item.

FIG.6is a diagram illustrating items to be narrowed down by character strings.

For example, as shown inFIG.6, two items COA and COB of the same brand with different sizes are sold at a store. Different items of the same brand have a common package design as in the example shown inFIG.6, and may not be identified in object recognition processing. That is, in the object recognition processing when the item COA illustrated inFIG.6is placed in the shopping basket SB, any of the items COA and COB can be set as a candidate item. In such a case, for example, when the character string “350 mL” can be recognized from the image of the placed items by character recognizing processing, the candidate items are narrowed down to the item COA. However, the character string “350 mL” represented in the placed item may not be shown in the images when that character string is hidden by the hand of the customer, the candidate items cannot be narrowed down.

In ACT36, the processor101checks whether there is one candidate item after narrowing down in ACT35. Then, the processor101determines NO if the candidate items have not been narrowed down to one, and proceeds to ACT37.

In ACT37, the processor101controls the touch panel104to display a selection screen. The selection screen is a GUI (graphical user interface) screen for allowing a customer to select one of the plurality of candidate items.

FIG.7is a diagram illustrating an example of a selection screen.

The selection screen illustrated inFIG.7is an exemplary screen in which the two items COA and COB illustrated inFIG.6are candidate items.

The selection screen includes four buttons BUA, BUB, BUC, and BUD for candidate items. InFIG.7, the items COA and COB are assigned to the buttons BUA and BUB respectively. The buttons BUA, BUB, BUC, and BUD are soft keys for the customer to designate one of the assigned items.

When such a selection screen is displayed, the customer designates one of the candidate items by tapping on a button to which the item placed in the shopping basket SB is assigned.

The processor101proceeds to ACT38inFIG.5with the selection screen displayed. In ACT38, the processor101waits for one of the candidate items to be designated. Then, if the item is designated by the above-described operation, the processor101determines YES, and proceeds to ACT39.

In ACT39, the processor101sets only the designated item as the candidate item.

That is, the processor101accepts the operation of designating one of the plurality of candidate items on the selection screen.

On the other hand, if no candidate item can be selected by object recognition processing in ACT31, the processor101determines NO in ACT33and proceeds to ACT40.

In ACT40, the processor101controls the touch panel104to display a guidance screen. The guidance screen is a screen for guiding a customer to scan a barcode attached to the item that has just been placed in the shopping basket SB (hereinafter, referred to as an item barcode) with the camera199.

FIG.8is a diagram of a guidance screen.

The guidance screen shown inFIG.8includes an area ARA. The processor101acquires image data output by the camera199and controls the touch panel104to display the image represented by the image data in the area ARA.

The processor101proceeds to ACT41inFIG.5with the guidance screen displayed.

In ACT41, the processor101waits for the retrieval timing of images. Then, the processor101determines YES at the retrieval timing, and proceeds to ACT42. It is assumed that the retrieval timing is, for example, a timing for each predetermined time interval, and may be appropriately determined by, for example, a provider of the cart terminal program PRA.

In ACT42, the processor101acquires image data output by the camera199. At this time, the processor101controls the touch panel104to display the image represented by the newly retrieved image data on the area ARA. This makes it easy for the customer to scan an item barcode with the camera199while confirming that the barcode is reflected in the image displayed on the area ARA.

In ACT43, the processor101performs item barcode reading processing on the image data acquired in ACT42. That is, the processor101attempts to detect and read an item barcode in the image represented by the image data acquired in ACT42.

In ACT44, the processor101checks whether any item barcode has been detected and read in ACT43. The processor101cannot detect any item barcode in ACT43when an item is not properly scanned with the camera199so that the item barcode is reflected. In such cases, the processor101determines NO in ACT44, returns to ACT41standby status, and repeats ACT41to ACT44until the item barcode can be extracted in ACT43. Then, if an item barcode can be detected in ACT43, the processor101determines YES in ACT44and proceeds to ACT45.

In ACT45, the processor101sets an item identified by the item code represented by the extracted item barcode as a candidate item.

When the processor101finishes setting one item as the candidate item in ACT39or ACT45as described above, the item identification processing ends. If there is only one candidate item selected in ACT31, the processor101determines YES in ACT32, passes the processing after ACT33, and ends the item identification processing as it is. The processor101determines YES in ACT36if there is only one candidate item after narrowing down in ACT35, passes the processing after ACT37, and ends the item identification processing as it is. That is, the processor101terminates the item identification processing in a state where only one candidate item is specified. When the processor101finishes the item identification processing In ACT18ofFIG.4in this way, it proceeds to ACT19.

As described above, when ACT35has been performed, the processor101identifies an item as one of the candidate items narrowed down based on the characters recognized by character recognition processing.

In ACT19, the processor101registers one candidate item specified by the item identification processing as an item for purchase. That is, for example, the processor101updates the transaction data so as to include the item code of the one candidate item specified by the item identification processing as the item code of the item for purchase. The processor101then returns to the standby status in ACT12and ACT13.

Once the customer changed their mind and decided not to purchase an item placed in the shopping basket SB, the customer removes the item from the shopping basket SB and returns the item to the store counter.

For example, the processor101checks whether an item has been removed from the shopping basket SB on the basis of a difference between the current data and the past data newly acquired in ACT14. The processor101may monitor the movement of the customer's hand holding an item on the basis of the current data and the plurality of past data, and determine that an item has been removed based on the result. Then, if the processor101determines that an item has been removed, it determines YES in ACT16and proceeds to ACT20.

In ACT20, the processor101detects an item (hereinafter, referred to as a removed item) extracted from the shopping basket SB. For example, the processor101detects an extracted item which is shown in an image represented by the past data and is not shown in an image represented by the current data by comparing those images. As a method of the process, for example, a well-known method such as deep learning using CNN can be used.

In ACT21, the processor101executes item identification processing for specifying which item has been removed. Here, the item identification processing is the process illustrated inFIG.5as in ACT18.

In ACT22, the processor101excludes one candidate item identified by the item identification processing from the registered items for purchase. That is, for example, the processor101updates the transaction data so as to reduce the quantity related to the item code of one candidate item identified by the item identification processing by one. The processor101then returns to the standby status in ACT12and ACT13.

Upon completion of the registration of the item for purchase, the customer performs a predetermined operation for instructing the start of checkout processing, for example, on the touch panel104. In response, the processor101determines YES in ACT13and proceeds to the checkout processing. The checkout processing may be similar to checkout processing performed by conventional cart POS systems. The illustration thereof is omitted. For example, the processor101performs processing for transferring transaction data to the checkout machine200as checkout processing. Alternatively, if a payment terminal (not shown) is attached to the cart900and the payment terminal is connected to the cart terminal100, processing for performing credit payment, barcode payment, or the like using the payment terminal may be performed as checkout processing.

As described above, when candidate items cannot be narrowed down to one by object recognition processing, the cart terminal100narrows down the candidate items based on the characters recognized by character recognition processing, and identifies the item included in the shopping basket SB as one of the candidate items narrowed down in this way. As a result, an item placed in the shopping basket SB and shown in a captured image can be accurately identified.

In addition, the cart terminal100identifies an item included in the shopping basket SB as one of the plurality of candidate items in accordance with the instruction of the customer made on the selection screen, so that the item included in the shopping basket SB can be identified even when the candidate items cannot be narrowed down to one using character recognition processing. In this case, a designation operation by the customer is required, but since the number of candidate items can be reduced by narrowing down the number of candidate items based on characters, a burden on the customer for designating one of the candidate items is small.

In addition, when one or more candidate items cannot be identified by object recognition processing, the cart terminal100identifies the item placed in the shopping basket SB based on its barcode. Accordingly, even if the object recognition fails, the item placed in the shopping basket SB can be identified.

Further, the cart terminal100registers, as an item for purchase, the item identified as being placed in the shopping basket SB. Thus, the transaction processing system1has a function as a registration system capable of recognizing an item placed in a shopping basket by a customer at a store from an image obtained by photographing the shopping basket and automatically registering the item as an item for purchase with high accuracy.

Various modifications can be made to this embodiment as follows.

It is also possible to recognize an object other than an item sold in a store.

The recognition result of the item is not limited to item registration in sales transactions, and may be used for any purpose.

A part of the processing executed by the processor101in the above-described embodiment may be executed by the POS server300or any one or a plurality of other information processing apparatuses. For example, the cart terminal100may perform only a user interface operation, and various information processing for specifying an item for purchase may be executed by a computer provided in the POS server300. In other words, each of the above-described steps may be performed by any information processing device other than the cart terminal100included in the transaction processing system1, or may be performed by a number of information processing devices included in the transaction processing system1as appropriate and dispersed.

Each of the functions performed by the processor101by the information processing may be performed by hardware that executes information processing that is not based on a program such as a logic circuit or the like. Each of the above-described functions can also be performed by a combination of software and hardware such as the above-described logic circuit.