COMMODITY INFORMATION READING DEVICE AND COMMODITY INFORMATION READING METHOD

A commodity information reading device includes an imaging device, a display device, and a controller configured to acquire an image of a commodity from an imaging device, calculate feature data of the commodity from the image, calculate a similarity value for the commodity using the calculated feature data and pre-registered feature data for a plurality of pre-registered commodities, cause commodity information of each pre-registered commodity that has a similarity value higher than a predetermined threshold value to be displayed as one candidate commodity in a plurality of candidate commodities on the display device, receive a selection designating one of the plurality of candidate commodities as a purchased commodity, and issue an alert when the designated candidate commodity does not have the highest similarity value from among the plurality of candidate commodities.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2017-085065, filed Apr. 24, 2017, the entire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to a commodity information reading device, and a commodity information reading method.

BACKGROUND

In an existing self-service checkout device, a customer carries out sales registration processing. The self-service checkout device includes a commodity identifying device for identifying a commodity being purchased. In general, the commodity identifying device can be a barcode scanner that reads a barcode label on the commodity or a camera that captures an image or images of the commodity for visual recognition purposes. Accuracy in visual recognition of commodities based on captured images has increased in recent years. However, there may still scenarios in which a particular instance of a commodity cannot be sufficiently recognized in an image due commodity position or lighting variations. In such cases, candidate matches for the commodity can be displayed on a display device so that the customer can decide which candidate matches the commodity being purchased and then select the candidate that matches the commodity.

However, in some cases, the customer may misidentify the commodity and make a selection from among the candidate matches that is different from the actual commodity being purchased. Such misidentification may be on purpose or simply due to innocent error. When the identification selection is different from the actual commodity, there is a concern that a price different from that of actually purchased commodity may be applied, and that this may lead to a failure in commodity inventory management.

The same is true when a sales person operating a point-of-sales (POS) terminal device connected to the commodity identification device selects a commodity different from an actual commodity among the displayed candidate commodities. In such cases, a price different from that of actually purchased commodity may be applied, and that this may lead to a failure in commodity inventory management.

DETAILED DESCRIPTION

In general, according to an embodiment, there is provided a commodity information reading device which includes an imaging device, a display device, and a controller configured to acquire an image of a commodity from an imaging device, calculate feature data of the commodity from the image, calculate a similarity value for the commodity using the calculated feature data and pre-registered feature data for a plurality of pre-registered commodities, cause commodity information of each pre-registered commodity that has a similarity value higher than a predetermined threshold value to be displayed as one candidate commodity in a plurality of candidate commodities on the display device, receive a selection designating one of the plurality of candidate commodities as a purchased commodity, and issue an alert when the designated candidate commodity does not have the highest similarity value from among the plurality of candidate commodities.

First Embodiment

In the example embodiment described hereinafter, a commodity information reading device provided in a self-service checkout device (hereinafter, referred to as a self-service checkout POS) is used in a store such as a supermarket. It should be noted that the particular commodity information reading devices described below are some possible examples of a commodity information reading device and do not limit the present disclosure. For example, a commodity information reading device may be connected to a sales data processing device, such as a POS terminal device, which is operated by a sales person.

FIG. 1is an external perspective view of a self-service checkout POS1. A main body2of the self-service checkout POS1is provided with a display3bthat includes a touch panel3aarranged on the surface, a scanner4, and a hand scanner5.

The display3bis an example of a predetermined display device and, for example, a liquid crystal display. The display3bdisplays a guide screen, an input screen, a registration screen, a settlement screen, or the like. The guide screen informs a customer of an operation method of the self-service checkout POS1. The input screen receives an input of commodity information of a commodity to which a code symbol is not attached. The registration screen displays commodity information of a commodity which is registered as purchased commodity. The settlement screen displays a total amount due, tendered amount, change due, or the like, and receives a selection of terms of payment.

The scanner4includes a reading window4aon a front face of the main body2, and an imaging unit which is built in at a position corresponding to a depth side of the reading window4ain the inside of the main body2. The imaging unit is, e.g., an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) array, and captures an image of an appearance of a commodity which is put up on the reading window4a. The self-service checkout POS1obtains commodity information such as a price, based on the captured image.

The hand scanner5is used for optically reading a code symbol such as a barcode, a two-dimensional code, or the like attached to a commodity which cannot easily be placed up in a fixed scanner4(for example, a large commodity or the like). The code symbol such as a barcode specifies a commodity code, or the like of a commodity to which the code symbol is attached.

The self-service checkout POS1includes commodity tables6and7. The commodity table6is a table on a right side of the main body2for placing an unsettled commodity in a basket. The commodity table7is a table on a left side of the main body2for placing a settled commodity. On the commodity table7, a hook for hanging bag8for bagging a settled commodity, or a temporary placing table9for temporarily placing a settled commodity before bagging is provided. The commodity placing tables6and7include a scale for checking whether or not a weight of a commodity is the same before and after settling.

The main body2further includes a card insertion slot10, a receipt issuing port11, a bill input-output port12, a coin inlet13, a coin outlet14, or the like. The card insertion slot10receives an insertion of a customer reward card or a credit card to be read by a card reader on the depth side of the card insertion slot10. The receipt issuing port11issues a receipt which is printed by a printer on the depth side of the receipt issuing port11. The bill input-output port12is an opening portion for receiving and paying of a bill by a bill input-output unit provided in the main body2. The coin inlet13is an opening portion for receiving coin by a coin input-output unit provided inside the main body2. The coin outlet14is an opening portion for paying a coin to the outside of the main body2by the coin input-output unit.

In addition, the main body2includes a pole on a top face, and a warning lamp15at a top portion of the pole. The warning lamp15is turned on for calling a sales person or the like when there is abnormality in operation in the self-service checkout POS1.

FIG. 2is a block diagram of a configuration of the self-service checkout POS1. The self-service checkout POS1includes a control unit21, a bus22, a communication I/F (interface)23, an input/output (I/O) device control unit24, a card reader25, a printer26, the bill input-output unit27, the coin input-output unit28, and a hard disk drive (HDD)29.

The HDD29stores various programs and various data such as a PLU file which are executed by a CPU in a rewritable manner. The HDD29is an example of a storage unit and the PLU file is an example of commodity information which includes a price. As the above described program, for example, there is a program for performing commodity sales registration processing, or object recognizing processing.

The PLU file is a file which stores commodity information of each commodity. A commodity code as a unique value of each commodity, or information related to an appearance of a commodity (referred to as appearance information) such as feature data which is used when recognizing an object is included in the commodity information. The PLU file is delivered and updated by the store server30, and is stored in the HDD29of the self-service checkout POS1. The examples described above are some possible examples of commodity information processing and do not limit other commodity information processing. For example, the store server30may store the PLU file and the control unit21may use commodity information by accessing the PLU file of the store server30through a network NT.

The control unit21has a computer configuration having a central processing unit (CPU), a read only memory (ROM) as a storage unit, a random access memory (RAM), and the like. The ROM stores various programs which are executed by the CPU, or various data. The RAM temporarily stores the data or the program when the CPU executes the various programs. The data stored in the RAM is for example, sales data which denotes a sales record of a commodity, or various screen display data which are displayed on the display3b.

The control unit21controls the touch panel3a, the display3b, the scanner4, the hand scanner5, the warning lamp15, the card reader25, the printer26, the bill input-output unit27, the coin input-output unit28, and an I/O device such as the HDD29through the bus22and an I/O device control unit24, and controls the entire system of the self-service checkout POS1.

The card reader25reads information of a customer reward card or a credit card which is inserted from the card insertion slot10. The printer26prints a receipt after sales transaction, and issues the receipt from the receipt issuing port11.

The bill input-output unit27treats a bill which is input or output from the bill input-output port12. In addition, the coin input-output unit28treats a coin input from the coin inlet13and a coin paid to the coin outlet14.

The control unit21and the store server30can communicate with each other using the network NT such as a local area network (LAN) through a communication interface23or attendant sales person terminal (not specifically depicted) for a sales person who supports self-service checkout sales transaction. The store server30generally performs a management of commodity sales registration processing, or the like, which is performed in one or more the self-service checkout POS1in a store.

The store server30is a device which has a computer configuration having a control unit31including a CPU, a ROM, and a RAM, and a communication interface32, a display33, and an input device such as a keyboard34. The control unit31performs a data communication with the self-service checkout POS1through the communication interface32, and the network NT.

FIG. 3is a block diagram of a functional configuration of the self-service checkout POS1. The control unit21of the self-service checkout POS1functions as a fetching unit40, a similarity calculating unit41, an option display unit42, an operation reception unit43, an informing unit44, a list display unit45, a sales registration unit46, a printing control unit47, a display control unit48, and a commodity reading unit49, by executing the program according to the embodiment which is stored in the ROM, by developing the program in the RAM.

The display control unit48controls display processing in the display3bthrough the I/O device control unit24, and causes various screens to be displayed. The printing control unit47controls printing of a receipt using the printer26, through the I/O device control unit24.

The fetching unit40obtains an image captured by an imaging unit of the scanner4.

The commodity reading unit49determines whether or not a code symbol such as a barcode is included in the captured image obtained by the fetching unit40, and when the code symbol is included, the commodity reading unit obtains commodity information which matches a value derived from the code symbol (including a commodity code or the like) from the PLU file.

When the hand scanner5reads a code symbol such as a barcode, the commodity reading unit49obtains commodity information which matches a value derived from the code symbol (including a commodity code or the like) from the PLU file.

The similarity calculating unit41firstly calculates feature data of a commodity in the image captured by the fetching unit40, and secondly calculates similarity between the calculated feature data and feature data of each pre-registered commodity stored in the PLU file.

The feature data is, for example, a color of the surface, a pattern, a state of unevenness, or the like. The similarity calculating unit41obtains feature data from a captured image by object recognition. The similarity calculating unit41compares feature data of the captured image with pre-registered feature data of each commodity recorded in the PLU file and pre-registered commodity having a high similarity (referred to as a degree of matching) with the commodity in the captured image.

When there is commodity having a similarity higher than a predetermined threshold value, the similarity calculating unit41determines that the commodity put up on the reading window4ais the commodity having the pre-registered feature, and outputs the commodity information as single candidate commodity.

When there is no commodity having a similarity higher than a predetermined threshold value, the similarity calculating unit41selects a plurality of commodity information in the order of high similarity and outputs thereof as candidate commodities. When there is no commodity having a similarity a predetermined lower threshold, the similarity calculating unit41outputs a message indicating that there is no candidate commodity.

When the similarity calculating unit41outputs single candidate commodity, the option display unit42displays the single candidate commodity on the display3b.

When there is a plurality of candidate commodities output from the similarity calculating unit41, the option display unit42causes the candidate commodity to be displayed on the display3bin the order of high similarity to the commodity which is put up on the reading window4a.

FIG. 4depicts an example of a candidate selecting screen50.

On the candidate selecting screen50, the option display unit42displays candidate commodities classified by type and size. Here, the “size” may mean a half, or a fourth of a same one commodity, or a combined number of one commodity in a bag or a package, or a difference in size or weight of one commodity.

The candidate selecting screen50includes selecting buttons51a,51b,51c,52a,52b,52c,53a, and53bwhich display a plurality of candidate commodities so as to be selected, a re-recognizing button54, and thick frames55,56, and57which denote a classification. The re-recognizing button54is a button which is pressed when there is no commodity corresponding to the selecting buttons51ato53b. When the re-recognizing button54is pressed, the process returns to a reading step using the scanner4, by clearing candidate commodity.

The selecting buttons51ato53bare arranged on the candidate selecting screen50in the order of high similarity from above. In the example illustrated inFIG. 4, a candidate having the highest similarity to a commodity which is put up on the reading window4ais a cabbage, a candidate having the second highest similarity is a Chinese cabbage, and a candidate having the third highest similarity is a bell pepper.

In a store in this example, a cabbage and a Chinese cabbage are sold in various sizes such as a while, a half, and a quarter. Bell peppers are sold individually and several packed in a bag. Each of the thick frames55to57in the candidate selecting screen50denotes a same commodity in different sizes. For example, the selecting button51afor a whole cabbage, the selecting button51bfor a half of a cabbage, and the selecting button51cof a quarter of are cabbage are disposed side by side in a line, and are surrounded with the thick frame55. The selecting buttons52ato52cfor a Chinese cabbage and the selecting buttons53aand53bfor a bell pepper are arranged similarly.

The operation reception unit43is a functional unit which receives various instructions from a customer operating the self-service checkout POS1through the touch panel3a, the scanner4and the hand scanner5. The instructions include, for example, a selection of one of the candidate displayed on the option display unit42as a commodity being purchased. When the option display unit42displays a single candidate commodity, the instruction includes a confirmation of the candidate commodity is determined (acknowledged) as a commodity being purchased. The instructions also include an instruction to start over the step of reading without selecting any one of the candidate commodities as a commodity being purchased.

The operation reception unit43reads a commodity code corresponding to a commodity to be purchased by a customer from the PLU file when a customer presses an input button corresponding to the commodity.

The sales registration unit46performs a sales registration with respect to commodity information of total commodities being purchased by a customer, and calculates a total amount due. The total commodities include a commodity which a customer has selected via the operation reception unit43from the candidate commodities on the option display unit42, and a commodity of which a code symbol such as a barcode has been read.

The list display unit45, in cooperation with the display control unit48, displays a list of commodity information which is subjected to a sales registration by the sales registration unit46on the registration screen on the display3b. Here,FIG. 5depicts an example list61on a registration screen60.

The informing unit44issues an alert (also referred to as a warning or an alarm) in a predetermined event or situation. The alert is, for example, a predetermined image displayed on a predetermined display device, or predetermined sound. The predetermined event or situation is, for example, when among a plurality of candidate commodities denoted by the option display unit42, a candidate commodity having a similarity that is not the highest is selected as a commodity being purchased by a customer in the operation reception unit43. The predetermined display device is for example, the display33of the store server30, or the display3bof the self-service checkout POS1.

With increased accuracy in object recognition in recent years, a candidate commodity not having the highest similarity is unlikely to be elected due to an error in object recognition. Thus, when a candidate commodity not having the highest similarity is selected, there is a probability of errors in the registration processing.

The informing unit44issues an alert on the display33of the store server30, which is a terminal device operated by a sales person.FIG. 6depicts an example alert screen70displayed on the display33.

The alert screen70includes, for example, a message71, a note72, an OK button73, and a terminal stop button74. The message71is, for example, “an unexpected item is selected.”. The note72is a note for denoting a detail of alarm contents, for example. The note72may be omitted. The OK button73receives an instruction of closing the alert screen70. The terminal stop button74receives an instruction of stopping the display of the alert.

The informing unit44issues an alert on the registration screen60of the self-service checkout POS1.FIG. 7depicts example alerts62and63which are displayed on the registration screen60.

The alert62on the registration screen60may be a mark such as a character, sign, badge, or the like in the vicinity of the selected commodity in the list61. The alert63explains the reason for displaying the mark to a customer.

The control unit21informs there is a possibility of misregistration (referred to as a misregistration informing processing).FIG. 8is a flowchart of an example flow of misregistartion informing processing by the control unit21.

The control unit21waits for an output of the fetching unit40(No in ACT1). The fetching unit40obtains an image captured by the imaging unit, when a commodity is put up on the reading window4a(Yes in ACT1).

When there is an output from the fetching unit40, the similarity calculating unit41performs object recognizing processing, and calculating and comparing of feature data, and outputs one or more candidate commodities. The option display unit42determines whether or not more than one output candidate commodities (ACT2). When there are more than one candidate commodities (Yes in ACT2), the option display unit42classifies the candidate commodities by type, size, or the like, and displays the candidate commodities on the candidate selecting screen50(ACT3).

The operation reception unit43waits for an instruction by pressing of the selecting buttons51ato53b(No in ACT4), and proceeds the processing to the subsequent ACT5when receiving the instruction (Yes in ACT4).

In ACT5, the control unit21determines whether or not the selected candidate commodity is the candidate commodity having the highest similarity, and when the selected candidate commodity is the candidate commodity having the highest similarity (Yes in ACT5), the control unit causes the selected candidate commodity to be displayed on the list61of the registration screen60(ACT6), and ends the processing.

In ACT5, when the selected candidate commodity is not the candidate commodity having the highest similarity (No in ACT5), the informing unit44issues an alert (ACT7), and the control unit21proceeds the processing to ACT6.

In ACT2, when there is a single candidate commodity (No in ACT2), and when the operation reception unit43receives an instruction from a customer for confirming the candidate commodity as a commodity being purchased (Yes in ACT8), the control unit21proceeds the processing to ACT6. In ACT8, when the confirmation from a customer is not received (No in ACT8), the control unit21prompts the customer for an instruction (ACT9).

In this manner, when a candidate commodity not having the highest similarity is selected, since it is possible to inform a sales person, or a customer of such possible registration errors in selecting a candidate commodity that corresponds to the actual commodity, an occurrence of registration errors may be reduced.

In the related art, an error in which a commodity different from the actual commodity is registered is detected based on a weight measured by a scale or the like on the commodity placing tables6and7. However, according to the example embodiments described above, it is possible to improve an accuracy in detecting such errors in combination with a weight measurement in the related art together, or it is possible to reduce a cost by omitting a scale.

In the example embodiments described above, when one commodity information is selected in the similarity calculating unit41, the commodity is displayed as single candidate commodity, and an acknowledging operation of a customer is prompted. However, an acknowledging operation is not limited to this example, and for example, when one commodity information selected by the similarity calculating unit41, the single candidate commodity may be determined as a commodity being purchased without an acknowledging operation by a customer.

In the example embodiments described above, an alert is issued when a candidate commodity not having a highest similarity is selected. However, an alert issuance is not limited to this example. For example, when there is a plurality of candidate commodities not having the highest similarity, the informing unit44may issue an alert only when a price of a selected candidate commodity has a lowest price among the plurality of candidate commodities. In this manner, it is possible to avoid issuing unnecessary alerts.

Second Embodiment

FIG. 9depicts another example alert on the display3bof the self-service checkout POS1. In the modification example, an alert screen80is displayed every time a commodity that may be different from the actual commodity is registered. The alert screen80includes a message81, and buttons82and83. The message81is a message which asks a customer operating the self-service checkout POS1such as “Is this an item you scanned?”. The button82is a button for the customer to confirm that “Yes, please proceed to a next item”, for example, and receives an instruction for moving to the subsequent step. The button83is a button for the customer to indicate that “No, please return to options,” for example, and receives an instruction for returning to the previous step.

In the example embodiments described above, the program which is executed in the commodity information reading device is stored in the ROM, or the like in advance.

The program which is executed in the commodity information reading device may be recorded in a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD).

The program which is executed in the commodity information reading device may be stored in a computer which is connected to a network such as the Internet, and be downloaded through the network. The program which is executed in the commodity information reading device may be provided or distributed through a network such as the Internet.

The program which is executed in the commodity information reading device may be formed in a module configuration which includes the above described each unit (fetching unit40, similarity calculating unit41, option display unit42, operation reception unit43, informing unit44, list display unit45, sales registration unit46, printing control unit47, display control unit48, and commodity reading unit49). The processor downloads the above described each unit to a main storage device, by reading the program from the above described storage medium, and executing the program. In this manner, the fetching unit40, the similarity calculating unit41, the option display unit42, the operation reception unit43, the informing unit44, the list display unit45, the sales registration unit46, the printing control unit47, the display control unit48, and the commodity reading unit49are generated in the main storage device.