Patent Description:
There are information processing devices, such as Point of Sale (POS) terminals, that identify a commodity to be purchased by a customer with generic object recognition or the like. Further, there are information processing devices that identify a commodity either by performing the object recognition or by scanning of a code symbol attached to the commodity, such as a barcode.

It is typically considered that a commodity is identified more accurately and rapidly by the scanning of the code symbol rather than by use of object recognition. Therefore, these information processing apparatuses preferentially perform scanning of the code symbol for any commodity on which the code symbol can be detected, and performs object recognition only on those commodities on which a code symbol is not attached or not detected. Often, commodity items, such as fruits and vegetables, do not have a code symbol attached and these items require analysis of exterior characteristics to identify.

However, in existing information processing devices capable of executing both scanning of the code symbol and object recognition, an inconvenience could occur in which object recognition is executed on an unintended article, that is, an article to which a code symbol has been attached, but which was not detected prior to initiation of object recognition processing on the article.

<CIT> relates to a system for the identification of products in a shopping cart comprising a computer comprising a database; an optical camera for monitoring the shopping cart; a software for identifying products in the shopping cart, which comprises an image recognition software adapted for discerning between different products in the shopping cart; an optional barcode scanner. Further, a respective method and computer program product is provided.

<CIT> relates to a commodity registration apparatus and a commodity registration method. In particular, it relates to a commodity registration apparatus that recognizes and registers a commodity on the basis of a commodity image captured by an image capturing section, and detects whether or not there is a barcode in the captured commodity image.

<CIT> relates to a storefront shopping technique for presenting information to a shopper aboutproximate for-sale items; specifically, both the shopper and the for-sale items can be associated with location beacons, and the presented information can be specific to the shopper.

<CIT> relates to a system whereby customers at a shopping center are first identified and then provided with a transport device for carrying any items that the customer picks up before reaching the corresponding cash register or dispenser/collection units.

An information processing device according to one embodiment can reduce unnecessary performance of generic object recognition on commodities having a bar code or the like.

According to one example, an information processing device on a shopping basket or a shopping cart includes a communication interface configured to communicate with an imaging device on the shopping basket or the shopping cart and a beacon at a product display area, and a processor configured to identify the commodity by either reading a commodity code in an image or object recognition on the image, and according to on a signal received from at least one beacon, switch between a first mode where the commodity is identified by reading the commodity code, and a second mode where the commodity can be identified by either reading the commodity code or object recognition.

A first embodiment is explained with reference to the drawings. <FIG> is a perspective view illustrating the exterior of an information processing device <NUM> in this embodiment. The information processing device <NUM> is attached to, together with a reading device <NUM>, a shopping cart <NUM> or a shopping basket used for storing commodities in a retail store. The information processing device <NUM> operates as a sales data processing device that registers the commodities to be purchased, that is, records information concerning the commodities, and a commodity information presenting device that presents information concerning substitutable commodities.

The information processing device <NUM> has an exterior and functions like a tablet computer or a tablet terminal and includes a main body <NUM>, a display <NUM>, and an input device <NUM>.

<FIG> is a diagram illustrating an example of a hardware configuration of the information processing device <NUM>. The information processing device <NUM> further includes a controller <NUM>, a storage device <NUM>, a short-range wireless communication interface <NUM>, and a communication interface <NUM>. The controller <NUM>, the storage device <NUM>, the short-range wireless communication interface <NUM>, the communication interface <NUM>, the display <NUM>, and the input device <NUM> are connected to one another via a system bus <NUM> such as a data bus or an address bus.

The display <NUM> is a tabular display device such as a liquid crystal display. The input device <NUM> is a touch panel superimposed on the surface of the display <NUM>. The main body <NUM> is a thin box-like housing having an opening on one surface. The main body <NUM> houses the hardware components explained above (e.g., the controller <NUM>, the storage device <NUM>, the short-range wireless communication interface <NUM>, the communication interface <NUM>, the display <NUM>, and the input device <NUM>). The opening of the main body <NUM> exposes a region where the input device <NUM> receives an input by an operator.

The controller <NUM> is a computer component and includes a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The ROM is a non-volatile memory that stores various computer programs and data. The RAM is a volatile memory that temporarily stores various computer programs and various data. The CPU executes the computer programs stored in the ROM and the storage device <NUM> to achieve various operations of the information processing device <NUM>.

The storage device <NUM> is a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage device <NUM> stores various computer programs executable by the controller <NUM> (e.g., CPU), setting information, and the like.

The storage device <NUM> stores a PLU (Price look-up) file therein. The PLU file stores commodity codes, which are uniquely allocated to commodities, and commodity information, such as names, unit prices, and classifications, in association with each other. The PLU file also stores feature values, reference images, and the like of commodities, which are usedby the informationprocessing device <NUM> in specifying the commodities through object recognition. The feature values are data such as colors of the commodities, unevenness of the surfaces of the commodities, and the like, read from the reference images of the commodities.

The communication interface <NUM> is a communication interface for communicating with a server apparatus and other information processing terminals, both of which are not illustrated in <FIG>, through wireless LAN communication, mobile communication, or the like.

In this embodiment, as shown in <FIG>, a beacon terminal <NUM> is installed on a product display area <NUM> such as display case, bin, shelf, or the like where commodities without code symbol <NUM>, such as vegetables and fruits, are displayed. The beacon terminal <NUM> may be installed on or near any object used for displaying a commodity in a store. The commodity without code symbol <NUM> is a commodity on which a code symbol such as a barcode or a two-dimensional code is not attached. The information processing device <NUM> includes a functional section (explained below) that performs predetermined processing upon reception of a radio wave emitted by the beacon terminal <NUM>.

<FIG> is a diagram illustrating an example of a hardware configuration of the beacon terminal <NUM>. The beacon terminal <NUM> includes a controller <NUM>, a storage device <NUM>, and a short-range wireless communication interface <NUM>. These hardware components are connected to one another via a system bus <NUM> such as a data bus or an address bus.

The controller <NUM> includes computer components such as a CPU, a ROM, and a RAM. The ROM is a nonvolatile memory that stores various computer programs and data. The RAM is a volatile memory that temporarily stores various computer programs and rewrites various data. The CPU executes the computer programs stored in the ROM and the storage device <NUM> to achieve various operations of the beacon terminal <NUM>.

The storage device <NUM> is a storage device such as a flash memory and stores beacon IDs, distribution information, and the like.

The short-range wireless communication interface <NUM> is a communication interface for short-range wireless communication such as BLE (Bluetooth Low Energy) or a Bluetooth-type protocol. The short-range wireless communication interface <NUM> communicates with the information processing device <NUM> and the like present within a communication zone.

The controller <NUM> of the beacon terminal <NUM> distributes the beacon IDs and the distribution information stored in the storage device <NUM> from the short-range wireless communication interface <NUM> by broadcasting the beacon IDs and the distribution information to each of information processing devices <NUM> present within the communication zone.

In the information processing device <NUM>, the short-range wireless communication interface <NUM> performs communication with the beacon terminal <NUM>. The short-range wireless communication interface <NUM> is a communication interface for performing short-range wireless communication such as BLE. The short-range wireless communication interface <NUM> communicates with the beacon terminal <NUM> and the reading device <NUM>.

The reading device <NUM> includes an imaging device such as a CCD (Charge Coupled Device) and reads information concerning a commodity from an image captured by the imaging device. More in detail, the reading device <NUM> outputs, to the information processing device <NUM>, code information obtained by reading or decoding a code symbol such as a barcode or a two-dimensional code attached on the surface of the commodity, and an image obtained by imaging the exterior of the commodity.

<FIG> is a diagram illustrating functional sections achieved by the controller <NUM> of the information processing device <NUM>. The controller <NUM> includes a short-range-communication control section <NUM>, a communication control section <NUM>, a read-information acquiring section <NUM>, a commodity-information acquiring section <NUM>, and a reading control section <NUM>. The controller <NUM> executes computer programs stored in the ROM and the storage device <NUM> to achieve the functional sections explained above.

The short-range-communication control section <NUM> controls the short-range wireless communication interface <NUM> to perform short-range wireless communication with the beacon terminal <NUM>. For example, the short-range-communication control section <NUM> receives distribution information, such as beacon IDs, distributed from the beacon terminal <NUM> by broadcast. Additionally, the short-range-communication control section <NUM> controls the short-range wireless communication interface <NUM> to establish one-to-one communication with the reading device <NUM>.

The communication control section <NUM> controls the communication interface <NUM> to perform communication with the server apparatus and the like. The read-information acquiring section <NUM> acquires the code information and the captured image input from the reading device <NUM>.

The commodity-information acquiring section <NUM> is one example of a first commodity-information acquiring section and is one example of a second commodity-information acquiring section. The commodity-information acquiring section <NUM> extracts, from the PLU file, information concerning a commodity matching the code information and the captured image acquired by the read-information acquiring section <NUM>.

More specifically, if the read-information acquiring section <NUM> acquires the code information, the commodity-information acquiring section <NUM> searches the PLU file for a commodity code matching the code information and acquires commodity information associated with the commodity code.

If the read-information acquiring section <NUM> acquires the captured image, the commodity-information acquiring section <NUM> performs object recognition based on the captured image to obtain a feature value, and acquires, from the PLU file, information concerning a commodity matching the feature value.

Concerning the object recognition for recognizing an object included in an image, a recognition technique is explained in, for example, a literature described below.

The commodity-information acquiring section <NUM> switches an operation state of the commodity-information acquiring section <NUM> between a first operation state and a second operation state under the control of the reading control section <NUM> explained below. The commodity-information acquiring section <NUM> in the first operation state performs only commodity recognition based on code information and does not perform commodity recognition based on a captured image. The commodity-information acquiring section <NUM> in the second operation state performs the commodity recognition based on the captured image in addition to the commodity recognition based on the code information.

Upon receipt of a beacon ID from the beacon terminal <NUM>, the reading control section <NUM> instructs the commodity-information acquiring section <NUM> to switch the operation state to the second operation state.

If a predetermined operation input is performed via the input device <NUM>, the reading control section <NUM> instructs the commodity-information acquiring section <NUM> to switch the operation state to the second operation state. That is, for example, if a customer using the shopping cart <NUM> desires to perform commodity recognition irrespective of presence or absence of reception of a beacon ID, the information processing device <NUM> allows the customer to do so by switching the operation state to the second operation state with a predetermined operation.

Upon startup, the reading control section <NUM> instructs the commodity-information acquiring section <NUM> to operate in the first operation state. Thereafter, basically, the commodity-information acquiring section <NUM> maintains the first operation state and switches the operation state if an instruction for switching the operation state is given from the reading control section <NUM>.

In such a configuration, the information processing device <NUM> stores, as information concerning a commodity to be purchased, in a predetermined region set in the storage device <NUM> or the like, commodity information of the commodity matching the code information or the captured image sent from the reading device <NUM>.

A customer, who uses the shopping cart <NUM> to which the information processing device <NUM> and the reading device <NUM> are attached, causes the reading device <NUM> to read a commodity and thereafter puts the commodity in the shopping cart <NUM>. The shopping cart <NUM> may include a scale that measures a total weight of commodities carried by the shopping cart <NUM>. The information processing device <NUM> may receive an output of the scale and confirm whether the total weight increases by weight corresponding to the commodity read by the reading device <NUM>.

The operation of the information processing device <NUM> is explained with reference to <FIG> is a flowchart illustrating an example of processing executed in the information processing device <NUM>.

If the controller <NUM> of the information processing device <NUM> functions as the short-range-communication control section <NUM> and receives a beacon ID of the beacon terminal <NUM> via the short-range wireless communication interface <NUM> (Yes in S1), the controller <NUM> functions as the reading control section <NUM> and turns on object recognition (aspect S2).

In this embodiment, the turning on the object recognition means instructing the commodity-information acquiring section <NUM> to switch the operation state to the second operation state. If the current operation state is the first operation state, the commodity-information acquiring section <NUM> receiving the instruction switches the operation state to the second operation state. If the current operation state is the second operation state, the commodity-information acquiring section <NUM> maintains the current operation state as it is.

In aspect S1, if the controller <NUM>, functioning as the short-range-communication control section <NUM>, does not receive the beacon ID of the beacon terminal <NUM> via the short-range wireless communication interface <NUM> (No in S1), the controller <NUM> functions as the reading control section <NUM> and turns off the object recognition (aspect S3).

In an example, turning off the object recognition means instructing the commodity-information acquiring section <NUM> to switch the operation state to the first operation state. If the current operation state is the second operation state, the commodity-information acquiring section <NUM> receiving the instruction switches the operation state to the first operation state. If the current operation state is the first operation state, the commodity-information acquiring section <NUM> maintains the current operation state as it is.

In this embodiment, if the shopping cart <NUM> approaches the product display area <NUM> and the information processing device <NUM> (attached to the shopping cart <NUM>) receives the beacon ID, the information processing device <NUM> switches the object recognition to ON in preparation for reading of the commodities without code symbol <NUM> such as vegetables and fruits. If the shopping cart <NUM> moves away from the product display area <NUM> and the information processing device <NUM> attached to the shopping cart <NUM> does not receive the beacon ID any more, the information processing device <NUM> switches the object recognition to OFF and performs commodity recognition based on a code symbol.

According to this embodiment explained above, the object recognition can be performed only near the product display area <NUM> in which the commodities without code symbol <NUM> are displayed. Consequently, for example, it is possible to abstain from the object recognition in a place where it is not assumed to perform reading of the commodities without code symbol <NUM>.

Even when in the first operation state, in which the object recognition is not performed, the state can be manually switched to the second operation state by the customer's operation. Therefore, for example, a commodity without code symbol <NUM> brought from a product display area <NUM> by a companion of the customer controlling the shopping cart <NUM> at a position away from the particular product display area <NUM> from which the companion acquired the commodity without the code symbol <NUM>, can be recognized by object recognition by manually switching the state to the second operation state as needed.

In this embodiment, the reading device <NUM> and the information processing device <NUM> communicate with each other by the wireless communication. However, in one embodiment, the reading device <NUM> and the information processing device <NUM> are not limited to this. For example, the reading device <NUM> and the information processing device <NUM> may communicate with each other by wired communication, the reading device <NUM> may be directly coupled to the information processing device <NUM>, or the reading device <NUM> may be incorporated in the information processing device <NUM>.

In this embodiment, the reading control section <NUM> instructs the commodity-information acquiring section <NUM> to switch the first operation state and the second operation state. However, in one embodiment, the reading control section <NUM> and the commodity-information acquiring section <NUM> are not limited to this.

For example, the commodity-information acquiring section <NUM> may be always in the second operation state in this embodiment. The reading control section <NUM> may instruct the read-information acquiring section <NUM> to switch execution and non-execution of acquisition of a captured image.

For example, the reading control section <NUM> may instruct the reading device <NUM> to switch the operation state. In this case, the commodity-information acquiring section <NUM> is always in the second operation state. The reading device <NUM> switches the operation state between the first operation state and the second operation state under the control of the reading control section <NUM>. If detecting a code symbol from a captured image, the reading device <NUM> in the first operation state reads code information from the code symbol and outputs the code information to the information processing device <NUM>. In addition, the reading device <NUM> in the second operation state outputs the captured image to the information processing device <NUM> if a code symbol cannot be detected from the captured image.

A second embodiment is explained below. In this explanation, the same components as the components in the first embodiment are denoted by the same reference numerals and signs. Detailed explanation of the components having the same functions is omitted. <FIG> is diagram illustrating disposition of beacon terminals in a store in this embodiment. In this embodiment, a plurality of beacon terminals <NUM> are generally uniformly disposed in the store. In the store, a plurality of access points covering wireless communication in the store are installed. The access points relay communication signals between the information processing device <NUM> and a store server.

The information processing device <NUM> in this embodiment stores map information in the storage device <NUM>. The map information stores information indicating beacon IDs and positions of the beacon terminals <NUM> in the store, information indicating positions of display cases, bins, s or shelves for commodities, and information for determining whether the commodities displayed in the shelves are commodities with code symbol in association with one another.

<FIG> is a diagram illustrating a positional relation between the information processing device <NUM> and the beacon terminals <NUM>. If the short-range wireless communication interface <NUM> is receiving signals of three or more beacons, the reading control section <NUM> in this embodiment calculates, based on the map information, the current position of the information processing device <NUM> from the positions of the beacon terminals <NUM> having the strongest radio wave intensity to the third strongest radio wave intensity of the signals. If the distance between the calculated position and the shelf <NUM> of the commodity without code symbol <NUM> is smaller than a predetermined threshold, the reading control section <NUM> sets the operation state of the information processing device <NUM> to the second operation state.

<FIG> is a flowchart illustrating a flow of the processing explained above executed by the information processing device <NUM>. The reading control section <NUM> calculates relative distances to the beacon terminals <NUM> having the strongest radio wave intensity to the third strongest radio wave intensity of the received signals (Step S11). Any known method can be used for calculating a relative distance between two positions based on radio wave intensity.

Subsequently, the reading control section <NUM> refers to the map information and calculates a position of the information processing device <NUM> from the relative distances to the beacon terminals <NUM> (Step S12). More specifically, for example, the reading control section <NUM> sets, as the current position of the information processing device <NUM>, an intersection of three circles drawn with the beacon terminals <NUM> as centers and with the relative distances calculated in Step S11 as radiuses.

Subsequently, the reading control section <NUM> determines whether the calculated position of the information processing device <NUM> is close to the commodity without code symbol <NUM> (Step S13). In Step S13, the reading control section <NUM> refers to the map information and determines whether the distance between the calculated position of the information processing device <NUM> and the shelf <NUM> of the commodity without code symbol <NUM> is smaller than a predetermined threshold. If the distance is smaller than the predetermined threshold, the reading control section <NUM> determines that the calculated position of the information processing device <NUM> is close to the commodity without code symbol <NUM>.

If determining that the calculated position of the information processing device <NUM> is close to the commodity without code symbol <NUM> (Yes in Step S13), the reading control section <NUM> turns on the object recognition (Step S14). If determining that the calculated position of the information processing device <NUM> is not close to the commodity without code symbol <NUM> (No in Step S13), the reading control section <NUM> turns off the object recognition (Step S15).

As explained above, according to this embodiment, even in the store where the plurality of beacon terminals <NUM> are disposed, the information processing device <NUM> performs object recognition near commodities without code symbol <NUM> while suppressing unnecessary object recognition.

The computer programs executed in the information processing device <NUM> in the embodiments explained above are stored in the ROM or the like in advance.

The computer programs executed in the information processing device <NUM> in the embodiments explained above may be provided by a computer-readable recording medium, such as a CD-ROM, a flexible disk (FD), a CD-R, and a DVD (Digital Versatile Disk) or provided as installable or executable files.

Further, the computer programs executed in the information processing device <NUM> in the embodiments explained above may be stored and provided by a computer connected to a network such as the Internet and downloaded through the network. The computer programs executed in the information processing device <NUM> in the embodiments explained above may be provided or distributed through a network such as the Internet.

Claim 1:
An information processing device (<NUM>) on a shopping basket or a shopping cart (<NUM>), comprising:
a communication interface (<NUM>) configured to communicate with an imaging device on the shopping basket or cart (<NUM>) and beacons (<NUM>) at product display areas (<NUM>); and
a processor configured to:
identify a commodity by either reading a commodity code (<NUM>) included in an image of the commodity from the imaging device or performing object recognition on the commodity in the image,
according to a signal received from at least one beacon (<NUM>), switch between a first mode in which the commodity is identified by reading the commodity code (<NUM>) and a second mode in which the commodity can be identified by either reading the commodity code (<NUM>) or object recognition,
calculate upon receipt of signals from at least three beacons (<NUM>) a current position of the shopping basket or cart (<NUM>) based on a strength of the signals,
calculate a distance from the shopping cart (<NUM>) or basket to a product display area (<NUM>) for a commodity without a commodity code (<NUM>) attached, and
switch to the second mode when the distance is less than a threshold value.