RECOGNITION SYSTEM, STORAGE SYSTEM INCLUDING RECOGNITION SYSTEM, SHOPPING ASSISTANCE SYSTEM INCLUDING STORAGE SYSTEM, STORAGE CONTROL METHOD, AND NON-TRANSITORY STORAGE MEDIUM

A recognition system includes a recognition processing unit and a determining processor. The recognition processing unit is configured to recognize a state in a container which allows one or more goods to be put in. The determining processor is configured to determine, based on a recognition result by the recognition processing unit in a transfer operation, whether or not the transfer operation is normal. The transfer operation is an operation of transferring each of the one or more goods from the container into a bag (different container) by a movement mechanism (a support, a guide section, and an elevator unit) configured to move each of the one or more goods.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is based upon and claims the benefit of priority of Japanese Patent Application No. 2017-228448, filed on Nov. 28, 2017, the entire contents of which are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure generally relates to recognition systems, storage systems including the recognition systems, shopping assistance systems including the storage systems, storage control methods, and non-transitory storage media. Specifically, the present disclosure relates to a recognition system for assisting customers in purchasing goods, a storage system including the recognition system, a shopping assistance system including the storage system, a storage control method, and a non-transitory storage medium.

BACKGROUND ART

Document 1 (JP H05-89364 A) discloses a system (POS system) attempting to realize retail stores without clerks. The system disclosed in Document 1 includes a basket and a transaction terminal apparatus. The basket allows goods to be put in. The basket includes a scanner configured to read goods information (goods data) and a transmission circuit configured to transmit the goods information to the outside. The transaction terminal apparatus includes a reception circuit configured to receive the goods information, a scale configured to measure a weight of the goods put in the basket, and a controller configured to compare weight data corresponding to the respective pieces of goods information with a measured value measured by the scale. When the measured value by the scale matches a total weight represented by the weight data, the transaction terminal apparatus performs a checkout process.

In the configuration described in Document 1, however, when goods are in the basket (container), an employee (clerk) of the store or a customer has to transfer the goods into another container such as a shopping bag after the checkout process. As a result, a time from the start of the checkout process until the customer receives the goods is increased, which may lead to an increased time for shopping.

To solve the above-described problems, a machine may be caused to perform a job of transferring the goods from the container into another container, but in this case, remaining of goods in the container or the like may cause a problem in the job.

SUMMARY

In view of the foregoing, it is an object of the present disclosure to provide a recognition system in which a problem is suppressed in a job of transferring goods from a container into a different container by a machine, a storage system including the recognition system, a shopping assistance system including the storage system, a storage control method, and a non-transitory storage medium.

A recognition system according to one aspect of the present disclosure includes a recognition processing unit and a determining processor. The recognition processing unit is configured to recognize a state in a container which allows one or more goods to be put in. The determining processor determines, based on a recognition result by the recognition processing unit in a transfer operation, whether or not the transfer operation is normal. The transfer operation is an operation of transferring each of the one or more goods from the container into a different container by a movement mechanism configured to move each of the one or more goods.

A storage system according to one aspect of the present disclosure includes: the recognition system; and a storage device. The storage device is configured to perform the transfer operation.

A shopping assistance system according to one aspect of the present disclosure includes: the above storage system; and a sales system configured to perform a sales process of the one or more goods put in the container.

A storage control method according to one aspect of the present disclosure includes recognizing a state in a container which allows one or more goods to be put in. The storage control method further includes determining, based on a recognition result in a transfer operation, whether or not the transfer operation is normal. The transfer operation is an operation of transferring each of the one or more goods from the container into a different container by a movement mechanism configured to move each of the one or more goods. The storage control method further includes controlling the movement mechanism based on a determination result.

A non-transitory storage medium according to one aspect of the present disclosure is a non-transitory storage medium recording a program for causing a computer system to execute a recognition process, a determining process, and a storage control process. The recognition process is a process of recognizing a state in a container which allows one or more goods to be put in. The determining process is a process of determining, based on a recognition result in a transfer operation, whether or not the transfer operation is normal. The transfer operation is an operation of transferring each of the one or more goods from the container into a different container by a movement mechanism configured to move each of the one or more goods. The storage control process is a process of controlling the movement mechanism based on a determination result by the determining process.

DETAILED DESCRIPTION

As illustrated inFIGS. 1 and 2, a recognition system1according to the present embodiment, together with a storage device31configured to perform a transfer operation, forms a storage system3configured to perform the transfer operation of goods8(or products). In the present disclosure, “transfer operation” refers to an operation of transferring each of the goods8from a container2into a different container10by a movement mechanism (a support311, a guide section312, and an elevator unit313which will be described later) configured to move each of the goods8. In the present embodiment, the container2is a shopping basket. Moreover, in the present embodiment, the different container10is made of polyethylene as what is called bag body, a so-called shopping bag (or a plastic shopping bag). The different container10is hereinafter also referred to as a “bag10”. It is only required that the bag has a dimension that allows the goods8to be put in, and the bag has a bag opening through which each of the goods8is put in or taken out.

As illustrated inFIG. 2, the storage system3, together with a sales system4, forms a shopping assistance system100for assisting customers in shopping. In other words, the shopping assistance system100includes the storage system3and the sales system4. The shopping assistance system100is introduced in retail stores such as convenience stores, supermarkets, department stores, drugstores, electronics retail stores, or hardware stores.

The sales system4is a system configured to directly or indirectly communicate with the container2and to perform a sales process (or a checkout process) of goods8put in the container2. As used herein, a “sales process” refers to various processes required for actions (sales and purchase) of transferring the ownership of goods from a seller (store) to a buyer (customer) and paying the seller the consideration (charge) for the goods by the buyer. The sales process includes, for example, an acquisition process, an order process, and the like. The acquisition process is a process of acquiring respective one or more goods information on one or more goods8picked up by a customer in a store. The order process is a process of receiving, from a customer, an order for one or more goods8which the customer wishes to buy.

Stores which the shopping assistance system100is introduced into enable customers to finish purchasing goods8by a series of actions of choosing the goods8in the stores, putting them in the container2, performing the checkout process thereof, and receiving the bag10in which the goods8have automatically been stored. Thus, with the shopping assistance system100, it is possible to reduce, for example, a time from the start of the checkout process to the customer getting the goods8while saving clerk's labor and customers' labor, thereby reducing customers' shopping time.

As illustrated inFIG. 1, the recognition system1includes a recognition processing unit11and a determining processor12. In the present embodiment, the recognition system1is applied to the container2placed at a predetermined location (in the present embodiment, on a bottom of a recess72in a counter desk7which will be described later). Note that the recognition system1is not applied to, for example, a container2stocked in a basket area near an entrance of the store. That is, the predetermined location does not include the basket area.

The recognition processing unit11is configured to recognize a state in the container2which allows one or more goods8to be put in. In the present disclosure, “state in the container” includes a state where the one or more goods8are put in the container2placed at the predetermined location as well as a state where no goods8are put in the container2placed at the predetermined location.

The determining processor12determines, based on a recognition result by the recognition processing unit11in the transfer operation, whether or not the transfer operation is normal. That is, in the transfer operation, the one or more goods8are transferred from the container2to the different container10, and thus, the state in the container2changes along with the transfer operation. The state in the container2is different between a case where the one or more goods8are successfully transferred from the container2into the different container10smoothly and a case where, for example, one or more goods8remain in the container2and an abnormality occurs in the process of transferring each of the one or more goods8from the container2into the different container10. This enables the determining processor12to determine whether or not the transfer operation is normal based on the recognition result by the recognition processing unit11in the transfer operation, that is, the state in the container2in the transfer operation.

As described above, in the present embodiment, it is possible to determine whether or not each of the one or more goods8is normally transferred from the container2into the different container10smoothly and whether or not an abnormality occurs in the process of transferring each of the one or more goods8from the container2into the different container10. Thus, in the present embodiment, based on the determination result, if the abnormality is included in, for example, the transfer operation, a customer, a clerk, or the like is allowed to perform correction so that the transfer operation is normally performed. Thus, the present embodiment provides the advantage that a problem is suppressed in a job of transferring each of the one or more goods8from the container2into the different container10by a machine (here, the movement mechanism) than in a case where normality/abnormality of the transfer operation is not determined.

The recognition system1according to the present embodiment, the storage system3including the recognition system1, and the shopping assistance system100including the recognition system1will be described in detail below. In the present embodiment, a convenience store is exemplified as a store which the shopping assistance system100is introduced into. That is, a “user” in the present disclosure is a customer in the convenience store and/or an employee (clerk) in the convenience store.

(2.1) Shopping Assistance System

Herein, the overall structure of the shopping assistance system100according to the present embodiment will first be explained. As shown inFIG. 2, the shopping assistance system100includes: the container2; the storage system3including the recognition system1; the sales system4; and a checkout system5. Note that the container2and the checkout system5do not have to be components included in the shopping assistance system100. The container2will be described in detail in “(2.2) Container” which will be described later. The recognition system1and the storage system3will be described in detail in “(2.3) Storage System” which will be described later.

The sales system4includes a management device41, a supply unit42, display device43, an input device44, a voice output device45, and an image-capturing device46. Each of the supply unit42, the display device43, the input device44, the voice output device45, and the image-capturing device46is a peripheral device of the management device41and is connected to the management device41.

The management device41is configured to receive goods information (which will be described later) transmitted from the container2. The management device41is connected to the checkout system5configured to perform a checkout process. The management device41is configured to communicate with a store terminal composed of, for example, a point of sale (POS) terminal. The management device41is configured to perform a predetermined process such as a transmission process of transmitting information required for the checkout process performed by the checkout system5to the checkout system5based on the goods information received from the container2. The management device41is configured to communicate also with the storage system3and a portable terminal6to control the storage system3and the portable terminal6.

The checkout system5includes a computer system as a main component and peripheral devices such as a cash unit configured to accept and dispense cash. The checkout system5is configured to cause a processor of the computer system to execute a program stored in a memory of the computer system, thereby performing the checkout process such as calculation of a checkout amount based on the goods information received from the sales system4. The checkout system5includes a card reader and a printer configured to issue a receipt. The card reader is configured to read, for example, credit card information and electronic money card information for payment, mobile phone terminal information, member's card information, and loyalty card information. Information such as for example, detailed statement, checkout amount, purchase date and time is described in the receipt.

In the present embodiment, the checkout system5stores price data corresponding to each piece of goods information. In addition, the checkout system5, the store server, or the center server may store: an information set such as advertisement information, discount information, producer information and recipe information, corresponding to each piece of goods information; and information such as gender, age, favorite and past purchase history, corresponding to each customer information. The information set, which is not stored in the sales system4, is available to the sales system4as a result of the sales system4communicating with the checkout system5, the store server, or the center server.

The supply unit42is configured to supply customers with one or more specific goods (e.g., cigarettes). The display device43is a device configured to display various kinds of information to customers. The input device44is a device configured to receive an operation performed by a customer through, for example, gesture detection. Moreover, the input device44includes a microphone and has a function of applying speech recognition and semantic analysis to a voice signal output from the microphone. Thus, a voice operation (voice input) by a customer is also possible. The voice output device45includes a loudspeaker and is configured to provide customers with various kinds of information by the voice.

That is, the sales system4uses the display device43, the input device44, and the voice output device45as user interfaces to provide customers with various kinds of information by the display or the voice and to receive operations (including voice input) by the customers. Note that provision of the information by the sales system4may be realized by at least one of the display and the voice. That is, the provision of the information may be realized by any one of the display and the voice, or realized by a combination of the display and the voice.

As an example, the sales system4may provide customers with messages such as “Welcome” and “Thank you very much” when the arrival of the customers is sensed and when the checkout process is completed. It is accordingly possible to give customers a sense of affinity like clerk service. For example, customers are to operate the input device44to select one of payment methods to pay the checkout amount displayed on the display device43. The sales system4may also provide customers with a utilization procedure for the shopping assistance system100by the display device43and the voice output device45. In this case, for example, when the container2is placed at a predetermined location (on the bottom of the recess72in the counter desk7), the display device43and the voice output device45then explain each step of the utilization procedure sequentially, thereby confirming whether or not the fast food or the cigarettes are purchased, etc.

The image-capturing device46is a camera configured to photograph customers. The image-capturing device46, together with the management device41and the portable terminal6, forms an age verification system9. In other words, the age verification system9includes the management device41, the image-capturing device46, and the portable terminal6. The age verification system9is a system configured to assist a clerk (checker) in checking the age of a customer (subject).

The portable terminal6is a terminal carried by a clerk. The portable terminal6is configured to wirelessly perform two-way communication with the management device41by a radio wave as a medium. The management device41transmits a request signal to the portable terminal6. The request signal includes image data of an image obtained by capturing an image of a subject (customer) as a target of the age verification. The portable terminal6receives the request signal, and based on the image data included in the request signal, the portable terminal6displays on-screen age verification including the image obtained by capturing an image of the subject. In this state, a checker (clerk) watches the on-screen age verification displayed on the portable terminal6to perform age verification of the subject. When receiving a result of the age verification performed by the checker as an operation by the checker, the portable terminal6transmits a response signal including the result of the age verification to the management device41.

Next, the container2will be described in detail. The container2is a shopping basket, includes a basket body201and a bottom panel202, and allows one or more goods8to be put in as illustrated inFIGS. 1 and 3. The basket body201is provided with a pair of grips204which are to be gripped by a customer. The basket body201has an opening at its bottom. The bottom panel202is slidable between a closed position closing the bottom (lower opening) of the basket body201and an open position. That is, the bottom panel202is attached to the basket body201so as to allow the bottom of the basket body201to be closed and opened. This enables the one or more goods8put in the container2to be discharged through the bottom of the basket body201.

That is, in the present embodiment, the bottom of the basket body201is an opening (outlet)203through which the one or more goods8are taken out of the container2in the transfer operation. The container2is configured to open and close the outlet203when the bottom panel202is opened and closed.

As illustrated inFIG. 2, the container2includes a receiver21, a transmitter22, a memory23, and a basket controller24.

The receiver21is configured to read respective goods information on one or more goods8put in the basket body201to acquire the respective goods information on the one or more goods8. In the present disclosure “goods information” is information for identifying each of goods8(information identification code), and examples thereof in Japan include Japanese Article Number (JAN) code and the like. Examples of the goods identification code include European Article Number (EAN) code in Europe, Universal Product Code (UPC) in USA and the like in addition to the JAN code. The receiver21is exposed from the outer side surface of, for example, the basket body201. The receiver21is a barcode reader and optically reads a barcode that each of the goods8carries to acquire goods information thereon. Therefore, a customer picks up each of the goods8to show it to the receiver21and then puts it in the basket body201. In this case, it can be said that the goods information on each of the goods8put in the basket body201has been acquired by the receiver21. The receiver21is not limited to the barcode reader, but examples thereof may further include a reader configured to read a two-dimensional code and a tag reader configured to read goods information from a radio-frequency identification (RFID) tag in a contactless manner.

The receiver21may also be configured to read goods information from, for example, each image of the goods8. In this example, the receiver21includes an image sensor configured to photograph the goods8inside or outside the basket body201. The receiver21may specify respective goods information of the goods8by performing image recognition processing with respect to each of the goods8, extracting feature such as outline, color and texture information therefrom, and verifying by comparing the feature with a template stored for each of goods8in advance. In this example, the receiver21may utilize information read with the barcode reader as auxiliary information for reading goods information from each image of the goods8. Alternatively, the barcode reader may be omitted when it is possible to sufficiently read goods information from only each image of goods8.

The transmitter22is configured to transmit goods information acquired through the receiver21to the sales system4. In the present embodiment, the transmitter22is composed of a communication module configured to perform two-way communication with the sales system4. The transmitter22communicates with the sales system4by optical wireless communication whose medium is radiation such as infrared radiation or visible radiation. Among different types of wireless communication, especially, near field type communication (near field wireless communication) having a narrow communication area is applicable because the transmitter22transmits goods information with the basket body201placed on the counter desk7.

The memory23is configured to store one or more pieces of goods information. For example, whenever the receiver21reads goods information, the goods information is stored in the memory23. The transmitter22may transmit the one or more pieces of goods information stored in the memory23all together to the sales system4or divide the one or more pieces of goods information into packets to transmit the packets to the sales system4. In the present embodiment, the memory23stores goods information acquired through the receiver21and the number thereof (i.e., acquisition count of goods information by the receiver21). For example, when there is one or more pieces of goods information as respective goods information on three goods8that include two goods8of “A” and one item of goods8(or one product) of “B”, the memory23stores goods information on “A”, “2” as count thereof, goods information “B”, and “1” as count thereof. For example, when the transmitter22succeeds in transmitting the one or more pieces of goods information to the sales system4, the one or more pieces of goods information stored in the memory23is deleted.

The basket controller24includes a microcontroller as a main component and is configured to cause a processor of the microcontroller to execute a program stored in a memory of the microcontroller, thereby controlling each part such as the receiver21. The program may be stored in the memory in advance, provided via a telecommunications network such as the Internet, or provided by a storage medium such as a memory card storing the program.

(2.3) Storage System

Next, the storage system3will be described in detail. As illustrated inFIG. 2, the storage system3includes a storage device31, a bag supply device32, and the recognition system1.

The storage device31performs, on an item of goods8in the container2, the transfer operation of transferring the item of goods8from the container2to the bag10, which enables the item of goods8to be stored in the bag (different container)10. That is, a customer picks up an item of goods8in a store, puts it in the container2, and places the container2, in which the item of goods8having been put, on the predetermined location (on the bottom of the recess72in the counter desk7), which causes the storage device31to perform transferring of the item of goods8. Thus, a customer does not have to take the item of goods8out of the container2for transferring them, and thus, the customer can easily transfer the item of goods8by the storage device31. When goods8are put in the container2, the storage device31is capable of collectively transferring the goods8.

In the present embodiment, the management device41is configured to control the storage system3in an interlocked manner with the sales system4such that each of one or more goods8are transferred while the customer performs a checkout process or the like by the checkout system5. For example, the storage system3starts transferring one or more goods8when the checkout process is started in the checkout system5.

The bag supply device32is a device configured to supply bags10to the storage device31. The bag supply device32takes a bag10one by one from a bag stocker in which a plurality of bags10are stocked, and the bag supply device32supplies the bag10which is taken out to the storage device31. In the present embodiment, the management device41controls the bag supply device32in an interlocked manner with the storage device31such that always one bag10is set (disposed) to the storage device31. That is, the storage device31is configured to use one bag10supplied from the bag supply device32to perform transferring. Each time the storage device31performs transferring for one time, a new bag10provided for the next transferring is supplied from the bag supply device32to the storage device31. In other words, the bag supply device32starts supplying a bag10to the storage device31when the storage device31completes transferring.

Next, the storage device31will be described in further detail. As illustrated inFIG. 1, the storage device31includes the support311, the guide section312, and the elevator unit313(driver). The support311, the guide section312, and the elevator unit313constitute a movement mechanism configured to move goods8.

The support311has a rectangular plate shape. An upper surface of the support311is a mount surface314on which goods8are to be mounted. In the present embodiment, the mount surface314has a rectangular shape with rounded corners.

The guide section312is in a shape of a cylinder. Of an upper surface and a lower surface of the cylinder, at least the upper surface is an opening. In the present embodiment, the guide section312is in a shape of a rectangular cylinder. Both an upper surface and a lower surface of the rectangular cylinder are openings. The openings of the guide section312are slightly larger than the mount surface314. The guide section312is disposed around the support311along the outer periphery of the mount surface314. Here, the shape of an inner peripheral edge of an upper end surface of the guide section312and the shape of an outer periphery of the mount surface314are in a similarity relationship.

The elevator unit313is configured to change the relative positional relationship between the support311and the guide section312from a first state to a second state and then to a third state. In the present embodiment, a state where the support311is located at an upper end of the guide section312is defined as the first state (seeFIGS. 5A and 5B). When the relative positional relationship between the support311and the guide section312corresponds to the first state, the upper surface (mount surface314) of the support311is located at a slightly higher level than the upper end surface of the guide section312. Thus, in the first state, the support311is disposed to close the opening at the upper surface of the guide section312. The second state is defined as a state where the location of the support311relative to the guide section312is at a lower level than in the first state (seeFIG. 5C). The third state is a state where the location of the support311relative to the guide section312is at a further lower level than in the second state (seeFIG. 6A).

In the present embodiment, the travel distance of the support311from the first state to the second state is preferably several percent to several ten percent (e.g., 10% to 20%) of the travel distance of the support311from the first state to the third state. For example, when the travel distance of the support311from the first state to the third state is 30 cm, the travel distance of the support311from the first state to the second state is preferably 3 cm to 6 cm. This is because, although one or more goods8may stick at an outlet203of the container2in the first state as described later, and the one or more goods8may be released from the outlet203in the second state and collide with the support311, the travel distance defined as described above leads to a relatively small influence of the collision on the one of more goods8.

The elevator unit313includes an electric motor (motor). The elevator unit313is realized by an appropriate mechanism which enables direct advance of the support311in the vertical direction by driving force generated by the electric motor of, for example, a pantograph type or a rack and pinion type. The elevator unit313is configured to move only the support311of the support311and the guide section312downward, thereby changing the relative positional relationship between the support311and the guide section312from the first state to the second state and then to the third state. That is, the support311is configured to move in the guide section312which is tubular and which is fixed at a fixed position in the vertical direction by the elevator unit313.

In the present embodiment, the storage device31includes a holding mechanism for holding a bottom part102of the bag10on the mount surface314, in addition to the support311, the guide section312, and the elevator unit313. The holding mechanism clamps the bottom part102of the bag10to hold the bottom part102. The storage device31keeps the bottom part102of the bag10held on the mount surface314by the holding mechanism at least while the relative positional relationship between the support311and the guide section312changes from the first state to the third state.

As illustrated inFIGS. 5A to 5CandFIG. 6A, when the relative positional relationship between the support311and the guide section312changes from the first state to the third state, an item of goods8discharged through the outlet203of the container2moves in the bag10. That is, in the present embodiment, the transfer operation of transferring the item of goods8from the container2into the bag (different container)10by the storage device31ends when the relative positional relationship between the support311and the guide section312has changed from the first state to the third state. In the present embodiment, the transfer operation is an operation of lowering the item of goods8with respect to the container2in a state where the outlet203of the container2is open, to change the positional relationship of the item of goods8relative to the container2to transfer the item of goods8from the container2through the outlet203into the bag (different container)10. Operation of the storage device31including the transfer operation will be described in detail in “(3.1) Operation of Storage Device” which will be described later.

Next, the recognition system1will be described. As illustrated inFIG. 1, the recognition system1includes the recognition processing unit11, the determining processor12, and a communication unit13. As described later, the recognition processing unit11(a ranging sensor111) is disposed outside the counter desk7, but the determining processor12and the communication unit13may be built in the counter desk7or may be disposed on an upper surface71of the counter desk7. Note that the broken line arrow shown inFIG. 1expresses an electromagnetic wave emitted from the ranging sensor111and traveling toward an object in the container2in a simulated manner.

The recognition processing unit11is configured to recognize the state in the container2. In the present embodiment, the recognition processing unit11includes the ranging sensor111. The ranging sensor111is, for example, a ranging image sensor adopting infrared radiation and is configured to measure the distance from the ranging sensor111to the object by a time-of-flight (TOF) method. In the present disclosure, “object” refers to an object in the container2and includes the support311of the storage device31in addition to the item of goods8put in the container2. The support311faces the interior of the container2through the outlet203in a state where the bottom panel202is open.

In the present embodiment, the ranging sensor111is, as illustrated inFIGS. 3 and 4, attached to an arch14installed on the upper surface71of the counter desk7. That is, the recognition processing unit11(ranging sensor111) is installed outside the counter desk7(more specifically, outside the container2). The arch14includes a pair of supports141and a beam142. The pair of supports141is installed on the upper surface71of the counter desk7to straddle the recess72and rises upward from the upper surface71. The beam142connects respective upper ends of the pair of supports141. The ranging sensor111is attached to a center portion in the lengthwise direction of the beam142.

The arch14is disposed at a location on a rear side from the central portion of the counter desk7as viewed from the front side of the counter desk7. Thus, the ranging sensor111is disposed at a location which is deviated from the center of the container2in a horizontal surface viewed in the gravity direction (the vertical direction of the upper surface71of the counter desk7). Thus, in the present embodiment, the ranging sensor111measures the distance to an object in the container2placed at a predetermined location (the bottom of the recess72in the counter desk7) not from directly above the container2but from obliquely above the container2.

The ranging sensor111measures the distance to the object in the container2to acquire a distance image. Each of a plurality of pixels included in the distance image includes distance information. In the present disclosure, “distance information” refers to information denoting the distance from the ranging sensor111to the object. The distance information is, for example, represented by luminance (color), but in the present embodiment, the distance information is represented by shade instead of the luminance (color). A lighter color represents a shorter distance from the ranging sensor111to the object. A darker color represents a longer the distance from the ranging sensor111to the object.

An example of the distance image is shown inFIG. 8A. The distance image shown inFIG. 8Arepresents a case where the item of goods8is a bottle containing beverage (hereinafter simply referred to as a “bottle”), and this distance image represents a state where the bottle lays on the mount surface314. In this case, the body of the bottle is closer to the ranging sensor111than the neck of the bottle is. Thus, in the distance image, an area corresponding to the body of the bottle has a lighter color than an area corresponding to the neck of the bottle. Moreover, the mount surface314is located farther away from the ranging sensor111than the bottle is. Thus, in the distance image, an area B2corresponding to the mount surface314has a darker color than an area B1corresponding to the bottle.

Here, the ranging sensor111in the present embodiment is disposed obliquely above the container2as described above. Thus, the distance image is corrected by appropriate image processing such as trapezoid correction so as to be an image equivalent to a distance image obtained when the distance to the object in the container2is measured from directly above the container2placed at the predetermined location. In the following description, a distance image after the image processing is simply referred to as a “distance image”. The image processing may be performed by the recognition processing unit11or may be performed by the determining processor12. Note that the relative positional relationship between the ranging sensor111and the container2placed at the predetermined location is substantially fixed. Thus, when parameters used for the image processing are preset before the recognition system1is used, the parameters do not have to be set each time measuring by the ranging sensor111is performed.

As described above, the ranging sensor111is disposed outside the container2and is configured to measure the distance to the object present in the container2. The recognition processing unit11(ranging sensor111) recognizes the state in the container2based on a measurement result by the ranging sensor111, that is, based on the distance from the ranging sensor111to the object in the container2.

The determining processor12includes one or more processors and a memory and executes a program stored in the memory by the one or more processors, thereby executing a determining process. The determining process is a process of determining whether or not the transfer operation is normal based on the recognition result by the recognition processing unit11in the transfer operation (in this embodiment, the distance from the ranging sensor111to the object in the container2). That is, the determining process is executed when the storage device31transfers the goods8from the container2into the bag (different container)10. The program may be stored in the memory in advance, provided via a telecommunications network such as the Internet, or provided by a storage medium such as a memory card storing the program.

In the present embodiment, the determining processor12executes, as the determining process, a first determining process and a second determining process. The first determining process is a process of determining whether or not one or more goods8are stuck at the outlet203of the container2so as to determine whether or not the transfer operation is normal. The second determining process is a process of determining, when the relative positional relationship between the support311and the guide section312corresponds to the third state, whether or not one or more goods8remain in the container2so as to determine whether or not the transfer operation is normal. The determining process by the determining processor12will be described in detail in “(3.3) Operation of Recognition System” which will be described later.

The communication unit13outputs a notification signal when the determining processor12determines that the transfer operation is abnormal. In the present disclosure, “abnormality of transfer operation” mainly refers to a situation where one or more goods8are stuck at the outlet203of the container2or a situation where one or more goods8remain in the container2when the relative positional relationship between the support311and the guide section312corresponds to the third state. In the present embodiment, the communication unit13outputs the notification signal to the sales system4(in this embodiment, the management device41). In the present embodiment, the communication unit13includes a communication module configured to perform one-way communication with the sales system4. The communication unit13performs, for example, communication with the sales system4through wireless communication whose medium is a radio wave such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). The communication unit13transmits the notification signal to the sales system4through wireless communication.

As illustrated inFIG. 3, the shopping assistance system100is disposed, for example, in the periphery of a counter desk7installed at a checkout counter of a store. That is, in the periphery of the counter desk7, the shopping assistance system100is provided. A plurality of counter desks7may be provided to one store.

The counter desk7is formed with a recess72that opens in a substantially central portion of the upper surface71in a lengthwise direction (lateral direction). The recess72serves as a space that allows the basket body201of the container2to be put in. The recess72opens backward as viewed from the front side of the counter desk7. Thus, a customer slides the basket body201of the container2along the bottom of the recess72in the horizontal direction, which enables the basket body201to be placed in the recess72.

The storage device31and the bag supply device32are installed under the recess72. On the bottom of the recess72, the mount surface314of the support311of the storage device31is exposed in a space (that is, predetermined location) in which the basket body201of the container2is to be placed. Thus, when the storage device31opens the bottom panel202of the container2placed at the predetermined location, an item of goods8in the container2are discharged onto the mount surface314. Then, the storage device31transfers the item of goods8placed on the mount surface314.

The supply unit42is suspended from a ceiling to be provided above the counter desk7. The management device41is built in the counter desk7. The checkout system5is disposed on the right side of the storage device31as viewed from the front side of the counter desk7. The display device43is fixed to a lower surface of the supply unit42and is configured to project an image onto a screen431by, for example, a projection mapping technique. Herein, the display device43projects an image onto a right region with respect to the storage device31in the upper surface71of the counter desk7as viewed from the front side of the counter desk7. That is, the right region with respect to the storage device31in the upper surface71of the counter desk7serves as the screen431of the display device43. The input device44is on the upper surface71of the counter desk7and is disposed behind the screen431as viewed from the front side of the counter desk7. The input device44is provided integrally with the voice output device45.

The positional relationship, shapes, and the like of components of the shopping assistance system100described above are mere examples and may accordingly be modified. For example, the storage device31and the recess72may be disposed at one end (left end or right end) in the width direction (lateral direction) of the counter desk7. Moreover, the recess72formed in the counter desk7is not an essential configuration for the shopping assistance system100. For example, the upper surface71of the counter desk7may have a space (predetermined location) in which the basket body201of the container2is to be placed.

Operation of the shopping assistance system100according to the present embodiment will be described below.

(3.1) Basic Operation

A basic operation of the shopping assistance system100from a customer's visit to a store until completion of transferring of goods8will be explained.

A customer visiting a store takes the container2from a basket area. The customer carrying the container2subsequently moves in the store and puts goods8that the customer desires to purchase one by one in the basket body201of the container2. At this moment, the customer shows the goods8picked up to the receiver21and then puts the goods8in the basket body201. Thus, the receiver21acquires respective goods information on the goods8.

The customer subsequently puts, at the predetermined location (on the bottom of the recess72in the counter desk7), the container2that all the goods8that the customer desires to purchase are put in. At this moment, a sensor installed, for example, in the bottom of the recess72in the counter desk7senses that the container2is placed on the predetermined location. The sales system4receives a sensed result by the sensor and then starts communication with the container2, thereby acquiring the goods information from the container2.

When acquiring the goods information, the sales system4transmits (transfers) the goods information to the checkout system5. The sales system4acquires price data transmitted as a reply (response) to the goods information from the checkout system5. Moreover, the sales system4creates a purchase list containing pieces of information on the goods8based on the price data thus acquired.

Moreover, at this time, the sales system4judges whether or not the purchase list includes specific goods which require age verification. As an example of the specific goods, the present embodiment includes alcohol (alcohol goods). Herein, the sales system4judges whether or not alcohol is included in the goods8put in the container2based on the purchase list. When alcohol is contained in the purchase list, the sales system4transmits a start signal to the age verification system9. The age verification system9receives the start signal and then starts a series of steps for the age verification.

When the age verification system9succeeds in the age verification, or when the purchase list includes no alcohol, the sales system4causes the display device43to display an on-screen confirmation for cigarettes. With the on-screen confirmation for cigarettes, it is confirmed whether or not cigarettes which are the specific goods are to be purchased. When “purchasing the cigarettes” is selected in the on-screen confirmation for cigarettes, the sales system4transmits the start signal to the age verification system9. When the age verification system9succeeds in the age verification, the sales system4causes the display device43to display an on-screen cigarette menu that allows customers to select one of cigarette brands. Note that when the purchase list includes alcohol and the age verification has already been successfully performed, the sales system4may cause the display device43to display the on-screen cigarette menu directly after “purchase cigarettes” is selected on the on-screen confirmation for cigarettes.

In the on-screen cigarette menu, when one or more specific cigarette brands are selected, the sales system4transmits the respective one or more pieces of goods information of the one or more specific cigarette brands to the supply unit42. The supply unit42selects corresponding one or more cigarette packs from cigarette packs stocked in the supply unit42and dispenses the one or more cigarette packs (i.e., cigarettes) thus selected to an outlet421(seeFIG. 3). This configuration enables customers to receive the cigarettes dispensed through the outlet421. Moreover, the sales system4acquires price data of the cigarettes from the checkout system5and adds the price data to the purchase list.

When supply of the cigarettes from the supply unit42is completed or “do not purchase cigarettes” is selected in the on-screen confirmation for cigarettes, the sales system4proceeds to the checkout process. At this time, the sales system4causes the display device43to display an on-screen checkout confirmation including the purchase list. When start checkout is selected in the on-screen checkout confirmation, the sales system4causes the checkout system5to start the checkout process.

Moreover, when the age verification system9fails the age verification, the sales system4stops the sales process of the specific goods (in this embodiment, alcohol and cigarettes). When the age verification in a sales process of cigarettes fails, the sales system4stops the sales process of the cigarettes and proceeds to the checkout process. When the age verification in a sales process of alcohol fails, the sales system4does not display the on-screen confirmation for cigarettes and stops the alcohol sales process.

After the checkout process is started, the sales system4causes the storage system3to perform transferring. That is, the sales system4causes the checkout system5to start the checkout process and then transmits, to the storage system3, a control signal for starting a storing process. Thus, the storage device31of the storage system3accordingly starts the transferring of the goods8from the container2to the bag10while the checkout process of the goods8is being performed. Note that preferably after the checkout process is completed, the customer is allowed to receive the bag10storing the goods8with the storing process by the storage device31being completed.

(3.2) Operation of Storage Device

Next, operation of the storage device31will be described with reference toFIGS. 5A to 5CandFIGS. 6A to 6C. InFIGS. 5A to 6C, the operation of the storage device31is schematically illustrated, and the bag supply device32and the like are accordingly omitted in the drawings. InFIGS. 5A to 5CandFIG. 6A, no goods8are stuck at the outlet203of the container2. When the relative positional relationship between the support311and the guide section312corresponds to the third state, no goods8remain in the container2. That is, in the examples shown inFIGS. 5A to 5CandFIG. 6A, the storage device31normally performs the transfer operation.

First, when the relative positional relationship between the support311and the guide section312corresponds to the first state as illustrated inFIG. 5A, the bag (different container)10is put on the support311and the guide section312with a bag opening101of the bag10facing downward. The bottom part102of the bag10is supported by the mount surface314. The storage device31shown inFIG. 5Ais in a stand-by state and remains in the stand-by state until the storage device31receives a control signal from the management device41.

When receiving the control signal from the management device41, the storage device31starts the storing process. When the storing process is started, the bottom panel202of the container2is opened as illustrated inFIG. 5B, so that an item of goods8in the container2is discharged through the outlet203onto the mount surface314of the storage device31. Thus, the item of goods8is placed on the bag10on the mount surface314. Here, the bottom panel202is held on the mount surface314by holding force of a holding mechanism in addition to the weight of the item of goods8. That is, the transfer operation includes an operation of opening and closing the bottom panel202to open and close the outlet203.

From this state (first state), the elevator unit313moves the support311downward relative to the guide section312, thereby changing the relative positional relationship between the support311and the guide section312to the second state. In the second state, the item of goods8is not stuck at the outlet203of the container2, and therefore, the item of goods8also moves downward as the support311moves. The item of goods8remains on the bag10on the mount surface314. The second state is maintained during an execution time period of a first determining process by the recognition system1which will be described later.

From the state (second state), the elevator unit313further moves the support311downward relative to the guide section312, so that the bag10is gradually drawn into an inner side of the guide section312. At this time, a side peripheral part103of the bag10which is continuous to the bottom part102of the bag10is drawn into the inner side of the guide section312while the side peripheral part103is squeezed by the upper end surface of the guide section312. Thus, the guide section312around the support311raises the side peripheral part103of the bag10from the outer periphery of the bottom part102.

When the relative positional relationship between the support311and the guide section312transitions to the third state as illustrated inFIG. 6A, the bag10is, in a space surrounded by the guide section312, turned inside out from the state shown inFIG. 5A, so that the item of goods8is stored in the bag10. At this time, the side peripheral part103is in a raised state along an inner peripheral surface of the guide section312. Here, the storage device31is configured such that in the second state, a level difference from the mount surface314to the upper end surface of the guide section312is larger than or equal to the total length of the bag10in the vertical direction. Herein “the total length of the bag10in the vertical direction” refers to a total length including a pair of handles104of the bag10, that is, a dimension from the bottom part102of the bag10to tips of the handles104. Thus, the bag10inclusive of the tips of the handles104is fit in a space surrounded by the guide section312. Thus, the bag10inclusive of not only the side peripheral part103but also the handles104is in the raised state. The state shown inFIG. 6A(that is, the third state, and the bottom panel202of the container2is open) is maintained during the execution time period in the second determining process which will be described later and which is performed by the recognition system1.

Next, the storage device31closes the bottom panel202of the container2as illustrated inFIG. 6B. Then, the container2is removed from the storage device31, and thereafter, the storage device31drives the support311by the elevator unit313to change the relative positional relationship between the support311and the guide section312from the third state to the first state. The bag10is thus released from the space surrounded by the guide section312.

That is, as the support311moves upward, the bag10on the mount surface314goes upward through the opening on a side facing the upper surface of the guide section312. Moreover, the support311moves upward, thereby completely releasing the bag10from the space surrounded by the guide section312as illustrated inFIG. 6C. At this time, the bag10inclusive of the handles104maintains an independent position. Moreover, the bag10is turned inside out from the state of the bag10supplied by the bag supply device32inFIG. 6Aso as to store the item of goods8. Therefore, when characters and the like are printed on the bag10, mirror printing is preferably adopted to the bag10. Thus, a customer can receive the item of goods8transferred in the bag10.

(3.3) Operation of Recognition System

Next, operation of the recognition system1will be described with reference toFIGS. 5A to 14. After the storing process by the storage device31is started, the recognition system1receives a state signal from the storage device31, for example, via the management device41so as to grasp the relative positional relationship between the support311and the guide section312and an open/close state of the bottom panel202of the container2.

First, the first determining process by the determining processor12will be described mainly with reference toFIG. 7. When the relative positional relationship between the support311and the guide section312corresponds to the first state, and the bottom panel202of the container2is open (seeFIG. 5B), the ranging sensor111of the recognition processing unit11acquires a distance image (51). The distance image at this time is hereinafter referred to as a “first image A1” (seeFIG. 8AandFIG. 8C). Next, when the relative positional relationship between the support311and the guide section312becomes the second state (seeFIG. 5C), the ranging sensor111performs acquisition again to acquire a distance image (S2). The distance image at this time is hereinafter referred to as a “second image A2” (seeFIGS. 8B and 8D).

The determining processor12determines, with reference to the first image A1and the second image A2, whether or not the item of goods8is stuck at the outlet203of the container2so as to determine whether or not the transfer operation is normal (S3). In other words, the determining processor12determines, based on a change of the state in the container2caused due to the transfer operation, whether or not the transfer operation is normal. Herein “the change of the state in the container” refers to a change of the state in the container2represented by the first image A1to the state in the container2represented by the second image A2. In particular, in the present embodiment, the determining processor12is configured to determine whether or not the transfer operation is normal based on a difference between a state in the container2prior to a start time point of an operation of the movement mechanism (seeFIG. 5B) and a state in the container2after the start time point (seeFIG. 5C). Moreover, in the present embodiment, the state in the container2after the start time point is a state in the container2prior to an end time point of the operation of the movement mechanism.

In the present embodiment, the determining processor12calculates the difference between the distance information of the first image A1and the distance information of the second image A2(in this embodiment, difference in shade), and if the difference exceeds the prescribed range, the determining processor12determines that the transfer operation is normal (S4: Yes). In contrast, if the difference is within the prescribed range, the determining processor12determines that the transfer operation is abnormal (S4: No). For example, the determining processor12calculates difference between distance information of each pixel of the first image A1and distance information of a corresponding pixel of the second image A2, and if the number of pixels with a difference in distance information more than the prescribed range is more than or equal to a certain number, the determining processor12determines that the transfer operation is normal. In contrast, if the number of pixels with a difference in distance information more than the prescribed range is less than the certain number, the determining processor12determines that the transfer operation is abnormal.

That is, when the item of goods8is not stuck at the outlet203, the item of goods8moves downward together with the support311along with a change of the relative positional relationship between the support311and the guide section312from the first state to the second state (seeFIG. 5C). Thus, in this case, as illustrated inFIGS. 8A and 8B, the area B1corresponding to the bottle (item of goods8) in the second image A2overall has a darker color than the area B1corresponding to the bottle in the first image A1. In this case, the number of pixels with a difference in distance information between the first and second images A1and A2more than the prescribed range is more than or equal to the certain number. Thus, the determining processor12determines that the transfer operation is normal.

In contrast, when the item of goods8is stuck at the outlet203as illustrated inFIG. 9, the item of goods8does not move together with the support311along with a change of the relative positional relationship between the support311and the guide section312from the first state to the second state. Thus, in this case, as illustrated inFIGS. 8C and 8D, the color in the area B1corresponding to the bottle (the item of goods8) in the second image A2does not substantially change from the color of the area B1corresponding to the bottle in the first image A1. In this case, the difference between the distance information of the first image A1and the distance information of the second image A2is within the prescribed range. Thus, the determining processor12determines that the transfer operation is abnormal.

When it is determined that the transfer operation is abnormal (S4: No), and a predetermined time (e.g., several seconds) has not elapsed since the start time point of the first determining process (S5: No), the determining processor12stands by for a constant time (e.g., several hundred milliseconds) (S6). When the constant time has elapsed, the ranging sensor111of the recognition processing unit11performs acquisition again to acquire a distance image (S7). At this time, the relative positional relationship between the support311and the guide section312remains in the second state. Thus, the distance image acquired by the ranging sensor111corresponds to the second image A2. That is, the ranging sensor111performs the acquisition again to acquire a distance image, thereby updating the second image A2. The determining processor12determines again whether or not the transfer operation is normal based on the first image A1and the second image A2which is updated (S3). The above-described process is repeated until a predetermined time has elapsed since the start time point of the first determining process or until the item of goods8is released from the outlet203before the predetermined time elapses.

When the item of goods8is not stuck at the outlet203, or when the item of goods8is released from the outlet203before a predetermined time has elapsed since the start time point of the first determining process, the determining processor12outputs a signal via the management device41to the storage device31. The signal denotes that the transfer operation is normal. Thus, the storage device31changes the relative positional relationship between the support311and the guide section312from the second state to the third state (seeFIG. 6A). That is, the storage device31is configured to perform the transfer operation based on the determination result by the determining processor12. The determining processor12then starts the second determining process.

On the other hand, when it is determined that the transfer operation is still abnormal (S4: No), and the predetermined time has elapsed since the start time point of the first determining process (S5: Yes), the determining processor12interrupts the checkout process by the checkout system5via the management device41(S8). The communication unit13then outputs a notification signal to the management device41(S9).

The management device41which receives the notification signal causes the display device43or the voice output device45to display or to reproduce, for example, a message such as “Goods are stuck at the outlet of the container. Please remove the goods from the outlet” for prompting correction. A customer who sees (or hears) this message may attempt to perform the correction by, for example, taking the item of goods8out of the outlet203and placing it on the mount surface314as shown inFIG. 10.

When the constant time has elapsed since the output of the notification signal, or when the input device44receives a manipulation representing that the correction is completed, the determining processor12causes the checkout system5to resume the checkout process via the management device41(S10). The determining processor12then outputs a signal denoting that the transfer operation is normal to the storage device31via the management device41, thereby starting the second determining process.

Next, the second determining process by the determining processor12will be described with reference to mainlyFIG. 11. When the relative positional relationship between the support311and the guide section312transitions to the third state (seeFIG. 6A), the ranging sensor111of the recognition processing unit11acquires a distance image (S11). A distance image at this time is hereinafter referred to as a “third image A3” (seeFIGS. 12A and 12B).

The determining processor12determines whether or not one or more goods8remain in the container2based on the third image A3, thereby determining whether or not the transfer operation is normal (S12). In other words, the determining processor12determines whether or not the transfer operation is normal based on a state in the container2at the end time point of the operation of the movement mechanism (seeFIG. 6A). In the present embodiment, if the third image A3does not include a pixel showing that corresponding distance information (shade) is shorter (lighter) than a prescribed distance (a prescribed shade) (the distance from the ranging sensor111to the outlet203), the determining processor12determines that the transfer operation is normal (S13: Yes). In contrast, if the third image A3includes a pixel showing that corresponding distance information is shorter than the prescribed distance, the determining processor12determines that the transfer operation is abnormal (S13: No).

That is, when no goods8remain in the container2, as illustrated inFIG. 12A, the area B1corresponding to bottles (the goods8) in the third image A3has a darker color than when an object is present in the above prescribed distance. In this case, the third image A3does not include a pixel showing that corresponding distance information is shorter than the prescribed distance. Thus, the determining processor12determines that the transfer operation is normal.

In contrast, when, for example, the goods8are stacked on each other, and some of the goods8remain in the container2as illustrated inFIG. 13, part of the area B1has a lighter color as illustrated inFIG. 12Bthan when an object is present in the prescribed distance. In this case, the third image A3includes a pixel showing that corresponding distance information is shorter than the prescribed distance. Thus, the determining processor12determines that the transfer operation is abnormal.

When it is determined that the transfer operation is abnormal (S13: No), the determining processor12interrupts the checkout process by the checkout system5via the management device41(S14). The communication unit13outputs the notification signal to the management device41(S15).

The management device41which receives the notification signal causes the display device43or the voice output device45to display or to reproduce, for example, a message such as “Goods remain in the container. Please move the goods so that the goods do not remain in the container” for prompting correction. A customer who sees (or hears) this message may attempt to perform the correction by, for example, moving the goods8so that no goods8remain in the container2as shown inFIG. 14.

When the constant time has elapsed since the output of the notification signal, or when the input device44receives a manipulation representing that the correction is completed, the determining processor12causes the checkout system5to resume the checkout process via the management device41(S16). The determining processor12then outputs a signal denoting that the transfer operation is normal to the storage device31via the management device41, thereby terminating the determining process. Then, the storage device31changes the relative positional relationship between the support311and the guide section312from the third state to the first state and completes the storing process (seeFIGS. 6A to 6C).

The above-described embodiment is a mere example of various embodiments of the present disclosure. Various modifications are possible depending on design and the like as long as the object of the present disclosure can be achieved. Moreover, functions similar to those of the storage system3may be realized by a storage control method, a computer program, a non-transitory storage medium storing a computer program, or the like.

A storage control method according to one aspect includes recognizing a state in a container2which allows one or more goods8to be put in. The storage control method further includes determining, based on a recognition result in a transfer operation, whether or not the transfer operation is normal. The transfer operation is an operation of transferring each of the one or more goods8from the container2into a bag (different container)10by a movement mechanism (a support311, a guide section312, and an elevator unit313) configured to move each of the one or more goods8. The storage control method further includes controlling the movement mechanism based on a determination result.

A program according to one aspect is a program that causes a computer system to execute a recognition process, a determining process, and a storage control process. The recognition process is a process of recognizing a state in a container2which allows one or more goods8to be put in. The determining process is a process of determining, based on a recognition result in a transfer operation, whether or not the transfer operation is normal. The transfer operation is an operation of transferring each of the one or more goods8from the container2into a bag (different container)10by a movement mechanism (a support311, a guide section312, and an elevator unit313) configured to move each of the one or more goods8. The storage control process is a process of controlling the movement mechanism based on a determination result in the determining process.

Variations of the above-described embodiment will be described below. The variations described below may be combined as appropriate.

In the shopping assistance system100in the present disclosure, the recognition system1, the storage system3, the sales system4, and the like include respective computer systems. Each computer system includes a processor as hardware and a memory as main components. The processor executes a program stored in the memory of each computer system, thereby realizing the function as the shopping assistance system100of the present disclosure. The program may be stored in the memory of each computer system in advance, provided via telecommunications network, or provided via a non-transitory computer-readable storage medium such as a memory card, an optical disk, a hard disk drive, or the like storing the program. The processor of each computer system includes one or more electronic circuits including a semiconductor integrated circuit (IC) or a large-scale integrated circuit (LSI). The integrated circuit such as IC or LSI mentioned herein may be referred to in another way, depending on the degree of the integration and includes integrated circuits called system LSI, very-large-scale integration (VLSI), or ultra-large-scale integration (VLSI). A field-programmable gate array (FPGA), which is programmable after fabrication of the LSI, or a logical device which allows reconfiguration of connections in LSI or reconfiguration of circuit cells in LSI may be adopted as the processor. The plurality of electronic circuits may be integrated into a single chip or may be distributed on a plurality of chips. The plurality of chips may be collected in one device or may be distributed in a plurality of devices. As mentioned herein, the computer system includes a microcontroller including one or more processors and one or more memories. Thus, the microcontroller is also composed of one or more electronic circuits including a semiconductor integrated circuit or a large-scale integrated circuit.

Collecting the plurality of functions in the shopping assistance system100in one housing is not an essential configuration for the shopping assistance system100. The components of the shopping assistance system100may be distributed in a plurality of housings. Moreover, at least some functions of the shopping assistance system100may be realized by, for example, cloud (cloud computing) or the like. In contrast, in the above-described embodiment, at least some of functions of the shopping assistance system100distributed in a plurality of device may be collected in one housing.

In the above-described embodiment, the ranging sensor111may be disposed directly above the container2placed at the predetermined location. In this case, the ranging sensor111measures the distance to the object in the container2from directly above the container2. Thus, image processing such as trapezoid correction is not required.

In the above-described embodiment, the recognition processing unit11may include a configuration for recognizing the state in the container2other than the ranging sensor111. For example, the recognition processing unit11may include an image sensor, transmission-type optical sensor (photo interrupter), a reflection-type optical sensor, or the like alternatively to the ranging sensor111or in addition to the ranging sensor111. The recognition processing unit11may include a transmission-type (or reflection-type) ultrasonic sensor alternatively to the ranging sensor111or in addition to the ranging sensor111.

In the above-described embodiment, the determining processor12determines, in the second determining process, whether or not the transfer operation is normal based on the state in the container2at an end time point of the operation of the movement mechanism, but the determining processor12is not limited to this example. For example, the determining processor12may determine, in the second determining process, whether or not the transfer operation is normal based on whether or not each of the goods8passes through the outlet203of the container2during the transfer operation. In this case, the recognition processing unit11is preferably configured to sense whether or not each of the goods8passes through the outlet203by the optical sensor or the ultrasonic sensor. For example, when one or more goods8remain in the container2with the relative positional relationship between the support311and the guide section312corresponding to the third state, the optical sensor or the ultrasonic sensor senses that each of the goods8passes through the outlet203. In this case, the determining processor12determines that the transfer operation is abnormal. In contrast, when no goods8remain in the container2, the optical sensor or the ultrasonic sensor senses that each of the one or more goods8do not pass through the outlet203. In this case, the determining processor12determines that the transfer operation is normal.

In the above-described embodiment, during the execution time period of the first determining process, the support311is stopped, and the relative positional relationship between the support311and the guide section312is maintained in the second state, but the first determining process is not limited to this example. For example, the first determining process may be performed while the support311is being moved downward. In this case, it is only required that the support311is stopped at the time point at which the first determining process determines that the transfer operation is abnormal.

In the above-described embodiment, when the determining processor12interrupts the checkout process by the checkout system5in the first determining process (S8inFIG. 7), the determining processor12causes the checkout process to be resumed without performing the first determining process again (S10inFIG. 7), but the determining processor12is not limited to this example. For example, in the above-described case, the determining processor12may restart the process from updating of the second image A2(S7inFIG. 7) with an assumption that the input device44receives a manipulation representing completion of correction in a state where a customer forgets to remove the one or more goods8from the outlet203. This aspect provides the advantage that even when a customer forgets to remove the one or more goods8from the outlet203, it is possible to determine at an early stage that the transfer operation still involves an abnormality. Note that in this aspect, if the determination performed again determines that the transfer operation is normal, the determining processor12causes the checkout system5to resume the checkout process, and then starts the second determining process.

In the above-described embodiment, when the determining processor12interrupts the checkout process by the checkout system5in the second determining process (S14inFIG. 11), the determining processor12causes the checkout process to be resumed without performing the second determining process again (S16inFIG. 11), but the determining processor12is not limited to this example. For example, in the determining processor12described above may restart the process from acquiring of the third image A3(S11inFIG. 11) with an assumption that the input device44receives a manipulation representing completion of correction in a state where a customer forgets to move the one or more goods8from the container2. This aspect provides the advantage that even when a customer forgets to move the one or more goods8from the container2, it is possible to determine at an early stage that the transfer operation still involves an abnormality. Note that in this aspect, if the determination performed again determines that the transfer operation is normal, the determining processor12causes the checkout system5to resume the checkout process.

In the above-described embodiment, the determining processor12performs both the first determining process and the second determining process as the determining processes, but the determining processor12is not limited to this example. For example, the determining processor12may be configured to perform only one of the first determining process and the second determining process as the determining process. That is, it is only required that the determining processor12is configured to determine at least one of whether or not the one or more goods8are stuck at the outlet203of the container2or whether or not the one or more goods8remain in the container2when the relative positional relationship between the support311and the guide section312corresponds to the third state.

In the above-described embodiment, the container2does not have to include the receiver21. For example, instead of providing the container2with the receiver21, the storage device31may include an information acquiring device for acquiring respective goods information on one or more goods8. The information acquiring device is configured to read respective goods information from radio frequency (RF) tags added to the one or more goods8so as to acquire the respective goods information. In this case, the information acquiring device reads the respective goods information when the storage device31transfers the one or more goods8. That is, the information acquiring device reads the respective goods information during a read time period after the start of the storing process until the bag (different container)10storing the one or more goods8is released from the storage device31. Moreover, in this case, the information acquiring device transmits the respective goods information to the sales system4.

In the above-described embodiment, the outlet203is a bottom surface of the basket body201of the container2, but the outlet203is not limited to this example. For example, the outlet203may be provided to a side part of the basket body201of the container2. In this case, the storage device31is disposed, for example, next to the recess72in the counter desk7, and an appropriate movement mechanism (e.g., belt conveyor) transfers one or more goods8from the container2through the outlet203into the different container10. In this case, the determining processor12determines whether or not one or more goods8remain in the container2with reference to the recognition result by the recognition processing unit11, which enables the determining processor12to determine whether or not the transfer operation is normal. The outlet203may be an upper opening in the basket body201of the container2.

In the above-described embodiment, when the determining processor12determines that the transfer operation is abnormal, the communication unit13outputs the notification signal to the sales system4, but the communication unit13is not limited to this example. For example, the communication unit13may be configured to output a notification signal to the portable terminal6carried by a clerk when the determining processor12determines that the transfer operation is abnormal. In this case, when the clerk checks the portable terminal6which receives the notification signal, it becomes possible to notify a customer that the abnormality occurs in the transfer operation, or to take appropriate measures, for example, a clerk goes to the counter desk7to perform correction instead of the customer.

In the above-described embodiment, the recognition system1, instead of a customer, may mechanically perform the correction. For example, the recognition system1has a mechanism for vibrating the container2, and the vibration may be given to the container2by the mechanism so as to attempt to release each of the one or more goods8from the outlet203of the container2or to move each of the one or more goods8remaining the container2downward.

In the above-described embodiment, the container2is not limited to the shopping basket but may be a shopping cart or the like.

In the above-described embodiment, the one or more goods8are bottles, but goods8are not limited to this example. For example, the goods8are not limited to including food and drinks such as PET-bottled beverage, canned beverage, paper-packed beverage, boxed confectionery, and bagged confectionery but may include articles of daily use such as ball-point pens and other products.

SUMMARY

As described above, a recognition system (1) according to a first aspect includes a recognition processing unit (11) and a determining processor (12). The recognition processing unit (11) is configured to recognize a state in a container (2) which allows one or more goods (8) to be put in. The determining processor (12) is configured to determine, based on a recognition result by the recognition processing unit (11) in a transfer operation, whether or not the transfer operation is normal. The transfer operation is an operation of transferring each of the one or more goods (8) from the container (2) into a different container (10) by a movement mechanism (a support (311), a guide section (312), and an elevator unit (313)) configured to move each of the one or more goods (8).

With this aspect, it is possible to determine whether or not each of the one or more goods (8) is normally transferred from the container (2) into the different container (10) smoothly, and whether or not an abnormality occurs in the process of transferring each of the one or more goods (8) from the container (2) into the different container (10). If the abnormality in the transfer operation is left as it is, a problem, for example, that the container (2) is removed with at least an item of goods (8) remaining in the container (2) may occur. In this case, a clerk has to solve the problem, and such a case is inconvenient for the clerk. In contrast, with this aspect, based on the determination result, if the abnormality is included in, for example, the transfer operation, a customer, a clerk, or the like is allowed to perform correction so that the transfer operation is normally performed. Thus, this aspect provides the advantage that a problem is suppressed in a job of transferring each of the one or more goods (8) from the container (2) into the different container (10) by a machine (here, the movement mechanism) than in a case where normality/abnormality of the transfer operation is not determined.

In a recognition system (1) according to a second aspect referring to the first aspect, the container (2) has an opening (outlet) (203) which is openable and closable. The transfer operation is an operation of changing a positional relationship of each of the one or more goods (8) relative to the container (2) in a state where the opening (203) in the container (2) is open, to transfer each of the one or more goods (8) from the container (2) through the opening (203) into the different container (10).

This aspect provides the advantage that the determining processor (12) can determine an abnormality such as sticking of the one or more goods (8) at the opening (203) in the transfer operation.

In a recognition system (1) according to a third aspect referring to the first or second aspect, the transfer operation is an operation of lowering each of the one or more goods (8) relative to the container (2) in a state where an opening (outlet) (203) in the container (2) is open, to change a positional relationship of each of the one or more goods (8) relative to the container (2) to transfer each of the one or more goods (8) from the container (2) through the opening (203) into the different container (10).

In a recognition system (1) according to a fourth aspect referring to any one of the first to third aspects, the determining processor (12) determines, based on a change of the state in the container (2) caused due to the transfer operation, whether or not the transfer operation is normal.

In this aspect, whether or not the transfer operation is normal is more easily determined than in a case where determination is performed based on only the state in the container (2) at any time point in the transfer operation.

In a recognition system (1) according to a fifth aspect referring to the fourth aspect, the determining processor (12) is configured to determine whether or not the transfer operation is normal based on a difference between a state in the container (2) prior to a start time point of an operation of the movement mechanism and a state in the container (2) after the start time point.

According to this aspect, it is possible to determine whether or not the transfer operation is normal at an earlier stage than in a case where determination is performed based on the difference between two states in the container (2) during operation of the movement mechanism.

In a recognition system (1) according to a sixth aspect referring to the fifth aspect, the state in the container (2) after the start time point is a state in the container (2) prior to an end time point of the operation of the movement mechanism.

According to this aspect, it is possible to easily determine whether or not the transfer operation is normal before the transfer operation ends.

In a recognition system (1) according to a seventh aspect referring to any one of the first to third aspects, the determining processor (12) determines whether or not the transfer operation is normal based on a state in the container (2) at an end time point of the operation of the movement mechanism.

According to this aspect, if in a job of transferring each of the one or more goods (8) from the container (2) into a different container (10), a next operation following the transfer operation is to be performed, it is possible to easily determine, based on a determination result by the determining processor (12), whether or not the next operation may be started.

In a recognition system (1) according to an eighth aspect referring to any one of the first to third aspects, the container (2) has an opening (outlet) (203) which is openable and closable. The determining processor (12) is configured to determine whether or not the transfer operation is normal based on whether or not each of the one or more goods (8) passes through the opening (203) in the container (2) during the transfer operation.

This aspect provides the advantage that an abnormality, such as remaining of at least an item of goods (8) in the container (2), in the transfer operation can be determined by the determining processor (12).

In a recognition system (1) according to a ninth aspect referring to any one of the first to eighth aspects, the recognition processing unit (11) includes a ranging sensor (111). The ranging sensor (111) is disposed outside the container (2) and is configured to measure a distance to an object in the container (2). The recognition processing unit (11) is configured to recognize the state in the container (2) based on the distance.

According to this aspect, a change of the state in the container (2) is easily recognized.

In a recognition system (1) according to a tenth aspect referring to the ninth aspect, the ranging sensor (111) is disposed at a location which is higher than the container (2) in a gravity direction and which is deviated from a center of the container (2) in a horizontal surface.

In this aspect, the distance to the object in the container (2) is more easily measured than in a case where the ranging sensor (111) is disposed at the center of the container (2) in the horizontal surface (i.e., directly above the container (2)). That is, for example, if, when the container (2) is a shopping basket having grips (204), the ranging sensor (111) is disposed directly above the container (2), the grips (204) may hinder the measurement. In contrast, if the ranging sensor (111) is disposed at a location deviated from the center of the container (2) in the horizontal surface, the ranging sensor (111) easily measures the distance to the object in the container (2) while avoiding the grips (204).

A recognition system (1) according to an eleventh aspect referring to any one of the first to tenth aspects further includes a communication unit (13). The communication unit (13) is configured to output a notification signal when the determining processor (12) determines that the transfer operation is abnormal.

With this aspect, outputting the notification signal easily prompts, for example, a customer or a clerk to take measures to correct the abnormality in the transfer operation.

A storage system (3) according to the twelfth aspect includes a recognition system (1) according to any one of the first to eleventh aspects and a storage device (31). The storage device (31) is configured to perform the transfer operation.

This aspect provides the advantage that a problem is suppressed in a job of transferring each of the one or more goods (8) from the container (2) into the different container (10) by a machine (a support (311), a guide section (312), and an elevator unit (313)).

In a storage system (3) according to a thirteenth aspect referring to the twelfth aspect, the storage device (31) is configured to perform the transfer operation based on a determination result by the determining processor (12).

This aspect provides the advantage that transfer operation according to situations can be easily performed. For example, when an abnormality occurs in the transfer operation, the transfer operation may be stopped, or when the abnormality is eliminated, the transfer operation may be resumed.

In a storage system (3) according to a fourteenth aspect referring to the twelfth or thirteenth aspect, the transfer operation includes an operation of opening and closing an opening (outlet) (203) which is formed in a container (2) and which is openable and closable.

This aspect provides the advantage that an abnormality, such as sticking of the goods (8) at the opening (203), in the transfer operation can be determined by the determining processor (12).

A shopping assistance system (100) according to a fifteenth aspect includes the storage system (3) according to any one of the twelfth to fourteenth aspects, and a sales system (4) configured to perform a sales process of the one or more goods (8) put in the container (2).

This aspect provides the advantage that a problem is suppressed in a job of transferring each of the one or more goods (8) from the container (2) into the different container (10) by a machine (a support (311), a guide section (312), and an elevator unit (313)).

A storage control method according to a sixteenth aspect includes recognizing a state in a container (2) which allows one or more goods (8) to be put in. The storage control method further includes determining, based on a recognition result in the transfer operation, whether or not the transfer operation is normal. The transfer operation is an operation of transferring each of the one or more goods (8) from the container (2) into a different container (10) by a movement mechanism (a support (311), a guide section (312), and an elevator unit (313)) configured to move each of the one or more goods (8). The storage control method further includes controlling the movement mechanism based on a determination result.

This aspect provides the advantage that a problem is suppressed in a job of transferring each of the one or more goods (8) from the container (2) into the different container (10) by a machine (a support (311), a guide section (312), and an elevator unit (313)).

A program according to a seventeenth aspect is a program that causes a computer system to execute a recognition process, a determining process, and a storage control process. The recognition process is a process of recognizing a state in a container (2) which allows one or more goods (8) to be put in. The determining process is a process of determining whether or not the transfer operation is normal based on a recognition result in the transfer operation. The transfer operation is an operation of transferring each of the one or more goods (8) from the container (2) into a different container (10) by a movement mechanism (a support (311), a guide section (312), and an elevator unit (313)) configured to move each of the one or more goods (8). The storage control process is a process of controlling the movement mechanism based on a determination result in the determining process.

This aspect provides the advantage that a problem is suppressed in a job of transferring each of the one or more goods (8) from the container (2) into the different container (10) by a machine (a support (311), a guide section (312), and an elevator unit (313)).

The configurations according to the second to eleventh aspects are not configurations essential for the recognition system (1) and may be accordingly omitted. Moreover, the configurations according to the thirteenth and fourteenth aspects are not configurations essential for the storage system (3) and may be accordingly omitted.