Apparatus, method for controlling the same, and storage medium

An apparatus acquires balance information indicating the balance of electronic money from a storage apparatus possessed by a user, and determines the total price of products picked up by the user using an image captured by an image capturing unit configured to capture an image in a store. In a case where the determined total price exceeds the balance, the apparatus outputs information notifying the user that the total price exceeds the balance.

BACKGROUND OF THE DISCLOSURE

Field of the Disclosure

The aspect of the embodiments relates to an apparatus, a method for controlling the apparatus, and a storage medium.

Description of the Related Art

Generally, in a conventional store such as a supermarket or a mass retailer, customers choose products from shelves, and sales clerks at checkout counters perform check-out operation. However, there has been an issue where the checkout area is crowded because the number of cash registers cannot be increased due to saving of labor and equipment costs. Japanese Patent Application Laid-Open No. 2019-145054 discusses a technique in which a plurality of cameras is installed in a store to capture images of customers to identify each customer, and to capture images of actions of the customers to identify which customer holds which product, and when each customer leaves the store, payment is automatically processed using a payment method that is pre-registered by the customer. Since the payment is cashless, the customer can reduce the time needed for shopping.

With the technique discussed in Japanese Patent Application Laid-Open No. 2019-145054, however, in a case where the customer is to pay in electronic money that has an upper limit, there may occur a case where the payment fails due to an insufficient balance of electronic money at the time of payment and the customer is to return a product to the original place in the store.

SUMMARY OF THE DISCLOSURE

According to an aspect of the embodiments, an apparatus includes an acquisition unit configured to acquire balance information indicating a balance of electronic money, from a storage apparatus possessed by a user, a determination unit configured to determine a total price of a product picked up by the user, using an image captured by an image capturing unit configured to capture an image in a store, and an output unit configured to output information in a case where the determined total price exceeds the balance indicated by the acquired balance information, the information notifying the user that the total price exceeds the balance.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the disclosure will now be described in detail in accordance with the accompanying drawings. The exemplary embodiments described below are mere examples and are not intended to limit the scope of the disclosure. In the drawings, the same or similar components are given the same reference numerals, and redundant descriptions thereof will be omitted.

FIG.1is a schematic diagram illustrating an example of a store100according to a first exemplary embodiment. The store100is provided with a store entrance101, a store exit105, store cameras108, voice notification apparatuses109, and store shelves110. There are customers111in the store100.

Each of the customers111carries a terminal apparatus (storage apparatus)112such as a smartphone. The terminal apparatus112has a display function and is connectable to a network. The terminal apparatus112may be an integrated circuit (IC) card. In this case, a money information reading apparatus102, an entrance area camera103, and an entrance gate104are provided at the store entrance101. In addition, an exit area camera106and an exit gate107are provided at the store exit105. There may be a plurality of the store entrances101and a plurality of the store exits105in the store100.

At the store entrance101, as processing for entry of each of the customers (users)111into the store100, the money information reading apparatus102reads, from the terminal apparatus112of the customer111, money information (the maximum amount of electronic money that the customer111can use at the time of entry into the store100) and terminal information of the terminal apparatus112. Then, the entrance area camera103captures an image of the customer111for which the money information has been read by the money information reading apparatus102, so that the entrance gate104is opened.

In the store100, video data of the store cameras108is analyzed to recognize a product held by the customer111.

At the store exit105, video data of the exit area camera106is analyzed, and when the customer111is detected from the video data, payment processing is performed as processing for exit of the customer111from the store100, based on the money information of the customer111and the product held by the customer111, so that the exit gate107is opened.

FIG.2is a block diagram illustrating examples of functions and configurations of an image processing apparatus (information processing apparatus)200according to the present exemplary embodiment and each apparatus connected to the image processing apparatus200. Each block illustrated inFIG.2can be implemented by hardware such as a central processing unit (CPU) of a computer, other elements, or mechanical devices, or by software such as a computer program. InFIG.2, functional blocks that are implemented by cooperation between such hardware and software components are illustrated. Thus, the persons skilled in the art having read the present specification should be able to understand that the functional blocks can be implemented by various combinations of hardware and software components.

The image processing apparatus200is connected to an image capturing apparatus201(corresponding to each of the various types of cameras illustrated inFIG.1), the money information reading apparatus102, each of the voice notification apparatuses109, and each of the terminal apparatuses112via a network (not illustrated inFIG.1) such as the Internet. The image processing apparatus200includes an operation unit202, a setting unit212, and a storage unit213. The operation unit202performs various operations. The setting unit212performs settings for the operations of the operation unit202. The storage unit213stores data. The operation unit202includes an image acquisition unit203, a detection unit204, a money information association unit207, a product association unit208, a payment determination unit209, a notification instruction unit210, and a payment unit211.

The image acquisition unit203acquires an image from the image capturing apparatus201and stores the acquired image in the storage unit213.

The detection unit204includes a person detection unit205and a product detection unit206. The person detection unit205detects a person in an image. The product detection unit206detects a product picked up by the person in the image. In addition, the product detection unit206tracks the detected product using a plurality of chronologically consecutive images, thereby determining whether the customer111has taken the product from the store shelf110or the customer111has returned the product to the store shelf110.

The money information association unit207acquires money information from the money information reading apparatus102, and stores the acquired money information in the storage unit213in association with information about the person detected by the person detection unit205and possessing the money information. The management of data representing the association between the person and the money information will be described below with reference toFIG.4.

The product association unit208stores information about the product detected by the product detection unit206in the storage unit213in association with the information about the person having picked up the product. The management of data representing the association between the person and the product will be described below with reference toFIG.5.

In a case where there is a change in the information about the product associated with the person by the product association unit208, the payment determination unit209determines whether the total price of the product is payable, based on the money information associated with the person by the money information association unit207.

In a case where the payment determination unit209determines that the total price is not payable, the notification instruction unit (output unit)210instructs each of the voice notification apparatus109nearest to the customer111and the terminal apparatus112of the customer111to provide a predetermined notification. At this time, the notification instruction unit210instructs the voice notification apparatus109to provide the notification by voice, and instructs the terminal apparatus112to provide the notification using a display function of the terminal apparatus112. As another exemplary embodiment, the notification instruction unit210may transmit the notification instruction to one of the voice notification apparatus109and the terminal apparatus112, but in a case where the terminal apparatus112is an IC card, the notification instruction unit210instructs the voice notification apparatus109to provide the notification by voice.

The payment unit211transmits a payment request to an external electronic money payment server (not illustrated) based on the money information associated with the person by the money information association unit207and the total price of the product.

FIG.3is a diagram illustrating an example of a hardware configuration of the image processing apparatus200according to the present exemplary embodiment. The image processing apparatus200includes a CPU301, a read-only memory (ROM)302, a random access memory (RAM)303, a hard disk drive (HDD)304, an input device305, a display device306, a disk drive307, and a network interface308. The network interface308is connected to a network309.

The CPU301is a control device that comprehensively controls the image processing apparatus200.

The ROM302stores a program for the CPU301to control the image processing apparatus200. Alternatively, a secondary storage device may be provided in place of the ROM302.

The RAM303is a memory into which the CPU301loads a program read from the ROM302to perform processing. The RAM303is also used as a temporary storage area for temporarily storing data to be subjected to various types of processing.

The HDD304stores image data and processing target data. The CPU301reads the image data and the processing target data from the HDD304.

The input device305includes a keyboard for inputting data such as numerical values, a pointing device for specifying a display position on a display region of the display device306, a mouse, and a touch panel.

The display device306is, for example, a display that displays images and region frames.

The disk drive307reads image data in a compact disk (CD), a digital versatile disk (DVD), a Blu-ray disk, or a flexible disk, and the CPU301acquires the image data via the disk drive307. In a case where the CPU301inputs image data from the HDD304or the network interface308, the disk drive307can be eliminated.

The network interface308is a circuit that performs communication via the network309. The CPU301acquires image data from an external apparatus (e.g., the image capturing apparatus201) via the network309. The CPU301transmits instructions to the voice notification apparatus109and the terminal apparatus112, which are external apparatuses, via the network309.

As described above, the image processing apparatus200has a hardware configuration including hardware components similar to those installed in a commonly- used personal computer (PC). Thus, each function performed by the image processing apparatus200can be implemented as software that operates on a PC. Various functions of the image processing apparatus200can be performed by the CPU301of the image processing apparatus200executing a program.

FIG.4illustrates an example of a structure of data representing associations between persons and money information that are managed by the image processing apparatus200according to the present exemplary embodiment.

A person identifier401is used to identify the customer111. The person identifier401is issued when the person detection unit205detects the customer111from the image data captured by the entrance area camera103at the store entrance101.

Money information402indicates the upper limit amount (balance information) of electronic money usable by the customer111identified by the person identifier401. The money information reading apparatus102at the store entrance101reads the money information402from the terminal apparatus112of the customer111, and the money information association unit207acquires the money information402from the money information reading apparatus102.

Terminal information403is used to identify the terminal apparatus112of the customer111identified by the person identifier401. The money information reading apparatus102at the store entrance101reads the terminal information403from the terminal apparatus112of the customer111, and the money information association unit207acquires the terminal information403from the money information reading apparatus102. Although not illustrated inFIG.4, information for providing notification of an insufficient balance of electronic money to the terminal apparatus112is also managed by the image processing apparatus200. The information for providing the notification to the terminal apparatus112is, for example, an email address of an electronic mail viewable on the terminal apparatus112. In a case where an application dedicated to the store100is installed in the terminal apparatus112, the information is used to provide the notification to the application.

FIG.5illustrates an example of a structure of data representing associations between persons and products that are managed by the image processing apparatus200according to the present exemplary embodiment.

A person identifier501is used to identify the customer111and is similar to the person identifier401.

A product502is the product picked up by the customer111and detected by the product detection unit206.

A price503is the price of the product502. The management of information about products in the store100including the price503will be described below with reference toFIG.6.

A quantity504is the number of pieces of the product502picked up by the customer111. The quantity504is detected by the product detection unit206.

A total (or a total price)505is the total price of the product(s)502picked up by the customer111.

FIG.6illustrates an example of a structure of data representing information about each product in the store100that is managed by the image processing apparatus200according to the present exemplary embodiment.

A product category601indicates a type of product. A product602is used to identify each product in the same product category601and is similar to the product502. A price603is the price of the product602and is similar to the price503.

FIG.7illustrates an example of the insufficient balance notification that is provided by the display function of the terminal apparatus112based on the notification instruction from the image processing apparatus200according to the present exemplary embodiment.

As another exemplary embodiment, in a case where the image processing apparatus200transmits the notification instruction to the terminal apparatus112using a dedicated application, the dedicated application can operate a vibration function of the terminal apparatus112to provide the notification.

FIG.8is a flowchart illustrating an example of processing that is performed by the image processing apparatus200according to the present exemplary embodiment during a period from when the customer111enters the store100to when the customer111exits the store100. The processing illustrated inFIG.8is performed by the functional units of the image processing apparatus200.

In step S801, the image processing apparatus200acquires the money information402and the terminal information403of the customer111from the money information reading apparatus102.

In step S802, the image processing apparatus200acquires, from the entrance area camera103, captured image data of the customer111for which the money information402has been acquired in step S801.

In step S803, the image processing apparatus200analyzes the image data acquired in step S802and detects the customer111. Then, the image processing apparatus200issues the person identifier401for identifying the customer111.

In step S804, the image processing apparatus200stores the person identifier401issued in step S803in association with the money information402and the terminal information403that are acquired in step S801.

In step S805, the image processing apparatus200acquires, from the store cameras108, captured image data of the inside area of the store100for analysis in step S806. The image processing apparatus200also acquires, from the exit area camera106, captured image data of the area at and around the store exit105.

In step S806, the image processing apparatus200analyzes the image data acquired in step S805.

In step S807, in a case where the image processing apparatus200detects the product502picked up by the customer111or detects the customer111at the store exit105as a result of the analysis in step S806(YES in step S807), the processing proceeds to step S808. On the other hand, in a case where the image processing apparatus200detects neither the product502nor the customer111(NO in step S807), the processing returns to step S805.

In step S808, in a case where the image processing apparatus200detects the product502picked up by the customer111as a result of the analysis in step S806, the image processing apparatus200determines that the customer111performs an action of either taking the product502from the store shelf110or returning the product502to the store shelf110(NO in step S808) and the processing proceeds to step S809. On the other hand, in a case where the customer111is detected at the store exit105in step S806, the image processing apparatus200determines that the action of the customer111is payment (YES in step S808), and the processing proceeds to step S812.

In step S809, the image processing apparatus200stores the person identifier401issued in step S803and information about the product502detected in step S806in association with each other. In a case where the image processing apparatus200determines that the customer111has performed the action of returning the product502to the store shelf110as a result of the analysis in step S806, in step S809, the image processing apparatus200cancels the association of the returned product502with the customer111.

In step S810, the image processing apparatus200determines whether the total price505is payable, based on the money information402of the customer111. In a case where the total price505is payable (YES in step S810), the processing returns to step S805. Otherwise (NO in step S810), the processing proceeds to step S811.

In step S811, the image processing apparatus200transmits an instruction to notify the customer111of the insufficient balance, to the voice notification apparatus109nearest to the customer111and the terminal apparatus112. At this time, the image processing apparatus200instructs the voice notification apparatus109to provide the notification by voice, and instructs the terminal apparatus112to provide the notification using the display function of the terminal apparatus112. After the transmission of the instruction, the processing returns to step S805.

In step S812, the image processing apparatus200transmits a payment request to the external electronic money payment server (not illustrated) based on the money information402of the customer111and the total price505, and the processing ends.

As described above, according to the present exemplary embodiment, the customer111is notified of an insufficient balance of electronic money before paying with electronic money, so that the customer111can take action such as charging electronic money or returning a product. In other words, an insufficient balance of electronic money at the time of payment with electronic money can be prevented.

In a second exemplary embodiment, a method for notifying the customer111that no products in the store100can be purchased with the current electronic money balance, in addition to the above-described processing according to the first exemplary embodiment, in a case where no products in the store100can be purchased with the current electronic money balance will be described. Redundant descriptions of portions similar to those according to the first exemplary embodiment will be omitted, and only characteristic features according to the second exemplary embodiment will be described.

FIG.9illustrates an example of a notification that no products in the store100can be purchased with the current electronic money balance, which is provided by the display function of the terminal apparatus112based on a notification instruction from the image processing apparatus200according to the present exemplary embodiment.

A message902notifies the customer111that no products in the store100can be purchased with the current balance. In a case where the image processing apparatus200transmits the notification instruction to the terminal apparatus112using an electronic mail, the message902is displayed by the customer111opening the electronic mail. In a case where the image processing apparatus200transmits the notification instruction to the terminal apparatus112using a dedicated application, the message902is displayed by push notification from the dedicated application.

FIG.10is a flowchart illustrating an example of processing that is performed by the image processing apparatus200according to the present exemplary embodiment during a period from when the customer111enters the store100to when the customer111exits the store100. InFIG.10, steps S1013and S1014are added to the processing illustrated inFIG.8.

In step S1013, the image processing apparatus200determines whether there is a product in the store100that can be purchased with the current electronic money balance, based on the money information402of the customer111and the total price505. To make this determination, data on the prices of the cheapest products in the store100is stored in advance in the image processing apparatus200. In a case where there is a product in the store100that can be purchased with the current balance (YES in step S1013), the processing returns to step S805. Otherwise (NO in step S1013), the processing proceeds to step S1014.

In step S1014, the image processing apparatus200transmits an instruction to notify the customer111that no products in the store100can be purchased with the current balance, to the voice notification apparatus109nearest to the customer111and the terminal apparatus112. At this time, the image processing apparatus200instructs the voice notification apparatus109to provide the notification by voice, and instructs the terminal apparatus112to provide the notification using the display function of the terminal apparatus112. After the transmission of the instruction, the processing returns to step S805.

As described above, according to the present exemplary embodiment, during the period after picking up a product and before picking up another product, the customer111is notified that no more products can be purchased with the current electronic money balance.

In the first exemplary embodiment, in a case where the total price505is not payable, the customer111is notified of the insufficient balance. In a third exemplary embodiment, a method for notifying the customer111of a combination of products that maximizes the total price505within the current balance, in addition to providing the insufficient balance notification will be described below. Redundant descriptions of portions similar to those according to the first exemplary embodiment will be omitted, and only characteristic features according to the third exemplary embodiment will be described.

FIG.11illustrates an example of an insufficient balance notification provided by the display function of the terminal apparatus112based on a notification instruction from the image processing apparatus200according to the present exemplary embodiment.

A message1102notifies the customer111of the insufficient balance, and also notifies the customer111of a combination of products that maximizes the total price505within the current balance (i.e., in the range less than or equal to the current balance). In a case where the image processing apparatus200transmits the notification instruction to the terminal apparatus112using an electronic mail, the message1102is displayed by the customer111opening the electronic mail. In a case where the image processing apparatus200transmits the notification instruction to the terminal apparatus112using a dedicated application, the message1102is displayed by push notification from the dedicated application.

FIG.12is a flowchart illustrating an example of processing that is performed by the image processing apparatus200according to the present exemplary embodiment during a period from when the customer111enters the store100to when the customer111exits the store100. InFIG.12, step S1213is added to the processing illustrated inFIG.8.

In step S1213, the image processing apparatus200determines a combination of products that maximizes the total price505within the current balance, based on the money information402of the customer111and the price503. The determined combination is used for the notification instruction in step S811.

As described above, according to the present exemplary embodiment, the customer111is notified of a combination of products that can be purchased using the current electronic money balance to the maximum extent possible.

In the first exemplary embodiment, in a case where the total price505is not payable, the customer111is notified of the insufficient balance. In a fourth exemplary embodiment, a method for notifying the customer111of a combination of products including a less expensive alternative product in a case where the alternative product is in the store100, in addition to providing the insufficient balance notification will be described below. Redundant descriptions of portions similar to those according to the first exemplary embodiment will be omitted, and only characteristic features according to the fourth exemplary embodiment will be described.

FIG.13illustrates an example of an insufficient balance notification that is provided by the display function of the terminal apparatus112based on a notification instruction from the image processing apparatus200according to the present exemplary embodiment.

A message1302notifies the customer111of the insufficient balance, and also notifies the customer111of a combination of products including a less expensive alternative product (a less expensive similar product) in a case where the alternative product is in the store100. In a case where the image processing apparatus200transmits the notification instruction to the terminal apparatus112using an electronic mail, the message1302is displayed by the customer111opening the electronic mail. In a case where the image processing apparatus200transmits the notification instruction to the terminal apparatus112using a dedicated application, the message1302is displayed by push notification from the dedicated application.

FIG.14is a flowchart illustrating an example of processing that is performed by the image processing apparatus200according to the present exemplary embodiment during a period from when the customer111enters the store100to when the customer111exits the store100. InFIG.14, step S1413is added to the processing illustrated inFIG.8.

In step S1413, the image processing apparatus200determines whether an alternative product that is of the same product type and is less expensive is in the store100, based on the money information402and the product502that are associated with the customer111and the information about the product602in the store100that is described above with reference toFIG.6. Then, in a case where there is an alternative product, a combination of products including the alternative product is determined. The determined combination is used for the notification instruction in step S811.

As described above, according to the present exemplary embodiment, the customer111can do the shopping without reducing the number of products to be purchased if the balance is insufficient.

In the above-described exemplary embodiments, after the product502picked up by the customer111is detected, whether the total price505is payable is determined. In a fifth exemplary embodiment, before the customer111picks up the product502, whether the total price505based on the assumption that the product502has been picked up is payable is determined.

More specifically, the image processing apparatus200detects a movement direction of the hand of the customer111, and in a case where the distance between the hand of the customer111and the product502ahead of the hand of the customer111becomes a predetermined value, the product502is determined to have been picked up by the customer111. Then, the image processing apparatus200performs step S808and the subsequent steps inFIG.8.

As described above, according to the present exemplary embodiment, in a case where the electronic money balance is insufficient, the customer111is notified of the insufficient balance before picking up the product502. This can save the customer from having to return the product502to the store shelf110.

While the exemplary embodiments of the disclosure have been described in detail above, the disclosure is not limited to the specific exemplary embodiments, and various forms within the spirit of the disclosure are encompassed in the scope of the disclosure. In addition, the above-described exemplary embodiments can partially be combined as needed.

Furthermore, processing in which a software program for implementing the functions according to the above-described exemplary embodiments is supplied to a system or an apparatus including a computer capable of executing the program directly from a recording medium or via wired or wireless communication and the program is executed is also included in the exemplary embodiments of the disclosure. Thus, program codes that are supplied to a computer and installed in the computer so that functional processing according to an exemplary embodiment of the disclosure is implemented by the computer also implement an exemplary embodiment of the disclosure. In other words, a computer program for implementing functional processing according to an exemplary embodiment of the disclosure is also encompassed in the exemplary embodiments of the disclosure. In this case, the program can be in any form having a program function, such as an object code, a program to be executed by an interpreter, or script data to be supplied to an operating system (OS).

A recording medium for supplying a program can be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical or magnetooptical storage medium, or a non-volatile semiconductor memory. Furthermore, one possible example of a method for supplying a program is that a computer program constituting an exemplary embodiment of the disclosure is stored on a server on a computer network and a client computer connected to the server downloads and executes the computer program.

This application claims the benefit of Japanese Patent Application No. 2020-114212, filed Jul. 1, 2020, which is hereby incorporated by reference herein in its entirety.