INFORMATION PROCESSING PROGRAM, INFORMATION PROCESSING METHOD, AND INFORMATION PROCESSING DEVICE

A non-transitory computer-readable recording medium stores an information processing program for causing a computer to execute a process including: obtaining video obtained by imaging an inside of a store; analyzing the obtained video to specify, from the video, a first area that includes a product arranged in the store, a second area that includes a customer to purchase the product, and a relationship in which interaction between the product and the customer is identified; generating attention information that indicates a degree of attention of a location where the product is arranged based on the specified relationship; and causing a display device to output the generated attention information.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2022-199826, filed on Dec. 14, 2022, the entire contents of which are incorporated herein by reference.

FIELD

The embodiment discussed herein is related to an information processing program, an information processing method, and an information processing device.

BACKGROUND

There is an approach to improve a conversion rate by analyzing behavior exhibited when a person who visits a retail store or the like purchases a product, which is what is called purchasing behavior. For example, in a case where a person who compares items less than five times has a high tendency to purchase an item while a person who compares items equal to or more than five times has a high tendency to leave without purchasing an item in a clothing store, a conversion rate may be improved by guiding the number of try-on items to less than five at the time of service.

Japanese Laid-open Patent Publication No. 2022-165483 is disclosed as related art.

SUMMARY

According to an aspect of the embodiments, a non-transitory computer-readable recording medium stores an information processing program for causing a computer to execute a process including: obtaining video obtained by imaging an inside of a store; analyzing the obtained video to specify, from the video, a first area that includes a product arranged in the store, a second area that includes a customer to purchase the product, and a relationship in which interaction between the product and the customer is identified; generating attention information that indicates a degree of attention of a location where the product is arranged based on the specified relationship; and causing a display device to output the generated attention information.

DESCRIPTION OF EMBODIMENTS

However, even for the same product, behavior exhibited by a customer may change depending on a location of a shelf or the like where the product is arranged. In other words, it is considered that the customer may easily exhibit purchasing behavior at some place and may hardly exhibit purchasing behavior at some other place depending on the location where the product is arranged. Therefore, it is considered that analysis of the purchasing behavior of the customer due to a difference in the location where the product is arranged leads to improvement in the conversion rate.

In one aspect, an object is to provide an information processing program, an information processing method, and an information processing device capable of more accurately analyzing purchasing behavior of a customer due to a difference in a location where a product is arranged.

Hereinafter, examples of an information processing program, an information processing method, and an information processing device according to the present embodiment will be described in detail with reference to the drawings. Note that the present embodiment is not limited by the examples. Furthermore, the individual examples may be appropriately combined within a range without inconsistency.

First, an information processing system for implementing the present embodiment will be described.FIG.1is a diagram illustrating an exemplary configuration of the information processing system according to the present embodiment. As illustrated inFIG.1, an information processing system1is a system in which an information processing device10, accounting machines100-1to100-m, and camera devices200-1to200-nare communicably coupled to each other via a network50. Note that m and n are any natural numbers, and hereinafter, the accounting machines100-1to100-mand the camera devices200-1to200-nmay be collectively referred to as an “accounting machine100” and a “camera device200”, respectively.

For the network50, various communication networks, such as an intranet used in a store of a retail store and the like, may be adopted regardless of whether it is wired or wireless, for example. Furthermore, the network50may not be a single network, and may be, for example, the intranet and the Internet configured via a network device such as a gateway or another device (not illustrated). Note that the expression “in a store” of a retail store or the like is not limited to indoors, and may include outdoors within the premises of the retail store or the like.

The information processing device10is, for example, an information processing device such as a desktop personal computer (PC), a laptop PC, a server computer, or the like installed in a store of a retail store and the like and used by a store staff, an administrator, and the like. Alternatively, the information processing device10may be a cloud computer device managed by a service provider that provides cloud computing services.

For example, the information processing device10receives, from the camera device200, video obtained by imaging a predetermined imaging range such as each sales floor, a checkout area, and the like in a store of a retail store and the like using the camera device200. Note that the video technically includes a plurality of captured images imaged by the camera device200, that is, a series of frames of a moving image.

Furthermore, the information processing device10extracts products arranged in the store and customers staying in the store from the video imaged by the camera device200using, for example, an existing object detection technique. Furthermore, the information processing device10specifies, for example, a relationship in which interaction between a product and a customer is identified, such as that the client views the product for a short time or a long time, touches the product, grasps the product, sits on the product, or the like. Alternatively, for example, the information processing device10generates attention information indicating a degree of attention of a location of a shelf or the like where the product is arranged based on the specified relationship, and causes a display device to output the attention information. Furthermore, for example, the information processing device10may specify a type of the relationship indicated by the customer, such as grasping the product, count the actual quantity for each type of the relationship, and cause the display device to output the result in association with the attention information. Note that the display device may be a device included in the information processing device10, may be an external device, or may be a device included in another information processing device communicably coupled to the information processing device10via the network50.

Furthermore, for example, the information processing device10tracks the customer extracted from the video, and further analyzes the video to identify the accounting machine100used by the tracked customer.

Furthermore, the information processing device10specifies the purchase quantity of products arranged at a location in the store where the customer has presented a predetermined relationship based on, for example, registration information obtained from the identified accounting machine100. Then, for example, the information processing device10causes the display device to output the specified purchase quantity for each location where the products are arranged in the store in association with the attention information indicating the degree of attention of the location where the products are arranged.

Then, by analyzing the attention information output to the display device, the store staff or the like is enabled to analyze, for example, whether the purchase quantity actually changes depending on a location of a shelf or the like where the product is arranged and where the customer has presented a predetermined relationship, such as grasping the product. In other words, the store staff or the like is enabled to analyze, for example, using the attention information, whether the purchase quantity of the product changes when the location where the product is arranged is changed even for the same product.

Furthermore, the information processing device10is enabled to determine whether the person extracted from the video is a salesclerk or a customer using, for example, a machine learning model, and to identify the customer. As a result, the information processing device10is enabled to exclude the salesclerk from the processing target and the analysis target.

Note that, whileFIG.1illustrates the information processing device10as one computer, it may be a distributed computing system including a plurality of computers.

The accounting machine100is, for example, a cash register terminal to be operated by a salesclerk to register information regarding a purchased product, or a self-checkout terminal to be operated by a customer him/herself to register information regarding a purchased product. For example, the customer pays for the purchased product with cash, a credit card, electronic money, or the like based on the information regarding the purchased product registered via the accounting machine100. Furthermore, the accounting machine100transmits, as registration information, the registered information regarding the purchased product to the information processing device10, for example. Note that the term accounting machine is an example, and may also be referred to as, for example, self checkout, automated checkout, a self-checkout machine, a self-check-out register, or the like.

The camera device200is, for example, a monitoring camera installed in each sales floor and a checkout area in a store of a retail store and the like. The video imaged by the camera device200is transmitted to the information processing device10. Furthermore, positional information, such as coordinates, for identifying each product and a sales floor area is allocated to each product and the sales floor area to be imaged by the camera device200, and for example, the information processing device10is enabled to identify each product and the sales floor area from the video received from the camera device200.

(Functional Configuration of Information Processing Device10)

Next, a functional configuration of the information processing device10will be described.FIG.2is a diagram illustrating an exemplary configuration of the information processing device10according to the present embodiment. As illustrated inFIG.2, the information processing device10includes a communication unit11, a storage unit12, and a control unit20.

The communication unit11is a processing unit that controls communication with another device such as the camera device200, and is, for example, a communication interface such as a network interface card.

The storage unit12has a function of storing various kinds of data and programs to be executed by the control unit20, and is implemented by, for example, a storage device such as a memory, a hard disk, or the like. The storage unit12stores an imaging DB13, a camera installation DB14, a product DB15, an accounting registration DB16, a model DB17, an attention DB18, and the like. Note that the DB is an abbreviation for a database.

The imaging DB13stores a plurality of captured images, which is a series of frames captured by the camera device200. The plurality of captured images imaged by the camera device200, which is the video, is transmitted from the camera device200as needed, received by the information processing device10, and stored in the imaging DB13.

The camera installation DB14stores, for example, information for identifying a location where each of the camera devices200is installed. The information stored here may be set in advance by, for example, the administrator or the like.FIG.3is a diagram illustrating exemplary information stored in the camera installation DB14according to the present embodiment. As illustrated inFIG.3, the camera installation DB14stores, for example, individual pieces of information such as “camera ID, installation location”, and the like in association with each other. In the “camera ID” stored here, for example, information such as an identifier for uniquely identifying each of the camera devices200is set, and in the “installation location”, for example, information for identifying a location where each of the camera devices200is installed is set. The example ofFIG.3illustrates an exemplary case where a camera with camera ID=1 is installed in a baby goods sales floor and a camera with camera ID=101 is installed in a checkout area where an accounting machine A is located.

Returning to the description ofFIG.2, the product DB15stores, for example, information regarding products arranged in each sales floor. The information stored here may also be set in advance by, for example, the administrator or the like.FIG.4is a diagram illustrating exemplary information stored in the product DB15according to the present embodiment. As illustrated inFIG.4, the product DB15stores, for example, individual pieces of information such as “product ID, product name, arrangement location, area coordinates”, and the like in association with each other. In the “product ID” stored here, for example, information such as an identifier for uniquely identifying each product is set, and in the “product name”, for example, names of individual products are set. Furthermore, in the “arrangement location”, for example, information for identifying a location where each product is arranged is set. Note that each product and a location where each product is arranged may be associated with each of the camera devices200and a location where the camera device200is installed by, for example, setting the same information as the “installation location” of the camera installation DB14in the “arrangement location”. Furthermore, in the “area coordinates”, for example, positional information of pixels of a location where each product is arranged in the captured image imaged by the camera device200may be set. The positional information will be described more specifically with reference toFIG.5.

FIG.5is a diagram illustrating an exemplary area of a product arrangement location according to the present embodiment.FIG.5is a diagram illustrating a captured image251of a certain sales floor imaged by the camera device200. The captured image251includes areas A to C including a location where individual products are arranged. The information processing device10is enabled to more precisely identify the location where each product is arranged by setting the positional information of each area in the captured image251as illustrated inFIG.5, for example, the coordinates of each vertex of each area in the “area coordinates” of the product DB15. Note that, while the captured image251including the plurality of product arrangement locations has been described with reference toFIG.5, one product arrangement location may be imaged by the camera device200, for example. In this case, the product arrangement location is determined for each captured image imaged by the camera device200, and thus the “area coordinates” are not necessarily stored in the product DB15.

Returning to the description ofFIG.2, the accounting registration DB16stores, for example, information regarding a purchased product registered via the accounting machine100. The information regarding the purchased product registered via the accounting machine100is transmitted from the accounting machine100as needed, received by the information processing device10, and stored in the accounting registration DB16.FIG.6is a diagram illustrating exemplary information stored in the accounting registration DB16according to the present embodiment. As illustrated inFIG.6, the accounting registration DB16stores, for example, individual pieces of information such as “accounting machine ID, customer ID, purchased product ID, purchase quantity”, and the like in association with each other. In the “accounting machine ID” stored here, for example, information such as an identifier for uniquely identifying the accounting machine100is set. Furthermore, in the “customer ID”, information such as an identifier for uniquely identifying the customer who uses the accounting machine100, the customer being extracted from the video and tracked by the information processing device10, is set. Furthermore, in the “purchased product ID”, for example, information such as an identifier for uniquely identifying each of the purchased products is set. Note that the product DB15and the accounting registration DB16may be associated with each other by, for example, setting the same information as the “product ID” of the product DB15in the “purchased product ID”. Furthermore, in the “purchase quantity”, for example, the number of items of the purchased products is set.

Returning to the description ofFIG.2, the model DB17stores, for example, information regarding a machine learning model for specifying an area including a product and a customer and a relationship between the product and the customer from a captured image in the store, and model parameters for constructing the model. The machine learning model is generated through machine learning using, for example, video, which is captured images obtained by imaging the inside of the store with the camera device200, as input data, and an area including a product and a customer and a type of a relationship between the product and the customer as ground truth labels. Note that the type of the relationship between the product and the customer may be, for example, an action of the customer such as viewing the product for a short time or a long time, touching the product, grasping the product, sitting on the product, or the like, but it is not limited to those actions. Furthermore, the area including the product and the customer may be, for example, a bounding box (Bbox) that rectangularly surrounds the area in the captured image.

Furthermore, the model DB17may store, for example, information regarding a machine learning model for determining whether a person identified from a captured image in the store is a salesclerk or a customer, and model parameters for constructing the model. The machine learning model is generated through machine learning using, for example, partial images of a person extracted from video obtained by imaging the inside of the store with the camera device200as input data and information indicating whether the person is a salesclerk or a customer as a ground truth label.

Note that, hereinafter, the machine learning model for specifying a relationship between a product and a customer and the like and the machine learning model for determining whether a person is a salesclerk or a customer may be referred to as a “product/customer relationship specifying model” and a “salesclerk/customer determination model”, respectively, to make them distinguishable from each other. Furthermore, the product/customer relationship specifying model and the salesclerk/customer determination model may be trained and generated by the information processing device10, or may be trained and generated by another information processing device.

The attention DB18stores, for example, attention information indicating a degree of attention of a location of a shelf or the like where products are arranged.FIG.7is a diagram illustrating exemplary information stored in the attention DB18according to the present embodiment. The attention information stored here may be generated and set by the generation unit23based on, for example, information such as the relationship between the product and the customer specified by the specifying unit22. As illustrated inFIG.7, the attention DB18stores, for example, individual pieces of information such as “product ID, product name, arrangement location, relationship, actual quantity, purchase quantity”, and the like in association with each other. In the “product ID” stored here, for example, information such as an identifier for uniquely identifying each product is set, and in the “product name”, for example, names of individual products are set. Furthermore, in the “arrangement location”, for example, information for identifying a location where each product is arranged is set. Furthermore, in the “relationship”, for example, a type of the specified relationship between the product and the customer is set. Furthermore, in the “actual quantity”, for example, an actual quantity for each type of the relationship presented by the customer is set. Furthermore, in the “purchase quantity”, for example, a purchase quantity of products arranged at a predetermined arrangement location and having a predetermined relationship presented by the customer. More specifically, in the example ofFIG.7, individual customers who present a relationship of grasping baby milk arranged at individual arrangement locations are tracked, and the purchase quantity is set based on the information regarding the purchased products registered via the accounting machine100used by the individual customers.

The example ofFIG.7indicates that the individual customers present the relationship of grasping the baby milk arranged in a baby goods sales floor 21 times, and that the purchase quantity is 19. Furthermore, it is indicated that the individual customers present the relationship of grasping the baby milk arranged in a powdered drink sales floor three times, and that the purchase quantity is 2. Note that, in the example ofFIG.7, since the product is baby milk, only the purchase quantity in the case where the relationship of grasping the product is presented is set on the premise that the relationship of grasping the product is presented to make a purchase. However, depending on the product, for example, in a case of a large-size product or the like, a customer may place an order with a salesclerk without grasping the product. Therefore, the purchase quantity of a relationship other than the relationship of grasping may be set by tracking a customer who presents a relationship other than the relationship of grasping.

Note that the information described above stored in the storage unit12is merely an example, and the storage unit12may store various kinds of information other than the information described above.

The control unit20is a processing unit that takes overall control of the information processing device10, and is, for example, a processor or the like. The control unit20includes an acquisition unit21, a specifying unit22, a generation unit23, and an output unit24. Note that each processing unit is an example of an electronic circuit included in a processor, or an example of a process to be performed by the processor.

For example, the acquisition unit21obtains video obtained by imaging the inside of the store with the camera device200from the imaging DB13. Note that the video obtained by imaging the inside of the store with the camera device200is transmitted to the information processing device10by the camera device200as needed, received by the information processing device10, and stored in the imaging DB13.

The specifying unit22analyzes the video obtained by the acquisition unit21to specify, from the video, a first area including a product arranged in the store, a second area including a customer to purchase the product, and a relationship in which interaction between the product and the customer is identified. Such a specifying process may include, for example, processing of inputting the video obtained by the acquisition unit21to the product/customer relationship specifying model, which is the machine learning model described, to specify the first area, the second area, and the relationship. Furthermore, the product/customer relationship specifying model may be a human object interaction detection (HOID) model trained to identify information regarding a first area, information regarding a second area, and interaction between a first class and a second class. Here, the information regarding the first area may be information indicating the first class indicating an object including a product and an area where the object appears, and the information regarding the second area may be information indicating the second class indicating a person who purchases the product and an area where the person appears. The HOID will be described more specifically with reference toFIG.8.

FIG.8is a diagram for explaining specification of a relationship by the HOID. As illustrated inFIG.8, the specifying unit22inputs image data, which is a captured image of the inside of the store, to the HOID machine learning model, and obtains an output result. As illustrated inFIG.8, the output result includes, for example, a Bbox261of a person, a class name of the person, a Bbox262of an object, a class name of the object, a probability value of interaction between the person and the object, a class name of the interaction between the person and the object, and the like.

Returning to the description ofFIG.2, the specifying unit22specifies a type of the relationship indicated by each customer at a predetermined location where the product is arranged in the store based on, for example, the specified relationship and the position of the customer when the specified relationship is indicated. In such a specifying process, the type of the relationship may be specified based on, for example, the probability value of the interaction between the person and the object, the class name of the interaction between the person and the object, and the like output from the HOID machine learning model described with reference toFIG.8. More specifically, for example, the specifying unit22specifies the type of the relationship indicated by the class name of the interaction having the highest probability value of the interaction between the person and the object, which is output from the HOID machine learning model, as a type of the relationship presented by the person with respect to the object. As an example, the specifying unit22specifies the action of “grasping” baby milk, which is the object, taken by the customer, who is the person present in the captured image, as a type of the relationship from the output result obtained by inputting the captured image of the baby goods sales floor to the HOID machine learning model. Furthermore, as another example, the specifying unit22specifies the action of “short-time viewing” of baby milk taken by the customer as a type of the relationship from the output result obtained by inputting the captured image of the powdered drink sales floor to the HOID machine learning model.

Furthermore, the specifying unit22counts, for example, the actual quantity for each type of the relationship presented by the customer at a predetermined location. More specifically, for example, the specifying unit22counts actual quantities of “short-time viewing”, “long-time viewing”, “grasping”, and the like specified as types of the relationship presented by the customer with respect to the baby milk at the baby goods sales floor, which are respective numbers of times of specification. According to the actual quantity counted in this manner, for example, it becomes possible to make an analysis such as the baby milk has been grasped a large number of times at the baby goods sales floor while it has been hardly grasped at the powdered drink sales floor and has been only viewed for a short time a large number of times.

Furthermore, the specifying unit22specifies a location in the store where the customer has grasped the product based on, for example, a predetermined relationship and the position of the customer when the predetermined relationship is presented. Then, for example, the specifying unit22tracks the customer included in the video and analyzes the video, thereby identifying the accounting machine100used by the tracked customer. The tracking of the customer will be described more specifically with reference toFIG.9.

FIG.9is a diagram for explaining the tracking of the customer. As illustrated inFIG.9, the specifying unit22extracts persons from a plurality of pieces of image data captured by the camera devices200installed at various places such as an entrance of the store, each sales floor, a checkout area, an exit, and the like, identifies the same person from the extracted persons, and tracks each of the persons. The person extraction and tracking from the image data will be described more specifically with reference toFIG.10.

FIG.10is a diagram illustrating exemplary person extraction and tracking from in-store images according to the present embodiment. As illustrated inFIG.10, the specifying unit22extracts a person from the in-store images using, for example, an existing detection algorithm such as you only look once (YOLO), single shot multibox detector (SSD), region based convolutional neural networks (RCNN), or the like. The in-store images are individual pieces of image data captured by the camera device200, and as illustrated inFIG.10, for example, a person extracted from the image data is indicated by a Bbox that rectangularly surrounds the area in the image.

Furthermore, as illustrated in the upper part ofFIG.10, it is a matter of course that a plurality of persons may be extracted from the in-store images. Therefore, as illustrated in the lower part ofFIG.10, the specifying unit22identifies the same person between frames based on, for example, a similarity level of Bboxes of the persons among the plurality of pieces of image data, that is, among a plurality of frames. For example, an existing tracking algorithm such as tracking learning detection (TLD), kernelized correlation filters (KCF), or the like may be used to identify the same person.

Returning to the description ofFIG.2, the specifying unit22further specifies the purchase quantity of products arranged at a location in the store where the customer has presented a predetermined relationship based on, for example, the registration information obtained from the identified accounting machine100. Note that the registration information is information regarding the purchased product registered via the accounting machine100, which is transmitted to the information processing device10by the accounting machine100as needed, received by the information processing device10, and stored in the accounting registration DB16. According to the purchase quantity specified in this manner, for example, it becomes possible to make an analysis such as whether or not the baby milk grasped by the customer at the baby goods sales floor has been purchased. Furthermore, by comparing the actual quantity in which the customer has grasped the baby milk at the baby goods sales floor with the purchase quantity, it becomes possible to make an analysis such as how much the baby milk grasped at the baby goods sales floor has been purchased.

Furthermore, the specifying unit22is enabled to determine whether the person extracted from the video is a salesclerk or a customer using, for example, the salesclerk/customer determination model, which is the machine learning model described above, and to identify the customer. As a result, the information processing device10is enabled to exclude the salesclerk from the processing target and the analysis target, and to reduce the processing load of the information processing device10. The process of the salesclerk/customer determination will be described more specifically with reference toFIG.11.

FIG.11is a diagram illustrating exemplary salesclerk/customer determination according to the present embodiment. As illustrated on the left side ofFIG.11, in a training phase, the information processing device10trains and generates a salesclerk/customer determination model using partial images of a person extracted from the video as input data and information indicating whether the person is a salesclerk or a customer as a ground truth label. The partial image of the person in the input data may be an image in a Bbox that rectangularly surrounds the area of the person. Furthermore, for example, a neural network (NN) may be used as the salesclerk/customer determination model.

Next, as illustrated on the right side ofFIG.11, in a determination phase, the information processing device10inputs a partial image of a person extracted from the video of the camera device200to the salesclerk/customer determination model trained in the training phase, and determines whether the person is a salesclerk or a customer. Note that the example ofFIG.11indicates that the person in the input partial image is determined to be a salesclerk.

Returning to the description ofFIG.2, the generation unit23generates attention information indicating a degree of attention of a location where the product is arranged based on, for example, the relationship in which the interaction between the product and the customer is identified, which is specified by the specifying unit22. Examples of the attention information generated by the generation unit23include information stored in the attention DB18as illustrated inFIG.7.

For example, the output unit24causes the display device to output the attention information generated by the generation unit23. Furthermore, the output unit24may cause the display device to output, for example, the attention information as illustrated inFIG.7in which the actual quantity counted for each type of a relationship between the product and the customer presented by the customer is associated with a predetermined location where the relationship is presented. In particular, the output unit24may cause the display device to output, for example, the attention information as illustrated inFIG.7in which the actual quantity of the relationship indicating that the customer has grasped the product is associated with each location where the products are arranged. Moreover, the output unit24may cause the display device to output, for example, the attention information as illustrated inFIG.7in which the purchase quantity of the products arranged at the location in the store where the customer has presented a predetermined relationship is associated with each location where the products are arranged in the store.

Next, a flow of an attention information output process to be executed by the information processing device10will be described.FIG.12is a flowchart illustrating a flow of the attention information output process according to the present embodiment.

First, as illustrated inFIG.12, the information processing device10obtains, from the imaging DB13, for example, the video obtained by imaging a predetermined imaging range such as the inside of the store of the retail store with the camera device200(step S101). Note that the video imaged by the camera device200is transmitted from the camera device200to the information processing device10as needed, and stored in the imaging DB13.

Next, the information processing device10inputs the video obtained in step S101to, for example, the product/customer relationship specifying model, thereby specifying an area including a product, an area including a customer, and a relationship between the product and the customer from the video (step S102). Note that the area including the product or the customer may be, for example, a Bbox that rectangularly surrounds the product or the customer in the video. Furthermore, the relationship between the product and the customer may be, for example, an action of the customer such as “short-time viewing” or “long-time viewing” of the product, “touching” the product, “grasping” the product, “sitting” on the product, or the like.

Next, the information processing device10generates attention information indicating a degree of attention of a location where the product is arranged based on, for example, the relationship between the product and the customer specified in step S102(step S103). Examples of the attention information generated in step S103may include information stored in the attention DB18as illustrated inFIG.7.

Next, the information processing device10causes the display device to output, for example, the attention information generated in step S103(step S104). After the execution of step S104, the attention information output process illustrated inFIG.12is terminated.

Then, by analyzing the attention information output to the display device, the store staff or the like is enabled to analyze, for example, whether the purchase quantity actually changes depending on a location of a shelf or the like where the product is arranged and where the customer has presented a predetermined relationship, such as grasping the product. In other words, the store staff or the like is enabled to analyze, for example, using the attention information, whether the purchase quantity of the product changes when the location where the product is arranged is changed even for the same product.

Effects

As described above, the information processing device10obtains the video obtained by imaging the inside of the store, analyzes the obtained image to specify, from the video, the first area including a product arranged in the store, the second area including a customer to purchase the product, and the relationship in which interaction between the product and the customer is identified, generates attention information indicating a degree of attention of a location where the product is arranged based on the specified relationship, and causes the display device to output the generated attention information.

In this manner, the information processing device10specifies the relationship between the product and the customer from the video obtained by imaging the inside of the store, and generates and outputs the attention information of the product arrangement location based on the specified relationship. As a result, it becomes possible to more accurately analyze the purchasing behavior of the customer due to a difference in the location where the product is arranged.

Furthermore, the information processing device10specifies a type of the relationship indicated by each customer at a predetermined location where the product is arranged in the store based on the specified relationship and the position of the customer when the specified relationship is indicated, counts the actual quantity for each type of the relationship indicated by the customer at the predetermined location, and causes the display device to output the attention information in which the counted actual quantity is associated with the predetermined location.

As a result, it becomes possible to more accurately analyze the purchasing behavior of the customer due to a difference in the location where the product is arranged.

Furthermore, the information processing device10specifies the relationship indicating that the customer has grasped the product for each location based on the specified relationship and the position of the customer when the specified relationship is indicated, counts the actual quantity of the relationship indicating that the customer has grasped the product for each location, and causes the display device to output the attention information in which the counted actual quantity is associated with each location.

As a result, it becomes possible to more accurately analyze the purchasing behavior of the customer due to a difference in the location where the product is arranged.

Furthermore, the information processing device10specifies a location in the store where the customer has grasped the product based on the predetermined relationship and the position of the customer where the predetermined relationship is indicated, tracks the customer included in the video, analyzes the video to identify the accounting machine used by the tracked customer, specifies the purchase quantity of the products arranged at the location in the store where the customer has indicated the predetermined relationship based on the registration information obtained from the identified accounting machine, and causes the display device to output the attention information in which the purchase quantity is associated with each location in the store where the products are arranged.

As a result, it becomes possible to more accurately analyze the purchasing behavior of the customer due to a difference in the location where the product is arranged.

Furthermore, the process of specifying the first area, the second area, and the relationship performed by the information processing device10includes a process of specifying the first area, the second area, and the relationship by inputting the obtained video to a machine learning model, and the machine learning model is a model for the HOID trained to identify the information regarding the first area indicating the first class indicating an object including a product and the area where the object appears, the information regarding the second area indicating the second class indicating a person who purchases the product and the area where the person appears, and the interaction between the first class and the second class.

As a result, it becomes possible to more accurately analyze the purchasing behavior of the customer due to a difference in the location where the product is arranged.

Pieces of information including the processing procedures, control procedures, specific names, various kinds of data and parameters described above or illustrated in the drawings may be changed as appropriate, unless otherwise specified. Furthermore, the specific examples, distribution, numerical values, and the like described in the embodiment are merely examples, and may be changed as appropriate.

Furthermore, specific forms of distribution and integration of components of individual devices are not limited to those illustrated in the drawings. That is, all or some of the components may be functionally or physically distributed or integrated in optional units depending on various kinds of loads, use situations, or the like. Moreover, all or some of the individual processing functions of each device may be implemented by a central processing unit (CPU) and a program to be analyzed and executed by the CPU, or may be implemented as hardware by wired logic.

FIG.13is a diagram for explaining an exemplary hardware configuration of the information processing device10. As illustrated inFIG.13, the information processing device10includes a communication device10a, a hard disk drive (HDD)10b, a memory10c, and a processor10d. Furthermore, the individual units illustrated inFIG.13are mutually coupled by a bus or the like.

The communication device10ais a network interface card or the like, and communicates with another information processing device. The HDD10bstores programs and DBs for activating the functions illustrated inFIG.2.

The processor10dis a hardware circuit that reads, from the HDD10bor the like, a program for performing processing similar to that of each processing unit illustrated inFIG.2and loads it into the memory10cto activate a process that implements each function described with reference toFIG.2and the like. In other words, this process implements a function similar to that of each processing unit included in the information processing device10. Specifically, the processor10dreads a program having functions similar to those of the acquisition unit21, the specifying unit22, the generation unit23, the output unit24, and the like from the HDD10bor the like. Then, the processor10dexecutes a process for executing processing similar to that of the specifying unit22or the like.

In this manner, the information processing device10operates as an information processing device that executes operation control processing by reading and executing the program for performing processing similar to that of each processing unit illustrated inFIG.2. Furthermore, the information processing device10may also implement functions similar to those of the examples described above by reading a program from a recording medium using a medium reading device and executing the read program. Note that the program referred to in another example is not limited to being executed by the information processing device10. For example, the present embodiment may also be applied to a case where another information processing device executes the program or a case where the information processing device10and the another information processing device cooperate to execute the program.

Furthermore, the program for performing processing similar to that of each processing unit illustrated inFIG.2may be distributed via a network such as the Internet. Furthermore, this program may be recorded in a computer-readable recording medium such as a hard disk, a flexible disk (FD), a compact disc read only memory (CD-ROM), a magneto-optical disk (MO), a digital versatile disc (DVD), or the like, and may be executed by being read from the recording medium by a computer.

FIG.14is a diagram for explaining an exemplary hardware configuration of the accounting machine100. As illustrated inFIG.14, the accounting machine100includes a communication device100a, an HDD100b, a memory100c, a processor100d, an input device100e, and an output device100f. Furthermore, the individual units illustrated inFIG.14are mutually coupled by a bus or the like.

The communication device100ais a network interface card or the like, and communicates with another information processing device. The HDD100bstores programs and data for activating the individual functions of the accounting machine100.

The processor100dis a hardware circuit that reads, from the HDD100bor the like, a program for performing processing of each function of the accounting machine100and loads it into the memory100cto activate a process that implements each function of the accounting machine100. In other words, this process implements a function similar to that of each processing unit included in the accounting machine100.

In this manner, the accounting machine100operates as an information processing device that executes operation control processing by reading and executing the program that executes processing of each function of the accounting machine100. Furthermore, the accounting machine100may also implement each function of the accounting machine100by reading a program from a recording medium using a medium reading device and executing the read program. Note that the program referred to in another example is not limited to being executed by the accounting machine100. For example, the present embodiment may also be applied to a case where another information processing device executes the program or a case where the accounting machine100and the another information processing device cooperate to execute the program.

Furthermore, the program that executes the processing of each function of the accounting machine100may be distributed via a network such as the Internet. Furthermore, this program may be recorded in a computer-readable recording medium such as a hard disk, an FD, a CD-ROM, an MO, a DVD, or the like, and may be executed by being read from the recording medium by a computer.

The input device100edetects various input operations made by a user, such as an input operation for the program executed by the processor100d. The input operation includes, for example, a touch operation or the like. In the case of a touch operation, the accounting machine100may further include a display device, and the input operation detected by the input device100emay be a touch operation performed on the display device. The input device100emay be, for example, a button, a touch panel, a proximity sensor, and the like.

The output device100foutputs data output from the program executed by the processor100dvia an external device coupled to the accounting machine100, for example, an external display device or the like. Note that the accounting machine100may not include the output device100fin the case where the accounting machine100includes the display device.