Patent ID: 12211313

DESCRIPTION OF EMBODIMENTS

Example embodiments of the present invention will be described below by using drawings. Note that, in every drawing, similar components are given similar signs, and description thereof is omitted as appropriate.

FIG.1is a diagram illustrating a use environment of a customer analysis apparatus10according to an example embodiment.FIG.2is a diagram for illustrating an image capture apparatus200. The customer analysis apparatus10determines a product50placed on a product shelf40or a product50taken out from the product shelf40, by processing an image generated by the image capture apparatus200. The product shelf40is an example of a product placement area. For example, the product shelf40is placed in a store and includes at least one shelf. A plurality of types of products50are placed on a shelf.

The image capture apparatus200captures an image of at least one of a shelf in the product shelf40and an area in front of the shelf. In the example illustrated inFIG.2, the image capture apparatus200captures an image of both a shelf in the product shelf40and an area in front of the shelf. The image capture apparatus200includes two image capturing units210. One image capturing unit210includes a lighting unit220and a first image capture unit22, and the other image capturing unit210includes a lighting unit220and a second image capture unit24.

A light emitting surface of the lighting unit220extends in one direction and includes a light emitting unit and a cover for covering the light emitting unit. The lighting unit220mainly emits light in a direction orthogonal to the extending direction of the light emitting surface. The light emitting unit includes a light emitting device such as an LED and emits light in a direction not being covered by the cover. Note that, when the light emitting device is an LED, a plurality of LEDs are arranged in a direction in which the lighting unit220extends (a vertical direction in the diagram).

Each of the first image capture unit22and the second image capture unit24is provided at one end of a lighting unit220and has a direction in which light of the lighting unit220is emitted as an image capture area. For example, in an image capturing unit210on the left side inFIG.1andFIG.2, the first image capture unit22has a lower area and a diagonally lower right area as an image capture area. Further, in an image capturing unit210on the right side inFIG.1andFIG.2, the second image capture unit24has an upper area and a diagonally upper left area as an image capture area.

As illustrated inFIG.2, the two image capturing units210are mounted on front frames42(or at the front of side walls on both sides) of the product shelf40. At this time, the image capturing unit210including the first image capture unit22is mounted on one front frame42and is oriented in such a way that the first image capture unit22is positioned in the upper part, and the image capturing unit210including the second image capture unit24is mounted on a front frame42on the opposite side of the image capturing unit210including the first image capture unit22and is oriented in such a way that the second image capture unit24is positioned in the lower part. Therefore, in a direction in which the shelf extends (an example of a first direction), the first image capture unit22, the product shelf40, and the second image capture unit24are arranged in this order. Further, the first image capture unit22is positioned above the shelf in the product shelf40, and the second image capture unit24is positioned below the shelf in the product shelf40. Thus, a space between the first image capture unit22and the second image capture unit24is widened.

Then, the first image capture unit22captures an image of a lower area and a diagonally lower area in such a way that the image capture area includes an opening of the product shelf40and an area in front of the opening. On the other hand, the second image capture unit24captures an image of an upper area and a diagonally upper area in such as way that the image capture area includes the opening of the product shelf40and the area in front of the opening. Thus using two image capturing units210enables image capture of the entire area including the opening of the product shelf40and the area in front of the opening. Therefore, the customer analysis apparatus10can determine the product name of a product50taken out from the product shelf40, by processing an image generated by the first image capture unit22(hereinafter described as a first image) and an image generated by the second image capture unit24(hereinafter described as a second image).

Further, a customer taking out the product50is also captured in at least the first image. Therefore, the customer analysis apparatus10can generate information indicating an attribute of the customer (hereinafter described as customer attribute information), by processing at least the first image. Attribute information of a customer includes an age, a gender, and a type and an attribute of wearing.

Then, the customer analysis apparatus10causes a result storage unit130illustrated inFIG.3to store information indicating the product name of the determined product50. Further, the customer analysis apparatus10determines display information to be displayed by a display apparatus30, based on the attribute information of the customer. For example, the display apparatus30is provided in the store in which the product shelf40is provided. The display apparatus30may be installed on the product shelf40or in the neighborhood thereof. An example of the display information is advertisement information.

FIG.3is a block diagram illustrating an example of a functional configuration of the customer analysis apparatus10. The customer analysis apparatus10includes an image acquisition unit110and an image processing unit120. The image acquisition unit110acquires a first image generated by the first image capture unit22and a second image generated by the second image capture unit24. The image processing unit120processes a first image and a second image. More specifically, the image processing unit120generates product information indicating a product50taken out from the product shelf40, by processing a first image and a second image. Further, the image processing unit120generates customer attribute information of a customer taking out the product50, by processing the first image. Then, the image processing unit120causes the result storage unit130to store the product information and the customer attribute information in association with each other. Specific examples of the customer attribute information will be described later.

The image processing unit120also uses a second image as needed when generating customer attribute information. Examples of customer attribute information generated by using a second image include attribute information related to a customer being short in height such as a child. Specific examples of the attribute information will also be described later.

Further, the image processing unit120determines whether a product50taken out by a customer is returned to the product shelf40, by processing a first image and a second image. Thus, whether the product50is purchased by the customer can be determined.

When processing a first image and a second image, the image processing unit120uses information stored in a processing method storage unit122. For example, the processing method storage unit122stores information required for determining the product name of a product50and information required for generating customer attribute information. While examples of the information described above include a feature value, a model generated by machine learning may also be included.

The customer analysis apparatus10further includes an output unit140. The output unit140is used with a display information storage unit142. The display information storage unit142stores a plurality of pieces of display information to be displayed by the display apparatus30, such as advertisement information, in association with customer attribute information related to the display information. Then, by using customer attribute information generated by the image processing unit120, the output unit140acquires display information to be displayed by the display apparatus30from the display information storage unit142and causes the display apparatus30to display the acquired display information.

Note that the processing method storage unit122, the result storage unit130, and the display information storage unit142are part of the customer analysis apparatus10in the example illustrated in this diagram. However, at least part of the units may be positioned outside the customer analysis apparatus10.

FIG.4is a diagram illustrating, in a table format, information stored in the result storage unit130. In the example illustrated in this diagram, for each act of a customer taking out a product50from the product shelf40, the result storage unit130stores a date and time when the act is performed, product information of the product50, customer attribute information of the customer, and purchase information indicating whether the product50is purchased by the customer.

FIG.5is a diagram illustrating a hardware configuration example of the customer analysis apparatus10. The customer analysis apparatus10includes a bus1010, a processor1020, a memory1030, a storage device1040, an input/output interface1050, and a network interface1060.

The bus1010is a data transmission channel for the processor1020, the memory1030, the storage device1040, the input/output interface1050, and the network interface1060to transmit and receive data to and from one another. Note that the method of interconnecting the processor1020and other components is not limited to a bus connection.

The processor1020is a processor provided by a central processing unit (CPU), a graphics processing unit (GPU), or the like.

The memory1030is a main storage provided by a random access memory (RAM) or the like.

The storage device1040is an auxiliary storage provided by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device1040stores program modules providing the functions of the customer analysis apparatus10(such as the image acquisition unit110, the image processing unit120, and the output unit140). By reading each program module into the memory1030and executing the program module by the processor1020, each function related to the program module is provided. Further, the storage device1040also functions as the processing method storage unit122, the result storage unit130, and the display information storage unit142.

The input/output interface1050is an interface for connecting the customer analysis apparatus10to various types of input/output equipment. For example, the customer analysis apparatus10communicates with the customer analysis apparatus10through the input/output interface1050.

The network interface1060is an interface for connecting the customer analysis apparatus10to a network. Examples of the network include a local area network (LAN) and a wide area network (WAN). The method of connecting the network interface1060to the network may be a wireless connection or a wired connection. The customer analysis apparatus10may communicate with the customer analysis apparatus10through the network interface1060.

FIG.5is a flowchart illustrating an example of processing performed by the customer analysis apparatus10. The first image capture unit22and the second image capture unit24repeatedly generate images. While any frame rate is applicable at this time, for example, the frame rate is equal to or greater than 24 frames/sec and equal to or less than 120 frames/sec. Further, a timing at which the first image capture unit22generates a first image is synchronized with a timing at which the second image capture unit24generates a second image. Every time the first image capture unit22and the second image capture unit24generate images, the image capture apparatus200transmits the generated first image and the generated second image to the customer analysis apparatus10. Note that intervals at which first images and second images are transmitted by the image capture apparatus200may be longer than the frame rates of the first image capture unit22and the second image capture unit24. In this case, the image capture apparatus200collectively transmits a plurality of first images and a plurality of second images to the customer analysis apparatus10. Then, every time a first image and a second image are transmitted, the customer analysis apparatus10performs the processing illustrated in this diagram.

First, the image acquisition unit110in the customer analysis apparatus10acquires a first image and a second image (Step S10). Next, the image processing unit120generates product information of a product50taken out by a customer, by processing the first image and the second image. Further, the image processing unit120generates customer attribute information by processing at least the first image (Step S20).

Further, the image processing unit120generates clothing information indicating clothing worn by the customer, by processing the first image, and makes the clothing information at least part of the customer attribute information. The image processing unit120may at times generate the clothing information by using both the first image and the second image. Examples of clothing indicated by the clothing information include business attire, sportswear, a winter scarf, and mourning.

Further, shoes of the customer are highly likely to be captured in the first image. Then, the image processing unit120generates shoe information indicating the shoes worn by the customer, by processing the first image, and makes the shoe information at least part of the customer attribute information. For example, the shoe information indicates the type of the shoes. Examples of the shoe type include business shoes, sports shoes, boots, high heels, pumps, loafers, and sandals. Further, the shoe information may indicate a state of the shoes. Examples of the shoe state include whether the shoes are dry or wet.

Note that, as described above, the image processing unit120may use a second image when generating customer attribute information. For example, the image processing unit120generates attribute information of a child being short in height by using a second image. In this case, the image processing unit120may also determine whether an adult customer is accompanied by a child by using a processing result of the first image and a processing result of the second image. For example, when an adult is captured in a first image and a child is captured in a second image generated at the same timing as the first image, the image processing unit120determines that the adult customer is accompanied by the child. When an adult customer visits a store with a child, at least part of attribute information of the customer becomes information indicating that the customer is accompanied by a child and attribute information of the child.

Further, the image processing unit120determines the type of a wearing article worn by the customer over the face or on the head, by processing the second image, and makes information indicating the wearing article (hereinafter described as wearing article information) at least part of the customer attribute information. For example, the wearing article information may indicate a mask, sunglasses or glasses, headwear, or a hair band.

Then, the image processing unit120causes the result storage unit130to store the generated information (Step S30).

Further, the output unit140reads display information related to the customer attribute information generated by the image processing unit120from the display information storage unit142and causes the display apparatus30to display the read display information (Step S40).

When the customer attribute information includes shoe information, the output unit140reads display information related to the shoe information from the display information storage unit142.

For example, when the shoe information indicates sneakers or running shoes, the output unit140reads at least one item out of advertisement information of a sports drink, advertisement information of protein, and advertisement information of a towel from the display information storage unit142. When the shoe information indicates high heels or pumps, the output unit140reads at least one item out of advertisement information of stockings and advertisement information of an adhesive bandage from the display information storage unit142. When the shoe information indicates boots or business shoes, the output unit140reads advertisement information of socks from the display information storage unit142. When the shoe information indicates that the shoes are wet, the output unit140reads at least one item out of advertisement information of socks and advertisement information of a towel from the display information storage unit142.

Further, when the customer attribute information includes clothing information of the customer, the output unit140reads display information related to the clothing information from the display information storage unit142.

For example, when the clothing information indicates gloves, tights, or a winter scarf, the output unit140reads at least one item out of advertisement information of a warm drink and advertisement information of a pocket body warmer from the display information storage unit142. When the clothing information is related to sports such as sportswear, the output unit140reads at least one item out of advertisement information of a sports drink, advertisement information of protein, and advertisement information of a towel from the display information storage unit142.

Further, when the clothing information indicates mourning, the output unit140reads advertisement information of a gift envelope from the display information storage unit142.

Furthermore, when the customer attribute information includes wearing article information, the output unit140reads display information related to the wearing article information from the display information storage unit142.

For example, when the wearing article information indicates a mask, the output unit140reads at least one item out of advertisement information of a drug for hay fever and/or a cold, advertisement information of a mask, advertisement information of an antiseptic, advertisement information of a warm drink, and advertisement information of a throat lozenge. When the wearing article information indicates a towel, the output unit140reads advertisement information of a sports drink from the display information storage unit142.

Further, when the customer attribute information indicates a customer accompanied by a child, the output unit140reads advertisement information of snacks from the display information storage unit142.

Note that, for example, information stored in the result storage unit130by the image processing unit120is statistically processed by an external information processing apparatus. Thus, a product50placed on the product shelf40can be determined, or a product50and/or a package of the product50can be determined.

As described above, the customer analysis apparatus10according to the present example embodiment is used with the first image capture unit22and the second image capture unit24. The first image capture unit22, the product shelf40, and the second image capture unit24are arranged in this order in the first direction. The first image capture unit22is positioned at a height identical to that of or above the product shelf40and generates a first image by capturing an image of at least a diagonally lower area. The second image capture unit24is positioned at a height identical to that of or below the product shelf40and below the first image capture unit22and generates a second image by capturing an image of at least a diagonally upper area. Then, the image processing unit120in the customer analysis apparatus10generates product information indicating a product50taken out from the product shelf40, by processing the first image and the second image. Further, the image processing unit120generates customer attribute information indicating an attribute of a customer taking out the product50, by processing the first image. Then, the image processing unit120causes the result storage unit130to store the product information and the customer attribute information in association with each other. Accordingly, product information and customer attribute information can be generated with a high degree of precision without increasing the image capture unit.

While the example embodiments of the present invention have been described above with reference to the drawings, the example embodiments are exemplifications of the present invention, and various configurations other than those described above may be employed.

Further, while a plurality of processes (processing) are described in a sequential order in each of a plurality of flowcharts used in the aforementioned description, the execution order of processes executed in each example embodiment is not limited to the order of description. The order of the illustrated processes may be modified without affecting the contents in each example embodiment. Further, the aforementioned example embodiments may be combined without contradicting one another.

The whole or part of the example embodiments described above may be described as, but not limited to, the following supplementary notes.1. A customer analysis apparatus used with a first image capture unit and a second image capture unit each capturing an image of at least one of a product placement location and an area in front of the product placement location, the customer analysis apparatus including:an image acquisition unit for acquiring a first image generated by the first image capture unit and a second image generated by the second image capture unit; andan image processing unit for processing the first image and the second image, whereinthe first image capture unit, the product placement location, and the second image capture unit are arranged in this order in a first direction,the first image capture unit is positioned at a height identical to that of or above the product placement location and captures an image of at least a diagonally lower area,the second image capture unit is positioned at a height identical to that of or below the product placement location and below the first image capture unit and captures an image of atleast a diagonally upper area, andthe image processing unitgenerates product information indicating a product taken out from the product placement location, by processing the first image and the second image,generates customer attribute information indicating an attribute of a customer taking out the product, by processing the first image, andcauses a storage unit to store the product information and the customer attribute information in association with each other.2. The customer analysis apparatus according to aforementioned 1, whereinthe image processing unit generates shoe information indicating a shoe worn by the customer, by processing the first image, and makes the shoe information at least part of the customer attribute information.3. The customer analysis apparatus according to aforementioned 1 or 2, whereinthe image processing unit generates clothing information indicating clothing worn by the customer, by processing the first image, and makes the clothing information at least part of the customer attribute information.4. The customer analysis apparatus according to any one of aforementioned 1 to 3, whereinthe image processing unit makes wearing article information indicating a wearing article worn over a face of the customer at least part of the customer attribute information, by further processing the second image.5. The customer analysis apparatus according to any one of aforementioned 1 to 4, whereinthe image processing unit generates child attribute information indicating an attribute of a child accompanying the customer, by further processing the second image, and makes the child attribute information at least part of the customer attribute information.6. The customer analysis apparatus according to any one of aforementioned 1 to 5, whereina display unit is provided at the product placement location or a store in which the product placement location is provided, andthe customer analysis apparatus further includesan information selection unit for acquiring display information to be displayed by the display unit, by using the customer attribute information, and causing the display unit to display the acquired display information.7. A customer analysis method performed by a computer used with a first image capture unit and a second image capture unit each capturing an image of at least one of a product placement location and an area in front of the product placement location, the customer analysis method including, by the computer:acquiring a first image generated by the first image capture unit and a second image generated by the second image capture unit;generating product information indicating a product taken out from the product placement location, by processing the first image and the second image;generating customer attribute information indicating an attribute of a customer taking out the product, by processing the first image; andcausing a storage unit to store the product information and the customer attribute information in association with each other, whereinthe first image capture unit, the product placement location, and the second image capture unit are arranged in this order in a first direction,the first image capture unit is positioned at a height identical to that of or above the product placement location and captures an image of at least a diagonally lower area, andthe second image capture unit is positioned at a height identical to that of or below the product placement location and below the first image capture unit and captures an image of at least a diagonally upper area.8. The customer analysis method according to aforementioned 7, further including, by the computer,generating shoe information indicating a shoe worn by the customer, by processing the first image, and making the shoe information at least part of the customer attribute information.9. The customer analysis method according to aforementioned 7 or 8, further including, by the computer,generating clothing information indicating clothing worn by the customer, by processing the first image, and making the clothing information at least part of the customer attribute information.10. The customer analysis method according to any one of aforementioned 7 to 9, further including, by the computer,generating wearing article information indicating a wearing article worn over a face of the customer, by further processing the second image, and making the wearing article information at least part of the customer attribute information.11. The customer analysis method according to any one of aforementioned 7 to 10, further including, by the computer,generating child attribute information indicating an attribute of a child accompanying the customer, by further processing the second image, and making the child attribute information at least part of the customer attribute information.12. The customer analysis method according to any one of aforementioned 7 to 11, whereina display unit is provided at the product placement location or a store in which the product placement location is provided, andthe customer analysis method further includes, by the computer,acquiring display information to be displayed by the display unit, by using the customer attribute information, and causing the display unit to display the acquired display information.13. A program executed by a computer used with a first image capture unit and a second image capture unit each capturing an image of at least one of a product placement location and an area in front of the product placement location, the program causing the computer to execute:a function of acquiring a first image generated by the first image capture unit and a second image generated by the second image capture unit;a function of generating product information indicating a product taken out from the product placement location, by processing the first image and the second image;a function of generating customer attribute information indicating an attribute of a customer taking out the product, by processing the first image; anda function of causing a storage unit to store the product information and the customer attribute information in association with each other, whereinthe first image capture unit, the product placement location, and the second image capture unit are arranged in this order in a first direction,the first image capture unit is positioned at a height identical to that of or above the product placement location and captures an image of at least a diagonally lower area, andthe second image capture unit is positioned at a height identical to that of or below the product placement location and below the first image capture unit and captures an image of at least a diagonally upper area.14. The program according to aforementioned 13, further causing the computer to generate shoe information indicating a shoe worn by the customer, by processing the first image, and make the shoe information at least part of the customer attribute information.15. The program according to aforementioned 13 or 14, further causing the computer togenerate clothing information indicating clothing worn by the customer, by processing the first image, and make the clothing information at least part of the customer attribute information.16. The program according to any one of aforementioned 13 to 15, further causing the computer togenerate wearing article information indicating a wearing article worn over a face of the customer, by further processing the second image, and make the wearing article information at least part of the customer attribute information.17. The program according to any one of aforementioned 13 to 16, further causing the computer togenerate child attribute information indicating an attribute of a child accompanying the customer, by further processing the second image, and make the child attribute information at least part of the customer attribute information.18. The program according to any one of aforementioned 13 to 17, whereina display unit is provided at the product placement location or a store in which the product placement location is provided, andthe program further causes the computer to executea function of acquiring display information to be displayed by the display unit, by using the customer attribute information, and causing the display unit to display the acquired display information.

REFERENCE SIGNS LIST

10Customer analysis apparatus22First image capture unit24Second image capture unit30Display apparatus40Product shelf42Front frame50Product110Image acquisition unit120Image processing unit122Processing method storage unit130Result storage unit140Output unit142Display information storage unit200Image capture apparatus210Image capturing unit220Lighting unit