SHELF-ALLOCATION INFORMATION GENERATING DEVICE AND SHELF-ALLOCATION INFORMATION GENERATING PROGRAM

A planogram information generating device includes: an image acquiring unit that acquires an image acquired by imaging product display shelves; a detection unit that detects product area images representing the products from the image of the product display shelves; a product recognizing unit that recognizes products represented by the product area images on the basis of information relating to images of products stored in advance; a determination unit that determines validity of recognition of one product area image as a first product on the basis of relevancy information between the first product recognized as a product represented by the one product area image and one or more second products recognized as products represented by one or more other product area images other than the one product area image; and a correction unit that corrects the first product recognized as the product represented by the one product area image.

TECHNICAL FIELD

The present invention relates to a planogram information generating device and a planogram information generating program.

BACKGROUND ART

A technique in which products are recognized from an image acquired by imaging product display shelves and information on products arranged on each shelf known as planogram information is acquired is performed. For example, a technology for generating planogram information representing a product display state on the basis of a product recognized from an image including a product display shelf and information relating to a designated position of the product is known (for example, see Patent Literature 1).

CITATION LIST

Patent Literature

SUMMARY OF INVENTION

Technical Problem

However, in a conventional technology of generating planogram information, due to difficulty in distinguishably identifying products of the same series with very similar external views, products with different sizes, and the like through image recognition, the accuracy of recognition of a single product through image recognition is not sufficient. For this reason, in the conventional technology, it is difficult to acquire arrangement of products with high accuracy in the generation of planogram information, and it is desired to correctly recognize products arranged on a product display shelf.

The present invention is realized in consideration of the problems described above, and an object thereof is to provide a planogram information generating device and a planogram information generating program capable of easily performing improvement of accuracy of recognition of products through image recognition for an image of products included in an image of product display shelves and checking and correction of a result of the recognition in generation of planogram information that is arrangement information of products arranged on a product display shelf.

Solution to Problem

In order to solve the problems described above, according to one embodiment of the present invention, there is provided a planogram information generating device that generates planogram information that is arrangement information of products arranged on product display shelves, the planogram information generating device including: an image acquiring unit that acquires an image acquired by imaging product display shelves on which a plurality of products are arranged; a detection unit that detects product area images representing the products from the image acquired by the image acquiring unit; a product recognizing unit that recognizes products represented by the product area images detected by the detection unit on the basis of information relating to images of products stored in advance; a determination unit that determines validity of recognition of one product area image as a first product on the basis of relevancy information between the first product recognized as a product represented by the one product area image and one or more second products recognized as products represented by one or more other product area images other than the one product area image; and a correction unit that corrects the first product recognized as the product represented by the one product area image by the product recognizing unit on the basis of information relating to the validity determined by the determination unit.

In addition, a planogram information generating program according to one embodiment of the present invention is a planogram information generating program causing a computer to function as a planogram information generating device that generates planogram information that is arrangement information of products arranged on product display shelves, the program causing the computer to realize: an image acquiring function of acquiring an image acquired by imaging product display shelves on which a plurality of products are arranged; a detection function of detecting product area images representing the products from the image acquired by the image acquiring function; a product recognizing function of recognizing products represented by the product area images detected by the detection function on the basis of information relating to images of products stored in advance; a determination function of determining validity of recognition of one product area image as a first product on the basis of relevancy information between the first product recognized as a product represented by the one product area image and one or more second products recognized as products represented by one or more other product area images other than the one product area image; and a correction function of correcting the first product recognized as the product represented by the one product area image by the product recognizing function on the basis of information relating to the validity determined by the determination function.

There is a feature in that the same products and products of the same category, the same manufacturer, and the same series/brand are arranged adjacently or closely in a product display shelf. In other words, one product arranged on a product display shelf has relevancy with other products adjacent to the one product and other products arranged within a predetermined distance from the one product. According to the embodiment described above, product area images are detected from an image of product display shelves, a product represented by each of the product area images is recognized, and validity of recognition of one product area image as a first product is determined on the basis of relevancy information between the first product recognized as a product represented by the one product area image and second products recognized as products represented by other product area images. Then, the first product recognized as the product represented by one product area image is corrected on the basis of information relating to the determined validity. In this way, a result of recognition of a product for a product area image is easily corrected, and the accuracy of the recognition can be improved.

Advantageous Effects of Invention

A planogram information generating device and a planogram information generating program capable of easily performing improvement of accuracy of recognition of products through image recognition for an image of products included in an image of product display shelves and checking and correction of a result of the recognition in generation of planogram information that is arrangement information of products arranged on a product display shelf can be provided.

DESCRIPTION OF EMBODIMENTS

A planogram information generating device according to an embodiment of the present invention will be described with reference to the drawings. If applicable, the same reference signs will be assigned to the same parts, and duplicate description thereof will be omitted.

FIG. 1is a diagram illustrating the functional configuration of a planogram information generating system1including a planogram information generating device10according to this embodiment. The planogram information generating device10is a device that generates planogram information that is arrangement information of products arranged on a product display shelf and includes a configuration for easily performing checking and correction of a result of recognition of an image of products of product display shelves.

As illustrated inFIG. 1, the planogram information generating system1includes the planogram information generating device10, an imaging/display device20, a product data storing unit30, a learning model storing unit40, and a recognition result storing unit50.

The planogram information generating system1may be configured as one device, or one or a plurality of planogram information generating devices10, imaging/display devices20, product data storing units30, learning model storing units40, and recognition result storing units50may configure respective devices. As one example, the imaging/display device20is configured as one terminal, and the planogram information generating device10, the product data storing unit30, the learning model storing unit40, and the recognition result storing unit50may be configured by a server. In addition, as another example, the planogram information generating device10and the imaging/display device20may be configured as one terminal. Each of the product data storing unit30, the learning model storing unit40, and the recognition result storing unit50may be configured by devices of any aspect as long as the devices are configured to be accessible from the planogram information generating device10. In a case in which the imaging/display device20is configured as one terminal, an imaging person can check whether there is an error in the planogram information generated through image recognition by referring to a display screen while imaging product display shelves. In addition, in the example of a terminal configured by only a display (display device)22, an operator or the like is assumed to perform an operation of checking and correcting planogram information generated through image recognition at an office or the like rather than an actual store.

In this embodiment, a terminal configuring the imaging/display device20or a terminal configuring the planogram information generating device10and the imaging/display device20, for example, is configured as a mobile terminal such as a high-function cellular phone (smartphone) or a cellular phone.

As illustrated inFIG. 1, the planogram information generating device10functionally includes an image acquiring unit11, a detection unit12, a product recognizing unit13, a planogram analyzing unit14, a determination unit15, a display unit16, a correction unit17, and a generation unit18. In addition, the imaging/display device20includes a camera21as an imaging device and a display22as a display device. Such functional units will be described later in detail.

The block diagram illustrated inFIG. 1illustrates blocks in units of functions. Such functional blocks (components) are realized by an arbitrary combination of hardware and/or software. In addition, a means that realizes each functional block is not particularly limited. In other words, each functional block may be realized by one device that is combined physically and/or logically or may be realized by directly and/or indirectly (for example, in a wired manner and/or a wireless manner) connecting two or more devices that are separated physically and/or logically and using the plurality of devices.

For example, the planogram information generating device10according to one embodiment of the present invention may function as a computer.FIG. 2is a diagram illustrating one example of the hardware configuration of the planogram information generating device10according to this embodiment. The planogram information generating device10may be physically configured as a computer device including a processor1001, a memory1002, a storage1003, a communication device1004, an input device1005, an output device1006, a bus1007, and the like.

In addition, in the following description, the term “device” may also refer to a circuit, a device, a unit, or the like. The hardware configuration of the planogram information generating device10may be configured to include one or a plurality of devices illustrated inFIG. 2or may be configured not to include some of the devices.

Each function of the planogram information generating device10is realized by the processor1001performing an arithmetic operation and controlling communication using the communication device1004and data reading and/or writing for the memory1002and the storage1003by causing the processor1001to read predetermined software (a program) onto hardware such as the memory1002or the like.

The processor1001, for example, controls the entire computer by operating an operating system. The processor1001may be configured by a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic operation device, a register, and the like. In addition, the processor1001may be configured to include a graphics processing unit (GPU). For example, functional units11to18illustrated inFIG. 1and the like may be realized by the processor1001.

In addition, the processor1001reads a program (program code), a software module, and data from the storage1003and/or the communication device1004into the memory1002and executes various processes in accordance with this. As the program, a program causing the computer to execute at least some of the operations described in the embodiment described above is used. For example, the functional units11to13of the planogram information generating device10may be realized by a control program that is stored in the memory1002and is operated by the processor1001. While various processes described above have been described as being executed by one processor1001, the processes may be executed by two or more processors1001simultaneously or sequentially. The processor1001may be realized using one or more chips. In addition, the program may be transmitted from a network through a telecommunication line.

The memory1002is a computer-readable recording medium and, for example, may be configured by at least one of a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), and the like. The memory1002may be referred to as a register, a cache, a main memory (a main storage device), or the like. The memory1002can store a program (a program code), a software module, and the like that can be executed to perform a planogram information generating method according to one embodiment of the present invention.

The storage1003is a computer-readable recording medium and, for example, may be configured by at least one of an optical disc such as a compact disc ROM (CD-ROM), a hard disk drive, a flexible disk, a magneto-optical disc (for example, a compact disc, a digital versatile disc, or a Blue-ray (registered trademark) disc), a smart card, a flash memory (for example, a card, a stick, or a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. The storage1003may be referred to as an auxiliary storage device. The storage medium described above, for example, may be a database including the memory1002and/or storage1003, a server, or any other appropriate medium.

The communication device1004is hardware (a transmission/reception device) for performing inter-computer communication through a wired and/or wireless network and, for example, may also be called a network device, a network controller, a network card, a communication module, or the like.

The input device1005is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, or the like) accepting an input from the outside. The output device1006is an output device (for example, a display, a speaker, an LED lamp, or the like) performing output to the outside. In addition, the input device1005and the output device1006may have an integrated configuration (for example, a touch panel).

In addition, devices such as the processor1001, the memory1002, and the like are interconnected through a bus1007for communication of information. The bus1007may be configured as a single bus or may be configured using different buses for different devices.

In addition, the planogram information generating device10may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or the like, and a part or the whole of each functional block may be realized by hardware. For example, the processor1001may be realized using at least one of such hardware components.

Referring back toFIG. 1again, each functional unit of the planogram information generating device10will be described. The image acquiring unit11acquires an image acquired by imaging product display shelves on which a plurality of products are arranged.FIG. 3is a diagram illustrating an example of an image PM0of product display shelves acquired by the image acquiring unit11. More specifically, when product display shelves are imaged by the camera21, the image acquiring unit11acquires an image that is imaged by the camera21as an image of product display shelves. As illustrated inFIG. 3, the image PM0of the product display shelves includes product display shelves and a plurality of products arranged on each shelf.

The product display shelves and the products arranged on the product display shelves, as illustrated inFIG. 3as an example, have features as described below. In other words, there is a feature in that the same products are arranged adjacently or closely. Particularly, in a large store or the like, a plurality of units are arranged for each product, and thus, this feature markedly appears. On the other hand, in a medium/small store or the like, there are many cases in which one unit is arranged for each product, and this feature is unlikely to appear. The number of units arranged for each product is represented by the word “face.” In other words, the number of faces of the same product is large, and there are a sufficient arrangement space in a large store, and the number of faces of the same product is small in a small store.

In addition, there is a feature in that products belonging to the same series and products belonging to the same brand are arranged at positions close to each other on a product display shelf. In addition, there is a feature in that products belonging to the same manufacturer or products belonging to the same category are arranged at positions close to each other on product display shelves or are arranged on the same product display shelf.

In consideration of the features of product display shelves and arranged products described above, the planogram information generating device10according this embodiment determines the validity of a result of recognition of products relating to one product area image using results of recognition of other product area images positioned on the vicinity of the one product area image and products relating to the other product area images.

The detection unit12detects a product area image representing products from an image of product display shelves acquired by the image acquiring unit11. More specifically, the detection unit12, for example, recognizes each object extracted using a technique such as a known edge detection technique or the like for an image of product display shelves as a product area image representing products. In addition, the detection unit12, for example, has learned a shape of each product in advance using a technique of known deep learning or the like and detects a product area image representing products from the image of product display shelves using learned data. Furthermore, the shape data of products learned in advance may be stored in the learning model storing unit40. The learning model storing unit40will be described later in detail. The technique used for detection of a product area image from an image of product display shelves is not limited to the example described above, and any technique may be used as long as it enables detection of each product area image.

FIG. 4is a diagram illustrating detection of product area images from an image PM1of product display shelves. As illustrated inFIG. 4, the detection unit12detects a plurality of product area images each representing a product from the image PM1of product display shelves, and a grid line corresponding to an outer frame of each product area image is attached in a detected product area image.

The product recognizing unit13recognizes a product represented by a product area image detected by the detection unit12on the basis of information relating to images of products stored in advance. In this embodiment, the information relating to images of products used for recognition of products is stored in the product data storing unit30.

FIG. 5is a diagram schematically illustrates an example of product image data31stored in the product data storing unit30. As illustrated inFIG. 5, the product image data31stores a plurality of pieces of product image data mb1to mb8representing outer views of products in association with product IDs used for identifying products. The product image data mb1to mb8represents outer views of a product identified by a product ID: X in various directions. In addition, product image data stored in the product data storing unit30is not limited to the example illustrated inFIG. 5, and the product data storing unit30may include one piece of product image data for each product ID or may include image data representing outer views in a case in which a product is gradually rotated in a vertical direction in addition to image data representing outer views in a case in which the product is gradually rotated in a horizontal direction as illustrated in the example illustrated inFIG. 5. Furthermore, the product data storing unit30may include one of one piece of product image data representing a front outer view of a product, one piece of product image data representing a rear outer view of the product, and a plurality of pieces of product image data acquired by imaging the product in a plurality of directions in association with each product ID and may include a combination of such product image data in association with each product ID.

The product recognizing unit13collates product image data mb stored in the product data storing unit30with each product area image detected by the detection unit12using a known collation technology and accordingly can recognize a product represented by the product area image. In addition, recognition of a product is not limited to the collation technology described above and the like, and any technique may be used. For example, the product recognizing unit13may learn outer views of various products in advance using a technique of deep leaning or the like and recognize a product represented by each product area image.

In this embodiment, the product data storing unit30stores a product master32that includes various attributes of products.FIG. 6is a diagram illustrating the configuration of the product master32. As illustrated inFIG. 6, the product master32stores a product name, a size, a series/brand, a manufacturer, a category, and the like of a product in association with a product ID used for identifying the product. In addition, product attributes associated with each product are not limited to those of the example illustrated inFIG. 6, and more sub-divided attributes may be included. Since a product represented by the product area image is recognized by the product recognizing unit13, each product area image can be associated with attributes of a product by referring to the product master32using the product ID of the recognized product as a key.

The product recognizing unit13stores a result of recognition of a product for each product area image in the recognition result storing unit50. The recognition result storing unit50is a storage means that stores a product area image and a product recognized in relation to the product area image in association with each other.

The planogram analyzing unit14acquires planogram data that is information relating to arrangement of products on each of product display shelves on the basis of a result of recognition of a product acquired by the product recognizing unit13and a positional relation of a product area image in an image of product display shelves.

More specifically, first, the planogram analyzing unit14recognizes the position of a shelf board from an image of product display shelves. Since products are arranged on a shelf board, for example, the planogram analyzing unit14acquires a distribution of pixels, which represent a product area image for each coordinate of a coordinate axis extending in the vertical direction, of an image of product display shelves and acquires coordinate values at which the pixel distribution is smaller than a predetermined value, an area near a minimal point of the pixel distribution, and the like as positions of the shelf board in the vertical direction.

In addition, the planogram analyzing unit14may learn images of shelf boards, images of price tags attached to shelf boards, and the like in advance in addition to images representing products and acquire positions of shelf boards using a known collation technology or a technique of deep learning or the like.

Then, the planogram analyzing unit14acquires planogram data on the basis of a positional relation between the position of a shelf board acquired from the image of product display shelves and the position of a product area image detected by the detection unit and information of products represented by a product area image recognized by the product recognizing unit13.FIG. 7is a diagram illustrating an example of planogram data acquired by the planogram analyzing unit14. As illustrated inFIG. 7, the planogram data includes information of a serial number, a shelf board number, a shelf position, a product ID, the number of faces, and the number of stacking stages in association with each other. The serial number is information used for identifying a product display shelf. The shelf board number is information used for identifying a shelf board in a product display shelf and, for example, numbers are assigned from a lower stage to an upper stage. The shelf position is information used for identifying a position of one shelf board in the horizontal direction, and, for example, numbers are assigned from the left side to the right side in a product display shelf image. The product ID is information used for identifying a product. The number of faces is the number of arranged products of the same kind. The number of stacking stages is the number of products of the same kind stacked at a certain position on a shelf board.

The determination unit15determines validity of recognition of one product area image as a first product on the basis of relevancy information between the first product recognized as a product represented by one product area image and one or more second products recognized as products represented by one or more other product area images other than the one product area image.

An example of the relevancy information will be described with reference toFIG. 8.FIG. 8is a diagram illustrating an example of acquisition of relevancy information relating to one product area image. As illustrated inFIG. 8, an image PM2of a product display shelf includes product area images mp11to mp17of products arranged at shelf positions1to7on a certain shelf. A reference sign and represents information of attributes of recognized products in relation to the product area images mp11to mp17of products arranged at the shelf positions1to7. For example, a product recognized in relation to a product area image of a product arranged at the shelf position1has attributes of a product name: N11, a series/brand: B1, and a manufacturer C1.

Here, an example in which relevancy information for a product area image mp13of a product arranged at the shelf position3is acquired is illustrated. In this example, the determination unit15acquires relevancy information me representing relevance between a product (first product) recognized as a product represented by the product area image mp13and products (second products) recognized as products represented by product area images mp12and mp14adjacent to the product area image mp13on the basis of attribute information of each product represented by a reference sign md.

In the relevancy information me illustrated inFIG. 8, data representing same/difference in the product name, the series/brand, and the manufacturer between products of the shelf positions2,3, and4(in the case of being different: 0, in the case of being the same: 1) are included as elements. More specifically, between products that are recognized as products represented by the product area image mp12of the shelf position2and the product area image mp13of the shelf position3, the product names are different, and the series/brands and the manufacturers are the same, and the determination unit15generates relevancy information me of which data representing such same/difference is “0, 1, 1”. Similarly, between products that are recognized as products represented by the product area image mp12of the shelf position2and the product area image mp14of the shelf position4, the product names, the series/brands, and the manufacturers are the same, and the determination unit15generates relevancy information me of which data representing such same/difference is “1, 1, 1”. In addition, between products that are recognized as products represented by the product area image mp13of the shelf position3and the product area image mp14of the shelf position4, the product names are different, and the series/brands and the manufacturers are the same, and the determination unit15generates relevancy information me of which data representing such same/difference is “0, 1, 1”.

In addition, although the same/difference of attributes such as the product names, the manufacturers, and the like are represented using binary values of “1” and “2” in the relevancy information illustrated inFIG. 8, this is merely one example, and, for example, the degree of coincidence of the attributes may be represented using a numerical value in the range of 0 to 1 or may be represented using any other range of numerical values.

In addition, in the example illustrated inFIG. 8, although products adjacent on both sides are focused when relevancy information relating to one product area image is generated, not only the products adjacent on both the sides (products of which the number of faces from one product area image is within “1”), products arranged within the number of faces that is two or more in each of the rightward direction and the leftward direction may be focused, and more product area images in the vicinity of one product area image may be included to be focused.

Other product area images focused for generating relevancy information relating to one product area image may include product area images present within a predetermined distance from one product area image in an image of a product display shelf.FIG. 9is a diagram illustrating another example of acquisition of relevancy information relating to one product area image.

As illustrated in an image PM3of a product display shelf illustrated inFIG. 9, when relevancy information relating to the product area image mp13is generated, the determination unit15may generate the relevancy information with product area images included in a range cr1within a predetermined distance from the product area image mp13focused. In addition, the determination unit15may generate relevancy information with product area images included in a range cr2within a distance that is further long from the product area image mp13focused. A distance from one product area image to another product area image may be counted using the number of pixels in an image of product display shelves or may be counted using the number of faces in a case in which planogram data is acquired.

Subsequently, the determination unit15generates a feature quantity relating to one product area image on the basis of relevancy information and determines validity of a result of recognition of a product relating to one product area image on the basis of the generated feature quantity.FIG. 10is a diagram illustrating a process of generation of a feature quantity and determination of validity using the determination unit15.

In this embodiment, for example, the determination unit15generates a feature quantity relating to one product area image mp13on the basis of the relevancy information me illustrated inFIG. 8. More specifically, the determination unit15generates a feature quantity ie1as below by using values representing same/difference of product names, series/brands, and manufacturers represented in the relevancy information me as values (feature) of each item.

In addition, the feature quantity represented in Equation (1) is one example of a case in which a validity determiner using a linear learner is used for denervation of validity, and a feature quantity and the relevancy information used for generation of the feature quantity are not limited to those of this example. The feature quantity described above is merely one example, is information on which relevancy between a product recognized as a product represented by one product area image and a product recognized as a product represented by a product area image that is adjacent or close to the one product area image or within a predetermined distance is reflected, and may be any information as long as the information has a form that is appropriate for being used by a predetermined learner and a determiner. As will be described later, the learner and the determiner are not limited to the case of being configured by a linear learner, and a known technology of linear learning, a SVM, a neural network, or the like may be employed.

As illustrated inFIG. 10, the determination unit15inputs the generated feature quantity ie1to a validity determiner CM1and acquires a result r1(validity score) of determination of validity of recognition of a product relating to one product area image. The validity determiner CM1is a machine-learned determiner relating to determination of validity based on a predetermined feature quantity.

For example, in the example of this embodiment, in a case in which the validity determiner CM1is configured as a linear learner, a validity score y of a result of determination of validity having a feature quantity ie1as a feature vector x having the number of features as the number of items is calculated using the following Equation (2).

Here, elements of the feature vector x is represented as below on the basis of a feature quantity ie1.

In addition, a vector w is a vector that is used for weighting a feature quantity and is acquired for determining validity based on the feature quantity through machine learning in advance.

While a value of the validity score y that is a result of the determination is different in accordance with details to be learned, for example, the value takes a value in the range of 0 to 1, and it can be learned such that the validity is the highest when y=1, and the validity is the lowest when y=0.

Since the relevancy information me illustrated inFIG. 8focuses on relevancy with product area images adjacent on the left and right sides of one product area image, the validity determiner CM1for a feature quantity ie1based on the relevancy information me illustrated inFIG. 10is configured through machine learning based on a feature quantity generated with relevancy between a product area image relating to determination of validity and product area images adjacent on the left and right sides thereof focused.

As described above, when relevancy information relating to one product area image is generated, focused product area images are not limited to product area images adjacent on the left and right sides of the one product area image, and accordingly, the validity determiner provided for determination of validity is configured through machine learning according to product area images focused when relevancy information is generated.

FIG. 11is a diagram illustrating another example of the process of generating a feature quantity and determining validity using the determination unit15. In the example illustrated inFIG. 11, when the validity of recognition of a product relating to one product area image mp13is determined, relevancy information generated using two product area images mp14and mp15arranged to the right side of the product area image mp13as targets is used. The determination unit15generates relevancy information representing relevancy between a product recognized for the product area image mp13and products recognized for the product area images mp14and mp15and generates a feature quantity on the basis of the generated relevancy information ie2.

A validity determiner CM2used here is different from the validity determiner CM1illustrated inFIG. 10. In other words, the validity determiner CM2is configured through machine learning based on a feature quantity generated with relevancy between a product area image relating to determination of validity and two product area images arranged to the right side thereof focused. Then, the determination unit15inputs the feature quantity ie2to the validity determiner CM2and acquires a result r2(validity score) of determination of validity of recognition of a product relating to the one product area image mp13.

FIG. 12is a diagram illustrating another example of the process of generating a feature quantity and determining validity using the determination unit15. In addition to the determination of validity (calculation of a validity score) described with reference toFIGS. 10 and 11, in the example illustrated inFIG. 12, when the validity of recognition of a product relating to one product area image mp13is determined, relevancy information generated for product area images mp11, mp12, mp14, and mp15that are respective two areas arranged on each of both left and right sides of the product area image mp13as targets is used. The determination unit15generates relevancy information representing relevancy between a product recognized for the product area image mp13and products recognized for the product area images mp11, mp12, mp14, and mp15and generates a feature quantity ie3on the basis of the generated relevancy information. Then, the determination unit15inputs the feature quantity ie2to the validity determiner CM3and acquires a result r3(validity score) of determination of validity of recognition of the product for the one product area image mp13. The validity determiner CM3is configured through machine learning based on a feature quantity generated with relevancy between a product area image relating to determination of validity and two respective product area images arranged on both left and right sides thereof focused.

In addition, the determination unit15may generate a feature quantity ie4originated from the determination results r1to r3output by a plurality of validity determiners CM1to CM3, acquire a determination result r4by inputting the generated feature quantity ie4to a validity determiner CM4, and determine validity of recognition of the product relating to the product area image mp13. The validity determiner CM4used here is configured on the basis of machine learning having results of determination of validity (validity scores) of predetermined three types relating to a product area image relating to determination of validity as a feature quantity.

In addition, in the examples illustrated inFIGS. 10 to 12, although a case in which the validity determiner is configured by a linear learner has been described, the validity determiner is not limited thereto and may be configured or using a known technology of a non-linear learner, a support vector machine (SVM), a neural network, or the like.

The learning model storing unit40illustrated inFIG. 1is a storage means that stores a learning model that is a result of machine learning in the validity determiner CM illustrated inFIGS. 10 to 12, and the determination unit15acquires an appropriate learning model according to the configuration of the feature quantity by referring to the learning model storing unit40and calculates a validity score.

The display unit16displays information relating to the validity determined by the determination unit15. More specifically, the display unit16displays information relating to validity relating to recognition of a product for a product area image on a display22. In addition, the display unit16is not an essential component of the planogram information generating device10according to this embodiment.

FIG. 13is a diagram illustrating an example of display of information relating to validity. In a case in which the validity of recognition of one product area image determined by the determination unit15is less than a predetermined degree, the display unit16displays information relating to the validity in a predetermined form in association with the one product area image.

As illustrated inFIG. 13, the display unit16displays an image of product display shelves PM4on the display22and displays information UI1relating to recognition results of products for product area images and validity of the recognition on the display22.

The information UI1, for example, includes recognition results mr1to mr7for some of product area images of a product display shelf of the fourth stage (uppermost stage). The recognition results mr1to mr7include product names and manufacturers that are some of attributes of the recognized products and, for example, illustrates shapes of a blow-off type and are respectively associated with product area images.

The example illustrated inFIG. 13illustrates a case in which the validity of recognition (for example, a validity score) of a product for a product area image mp21is less than a predetermined degree. As illustrated inFIG. 13, the display unit16displays the recognition result mr4of a product for a product area image mp21in association with the product area image mp21. In the example illustrated inFIG. 13, the product area image mp21represents a product having a product name: N21and a manufacturer: C2. On the other hand, a product represented by the product area image mp21is recognized by the product recognizing unit13as a product having a product name: N22and a manufacturer: C2. For this reason, it is determined that the validity of recognition of a product for the product area image mp21using the determination unit15is low. Accordingly, the display unit16displays the recognition result mr4in a form emphasized by applying a color or the like thereto. In this way, a product area image for which the validity of the result of recognition of a product is low can be recognized by a user.

FIG. 14is a diagram illustrating another example of display of information relating to validity. As illustrated inFIG. 14, the display unit16displays an image PM5of a product display shelf on the display22and displays information UI2relating to a recognition result of a product for a product area image and validity of recognition on the display22.

In the example illustrated inFIG. 14, more specifically, by touching a product area image represented in an image PM5of a product display shelf using a user's finger F, the display unit16displays information of a product recognized for each product area image (an outer view image of a product in the example illustrated inFIG. 14) in association with each product area image. In this way, a user can compare a recognition result of each product area image with the product area image, and accordingly, a result of recognition of an image of a product can be easily checked and corrected. In addition, in a case in which the validity of recognition of a product for the product area image mp31is less than a predetermined degree, the display unit16, as illustrated in information UI2, displays an outer view image representing a result of recognition of the product for the product area image mp31in a form emphasized by applying a color or the like thereto. In this way, a user can recognize a product area image for which validity of a result of product recognition is low.

FIG. 15is a diagram illustrating another example of display of information relating to validity. As illustrated inFIG. 15, the display unit16displays an image PM6of a product display shelf on the display22and displays information UI3relating to a result of recognition of a product relating to a product area image and validity of the recognition on the display22.

In this example, the product recognizing unit13recognizes a plurality of candidates for a product represented by one product area image for the one product area image. For example, as illustrated inFIG. 15, the product recognizing unit13outputs three products (product names: C21, C22, and C33) in order of highest to lowest score representing the reliability of product recognition as a result of recognition of one product area image mp41. Here, it is assumed that scores representing reliability of image recognition are higher in order of a product having the product name C22, a product having the product name C33, and a product having the product name C21.

The determination unit15determines validity of recognition for each of a plurality of products output by the product recognizing unit13as a result of recognition of one product area image. Then, the display unit16displays at least one product candidate among a plurality of product candidates and information relating to validity thereof in association with one product area image.

For example, the display unit16may sort information of a plurality of products recognized by the product recognizing unit13in order of highest to lowest validity of the recognition result of a product and displays the sorted information. Here, it is assumed that the validity of a recognition result for products is higher in order of the product having the product name C21, the product having the product name C22, and the product having the product name C33. As illustrated inFIG. 15, the display unit16displays product information mr31of the product name C21, product information mr32of the product name C22, and product information mr33of the product name C33in order of highest to lowest validity of recognition result for products as information relating to validity of product recognition for the product area image mp41.

In the example illustrated inFIG. 15, each of pieces of the product information mr31to mr32includes a check box. When a user inputs a check mark in a check box of a correct product as a result of recognition of the product area image mp41, the product recognizing unit13can store the product to which the check mark has been input in the recognition result storing unit50as a result of recognition of the product area image mp41. In this way, a result of recognition of a product for one product area image can be easily checked and corrected.

In addition, instead of displaying information of a plurality of products as illustrated inFIG. 15, the display unit16may display only a candidate for a product of which the validity of a recognition result of the product determined by the determination unit15is the highest among a plurality of candidates for a product recognized by the product recognizing unit13as a result of recognition of one product area image in association with the one product area image. Accordingly, a result of recognition of a product for one product area image can be easily corrected.

In addition, in a case in which validity of recognition of a product recognized for one product area image is less than a predetermined degree, the display unit16may display the product recognized for a product area image adjacent to the one product area image in an image of product display shelves as a candidate for a product represented by the one product area image.

More specifically, the product recognizing unit13performs recognition of a product for one product area image area and performs recognition of a product for a product area image adjacent to the one product area image. Then, the determination unit15assumes that the product recognized for the adjacent product area image is the product recognized for the one product area image and determines validity of the assumed recognition. Then, the display unit16may display information of the product recognized for the adjacent product area image as a candidate for the product represented by the one product area image together with information relating to the validity thereof.

For example, in a case in which the validity of recognition of the product recognized for the one product area image is less than a predetermined degree and in a case in which the product represented by the one product area image is not included in a result of recognition of the one product area image, candidates for a product that has a possibility of being the product represented by the one product area image can be presented to the user.

The correction unit17corrects a first product recognized as the product represented by the one product area image by the product recognizing unit13on the basis of the information relating to validity determined by the determination unit15.

For example, in a case in which it is determined by the determination unit15that a product other than the first product recognized by the product recognizing unit13has the highest validity as a result of recognition of the one product area image, the correction unit17corrects the result of recognition of a product represented by the one product area image to the product determined to have the highest validity. More specifically, for example, in a case in which a plurality of candidates for a product represented by one product area image are recognized by the product recognizing unit13for the one product area image, and validity of recognition is determined by the determination unit15for each of a plurality of products output as a result of recognition of the one product area image, the correction unit17corrects the result of recognition of a product represented by the one product area image to a product having the highest validity of recognition.

In addition, as described above, in a case in which a correcting input is made by the user in accordance with display of information relating to validity of recognition of the one product area image as the first product using the display unit16, the correction unit17may correct the result of recognition of a product represented by the one product area image to an input product.

The generation unit18generates planogram information that is information of arrangement of products arranged on product display shelves. More specifically, the generation unit18generates planogram information on the basis of a positional relation between the position of a shelf board acquired from an image of the product display shelf and the position of a product area image detected by the detection unit and information of a product represented by the product area image recognized by the product recognizing unit13. One example of the planogram information is illustrated inFIG. 7. In this embodiment, the generation unit18generates planogram information on which a correction of the result of recognition of a product area image using the correction unit17is reflected. The generation unit18can output the generated planogram information to a predetermined storage means. In addition, the generation unit18may display the generated planogram information on the display22.

Variations of information that is further used for determining validity of recognition of a product for one product area image using the determination unit15will be described below.

The determination unit15may determine validity by further referring to color information representing a difference between information relating to a color of one product area image and information relating to a color of another product area image. In other words, the determination unit15can determine validity by further using color information as relevancy information.

More specifically, the determination unit15generates a color histogram of one product area image and a color histogram of another product area image (a product area image that is adjacent or close to the one product area image or the like) and calculates a similarity between the generated color histograms (information representing a difference). The similarity, for example, is realized by calculation of a so-called cosine distance or the like, and a calculation method thereof is not particularly limited, and any known technique may be used.

The determination unit15may determine validity of recognition of a product of one product area image using a validity determiner further using the similarity between the color histograms as feature quantity. The validity determiner used in this case can be acquired through machine learning using the similarity between the color histograms as a feature quantity. In addition, a similarity between color histograms between different product area images may be further used as a feature quantity. In addition, in the generation of color histograms, appropriate subtractive color processing may be performed in accordance with conditions such as a processing load and the like.

The determination unit15may determine validity by further referring to information relating to a distance between one product area image and another product area image in an image of product display shelves. In other words, the determination unit15may determine validity by further using information relating to a distance between one product area image and another product area image as relevancy information. More specifically, the determination unit15calculates a distance between one product area image and another product area image (a product area image that is adjacent or close to the one product area image or the like). The distance between the product area images, for example, may be represented in the number of pixels in an image of product display shelves or the number of faces.

The determination unit15may determine validity of recognition of a product of one product area image using a validity determiner further using a distance between the one product area image and another product area image as a feature quantity. The validity determiner used in this case is acquired through machine learning having a distance between product area images as a feature quantity. In addition, a distance between other product area images may be further used as a feature quantity.

In addition, the determination unit15may use a recognition score representing the accuracy of recognition of a product for a product area image that is output by the product recognizing unit13for determining the validity. In other words, the determination unit15may determine the validity by further referring to recognition scores relating to recognition of products for one product area image and another product area image as relevancy information. More specifically, the determination unit15acquires recognition scores for one product area image and another product area image (a product image area adjacent or close to the one product area image or the like). Then, the determination unit15may determine validity of recognition of a product of one product area image using a validity determiner by further using a recognition score relating to each product area image as a feature quantity. The validity determiner used in this case is acquired through machine learning having a recognition score relating to the product area image as a feature quantity.

Next, a planogram information generating method used in the planogram information generating device10will be described with reference toFIG. 16.FIG. 10is a flowchart illustrating processing details of the planogram information generating method according to this embodiment.

In Step S1, the image acquiring unit11acquires an image of product display shelves, which is imaged by the camera21, on which a plurality of products are arranged. In the subsequent Step S2, the detection unit12detects product area images representing products from the image of product display shelves acquired by the image acquiring unit11in Step S1.

Next, in Step S3, the product recognizing unit13recognizes a product represented by each product area image detected by the detection unit12in Step S2on the basis of information relating to images of products stored in advance.

In Step S4, the planogram analyzing unit14acquires planogram data that is information relating to arrangement of products on each of product display shelves on the basis of a result of recognition of products using the product recognizing unit13in Step S3and a positional relation of product area images in the image of product display shelves. In addition, the process of Step S4is not an essential process in the planogram information generating method according to this embodiment.

In Step S5, the determination unit15selects one product area image among a plurality of product area images recognized in Step S3. In Step S6, the validity of recognition of a product represented by one product area image is determined.

In Step S7, the display unit16displays information relating to the validity of recognition of a product for one product area image, which has been determined by the determination unit15in Step S6, on the display22in association with the one product area image. In addition, the process of Step S7is not an essential process in the planogram information generating method according to this embodiment.

In Step S8, the correction unit17corrects the product recognized by the product recognizing unit13as a product represented by one product area image in Step S3on the basis of the information relating to the validity determined by the determination unit15in Step S6.

In Step S9, the determination unit15determines whether or not determination of validity of recognition of products of all the product area images has been performed. In a case in which it is determined that determination of validity of all the product area images has been performed, the process proceeds to Step S10. On the other hand, in a case in which it is determined that determination of validity of all the product area images has not been performed, the processes of Steps S5to S8are repeated. In addition, in the planogram information generating method according to this embodiment, determination of validity for all the product area images and correction for a recognized product are not essential, and, when the processes of Steps S5to S8for some product area images are completed, the processing sequence may proceed to Step S10.

In Step S10, the generation unit18generates planogram information. More specifically, the generation unit18generates planogram information on which correction for a product recognized in Step S8is reflected.

Next, a planogram information generating program for causing a computer to function as the planogram information generating device10according to this embodiment will be described.FIG. 17is a diagram illustrating the configuration of the planogram information generating program P1.

The planogram information generating program P1is configured to include a main module m10that performs overall control of a planogram information generating process in the planogram information generating device10, an image acquiring module m11, the detection module m12, the product recognizing module m13, a planogram analyzing module m14, a determination module m15, a display module m16, a correction module m17, and a generation module m18. Functions for the image acquiring unit11, the detection unit12, the product recognizing unit13, the planogram analyzing unit14, the determination unit15, the display unit16, the correction unit17, and the generation unit18of the planogram information generating device10are realized by the modules m11to m18. In addition, the planogram information generating program P1may be in the form of being transmitted through a transmission medium such as a communication line or, as illustrated inFIG. 17, may be in the form of being stored on a recording medium M1.

In the planogram information generating device10, the planogram information generating method, and the planograin information generating program P1according to this embodiment described above, in consideration of a feature in that one product arranged on a product display shelf has relevancy with another product adjacent to the one product and another product arranged within a predetermined distance from the one product, product area images are detected from an image of product display shelves, a product represented by each of the product area images is recognized, and validity of recognition of one product area image as a first product is determined on the basis of relevancy information between the first product recognized as a product represented by the one product area image and a second product recognized as a product represented by another product area image. Then, the first product recognized as a product represented by the one product area image is corrected on the basis of information relating to the determined validity. In this way, a result of recognition of a product for a product area image is easily corrected, and the accuracy of the recognition can be improved.

In addition, in a planogram information generating device according to another embodiment, the other product area images may include product area images adjacent to the one product area image in the image of the product display shelves.

Products adjacent to each other on a product display shelf tend to have strong relevance in such attributes and the like. According to the embodiment described above, information representing relevance between a product represented by one product area image and products represented by product area images adjacent to the one product area image is included in the relevancy information, and accordingly, the accuracy of determination of the validity of recognition of the product for one product area image is improved.

In addition, in a planogram information generating device according to another embodiment, the other product area images may include product area images present within a predetermined distance from the one product area image in the image of the product display shelves.

Products arranged to be close to each other on a product display shelf tends to have relevance in such attributes and the like. According to the embodiment described above, information representing relevance between a product represented by one product area image and products represented by product area images present within a predetermined distance from the one product area image is included in the relevancy information, and accordingly, the validity of recognition of a product for one product area image is appropriately determined.

In addition, a planogram information generating device according to another embodiment further includes a planogram analyzing unit that acquires information relating to arrangement of products on each shelf of the product display shelves on the basis of a result of recognition of products using the product recognizing unit and a positional relation of the product area images in the image of the product display shelves, and the other product area images may be product area images having a predetermined positional relation with the one product area image among product area images of products arranged on each shelf of the product display shelves.

According to the embodiment described above, on the basis of the information relating to the arrangement of products on each shelf of product display shelves, information representing relevance between a product represented by one product area image and products represented by product area images having a predetermined positional relation with the one product area image is included in the relevancy information, and accordingly, the validity of recognition of a product for one product area image is appropriately performed.

In addition, in a planogram information generating device according to another embodiment, the relevancy information may be information representing a degree of coincidence between an attribute of the first product and an attribute of the second product.

According to the embodiment described above, the degree of coincidence between an attribute of the first product and an attribute of the second product is employed as relevancy information, and accordingly, the relevance between the first product and the second product is appropriately represented using the relevancy information.

In addition, a planogram information generating device according to another embodiment may further include a display unit that displays the information relating to the validity determined by the determination unit.

According to the embodiment described above, by referring to the information relating to the displayed validity, a user can easily check and correct a result of recognition of a product for one product area image.

In addition, in a planogram information generating device according to another embodiment, in a case in which the validity of recognition of the one product area image determined by the determination unit is less than a predetermined degree, the display unit may display information relating to the validity in a predetermined form in association with the one product area image.

According to the embodiment described above, a user can be allowed to recognize a product area image having low validity of the result of product recognition.

In addition, in a planogram information generating device according to another embodiment, the product recognizing unit recognizes a plurality of candidates for a product represented by the one product area image for the one product area image, the determination unit determines validity in a case in which each of the plurality of candidates for the product is recognized as the first product, and the display unit may display at least one candidate for the product among the plurality of candidates for the product and information relating to validity of the candidate in association with the one product area image.

According to the embodiment described above, a more valid candidate for a product can be presented as a product represented by the one product area image, and accordingly, a result of recognition of a product for the one product area image can be easily checked and corrected.

In addition, in a planogram information generating device according to another embodiment, in a case in which the validity of recognition of the one product area image determined by the determination unit is less than a predetermined degree, the display unit may display the second products as candidates for a product represented by the one product area image.

According to the embodiment described above, in consideration of high relevancy between a product represented by one product area image and products represented by other product area images, the second products recognized as products represented by other product area images are presented as candidates for a product represented by the one product area image. Accordingly, products having a high possibility of being a product represented by the one product area image are presented, and accordingly, a result of recognition of a product for the one product area image can be easily corrected.

In addition, in a planogram information generating device according to another embodiment, the product recognizing unit recognizes a plurality of candidates for the product represented by the one product area image for the one product area image, the determination unit determines validity in a case in which each of the plurality of candidates for the product is set as the first product, and display unit may display a candidate for the product having the highest validity among the plurality of candidates for the product in association with the one product area image as a result of recognition of the one product area image.

According to the embodiment described above, the most valid candidate for a product as a product represented by one product area image can be presented, and accordingly, a result of recognition of the product for the one product area image can be easily corrected.

In addition, in a planogram information generating device according to another embodiment, the correction unit may correct the first product recognized as the product represented by the one product area image using the product recognizing unit on the basis of a correction input that is input by a user in accordance with the information relating to the validity displayed by the display unit.

According to the embodiment described above, the result of recognition of a product is corrected on the basis of information input by a user who has recognized the information relating to validity of recognition of the product for the product area image, and accordingly, the result of recognition of the product for the one product area image can be corrected reliably and appropriately.

In addition, in a planogram information generating device according to another embodiment, the determination unit may determine validity by further referring to color information representing a difference between information relating to a color of one product area image and information relating to colors of other product area images.

According to the embodiment described above, by considering that there is a high possibility that the difference between the color of one product area image and colors of other product area images is small due to strong relevancy between a product represented by the one product area image and products represented by other product area images, the validity of recognition of the product for the one product area image is determined by further referring to the difference between the information relating to the color of the one product area image and the information relating to the colors of other product area images. Accordingly, the accuracy of determination of the validity is improved.

In addition, in a planogram information generating device according to another embodiment, the determination unit may determine the validity by further referring to information relating to a distance between one product area image and other product area images in the image of the product display shelf.

According to the embodiment described above, by considering that a degree of relevancy between a product represented by one product area image and products represented by other product area images depends on a distance between the product area images, the validity of recognition of the product for one product area image is determined by further referring to information relating to a distance between the product area images. Accordingly, the accuracy of determination of the validity is improved.

In addition, in the planogram information generating device according to another embodiment, the product recognizing unit outputs a recognition score representing accuracy of recognition of a product for a product area image, and the determination unit may determine the validity by further referring to recognition scores relating to recognition of products for one product area image and other product area images.

According to the embodiment described above, the validity of recognition of a product relating to one product area image is determined using a recognition score representing the accuracy of recognition of a product for a product area image. Accordingly, the accuracy of determination of the validity is improved.

As above, while the present invention has been described in detail, it is apparent to a person skilled in the art that the present invention is not limited to the embodiments described in this specification. The present invention may be modified or changed without departing from the concept and the scope of the present invention set in accordance with the claims. Thus, the description presented in this specification is for the purpose of exemplary description and does not have any limited meaning for the present invention.

Each aspect/embodiment described in the present disclosure may be applied to long term evolution (LTE), LTE-advanced (LTE-A), Super 3G, IMT-advanced, 4G, 5G, future ratio access (FRA), W-CDMA (Registered trademark), GSM (registered trademark), CDMA 2000, ultra mobile broadband (UMB), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, ultra-wideband (UWB), Bluetooth (registered trademark), a system using another appropriate system and/or a next generation system extended based on these.

The processing sequence, the sequence, the flowchart, and the like of each aspect/embodiment described in the present disclosure may be changed in order as long as there is no contradiction. For example, in a method described in the present disclosure, elements of various steps are presented in an exemplary order, and the method is not limited to the presented specific order.

Information and the like may be output from an upper layer (or a lower layer) to a lower layer (or an upper layer). The information and the like may be input and output through a plurality of network nodes.

The input/output information and the like may be stored in a specific place (for example, a memory) or managed using a management table. The input/output information and the like may be overwritten, updated, or additionally written. The output information and the like may be deleted. The input information and the like may be transmitted to another device.

A judgment may be performed using a value (“0” or “1”) represented by one bit, may be performed using a Boolean value (true or false), or may be performed using a comparison between numerical values (for example, a comparison with a predetermined value).

The aspects/embodiments described in the present disclosure may be individually used, used in combination, or be switched therebetween in accordance with execution. In addition, a notification of predetermined information (for example, a notification of being X) is not limited to be performed explicitly and may be performed implicitly (for example, a notification of the predetermined information is not performed).

It is apparent that software, regardless whether it is called software, firmware, middleware, a microcode, a hardware description language, or any other name, be widely interpreted to mean a command, a command set, a code, a code segment, a program code, a program, a subprogram, a software module, an application, a software application, a software package, a routine, a subroutine, an object, an executable file, an execution thread, an order, a function, and the like.

In addition, software, a command, and the like may be transmitted and received via a transmission medium. For example, in a case in which software is transmitted from a website, a server, or any other remote source using wiring technologies such as a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL) and the like and/or radio technologies such infrared rays, radio waves, and microwaves, and the like, such wiring technologies and/or radio technologies are included in the definition of the transmission medium.

Information, a signal, and the like described in the present disclosure may be represented using any one among other various technologies. For example, data, an instruction, a command, information, a signal, a bit, a symbol, a chip, and the like described over the entire description presented above may be represented using a voltage, a current, radiowaves, a magnetic field or magnetic particles, an optical field or photons, or an arbitrary combination thereof.

In addition, a term described in the present disclosure and/or a term that is necessary for understanding the present disclosure may be substituted with terms having the same meaning or a meaning similar thereto.

Terms “system” and “network” used in the present disclosure are compatibly used.

In addition, information, a parameter, and the like described in the present disclosure may be represented using absolute values, relative values from predetermined values, or other corresponding information.

Description of “on the basis of” used in the present invention does not mean “only on the basis of” unless otherwise mentioned. In other words, description of “on the basis of” means both “only on the basis of and at least on the basis of.”

In the present disclosure, in a case in which names such as “first,” “second,” and the like is used, referring to each element does not generally limit the amount or the order of such an element. Such names may be used in the present disclosure as a convenient way for distinguishing two or more elements from each other. Accordingly, referring to the first and second elements does not mean that only the two elements are employed therein or the first element precedes the second element in a certain form.

As long as “include,” “including,” and modifications thereof are used in the present disclosure or the claims, such terms are intended to be inclusive like a term “comprising.” In addition, a term “or” used in the present disclosure or the claims is intended to be not an exclusive logical sum.

In the present disclosure, other than a case in which clearly only one device is present in a context or technically, a device includes a plurality of devices.

In the entirety of the present disclosure, unless a singular form is represented clearly from the context, plural forms are included.

REFERENCE SIGNS LIST