Object identification device, object identification method, calculation device, system, and recording medium

Provided is a technology to identify the number of objects to be identified for each packaging unit with a high degree of accuracy. This object identification device comprises: a comparison unit which compares a captured image with an identification image for identifying objects; and a determination means which, if a plurality of objects are identified as being contained within the captured image, determines whether the plurality of identified objects are articles bundled together in accordance with predetermined conditions, based on a first parameter which represents the geometrical relationship between a first position in a first space, and a second position corresponding to the first position in a second space which represents the image space if the plurality of articles represented by the identification image are bundled together, and a second parameter which represents the geometrical relationship between the first position and a third relationship corresponding to the first position.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is based upon and claims the benefit of the priority of International Application No. PCT/JP2017/024924, entitled OBJECT IDENTIFICATION DEVICE, OBJECT IDENTIFICATION METHOD, CALCULATION DEVICE, SYSTEM, AND RECORDING MEDIUM, filed on Jul. 7, 2017, the disclosure of which is incorporated herein in its entirety. Embodiments of this invention relate to an object identification device, an object identification method, a calculation device, a system, and a recording medium.

BACKGROUND ART

As a method of enhancing recognition precision, increasing an amount of information in a database used for recognition is disclosed (for example, PTL 1). Further, a technology of estimating a position and an attitude of a three-dimensional object, based on a two-dimensional image, by use of a database is disclosed (for example, PTL 2). The database is generated by use of images in which an object for reference is captured from a plurality of different viewpoints.

CITATION LIST

Patent Literature

SUMMARY OF INVENTION

Technical Problem

A predetermined number of articles may be grouped together and packed as one object. When an image of single article is registered as for identification (database image) in identification of such grouped-together and packed articles, individual articles may be identified, but whether the object is a bundle of grouped articles or single article may not be determined accurately.

Further, when an image of a plurality of articles grouped together is set as one object in a database image, the image is checked against database images prepared for each of grouping units, and as varieties of grouping unit increase, a number of checking process increases. This causes overload of check processing on an object identification device, thus not practical.

The present disclosure has been made in view of the aforementioned problem, and one of the objects of the present disclosure is to provide a technology capable of precisely identifying a number of objects to be identified for each packing unit.

Solution to Problem

One embodiment of the present disclosure is an object identification device comprising:

check means for checking a captured image against an identification image for identifying objects; and

determination means for, in a case where a plurality of objects are identified to be included in the captured image, determining whether or not the identified objects are articles grouped together under a predetermined condition,

by usinga first parameter indicating a geometric relation between a first position in a first space being an image space of the identification image and a second position corresponding to the first position in a second space, the second space representing an image space in a case where objects indicated by the identification image are articles grouped together under the predetermined condition, anda second parameter indicating a geometric relation between the first position and a third position corresponding to the first position on the captured image.

One embodiment of the present disclosure is an object identification method comprising:

checking a captured image against an identification image for identifying objects; and

in a case where a plurality of objects are identified to be included in the captured image, determining whether or not the identified objects are articles grouped together under a predetermined condition,

by usinga first parameter indicating a geometric relation between a first position in a first space being an image space of the identification image and a second position corresponding to the first position in a second space, the second space representing an image space in a case where objects indicated by the identification image are articles grouped together under the predetermined condition, anda second parameter indicating a geometric relation between the first position and a third position corresponding to the first position on the captured image.

One embodiment of the present disclosure is a calculation device comprising:

extraction means for extracting feature information indicating a feature of an object, from an identification image for identifying the object; and

calculation means for, in a case where a plurality of objects are identified on a captured image by a check using the identification image, calculating a first parameter used along with a second parameter when determining whether or not the plurality of identified objects are articles grouped together under a predetermined condition, based on the extracted feature information,the first parameter indicating a geometric relation between a first position in a first space being an image space of the identification image and a second position related to the first position in a second space representing an image space when a plurality of objects indicated by the identification image are articles grouped together under the predetermined condition, andthe second parameter indicating a geometric relation between the first position and a third position related to the first position on the captured image.

Further, a system including the aforementioned object identification device and calculation device also falls under the category of the present disclosure.

A computer program providing each of the aforementioned devices or the aforementioned method by a computer, and a computer readable non-transitory recording medium storing the computer program also fall under the category of the present disclosure.

Advantageous Effects of Invention

The present disclosure can precisely identify a number of objects to be identified for each packing unit.

EXAMPLE EMBODIMENT

First Example Embodiment

A first example embodiment of the present disclosure will be described with reference to drawings.FIG. 1is a diagram illustrating an example of an overall configuration of system1according to the present example embodiment. System1illustrated inFIG. 1includes object identification device100, calculation device200, image capture device300, and output device400.FIG. 1illustrates a configuration characteristic of the present disclosure, and it goes without saying that system1may include a device not illustrated inFIG. 1. Further, the devices illustrated inFIG. 1may be configured to be combined with one another. For example, object identification device100and calculation device200may be integrated with each other, or object identification device100and image capture device300may be integrated with each other.

Object identification device100is mutually and communicably connected to calculation device200, image capture device300, and output device400. A communication means between the devices may be either of wired and wireless communication, or may be communication through any of a mobile communication network, a public network, a local area network (LAN), and a wide area network (WAN). Thus, various methods may be considered as the communication method between the devices.

Image capture device300captures an image including an object being an image capture target. For example, image capture device300is provided by a camera including a lens and an imaging element. For example, image capture device300is provided by a digital camera or a web camera. Image capture device300supplies a captured image to object identification device100. Image capture device300may store a captured image inside image capture device300or in a storage device different from object identification device100. In this case, object identification device100may acquire a captured image from a location where the captured image is stored.

Output device400performs output based on a control signal from object identification device100. For example, output device400may be a display device such as a display or a terminal device including a display. Further, without being limited to the above, output device400may be a speaker, a printer, or a device outputting information included in a received signal as a file.

FIG. 2is a diagram for illustrating an example of a use scene in which system1according to the present example embodiment is used. InFIG. 2, camera3is illustrated as an example of image capture device300. Camera3is installed at a position allowing image capture of an object5(a bundle or an article; a four-article bundle is shown inFIG. 2), as an example, grouping together a plurality of the same type of article packed in a predetermined container, that is, an article constitutes a portion of a bundle. Further, inFIG. 2, mobile terminal4is illustrated as an example of output device400. The aforementioned predetermined container may contain an article in a state of a plurality of articles not being bundled together.

A captured image including object5captured by camera3is transmitted to object identification device100. Object identification device100determines whether object5is an article or a bundle, based on an identification result of object5and a parameter calculated by calculation device200, and further determines a number of object5for each packing unit. Specifically, object identification device100determines a number of articles when object5are articles or determines a number of bundles when object5are bundles. Then, object identification device100outputs the determination result to mobile terminal4. Consequently, operator2can check the determination result displayed on a screen on mobile terminal4.

Next, configurations of object identification device100and calculation device200that are included in system1will be described with reference toFIG. 3. Object identification device100receives a captured image captured by image capture device300, checks objects included in the captured image, and determines a number of articles for each packing unit. Object identification device100includes first feature extraction unit110, check unit120, determination unit130, and output control unit140.

Calculation device200stores an image for identification (also referred to as a database image) for identifying an object, the image being used when object identification device100makes a check on an object included in a captured image. Further, calculation device200calculates a parameter used when object identification device100determines a number of articles for each packing unit. Calculation device200includes image storage210, second feature extraction unit220, parameter calculation unit230, and Parameter storage240.

First, each unit in calculation device200will be described. Image storage210stores a database image for identifying an object. A database image is used when making an object is checked against a captured image. An example of a database image stored by image storage210is illustrated inFIG. 4.FIG. 4is a diagram illustrating an example of image information41including a database image and being stored in image storage210. As illustrated inFIG. 4, image information41includes image number42, object name43, and database image44. Image number42is an identifier for identifying database image44. Object name43is a name of an object represented by database image44. Database image44is a captured image in which an image of an object indicated by object name43is captured.

As illustrated inFIG. 4, image information41includes database image44A being a captured image in which an image of an object with object name43“XXX” is captured. Further, image information41includes database image44B being a captured image in which an image of an object with object name43“YYY” is captured. While it is assumed in the present example embodiment that image information41includes one captured image for one object as database image44, a plurality of captured images may be included for one object as database images44.

Furthermore, database image44may be an image generated as an image for checking instead of an actually captured image. Further, database image44may be an image captured by image capture device300or an image acquired by another method. Further, image storage210may be provided by a storage device separate from calculation device200.

Second feature extraction unit220extracts a feature value from each database image44stored in image storage210and extracts (detects) a feature point having the feature value. Information related to a feature value and a feature point that are extracted by second feature extraction unit220is hereinafter referred to as feature information.

Second feature extraction unit220may extract feature information by any method, and the method is not particularly limited. Second feature extraction unit220may extract feature information used when object identification device100makes a check. Second feature extraction unit220stores the extracted feature information into the own unit or a storage such as image storage210, along with information (for example, image number42) indicating database image44being an extraction source. Aforementioned image storage210may store feature information extracted by second feature extraction unit220, in addition to database image44or in place of database image44. Further, second feature extraction unit220may extract a plurality of types of feature values as feature information. Second feature extraction unit220supplies the extracted feature information to parameter calculation unit230, along with information (for example, image number42) indicating database image44being an extraction source.

Parameter calculation unit230calculates a parameter (first parameter) indicating a geometric relation between database images, by use of feature information extracted by second feature extraction unit220.FIG. 5is a diagram for illustrating parameter calculation by parameter calculation unit230.

Database image51illustrated in the upper part ofFIG. 5is database image44A with image number42“P401” illustrated inFIG. 4. Further, bundle image52indicating a bundle grouping four articles indicated by database image51together is illustrated in the lower part ofFIG. 5. The bundle is grouped articles indicated by database image51together in the same direction in two rows and two columns. This bundle image52may be an image virtually generated based on database image51or may be acquired by capturing an image of a bundle grouping articles together, based on a predetermined condition. In the case of the latter, second feature extraction unit220or another component may extract feature information from a captured image in which an image of the bundle is captured. It is assumed that the bundle represented by bundle image52inFIG. 5includes a plurality of articles (B401A to B401D).

By use of feature information extracted from database image51by second feature extraction unit220, parameter calculation unit230specifies a feature point on bundle image52related to each feature point in an image space of database image51. Dot-and-dash lines53A and broken lines53B inFIG. 5are examples of segments connecting related feature points between database image51and bundle image52. Parameter calculation unit230specifies thus related feature points and calculates a parameter indicating a geometric relation between positions of feature points in database image51and bundle image52. For convenience of description, only part of dot-and-dash lines53A and broken lines53B are illustrated. In other words, parameter calculation unit230also similarly calculates parameters indicating geometric relations between database image51, and an image of an article B401C part and an image of an article B401D part in bundle image52, respectively.

An image space of database image51is herein referred to as an article image space (first space). Then, a position of a feature point on database image51in the article image space is also referred to as a first position. Further, an image space of bundle image52is referred to as a bundle image space (second space). Then, a position of a feature point in the bundle image space related to a feature point on database image51is also referred to as a second position. In other words, a parameter calculated by parameter calculation unit230indicates a geometric relation between a first position and a second position.

A parameter is assumed to be a homography matrix in the description of the present example embodiment but is not limited thereto, and may be a transformation factor used in an affine transformation or a parameter used in another transformation formula.

Then, parameter calculation unit230stores, into Parameter storage240, information indicating a database image being a target of parameter calculation, information indicating an article in the bundle image space, and a parameter. The pieces of information and the parameter are associated with one another.

As illustrated inFIG. 6, parameter information61includes transformation source image number62, transformation destination position number63, and parameter64. Transformation source image number62is an image number indicating a database image being a source of a geometric transformation. Transformation destination position number63is a number indicating a position of an article in a bundle being a destination of a geometric transformation. A position of an article is indicated by a sign given to each of a plurality of articles illustrated inFIG. 5, according to the present example embodiment.

For example, parameter64in the first row in parameter information61is a homography matrix in a case of transforming coordinates on database image44A being database image44with image number42“P401” into coordinates in a region taken up by article B401A at a position indicated by “B401A” in the bundle image space. For convenience of description, parameter64in this case is hereinafter denoted as H[P401→B401A]. As described above, transformation destination position number63has only to be information indicating a position of an article in a bundle and for example, may be “upper left.”

Similarly, H[P401→B401B] is a homography matrix in a case of transforming coordinates on database image44A with image number42“P401” into coordinates in a region taken up by article B401B at a position indicated by “B401B” in the bundle image space. The homography matrix indicates parameter64included in the second row inFIG. 6.

Parameter storage240may be provided by a storage device separate from calculation device200. Further, Parameter storage240may be integrally configured with image storage210. Further, Parameter storage240may be built into object identification device100.

Next, object identification device100will be described. First feature extraction unit110extracts a feature value from a captured image and extracts a feature point having the feature value. First feature extraction unit110acquires a captured image from image capture device300or a storage device storing the captured image. First feature extraction unit110extracts feature information being information about a feature value and a feature point from the acquired captured image. First feature extraction unit110may extract feature information by any method, and the method is not particularly limited. First feature extraction unit110may extract feature information by a method similar to that by second feature extraction unit220. When second feature extraction unit220extracts a plurality of types of feature values as feature information, first feature extraction unit110may extract any type of feature value as feature information. Then, first feature extraction unit110supplies the extracted feature information to check unit120, along with a captured image being an extraction source.

When a captured image acquired by first feature extraction unit110is stored in a storage or the like in object identification device100, first feature extraction unit110may supply extracted feature information to check unit120, along with information indicating a captured image being an extraction source.

Check unit120checks a captured image against a database image by use of feature information extracted by first feature extraction unit110and feature information extracted by second feature extraction unit220. Then, check unit120supplies the check result to determination unit130. For example, a check result includes a degree of similarity between a captured image and database image44, information indicating database image44, information indicating a related feature point, and a position of the feature point on the captured image. However, the check result is not limited to the above. Further, for example, information indicating database image44may be image number42or other information. A method of checking performed by check unit120is not particularly limited, and any method may be employed.

Determination unit130refers to a check result and determines whether or not the check result indicates that a plurality of objects are included in a captured image. When determining that the captured image includes a plurality of objects, based on the check result, determination unit130determines whether or not the plurality of objects constitute a bundle grouping some articles together, based on a predetermined condition. The determination is based on parameter64(a first parameter) stored in Parameter storage240and a second parameter indicating a geometric relation between database image44and the captured image.

It is preferable that a second parameter indicating a geometric relation between database image44and a captured image be the same as aforementioned parameter64. A second parameter is assumed to be a homography matrix in the description of the present example embodiment but is not limited thereto. The second parameter may be a transformation factor used in an affine transformation or a parameter used in another transformation formula.

First, determination unit130receives a check result from check unit120. Then, determination unit130refers to the check result and determines whether or not a plurality of objects are identified and also the identified plurality of objects indicate the same object. For example, whether the identified plurality of objects indicate the same object may be determined by determining whether or not the objects (articles) have the same name (for example, an object name). Then, when a plurality of objects having the same object name are identified, determination unit130acquires, from Parameter storage240, parameter64related to database image44related to the object in the captured image. The acquisition is performed by use of information indicating database image44, the information being included in the check result.

Determination unit130determines whether or not the plurality of identified objects constitute a bundle grouping the objects (articles) together, based on a predetermined condition. The determination is made based on acquired parameter64, and a second parameter indicating a geometric relation between database image44and the captured image.

Determination processing performed by determination unit130will be further described with reference toFIG. 4,FIG. 5, andFIG. 7toFIG. 11.FIG. 7is a diagram for illustrating the determination processing performed by determination unit130.

Captured image70illustrated inFIG. 7is an image acquired by capturing an image of four articles (71A to71D). Four articles (71A to71D) are grouped together to form bundle72in such a way as to be directed in the same direction in two rows and two columns. As illustrated inFIG. 7, captured image70includes one bundle72. Four articles (71A to71D) are herein referred to as articles71when the articles are not distinguished or generically named.

Check unit120checks captured image70illustrated inFIG. 7against database image44by use of feature information extracted from database image44and feature information extracted from captured image70. It is assumed that, from the acquired check result, a check result that four of the article represented by database image44A illustrated inFIG. 4appear in captured image70is consequently acquired. In other words, the check result tells that four articles71appear in captured image70.

A calculation method of a second parameter may be the same as the method performed by parameter calculation unit230. For example, a homography matrix for transforming coordinates on database image44into coordinates on an image of an article71A part in captured image70is hereinafter denoted as H[P401→71A]. A position of a feature point on a captured image related to a feature point on database image44is hereinafter also referred to as a third position. In other words, a second parameter calculated by determination unit130indicates a geometric relation between a first position and a third position.

Similarly, a homography matrix for transforming coordinates on database image44into coordinates on an image of an article71B part of captured image70is denoted as H[P401→71B]. Similarly, a homography matrix related to article71C is denoted as H[P401→71C], and a homography matrix related to article71D is denoted as H[P401→71D]. Thus, determination unit130calculates second parameters correspondingly to a number of identified articles.

Frame73A in broken lines illustrated inFIG. 7is a region on captured image70when coordinates on database image44are transformed into coordinates on captured image70by use of H[P401→71A], the region being indicated by the coordinates. Similarly, frame73B illustrated inFIG. 7is a region on captured image70when coordinates on database image44are transformed into coordinates on captured image70by use of H[P401→71B], the region being indicated by the coordinates. Frame73C is a region on captured image70when coordinates on database image44are transformed into coordinates on captured image70by use of H[P401→71C], the region being indicated by the coordinates. Frame73D is a region on captured image70when coordinates on database image44are transformed into coordinates on captured image70by use of H[P401→71D], the region being indicated by the coordinates.

Based on the check result, determination unit130acquires, from Parameter storage240, parameter64associated with “P401” being image number42indicating database image44A. In this case, determination unit130acquires, from Parameter storage240, parameters64in the first to fourth rows being indicated inFIG. 6and having transformation source image number62“P401.” As described above, acquired parameters64are denoted as H[P401→B401A], H[P401→B401B], H[P401→B401C], and H[P401→B401D] in an order from the first row inFIG. 6.

Then, in order to determine whether or not the plurality of identified objects are articles grouped together based on a predetermined condition, determination unit130transforms coordinates on database image44into coordinates on a captured image. The transformation is performed by use of the first parameter and the second parameter.

Article71A (first article) out of four identified articles (71A to71D) is set as a reference. A reference article may be set arbitrarily and is not particularly limited. Further, in this example, assuming that reference article71A and other articles (71B to71D) form a bundle, determination unit130performs determination processing by performing a homography transformation, to be described below, on article71B (second article) positioned adjacently on the right side of article71A.

First, determination unit130transforms coordinates on database image44into coordinates in the article image space.FIG. 8is a diagram conceptually illustrating a correspondence relation between the bundle image space and the article image space.

Article image space81illustrated inFIG. 8is an image space including database image81acorresponding to database image44A. A position (coordinates) of reference article71A in bundle image space82related to a position (upper left) of reference article71A under an assumption that reference article71A is included in a bundle is a position of article B401A, describing by use ofFIG. 5. Further, a position of a second article in bundle image space82related to a position of the second article (in this case, adjacent on the right) relative to the position (upper left) of reference article71A relates to a position of article B401B positioned adjacently on the right side of article B401A, describing by use ofFIG. 5. Determination unit130transforms coordinates on database image81ain article image space81into coordinates in bundle image space82by use of H[P401→B401B] (S1). Image82B illustrated inFIG. 8is an image in bundle image space82related to database image81a.

Next, determination unit130further transforms coordinates on database image81atransformed into coordinates in bundle image space82into coordinates in article image space81. The transformation is performed by use of an inverse matrix of H[P401→B401A] being a homography matrix between a position (coordinates) of reference article71A in bundle image space82related to a position (upper left) of reference article71A under the assumption that reference article71A is included in the bundle, and coordinates of database image44A.

An inverse matrix of H[P401→B401A] is denoted as H−1[P401→B401A]. Determination unit130transforms coordinates on database image81atransformed into coordinates in bundle image space82(corresponding to coordinates on image82B) into coordinates in article image space81by use of H−1[P401→B401A] (S2). Image81billustrated inFIG. 8is an image related to image82B in article image space81.

Then, determination unit130transforms the coordinates transformed in S2into coordinates on captured image70. The transformation is performed by use of H[P401→71A] being a homography matrix between coordinates in a region of reference article71A on captured image70and coordinates on database image44A.

FIG. 9is a diagram illustrating an example of a region (first region) indicated by coordinates after transformation when the coordinates on database image81aare transformed into the coordinates on captured image70. Captured image70illustrated inFIG. 9is similar to captured image70illustrated inFIG. 7.

Frame91B in dot-and-dash lines illustrated inFIG. 9illustrates an example of a region indicated by coordinates after transformation when the coordinates of database image81aare transformed into the coordinates on captured image70. The region indicated by this frame91B is a region estimated by use of H[P401→B401B], H−1[P401→B401A], and H[P401→71A]. The estimated region indicates a region of coordinates on image81bestimated to be positioned adjacently on the right side of an article indicated by database image81ain a coordinate space of captured image70under an assumption that the article constitutes a bundle.

A comparison between a region (second region) indicated by frame73B and a region indicated by frame91B tells that the frames almost overlap each another. Determination unit130calculates an overlap rate being a value indicating a degree of overlap between the frame73B region and the frame91B region. Then, when the overlap rate is equal to or greater than a predetermined threshold value, determination unit130determines that article71B included in frame73B is an article constituting a bundle along with article71A. Determination unit130makes a similar determination for article71C and article71D. Consequently, determination unit130determines whether or not a plurality of articles are articles constituting one bundle.

Thus, by using an overlap rate of regions for determination of whether or not a plurality of articles constitute a bundle, determination unit130can determine whether or not the plurality of articles constitute a bundle with a less amount of calculation.

Based on an overlap rate indicating a ratio between an area of one region and an area of a region included in both of the one region and another region, determination unit130may determine whether or not an article related to regions (article71B in the case ofFIG. 9) is an article constituting a bundle along with reference article71A. For example, determination unit130may make a determination, based on an overlap rate between an area of a frame73B region and an area of a region included in both of the frame73B region and the frame91B region. Further, determination unit130may make a determination, based on an overlap rate between an area of the frame91B region and an area of a region included in both of the frame73B region and the frame91B region. Further, determination unit130may make a determination by use of the two overlap rates. Further, without being limited to the above, an overlap rate may be a ratio between the sum of two areas and an area of a region included in both or may be a ratio between an area of a part included in at least either of two regions and an area of a region included in both.

Further, determination unit130may determine whether or not an article related to regions is an article constituting a bundle along with a reference article, not by regions but by closeness between parameters. For example, determination unit130calculates H[P401→71A]×H−1[P401→B401A]×H[P401→B401B] and calculates a degree of similarity between the calculation result and H[P401→71B]. The degree of similarity may be any indicator. For example, the degree of similarity may be closeness between values in matrices or may be closeness between values based on any indicator derived based on a matrix. Then, when the degree of similarity satisfies a predetermined condition, such as the degree of similarity being equal to or greater than a predetermined threshold value, determination unit130may determine that the article related to the regions is an article constituting a bundle along with the reference article.

Thus, by using a degree of similarity between matrices in determination of whether or not an article related to regions is an article constituting a bundle along with a reference article, object identification device100can more accurately determine whether or not the article related to the regions is an article constituting a bundle along with the reference article, compared with a case of using an overlap rate between the regions. When determining a degree of similarity, determination unit130may calculate, after acquiring parameter64, based on a check result, a second parameter being the same type as aforementioned parameter64.

Further, determination unit130may determine whether or not an article related to regions is an article constituting a bundle along with a reference article, by use of a feature point instead of all coordinates on database image44. For example, determination unit130may determine that an article related to regions is an article constituting a bundle along with a reference article when a distance between coordinates of a feature point transformed by use of H[P401→B401B], H−1[P401→B401A], and H[P401→71A], and coordinates of the feature point transformed by use of H[P401→71B] is equal to or less than a predetermined threshold value.

FIG. 10is another diagram for illustrating the determination processing performed by determination unit130. Captured image101illustrated inFIG. 10is an image acquired by capturing an image of four articles (102A to102D). Four articles (102A to102D) are not grouped together. As illustrated inFIG. 10, captured image101includes four articles (102A to102D). Check unit120checks such captured image101against database image44by use of feature information extracted from database image44and feature information extracted from captured image101. A method of correctly identifying each of four articles (102A to102D) to be an article and not to constitute a bundle will be described.

Each of frames103A to103D in broken lines illustrated inFIG. 10indicates a region on captured image101indicated by coordinates on captured image101transformed from coordinates on database image44by use of the aforementioned homography matrix.

In this case, article102A is set as a reference article. Determination unit130transforms coordinates on database image44in the article image space into coordinates in the bundle image space by use of H[P401→B401B]. Then, determination unit130performs a transformation of coordinates. The transformation is performed by use of a homography matrix between coordinates of a position in the bundle image space related to a position of the reference article in a bundle and coordinates on database image44A. As illustrated inFIG. 10, reference article102A is an article positioned in the upper left corner of the four articles. Therefore, determination unit130further transforms the coordinates on database image44A transformed into the coordinates in the bundle image space into coordinates in the article image space, by use of H−1[P401→B401A]. Determination unit130further transforms the coordinates transformed into the coordinates in the article image space into coordinates on captured image101. The transformation is performed by use of H[P401→102A] being a homography matrix between coordinates in the reference article102A region and coordinates on database image44A.

FIG. 11is a diagram illustrating an example of a region indicated by coordinates after transformation when coordinates on database image44A are transformed into coordinates on captured image101. Captured image101illustrated inFIG. 11is similar to captured image101illustrated inFIG. 10.

Frame104B in dot-and-dash lines illustrated inFIG. 11represents an example of a region indicated by coordinates after transformation when coordinates on database image44A are transformed into coordinates on captured image101. The region indicated by this frame104B is a region estimated by use of H[P401→B401B], H−1[P401→B401A], and H[P401→102A]. The estimated region indicates a region of coordinates estimated to be positioned adjacently on the right side of an article under an assumption that the article indicated by database image44constitutes a bundle, in a coordinate space of captured image101.

A comparison between the region indicated by frame103B and the region indicated by frame104B indicates a greater non-overlapping portion compared with a region indicated by frame73B and a region indicated by frame91B inFIG. 9. Consequently, determination unit130determines that article102B is not an article constituting a bundle along with article102A. In other words, determination unit130determines that each of article102B and article102A is an article. By making similar determinations on article102C and article102D, determination unit130determines that each object is an article. Consequently, determination unit130can determine that the articles included inFIG. 11are four articles with an object name “XXX.”

Thus, object identification device100according to the present example embodiment can correctly determine whether an object is an article or a bundle. Accordingly, object identification device100can precisely identify a number of objects for each packing unit.

Determination unit130supplies information indicating an article (for example, an object name) and a number of articles for each packing unit to output control unit140as a determination result.

Based on a determination result supplied from determination unit130, output control unit140controls output of information indicating the determination result. For example, when output device400is a display device such as a display (display unit) or a terminal device including a display, output control unit140outputs a control signal to output device400. The control signal is for displaying the information indicating the determination result on a screen. Consequently, output device400displays the information indicating the determination result on the screen.

Further, when output device400is a device outputting received information as a file, output control unit140outputs, to output device400, a control signal for outputting the information indicating the determination result as a file. Consequently, output device400outputs the information indicating the determination result as a file.

FIG. 12is a diagram illustrating an example of a display screen output by output device400when output device400is a display device or a terminal device including a display.

Output control unit140generates a control signal for causing a display unit such as a display to display a display screen in which information indicating a determination result is superimposed on a captured image. Based on the control signal generated by output control unit140, output device400displays, on the display unit, a display screen in which the information indicating the determination result is superimposed on the captured image.

Display screen121illustrated inFIG. 12is a screen displaying captured image70including bundle72composed of articles (71A to71D) illustrated inFIG. 7on a display unit. By output control unit140generating a display screen in which information122indicating a determination result is superimposed on captured image70, output device400displays display screen121illustrated inFIG. 12. A display method of a packing unit is not particularly limited; and for example, in a case of an article, information indicating an article may be displayed. Further, in a case of a bundle, a shape or a number of articles grouped together may be displayed.

Output control unit140may generate a display screen in which information indicating a determination result is associated with a related article for each object name and for each packing unit of an article. For example, when captured image70includes a bundle for one article and a bundle for another article, output control unit140receives a determination result along with information indicating a position of an identified object (an article and/or a bundle). Then, based on a received determination result and received information indicating a position, output control unit140generates a display screen in which information indicating a determination result for the one object is associated with the bundle of articles. Similarly, output control unit140associates information indicating a determination result for the other object with the bundle of the other article in the display screen. Thus, by output control unit140generating a display screen in which information indicating a determination result is associated with a related article for each object name and for each packing unit of an object, output device400displays the display screen in which the determination result is associated with the object. Accordingly, object identification device100allows operator2viewing a display screen to readily grasp a quantity for each article and each packing unit of an article.

Further, output control unit140may generate a control signal causing a display unit such as a display to display a display screen in which positional information is superimposed on a captured image. The positional information indicates a position of an identified article on the captured image when a packing unit is an article and indicates a bundle part when the packing unit is a bundle. For example, output control unit140may generate a display screen in which positional information indicating a position of bundle72on a captured image is superimposed on captured image70. As illustrated inFIG. 12, output control unit140may generate frame123including all regions (regions represented by frames73A to73D) of articles (71A to71D) included in bundle72at the position of bundle72. Then, output control unit140may generate a display screen in which frame123is superimposed on captured image70.

Thus, by displaying, on a display unit, a display screen in which information indicating a determination result is superimposed on a captured image, based on a control signal generated by output control unit140, output device400can allow an operator2viewing the display screen to readily grasp a number of articles for each packing unit.

FIG. 13is a diagram illustrating an example of an output file when output device400is a device outputting received information as a file. Output file131illustrated inFIG. 13is tabular data including a column for object name132, a column for packing unit133, and a column for quantity134. Output control unit140may generate a control signal for controlling output device400in such a way as to output such output file131, based on a determination result. Consequently, operator2can readily grasp which article is included in a captured image in which packing unit and in what quantity. Further, for example, output control unit140may compare a list of articles to be included in a captured image with a determination result and perform control in such a way as to display the comparison result on output device400.

FIG. 14is a flowchart illustrating an example of a processing flow in calculation device200in system1according to the present example embodiment. As described inFIG. 14, second feature extraction unit220extracts feature information from database image44stored in image storage210(Step S141).

Then, based on the extracted feature information, parameter calculation unit230calculates a parameter indicating a geometric relation between a first position in the article image space and a second position in the bundle image space (Step S142).

As described above, the parameter is used along with a second parameter indicating a geometric relation between a first position and a third position related to the first position on a captured image, when determining, in a case of a plurality of articles being identified on a captured image by a check using database image44, whether or not the plurality of identified articles constitute an object (bundle) grouping the articles together, based on a predetermined condition.

FIG. 15is a flowchart illustrating an example of a processing flow in object identification device100in system1according to the present example embodiment. As described inFIG. 15, first feature extraction unit110in object identification device100extracts feature information from an acquired captured image (Step S151).

Then, by use of the feature information extracted from the captured image and the feature information extracted from database image44in Step S141, check unit120checks the captured image against database image44(Step S152).

Determination unit130refers to the check result and determines whether or not a plurality of objects are identified and also the identified objects indicate the same objects name (Step S153). When a plurality of objects with the same object name are not identified (NO in Step S153), the processing proceeds to Step S156.

When a plurality of articles with the same object name are identified (YES in Step S153), determination unit130calculates a parameter (second parameter) indicating a geometric relation between database image44related to the identified article (matching the captured image) and the captured image (Step S154).

Then, by use of the parameter calculated in Step S142and the parameter calculated in Step S154, whether or not the plurality of objects are articles grouped together based on a predetermined condition is determined (Step S155).

Subsequently, output control unit140generates a control signal for controlling output of information by output device400(Step S156). In a case of after execution of Step S155, output control unit140generates a control signal for outputting information indicating the determination result, based on the determination result. Then, output control unit140outputs the generated control signal to output device400(Step S157).

Consequently, based on the control signal output by output control unit140, output device400can display, for example, display screen121as illustrated inFIG. 12on a display unit.

An example of the predetermined condition including (1) to (3) described below has been described in the present example embodiment.(1) Number of articles is four.(2) All articles are directed in the same direction.(3) Articles are arranged in two rows and two columns.

However, the predetermined condition is not limited to the above. Examples of the predetermined condition in a case of four articles are illustrated inFIG. 16. A part (a) inFIG. 16is an example when the predetermined condition includes: a number of articles is four; all four articles are directed in the same direction; and the articles are vertically arranged in one column. A part (b) inFIG. 16is an example when the predetermined condition includes: a number of articles is four; articles are arranged in two rows and two columns; and an upper part of an article faces outside. A part (c) inFIG. 16is an example when the predetermined condition includes: a number of articles is four; all articles are directed in the same direction; and articles are lined up side by side.

As described above, check unit120in object identification device100in system1according to the present example embodiment checks a captured image against a database image for identifying an object. Then, when the check result indicates that a plurality of objects are included in the captured image, determination unit130determines whether or not the plurality of identified objects are articles grouped together based on a predetermined condition. The determination is made by use of a first parameter and a second parameter. The first parameter indicates a geometric relation between a first position on a database image in an image space and a second position related to the first position in the image space when a plurality of articles indicated by the database image are grouped together based on the predetermined condition. Further, the second parameter indicates a geometric relation between the first position and a third position related to the first position on the captured image.

For example, when a captured image is checked against a database image in a case that the captured image includes one bundle composed of four articles, four articles are assumed to be included in the captured image from the check result. Determination unit130according to the present example embodiment determines whether or not the plurality of identified articles constitute a bundle, based on the check result by determining whether or not an article is positioned at a position forming a bundle relative to another article. The latter determination is based on a parameter indicating a geometric relation between a position on a database image and a position in a bundle image space when articles constitute a bundle, and a parameter indicating a geometric relation between the captured image and the database image. Consequently, object identification device100can determine whether the plurality of objects included in the captured image are articles grouped together based on a predetermined condition or individual articles.

Accordingly, even when a plurality of objects are articles grouped together as one bundle, object identification device100according to the present example embodiment can accurately determine the articles as one bundle instead of a plurality of articles.

Accordingly, object identification device100according to the present example embodiment can precisely identify a number of articles to be identified for each packing unit.

In system1according to the present example embodiment, image capture device300may be integrally formed with object identification device100and function as, for example, an intelligence camera. Further, part of the functions of calculation device200may be built into object identification device100. For example, first feature extraction unit110and second feature extraction unit220may be provided by the same circuit.

Thus, the devices in system1illustrated inFIG. 1may be configured in combination.

Second Example Embodiment

A second example embodiment of the present disclosure will be described. A basic configuration according to the present example embodiment resolving the problem to be resolved by the present disclosure will be described.

FIG. 17is a functional block diagram illustrating an example of a functional configuration of object identification device10according to the present example embodiment. As illustrated inFIG. 17, object identification device10according to the present example embodiment includes check unit12and determination unit13.

Check unit12has the function of aforementioned check unit120. Check unit12checks a captured image against an image for identification for identifying an object. Check unit12supplies the check result to determination unit13. A method of checking performed by check unit12is not particularly limited, and a method similar to that by aforementioned check unit120may be employed.

Determination unit13has the function of aforementioned determination unit130. When a check result indicates that a plurality of objects are included in a captured image, determination unit13determines whether or not the plurality of identified objects are articles grouped together based on a predetermined condition, based on a first parameter and a second parameter. The first parameter indicates a geometric relation between a first position in a first space being an image space of an image for identification and a second position related to the first position in a second space representing an image space when a plurality of articles indicated by the image for identification are grouped together based on a predetermined condition. Further, the second parameter indicates a geometric relation between the first position and a third position related to a first position on the captured image.

Since it can be understood that a plurality of articles are identified based on a check result, the plurality of articles are assumed to include a first article and a second article different from the first article in the description.

Similarly to aforementioned determination unit130, when an overlap rate is equal to or greater than a predetermined threshold value, determination unit13may determine that the plurality of identified objects are articles grouped together. The overlap rate indicates a degree of overlap between a first region indicated by coordinates transformed by performing (1) to (3) described below and a second region on a captured image of the second article, the second region being acquired from the second parameter related to the second article on the captured image.(1) Transform coordinates of an image for identification into coordinates in the second space by use of the first parameter related to the second article in the second space, based on a position of the second article relative to a position of the first article.(2) Transform the coordinates transformed into coordinates in the second space into coordinates in the first space, based on the first parameter related to the first article in the second space.(3) Transform the coordinates transformed into coordinates in the first space into coordinates on the captured image by use of the second parameter related to the first article on the captured image.

FIG. 18is a flowchart illustrating an example of a processing flow in object identification device10according to the present example embodiment. As described inFIG. 18, check unit12checks a captured image against an image for identification for identifying an object (Step S181). Then, determination unit13determines whether or not a plurality of objects are articles grouped together based on a predetermined condition, based on a first parameter and a second parameter (Step S182).

FIG. 19is a functional block diagram illustrating an example of a functional configuration of calculation device20calculating a first parameter referred to by aforementioned object identification device10. As illustrated inFIG. 19, calculation device20includes extraction unit22and calculation unit23.

Extraction unit22has the function of aforementioned second feature extraction unit220. Extraction unit22extracts feature information indicating a feature of an object from each of a plurality of images for identification for identifying articles. Extraction unit22supplies the extracted feature information to calculation unit23.

Calculation unit23has the function of aforementioned parameter calculation unit230. Based on extracted feature information, calculation unit23calculates a first parameter indicating a geometric relation between a first position in a first space being an image space of an image for identification and a second position related to the first position in a second space representing an image space when a plurality of articles indicated by the image for identification are grouped together based on a predetermined condition. As described above, the first parameter is used along with a second parameter indicating a geometric relation between the first position and a third position related to the first position on a captured image, when determining whether or not a plurality of identified objects are articles grouped together based on a predetermined condition, in a case of the plurality of articles being identified on the captured image by a check by object identification device10using the image for identification.

Further, object identification device10and calculation device20, according to the present example embodiment, may be configured as system6illustrated inFIG. 20.FIG. 20is a block diagram illustrating a configuration example of a system including object identification device10and calculation device20. As illustrated inFIG. 20, system6includes object identification device10and calculation device20. Object identification device10and calculation device20are communicably connected to each another.

Object identification device10may be integrally formed with calculation device20, or part of the functions of calculation device20may be built into object identification device10. Further, system6may be configured to include image capture device300and output device400, similarly to aforementioned system1.

For example, when a captured image including an object (one bundle) composed of four articles grouped together is checked against an image for identification, the captured image is assumed to include four articles from the check result. Based on the check result, determination unit13according to the present example embodiment determines whether or not the plurality of identified articles constitute a bundle by determining whether or not an article is positioned at a position forming a bundle relative to another article, based on a first parameter and a second parameter. The first parameter indicates a geometric relation between a position on the image for identification and a position in a bundle image space when an object is a bundle. The second parameter indicates a geometric relation between the captured image and the image for identification. Consequently, object identification device10can determine whether the plurality of objects included in the captured image are articles grouped together based on a predetermined condition or individual articles.

Accordingly, when a plurality of objects are articles grouped together based on a predetermined condition, object identification device10according to the present example embodiment can accurately determine the objects as articles grouped together instead of individual articles.

Accordingly, object identification device10according to the present example embodiment can precisely identify a number of articles to be identified for each packing unit.

Hardware Configuration

Each component in each device according to each example embodiment of the present disclosure represents a function-based block. For example, a part or the whole of each component in each device is provided by any combination of information processing device900as illustrated inFIG. 21and a program.FIG. 21is a block diagram illustrating an example of a hardware configuration of information processing device900providing each component in each device. As an example, information processing device900includes the following components.Central processing unit CPU)901Read only memory (ROM)902Random access memory (RAM)903Program904loaded on RAM903Storage device905storing program904Drive device907performing a read and a write from and into recording medium906Communication interface908connected to communication network909Input-output interface910performing input and output of dataBus911connecting the components

Each component in each device according to each example embodiment is provided by CPU901acquiring and executing program904providing the function of the component. For example, program904providing the function of each component in each device is previously stored in storage device905or ROM902, and is loaded into RAM903and executed by CPU901, as needed. Program904may be supplied to CPU901through communication network909or may be previously stored in recording medium906, and drive device907may read the program and supply the program to CPU901.

There are various modified examples of the method of providing each device. For example, each device may be provided by any combination of a separate information processing device900and a program, for each component. Further, a plurality of components included in each device may be provided by any combination of one information processing device900and a program.

Further, a part or the whole of each component in each device is provided by another general-purpose or dedicated circuit, a processor or the like, or a combination thereof. The above may be configured with a single chip or may be configured with a plurality of chips connected through a bus.

A part or the whole of each component in each device may be provided by a combination of the aforementioned circuit or the like, and a program.

When a part or the whole of each component in each device is provided by a plurality of information processing devices, circuits, or the like, the plurality of information processing devices, circuits, or the like may be placed in a concentrated manner or in a distributed manner. For example, the information processing devices, the circuits, or the like may be provided in a form of being connected to one another through a communication network, such as a client-server system or a cloud computing system.

The respective aforementioned example embodiments are preferred example embodiments of the present disclosure, and the scope of the present disclosure is not limited to the respective aforementioned example embodiments. A person skilled in the art can make example embodiments which include various changes through modifying and substituting the respective aforementioned example embodiments without departing from the spirit and scope of the present disclosure.

For example, object identification device (10,100) according to the present disclosure may be suitably used in a system including a camera installed in the neighborhood of a cash register such as a point of sales (POS) register. In this case, aforementioned object identification device (10,100) may be provided by being built into the POS register. The POS register may further have the function of calculation device (20,200). Further, object identification device (10,100) may cause the POS register to perform output similarly to output device400. Even in this case, by determining whether or not a plurality of recognized objects are the same object from a captured image acquired by capturing an image of a commodity by the camera, object identification device (10,100) can precisely identify a number of the objects to be identified.

Further, object identification device (10,100) and calculation device (20,200), according to the present disclosure, may be provided by a server device capable of communicating with the POS register. Then, object identification device (10,100) may cause the POS register to perform output similarly to output device400. Thus, object identification device (10,100) and calculation device (20,200), according to the present disclosure, may also be suitably applied to a configuration using a POS register already installed at a store.

The aforementioned POS register is not limited to a cash register operated by a clerk and may be either of a sharing-type cash register (semi-self-register) at which a customer performs a checkout and a cash register (self-register) at which a clerk is not involved when passing through the register (a series of operations up to a checkout is performed only by a customer).

A part or the whole of the aforementioned example embodiments may also be described as the following supplementary notes but are not limited thereto.

check means for checking a captured image against an identification image for identifying objects; and

determination means for, in a case where a plurality of objects are identified to be included in the captured image, determining whether or not the identified objects are articles grouped together under a predetermined condition,

by usinga first parameter indicating a geometric relation between a first position in a first space being an image space of the identification image and a second position corresponding to the first position in a second space, the second space representing an image space in a case where objects indicated by the identification image are articles grouped together under the predetermined condition, anda second parameter indicating a geometric relation between the first position and a third position corresponding to the first position on the captured image.
(Supplementary Note 2)

The object identification device according to Supplementary Note 1, wherein

the second position is a position of each of the articles grouped together, and

the determination meanscalculates the second parameters for the number of the identified objects, based on a third position placed each of the identified objects, anddetermines whether or not the identified objects are the articles grouped together, by using the first parameters for the number of the articles grouped together and the second parameters.
(Supplementary Note 3)

The object identification device according to Supplementary Note 2, wherein

the identified objects include a first object and a second object which is different from the first object, and

the determination means

(i) transforms coordinates on the identification image into coordinates in the second space by use of the first parameter related to a second object in the second space, based on a position of the second object relative to a position of the first object, (ii) transforms the transformed coordinates into coordinates in the first space, based on the first parameter related to the first object in the second space, and (iii) transforms coordinates transformed into coordinates in the first space into coordinates on the captured image by use of the second parameter related to the first object on the captured image, and

determines the identified objects are the articles grouped together in a case where an overlap rate indicating a degree of overlap between a first region indicated by coordinates transformed into coordinates on the captured image and a second region on the captured image of the second object, the second region being acquired from the second parameter related to the second object on the captured image, is equal to or greater than a predetermined threshold value.

The object identification device according to Supplementary Note 2, wherein

identified objects include a first object and a second object which is different from the first object, and

the determination means determines whether or not the identified objects are articles grouped together under the predetermined condition, by use of a degree of similarity, based on a predetermined indicator between a value acquired from the first parameter related to the second object in the second space, the first parameter related to the first object in the second space, and the second parameter related to the first object on the captured image, and the second parameter related to the second object on the captured image.

The object identification device according to any one of Supplementary Notes 1 to 4, further comprising

output control means for controlling output of information indicating a determination result by the determination means, wherein

the output control means controls a display unit to display a display screen in which information indicating the determination result is superimposed on the captured image.

The object identification device according to Supplementary Note 5, wherein

the output control means controls the display unit to display the display screen in which information indicating the determination result is associated with the articles grouped together or an individual article in the captured image.

An object identification method comprising:

checking a captured image against an identification image for identifying objects; and

in a case where a plurality of objects are identified to be included in the captured image, determining whether or not the identified objects are articles grouped together under a predetermined condition,

by usinga first parameter indicating a geometric relation between a first position in a first space being an image space of the identification image and a second position corresponding to the first position in a second space, the second space representing an image space in a case where objects indicated by the identification image are articles grouped together under the predetermined condition, anda second parameter indicating a geometric relation between the first position and a third position corresponding to the first position on the captured image.
(Supplementary Note 8)

The object identification method according to Supplementary Note 7, wherein

the second position is a position of each of the articles grouped together, and the method further comprising:

calculating the second parameters for the number of the identified objects, based on a third position placed each of the identified objects, and

determining whether or not the identified objects are the articles grouped together, by using the first parameters for the number of the articles grouped together and the second parameters.

A non-transitory computer readable recording medium having a program recorded thereon, the program causing a computer to execute processing of:

checking a captured image against an identification image for identifying objects; and

in a case where a plurality of objects are identified to be included in the captured image, determining whether or not the identified objects are articles grouped together under a predetermined condition,

by usinga first parameter indicating a geometric relation between a first position in a first space being an image space of the identification image and a second position corresponding to the first position in a second space, the second space representing an image space in a case where objects indicated by the identification image are articles grouped together under the predetermined condition, anda second parameter indicating a geometric relation between the first position and a third position corresponding to the first position on the captured image.
(Supplementary Note 10)

The recording medium according to Supplementary Note 9, wherein

the second position is a position of each of the articles grouped together, and the determining further comprising:

calculating the second parameters for the number of the identified objects, based on a third position placed each of the identified objects, and

determining whether or not the identified objects are the articles grouped together, by using the first parameters for the number of the articles grouped together and the second parameters.

A calculation device comprising:

extraction means for extracting feature information indicating a feature of an object, from an identification image for identifying the object; and

calculation means for, in a case where a plurality of objects are identified on a captured image by a check using the identification image, calculating a first parameter used along with a second parameter when determining whether or not the plurality of identified objects are articles grouped together under a predetermined condition, based on the extracted feature information,the first parameter indicating a geometric relation between a first position in a first space being an image space of the identification image and a second position related to the first position in a second space representing an image space when a plurality of objects indicated by the identification image are articles grouped together under the predetermined condition, andthe second parameter indicating a geometric relation between the first position and a third position related to the first position on the captured image.
(Supplementary Note 12)

A system comprising:

the object identification device according to any one of Supplementary Notes 1 to 6; and

the calculation device according to Supplementary Note 11.

REFERENCE SIGNS LIST