Patent ID: 12205350

DESCRIPTION OF EMBODIMENTS

Embodiments of the present invention will be described below with reference to the drawings.

FIG.1is a block diagram illustrating a specific exemplary functional configuration of an object-specific keypoint separation apparatus10according to the present invention. The object-specific keypoint separation apparatus10is an apparatus that separates keypoints of objects that are persons captured in an image (hereinafter referred to as a “captured image”) for each of the objects. More specifically, the object-specific keypoint separation apparatus10separates the keypoints for each of the objects using the captured image and a pre-trained model generated from machine learning. A keypoint of an object in the present embodiment is a part defined for an object such as a joint, an eye, an ear, and the nose of the object.

The pre-trained model in the present embodiment is model data trained to receive a captured image as an input and output a vector field map group with a low resolution and a heat map group with a low resolution. The vector field map group with a low resolution is a set of vector field maps with a low resolution (first maps) generated using the captured image and collected for all the keypoints. The heat map group with a low resolution is a set of heat maps with a low resolution (second maps) generated using the captured image and collected for all the keypoints. An operation using the pre-trained model will now be described. Specifically, first the pre-trained model generates a vector field map with a low resolution describing the connection relationship of keypoints of the object and a heat map with a low resolution for the keypoints from the input captured image. Thereafter, the pre-trained model outputs a vector field map group with a low resolution obtained from the generated vector field maps with a low resolution and a heat map group with a low resolution obtained from the generated heat maps with a low resolution. Here, a low resolution is a resolution that is lower than the resolution of the input image. A vector field with a low resolution map is a vector field map with a resolution lower than the resolution of the input image. A heat map with a low resolution is a heat map with a resolution lower than the resolution of the input image. In the following description, a vector field map group with a low resolution will be described as a low-resolution vector field map group, and a heat map group with a low resolution will be described as a low-resolution heat map group.

A vector field map shows a vector in the direction from a child keypoint to a parent keypoint. For example, a vector describing the connection relationship of keypoints of an object stored on a vector field map is only the vector around the keypoints. A heat map is a map having a peak at the coordinates at which a keypoint of an object appears. The heat map is similar to a heat map used in object-specific keypoint separation of the related art. The present invention is characterized in that a heat map obtained by upsampling a low-resolution heat map to have an equal resolution is corrected by a coordinate value indicated by a low-resolution vector field. Here, an equal resolution is the same resolution as that of the input image. The object-specific keypoint separation apparatus10is configured using an information processing apparatus, for example, a personal computer.

The object-specific keypoint separation apparatus10includes a central processing unit (CPU), a memory, an auxiliary storage device, and the like connected to one another through a bus and executes a program. Executing the program enables the object-specific keypoint separation apparatus10to function as an apparatus including an inference execution unit101, a heat map correction unit102, an upsampling unit103, and an object-specific separation unit104. Further, all or some functions of the object-specific keypoint separation apparatus10may be implemented using hardware such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). In addition, the program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disc, a ROM, or a CD-ROM, or a storage device such as a hard disk incorporated into a computer system. In addition, the program may be transmitted and/or received via an electrical communication line.

The inference execution unit101uses a captured image and a pre-trained model as an input. The inference execution unit101uses an input captured image and pre-trained model to output a low-resolution heat map group and a low-resolution vector field map group. The inference execution unit101outputs a low-resolution heat map group to the heat map correction unit102, and outputs a low-resolution vector field map group to the heat map correction unit102and the upsampling unit103.

The heat map correction unit102receives the low-resolution heat map group and the low-resolution vector field map group as an input. The heat map correction unit102uses the input low-resolution heat map group and low-resolution vector field map group to generate a heat map group with an equal resolution. In this way, the heat map correction unit102uses the input low-resolution heat map group and low-resolution vector field map group to correct the low-resolution heat map group. The heat map correction unit102outputs the generated heat map group to the object-specific separation unit104. The heat map correction unit102is an aspect of a map correction unit.

The upsampling unit103receives the low-resolution vector field map group as an input. The upsampling unit103upsamples the input low-resolution vector field map group. For example, the upsampling unit103upsamples the input low-resolution vector field map group so that the low-resolution vector field map group has an equal resolution. The technique of bicubic interpolation may be used for upsampling. The upsampled low-resolution vector field map will be described as a vector field map. The upsampling unit103outputs a vector field map group to the object-specific separation unit104.

The object-specific separation unit104receives the heat map group and the vector field map group as an input. The object-specific separation unit104uses the input heat map group and vector field map group to separate keypoints for each object. The object-specific separation unit104separates the keypoints set in a tree-shaped hierarchical structure for each object and outputs a coordinate group indicating the separation result (a coordinate group of the keypoints separated for each object) to the outside.

FIG.2is a block diagram illustrating a specific exemplary functional configuration of a training apparatus20according to the present invention.

The training apparatus20is an apparatus that generates a pre-trained model to be used by the object-specific keypoint separation apparatus10. The training apparatus20is communicably connected to the object-specific keypoint separation apparatus10.

The training apparatus20includes a CPU, a memory, an auxiliary storage device, and the like connected to one another through a bus and executes a program. Executing the program enables the training apparatus20to function as an apparatus including a training model storage unit201, a training data input unit202, and a training unit203. Further, all or some functions of the training apparatus20may be realized using hardware such as an ASIC, a PLD, or an FPGA. In addition, the program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disc, a ROM, or a CD-ROM, or a storage device such as a hard disk incorporated in a computer system. In addition, the program may be transmitted and/or received via an electrical communication line.

The training model storage unit201is configured using a storage device such as a magnetic storage device or a semiconductor storage device. The training model storage unit201stores a training model for machine learning in advance. Here, the training model is information representing a machine learning algorithm used to train a relationship between input data and output data. Although there are various learning algorithms for supervised learning including various regression analysis methods, a decision tree, a k-nearest neighbor method, a neural network, a support vector machine, deep learning, and the like, a case in which deep learning is used will be described in the present embodiment. Further, for the learning algorithm, another training model described above may be used.

The training data input unit202has a function of randomly selecting samples from a plurality of pieces of input training data and outputting the selected samples to the training unit203. The training data is data for learning used in supervised learning and is data represented by a combination of input data and output data that is assumed to be correlated with the input data. Here, the input data is a captured image, and the output data is a low-resolution heat map group and a low-resolution vector field map group for the keypoints paired with the captured image.

The training data input unit202is communicably connected to an external apparatus (not illustrated) storing a training data group and receives the training data group as an input from the external apparatus via the communication interface of the apparatus. In addition, for example, the training data input unit202may be configured to receive the training data group as an input by reading the training data group from a recording medium (for example, a universal serial bus (USB) memory, a hard disk, or the like) storing the training data group in advance.

The training unit203generates a pre-trained model by performing training so as to minimize a difference between a first set of a low-resolution heat map group for keypoints and a low-resolution vector field map group for the keypoints obtained by converting the captured image of the training data output from the training data input unit202in accordance with the training model and a second set of a low-resolution heat map group for the keypoints and a low-resolution vector field map group for the keypoints in the training data. The generated pre-trained model is input to the object-specific keypoint separation apparatus10. Further, the input of the pre-trained model to the object-specific keypoint separation apparatus10may be performed through communication between the object-specific keypoint separation apparatus10and the training apparatus20, or may be performed using a recording medium on which the pre-trained model has been recorded.

FIG.3is a diagram illustrating specific exemplary processing of the object-specific keypoint separation apparatus10according to an embodiment. An image21illustrated inFIG.3is a low-resolution heat map of the right elbow of the low-resolution heat map group output from the inference execution unit101. A region211in the image21is a region of the right wrist, and a region212is a region of the right elbow. Here, it is assumed that the right wrist is a child keypoint and the right elbow is a parent keypoint. In this case, a vector field map in the direction to the parent keypoint (right elbow) from the child keypoint (right wrist) is seen as in an image22. The image22is a low-resolution vector field map of the low-resolution heat map group output from the inference execution unit101with the direction of the right elbow seen from the right wrist.

An image23inFIG.3represents a heat map obtained by upsampling the resolution of the region212of the right elbow to have the same resolution as that of the input image. It is assumed that the correct coordinates of the keypoints in the image23are coordinates indicated by a region231and the peak coordinates of the upsampled heat map are the coordinates indicated by a region232. In a case in which the coordinates indicating the peak value when the low-resolution heat map is upsampled using bicubic interpolation deviates from the correct coordinates to a predetermined range or more, accuracy deteriorates in estimation of keypoint coordinates. Thus, in the present invention, a vector density map indicating a degree of vector density is generated from the vectors describing the connection relationships between the keypoints and the child keypoints, and the final keypoints are determined using the synthesis values with respect to the upsampled low-resolution heat map. As a result, the coordinates can be closer to the correct coordinates, and thus accuracy deterioration can be reduced in estimation of keypoint coordinates.

A vector density map24which indicates a degree of vector density from vectors describing the connection relationships between the keypoints and the child keypoints is generated from the low-resolution vector field map (image22). A specific generation method for the vector density map24will be described below. A point241on the vector density map24represents the coordinates of the parent keypoint indicated by each vector on the low-resolution vector field map. In addition, the object-specific keypoint separation apparatus10multiplies each pixel of the image23(heat map) by the value of (1.0−α), and multiplies each pixel of the vector density map24by the value of α. Further, it is assumed that the synthesis ratio α for the vector density map24and the upsampled low-resolution heat map (image23) is a variable value in accordance with a degree of vector density. For example, the value of a is set to be high (a value closer to 1) if the convergence density is significantly high (equal to or higher than a threshold value), and the value of a is set to be low (a value closer to 0) if the convergence density is low (lower than the threshold value). Then, the object-specific keypoint separation apparatus10acquires a final heat map25by adding the multiplication result. Then, a peak position251on the heat map25is the position of the keypoint. In the object-specific keypoint separation apparatus10, a heat map group is generated by performing the above-described processing on the low-resolution heat map group output from the inference execution unit101.

FIG.4is a flowchart showing the processing of the object-specific keypoint separation apparatus10according to the embodiment.

The inference execution unit101receives a captured image and a pre-trained model from the outside as an input (step S101). The captured image and the pre-trained model do not need to be input at the same timing. In a case in which the inference execution unit101has already acquired the pre-trained model from the training apparatus20before starting the processing ofFIG.4, the inference execution unit101receives only the captured image as an input in the processing of step S101.

Inputting the captured image into the pre-trained model that has been input causes the inference execution unit101to output a low-resolution heat map group and a low-resolution vector field map group of the object captured in the captured image (step S102). The inference execution unit101outputs the low-resolution heat map group to the heat map correction unit102. The inference execution unit101outputs the low-resolution vector field map group to the heat map correction unit102and the upsampling unit103.

The heat map correction unit102upsamples the low-resolution heat map group output from the inference execution unit101to be an image with the equal resolution (step S103). For example, the heat map correction unit102upsamples the low-resolution heat map group to be an image with the equal resolution using bicubic interpolation. The heat map correction unit102initializes a vector density map (step S104). That is, the heat map correction unit102initializes the value of each pixel of the vector density map to 0 to generate a vector density map group based on the low-resolution vector field map group.

The heat map correction unit102first selects one low-resolution heat map of the plurality of input low-resolution heat map groups. Next, the heat map correction unit102generates a vector density map using the selected low-resolution heat map and a low-resolution vector field map corresponding to the selected low-resolution heat map. Specifically, the heat map correction unit102performs addition and synthesis on the vector density map based on a normal distribution set with a predetermined variance value around the coordinates indicated by each vector on the low-resolution vector field map indicating the direction of the keypoints indicated by the low-resolution heat map (step S105).

FIG.5is a diagram illustrating exemplary generation of the vector density map according to the embodiment. The image21and the image22shown inFIG.5are the same images as those shown inFIG.3. The heat map correction unit102generates an image26using the low-resolution vector field map shown as the image22. A point261in the image26represents the coordinates of the parent keypoint indicated by each vector on the low-resolution vector field map. A region262in the image26represents coordinates on the map. Then, the heat map correction unit102calculates a coordinate value V assigned to each coordinate of the vector density map based on the following equation (1).

[Math.1]V=∑n=1Ne-Pn-C22σ2(1)

In equation (1), N represents the number of coordinates of the parent keypoint indicated by each vector on the low-resolution vector field map, C represents respective coordinates of the vector density map, and σ2represents a known variance value. The heat map correction unit102assigns the calculated coordinate value V of respective coordinates to respective coordinates of the initialized vector density map to generate a vector density map. The vector density map generated in the processing is the image24inFIG.5.

The heat map correction unit102performs normalization by dividing the value on the vector density map by the maximum value in the generated vector density map so that the maximum value is 1.0 (step S106). Thereafter, the heat map correction unit102multiplies the normalized vector density map and the upsampled heat map by a predetermined synthesis ratio, respectively.

The heat map correction unit102multiplies each pixel of the upsampled heat map by a value of (1.0−α) and multiplies each pixel of the vector density map by the value of a. For example, the heat map correction unit102compares differences between the average values of samples in a given distance from the coordinates indicating the peaks of the vector density map and the upsampled heat map and the peak values, and sets a value of a to 0.5 or greater if the difference value indicating the vector density map is greater than or equal to a threshold value.

Then, the heat map correction unit102adds and synthesizes one multiplied vector density map and one multiplied heat map to generate one final heat map (step S107). The heat map correction unit102outputs the generated heat map to the object-specific separation unit104. The heat map correction unit102performs the processing of steps S103to S107on all of the low-resolution heat map groups and the low-resolution vector field map groups output from the inference execution unit101. As a result, a plurality of heat map groups are output from the heat map correction unit102to the object-specific separation unit104. The heat map correction unit102determines whether the processing of step S103to step S107has been performed on all of the low-resolution heat map groups and the low-resolution vector field map groups output from the inference execution unit101(step S108).

If the processing of step S103to step S107has been performed on all of the low-resolution heat map groups and the low-resolution vector field map groups output from the inference execution unit101(Yes in step S108), the object-specific keypoint separation apparatus10performs the processing of step S109.

On the other hand, if the processing of step S103to step S107has not been performed on all of the low-resolution heat map groups and the low-resolution vector field map groups output from the inference execution unit101(No in step S108), the object-specific keypoint separation apparatus10performs the processing of step S103to step S107on the low-resolution heat maps and the low-resolution vector field maps that have not been processed.

The upsampling unit103upsamples all of the low-resolution vector field map groups output from the inference execution unit101to be an image with the equal resolution (step S109). For example, the upsampling unit103upsamples the low-resolution vector field map groups to be an image with the equal resolution using bicubic interpolation. The upsampling unit103outputs the upsampled vector field map groups to the object-specific separation unit104.

The object-specific separation unit104uses the heat map groups output from the heat map correction unit102and the vector field map groups output from the upsampling unit103to separate the keypoints for each object (step S110). The object-specific separation unit104outputs a coordinate group of the keypoints separated for each object.

According to the object-specific keypoint separation apparatus10configured as described above, it is possible to reduce accuracy deterioration in estimation of a keypoint in object-specific keypoint separation while reducing a computational load. Specifically, the object-specific keypoint separation apparatus10outputs a low-resolution heat map group and a low-resolution vector field map group using the input image. In addition, the object-specific keypoint separation apparatus10corrects a heat map group obtained by upsampling a low-resolution heat map group to have the equal resolution with a coordinate value indicated by a low-resolution vector field map. As a result, even in a case in which an image captured using a high-resolution camera is input, a heat map with the equal resolution and a vector field map are not generated. Therefore, a computational load can be reduced. Furthermore, the object-specific keypoint separation apparatus10generates a vector density map indicating a degree of vector density from vectors each describing a connection relationship between a keypoint and a child keypoint, determines a final keypoint using a synthesis value with respect to the upsampled low-resolution heat map, and thus the coordinates of the keypoint can be closer to the correct coordinates. As a result, accuracy deterioration can be reduced in estimation of the keypoint coordinates.

Modified Example

The object-specific keypoint separation apparatus10and the training apparatus20may be configured to be integrated. Specifically, the object-specific keypoint separation apparatus10may be configured to have the learning function of the training apparatus20. With this configuration, the object-specific keypoint separation apparatus10has a learning mode and an inference mode, and performs operations corresponding to each of the modes. Specifically, in the learning mode, the object-specific keypoint separation apparatus10generates a pre-trained model by performing the same processing as that performed by the training apparatus20. In the inference mode, the object-specific keypoint separation apparatus10executes the processing shown inFIG.4using the generated pre-trained model.

The processing of step S105inFIG.4above may be performed in other ways. For example, for each coordinate on the vector density map, 1.0/τL, which is the reciprocal of the sum ΣL of the distances L between the coordinates indicated by each vector on the low-resolution vector field map and the correct coordinates of the keypoints, and the like may be used.

FIG.6is a diagram illustrating exemplary generation of a vector density map. The image21and the image22shown inFIG.6are the same images as those shown inFIG.3. The heat map correction unit102generates an image26using the low-resolution vector field map shown as the image22. Then, the heat map correction unit102calculates a coordinate value V assigned to respective coordinates of the vector density map based on the following equation (2).

[Math.2]V=∑n=1N1.Pn-C2(2)

As such, the processing of step S105inFIG.4can be substituted with a method of scaling the density of vectors.

Although the example in which accuracy deterioration is reduced in estimation of keypoint coordinates even in a situation where a low-resolution heat map and a low-resolution vector field map are used has been described in the embodiments described above, a vector density map may be used to improve accuracy in estimation of a heat map with an equal resolution by using a heat map with an equal resolution and a vector field with an equal resolution.

INDUSTRIAL APPLICABILITY

The present invention can be applied to a technology to separate keypoints of objects detected from an image capturing the objects for each object.

REFERENCE SIGNS LIST

10Object-specific keypoint separation apparatus20Training apparatus101Inference execution unit102Heat map correction unit103Upsampling unit104Object-specific separation unit201Training model storage unit202Training data input unit203Training unit