An apparatus includes an extraction unit configured to extract a feature amount from each of a plurality of pieces of input data, a calculation unit configured to calculate, based on an identification model for identifying to which one of a plurality of labels each of the plurality of pieces of input data belongs, which is generated using the feature amount, a likelihood indicating how likely each of the plurality of pieces of input data belongs to the labels, and a presenting unit configured to present attribute information about the input data based on the feature amount and the likelihood.

BACKGROUND OF THE INVENTION

Field of the Invention

Aspects of the present invention relate to an information processing apparatus, an information processing method, and a storage medium.

Description of the Related Art

In Japanese Patent Application Laid-Open No. 2010-54346, a neural network is used to calculate an identification criterion for classifying a plurality of types of defects. In Japanese Patent Application Laid-Open No. 2010-54346, data that indicates a type of a defect is automatically extracted on a space constituted by two feature amounts determined by a user, and the user instructs a defect type with respect to the extracted data to update the identification criterion.

In Japanese Patent Application Laid-Open No. 2010-54346, the identification criterion is calculated based on data to which a label of a few defect types is given, and the data distribution on the feature space constituted by the two feature amounts determined by the user and the identification criterion for classifying defects in the feature space are presented to the user. However, when a data distribution and an identification criterion are presented to the user, the user can understand a space of up to three dimensions. Thus, in a case where an identification criterion is calculated using four or more feature amounts, there arises a situation that a data distribution on the feature space cannot be displayed.

SUMMARY OF THE INVENTION

According to an aspect of the present invention, an apparatus includes an extraction unit configured to extract a feature amount from each of a plurality of pieces of input data, a calculation unit configured to calculate, based on an identification model for identifying to which one of a plurality of labels each of the plurality of pieces of input data belongs, which is generated using the feature amount, a likelihood indicating how likely each of the plurality of pieces of input data belongs to the labels, and a presenting unit configured to present attribute information about the input data based on the feature amount and the likelihood.

Further features of aspects of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.

DESCRIPTION OF THE EMBODIMENTS

In a first exemplary embodiment of aspects of the present invention, images of a specific inspection target object are captured, and whether the inspection target object is normal is identified based on the captured images. In the present exemplary embodiment, feature amounts serving as elements for the identification between normal and abnormal are calculated from the images. A likelihood indicating how likely the inspection target object is to be normal, which is to be a criterion for the identification between normal and abnormal, is calculated based on the feature amounts calculated from a plurality of normal images and a plurality of abnormal images.

Meanwhile, when a data distribution on a feature space is visualized, in a case where only the data distribution on the feature space is visualized, the likelihood of data that is an identification criterion is not taken into consideration. Thus, although two pieces of neighboring data in the visualized result may have completely different likelihoods, the user may erroneously determine that pieces of neighboring data in the visualized result have close likelihoods. In view of the foregoing, in the present exemplary embodiment, a data distribution on a feature space is visualized while taking the likelihood of data, in addition to a distance relationship on the feature space, into consideration. In this way, the data distribution on the feature space and the identification performance based on the identification criterion can simultaneously be presented.

FIG. 1is a diagram illustrating an example of a presentation result by an information processing apparatus according to the present exemplary embodiment. The information processing apparatus is for simultaneously visualizing a data distribution on a feature space constituted by a plurality of feature amounts, and a likelihood that is an identification criterion for the identification between normal and abnormal. InFIG. 1, axes105and106of a visualized space indicate bases for displaying a visualized result. Details of the bases will be described below. Further, distances between respective pieces of data reflect the positional relationships on the feature space. A contour line103indicates positional coordinates of the same likelihood. The information processing apparatus displays a presentation result as illustrated inFIG. 1, thereby simultaneously presenting the positional relationships between normal data100and abnormal data101on the feature space, and the likelihoods. On the other hand, the technique discussed in Japanese Patent Application Laid-Open No. 2010-54346 displays a feature space and an identification criterion on the feature space, so that, when the feature space exceeds the number of dimensions that can directly be presented, the feature space cannot be displayed.

FIG. 2is a block diagram illustrating an example of a configuration of the information processing apparatus according to the present exemplary embodiment. The information processing apparatus includes a data record unit200, a feature amount extraction unit201, an identification model learning unit202, a likelihood calculation unit203, a likelihood record unit204, a data analysis processing unit205, and a presenting unit206.

FIG. 3is a flow chart illustrating a method of information processing performed by the information processing apparatus according to the present exemplary embodiment. First, in step S300, the data record unit200stores, in association with image numbers, a plurality of pieces of image data obtained by capturing images of normal inspection target objects and abnormal inspection target objects, as illustrated inFIG. 4. At this time, the data record unit200stores each of the plurality of pieces of image data in association with, a normal label indicating a piece of image data obtained by capturing a normal inspection target object, or an abnormal label indicating a piece of image data obtained by capturing an abnormal inspection target object. The feature amount extraction unit201, which is a means for extracting a feature amount, reads image data as input data from the data record unit200. The present exemplary embodiment is described while taking the images as an example. However, any data exhibiting different tendencies between a normal inspection target object and an abnormal inspection target object may be used. Examples of such data include acoustic data and data obtained by other sensors.

Next, in step S301, the feature amount extraction unit201calculates a feature amount that is to be an element for the identification between normal and abnormal, with respect to each of the pieces of image data stored in the data record unit200. While there are various examples of a feature amount, statistics such as mean, variance, skewness, kurtosis, mode, entropy, etc. of luminance values of the images are used in the present exemplary embodiment. Besides the foregoing examples, a texture feature amount using a co-occurrence matrix, a local feature amount using scale-invariant feature transform (SIFT) can be used. The feature amount extraction unit201extracts an N-dimensional feature amount with respect to all of the pieces of the normal image data and the abnormal image data that are stored in the data record unit200.

Next, in step S302, the identification model learning unit202, which is a means for learning an identification model, calculates parameters of an identification model by use of a given identification model for the separation between normal data and abnormal data and the feature amounts calculated by the feature amount extraction unit201. More specifically, the identification model learning unit202learns (generates), using the feature amounts, an identification model for identifying to which one of the normal label and the abnormal label each of the plurality of pieces of image data belongs. In the present exemplary embodiment, the Mahalanobis distance is used as the identification model. The identification model learning unit202calculates the mean and the variance-covariance matrix using the feature amounts extracted from the pieces of image data stored in association with the normal label in the data record unit200. In this way, the identification can be made in such a manner that the smaller a Mahalanobis distance calculated using a feature amount extracted from data of an arbitrary image, the more likely the arbitrary image is normal. On the other hand, the identification can be made in such a manner that the greater a Mahalanobis distance calculated using a feature amount extracted from data of an arbitrary image, the more likely the arbitrary image is abnormal. An N-dimensional feature amount extracted by the feature amount extraction unit201from a piece of image data stored in the data record unit200is denoted by ci(i is the image number). A mean value and a variance-covariance matrix that are calculated using only the feature amounts extracted from the pieces of image data stored in association with the normal labels are denoted by μ and σ, respectively. The identification model learning unit202calculates the mean value μ and the variance-covariance matrix σ as the parameters of the identification model. While the Mahalanobis distance is used as the identification model in the present exemplary embodiment, any identification model by which the identification between normal and abnormal can be made may be used. Examples of such an identification model include one-class support vector machines (SVM) and k-nearest neighbor.

Next, in step S303, the likelihood calculation unit203, which is a means for calculating a likelihood, calculates a likelihood L(ci), which indicates how likely an image stored in the data record unit200is to be normal, by use of the identification model calculated by the identification model learning unit202. More specifically, first, the likelihood calculation unit203calculates a Mahalanobis distance D(ci) for the N-dimensional feature amount ciusing the mean value μ and the variance-covariance matrix σ that have been calculated by the identification model learning unit202using only the feature amounts extracted from the pieces of image data stored in association with the normal labels, as specified by formula (1) below. In formula (1), T represents the transpose of the matrix, and σ1represents the inverse of the variance-covariance matrix o.

Next, the likelihood calculation unit203calculates the likelihood L(ci) using the Mahalanobis distance D(ci) as specified by formula (2) below. In formula (2), Z represents a normalization coefficient. In other words, the likelihood calculation unit203calculates, with respect to each of the plurality of pieces of data, the likelihood L(ci) that indicates how likely each of the plurality of pieces of data belongs to the normal label, which is a first label, using the feature amount ciand the mean value μ of the feature amounts extracted from the data belonging to the normal label that is the first label.

Next, as illustrated inFIG. 5, the likelihood record unit204stores the likelihood L(ci) calculated for the feature amount ciby the feature amount extraction unit201, in association with the image number used by the data record unit200inFIG. 4. While the likelihood record unit204stores the likelihood L(ci) separately from the data record unit200, the likelihood L(ci) may be recorded in any form as long as the likelihood L(ci) is stored in such a manner that the feature amount ciis associated with the likelihood L(ci).

Next, in step S304, if the feature amount ciand the likelihood L(ci) are data having greater dimensions than three dimensions, the data analysis processing unit205, which is a means for processing data analysis, reduces the number of dimensions and calculates positional coordinates on a space of three or fewer dimensions. More specifically, the data analysis processing unit205calculates positional coordinates of each of the plurality of pieces of data on the visualized space in order to simultaneously visualize the relationship between the pieces of data on the feature space and the likelihood L(ci) that is the identification criterion. For example, the data analysis processing unit205calculates the positional coordinates of the data on the visualized space by use of a unified vector ui=[ci, L(ci)] obtained by combining the feature amount cicalculated by the feature amount extraction unit201and the likelihood L(ci) stored in the likelihood record unit204.

For example, the data analysis processing unit205performs the visualization so that an index S, which is referred to as “stress” and specified by formula (3) below, is minimized.

In formula (3), M represents the number of pieces of data to be visualized. As specified by formula (4) below, d1ijrepresents the distance between the i-th data and the j-th data on the visualized space.

As illustrated inFIG. 1, the data analysis processing unit205determines the visualized space as a two-dimensional space and calculates the distance d1ijbetween the i-th data and the j-th data on the visualized space using the Euclidean distance. In the present exemplary embodiment, the coordinates of the i-th data on the visualized space are vi=[xi, yi]T, and the coordinates of the j-th data on the visualized space are vj=[xj, yj]T. In this case, the axis105of the visualized space is the coordinate axis for the positions of xiand xj, and the axis106of the visualized space is the coordinate axis for the positions of yiand yj.

Further, dijrepresents the dissimilarity between the i-th data and the j-th data. In general, the dissimilarity dijis calculated using the positional relationship on the feature space. Thus, the dissimilarity dijis calculated using the feature amount ciof the i-th data and the feature amount cjof the j-th data. However, if the dissimilarity dijis calculated using only the positional relationship on the feature space, the positional relationship between the pieces of data that is expressed on the visualized space does not reflect the likelihood L(ci) that is the identification criterion. Thus, the data analysis processing unit205takes the likelihood L(ci) that is the identification criterion into consideration when calculating the dissimilarity diIn the present exemplary embodiment, the data analysis processing unit205calculates the dissimilarity dijusing the Euclidean distance using the unified vector ui=[ci, L(ci)] obtained by unifying the likelihood L(ci) and the feature amount ci, as specified by formula (5) below.

As the foregoing describes, the data analysis processing unit205calculates the coordinates viand vjof data on the visualized space so that the index S as specified by the formula (3) above is minimized. More specifically, the data analysis processing unit205calculates the positional coordinates viand vjof each of the plurality of pieces of data so that an error between the distance between two pieces of the data on the feature amount ciand the likelihood L(ci), and the distance between the positional coordinates of two pieces of the data on the space is minimized. At this time, the data analysis processing unit205calculates the dissimilarity dijbetween the data using the unified vectors uiand uj, whereby the positional relationship between the data on the likelihood L(ci) that is the identification criterion can be simultaneously reflected on the positional relationship between the data on the visualized space.

While the distance d1ijbetween the two pieces of data on the visualized space and the dissimilarity dijare calculated using the Euclidean distance in the present exemplary embodiment, the Mahalanobis distance, the city block distance, or the Pearson distance may be used as long as the relationship between the two pieces of data can be defined. Further, any other index may be used as the index S of formula (3) above.

Further, while the unified vectors uiand ujare used to reflect the influence of the likelihood L(ci) that is the identification criterion in the positional relationship between the data on the visualized space in the present exemplary embodiment, the present invention is not limited thereto. The index S of formula (3) above may be defined as an index that provides the influence of the likelihood L(ci) that is the identification criterion. In this case, for example, an index S1 of formula (6) below may be used in place of the index S of formula (3) above.

In formula (6), d2iis the dissimilarity between the feature amounts ciand cjof the two pieces of data and is equal to the dissimilarity dijin the case where ui=ci. Further, pijis the dissimilarity between the likelihoods L(ci) and L(cj) of the two pieces of data and is obtained by pij={L(ci)−L(cj)}2. The dissimilarities d2ijand pijcan be calculated using the Mahalanobis distance, Pearson distance, etc. Further, α is a parameter that determines the intensity of the influence of the dissimilarity on the feature space and the dissimilarity obtained using the Mahalanobis distance. As α becomes close to 0, the influences of the likelihoods L(ci) and L(cj) decrease, and the dissimilarity d2ijon the feature space is maintained. On the other hand, as a increases, the dissimilarity pijbetween the likelihoods L(ci) and L(cj) is maintained on the visualized space.

While the positional relationship between data on the visualized space is determined by the method described above in the present exemplary embodiment, the method for the determination is not limited to the method described above. Any method that can reduce the number of dimensions may be used, such as principal component analysis, Fisher's discriminant analysis, etc.

Next, in step S305, the presenting unit206, which is a presentation means, presents attribute information including the positional relationship between the data and the likelihood L(ci) that is the identification criterion using the coordinates viof the data on the visualized space that are calculated by the data analysis processing unit205. More specifically, the presenting unit206displays the positions of the positional coordinates of the respective pieces of the normal data100and the abnormal data101on the two-dimensional space, as illustrated inFIG. 1. Further, the presenting unit206displays the contour line103along the positional coordinates of the same likelihood L(ci) that is the identification criterion.

In order to display the contour line103specified inFIG. 1, the presenting unit206is to join points of the same likelihood L(ci). Meanwhile, the coordinates viof data points that are calculated by the data analysis processing unit205do not exist at regular intervals, so the presenting unit206is to interpolate points of the same likelihood L(ci). Thus, the presenting unit206performs interpolation of the likelihood L(ci) by cubic interpolation using the likelihood L(ci) of the coordinates viof data points that are calculated by the data analysis processing unit205, and joins points of the same likelihood L(ci) on the visualized space, thereby displaying the contour line103specified inFIG. 1. While the interpolation of points of the same likelihood L(ci) on the visualized space is performed using bicubic interpolation in the present exemplary embodiment, any method that enables such interpolation may be used, such as bilinear interpolation, etc.

As the foregoing describes, in the present exemplary embodiment, the likelihood L(ci), which is the identification criterion for the identification between normal and abnormal, and the feature amount that is the information to be an element for the identification between normal and abnormal can be presented simultaneously. While the identification between normal and abnormal in the one-class identification situation is described as an example in the present exemplary embodiment, an exemplary embodiment of aspects of the present invention is also applicable to a binary or multiclass identification situation. For example, in the case of a multiclass identification situation, the likelihood L(ci) is calculated for every one of the classes. Thus, the unified vector uican be realized by combining the likelihoods L1(ci) to Ln(ci) for all the classes to obtain ui=[ci, L1(ci), L2(ci), . . . , Ln(ci)]. Further, in a case where a limitation by the likelihood is to be set, the dissimilarity between the likelihood vectors may be calculated using the Euclidean distance, Mahalanobis distance, Pearson distance, etc.

An information processing apparatus according to a second exemplary embodiment of aspects of the present invention will be described below. In the first exemplary embodiment, the information processing apparatus extracts the feature amount cifrom target data and learns the identification model for the identification between normal and abnormal by use of the extracted feature amount ciIn the present exemplary embodiment, the case where input data contains data given a low-reliability normal or abnormal label will be considered. If data with an incorrect label is used in identification model learning, an appropriate identification boundary between normal and abnormal cannot be acquired, and the identification accuracy may decrease. Thus, the user corrects the given label to regive an appropriate label. By performing the identification model leaning using the regiven label, the identification model can be learned with higher identification performance.

Thus, in the present exemplary embodiment, data that may have an incorrect label is presented to the user using the feature amount ciand the likelihood L(ci) to prompt the user to give an appropriate label. At this time, not only the data that may have an incorrect label but also useful data for the correction of other labels may be presented to the user so that an appropriate label can be given. While the two types of labels that are the normal label and the abnormal label are used in the present exemplary embodiment, an exemplary embodiment of aspects of the present invention is also applicable to a case where a plurality of other labels is given. Points in which the present exemplary embodiment is different from the first exemplary embodiment will be described below.

FIG. 6is a block diagram illustrating an example of a configuration of the information processing apparatus according to a second exemplary embodiment of aspects of the present invention. The information processing apparatus includes a data record unit200, a feature amount extraction unit201, an identification model learning unit202, a likelihood calculation unit203, a likelihood record unit204, a clustering unit905, a presentation data determination unit906, a display unit907, and a label correction unit908. The data record unit200, the feature amount extraction unit201, the identification model learning unit202, the likelihood calculation unit203, and the likelihood record unit204are similar to those in the first exemplary embodiment (FIG. 2).

FIG. 7is a flow chart illustrating a method of information processing performed by the information processing apparatus according to the present exemplary embodiment. In steps S300to S303, the information processing apparatus performs processing similar to those in the first exemplary embodiment (FIG. 3). More specifically, in step S300, the feature amount extraction unit201inputs data stored in the data record unit200. Next, in step S301, the feature amount extraction unit201calculates a feature amount cifor data stored in the data record unit200. Next, in step S302, the identification model learning unit202learns using the calculated feature amount cian identification model for the identification between normal and abnormal. Next, in step S303, the likelihood calculation unit203calculates using the identification model a likelihood L(ci) for the feature amount cicalculated by the feature amount extraction unit201. The likelihood record unit204stores the likelihood L(ci).

Next, in step S1004, the clustering unit905, which is a clustering means, calculates positional coordinates of each of a plurality of pieces of data on a space based on the feature amount ciand the likelihood L(ci), as in the data analysis processing unit205illustrated inFIG. 2. Next, the clustering unit905performs data clustering using the feature amount cicalculated by the feature amount extraction unit901and the likelihood L(ci) stored in the likelihood record unit904. For example, the clustering unit905classifies the plurality of pieces of data into predetermined k pieces of clusters B1 to Bk. More specifically, the clustering unit905determines the clusters B1 to Bk to which all the pieces of data belong so that an error between the center of gravity wiof the cluster Biand the unified vector ujcontained in the cluster Bi, as specified by formula (7) below, is minimized.

As in the first exemplary embodiment, the unified vector ujis a vector obtained by combining the feature amount cjand the likelihood L(cj), and uj=[cj, L(cj)]. In this way, the feature amount cjand the likelihood L(cj) obtained using the identification model can be reflected in the clustering result.

The number of clusters k may be predetermined by the user, or data may be displayed to prompt the user to input the number of clusters k as in the first exemplary embodiment. Further, the number of clusters k may be determined by an x-means method in which the number of clusters k is determined using the Bayesian information criterion (BIC), or by any other methods. Further, besides the foregoing clustering method, any other methods may be used such as a hierarchical clustering method, etc.

Next, in steps S1005to S1007, the presentation data determination unit906, which is a means for determining presentation data, determines data the label of which is to be reconfirmed by the user, using the clusters B1 to Bk calculated by the clustering unit905. First, in step S1005, the presentation data determination unit906extracts data with a low-reliability label as a label confirmation candidate. In order to extract low-reliability data, the presentation data determination unit906is to determine what data each of the clusters B1 to Bk of the clustering result contains. Thus, the presentation data determination unit906assigns labels that occur most frequently in the clusters B1 to Bk, respectively, as labels of the clusters B1 to Bk, respectively. Then, the presentation data determination unit906extracts data having a different label from the labels assigned to the respective clusters B1 to Bk as low-reliability data.

FIG. 8is a diagram illustrating an example of a clustering result. The clustering unit905classifies, for example, a plurality of pieces of data into a plurality of clusters1100to1103. The presentation data determination unit906, for example, assigns a normal label to the cluster1100, which contains a large number of pieces of normal data100, and assigns an abnormal label to the clusters1101,1102and1103, each of which contains a large number of pieces of abnormal data101. At this time, the cluster1100assigned the normal label contains a few pieces of abnormal data1104. The presentation data determination unit906extracts such a few pieces of abnormal data1104as a label confirmation candidate. In other words, the presentation data determination unit906extracts as a label confirmation candidate the data1104belonging to the abnormal label having a smaller number of pieces of data than other normal labels, among the pieces of data belonging to the cluster1100.

Next, in step S1006, the presentation data determination unit906determines whether there is a label confirmation candidate extracted in step S1005. If there is a label confirmation candidate (YES in step S1006), the processing proceeds to step S1007. On the other hand, if there is no label confirmation candidate (NO in step S1006), the processing proceeds to step S1010, and the processing illustrated inFIG. 7is ended.

In step S1007, the presentation data determination unit906determines as presentation data the abnormal data1104extracted as a label confirmation candidate in step S1005. Meanwhile, when the abnormal data1104alone is presented to the user, it is difficult for the user to judge a label that should be given to the abnormal data1104. Thus, simultaneously present data belonging to the current cluster and data belonging to a neighborhood cluster in addition to the abnormal data1104being a label confirmation candidate is performed. For example, the presentation data determination unit906determines normal data1105located in the neighborhood of the abnormal data1104, abnormal data1106belonging to the cluster1103of the abnormal label that is located in the neighborhood of the cluster1100to which the abnormal data1104belongs, etc., as presentation data.

In the search for neighborhood data, the presentation data determination unit906does not search for neighborhood data on the feature space but searches for neighborhood data with the feature space and the likelihood taken into consideration, whereby data determined by the learned identification model as being located in the neighborhood can be presented. By presenting the neighborhood data together with the abnormal data1104being the label confirmation candidate, it becomes possible to prompt the user to input a more appropriate label.

Next, in step S1008, the display unit907, which is a presenting means, displays (presents) to the user the positions of the positional coordinates of the presentation data containing the label confirmation candidate data determined by the presentation data determination unit906on the space.

Next, in step S1009, the user performs reconfirmation of the label based on the display on the display unit907, and the label correction unit908, which is a means for correcting a label, corrects the label of the label confirmation candidate data based on an instruction from the user. If an instruction is given to correct the label to which the presentation data displayed by the display unit907belongs, the label correction unit908corrects the label to which the presentation data belongs.

Thereafter, the information processing apparatus repeats step S302and subsequent steps using the corrected label. In step S302, the identification model learning unit202relearns the identification model using the data containing the presentation data of the label corrected by the label correction unit908, whereby the identification model can be learned more appropriately.

As the foregoing describes, in the present exemplary embodiment, data with a low-reliability label can be extracted with the likelihood L(ci) that is the identification criterion taken into consideration, and a label confirmation candidate can be presented to the user.

An information processing apparatus according to a third exemplary embodiment of aspects of the present invention will be described below. In the first exemplary embodiment, the information processing apparatus extracts the feature amount cifrom target data and learns the identification model for the identification between normal and abnormal by use of the extracted feature amount ci. Then, the information processing apparatus calculates the likelihood L(ci) of the data using the identification model and simultaneously displays the data distribution and the contour line103of the likelihood L(ci) on the feature space. The present exemplary embodiment will consider a case where a label given to input data is reliable but the number of pieces of data is insufficient. An example is a state in which a plurality of types of abnormal patterns exists in abnormal data. When a plurality of types of abnormal patterns exists in abnormal data, there may be a case where the number of pieces of data of an abnormal pattern is sufficient while the number of pieces of data of another abnormal pattern is extremely small. In such a case, the identification performance for the abnormal pattern that is small in the number of data decreases.

Thus, in the present exemplary embodiment, the information processing apparatus prompts the user to add data necessary for improving the identification performance by use of the data distribution on the feature space and the likelihood L(ci). The information processing apparatus enables the user to select abnormal data104close to normal data from the visualized result and confirm data to be added, as illustrated inFIG. 1. Further, the information processing apparatus can display additional data and a trend of the data without requiring user selection. Points in which the present exemplary embodiment is different from the second exemplary embodiment will be described below.

FIG. 9is a block diagram illustrating an example of a configuration of the information processing apparatus according to the third exemplary embodiment of aspects of the present invention. The information processing apparatus illustrated inFIG. 9is different from the information processing apparatus illustrated inFIG. 6in that an additional data input unit608and an additional data record unit609are provided in place of the label correction unit908.

FIG. 10is a flow chart illustrating a method of information processing performed by the information processing apparatus according to the present exemplary embodiment. In steps S300to S303and S1004, the information processing apparatus performs processing similar to those in the second exemplary embodiment (FIG. 7). More specifically, in step S300, the feature amount extraction unit201inputs data stored in the data record unit200. Next, in step S301, the feature amount extraction unit201calculates a feature amount cifor data stored in the data record unit200. Next, in step S302, the identification model learning unit202learns using the calculated feature amount cian identification model for the identification between normal and abnormal. Next, in step S303, the likelihood calculation unit203calculates using the identification model a likelihood L(ci) for the feature amount cicalculated by the feature amount extraction unit201. The likelihood record unit204stores the likelihood L(ci). Next, in step S1004, the clustering unit905classifies a plurality of pieces of data into k pieces of clusters B1 to Bk by data clustering using the likelihood L(ci) and the feature amount ci.

Next, in step S705, the presentation data determination unit906assigns labels that occur most frequently in the clusters B1 to Bk, respectively, as labels of the clusters B1 to Bk, respectively. Then, the presentation data determination unit906determines from a result of the clustering performed by the clustering unit905a cluster lacking in data for leaning the identification model. Then, the presentation data determination unit906determines data to be presented to the user as similar data of the cluster lacking in data from the cluster lacking in data.

FIG. 11Ais a diagram illustrating an example of a clustering result. The clustering unit905, for example, classifies a plurality of pieces of data into clusters800to803. The presentation data determination unit906, for example, assigns a normal label to the cluster800, which contains a large number of pieces of normal data100, and assigns an abnormal label to the clusters801,802, and803, each of which contains a large number of pieces of abnormal data101.

The presentation data determination unit906determines a cluster lacking in data for the learning of the identification model. For example, the presentation data determination unit906determines as a cluster lacking in data the cluster800to which the normal label is assigned and that contains abnormal data804. In the cluster800, the identification between normal and abnormal is not adequately conducted, and there exists abnormal data804causing the identification accuracy to decrease. The cluster800contains a large number of pieces of normal data100and a small number of pieces of abnormal data804. The abnormal data804classified into the cluster800to which the normal label is assigned is data causing the identification performance to decrease. The presentation data determination unit906determines the cluster800to which the abnormal data804belongs as a cluster lacking in data.

In order to determine a cluster lacking in data, the presentation data determination unit906is to set the normal cluster800to which a large number of pieces of normal data100belong. Thus, the presentation data determination unit906determines as a normal cluster the cluster800to which the largest number of pieces of normal data100belong. In the present exemplary embodiment, it is assumed that there is one normal cluster among all the clusters. However, there may be a case where two or more normal clusters exist. In such a case, two or more normal clusters may be set. For example, a cluster to which a large number of pieces of normal data belong among 80 or higher percent of the total number of pieces of normal data may be determined as a normal cluster.

Next, the presentation data determination unit906extracts the abnormal data804belonging to the normal cluster800. More specifically, the presentation data determination unit906extracts the data804belonging to the abnormal label having a smaller number of pieces of data than other normal labels, among the pieces of data belonging to the cluster800. Then, the presentation data determination unit906determines as a cluster lacking in data the normal cluster800to which the extracted abnormal data804belongs.

Next, in step S706, if there is no cluster lacking in data (NO in step S706), the processing is ended in step S710. On the other hand, if there is a cluster lacking in data (YES in step S706), the processing proceeds to step S707.

In step S707, the presentation data determination unit906determines the abnormal data804extracted in step S705as presentation data. The abnormal data804extracted in step S705is the data determined as belonging to the normal cluster800. Thus, the abnormal data804has a small difference from the normal data. When the abnormal data804having a small difference from the normal data is presented to the user, it is difficult for the user to judge data that is appropriate as additional data. In order to present a trend of additional data as appropriate to the user, data is presented apart from the normal cluster800and simultaneously present data from which the user can clearly understand a difference. By presenting the abnormal data804together with the data from which the user can understand a difference with ease, it becomes possible to prompt the user to add data that is effective for improving the identification performance.

As to the presentation data, data that has the same abnormal pattern as that of the extracted abnormal data804and is located apart from the normal cluster800may be needed. In order to select such data, the cluster803to which the abnormal data804is supposed to belong is determined. Thus, the presentation data determination unit906performs clustering of abnormal data excluding normal data from all the data illustrated inFIG. 11Aand generates abnormal data clusters805to807as illustrated inFIG. 11B. Next, the presentation data determination unit906determines the abnormal data cluster807to which the extracted abnormal data804belongs as a cluster to which the extracted abnormal data804is supposed to belong. Then, the presentation data determination unit906determines data to be presented other than the extracted abnormal data804from the abnormal data belonging to the abnormal data cluster807. Abnormal data808located in the neighborhood of the extracted abnormal data804among the data belonging to the abnormal data cluster807may be presented as presentation data. In this way, a plurality of pieces of similar data can be presented to present to the user more information about data that needs to be added. Further, as another method, abnormal data809located at a great distance from the extracted abnormal data804, abnormal data810close to the center of gravity811of the abnormal data cluster807, etc. in the same abnormal data cluster807may be determined as presentation data. Any selection method may be used by which data that can provide more information to the user can be selected.

Further, not only the abnormal data cluster807to which the extracted data804belongs but also data belonging to another abnormal data cluster806located in the neighborhood may be determined as presentation data. In this case, as a comparison, presentation data is determined as data of the cluster806different from the abnormal data cluster807that requires additional data. By presenting such data, the difference from originally needed data becomes clearer to the user.

In the present exemplary embodiment, the cluster807to which the extracted abnormal data804is supposed to belong is determined by the clustering. As to other methods, for example, if a label other than the normal label and the abnormal label is assigned as input data, the cluster to which the extracted abnormal data is supposed to belong may be determined using the label information.

Next, in step S708, the display unit907displays (presents) to the user the position of the positional coordinates of the presentation data containing the abnormal data804extracted by the presentation data determination unit606on the space and prompts the user to input additional data.

Next, in step S709, the additional data input unit608receives input of additional data from the user. In the present exemplary embodiment, the user inputs data close to the abnormal data804displayed by the display unit607. The additional data record unit609stores the input data in the format illustrated inFIG. 4. Thereafter, the processing returns to step S301, and the information processing apparatus repeats the learning of the identification model again using the data stored in the data record unit200and the additional data record unit609. In other words, if data is added based on the display by the display unit607, the feature amount extraction unit201extracts a feature amount cifrom the added input data, and the identification model learning unit202learns the identification model using the feature amount ciof the added data. In this way, the identification model is learned with the additional data taken into consideration so that the likelihood L(ci) that is the identification criterion can be calculated more appropriately and the clustering is performed as appropriate. For example, as illustrated inFIG. 11B, the appropriate abnormal data cluster807to which the abnormal data804belongs can be generated.

In the present exemplary embodiment, in step S706, the processing is repeated until the presentation data determination unit906determines that there is no cluster lacking in data. Further, if the user selects not to input additional data, the processing proceeds to step S710to end the processing.

As the foregoing describes, in the present exemplary embodiment, the clustering is performed using the likelihood L(ci), which is an identification criterion, in addition to the feature amount ciof data so that the influence of the identification model can be taken into consideration to present to the user the image data that is effective as additional data.

In the first to third exemplary embodiments, the data distribution on the feature space and the likelihood that is the identification criterion can be displayed simultaneously even in the case where feature amounts of four or greater dimensions are used. Further, in the second and third exemplary embodiments, data that is effective for improving the identification performance can be presented to the user based on the data distribution on the feature space and the likelihood that is the identification criterion.

The foregoing exemplary embodiments are mere illustration of examples of implementation of aspects of the present invention, and the interpretation of the technical scope of aspects of the present invention should not be limited by the disclosed exemplary embodiments. In other words, aspects of the present invention can be implemented in various forms without departing from the spirit features thereof.

OTHER EMBODIMENTS

While aspects of the present invention have been described with reference to exemplary embodiments, it is to be understood that aspects of the invention are not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No. 2015-204016, filed Oct. 15, 2015, which is hereby incorporated by reference herein in its entirety.