Data analysis apparatus, data analysis method, and data analysis program

A data analysis apparatus executes: a selection process selecting a first feature variable group that is a trivial feature variable group contributing to prediction and a second feature data group other than the first feature variable group from a set of feature variables; an operation process operating a first regularization coefficient related to a first weight parameter group corresponding to the first feature variable group in a manner that the loss function is larger, and operating a second regularization coefficient related to a second weight parameter group corresponding to the second feature variable group in a manner that the loss function is smaller, among a set of weight parameters configuring a prediction model, in a loss function related to a difference between a prediction result output in a case of inputting the set of feature variables to the prediction model and ground truth data corresponding to the feature variables.

CLAIM OF PRIORITY

The present application claims priority from Japanese patent application JP 2019-100316 filed on May 29, 2019, the content of which is hereby incorporated by reference into this application.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a data analysis apparatus, a data analysis method, and a data analysis program for analyzing data.

2. Description of the Related Art

Machine learning is one of technologies for realizing artificial intelligence (AI). The machine learning technology is configured with a learning process and a prediction process. First, in the learning process, learning parameters are calculated in such a manner that an error between a predicted value obtained from a feature variable vector that is an input and an actual value (true value) is minimum. Subsequently, in the prediction process, a new predicted value is calculated from data not used in learning (hereinafter, referred to as test data).

Learning parameter calculation methods and learning parameter computing methods to attain maximum prediction accuracy have been invented so far. With an approach called perceptron, for example, a predicted value is output on the basis of a computing result of linear combination between an input feature variable vector and a weight vector. A neural network is also called multilayer perceptron and capable of solving linearly inseparable problems by using a plurality of perceptrons in a multi-layered fashion. Deep learning is an approach of introducing new technologies such as dropout to the neural network and stepped into the limelight as an approach capable of achieving high prediction accuracy.

In this way, development of machine learning technologies has been underway so far with a view to improving prediction accuracy. There is known an approach, other than the development of the machine learning technologies, of improving prediction accuracy by selecting data for use in learning in advance as disclosed in International Publication No. WO2010/016110. According to International Publication No. WO2010/016110, feature variables important for prediction are selected by making use of the possibility of using a magnitude of each element value of the weight vector that is one of the learning parameters as an importance degree of each feature variable contributing to prediction in multiple regression analysis.

The machine learning is often used as a technology, other than the prediction of a probability of contracting a disease or a probability of a machine failure, for identifying feature variables contributing to the prediction of the probability of contracting a disease or the probability of a machine failure on condition that a highly accurate prediction result can be obtained.

For example, in analysis of healthcare information, it is predicted whether a person is a patient using data about blood tests performed on patients with a disease X and on other people, feature variables contributing to the prediction are extracted as important feature variables, and the important feature variables are made much use of in establishment of a treatment policy or a daily life guidance for a patient.

With the approach of prediction by computing the linear combination as described in International Publication No. WO2010/016110 and the perceptron, the feature variables contributing to the prediction are extracted by an approach of identifying the important feature variables using the magnitude of each element value of the weight vector. Furthermore, with an approach of prediction by computing non-linear combination, feature variables contributing to prediction are extracted by an approach of identifying important feature variables using an out-of-bag error rate in random forests, which is one of approaches using a decision tree.

As described in Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. “Why should I trust you?: Explaining the predictions of any classifier.” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2016, development of an approach capable of extracting important feature variables is underway in deep learning capable of solving linearly inseparable problems and the like. The development of these approaches has helped establish new effective treatment policies and daily life guidances.

For example, in a case in which specific feature variables are nearly equivalent to a true value, it is possible to make highly accurate prediction using only the specific feature variables. In addition, in a case, for example, in which feature variables other than the specific feature variables also contribute to prediction of the true value, such possibilities are considered to occur that importance degrees of the feature variables other than the specific feature variables are relatively reduced and that it is impossible to extract the other feature variables as the feature variables contributing to the prediction. It is estimated that the specific feature variables, in particular, are trivial feature variable related to the disease X by the analysis and the like performed so far.

Furthermore, to make it clear that the feature variables (hereinafter, “non-trivial feature variables”) other than the trivial feature variables contributing to prediction (hereinafter, simply “trivial feature variables”) among the specific feature variables contribute to prediction, it is necessary to perform prediction using only the non-trivial feature variables. In this case, a reduction in prediction accuracy is conceivable because of no use of the trivial feature variables.

SUMMARY OF THE INVENTION

An object of the present invention is to provide a data analysis apparatus, a data analysis method, and a data analysis program capable of extracting non-trivial feature variables contributing to prediction as important feature variables.

A data analysis apparatus according to one aspect of the present invention is a data analysis apparatus including: a processor that executes a program; and a storage device that stores the program, the processor executing: a selection process for selecting a first feature variable group that is a trivial feature variable group contributing to prediction and a second feature variable group other than the first feature variable group from a set of feature variables; an operation process for operating a first regularization coefficient related to a first weight parameter group corresponding to the first feature variable group among a set of weight parameters configuring a prediction model in such a manner that the loss function is larger, and operating a second regularization coefficient related to a second weight parameter group corresponding to the second feature variable group among the set of weight parameters configuring the prediction model in such a manner that the loss function is smaller, in a loss function related to a difference between a prediction result output in a case of inputting the set of feature variables to the prediction model and ground truth data corresponding to the feature variables; and a learning process for learning the set of weight parameters of the prediction model in such a manner that the loss function is minimum as a result of operating the first regularization coefficient and the second regularization coefficient by the operation process.

According to a representative embodiment of the present embodiment, it is possible to extract feature variables contributing to prediction among non-trivial feature variables contributing to prediction as important feature variables. Objects, configurations, and advantages other than those described above will be readily apparent from the description of embodiments given below.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

A data analysis apparatus according to a first embodiment will first be described. The data analysis apparatus according to the first embodiment selects trivial feature variables and non-trivial feature variables, and displays a prediction result by increasing degrees of contribution of the non-trivial feature variables to prediction and suppressing degrees of contribution of the trivial feature variables in a learning process.

<Trivial Feature Variables and Non-Trivial Feature Variables>

FIG.1is an explanatory diagram1depicting a trivial feature variable and non-trivial feature variables.FIG.1depicts an example of logistic regression for predicting whether a senior high school student can graduate from senior high school. InFIG.1, a feature variable x1,nindicates an age, a feature variable x2,nindicates the number of attendances, a feature variable x3,nindicates a body height, and a predicted value ynindicates whether a senior high school student can graduate from senior high school. n indicates an n-th (where n is an integer equal to or greater than 1) senior high school student. It is assumed that among the feature variables x1,nto x3,n, the feature variable x1,nis a specific feature variable nearly equivalent to a true value of a predicted value yn.

(A) indicates logistic regression for predicting the predicted value ynusing the feature variables x1,nto x3,n. σ indicates a sigmoid function, w1to w3indicate degrees of contribution (also referred to as weight parameters) to prediction of the predicted value yn, and an area under curve (AUC) indicates prediction accuracy (0.00≤AUC≤1.00). A higher value of the AUC indicates higher prediction accuracy. Since the specific feature variable x1,nis nearly equivalent to a true value of the predicted value ynindicating whether the senior high school student can graduate from senior high school, the specific feature variable x1,nis regarded as a trivial feature variable.

The AUC is an abbreviation of area under an ROC curve, which is an area of a part surrounded by a horizontal axis and a vertical axis of a receiver operating characteristic curve (ROC curve), and the AUC closer to 1 means that accuracy of a model is higher. The ROC is plotted with the horizontal axis indicating a false-positive rate and the vertical axis indicating a true-positive rate. In other words, the AUC closer to 1 refers to achieving a high true-positive rate at a time at which a value of a false-positive rate is low; thus, it is possible to evaluate that the model has high accuracy with a smaller bias. It is noted herein that the false-positive rate is a rate obtained by dividing the number of false-positive samples by a sum of the number of false-positive samples and the number of true-negative samples, and the true-positive rate is a rate obtained by dividing the number of true-positive samples by a sum of the number of true-positive samples and the number of false-negative samples.

In the first embodiment, in a case, for example, in which the predicted value ynis a test result (positive) and a correct label tnis having a disorder, samples (feature variables xn) are true-positive. Furthermore, in a case in which the predicted value ynis the test result (positive) and a correct label tnis not having a disorder, samples (feature variables xn) are false-positive. Moreover, in a case in which the predicted value ynis a test result (negative) and a correct label tnis having a disorder, samples (feature variables xn) are false-negative. Furthermore, in a case in which the predicted value ynis the test result (negative) and a correct label tnis not having a disorder, samples (feature variables xn) are true-negative.

In a case in which a degree of contribution w1to prediction of the predicted value ynof the feature variable x1,nis high, degrees of contribution of the other feature variables, that is, a degree of contribution w2of the feature variable x2,nand a degree of contribution w3of the feature variable x3,nare relatively low. Owing to this, it is impossible to extract the other feature variables as feature variables contributing to prediction although the other feature variables actually include the feature variables contributing to the prediction.

(B) indicates logistic regression for predicting the predicted value ynwhile excluding the trivial feature variable x1,namong the feature variables x1,nto x3,n. In this case, excluding the trivial feature variable x1,ncauses an increase in a value of the degree of contribution w2of the feature variable x2,n(w2=0.95) in (B), while the value of the degree of contribution w2of the feature variable x2,nis low (w2=0.15) in (A). In this way, the specific feature variable x2,n, which also contributes to the prediction, is regarded as a feature variable contributing to prediction although being the non-trivial feature variable.

Therefore, the data analysis apparatus according to the present embodiment operates parameters of a loss function in such a manner that a weight of the trivial feature variable nearly equivalent to the true value of the predicted value ynis suppressed, operates the parameters of the loss function in such a manner that a weight of the non-trivial feature variable is increased, and maintains the prediction accuracy AC for the predicted value ynwithout reducing the prediction accuracy AC.

FIG.2is an explanatory diagram2depicting a trivial feature variable and non-trivial feature variables.FIG.2depicts an example of logistic regression for predicting whether a college student can graduate from college. InFIG.2, the feature variable x1,nindicates the number of attendances, the feature variable x2,nindicates a test score, the feature variable x3,nindicates a body height, and the predicted value ynindicates whether a college student can graduate from college. n indicates an n-th (where n is an integer equal to or greater than 1) college student. It is assumed that among the feature variables x1,nto x3,n, the feature variable x1,nis a specific feature variable which is known to contribute to prediction of the predicted value yndespite low equivalence to a true value.

(A) indicates logistic regression for predicting the predicted value ynusing the feature variables x1,nto x3,n. Since the specific feature variable x1,nis the number of attendances, it is assumed that a college student the number of attendances of whom is large is a serious student and evaluated as an excellent student. Since the specific feature variable x1,nis known to contribute to prediction of the predicted value yn, the specific feature variable x1,nis regarded as a trivial feature variable.

In a case in which the degree of contribution w1of the feature variable x1,nto prediction of the predicted value ynis considerably high, the degrees of contribution of the other feature variables, that is, the degree of contribution w2of the feature variable x2,nand the degree of contribution w3of the feature variable x3,nare relatively low. Owing to this, it is impossible to extract the other feature variables as feature variables contributing to prediction although the other feature variables actually include the feature variables contributing to the prediction.

(B) indicates logistic regression for predicting the predicted value ynwhile excluding the trivial feature variable x1,namong the feature variables x1,nto x3,n. In this case, the machine learning enables an increase in the value of the degree of contribution w2of the feature variable x2,n(w2=0.95) in (B), while the value of the degree of contribution w2of the feature variable x2,nis low (w2=0.35) in (A). In this way, the specific feature variable x2,nis regarded as a non-trivial feature variable contributing to prediction.

Therefore, the data analysis apparatus according to the present embodiment operates parameters of a loss function in such a manner that the weight of the trivial feature variable known to contribute to prediction of the predicted value ynis reduced, operates the parameters of the loss function in such a manner that the weight of the non-trivial feature variable is increased, and maintains the prediction accuracy AC for the predicted value ynwithout reducing the prediction accuracy AC.

<Example of Hardware Configuration of Data Analysis Apparatus>

FIG.3is a block diagram depicting an example of a hardware configuration of the data analysis apparatus according to the first embodiment. A data analysis apparatus300has a processor301, a storage device302, an input device303, an output device304, and a communication interface (communication IF)305. The processor301, the storage device302, the input device303, the output device304, and the communication IF305are connected to one another by a bus306. The processor301controls the data analysis apparatus300. The storage device302serves as a work area of the processor301. Furthermore, the storage device302is a non-transitory or transitory recording medium that stores various programs and data. Examples of the storage device302include a read only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), and a flash memory. Data is input through the input device303. Examples of the input device303include a keyboard, a mouse, a touch panel, a numeric keypad, and a scanner. The output device304outputs data. Examples of the output device304include a display and a printer. The communication IF305is connected to a network to transmit and receive data.

<Example of Functional Configuration of Data Analysis Apparatus300>

FIG.4is a block diagram depicting an example of a functional configuration of the data analysis apparatus300according to the first embodiment. The data analysis apparatus300has a data storage section401, a model storage section402, a result storage section403, a selection section411, a learning section412, an operation section413, a prediction section414, a degree-of-importance calculation section415, and an output section416. The data storage section401, the model storage section402, and the result storage section403are specifically realized by, for example, the storage device302depicted inFIG.3. Furthermore, the selection section411, the learning section412, the operation section413, the prediction section414, the degree-of-importance calculation section415, and the output section416are specifically realized by, for example, causing the processor301to execute a program stored in the storage device302depicted inFIG.3.

The data storage section401stores training data for use in a learning process by the learning section412and test data for use in a prediction process by the prediction section414.

The training data is sample data configured with, for example, each combination {xd,n, tn} of feature variables xd,nand a correct label tnthat is a true value thereof (where d=1, 2, . . . , and D, n=1, 2, . . . , and N. D is the number of types (dimensions) of feature variables and N is the number of samples). The feature variables xd,nare, for example, test data or image data about each patient.

The test data is feature variables xd,ndifferent from the training data. The combination of feature variables xd,nas the test data from which the predicted value ynis obtained and the correct label tnthat is the true value thereof is handled as the training data.

The model storage section402stores output data from the learning section412. The output data contains a weight vector wn, which indicates the degrees of contribution, of the feature variables xd,n.

The result storage section403stores the predicted value yncalculated by a prediction process by the prediction section414, the weight parameters wnthat are learning parameters, and important feature variables contributing to prediction and extracted by the degree-of-importance calculation section415.

The selection section411selects trivial feature variables and non-trivial feature variables from a set of feature variables xd,nthat are the training data. The selection section411may select, as the trivial feature variables, feature variables suggested to be academically important in an accumulation of findings made by developers or engineers so far, documents, or the like.

In addition, the selection section411selects, as the non-trivial feature variables, remaining feature variables xd,nthat are not selected as the trivial feature variables from among the set of feature variables xd,n. InFIGS.1and2, for example, the selection section411selects the feature variable x1,nas the trivial feature variable and selects the feature variables x2,nand x3,nas the non-trivial feature variables.

The learning section412updates a hyperparameter and the weight parameters wnin the following Equation (1) in such a manner that an error between the predicted value ynobtained from the feature variables xd,nthat are inputs and the correct label tnis minimum.

Equation (1) above is an example of a prediction expression of the logistic regression that is one approach of the machine learning using computing of linear combination in calculation of the predicted value yn. The predicted value ynis calculated on the basis of the feature variables xd,nand the weight parameters wn∈RD(where D is an integer equal to or greater than 1). wtis a weight vector having the weight parameters wnas elements, and t in the weight vector wtmeans transpose. σ denotes an activation function such as the sigmoid function. xnis a feature variable vector having the feature variables xd,nas elements.

The learning section412sets a loss function L (wn) for calculating the learning parameters (weight vector wt) using above Equation (1) in such a manner that the error between the predicted value ynobtained from the feature variable vector xnthat is the inputs and the correct label tnthat is an actual value (true value) is minimum. Specifically, the learning section412sets, for example, weight parameters wk,nof trivial feature variables xk,nselected by the selection section411and weight parameters wh,nof non-trivial feature variables xh,nselected by the selection section411to a degree-of-contribution operation term RP(wtn).

The loss function L(wn) is represented by a sum of an error function E(wn) and a degree-of-contribution operation term RP(wn), as depicted in the following Equations (2) and (3).

wnis a weight vector having, as elements, weight parameters w1to wDcorresponding to feature variables x1,nto xd,nof the feature variable vector xnthat is an n-th sample. The error function E(wn) may be, for example, a mean squared error or a cross entropy error between the predicted value ynand the correct label tn.

Furthermore, Equation (3) is the degree-of-contribution operation term RP(wn). A hyperparameter in the degree-of-contribution operation term RP(wn) is set by the operation section413. In Equation (3), λ (0.0≤λ≤1.0) is a loss coefficient. As λ is larger, a value of the loss function L(wn) becomes higher. p denotes a norm dimension.

Moreover, a prediction expression in a case in which the weight vector wnis present in each feature variable xd,nis expressed by, for example, the following Equation (4) by the machine learning approach.

Furthermore, a loss function L(wtn) is represented by a sum of an error function E(wtn) and a degree-of-contribution operation term RP(wtn), as depicted in the following Equations (5) and (6).

Furthermore, the degree-of-contribution operation term RP(wtn) of Equation (6) may be replaced by a degree-of-contribution operation term R1(wn) of the following Equation (7) with norm dimension p=1.

In the degree-of-contribution operation term R1(wn) of Equation (7), λ is the loss coefficient described above, μ is a first regularization coefficient related to the weight parameters wk,nof the trivial feature variables xk,nand ν is a second regularization coefficient related to the weight parameters wh,nof the non-trivial feature variables xh,n. A relationship between the first regularization coefficient μ and the second regularization coefficient ν is, for example, ρ+ν=1.0. λ, μ, and ν are hyperparameters. Furthermore, k indicates a number representing the trivial feature variables xk,n, T indicates the number of the trivial feature variables xk,n, h indicates a number representing the non-trivial feature variables, and U indicates the number of the non-trivial feature variables xh,n.

Adding the degree-of-contribution operation term R1(wn) to the error function E(wnt) by the learning section412makes it possible to produce effects of preventing growth of the weight parameters wk,nof the trivial feature variables xk,nand of obtaining a sparse model.

Moreover, the degree-of-contribution operation term RP(wtn) of Equation (6) may be replaced by a degree-of-contribution operation term R2(wn) of the following Equation (8) with the norm dimension p=2.

Adding the degree-of-contribution operation term R2(wn) to the error function E(wn) by the learning section412makes it possible to produce effects of preventing growth of the weight parameters wk,nof the trivial feature variables xk,nand of suppressing overfitting to obtain a smooth prediction model.

Furthermore, the degree-of-contribution operation term RP(wtn) may be replaced by a degree-of-contribution operation term Rels(wn) of the following Equation (9).

Equation (9) is an equation of an elastic net of linear combination between an L1 norm and an L2 norm of each weight vector wn, and is a degree-of-contribution operation term obtained by the linear combination between Equations (7) and (8). In Equation (9), α(0.0≤α≤1.0) is an elastic coefficient. The elastic coefficient α is also a hyperparameter.

Adding the degree-of-contribution operation term Rels(wn) to the error function E(wn) makes it possible to produce an effect of preventing the growth of the weight parameters wk,nof the trivial feature variables xk,nto obtain a sparse model as depicted in Equation (7) and an effect of suppressing overfitting to obtain a smooth model as depicted in Equation (8).

The operation section413operates the hyperparameter in the degree-of-contribution operation term for increasing the degrees of contribution of the non-trivial feature variables to prediction and reducing the degrees of contribution of the trivial feature variables to prediction. The operation section413operates the hyperparameters described above, that is, the loss coefficient λ, the first regularization coefficient μ related to the weight parameters wk,nof the trivial feature variables xk,nthe second regularization coefficient ν related to the weight parameters wh,nof the non-trivial feature variables xh,n, and the elastic coefficient α. Since the first regularization coefficient μ and the second regularization coefficient ν are each set to a value from 0.0 to 1.0, it is possible to facilitate control over a degree of suppression of the weight parameters wk,nof the trivial feature variables xk,n.

Furthermore, the operation section413operates the first regularization coefficient μ and the second regularization coefficient ν in such a manner that a sum of the first regularization coefficient μ and the second regularization coefficient ν is, for example, 1.0. The operation section413operates the first regularization coefficient μ and the second regularization coefficient ν in such a manner that the first regularization coefficient μ is greater than the second regularization coefficient ν. The operation section413may operate the hyperparameters on condition that the first regularization coefficient μ related to the weight parameters wk,nof the trivial feature variables xk,nis greater than 0.5.

On this condition, if the first regularization coefficient μ is set greater than the second regularization coefficient ν, a value of the term multiplied by the first regularization coefficient μ becomes greater within the degree-of-contribution operation term RP(wtn). Owing to this, learning is carried out in such a manner that values of a first weight parameter group, that is, the trivial feature variables xk,nare made smaller than values of a second weight parameter group, that is, the non-trivial feature variables xh,nto make the loss function L (wtn) smaller. As a result, it is possible to suppress the weight parameters wk,nof the trivial feature variables xk,n, compared with a case of not using the degree-of-contribution operation term RP(wtn). Furthermore, a range of the value of the first regularization coefficient μ may be set, for example, to be equal to or greater than 0.7.

Moreover, the norm dimension p may be set to 0.5 or the like while examples of the L1 norm, the L2 norm, and the elastic net have been illustrated.

The prediction section414executes a prediction process by giving the feature variables xd,nas the test data to a prediction model for which the weight vector wnis applied to Equations (1) and (2), and outputs the predicted value ynto the result storage section403and the output section416.

In addition, the prediction section414calculates the AUC for the predicted value yndescribed above. A case in which the AUC is equal to or smaller than a threshold means a failure in prediction. In this case, the operation section413may re-operate each hyperparameter and the learning section412may perform relearning of the weight vector wn.

The degree-of-importance calculation section415aligns the feature variables xd,nin descending order of contribution to prediction using the weight vector wnstored in the model storage section402, and carries out calculation to regard the feature variables xd,nas important feature variables in the descending order of contribution to prediction. The descending order of contribution to prediction is, for example, descending order of norms of the weight vector wn. The degree-of-importance calculation section415calculates the norms of the weight vector wn.

The degree-of-importance calculation section415assigns degrees of importance to the feature variables xd,nin the descending order of contribution to prediction. The degree of importance is proportional to the norm and takes on a greater value as the norm is higher. The degree-of-importance calculation section415may add a weight of a value equal to or greater than 0.0 and smaller than 1.0 to the norms of the weight vector wnof the trivial feature variables. Furthermore, the degree-of-importance calculation section415may exclude the trivial feature variables at a time of aligning the feature variables xd,nin the descending order of contribution to prediction.

Moreover, the degree-of-importance calculation section415may assign the norms themselves as the degrees of importance. To calculate the degree of importance, an out-of-bag error rate may be used without using the weight vector wndepending on the machine learning approach to be used.

The selection section411may thereby further select the trivial feature variables and the non-trivial feature variables while referring to the degrees of importance calculated by the degree-of-importance calculation section415.

It is noted that a plurality of apparatuses may configure the data analysis apparatus300. For example, a plurality of data analysis apparatuses300may be present for load distribution. Alternatively, a plurality of apparatuses including one or more functions may configure the data analysis apparatus300.

<Example of Data Analysis Process Procedure>

FIG.5is a flowchart describing an example of a data analysis process procedure by the data analysis apparatus300according to the first embodiment. The data analysis apparatus300reads, by the selection section411, a training data set from the data storage section401(Step S501), and then selects, by the selection section411, the trivial feature variables and the non-trivial feature variables from among the training data set (Step S502).

The data analysis apparatus300then generates the weight parameters wnusing the loss function L of Equation (2) or (5) in such a manner that the error between the predicted value ynobtained by giving the feature variables xd,nof the training data set and the correct label tnis minimum (Step S503). Steps S501to S503correspond to the learning process.

The data analysis apparatus300reads, by the prediction section414, a test data set from the data storage section401(Step S504). The data analysis apparatus300calculates the predicted value ynby giving the feature variables xd,nof the test data set to the prediction model for which the weight parameters wnare set to the prediction expression of Equation (1) or (4) (Step S505).

The data analysis apparatus300extracts, by the degree-of-importance calculation section415, the degrees of importance of the feature variables (Step S506). Next, the data analysis apparatus300saves a combination of the predicted value ynand the degrees of importance in the result storage section403(Step S507). The data analysis apparatus300then outputs, by the output section416, the combination of the predicted value ynand the degrees of importance (Step S508).

Further, the data analysis apparatus300operates, by the operation section413, the hyperparameters, that is, the loss coefficient λ, the first regularization coefficient μ related to the weight parameters wk,nof the trivial feature variables xk,nthe second regularization coefficient ν related to the weight parameters wh,nof the non-trivial feature variables xh,n, and the elastic coefficient α (Step S509).

<Examples of Display Screen>

FIG.6is an explanatory diagram depicting a display example 1 of a display screen. A display screen600is displayed on a display that is an example of the output device304in the data analysis apparatus300or a display of a computer to which the output section416outputs data.

The display screen600includes an import file button601, a feature select button602, a train button603, a predict button604, a save button605, a file name box606, and a select screen610.

Upon detecting depression of the import file button601by a user's operation, the data analysis apparatus300selects the training data for use by the learning section412, the test data for use by the prediction section414, a determined optimum model, a prediction result, the degrees of importance, and the like by a user's operation. Names of the selected data are displayed in the file name box606. Subsequently, upon depression of the Feature select button602by a user's operation, the data analysis apparatus300displays, by the selection section411, the select screen610of the feature variables.

A user places a checkmark in each feature variable to be selected as the trivial feature variable, for example, in a checkbox611. The selection section411selects the feature variables in the checked checkboxes as the trivial feature variables. When the selection of the feature variables is over and learning is started, the user depresses the train button603. The learning section412thereby starts the learning process. Subsequently, the user selects test data and depresses the predict button604. The prediction section414thereby starts the prediction process.

FIG.7is an explanatory diagram depicting a display example 2 of the display screen. On the display screen600, a degree of importance of the predicted value yn, and a suppression effect of the weight parameters wk,nof each trivial feature variable xk,nare displayed after end of the prediction process. The predicted value ynis displayed in an accuracy display area711. In addition, the weight parameter wd,nof each feature variable xd,nin ordinary prediction and a result of suppressing the weight parameter wk,nof each trivial feature variable xk,nby the operation section413are displayed side by side in a suppression effect display area712.

While a comparison between the ordinary prediction and the suppression result is displayed inFIG.7, only the suppression result may be displayed. Furthermore, a value displayed as the weight parameter wk,nof each trivial feature variable xk,nmay be a value of the actual weight parameter wk,n, a normalized value in each sample n, or an average value obtained by normalizing the value in each sample n and then adding up the normalized values in all samples1to N or by a total cross-validation.

In a case of saving these analysis results, the user depresses the save button605. A screen on which a memory space in which the analysis results are to be saved can be designated is thereby displayed. Upon user's designating the memory space and depressing an execution button, the analysis results are saved in the designated memory space. The memory space in which the analysis results are saved is displayed in an export file name box701or the like.

In this way, according to the first embodiment, using the loss function for setting different penalties between the trivial feature variables and the non-trivial feature variables in the machine learning accountable for grounds for prediction makes it possible to realize prediction while suppressing the degrees of contribution (weight parameters wk,n) of the trivial feature variables xk,nto prediction and making active use of the other non-trivial feature variables xh,n. This makes it possible to extract unknown feature variables that may be feature variables not discovered yet in academic findings or the like contributing to prediction.

Second Embodiment

A second embodiment will be described. In the first embodiment, the two feature variable groups, that is, a trivial feature variable group and a non-trivial feature variable group are selected. The second embodiment is an example of increasing the number of groups of feature variables for operating the degrees of contribution such as trivial feature variables, non-trivial feature variables, and trivial feature variables not contributing to prediction, compared with the number of feature variable groups in the first embodiment. It is noted that the same constituent elements as those in the first embodiment are denoted by the same reference characters and are not often described.

The selection section411selects trivial feature variables, in which contributing to prediction is trivial, and feature variables, in which contributing to prediction is non-trivial, as well as trivial feature variables, in which not contributing to prediction is trivial, from the set of feature variables xd,nas the training data. The selection section411may select, as the trivial feature variables, feature variables suggested to be academically important in the accumulation of findings made by developers or engineers so far, documents, or the like.

In addition, the selection section411may select, as the trivial feature variables not contributing to prediction, feature variables suggested to be not academically important in the accumulation of findings made by developers or engineers so far, documents, or the like. Furthermore, the selection section411may select, as the non-trivial feature variables, remaining feature variables xd,nthat are not selected as the trivial feature variables and the trivial feature variables not contributing to prediction from among the set of feature variables xd,n. InFIGS.1and2, for example, the selection section411selects the feature variable x1,nas the trivial feature variable, selects the feature variable x2,nand x3,nas the non-trivial feature variable, and selects the feature variable x3,nas the trivial feature variable not contributing to prediction.

The operation section413operates the hyperparameters in the degree-of-contribution operation term for increasing the degrees of contribution of the non-trivial feature variables to prediction and reducing the degrees of contribution of the trivial feature variables to prediction and the degrees of contribution of the trivial feature variables not contributing to prediction. The degree-of-contribution operation term RP(wtn) is replaced by a degree-of-contribution operation term R1(wn) of the following Equation (10) with the norm dimension p=1.

Equation (10) is an example of the degree-of-contribution operation term R1(wn) of the L1 norm. τ is a third regularization coefficient related to weight parameters w1,nof trivial feature variables x1,nin which not contributing to prediction is trivial. τ is also a hyperparameter. l indicates a number of the trivial feature variables, in which not contributing to prediction is trivial, and V is the number of the non-trivial feature variables. The degree-of-contribution operation term R1(wn) of Equation (10) is added to the error function E(wnt) by the learning section412as depicted in Equation (5). The learning section412thereby calculates the loss function L(wnt) and updates the weight parameters wk,n, wh,n, and w1,n.

By doing so, it is possible to produce effects of preventing growth of the weight parameters wk,nof the trivial feature variables xk,nand the weight parameters w1,nof the trivial feature variables x1,n, in which not contributing to prediction is trivial, and of obtaining a sparse model.

Moreover, the degree-of-contribution operation term RP(wtn) of Equation (6) may be replaced by a degree-of-contribution operation term R2(wn) of the following Equation (11) with the norm dimension p=2.

Adding the degree-of-contribution operation term R2(wn) to the error function E(wn) by the learning section412makes it possible to produce effects of preventing growth of the weight parameters wk,nof the trivial feature variables xk,nand the weight parameters w1,nof the trivial feature variables, in which not contributing to prediction is trivial, and of suppressing overfitting to obtain a smooth prediction model.

Furthermore, the degree-of-contribution operation term RP(wtn) of Equation (6) may be replaced by a degree-of-contribution operation term Rels(wn) of the following Equation (12).

Equation (12) is an equation of an elastic net of linear combination between the L1 norm and the L2 norm of each weight vector wn, and is a degree-of-contribution operation term obtained by the linear combination between Equations (10) and (11). In Equation (12), α(0.0≤α≤1.0) is the elastic coefficient. The elastic coefficient α is also a hyperparameter.

Adding the degree-of-contribution operation term Rels(wn) to the error function E(wn) makes it possible to produce an effect of preventing the growth of the weight parameters wk,nof the trivial feature variables xk,nand the weight parameters w1,nof the trivial feature variables x1,n, in which not contributing to prediction is trivial, to obtain a sparse model as depicted in Equation (10) and an effect of suppressing overfitting to obtain a smooth model as depicted in Equation (11).

Furthermore, the operation section413operates the first regularization coefficient μ, the second regularization coefficient ν, and the third regularization coefficient T in such a manner that a sum of the first regularization coefficient μ, the second regularization coefficient ν, and the third regularization coefficient τ is, for example, 1.0. The operation section413operates the first regularization coefficient μ, the second regularization coefficient ν, and the third regularization coefficient τ in such a manner the first regularization coefficient μ and the third regularization coefficient τ are greater than the second regularization coefficient ν. The operation section413may operate the first regularization coefficient μ, the second regularization coefficient ν, and the third regularization coefficient τ on condition that one of the first and third regularization coefficients μ and τ is greater than 0.5.

By doing so, with growth of the weight parameters wk,nof the trivial feature variables xk,nand the weight parameters of the trivial feature variables x1,nin which not contributing to prediction is trivial, regularization terms of the first regularization coefficient μ and the third regularization coefficient τ grow. It is, therefore, possible to suppress the weight parameters wk,nof the trivial feature variables xk,nand the weight parameters w1,nof the trivial feature variables x1,n, in which not contributing to prediction is trivial, and grow the values of the weight parameters wh,nof the non-trivial feature variables xh,n, compared with the case of not using the degree-of-contribution operation term RP(wtn). Furthermore, a range of the value of one of the first regularization coefficient μ and the third regularization coefficient τ may be set, for example, to be equal to or greater than 0.7.

Moreover, the selection section411may comprehensively change feature variables designated as trivial feature variables and select the trivial feature variables on the basis of a result of carrying out the first embodiment. Specifically, the selection section411selects, for example, only one trivial feature variable, carries out the first embodiment, and obtains the prediction accuracy (AUC, a determination coefficient r2, or the like) and the degree of importance.

Subsequently, the data analysis apparatus300changes the only feature variable to be selected and carries out the first embodiment by as much as the number of all feature variables. Furthermore, the data analysis apparatus300increases the number of designated feature variables to two, similarly carries out the first embodiment by combinations of all feature variables, further increases the number of designated feature variables, and carries out the first embodiment in all patterns in which feature variables can be selected as the trivial feature variables. Subsequently, in a case in which the prediction accuracy is equal to or higher than a threshold, the selection section411lists up the feature variables selected as the trivial feature variables and the combinations of the feature variables, and selects the trivial feature variables from among the feature variables and the combinations.

The listed feature variables can be interpreted as important feature variables for realizing accurate prediction. At this time, the data analysis apparatus300may select the feature variables in descending order of frequency of appearance in the listed feature variables and the combinations of the feature variables in sequence. The selection section411can thereby dynamically select the trivial feature variables and the non-trivial feature variables.

Moreover, as a result of carrying out the first embodiment in all patterns in which feature variables can be selected as trivial feature variables by the data analysis apparatus300, the selection section411may refer to the obtained degrees of importance and select the feature variables having higher degrees of importance as trivial feature variables although the feature variables are designated as the trivial feature variables. It can be interpreted that the feature variables having higher degrees of importance despite suppression of the degrees of contribution to prediction are the important feature variables for realizing accurate prediction. At this time, the data analysis apparatus300may select the feature variables in descending order of frequency of appearance in the feature variables listed as those having the degrees of importance equal to or higher than a threshold and the combinations of the feature variables in sequence although the feature variables are designated as the trivial feature variables. The selection section411can thereby dynamically select the trivial feature variables and the non-trivial feature variables.

In this way, according to the second embodiment, using the loss function L (wnt) for setting different penalties among the trivial feature variables, the non-trivial feature variables, and the trivial feature variables, in which not contributing to prediction is trivial, in the machine learning accountable for grounds of prediction makes it possible to realize prediction while suppressing the degrees of contribution of the trivial feature variables to prediction and those of the trivial feature variables, in which not contributing to prediction is trivial, and making active use of the non-trivial feature variables. This makes it possible to extract unknown feature variables that may be feature variables not discovered yet in academic findings or the like despite contribution to prediction.

Third Embodiment

A third embodiment will be described. The third embodiment is an example related to a method of selecting trivial feature variables and non-trivial feature variables by the selection section411. It is noted that the same constituent elements as those in the first and second embodiments are denoted by the same reference characters and are not often described.

In the first and second embodiments, the selection section411designates the trivial feature variables from among the feature variables already indicated to be academically important in documents or the like or the accumulation of findings made by developers and engineers so far in selecting the trivial feature variables. In the third embodiment, the selection section411selects trivial feature variables on the basis of degrees of actual contributions to prediction. To describe a selection method based on the degree of contributing to prediction, as an example of predicting Boston house prices, a performance verification is carried out on the basis of data used in Harrison, D. and Rubinfeld, D. L. (1978) Hedonic prices and the demand for clean air. J. Environ. Economics and Management 5, 81-102.

FIG.8is an explanatory diagram depicting a feature variable vector Features and ground truth data Target. In an experiment, prediction is applied in a case of using 10-fold cross validation and all thirteen feature variables in (1) to (13), two feature variables having the degrees of importance accounting for top 20% are selected as the trivial feature variables from among the feature variables contributing to prediction, and the first embodiment is carried out.

FIG.9is an explanatory diagram depicting an experimental result. A prediction result by the data analysis apparatus300in a case in which the operation section413does not operate the hyperparameters is a Normal graph, while a prediction result by the data analysis apparatus300in a case in which the operation section413operates the hyperparameters is a Suppression graph. Since the determination coefficient r2(=0.75) in the Normal graph exceeds 0.7, the data analysis apparatus300calculates the degrees of importance with respect to degrees of contribution to accurate prediction.

The selection section411compares a magnitude of the weight vector wnamong the feature variables, and selects the feature variables (6) and (13) that are top two feature variables as trivial feature variables. The operation section413operates the first regularization coefficient μ related to the weight parameters wk,nof the trivial feature variables xk,nto be equal to or greater than 0.5 using Equation (7). The learning section412generates learning parameters (weight vector wn) in the learning process. The selection section411compares again the magnitude of the weight vector wnamong the feature variables.

Since the determination coefficient r2(=0.82) exceeds 0.7, it is understood that prediction is carried out with high prediction accuracy even after operating the first regularization coefficient μ. Comparison of the magnitude of the weight vector wnin Normal prediction with the magnitude of the weight vector wnin Suppression prediction indicates that the weight vector wnof the feature variables (6) and (13) can be suppressed and that the magnitude of the weight vector wnsmall in the Normal prediction can be grown.

While the feature variables having the top 20% degrees of importance among the feature variables contributing to prediction are selected as the trivial feature variables in the third embodiment, a percentage may be 50% or the like or the number of the trivial feature variables may be determined in advance. Furthermore, while the selection method based on the degrees of contribution to prediction has been described in the third embodiment, the selection section411may select the trivial feature variables on the basis of a prediction result. The selection section411may select the trivial feature variables until the prediction result indicating, for example, that the determination coefficient r2or the AUC is equal to or smaller than 0.8.

In this way, according to the third embodiment, using the loss function for setting different penalties between the trivial feature variables and the non-trivial feature variables in the machine learning accountable for grounds for prediction makes it possible to realize prediction while suppressing the degrees of contribution (weight parameters wk,n) of the trivial feature variables xk,nto prediction and making active use of the other non-trivial feature variables xh,n. This makes it possible to extract unknown feature variables that may be feature variables not discovered yet in academic findings or the like and contributing to prediction.

Fourth Embodiment

A fourth embodiment will be described. The fourth embodiment is an example related to a method of determining the first regularization coefficient μ of the trivial feature variables and the second regularization coefficient ν of the non-trivial feature variables by the operation section413. It is noted that the same constituent elements as those in the first embodiment are denoted by the same reference characters and are not often described.

In the first embodiment, the operation section413determines the regularization term of the trivial feature variables and that of the non-trivial feature variables on condition that the range of the value of each of the first and second regularization coefficients μ and ν is set in such a manner that the sum of the first regularization coefficient μ of the trivial feature variables and the second regularization coefficient ν of the non-trivial feature variables is equal to 1 and that the first regularization coefficient μ of the trivial feature variables is greater than 0.5. In the fourth embodiment, an example of generating learning parameters having highest prediction accuracy in a designated range of values on the above condition by the learning section412will be described.

FIG.10depicts an example of screen display of the data analysis apparatus300according to the fourth embodiment. As depicted inFIG.10, a slider1001that is an example of a user interface and that adjusts values of the first regularization coefficient μ and the second regularization coefficient ν may adjust the values of the first regularization coefficient μ of the trivial feature variables and the second regularization coefficient ν of the non-trivial feature variables in determining the first regularization coefficient μ and the second regularization coefficient ν. Furthermore, the values of the first regularization coefficient μ and the second regularization coefficient ν may be subsequently changed again after confirming the magnitude of the weight vector wnas depicted inFIG.7.

Moreover, as a method of determining values, the user may set the first regularization coefficient μ of the trivial feature variables to a fixed value such as 0.9 or values in a desired balance may be selected on the basis of a degree of suppression of the weight vector wnand the prediction accuracy.

In this way, according to the fourth embodiment, using the loss function for setting different penalties between the trivial feature variables and the non-trivial feature variables in the machine learning accountable for grounds for prediction makes it possible to realize prediction while suppressing the degrees of contribution (weight parameters wk,n) of the trivial feature variables xk,nto prediction and making active use of the other non-trivial feature variables xh,n. This makes it possible to extract unknown feature variables that may be feature variables not discovered yet in academic findings or the like and contributing to prediction.

Fifth Embodiment

In a fifth embodiment, an example of calculating degrees of importance used in the first to fourth embodiments will be described. It is noted that the same constituent elements as those in the first to fourth embodiments are denoted by the same reference characters and are not often described.

<Example of Reallocation of Feature Vectors>

While artificial intelligence (AI) has a capability of solving linearly inseparable problems, it is unclear why the AI made such a judgment. A machine learning approach such as deep learning, in particular, is high in prediction accuracy but low in accountability. For example, in a case in which the AI output a diagnosis result that “prone to catch a cold” to a certain patient, a doctor is unable to answer a question why the AI obtained such a result. If the AI can make a judgment up to a cause of a symptom, the doctor can give proper treatment to the patient.

FIGS.11A and11Bare explanatory diagrams depicting an example of reallocation of feature variable vectors. In (A), in a feature variable space SP1, a plurality of feature variable vectors xn(n=1, 2, . . . , N, where N is the number of images) are present. The plurality of feature variable vectors xnare discriminated as correct labels La and Lb by, for example, a nonlinear prediction model PM1. In (B), in a feature space SP2, a plurality of feature variable vectors xnare present. The plurality of feature variable vectors xnare discriminated as correct labels La and Lb by, for example, a nonlinear prediction model PM2.

In (A), the machine learning approach such as the deep learning learns linear regression anew for explaining the prediction model PM1that is a discrimination result. Specifically, the machine learning approach executes, for example, a retrofitted process of obtaining the prediction model PM1and then locally performing straight-line approximation on the prediction model PM1. However, it is unclear in such a retrofitted process whether a straight-line approximated local part of the prediction model PM1can correctly explain the feature variable vectors xn. Furthermore, and more importantly, executing logistic regression called straight-line approximation makes it necessary to execute machine learning twice after all.

Since the prediction model PM2in (B) is linear, referring to an inclination of the prediction model PM2makes it possible to grasp with which parameter in the feature variable space SP2each of the feature variable vectors xnis weighted and to correctly explain the feature variable vectors xn. In the first embodiment, the plurality of feature variable vectors xnin the feature variable space SP1are reallocated to the other feature variable space SP2without obtaining the nonlinear prediction model PM1like (A) for the plurality of feature vectors xn. The linear prediction model PM2is thereby obtained; thus, it is possible to grasp with which parameter in the feature variable space SP2each of the feature variable vectors xnis weighted and to correctly explain each feature variable vector xnin response to the degree of importance.

In other words, the user can grasp which factor (feature) included in the feature variables xncontributes to the prediction result for every sample (for example, for every patient) with the feature variable vectors xn; thus, it is easy to explain why such a prediction result is obtained. Therefore, it is possible to improve accountability of the machine learning. According to the above example, the user can grasp why the AI output the diagnosis result of “prone to catch a cold” (for example, because of slimness) with respect to the certain patient. Furthermore, it is possible to improve efficiency of the machine learning since it is unnecessary to execute the machine learning twice as in (A). Therefore, it is possible to promptly provide an explanation described above.

<Example of Structure of Neural Network>

FIG.12is an explanatory diagram depicting an example of a structure of a neural network according to the fifth embodiment. A neural network1200has a data unit group DU, a reporting unit group RU, a harmonizing unit group HU, a reallocation unit RAU, a unifying unit UU, a decision unit DCU, and an importance unit IU.

The data unit group DU is configured such that a plurality of data units DU1(l is a layer number and 1≤l≤L. L is the layer number of a lowest layer and L=4 inFIG.12) are connected in series. The data unit DU1that is a highest layer of l=1 corresponds to an input layer1201and the data units DU1of l≤2 correspond to intermediate layers (also referred to as hidden layers) of the neural network1200. Each data unit DU1is a perceptron to which output data from the data unit DU(l−1) of a previous layer is input, in which calculation is performed using a learning parameter of the own data unit DU1, and from which output data is output.

It is noted, however, that the data unit DU1holds the training data at a time of learning by the learning section412. The training data means herein, for example, sample data configured with a combination {xn, tn} of images xnas an example of the feature variable vector xnand the correct label tnthat is a true value of the images (n=1, 2, . . . , N, where N is the number of images). The images xnare data having a two-dimensional matrix structure and hereinafter handled as a d-dimensional vector (where d is an integer satisfying d≥1) obtained by raster scanning. In a case of denoting “x” for easier description, it is assumed that the vector is a one-dimensional vector obtained by raster scanning the image xnin a matrix form.

The correct label tnis a K-dimensional vector that indicates a type (for example, an animal such as dog or cat) in a one-hot representation with respect to the number of types K of the images xn. In the one-hot representation, one element of the vector corresponds to the type of the images xnand 1.0 is stored in the only one element, while 0.0 is stored in all the other elements. The type (for example, a dog) corresponding to the element of 1.0 is the type that is a ground truth. It is noted that in a case in which medical images xnsuch as CT images, MRI images, or ultrasound images are an input, the label tnis a true value that represents a type of disease or a prognosis (good or bad) of a patient.

It is assumed that the images xn∈Rd(where Rdis d-dimensional real numbers) are a feature variable vector configured with the d-dimensional real numbers Rd. A function hl+1Dthat indicates the data unit DU(l+1) is expressed by the following Equation (13).
[Expression 11]
Equation (13)
hDi+1=fDl(WDlhDl)  (13)
where hDi∈dlis an input/output vector of the data unit WDl∈dl+1×dlis a learning parameter
when l=1, hD1=xn

In Equation (13), an index l (integer satisfying 1≤l≤L) denotes the layer number (the same applies to the following equations). L is an integer equal to or greater than 1 and denotes a layer number of the lowest layer. fDlon a right side is an activation function. As the activation function, any of various activation functions such as the sigmoid function, a hyperbolic tangent function (tank function), and a rectified linear unit (ReLU) function may be used. A matrix WlDis the learning parameter of the data unit DU1. A vector hlDon the right side is an input vector input to the data unit DU1, that is, an output vector from the data unit DU1that is the previous layer of the data unit DU1. It is noted that the output vector hlDfrom the data unit DU1in a case in which the number of layers l=1 is hlD=xn.

It is noted that the data unit DU1holds the images xnthat are the feature variable vector as the test data at a time of prediction by the prediction section414.

An output vector hlDfrom the data unit DU1on the same layer is input to each reporting unit RUI (2≤l≤L), and the reporting unit RU1contracts the number of dimensions of the output vector hlD. A function h1R that indicates the reporting unit RU1is expressed by the following Equation (14).
[Expression 12]
Equation (14)
hRl=σ(WRlhDl)  (14)

In Equation (14), a matrix WlRis a learning parameter of the reporting unit RU1. The d-dimensional output vector hlDfrom each data unit DU1is contracted to an m-dimensional output vector hlRby Equation (14). Further, σ is the sigmoid function.

Each harmonizing unit HU1(2≤l≤L) is provided between each data unit DU1on the intermediate layer and the reallocation unit RAU per data unit DU1on the intermediate layer. The harmonizing units HU1each convert the number of dimensions of the output data from the data unit DU1on the intermediate layers into the same size. Therefore, pieces of the output data made to have the same number of dimensions by the harmonizing units HU1are input to the reallocation unit RAU.

In other words, the output vector hlDis input to each harmonizing unit HU1from the data unit DU1on the same layer, and the harmonizing units HU1each convert the number of dimensions of the output vector hlDinto the same number of dimensions. A function h′H that indicates each harmonizing unit HU1is expressed by the following Equation (15).
[Expression 13]
Equation (15)
hHl=fH(WHlhDl)  (15)
where WHldl+1×dlis a learning parameter

In Equation (15), a matrix WlHis a learning parameter of the harmonizing unit HU1. The d-dimensional output vector hlDfrom the data unit DU1is thereby converted into an m-dimensional output vector hlH. It is noted that m is a hyperparameter that determines the number of dimensions. d and m may differ from d and m in the reporting units RU1. Furthermore, fHis an activation function.

The attention unit AU calculates a weight α of each data unit DU1using the output vector hlRfrom each reporting unit RU1. A function a that indicates the attention unit AU is expressed by the following Equation (16).
[Expression 14]
Equation (16)
α=softmax(WAhR)  (16)
where WA(L−1)×m(m=r(L−1)) is a learning parameter

In Equation (16), a matrix WAis a learning parameter of the attention unit AU. A softmax function that is one type of activation function calculates a vector hRin dimensions equal to the number of layers (L=4 in an example of Equation (17) below). As indicated by the following Equation (17), the vector hRon the right side of Equation (16) is a vector obtained by stacking hlRin a perpendicular direction.

Therefore, the matrix WAis a matrix of L rows by M columns (where M is the number of elements of the vector hR). By adopting the softmax function in the attention unit AU, each element (a sum of all the elements is 1) of the vector hRwith the number of layers being L represents the weight of the corresponding data unit DU1.

The reallocation unit RAU reallocates the feature variable vectors (images xn) in one feature variable space to the other feature variable space. Specifically, as depicted inFIGS.11A and11B, for example, the prediction model obtained by a feature variable vector group in the feature variable space SP1can be nonlinear; thus, the reallocation unit RAU transfers the feature variable vector group to the feature variable space SP2so that a linear prediction model can be obtained in the feature variable space SP2. A function hlTthat indicates the reallocation unit RAU is expressed by the following Equation (18).
[Expression 16]
Equation (18)
hTl=fT(hHl,xn)  (18)

As the function fT, the Hadamard product between the vectors, the element addition, or the like can be used. In the present embodiment, the Hadamard product is used (refer to the following Equation (19)). In Equation (19), the Hadamard product between the output vector hlHfrom the harmonizing unit HU1and the feature variable vector xnis obtained.
[Expression 17]
Equation (19)
hTl=hHl⊙xn(19)

The unifying unit UU unifies the output vector hlTfrom the reallocation unit RAU with the output vector α from the attention unit AU. In other words, the unifying unit UU weights the output vector hlTfrom the reallocation unit RAU with the output vector α from the attention unit AU. A function hUthat indicates the unifying unit UU is expressed by the following Equation (20).
[Expression 18]
Equation (20)
hU=Σk=1L−1α[k]hTk+1(20)

In Equation (20), α[k] on the right side indicates an element (weight) in a k-th dimension of the output vector α of Equation (16).

The decision unit DCU decides on the predicted value ynand outputs the predicted value ynto an output layer1203. Specifically, the decision unit DCU weights, for example, the output vector hUfrom the unifying unit UU with a weight vector wothat is one of the learning parameters and gives the resultant vector to the sigmoid function σ, thereby obtaining the predicted value yn. A function ynthat indicates the decision unit DCU is expressed by the following Equation (21). In Equation (21), t in wotmeans transpose.
[Expression 19]
ynEquation (21)

The importance unit IU calculates a degree-of-importance vector slnthat indicates the degree of importance of a feature variable on each layer of the neural network and outputs the degree-of-importance vector slnto the output layer1203. A function slnthat indicates the importance unit IU is expressed by the following Equation (22).
[Expression 20]
Equation (22)
snl=α[l]fT(wo,hHl(22)

In Equation (22), α[l] on the right side indicates an element (weight) in the 1-th layer of the output vector α of Equation (12). As the function fT, the Hadamard product between the vectors, the element addition, or the like can be used, similarly to Equation (18). In the first embodiment, the Hadamard product is used. In Equation (22), the degree-of-importance vector slnis the Hadamard product between the weight vector woand the output vector hlHfrom the harmonizing unit HU1. The degree-of-importance vector sin is a degree of importance of the n-th feature variable vector (image) xnon the layer l.

<Example of Functional Configuration of Data Analysis Apparatus300>

FIG.13is a block diagram depicting an example of a functional configuration of the data analysis apparatus300according to the fifth embodiment. The data analysis apparatus300has the input layer1201, the intermediate layers1202, the output layer1203, a conversion section1301, a reallocation section1302, a predicted data calculation section1303, a degree-of-importance calculation section1304, a setting section1305, a unifying section1306, and a contraction section1307. These are an example of internal configurations of the learning section412and the prediction section414.

As indicated by Equation (15), the conversion section1301contracts the number of dimensions d of the output vector hlDfrom the DU1(l≥2) on each intermediate layer on the basis of the output vector hlDand the matrix WlH, and outputs the output vector hlHafter conversion. The conversion section1301is the harmonizing unit group HU described above.

As indicated by Equations (18) and (19), the reallocation section1302reallocates each feature variable vector xnin the feature variable space SP1given to the input layer1201to the feature variable space SP2on the basis of the output vector hlHafter conversion from the conversion section1301and the feature variable vector xn. The reallocation section1302is the reallocation unit RAU described above.

As indicated by Equation (21), the predicted data calculation section1303calculates the predicted vector ynwith respect to each feature variable vector xnin the feature space SP1on the basis of a reallocation result hTlby the reallocation section1302and the weight vector wo. The predicted data calculation section1303is the decision unit DCU described above.

As indicated by Equation (22), the degree-of-importance calculation section1304calculates the degree-of-importance vector slnof the feature variable vector xnon each layer l of the intermediate layers1202on the basis of the output vector hlHafter conversion and the weight vector wo. The degree-of-importance calculation section1304is the importance unit IU described above.

For example, as for the images xnthat express an animal, it is assumed that an output vector hlaDon one layer la is a feature variable that indicates whether a contour of a face is suitable for a cat and assumed that an output vector hlbDon one layer lb (≠la) is a feature variable that indicates whether a shape of an ear is suitable for the cat. In this case, referring to corresponding degree-of-importance vectors slanand slanenables a user to explain in light of which feature of the face in the images xnthe data analysis apparatus300discriminates the animal as a cat. For example, in a case in which the degree-of-importance vector slanis low but the degree-of-importance vector slanis high, the user can explain that the data analysis apparatus300discriminates the animal as a cat in light of the shape of the ear in the images xn. It is noted that the calculated degree-of-importance vectors slnare extracted by the degree-of-importance calculation section415.

As indicated by Equations (16) and (17), the setting section1305sets the weight α of each intermediate layer1202on the basis of the output vector hip from the intermediate layer1202and the matrix WA. The setting section1305is the attention unit AU described above.

As indicated by Equation (20), the unifying section1306unifies the reallocation result hTlwith the weight α set by the setting section1305. The unifying section1306is the unifying unit UU described above. In this case, the predicted data calculation section1303calculates the predicted vector ynon the basis of a unifying result hUby the unifying section1306and the weight vector wo. Furthermore, the degree-of-importance calculation section1304calculates the degree-of-importance vector snlon the basis of the weight α set by the setting section1305, the output vector hlHafter conversion and the weight vector wo.

As indicated by Equation (14), the contraction section1307contracts the number of dimensions d of the output vector hlDfrom each intermediate layer1202on the basis of the output vector hlDfrom the intermediate layer1202and the matrix W1R, and outputs the output vector hlRafter contraction. The contraction section1307is the reporting unit group RU described above. In this case, the setting section1305sets the weight α of each intermediate layer1202on the basis of the output vector hlRafter contraction from the contraction section1307and the matrix WA.

In a case in which the training data that contains each feature variable vector xnin the feature space SP1and the correct label tnwith respect to the predicted vector ynis given, the learning section412optimizes the matrix WlDthat is a first learning parameter, the matrix W1Hthat is a second learning parameter, the weight vector wothat is a third learning parameter, the matrix WAthat is a fourth learning parameter, and the matrix WlRthat is a fifth learning parameter using the predicted vector ynand the correct label tnin such a manner, for example, that a cross entropy between the correct label tnand the predicted value ynbecomes a minimum.

The prediction section414sets the optimized learning parameters to the first neural network1200and gives each feature variable vector x′nas the test data to the input layer1201, thereby causing the predicted data calculation section1303to calculate a predicted vector y′n.

In this way, according to the fifth embodiment, reallocating each feature variable vector xnas the sample data in advance makes it possible to calculate the degree of importance of each feature variable even if the neural network is multi-layered, and to realize accountability per sample (feature variable vector xn) with high accuracy and with high efficiency. Moreover, since the linear prediction model is obtained by reallocating each sample (feature variable vectors xn) in advance, it is possible to calculate the predicted value with high accuracy and with low load at times of learning and prediction.

As described so far, according to the fifth embodiment, the data analysis apparatus300has the conversion section1301, the reallocation section1302, and the degree-of-importance calculation section1304. Therefore, the linear prediction model is obtained by reallocating the feature variable vector (xn, x′n) in advance; thus, it is possible to calculate the predicted value with high accuracy and with low load at times of learning and prediction. Furthermore, it is possible to grasp features possessed by the feature variable vector (xn, x′n) by the degree of importance of every layer l from the degree-of-importance calculation section1304. It is thereby possible to realize accountability about the feature variable vector (xn, x′n) given to the neural network as an object to be analyzed with high accuracy and with high efficiency.

Moreover, the data analysis apparatus300has the predicted data calculation section1303; thus, it is possible to realize accountability about the reason for obtaining the prediction result (yn, y′n) from the neural network as an object to be analyzed with respect to the feature variable vector (xn, x′n) with high accuracy and with high efficiency.

Furthermore, the data analysis apparatus300has the setting section1305and the unifying section1306; thus, the predicted data calculation section1303can calculate the prediction result based on the reallocation result with high accuracy.

Moreover, the data analysis apparatus300has the contraction section1307; thus, it is possible to improve efficiency of data analysis by contraction of dimensions.

Furthermore, the data analysis apparatus300can construct a high accuracy prediction model by learning of the learning parameters.

Since the degree of importance is obtained per feature variable, the selection section411can select the non-trivial feature variables on the basis of the degrees of importance.

As described so far, the data analysis apparatus300described above can extract unknown feature variables that may be feature variables not discovered yet in academic findings or the like and contributing to prediction by increasing the degrees of contribution of the non-trivial feature variables to prediction, reducing the degrees of contribution of the trivial feature variables, and minimizing a reduction in prediction accuracy in the machine learning accountable for grounds for prediction.

The present invention is not limited to the embodiments described above and encompasses various modifications and equivalent configurations within the meaning of the accompanying claims. For example, the embodiments have been described above in detail for describing the present invention so that the present invention is easy to understand, and the present invention is not always limited to the embodiments having all the described configurations. Furthermore, apart of configurations of one embodiment may be replaced by configurations of the other embodiment. Moreover, the configurations of the other embodiment may be added to the configurations of the one embodiment. Further, for part of the configurations of each embodiment, addition, deletion, or replacement may be made of the other configurations.

Moreover, a part of or all of the configurations, the functions, the processing sections, processing means, and the like described above may be realized by hardware by being designed, for example, as an integrated circuit, or may be realized by software by causing the processor to interpret and execute programs that realize the functions.

Information in programs, tables, files, and the like for realizing the functions can be stored in a storage device such as a memory, a hard disk, or a solid state drive (SSD), or in a recording medium such as an integrated circuit (IC) card, a secure digital (SD) card, or a digital versatile disk (DVD).

Furthermore, control lines or information lines considered to be necessary for the description are illustrated and all the control lines or the information lines necessary for implementation are not always illustrated. In actuality, it may be contemplated that almost all the configurations are mutually connected.