INFORMATION PROCESSING APPARATUS, LEARNING APPARATUS, INFORMATION PROCESSING SYSTEM, INFORMATION PROCESSING METHOD, LEARNING METHOD, INFORMATION PROCESSING PROGRAM, AND LEARNING PROGRAM

An information processing apparatus includes at least one processor, in which the processor is configured to: for a machine learning model that uses a document data group including a plurality of document data as input and outputs output data, derive an evaluation value in the machine learning model for each document data included in the document data group.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from Japanese Patent Application No. 2022-138807, filed on Aug. 31, 2022, and Japanese Patent Application No. 2023-030561, filed on Feb. 28, 2023, the entire disclosures of which are incorporated by reference herein.

BACKGROUND

Technical Field

The present disclosure relates to an information processing apparatus, a learning apparatus, an information processing system, an information processing method, a learning method, an information processing program, and a learning program.

Related Art

There is known a technique of deriving an evaluation value of input data in a machine learning model in order to interpret the machine learning model. There is known a technique of deriving a degree of contribution of the input data to the derivation of output data in the machine learning model as such an evaluation value, for example, in order to interpret the machine learning model. Examples of the technique of deriving the degree of contribution include a method, such as local interpretable model-agnostic explanations (LIME). In addition, a data group in which a plurality of data are grouped is used as the input data to output the output data from the machine learning model. For example, JP 2020-113218 A discloses a machine learning model in which a text including a plurality of word data is used as the input data. JP 2020-113218 A describes a technique of assigning a degree of contribution to a classification result for each word obtained by dividing the text in the machine learning model that uses the text as the input data and outputs the classification result.

However, it cannot be said that in the related art is sufficient to obtain the evaluation value in the machine learning model that uses a document data group including a plurality of document data as input. For example, in the technique described in JP 2020-113218 A, in a case in which the text is the document data group including the plurality of document data, the degree of contribution for each word can be derived, whereas it is insufficient to derive the degree of contribution for each document data.

SUMMARY

The present disclosure has been made in view of the above circumstances, and is to provide an information processing apparatus, a learning apparatus, an information processing method, an information processing system, a learning method, an information processing program, and a learning program which can obtain, for each document data, an evaluation value in a machine learning model that uses a document data group including a plurality of document data as input.

In order to achieve the above object, a first aspect of the present disclosure relates to an information processing apparatus comprising: at least one processor, in which the processor is configured to: for a machine learning model that uses a document data group including a plurality of document data as input and outputs output data, derive an evaluation value in the machine learning model for each document data included in the document data group.

A second aspect relates to the information processing apparatus according to the first aspect, in which the processor is configured to: perform at least one of specification of the document data, which is a display target, from the document data group or specification of a display order of a document according to the document data based on the derived evaluation value.

A third aspect relates to the information processing apparatus according to the first aspect, in which the processor is configured to: use each document data as input of the machine learning model to acquire document unit output data which is output for each document data; and derive the evaluation value for each document data based on the document unit output data.

A fourth aspect relates to the information processing apparatus according to the third aspect, in which the evaluation value has a correlation with the document unit output data.

A fifth aspect relates to the information processing apparatus according to the first aspect, in which the processor is configured to: normalize each document data included in the document data group; and derive the evaluation value for each normalized document data.

A sixth aspect relates to the information processing apparatus according to the first aspect, in which the processor is configured to: extract a plurality of word data from each document data included in the document data group; derive the evaluation value in the machine learning model as a word unit evaluation value for each word data; and derive the evaluation value according to a statistical value of the word unit evaluation value of the word data included in the document data for each document data.

A seventh aspect relates to the information processing apparatus according to the first aspect, in which the document data having a greatest first evaluation value, which is derived for each document data, is used as first document data, and each of the plurality of document data other than the first document data included in the document data group is used as second document data, and the processor is configured to: use each combination data in which the first document data and the second document data are combined as input of the machine learning model to derive a second evaluation value from output data which is output for each combination data.

An eighth aspect relates to the information processing apparatus according to the seventh aspect, in which the processor is configured to: give a first display priority to the first document data; and give a second display priority, which is lower than the first display priority, to the second document data based on the second evaluation value.

A ninth aspect relates to the information processing apparatus according to the first aspect, in which the processor is configured to: extract a plurality of word data from each document data included in the document data group; derive the evaluation value in the machine learning model as a word unit evaluation value for each word data; derive a first statistical value of the word unit evaluation value of the word data included in the document data for each document data to give a first evaluation value to first evaluation value document data which is the document data having a greatest first statistical value; derive, for a plurality of combination data in which the first evaluation value document data, and each of the plurality of document data other than the first evaluation value document data included in the document data group are combined, a second statistical value of the word unit evaluation value of the word data included in the combination data for each combination data to give a second evaluation value, which is lower than the first evaluation value, to second evaluation value document data which is the document data having a greatest second statistical value; and set, in derivation of the second statistical value, the word unit evaluation value of the word data included in the first evaluation value document data among the word data included in the document data combined with the first evaluation value document data to be relatively lower than the word unit evaluation value of the word data which is not included in the first evaluation value document data.

A tenth aspect relates to a learning apparatus of a machine learning model that uses a plurality of document data as input and outputs output data, the learning apparatus comprising: at least one processor, in which the processor is configured to: use, for a plurality of document data for training, each document data for training as input of the machine learning model to acquire output data which is output for each document data for training; calculate, for a part of the document data for training from the plurality of document data for training, a loss function representing a degree of difference between correct answer data and the output data for each document data for training based on the output data obtained for each document data for training and the correct answer data; and update the machine learning model based on the loss function.

An eleventh aspect relates to the learning apparatus according to the tenth aspect, in which the processor is configured to: extract the part of document data for training based on a degree of similarity between the output data and the correct answer data.

A twelfth aspect relates to the learning apparatus according to the tenth aspect, in which the processor is configured to: calculate, also for another document data for training other than the part of document data for training, the loss function with a weight smaller than a weight of the part of document data for training for each data for training; and update the machine learning model based also on the loss function of the other document data for training.

A thirteenth aspect relates to the learning apparatus according to the tenth aspect, in which the processor is configured to: calculate, for the part of document data for training, the loss function by performing weighting based on the output data obtained for each document data for training and the correct answer data.

A fourteenth aspect relates to the learning apparatus according to the thirteenth aspect, in which the processor is configured to: set weighting to be larger as a degree of similarity between the output data and the correct answer data is higher.

A fifteenth aspect relates to the learning apparatus according to the tenth aspect, in which the processor is configured to: repeatedly update the machine learning model based on the loss function obtained from the part of document data for training; and change the number of the part of document data for training to be extracted, according to the number of updates of the machine learning model.

A sixteenth aspect relates to the learning apparatus according to the tenth aspect, in which each document data for training is given with a label representing a type of an associated prediction result of the machine learning model, and the processor is configured to: extract the document data for training for each type of the label.

A seventeenth aspect relates to an information processing apparatus comprising: at least one processor, in which the processor is configured to: for a machine learning model that uses a document data group including a plurality of document data as input and outputs output data, derive an evaluation value in the machine learning model for each document data included in the document data group, and the machine learning model is a machine learning model trained by a learning apparatus of the machine learning model that uses the document data group including the plurality of document data as input and outputs the output data, the learning apparatus including: at least one processor for training, in which the processor for training is configured to: use each document data for training included in a document data group for training as input of the machine learning model to acquire output data which is output for each document data for training; calculate, for a part of the document data for training from the document data group for training, a loss function representing a degree of difference between correct answer data and the output data for each document data for training based on the output data obtained for each document data for training and the correct answer data; and update the machine learning model based on the loss function.

An eighteenth aspect relates to an information processing system comprising: the information processing apparatus according to the present disclosure; and the learning apparatus according to the present disclosure.

In addition, in order to achieve the above object, a nineteenth aspect of the present disclosure relates to an information processing method executed by a processor of an information processing apparatus including at least one processor, the information processing method comprising: for a machine learning model that uses a document data group including a plurality of document data as input and outputs output data, deriving an evaluation value in the machine learning model for each document data included in the document data group.

In addition, in order to achieve the above object, a twentieth aspect of the present disclosure relates to an information processing program causing a processor of an information processing apparatus including at least one processor, to execute a process comprising: for a machine learning model that uses a document data group including a plurality of document data as input and outputs output data, deriving an evaluation value in the machine learning model for each document data included in the document data group.

In addition, a twenty-first aspect of the present disclosure relates to a learning method comprising: via a processor, using, for a plurality of document data for training, each document data for training as input of a machine learning model to acquire output data which is output for each document data for training; calculating, for a part of the document data for training from the plurality of document data for training, a loss function representing a degree of difference between correct answer data and the output data for each document data for training based on the output data obtained for each document data for training and the correct answer data; and updating the machine learning model based on the loss function.

In addition, a twenty-second aspect of the present disclosure relates to a learning program causing a processor to execute a process comprising: using, for a plurality of document data for training, each document data for training as input of a machine learning model to acquire output data which is output for each document data for training; calculating, for a part of the document data for training from the plurality of document data for training, a loss function representing a degree of difference between correct answer data and the output data for each document data for training based on the output data obtained for each document data for training and the correct answer data; and updating the machine learning model based on the loss function.

According to the present disclosure, it is possible to obtain the evaluation value for each document data in the machine learning model that uses the document data group including the plurality of document data as input.

DETAILED DESCRIPTION

Hereinafter, the description of embodiments of the present disclosure will be made in detail with reference to the drawings. It should be noted that the present embodiment does not limit the technique of the present disclosure.

First Embodiment

First, one example of an overall configuration of an information processing system according to the present embodiment will be described.FIG.1shows a configuration diagram showing one example of an overall configuration of an information processing system1according to the present embodiment. As shown inFIG.1, the information processing system1according to the present embodiment comprises an information processing apparatus10and a patient information database (DB)14. The information processing apparatus10and the patient information DB14are connected to each other via a network19by the wired communication or the wireless communication.

Patient information15related to a plurality of patients is stored in the patient information DB14. The patient information DB14is realized by a storage medium, such as a hard disk drive (HDD), a solid state drive (SSD), and a flash memory, provided in a server apparatus in which a software program for providing functions of a database management system (DBMS) to a general-purpose computer is installed.

As one example, the patient information15according to the present embodiment is document data15D representing a document related to medical care of a specific patient. As shown inFIG.2, the document data15D includes, for example, medical record information, patient profile information, and examination result information. It should be noted that, in the present embodiment, the “document” is information in which at least one of a word or a sentence is a constituent element. For example, the document may include only one word, or may include a plurality of sentences. In the example shown inFIG.2, as the document data15D which is the medical record information, five of “9/5S”, “9/50”, “9/5A”, “9/7O”, and “9/7P” are shown. In addition, as the document data15D which is the patient profile information, two of “age/gender” and “previous disease” are shown. In addition, as the document data15D which is the examination result information, two of “albumin” (examination value of albumin) and “urea/nitrogen” (examination value of urea and examination value of nitrogen) are shown.

The patient information15is stored in the patient information DB14in association with identification information for identifying the patient for each specific patient. The patient information15according to the present embodiment is one example of a document data group according to the present disclosure, and the document data15D according to the present embodiment is one example of document data according to the present disclosure.

The information processing apparatus10is an apparatus having a function of providing a user with a prognosis prediction result using a prognosis prediction model32, and the patient information15according to a degree of influence on the prognosis prediction result, regarding any patient. The prognosis prediction model32according to the present embodiment is one example of a machine learning model according to the present disclosure.

In the prognosis prediction model32according to the present embodiment is a model that outputs a probability that the patient is in a death state, specifically, a death probability as a prognosis prediction result16in a case in which the patient information15is input, as shown inFIG.2. It should be noted that, in the present embodiment, a prognosis prediction result16A (seeFIG.6) output in a case in which all the document data15D included in the patient information15are input, a prognosis prediction result16B (seeFIG.6) output in a case in which each document data15D is input, and the like are collectively referred to without distinction, the prognosis prediction result output from the prognosis prediction model32is simply referred to as the prognosis prediction result16.

As shown inFIG.3as one example, the prognosis prediction model32according to the present embodiment is trained by being given with training data90, which is also called train data or teacher data, in a training phase. The training data90is a set of patient information for training95and a correct answer prognosis prediction result96C. The patient information for training95includes a plurality of document data for training95D related to the medical care of a certain patient. The correct answer prognosis prediction result96C is, for example, the death probability obtained from a result of actually observing the prognosis of the patient. Specifically, it is assumed that “the death probability of the patient who has actually died is 1 (100%)” and “the death probability of a patient who has not died is 0 (0%)”. It should be noted that the death probability is not limited to 100% and 0%, and various adjustments can be made. For example, in a case in which a period until death is long, the death probability may be reduced from 100%. It should be noted that, the present disclosure is not limited to the present embodiment, and as the correct answer prognosis prediction result96C, for example, the death probability actually given to the patient by a doctor with reference to the document data for training95D may be used.

In the training phase, the patient information for training95is vectorized and input to the prognosis prediction model32for each document data for training95D. The prognosis prediction model32outputs a prognosis prediction result for training96to the patient information for training95. A loss calculation of the prognosis prediction model32using a loss function is performed based on the prognosis prediction result for training96and the correct answer prognosis prediction result96C. Then, various coefficients of the prognosis prediction model32are subjected to update setting according to a result of the loss calculation, and the prognosis prediction model32is updated according to the update setting.

In the training phase, the series of pieces of processing of the input of the patient information for training95to the prognosis prediction model32, the output of the prognosis prediction result for training96from the prognosis prediction model32, the loss calculation, the update setting, and the update of the prognosis prediction model32are repeatedly performed while exchanging the training data90. The series of repetitions are terminated in a case in which the prediction accuracy of the prognosis prediction result for training96with respect to the correct answer prognosis prediction result96C reaches a predetermined set level. As described above, the trained prognosis prediction model32is generated.

As shown inFIG.4, the information processing apparatus10according to the present embodiment comprises a controller20, a storage unit22, a communication interface (I/F) unit24, an operation unit26, and a display unit28. The controller20, the storage unit22, the communication OF unit24, the operation unit26, and the display unit28are connected to each other via a bus29such as a system bus or a control bus so that various types of information can be exchanged.

The controller20according to the present embodiment controls an overall operation of the information processing apparatus10. The controller20is a processor, and comprises a central processing unit (CPU)20A. In addition, the controller20is connected to the storage unit22to be described below. It should be noted that the controller20may comprise a graphics processing unit (GPU).

The operation unit26is used by the user to input, for example, an instruction or various types of information related to the prognosis prediction of the specific patient. The operation unit26is not particularly limited, and examples thereof include various switches, a touch panel, a touch pen, and a mouse. The display unit28displays the prognosis prediction result16, the document data15D, various types of information, and the like. It should be noted that the operation unit26and the display unit28may be integrated into a touch panel display.

The communication OF unit24performs communication of various types of information with the patient information DB14via the network19by the wireless communication or the wired communication. The information processing apparatus10receives the patient information15from the patient information DB14via the communication OF unit24by the wireless communication or the wired communication.

The storage unit22comprises a read only memory (ROM)22A, a random access memory (RAM)22B, and a storage22C. Various programs and the like executed by the CPU20A are stored in the ROM22A in advance. Various data are transitorily stored in the RAM22B. The storage22C stores an information processing program30, the prognosis prediction model32, various types of other information, and the like executed by the CPU20A. The storage22C is a non-volatile storage unit, and is, for example, an HDD or an SSD.

Further,FIG.5shows a functional block diagram of one example of the configuration of the information processing apparatus10according to the present embodiment. As shown inFIG.5, the information processing apparatus10comprises an acquisition unit40, a prognosis prediction result derivation unit41, a document extraction unit42, a pre-processing unit44, a prognosis prediction result derivation unit46, a post-processing unit48, an evaluation value derivation unit49, and a display controller50. As one example, in the information processing apparatus10according to the present embodiment, in a case in which the CPU20A of the controller20executes the information processing program30stored in the storage22C, the CPU20A functions as the acquisition unit40, the prognosis prediction result derivation unit41, the document extraction unit42, the pre-processing unit44, the prognosis prediction result derivation unit46, the post-processing unit48, the evaluation value derivation unit49, and the display controller50.

The acquisition unit40has a function of acquiring the patient information15of the specific patient from the patient information DB14. As one example, in a case in which the acquisition unit40according to the present embodiment receives patient identification information representing the specific patient who is a target of the prognosis prediction, the acquisition unit40acquires the patient information15corresponding to the received patient identification information from the patient information DB14via the network19. The acquisition unit40outputs the acquired patient information15to the prognosis prediction result derivation unit41and the document extraction unit42.

The prognosis prediction result derivation unit41is used to train the prognosis prediction model32. As shown inFIG.6, the prognosis prediction result derivation unit41vectorizes all the document data15D included in the patient information15, inputs the vectorized document data15D to the prognosis prediction model32, and acquires the output prognosis prediction result16A in a unit of the patient information. In other words, the prognosis prediction result derivation unit41derives the prognosis prediction result16A for each patient information15by using the prognosis prediction model32.

The document extraction unit42has a function of extracting the document data15D from the patient information15based on a predetermined reference. As one example, the document extraction unit42according to the present embodiment extracts the document data15D in a unit of a single sentence, by using one single sentence included in the patient information15as one document data15D. It should be noted that the reference for extracting the document data15D from the patient information15is not particularly limited, and for example, the association date may be the same as the reference. In such a case, for example, in the example shown inFIG.6, as one document data15D, “9/5: A, 9/5: P, 9/5: S, 9/5: O” is extracted. The document extraction unit42outputs the extracted document data15D to the pre-processing unit44.

The pre-processing unit44has a function of performing pre-processing with respect to the extracted document data15D before inputting to the prognosis prediction model32. A length of a text is different between the entire patient information15and the extracted document data15D. Therefore, in the present embodiment, the normalization for adjusting the length of the text of the document data15D to the connected length of the texts of all the document data15D included in the patient information15is performed as the pre-processing. It should be noted that the normalization method is not particularly limited. For example, a method may be adopted in which a value in a case of vectorizing the document data15D for inputting to the prognosis prediction model32is normalized by the number of the document data15D included in the patient information15. Further, for example, a method may be adopted in which the extracted document data15D are repeatedly connected to obtain the length that can be regarded as equivalent to the connected length of the texts of all the document data15D included in the patient information15.

It should be noted that the pre-processing by the pre-processing unit44is not always needed. For example, in a case in which a machine learning model which is not affected by the length of the input document (text), such as averaging the values of the input vectors, is adopted as the prognosis prediction model32, pre-processing does not have to be performed.

The pre-processing unit44outputs the document data15D, which is subjected to the pre-processing, to the prognosis prediction result derivation unit46. It should be noted that, in a case in which the normalization is performed as described above, in a case of the document data15D in which the text is short, particularly the document data15D in which the text is a word sentence including only one word, an evaluation value17to be described in detail below tends to be high. Therefore, the pre-processing unit44does not have to output the document data15D in which the length of the sentence (text) is relatively short, for example, the document data15D in which the total number of included words is equal to or lower than a predetermined number to the prognosis prediction result derivation unit46.

As shown inFIG.6, the prognosis prediction result derivation unit46has a function of, for each document data15D, vectorizing the document data15D, inputs the vectorized document data15D to the prognosis prediction model32, and acquiring the output prognosis prediction result16B in a unit of the document. It should be noted that, in addition to the vectorized document data15D, the patient profile information and the examination result information used in the training phase of the prognosis prediction model32may be used as input information. The prognosis prediction result derivation unit46outputs the acquired prognosis prediction result16B for each patient information15to the post-processing unit48.

The post-processing unit48has a function of performing post-processing on the prognosis prediction result16B with respect to the pre-processing which is performed. As described above, in a case in which the normalization is performed, the evaluation value17to be described in detail below tends to be high in the document data in which the sentence (text) is short. Therefore, the post-processing unit48performs correction as the post-processing. For example, the post-processing unit48may perform the post-processing of performing the normalization by adding a sentence (text) length to the prognosis prediction result16B. As the post-processing in such a case, for example, the post-processing unit48may normalize the prognosis prediction result16B by the following expression (1).

log(number of words included in document data 15D)×prognosis prediction result 16B(1)

The post-processing unit48outputs the prognosis prediction result16B, which is subjected to the post-processing, to the evaluation value derivation unit49.

The evaluation value derivation unit49derives the evaluation value17for each document data15D according to the prognosis prediction result16, which is subjected to the post-processing. The evaluation value17according to the present embodiment has a correlation with the prognosis prediction result16B in a unit of the document. As one example, in the present embodiment, since the prognosis prediction model32is a model that derives the probability that the patient is in the death state and outputs the death probability as the prognosis prediction result16B, the value of the evaluation value17is higher as the value of the prognosis prediction result16B in a unit of the document is higher. It should be noted that, in a case in which the prognosis prediction model32is a model that outputs a survival probability having a reciprocal relationship with the death probability as the prognosis prediction result16B as the derivation of the probability that the patient is in the death state, unlike the present embodiment, the value of the evaluation value17is higher as the value of the prognosis prediction result16B in a unit of the document is lower. As described above, in the present embodiment, the value of the evaluation value17is higher as it is predicted that the death state is more likely to occur. Stated another way, the value of the evaluation value17is higher as the prognosis prediction result16B shows a more extreme value. It should be noted that, in the present embodiment, the evaluation value17is represented as a specific numerical value, but may be represented by, for example, “high”, “medium”, “low”, or the like. The evaluation value derivation unit49outputs the evaluation value17derived for each document data15D to the display controller50.

The display controller50specifies the document data15D, which is a display target, from among all the plurality of document data15D included in the patient information15based on the evaluation value17for each document data15D. For example, the display controller50specifies a predetermined number of the document data15D as the display targets in descending order of the evaluation value17. In addition, the display controller50specifies the document data15D of which the evaluation value17is equal to or higher than a predetermined value as the display target.

Further, in a case of specifying the display target, the document data15D may be selected one by one by using a method of Beam Search. In such a case, first, the display controller50extracts the highest K document data15D in the ranking of the evaluation value17from all the document data15D included in the patient information15as the document data15D to which a first display priority having the highest display priority is given. Then, other document data15D included in the remaining document data15D included in the patient information15are added to the extracted document data15D and ranked based on the evaluation value17, and a second display priority, which is the next to the first display priority, is given to the highest K document data15D. This processing is repeated until a predetermined number of the document data15D are specified or the total length obtained by adding the lengths of all the document data15D to which the display priority is given reaches a predetermined length.

In addition, the display controller50specifies a display order in which the document data15D is displayed based on the evaluation value17. For example, the display controller50specifies the display order such that the display priority is raised in descending order of the evaluation value17. It should be noted that the display controller50may adopt, as the display order, a time-series order based on the date and time associated with the document data15D. In a case in which the display order is the time-series order, the display priority is higher as the date and time are newer. In addition, the display order in which the order according to the evaluation value17and the time-series order are combined may be adopted. It should be noted that, in such a case, the burden on the user who reads the document data15D is larger as the document data15D is longer, and thus the length of the document data15D may be added as a penalty. Specifically, the penalty that is larger as the length of the document data15D is longer may be added.

It should be noted that, in a case in which at least one of the display target or the display order is determined in advance, the display controller50need only specify which of the display target and the display order is not determined in advance, and may omit the specification of the display target and the specification of the display order in a case in which both the display target and the display order are determined in advance. For example, in a case in which it is determined in advance that all the document data15D are used as the display targets, the display controller50need only specify the display order.

In addition, the display controller50performs control of displaying the document data15D specified as the display target on the display unit28in the specified display order. It should be noted that the display controller50may also control of displaying the prognosis prediction result16A derived by the prognosis prediction result derivation unit41on the display unit28.

Hereinafter, an action of the information processing apparatus10according to the present embodiment will be described with reference to the drawings.FIG.7shows a flowchart showing one example of a flow of information processing executed by the information processing apparatus10according to the present embodiment. The information processing apparatus10according to the present embodiment executes the information processing shown inFIG.7in a case in which the CPU20A of the controller20executes the information processing program30stored in the storage22C based on a start instruction or the like of the user performed by the operation unit26, as one example.

In step S100ofFIG.7, as described above, the acquisition unit40receives the patient identification information designated by the user using the operation unit26. In next step S102, as described above, the acquisition unit40acquires the patient information15associated with the patient identification information from the patient information DB14via the network19.

In next step S104, as described above, the prognosis prediction result derivation unit41derives the prognosis prediction result16A in a unit of the patient information by using all the document data15D included in the patient information15as input of the prognosis prediction model32. In next step S106, as described above, the document extraction unit42extracts one document data15D from the patient information15. In next step S108, as described above, the pre-processing unit44performs the pre-processing on the document data15D and normalizes the length of the document data15D.

In next step S110, as described above, the prognosis prediction result derivation unit46derives the prognosis prediction result16B in a unit of the document by using the document data15D extracted in step S106as input of the prognosis prediction model32. In next step S112, as described above, the post-processing unit48performs the post-processing on the prognosis prediction result derivation unit46B in a unit of the document, and performs the normalization.

In next step S114, the document extraction unit42determines whether or not the prognosis prediction result16B is derived for all the document data15D included in the patient information15. In a case in which the prognosis prediction result16B is not yet derived for all the document data15D, a negative determination is made in the determination of step S114, the processing returns to step S106, and the pieces of processing of steps S106to S112are repeated. On the other hand, in a case in which the prognosis prediction result16B is derived for all the document data15D, a positive determination is made in the processing of step S114, and the processing proceeds to step S116.

In step S116, as described above, the evaluation value derivation unit49derives the evaluation value17having the correlation with the prognosis prediction result16B in a unit of the document for each document data15D. In next step S118, as described above, the display controller50specifies the display target from among all the document data15D included in the patient information15, and also specifies the display order of the document data15D, which is the display target.

In next step S119, as described above, the display controller50displays the document corresponding to the document data15D, which is the display target, on the display unit28in the specified display order.FIG.8is a diagram showing one example of a state in which the document data15D, which is the display target, is displayed on the display unit28in the specified display order. In the example shown inFIG.8, the first display priority is given to document data15D1, and the second display priority is given to document data15D2. In this way, by displaying the document data15D which is the display target specified in step S118on the display unit28in the specified display order, useful information for the specific patient for which the prognosis prediction is performed by the prognosis prediction model32is displayed in descending order of a degree of importance. In a case in which the processing of step S119is terminated, the information processing shown inFIG.7is terminated.

It should be noted that, in the present embodiment, the embodiment is described in which the evaluation value17is derived based on the prognosis prediction result16B in a unit of the document output from the prognosis prediction model32, but the present disclosure is not limited to the present embodiment. For example, information processing according to a modification example 1 may be applied.

Modification Example 1

FIG.9shows a flowchart showing one example of a flow of information processing executed by the information processing apparatus10according to the present modification example. The pieces of processing of steps S100to S114are the same as the pieces of processing of steps S100to S114of the information processing described above with reference toFIG.7, and thus the description thereof will be omitted.

In next step S116, the evaluation value derivation unit49derives the evaluation value17having the correlation with the prognosis prediction result16B in a unit of the document for each document data15D, in the same manner as in step S116of the information processing shown inFIG.9. It should be noted that the evaluation value given by this processing is used as a first evaluation value.

In next step S120, the document extraction unit42specifies first document data and second document data from among the document data15D included in the patient information15based on the first evaluation value. As one example, the document extraction unit42according to the present embodiment specifies the document data15D having the highest first evaluation value as the first document data, and specifies the document data15D other than the first document data included in the patient information15as the second document data.

In the example shown inFIG.10, the document extraction unit42specifies, from among the document data15D1to15D3, the document data15D2as the first document data and specifies the document data15D1and15D3as the second document data, based on the evaluation value17.

In next step S122, the document extraction unit42extracts combination data in which one of a plurality of second document data is combined with the first document data specified in step S120. In the example shown inFIG.10, the combination data in which the document data15D2, which is the first document data, and the document data15D1, which is the second document data, are combined, and the combination data in which the document data15D2, which is the first document data, and the document data15D3, which is the second document data, are combined are shown.

In next step S124, as described above, the pre-processing unit44performs the pre-processing on the combination data extracted in step S122, and normalizes the length of the combination data.

In next step S126, as shown inFIG.10, the prognosis prediction result derivation unit46derives a prognosis prediction result16C in a unit of the combination data as input of the prognosis prediction model32by using the combination data extracted in step S122. In next step S128, as described above, the post-processing unit48performs the post-processing on the prognosis prediction result16C in a unit of the combination data, and performs the normalization.

In next step S130, the document extraction unit42determines whether or not the prognosis prediction result16C is derived for all the combination data. In a case in which the prognosis prediction result16C is not yet derived for all the combination data, a negative determination is made in the determination of step S130, the processing returns to step S122, and the pieces of processing of steps S122to S128are repeated. In other words, the processing of deriving the prognosis prediction result16C in a unit of the combination data is sequentially repeated by varying the second document data to be combined with the first document data. On the other hand, in a case in which the prognosis prediction result16C is derived for all the combination data, a positive determination is made in the processing of step S130, and the processing proceeds to step S132.

In step S132, as described above, the evaluation value derivation unit49derives the evaluation value17having the correlation with the prognosis prediction result16C in a unit of the combination data for each combination data. The evaluation value17derived here is used as a second evaluation value.

In next step S134, the display controller50specifies the display target. Here, the first document data is specified as the display target. In addition, the document data15D, which is the display target, is specified from among the plurality of document data15D as the second document data based on the second evaluation value. For example, the display controller50specifies the document data15D having the highest second evaluation value as the display target. It should be noted that, from the meaning that the document data15D specified as the display target from among the document data15D used as the second document data is added to the first document data and used as the display target, the term “additional document data” is used.

In next step S136, the document extraction unit42determines whether or not to terminate the addition of the document data15D, which is the display target. As one example, the document extraction unit42according to the present embodiment terminates the addition of the document data15D in a case in which a predetermined termination condition is satisfied. Examples of the predetermined termination condition include a case in which the number of the document data15D, which is the display target, reaches a predetermined number, and a case in which the total length of the lengths of the texts of the plurality of document data15D, which are the display targets, is equal to or longer than a predetermined length. In a case in which the predetermined termination condition is not satisfied, a negative determination is made in the determination in step S136, and the processing proceeds to step S138. In step S138, the document extraction unit42specifies the first document data and the second document data again. Here, the document data in which the document data15D, which is the additional document data, is added to the document data15D previously used as the first document data is specified as new first document data. In addition, the document data15D other than the new first document data included in the patient information15is specified as the second document data.

On the other hand, in step S136, in a case in which the termination condition is satisfied, a negative determination is made in the determination, and the processing proceeds to step S140. In step S140, the display controller50displays the document corresponding to the document data15D, which is the display target, on the display unit28, in the same manner as in step S119of the information processing shown inFIG.7. In a case in which the processing of step S119is terminated, the information processing shown inFIG.9is terminated.

Second Embodiment

In the present embodiment, an embodiment will be described in which the evaluation value17is derived based on the prognosis prediction result16B in a unit of the word output from the prognosis prediction model32.FIG.11shows a functional block diagram of one example of the configuration of the information processing apparatus10according to the present embodiment. The information processing apparatus10according to the present embodiment is different from the information processing apparatus10(seeFIG.5) according to the first embodiment in that the pre-processing unit44, the prognosis prediction result derivation unit46, and the post-processing unit48are not provided, and a word extraction unit43is further provided.

The word extraction unit43has a function of extracting word data15W from all the document data15D included in the patient information15acquired by the acquisition unit40. It should be noted that the method by which the word extraction unit43extracts the word data15W from the document data15D is not particularly limited. For example, the word extraction unit43may extract the morphological elements obtained by performing the morphological element analysis with a known morphological element analyzer, such as JUMAN, as the word data15W. The word extraction unit43outputs all the extracted word data15W to the prognosis prediction result derivation unit41and the prognosis prediction result derivation unit46.

As shown inFIG.12, the prognosis prediction result derivation unit41according to the present embodiment vectorizes all the word data15W, inputs the vectorized word data15W to the prognosis prediction model32, and acquires the output prognosis prediction result16D in a unit of the patient information. In other words, the prognosis prediction result derivation unit41derives the prognosis prediction result16D for each patient information15by using the prognosis prediction model32. It should be noted that, in a case in which the word data15W is vectorized, a known term frequency-inverse document frequency (TF-IDF) or bag of words (BoW) may be applied to perform the vectorization.

On the other hand, the evaluation value derivation unit49according to the present embodiment derives an evaluation value17A for each word data15W according to the prognosis prediction result16D. As the evaluation value used here, a so-called “degree of contribution” to the machine learning model obtained by a method, such as the LIME, a so-called “contribution feature amount” to the machine learning model obtained by a gradient boosting decision tree (GBDT), and the like can be applied. In addition, the evaluation value derivation unit49derives the evaluation value17(evaluation value17in a unit of the document) for each document data15D based on the evaluation value17A in a unit of the word. As one example, as shown inFIG.12, the evaluation value derivation unit49according to the present embodiment derives an addition value (total value) obtained by adding the evaluation value17A of the word data15W included in the document data15D as the evaluation value17of the document data15D. In the example shown inFIG.12, for the document data15D that “right hand is numb”, the word data15W of “numb” is included, and thus an evaluation value 1 of the word data15W of “numb” is used as a value of the evaluation value17of the document data15D. In addition, for the document data15D of “acute phase treatment for cerebral infarction”, two word data15W of “cerebral infarction” and “acute phase treatment” are included, and thus a value (8+4=12) obtained by adding an evaluation value 8 of the word data15W of “cerebral infarction” and an evaluation value 4 of the word data15W of “acute phase treatment” is used as a value of the evaluation value17of the document data15D. It should be noted that, in a case in which the document data15D includes a plurality of the same word data15W, for the same word data15W, the evaluation value17A may be lowered and added, or the evaluation value17A does not have to be added. It should be noted that the total value of the evaluation values17A according to the present embodiment is one example of a statistical value according to the present disclosure.

As described above, in the present embodiment, the total value of the evaluation values17A of the word data15W included in the document data15D is used as the evaluation value17in a unit of the document, but a value other than the total value may be used, and the statistical value obtained from the evaluation value17in a unit of the document need only be used. For example, an average value obtained by dividing the total value by the number of added word data15W or the number of nouns may be used as the evaluation value17of the document data15D.

The evaluation value derivation unit49outputs the derived evaluation value17in a unit of the document to the display controller50.

Similar to the display controller50according to the first embodiment, the display controller50according to the present embodiment specifies the document data15D, which is the display target, and specifies the display order based on the evaluation value17in a unit of the document.

Hereinafter, an action of the information processing apparatus10according to the present embodiment will be described with reference to the drawings.FIG.13shows a flowchart showing one example of a flow of information processing executed by the information processing apparatus10according to the present embodiment.

In step S200ofFIG.13, the acquisition unit40receives the patient identification information in the same manner as in step S100of the information processing (seeFIG.7) according to the first embodiment. In next step S202, the acquisition unit40acquires the patient information15associated with the patient identification information from the patient information DB14via the network19in the same manner as in step S102of the information processing (seeFIG.7) according to the first embodiment.

In next step S204, as described above, the word extraction unit43extracts all the word data15W from all the document data15D included in the patient information15acquired in step S202by the morphological element analysis or the like.

In next step S206, as described above, the prognosis prediction result derivation unit41derives the prognosis prediction result16D in a unit of the patient information (in a unit of all the words) by using all the word data15W included in the patient information15as input of the prognosis prediction model32.

In next step S210, as described above, the evaluation value derivation unit49derives the evaluation value17A in a unit of the word, which is the degree of contribution or the like.

In next step S212, the evaluation value derivation unit49extracts one document data15D from the patient information15. Next step S214, the evaluation value derivation unit49derives the evaluation value17(evaluation value17in a unit of the document) of the document data15D extracted in step S212based on the evaluation value17A in a unit of the word.

In next step S216, the evaluation value derivation unit49determines whether or not the evaluation value17in a unit of the document is derived for all the document data15D included in the patient information15. In a case in which the evaluation value17in a unit of the document is not yet derived for all the document data15D, a negative determination is made in the determination in step S216, the processing returns to step S212, and the pieces of processing of steps S212and S214are repeated. On the other hand, in a case in which the evaluation value17in a unit of the document is derived for all the document data15D, a positive determination is made in the determination in step S216, and the processing proceeds to step S218.

In step S218, the display controller50specifies the display target and the display order from among all the document data15D included in the patient information15based on the evaluation value17in a unit of the document, as described above. In next step S220, in the same manner as in step S119of the information processing (seeFIG.7) according to the first embodiment, the display controller50displays the document corresponding to the document data15D, which is the display target, on the display unit28. In a case in which the processing of step S220is terminated, the information processing shown inFIG.13is terminated.

It should be noted that the display controller50may display at least one of the evaluation value17of the document data15D or the word data15W having a high the evaluation value17A included in the document data15D in association with each document data15D. For example, the display controller50may display the word data15W of which the evaluation value17A is higher than a certain threshold value, or the word data15W whose number is equal to or larger than a predetermined threshold value (for example, 3) in descending order of the evaluation value17A among the word data15W included in the document data15D. Further, in the above, a case in which the display controller50specifies the display target and the display order, have been described. However, the present disclosure is not limited thereto. For example, a display form may be changed based on the evaluation value17, instead of the display target and the display order. Here, the display form may include, for example, color of cell, term, or sentence of the document data15D. Further, changing the display form may include, for example, changing the color of the document data that has respectively high evaluation value to a color that a user can easily pay attention to, compared to the document data that has respectively low evaluation value.

It should be noted that, in the present embodiment as well, the pre-processing, the post-processing, or the like performed in the information processing of the first embodiment may be performed.

It should be noted that, in the present embodiment, the embodiment of the modification example 1 of the first embodiment may be combined.FIG.14shows a flowchart showing one example of a flow of information processing executed by the information processing apparatus10in such a case.

In such a case, the word extraction unit43extracts a plurality of word data15W from each document data15D included in the patient information15, as described above. In addition, the evaluation value derivation unit49derives the evaluation value17A in a unit of the word for each word data15W, as described above. In addition, the evaluation value derivation unit49derives the evaluation value17for each document based on the evaluation value17A of the word data15W included in the document data15D for each document data15D, and gives the first evaluation value to the document data15D having the greatest evaluation value17for each document as first evaluation value document data in step S230ofFIG.14. The evaluation value17for each document in such a case is one example of a first statistical value according to the present disclosure. It should be noted that the first evaluation value need only be any value having a correlation with the evaluation value17, and specific numerical values and the like are not particularly limited.

In next step S232, the evaluation value derivation unit49extracts a plurality of combination data in which the first evaluation value document data and each of the plurality of document data15D other than the first evaluation value document data included in the patient information15are combined. In next step S234, the evaluation value derivation unit49derives the evaluation value17for each combination data based on the evaluation value17A of the word data15W included in the combination data for each combination data. It should be noted that, among the word data15W included in the document data15D to be combined with the first evaluation value document data, the evaluation value17A in a unit of the word of the word data15W included in the first evaluation value document data is made to be relatively lower than the evaluation value17A in a unit of the word of the word data15W which is not included in the first evaluation value document data. It should be noted that, as a result of setting the evaluation value17A to be relatively low, the evaluation value17A may be made to “0”. In other words, an embodiment may be adopted in which the evaluation value17A is counted only once for each word data15W.

In next step S236, the evaluation value derivation unit49determines whether or not all the combination data are extracted. In a case in which all the combination data are not yet extracted, a negative determination is made in the determination in step S236, the processing returns to step S232, and the pieces of processing of steps S232and S234are repeated. On the other hand, in a case in which all the combination data are extracted, a positive determination is made in the determination in step S236, and the processing proceeds to step S238.

In step S238, the evaluation value derivation unit49gives the second evaluation value lower than the first evaluation value to the combination data having the greatest evaluation value17derived in step S234as the second evaluation value document data. The evaluation value17for each combination data in such a case is one example of a second statistical value according to the present disclosure.

In next step S240, the display controller50specifies the document data15D, which is the display target, from among all the document data15D included in the patient information15, and specifies the display order of the document data15D, which is the display target, based on the first evaluation value and the second evaluation value.

In next step S242, as described above, the display controller50displays the document corresponding to the document data15D, which is the display target, on the display unit28in the specified display order. In a case in which the processing of step S242is terminated, the information processing shown inFIG.14is terminated.

As described above, for the prognosis prediction model32that uses the patient information15including the plurality of document data15D as input and outputs the output data, the CPU20A of the information processing apparatus10according to each embodiment described above derives the evaluation value17in the prognosis prediction model32for each document data15D included in the patient information15.

As described above, with the information processing apparatus10of each embodiment described above, the evaluation value17in the prognosis prediction model32that uses the patient information15including the plurality of document data15D as input can be obtained for each document data. As a result, since at least one of the specification of the document data15D to be provided to the user or the specification of the order of the provision can be performed based on the evaluation value17, it is possible to provide the user with useful information for the specific patient in descending order of the degree of importance.

Third Embodiment

In the present embodiment, a learning method of the machine learning model used in each embodiment described above will be described.

FIG.15shows a configuration diagram showing one example of the overall configuration of the information processing system1according to the present embodiment. As shown inFIG.15, the information processing system1according to the present embodiment is different from the information processing system1(seeFIG.1) according to the embodiment described above in that a learning apparatus60and a training information DB62are further provided. The learning apparatus60is connected to the information processing apparatus10by the wired communication or the wireless communication via the network19, and is also connected to the training information DB62by the wired communication or the wireless communication.

Training data63used to train the machine learning model is stored in the training information DB62. The training information DB62is realized by a storage medium, such as an HDD, an SSD, and a flash memory, provided in a server apparatus in which a software program for providing functions of a database management system to a general-purpose computer is installed.

As one example, as shown inFIG.16, the training data63according to the present embodiment is a set of patient information for training65and a correct answer prognosis prediction result66C. The patient information for training65includes a plurality of document data for training65D related to the medical care of a certain patient. The document data for training65D is, for example, data for each document included in each of the medical record information, the patient profile information, the examination result information, and the like. For example, inFIG.16, each of “CRP high value”, “meal is consumed”, “interview conduction”, and “slight fever continues” corresponding to “9/5 13:00” in “medical record information” is the document data for training65D. It should be noted that the document data for training65D may be in a unit of one sentence, or may be in a unit of a plurality of sentences that satisfy a predetermined reference. Examples of the predetermined reference include a reference for each category, such as “S”, “O”, “A”, and “P”, and a reference for each medical record. On the other hand, similar to the correct answer prognosis prediction result96C (seeFIG.3) according to the embodiment described above, the correct answer prognosis prediction result66C is, for example, the death probability obtained from the result of actually observing the prognosis of the patient. The patient information for training65according to the present embodiment is one example of a document data group for training according to the present disclosure, and the document data for training65D according to the present embodiment is one example of document data for training according to the present disclosure. In addition, the correct answer prognosis prediction result66C according to the present embodiment is one example of correct answer data (also called as “gold data” or “target document”) according to the present disclosure.

As shown inFIG.17, the learning apparatus60according to the present embodiment comprises a controller70, a storage unit72, a communication I/F unit74, an operation unit76, and a display unit78. The controller70, the storage unit72, the communication I/F unit74, the operation unit76, and the display unit78are connected to each other via a bus79such as a system bus or a control bus so that various types of information can be exchanged.

The controller70according to the present embodiment controls an overall operation of the learning apparatus60. The controller70is a processor, and comprises a CPU70A. It should be noted that the controller70may comprise a GPU.

The operation unit76is used for the user to input an instruction, information, and the like related to training of the prognosis prediction model82. The operation unit76is not particularly limited, and is, for example, various switches, a touch panel, a touch pen, a mouse, a microphone for voice input, and a camera for gesture input. The display unit78displays information related to training of the prognosis prediction model82and the like. It should be noted that the operation unit76and the display unit68may be integrated into a touch panel display.

The communication I/F unit64performs communication of various types of information with the information processing apparatus10via the network19by the wireless communication or the wired communication. In addition, the learning apparatus60receives the training data63from the training information DB62via the communication I/F unit74by the wireless communication or the wired communication.

The storage unit72comprises a ROM72A, a RAM72B, and a storage72C. Various programs and the like executed by the CPU70A are stored in the ROM72A in advance. Various data are transitorily stored in the RAM72B. The storage72C stores a learning program80executed by the CPU70A, a trained prognosis prediction model82, various types of other information, and the like. The storage72C is a non-volatile storage unit, and is, for example, an HDD or an SSD.

Further,FIG.18shows a functional block diagram of one example of the configuration of the learning apparatus60according to the present embodiment. As shown inFIG.18, the learning apparatus60comprises a training data acquisition unit100, a document extraction unit102, an output data acquisition unit104, an update data extraction unit106, a loss function calculation unit108, and an update unit110. As one example, in the learning apparatus60according to the present embodiment, in a case in which the CPU70A of the controller70executes the learning program80stored in the storage72C, the CPU70A functions as the training data acquisition unit100, the document extraction unit102, the output data acquisition unit104, the update data extraction unit106, the loss function calculation unit108, and the update unit110.

The training data acquisition unit100has a function of acquiring the training data63from the training information DB62. The training data acquisition unit100outputs the patient information for training65among the acquired training data63to the document extraction unit102, and outputs the correct answer prognosis prediction result66C to the update data extraction unit106.

The document extraction unit102has a function of extracting the document data for training65D from the patient information for training65based on a predetermined reference. The document extraction unit102outputs the extracted document data for training65D to the output data acquisition unit104.

The output data acquisition unit104has a function of acquiring the output data which is output from the prognosis prediction model82as a result of inputting the document data for training65D to the prognosis prediction model82. As one example, as shown inFIG.19, the output data acquisition unit104according to the present embodiment inputs the patient information for training65D extracted by the document extraction unit102one by one to the prognosis prediction model82. Specifically, the output data acquisition unit104vectorizes the document data for training65D and inputs vectorized document data for training65D to the prognosis prediction model82. Then, the output data acquisition unit104has a function of acquiring each output data120output from the prognosis prediction model82. The output data acquisition unit104outputs a plurality of acquired output data120to the update data extraction unit106. It should be noted that, in the present embodiment, as the output data120, a value obtained by converting the death probability expressed by a percentage to a small number is used. For example, “0.9” for the output data120inFIG.19means that the death probability is 90%. It should be noted that, similar to the output data120, as correct answer data124corresponding to the correct answer prognosis prediction result66C, a value obtained by correcting the death probability of the correct answer expressed by a percentage to a small number is used. For example, “1.0” for the correct answer data124inFIG.19means that the death probability is 100%.

The update data extraction unit106has a function of extracting a part of the document data for training65D, as update data for updating the prognosis prediction model82, from the plurality of document data for training65D based on the output data120and the correct answer prognosis prediction result66C. As one example, the update data extraction unit106according to the present embodiment extracts a part of the document data for training65D, as the update data, from the plurality of document data for training65D based on a degree of similarity between the output data120and the correct answer data124.

It can be regarded that the degree of similarity between the output data120and the correct answer data124is higher as the output data120has a smaller difference between the correct answer data124and the output data120. Therefore, the update data extraction unit106according to the present embodiment extracts, as the update data, the document data for training65D in a case in which the output data120of the highest X % (X is a predetermined threshold value) in descending order of the degree of similarity is output among the plurality of output data120. It should be noted that, unlike the present embodiment, an embodiment may be adopted in which the update data extraction unit106extracts, as the update data, the document data for training65D in a case of outputting the output data120in which the difference between the output data120and the correct answer data124is equal to or higher than the threshold value. The update data extraction unit106outputs the document data for training65D extracted as the update data to the loss function calculation unit108.

For the document data for training65D extracted as the update data by the update data extraction unit106, the loss function calculation unit108calculates a loss function122representing a degree of difference between the correct answer data124and the output data120for each document data for training65D. Specifically, a loss function122according to the present embodiment is an absolute value of the difference between the correct answer data124and the output data120. The loss function calculation unit108outputs the loss function122, which is a calculation result, to the update unit110.

It should be noted that, in the present embodiment, the embodiment is described in which the update data extraction unit106extracts the document data for training65D as the update data, and then the loss function calculation unit108calculates the loss function for the extracted document data for training65D, but an embodiment may be adopted in which, unlike the present embodiment, the update data extraction unit106extracts the update data simultaneously with the calculation of the loss function by the loss function calculation unit108. Specifically, the following expressions (2) and (3) may be used in a form of a calculation expression. It should be noted that, in Expressions (2) and (3), Lirepresents a loss for i-th training data, T represents a sentence set of the medical record for a certain hospitalization, yt represents a correct answer label of t-th sentence, y{circumflex over ( )}t represents an output value of the prognosis prediction model82for the t-th sentence, r represents an output order of the sentence in the hospitalization, and lTrepresents a document of the medical record for the hospitalization. α and γ are hyper parameters for determining a degree to which only a part of sentences are considered for each hospitalization.

The update unit110has a function of updating the prognosis prediction model82based on the loss function122.

By repeating each processing of the output data acquisition unit104, the update data extraction unit106, the loss function calculation unit108, and the update unit110, the accuracy of the prognosis prediction model82is improved, and the trained prognosis prediction model82is generated.

Hereinafter, an action of the learning apparatus60according to the present embodiment will be described with reference to the drawings.FIG.20shows a flowchart showing one example of a flow of learning processing executed by the information processing apparatus10according to the present embodiment. The learning apparatus60according to the present embodiment executes the learning processing shown inFIG.20in a case in which the CPU70A of the controller70executes the learning program80stored in the storage72C based on a start instruction or the like of the user performed by the operation unit76, as one example.

In step S300ofFIG.20, as described above, the training data acquisition unit100acquires the training data63from the training information DB62.

In next step S302, the document extraction unit102extracts the plurality of document data for training65D from the patient information for training65of the training data63, as described above.

In next step S304, the output data acquisition unit104inputs one of the plurality of document data for training65D extracted in step S302to the prognosis prediction model82. In next step S306, the output data acquisition unit104acquires the output data120output from the prognosis prediction model82as a result of the processing of step S304.

In next step S308, the output data acquisition unit104determines whether or not the output data120is acquired for all the document data for training65D extracted in step S302. Until the output data120is acquired for all the document data for training65D, a negative determination is made in the determination in step S308, the processing returns to step S304, and the pieces of processing of steps S304and306are repeated. On the other hand, in a case in which the output data120is acquired for all the document data for training65D, a positive determination is made in the determination in step S308, and the processing proceeds to step S310.

In step S310, as described above, the update data extraction unit106extracts the document data for training65D based on the degree of similarity between the output data120and the correct answer data124.

In next step S312, as described above, the loss function calculation unit108calculates the loss function122for the document data for training65D extracted in step S310.

In next step S314, the update unit110updates the prognosis prediction model82based on the loss function122calculated in step S312, as described above.

In next step S316, the update unit110determines whether or not to terminate the learning processing shown inFIG.20. As one example, the update unit110according to the present embodiment terminates the learning processing shown inFIG.20in a case in which the prediction accuracy of the prognosis prediction model82with respect to the correct answer data124reaches the predetermined set level. In the update unit110, in a case in which the prediction accuracy of the prognosis prediction model82with respect to the correct answer data124does not reach the predetermined set level, a negative determination is made in the determination of step S316, the processing returns to step S304, and the pieces of processing of steps S304to S314are repeated. On the other hand, in a case in which the prediction accuracy of the prognosis prediction model82with respect to the correct answer data124reaches the predetermined set level, a positive determination is made in the determination in step S316, and the learning processing shown inFIG.19is terminated.

It should be noted that a condition for terminating the learning processing is not limited to the condition described above. For example, a condition may be used in which the value of the loss function described above is not updated as compared with the previous step, or a condition may be used in which an index for measuring the performance of the document extraction is prepared and the value is not updated. It should be noted that, as the index for measuring the performance of the document extraction, a rate of match or the degree of similarity in a case of comparison between a document list extracted as the document having the degree of contribution to the prognosis prediction by using the prognosis prediction model82and a document list determined to be important academically or by the user can be considered.

Modification Example

The learning apparatus60according to the present embodiment is not limited to the embodiment described above, and various modification examples can be made.

For example, in the embodiment described above, the document data for training65D having a low relevance to the prediction result is not used to update the prognosis prediction model82, but an embodiment may be adopted in which the document data for training65D is also used to update the prognosis prediction model82. For example, an embodiment may be adopted in which, for the document data for training65D having a low relevance to the prediction result, the loss function calculation unit108calculates the loss function122with lower weighting than the document data for training65D having a high relevance to the prediction result, and the update unit110updates the prognosis prediction model82by using the loss function as well.

In addition, for the document data for training65D having a high relevance to the prediction result, the loss function calculation unit108may calculate the loss function122by performing weighting based on the output data120and the loss function122. For example, the loss function calculation unit108may calculate the loss function122by performing weighting that is larger as the degree of similarity is higher according to the degree of similarity. As a specific example, a value obtained by the following expression (1) in which G is a reverse order of the descending order of the degree of similarity, that is, an order of arrangement in the order of a low degree of similarity, and the weighting is performed by using a preset λ may be used as a weight.

Gλ/total number of document data for training 65Dthathas ahigh relevance to prediction result  (1)

In addition, instead of the expression (1) described above, a weighting value set according to the value of the output data120may be used.

In addition, in a case in which the number of updates satisfies a specific condition in a case in which the update of the prognosis prediction model82is repeated, the learning apparatus60may maintain or increase the number of the document data for training65D extracted in the processing of step S310. For example, an embodiment may be adopted in which, in every ten updates, all the document data for training65D may be extracted one time, and the document data for training65D of the highest X % having a high degree of similarity may be extracted one time.

In addition, different labels may be given to the document data for training65D based on the correct answer prognosis prediction result66C. For example, in a case in which the correct answer prognosis prediction result66C of the prognosis prediction model82indicates the probability that the correct answer prognosis prediction result66C is that immediately before discharge from the hospital, the document data for training65D is separated into the document data immediately before discharge from the hospital and the other document data, and labels corresponding to the document data immediately before discharge from the hospital and the other document data, respectively, are given. The loss function may be calculated for each of the document data groups to which the respective labels are given, and the prognosis prediction model82may be updated by using a plurality of calculated loss functions. By doing so, it is possible to generate the prognosis prediction model82suitable for extracting the document data indicating that the state is good, focusing on the fact that the state of the patient is good immediately before discharge from the hospital.

The learning apparatus60trained as described above trains the prognosis prediction model82by updating the prognosis prediction model82by preferentially using a part of the document data for training65D in which the output data120is similar to the correct answer data124among the plurality of document data for training65D. Since the prognosis prediction model82is updated without using the document data for training65D having a low relevance to the prediction result, which is included in the plurality of document data for training65D, or by using the document data for training65D having a low relevance to the prediction result while decreasing the importance, the prognosis prediction model82with higher accuracy can be generated.

In addition, the prognosis prediction model82trained by the learning apparatus60according to the present embodiment is a high-performance machine learning model that receives the document data as input. Therefore, instead of inputting each document data group to the prognosis prediction model82, each document data can be input to the prognosis prediction model82and used to obtain the prediction result.

It should be noted that, in each embodiment described above, as one example of the machine learning model according to the present disclosure, the prognosis prediction model32is described, which outputs, as the output data, the probability that a certain patient is in the death state, which is one example of a state according to a predetermined task, but the machine learning model is not limited to the prognosis prediction model32. For example, the present disclosure can also be applied to the prediction model that uses, as the input data, a document data group including a plurality of company reports including words related to personnel change information, product information, and the like as the document data15D, and outputs, as the output data, a prediction result for company trends, such as a probability that a business status of the company is deteriorated.

Further, in the embodiment described above, for example, as the hardware structure of the processing unit that executes various processing, such as the acquisition unit40, the prognosis prediction result derivation unit41, the document extraction unit42, the word extraction unit43, the pre-processing unit44, the prognosis prediction result derivation unit46, the post-processing unit48, the evaluation value derivation unit49, and the display controller50, the following various processors can be used. As described above, in addition to the CPU that is a general-purpose processor that executes software (program) to function as various processing units, the various processors include a programmable logic device (PLD) that is a processor of which a circuit configuration can be changed after manufacture, such as a field programmable gate array (FPGA), and a dedicated electric circuit that is a processor having a circuit configuration that is designed for exclusive use in order to execute specific processing, such as an application specific integrated circuit (ASIC).

One processing unit may be configured by using one of the various processors or may be configured by using a combination of two or more processors of the same type or different types (for example, a combination of a plurality of FPGAs or a combination of a CPU and an FPGA). In addition, a plurality of the processing units may be configured by using one processor.

A first example of the configuration in which the plurality of processing units are configured by using one processor is an embodiment in which one processor is configured by using a combination of one or more CPUs and the software and this processor functions as the plurality of processing units, as represented by computers, such as a client and a server. A second example thereof is an embodiment of using a processor that realizes the function of the entire system including the plurality of processing units by one integrated circuit (IC) chip, as represented by a system on chip (SoC) or the like. In this way, as the hardware structure, the various processing units are configured by using one or more of the various processors described above.

Further, more specifically, as the hardware structure of the various processors, an electric circuit (circuitry) in which circuit elements, such as semiconductor elements, are combined can be used.

In addition, in each embodiment described above, an aspect is described in which the information processing program30is stored (installed) in the storage unit22in advance, but the present disclosure is not limited to this. The information processing program30may be provided in a form of being recorded in a recording medium, such as a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), and a universal serial bus (USB) memory. Moreover, each information processing program30may be provided in a form of being downloaded from an external device via a network. That is, an embodiment may be adopted in which the program described in the present embodiment (program product) is distributed from an external computer, in addition to the provision by the recording medium.

In regard to the embodiments described above, the following appendixes will be further disclosed.

An information processing apparatus comprising: at least one processor, in which the processor is configured to: for a machine learning model that uses a document data group including a plurality of document data as input and outputs output data, derive an evaluation value in the machine learning model for each document data included in the document data group.

The information processing apparatus according to appendix 1, in which the processor is configured to: perform at least one of specification of the document data, which is a display target, from the document data group or specification of a display order of a document according to the document data based on the derived evaluation value.

The information processing apparatus according to appendix 1 or 2, in which the processor is configured to: use each document data as input of the machine learning model to acquire document unit output data which is output for each document data; and derive the evaluation value for each document data based on the document unit output data.

The information processing apparatus according to appendix 3, in which the evaluation value has a correlation with the document unit output data.

The information processing apparatus according to any one of appendixes 1 to 4, in which the processor is configured to: normalize each document data included in the document data group; and derive the evaluation value for each normalized document data.

The information processing apparatus according to appendix 1 or 2, in which the processor is configured to: extract a plurality of word data from each document data included in the document data group; derive the evaluation value in the machine learning model as a word unit evaluation value for each word data; and derive the evaluation value according to a statistical value of the word unit evaluation value of the word data included in the document data for each document data.

The information processing apparatus according to appendix 1 or 2, in which the document data having a greatest first evaluation value, which is derived for each document data, is used as first document data, and each of the plurality of document data other than the first document data included in the document data group is used as second document data, and the processor is configured to: use each combination data in which the first document data and the second document data are combined as input of the machine learning model to derive a second evaluation value from output data which is output for each combination data.

The information processing apparatus according to appendix 7, in which the processor is configured to: give a first display priority to the first document data; and give a second display priority, which is lower than the first display priority, to the second document data based on the second evaluation value.

The information processing apparatus according to appendix 1 or 2, in which the processor is configured to: extract a plurality of word data from each document data included in the document data group; derive the evaluation value in the machine learning model as a word unit evaluation value for each word data; derive a first statistical value of the word unit evaluation value of the word data included in the document data for each document data to give a first evaluation value to first evaluation value document data which is the document data having a greatest first statistical value; derive, for a plurality of combination data in which the first evaluation value document data, and each of the plurality of document data other than the first evaluation value document data included in the document data group are combined, a second statistical value of the word unit evaluation value of the word data included in the combination data for each combination data to give a second evaluation value, which is lower than the first evaluation value, to second evaluation value document data which is the document data having a greatest second statistical value; and set, in derivation of the second statistical value, the word unit evaluation value of the word data included in the first evaluation value document data among the word data included in the document data combined with the first evaluation value document data to be relatively lower than the word unit evaluation value of the word data which is not included in the first evaluation value document data.

The information processing apparatus according to any one of appendixes 1 to 9, in which the machine learning model is a model that is used to carry out a predetermined task and outputs a probability of a state according to the task as the output data.

The information processing apparatus according to any one of appendixes 1 to 10, in which the plurality of document data included in the document data group is document data representing a document related to a medical care of a specific patient, and the machine learning model is a model that predicts a state of the specific patient.

An information processing method executed by a processor of an information processing apparatus including at least one processor, the information processing method comprising: for a machine learning model that uses a document data group including a plurality of document data as input and outputs output data, deriving an evaluation value in the machine learning model for each document data included in the document data group.

An information processing program causing a processor of an information processing apparatus including at least one processor, to execute a process comprising: for a machine learning model that uses a document data group including a plurality of document data as input and outputs output data, deriving an evaluation value in the machine learning model for each document data included in the document data group.

A learning apparatus of a machine learning model that uses a plurality of document data as input and outputs output data, the learning apparatus comprising: at least one processor, in which the processor is configured to: use, for a plurality of document data for training, each document data for training as input of the machine learning model to acquire output data which is output for each document data for training; calculate, for a part of the document data for training from the plurality of document data for training, a loss function representing a degree of difference between correct answer data and the output data for each document data for training based on the output data obtained for each document data for training and the correct answer data; and update the machine learning model based on the loss function.

The learning apparatus according to appendix 14, in which the processor is configured to: extract the part of document data for training based on a degree of similarity between the output data and the correct answer data.

The learning apparatus according to appendix 14 or 15, in which the processor is configured to: calculate, also for another document data for training other than the part of document data for training, the loss function with a weight smaller than a weight of the part of document data for training for each data for training; and update the machine learning model based also on the loss function of the other document data for training.

The learning apparatus according to any one of appendixes 14 to 16, in which the processor is configured to: calculate, for the part of document data for training, the loss function by performing weighting based on the output data obtained for each document data for training and the correct answer data.

The learning apparatus according to appendix 17, in which the processor is configured to: set weighting to be larger as a degree of similarity between the output data and the correct answer data is higher.

The learning apparatus according to any one of appendixes 14 to 18, in which the processor is configured to: repeatedly update the machine learning model based on the loss function obtained from the part of document data for training; and change the number of the part of document data for training to be extracted, according to the number of updates of the machine learning model.

The learning apparatus according to any one of appendixes 14 to 19, in which each document data for training is given with a label representing a type of an associated prediction result of the machine learning model, and the processor is configured to: extract the document data for training for each type of the label.

An information processing apparatus comprising: at least one processor, in which the processor is configured to: for a machine learning model that uses a document data group including a plurality of document data as input and outputs output data, derive an evaluation value in the machine learning model for each document data included in the document data group, and the machine learning model is a machine learning model trained by a learning apparatus of the machine learning model that uses the document data group including the plurality of document data as input and outputs the output data, the learning apparatus including: at least one processor for training, in which the processor for training is configured to: use each document data for training included in a document data group for training as input of the machine learning model to acquire output data which is output for each document data for training; calculate, for a part of the document data for training from the document data group for training, a loss function representing a degree of difference between correct answer data and the output data for each document data for training based on the output data obtained for each document data for training and the correct answer data; and update the machine learning model based on the loss function.

An information processing system comprising: the information processing apparatus according to any one of appendixes 1 to 11; and the learning apparatus according to any one of appendixes 14 to 20.

A learning method comprising: via a processor, using, for a plurality of document data for training, each document data for training as input of a machine learning model to acquire output data which is output for each document data for training; calculating, for a part of the document data for training from the plurality of document data for training, a loss function representing a degree of difference between correct answer data and the output data for each document data for training based on the output data obtained for each document data for training and the correct answer data; and updating the machine learning model based on the loss function.

A learning program causing a processor to execute a process comprising: using, for a plurality of document data for training, each document data for training as input of a machine learning model to acquire output data which is output for each document data for training; calculating, for a part of the document data for training from the plurality of document data for training, a loss function representing a degree of difference between correct answer data and the output data for each document data for training based on the output data obtained for each document data for training and the correct answer data; and updating the machine learning model based on the loss function.