# From Texts to Classification Knowledge

Shusaku Tsumoto, Tomohiro Kimura, Shoji Hirano

► To cite this version:

Shusaku Tsumoto, Tomohiro Kimura, Shoji Hirano. From Texts to Classification Knowledge. 4th International Conference on Intelligence Science (ICIS), Feb 2021, Durgapur, India. pp.173-184, 10.1007/978-3-030-74826-5_15. hal-03741713

HAL Id: hal-03741713

https://inria.hal.science/hal-03741713v1

Submitted on 1 Aug 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# From Texts to Classification Knowledge *

Shusaku Tsumoto¹, Tomohiro Kimura², and Shoji Hirano¹

¹ Department of Medical Informatics, Faculty of Medicine, Shimane University
{tsumoto,hirano}@med.shimane-u.ac.jp

² Medical Services Division, Faculty of Medicine, Shimane University
t-kimura@med.shimane-u.ac.jp

**Abstract.** Hospital information system stores all clinical information, whose major part is electronic patient records written by doctors, nurses and other medical staff. Since records are described by medical experts, they are rich in knowledge about medical decision making. This paper proposes an approach to extract clinical knowledge from the texts of clinical records. The method consists of the following three steps. First, discharge summaries, which include all clinical processes during the hospitalization, are extracted from hospital information system. Second, morphological and correspondence analysis generates a term matrix from text data. Then, finally, machine learning methods are applied to a term matrix in order to acquire classification knowledge. We compared several machine learning methods by using discharge summaries stored in hospital information system. The experimental results show that random forest is the best classifier, compared with deep learning, SVM and decision tree. Furthermore, random forest gains more than 90% classification accuracy.

**Keywords:** Discharge Summary · Hospital Information System · Text Mining · Classification Learning · Rough Sets

## 1 Introduction

Computerization of patient records enables to store “big unstructured text data” in a Hospital Information System (HIS). For example, our system in Shimane university hospital, where about 1000 patients visit outpatient and about 600 patients stays in-patient, additionally stores about 200GB text data per year, including patient records, discharge summaries and reports of radiology and pathology. Application of text mining to these resources is very important to discover useful knowledge from text data, which can be viewed as a new type of support of clinical actions, researches and hospital management.

This paper is a first step to acquire clinical knowledge from texts: We focus on discharge summaries, which includes all clinical processes during the hospitalization, whose written style is formal, compared with regular records. Thus, conventional text mining approach can be used to extract enough keywords from each text. Then, we

* This research is supported by Grant-in-Aid for Scientific Research (B) 18H03289 from Japan Society for the Promotion of Science (JSPS).
propose a method for construction of classifiers of diseases codes from discharge summaries, which consists of the following five steps. First, discharge summaries are extracted from hospital information system. Second, morphological analysis is applied to a set of summaries and a term matrix is generated. Second, correspond analysis is applied to the term matrix with the class labels, and two dimensional coordinates are assigned to each keyword. By measuring the distance between categories and the assigned points, ranking of key words for each category will be generated. Then, keywords are selected as attributes according to the rank, and training example for classifiers will be generated. Finally learning methods are applied to the training examples. We conduct experimental validation with four methods, random forest, deep learning (multi-layer perceptron), SVM and decision tree induction method. The results show that random forest achieved the best performance and the second best was the deep learner with a small difference, but decision tree methods with many keywords performed only a little worse than neural network or deep learning methods. The paper is organized as follows. Section 2 explains our motivation. Section 3 gives a proposed mining process. Section 4 shows the experimental results. Section 5 discusses the results obtained. Finally, Section 6 concludes this paper.

## 2 Motivation

Knowledge acquisition from medical experts is a very classical problem, but still a bottleneck for developing medical expert systems [10, 12].

The reason why knowledge acquisition is difficult is that medical experts had difficulties in reformatting their knowledge into if-then rules. Thus, although deeper reasoning for experts emerged in the research of artificial intelligence in medicine in order to solve such bottle neck problems [1, 11], it is still a difficult problem.

However, medical staff write down the process of clinical decision making as texts: it suggests that such knowledge about decision making can be represented as language texts, but not as rule-based. Thus, if we can analyze texts written by medical staff, we will be able to get knowledge about medical decision making.

## 3 Methods

### 3.1 Discharge Summary

In this paper, we focus on discharge summaries. A discharge summary includes all clinical processes during the hospitalization, whose written style is formal, compared with regular records. Thus, conventional text mining approach can be used to extract enough keywords from each text. We apply ordinary text mining process as preprocessing.

### 3.2 Motivation for Feature Selection

It is well known that feature selection is important even for deep learners [8, 9]. Although deep learners gain good performance in image analysis, in other cases, differences between deep learners and other classification method are very small. It may be
due to that we have not yet suitable network structure and we may not use suitable features for classification. The empirical fact that deep learners are good at recognition of images suggests that some kind of topological relations should be explicitly embedded into training data. So, here we propose a new feature selection method based on correspondence analysis, which calculates mapping attributes to points of multi-dimensional coordinates. The method can embed the topological relations between keywords and concepts into the data.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Mining Process</figcaption></figure>

## 3.3 Mining Process

Fig. 1 shows the proposed total mining process pipeline, composed of the following five steps. The first three methods can be called text mining process and the next two methods can be called information granulation process, which determines the granularity of knowledge for classification. Please note that these five methods are automated: after the users impose some conditions for text retrieval, such as collection period, and select the classification model, the process will start and the classification model will be generated.

**Extraction of Discharge Summaries** First, discharge summaries are extracted from the hospital information system as texts.

**Morphological Analysis.** Next, morphological analysis (MeCab [2]), is applied, which outputs a term matrix, that is, a contingency table for keywords and concepts are generated.

**Correspondence Analysis.** Thirdly, correspondence analysis is applied to a term matrix. Although high dimensional coordinates can be selected, since a very large table is obtained, we focus on two dimensional analysis which can be easily used for visualization. To each key word and concept, two dimensional coordinate is assigned.

**Ranking.** The coordinates of a concept and a keyword will be used to calculate the euclidean distance between them. We use the distance values for ranking of keywords to each concept: the smaller the distance is, the higher its ranking is.
<table><caption>**Table 1.** DPC top twenty diseases (fiscal year: 2015)</caption><thead><tr><td>No DPC</td><td>Cases</td></tr></thead><tbody><tr><td>1 Cataract (lateral)</td><td>445</td></tr><tr><td>2 Cataract (bilateral)</td><td>152</td></tr><tr><td>3 Type II Diabetes Mellitus (except for keto-acidosis)</td><td>145</td></tr><tr><td>4 Lung Cancer (with surgical operation)</td><td>131</td></tr><tr><td>5 Uterus Cancer (without surgical operation)</td><td>121</td></tr><tr><td>6 Lung Cancer (without surgical operation, chemotherapy)</td><td></td></tr><tr><td>7 Uterus benign tumor</td><td>111</td></tr><tr><td>8 Lung Cancer (without surgical operation, with chemotherapy)</td><td>110</td></tr><tr><td>9 Shortage of Pregnancy</td><td>110</td></tr><tr><td>10 Injury of Elbow and Knee</td><td>99</td></tr><tr><td>11 Autoimmune Disease</td><td>96</td></tr><tr><td>12 Non-Hodgkin Disease</td><td>94</td></tr><tr><td>13 Pneumonia</td><td>86</td></tr><tr><td>14 Lung Tumor (without surgical operation nor chemotherapy)</td><td>85</td></tr><tr><td>15 Chronic Nephritis</td><td>83</td></tr><tr><td>16 Liver Cancer</td><td>82</td></tr><tr><td>17 Gallbladder Stone</td><td>82</td></tr><tr><td>18 Cerebral Infarction</td><td>80</td></tr><tr><td>19 Retinal Detachment</td><td>75</td></tr><tr><td>20 Fetal Abnormalities</td><td>75</td></tr></tbody></table>

**Keyword Selection.** We assume the number of keywords selected before the analysis, say 100. Usually, since the lower rank keywords give information about specific cases, the number of selected keywords will determine the granularity of induced knowledge. Then, keywords whose ranking are up to 100 are selected keywords for classification. Some keywords may overlap, so such overlapped keywords will be deleted. Then, training examples with a classification label and the value of selected keywords (binary attributes) are constructed.

**Classification.** Finally, classification learning methods will be applied. In this paper, we selected random forest [6], deep learning (multi-layer perceptron) (darch), Support Vector Machine (SVM) [4], Backpropagation Neural Network (BNN) [16] and decision tree (rpart) [13] for comparison.
# 4 Experimental Evaluation

We selected top 20 frequent DPC codes in fiscal year of 2015 extracted discharge summaries from HIS in Shimane University Hospital. Table 1 shows their statistics and includes the averaged number of characters used in summaries.

Except for extraction from data from HIS, all the processes are implemented on R 3.5.0.

## 4.1 Mining Process

**Correspondence Analysis.** For morphological analysis, RMeCab [2] is used and the bag of keywords was generated. From the bag of keywords, a contingency table for these summaries are obtained. Then, correspondence analysis which is implemented in MASS package on R3.5.0 was applied to the table and two dimensional coordinate was assigned to each keyword and each class.<fnref n="3" /> Fig 2 shows the two-dimensional plot of correspondence analysis.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Correspondence Analysis</figcaption></figure>

**Ranking.** Next, the distances between the coordinate of a keyword and that of a class is calculated, and the ranking of keywords for each class was obtained. By using the ranking, a given number of keywords were selected to generate a classification table.

**Keyword Selection.** In the case of 250 keywords, 250 keywords were selected for each DPC code and 5000 keywords in total. But, since the overlapped words should

<footnote n="3">³ The method can also generate $p(p \ge 3)$-dimensional coordinates. However, higher dimensional coordinates did not give better performance than the experiments below.</footnote>
<table><caption>**Table 2.** Number of Selected Keywords and Actually Used Keywords</caption><thead><tr><th rowspan="2">#Keyword</th><th colspan="2">Selected Keywords</th></tr><tr><th>DPC Top 10</th><th>DPC Top 20</th></tr></thead><tbody><tr><td>1</td><td>10</td><td>19</td></tr><tr><td>2</td><td>19</td><td>37</td></tr><tr><td>3</td><td>27</td><td>54</td></tr><tr><td>4</td><td>36</td><td>71</td></tr><tr><td>5</td><td>44</td><td>88</td></tr><tr><td>10</td><td>88</td><td>167</td></tr><tr><td>20</td><td>115</td><td>309</td></tr><tr><td>30</td><td>247</td><td>449</td></tr><tr><td>40</td><td>334</td><td>597</td></tr><tr><td>50</td><td>406</td><td>718</td></tr><tr><td>100</td><td>724</td><td>1125</td></tr><tr><td>150</td><td>1000</td><td>1472</td></tr><tr><td>200</td><td>1192</td><td>1782</td></tr><tr><td>250</td><td>1382</td><td>1932</td></tr><tr><td>300</td><td>1547</td><td>2031</td></tr><tr><td>350</td><td>1676</td><td>2113</td></tr><tr><td>400</td><td>1797</td><td>2192</td></tr><tr><td>450</td><td>1929</td><td>2273</td></tr><tr><td>500</td><td>2028</td><td>2364</td></tr><tr><td>750</td><td>2304</td><td>2808</td></tr><tr><td>1000</td><td>2545</td><td>3000</td></tr><tr><td>ALL</td><td>13944</td><td>20417</td></tr></tbody></table>

be removed, so only 1932 keywords were used for classification. Thus, some important keywords may be deleted due to the overlap if such words are frequently used at least in two diseases.

**Classification.** Finally, decision tree (package: rpart [13]), random forest (package: randomForest [6], SVM (kernlab [4]), BNN/package: nnet [16]) and Deep Learner (multi-layer perceptron) (darch<fnref n="4" />) were applied to the generated table. For parameters of Darch, the number of intermediate neurons are 10, (10,5), (40,10) and (100,10), whose epoch was 100. For all other packages, the default settings of parameters were used.

**Evaluation Process.** Evaluation process was based on repeated 2-fold cross validation [5]<fnref n="5" />.

First, a given dataset was randomly split into training examples and test samples half in half. Then, training examples was used for construction of a classifier, and the derived

<footnote n="4">⁴ Darch was removed from R package. Please check the github: https://github.com/maddin79/darch</footnote>

<footnote n="5">⁵ The reason why 2-fold is selected is that the estimator of 2-fold cross-validation will give the lowest estimate of parameters, such as accuracy and the estimation of bias will be minimized.</footnote>
classifier was evaluated by using remaining test samples. The above procedures were repeated for 100 times in this experiment, and the averaged accuracy was calculated.

The number of keywords varied from 1 to 1000, selected according to the rank given by correspondence analysis. For analysis, two units of HP Proliant ML110 Gen9 (Xeon E5-2640 v3.2 2.6GHz 8Core, 64GBDRAM) was used.

<table><caption>**Table 3.** Experimental results (averaged accuracies)</caption><thead><tr><th>#keywords</th><th>Darch<br>one layer<br>(20)</th><th>Darch<br>two layers<br>(40,20)</th><th>Darch<br>two layers<br>(80,20)</th><th>SVM</th><th>Rpart</th><th>Random<br>Forest</th><th>BNN</th></tr></thead><tbody><tr><td>1</td><td><strong>0.247</strong></td><td>0.236</td><td>0.237</td><td>0.233</td><td>0.202</td><td>0.239</td><td>0.264</td></tr><tr><td>2</td><td><strong>0.442</strong></td><td>0.407</td><td>0.43</td><td>0.24</td><td>0.218</td><td>0.288</td><td>0.429</td></tr><tr><td>3</td><td>0.569</td><td>0.581</td><td><strong>0.584</strong></td><td>0.324</td><td>0.145</td><td>0.295</td><td>0.541</td></tr><tr><td>4</td><td>0.632</td><td>0.628</td><td>0.633</td><td>0.424</td><td>0.254</td><td><strong>0.676</strong></td><td>0.582</td></tr><tr><td>5</td><td>0.662</td><td>0.655</td><td>0.657</td><td>0.295</td><td>0.315</td><td><strong>0.714</strong></td><td>0.597</td></tr><tr><td>10</td><td>0.716</td><td>0.704</td><td>0.71</td><td>0.323</td><td>0.315</td><td><strong>0.767</strong></td><td>0.633</td></tr><tr><td>20</td><td>0.786</td><td>0.772</td><td>0.778</td><td>0.664</td><td>0.598</td><td><strong>0.826</strong></td><td>0.698</td></tr><tr><td>30</td><td>0.804</td><td>0.792</td><td>0.796</td><td>0.694</td><td>0.652</td><td><strong>0.841</strong></td><td>0.718</td></tr><tr><td>40</td><td>0.821</td><td>0.809</td><td>0.814</td><td>0.739</td><td>0.656</td><td><strong>0.855</strong></td><td>0.74</td></tr><tr><td>50</td><td>0.823</td><td>0.813</td><td>0.818</td><td>0.742</td><td>0.673</td><td><strong>0.855</strong></td><td>0.748</td></tr><tr><td>100</td><td>0.849</td><td>0.841</td><td>0.851</td><td>0.749</td><td>0.577</td><td><strong>0.875</strong></td><td>0.785</td></tr><tr><td>150</td><td>0.864</td><td>0.857</td><td>0.867</td><td>0.778</td><td>0.747</td><td><strong>0.896</strong></td><td>0.806</td></tr><tr><td>200</td><td>0.865</td><td>0.855</td><td>0.864</td><td>0.784</td><td>0.741</td><td><strong>0.907</strong></td><td>0.805</td></tr><tr><td>250</td><td>0.868</td><td>0.862</td><td>0.867</td><td>0.783</td><td>0.744</td><td><strong>0.906</strong></td><td>0.807</td></tr><tr><td>300</td><td>0.82</td><td>0.814</td><td>0.821</td><td>0.77</td><td>0.768</td><td><strong>0.907</strong></td><td>0.798</td></tr><tr><td>350</td><td>0.826</td><td>0.815</td><td>0.824</td><td>0.767</td><td>0.761</td><td><strong>0.907</strong></td><td>0.799</td></tr><tr><td>400</td><td>0.825</td><td>0.818</td><td>0.826</td><td>0.771</td><td>0.764</td><td><strong>0.908</strong></td><td>0.808</td></tr><tr><td>450</td><td>0.825</td><td>0.819</td><td>0.83</td><td>0.77</td><td>0.767</td><td><strong>0.908</strong></td><td>0.802</td></tr><tr><td>500</td><td>0.832</td><td>0.821</td><td>0.831</td><td>0.77</td><td>0.768</td><td><strong>0.908</strong></td><td>0.804</td></tr><tr><td>750</td><td>0.836</td><td>0.831</td><td>0.841</td><td>0.757</td><td>0.782</td><td><strong>0.907</strong></td><td>0.81</td></tr><tr><td>1000</td><td>0.846</td><td>0.836</td><td>0.845</td><td>0.753</td><td>0.79</td><td><strong>0.909</strong></td><td>0.82</td></tr></tbody></table>

All results are obtained by repeated two-fold cross validation (100 repetitions).

Layer(s) denote the number of intermediate layers.

and (a,b) shows the numbers of neurons for intermediate layers.

## 4.2 Classification Results

Table 3 shows the evaluation results of the top 20 diseases. For four or fewer keywords, all the classifiers showed an accuracy of about 70%. At five or more keywords, however, SVM showed a decrease in accuracy, whereas the other methods showed monotonic increases in accuracy, with the latter plateauing at 200 keywords. The Random Forest method performed better than the other classifiers, followed by Darch deep learning. If more than 250 keywords were selected, the performance of Darch decreased,
whereas the performances of random forest and decision trees increased monotonically. Although BNN showed poorer accuracy than Darch (default setting) with 5 to 100 selected keywords, the accuracy of BNN approached that of Darch classifiers with a larger number of keywords. Interestingly, the accuracy of the decision tree method increased monotonically, becoming maximal when all the keywords were used for analysis.

# 5 Discussion

## 5.1 Misclassified Cases

Fig 3 and 4 show confusion matrices of random forest and darch (multi-layer percep-
tron), where DPC codes are set in order which means that similar diseases are assigned
to similar codes.<fnref n="6" /> Yellowed region indicates misclassified cases. It can be observed that
whereas errors in random forest are located near the diagonal, errors in darch are more
scattered. This suggests that random forest almost correctly classifies a case if similar
dpc codes are grouped into one generalized class, while Darch has unexpected errors.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Confusion Matrix of Random Forest</figcaption></figure>

<footnote n="6">⁶ DPC codes are three-level hierarchical system and each DPC code is defined as a tree. The first-level denotes the type of a disease, the second-level gives the primary selected therapy and the third-level shows the additional therapy. Thus, in the tables, characteristics of codes are used to represent similarities.</footnote>
<figure><img src="image_5.png" /><figcaption>Fig. 4. Confusion Matrix of Deep Learner</figcaption></figure>

## 5.2 Execution Time

Tab. 4 shows an empirical comparison of repeated two-fold cross validations (100 trials). The times need for Random Forest and SVM were 183 and 156 minutes for 250 keywords, whereas Darch (20) required 672 minutes. For 1000 keywords, the times needed for Random Forest, SVM and Darch (20) were 261, 288, and 1101 minutes, respectively. The times required by random forest and BNN methods were close to those of Deep Learners. In the case of Darch, the number of intermediate layers resulted in greater computation times, although the growth rate was smaller than that of BNN. Although the performance of BNN and Deep Learners were high, the problem is that they need more time for computation.

## 5.3 Next Step

Experimental Validation shows that discharge summaries, as natural language texts, include enough information for classification of disease codes. And it also shows that 200 keywords for each disease are enough for classification. Thus, the next step is to apply automated knowledge acquisition process [14, 15] based on rough sets to the term matrix generated before application of learning methods
<table><caption>**Table 4.** Times required for construction of classifications for the top 20 diseases</caption><thead><tr><th rowspan="2">#keyword</th><th>Darch</th><th>Darch</th><th>Darch</th><th>SVM</th><th>Rpart</th><th>Random Forest</th><th>BNN</th></tr><tr><th>one layer</th><th>two layers</th><th>two layers</th><th></th><th></th><th></th><th></th></tr><tr><th></th><th>(20)</th><th>(40,20)</th><th>(80,20)</th><th></th><th></th><th></th><th></th></tr></thead><tbody><tr><td>1</td><td>172</td><td>226</td><td>230</td><td>8</td><td>0</td><td>3</td><td>3</td></tr><tr><td>2</td><td>175</td><td>231</td><td>247</td><td>10</td><td>0</td><td>6</td><td>4</td></tr><tr><td>3</td><td>177</td><td>239</td><td>257</td><td>12</td><td>1</td><td>9</td><td>4</td></tr><tr><td>4</td><td>182</td><td>245</td><td>276</td><td>12</td><td>0</td><td>10</td><td>5</td></tr><tr><td>5</td><td>186</td><td>254</td><td>288</td><td>14</td><td>1</td><td>11</td><td>6</td></tr><tr><td>10</td><td>201</td><td>277</td><td>368</td><td>21</td><td>1</td><td>20</td><td>11</td></tr><tr><td>20</td><td>231</td><td>362</td><td>532</td><td>31</td><td>2</td><td>35</td><td>42</td></tr><tr><td>30</td><td>269</td><td>435</td><td>729</td><td>40</td><td>3</td><td>48</td><td>95</td></tr><tr><td>40</td><td>302</td><td>516</td><td>891</td><td>48</td><td>4</td><td>61</td><td>171</td></tr><tr><td>50</td><td>331</td><td>575</td><td>1063</td><td>57</td><td>4</td><td>76</td><td>239</td></tr><tr><td>100</td><td>453</td><td>752</td><td>1574</td><td>90</td><td>7</td><td>115</td><td>577</td></tr><tr><td>150</td><td>540</td><td>922</td><td>1948</td><td>114</td><td>9</td><td>152</td><td>943</td></tr><tr><td>200</td><td>635</td><td>1099</td><td>2431</td><td>142</td><td>12</td><td>172</td><td>1429</td></tr><tr><td>250</td><td>672</td><td>1281</td><td>2902</td><td>156</td><td>13</td><td>183</td><td>1679</td></tr><tr><td>300</td><td>751</td><td>1263</td><td>2701</td><td>164</td><td>13</td><td>198</td><td>1850</td></tr><tr><td>350</td><td>789</td><td>1507</td><td>3085</td><td>172</td><td>14</td><td>204</td><td>2010</td></tr><tr><td>400</td><td>817</td><td>1500</td><td>3114</td><td>179</td><td>15</td><td>202</td><td>2136</td></tr><tr><td>450</td><td>833</td><td>1550</td><td>3311</td><td>194</td><td>15</td><td>222</td><td>2296</td></tr><tr><td>500</td><td>883</td><td>1650</td><td>3381</td><td>201</td><td>17</td><td>223</td><td>2509</td></tr><tr><td>750</td><td>1027</td><td>2177</td><td>4802</td><td>244</td><td>19</td><td>271</td><td>3542</td></tr><tr><td>1000</td><td>1110</td><td>2504</td><td>5030</td><td>261</td><td>22</td><td>288</td><td>4062</td></tr></tbody></table>

All results are obtained by repeated two-fold cross validation (100 repetitions).

Layer(s) denote the number of intermediate layers

and (a,b) shows the numbers of neurons for intermediate layers.
<figure><img src="image_6.png" /><figcaption>Fig. 5. Mining Process</figcaption></figure>

# 6 Conclusion

Knowledge acquisition from medical experts is a very classical problem, but still a bottleneck for developing medical expert systems.

The reason why knowledge acquisition is difficult is that medical experts had difficulties in reformatting their knowledge into if-then rules. Thus, although deeper reasoning for experts emerged in the research of artificial intelligence in medicine in order to solve such bottle neck problems: it is still a difficult problem.

However, medical staff write down the process of clinical decision making as texts: it suggests that such knowledge about decision making can be represented as language texts, but not as rule-based. Thus, if we can analyze texts written by medical staff, we will be able to get knowledge about medical decision making.

We propose a method for construction of classifiers of disease codes from discharge summaries as follows. First, discharge summaries are extracted from hospital information system. Then, second, morphological analysis is applied to a set of summaries and a term matrix is generated. Third, correspond analysis is applied to the classification labels and the term matrix and generates two dimensional coordinates. By measuring the distances between categories and the assigned points, ranking of key words will be generated. Then, keywords are selected as attributes according to the rank, and training example for classifiers will be generated. Finally learning methods are applied to the training examples. Experimental validation was conducted by discharge summaries stored in Shimane University Hospital in fiscal year of 2015. The results shows that random forest achieved the best performance around 93% classification accuracy. The second best was deep learners with a small difference, around 91%. Thus, they suggest that discharge summaries includes enough information of clinical processes for each disease code. It will be our future work to acquire more structured knowledge from texts, such as diagnostic rules, rules for therapies, rules for prediction of outcome and so on.

# References

1. Amisha, Malik, P., Pathania, M., Rathaur, V.: Overview of artificial intelligence in medicine. Journal of family medicine and primary care 8(7), 2328-2331 (2019)

2. Ishida, M.: Rmecab. http://rmecab.jp/wiki/index.php?RMeCabFunctions (2016)

3. JONES, K.S.: A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation 28(1), 11-21 (1972). https://doi.org/10.1108/eb026526
4. Karatzoglou, A., Smola, A., Hornik, K., Zeileis, A.: kernlab – an S4 package for kernel methods in R. Journal of Statistical Software **11**(9), 1–20 (2004), http://www.jstatsoft.org/v11/i09/

5. Kim, J.H.: Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap. Comput. Stat. Data Anal. **53**(11), 3735–3745 (Sep 2009). https://doi.org/10.1016/j.csda.2009.04.009, http://dx.doi.org/10.1016/j.csda.2009.04.009

6. Liaw, A., Wiener, M.: Classification and regression by randomforest. R News **2**(3), 18–22 (2002), http://CRAN.R-project.org/doc/Rnews/

7. Luhn, H.P.: A statistical approach to mechanized encoding and searching of literary information. IBM Journal of Research and Development **1**(4), 309–317 (Oct 1957). https://doi.org/10.1147/rd.14.0309

8. Mares, M.A., Wang, S., Guo, Y.: Combining multiple feature selection methods and deep learning for high-dimensional data. Transactions on Machine Learning and Data Mining **9**, 27–45 (2016)

9. Nezhad, M.Z., Zhu, D., Li, X., Yang, K., Levy, P.: SAFS: A deep feature selection approach for precision medicine. CoRR abs/1704.05960 (2017), http://arxiv.org/abs/1704.05960

10. Persidis, A., Persidis, A.: Medical expert systems: An overview. Journal of Management in Medicine **5**(3), 27–34 (1991). https://doi.org/10.1108/EUM0000000001316

11. Riaño, D., Wilk, S., ten Teije, A. (eds.): Artificial Intelligence in Medicine - 17th Con-ference on Artificial Intelligence in Medicine, AIME 2019, Poznan, Poland, June 26-29, 2019, Proceedings, Lecture Notes in Computer Science, vol. 11526. Springer (2019). https://doi.org/10.1007/978-3-030-21642-9

12. Shortliffe, E.: Medical expert systems–knowledge tools for physicians. The Western journal of medicine **145**(6), 830–839 (1986)

13. Therneau, T.M., Atkinson, E.J.: An Introduction to Recursive Partitioning Using the RPART Routines (2015), https://cran.r-project.org/web/packages/rpart/vignettes/longintro.pdf

14. Tsumoto, S.: Automated induction of medical expert system rules from clinical databases based on rough set theory. Information Sciences **112**, 67–84 (1998)

15. Tsumoto, S., Hirano, S.: Incremental induction of medical diagnostic rules based on incremental sampling scheme and subrule layers. Fundam. Informaticae **127**(1-4), 209–223 (2013). https://doi.org/10.3233/FI-2013-905, https://doi.org/10.3233/FI-2013-905

16. Venables, W.N., Ripley, B.D.: Modern Applied Statistics with S. Springer, New York, fourth edn. (2002), http://www.stats.ox.ac.uk/pub/MASS4, iSBN 0-387-95457-0