Knowledge Injection for Disease Names in Logical Inference between Japanese Clinical Texts
Year: 2023

Abstract:
In the medical field, there are many clinical texts such as electronic medical records, and research on Japanese natural language processing using these texts has been conducted. One such research involves Recognizing Textual Entailment (RTE) in clinical texts using a semantic analysis and logical inference system, ccg2lambda. However, it is difficult for existing inference systems to correctly determine the entailment relations, if the input sentence contains medical domain specific paraphrases such as disease names. In this study, we propose a method to supplement the equivalence relations of disease names as axioms by identifying candidates for paraphrases that lack in theorem proving. Candidates of paraphrases are identified by using a model for the NER task for disease names and a disease name dictionary. We also construct an inference test set that requires knowledge injection of disease names and evaluate our inference system. Experiments showed that our inference system was able to correctly infer for 106 out of 149 inference test sets.

heading: 1 Introduction
text: In the medical field, there are many electronic texts, such as image detections and electronic medical records, and using such texts becomes more active in research on natural language processing (NLP) in Japanese (Aramaki et al., 2018; Doi et al., 2011). However, many of these studies utilize machine learning approaches, and it is argued that the machine learning approaches have problems in dealing with challenging linguistic phenomena such as negation and quantification. Logic-based approaches have been proposed to perform systematic inferences involving these challenging linguistic phenomena, and one task of these inferences can be referred to as recognizing textual entailment (RTE). RTE is the task of determining whether a hypothesis sentence H can be inferred from a premise
sentence T. For example, the following example illustrates a case where T entails H.
T : Some patients are given Loxoprofen.
H : There are patients who are given headache medicines.
One such effort at RTE between clinical texts is the logical inference system in the medical domain proposed by Ishida et al. (2022). Compounds words are often appeared in Japanese clinical texts, and methods to analyze linguistic phenomena in compound words were desirable. The system is an inference system based on ccg2lambda (MartínezGómez et al., 2016), a semantic analysis and logical inference system: the system extends ccg2lambda to enable an analysis of compound words that are frequently appeared in clinical texts. However, this system fails to perform inference when there are differences in the notation of disease names in medical texts. For example, the disease name “Deep vein thrombosis” has multiple paraphrases, such as “DVT” and “Homann’s sign”, and different clinical texts use different phrases that refer to the same disease name. The premise sentence is “The patient developed Homann’s sign.” and the hypothesis sentence is “The patient developed deep vein thrombosis.”, then empirically the premise sentence implies the hypothesis sentence. To show this entailment relation, the knowledge that “Homann’s sign means deep vein thrombosis” must be supplemented in the theorem prover.
In this study, we propose our logical inference system with knowledge injection in the medical field. We identify candidates of paraphrase of disease names that are necessary for theorem proving by a named entity recognition (NER) model for disease names and the Japanese disease name dictionary called J-MeDic (Ito et al., 2018). By generating axioms according to the combination of compound word semantic tags assigned to the identified
1 108
disease names, we inject paraphrase knowledge of disease names as axioms. We also evaluate the effectiveness of our inference system by constructing an inference test set that requires knowledge injection of disease names.

heading: 2 Background

heading: 2.1 Inference systems for clinical texts
text: There has been growing progress in research on neural network models for RTE with large-scale datasets using crowdsourcing such as SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018). However, neural network models for RTE generally require a large amount of training data, and inference is black-box, which makes it difficult to correct errors. Therefore, there is a need to develop RTE systems that are applicable to fields such as medical field, where it is difficult to prepare a large amount of training data and transparency is necessary.
The logical inference system ccg2lambda has the advantage of being able to judge entailment relations between sentences without using a large amount of training data and being easy to personalize and modify processing. However, the original ccg2lambda fails to analyze the semantic relations within compound words because they are treated as one word.
Ishida et al. (2022) addressed this problem by adding a compound word analysis module to ccg2lambda. This module extracts compound words from the Combinatory Categorial Grammar (CCG) (Steedman, 2000; Bekki, 2010) syn-
tactic structures obtained by the CCG parser of ccg2lambda and assigns compound word semantic tags that represent semantic relations within compound words, using a compound word semantic tagger. Based on syntactic structures, semantic tags, and lambda calculus, the semantic representation was derived by taking into account the semantic relations within the compound words, and inference between clinical texts containing compound words was realized.

heading: 2.2 Related studies on axiom injection
text: As for related studies on axiom injection of logical inference systems, including ccg2lambda, Martínez-Gómez et al. (2017) proposed word axiom injection by using lexical knowledge. Hokazono et al. (2018) used this word abduction mechanism to inject word knowledge specific in the financial texts as the lexical knowledge. However, these previous studies were limited to handle word-toword relations in natural deduction proofs. Yanaka et al. (2018) proposed a method for paraphrase detection by natural deduction proofs of semantic relations between sentence pairs to complement phrasal knowledge. In this study, we propose how to detect phrasal knowledge of disease names necessary for proving entailment relations between clinical texts and inject the knowledge into logical inference.

heading: 2.3 J-MeDic
text: In this study, we use J-MeDic to inject disease name knowledge into logical inference. J-MeDic is a Japanese dataset that extensively extracts
2
words related to symptoms and disease names from progress records and discharge summaries recorded in electronic medical records by medical professionals. The dataset contains not only the formal names of diseases but also abbreviations and English names. The dataset covers 362,866 disease names in total. Table 1 provides an example where the standard disease name column in J-MeDic is “Deep vein thrombosis.” J-MeDic records include the surface forms of the disease name, their pronunciation, ICD-10 codes, standard disease names, levels of reliability, and levels of frequency.

heading: 2.4 Related studies on NER in the medical domain
text: There are related studies on NER for disease names, such as the work by Goino and Hamagami (2021) and MedNER (Nishiyama et al., 2022). In Goino-Hamagami’s work, NER and modality estimation of medical conditions such as disease names and symptoms along with their modalities (five types) were performed using BERT and CRF. Experiments were conducted using three BERT models released by Tohoku University (tohokucharBERT, tohoku-BERT, tohoku-wwm-BERT) and UTH-BERT1, a Japanese BERT model pretrained on medical documents by the University of Tokyo. MedNER2 is a tool for the NER for disease names using word embeddings of BERT. MedNER follows J-MeDic and also performs modality estimation.

heading: 3 System Overview
text: The overview of our proposed system is shown in Figure 1. We provide our inference system by extending Ishida et al. (2022)’s inference system. The
1https://ai-health.m.u-tokyo.ac.jp/home/research/uth-bert 2https://github.com/sociocom/MedNER-J
system consists of syntactic analysis, compound word analysis, semantic analysis, and theorem proving. In this study, we add a knowledge injection module in the medical domain to the previous system.
In the knowledge injection module, we first apply an NER model based on a pretrained language model to extract disease names from the input sentence. For the extracted disease names, we perform an exact match search for the surface form column of disease names in J-MeDic. If the disease name and the surface form match, we inject the knowledge about the disease name in J-MeDic as an axiom to the automated theorem prover Coq (Bertot and Castéran, 2013). Since the additional axioms vary depending on the semantic relations between morphemes, we check the semantic tags of compound words assigned by the compound word semantic tagger, derive the axiom using the semantic tag and the knowledge of J-MeDic, and then inject the knowledge. We describe the details to provide an NER model in Section 4 and Subsection 6.1 and the details of axiom injection in Section 5.

heading: 4 Building an NER dataset for disease names
text: To train the model for the NER task for disease names in our inference system, we constructed a dataset newly for NER using clinical texts. We use a corpus of case reports, J-MedStd-CR3, which was extracted through OCR from case report papers in PDF format that are openly accessible on J-Stage. In this study, we manually annotated the appearances of disease names from J-MeDic in the 2,626 sentences of the J-MedStd-CR corpus.
3https://sociocom.naist.jp/medtxt/cr/
3

heading: 5 Axiom Injection
text: In the proposed method, after identifying candidate paraphrases of the disease names, the knowledge of paraphrases is injected as axioms. In axiom injection, it is necessary to generate the knowledge about paraphrases of disease names as axioms according to the semantic representations in ccg2lambda. For example, when the premise sentence is (1) and the hypothesis sentence is (2), the underlined disease name’s semantic tag in (1) is “EN” because “PE” is one word and entity. The semantic representation in this case is (1a). Semantic tags for the disease name underlined in (2) are “PA EN” because “pulmonary” in “ pulmenary embolism” is a body part and “embolism” is an entity. The semantic representation is (2a). PartOf(e1, e2) indicates hai(e2) is a body part for sokusensyo(e1). As in (1a) and (2a), semantic representations differ depending on the semantic relations between morphemes within the compound words. In this study, we realize axiom injection by defining axioms generated through the combination of semantic tag assigned to both the surface form and standard disease names.
(1) PE ga zouaku-si-teita PE NOM worsen-EUPH-PST
PEが増悪していた。 PE worsened.
a. ∃e1(PE(e1)) (2) hai-sokusensyo ga zouaku-si-teita
pulmonary-embolism NOM worsen-EUPH-PST
肺塞栓症が増悪していた。 Pulmonary embolism worsend.
a. ∃e2(hai(e2) ∧ ∃e1(sokusensyo(e1) ∧ PartOf(e1, e2)))

heading: 5.1 Trends in combinations of semantic tags in disease names
text: Table 2 shows some of the semantic tag assigned by Ishida et al. (2022)’s compound word semantic tagger. We investigated the composition of semantic tag to generate axioms based on the combination of semantic tag assigned to disease names. We applied the compound word semantic tagger to assign semantic tag to 651 disease names and their standard disease names extracted from randomly selected sentences containing disease names in clinical texts of the J-MedStd-CR corpus. The compound word semantic tagger is based on BiLSTM and BERT models. BERT model is the model released by
4
Tohoku University4 that was trained on Japanese Wikipedia data, and the tokenizers are MeCab and WordPiece. The top 5 combinations of semantic tags assigned to disease names are shown in Table 3. We perform axiom injection according to these combinations.

heading: 5.2 Axiom injection based on semantic tags
text: Table 4 shows the combination examples of semantic tag for the disease names in axiom injection. The asterisk ∗ indicates that the combinations of semantic tags are the same for both the surface form and the standard disease name. For example, the first row “∗ ⇒ ∗” indicates that the same semantic tags are assigned, such as “EV EN ⇒ EV EN” for “walking-impossibility ⇒ walking-difficulty”. Similarly, the second row “M_EN ∗ ⇒ ∗” indicates that the semantic tags “M_EN EN” are assigned to the phrase “pseudointestinal obstruction” included in “chronic-pseudointestinal obstruction”, and they match the tags assigned to “pseudo-ileus”. “M_EN+” indicates that there are one or more “M_EN” tag present. As in the example where the semantic tags “EN
4https://huggingface.co/cl-tohoku/bert-base-japanese
⇒ M_EN M_EN EN” are assigned to “PBC ⇒ primary-biliary-liver cirrhosis”, even when there are multiple repetitions of the M_EN tag in the semantic tags assigned to the standard disease name, it is possible to generate an axiom. As an example of the generated axioms, we show the axiom to be injected when the premise sentence is (1) and the hypothesis sentence is (2) in Figure 2. The generated axioms are injected to the theorem prover as callable axioms during automated theorem proving and used for inference.
5

heading: 6 Experiments

heading: 6.1 Experiments on the NER task of disease names

heading: 6.1.1 Experimental settings
text: To select a best performance model for the NER task for disease names to be combined with our inference system, we evaluate three RoBERTa models shown in Table 5. The Japanese RoBERTa base5 (hereafter referred to as the RIKEN model), which is publicly available from RIKEN, was pretrained only on Japanese Wikipedia. The tokenizer uses MeCab (Kudo et al., 2004) for word segmentation and BPE (Sennrich et al., 2016) for subword segmentation. The Japanese RoBERTa base model publicly available from RINNAI (Zhao and Sawada, 2021) (hereafter referred to as the RINNA model), and the roberta-large-japanese model released by Waseda University6 (hereafter referred to as Waseda model), were pretrained on both Japanese Wikipedia and the Japanese portion of the CC-100 dataset. Both models use Juman++ (Tolmachev et al., 2018) as a tokenizer for word segmentation and SentencePiece (Kudo and Richardson, 2018) for a subword segmentation.
For the NER task, we used 2,303 sentences from J-MedStd clinical texts as training data. We used 85% of the data as development data and 15% as validation data to finetune the pretrained language
5https://huggingface.co/liat-nakayama/japanese-robertabase-20220905
6https://huggingface.co/nlp-waseda/roberta-largejapanese
models. The training data contains 2,551 appearances of disease names. We randomly selected 326 sentences from J-MedStd clinical texts for evaluation. The 326 sentences that are used for evaluation do not overlap with the training data.

heading: 6.1.2 Evaluation
text: We trained and evaluated three BERT models, shown in Table 5, using our NER dataset in the experiment. We also performed a comparison with MedNER using our NER dataset and the experimental results are shown in Table 6. The RIKEN model had the highest score in terms of f1-score for predicting disease names. In the RINNA model, even unrelated text around disease names were extracted. The Waseda model had some disease names that were split in the middle of the name. MedNER tended to extract disease names that are closely related, such as “necrotic” and “granuloma” for “granuloma with necrosis”, and “pain” and “pruritus” for “pain and pruritus”. F1-score of MedNER decreased because the evaluation dataset created in this study was annotated with one disease name per annotation.
Based on the results of this experiment, we adopted the RIKEN model, which showed the highest performance, as the model for the NER task to be combined with our inference system.

heading: 6.2 Experiments on the RTE task
text: To evaluate the effectiveness of our inference system, we performed a comparison between our inference system and the previous inference system (Ishida et al., 2022).
6

heading: 6.2.1 Inference test set involving disease names
text: We constructed an inference test set that requires disease name knowledge injection and evaluated our proposed system. The inference test set is constructed to consist of sentence pairs whose relation is entailment and the experiment is conducted to test whether the system can correctly predict entailment relations. Table 7 shows examples of the inference tests set. For constructing the test set, we used sentences from J-MedStd-CR, a corpus of clinical case reports where the disease names mentioned in the sentences are different from the standard disease names in J-MeDic. We manually constructed a set of 149 pairs of simplified hypothesis sentences and corresponding premise sentences, where the hypothesis sentences were simplified versions of sentences containing disease names in the J-MedStd-CR corpus, and the disease names in the hypothesis sentences were replaced with their corresponding standard names.

heading: 6.2.2 Evaluation
text: We compared the accuracy between our inference system with the knowledge injection module and the previous system by Ishida et al. (2022) on our inference test set. Table 8 shows the results of the evaluation of inference. While the previous system failed to predict entailment relations for all examples, our system was able to make correct predictions for 106 out of 149 test sets.

heading: 6.2.3 Error analysis
text: We performed an error analysis on the cases where our inference system made incorrect predictions. Table 9 shows examples of error types and sentence pairs for the analysis of errors. There were many cases where the disease names written in English were not correctly extracted due to errors in NER. For errors related to syntactic analysis, the morphemes “itching” in (3) and “necrosis” in (4) were misclassified as verbs by the morphological analyzer, janome7, when they should have been treated as nouns. These morphemes were treated as verbs such as “itch” and “become necrosis”, and the wrong axioms were provided, which resulted in the failure of inference.
(3) sou-you scraching-itching そう_痒 (pruritus)
(4) kan-saibou-esi liver-cell-necrosis 肝_細胞_壊死 (hepatic necrosis)
For the error caused by compound word analysis, the compound word semantic tagger by Ishida et al. (2022) classified the “cancer” in (5) as “PA” instead of being tagged with “EN”, which resulted in a failure of inference. As a result, axiom injection could not be performed correctly.
(5) nyoukan-nyourozyouhi-gan ureter-urothelium-cancer 尿管_尿路上皮_癌 (urothelial cancer)
Regarding errors due to syntactic analysis, an example is shown in Figure 3. Since “limbs pain” is a
7https://github.com/mocobeta/janome
7
Type Number Example NER error 25 kanseiken ha AIH-you no syoken o teisi-teita
liver-biopsy NOM AIH-like GEN finding ACC present-PST 肝生検ではＡＩＨ様の所見を呈していた。
kanseiken ha jikomen’ekisei-kan’en-you no syoken o teisi-teita Liver-biopsy NOM autoimmune-hepatitis–like GEN finding ACC present-PST
⇒肝生検では自己免疫性肝炎様の所見を呈していた。 (Liver biopsy showed findings suggestive of AIH
⇒ Liver biopsy showed findings suggestive of autoimmune hepatitis.) Syntactic analysis error 14 sou-you-kan ga at-ta sou-you ga at-ta
scratching-itching-sensation NOM be-PST scratching-itching NOM be-PST 掻痒感があった。 ⇒ そう痒があった。 (The patient had itch sensation. ⇒ The patient had pruritus.)
疼痛(pain) NPnc,nm,f が(NUM) NPga,nm,f\NPnc,nm,f
増悪(worsen) Snm,stem,f\NPga,nm,f し(EUPH) Snm,cont,f\Snm,stem,f
Snm,base,f\NPga,nm,f <B1 た(PST) Snm,base,f\Snm,cont,f
disease name, “limbs” and “pain” need to be combined first. However, according to the result of the syntax analysis, “pain worsened” was combined first, and then “limbs” was combined afterwards. This illustrates a case where the CCG syntactic structure for the disease name was not constructed correctly, leading to a failure to perform correct inference.

heading: 7 Conclusion
text: In this study, to flexibly perform inference involving knowledge for disease names, we extended the previous semantic analysis and logical inference system in the medical domain (Ishida et al., 2022). Specifically, we developed the knowledge injection module for the logical inference system, performing NER for disease names, searching for relevant knowledge using J-MeDic, and adding the resulting axioms to the theorem prover.
We also constructed a dataset for the NER task of disease names and an inference test set that requires knowledge injection of disease names. We evaluated our inference system using the constructed test set and as a result, we were able to perform correct inference for 106 out of 149 inference test cases. The future challenges are to expand the NER dataset and the inference test set, and to improve
the knowledge injection module to further enhance the performance of the inference system. Furthermore, a comparison will be performed between the neural models trained on the expanded medical inference test set and the proposed method.

heading: Acknowledgements
text: We thank the three anonymous reviewers for their helpful comments and suggestions, which improved this paper. This work was supported by JST, PRESTO grant number JPMJPR21C8, Japan.

heading: Future Work
text: The future challenges are to expand the NER dataset and the inference test set, and to improve
the knowledge injection module to further enhance the performance of the inference system. Furthermore, a comparison will be performed between the neural models trained on the expanded medical inference test set and the proposed method