# Research on Dual-Dimensional Entity Association-Based Question and Answering Technology for Smart Medicine

Authors: Pengjun Zhai; Yu Fang; Xue Cui
Journal: Mobile Information Systems
 (2022)

---

## Abstract

With the development of the Internet of Things, intelligent medical devices and intelligent consultation platforms have been rapidly popularized, providing great convenience for medical treatment to patients and consultation to doctors. In the face of large-scale medical electronic information data, how to automatically and accurately learn professional knowledge and realize application is very important. The existing intelligent medical question answering models typically use query expansion to improve the accuracy of model matching answers but ignore the corresponding entity association between questions and answers, and the method of randomly generating negative samples cannot train the model to capture more semantic information. To solve these problems, a question answering method based on dual-dimensional entity association for intelligent medicine is proposed. This method learns semantics from the dual-dimension of question and answer respectively. In the question dimension, query extension words with strong relevance to query intention are obtained through entity association in the medical knowledge graph. In the answer dimension, answer sentences are segmented and sampled by employing a variety of similarity distances to generate negative samples in different ranges, provide different levels of correlation information between entities for model training, and then integrate the trained model to improve the accuracy and robustness of the question answering model. The experimental results show that the question answering model proposed in this paper has a good improvement in accuracy.


---

## Body

## 1. Introduction

As an applied branch of the automatic question answering (QA) model, the medical question answering model has been a highline of research and application with the improvement of NLP (natural language processing) technology. The QA model consists of two stages [1]: question processing and question-answer matching. The former analyses and classifies the questions, extracts keywords, and reconstructs questions, while the latter searches and matches the answers based on semantic and syntactic analysis. At present, query expansion [2] is widely used to narrow the deviation between questions and answers to understand questions more accurately. Meanwhile, many research studies match questions and answers by rules [3], clustering [4], similarity [5], and neural network [6, 7] for training the model to make the obtained answers closer to the golden answer.However, query expansion methods based on keywords or semantics only start with the surface information such as statistics, medical dictionaries, and mutual information to mine candidate extension words, ignoring the key role of the relation between medical entities in question-answer corpus and negative medical entity recognition in obtaining extension words. In addition, the training methods of the QA model based on a single similarity or neural network collecting negative samples only focus on the relation between entities in a sample at a specific hierarchy, which lacks diversity and stability. Table1 gives two examples of our QA data, and there is a relation of disease causes symptoms between symptom entities such as “vomiting,” “abdominal pain,” and the disease entity of “gastrointestinal dysbacteriosis” in question 1 (Q1). However, the symptom entities of “protruding navel” and “reducible tumor” are ignored because of the query deviation, and the negative entities such as “no fever” also interfere with the QA model because they are not recognized. In addition, the answer 2 (A2) and A2’ of Q2 have high lexical similarity because they contain such entities as “blister shaped bulge,” “fever,” and “loss of appetite.”Table 1
QA examples from the Chinese pediatric corpus.
Example#1Q1: Baby has a protruding navel, reducible tumor, no fever, occasionally vomiting, and abdominal pain. How is this going on.A1: Baby abdominal distension, acid reflux, abdominal pain sometimes accompanied by vomiting, fever, consider caused by gastrointestinal dysbacteriosis, and imperfectDigestive system is also an essential factor.Example#2Q2: Small blisters appeared on the child’s hands and feet, a few days later, had a fever, and have been a loss of appetite. How is this going on.A2: According to the description of symptoms, it is supposed to herpes zoster is caused by virus infection. There will be blister shaped bulges in the area of nerve distribution, accompanied by fever and loss of appetite. It is recommended that you go to theHospital for examination and treatment.A2’: According to the description of symptoms, it is supposed to hand-foot-mouth disease. The disease often occurs in the mouth, hands and feet, with blister shaped bulge, fever, loss of appetite and other symptoms. It is recommended that you go to the hospital for antiviral treatment.An elegant framework is proposed in this paper to capture the potential entity association in Chinese medical QA corpus with the help of the medical knowledge graph, and adopt the idea of model integration [8] to train model focusing on multiple similarities at the same time by combining multiple base learners. Specifically, the main contributions are as follows:(i)
We harvest the Chinese medical QA sentences from the XunYiWenYao (http://3g.xywy.com/) and 39 Health (http://www.39.net/). The query intention classifier is trained on the pre-processed data by using the semi-supervised self-training method for realizing the automatic annotation of the query intentions in the original questions.(ii)
The proposed medical integrated QA approach focusing on dual-dimensional entity association, which focuses on the entity association from the dimensions of question and answer, reduces the noise of introducing extension words, and captures the entity relevance at different hierarchy through integrated learning so that to the model can match the golden answer more accurately.(iii)
The results on the QA dataset show that the effectiveness of the proposed model DDEA (a dual-dimensional entity association-based question and answering technology for smart medicine) in obtaining query extension words and matching questions and answers, and can more effectively capture the potential semantic information under different query intentions.


## 2. Related Work

### 2.1. Question Answering Based on Query Expansion

Since the content between the question sentence and answer sentence is different, the semantic deviation is caused, which significantly affects the accuracy of the QA model. Therefore, the query expansion method is introduced into the QA model, which makes up the semantic gap between questions and answers by adding words related to the answers to the original query. In the field of medical, external medical knowledge resources such as MeSH [9], UMLS [10], and several medical ontology databases [11] are employed as the source of extension words. However, the query expansion only based on synonyms is incapable of accurately capturing the semantic information in the corpus. Yang et al. [12] developed a QA system, which trained the classifier based on the Euclidean distance and word to select the appropriate words from the medical expert vocabulary to enhance the query of users. Although the performance of the QA model improved by ontology databases, it is difficult to construct and introduce in a specific system because of the large scale. Wang et al. [13] extended queries by finding the most relevant semantic association with potential feature vectors and association terms with triples. Li et al. [14] based on the weight of medical terms propose a query reconstruction method that weighted medical terms with self-information and then combined the weighted medical terms and the original query in proportion. Shen et al. [15] reflected the degree of association between words by utilizing the mutual information and took the entity concept with the highest mutual information value as the query extension word. To promote the performance of the QA model, Aicha et al. [16] extended queries with medical entities and semantic relations based on the external resource of OWL. Nasir et al. [17] presented a method of joint knowledge and related feedback, which analysed the diversity of query words and finds synonyms through different methods. Hu et al. [18] employed domain concepts to extend queries by fusing the domain knowledge graph and extracted context features to obtain context-aware of queries. However, the method of query expansion only adopts the concept of a domain entity and ignores the critical role of the relation between entities.


### 2.2. Question Answering Based on Model Integration

Computing the similarity between sentences or texts is greatly helpful in the QA model [19]. Scott et al. [20] proposed an enhanced lexical semantic resource model to raise the effectiveness of the QA model. Zhang et al. [6] employed the end-to-end multi-scale CNN based on the word vector to model question and answer respectively to find the correct answer using the similarity of question and answer. Based on distributed representation technology, Word2Vec, Si et al. [21] matched the correct answer by obtaining the cosine similarity between the sentence pairs. However, the performance of these models based on single models is unstable, which affects QA models’ accuracy.The method of integrating multiple excellent base models began to appear and be applied to the QA model to increase the effectiveness of the model. Hou et al. [22] selected the first N optimal classifiers for integration by training samples used the cross validation. Wang et al. [23] trained the models with the same hyperparameter and structure for several rounds and integrate them to select the answers for questions. Chen et al. [24] trained several stochastic artificial neural networks and integrated the models to predict the answers with high accuracy. Liu et al. [25] trained heterogeneous base models by the bagging method and then integrated them. Bandyopadhyay et al. [26] integrated BioBERT models pre-trained on the different corpus to achieve better results. Yang et al. [27] presented an adaptive decision fusion method, which adaptively combines classifiers with different levels of features to cultivate an answer selection model with robustness and effectiveness. However, all negative samples above with integration strategies come from random sampling, which cannot explicitly control the diversity of the base models. Therefore, Mehri et al. [28] put forward the idea of Multi-Granularity Training (MGT). The concept of MGT is to obtain the training samples set by calculating the semantic similarity between the positive and negative sample sets, and then sorting and segmenting the negative samples, thereby the MGT explicitly controlling the granularity of the semantic representation learned by the base model. However, the single dimension of semantic similarity distance does not guarantee that the model can fully employ the diversity of negative samples, resulting in insufficient diversity of the base models.


## 2.1. Question Answering Based on Query Expansion

Since the content between the question sentence and answer sentence is different, the semantic deviation is caused, which significantly affects the accuracy of the QA model. Therefore, the query expansion method is introduced into the QA model, which makes up the semantic gap between questions and answers by adding words related to the answers to the original query. In the field of medical, external medical knowledge resources such as MeSH [9], UMLS [10], and several medical ontology databases [11] are employed as the source of extension words. However, the query expansion only based on synonyms is incapable of accurately capturing the semantic information in the corpus. Yang et al. [12] developed a QA system, which trained the classifier based on the Euclidean distance and word to select the appropriate words from the medical expert vocabulary to enhance the query of users. Although the performance of the QA model improved by ontology databases, it is difficult to construct and introduce in a specific system because of the large scale. Wang et al. [13] extended queries by finding the most relevant semantic association with potential feature vectors and association terms with triples. Li et al. [14] based on the weight of medical terms propose a query reconstruction method that weighted medical terms with self-information and then combined the weighted medical terms and the original query in proportion. Shen et al. [15] reflected the degree of association between words by utilizing the mutual information and took the entity concept with the highest mutual information value as the query extension word. To promote the performance of the QA model, Aicha et al. [16] extended queries with medical entities and semantic relations based on the external resource of OWL. Nasir et al. [17] presented a method of joint knowledge and related feedback, which analysed the diversity of query words and finds synonyms through different methods. Hu et al. [18] employed domain concepts to extend queries by fusing the domain knowledge graph and extracted context features to obtain context-aware of queries. However, the method of query expansion only adopts the concept of a domain entity and ignores the critical role of the relation between entities.


## 2.2. Question Answering Based on Model Integration

Computing the similarity between sentences or texts is greatly helpful in the QA model [19]. Scott et al. [20] proposed an enhanced lexical semantic resource model to raise the effectiveness of the QA model. Zhang et al. [6] employed the end-to-end multi-scale CNN based on the word vector to model question and answer respectively to find the correct answer using the similarity of question and answer. Based on distributed representation technology, Word2Vec, Si et al. [21] matched the correct answer by obtaining the cosine similarity between the sentence pairs. However, the performance of these models based on single models is unstable, which affects QA models’ accuracy.The method of integrating multiple excellent base models began to appear and be applied to the QA model to increase the effectiveness of the model. Hou et al. [22] selected the first N optimal classifiers for integration by training samples used the cross validation. Wang et al. [23] trained the models with the same hyperparameter and structure for several rounds and integrate them to select the answers for questions. Chen et al. [24] trained several stochastic artificial neural networks and integrated the models to predict the answers with high accuracy. Liu et al. [25] trained heterogeneous base models by the bagging method and then integrated them. Bandyopadhyay et al. [26] integrated BioBERT models pre-trained on the different corpus to achieve better results. Yang et al. [27] presented an adaptive decision fusion method, which adaptively combines classifiers with different levels of features to cultivate an answer selection model with robustness and effectiveness. However, all negative samples above with integration strategies come from random sampling, which cannot explicitly control the diversity of the base models. Therefore, Mehri et al. [28] put forward the idea of Multi-Granularity Training (MGT). The concept of MGT is to obtain the training samples set by calculating the semantic similarity between the positive and negative sample sets, and then sorting and segmenting the negative samples, thereby the MGT explicitly controlling the granularity of the semantic representation learned by the base model. However, the single dimension of semantic similarity distance does not guarantee that the model can fully employ the diversity of negative samples, resulting in insufficient diversity of the base models.


## 3. Methods

The QA model in this paper is applied to the Chinese medical automatic question answering system. We are crawling the medical question-answer pairs as the corpus and then training the query intention classifier by utilizing a semi-supervised and self-training method to tag the intention of the questions. In addition, we develop a dual-dimensional entity association-based question and answering technology for smart medicine, DDEA. Figure1 shows the framework that gives a question and its answers as an example, which extends the question by the relation between medical entities and mutual information in the question dimension and then completes the negative sample segmentation sampling based on the multi-similarity distance in the answer dimension to training the base models that focus on the relevance between entities of different levels. Finally, the base models are integrated to achieve more accurate automatic QA matching for different query intentions.Figure 1
The framework of the medical integrated QA model focusing on the dual dimensional entity relation. Take the question and answer sample of question 2 as an example,Q2 and AQ2 are the question 2 and its positive answer, and ap2 is the negative answer sample. For the given question 2, the query extension words are first obtained based on the knowledge graph and mutual information in the question dimension, and then segment sampling is realized for the expanded answer from the two perspectives of lexical similarity and semantic similarity in the answer dimension. After that, train the corresponding base model and integrate them to complete the matching of questions and answers.Table 2
Examples of interrogative words in questions.
Question typesInterrogative wordsDiagnosisWhat diseases, what is it, what is going on…TreatmentHow to heal/treat, what should I do, how can I…SymptomSymptom, show that, display…PathogenyWhy, how is it that, what causes…Diagnosis-treatmentWhat disease and how to treat, how did this and what to do…


### 3.1. Corpus Annotation

Data cleaning: delete invalid data such as unreadable expression and without answers, and 7987 pairs of eligible question-answer pairs are obtained by deleting individual questions other than the classify of disease diagnosis, symptom, treatment, and pathogeny in the corpus to ensure the balance of the corpus. Next, unify disease names for the corpus and about 20% of the questions, i.e., 1408 questions, are selected to tag the intent category as an initial training set for the classifier.Intention classification: questions with the same intention often contain similar interrogative words even though they are different in various questions. The type of question is divided into five categories: disease diagnosis, treatment, symptom, pathogeny, and diagnosis-treatment. The latter is a compound type of question, which has both disease diagnosis interrogative words and treatment interrogative words. The examples of interrogative words are given in Table2.Automatic annotation: the query intention has a positive impact on extracting the key medical entities in questions, making the QA model accurately infer the medical entity types that may appear in the answers to avoid noises for the original query caused by introducing irrelevant extension words. Considering the sample imbalance in the dataset, the SVM [29] is selected, which is insensitive to the sample imbalance, as the initial classifier, and train the classifier by utilizing the interrogative features and TF-IDF features of questions. Finally, two physicians are invited to check and correct the annotation results of the classifier.


### 3.2. Obtaining Query Expansion Words

Query expansion is a vital step in the stage of question processing. Its accuracy directly affects the QA model performance. There, the candidate extension words more relevant with question are obtained by focusing on the relation between entities for questions under different query intentions, and the interference of negative medical entities on the value of mutual information between entities is eliminated through the negative medical entity recognition to obtain more accurate extension words.Due to lack of a complete medical knowledge base in Chinese, we first obtain a domain dictionary of disease, symptom, drug, and test by integrating 39 Health, Sogou thesaurus (https://pinyin.sogou.com/dict/cate/index/132), ICD-9-CM (https://www.cdc.gov/nchs/icd/icd9cm.htm), and ICD-10 (https://www.cdc.gov/nchs/icd/icd10.htm). The dictionary information is shown in Table 3. Meanwhile, we extract the triples with labeled departments from an open Chinese medical general knowledge graph (https://github.com/liuhuanyong/QASystemOnMedicalKG) and develop a Chinese medical knowledge graph by utilizing the medical data crawled from 39 Health. Then, combined with the medical domain dictionary, the initial query keywords are selected for the questions with intention labels. Here, the symptom entities are extracted as the initial query keywords for the question of disease diagnosis and diagnosis-treatment, the disease entities are extracted as the initial query keywords for the question of treatment and pathogeny. The query keywords Wk are obtained by removing the negative medical entities through trigger feature words and stop feature words based on the context algorithm [30] from the initial query keywords. Table 4 shows the trigger feature word and stop feature word included in the corpus.Table 3
Statistics of the domain dictionary.
Word typesDiseaseSymptomDrugTest#Entity87948498641280369870#Entity represents the number of entities.Table 4
Statistics of feature words.
Trigger feature wordsDeny, no, not exist, excludeStop feature wordsBut, however, yetThe possible types of medical entities in the answer corresponding to the question are predicted based on the query keywordsWk by using the inference rule R and the types of query intentions, and the inference rule is as follows:(1)QbelongsToI⟶QhasEntityTQ⟶AhasEntityTA,where Q and A are question and answer, respectively; I and TQ are the query intention and the type of query keyword Wk in question, respectively; and TA is the type of medical entity in answer. Then, if there are symptom entities in the questions of disease diagnosis, it may exist in the answer, i.e., TA≈TQ.Based on the medical knowledge graph KG in the Chinese language, the questions with various intentions are classified and expanded by adopting reasoning ruleR combining the query keyword Wk and the type of query intention T in the question. The query keyword Wk is normalized based on the entity in the KG to avoid the negative impact of the deviation between colloquial words in corpus and the same concept in KG. Then, combined with the medical entity type TA in answers obtained by reasoning rule R, the medical entity Mt is extracted from the entities of disease, symptom, drug, and pathogeny in KG. For the question of disease diagnosis, the intersection of disease medical entities corresponding to symptoms is regarded as candidate query extension words We′ because multiple diseases may cause several symptoms. In addition, the union of the corresponding drug and symptom medical entities is considered as candidate query extension words of treatment and symptom questions, respectively. The candidate query extension words of the diagnosis-treatment question are the union of the disease and drug entities. It is difficult to generalize with a few extension words because the pathogeny usually contains many sentences for the question of pathogeny, and this type of question will not be dealt with temporarily to avoid the noise.Considering that there are some infrequent medical entities in the answers, which will bring noises to obtain the extension words, this paper filters candidate query extension wordsWe′ through negative medical entity recognition and mutual information. Here, mutual information refers to the correlation between two words. It is obtained by calculating the frequency of the two words appearing in the common window,(2)Iw1,w2=logcw1,w2×Ncw1×cw2,where cw1,w2 is the number of occurrences w1 and w2 appearing in the co-occurrence window at the same time, cw1 and cw2 are the number of occurrences w1 and w2 appearing in the corpus, respectively, and N is the quantity of entities in the corpus.To avoid the interference of the word frequency of negative medical entities to calculate the mutual information of medical entities, the context algorithm is employed to recognize the negative medical entities, and the associated word frequency is mapped to 0. Supposing that medical entities in a question are independent of each other, the degree of association between the extension wordWe′ and the question Q can be obtained by calculating the sum of mutual information of each word pair, and the normalized result NMQ is compared with the hyperparameter of extension threshold to select the final expansion words We,(3)MQ=∑Wk∈Q∑We′∈AIWk,We′,NMQ=Mmax−MQMmax−Mmin,where Mmax and Mmin are the maximum and minimum of MQ, respectively. Algorithm 1 illustrates the specific process.Algorithm 1: Query expansion based on knowledge graph.
Input: question Q, question type T, interface rule R, Chinese Pediatric knowledge graph KG, threshold θOutput: expansion words list we0,…,l(1)
we0,…,j ← getKeywordsByType (checkNegative (Q, type))(2)
fori in 1, 2, …, jdo(3)
Normalizewki with KGInfer Medical Entity typeTA by RSearch Medical Entity by TypeTA with KG: Mt(4)
end for(5)
ifT equal diagnose then(6)
Merge all response entity:M=M1∩M2⋯∩Mt(7)
else(8)
M=M1∩M2⋯∩Mt(9)
end if(10)
Get Candidate query extension words:we′0,…,h←M(11)
forn in 0, 1, …, hdo(12)
Calculate the degree of associationMQNMQ←NormalizeMQ(13)
ifNMQ>0then(14)
We+=We′n(15)
else(16)
We==We′n(17)
end if(18)
returnWe0,…,l(19)
end for


### 3.3. Question and Answer Matching

The randomly generated negative samples may have low similarity and poor correlation with the positive samples, which will harm the model training and reduce the effectiveness of the model. Inspired by the integration method of the Multi-Granularity Training (MGT) model [28], we develop an idea of negative sample generation based on Multi-Similarity Segmented Sampling (MSSS) in the answer dimension, that is, the initial negative samples are segmented and sampled at two levels of semantic similarity and lexical similarity to construct training sample sets focusing on different levels of relevance between entities, which is used to train distinct base models, and then these base models are integrated to match the questions and answers. The negative sample generation strategy of MSSS enables the base models to learn multi-type statement representation such as semantic and lexical, and multi-granularity statement representation from subtle to abstract.The TF-IDF algorithm [31] is frequently employed to calculate text similarity based on lexical due to it can capture the importance of words in the text through the statistical method. The domain words are often more discriminative and important than common words for the medical question answering corpus in the Chinese language. Thus, a TF-IDF algorithm of domain weighting is utilized to calculate the lexical similarity between positive and negative samples:(4)TnWn=tfWn∗idfWn,TnWf=α∗tfWf∗idfWf,where TnWn and TnWf are the TFIDF of the common words Wn and domain words Wf, respectively, tf is the term frequency and idf is the inverse document frequency, respectively, and α is the weight.In addition, the tree structure Chinese Medical Subject Headings (CMeSH,http://cmesh.imicams.ac.cn/index.action?action=index), which can clearly show the semantic relation between medical vocabulary, is adopted to calculate the similarity between positive and negative samples at the semantic level. Specifically, based on CMeSH and referring to the calculation method of semantic similarity proposed by Jiang et al. [32], we first gain the semantic similarity SimWf1,Wf2 between domain words and then the semantic similarity SimS+,S− between positive and negative samples is calculated:(5)SimWf1,Wf2=11+DistWf1,Wf2,SimS+,S−=2∑Wf+∈S+∑Wf−∈S−MaxSimWf+,Wf−S++S−,where DistWf1,Wf2 is the semantic distance between domain words Wf1 and Wf2, and S+ and S− are the number of domain words in the sentence of positive and negative samples.By sortingTnWn, TnWf, and SimWf1,Wf2, two kinds of negative sample sequences of similarity are obtained and then sample them by segments s˜ to gain different negative sample sets SQes˜−. under different segments for extended questions Qe,(6)SQes˜−=a∈AdAQe,a∈MaxdSimQe,s˜−1,MaxdSimQe,s˜.Here, A denotes the answer, AQe is the correct answer to question Qe, dAQe,a is the cosine similarity between positive and negative answers, MaxdSimQe,s˜ is the maximum semantic similarity value in segment s˜, and MaxdTQe,s˜ is the maximum lexical similarity value in segment s˜. Here, for any i∈1,k−1, the ith negative sample NQ,il in segment s˜ of question Qe satisfies NQ,il∼UniformSQes˜−. Figure 2 illustrates that the negative sample sets are generated by segmented sampling under the given similarity, which takes the number of segments s˜=3 as an example, where AQ is the positive sample and a1∼a9 are the negative samples, and there are three negative sample sets in Figure 2 such as a1∼a3 in the first segment, a4∼a6 in the second segment, and a8∼a9 in the third segment. As this example, multiple training sample sets can be generated. In this paper, we train a base model M¯i on each training sample set to make the various training sample sets fully utilized by QA, and the base models are integrated through a combination method of weighted average. The prediction probability PAQ|Q of the final integrated model is as follows:(7)PAQ|Q=∑i=1GμipiAQ|Q,μi=p^iAQ|Q∑i=1Gp^iAQ|Q,where G is the total number of the base models, μi is the weight of the ith base model, and piAQ|Q is the predicted result of the ith base model. Here, the weight μi depends on the accuracy p^iAQ|Q of the base model on the validation set, and the weight ratio of the base model with high accuracy in the integrated model is relatively large.Figure 2
Diagram of segmented sampling.We minimize formula (8) to train the model:(8)ℒΘ=−λ1NMQ+λ2PAQ|Q,where λ1 and λ2 are the weights and Θ is the set of all the parameters of the DDEA. The question answer matching algorithm is shown in Algorithm 2.Algorithm 2: Question answer matching based on MSSS.
Input: Question Q, segment number s¯, Algorithm 1Output: Integration model IM(1)
Qe←Algorithm1Get positive answerAOo and source negative sample S_(2)
forW in S_ do(3)
GetTFIDTTnW for common and domain words respectively(4)
forW˜ in AOo do(5)
CalculateSimS+,S−(6)
end for(7)
end for(8)
S1−=keywordSimilarityAOo,S−←getNegativeSampleByTnWS2−=sematicSimilarityAOo,S−←getNegativeSampleBySimS+,S−(9)
fori in 1,2,…,s˜do(10)
Get NegativeSamplesSOoi−forS1−andS2−Triple.append (constructQe,AOo,SOoi−)(11)
end for(12)
forj in 1,2,…,2s˜ do(13)
Get base model←trainTripleiMeasure base modelsp and get weights μIM=sign∑μp(14)
returnIM(15)
end for


## 3.1. Corpus Annotation

Data cleaning: delete invalid data such as unreadable expression and without answers, and 7987 pairs of eligible question-answer pairs are obtained by deleting individual questions other than the classify of disease diagnosis, symptom, treatment, and pathogeny in the corpus to ensure the balance of the corpus. Next, unify disease names for the corpus and about 20% of the questions, i.e., 1408 questions, are selected to tag the intent category as an initial training set for the classifier.Intention classification: questions with the same intention often contain similar interrogative words even though they are different in various questions. The type of question is divided into five categories: disease diagnosis, treatment, symptom, pathogeny, and diagnosis-treatment. The latter is a compound type of question, which has both disease diagnosis interrogative words and treatment interrogative words. The examples of interrogative words are given in Table2.Automatic annotation: the query intention has a positive impact on extracting the key medical entities in questions, making the QA model accurately infer the medical entity types that may appear in the answers to avoid noises for the original query caused by introducing irrelevant extension words. Considering the sample imbalance in the dataset, the SVM [29] is selected, which is insensitive to the sample imbalance, as the initial classifier, and train the classifier by utilizing the interrogative features and TF-IDF features of questions. Finally, two physicians are invited to check and correct the annotation results of the classifier.


## 3.2. Obtaining Query Expansion Words

Query expansion is a vital step in the stage of question processing. Its accuracy directly affects the QA model performance. There, the candidate extension words more relevant with question are obtained by focusing on the relation between entities for questions under different query intentions, and the interference of negative medical entities on the value of mutual information between entities is eliminated through the negative medical entity recognition to obtain more accurate extension words.Due to lack of a complete medical knowledge base in Chinese, we first obtain a domain dictionary of disease, symptom, drug, and test by integrating 39 Health, Sogou thesaurus (https://pinyin.sogou.com/dict/cate/index/132), ICD-9-CM (https://www.cdc.gov/nchs/icd/icd9cm.htm), and ICD-10 (https://www.cdc.gov/nchs/icd/icd10.htm). The dictionary information is shown in Table 3. Meanwhile, we extract the triples with labeled departments from an open Chinese medical general knowledge graph (https://github.com/liuhuanyong/QASystemOnMedicalKG) and develop a Chinese medical knowledge graph by utilizing the medical data crawled from 39 Health. Then, combined with the medical domain dictionary, the initial query keywords are selected for the questions with intention labels. Here, the symptom entities are extracted as the initial query keywords for the question of disease diagnosis and diagnosis-treatment, the disease entities are extracted as the initial query keywords for the question of treatment and pathogeny. The query keywords Wk are obtained by removing the negative medical entities through trigger feature words and stop feature words based on the context algorithm [30] from the initial query keywords. Table 4 shows the trigger feature word and stop feature word included in the corpus.Table 3
Statistics of the domain dictionary.
Word typesDiseaseSymptomDrugTest#Entity87948498641280369870#Entity represents the number of entities.Table 4
Statistics of feature words.
Trigger feature wordsDeny, no, not exist, excludeStop feature wordsBut, however, yetThe possible types of medical entities in the answer corresponding to the question are predicted based on the query keywordsWk by using the inference rule R and the types of query intentions, and the inference rule is as follows:(1)QbelongsToI⟶QhasEntityTQ⟶AhasEntityTA,where Q and A are question and answer, respectively; I and TQ are the query intention and the type of query keyword Wk in question, respectively; and TA is the type of medical entity in answer. Then, if there are symptom entities in the questions of disease diagnosis, it may exist in the answer, i.e., TA≈TQ.Based on the medical knowledge graph KG in the Chinese language, the questions with various intentions are classified and expanded by adopting reasoning ruleR combining the query keyword Wk and the type of query intention T in the question. The query keyword Wk is normalized based on the entity in the KG to avoid the negative impact of the deviation between colloquial words in corpus and the same concept in KG. Then, combined with the medical entity type TA in answers obtained by reasoning rule R, the medical entity Mt is extracted from the entities of disease, symptom, drug, and pathogeny in KG. For the question of disease diagnosis, the intersection of disease medical entities corresponding to symptoms is regarded as candidate query extension words We′ because multiple diseases may cause several symptoms. In addition, the union of the corresponding drug and symptom medical entities is considered as candidate query extension words of treatment and symptom questions, respectively. The candidate query extension words of the diagnosis-treatment question are the union of the disease and drug entities. It is difficult to generalize with a few extension words because the pathogeny usually contains many sentences for the question of pathogeny, and this type of question will not be dealt with temporarily to avoid the noise.Considering that there are some infrequent medical entities in the answers, which will bring noises to obtain the extension words, this paper filters candidate query extension wordsWe′ through negative medical entity recognition and mutual information. Here, mutual information refers to the correlation between two words. It is obtained by calculating the frequency of the two words appearing in the common window,(2)Iw1,w2=logcw1,w2×Ncw1×cw2,where cw1,w2 is the number of occurrences w1 and w2 appearing in the co-occurrence window at the same time, cw1 and cw2 are the number of occurrences w1 and w2 appearing in the corpus, respectively, and N is the quantity of entities in the corpus.To avoid the interference of the word frequency of negative medical entities to calculate the mutual information of medical entities, the context algorithm is employed to recognize the negative medical entities, and the associated word frequency is mapped to 0. Supposing that medical entities in a question are independent of each other, the degree of association between the extension wordWe′ and the question Q can be obtained by calculating the sum of mutual information of each word pair, and the normalized result NMQ is compared with the hyperparameter of extension threshold to select the final expansion words We,(3)MQ=∑Wk∈Q∑We′∈AIWk,We′,NMQ=Mmax−MQMmax−Mmin,where Mmax and Mmin are the maximum and minimum of MQ, respectively. Algorithm 1 illustrates the specific process.Algorithm 1: Query expansion based on knowledge graph.
Input: question Q, question type T, interface rule R, Chinese Pediatric knowledge graph KG, threshold θOutput: expansion words list we0,…,l(1)
we0,…,j ← getKeywordsByType (checkNegative (Q, type))(2)
fori in 1, 2, …, jdo(3)
Normalizewki with KGInfer Medical Entity typeTA by RSearch Medical Entity by TypeTA with KG: Mt(4)
end for(5)
ifT equal diagnose then(6)
Merge all response entity:M=M1∩M2⋯∩Mt(7)
else(8)
M=M1∩M2⋯∩Mt(9)
end if(10)
Get Candidate query extension words:we′0,…,h←M(11)
forn in 0, 1, …, hdo(12)
Calculate the degree of associationMQNMQ←NormalizeMQ(13)
ifNMQ>0then(14)
We+=We′n(15)
else(16)
We==We′n(17)
end if(18)
returnWe0,…,l(19)
end for


## 3.3. Question and Answer Matching

The randomly generated negative samples may have low similarity and poor correlation with the positive samples, which will harm the model training and reduce the effectiveness of the model. Inspired by the integration method of the Multi-Granularity Training (MGT) model [28], we develop an idea of negative sample generation based on Multi-Similarity Segmented Sampling (MSSS) in the answer dimension, that is, the initial negative samples are segmented and sampled at two levels of semantic similarity and lexical similarity to construct training sample sets focusing on different levels of relevance between entities, which is used to train distinct base models, and then these base models are integrated to match the questions and answers. The negative sample generation strategy of MSSS enables the base models to learn multi-type statement representation such as semantic and lexical, and multi-granularity statement representation from subtle to abstract.The TF-IDF algorithm [31] is frequently employed to calculate text similarity based on lexical due to it can capture the importance of words in the text through the statistical method. The domain words are often more discriminative and important than common words for the medical question answering corpus in the Chinese language. Thus, a TF-IDF algorithm of domain weighting is utilized to calculate the lexical similarity between positive and negative samples:(4)TnWn=tfWn∗idfWn,TnWf=α∗tfWf∗idfWf,where TnWn and TnWf are the TFIDF of the common words Wn and domain words Wf, respectively, tf is the term frequency and idf is the inverse document frequency, respectively, and α is the weight.In addition, the tree structure Chinese Medical Subject Headings (CMeSH,http://cmesh.imicams.ac.cn/index.action?action=index), which can clearly show the semantic relation between medical vocabulary, is adopted to calculate the similarity between positive and negative samples at the semantic level. Specifically, based on CMeSH and referring to the calculation method of semantic similarity proposed by Jiang et al. [32], we first gain the semantic similarity SimWf1,Wf2 between domain words and then the semantic similarity SimS+,S− between positive and negative samples is calculated:(5)SimWf1,Wf2=11+DistWf1,Wf2,SimS+,S−=2∑Wf+∈S+∑Wf−∈S−MaxSimWf+,Wf−S++S−,where DistWf1,Wf2 is the semantic distance between domain words Wf1 and Wf2, and S+ and S− are the number of domain words in the sentence of positive and negative samples.By sortingTnWn, TnWf, and SimWf1,Wf2, two kinds of negative sample sequences of similarity are obtained and then sample them by segments s˜ to gain different negative sample sets SQes˜−. under different segments for extended questions Qe,(6)SQes˜−=a∈AdAQe,a∈MaxdSimQe,s˜−1,MaxdSimQe,s˜.Here, A denotes the answer, AQe is the correct answer to question Qe, dAQe,a is the cosine similarity between positive and negative answers, MaxdSimQe,s˜ is the maximum semantic similarity value in segment s˜, and MaxdTQe,s˜ is the maximum lexical similarity value in segment s˜. Here, for any i∈1,k−1, the ith negative sample NQ,il in segment s˜ of question Qe satisfies NQ,il∼UniformSQes˜−. Figure 2 illustrates that the negative sample sets are generated by segmented sampling under the given similarity, which takes the number of segments s˜=3 as an example, where AQ is the positive sample and a1∼a9 are the negative samples, and there are three negative sample sets in Figure 2 such as a1∼a3 in the first segment, a4∼a6 in the second segment, and a8∼a9 in the third segment. As this example, multiple training sample sets can be generated. In this paper, we train a base model M¯i on each training sample set to make the various training sample sets fully utilized by QA, and the base models are integrated through a combination method of weighted average. The prediction probability PAQ|Q of the final integrated model is as follows:(7)PAQ|Q=∑i=1GμipiAQ|Q,μi=p^iAQ|Q∑i=1Gp^iAQ|Q,where G is the total number of the base models, μi is the weight of the ith base model, and piAQ|Q is the predicted result of the ith base model. Here, the weight μi depends on the accuracy p^iAQ|Q of the base model on the validation set, and the weight ratio of the base model with high accuracy in the integrated model is relatively large.Figure 2
Diagram of segmented sampling.We minimize formula (8) to train the model:(8)ℒΘ=−λ1NMQ+λ2PAQ|Q,where λ1 and λ2 are the weights and Θ is the set of all the parameters of the DDEA. The question answer matching algorithm is shown in Algorithm 2.Algorithm 2: Question answer matching based on MSSS.
Input: Question Q, segment number s¯, Algorithm 1Output: Integration model IM(1)
Qe←Algorithm1Get positive answerAOo and source negative sample S_(2)
forW in S_ do(3)
GetTFIDTTnW for common and domain words respectively(4)
forW˜ in AOo do(5)
CalculateSimS+,S−(6)
end for(7)
end for(8)
S1−=keywordSimilarityAOo,S−←getNegativeSampleByTnWS2−=sematicSimilarityAOo,S−←getNegativeSampleBySimS+,S−(9)
fori in 1,2,…,s˜do(10)
Get NegativeSamplesSOoi−forS1−andS2−Triple.append (constructQe,AOo,SOoi−)(11)
end for(12)
forj in 1,2,…,2s˜ do(13)
Get base model←trainTripleiMeasure base modelsp and get weights μIM=sign∑μp(14)
returnIM(15)
end for


## 4. Experiments

### 4.1. Dataset

We are crawling the question-answer pairs in Chinese medical from XunYiWenYao and 39 Health and annotating them by the automatic annotation method of query intention mentioned in Section3.1. Therefore, the Chinese medical question answering (CMQA) dataset is developed, which is composed of triples qi,ai+,ai−, where qi, ai+, and ai− are the question, positive answer, and negative answer, respectively. The negative samples in the triples are generated by the method based on MSSS in Section 3.3. The training set, validation set, and test set of the experimental data are basically divided according to the proportion of 10%, 10%, and 80% of the data volume and also combined with the length of characters in question sentences and answer sentences. To reduce the experimental error caused by the data, we try to keep the length of characters in question sentences and answer sentences consistent. Therefore, 6287, 850, and 850 question-answer pairs in the training set, validation set, and test set of the experimental data are finally selected. Table 5 shows the CMQA.Table 5
Statistics of CMQA for the question answering model in Chinese pediatric.
Dataset#Question1#Answer2Ave. #CharQ3Ave. #CharA4Positive : negative5Train62876287971711 : 50Valid850850941731 : 100Test850850951691 : 100Total7987798795170—1,2#Question and #Answer represents the quantity of questions and answers, respectively. 3,4Ave. #CharQ and Ave. #CharA represent the average of questions and answers, respectively. 5Positive: negative is the ratio of positive and negative samples.


### 4.2. Details

In this study, a few comparative experiments are implemented to highlight the positive influence of entity relation in the knowledge graph for obtaining the query extension words and validate the improvement of the integration model based on multi-similarity negative sample generation on the effect of questions-answers matching. The ACC@1, ACC@3, MRR, MAP [33], and NDCG [34] are selected to evaluate the performance of models.The experiments employ a scikit-learn machine learning module based on the Pytorch framework (https://pytorch.org/) and utilize the word vector published by Chinese-word-vectors [35]. The Adam Optimizer is selected, and the learning rate is initially set to 0.001, the dropout is set to 0.3, and the weight of domain words in the TF-IDF algorithm and the number of segments for segmented sampling are set to 0.6 and 3, respectively.


### 4.3. Results

From Table6, the statistics of query intention classification for questions in the CMQA that are automatically tagged with query intention and the results of this annotation are checked by doctors.Table 6
Statistics of the query intention category for questions.
Question typeDiagnosisTreatmentSymptomPathogenyDia-TreNumber28151578257493981Dia-Tre represents diagnosis-treatment.To evaluate the significance of the proposed query expansion method based on the entity relation in the knowledge graph and the mutual information of negative medical entities, the QA models based on the hybrid neural network designed by Zhang [36] are selected as the QA encoder, such as stack-CNN, multi-CNN, BIGRU, BIGRU-CNN, and multi-stack-CNN, where the last one is the combination of the first two frameworks. Table 7 illustrates the answer select performance of models based on the original query, thesaurus expansion (QE-T), and our query expansion (QE-KG), and extending the question can improve the precision. However, the results of multi-stack-CNN under QE-T are lower than those of the original query, which indicates that extending the question only by matching the thesaurus of keywords may introduce noise. The ACC@1 of QE-KG is 3.88% higher than that of the original query on average. Comparing to QE-T, it increased by 3.23%. These results show that the method of query expansion based on the entity relation can improve the accuracy of extension words by utilizing the potential knowledge information and reduce the noise introduced by extension words by recognizing negative medical entities. In addition, the MAP and NDCG of QE-KG are improved, which respectively shows that the accuracy and the correlation between the returned answers of QE-KG are higher.Table 7
Statistics of CMQA for the question answering model in Chinese pediatric.
ModelsACC@1ACC@3MRRMAPNDCGOriginal queryBIGRU32.0454.8947.3047.3058.78Stack-CNN35.6959.4850.9150.9161.63Multi-CNN43.7064.7857.6457.6466.98Multi-stack-CNN44.6466.9058.2758.2767.43BIGRU-CNN44.7665.7258.4958.4967.75Mean value40.1762.3554.5254.5264.51Query expansion method based on thesaurus (QE-T)BIGRU34.6354.4248.5448.5459.64Stack-CNN36.2860.9051.8651.8662.48Multi-CNN43.9365.4957.3357.3366.70Multi-stack-CNN44.29↓65.25↓58.12↓58.12↓67.35↓BIGRU-CNN44.9966.6758.5858.5867.76Mean value40.8262.5554.8954.8964.79Our query expansion method (QE-KG)BIGRU38.1659.2552.5252.5262.93Stack-CNN40.1663.0254.8254.8264.78Multi-CNN46.5368.4360.0060.0068.95Multi-stack-CNN47.4769.2661.0461.0469.71BIGRU-CNN47.9469.8561.6261.6270.16Mean value44.0565.9658.0058.0067.31To prove the validity of the integrated learning method of base models trained by negative samples generated through segmented sampling based on multi-similarity in the accuracy and robustness of the QA method, this paper takes the BIGRU-CNN with the best performance as the base model structure, and implements and compares the integrated models based on a single model, negative random sampling, MGT [28], and entity relation of our DDEA. Table 8 shows the results of different models by integrate learning. Here, we design the experiments of the integrated model based on the negative random sampling from three aspects: parameter random initialization, different checkpoints, and training set with random sampling. The number of base models is set to 6, and the specific reasons are explained in Section 4.4. There are two negative samples generated methods by segmented sampling based on the lexical similarity of domain word weight and the semantic similarity of CMeSH for the integrated model based on MGT, and three base models are selected from the perspective of lexical similarity and semantic similarity for DDEA to ensure that the number of base models is consistent with that of the contrast models. From Table 8, the single model is inferior to other integrated models, and compared with the integrated models based on negative random sampling, the MGT based on similarity sampling shows more than 4.65% improvement on the ACC@1 on average, and that of DDEA outperforms the MGT-based methods by up to 1.57%, which show that the accuracy of the QA model is improved by learning more comprehensive language representation from multiple levels such as lexical and semantic through DDEA. In addition, the result of DDEA on NDCG is 0.99% higher than that of MGT, which shows that DDEA is more stable.Table 8
Results of different models by integrate learning.
Integrate modelsBase model typeACC@1ACC@3MRRNDCGSingle model—47.9469.8561.6270.16Negative random samplingRandom initialization (6)49.1271.5063.0171.43Checkpoint (6)50.7772.2063.9872.11Training set (6)54.1876.2167.4274.94Mean value51.3673.3064.8072.83MGTLexical similarity (6)55.6377.4768.3075.37Semantic similarity (6)56.3978.3368.9176.20Mean value56.0177.9068.6175.79DDEA (our)Lexical similarity (3) + semantic similarity (3)57.5879.6569.5776.78


### 4.4. Discussion

In this paper, the QA model DDEA starts from the relation between entities to mine the extension words that have higher relevance with the key medical entities of the questions, and then trains base models based on negative samples generated by the segmented sampling of multiple similarities to promote the precision of the model. The classification of query intention can help the model specifically extract the key medical entities in questions. The threshold of confidence has a significant influence on the query intention classifier accuracy. From Figure3, the classifier accuracy is up to the highest when it is set to 0.8.Figure 3
Accuracy of the intention classifier under different confidence levels.According to the characteristics of the Chinese medical QA dataset, the words are divided into common words and domain words in the aspect of lexical similarity and the importance of domain words is emphasized by weight, which will directly affect the performance of the results of lexical similarity. Therefore, we evaluate the integrated model of BIGRU-CNN trained by six negative samples generated by segmented sampling based on the lexical similarity with different weights, and the ACC@1 is highest when the weight is 0.6, which is shown in Figure4. In addition, the number of base models and the learning granularity of the base model are directly determined by the number of segments in the process of negative sample generation. Figure 4 explains the ACC@1 results for various numbers of segments, and the optimal number of segments is three. In addition, it shows a drastical decline when the number of segments goes down, making it hard for the model to take advantage of multi-granularity learning. The degree of discrimination between the base models is reduced when the number of segments is four or five, which leads to the reduction of ACC@1 and the calculation time is longer. Therefore, the number of segments is set to three, i.e., the three base models for each similarity level, a total of six base models.Figure 4
ACC@1 according to the value of different parameters.Table9 provides several examples of QA models for a disease diagnosis question to show the highest score answers returned by these models, including original query, query expansion based on thesaurus (QE-T), query expansion based on knowledge graph (QE-KG), integrated model based on semantic similarity with QE-KG (QE-KG + MGT), and our model DDEA for illustrating the advantages of DDEA more intuitively. BIGRU-CNN with the best performance is selected as the model structure. The answers in Table 9 show that the query expansion model QE-KG has a higher matching degree between the extension words and correct answers. In contrast, the query expansion model QE-T causes query deviation due to the introduction of medical entities with low relevance with the correct answers. Through the comparison of answers, QE-KG outperforms the QE-T because QE-KG can usefully shorten the gap of semantic among questions and answers by matching potential medical entities in answers for the original query to obtain the correct disease entity of “umbilical hernia,” but there are still differences between the returned answers and the correct answers. The integrated model DDEA successfully captures the key information, either the correct diseases or “protruding navel, vomiting” in the question, and DDEA is based on the negative samples generated by segmented sampling with multi-similarity, which makes the model learn the language representation in multi-granularity from subtle to abstract but also ensure it to learn the language representation in multi-level that increases the precision of the model prediction.Table 9
Summary of different works pertaining to face and speech fusion.
ModelsQuestionReturned answersCorrect answersOriginal queryBaby is born more than 40 days: has protruding navel, reducible tumor, occasionally vomiting, and abdominal pain, how is this going on.Baby abdominal distension, acid reflux, abdominal pain sometimes accompanied by vomiting, fever, consider caused by gastrointestinal dysbacteriosis, and imperfect digestive system is also an important factor.If the child’s navel is raised, consider that it is related to mild umbilical hernia. You can observe whether the child’s abdomen has a mass raised from the body surface. If the mass cannot disappear after lying on his back, and the child has vomiting, abdominal distension, and other symptoms, you should see a doctor immediately.QE-TBaby is born more than 40 days: has protruding navel, reducible tumor, occasionally vomiting, and abdominal pain, how is this going on.cysts, nauseaBrain cyst is a swelling formed by cerebrospinal fluid surrounded in the pia mater. If not treated in time, with the increase of the cyst, it can cause headache, dizziness, nausea and vomiting, and other symptoms.QE-KGBaby is born more than 40 days: has protruding navel, reducible tumor, occasionally vomiting, and abdominal pain, how is this going on.umbilical hernia, infantile umbilical herniaUmbilical hernia includes congenital type, adult type, and infant type. Infantile umbilical hernia is characterized by raised walnut size tumor in the umbilicus. In crying and after exercise, the tumor becomes larger and feels more solid.QE-KG + MGTHernias include femoral hernia, indirect inguinal hernia and umbilical hernia. When the baby is crying or defecating, infantile umbilical hernia can press the raised abdominal contents back to the umbilical part. If the bulge progressive increase, it can be pressed by the abdominal belt.DDEAIf the child’s navel is raised, consider that it is related to mild umbilical hernia. You can observe whether the child’s abdomen has a mass raised from the body surface. If the mass cannot disappear after lying on his back, the child has vomiting, abdominal distension and other symptoms, you should see a doctor immediately


## 4.1. Dataset

We are crawling the question-answer pairs in Chinese medical from XunYiWenYao and 39 Health and annotating them by the automatic annotation method of query intention mentioned in Section3.1. Therefore, the Chinese medical question answering (CMQA) dataset is developed, which is composed of triples qi,ai+,ai−, where qi, ai+, and ai− are the question, positive answer, and negative answer, respectively. The negative samples in the triples are generated by the method based on MSSS in Section 3.3. The training set, validation set, and test set of the experimental data are basically divided according to the proportion of 10%, 10%, and 80% of the data volume and also combined with the length of characters in question sentences and answer sentences. To reduce the experimental error caused by the data, we try to keep the length of characters in question sentences and answer sentences consistent. Therefore, 6287, 850, and 850 question-answer pairs in the training set, validation set, and test set of the experimental data are finally selected. Table 5 shows the CMQA.Table 5
Statistics of CMQA for the question answering model in Chinese pediatric.
Dataset#Question1#Answer2Ave. #CharQ3Ave. #CharA4Positive : negative5Train62876287971711 : 50Valid850850941731 : 100Test850850951691 : 100Total7987798795170—1,2#Question and #Answer represents the quantity of questions and answers, respectively. 3,4Ave. #CharQ and Ave. #CharA represent the average of questions and answers, respectively. 5Positive: negative is the ratio of positive and negative samples.


## 4.2. Details

In this study, a few comparative experiments are implemented to highlight the positive influence of entity relation in the knowledge graph for obtaining the query extension words and validate the improvement of the integration model based on multi-similarity negative sample generation on the effect of questions-answers matching. The ACC@1, ACC@3, MRR, MAP [33], and NDCG [34] are selected to evaluate the performance of models.The experiments employ a scikit-learn machine learning module based on the Pytorch framework (https://pytorch.org/) and utilize the word vector published by Chinese-word-vectors [35]. The Adam Optimizer is selected, and the learning rate is initially set to 0.001, the dropout is set to 0.3, and the weight of domain words in the TF-IDF algorithm and the number of segments for segmented sampling are set to 0.6 and 3, respectively.


## 4.3. Results

From Table6, the statistics of query intention classification for questions in the CMQA that are automatically tagged with query intention and the results of this annotation are checked by doctors.Table 6
Statistics of the query intention category for questions.
Question typeDiagnosisTreatmentSymptomPathogenyDia-TreNumber28151578257493981Dia-Tre represents diagnosis-treatment.To evaluate the significance of the proposed query expansion method based on the entity relation in the knowledge graph and the mutual information of negative medical entities, the QA models based on the hybrid neural network designed by Zhang [36] are selected as the QA encoder, such as stack-CNN, multi-CNN, BIGRU, BIGRU-CNN, and multi-stack-CNN, where the last one is the combination of the first two frameworks. Table 7 illustrates the answer select performance of models based on the original query, thesaurus expansion (QE-T), and our query expansion (QE-KG), and extending the question can improve the precision. However, the results of multi-stack-CNN under QE-T are lower than those of the original query, which indicates that extending the question only by matching the thesaurus of keywords may introduce noise. The ACC@1 of QE-KG is 3.88% higher than that of the original query on average. Comparing to QE-T, it increased by 3.23%. These results show that the method of query expansion based on the entity relation can improve the accuracy of extension words by utilizing the potential knowledge information and reduce the noise introduced by extension words by recognizing negative medical entities. In addition, the MAP and NDCG of QE-KG are improved, which respectively shows that the accuracy and the correlation between the returned answers of QE-KG are higher.Table 7
Statistics of CMQA for the question answering model in Chinese pediatric.
ModelsACC@1ACC@3MRRMAPNDCGOriginal queryBIGRU32.0454.8947.3047.3058.78Stack-CNN35.6959.4850.9150.9161.63Multi-CNN43.7064.7857.6457.6466.98Multi-stack-CNN44.6466.9058.2758.2767.43BIGRU-CNN44.7665.7258.4958.4967.75Mean value40.1762.3554.5254.5264.51Query expansion method based on thesaurus (QE-T)BIGRU34.6354.4248.5448.5459.64Stack-CNN36.2860.9051.8651.8662.48Multi-CNN43.9365.4957.3357.3366.70Multi-stack-CNN44.29↓65.25↓58.12↓58.12↓67.35↓BIGRU-CNN44.9966.6758.5858.5867.76Mean value40.8262.5554.8954.8964.79Our query expansion method (QE-KG)BIGRU38.1659.2552.5252.5262.93Stack-CNN40.1663.0254.8254.8264.78Multi-CNN46.5368.4360.0060.0068.95Multi-stack-CNN47.4769.2661.0461.0469.71BIGRU-CNN47.9469.8561.6261.6270.16Mean value44.0565.9658.0058.0067.31To prove the validity of the integrated learning method of base models trained by negative samples generated through segmented sampling based on multi-similarity in the accuracy and robustness of the QA method, this paper takes the BIGRU-CNN with the best performance as the base model structure, and implements and compares the integrated models based on a single model, negative random sampling, MGT [28], and entity relation of our DDEA. Table 8 shows the results of different models by integrate learning. Here, we design the experiments of the integrated model based on the negative random sampling from three aspects: parameter random initialization, different checkpoints, and training set with random sampling. The number of base models is set to 6, and the specific reasons are explained in Section 4.4. There are two negative samples generated methods by segmented sampling based on the lexical similarity of domain word weight and the semantic similarity of CMeSH for the integrated model based on MGT, and three base models are selected from the perspective of lexical similarity and semantic similarity for DDEA to ensure that the number of base models is consistent with that of the contrast models. From Table 8, the single model is inferior to other integrated models, and compared with the integrated models based on negative random sampling, the MGT based on similarity sampling shows more than 4.65% improvement on the ACC@1 on average, and that of DDEA outperforms the MGT-based methods by up to 1.57%, which show that the accuracy of the QA model is improved by learning more comprehensive language representation from multiple levels such as lexical and semantic through DDEA. In addition, the result of DDEA on NDCG is 0.99% higher than that of MGT, which shows that DDEA is more stable.Table 8
Results of different models by integrate learning.
Integrate modelsBase model typeACC@1ACC@3MRRNDCGSingle model—47.9469.8561.6270.16Negative random samplingRandom initialization (6)49.1271.5063.0171.43Checkpoint (6)50.7772.2063.9872.11Training set (6)54.1876.2167.4274.94Mean value51.3673.3064.8072.83MGTLexical similarity (6)55.6377.4768.3075.37Semantic similarity (6)56.3978.3368.9176.20Mean value56.0177.9068.6175.79DDEA (our)Lexical similarity (3) + semantic similarity (3)57.5879.6569.5776.78


## 4.4. Discussion

In this paper, the QA model DDEA starts from the relation between entities to mine the extension words that have higher relevance with the key medical entities of the questions, and then trains base models based on negative samples generated by the segmented sampling of multiple similarities to promote the precision of the model. The classification of query intention can help the model specifically extract the key medical entities in questions. The threshold of confidence has a significant influence on the query intention classifier accuracy. From Figure3, the classifier accuracy is up to the highest when it is set to 0.8.Figure 3
Accuracy of the intention classifier under different confidence levels.According to the characteristics of the Chinese medical QA dataset, the words are divided into common words and domain words in the aspect of lexical similarity and the importance of domain words is emphasized by weight, which will directly affect the performance of the results of lexical similarity. Therefore, we evaluate the integrated model of BIGRU-CNN trained by six negative samples generated by segmented sampling based on the lexical similarity with different weights, and the ACC@1 is highest when the weight is 0.6, which is shown in Figure4. In addition, the number of base models and the learning granularity of the base model are directly determined by the number of segments in the process of negative sample generation. Figure 4 explains the ACC@1 results for various numbers of segments, and the optimal number of segments is three. In addition, it shows a drastical decline when the number of segments goes down, making it hard for the model to take advantage of multi-granularity learning. The degree of discrimination between the base models is reduced when the number of segments is four or five, which leads to the reduction of ACC@1 and the calculation time is longer. Therefore, the number of segments is set to three, i.e., the three base models for each similarity level, a total of six base models.Figure 4
ACC@1 according to the value of different parameters.Table9 provides several examples of QA models for a disease diagnosis question to show the highest score answers returned by these models, including original query, query expansion based on thesaurus (QE-T), query expansion based on knowledge graph (QE-KG), integrated model based on semantic similarity with QE-KG (QE-KG + MGT), and our model DDEA for illustrating the advantages of DDEA more intuitively. BIGRU-CNN with the best performance is selected as the model structure. The answers in Table 9 show that the query expansion model QE-KG has a higher matching degree between the extension words and correct answers. In contrast, the query expansion model QE-T causes query deviation due to the introduction of medical entities with low relevance with the correct answers. Through the comparison of answers, QE-KG outperforms the QE-T because QE-KG can usefully shorten the gap of semantic among questions and answers by matching potential medical entities in answers for the original query to obtain the correct disease entity of “umbilical hernia,” but there are still differences between the returned answers and the correct answers. The integrated model DDEA successfully captures the key information, either the correct diseases or “protruding navel, vomiting” in the question, and DDEA is based on the negative samples generated by segmented sampling with multi-similarity, which makes the model learn the language representation in multi-granularity from subtle to abstract but also ensure it to learn the language representation in multi-level that increases the precision of the model prediction.Table 9
Summary of different works pertaining to face and speech fusion.
ModelsQuestionReturned answersCorrect answersOriginal queryBaby is born more than 40 days: has protruding navel, reducible tumor, occasionally vomiting, and abdominal pain, how is this going on.Baby abdominal distension, acid reflux, abdominal pain sometimes accompanied by vomiting, fever, consider caused by gastrointestinal dysbacteriosis, and imperfect digestive system is also an important factor.If the child’s navel is raised, consider that it is related to mild umbilical hernia. You can observe whether the child’s abdomen has a mass raised from the body surface. If the mass cannot disappear after lying on his back, and the child has vomiting, abdominal distension, and other symptoms, you should see a doctor immediately.QE-TBaby is born more than 40 days: has protruding navel, reducible tumor, occasionally vomiting, and abdominal pain, how is this going on.cysts, nauseaBrain cyst is a swelling formed by cerebrospinal fluid surrounded in the pia mater. If not treated in time, with the increase of the cyst, it can cause headache, dizziness, nausea and vomiting, and other symptoms.QE-KGBaby is born more than 40 days: has protruding navel, reducible tumor, occasionally vomiting, and abdominal pain, how is this going on.umbilical hernia, infantile umbilical herniaUmbilical hernia includes congenital type, adult type, and infant type. Infantile umbilical hernia is characterized by raised walnut size tumor in the umbilicus. In crying and after exercise, the tumor becomes larger and feels more solid.QE-KG + MGTHernias include femoral hernia, indirect inguinal hernia and umbilical hernia. When the baby is crying or defecating, infantile umbilical hernia can press the raised abdominal contents back to the umbilical part. If the bulge progressive increase, it can be pressed by the abdominal belt.DDEAIf the child’s navel is raised, consider that it is related to mild umbilical hernia. You can observe whether the child’s abdomen has a mass raised from the body surface. If the mass cannot disappear after lying on his back, the child has vomiting, abdominal distension and other symptoms, you should see a doctor immediately


## 5. Conclusions

In this paper, we propose a new model termed DDEA, which effectively avoids weak relevance between the query extension words and the query intentions through the relation of entities from different medical scenarios in the question dimension. Meanwhile, the interference to mutual information value from the negative medical entities is reduced by recognizing these entities in the stage of extension word screening, thus improving the accuracy of query expansion. In the answer dimension, DDEA adopts a negative sample generation strategy of segmentation sampling based on the multi-level similarity to integrate the base models trained by these sample sets so that the model focuses on the relevance of entities at different levels, which improves the accuracy and stability of the QA model. The results show that DDEA outperforms the MGT integration method, and DDEA is able to capture multi-granularity knowledge information.In the future, we will employ a more complex integration method for training a more accurate and intelligent QA model, and make the model automatically generate answers in the case of low matching for questions and answers by the deep learning methods.



---

*Source: 7229389-2022-09-09.xml*