DEIM Forum 2019 B1-3

Application of Rough Set as Knowledge in Transfer Learningfor Medical Corpus Analysis
Huaze Xie, Yuanyuan Wang, Mohd Anuaruddin Bin Ahmadon, Shingo Yamaguchi
Graduate School of Sciences and Technology for Innovation,
Yamaguchi University,
2-16-1 Tokiwadai, Ube, Yamaguchi, 755-8611, Japan
E-mail: {g501vg, y.wang, anuar, shingo}@yamaguchi-u.ac.jp
Abstract Supervised transfer learning working on natural language processing, we propose a new rough set embedding
model. For a query about a given context paragraph from medical documents semantic analysis, requires modeling complex
interactions between the context and the query. In our previous work, we proposed a rough set preprocessing work in diabetes
medical records. For the standardized diabetes corpus analysis, the RNN with rough set works better. And we expect thismodel to work on transfer learning. In this paper, we introduce the Rough Set Attention Flow Network, we use the rough set as
the knowledge work in an unstandardized diabetes corpus semantic analysis.
Keyword Transfer learning, Rough set, Semantic analysis, Medical document.
simple questions or extracting answe rs from relevant

1. INTRODUCTION
The recurrent neural network (RNN) and transfer
learning have gained significant popularity over the past
few years within the natural language processing andliterature. The AS system is improved for medical corpus
vector generation.
In this paper, we introduce the Rough Set Attention

documents semantic analysis. In many different tasks andFlow Network (RS-AFN), a hierarchicalimage domains, the end-to-end training system can getarchitecture for modeling the representations of themore promising results. For the application of NLP incontext paragraph at different levels of granularity (Figure

medical, due to the popularity of electronic medical1). For the neural machine translation by jointly learning

records and the high reliability of medical literatur e,

to align and translate [5], we improved the retrieval

mechanical learning, deep learning and transfer learning

efficiency by rough set preprocessing. Initially, we only

in medically are more useful. This is because most of the

expected to separate the overly close word relationshipmedical literature is based on the validated medical theory.

through the rough set. Then we found that rough set canOver time, a complete system will be formed. Therefore,

absorb the results after each training and transform it into

we expect to propose a more suitable natural language

the target vocabulary before the next training. So we makeanalysis model for medical documents.

use of the advantage of the rough set to transform it in to

In our previous work, we proposed the semantic
analysis model with rough set preprocessing [1]. And forthis model, we expect to apply the rough set as themulti-stage

knowledge that can be transferred between feature -mapped
corpora in transfer learning.
Our rough set attention mechanism offers following

knowledge work in an attention transfer learning flow, so

improvements

we proved the effectiveness of rough set as a supervised

paradigms. First, our attention layer is inherited from

attention model retrieval ability [2]. In order to make our

BI-DAF model, attention layer is not used to summarize

model convenient for practical application. We designed

the context paragraph into a fixed -size vector. Instead, thesome tests in the medical corpus and improved our model

attention is computed for every time step, and the attended

to make it more suitable for machine comprehension (MC).

vector at each time step, along with the representations

This MC model is improved from the Au tomatic Scoringfrom previous layers, is allowed to flow through to the

(AS)

the

subsequent modeling layer. This reduces the information

BI-Directional Attention Flow (BI-DAF) model (M. Seo etloss caused by early summarization [4]. Second, when theal. 2017) [4]. And this MC test makes our medicalrough set as the knowledge transforms in the unlabeledsemantic analysis model more suitable for answering

corpora. We designed a selective absorption algorithm toSystem

(Y.

Wang

et

al.

2015) [3]

and

to

the

previously

popularattention

Figure 1: Rough Set Attention Flow Network Modelassist the attention layer, it does not mean we abandoned

2. PRELIMINARY

memory-less attention mechanism, the ordinary attention

A. Rough set

at each time step is a function of only the query and theRough set theory is a mathematical method tocontext paragraph at the current time step and does notdescribe the incompleteness and uncertainty of data. It

directly depend on the attention at the previous time step.

can effectively analysis and dispose of inaccurate and

We adopt the absorbability of rough sets to absorb

incomplete data, finds hidden information from theknowledge from the current attention model and transmit

rough set.

it to an automatic scoring system, this process does not

Rough set advantages:

affecta) It can dispose of all kinds of data, including

thenormalmemory-lessattentionmechanismworking. Third, in the RNN -LSTM training, we continue

incomplete data and many variables.

to

b)

use

attentionmechanisms

in

both

directions,

It

can

disposeofdeterministic

query-to-context and context-to-query, which provide

non-deterministic data.

complementary information to each other.

c) The minimum expression of knowledge and the
granularity of knowledge can be obtained.

andRoughsets

attribute

reductionistoeliminate

redundant attributes or redundant features and play arole of dimensionality reduction.
B.

Transfer learning
Compared with the traditional training model of data
mining, transfer learning is a learning model based ona training set. In actual data testing, the data
distribution of the training set and test set has big
differences. This is also the main reason that eventhough the training model effect is worki ng better, but

Figure 2: Rough set in transfer

the testing effect maybe have too many problems.

the words. These first three layers are applied to bothThis is not caused by the over-fitting of the model, thethe query and context.

inconsistency between the training set and the test set
is the main cause.

4) Attention Flow Layer: produces a set of query
aware feature vectors for each word in the context.

In our testing of open source medical data on diabetes,

5) Rough Set Transform Layer: the strong correlation

our task requires training about 50,000 medical

is extracted from the mapping set of the target word

records. But in fact, our training set collection only

relationship from the pre-rough set.

comes from about 800 medical documents on diabetes.
So we use transfer learning to make our method adapt

6) Knowledge Layer: rearrangement of existing roughset target word relations.

to the test set.
Different from the general transfer learning model, we7) Modeling Layer: an RNN-LSTM training to scan
the context [1].

decided to start transfer learning after modeling. It

8) Output Layer: the answers in the query are sortedmeans if the test result is not very bad, we expect toout and the knowledge learned in this training is added

improve the accuracy of the model, so we will use theto the rough set.

test set with its predicted results as training samples.
It is like there is only one correct answer, but the

4. ROUGH SET IN TRANSFER
In our transfer learning training model shows in Figure

wrong answer is different from each other.

2, we proposed the rough set training method to collect

The correct answers given in the test set willand disseminate target words in medical documents [1].

accumulate because they are the results of the correct

And the new target words also can be added into rough set

or the same model, but the wrong answers have[1].

different error patterns. If the parameters of the model

The rough set attention training method task is given by

are limited and not over-fitting, the model willrough set target words at first (see Figure 3. 1) ). The

eliminate these wrong patterns and tend to the right

input is target words and outputs a sequence, this is like aanswers.

very simple question answering s ystem by RNN, we can

It is similar to Figure 1, the basic word embedding is

get the output by (1).

training from the test corpus and the semantic is
obtained by training Word2Vec with training corpus
and test corpus.

3. METHOD
In our machine comprehension model (Figure 1), we
make the processing work in a hierarchical mu lti-stage andit consists of eight layers:
1) Character Embedding Layer: maps each word tovector space by word2vec
2) Word Embedding Layer: maps rough set targetwords to another vector space.

h1  f (U T  Wh 0  b), y1  Softmax (Vh1  c) (1)Of cause we also need to find out the most relevant
words in the output keywords and add them to the roughset (see Figure 3. 2) ). We get documents vectors fromtarget words by (2).

h1  f (U x1  Wh 0  b), T   (h1 , h2 , h3 , h4 )(2)

For training tasks with excessive length, we usually use
Encoder-Decoder structure to solve this problem (seeFigure 3. 3) ). At first Encoder-Decoder structure encodes

3) Contextual Embedding Layer: utilizes contextual

the input data into a context vector c. After getting c,cues from surrounding words to refine the embedding of

decode it with another RNN network. This part of RNN

Figure 3: Rough set attention training strategy

network is called Decoder. This is done by using c as input

automaticallyfor each step in Decoder. Because this Encoder -Decoder

information to y being output. Specifically, we use a i j tostructure does not limit the sequence length of input andmeasure the correlation between the h j of phase j inoutput, we will detect the context containing keywords

Encoder and the phase i in decoding. Finally, the input

again to find the hidden attributes of the target words. And

context information c i of phase i in Decoder comes from

we also use Encoder-Decoder to encode and decode thethe weighting of a ij by all h i .

questions in the input medical literat ure and medical
records to get the answers.

selects

themostappropriate

context

In the attention mechanism, we can easily apply the it to
the semantic learning of medical literature and the medical

In the Encoder-Decoder structure, Encoder encodes allrecords question answering system. However, due to the

input sequences into a unified semantic feature c , andkeywords strong relationship in the medical literature,then decodes them. Therefore, c must contain all theattention mechanism is difficult to separate the keywords,

information in the original sequence, and its lengthso, it will lead to the difficulty in obtaining the relevance

becomes the bottleneck restricting the performance of thefrom the target words in the medical paper. So we propose

model. Attention mechanism solves this problem by

an attention mechanism with rough sets in order toentering different c at each time. We introduce Decoder

improve the performance of the automatic scoring systemwith attention mechanism (see Figure 3. 4) ). Each c(see Figure 3. 5) ).

5. RESULTWe trained the rough set attention flow network models

In the ordinary training methods, there are manyin Keras. And we use about 300 labeled medical papers todefinitions of professional nouns in medical theory andtrain our method, we also have 500. All the medical papers

training results of diversified information, the ordinary

are about diabetes, in order to make the training set andtraining method usually cannot give reasonable answers or

the test set are in the same feature space.
Question: How to increase satiety of diabetic patients?
Answer: incretin hormone glucagon-like peptide 1, GLP1,
insulin release.
Context: The incretin hormone glucagon-like peptide 1(GLP1) potentiates insulin release and suppresses
glucagon secretion in response to the ingestion ofnutrients. GLP1 also delays gastric emptying andincreases satiety.

objective information.

Form 1: Examples from RS-AFN training model to obtain the

So there are three kinds ofexamples given to observe our RS-AFN model training
results.
The example test documents are the medical paper fromNature Reviews, here we use th e paper: “Glucagon-like
peptide 1 in health and disease”, and “Familial forms of
diabetes insipidus: clinical and molecular characteristics ”
to talk our RS-AFN model test results.

correct answer span .

From the form 1, we can see the RS-AFN model can

Question: What is the effect of GLP1?
Answer: normalize
Context: In patients with type 2 diabetes mellitus(T2DM), supraphysiological doses of GLP1 normalize
the endogenous insulin response during a hyperglycaemic
clamp.
Question: How the GLP1 treatment the T2DM?
Answer: receptor agonists
Context: Owing to the short plasma half-life of native
GLP1, several GLP1 receptor agonists (GLP1RAs) withlonger half-lives have been developed for the treatmentof T2DM.
Question: What are the genetic diseases caused bydiabetes mellitus?
Answer: diabetes insipidus
Context: The genetic and molecular basis of familial
forms of diabetes insipidus has been elucidated. Sevendifferent familial forms of diabetes insipidus are known
to exist. The clinical presentation, genetic basis andcellular mechanisms responsible for them varyconsiderably.

give the correct answer span. It means our rough set obtain

Form 2: Examples from RS-AFN training model to observe thecausalit y.

Question: What is diabetes insipidus?
Answer: diabetes insipidus, diluted urine, polyuria
Context: Diabetes insipidus is a clinical syndrome
characterized by the excretion of abnormally large
volumes of diluted urine (polyuria) and increased fluid
intake (polydipsia).
Question: What is neurohypophyseal diabetes insipidus ?
Answer: neurohypophyseal diabetes insipidus
Context: The most common type of diabetes insipidus iscaused by lack of the antidiuretic hormone arginine
vasopressin (vasopressin), which is produced in the
hypothalamus and secreted by the neurohypophysis. This
type of diabetes insipidus is referred to here asneurohypophyseal diabetes insipidus .
Question: What is nephrogenic diabetes insipidus?
Answer: nephrogenic diabetes insipidus
Context: The syndrome can also result from resistance tothe antidiuretic effects of vasopressin on the kidney, either
at the level of the vasopressin 2 receptor or the aquaporin
2 water channel (which mediates the re-absorption ofwater from urine), and is referred to as renal or
nephrogenic diabetes insipidus.
Form 3: Examples from the result comparison of similar problems .

the correct target words from paper title and abstract.
From the from 2, for the causality from words
relationship, our RS-AFN model can successfully grasp
the relationship between target words in the rough set.
From the from 3, the similar problems are given in orderto observe the same target word or the target word in theothertarget

words

like

“diabetes

insipidus”

and“neurohypophyseal diabetes insipidus ”, we can see thegiven answers are from the different paragraph, instead of
the same paragraph shows many times.
The results of our RS-AFN training model and the targetwords with keywords competing for approaches on the
medical documents, the visualization by dimensionality
reduction of test set are showed in Figure 4.
In Figure 4, the star shows the target word in rough set,and each point is a keyword trained by target word.
We find that by increasing the number of target words inrough set, we can get more keyword output.
And the incompleteness of rough sets can allow strong
relational target words to be added.

Figure 4: Different initial rough sets in the transfer

6. CONCLUSION
In this paper, we introduce RS -AFN a multi-stage
hierarchical process that represents the context at different
levels of granularity and uses a rough set attention flowmechanismtoachieveatargetwords

context

representation. The experimental evaluations show that
our model achieves the output of keywords in attention
model is controlled by rough sets with target words .
The visualizations and results show that our model islearning the test set containing medical vocabulary
representation for MC and is capable of answering
complex questions by attending to correct locations in thegiven target words. And the target words lead the
keywords

toconstitutetherelevanceof

medicalvocabulary. This relevance will be used to the electronic
medical record. In our future work, we expect to combine
our learning model with the actual medical record s,
accordingly, we can give some advice for the records.

References[1] H. Xie, M.A.B. Ahmadon, and S. Yamaguchi.
“Evaluation of Rough Sets Data Preprocessing onContext-Driven Semantic Analysis with RNN, ”
Global Conference on Consumer Electronics (GCCE),
pp.379-382, 2018.
[2] H. Xie, M.A.B. Ahmadon, S. Yamaguchi, and I.
Toyoshima. “Random Sampling and Inductive Ability
Evaluation of Word Embedding in Medical
Literature,” International Conference on Consumer
Electronics (ICCE), 2019.
[3] Y. Wang, Y. Kawai, S. Miyamoto and K. Sumiya. “AnAutomatic Scoring System for E-Reports based onStudent Peer Evaluation using Groupware, ” The
Database Society of Japan Journal, pp.71-76, 2015.
[4] M. Seo, A. Kembhavi, A. Farhadi and H. Hajishirzi.
“Bidirectional
Attention
Flow
forMachineComprehension,”International
Conferenceon
Learning Representations (ICLR), 2017.
[5] C. Clark and M. Gardner. “Simple and Effective
Multi-Paragraph Reading Comprehension, ” Annual
Meeting of the Association for Computational
Linguistics, pp.845-855, 2018.
[6] R. T. Sousa, L. A. Pereira and A. S. Soares.
“Predicting Diabetes Disease Evolution Using
Financial Records and Recurrent Neural Networks, ”
Machine Learning for Health, 2018.
[7] J. Howard and S. Ruder. “Universal Language Model
Fine-tuning for Text Classification ,” Association for
Computational Linguistics, 2018.
[8] R. Volpi, H. Namkoong, O. Sener, J. Duchi and V.
Murino, Silvio Savarese “Generalizing to Unseen
Domains via Adversarial Data Augmentation,”
Neural Information Processing Systems, 2018.
[9] D. Xiao, G. Wang, F. Hu. “Fast Parallel Attribute
Reduction Algorithm Based on Rough Set Theory ”
Computer Science, 2009.

[10] M. J. Kusner , Y. Sun, N. I. Kolkin, K. Q. Weinberger.
“From word embeddings to document distances, ”
International Conference on International Conference
on Machine Learning, 2015.
[11] A.B. Dieng, C. Wang , J. Gao , J. Paisley. “TopicRNN:
A Recurrent Neural Network with Long-Range
Semantic Dependency” International Conference on
Learning Representations, 2017.
[12] T. Mikolov, W. Yih, and G. Zweig. “Linguistic
Regularities
inContinuous
Space
Word
Representations”.North American Chapter of the
Association for Computational Linguistics: Human
Language Technologies, pp.746 -751, 2013.
[13] T. Mikolov, K. Chen, G. Corrado, and J. Dean.
“Efficient Estimation of Word Represen tations in
Vector Space” International Conference on Learning
Representations, 2013.
[14] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J.
Dean. “Distributed Representations of Words andPhrasesandtheirCompositionality”
NeuralInformation Processing Systems, pp.3111-3119,
2013.
[15] Z.
Pawlak.
“Ai
andIntelligent
Industrial
Applications:
the
Rough
Set
Perspective ”
Cybernetics and Systems , pp.227 -252, 2000.