Discussion Paper No. 79
In-hospital Mortality Prediction for Trauma
Patients Using Cost-sensitive MedLDA
Haruya Ishizuka, Tsukasa Ishigaki,
Naoya Kobayashi, Daisuke Kudo,
and Atsuhiro Nakagawa

March , 2018

Data Science and Service Research
Discussion Paper

Center for Data Science and Service Research
Graduate School of Economic and Management
Tohoku University27-1 Kawauchi, Aobaku
Sendai 980-8576, JAPANIn-hospital Mortality Prediction for Trauma
Patients Using Cost-sensitive MedLDA
Haruya Ishizuka1 , Tsukasa Ishigaki2 , Naoya Kobayashi3 , Daisuke Kudo3 , andAtsuhiro Nakagawa3
1

Graduate School of Economics and Management, Tohoku University, Miyagi,
JAPAN isizuka19931114@gmail.com⋆⋆2
Graduate School of Economics and Management, Tohoku University, Miyagi,
JAPAN
3
Tohoku University School of Medicine, Miyagi, JAPAN

Abstract. In intensive care units (ICUs), mortality prediction using
vital sign or demographics of patients yields helpful information to support the decision-making of intensivists. Clinical texts recorded by medical staﬀ tend to be valuable for prediction. However, text data are notapplicable to outcome prediction of the regression framework in a direct
way. In addition, learning of prediction models of such outcomes is a classof imbalanced data problem because the number of survivors is greaterthan the number of dead patients in most ICUs. To address these difficulties, we present Cost-Sensitive MedLDA: a supervised topic model
employing cost-sensitive learning. The model realizes a prediction modelfrom heterogeneous data such as vital signs, demographic information,and clinical text in an imbalanced class problem. Through experimentation and discussion, we demonstrate that the model has two benefits
for use in medical fields: 1) our model has high prediction performancefor minority instances while maintaining good performance for majority
instances even if the training set is imbalanced data; 2) our model canreveal some characteristics that are associated with bad outcomes fromthe use of clinical texts.

1

Introduction

Intensive Care Units (ICUs) are closely associated with patient mortality. Intensivists must make prompt and accurate decisions to provide adequate treatment. Nevertheless, doing so manually imposes heavy burdens on them becauseof constraints related to time and available personnel. To address this issue, clinical decision support systems (CDSSs) able to detect critically ill patients haveattracted attention. Accurate mortality prediction methods are fundamentally
important for realizing practical CDSSs. Severity scores that are calculated fromnumerical data recorded in structured relational databases, have been used toevaluate illness severity. For instance, Acute Physiology and Chronic Health II⋆⋆

The first author currently is an employee of a private company, however, the work
was done at Tohoku University

2

H. Ishizuka et al.

(APACHE II)[7] and Simplified Acute Physiology Score II (SAPS II)[14] (LeGall et al., 1993) are widely used at many hospitals.
In addition to numerical data, some researchers have recently developed mortality prediction methods using unstructured clinical text, and have reported its
results [22, 8, 3, 16]. However, their methods are based mainly on a two-stage
mortality prediction procedure. In this two-stage procedure, we first extract topics that represent underlying semantic themes of documents, from documents
using an unsupervised topic model, e.g. Latent Dirichlet Allocation (LDA)[1]. In
addition, we use the learned topic distribution as an input to a classifier. Someearlier works have demonstrated that supervised topic models (STMs), which
can extract more predictive topics using label information associated with documents, outperform two-stage procedures in terms of classification accuracy[2, 11].
It is preferred to use STM for more accurate outcome prediction, but two main
problems in earlier STMs arise when we use them for mortality prediction.
First, it is necessary to develop a cost-sensitive model. The imbalanced data
problem is a well-known diﬃculty that poses challenges to the development anduse of machine learning applications. Generally, in-hospital mortality rates in
ICUs are from 10% to 15%, representing one example of imbalanced data. Learning algorithms for a classifier typically assume that instances in each class are
distributed uniformly. When the training set comprises imbalanced data, a classifier rarely identifies minority instances. Consequently, the predictive accuracy
becomes poor. Cost-sensitive learning is one approach to mitigating this diﬃculty. In cost-sensitive learning, if instances that actually belong to a minority
class are misclassified to a majority class, a severer penalty is assigned to the objective function. However, no STM takes a cost-sensitive approach in its learning
stage.
Secondly, for ICU settings, it is helpful to develop an STM that integrates
textual information and numerical data. However, STMs have been developed
mainly in the fields of document classification and image categorization. Presumably, topic representations and their discriminant functions are trained only
from text or images. Electronic health records at an ICU include not only clinical text but also various numerical data. This information can be beneficial foroutcome prediction.
To address these issue, we propose Cost-Sensitive Maximum entropy discriminative LDA (CS-MedLDA). Maximum entropy discriminative LDA (MedLDA)(Zhu et al., 2012[24]) is a supervised topic model that integrates discriminative
max-margin learning with topic models. CS-MedLDA is its extension, which
adopts a cost-sensitive approach in the training phase. In addition, our model
can use numerical data as features of discriminant function, unlike conventional
STM. As described herein, we demonstrate that the method can detect minority
instances correctly and that it can discover characteristics that are associatedwith a minority group in an imbalanced dataset.
This paper is organized as follows. Sect.2 presents a summary of some related work. Sect.3 presents a description of the formulation and learning algorithm. Sect.4 explains the pre-experiment: one-versus-rest classification in 20

Mortality Prediction using cost-sensitive MedLDA

3

newsgroups dataset. Sect.5 presents experimentally obtained results in a trauma
patient dataset. Finally, Sect.6 presents the clinical significance, conclusions, andavenues of future work.

2

Related Work

2.1

Cost-sensitive learning

Many works have specifically examined cost-sensitive learning[10]. Especially,
the application of Support Vector Machine (SVM), a classifier based on themax-margin principle, has been investigated actively in this field because of
its rigorous mathematical foundation and predictive power. As a well-known
SVM application, cost-sensitive SVM (CS-SVM) [19], Boosting SVM[20], andOptimized cost-sensitive SVM[5] can be cited.
2.2

Supervised Topic Models

STMs, which are topic models that can learn predictive topic structures usinglabel information of documents, have been developed mainly for document classification and image categorization[2, 15, 24]. Specifically, MedLDA has shown
great promise for document classification, image categorization and link prediction in social networks[17, 21, 4]. In addition, some authors have developed
sophisticated learning algorithms to reduce the training time of MedLDA[12, 25].
However, STMs are generally assumed as models that are trained with asingle data type. In addition, no supervised topic model exists which uses costsensitive learning to manage an imbalanced data problem.
2.3

Mortality Prediction using LDA and Its Extension

Outcome prediction methods using textual information have been developed
recently because of the advancement of Natural Language Processing techniques
and text mining. Most of them use LDA and its extension to extract latent topics
from clinical text[16, 8, 3, 22]. All of them reported the eﬀectiveness of using
clinical text and LDA variants.
However, these methods have used 2-stage procedure that learns topic representations and classifiers separately. Halpern et al. (2012) claim that STM hasa higher prediction performance than two-stage procedures when the dimensionof latent topic dimensions are low. Zhang et al.,2017[23] proposes a survival
topic model which is an STM to estimate the hazard function of mortality. It
is an example of using STM for mortality prediction. However, few works have
explained the use of STM for mortality prediction. No report of the relevant literature describes the use of MedLDA to predict outcomes in spite of its superior
classification ability.

43

H. Ishizuka et al.

Cost-sensitive MedLDAThis section presents the formulation and learning algorithm of CS-MedLDA.
First, we introduce LDA, Regularized Bayesian Inference, and MedLDA as preliminaries. Next, we explain details of CS-MedLDA. In the following section,D
we respectively denote W = {wd }D
d=1 and y = {yd }d=1 as the training documents and documents label. Here, yd ∈ Y = {−1, 1}. KL[q∥p] stands for the
Kullback–Leibler divergence. Also, Eq [·] represents the expectation.
3.1

Preliminaries

LDA LDA is a hierarchical Bayesian model that posits each document as an
admixture of K topics, where each topic Φk is a multinomial distributions overV vocabularies. For document d, the generating process can be described as presented below.
1. Draw a topic proportion θ d ∼ Dir(α).
2. For each word n (0 ≤ n ≤ Nd )(a) draw a topic assignment zdn ∼ Mult(θ d ).
(b) draw the observed word wdn ∼ Mult(Φzdn ).

Therein, Dir(·) represents a Dirichlet distribution, Mult(·) denotes a multinomial, and Φzdn expresses the topic selected by the topic assignment zdn . For
Bayesian LDA, the topics are random samples drawn from a Dirichlet prior∀k, ϕk ∼Dir(β).
dGiven a training set of documents W, we let zd = {zdn }Nn=1 denote the set
Dof topic assignments for document d and let Z = {zd }d=1 and Θ = {θ d }D
d=1
respectively denote all the topic assignments and mixing proportions for thewhole corpus. The goal of LDA is to infer the posterior distribution as
p0 (Θ, Φ, Z)p(W|Z, Φ)
,
(1)
p(W)
∏
∏
∏ ∏
∏ ∏
where p0 (Θ, Φ, Z) = k p(Φk |β) d p(θ d |α) d n p(zdn |θ d ),p(W|Z, Φ) = d n p(wdn |zdn , Φ)according to the generation process.
p(Θ, Φ, Z|W) =Regularized Bayesian Inference Jiang et al. (2012) show that (1) inferredfrom the Bayes rule is equivalent to the solution of the optimization problem on
probability distribution below q(Z, Θ, Φ) as
minq(Z,Θ,Φ)∈P

KL[q(Z, Θ, Φ)∥p0 (Z, Θ, Φ)] − Eq [log p(W|Z, Φ)],where P signifies the space of probability distributions. Regularized Bayesian Inference (RegBayes) is the framework which regards posterior inference by Bayes
rule as optimization problem including objective function and constraint on
probability distribution. One benefit of this framework is that it can be extended
naturally to include some regularization terms on q.

Mortality Prediction using cost-sensitive MedLDA

5

MedLDA As a supervised topic model, MedLDA learns topic representations
and max-margin classifiers jointly. Training is conducted by solving the RegBayes
problem below asminq(η,Z,Θ,Φ)∈P

L(q(η, Z, Θ, Φ)) − 2CR(q(η, Z, Θ, Φ))subject to ∀d, 1 − yd Eq [η T z¯d ] − ξd ≤ 0, ξd ≥ 0(2)where η represents the coeﬃcient vector of classifiers L(q) is KL[q(η, Z, Θ, Φ)∥p(η, Z, Θ, Φ|W)];∑D
R(q) = d=1 Eq [max(0, 1 − yd η T z̄d )] denotes the expected hinge loss, and C
signifies the penalty for misclassification. Also, z̄ is the average topic assignment
1 ∑N
vector with each element being z̄k =I(zn = k). The label prediction
N n=1
rule is given as the signature function below:
sign(f (η T z̄)).
3.2

Cost-sensitive MedLDA

Formulation CS-MedLDA is a MedLDA variant that adopts cost-sensitive
learning to learn topic representations and classifiers. In addition, this model
can use not only documents but also numerical data, unlike previous supervised
topic models. We define sd , sd ∈ RP as numerical data. Let xd = {z̄d , sd }D
d=1
and η = {η 1 , η 2 }, η 1 ∈ RK , η 2 ∈ RP respectively denote the feature vector
and coeﬃcient vector. Then, the training of CS-MedLDA is formulated with the
optimization problem below asminq(η,Z,Θ,Φ)∈P

L(q(η, Z, Θ, Φ)) − 2C + R+ (q(η, Z, Θ, Φ))− 2C − R− (q(η, Z, Θ, Φ))(3)subject to ∀d, 1 − yd Eq [η T xd ] − ξd ≤ 0, ξd ≥ 0
where
R+ (q) =∑Eq [max(0, 1 − yd η T xd )];d:{d∈yd =1}−

R (q) =∑

Eq [max(0, 1 − yd η T xd )].

d:{d∈yd =−1}

The objective function is penalized with C + if a minority instance is misclassified
to a majority group. One can mitigate the imbalanced data problem by adjusting
C − and C + . Here, CS-MedLDA is equivalent to MedLDA if C + = C − andsd = 0.
Learning Algorithm We use data augmentation and Gibbs sampling[25] asa learning algorithm of CS-MedLDA. Furthermore, we designate y+ , W+ and6

H. Ishizuka et al.

X+ = {Z+ , S+ } as the set of variables associated with a minority class, letting y− , W− and X− = {Z− , S− } b variables belonging to a majority class.
Introducing the unnormalized pseudo-likelihood as∏ϕ(y+ |X+ , C + , η) =exp(−2C + max{0, ξd }),d∈{d:yd =1}
−

−

∏

−

ϕ(y |X , C , η) =exp(−2C − max{0, ξd }).

d∈{d:yd =−1}

Then, the optimization problem (3) is rewritten as shown below:
L(q(η, Z, Θ, Φ)) − Eq [log ϕ(y+ |X+ , C + , η)].
− Eq [log ϕ(y− |X− , C − , η)]minq(η,Z,Θ,Φ)∈P(4)Using the ideas of data augmentation[18], the unnormalized pseudo-likelihood
for a minority group can be expressed as∫ ∞∏ϕ(y+ |X+ , C + , η) =ϕ(yd , λd |xd , C + , η)dλd
d∈{d:yd =1}=∏d∈{d:yd =1}

0

∫ ∞
0

(
)
1
(λd + C + ξd )2
√
exp −
dλd
2λd
2πλd

(5)

Then, if yd = 1,()
(λd + C + ξd )2
1
exp −ϕ(yd , λd |xd , C , η) = √
.
2λd
2πλd
+

In the case of yd = −1, we obtain ϕ(yd , λd |xd , C − , η) by changing C + to C − .
Using ϕ(y+ , λ+ |X+ , C + , η) and ϕ(y− , λ− |X− , C − , η), we can rewrite (3) as a
new optimization problem including augmented variable λ as shown below.
minq(η,Z,Θ,Φ)L(q(η, Z, Θ, Φ) − Eq [log ϕ(y+ , λ+ |X+ , C + , η)]− Eq [log ϕ(y− , λ− |X− , C − , η)](6)Solving this optimization problem, we obtain the complete posterior asq(η, λ, Z, Θ, Φ) =p0 (η, Z, Θ, Φ)p(W|Z, Φ)ϕ(y+ , λ+ |X+ , C + , η)ϕ(y− , λ− |X− , C − , η),ψ(y,W)(7)

where ψ(y,W) is the normalized constant. In addition, we employ collapsed
Gibbs sampling[9] by setting prior p0 (η, Z, Θ, Φ) = p(η)p0 (Z, Θ, Φ) and integrating out Θ, Φ from q(η, λ, Z, Θ, Φ).
Letting q ∗ (η|rest), q ∗ (zdn = k|rest) and q ∗ (λ−1d |rest), and assuming the fullconditional distribution of η, zdn and λ−1,
theseprobability distributions aredMortality Prediction using cost-sensitive MedLDA

7

Algorithm 1 Gibbs sampling for CS-MedLDA.
Initialization: set λ = 1 and randomly draw zdn from a uniform distribution
for m = 1 to M dodraw the classifier η from the normal distribution N (µ, Σ)
for d = 1 to D dofor each word n in document d dodraw a topic from the multinomial distribution M (C + ) or M (C − )end for+draw λ−1d (and thus λd ) from the inverse Gaussian distribution IG(1/C |ξd |, 1)−
or IG(1/C |ξd |, 1)
end forend forgiven in the form presented below. Derivation details are presented in Appendix
1.
q ∗ (η|rest) = N (µ, Σ)
)−1
(
∑
∑
xd xT
IK+P
xd xT
+ 2
− 2
d
d
+ (C )
Σ=+ (C )
ν2λdλdd∈{d:yd =1}d∈{d:yd =−1}()∑∑
yd (λd + C + )yd (λd + C − )µ = Σ C+xd + C −xdλdλdd∈{d:yd =−1}d∈{d:yd =−1}(8)tk(Ck.−n+ βt )(Cd.−n+ αk )∑V∑V
t
t=1 Ck.−n +t=1 βt[ 2γη1k {λd yd + C + (yd − Eq [η 2 ]T sd )}
× exp
λd
]
2 2
k
+ 2 γ η1k + 2γ(1 − γ)η1k Λdn− (C )2λdq ∗ (zdn = k|rest) ∝(9)= M (C + )If yd = −1, we obtain M (C − ) with changing C + to C − . Table 1 shows thektdefinition of Cd.−n,Ck.−n,γ and Λkdn .
+q ∗ (λ−1d |rest) = IG(1/C |ξd |, 1)(10)

Here, IG(a, b) is the inverse Gaussian distribution. After sampling λ−1
d , we obtain λd by inversion. If yd = −1, then in the same fashion as zdn , we obtain
IG(1/C − |ξd |, 1) with changing C + to C − .
Algorithm 1 shows pseudo-code for the learning algorithm of CS-MedLDA.
In our research, we set λ = 1 initially and draw Z randomly from a uniform
distribution.

8

H. Ishizuka et al.
ktTable 1. Definition of Cd.−n,Ck.−n,γ and Λkdn

∑Nd
k
Cd.−n
I(z = k) − I(zdn = k)∑n=1 ∑NdntdCk.−n D
d=1
n=1 I(zdn = k)I(wdn = t) − I(zdn = k)I(wdn = t)
1
γ
Nd
∑K
1
k
k′
Λdn
ηk′ Cd.−n′
Nd − 1 k =1
Table 2. Distribution of document labels in our experimental settings
Condition
1
2
3
4
5

|D+ |
798
798
798
798
798

4

Pre-Experiment

4.1

Experimental Settings

|D− |
4,852
5,816
6,803
7,796
8,787

imbalance ratio
14.1%
12.1%
10.4%
9.1%
8.3%

Sect.4 presents the pre-experiment results: one-versus-rest classification in a 20
newsgroup dataset. This dataset includes about 20,000 articles within 20 newsgroup categories. If an article d is labeled as alt.athism, then yd = 1, else yd = −1.
In one-versus-rest classification, an imbalanced data problem occurs when numerous categories area used. Table 2 presents the distribution of labels in our
settings. Column |D+ | is the number of articles categorized into alt.atishm. Column |D− | is the number of articles with other labels. Column imbalance ratiois the ratio of alt.athism articles to the whole dataset. The training set contains
60% of total documents. The test set is the rest.
We compare the proposed model with four competitors: LDA+SVM, LDA+wSVM,
sLDA, and MedLDA. Here, LDA+SVM and LDA+wSVM use LDA to learn the
topic representations, and predict labels, respectively, with linear SVM and linear CS-SVM. sLDA is an STM proposed by Blei & McAliﬀe, 2007. sLDA and
MedLDA are trained with Gibbs sampling[26, 25]. For LDA+SVM, sLDA and
MedLDA, we set penalty parameter C = 1. For LDA+wSVM and CS-MedLDA,
we penalize their objective function with penalty C + = 2 when a minority instance is misclassified to a majority group. In the opposite case, penalizing with
C − = 1. The η’s prior p( η) for sLDA, MedLDA, and CS-MedLDA is the normal
distribution N (0, IK+P ). For all five models, we use the symmetric Dirichletprior α = 1/K and β = 1/V, V = 61, 118. The number of MCMC iterations M
is 100. The number of latent topics K is shifted from 10 to 60 in steps of 5.
To evaluate the predictive power, we use accuracy and the G-mean. The
G-meanis the geometric mean of the accuracy of each class, i.e., G-mean =√#T N #T P·, where the sizes of diﬀerent classes have already been consid|D− | |D+ |Mortality Prediction using cost-sensitive MedLDA

9(a) Accuracy(b) G-mean

Fig. 1. (a) Accuracy and (b) G-mean of each model when conducting one-versus-rest
classification with a 20 newsgroup dataset. The x-axis shows the number of topics. The
y-axis shows the value of indices. The imbalance ratio is shifted from 14.1% to 8.3%.

ered. Therefore, it is a good candidate for evaluating class-imbalance learning
performance.
4.2

ResultFigure 1 presents the accuracy and G-mean results obtained using diﬀerent methods. All models except CS-MedLDA show a low G-mean instead of high accuracy.
This result indicates that these models cannot discriminate minority instancesfrom majority instances because of the influence of an imbalanced data problem.
However, the proposed model shows much higher G-mean. Moreover, its accuracy almost equals that of the others. Therefore, CS-MedLDA can more correctly
identify alt.athism articles belonging to a minority group than competitors can.

5

Experimental Results in Trauma Patient Dataset

This section presents the results of in-hospital mortality prediction and topic
evaluation for trauma patients in the Medical Information Mart for the Intensive Care-III (MIMIC-III) database[13]. MIMIC-III includes numerical data andclinical text of patients who were admitted to the Beth Israel Deaconess Medical
Center during 2001–2012. We extract 1,546 trauma patients (in-hospital mortality rate: 12.4%) who were older than 16 years old and who had stayed in thisICU for 24 hr or longer.

10

H. Ishizuka et al.

(a) Accuracy(b) G-mean(c) Precision

Fig. 2. (a) Accuracy, (b) G-mean, and (c) Precision of each model when predicting
patient outcomes in a trauma patient dataset.

The material documents are nursing notes and radiology reports gathered
within the first 24 hr of the ICU stay. As preprocessing, we first eliminate punctuation, numbers, and clinical stopwords4 . Next, we extracted noun phrases using morphological analysis. Finally, we obtained 18,041 documents and 28,080
vocabularies from 1,546 trauma patients.
Additionally, we obtained 12 numerical data such as gender, heart rate, andhematocrit. All of these data are used to calculate SAPS II or APACHE II,
intensivists normally use in their work.
5.1

In-hospital Mortality Prediction

This experiment investigated the in-hospital mortality prediction accuracy ofthe proposed model. We compare the proposed model with five competitors:
Numeric, LDA+SVM, LDA+wSVM, sLDA, and MedLDA. The numeric model
predicts patient outcomes using linear SVM, but this model is based only on
numerical data without text. Other models are the same as those described in
Sect.4 except that models predict outcomes using both numerical and textual
information. Hyperparameters are also the same settings as those described in
Sect.4.
For evaluating predictive capability, we use precision in addition to accuracy#T P
and the G-mean. Precision is defined as precision =, indicating#T P + #F P
the percentage of false alerts.
Results Fig. 2 presents results of this experiment. MedLDA and LDA+wSVM
show the highest accuracy of the six models, although their G-means are much
lower than those of other competitors. These models cannot detect critically
4

https://github.com/kavgan/clinical-concepts/blob/master/clinical-stopwords.txt

Mortality Prediction using cost-sensitive MedLDA(a) MedLDA

11(b) CS-MedLDA

Fig. 3. (a) is the Monte-Carlo estimates of η 1 of original MedLDA, and (b) is CSMedLDA’s one when the number of topics K = 40.

ill patients by the influence of an imbalanced data problem. When a classifier
faces an imbalanced data problem, its precision becomes unstable because thedenominator is extremely small. Therefore, we did not calculate their precision.
Numeric and LDA+wSVM show higher G-mean than either LDA+SVM or
MedLDA. However, their accuracy and precision were lower than that of either sLDA or CS-MedLDA. This result indicates that these models classified
instances which actually belong to a majority group to a minority group in
surplus. Comparing CS-MedLDA with sLDA, the former shows higher values
than sLDA with all indices. Therefore, our proposed model can correctly predictpatient outcomes as compared to its competitors.
5.2

Topic Evaluation

This section presents the result of investigation into the topics extracted using
CS-MedLDA. First, we explore the diﬀerence of coeﬃcient vector η 1 between
standard MedLDA and our proposed model. Next, based on estimates of coefficients η1k and frequently occurring words in each topic, we strive to discover
topics that have a positive correlation with bad outcomes.
coeﬃcient η Analysis Fig. 3 presents the estimated values of coeﬃcient η 1 of
MedLDA and CS-MedLDA in the trauma patient dataset. We considered first
50 MCMC iterations as the burn-in period. The estimates of η 1 of the original
MedLDA tend to be negative values. Based on the results of two classification
experiments, MedLDA produces poor predictions because it is unable to capture topics that are positively correlated with minority instances. However, CSMedLDA can capture such topics by giving a larger penalty for misclassifying
minority instances. As a result, CS-MedLDA produces better predictions.
Qualitative Evaluation Table 3 presents the most frequently observed wordsof 10 topics extracted using CS-MedLDA. The estimated value of η1k corre-

12

H. Ishizuka et al.

sponding to each topic is positive. Topic 1 has the strongest positive correlationfrom the viewpoint of the value of η1k .
Hematoma, scan and subarachnoid hemorrhage (sah) appear as frequent words of topic 1. Therefore, this topic is the result of CT scan over subarachnoid hemorrhage patients. In addition, dislocation of a brain might occur
from the fact that dislocation appears frequently in this topic. Therefore, patients whose ratio of this topic is high might be aﬄicted by a severe subarachnoid
hemorrhage.
In topic 3, malignant disease spreads to the subclavian lymph node: lymph
and clavicle were observed frequently. In addition, patients were in postop(post-operation) agitation. Metastasis of malignant disease and post-operation
agitation are known as risk factors related to bad outcomes.
Topic 4 represents tracheal intubation for mechanical ventilation. Actually,
ett is an abbreviation of endotracheal tube. Generally, endotracheal tubes areused to secure a patient airway with consciousness disturbance or respiratory
failure. Because lobe also appeared among the most frequently observed words,
this topic represents respiratory care.
Both of these topics include factors related to deceased patients from pastfindings. From the above, CS-MedLDA can discover characteristics associatedwith patient mortality. However, some topics are diﬃcult to interpret. Improvement of interpretability is an important task for future work.

6

Clinical Significance and Conclusion

As described herein, we proposed a CS-MedLDA, a supervised topic model using cost-sensitive learning to ascertain the latent topic structure and max-marginclassifiers. Experimentation revealed that CS-MedLDA has two preferred properties. The proposed model detects minority instances more correctly than previous methods under an imbalanced data problem; it discovers some characteristics
associated with a minority group from documents. These properties bring two
benefits when applied in medical fields.
Table 3. Most frequently observed words in 10 topics with coeﬃcient (η ≥ 0).
Topic 1 hematoma,mm,sinus,optiray,dislocation,scan,npn,foci, opacities,car,sah
Topic 2 ct,year,number,radiology,head,spine,hemorrhage, status,resp,structures
Topic 3 process,checks,stool,lymph,tol,clavicle,multiplanar, reformats,need,postop ,agitation
Topic 4 lobe,line,namepattern,aorta,ett,opacification,site, endotracheal,elbow,ribs,vein
Topic 5 foley,sbp,side,pupils,gallbladder,eye,qhr, remains,mcgkgmin,diameter,tree
Topic 6 herniation,service,read,flow,extubation,scale,spaces, question,rule,sicu,ed
Topic 7 indication,blood,rib,lat,detail,foot,transfer,units, volumes,data,drain
Topic 8 wcontrast,atelectasis,bilat,ventricle, lymph,adenopathy,rt,arrival,breath, reduction,heights
Topic 9 urine,midline,bp,npo,nursing,collar,wean,pancreas, stenosis,displacement,tls
Topic 10 findings,lung,time,abd,neck,position,exam,leg, mdct,calcifications,hegiht

Mortality Prediction using cost-sensitive MedLDA

13

First, our model is expected to work well for other imbalanced data classification problems. In medical fields, various imbalance data are generated aside frompatient mortality. One example is disease recurrence. In addition, numerical dataand clinical text are recorded all over the hospital to monitor patients’ diseasestate. Using the proposed model, we can readily construct a prediction modelthat integrates numerical and textual information with a simple Gibbs samplingalgorithm. These two properties, i.e. detectability over minority instances andthe ease of integrating numerical and textual information, are helpful for otherimbalanced data classification.
Secondly, the proposed model might become a powerful method by which
we can explore the characteristics associated with a minority group. In conventional clinical studies, researchers have attempted to seek such characteristics
for preventing disease onset or finding care methods from numerical data. In
other words, researchers have often been interested in the characteristics of minority instances in their studies because patients with bad outcomes or thoseaﬄicted with some disease are rare examples. Through interpretation of topics
extracted using our model, we can uncover them not from numerical data but
from unstructured text. Therefore, CS-MedLDA might become another tool toelucidate minority group characteristics.
Future studies will be conducted to exploit dynamic supervised topic models
that learn topic representations and its dynamics from documents and fromsupervision. Patient conditions might change an instant after ICU admission.
Specifically, one’s health condition tends to be unstable in his/her acuity phase.
Use of this dynamic information of clinical data has presented many benefits foroutcome prediction. Dynamic supervised topic models might become an eﬀective
outcome prediction method based on dynamic information and supervision, anda text mining method that captures the time-variant evolution of illness severity
from clinical texts.

A

Derivation Detail of Conditional Distributions

By integrating out the Dirichlet variables (Θ, Φ) in the complete posterior distribution (7), we obtain the collapsed posterior distributionq ∗ (η, λ, Z) ∝ p(η)p(W, Z|α, β)ϕ(y+ , λ+ |X+ , C + , η)ϕ(y− , λ− |X− , C − , η)[∏] K
D
δ(Cd + α) ∏ δ(Ck + β)
= p(η)
δ(α)
δ(β)
k=1
d=1
(
)
∏
1
(λd + C + ξd )2
√
×
exp −
2λd
2πλd
d∈{yd =1}()∏
1
(λd + C − ξd )2
√
×
exp −
2λd
2πλd
d∈{yd =1}

(11)

14

H. Ishizuka et al.

∏dim(x)Γ (xi )i=1
,Γ (·) is the Gamma function. Also, Ck = {Ckt }Vt=1∑dim(x)Γ ( i=1 xi )is the set of word counts associated with topic k; Cd = {Cdk }K
k=1 is the number ofwhere δ(x) =times that terms are associated with topic k. The full conditional distributions
used in collapsed Gibbs sampling are the following.
For η Setting Gaussian prior N (0, ν 2 IK+P ) for η, the conditional distributionq ∗ (η|rest) is given by as the following Gaussian distribution.
( ∑
)
(
)
∑
−(λd + C + ξd )2−(λd + C + ξd )2q ∗ (η|rest) ∝ p(η) exp
exp
2λd
2λd
d∈{d:yd =1}d∈{d:yd =−1}

∝ N (µ, Σ)
(
)−1
∑
∑
IK+P
xd xT
xd xT
+ 2
− 2
d
d
Σ=
+
(C
)
+
(C
)
ν2
λd
λd
d∈{d:yd =−1}d∈{d:yd =1}()∑∑
yd (λd + C + l)yd (λd + C − l)
+
−
µ=Σ C
xd + Cxd
λd
λdd∈{d:yd =−1}d∈{d:yd =−1}

(12)Therefore, we can draw coeﬃcient vector η from q(η|rest) = N (µ, Σ)
For Z . The conditional distribution of Z given the other variable is the following.
()δ(Cd + α)(λd + C + ξd )2
exp −
δ(β)
δ(α)
2λd
k=1
d∈{yd =1}()∏
δ(Cd + α)(λd + C − ξd )2
×
exp −
δ(α)
2λd

q ∗ (Z) ∝K∏
δ(Ck + β)∏d∈{yd =−1}

By canceling common factors, we can derive the conditional distribution asq ∗ (zdn |rest).
For λd Finally, the conditional distribution of the augmented variable λ given
the rest variable is a generalized inverse Gaussian distribution (Devroye, 1986)as()
1(λd + C + ξd )2q ∗ (λd |rest) ∝ √
exp −
2λd
2πλd
(
)
1
(C + )2 ξd2
λd
∝√
exp −
−
2λd
2
2πλd
1
= GIG( , 1, (C + )2 ξd2 ),2Mortality Prediction using cost-sensitive MedLDA

15

(
))
(
1 b
where GIG(p, a, b) = C(p, a, b)xp−1 exp −+ ax
and C(p, a, b) is a
2 xnormalization constant. As q ∗ (λd |X, η) is the generalized inverse Gaussian dis∗ −1tribution, λ−1d follows an inverse Gaussian distribution. We denote q (λd |rest)−1∗ −1
as the conditional distribution of λd , and q (λd |rest) is the following+q(λ−1d |rest) = IG(1/C |ξd |, 1),√()
bb(x − a)2
where IG(a, b) =exp −, a, b > 0.
2πx3
2a2 x

References[1] D.M. Blei, A. Ng, and M.I. Jordan. Latent Dirichlet allocation. Journal of
Machine Learning Research (JMLR), 3:993–1022, 2003.
[2] D.M. Blei and J.D. McAuliﬀe. Supervised topic models. In Advances in
Neural Information Processing Systems (NIPS), pages 121–128, 2007.
[3] K.L. Caballero and R. Akella. Dynamic modelling patient’s health statefrom electric medical records: A time series approach. Proc. 21st ACM
SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.69-78, August, 2015.
[4] N. Chen, J. Zhu, F. Xia, and B. Zhang. Discriminative relational topic
models. IEEE Trans. on PAMI, 37(5):973–986, 2015.
[5] P. Cao, D. Zhao, and O. Zaiane. An optimized cost-sensitive SVM for imbalanced data learning. In Advances in Knowledge Discovery and Data Mining:2013. Berlin Heidelberg: Springer; 2013:28092.
[6] L. Devroye. Non-uniform random variate generation. Springer-Verlag, 1986.
[7] J.J. Escarce and M.A. Kelly. Admission source to the medical intensive careunit predicts hospital death independent of APACHE II score. Journal of
American Medical Association, 264(18):.2389–2394, 1985.
[8] M. Ghassemi, T. Naumann, F. Doshi-Velez, N. Brimmer, R. Joshi, A.
Rumshisky, and P. Szolovits. Unfolding physiological state: Mortality modelling in intensive care units, Proc. 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.75–84, 2014.
[9] T.L. Griﬃths and M. Steyvers. Finding scientific topics. Proceedings of the
National Academy of Science (PNAS), 5228–5235, 2004.
[10] H. He and E.A. Garcia. Learning from imbalanced data, IEEE Trans.
Knowl. Data Eng., 21(9):1263–1284, 2009.
[11] Y. Halpern, S. Horng, L. Nathanson, N. Shapiro, and D. Sontag. A comparison of dimensionality reduction techniques for unstructured clinical text,
In ICML 2012 Workshop on Clinical Data Analysis, 2012.
[12] Q. Jiang, J. Zhu, M. Sun, and E.P. Xing. Monte Carlo methods for maximum margin supervised topic models. Advances in Neural Information Processing Systems (NIPS), 1601–1609, 2012.
[13] A. Johnson, T. Pollard, L. Shen, L. Lehman, M. Fen, M. Ghassemi, B.
Moody, P. Szolovits, L. Celi, and R. Mark. MIMIC-III a freely accessible
critical care database. Scientific Data, 2016.

16

H. Ishizuka et al.

[14] L.G. JR, L.S, and S.F. A new simplified acute physiology score (SAPS
II) based on a European/North American multicenter study. Journal of
American Medical Association, 270(24):2957–2963, 1993.
[15] S. Lacoste-Jullien, F. Sha, and M.I. Jordan. DiscLDA: Discriminative learning for dimensionality reduction and classification. NIPS, 897–904, 2008.
[16] L. Lehman, M. Mohammed Saeed, W. Long, J. Lee, and R. Mark. Risk
stratification of icu patients using topic models inferred from unstructured
progress notes. AMIA Annual Symposium Proc., 2012:505–511, 2012.
[17] D. Li, S. Somasundaran, and A. Chakraborty. A combination of topic models with max-margin learning for relation detection. In ACL TextGraphs-6
Workshop, 2011.
[18] N.G. Polson and S.L Scott. Data augmentation for support vector machines.
Bayesian Analysis, 6(1):1–4, 2011.
[19] K. Veropoulos, C. Campbell, and N. Cristianini. Controlling the sensitivity of support vector machines. In: Proceedings of the International Joint
Conference on Artificial Intelligent, pp. 55–60, 1999.
[20] B.X. Wang and N. Japkowicz, Boosting support vector machines for imbalanced datasets. Journal of Knowledge and Information Systems, 4994:38–47,
2008.
[21] Y. Wang and G. Mori. Max-margin latent Dirichlet allocation for image
classification and annotation. In BMVC, 2011.
[22] J. Yohan and C.P. Rose. Time Series Analysis of Nursing Notes for Mortality Prediction via a State Transition Topic Model. Proceedings of the24th ACM International Conference on Information and Knowledge Management, 2015.
[23] Y. Zhang, R. Jiang, and L. Petzold, Survival Topic Models for Predicting
Outcomes for Trauma Patients. Data Engineering (ICDE), 2017 IEEE 33rd
International Conference.
[24] J. Zhu, A. Ahmed, and E.P. Xing. MedLDA: maximum margin supervised
topic models. Journal of Machine Learning Research (JMLR), (13):2237–
2278, 2012.
[25] J. Zhu, N. Chen, H. Perkins, and B. Zhang. Gibbs max-margin topic models with fast inference algorithms. In International Conference on Machine
Learning (ICML), pp. 124–132, 2013.
[26] J. Zhu, X. Zheng, and B. Zhang. Bayesian logistic supervised topic models
with data augmentation. In The Annual Meeting of the Association for
Computational Linguistics, 2013.