JMIR MEDICAL INFORMATICSP Tafti et al

Original Paper

Adverse Drug Event Discovery Using Biomedical Literature: A Big
Data Neural Network Adventure
Ahmad P Tafti1, PhD; Jonathan Badger1, PharmD; Eric LaRose1, BS; Ehsan Shirzadi2, MSc; Andrea Mahnke1, MSc;
John Mayer1, PhD; Zhan Ye1, PhD; David Page3, PhD; Peggy Peissig1, MBA, PhD
1

Biomedical Informatics Research Center, Marshfield Clinic Research Institute, Marshfield, WI, United States

2

Institute of Electrical and Electronics Engineers, Dublin, Ireland

3

Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison, WI, United States

Corresponding Author:
Ahmad P Tafti, PhD
Biomedical Informatics Research Center
Marshfield Clinic Research Institute1000 N Oak Ave
Marshfield, WI, 54449
United States
Phone: 1 7153893514
Email: pahlavantafti.ahmad@marshfieldresearch.org

Abstract
Background: The study of adverse drug events (ADEs) is a tenured topic in medical literature. In recent years, increasing
numbers of scientific articles and health-related social media posts have been generated and shared daily, albeit with very limited
use for ADE study and with little known about the content with respect to ADEs.
Objective: The aim of this study was to develop a big data analytics strategy that mines the content of scientific articles andhealth-related Web-based social media to detect and identify ADEs.
Methods: We analyzed the following two data sources: (1) biomedical articles and (2) health-related social media blog posts.
We developed an intelligent and scalable text mining solution on big data infrastructures composed of Apache Spark, natural
language processing, and machine learning. This was combined with an Elasticsearch No-SQL distributed database to exploreand visualize ADEs.
Results: The accuracy, precision, recall, and area under receiver operating characteristic of the system were 92.7%, 93.6%,
93.0%, and 0.905, respectively, and showed better results in comparison with traditional approaches in the literature. This work
not only detected and classified ADE sentences from big data biomedical literature but also scientifically visualized ADEinteractions.
Conclusions: To the best of our knowledge, this work is the first to investigate a big data machine learning strategy for ADE
discovery on massive datasets downloaded from PubMed Central and social media. This contribution illustrates possible capacities
in big data biomedical text analysis using advanced computational methods with real-time update from new data published on adaily basis.
(JMIR Med Inform 2017;5(4):e51) doi: 10.2196/medinform.9170
KEYWORDS
adverse drug event; adverse drug reaction; drug side effects; machine learning; text mining

Introduction
Background
Adverse drug events (ADEs), defined as the set of detriments
or injuries caused by a medication, have led to additional
medical costs, prolonged hospitalization, morbidity, andhttp://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderX

ascribable disability worldwide [1-4]. ADEs encompass all
adverse drug reactions but also include preventable causes oferrors such as inappropriate dosing, dispensing errors, and drug
abuse. Discovery of ADEs has gained great attention in thehealth care community, and in the last few years, several drug
risk-benefit assessment strategies have been developed toanalyze drug efficacy and safety using different medical data
JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 1
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
sources, ranging from electronic health records (EHRs) tohuman-health–related social media and drug reviews [5-14]. A
variety of combined computational methods using natural
language processing (NLP), machine learning strategies, andtext retrieval algorithms have been employed to extract ADEs
from such data sources [15-23]. Clinical trials, EHRs, andmedical case reports are additional biomedical data-rich sourcesthat have been utilized for ADE extraction [24-28].
In recent years, biomedical articles produced by scientists all
across the world have grown extensively. Figure 1 [29] showsthat the number of journal and conference papers published indifferent medication studies (eg, ADEs and drug analysis, drug
evaluation, and drug repositioning) rapidly grew in number
from year 2007 to 2016. The total number of publications in
those years is approximately 342,301 articles. To roughly
estimate the size of such scientific papers, we assumed a PDF
file format for each article. The size of a PDF file depends on
the number of pages and pictures or metadata inside the file.
Considering a 9 to 11 page PDF including plain text along witha few pictures, it may equal almost 3 MB size in average, andit appears, approximately 1.02 TB articles were generated in
drug associated studies from 2007 to 2016. The other fileformats such as extensible markup language (XML), may bemuch larger in size. Scientific articles published in biomedical
research are usually generated using standardized and principled
methods and therefore, are especially valuable for high-quality
knowledge discovery. This great deluge of information includes
an enormous number of scientific publications on ADEs’ study,
an area of focus into which many biomedical researchers haveentered, developing a variety of research activities fordiscovering, analyzing, and monitoring ADEs [30-38].

P Tafti et al
It is impossible for researchers, scientists, and physicians toread and process the large body of scientific articles and remain
abreast of the foremost information regarding ADEs. Therefore,
there is a pressing need to develop intelligent computational
methods, particularly big data analytics solutions, to efficiently
process this wealth of data. Big data biomedical text analysis
utilizes advanced computational technologies including big data
infrastructure, NLP, statistical analytics, and machine learningalgorithms to extract facts from text data. This in turn generates
new hypotheses by systematically analyzing large numbers ofscientific publications.

Objectives and the Main Contributions
Whereas ADE discovery from diverse biomedical data sourcesin general has been studied historically in health careinformatics, the use of big data scientific articles andhealth-related social media for ADE discovery has been very
limited so far. The motivation of this work is to study big data
machine learning solutions, particularly big data neural networks(bigNN), to analyze ADEs from large-scale biomedical text
data, developing a scalable framework to fulfill the following
objectives: (1) to extract current knowledge and high-quality
information about ADEs using full text scientific articles andsocial media, (2) to utilize and adapt advanced NLP and machinelearning algorithms in a large-scale fashion by the use of big
data infrastructures, and (3) to provide better insights andtendencies in large-scale biomedical text analytics and identify
the challenges and potential enhancements toward efficient andaccurate ADE discovery. We briefly summarize our main
contributions as follows:

Figure 1. The number of publications in several medication studies available at PubMed over the last 10 years. The results obtained by submitting aquery: ((((((((((drug analysis[MeSH Terms]) OR drug analysis[MeSH Subheading]) OR adverse drug event[MeSH Terms]) OR adverse drug event[MeSHSubheading]) OR adverse drug reaction[MeSH Terms]) OR adverse drug reaction[MeSH Subheading]) OR drug evaluation[MeSH Terms]) OR drug
evaluation[MeSH Subheading])) OR drug repositioning[MeSH Terms]) OR drug repositioning[MeSH Subheading].

http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderX

JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 2
(page number not for citation purposes)

JMIR MEDICAL INFORMATICSP Tafti et al

We initiated a study of big data literature mining for ADE
discovery with the use of two different data sources: (1)published full text scientific articles available on PubMed
Central [39], and (2) posts available in health-related social
media, including MedHelp [40], patient [41], and WebMD [42].
Although several promising approaches have been designed forbiomedical text mining, the development of scalable machinelearning frameworks capable of ADE extraction from big data
is very limited. To the best of our knowledge, our work is thefirst to investigate a bigNN strategy for ADE discovery on
massive datasets downloaded from PubMed Central and social
media.

••

Tier 1 systematically collects the expanding body of scientificarticles and social media blog posts through different data
sources available on the Internet. A multi-threaded crawler or
downloader was developed to provide timely and efficient
processing of the diverse big data content found on the Internet.
Scrapy [43], a free and open source Web crawling system has
been used to allow multiple threads to automatically fetch URLs
from different sources. A queuing system and a scheduler havealso been established as a part of the data access tier.

•With the current work and using big data analytics platforms
such as Elasticsearch and Apache Spark, we developed ascalable framework to analyze and visualize ADEs fromhundreds of thousands of published scientific articles andsocial media blog posts.
Combining a variety of the internal neural network
parameters, we presented a predictive model that obtained
accuracy, precision, recall, and area under 92.7%, 93.6%,
93.0%, and 0.905, respectively, on a massive dataset
downloaded from PubMed Central plus health-related social
media.

http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderX

This paper opens the door to pursue large-scale biomedical
literature mining and its application in health careinformatics in general and introduces several possible
enhancements to advance the level of the impact of thisresearch area.

Methods
The general pipeline of the proposed ADE extraction framework
is illustrated in Figure 2. In this section, we shall explain theunderlying tiers of the proposed bigNN framework.

Tier 1: Data Access

JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 3
(page number not for citation purposes)

JMIR MEDICAL INFORMATICSP Tafti et al

Figure 2. The proposed system for adverse drug event (ADE) discovery. All tiers developed on top of the Apache Spark 2.0 that utilizes an Elasticsearchdatabase 2.4.1 to data storage and retrieval.

All collected data (eg, XML files) are first turned into plain text
data, and together with associated metadata (eg, journal name,author list, and publication date) are stored in a single type (tabletype) inside a No-SQL database, namely Elasticsearch [44],which provides a distributed, open source, RESTful and fulltext search and analytics engine.

Tier 2: Natural Language Processing
Tier 2 includes several computational procedures to process
raw text data, preparing potential ADE sentences to feed thenext tier.

http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderX

Selecting Relevant Documents
One of the major goals of the proposed system was to collectgroups of sentences that provide evidence about drug-event
pairs. Toward that outcome, there is an emergent need to identify
trustworthy and reputable data sources (eg, well-founded andprestigious journals such as Nature, PNAS, and PLOS). As the
proposed framework accumulates data from two separate data
sources, including scientific journals and messages posted onsocial media, we established two different criteria to yield morecredible data. Section A.1 of Multimedia Appendix 1 further
discusses the proposed method and criteria.

JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 4
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Text Processing
We first normalize all documents by converting corpora into astandard consistent form. This process (1) converts all characters
to lower case, (2) transliterates to American Standard Code forInformation Interchange if needed, and (3) deletes a set ofexisting substrings and patterns (eg, [], , ?, !, and ()). Once wecomplete the proposed text normalization process, we convert
every document into a set of sentences. Although several ADEs
could be captured among different sentences, extracting ADEs
interactions across sentences is significantly more challenging
than within sentences [6,45]. To feed the bigNN system, arandom subset of sentences was selected for manual annotationby three domain experts; see Section A.2 of Multimedia
Appendix 1 for details. The random subset of sentences includes
health and medical-related text data either with or without ADE
interactions. Sentences that are missing either a drug name oran adverse drug effect term were excluded. Using a Web
application (Multimedia Appendix 1), the domain experts
labeled individual sentences as ADEs, No-ADEs, or Not
Decided. To focus on binary classification, we omitted sentences
labeled “Not Decided” leaving two different classes: (1) ADEsand (2) No-ADEs. Section A.2 of Multimedia Appendix 1
explains how we made a training set for the machine learning
tier.

Tier 3: Machine Learning
Tier 3 implements the core functionality needed for making a
predictive model to distinguish ADEs sentences versus thosethat are No-ADEs. From the machine learning perspective, thismeans making a binary classification predictive model thatassigns one of two trained labels to new unlabeled sentences.
Therefore, when an arbitrary sentence arrives, the predictivemodel chooses one category exclusively from two predefinedclasses as ADEs or No-ADEs. There are two basic steps. The
first step is to extract effective content as a set of features. The
next step is the text classification assignment. The bag-of-words

http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderXP Tafti et al(BoW) representation, a widely used content extractor algorithm,
has been around for several years in the text analytics domain,and it provides an easy way to turn text-based data records into
a set of feature vectors such that the frequency of occurrenceof words (eg, uni-grams and/or bi-grams, along with
part-of-speech [POS] tagging) in the corpus is used as a feature
vector to train a classifier (eg, support vector machine [SVM],
decision tree, and/or logistic regression) [46-51]. The BoW
representation maintains word intensities across the corpus, but
it dissembles grammar, syntactic, semantic, and word order. In
contrast to the BoW representation, the word2vector (word2vec)
algorithm, originally developed at Google [52], includes a setof computational methods that turns a corpus of text data intoa meaningful vector space that encompasses grammar, semantic,
and word order. Word2vec comes with a two-layer neuralnetwork and is able to tackle several text analytics functions
including dependency parsing [53,54], named entity recognition[55,56], text classification [57,58], and word clustering [59].
The model takes a text corpus as an input and turns each wordin the corpus into a vector as illustrated in Section A.3 ofMultimedia Appendix 1. It then groups vectors of similar words
together in a vector space, training words against other words
that neighbor them in the input text corpus [60,61]. Our bigNN
system implemented the word2vec neural network, which is
fully explained in Section A.3 of Multimedia Appendix 1.
An abstract view of the proposed learning algorithm is shownin Figure 3. As mentioned in Section A.3 of Multimedia
Appendix 1, the algorithm builds a vector representation forwords and/or sentences. If we train a learning model with
98-dimension, then we will obtain a 98-digit in front of eachword (eg, “aspirin”). The cosine distance similarity [62,63],
which is the normalized dot product between vectors, is thenmeasured to find the best fitness class for a sentence. Once wehave the labeled sentences as ADEs or No-ADEs, we focus on
ADE sentences and find the positive adverse-drug interactions
using cTAKES [64].

JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 5
(page number not for citation purposes)

JMIR MEDICAL INFORMATICSP Tafti et al

Figure 3. This figure depicts the abstract view of the proposed learning model where the cosine distance similarity is measured to select an appropriate
category for a given sentence. Every single word is represented as a vector, and eventually every sentence (eg, adverse drug events [ADEs] or No-ADEs)
turns in to a vector space. Once we have done with training the model, for every new coming sentence, the highest cosine distance value is then measured
to find the best fitness class.

Results
Implementation and Test Bed
We investigated two different data sources: (1) biomedical
articles and (2) health-related social media blog posts. The firstdata source included 97,246,719 sentences obtained from almost
1,451,413 abstracts and full text articles available on PubMed
Central, and the second one consisted of 2,524,622 sentences
obtained from 419,915 blog posts at MedHelp, Patient, and
WebMD. To train our proposed predictive model illustrated in
Figure 2, we first randomly selected different subsets for the
http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderX

purpose of manual annotation. Those subsets were annotatedby three domain experts who have been working in medical andpharmacy domains. We defined two different classes as ADEs
and No-ADEs to indicate drug-event interactions in a sentence.
The sentences that included positive drug-event interaction were
tagged as ADEs, and the others were tagged as No-ADEs.
Before the large-scale manual extraction, we evaluated theinterrater reliability among our three domain experts, the resultsof kappa statistics, .84, which indicates very good concordance
and agreement; more details are reported in Section A.2 ofMultimedia Appendix 1.

JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 6
(page number not for citation purposes)

JMIR MEDICAL INFORMATICSP Tafti et al

Table 1. Nine datasets were employed to make and evaluate the big data neutral network (bigNN) system. Each row identifies the dataset along with
the number of ADEs and No-ADEs sentences within the dataset. Every row also shows how many of these sentences are human-labeled and/or
machine-labeled. The datasets are separated into three main categories: biomedical articles (*_BA), social media posts (*_SM), and the combinationof the two (*_Combined).

aDataset ID

Total number ofsentences

Number of ADEsa
sentences

Number of NoADEs sentences

Number of humanlabeled sentences

Number of machinelabeled sentences

ADEs#1_BA

6960

3311

3649

6960

0

ADEs#1_SM

400

160

240

400

0

ADEs#1_Combined

7360

3471

3889

7360

0

ADEs#2_BA

13,545

6359

7186

7015

6530

ADEs#2_SM

472

195

277

405

67

ADEs#2_Combined

14,017

6554

7463

7420

6597

ADEs#3_BA

21,27810,307

10,9717015

14,263

ADEs#3_SM

565

241

324

405

160

ADEs#3_Combined

21,843

10,548

11,2957420

14,423

ADEs: adverse drug events.

Next, we developed nine different datasets from those sentences
to train the proposed bigNN system. The first set of datasets(the first three rows in Table 1) used only human-labeleddatasets from our domain experts. Within these three datasets,
three types of sentences were generated: the first type wasbiomedical articles, the second was social media, and the third
was the combination of the two. For the second set of datasets(the three datasets in the middle of Table 1), we added somemachine-labeled datasets (the number of datasets were reported
in the last column, “number of machine-labeled sentences” ofTable 1) with human-labeled datasets to increase the sample
sizes. We also generated three types of sentences as illustrated
above. For the third set of datasets (the last three rows of Table1), we added in more machine-labeled datasets withhuman-labeled datasets; similar three types of sentences weregenerated.
We also utilized three smaller datasets, including 2600, 3500,
and 4000 human-labeled data records and plotted learning curves
to see how the accuracy of the predictive model varies withincreasing amount of training data. The results were notpromising enough, and we started with a larger human-labeleddataset as illustrated in the first three rows in Table 1. For all
the experiment, we utilized 75% of every dataset to train themodel and 25% to test it using four-fold cross validation, withno sentence to appear in both the training and testing sets at thesame time.

Experimental Setup
Every programming module in Tier 1 and Tier 2 was developedby Python 2.7.13. Tier 3, the machine learning tier, wasimplemented by Java j2SE 8. All of these tiers were developed
on top of the Apache Spark 2.0 [65] and Elasticsearch DB 2.4.1[44] just to tackle the problem of big data analytics in an
efficient and timely fashion. From the computational side, a
dedicated computational resource, including two virtual
machines in a VMWARE cluster environment, each running a
64-bit CentOS 6.8 operating system with 8 vCPUs, 16 GB RAM,

http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderXand 1 TB HDD in total, hosted on a Xeon E5-2690V3 2.6 GHz
CPU, were used to obtain the experimental results.

Experimental Validations
We analyzed the performance of the predictive model acrossall the datasets. Accuracy, precision, and recall obtained by theexperiments are shown in Multimedia Appendix 2. The firstcolumn shows the dataset used to in the experiments. The second
column describes a configuration setup for a set of internalparameters of the proposed neural network model. Minimum
word frequency (MWF) allows for ignoring all words in the
vocabulary with total occurrences lower than MWF value. Epoch(EP) is the number of forward and backward passes of all
training examples. Window size (WS) defines context windows
size to generate a vector representation for words across the
documents. Iteration (ITR) defines the number of iterations
done for each mini-batch during a training process. The lastcolumn shows elapsed time for the training stage. This does notreflect the time of text-preprocessing tasks such as normalizationand tokenization. The current table shows that greater EP and
ITR with the use of WS of two will provide better performanceacross all three datasets. To further analyze our proposed ADEs
sentence discovery system, we also compared the proposed
predictive model with the combination of BoW feature selection
method and SVM, decision tree, and naïve Bayes classifiers.
Uni-grams, bi-grams, along with POS tagging were used as
BoW features to make a predictive model. One can see inMultimedia Appendix 2, the most promising accuracy results
across all datasets obtained by the (MWF=2%, EP=25, WS=2,
and ITR=10) configuration. Furthermore, we demonstrated thatthe performance of bigNN system, both accuracy and time ofcompleting the task, is comparable with traditional SVM, naïve
Bayesian, and decision tree with BoW strategy. The results ofthis experiment are shown in Table 2. All the measures in Table
2 are selected from the best performed model by tuning themodels using different parameters for all the bigNN system andSVM, naïve Bayesian, and decision tree with BoW strategy.
Regarding the BoW feature set, we obtained the best results by
utilizing a combination of uni-grams and bi-grams, together
JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 7
(page number not for citation purposes)

JMIR MEDICAL INFORMATICSP Tafti et alwith POS tagging. With respect to the traditional machinelearning classifiers, and for example SVM, the best performance
was achieved with the use of radial basis function kernel, loss
of 0.12, seed of 1, and without normalization of input data.
For each of the datasets shown in Table 2, we split the data
randomly as 75% to train and 25% to test the proposed sentenceclassifier system. The best accuracy results using our proposed
predictive model were obtained by MWF=2%, EP=25, WS=2,
ITR=10 configuration. Regarding the BoW features set, we
utilized a combination of uni-grams, bi-grams, and POS tagging.
Using the proposed predictive model, the vocabulary size for
ADEs#1_Combined,
ADEs#2_Combined,
andADEs#3_Combined datasets were 15125, 26,448, and 37,524,
respectively.
Whereas with the use of BoW, when it utilized uni-grams,
bi-grams, and POS, the vocabulary size was 33,567, 59,941,
and76,758fordatasets
ADEs#1_Combined,
ADEs#2_Combined, and ADEs#3_Combined. For SVM,
decision tree, and naïve Bayes classifiers, we utilized Wekalibrary (version 3.7.12) [66] running on hadoop-2.7 [67] by theuse of Hadoop distributed file system.
The study shows that bigNN system generates better results incomparison with traditional BoW along with SVM, decision
tree, and naïve Bayes classification algorithms. Area under the
curve (AUC) of our proposed predictive model across all threedatasets was also analyzed, and it is shown in Figure 4.

experiments just to make sure that the predictive model issufficiently accurate in assigning appropriate classes for new
unlabeled data records. Our approach was to rely on humanreviews. We fed the proposed word2vec predictive model withnew unlabeled data records and gave a random subset of thesystem output five hundred system-labeled instances to two
domain experts to review. We got 87.6%, 86.1%, and 88.7% inaverage for accuracy, precision, and recall, respectively.
We briefly summarized the experimental results as follows:
The results we obtained showed that the use of combined dataset
was better than the use of either source individually (Table 1)
for all the models and is statistically significant (at P=.04).
The results illustrated in the Multimedia Appendix 2 show thata greater epoch along with a greater iteration with the use of
window size of two tend to be useful over all datasets using
bigNN system, and the result is statistically significant (atP=.02). However, it requires a longer training time.
The comparative study shown in Table 2 demonstrates that the
bigNN system was able to generate better results in comparison
with traditional BoW along with SVM, decision tree, and naïve
Bayes classification algorithms. Performing a t test on AUC
matched by those models shows statistically significant
differences (at P=.03) between our bigNN system and those
two models utilizing BoW along with decision tree and naïve
Bayes. It also shows no statistically differences employing BoW
and SVM.

The current test results are only as good as the predictive model
developed in the training phase. We accomplished further
Table 2. The comparisons of our big data neutral network (bigNN) system with traditional bag-of-words (BoW) method using support vector machine(SVM), decision tree, and naïve Bayes classifiers.

aDataset ID

Learning method

Number ofsentences

Accuracy(%)

Precision(%)

Recall (%)

Area under the receiver Training
operating characteristic time (min)ADEs#1_Combineda

bigNNb system

7360

88.7

88.5

89.4

0.842

45.7

ADEs#1_Combined

BoWc + SVMd

7360

89.4

88.3

88.0

0.841

66.3

ADEs#1_Combined

BoW + decision tree

7360

84.0

83.7

82.1

0.775

49.5

ADEs#1_Combined

BoW + naïve Bayes

7360

83.7

82.1

83.5

0.763

48.9

ADEs#2_Combined

bigNN system

14,017

89.1

88.9

89.3

0.874

69.5

ADEs#2_Combined

BoW + SVM

14,017

89.5

88.0

89.7

0.875

88.9

ADEs#2_Combined

BoW + decision tree

14,017

85.5

84.9

84.5

0.861

75.2

ADEs#2_Combined

BoW + naïve Bayes

14,017

84.3

84.0

85.7

0.855

73.8

ADEs#3_Combined

bigNN system

21,843

92.7

93.6

93.0

0.905

121.7

ADEs#3_Combined

BoW + SVM

21,843

92.5

94.0

93.2

0.911

159.5

ADEs#3_Combined

BoW + decision tree

21,843

88.3

87.5

87.2

0.868

131.5

ADEs#3_Combined

BoW + naïve Bayes

21,843

87.5

86.2

85.8

0.851

135.3

ADEs: adverse drug events.

b

bigNN: big data neutral network.

c

BoW: bag-of-words.

d

SVM: support vector machine.

http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderX

JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 8
(page number not for citation purposes)

JMIR MEDICAL INFORMATICSP Tafti et al

Figure 4. This figure shows the area under the curve (AUC) of our proposed predictive model. ADEs: adverse drug events.

Table 2 also shows that our bigNN system was faster than the
BoW method along with SVM, decision tree, and naïve Bayesclassification algorithms. Performing a t test on training time
matched by those methods presents statistically significant
differences (at P=.03) between our proposed model and all those
three models developed by BoW along with SVM, decision
tree, and naïve Bayes.
The AUCs presented in Figure 4 show that larger training
samples tend to add benefit when making an accurate andreliable predictive model.
To further analyze the results, the next section will present a
set of ADEs’ scientific visualizations obtained using the
proposed framework.

Scientific Visualization of ADEs
Scientific visualization is concerned with representing large
and highly dimensional information by means of charts, graphs,
and images. The general objective of any scientific visualization
is to improve understanding of the data being investigated. In

http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderX

this section, we scientifically visualize the ADE information
extracted by our proposed system. Once the system was trained
across the different datasets, we fed the system new unlabeled
data. This included 92,681,359 sentences from biomedical
articles downloaded from PubMed Central and 1,624,117
sentences from social media, including MedHelp, Patient, and
WebMD. There are considerable amounts of drugs and adverse
events, and it is beyond the scope of this paper to visualize allof them. Here, we limited ourselves to the list of 28 drugs as
illustrated in Table 3. The proposed system could find 12,265
ADE sentences from biomedical articles and 181 ADE sentencesfrom three social media sources using the drug list defined.
Figures 5 and 6 show ADE discovery visualization results
obtained from the biomedical articles and social media,
respectively. Figure 7 represents a set of word cloud examples
generated for ADEs associated with “aspirin,” “atenolol,”“gabapentin,” and “statins.” A word cloud is a graphicalrepresentation composed of different words contained in a
corpus, in which the size of every word indicates its frequency
or importance to the text.

JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 9
(page number not for citation purposes)

JMIR MEDICAL INFORMATICSP Tafti et al

Table 3. The list of the drugs used to make the scientific visualization results.
Row ID

Drug name1

Anesthesia

2

Antihistamine

3

Antipsychotic

4

Aspirin5

Atenolol6

Atorvastatin

7

Azithromycin

8

Dexamethasone

9

Diazepam

10

Dopamine

11

Ephedrine

12

Gabapentin

13

Galantamine

14

Heparin

15

Ibuprofen

16

Lamotrigine

17

Lorazepam

18

Melatonin

19

Meloxicam

20

Metformin

21

Methylphenidate

22

Ondansetron

23

Orlistat24

Sildenafil

25

Statins

26

Vioxx

27

Warfarin

28

Wellbutrin

http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderX

JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 10(page number not for citation purposes)

JMIR MEDICAL INFORMATICSP Tafti et al

Figure 5. The adverse drug events’ (ADEs’) visualization results obtained from biomedical articles. One can see the number of “anesthesia” observations
is 2186, where the most frequent adverse drug events are “hypotension,” “nausea,” “aspiration,” and “depression,” respectively. Using the CI of 95%,
Pr(hypotension|anesthesia) is between 20.5% and 23.5%, and Pr(nausea|anesthesia) is some point between 12.0% and 14.7%. Another example is“gabapentin” where the most frequent adverse drug events based on ADE sentences extracted from biomedical articles are “dizziness,” “nausea,”“fatigue,” and “edema” whereas the number of “gabapentin” is 261. Using the CI of 95%, Pr(dizziness|gabapentin) is between 33.0% and 44.2%.

http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderX

JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 11(page number not for citation purposes)

JMIR MEDICAL INFORMATICSP Tafti et al

Figure 6. The adverse drug events’ (ADEs’) visualization results obtained from health-related social media. One can see the number of “metformin”
observations is 28, where its most frequent ADEs are “nausea,” “diarrhea,” “vomiting,” “dizziness,” and “stomach pain,” respectively. Another example
is “atenolol” and its most frequent adverse events are “depression,” “bradycardia,” “hypotension,” “tiredness,” and “dizziness.” From the adverse events
aspect, this figure shows, for example, “nausea” as an adverse event is mostly associated with “metformin,” “dexamethasone,” “antihistamine,”“wellbutrin,” and “sildenafil”.

http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderX

JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 12(page number not for citation purposes)

JMIR MEDICAL INFORMATICSP Tafti et al

Figure 7. The word cloud representations for “aspirin,” “atenolol,” “gabapentin,” and “statins.” (1) These results are obtained from biomedical articles.
Whereas the most frequent adverse dug events for “statins” are “myopathy,” “rhabdomyolysis,” “myalgia,” “fatigue,” and “hepatic dysfunction” asshown in (D), the most frequent ADEs for “atenolol” are “depression,” “hypotension,” “bradycardia,” and “impotence,” respectively (B). (2) These
results are extracted from social media. Whereas the most frequent ADEs for “statins” are “rhabdomyolysis” and “fatigue” (D), the most frequent ADEs
for “atenolol” are “depression,” “bradycardia,” “hypotension,” and “dizziness,” respectively (B).

Principal Findings

there is a pressing need to develop efficient solutions to harness
this wealth of data using advanced computational methods suchas artificial intelligence and big data machine learning.

Modern medical data sources ranging from clinical trials, EHRs,
and medical case reports to scientific articles and patients’ blog
posts are rapidly growing in size and complexity, and scientificbiomedical articles, as well as health-related social media areunder-researched data sources for biomedical studies. Thus,

With this contribution, an attempt was made to design anddevelop a computational bigNN system to detect, analyze, andvisualize ADEs from massive data sources obtained from
PubMed Central, a widely referenced repository of scientificarticles, and the social media blog posts existing in MedHelp,

Discussion

http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderX

JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 13(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
Patient, and WebMD. The model was developed using theword2vec neural network architectural on top of the Apache
Hadoop cluster, Apache Spark, and Elasticsearch No-SQL
distributed database to tackle efficient big data ADE
identification. We accomplished extensive experimental
validations to ensure that the proposed predictive model canaccurately assign appropriate classes (eg, ADEs or No-ADEs)
for both current and also new unlabeled data records. Our trained
system was able to detect a number of well-known ADEs fromunlabeled data taken directly from the literature. A list ofwell-known ADEs can be found in Multimedia Appendix 1.
Warfarin is an extremely effective anticlotting agent primarily
used for patients at high risk for stroke or heart attack becauseof atrial fibrillation that, up until recently, has been a cornerstoneof treatment. One common and potentially serious adverse effectof warfarin therapy is bleeding. This can occur because ofchanges in diet, drug interactions, or spurious physiologicalchanges. The effectiveness of warfarin coupled with a high risk
for serious bleeding events led to extensive research andpublication in the medical literature, which is very apparent inour data visualization. Similarly, the widespread use of aspirin,
which also carries a risk of bleeding, has been extensively
studied for primary prevention of heart attack, colorectal cancer,
and for secondary preventions of cardiovascular events, whichis also captured in our results. Our system also identifies serious
but rare side effects such as lactic acidosis caused by metformin
use and rhabdomyolysis attributed to statin therapy. Common
drug side effects are also captured, although with a smaller
number of hits. Examples in Figure 5 include ADE pairs of
metformin and diarrhea, antipsychotics and weight gain,
gabapentin and dizziness, and drowsiness with antihistamines.
The findings of our system shed light on areas for future workand on inherent challenges with semantics and context in NLP.
The following examples will illustrate some of these challenges.
Our system identifies nausea and vomiting as ADEs associatedwith dexamethasone. Although it is true that dexamethasone
can cause these reactions, it is commonly prescribed to prevent
nausea and vomiting associated with chemotherapy. Without
more contextual clues from the free text in the articles, it is
impossible for us to decide whether dexamethasone is being
identified as a treatment or a causal agent in this case. Anotherexample where our system identified an indication as an ADE
was with lamotrigine and seizures. Seizure is a primary
indication for the use of lamotrigine and not a causal agent.
Successfully classifying these edge cases may require additionallabeled data, a larger window size, or other unexploredtechniques and is an area for further study.
The ADEs identified from our analysis of social media provide
a number of interesting similarities and differences with thosein the literature. In our social media results, as might be
expected, we see a larger proportion of ADEs related to themore common side effects of drugs as compared with theliterature. For example, we see a large proportion of sentences
identified for nausea and diarrhea with the use of metformin
and fewer mentions of abdominal pain and vomiting. This
parallels nicely with the incidence expected in real-world use.
In contrast, we see a high proportion of sentences labeled forlactic acidosis, an extremely rare ADE associated with
http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderXP Tafti et al
metformin use from the literature. The number of sentences
describing adverse drug events in biomedical text articles ishighly variable and includes influencing factors such as theseverity of the drug reaction, safety concerns eliciting directedstudy, and the goals and intent of the research paper.
Non-life-threatening ADEs are less important to clinicalresearchers, assuming they do not result in discontinuation,compared with serious reactions. In a similar way, side effects
reported in social media will naturally include more commonside effects, particularly because they are impactful to the patient
taking the medication. In future work, text mining should takeadvantage of these naturally occurring differences. Publicationsin the biomedical literature or postings in social media,
especially early after the release of a novel drug, may include
case studies or reports of side effects not seen in clinical trials
that could be detected by our system before the signal reaches
the critical detection threshold of reporting systems such as the
FDA Adverse Event Reporting System.

Limitations
We acknowledge some limitations to this research study.
Assessing the quality of scientific journals is a difficult task but
important for narrowing the search space of candidate articles.
In this work, we attempt to combine three ranking indices in
the hope of identifying journals with the most credible
information without generating a hand-curated list. It may be
the case that our approach excludes journals that would be
extremely useful in identifying ADEs but are excluded basedon a low combined score. In addition, some journals will provide
a richer source of information on ADEs than others based on
their intended audience and subject matter irrespective of any
ranking criterion. This leaves the question of which journals tofocus on for ADE text mining open for further exploration. In
the scientific articles, as well as social media blog posts, we
noticed that different people may use different terms to discussa similar single adverse event. For example, the terms “moodchanges” and “mood swings” are often used interchangeably,
equally meaning “mood changes.” A robust dictionary-based
methodology may help address this issue. Additionally, text
mining of the social media comments posted by patients is areally challenging task, as the comments are often written in an
informal way. As we have a smaller number of labeled sentencesfrom this source, we didn’t address overcoming the differencesin phrasing, spelling errors, or other problems introduces fromthese posts. No fuzzy matching or specialized dictionaries wereused on either source, so if there are spelling errors in the drug
name, adverse event, or indication, the sentence would havebeen excluded from evaluation. Furthermore, our proposed big
data neural network model is more appropriate for the
short-length text data (eg, a sentence) classification rather thanthe long-length text data (eg, full text articles) categorization.
Advanced tokenization systems, and in particular, a medical
literature–based tokenization system will be useful withshort-length text data.

Conclusions
The present contribution utilized a bigNN system to discover
only ADEs; however, there are several interesting applications
to leverage the proposed system. For example, the proposed big
JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 14(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
data analytics pipeline could help study drug repurposing andimpact drug development, particularly in analyzing the successrates of new medications. The social media demographic
information (eg, age, gender, ethnicity, and location) supports
use of the current contribution to explore ADEs and drug
indications discussed by different demographic groups. For
future work, we intend to further explore the application of the
proposed framework to medical informatics, and particularly

P Tafti et al
drug analyses, extending the work for social media–based ADE
discovery discussed by different demographics groups. We
would enhance the proposed framework to tackle the problem
of semisupervised learning with multiple labels, rather than
only a single label. We also plan to make a sentence-based ADE
discovery dataset and present it publicly and make it freely
available to the research community.

Acknowledgments
The authors gratefully acknowledge the support of the Center for Predictive Computational Phenotyping, supported by the National
Institute of Health Big Data to Knowledge Initiative, US4AI117924; the University of Wisconsin Institute for Clinical andTranslational Research (UL1TR000427) and the support of NIGMS grant 2RO1 GM097618. The authors of the paper wish to
thank Richard Berg, Ryan Frahm, and Joseph Ellefson at Marshfield Clinic Research Institute and Mehdi Assefi and Sanaz
Haghani at University of Georgia for their valuable contributions in big data analytics and scientific visualizations presented inthis work. Special thanks goes to Anne Nikolai at Marshfield Clinic Research Institute for her valuable contributions in manuscript
preparation.

Authors' Contributions
PP, DP, and APT conceived and designed the study. PP and DP secured the funding for this contribution. APT, JB, EL, and ES
designed and developed the software framework. APT, JB, EL, ES, AM, JM, and ZY conducted the data collection. All authors
contributed to the analysis and interpretation of the results and scientific visualization. PP and APT led the writing of thismanuscript with all coauthors comments. All authors read and approved the final manuscript.

Conflicts of Interest
None declared.

Multimedia Appendix 1
Selecting the relevant documents, Building the training set, Word2vec neural network model, Well-known ADEs.
[PDF File (Adobe PDF File), 640KB-Multimedia Appendix 1]

Multimedia Appendix 2
bigNN experimental validations.
[PDF File (Adobe PDF File), 195KB-Multimedia Appendix 2]

References
1.
2.

3.
4.
5.
6.
7.
8.

Gottlieb A, Hoehndorf R, Dumontier M, Altman RB. Ranking adverse drug reactions with crowdsourcing. J Med Internet
Res 2015;17(3):e80 [FREE Full text] [doi: 10.2196/jmir.3962] [Medline: 25800813]
Bates DW, Cullen DJ, Laird N, Petersen LA, Small SD, Servi D, et al. Incidence of adverse drug events and potential
adverse drug events. Implications for prevention. ADE Prevention Study Group. J Am Med Assoc 1995 Jul 05;274(1):29-34.
[Medline: 7791255]
Classen DC, Pestotnik SL, Evans RS, Lloyd JF, Burke JP. Adverse drug events in hospitalized patients. Excess length ofstay, extra costs, and attributable mortality. J Am Med Assoc 1997;277(4):301-306. [Medline: 9002492]
Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis ofprospective studies. J Am Med Assoc 1998 Apr 15;279(15):1200-1205. [Medline: 9555760]
Trame MN, Biliouris K, Lesko LJ, Mettetal JT. Systems pharmacology to predict drug safety in drug development. Eur J
Pharm Sci 2016 Oct 30;94:93-95. [doi: 10.1016/j.ejps.2016.05.027] [Medline: 27251780]
Liu J, Zhao S, Zhang X. An ensemble method for extracting adverse drug events from social media. Artif Intell Med 2016
Jun;70:62-76. [doi: 10.1016/j.artmed.2016.05.004] [Medline: 27431037]
Price J. What can big data offer the pharmacovigilance of orphan drugs? Clin Ther 2016 Dec;38(12):2533-2545. [doi:10.1016/j.clinthera.2016.11.009] [Medline: 27914633]
van Walsem A, Pandhi S, Nixon RM, Guyot P, Karabis A, Moore RA. Relative benefit-risk comparing diclofenac to othertraditional non-steroidal anti-inflammatory drugs and cyclooxygenase-2 inhibitors in patients with osteoarthritis or rheumatoid
arthritis: a network meta-analysis. Arthritis Res Ther 2015 Mar 19;17:66 [FREE Full text] [doi: 10.1186/s13075-015-0554-0][Medline: 25879879]

http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderX

JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 15(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
9.

10.

11.
12.

13.
14.
15.

16.

17.
18.
19.
20.

21.
22.
23.
24.

25.
26.

27.
28.
29.
30.
31.
32.

Pignatti F, Ashby D, Brass EP, Eichler H, Frey P, Hillege HL, et al. Structured frameworks to increase the transparency of
the assessment of benefits and risks of medicines: current status and possible future directions. Clin Pharmacol Ther 2015
Nov;98(5):522-533. [doi: 10.1002/cpt.203] [Medline: 26261064]
Harpaz R, Callahan A, Tamang S, Low Y, Odgers D, Finlayson S, et al. Text mining for adverse drug events: the promise,challenges, and state of the art. Drug Saf 2014 Oct;37(10):777-790 [FREE Full text] [doi: 10.1007/s40264-014-0218-z][Medline: 25151493]
Singh S, Loke YK. Drug safety assessment in clinical trials: methodological challenges and opportunities. Trials 2012 Aug
20;13:138 [FREE Full text] [doi: 10.1186/1745-6215-13-138] [Medline: 22906139]
Guo JJ, Pandey S, Doyle J, Bian B, Lis Y, Raisch DW. A review of quantitative risk-benefit methodologies for assessing
drug safety and efficacy-report of the ISPOR risk-benefit management working group. Value Health 2010 Aug;13(5):657-666[FREE Full text] [doi: 10.1111/j.1524-4733.2010.00725.x] [Medline: 20412543]
Eichler H, Abadie E, Raine JM, Salmonson T. Safe drugs and the cost of good intentions. N Engl J Med 2009 Apr
02;360(14):1378-1380. [doi: 10.1056/NEJMp0900092] [Medline: 19339718]
Garrison LP, Towse A, Bresnahan BW. Assessing a structured, quantitative health outcomes approach to drug risk-benefit
analysis. Health Aff (Millwood) 2007;26(3):684-695 [FREE Full text] [doi: 10.1377/hlthaff.26.3.684] [Medline: 17485745]
Jonnagaddala J, Jue T, Dai H. Binary classification of Twitter posts for adverse drug reactions. 2016 Jan Presented at: In
Proceedings of the Social Media Mining Shared Task Workshop at the Pacific Symposium on Biocomputing; 2016; Big
Island, HI. USA p. 4-8.
Rastegar-Mojarad M, Elayavilli R, Wang L, Prasad R, Liu H. Prioritizing Adverse Drug Reaction and Drug Repositioning
Candidates Generated by Literature-Based Discovery. In: Proceedings of the 7th ACM International Conference on
Bioinformatics, Computational Biology, and Health Informatics. 2016 Presented at: BCB '16; October 02 - 05, 2016; Seattle,
WA, USA p. 289-296. [doi: 10.1145/2975167.2975197]
Yang M, Kiang M, Shang W. Filtering big data from social media--building an early warning system for adverse drug
reactions. J Biomed Inform 2015 Apr;54:230-240 [FREE Full text] [doi: 10.1016/j.jbi.2015.01.011] [Medline: 25688695]
Sarker A, Gonzalez G. Portable automatic text classification for adverse drug reaction detection via multi-corpus training.
J Biomed Inform 2015 Feb;53:196-207 [FREE Full text] [doi: 10.1016/j.jbi.2014.11.002] [Medline: 25451103]
Liu X, Chen H. AZDrugMiner: an information extraction system for mining patient-reported adverse drug events in online
patient forums. 2013 Presented at: International Conference on Smart Health; August 3, 2013; Beijing, China p. 134-150.
Bian J, Topaloglu U, Yu F. Towards large-scale twitter mining for drug-related adverse events. In: Proceedings of the 2012
international workshop on Smart health and wellbeing. 2012 Oct 29 Presented at: SHB '12; October 29 - 29, 2012; Maui,
Hawaii, USA p. 25-32 URL: http://europepmc.org/abstract/MED/28967001 [doi: 10.1145/2389707.2389713]
Chee BW, Berlin R, Schatz B. Predicting adverse drug events from personal health messages. AMIA Annu Symp Proc
2011;2011:217-226 [FREE Full text] [Medline: 22195073]
Paul M, Dredze M. Drug Extraction from the Web: Summarizing Drug Experiences with Multi-Dimensional Topic Models.
2013 Presented at: Proceedings of NAACL-HLT 2013; 9-14 June, 2013; Atlanta, Georgia p. 168-178.
Tatonetti NP, Ye PP, Daneshjou R, Altman RB. Data-driven prediction of drug effects and interactions. Sci Transl Med
2012 Mar 14;4(125):125ra31 [FREE Full text] [doi: 10.1126/scitranslmed.3003377] [Medline: 22422992]
Liang Z, Huang JX, Zeng X, Zhang G. DL-ADR: a novel deep learning model for classifying genomic variants into adverse
drug reactions. BMC Med Genomics 2016 Aug 10;9(Suppl 2):48 [FREE Full text] [doi: 10.1186/s12920-016-0207-4][Medline: 27510822]
Iyer SV, Harpaz R, LePendu P, Bauer-Mehren A, Shah NH. Mining clinical text for signals of adverse drug-drug interactions.
J Am Med Inform Assoc 2014;21(2):353-362 [FREE Full text] [doi: 10.1136/amiajnl-2013-001612] [Medline: 24158091]
Harpaz R, DuMouchel W, Shah NH, Madigan D, Ryan P, Friedman C. Novel data-mining methodologies for adverse drug
event discovery and analysis. Clin Pharmacol Ther 2012 Jun;91(6):1010-1021 [FREE Full text] [doi: 10.1038/clpt.2012.50][Medline: 22549283]
Gurulingappa H, Mateen-Rajput A, Toldo L. Extraction of potential adverse drug events from medical case reports. J
Biomed Semantics 2012;3(1):15 [FREE Full text] [doi: 10.1186/2041-1480-3-15] [Medline: 23256479]
Aramaki E, Miura Y, Tonoike M, Ohkuma T, Masuichi H, Waki K, et al. Extraction of adverse drug effects from clinicalrecords. Stud Health Technol Inform 2010;160(Pt 1):739-743. [Medline: 20841784]
NCBI. PubMed URL: https://www.ncbi.nlm.nih.gov/pubmed [accessed 2017-11-29] [WebCite Cache ID 6vLTyklNq]
Naples JG, Hanlon JT, Schmader KE, Semla TP. Recent literature on medication errors and adverse drug events in older
adults. J Am Geriatr Soc 2016 Feb;64(2):401-408 [FREE Full text] [doi: 10.1111/jgs.13922] [Medline: 26804210]
Noguchi C, Sakuma M, Ohta Y, Bates DW, Morimoto T. Prevention of medication errors in hospitalized patients: the Japanadverse drug events study. Drug Saf 2016 Nov;39(11):1129-1137. [doi: 10.1007/s40264-016-0458-1] [Medline: 27638660]
Meier F, Maas R, Sonst A, Patapovas A, Müller F, Plank-Kiegele B, et al. Adverse drug events in patients admitted to an
emergency department: an analysis of direct costs. Pharmacoepidemiol Drug Saf 2015 Feb;24(2):176-186. [doi:10.1002/pds.3663] [Medline: 24934134]

http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderXP Tafti et al

JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 16(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
33.

34.

35.

36.
37.
38.

39.
40.
41.
42.
43.
44.
45.

46.

47.
48.
49.
50.

51.
52.
53.
54.
55.

56.
57.

58.

59.

Nuckols TK, Smith-Spangler C, Morton SC, Asch SM, Patel VM, Anderson LJ, et al. The effectiveness of computerizedorder entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review andmeta-analysis. Syst Rev 2014 Jun 04;3:56 [FREE Full text] [doi: 10.1186/2046-4053-3-56] [Medline: 24894078]
Kanaan AO, Donovan JL, Duchin NP, Field TS, Tjia J, Cutrona SL, et al. Adverse drug events after hospital discharge inolder adults: types, severity, and involvement of Beers Criteria Medications. J Am Geriatr Soc 2013 Nov;61(11):1894-1899[FREE Full text] [doi: 10.1111/jgs.12504] [Medline: 24116689]
Dormann H, Sonst A, Müller F, Vogler R, Patapovas A, Pfistermeister B, et al. Adverse drug events in older patients
admitted as an emergency: the role of potentially inappropriate medication in elderly people (PRISCUS). Dtsch Arztebl
Int 2013 Mar;110(13):213-219 [FREE Full text] [doi: 10.3238/arztebl.2013.0213] [Medline: 23596501]
Finkelstein J, Chen Q, Adams H, Friedman C. Automated summarization of publications associated with adverse drug
reactions from PubMed. AMIA Jt Summits Transl Sci Proc 2016;2016:68-77 [FREE Full text] [Medline: 27570654]
Natarajan S, Bangera V, Khot T, Picado J, Wazalwar A, Costa VS, et al. Markov logic networks for adverse drug event
extraction from text. Knowl Inf Syst 2017 May;51(2):435-457. [doi: 10.1007/s10115-016-0980-6] [Medline: 29123330]
Kang N, Singh B, Bui C, Afzal Z, van Mulligen EM, Kors JA. Knowledge-based extraction of adverse drug events frombiomedical text. BMC Bioinformatics 2014 Mar 04;15:64 [FREE Full text] [doi: 10.1186/1471-2105-15-64] [Medline:
24593054]
NCBI. PubMed Central URL: https://www.ncbi.nlm.nih.gov/pmc/ [accessed 2017-11-29] [WebCite Cache ID 6vLUJCock]
MedHelp. URL: http://medhelp.org/ [accessed 2017-11-29] [WebCite Cache ID 6vLUQL3F0]
Patient. URL: https://patient.info/ [accessed 2017-11-29] [WebCite Cache ID 6vLUVsMfF]
WebMD. URL: https://www.webmd.com/default.htm [accessed 2017-11-29] [WebCite Cache ID 6vLUcTEob]
Scrapy. URL: https://scrapy.org/ [accessed 2017-11-29] [WebCite Cache ID 6vLUjwo7C]
Elastic. Elasticsearch URL: https://www.elastic.co/ [accessed 2017-11-29] [WebCite Cache ID 6vLUpqeIv]
Henriksson A, Kvist M, Dalianis H, Duneld M. Identifying adverse drug event information in clinical notes with distributionalsemantic representations of context. J Biomed Inform 2015 Aug 17;57:333-349 [FREE Full text] [doi:10.1016/j.jbi.2015.08.013] [Medline: 26291578]
Sriram B, Fuhry D, Demir E, Ferhatosmanoglu H, Demirbas M. Short text classification in twitter to improve information
filtering. 2010 Presented at: The 33rd international ACM SIGIR conference on Research and development in information
retrieval; Jul 19, 2010; Geneva, Switzerland p. 841-842.
Sivic J, Zisserman A. Efficient visual search of videos cast as text retrieval. IEEE Trans Pattern Anal Mach Intell 2009
Apr;31(4):591-606. [doi: 10.1109/TPAMI.2008.111] [Medline: 19229077]
Sahlgren M, Cöster R. Using bag-of-concepts to improve the performance of support vector machines in text categorization.
2004 Presented at: the 20th International Conference on Computational Linguistics; August 23, 2004; Geneva, Switzerland.
Lodhi H, Saunders C, Shawe-Taylor J, Cristianini N, Watkins C. Text classification using string kernels. J Mach Learn
Res 2002;2:419-444.
Pedersen T. A decision tree of bigrams is an accurate predictor of word sense. 2001 Presented at: NAACL '01 Proceedingsof the second meeting of the North American Chapter of the Association for Computational Linguistics on Language
technologies; June 01 - 07, 2001; Pittsburgh, Pennsylvania p. 1-8.
Scott S, Matwin S. Feature engineering for text classification. 1999 Presented at: ICML '99 Proceedings of the SixteenthInternational Conference on Machine Learning; June 27 - 30, 1999; San Francisco, CA, USA p. 379-388.
Mikolov T, Chen K, Corrado G, Dean J. arxiv.org. 2013. Efficient estimation of word representations in vector space URL:
https://arxiv.org/pdf/1301.3781.pdf [accessed 2017-11-30] [WebCite Cache ID 6vNQxsjjp]
Bansal M, Gimpel K, Livescu K. Tailoring Continuous Word Representations for Dependency Parsing. In ACL (2) 2014
Jun (pp. 809-815) 2014.
Dyer C, Ballesteros M, Ling W, Matthews A, Smith N. Transition-based dependency parsing with stack long short-term
memory. arXiv preprint arXiv.08075 May 2015:29.
Lample G, Ballesteros M, Subramanian S, Kawakami K, Dyer C. Statnlp.org. 2016. Neural architectures for named entity
recognition URL: http://www.statnlp.org/wp-content/uploads/2017/06/nner.pdf [accessed 2017-11-30] [WebCite Cache
ID 6vNRB665R]
Santos CN, Guimaraes V. Aclweb.org. 2015. Boosting named entity recognition with neural character embeddings URL:
http://www.anthology.aclweb.org/W/W15/W15-3904.pdf [accessed 2017-11-30] [WebCite Cache ID 6vNRHxjLX]
Lilleberg J, Zhu Y, Zhang Y. Support vector machines and word2vec for text classification with semantic features. 2015
Presented at: Cognitive Informatics & Cognitive Computing (ICCI*CC), 2015 IEEE 14th International Conference on;
July 6, 2015; Beijing, China p. 136-140.
Zhang X, Zhao J, LeCun Y. Courant Institute of Mathematical Sciences, New York University. 2015. Character-level
convolutional networks for text classification URL: https://papers.nips.cc/paper/5782-character-level-convolutional-networks-for-text-classification.pdf [accessed 2017-11-30] [WebCite Cache ID
6vNRQDFcC]
Xue B, Fu C, Shaobin Z. A new clustering model based on Word2vec mining on Sina Weibo users' tags. IJGDC
2014;7(3):41-48. [doi: 10.14257/ijgdc.2014.7.3.05]

http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderXP Tafti et al

JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 17(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
60.
61.
62.
63.

64.
65.
66.
67.

P Tafti et al

Goldberg Y, Levy O. Arxiv.org. 2014. word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding
Method URL: https://arxiv.org/pdf/1402.3722.pdf [accessed 2017-12-01] [WebCite Cache ID 6vNReVCAM]
Rong X. arxiv.org. 2014. word2vec parameter learning explained URL: https://arxiv.org/pdf/1411.2738.pdf [accessed2017-12-01] [WebCite Cache ID 6vNRhUI42]
Ye J. Cosine similarity measures for intuitionistic fuzzy sets and their applications. Math Comput Model 2011;53(1-2):91-97.
[doi: 10.1016/j.mcm.2010.07.022]
Muflikhah L, Baharudin B. Document clustering using concept space and cosine similarity measurement. 2009 Presented
at: Computer Technology and Development, 2009. ICCTD '09. International Conference on; November 13, 2009; Kota
Kinabalu, Malaysia p. 58-62.
cTAKES: clinical Text Analysis and Knowledge Extraction System. URL: http://ctakes.apache.org/ [accessed 2017-11-29][WebCite Cache ID 6vLTlneFL]
Apache Spark. URL: https://spark.apache.org [accessed 2017-11-29] [WebCite Cache ID 6vLVIS1gd]
The University of Waikato. Weka Library URL: https://www.cs.waikato.ac.nz/ml/weka/ [accessed 2017-11-29] [WebCite
Cache ID 6vLUwaBCP]
Hadoop. URL: http://hadoop.apache.org/ [accessed 2017-11-29] [WebCite Cache ID 6vLVBwzjO]

Abbreviations
ADE: adverse drug event
AUC: area under the curve
BigNN: big data neutral network
BoW: bag-of-words
EHR: electronic health record
EP: epoch
ITR: iteration
MWF: minimum word frequency
NLP: natural language processing
POS: part-of-speech
SVM: support vector machine
WS: window size
XML: extensible markup language

Edited by G Eysenbach; submitted 13.10.17; peer-reviewed by V Hristidis, R Segall, F Rohani; comments to author 03.11.17; revised
version received 07.11.17; accepted 08.11.17; published 08.12.17
Please cite as:
P Tafti A, Badger J, LaRose E, Shirzadi E, Mahnke A, Mayer J, Ye Z, Page D, Peissig P
Adverse Drug Event Discovery Using Biomedical Literature: A Big Data Neural Network Adventure
JMIR Med Inform 2017;5(4):e51
URL: http://medinform.jmir.org/2017/4/e51/
doi: 10.2196/medinform.9170
PMID: 29222076©Ahmad P Tafti, Jonathan Badger, Eric LaRose, Ehsan Shirzadi, Andrea Mahnke, John Mayer, Zhan Ye, David Page, Peggy
Peissig. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 08.12.2017. This is an open-access article
distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which
permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR
Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on
http://medinform.jmir.org/, as well as this copyright and license information must be included.

http://medinform.jmir.org/2017/4/e51/

XSL• FO
RenderX

JMIR Med Inform 2017 | vol. 5 | iss. 4 | e51 | p. 18(page number not for citation purposes)