Introduction
Adverse drug reactions (ADR)—unwanted or harmful reactions resulting from correct medical drug use—present a significant and costly public health problem. BIBREF0 Detecting, assessing, and preventing these events are the tasks of pharmacovigilance. In the pre-trial and trial stages of drug development, the number of people taking a drug is carefully controlled, and the collection of ADR data is centralized. However, after the drug is available widely, post-marketing surveillance often requires the collection and merging of data from disparate sources, BIBREF1 including patient-initiated spontaneous reporting. Unfortunately, adverse reactions to drugs are grossly underreported to health professionals. BIBREF2 , BIBREF3 Considerable issues with patient-initiated reporting have been identified, including various types of reporting biases and causal attributions of adverse events. BIBREF4 , BIBREF5 , BIBREF6 Nevertheless, a large number of people, freely and spontaneously, report ADRs on social media. The potential availability of inexpensive, large-scale, and real-time data on ADRs makes social media a valuable resource for pharmacovigilance.
Information required for pharmacovigilance includes a reported adverse drug reaction, a linked drug referred to by its full, abbreviated, or generic name, and an indication whether it was the social media post author that experienced the adverse event. However, there are considerable challenges in automatically extracting this information from free-text social media data. Social media texts are often short and informal, and include non-standard abbreviations and creative language. Drug names or their effects may be mis-spelled; they may be used metaphorically (e.g., Physics is like higher level maths on steroids). Drug names might have other non-drug related meanings (e.g., ecstasy). An adverse event may be negated or only expected (e.g., I bet I'll be running to the bathroom all night), or it may not apply to the author of the post at all (e.g., a re-tweet of a press release).
The shared task challenge organized as part of the AMIA-2017 Workshop on Social Media Mining for Health Applications (SMM4H) focused on Twitter data and had three tasks: Task 1 - recognizing whether a tweet is reporting an adverse drug reaction, Task 2 - inferring whether a tweet is reporting the intake of a medication by the tweeter, and Task 3 - mapping a free-text ADR to a standardized MEDDRA term. Our team made submissions for Task 1 and Task 2. For both tasks, we trained Support Vector Machine classifiers using a variety of surface-form, sentiment, and domain-specific features. Handling class imbalance with under-sampling was particularly helpful. Our submissions obtained F-scores of 0.435 on Task 1 and 0.673 on Task 2, resulting in a rank of first and third, respectively. (Nine teams participated in each task.) We make the resources created as part of this project freely available at the project webpage: http://saifmohammad.com/WebPages/tweets4health.htm.
Task and Data Description
Below we describe in detail the two tasks we participated in, Task 1 and Task 2.
Task 1: Classification of Tweets for Adverse Drug Reaction
Task 1 was formulated as follows: given a tweet, determine whether it mentions an adverse drug reaction. This was a binary classification task:
The official evaluation metric was the F-score for class 1 (ADR): INLINEFORM0
The data for this task was created as part of a large project on ADR detection from social media by the DIEGO lab at Arizona State University. The tweets were collected using the generic and brand names of the drugs as well as their phonetic misspellings. Two domain experts under the guidance of a pharmacology expert annotated the tweets for the presence or absence of an ADR mention. The inter-annotator agreement for the two annotators was Cohens Kappa INLINEFORM0 . BIBREF7
Two labeled datasets were provided to the participants: a training set containing 10,822 tweets and a development set containing 4,845 tweets. These datasets were distributed as lists of tweet IDs, and the participants needed to download the tweets using the provided Python script. However, only about 60–70% of the tweets were accessible at the time of download (May 2017). The training set contained several hundreds of duplicate or near-duplicate messages, which we decided to remove. Near-duplicates were defined as tweets containing mostly the same text but differing in user mentions, punctuation, or other non-essential context. A separate test set of 9,961 tweets was provided without labels at the evaluation period. This set was distributed to the participants, in full, by email. Table TABREF1 shows the number of instances we used for training and testing our model.
Task 1 was a rerun of the shared task organized in 2016. BIBREF7 The best result obtained in 2016 was INLINEFORM0 . BIBREF8 The participants in the 2016 challenge employed various statistical machine learning techniques, such as Support Vector Machines, Maximum Entropy classifiers, Random Forests, and other ensembles. BIBREF8 , BIBREF9 A variety of features (e.g., word INLINEFORM1 -grams, word embeddings, sentiment, and topic models) as well as extensive medical resources (e.g., UMLS, lexicons of ADRs, drug lists, and lists of known drug-side effect pairs) were explored.
Task 2: Classification of Tweets for Medication Intake
Task 2 was formulated as follows: given a tweet, determine if it mentions personal medication intake, possible medication intake, or no intake is mentioned. This was a multi-class classification problem with three classes:
The official evaluation metric for this task was micro-averaged F-score of the class 1 (intake) and class 2 (possible intake): INLINEFORM0 INLINEFORM1
Information on how the data was collected and annotated was not available until after the evaluation.
Two labeled datasets were provided to the participants: a training set containing 8,000 tweets and a development set containing 2,260 tweets. As for Task 1, the training and development sets were distributed through tweet IDs and a download script. Around 95% of the tweets were accessible through download. Again, we removed duplicate and near-duplicate messages. A separate test set of 7,513 tweets was provided without labels at the evaluation period. This set was distributed to the participants, in full, by email. Table TABREF7 shows the number of instances we used for training and testing our model.
For each task, three submissions were allowed from each participating team.
System Description
Both our systems, for Task 1 and Task 2, share the same classification framework and feature pool. The specific configurations of features and parameters were chosen for each task separately through cross-validation experiments (see Section SECREF31 ).
Machine Learning Framework
For both tasks, we trained linear-kernel Support Vector Machine (SVM) classifiers. Past work has shown that SVMs are effective on text categorization tasks and robust when working with large feature spaces. In our cross-validation experiments on the training data, a linear-kernel SVM trained with the features described below was able to obtain better performance than a number of other statistical machine-learning algorithms, such as Stochastic Gradient Descent, AdaBoost, Random Forests, as well SVMs with other kernels (e.g., RBF, polynomic). We used an in-house implementation of SVM.
Handling Class Imbalance: For Task 1 (Classification of tweets for ADR), the provided datasets were highly imbalanced: the ADR class occurred in less than 12% of instances in the training set and less than 8% in the development and test sets. Most conventional machine-learning algorithms experience difficulty with such data, classifying most of the instances into the majority class. Several techniques have been proposed to address the issue of class imbalance, including over-sampling, under-sampling, cost-sensitive learning, and ensembles. BIBREF10 We experimented with several such techniques. The best performance in our cross-validation experiments was obtained using under-sampling with the class proportion 1:2. To train the model, we provided the classifier with all available data for the minority class (ADR) and a randomly sampled subset of the majority class (non-ADR) data in such a way that the number of instances in the majority class was twice the number of instances in the minority class. We found that this strategy significantly outperformed the more traditional balanced under-sampling where the majority class is sub-sampled to create a balanced class distribution. In one of our submissions for Task 1 (submission 3), we created an ensemble of three classifiers trained on the full set of instances in the minority class (ADR) and different subsets of the majority class (non-ADR) data. We varied the proportion of the majority class instances to the minority class instances: 1:2, 1:3, and 1:4. The final predictions were obtained by majority voting on the predictions of the three individual classifiers.
For Task 2 (Classification of tweets for medication intake), the provided datasets were also imbalanced but not as much as for Task 1: the class proportion in all subsets was close to 1:2:3. However, even for this task, we found some of the techniques for reducing class imbalance helpful. In particular, training an SVM classifier with different class weights improved the performance in the cross-validation experiments. These class weights are used to increase the cost of misclassification errors for the corresponding classes. The cost for a class is calculated as the generic cost parameter (parameter C in SVM) multiplied by the class weight. The best performance on the training data was achieved with class weights set to 4 for class 1 (intake), 2 for class 2 (possible intake), and 1 for class 3 (non-intake).
Preprocessing: The following pre-processing steps were performed. URLs and user mentions were normalized to http://someurl and @username, respectively. Tweets were tokenized with the CMU Twitter NLP tool. BIBREF11
Features
The classification model leverages a variety of general textual features as well as sentiment and domain-specific features described below. Many features were inspired by previous work on ADR BIBREF12 , BIBREF8 , BIBREF9 and our work on sentiment analysis (such as the winning system in the SemEval-2013 task on sentiment analysis in Twitter BIBREF13 and best performing stance detection system BIBREF14 ).
General Textual Features
The following surface-form features were used:
INLINEFORM0 -grams: word INLINEFORM1 -grams (contiguous sequences of INLINEFORM2 tokens), non-contiguous word INLINEFORM3 -grams ( INLINEFORM4 -grams with one token replaced by *), character INLINEFORM5 -grams (contiguous sequences of INLINEFORM6 characters), unigram stems obtained with the Porter stemming algorithm;
General-domain word embeddings:
dense word representations generated with word2vec on ten million English-language tweets, summed over all tokens in the tweet,
word embeddings distributed as part of ConceptNet 5.5 BIBREF15 , summed over all tokens in the tweet;
General-domain word clusters: presence of tokens from the word clusters generated with the Brown clustering algorithm on 56 million English-language tweets; BIBREF11
Negation: presence of simple negators (e.g., not, never); negation also affects the INLINEFORM0 -gram features—a term INLINEFORM1 becomes INLINEFORM2 if it occurs after a negator and before a punctuation mark;
Twitter-specific features: the number of tokens with all characters in upper case, the number of hashtags, presence of positive and negative emoticons, whether the last token is a positive or negative emoticon, the number of elongated words (e.g., soooo);
Punctuation: presence of exclamation and question marks, whether the last token contains an exclamation or question mark.
Domain-Specific Features
To generate domain-specific features, we used the following domain resources:
Medication list: we compiled a medication list by selecting all one-word medication names from RxNorm (e.g, acetaminophen, nicorette, zoloft) since most of the medications mentioned in the training datasets were one-word strings.
Pronoun Lexicon: we compiled a lexicon of first-person pronouns (e.g., I, ours, we'll), second-person pronouns (e.g., you, yourself), and third-person pronouns (e.g., them, mom's, parents').
ADR Lexicon: a list of 13,699 ADR concepts compiled from COSTART, SIDER, CHV, and drug-related tweets by the DIEGO lab; BIBREF16
domain word embeddings: dense word representations generated by the DIEGO lab by applying word2vec on one million tweets mentioning medications; BIBREF16
domain word clusters: word clusters generated by the DIEGO lab using the word2vec tool to perform K-means clustering on the above mentioned domain word embeddings. BIBREF16
From these resources, the following domain-specific features were generated:
INLINEFORM0 -grams generalized over domain terms (or domain generalized INLINEFORM1 -grams, for short): INLINEFORM2 -grams where words or phrases representing a medication (from our medication list) or an adverse drug reaction (from the ADR lexicon) are replaced with <MED> INLINEFORM3 and <ADR>, respectively (e.g., <MED> INLINEFORM4 makes me);
Pronoun Lexicon features: the number of tokens from the Pronoun lexicon matched in the tweet;
domain word embeddings: the sum of the domain word embeddings for all tokens in the tweet;
domain word clusters: presence of tokens from the domain word clusters.
Sentiment Lexicon Features
We generated features using the sentiment scores provided in the following lexicons: Hu and Liu Lexicon BIBREF17 , Norms of Valence, Arousal, and Dominance BIBREF18 , labMT BIBREF19 , and NRC Emoticon Lexicon BIBREF20 . The first three lexicons were created through manual annotation while the last one, NRC Emoticon Lexicon, was generated automatically from a large collection of tweets with emoticons. The following set of features were calculated separately for each tweet and each lexicon:
the number of tokens with INLINEFORM0 ;
the total score = INLINEFORM0 ;
the maximal score = INLINEFORM0 ;
the score of the last token in the tweet.
We experimented with a number of other existing manually created or automatically generated sentiment and emotion lexicons, such as the NRC Emotion Lexicon BIBREF21 and the NRC Hashtag Emotion Lexicon BIBREF22 (http://saifmohammad.com/ WebPages/lexicons.html), but did not observe any improvement in the cross-validation experiments. None of the sentiment lexicon features were effective in the cross-validation experiments on Task 1; therefore, we did not include them in the final feature set for this task.
Official Submissions
For each task, our team submitted three sets of predictions. The submissions differed in the sets of features and parameters used to train the classification models (Table TABREF32 ).
While developing the system for Task 1 we noticed that the results obtained through cross-validation on the training data were almost 13 percentage points higher than the results obtained by the model trained on the full training set and applied on the development set. This drop in performance was mostly due to a drop in precision. This suggests that the datasets had substantial differences in the language use, possibly because they were collected and annotated at separate times. Therefore, we decided to optimize the parameters and features for submission 1 and submission 2 using two different strategies. The models for the three submissions were trained as follows:
Submission 1: we randomly split the development set into 5 equal folds. We trained a classification model on the combination of four folds and the full training set, and tested the model on the remaining fifth fold of the development set. The procedure was repeated five times, each time testing on a different fold. The feature set and the classification parameters that resulted in the best INLINEFORM0 were used to train the final model.
Submission 2: the features and parameters were selected based on the performance of the model trained on the full training set and tested on the full development set.
Submission 3: we used the same features and parameters as in submission 1, except we trained an ensemble of three models, varying the class distribution in the sub-sampling procedure (1:2, 1:3, and 1:4).
For Task 2, the features and parameters were selected based on the cross-validation results run on the combination of the training and development set. We randomly split the development set into 3 equal folds. We trained a classification model on the combination of two folds and the full training set, and tested the model on the remaining third fold of the development set. The procedure was repeated three times, each time testing on a different fold. The models for the three submissions were trained as follows:
Submission 1: we used the features and parameters that gave the best results during cross-validation.
Submission 2: we used the same features and parameters as in submission 1, but added features derived from two domain resources: the ADR lexicon and the Pronoun lexicon.
Submission 3: we used the same features as in submission 1, but changed the SVM C parameter to 0.1.
For both tasks and all submissions, the final models were trained on the combination of the full training set and full development set, and applied on the test set.
Results and Discussion
Task 1 (Classification of Tweets for ADR)
The results for our three official submissions are presented in Table TABREF39 (rows c.1–c.3). The best results in INLINEFORM0 were obtained with submission 1 (row c.1). The results for submission 2 are the lowest, with F-measure being 3.5 percentage points lower than the result for submission 1 (row c.2). The ensemble classifier (submission 3) shows a slightly worse performance than the best result. However, in the post-competition experiments, we found that larger ensembles (with 7–11 classifiers, each trained on a random sub-sample of the majority class to reduce class imbalance to 1:2) outperform our best single-classifier model by over one percentage point with INLINEFORM1 reaching up to INLINEFORM2 (row d). Our best submission is ranked first among the nine teams participated in this task (rows b.1–b.3).
Table TABREF39 also shows the results for two baseline classifiers. The first baseline is a classifier that assigns class 1 (ADR) to all instances (row a.1). The performance of this baseline is very low ( INLINEFORM0 ) due to the small proportion of class 1 instances in the test set. The second baseline is an SVM classifier trained only on the unigram features (row a.2). Its performance is much higher than the performance of the first baseline, but substantially lower than that of our system. By adding a variety of textual and domain-specific features as well as applying under-sampling, we are able to improve the classification performance by almost ten percentage points in F-measure.
To investigate the impact of each feature group on the overall performance, we conduct ablation experiments where we repeat the same classification process but remove one feature group at a time. Table TABREF40 shows the results of these ablation experiments for our best system (submission 1). Comparing the two major groups of features, general textual features (row b) and domain-specific features (row c), we observe that they both have a substantial impact on the performance. Removing one of these groups leads to a two percentage points drop in INLINEFORM0 . The general textual features mostly affect recall of the ADR class (row b) while the domain-specific features impact precision (row c). Among the general textual features, the most influential feature is general-domain word embeddings (row b.2). Among the domain-specific features, INLINEFORM1 -grams generalized over domain terms (row c.1) and domain word embeddings (row c.3) provide noticeable contribution to the overall performance. In the Appendix, we provide a list of top 25 INLINEFORM2 -gram features (including INLINEFORM3 -grams generalized over domain terms) ranked by their importance in separating the two classes.
As mentioned before, the data for Task 1 has high class imbalance, which significantly affects performance. Not applying any of the techniques for handling class imbalance, results in a drop of more than ten percentage points in F-measure—the model assigns most of the instances to the majority (non-ADR) class (row d). Also, applying under-sampling with the balanced class distribution results in performance significantly worse ( INLINEFORM0 ) than the performance of the submission 1 where under-sampling with class distribution of 1:2 was applied.
Error analysis on our best submission showed that there were 395 false negative errors (tweets that report ADRs, but classified as non-ADR) and 582 false positives (non-ADR tweets classified as ADR). Most of the false negatives were due to the creative ways in which people express themselves (e.g., i have metformin tummy today :-( ). Large amounts of labeled training data or the use of semi-supervised techniques to take advantage of large unlabeled domain corpora may help improve the detection of ADRs in such tweets. False positives were caused mostly due to the confusion between ADRs and other relations between a medication and a symptom. Tweets may mention both a medication and a symptom, but the symptom may not be an ADR. The medication may have an unexpected positive effect (e.g., reversal of hair loss), or may alleviate an existing health condition. Sometimes, the relation between the medication and the symptom is not explicitly mentioned in a tweet, yet an ADR can be inferred by humans.
Task 2 (Classification of Tweets for Medication Intake)
The results for our three official submissions on Task 2 are presented in Table TABREF41 (rows c.1–c.3). The best results in INLINEFORM0 are achieved with submission 1 (row c.1). The results for the other two submissions, submission 2 and submission 3, are quite similar to the results of submission 1 in both precision and recall (rows c.2–c.3). Adding the features from the ADR lexicon and the Pronoun lexicon did not result in performance improvement on the test set. Our best system is ranked third among the nine teams participated in this task (rows b.1–b.3).
Table TABREF41 also shows the results for two baseline classifiers. The first baseline is a classifier that assigns class 2 (possible medication intake) to all instances (row a.1). Class 2 is the majority class among the two positive classes, class 1 and class 2, in the training set. The performance of this baseline is quite low ( INLINEFORM0 ) since class 2 covers only 36% of the instances in the test set. The second baseline is an SVM classifier trained only on the unigram features (row a.2). The performance of such a simple model is surprisingly high ( INLINEFORM1 ), only 4.7 percentage points below the top result in the competition.
Table TABREF42 shows the performance of our best system (submission 1) when one of the feature groups is removed. In this task, the general textual features (row b) played a bigger role in the overall performance than the domain-specific (row c) or sentiment lexicon (row d) features. Removing this group of features results in more than 2.5 percentage points drop in the F-measure affecting both precision and recall (row b). However, removing any one feature subgroup in this group (e.g., general INLINEFORM0 -grams, general clusters, general embeddings, etc.) results only in slight drop or even increase in the performance (rows b.1–b.4). This indicates that the features in this group capture similar information. Among the domain-specific features, the INLINEFORM1 -grams generalized over domain terms are the most useful. The model trained without these INLINEFORM2 -grams features performs almost one percentage point worse than the model that uses all the features (row c.1). The sentiment lexicon features were not helpful (row d).
Our strategy of handling class imbalance through class weights did not prove successful on the test set (even though it resulted in increase of one point in F-measure in the cross-validation experiments). The model trained with the default class weights of 1 for all classes performs 0.7 percentage points better than the model trained with the class weights selected in cross-validation (row e).
The difference in how people can express medication intake vs. how they express that they have not taken a medication can be rather subtle. For example, the expression I need Tylenol indicates that the person has not taken the medication yet (class 3), whereas the expression I need more Tylenol indicates that the person has taken the medication (class 1). In still other instances, the word more might not be the deciding factor in whether a medication was taken or not (e.g., more Tylenol didn't help). A useful avenue of future work is to explore the role function words play in determining the semantics of a sentence, specifically, when they imply medication intake, when they imply the lack of medication intake, and when they are not relevant to determining medication intake.
Conclusion
Our submissions to the 2017 SMM4H Shared Tasks Workshop obtained the first and third ranks in Task1 and Task 2, respectively. In Task 1, the systems had to determine whether a given tweet mentions an adverse drug reaction. In Task 2, the goal was to label a given tweet with one of the three classes: personal medication intake, possible medication intake, or non-intake. For both tasks, we trained an SVM classifier leveraging a number of textual, sentiment, and domain-specific features. Our post-competition experiments demonstrate that the most influential features in our system for Task 1 were general-domain word embeddings, domain-specific word embeddings, and INLINEFORM0 -grams generalized over domain terms. Moreover, under-sampling the majority class (non-ADR) to reduce class imbalance to 1:2 proved crucial to the success of our submission. Similarly, INLINEFORM1 -grams generalized over domain terms improved results significantly in Task 2. On the other hand, sentiment lexicon features were not helpful in both tasks.
.2
Appendix
We list the top 25 INLINEFORM0 -gram features (word INLINEFORM1 -grams and INLINEFORM2 -grams generalized over domain terms) ranked by mutual information of the presence/absence of INLINEFORM3 -gram features ( INLINEFORM4 ) and class labels ( INLINEFORM5 ): INLINEFORM6
where INLINEFORM0 for Task 1 and INLINEFORM1 for Task 2.
Here, <ADR> INLINEFORM0 represents a word or a phrase from the ADR lexicon; <MED> INLINEFORM1 represents a medication name from our one-word medication list.
4 Task 1
1. me
2. withdraw
3. i
4. makes
5. <ADR> INLINEFORM0 .
6. makes me
7. feel
8. me <ADR>
9. <MED> INLINEFORM0 <ADR>
10. made me
11. withdrawal
12. <MED> INLINEFORM0 makes
13. my
INLINEFORM0
14. <MED> INLINEFORM0 makes me
15. gain
16. weight
17. <ADR> INLINEFORM0 and
18. headache
19. made
20. tired
21. rivaroxaban diary
22. withdrawals
23. zomby
24. day
25. <MED> INLINEFORM0 diary
Task 2
1. steroids
2. need
3. i need
4. took
5. on steroids
6. on <MED>
7. i
8. i took
9. http://someurl
10. @username
11. her
12. on
13. him
INLINEFORM0
14. you
15. he
16. me
17. need a
18. kick
19. i need a
20. she
21. headache
22. kick in
23. this <MED>
24. need a <MED>
25. need <MED>

Question: why do they think sentiment features do not result in improvement?
Answer: Because sentiment features extracted the same information as other features.

did not observe any improvement in the cross-validation experiments

Question: what was the size of the datasets?
Answer: 10822, 4845

training set containing 10,822 tweets and a development set containing 4,845 tweets, test set of 9,961 tweets was provided without labels, training set containing 8,000 tweets and a development set containing 2,260 tweets, test set of 7,513 tweets

Question: what were the evaluation metrics?
Answer: micro-averaged F-score of the class 1 (intake) and class 2 (possible intake)

F-score for class 1 (ADR), micro-averaged F-score of the class 1 (intake) and class 2 (possible intake)

Question: what were their results on both tasks?
Answer: Unanswerable

0.435 on Task1 and 0.673 on Task2.

Question: what domain-specific features did they train on?
Answer: INLINEFORM0 -grams generalized over domain terms, Pronoun Lexicon features, domain word embeddings, domain word clusters

INLINEFORM0 -grams generalized over domain terms, Pronoun Lexicon features, domain word embeddings, domain word clusters

Question: what are the sentiment features used?
Answer: the number of tokens with INLINEFORM0, the total score = INLINEFORM0, the maximal score = INLINEFORM0, the score of the last token in the tweet

The following set of features were calculated separately for each tweet and each lexicon:

the number of tokens with INLINEFORM0 ;

the total score = INLINEFORM0 ;

the maximal score = INLINEFORM0 ;

the score of the last token in the tweet.

Question: what surface-form features were used?
Answer: INLINEFORM0 -grams, General-domain word embeddings, General-domain word clusters, Negation: presence of simple negators, the number of tokens with all characters in upper case, the number of hashtags, presence of positive and negative emoticons, whether the last token is a positive or negative emoticon, the number of elongated words, presence of exclamation and question marks, whether the last token contains an exclamation or question mark

INLINEFORM0 -grams, General-domain word embeddings, General-domain word clusters, Negation, Twitter-specific features, Punctuation