Abstract: Our team, NRC-Canada, participated in two shared tasks at the AMIA-2017 Workshop on Social Media Mining for Health Applications (SMM4H): Task 1 - classification of tweets mentioning adverse drug reactions, and Task 2 - classification of tweets describing personal medication intake. For both tasks, we trained Support Vector Machine classifiers using a variety of surface-form, sentiment, and domain-specific features. With nine teams participating in each task, our submissions ranked first on Task 1 and third on Task 2. Handling considerable class imbalance proved crucial for Task 1. We applied an under-sampling technique to reduce class imbalance (from about 1:10 to 1:2). Standard n-gram features, n-grams generalized over domain terms, as well as general-domain and domain-specific word embeddings had a substantial impact on the overall performance in both tasks. On the other hand, including sentiment lexicon features did not result in any improvement.

Introduction: Adverse drug reactions (ADR)—unwanted or harmful reactions resulting from correct medical drug use—present a significant and costly public health problem. BIBREF0 Detecting, assessing, and preventing these events are the tasks of pharmacovigilance. In the pre-trial and trial stages of drug development, the number of people taking a drug is carefully controlled, and the collection of ADR data is centralized. However, after the drug is available widely, post-marketing surveillance often requires the collection and merging of data from disparate sources, BIBREF1 including patient-initiated spontaneous reporting. Unfortunately, adverse reactions to drugs are grossly underreported to health professionals. BIBREF2 , BIBREF3 Considerable issues with patient-initiated reporting have been identified, including various types of reporting biases and causal attributions of adverse events. BIBREF4 , BIBREF5 , BIBREF6 Nevertheless, a large number of people, freely and spontaneously, report ADRs on social media. The potential availability of inexpensive, large-scale, and real-time data on ADRs makes social media a valuable resource for pharmacovigilance. Information required for pharmacovigilance includes a reported adverse drug reaction, a linked drug referred to by its full, abbreviated, or generic name, and an indication whether it was the social media post author that experienced the adverse event. However, there are considerable challenges in automatically extracting this information from free-text social media data. Social media texts are often short and informal, and include non-standard abbreviations and creative language. Drug names or their effects may be mis-spelled; they may be used metaphorically (e.g., Physics is like higher level maths on steroids). Drug names might have other non-drug related meanings (e.g., ecstasy). An adverse event may be negated or only expected (e.g., I bet I'll be running to the bathroom all night), or it may not apply to the author of the post at all (e.g., a re-tweet of a press release). The shared task challenge organized as part of the AMIA-2017 Workshop on Social Media Mining for Health Applications (SMM4H) focused on Twitter data and had three tasks: Task 1 - recognizing whether a tweet is reporting an adverse drug reaction, Task 2 - inferring whether a tweet is reporting the intake of a medication by the tweeter, and Task 3 - mapping a free-text ADR to a standardized MEDDRA term. Our team made submissions for Task 1 and Task 2. For both tasks, we trained Support Vector Machine classifiers using a variety of surface-form, sentiment, and domain-specific features. Handling class imbalance with under-sampling was particularly helpful. Our submissions obtained F-scores of 0.435 on Task 1 and 0.673 on Task 2, resulting in a rank of first and third, respectively. (Nine teams participated in each task.) We make the resources created as part of this project freely available at the project webpage: http://saifmohammad.com/WebPages/tweets4health.htm.

Question: why do they think sentiment features do not result in improvement?
Answer: Because sentiment features extracted the same information as other features.