SFU Professional Master’s Program in Computer Science
By: Rohan Harode
Tags: Drugs, Machine Learning, NLP, Data Science, Sentiment Analysis

This blog is written and maintained by students in the Professional Master’s Program in the School of Computing Science at Simon Fraser University as part of their course credit. To learn more about this unique program, please visit {sfu.ca/computing/pmp}.

Motivation and Background

The safety of a medical product is evaluated through pre-defined protocols and clinical trials. These clinical studies are carried under regulated conditions on a limited number of aspects and within a specified time constraint. In this process, adverse side effects and potential risks associated with drugs may not be detected before the drug is launched in the market. A preeminent solution to this problem can be achieved by looking at post-marketing drug surveillance methods such as drug review analysis to monitor drug-related issues. Text mining on drug reviews will be useful not only for patients to decide which drug to take, but also for pharmacy companies and clinicians to improve consumer safety by assisting in the reduction of medication errors and obtain valuable summaries of public opinion.

Related Work:

Previous studies have shown that health-related user-generated content is useful from different points of view. One of the benchmark papers in this area was written by Jane Sarasohn-Kahn. It states that users are often looking for stories from “patients like them” on the Internet, which is hard to find among their friends and family. For the past decade, researchers have been analyzing the emotional impact of user experience and the severity of adverse drug reactions by extracting sentiment and semantic information (Sharif, Zaffar, Abbasi, and Zimbra (2014).

Due to a lack of trust and quality of user-expressed medical language, extensive research in the medical and health domain has not been done. Therefore, we aim to build a platform where patients and clinicians can search by symptoms and get drug recommendations, side effects of drugs and obtain insights into patients’ portfolio.

Problem Statement

We understood how Data Science and Text Mining have been of significant importance in the health care industry and aim to answer the following questions through our platform:

How to use sentiment analysis and predictive modelling to recommend the most effective drugs for the given condition?

What is the emotional inclination of users towards a chosen drug?

To discover the most influential age group and gender of the target audience for a chosen drug?

How can we build a real-time sentiment classification and predict the polarity from a drug review text?

These problems are challenging as it required us to do a lot of domain research and understanding of the medicinal market. Besides this, several features and structures of data are not consistent across sites and therefore transformation and integration of the data was a big task. Another big challenge we encountered is that we need to make sure drug names and medical conditions are uniform and reduce data redundancy. For example, on WebMD, a condition is stated as “Cancer of the Pancreas” whereas on Drugs.com it is “Pancreatic Cancer” that has the same meaning.

Data Science Pipeline

Data Sources and Collection

The first step in the data scinec pipeline is to obtain datasets. We scraped 3 healthcare websites (Drugs.com, WebMD, DrugLib) that contained drug information like drug name, side effects, medical condition, reviews, age, and gender.

We created a Scrapy web crawler using Spider to scrape WebMD. Few features like gender and age group were hard to extract from the web page since text patterns for each user differed. Therefore, we used WebMD’s REST API and extracted the corresponding key-value pair from JSON.

DrugLib was scraped using BeautifulSoup and requests — a python library for processing and downloading HTML files. The only challenge we faced here was to extract age from the user description. For this reason, we used a regular expression to resolve the issue.

We scraped around 580k reviews that had 4229 unique conditions and 11k unique drugs present in them. Below is the data distribution among all 3 websites:

Data source distribution

Due to crawling restrictions on drugs.com, we extracted the data from Kaggle that had drug reviews listed only till 2017.

Data Cleaning and Integration

Cleaning and Filtering:

As we can observe from the table in the data collection section, DrugLib has around 1500 conditions among 4200 reviews which seems quite inconsistent when compared with other data sources. That is because the scraped data was case sensitive, had a lot of misspelled conditions and multiple conditions were grouped into one (“anxiety, panic attacks, nervousness”). To resolve the above issues across all 3 data sources, we performed a lot of cleaning operations on conditions like converting to lowercase, trimming and filtered irrelevant conditions with <span> and “Not Listed” mentioned in them.

Jaccard Similarity and Fuzzy Matching:

As discussed earlier, drug names and conditions varied a lot across 3 data sources and therefore need to be transformed using Entity Resolution methods. Jaccard Similarity technique is used to find a set of matching pairs (drug, condition). Due to this join, there will be multiple pairs whose conditions match but not the drug name and might lead to incorrect results. Therefore, we applied another Entity Resolution method named Fuzzy Matching to match drug names first and assign a new condition to a pair based on the Fuzzy score calculated between new condition and original condition. Below table shows one of the problems encountered and its transformation:

Since most of the conditions on drugs.com are concise and didn’t include stopwords we decided to use drugs.com as our base platform and altered WebMD and DrugLib conditions using the above technique.

We aggregated the matching data from all 3 data sources and stored it in PostgreSQL database.

Exploratory Data Analysis

Correlation Analysis

Since the overall rating for a drug review is not present on WebMD, it is interesting to see the correlation between different types of ratings. We found that drug effectiveness is highly related to satisfaction.

Below joint plot shows how sentiment polarity is highly related to the overall rating using regression line and kernel density fit.

Rating vs Sentiment Score

Users on online health forums can rate the comments of other users through useful count feature. The plot shows a positive increasing trend of the average useful count of a review versus overall rating.

Average Useful Count vs Rating

2. Rating distribution

Pie chart shows how ratings from a scale of 1 to 10 are distributed in our dataset. As we can see the majority of users have given a higher rating as compared to lower ones.

3. Drugs distribution per condition:

The below figure illustrates the conditions which have the maximum number of drugs associated with them. There are 200+ drugs available to treat conditions like birth control, high blood pressure and pain.

Drugs count per condition

4. Most common condition

The following bar plot points out the most common condition among drug users. Birth control, pain, and depression are the top conditions.

Most common conditions among drug users

Data pre-processing & Feature Extraction

Machine learning algorithms cannot work with raw text data and therefore text must be converted to numeric representation or more specifically vectors of numbers.

We chose Bag-of-Words (BoW) models as our feature extraction method which describes the occurrence of words in a document. We experimented with two different approaches to convert text into vectors.

TF-IDF vectorizer: TF-IDF stands for Term frequency-inverse document frequency. The importance of a word increases proportionally to the number of times it appears in the document but is compensated by the frequency of the word in the corpus. Hashing vectorizer: This vectorizer converts a text to a sequence of indexes in a fixed-size hashing space for faster lookup. Therefore, making it suitable for a large dataset and also saves a lot of computation and time.

Predictive Modelling

Sentiment Classification

We trained various machine learning models and deep neural networks to classify the sentiment of review text into 3 main categories (Positive, Neutral, Negative).

We created the target labels for sentiment classification by categorizing the rating as follows:

Rating categorization

Machine learning models:

We chose 6 baseline models: Naïve Bayes, Logistic Regression, Stochastic Gradient Descent classifier, Support Vector Machine and XGBoost for prediction of sentiment polarity. We investigated our models’ performance by training on combinations of different vectorizers and tokenizers. The results are shown in the evaluation section.

Deep learning models:

Multilayer Perceptron: We developed MLP for multi-class softmax classification using Keras. Recurrent Neural Network: We implemented a stacked Bi-directional LSTM using Keras and TensorFlow backend. Long short-term memory (LSTM) is a special kind of RNN that can recognize patterns in a sequence of data and learn long-term dependencies. Thus, we determine it to be suitable for drug review analysis.

2. Review Rating Prediction

We built a 10-class rating classifier using Bi-directional LSTM + Bi-directional GRU using Keras and TensorFlow backend. As part of text preprocessing, we used Keras Tokenizer class and text_to_sequences to transform the text into a sequence of integers.

Model Selection

In applied machine learning, the real challenge is how to choose among a range of different models and what are the considerations we should be concerned about. Model selection is the process of picking one final model out of several candidate models that addresses our problem statement. In our project, both sentiment classification and review rating prediction modules fall under the classification problem. Resampling model selection methods like Random train/test splits and cross-validation techniques were utilized to estimate the performance of our models.

Web Application

We created an interactive web application using Plotly and Dash to demonstrate top-recommended drugs per condition, emotions inclination towards a drug, age-gender distribution, real-time sentiment prediction and collective side effects observed by users. We utilized Dash inbuilt webserver to host our platform. Drug data was fetched from the PostgreSQL database. These visualizations will provide useful insights and at the same time motivation to drug makers by improving the quality of drugs and get a better understanding of larger audiences.

Methodology

Sentiment analysis using VADER

We used NLTK VADER (Valence Aware Dictionary and sentiment Reasoner) sentiment analyzer which is a lexicon and rule-based sentiment analysis tool. Since most of the drug reviews are written in informal language, we decided to use VADER as it works well on the non-technical, social media content and product reviews.

In this approach, each word in the lexicon is rated as to whether it is positive, negative or neutral. The final sentiment value which is an aggregate score is called compound score. The compound score is a metric that calculates the sum of all lexicon ratings and normalizes between -1 to 1. As we can see from the below plot, sentiment polarity is quite uniformly distributed with a peak at 0, which implies that the majority of reviews were classified as neutral.

Sentiment classification using predictive models

We preferred Bag of words model rather than word embeddings as it works better when the dataset is small, and context is very domain-specific which holds true in our case.

TF-IDF: We used NLTK’s pre-trained word tokenizer and Snowball stemmer for text pre-processing and incorporated both unigram and bigrams models. We chose Snowball stemmer (improved version of Porter Stemmer) as it prevents over-stemming, provides speed, precision, and multiple language support.

Hashing: The TF-IDF performs decent in our case but brings two limitations to our approach. First, it is not memory efficient as the vocabulary can become very large. Secondly, it takes longer to converge. Therefore, we tried Hashing vectorizer which overcomes these limitations by applying feature hashing to encode tokens as numerical indexes. With Hashing vectorizer, we used RegExp Tokenizer and took unigram and bigram models into account. Lastly, compared the accuracy of models using the above techniques which are discussed in evaluation.

ML models: To achieve higher accuracy in tree-based models, we used RandomizedSearchCV for hyper-parameter tuning in which it draws a random value during each iteration from the range of specified values for each hyperparameter searched over and evaluate the model with those hyperparameters. The best parameters are as follows:

Best parameters found: {‘subsample’: 0.70, ‘n_estimators’: 200, ‘max_depth’: 7, ‘learning_rate’: 0.99

Deep neural networks: We chose Keras models to train our sentiment classification as it provides us with a lot of flexibility like hyperparameter tuning and designs a network suitable for our data. We also employed techniques like Early Stopping and Model Checkpoint to save the best model with higher validation accuracy.

Here are some of the noteworthy inferences we collected during training:

softmax turned out to be the best activation function for the output layer.

256 input units were optimum for accuracy without overfitting.

Adding additional units or layers to the model resulted in the loss of accuracy.

Emotion Classification

Emotion analysis or Affective Computing is essential to understand human behaviour and underlying opinion. We used the NRC Emotion Lexicon which has around 14k unigrams where each word is associated with 8 universal emotions — anger, fear, anticipation, trust, surprise, sadness, joy, and disgust and two sentiments — negative and positive. The emotion and sentiment values are in binary form.

With the Word-Emotion Association, we tracked emotions in the drug reviews. We matched every word in the review text with unigram in the lexicon file and increased the count of respective emotions for each review.

Finally, we took an average of the aggregated count for each drug to show the overall distribution.

Drug Recommendation

Drug recommender systems can assist the medical care providers with the selection of a suitable medication for the patients. The intuition behind our drug recommendation system is to utilize the overall opinion of the audience through sentiment analysis and predicted review rating from an LSTM model. Since, the sentiment score from VADER was between -1 to 1, therefore we designed a custom rating mapper to normalize the original score to match the overall rating scale (1 to 10).

Visualizing fastText Word Embeddings using t-SNE

t-SNE is used to understand the high-dimensional vector space of word embeddings to interpret relations between vectors captured from the text. After removing stop words from the dataset, we used the pre-trained fastText English Wikipedia model to fit the drug reviews as well as side effects and plot the most influential 100 bigrams to see the relation in word grouping. Similar words are clustered together in the plot which indicates that the distance between these words is low and their semantics are more similar. We found that bigrams in our case make sense like “gained pounds” and “lost pounds” instead of unigrams like “gained”, “lost” and “pounds”. Pairs of consecutive words make tokens more understandable (for example, “gain pounds” is more informative than “gain”).

Evaluation

Model evaluation is an integral part of the data science pipeline. Various evaluation metrics are used to assess the goodness of fit between model and data and to predict how predictions (associated with a specific model and data set) are expected to be accurate.

a. Sentiment Classification

Baseline models: We evaluated our classification models using confusion matrix and accuracy as shown below.