My First Adventures in NLP
By: Britton Winterrose
Tags: Machine Learning, NLP, Naturallanguageprocessing, Sentiment Analysis, Data Visualization

My First Adventures in NLP

A journey in exploring sentiment analysis as a “black box” and breaking it down to find insights.

November 15, 2018 I set out on a mission to create a drug-review sentiment analysis model.

It began as most ideas do… with a problem.

For the past 24 months my wife has struggled through a major depressive episode. Treatment protocol has taken her from one medication trial (and failure) to the next. Each drug trial takes anywhere between 8 to 16 weeks depending on titration, follow-up appointments, and tapering off. Throughout this process she’s experienced the whole gambit of side effects. Nausea, fatigue, drowsiness, and severe anxiety to name a few. Some drugs helped a little with the depression. Others made it noticeably worse. Terrifyingly worse.

In those moments I felt powerless to help the one that I love. My only course of action was to contact the physician and leave a message or note. And often by the time I heard back from the physician the side effects would have changed.

I don’t blame the physician for the level of responsiveness. They communicated as best they could given the volume of individuals that they help in their practice. But the thought often crossed my mind “Why even (call or) email them? It’s not like they’re going to get this message soon enough to do anything about it.”

Often we’d end up waiting to report back to the doctor until the follow-up appointment. But it is hard to remember how difficult each day, or week, of trying a new medication was when you’re on the spot. Especially when it was 8 weeks into using a new drug. This creates a slow, inconsistent and labor intensive medication-experience feedback cycle.

These breakdowns & bottlenecks in doctor — patient communication slow down the treatment process. If doctors can find the correct medications immediately then patients care outcomes improve. Correct drug fit means fewer followup visits, which helps to reduce the cost of patient care. A problem with real costs & lives attached.

So what if I could create a system to allow health care providers to identify effective / ineffective drugs more quickly?

An idea: medication experience feedback analysis

Quantified reviews create clean simple data. Data that can visualized, analyzed, monitored, and compared.

By gathering daily patient drug-feedback and performing sentiment analysis one could:

Examine the patient’s experience with a medication over the trial period to track how long the drug takes to begin working. (Or whether it works at all)

Identify when patients are struggling and notify the physician to provide support.

Evaluate drugs over time and across various patient types to monitor non-clinical performance.

Provide doctors with a chart level view of the patient’s ongoing experience during follow-up visits.

But this system’s success would hinge on one key component. A model that can consistently and accurately predict the sentiment of a patient’s drug review.

So that is what I set out to see if I could build.

Into the Unknown

A bit like wandering down an unfamiliar road.

Walking into this project I knew very little about NLP and sentiment analysis other than what it can do conceptually. But I needed some quality data to start with.

The internet delivered some terrific data. The “Drug Review (Drugs.com) Dataset” from the UCI Machine Learning Repository contributed by Felix Gräßer & Surya Kallumadi.

“The dataset provides patient reviews on specific drugs along with related conditions and a 10 star patient rating reflecting overall patient satisfaction. The data was obtained by crawling online pharmaceutical review sites. (Drugs.com)”

An excellent dataset that contains the text reviews, a rating, as well as category fields for condition and medication.

I started by importing & exploring the dataset. Once satisfied that I grasped the structure of the data I began looking for guidance on sentiment analysis.

Attempt #1 — A shot in the dark

I googled, read many, and selected a simple starting tutorial to launch my NLP journey. My first attempt at NLP began with taking a bit of code from the “Sentiment Analysis with Python” tutorial and applying it to my dataset. (See my notebook here) I quickly learned that adapting even the best written code to a new dataset can be quite the challenge. The RegEx provided for cleaning the data was insufficient, so I wrote and tested my own.

After normalizing and vectorizing the review text I binned the review-ratings from 11 categories into three. “Positive”, “Negative”, and “Neutral”. Then using scikit-learn’s Logistic Regression I trained a model and checked the accuracy of the model’s performance.

After a small amount of tuning, this model produced a final accuracy of 81.95%. At this point I was uncertain how to measure the performance of this model and unsure whether accuracy alone was a valid metric.

But as I inspected the weights for each of the top and bottom 10 features (words) I realized that there were some discrepancies that I was unable to explain.

Hmm. These negative weighted words seem pretty positive…

I’m not the type to give up. So, I needed to learn more and sought out after a more in depth tutorial.

Attempt #2 — Deconstructing the Black Box

After some questioning with one of my friends he recommended a post from Insight AI’s Emmanuel Ameisen. Using that post as a guide and Leonardo Apolonio’s “How-to-solve-NLP” code as a framework I set off on my next journey in NLP with my freshly inspected dataset. (My notebook is here.)

I first reviewed the original paper that was published using the dataset. I’ve included notes about the methodology for data collection, preprocessing, tokenizing, and modeling used in that study.

As I worked through the tutorial I followed a simple process of “stop and learn” anytime there was ML vocabulary I didn’t understand. I learned about embedding, Word2Vec, Bag-of-Words, corpus, tokenizing, normalizing, non-ASCII characters, stemming, lemmatizing, and vectorizing in data preparation. This process is time consuming but effective.

My mindset: Make the black box work, then tear apart the black box

My goal with each of these tutorials is simple: make it work, then gain understanding. This process works well for people like me who could be described as “mechanically inclined”. It allows us to build an understanding of the entire system’s structure first, then take a deeper dive into knowledge of each part. I find that this method helps me to retain what I learn. My brain locks in the explanation of the details because they are perceived as relevant and essential parts of the whole. (Here’s a short presentation I did on this method.)

“Take chances, make mistakes, GET MESSY!” — Ms. Frizzle

The danger of this method is when the working internals are not explored. If the mechanisms of the model are treated like a black box forever the user misses the opportunity to learn.

But did it actually work?

It took a lot of work to get it working cleanly with my dataset but work it did. The tutorial clearly illustrated new-to-me concepts within a framework of evaluating model performance including:

Evaluating the effectiveness of various embeddings through projected visualization.

Utilizing precision, recall, f1, and accuracy for evaluating Logistic Regression performance.

Use of a confusion matrix for model performance visualization.

Visualizing feature inspection through the use of an Important Words plot.

Using Lime, a blackbox explainer, for word2vec to retrieve and inspect features.

The results of all this work?

The highest performing embedding + logreg model was my Bag-of-Words model at 78% accuracy. The CNN ended up coming in at ~75% accurate leaving plenty of room for improvement.