Using Sentiment analysis to predicting Depression on Social Media
Tags: Machine Learning, NLP, Naturallanguageprocessing, Sentiment Analysis

image @ learn.g2.com

Methods in predictive techniques to model mental well-being, and for understanding health outcomes

Motivation:

According to WHO:

Globally, more than 264 million people of all ages suffer from depression.

Depression is a leading cause of disability worldwide and is a major contributor to the overall global burden of disease.

Close to 800 000 people die due to suicide every year. Suicide is the second leading cause of death in 15–29-year-olds.

Although there are known, effective treatments for mental disorders, between 76% and 85% of people in low- and middle-income countries receive no treatment for their disorder.

However it doesn’t have to be this way

Social media provides an unprecedented opportunity to transform early depression intervention services, particularly in young adults.

Every second, approximately 6,000 Tweets are tweeted on Twitter, which corresponds to over 350,000 tweets sent per minute, 500 million tweets per day and around 200 billion tweets per year [refer].

By using linguistics and behavioral cues it is possible to detect mood and psychosocial disorders.

In addition to mental disorders, these approaches are starting to assess related symptomatology, such as self-harm, stress, and the severity of mental illness without the use of in-person, clinical assessment.

In current case-study, we’ll analysis how NLP techinques can be used to predict mental heath (here we’ll take depression as an example) using social media (twitter tweets).

We’ll be using kaggle dataset (here).

Steps:

Data cleaning/Preparation EDA: Data visualization and Understanding Model building:

we’ll explore following methods: LogisticRegression, MultinomialNB, DecisionTreeClassifier(), RandomForestClassifier(), KNeighborsClassifier(), XGBClassifier and SVC

Let’s look at the dataset which contains tweets and positive negative labels:

We’ll then clean data, by using below steps:

remove punctuation and special characters

remove stop words

split text and tokenize

apply stemmer

remove small words

First let’s see frequency of words in positive tweets and negative tweets separately.

1. Positive tweets:

positive tweet has “love”, “Thank”, “today”, “know, “work” as more frequent words:

2. Negative tweets:

negative tweet has “miss”, “work” “today”, “time,” sleep, “sorry” as more frequent words:

We’ll split our data to test and train. And then tokenize using TF-IDF vectorizer. TF-IDF is a measure of originality of a word by comparing number of times a word appears in a docs with number of docs the word appear.

After vectorizing our data-set looks like this:

We’ll now run the following set of classifiers:

Which we can compare and see the accuracy score:

Compare Accuracies

Conclusion

We are able to achieve good accuracy with all the models, (we also need to validate the possibility if the model has been inaccuracies since our data is likely to be affected by data imbalanced by exploring data balancing techniques).

While classification techniques are good for bench-marking, for more generalized models it would make more sense to explore Long Short Term Memory network (LSTM) model along with Convolutional neural networks (CNNs). [refer]

My kaggle Notebook can be found here

Linkedin article can be found here