Feeling Better? Analyzing the Effects of Social Media Engagement on Depression Using Natural Language Processing
By: Aadit Barua
Tags: NLP, Depression, Sentiment Analysis, Health, Data Science

Feeling Better? Analyzing the Effects of Social Media Engagement on Depression Using Natural Language Processing Aadit Barua Follow Apr 13 · 6 min read

Photo by Stefano Pollio on Unsplash

Aadit Barua, Westlake High School, Austin, TX

A global challenge

Depression is a mental health condition that affects over 250 million people, according to the World Health Organization (WHO). The situation has become much worse due to COVID-19; the World Economic Forum (WEF) quotes a study that found one out of every three individuals across multiple countries to be dealing with some level of depression during the pandemic. Many people fighting depression turn to online communities for support and help. Depression can vary from mild to severe, and online platforms have emerged to serve the needs of individuals with this ailment. I want to explore various issues related to depression using natural language processing (NLP) techniques on social media data.

What, if anything, separates messages in depression forums from those posted in forums dealing with suicidal thoughts and self-harm (STSH)? Are there any obvious indicators of a more serious level of depression that can be quickly detected in social media posts?

What are the dominant topics in messages involving severe depression?

Does joining an online community focused on depression help individuals feel better over time? If yes, how does the engagement level affect such benefits?

I use classification, topic modeling, and sentiment analysis to address these questions. What I find is a bit surprising and yet highly encouraging. Contrary to common wisdom, the differences between messages posted in depression and STSH forums are subtle, with the former emphasizing job-related issues - which could be attributed to COVID-19 — and the latter focusing more on family and friends. The good news is that on average, participants in both forums feel better from the time a thread was created to when the last message was posted in that thread. Further, the improvement has a strong positive correlation with engagement, i.e., the number of messages posted by others on the thread. Taken together, my findings suggest that these online platforms can be extremely valuable for those dealing with depression.

My method

Figure 1 shows a summary of my analysis. First, I classify messages posted in online communities based on the level of depression. An accurate model would help detect early signs of severe depression and the dangers associated with such a condition. Second, if the classification model works reasonably well, I will study the differences in topics in the two forums’ messages. Third, I want to understand if users who start a new topic or thread in the depression forum feel better over time due to the interactions. Last but not least, I will run a correlation between the change in the sentiment score and the engagement level to find the extent to which more interactions with others may help someone feel better.

Figure 1: Understanding the severity of depression and the role of online communities

Data and analysis

I obtained the data from two public online forums, one focusing on depression and the other on “suicidal thoughts and self-harm” (STSH). I used a neural network to distinguish between messages posted in the depression and the STSH forums. I created a balanced data set, even though there are significantly more messages posted in the depression forum. The code for the neural network with text is the same that I used in an earlier article on sentiment analysis: https://medium.com/swlh/adding-context-to-unsupervised-sentiment-analysis-7b6693d2c9f8

Table 1: Confusion matrix

As shown in Table 1, the model’s overall accuracy is 81.3%, which is quite acceptable. Moreover, the errors on the two classes are reasonably balanced, with slightly lower errors on predicting posts in STSH. Some oversampling on the STSH posts may further lower that error rate.

Topic modeling on original posts

To take a deeper dive into the differences between the two forums, instead of just looking at high TF-IDF words, I performed topic modeling (after all, there are a lot fewer topics than words). I tried 4, 5, and 6 topics and settled on five after some trial and error. The topics are related to (i) jobs (either actual loss or the fear of loss), (ii) family, (iii) dark themes such as physical harm, (iv) friends, and (v) hope and positive thoughts. Table 2 shows the average topic percentages in messages posted in the two forums. It would be natural to think that dark themes will dominate the STSH forum. However, the reality is more subtle and nuanced. Dark themes have nearly identical and small shares of posts in the two forums. The depression forum has many posts with jobs as the main topic, while STSH posts are more focused on family and friends.

Table 2: Topics in the two forums

Do the sentiments of the participants improve over time?

An important question is whether users who create an original thread to share their stories and seek help or support feel better as they interact with others. I tracked the difference in sentiments of messages posted across time by a user who started a new thread:

Change in sentiment = Sentiment score in the last post by the user who created the thread — sentiment score of the first post in the thread by the same user. The code is shared on GitHub at

https://github.com/AdiB2002/Depression-Analysis/blob/main/Changing%20Sentiments.py. The results are shown in Table 3.

Table 3: Changes in sentiment scores over the life of a thread

There is a significant increase in the sentiment scores across time in both forums, which is highly encouraging. For example, if a user had a sentiment score of -.14 in the first post, his/her last post on that thread had an average sentiment score of .18 — .14 = .04 (slightly positive). Similarly, for the user in the STSH forum, the final post would have a sentiment score of .30 — .35 = -.05, which is slightly negative, but which represents a substantial decrease in the negative tone in the first post. It is evident that users who come to these communities feel better as they interact with others who empathize and offer different types of help.

Does the level of engagement matter?

Not all threads experience the same levels of engagement and interaction. To test if the number of messages posted by others in a thread makes the thread creator feel better, I ran a simple correlation between the change in sentiment and the number of messages in the thread, not counting the messages posted by the thread creator. The overall correlation is 0.39, with a significance level of less than 1%. I have not had the chance to dig deep into whether it is just encouragement or real help that makes users feel better, but it is safe to conclude that more engagement has a significantly positive effect.

Conclusion

Depression is a serious mental health issue across the world, and those who suffer from it are often hesitant to discuss their situation with others in face-to-face conversations. However, many turn to online communities perhaps due to the anonymity such platforms provide. I studied two forums that focus on two different levels of depression. Using a neural network-based classification with NLP, I showed that we can predict the forum in which a message is posted, which indicates the severity of depression. The differences in the messages are surprisingly subtle yet recognizable, with no obvious emphasis on dark themes in the STSH forum. The most encouraging finding is that people feel better from participating in these online forums and that such improvement is correlated with the level of interactions on a thread.

My analysis is based on one online platform from a country that has managed the COVID-19 pandemic relatively well. It will be important to test the ideas above with other online communities to find if the results are similar, and whether such platforms are equally successful.