How to Protect Your Mental Health During the COVID-19?
By: Jenn Huang
Tags: Machine Learning, Data Science, NLP, Coronavirus, Covid 19

This project has won the 2nd Prize and the Most Valuable Dataset in the UC Davis MSBA COVID-19 Challenge. This is a collaborative work of Yijun Huang, Linyan Dai, Kayla Zou, and Haonan Wang. The step-by-step guidance of python coding has been shared at Github: https://github.com/jennnh/4G_COVID-19_Challenge

Overview

The COVID-19 pandemic has brought a great threat to people’s physical health and well-being. However, many mental illnesses were aroused simultaneously without being noticed. In the USA, every single person has been faced with a mental health issue in this COVID crisis, caused by jobs disappearing, businesses shutting down, and separation from friends and family. What’s worse, former congressman Patrick J Kennedy says calls at suicide hotlines have increased by 800 percent as resources shift to COVID-19 relief.

In this case, this project aims to make detailed suggestions about activities to release people’s emotional strain during the quarantine period. In detail, our team first scraped Tweets with the hashtag #StayHome on Twitter in various regions and dates as the data sources. Second, we leveraged sentiment analysis and word clouds to understand and visualize the emotions of people. Third, we found 13 activity-related topics using topic modeling and across-analyzed them with other columns to identify factors contributing to people’s positive and negative emotions. In the end, we made suggestions to people about what activities they should participate in and what activities they should avoid to keep happy.

Web Scraping

First of all, we needed to scrape data from Twitter through web scraping. We didn’t use the free API of Twitter because it only provided Tweets within the last 7 days but we needed Tweets since one month ago. Therefore, we used a package called “GetOldTweets3”, which provided enough Tweets for our analysis.

We wanted to select some representative cities to conduct our analysis. To that end, we selected 10 cities that have both a large population and a large number of confirmed cases based on data from John Hopkins University on April 13th, 2020 to scrape. The 10 cities are New York City (New York), Boston (Massachusetts), Chicago (Illinois), Detroit (Michigan), Los Angeles (California), Houston (Texas), Newark (New Jersey), Miami (Florida), Philadelphia (Pennsylvania), and New Orleans (Louisiana).

Afterward, we scraped the Tweets that were posted nearby those cities. For each Tweet, we collected the user’s name, text, the date and time of the Tweet, the number of retweets, the number of favorites, and the HashTags. After scraping the data, we also did a stratified sampling based on the proportion of new users in each city. New users are defined by whether it is the first time that user posts about #StayHome.

Exploratory Data Analysis (EDA)

After getting the dataset, we started to do some exploratory analysis. The dataset that we have collected contains Tweets from March 5th, 2020 to April 11th, 2020, and it includes 11631 Tweets and 6741 unique users. The graph below shows the distribution of the number of Tweets in different cities across time.

The Number of Tweets in Different Cities across Time

According to the graph, in most cities, there is a sudden increase in Tweets number around late March. This is when most cities announced the shelter in place order. For example, the grey area increased greatly after the blue line, where New York just announced the order. This means that the shelter in place announcement increased people’s awareness and attention of COVID-19, so people started to attach great importance to “stay home” and talked about it more on Twitter. After several days, the number of Tweets decreased, which indicates that people started to pay less attention to that and they got accustomed to staying at home.

Natural Language Processing Analysis

Text Preprocessing

After the EDA, we have done text preprocessing to improve the performance of NLP models, including:

Making everything lower-case.

Text decontraction. For instance, replace “won\’t” with “will not”, replace “\’ve” with “ have”, and replace “\’d” with “ would”.

Removing strange symbols, URLs, emojis, and #hashtags

Word lemmatization using NLTK stem. Lemmatization is the process of grouping together the different inflected forms of a word so they can be analyzed as a single item. For instance, after Lemmatization, “cries” would be “cry”.

The comparison of a review between before text preprocessing and after text preprocessing is as follows:

Before

A sunny day haiku #quarantinepoetry for an old friend down in Texas summerdaenen THANK YOU!!! VENMO @randsomnotes for a poem of your very own! #quarantinelife #stayhome #summer @New York, New York https://www.instagram.com/p/B-29p1rp1la/?igshid=12yeaszfhp7o2

After

A sunny day haiku for an old friend down in Texas summerdaenen THANK YOU!!! VENMO @randsomnotes for a poem of your very own! @New York, New York

Sentiment Analysis

We used the package “Textblob” to assign each Tweet a sentiment score called “polarity” to measure how positive or how negative the review is. Afterward, we classified sentiment into positive and negative ones with a threshold of 0. Tweets with a polarity score larger than 0 are positive ones, with a polarity below 0 are negative ones, and with a polarity equal to 0 are neutral ones. The proportion of Tweets with different emotions is as follows:

Based on the graph, half of the Tweets are positive and only a small part of the Tweets are negative. It means that not a lot of people are doing bad while staying at home because of COVID-19.

We also draw a line graph to see how sentiment changed as time went by, and the chart shows that people started to feel a little better after being suited to this situation.

Average Sentiment of Tweets across Time

World Cloud

To understand what are the most common words in negative Tweets and positive Tweets, we created two word clouds separately in these two emotion categories.

Left: Positive World Cloud / Right: Negative World Cloud

After removing common words in both of these word clouds, we can see that Tweets in the positive ones often contain positive words like “safe”, “friend”, or “happy”. There are also words like “video”, “music” that imply positive activities. On the contrary, in the negative word cloud, we can infer that Tweets are mainly expressing worries about being sick, or complaining about being bored after staying at home for a long time. Plus, the word “game” also shows up in the negative word cloud surprisingly.

Topic Modeling

The third part of the NLP analysis is the topic modeling. First of all, we preprocessed texts further to improve the performance of topic modeling, including:

Tokenization: Tokenizing each sentence into a list of words

Removing punctuations.

Removing stop words using the NLTK library. Stop words are the very common words like “if”, “but”, “we”, “he”, and “they”. We also added multiple new common words based on this corpus.

After the preprocessing, we used Latent Dirichlet Allocation via Mallet (LdaMallet) to extract topics from all Tweets. First, we tried out different numbers of topics and used coherence values to choose the best number of topics. After determining to choose 30 topics, we got the keywords for each topic and filtered the 13 ones that were related to people’s activities for further analysis.

Afterward, we combined the results from sentiment analysis and topic modeling to analyze which topics will contribute to people’s positive emotions and which will lead to negative ones. More specifically, we calculated the average polarity for each topic and visualized the result in the following Treemap.

The Treemap Showing the Average Sentiment of 13 Activity-related Topics

The area and color are determined by the polarity of each topic, which means the larger and the darker the area, the more positive the topics. Therefore, activities at the top-left are the most positive ones, such as watching videos, donating, and communicating through word and message. On the other hand, activities at the bottom-right are the least positive ones, such as cooking dinner and cleaning the house.

Besides the sentiment of topics, we also analyzed what are the most popular activities across different cities. Therefore, we listed the top three activities in each city based on the number of Tweets related to the topic.

The Top3 Activities across Different Cities

According to the chart, we got several interesting findings:

Watching videos, workouts, and delivery are popular activities in many cities.

People in New York prefer staying with their friends and family, while people in Los Angeles most like watching shows and movies.

Communication through words, messages, and pictures is very popular in Boston.

People in Chicago especially love to say prayers.

Besides topics, we want to know more about specific words that influence people’s feelings, so we picked a list of high-frequency words from these topics and calculated the average polarity of all Tweets containing each word. The bar chart ranks the words according to its polarity. The words colored light red are the most positive ones, and then the ones colored dark red and grey.

The Average Polarity of All Tweets Containing a Keyword

Consistent with the topic modeling result, watching videos, virtual connection, and going to the gym are related to extremely positive sentiments. Besides, activities like singing songs, receiving deliveries, drinking wine, and saying a prayer are also very positive. It is worth noting that the word “game” ranks the last in the bar chart, which disagrees with our knowledge that playing games helps people feel better.

Recommendations

Based on the analysis of topics and single words, we give the following suggestions to people who feel mentally discomfort:

You can spend more time on these activities: listening to and singing songs, watching movies or TV series, workout, reading, and virtually connecting with your family and friends even though you are physically distant.

Besides, these activities are surprisingly very beneficial to your emotions based on our analysis: