Analyzing the Impact of COVID-19 on Frontline Workers Using Machine Learning
By: Snigdha Ramkumar
Tags: Artificial Intelligence, Machine Learning, Covid 19, NLP, Coronavirus

The recent pandemic drastically affected many lives and changed the way society interacts. The whole world was encouraged to stay in lockdown– except for the frontline and essential workers. As the name suggests, essential workers are those whose jobs are deemed “essential”, without whom the core functions of the economy and society would not function. Among these essential services are the frontline workers, who have to provide their services in person. Frontline workers include, but are not limited to, doctors, nurses, cashiers, fast food workers, agricultural workers, and truck drivers.

OVERVIEW

For this, we undertook a project to study the impact of COVID-19 on frontline workers by analyzing the posts that these frontline workers put on social media. The reason for the same was to analyze the intangible impact — psychological, physical and material — that this crisis was having on these warriors. For this purpose, a novel dataset consisting of posts from various frontline workers was extracted from social media platform, Reddit.

Through the pandemic, frontline workers have been the most impacted because of the nature of their job, poor management, the lack of benefits or monetary compensation, anxiety due to the pandemic, anxiety because of possible job loss, and frustration towards society.

After understanding the general emotions of these workers through their posts, a classifier was developed to detect the users who were at enhanced risk to severe mental distress (such as depression or anxiety leading to suicidal ideation), using models pre-trained on social media posts.

Different models were tested for this purpose, with different features (statistical, syntactic, word and sentence embeddings), and their overall accuracy was compared and contrasted. These trends were visualized using Tableau, leading to interesting conclusions.

DATA COLLECTION

Social platforms make it easier to find like-minded peers, making interaction more likely. Reddit is one such platform, which allows its users to generate content anonymously, with only a username for identity.

For this research, posts and comments were extracted using the Pushshift.io API and communities (“subreddits”) meant for frontline workers were targeted.

From these subreddits, only posts related to coronavirus were extracted, using a keyword-specific search like coronavirus, covid19, corona, wuhan, china flu, covid, mask, social distancing, and pandemic. Related posts between December 1st 2019 and August 24th 2020 were obtained. A new novel dataset with more than 10,000 entries from frontline workers was created.

Post from r/Nurse

SENTIMENT ANALYSIS

For this analysis, the text body as well as the title of the posts were used, and the overall sentiment was extracted using VADER and TextBlob.

On the text posts, VADER displayed the overall sentiment — positive, negative, or neutral — for the entire collection of sentences. There is a fourth value, ‘compound’ which gives the overall sentiment of the text.

With TextBlob, the polarity and subjectivity of the text posts was found, with polarity ranging from -1 to +1, and subjectivity between 0 and 1. Polarity is related to emotions; a high polarity score implies that the text is positive while a low polarity implies negativity.

MENTAL DISTRESS DETECTION

A balanced dataset from Reddit comprising of more than 100,000 posts, prelabelled as suffering from extreme mental distress and not possessing such tendencies was used for building the model.

This dataset was created by extracting posts from r/SuicideWatch, a community where users generally post about their suicidal thoughts. Common themes involve discussing why they want to commit suicide and talking about previous attempts.

Non-suicidal posts, i.e. posts with no suicidal tendencies, were extracted from other popular subreddits, r/books, r/jokes, r/legaladvice, r/casualconversation, and r/college, to create a roughly balanced dataset.

Using this dataset, models were created to detect differences in how the users suffering from severe mental health issues may have different posting patterns and habits compared to other users.