Text mining: What are people doing at home during the Coronavirus quarantine?
By: Feng Lim
Tags: Data Science, Naturallanguageprocessing, Text Mining, Coronavirus, Covid 19

As more countries declare a nationwide shutdown, most of the people are asked to stay at home and quarantined. I wanted to know how people are spending their time and how they are feeling during this “closedown ” period, so I analyzed some tweets in this article, hoping that the data will give me some insights.

Importing and pre-processing the dataset

For the dataset, I extracted 20,000 tweets with “#quarantine” and “#stayhome” hashtags from Twitter using the twitteR library.

After importing the data into R, we need to pre-process and tokenize the tweets into words (tokens) for analysis

tweet_words <- tweets %>%

select(id,

screenName,

text,

created) %>%

mutate(created_date = as.POSIXct(created, format="%m/%d/%Y %H")) %>%

mutate(text = replace_non_ascii(text, replacement = "", remove.nonconverted = TRUE)) %>%

mutate(text = str_replace_all(text, regex("@\\w+"),"" )) %>%

mutate(text = str_replace_all(text, regex("[[:punct:]]"),"" )) %>%

mutate(text = str_replace_all(text, regex("http\\w+"),"" )) %>%

unnest_tokens(word, text)

Removing common (stop) words from the dataset

After the dataset is tokenized and pre-processed, we need to remove stop words such as “for,” “the,” “an,” etc. that are not useful for analysis.

#Remove stop words

my_stop_words <- tibble(

word = c(

"https","t.co","rt","amp","rstats","gt",

"cent","aaya","ia","aayaa","aayaaaayaa","aaaya"

),

lexicon = "twitter"

) #Prepare stop words tibble

all_stop_words <- stop_words %>%

bind_rows(my_stop_words) #Remove numbers

suppressWarnings({

no_numbers <- tweet_words %>%

filter(is.na(as.numeric(word)))

}) #Anti-join the stop words and tweets tibbles

no_stop_words <- no_numbers %>%

anti_join(all_stop_words, by = "word")

We can also do a quick check to see how many stop words have been removed from the dataset by using the codes below:

tibble(

total_words = nrow(tweet_words),

after_cleanup = nrow(no_stop_words)

)

And the result would look like this:

The figure on the right (155,940) is the number of tokens left after removing the stop words.

At this point, our dataset should be clean and ready to be analyzed!

Word frequency analysis

A common method of doing text mining is to look at word frequencies. First, let’s look at some most common words in our tweets.

Unigram word cloud

The top five words are:

Quarantine — appeared 13358 times

Covid19– appeared 1628 times

Coronavirus — appeared 1566 times

Day — appeared 1200 times

Home — appeared 1122 times

Clearly, the quarantine is related to the Coronavirus / COVID-19 situation, and people are staying home to avoid being exposed to the virus.

Unigram word cloud

no_stop_words %>%

count(word) %>%

with(wordcloud(word, n, max.words = 100, random.order = FALSE,scale=c(4,0.7),

colors=brewer.pal(8, "Dark2"),random.color = TRUE))

Most common positive and negative words

After obtaining the word frequencies, we can use the “NRC” lexicon to assign a label (positive or negative) to each word. Then, we can create a word cloud tagged to the labels.

Word cloud of the most common positive and negative words

From the word cloud, we know that most people are stressed and bored during this isolation period. But on the bright side, we also learn that people are sending out kind messages, telling others to stay safe and healthy.

#Positive and negative terms word cloud

no_stop_words %>%

inner_join(get_sentiments("bing"), by = c("word" = "word")) %>%

count(word, sentiment, sort = TRUE) %>%

acast(word ~ sentiment, value.var = "n", fill = 0) %>%

comparison.cloud(colors = brewer.pal(2, "Dark2"),

max.words = 100)

Sentiment analysis

Sentiment analysis helps us identify expressed text and opinions from text data. It helps us understand people’s attitudes and feelings towards a particular topic.