Twitter Text Analysis in R. Helping communication practitioners get... | by Kieran Tan Kah Wang | Towards Data Science

Open data sources are one of the best gifts for data scientists or analysts as they allow them to draw valuable insights for free, without having to worry about the data licenses. Twitter is one of the most popular social media application in the world as it’s free, and also allow users to tweet on any topics that come to their mind. This article will focus on how can we use Twitter through R programming to extract valuable insights and communicate these findings to the relevant stakeholders using Tableau.
“How might we help the communication practitioners to get actionable insights from Twitter so that they can create more effective communication that caters to the needs & concerns of general public”
The chosen target user of this problem statement is the military communication practitioner, who is keen to understand what are the concerns the general public is having with regards to the military (i.e. National Service) in Singapore.
For every data source, we can do a simple representation with the four ‘V’s of big data, mainly Volume, Velocity, Variety, and Veracity for a general understanding of this data source.
Volume — Scale of dataVelocity — Analysis of streaming dataVariety — Different forms of dataVeracity — Uncertainty of data
Volume — Scale of data
Velocity — Analysis of streaming data
Variety — Different forms of data
Veracity — Uncertainty of data
We can use twitteR library package in R Programming to access the Twitter API. Note that we will need to sign up for a Twitter developer account to access the API, as each user will be provided with a unique set of consumer key, consumer secret key, access token, and access secret key. Once we have set up the connection to Twitter API, we will request the tweets we want from Twitter by stating the search term (i.e. “National Service”), the maximum number of tweets (i.e. n = 1000), latitude longitude of Singapore with the search radius in miles (i.e. geocode = ‘1.3521, 103.8198, 279mi’) and the language (i.e. lang= ‘en’).
# import necessary library for Twitter API, data manipulation and text cleaninglibrary("twitteR")library("ROAuth")library("dplyr")library("tidytext")# Set up Twitter Connectionconsumer_key <- '' # removed due to confidentialityconsumer_secret<- '' # removed due to confidentialityaccess_token <- '' # removed due to confidentialityaccess_secret <- '' # removed due to confidentialitysetup_twitter_oauth(consumer_key ,consumer_secret,access_token ,access_secret)# extract english tweets using 'National Service' tweeted in Singapore, with retweets removedtweets <- strip_retweets(searchTwitter('National Service',n=1000, geocode='1.3521,103.8198,279mi',lang='en'))# print length of tweetslength(tweets)
Note that for the free version of Twitter public API, we are only able to request the last 7 days’ worth of tweets. As a result, the number of tweets we got in our request was just 17.
When we extract tweets via the Twitter API, the results will not be clean as tweets can be in multiple languages and also include non-readable words like emoticons. Looking at the screenshot below, obvious data cleaning steps to be taken are removing Twitter usernames, removing emoticons (represented by <U+...>), and removing the URL links to the tweets (highlighted in blue).
As mentioned in the data characteristic section above, the tweets returned are in JSON format and we will have to first convert it to a data frame in R before we can use common data handling library like dplyr . In our analysis, we do not want any punctuations, numbers, or words of length lower than 2. Therefore in our text cleaning code, we will remove them.
# convert tweets to dfdf <- twListToDF(tweets)# text cleaningdf$text <- as.character(df$text)df$text <- gsub("\\$", "", df$text) df$text <- gsub("@\\w+", "", df$text)df$text <- gsub("[[:punct:]]","", df$text)df$text <- gsub("http\\w+", "", df$text)df$text <- gsub("[ |\t]{2,}", "", df$text)df$text <- gsub("^ ", "", df$text)df$text <- gsub(" $", "", df$text)df$text <- gsub("RT","",df$text)df$text <- gsub("href", "", df$text)df$text <- gsub("([0-9])","", df$text)
In every text analysis problem, breaking big chunks of text into individual words (or tokens) is essentially the first step to take, which we referred to as tokenization and can be done using tidytext library. For our analysis, once we break the texts into individual tokens, we will aggregate the counts of each unique word so that we can know what are some common words tweeted by users.
# split the text into individual tokens, remove stopwords,'National Service', sort by descending counttokens <- data_frame(text = df$text) %>%    unnest_tokens(word, text) %>%    anti_join(stop_words) %>%    count(word, sort = TRUE)# output to csv for further usagewrite.csv(tokens, './bukitmerah.csv',row.names = FALSE)
As above, we have outputted our results containing the individual word/token along with their frequencies in a CSV file, which we will load into Tableau later for our visualization purpose.
Another possible analysis we can make with the tweets is sentiment analysis, which is the interpretation and classification of emotions in the data. This sentiment analysis can be easily conducted in R using the nrc lexicon which will categorize each word/token to one of the 10 sentiment categories mainly, anger, anticipation, disgust, fear, joy, negative, positive, sadness, surprise, trust. The aggregation will then be done to categorize the positive sentiments and all that's negative. A score of +1 will be tagged to each positive word and -1 to each negative word. Once done, the sentiment score of each tweet will be consolidated and if the score is above 0, that tweet will be classified as positive. If it's below 0, it will be negative. If the score is exactly 0, it will be neutral.
# get sentiment score for each tweets by using the sentiment lexicon "nrc"text_df <- data_frame(id=df$id, text = df$text) %>%     unnest_tokens(word, text) %>%    anti_join(stop_words) %>%    inner_join(get_sentiments("nrc")) %>%    mutate(score = ifelse(sentiment=='positive',1,ifelse(sentiment=='joy',1,ifelse(sentiment=='anticipation',1,ifelse(sentiment=='trust',1,ifelse(sentiment=='surprise',1,-1)))))) %>%    group_by(id) %>%    summarise(total_score = sum(score)) %>%    mutate(sentiment = ifelse(total_score>0,'positive',ifelse(total_score<0,'negative','neutral')))# get the dataframe which contains tweet message, id and it's sentimentsentiments <- df %>% inner_join(text_df, by='id') %>% select('text','id','created','sentiment')# output to csv for further usagewrite.csv(sentiments, './sentiments.csv',row.names = FALSE)
As above, we have outputted our results containing the whole tweet along with their sentiment score (i.e. positive/negative/neutral) in a CSV file, which we will load into Tableau later for our visualization purpose.
From the previous sections, we have outputted two files into CSV formats which we will load into Tableau to discover possible insights through data visualization.
Using the first file containing the individual word/token along with their tweeted frequencies by all users in their tweets, we can plot a word cloud in Tableau, with bigger sized words indicating a higher frequency.
From the word cloud, we can identify commonly used words like “thread”, “mandatory”, “volunteer”, “February” and “americorps” which do not suggest much in terms of a military context.
Let’s see if we can discover more useful insights with our second CSV file that contains the tweets and their sentiment score. As the time of the text being tweeted was being captured as well, we can plot a bar graph over time to see the sentiments of people towards National Service over the week (remember that Twitter public API only allows data requests up to 7 days).
Out of the 17 relevant tweets, 10 were positive, 5 were negative and 2 were neutral. We also observed that the sentiments were moving towards the negative side over the week.
In this analysis of using Twitter Public API to gather insights from the general public on National Service, three major drawbacks can be identified.
Twitter API
Twitter API
> Free account only allows pulling of past 7 days tweets from the API
> 7 days of data will not provide a comprehensive analysis of public views towards National Service
> Unable to provide a drill-down of the users on their demographics
2. Accuracy of Sentiment Analysis
> More in-depth sentiment analysis should be conducted as current software do not consider sentiments of whole tweets resulting in tweets categorized in the wrong sentiment based on a few words
> Inappropriate words or remarks cannot be filtered off
3. Irrelevant tweets
> Irrelevant tweets on “National Service” in other countries might be captured as Twitter is open for everyone to tweet anything regardless of their locations
In this article, we have discussed how we can use open data sources like Twitter to gather valuable insights from the general public on any issues a particular stakeholder is interested to look into, in our case the communication practitioners on military/national service. We have also discussed using in-built sentiment analysis packages in R to interpret the emotions involved in the texts. Most of the time in the world of data science, data scientists often build their sentiment analysis model as each problem calls for a different model to derive more appropriate and accurate sentiments. Hope you guys now have an idea how we can consider using Twitter API and inbuild sentiment analysis library to kickstart your text mining problems!