Twitter Sentiment Analysis: Using PySpark to Cluster Members of Congress
By: Noah Muluberhan-Berhe
Tags: Python, Clustering, Sentiment Analysis, NLP, Spark

Having used quite a lot of Python in my previous projects I wanted to take on a large dataset which would require PySpark’s parallelised computing abilities. I found a large dataset on tweets sent by Members of the US Congress which looked interesting, and came up with an idea…

My hypothetical client:

I’m helping Lobbyists4America better understand relationships between congress members through their tweets: the dataset is comprised of tweets sent between 2008–2017. The aim being to use this intelligence to help them better target their clients’ lobbying efforts in Congress.

Which of these members of Congress are most alike?

Hypotheses:

1. There will be clusters of like-minded politicians that can be drawn largely along party lines.

2. There will be centres of influence (loud / influential voices) in these clusters that clients can target.

3. Some themes will emerge as more topical in this time period

Approach:

1. Import data and conduct EDA.

2. Theme extraction: Bag-of-Words

3. Entity Recognition: Spark-NLP

4. Sentiment Analysis: using TextBlob for sentiment scoring

5. Spark-ML to cluster like-minded members.

Step 1: Import data and conduct EDA.

Import Python libraries

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns

Import Spark functions and Spark-NLP, in the process building a SparkSession that ensures Spark-NLP jars files will be found in our module (I found I had to do this otherwise there was a No Module Found error)

import sparknlp

from pyspark.sql import SparkSession

spark = SparkSession.builder \

.master("local[4]")\

.config("spark.driver.memory","4G")\

.config("spark.driver.maxResultSize", "2G") \

.config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.11:2.4.5")\

.config("spark.kryoserializer.buffer.max", "1G")\

.getOrCreate()

from sparknlp.pretrained import PretrainedPipeline

from pyspark.sql.functions import from_unixtime, to_date, year, udf, explode, split, col, length, rank, dense_rank, avg, sum

from pyspark.sql.window import Window

Spark ML to be used later when applying Clustering

from pyspark.ml.linalg import Vectors

from pyspark.ml.feature import VectorAssembler, StandardScaler

from pyspark.ml.stat import Correlation

from pyspark.ml.clustering import BisectingKMeans

from pyspark.ml.evaluation import ClusteringEvaluator

from pyspark.ml import Pipeline

from pyspark.ml.tuning import CrossValidator, ParamGridBuilder

NLP libraries I will implement:

import nltk

from nltk.corpus import stopwords

from collections import Counter

from wordcloud import WordCloud from textblob import TextBlob

Import data into Spark dataframe:

It’s relatively simple to read in JSON files as Spark Dataframes, although there were some considerations to be made to enable more efficient processing when working with the dataframes.

1. A Spark dataframe should be split into partitions = 2–3 times the number of threads available in your CPU or cluster. My machine has 4 logical cores so I chose 3x, ie. 12 partitions, based on experimentation.

2. Then I cached the tables (‘persist’) to improve query performance later: you can check the Storage tab of the Spark GUI that 12 partitions have indeed been cached for each file.

3. The Shuffle Read partitions parameter is default to 200, we don’t want this to be the bottleneck, so we set this equal to partitions in our data, using spark.sql.shuffle.partitions. This will help specifically with wide shuffle transformations (e.g. GROUP BY or ORDER BY) that may be performed later on.

# Tweets

tweets = spark.read.json('../Lobbyists4America1/US_PoliticalTweets/tweets.json').repartition(12).persist() # Users

users = spark.read.json('../Lobbyists4America1/US_PoliticalTweets/users.json').repartition(12).persist()

I used Spark-SQL to create different slices of the data to get a better understanding of it.

One observation pertinent to my hypotheses is that few members have a large following on Twitter, these would powerful advocates for my clients’ policies

Namely…

+------------------+---------------+

| name|followers_count|

+------------------+---------------+

| Donald J. Trump| 31712585|

| President Trump| 18545354|

| Bernie Sanders| 5072538|

| Cory Booker| 3094413|

| Marco Rubio| 2554822|

| Elizabeth Warren| 2412087|

| John McCain| 2274034|

| Nancy Pelosi| 1126205|

| Jerry Brown| 1119275|

| Senator Ted Cruz| 960413|

| Governor Christie| 869256|

| John Kasich| 755475|

| Chuck Schumer| 591618|

| Senator Tim Kaine| 553125|

| Trey Gowdy| 552930|

| John Lewis| 512433|

|Kirsten Gillibrand| 396587|

| Maxine Waters| 379015|

| Mike Lee| 314821|

|Rep. Keith Ellison| 313271|

+------------------+---------------+

only showing top 20 rows

Step 2: Theme extraction: Bag-of-Words

After having a quick look at the data and some descriptive stats, I wanted to go a little deeper and understand what the main themes were.

NLTK has a corpus of stopwords that I downloaded to help me remove them from the tweets. Spark RDDs can be manipulated such that we can derive a word count from a collection of documents / tweets: using flatMap, reduceByKey and sort. I then fed this into WordCloud to visualise this for each year in the dataset

Note: I focused on years 2013 onwards, as they had large enough sample sizes

Step 3: Entity Recognition: Spark-NLP

The problem with the Bag-of-Words approach is that there were many words that didn’t constitute topics or themes, so I fed the corpus generated above into Spark-NLP’s pre-trained pipeline and essentially asked it whether each word was an entity.

Recreating the wordclouds on this cleaned corpus, it’s much clearer to see the hot topics at this time:

This step gave me some comfort in my direction of travel: I am going to focus on Healthcare as the main theme for analysis…

Step 4: Sentiment Analysis: using TextBlob for sentiment scoring

Assigning each tweet a sentiment score was incredibly simple with TextBlob, simply 3 lines of code which took less than a second to run over the entire dataset:

sentiment = udf(lambda x: TextBlob(x).sentiment[0])

spark.udf.register(“sentiment”, sentiment)

tweets = tweets.withColumn(‘sentiment’,sentiment(‘text’).cast(‘double’))

I looked at the overall trend in sentiment and also number of tweets. Using a 90 day daily moving average we can see that Twitter started to gain popularity as a medium for communication by members of Congress from 2013 onwards. It’s also interesting to see a very large increase at the back end of 2016: perhaps to do with Trump’s high engagement in Twitter and what appears to be other politicians responding to his tactic and increasing their own presence on Twitter. This increase was accompanied by a slight drop in sentiment, can we infer that tweets started becoming more confrontational in tone?

Looking at tweets containing “Obamacare” we see an interesting trend: in 2013 the number of these tweets really started to pick up sharply and fall away just as sharply in 2014, was this a deliberate tactic? Also interestingly: during this pick up there was a drop in sentiment to neutral, looks like if you’re tweeting about “Obamacare” instead of “ACA” then you’re likely to be negative about it.

This is borne out when looking at those members who tended to use “Obamacare”: they were Republicans.

Something else interesting: was this drop in “Obamacare” sentiment also the cause of the overall drop in sentiment at this time?

Explaining overall changes in sentiment by theme could be an interesting way to shed light on overall trends, perhaps creating some sort of weighted sentiment measure at the thematic level: but that’s for another time (unrelated to my hypotheses).

Step 5: Spark-ML to cluster like-minded members

Focusing on Healthcare, I wanted to get a more nuanced idea of likeminded and similarly-energised Congress members. It’s easy enough to split them into R vs D, but what if you wanted to know which of those members of Congress held the strongest opinions on the topic, and which of them were also energised by the issue to be active on Twitter about it. This would be valuable intel for a lobbyist.

So I used Spark-ML’s unsupervised learning models (namely Bisecting K-Means) to create these clusters based on the number of tweets and sentiment expressed by members in tweets containing either “ACA” or “Obamacare”

VectorAssembler to create the features column Applied StandardScaler to said features column Optimise for k in Bisecting K-Means, by iterating through different options and evaluating using the silhouette score. Chose k=6 as this had the highest score: 0.502 Scatterplot…

I chose to annotate each point with the member’s name and also that member’s ranking based on number of followers. So, based on number of followers Cory Booker appears to be the most influential member within his cluster being the one with the 4th highest number of followers among all members of Congress on Twitter.

Conclusions

Very interesting to see clusters based on the views / strength of opinion expressed on Healthcare reforms. The clusters appear to make sense and give us a more nuanced view than just Republican vs Democrat. Here we now have 6 clusters of likeminded (sentiment) members who are also similarly motivated (num_tweets) by the issue at hand. The green cluster appears to be highly opposed to Obamacare, among them Mike Lee appears to have the highest following, and very vocal on the issue as well. The red cluster are even more negative in their sentiment, although have tweeted far fewer times than those in the green cluster. This could be to do with sample size: the smaller the sample size the more susceptible it is to extremes in sentiment, while the larger the sample size the more it tends towards neutral. This is something I saw in different slices of the data: the more you zoom out, the more sentiment neutralises.

Hypotheses: Verdict

There will be clusters of like-minded politicians that can be drawn largely along party lines.

Correct There will be centres of influence (loud / influential voices) in these clusters that clients can target.

Correct Some themes will emerge as more topical in this time period

Correct

Potential Next Steps

I have created these clusters using the views of politicians across two different themes: ACA and Obamacare, the potential is there to create different clusters on the issue of Foreign Policy, or Energy, or Tax Reforms. Or even one set of clusters across different issues: to see which members are likeminded in general, and not just on specific issues…. this could lead us to extrapolate that … “if politician A is like politician B on this issue, then they may also come round on this issue as well”

Thanks for reading, for those interested… you can find the code on my GitHub, any feedback for this budding Data Scientist would be greatly appreciated!