The AOC Reply Dataset
By: Nick Doiron
Tags: Naturallanguageprocessing, Dataset, Supervised Learning, Machine Learning, Unsupervised Learning

Rep. Alexandria Ocasio-Cortez’s ‘@AOC’ Twitter account is a flashpoint for political discussions. The experienced internet surfer warns ‘don’t read the comments,’ but I’ve spent much time there, reading, baffled, fortunately only once baited into replying (someone said she could only get votes in Brooklyn).

On political Twitter, the most vitriolic users get called out for being ‘bots’. I can’t know if these accounts are real conservatives, burner accounts for trolling, or organized political or spam bots. All I know is that even when I block the worst accounts, more appear.

An opportunity for machine learning

I thought it would be interesting to analyze and cluster a large sample of replies to AOC. Treat the account as a honeypot. Maybe the resulting model could learn to filter out bad Tweets everywhere?

The ideal system (in my view) would automatically sort Tweets into 5–10 archetypes, which a user can then choose to see or not.

I don’t have the resources to label every Tweet manually, as I would expect for supervised learning.

‘Sentiment analysis’ on keywords within Tweets has been overdone, and doesn’t help with context — is someone angrily responding to a news article, or screaming at a bodega cat?

Scraping Twitter

Twitter’s official API offers a narrow range of options to their data. Unfortunately none fetch replies to a specific Tweet… There are scripts circulating on GitHub and StackOverflow which search recent Tweets and pick out replies, but at the speed of Twitter I would never capture a week or a month of discussions. The alternative is to hack a little.

(not Terms-of-Service-friendly, continue at your own risk)

I wrote a TamperMonkey script for my browser to download each Tweet’s text and metadata as I scrolled through replies. I had signed out of my account to get a less biased view with no user blocks (note that Twitter has hidden some users in older threads). I opened every Tweet and Retweet made by the ‘@AOC’ account in March 2019. The end result was around 110,000 replies (a 27.1 MB CSV). I am open to sharing the code and data with researchers.

Related Works

OpenAI’s post on unsupervised learning for NLP is beautiful and covers many different options.

After the 2016 election, Twitter released Russian agent Tweets, which were then analyzed by outside researchers using unsupervised NLP: https://arxiv.org/abs/1810.01466

Tutorial using word2vec and k-means clustering to analyze Reddit comments and Tweets:

https://medium.com/ml2vec/using-word2vec-to-analyze-reddit-comments-28945d8cee57

https://medium.com/ml2vec/using-word2vec-to-analyze-reddit-comments-28945d8cee57 Hierarchical clustering of data using SciKit-Learn https://stackabuse.com/hierarchical-clustering-with-python-and-scikit-learn/

Methodology

I came up with options to analyze the Tweets:

A : Supervised learning on text of suspect users — select a set of ‘probable bad faith’ Tweets based on username and screen name (ending with numbers, names with MAGA, QAnon, or Trump). This is NOT banning users for politics, but developing a model patterned on burner and troll accounts. Train a model on the text of replies and rate a Tweet based on their score.

: select a set of ‘probable bad faith’ Tweets based on username and screen name (ending with numbers, names with MAGA, QAnon, or Trump). This is NOT banning users for politics, but developing a model patterned on burner and troll accounts. Train a model on the text of replies and rate a Tweet based on their score. B: Supervised learning on text of profane and inflammatory users — similar to previous — search replies for keywords in Tweet text (profanity, inflammatory topics in unrelated threads, personal insults, hashtags). Put these Tweets, along with any others by these users, in the ‘bad faith’ set and rate other Tweets on their score.

similar to previous — search replies for keywords in Tweet text (profanity, inflammatory topics in unrelated threads, personal insults, hashtags). Put these Tweets, along with any others by these users, in the ‘bad faith’ set and rate other Tweets on their score. C: Unsupervised learning with word2vec — run word2vec on Tweet text and then use k-means clustering to divide into groups.

run word2vec on Tweet text and then use k-means clustering to divide into groups. D: Unsupervised learning with scikit-learn — do hierarchical clustering using scikit-learn and keyword flags (as full text doesn’t seem so easily applicable to this program).

Analysis and Results

Results of Supervised Learning

I didn’t want to label Tweets at first because of the manual process, but with the usernames option it would take only a few SQL statements. First, the stats:

of 110k replies, 17k came from ‘number users’ ending in 3+ digits, and around 2k came from users with a Trump/MAGA/Anon/nationalist name

no obvious leaders: only 12 of these posted 30+ replies in a month

of 3,300 users who wrote multiple replies, they still appeared less than once per week on average

For me this underlined a missing feature on Twitter: a simple regex could find thousands of users that I don’t have time to address one by one. Existing ‘block lists’ ignore how easy it is to create new accounts.

I have experience setting up a supervised NLP model with Google AutoML, but given all the original Tweets and replies, it initially failed to categorize any as ‘bad tweets’. When your data is so unbalanced (6:1 or worse), AutoML can ignore the minor category and claim an accuracy in the mid-80-percent range.

For my second run, I decided to collect the profane Tweets (and additional Tweets by their authors), 15k in total. I balanced these Tweets with 15k randomly selected normal Tweets, rather than the full dataset.

Confusion matrix for “reply” and “original tweet | reply” input formats were similar

The confusion matrix is tricky to read here, but predicting ‘True’ means the model believed it was profane. The usefulness of this model, rather unusually, depends on which Tweets fall into Predicted-true/profane but had been labelled False in my original selection process. I checked this category:

Another category shows supposedly profane users’ Tweets which the model would have labelled as non-profane.

Both lists were encouraging — I am pleased that a Python script and SQL query template can do rough labeling, that AutoML can build more general rules from that, and that the resulting classifier appears to work well.

I would prefer a classifier that considers the text of original Tweets and the replies as separate columns, and uses other Tweet attributes as well. Luckily for me, Microsoft just added an automated experiments platform to Azure, with this getting started guide appearing in May 2019. Azure runs a variety of algorithms and reports back on which have the best accuracy, in this case “MaxAbsScaler, SGD” (scaling each column from 0–1 then using stochastic gradient descent).

‘VotingEnsemble’ combines multiple methods. Stochastic Gradient Descent was the winner here

High accuracy is reported, but it isn’t possible for me to review the Azure version of should-be-bad Tweets. I can download the model as a 145 MB pickle file, but it’s unclear how to continue working with it in the Azure environment or on my local machine. If better documentation and walkthroughs appear, watch this space.

Results of Unsupervised Learning

The Reddit NLP tutorial was so readily usable, that I was quickly clustering words from my own dataset. I requested 10 clusters and printed out the top words from each:

At first I was discouraged. Cluster #1 picked out some infrastructure and policy terms, but it picked out lots of them. Cluster #9 appears to just be ‘verbs’. Another cluster had ‘faster’, ‘cheaper’, ‘smarter’, ‘greater’, ‘less’… words related in grammar, but not in meaning.

After expanding my script to show 50 clusters, they included topics like:

environment (‘temperature’, ‘cooling’, ‘extinction’, ‘flood’, ‘flooding’, ‘disasters’, ‘greenhouse’, ‘natural’, ‘epidemic’, ‘snow’, ‘industrial’)

(‘temperature’, ‘cooling’, ‘extinction’, ‘flood’, ‘flooding’, ‘disasters’, ‘greenhouse’, ‘natural’, ‘epidemic’, ‘snow’, ‘industrial’) name-calling (‘hack’, ‘dictator’, ‘activist’, ‘crook’, ‘actor’, ‘phony’, ‘con’, ‘loser’)

(‘hack’, ‘dictator’, ‘activist’, ‘crook’, ‘actor’, ‘phony’, ‘con’, ‘loser’) ethics (‘funneled’, ‘contributions’, ‘violations’, ‘pacs’, ‘aide’, ‘accounts’, ‘hidden’, ‘ethics’)

(‘funneled’, ‘contributions’, ‘violations’, ‘pacs’, ‘aide’, ‘accounts’, ‘hidden’, ‘ethics’) minority groups

The utility of the cluster analysis is mixed, because it does surface keywords that belong in topic groups, but I don’t see this helping uncover good vs. bad Tweets. Update: I am watching a Stanford NLP course and Gensim syn0 is explained — the original data for placing words in vector-space was created before the 2016 election, so the association of Trump, Clinton, and other words have changed since.

I decided to follow the path of this Twitter analysis tutorial, and generate some D3 histograms to show environment-related replies were motivated by AOC-originating environment and Green New Deal Tweets.

histogram of original environment Tweets and replies; each bar represents 4 hours

Environment-related replies to AOC peak daily and soared after a town hall with Chris Hayes on MSNBC.

Let’s set new axes for our next chart. Starting with a new environment Tweet at zero-hour, when do relevant replies typically come in?

decay of interest in a Tweet by minute (environmental topics and environmental replies only)

How does that compare to name-calling replies?

after environmental tweets, environmental replies (green) and name-calling replies (orange) are almost even in the first 30 seconds; name-calling immediately tapers off to a lesser share of Tweets

Finally I decided to build the hierarchical clustering model with a focus on classifying the sum of a user’s replies. I created several columns which count occurrences of terms in an expanded profanity list, environment words, name-calling words, etc that are informed by the clusters in the previous example.

I viewed Tweets from a sample few users in 10 cluster-division and 30 cluster-division, but couldn’t see a meaningful commonality in their Tweets? There were definitely offensive statements in all but one of the samples which I viewed. I’m wondering if they were grouped by high-to-low counts of occurrences, which would be more driven by number of posts than actual content.

Testing it out

On a new AOC Tweet, what Tweets would get flagged? Let’s refer back to the Supervised Learning model which I made in AutoML. I opened up the Predict tab and ran a few Tweets through (these were marked as ‘bad faith’).

here wew go: 99% positive

even though this one is anti-GOP, it was flagged

A disappointing number of bad Tweets below would be untouched by the model.

(updated May 7: I chose some samples which are not abusive or block-worthy… their good-ness or bad-ness depends on context and your personal politics, but I considered them to be bad replies and the ML model did not)

Conclusions

I’m not coming into this with the platform to track sophisticated trolls. Back when I started blocking, I would see if an account was ‘worth it’ by checking their photo, profile, and follower count to confirm if the account was just a burner / politics troll account. I would find accounts with ‘travel deals’ which one day switched to over ALL CAPS political discussions, stock photo profile pics and all number usernames. Twitter does have all of this data, plus the social graph, location/IP, login behavior, user-to-user block and mute and report history to work with. It would be trivial for them to predict who I am likely to block.

Where I did succeed is collecting an interesting dataset (which I hope to share with more researchers) and producing the next best thing to a platform: a partial filter for trolls on politics and news threads. It uses machine learning and not simply regular expressions (though, to be honest, it looks like that alone could filter out a lot). I’m also comfortable that it isn’t a broad political classifier, as it is trained on ‘bad’ Tweets, not a dataset of conservative Tweets or Breitbart articles or content from Gab.

Unsupervised learning, at least what is generally available to a tutorial-hopper like me, is more useful for analytics than it is for classifying content. That was surprising to me, and I hope to learn more about this.

Azure has a good set of libraries and Jupyter notebook infrastructure that I would use in future supervised learning projects, but I get lost in their interface.

I expect to see other algorithms which block socialist views or other twists on this. How can we use filter bubbles on Twitter, and still get a steady supply of well-written stories that challenge us?

Sounds like a job for another algorithm 🤔