Analyze YouTube Discourse and Find Troll Bots with Python
By: Thomas Hikaru Clark
Tags: Machine Learning, Python, Politics, Data Science

To make bar charts like the one above, I am using Plotly Express, whose bar() function makes creating aesthetically pleasing, interactive charts very easy:

# take video database's DataFrame: sort, filter, plot

df = vdb.df.sort_values(by="publishedAt")

if channelName:

df = df[df["channelName"] == channelName]

fig = px.bar(df, x="videoId", y=["likes", "dislikes"], hover_data=["channelName", "title", "publishedAt"])

fig.show()

Intermediate Analysis: word2vec and t-SNE

Keyword search is great, but it should leave you feeling like we’ve barely tapped the potential our trove of 100,000 comments. Let’s ratchet up our analysis with word embeddings, specifically word2vec.

As its name suggestions, word2vec converts words to high-dimensional vectors. The main advantage of this is that words that are used in similar ways will be converted to similar vectors (vectors with a high cosine similarity). In this way, different lexical tokens with similar meanings — like “GOP” and “Republicans” — can be mapped to similar locations in 100-dimensional hyperspace (sounds like sci-fi, I know). This method allows us to find much more interesting patterns in our dataset, and fortunately there is a Python implementation available from gensim.

Although visualizing high-dimensional hyperspace is impossible for humans, methods fortunately exist to project these high-dimensional embeddings into two or three dimensions so we can wrap our minds around them. The t-SNE method converts the word2vec vectors into something that we can plot and see, with the crucial constraint that it approximately preserves distance relationships. So vectors that are far away in hyperspace will still be far away after t-SNE, and vectors that are originally close will remain close.

We can use word2vec and t-SNE to plot clusters of vectors that are similar to each other. For a great tutorial on this, check out this post. Given a list of words, we plots the words that are most similar to them according to their word embeddings. Note that similarity is defined based on whether words are used in similar contexts or not, so clusters may not always contain semantic synonyms.

Word embeddings trained on comments from Fox News videos, projected into two dimensions using t-SNE. Words that are similar to “biden” and “blm” (i.e. “black lives matter”) are shown. This suggests that the comments section has a pretty negative view of BLM. Image by Author.

Word embeddings trained on comments from MSNBC videos, projected into two dimensions using t-SNE. Words that are similar to “trump” and “gop” are shown. Image by Author.

Vectorizing and Plotting Entire Comments

While plotting individual words is insightful, it would be useful to be able to plot entire comments such that similar comments are clustered near each other. Since a comment is a series of words, one thing we could try is to average the word vectors of all the words in a comment, and then plot this resulting vector to represent the entire comment. However, this would give rhetoricexcessive importance to common words like “the” or “and”. To solve this, we can perform a Tf-Idf vectorization and use the inverse document frequencies to weight the words in a comment when we take the average. This gives rare words a higher weight than common words, and there’s an easy-to-use implementation of this in Scikit-Learn. This approach has shortcomings, but it should begin to capture some of the meaning and themes of the YouTube comments.

A clustering of closely related comments (red points in the middle of the plot). On inspection, they all were similar comments such as “Trump lied, Americans died!”. Importantly, they were clustered together despite having small differences in wording, so comment vectorization is working OK. Image by Author.

Hunting Troll Bots

You may have heard of various attempts by certain state actors to influence the US election with troll armies and automated comment bots. I wanted to see if there is a way to automatically detect likely bot activity purely based on our comments database.

There are a few things that I propose as red flags for bot activity. One is posting a large number of comments in a short span of time. This would either suggest that a person is copying and pasting comments rather than typing them out word for word, or that an automated bot is at work. Another red flag is an account posting the same exact comment on multiple different videos.

Using our database, we can easily filter for the most prolific commenters on a given channel and plot their commenting activity. By plotting comment activity in increments of 10 minutes, I noticed accounts with activity that would be very unusual for a human — like commenting multi-paragraph comments a dozen times in five minutes. Once account posted the same exact comment 60 times in 10 minutes:

“Nationwide civil unrest GOP down 33% Unemployment tripled 190 ,000 Americans dead. Your LIFE is getting WORSE with TRUMP”

Interestingly, the account posted slight variations of the same comment, differing only in a few extra punctuation marks placed at the end of the comment. This makes me suspect that the bot is programmed to evade YouTube’s bot detection measures. In the visualization below, the x-axis is split into 30-minue time increments, and each stacked block represents one comment by the same account. The blocks are color-coded by video.