Twitter Bot or Not
By: Chris Doenlen
Tags: Flask, Twitterbots, Machine Learning, Twitter, Data Science

Dataset and model features

For this project, I used the Twitter Bot Accounts dataset on Kaggle, which has account-level information approximately 37,000 Twitter users, labeled as ‘bot’ or ‘human’. Account-level information means we’re not actually looking at tweets, but rather activity information like number of tweets and likes, network information like number of follows and friends, whether or not the user is verified, etc.

When exploring the dataset, right off the bat we can see that bot and human accounts behave differently — for example, humans tweet more frequently than their robot counterparts:

Average Tweets per Day by Account Type

Their networks also look differently, too — humans have bigger networks, both in terms of accounts they follow and that follow them.

Distributions of Followers and Friends by Account Type

I used these account-level details, along with a few other engineered metrics such as overall network size and reach as the features in the model.

Model scoring and evaluation

I wanted this model to accurately label bots as such, but not at the expense of labeling everything a bot — so I sought to find and tune a model that would achieve a balance between both precision and recall.

I tried a number of classification models from the sklearn library: Logistic Regression, K-Nearest Neighbors, Naive Bayes, Decision Tree, Random Forest, and XGBoost. Recognizing the non-linear nature of the data and the number of features, it wasn’t a surprise that the last two performed best ‘out of the box’ with regards to balanced precision and recall scores.

With some additional parameter tuning, XGBoost stood out as the best model.

KFolds Cross-Validation Precision-Recall Curves for XGBoost Model

Building the Flask app

After training on the full dataset, I pickled the final model to be used in a Flask app that would allow others to the check bot-likeliness of any Twitter users.

The app itself would be minimal — just a search bar where users could type in the Twitter account handle and the results. The structure of the app was easily coded up with a few lines of HTML, and I used the Bulma CSS framework to help with a simple and clean style.

To pull live Twitter information via their API, I set up a developer account, which is a straightforward application of providing a few details of who you are and your intentions. Within a few hours of submission, I had my access keys.

With the help of Tweepy, a Python library for interfacing with the Twitter API, I was able to pull the same account-level information I needed to make predictions on new, live users.

If you’re new to Tweepy, here’s a nice chunk of starter code that’ll allow you to access your home timeline, create a Pandas DataFrame of tweets by a specific user or topic, and actually send tweets from Python.

This was my first time creating a Flask app, let alone deploying anything. Fortunately, Heroku is pretty straightforward for new developers like myself to get their first projects live.

Final thoughts

I’m quite pleased with how this project turned out. I think the model does a reasonable job of making predictions on live data and I found myself lurking Twitter feeds playing ‘bot or not’: looking for trollish or bot-like comments and running a search on their handle with the app.

Of course there’s no way to be sure, but it’s pretty fun when your hunch is validated by AI.

I’m extremely interested in using Twitter more in the future and I’m excited to add natural language processing to my toolkit to dig deeper into the data. I see this project as the first piece in a larger body of work that explores networks, bubbles, and the nature of our curated realities on these platforms.

Thanks for reading — please check out the app and let me know your bot-likelihood!