TPOT: Automated Machine Learning in Python | by Dario Radečić | Towards Data Science

Everybody and their mother wants to get into machine learning. That’s great, but the field got automated over the years — to a degree. Nowadays, you don’t have to be a machine learning engineer to produce good quality models. How? That’s where libraries like TPOT come into play.
If you’re more of a video person, or just want to reinforce your knowledge, feel free to watch our video on the topic. Source code is included:
Let’s face the facts — machine learning isn’t as interesting as hyped by the media. You can do amazing things with it, don’t get me wrong. I’m only talking about the actual machine learning process. You know, the process of finding an optimal algorithm and its hyperparameters.
It’s good this part is getting (or already got) automated — since we as engineers can focus on more interesting things.
The ideal reader is someone familiar with the Python programming language and with the general flow of a machine learning project. You don’t have to be an expert, or even work in the field, but it’s expected that you’re familiar with the most common algorithms and how to use them.
The article is structured as follows:
What’s TPOT?Genetic programming introductionDataset loading and preparationTPOT in action
What’s TPOT?
Genetic programming introduction
Dataset loading and preparation
TPOT in action
So without much ado, let’s get started!
TPOT, or Tree-based Pipeline Optimization Tool is a Python library for automated machine learning. Here’s the official definition:
Consider TPOT your Data Science Assistant. TPOT is a Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming.[1]
In a way, TPOT is your data science assistant. It automates the “boring” stuff and leaves you with the tasks of data gathering and preparation. I know this segment isn’t fun to some of you — but it’s 80% of the job. With libraries like TPOT, that percentage might only get higher.
Plus, data gathering and preparation can’t be fully automated, for obvious reasons. Another sentence over at the official documentation drives the point home:
TPOT will automate the most tedious part of machine learning by intelligently exploring thousands of possible pipelines to find the best one for your data.[1]
Let’s take a quick look at your average machine learning pipeline to see where TPOT fits in:
As you can see, the library automates every tedious portion of the process.
But how? By using something known as “genetic programming”. Let’s explore this term in the next section.
The official website summarizes the concept pretty well:
Genetic Programming (GP) is a type of Evolutionary Algorithm (EA), a subset of machine learning. EAs are used to discover solutions to problems humans do not know how to solve, directly. Free of human preconceptions or biases, the adaptive nature of EAs can generate solutions that are comparable to, and often better than the best human efforts.
Inspired by biological evolution and its fundamental mechanisms, GP software systems implement an algorithm that uses random mutation, crossover, a fitness function, and multiple generations of evolution to resolve a user-defined task. GP can be used to discover a functional relationship between features in data (symbolic regression), to group data into categories (classification), and to assist in the design of electrical circuits, antennae, and quantum algorithms. GP is applied to software engineering through code synthesis, genetic improvement, automatic bug-fixing, and in developing game-playing strategies, ... and more.[2]
With the right data and the right amounts of data, we can use machine learning to learn any function. Still, knowing which algorithm to use can be daunting — there’s just too many of them. And we still haven't touched the idea of hyperparameter tweaking.
That’s where genetic programming really shines, because it is inspired by the Darwinian process of Natural Selection, and they are used to generate solutions to optimization in computer science.
Genetic algorithms have 3 properties:
Selection: where you have a population of a possible solution and a fitness function — then at every iteration each fit is evaluatedCrossover — selecting the fittest solution and performing crossover to create a new populationMutation — taking their children and mutating them with some random modification until you get the fittest or best solution
Selection: where you have a population of a possible solution and a fitness function — then at every iteration each fit is evaluated
Crossover — selecting the fittest solution and performing crossover to create a new population
Mutation — taking their children and mutating them with some random modification until you get the fittest or best solution
Sounds like a lot, definitely, but TPOT doesn’t require us to be experts in genetic programming. Knowing the concepts is, as always, beneficial.
Okay, we have the theory part covered thus far. We’ll gather and prepare some data in the next section.
I love using the Titanic dataset for most of my examples. The dataset isn’t too basic, as it requires some preparation, but it isn’t too complex in the means that the preparation process would take too long. You could definitely spend more time preparing this dataset then I did, but this will be enough for our needs.
Imports-wise, we’ll need 3 things for now: Pandas, StandardScaler and train test split:
import pandas as pdfrom sklearn.preprocessing import StandardScalerfrom sklearn.model_selection import train_test_split
Next, we can load in the dataset and display the first couple of rows:
data = pd.read_csv(‘https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv')data.head()
Here’s what we’ll do, preparation-wise:
Drop irrelevant columns (Ticket and PassengerId)
Remap Sex column to zeros and ones
Check if a passenger had a unique title (like doctor) or had something more generic (like Mr., Miss.) — can be extracted from the Name column
Check if cabin information was known — if the value of Cabin column is not NaN
Create dummy variables from the Embarked column — 3 options
Fill Age values with the simple mean
And here’s the code for the preparation process:
data.drop([‘Ticket’, ‘PassengerId’], axis=1, inplace=True)gender_mapper = {‘male’: 0, ‘female’: 1}data[‘Sex’].replace(gender_mapper, inplace=True)data[‘Title’] = data[‘Name’].apply(lambda x: x.split(‘,’)[1].strip().split(‘ ‘)[0])data[‘Title’] = [0 if x in [‘Mr.’, ‘Miss.’, ‘Mrs.’] else 1 for x in data[‘Title’]]data = data.rename(columns={‘Title’: ‘Title_Unusual’})data.drop(‘Name’, axis=1, inplace=True)data[‘Cabin_Known’] = [0 if str(x) == ‘nan’ else 1 for x in data[‘Cabin’]]data.drop(‘Cabin’, axis=1, inplace=True)emb_dummies = pd.get_dummies(data[‘Embarked’], drop_first=True, prefix=’Embarked’)data = pd.concat([data, emb_dummies], axis=1)data.drop(‘Embarked’, axis=1, inplace=True)data[‘Age’] = data[‘Age’].fillna(int(data[‘Age’].mean()))
Feel free to just copy it over, as this isn’t the article on data preparation.
Here's how our dataset looks now:
Looks fine, but some attributes are on a much greater scale than the others — Age and Fare. We’ll use the standard scaler to address this, but first, let’s perform the train test split:
X = data.drop(‘Survived’, axis=1)y = data[‘Survived’]X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8)
And now let’s fit the standard scaler to the train data, and transform both train and test data:
ss = StandardScaler()X_train_scaled = ss.fit_transform(X_train)X_test_scaled = ss.transform(X_test)
That’s it for this section. In the next one, we’ll see the TPOT library in action.
We’ve covered a lot thus far, and now we’ll shift our focus to the meat of the article. First, we need to import TPOT — either the regressor or the classifier. We’re dealing with the classification problem here, so guess which one we’ll use.
from tpot import TPOTClassifier
The training process can’t be easier:
tpot = TPOTClassifier(verbosity=2, max_time_mins=10)tpot.fit(X_train_scaled, y_train)
Here we’ve specified the verbosity parameter as 2, as we want more information printed out, and also specified for how long we want training to last. Feel free to train it for longer, but 10 minutes is perfectly fine for a demonstration case.
We’ll get some info printed out during the training:
It’s only 4 generations, so the model won’t be perfect, but we didn’t want to spend an entire day training the model. You could.
We can now check for the best pipeline:
tpot.fitted_pipeline_
As you can see, the gradient boosting method was used with tweaked hyperparameter values. TPOT got to this point in only 10 minutes! Think about how much time would it take to get here manually.
We can now check for the accuracy on the test set:
tpot.score(X_test_scaled, y_test)>>> 0.8491620111731844
Not terrible, by any means.
And that’s just enough for an introduction to the TPOT library. Feel free to explore further on your own. The official documentation is a great place to start.
Loved the article? Become a Medium member to continue learning without limits. I’ll receive a portion of your membership fee if you use the following link, with no extra cost to you.