Building an Automated Machine Learning Pipeline: Part One | by Ceren Iyim | Towards Data Science

Part 1: Understand, clean, explore, process data (you are reading now)
Part 2: Set metric and baseline, select and tune model (live!)
Part 3: Train, evaluate and interpret model (live!)
Part 4: Automate your pipeline using Docker and Luigi (live!)
During my data science learning journey, I learned through reading articles on Towards Data Science as much as I learned from online courses and various coding challenges. Following such diverse sources of learning that come with their own interpretations of machine learning (ML) helped me a lot. Today, I will bring a part of my perspective to the ML applications that I distilled throughout this journey and start an article series. It will hopefully show how to build an automated ML pipeline with a beginner-friendly language and structure.
We are going to follow an intuitive ML pipeline to build a predictor to rate wine quality:
Understand & Clean & Format DataExploratory Data AnalysisFeature Engineering & Pre-processingSet Evaluation Metric & Establish BaselineSelect an ML Model based on the Evaluation MetricPerform Hyperparameter Tuning on the Selected ModelTrain and Evaluate the ModelInterpret Model PredictionsDraw Conclusions & Document Work
Understand & Clean & Format Data
Exploratory Data Analysis
Feature Engineering & Pre-processing
Set Evaluation Metric & Establish Baseline
Select an ML Model based on the Evaluation Metric
Perform Hyperparameter Tuning on the Selected Model
Train and Evaluate the Model
Interpret Model Predictions
Draw Conclusions & Document Work
This first article covers steps 1, 2 and 3. The code behind this article can be found in this notebook. The second and third articles in the series will have covered the entire pipeline.
After we build the pipeline, the fourth article will focus on how we can automate the wine rating predictor with Docker and Luigi. This part is especially exciting for me because it introduced me to concepts and tooling that let me understand how an ML solution runs on production systems.
Disclaimer: The core structure and utilities in this automation are taken from a coding challenge that I took on recently as part of an ML Engineer interview process at an amazing company called Data Revenue. I did not get the job because they were looking for someone with more emphasis and experience in engineering whereas I’m more on the science part. Ultimately, I wore a software engineer’s hat and learned a lot during the interview process.
The complete project is available on GitHub for the impatient:
github.com
In this project, we are provided with a sample dataset. In a real-world scenario, one can possibly obtain (perhaps scrape) this data from online sources like winemag.
The goal of the project is to build a wine rating predictor using a sample dataset to show good prediction is possible, perhaps as a proof of concept for a larger project.
We are going to use the results of the wine rating predictor to confirm that ML applications are effective solutions to business problems such as optimizing a wine portfolio for a seller.
We should always keep in mind that the “science” in data science serves a purpose, ultimately for a non-scientist. A typical project has some implied base requirements.
Our wine rating predictor should be understandable because our audience will likely have limited knowledge of statistics and ML. Furthermore, it should be performant because the complete production dataset could have millions of rows. Finally, it should be automated in a way that it can run on any production system without requiring specialized configurations and setups.
The data for this project is available in data_root/raw/wine_dataset in the repository. For now, we will use 90% of it as the training set and the remaining 10% as the test set. They are both available in data_root/interim as train.csv and test.csv .
We are going to use Python 3.7 and will need the following libraries in this notebook:
In this step, we will get ourselves familiar with the dataset, understand what our dataset conveys and clean it by removing redundant rows and columns.
Let’s load the training and test datasets into data frames and have a look at some rows of the training dataset:
train = pd.read_csv("../data_root/interim/train.csv") test = pd.read_csv("../data_root/interim/test.csv")train.sample(5)
There are some empty rows in the dataset and we have only 2 numeric variables: points and the price.
Points is the dependent variable that we are trying to predict, and will be referred to as the target. The rest are the independent variables that can be used in determining the target and will be referred to as features.
We are trying to grasp the relationships between independent variables and a dependent variable and use this relationship to generate predictions, which makes the wine rating predictor a supervised regression machine learning model:
Supervised: Dataset includes the wine ratings (labelled as points) and independent variables which can be used in determining the points.
Regression: Our target variable — points consist of continuous integers ranging from 80 to 100.
Most of the column names are self-explanatory if you ever thought about which wine to buy at a supermarket. However, I think it is important to understand how a particular wine receives points and domain-specific features.
Points are assigned to a wine by a taster. A wine may receive different points from the same taster. When a taster is assigning points to a wine, s/he also provides a description of the notes and taste about the wine.
The designation is the name of the vineyard and region_1 & region_2 features describe the places where the grapes of the wine are grown. Here is how a Barlow 2005 Barrouge Red looks like:
Now, let’s have a look at the data types and number of non-empty (non-null) of the training and test dataset:
train.info()
test.info()
This is an interesting dataset because 90% of the features are categorical — variables that are in object Dtype. ML models can only work with numerical data and non-null values. Thus, we are going to develop some strategies in the Feature Engineering & Pre-processing to handle the categorical data and missing values in our model.
The function below will enable us to observe the missing values as a percentage per feature.
Features that have missing values of more than 50% are not likely to provide significant information to our model. Therefore, region_2 will be dropped from both datasets.
In this sample dataset, target value has no missing values. If we would have missing values in the target, we would have dropped all the missing values not to distort the existing distribution of the target.
Cardinality is the number of unique values that a categorical feature has. Let’s have a look at it for our training dataset:
object_columns = (train                  .select_dtypes(include="object")                  .columns)for column in object_columns:    print("{} has {} unique values."          .format(column,                   train[column]                  .nunique()))
We can infer that a feature has a
high cardinality if it has more than 1000 rows out of 9000: description, designation, title and winery.
moderate cardinality if it has more than 100 rows: province, region_1 and variety.
low cardinality if it has less than 100 rows: country, region_2, taster_name and taster_twitter_handle.
High cardinality features may cause problems when we are training our model. Even though we transform each unique value of a categorical feature to a new feature, they may bring the curse of dimensionality. So, we are going to use the high cardinality features for adding new features, then we are going to remove them from the datasets.
Remember that this is a sample dataset. So we cannot guarantee that we have seen all the possible values of each categorical feature when we are training the model. If our wine rating predictor encounters with a new value in the test set, it will throw errors. That is why we are going to revisit this in the Feature Engineering & Pre-processing.
Two variables are called collinear if there exists a linear association between them. In case of categorical features, they are called collinear if they are providing the same information to predict a target.
Collinear features decrease the generalization performance on the test set and interpretability of the model. Recall that one of the requirements for our wine rating predictor was interpretability.
In our sample dataset, taster_name and taster_twitter_handle signals for the collinearity. After making sure that all values of the taster_twitter_handle are covered by the taster_name (not shown here but checked here in the notebook), it is removed to build a good and explicable wine rating predictor.
Up until now, we had the first glance of the dataset and detected the features to remove:
designation and winery due to the high cardinality
region_2 due to having missing values more than 50%
taster_twitter_handle due to collinearity
Moreover, the training set contains some duplicate rows. In addition to the list above, we are going to drop them and clean the data with the below function:
drop_columns = ["designation", "winery", "taster_twitter_handle", "region_2"]train_cleaned = CleanDatat(train, drop_columns, "points")test_cleaned = CleanData(test, drop_columns, "points"
Now, we are going to use the train_cleaned.csv and test_cleaned.csv to visualize and explore the features and target further in the Exploratory Data Analysis.
Exploratory data analysis (EDA) is the step where we have a detailed look at the variables and visualize the relationships between features and target. In this step, we are going to seek and follow the clues of strong determiners of the target. At the end of this step, we will decide which features to process in the Feature Engineering & Pre-processing.
First, we will examine the points (target), numeric features and link between them. Then, we will investigate the low-cardinality features and their relationship to the target. As we find valuable insights, we will elaborate on them through the steps of EDA.
Distribution is a description of a variable’s range and how data is spread in that range. We can easily observe a distribution by plotting a histogram:
figsize(10, 8)plt.rcParams['font.size'] = 14plot_histogram(train_cleaned, "points")
Points show a normal distribution (a bell-shaped curve is obvious in the distribution) as expected from a random variable. Range of the points is distributed between 80 and 100 with a mean of 88.45 and a median of 88. Moreover, standard deviation — a measure of the spread of a data range is 3.03 resulting in a variance of 9.1 as the squared deviation. Based on the normal distribution, we can confidently say that 95% of the points lie in the 82.5–94.5 range.
When we plot the histogram of the price it showed a vague picture because it was highly right-skewed (available here in the notebook). To get a reasonable one, we will observe it in the 0–200 range:
figsize(10, 8) plt.rcParams['font.size'] = 14plot_histogram(train_cleaned, "price", 200) plt.xlim(0,200)
This is still a right-skewed or positively skewed distribution where the right tail is longer and most of the values are concentrated on the left-hand side. Median of the price is 25 and the mean price is 35.5.
In order to use price as a predictor, we need to assume that the points received do not cause any price change. Otherwise, this will cause data leakage since we’ll be including a predictor at a later point in time. This could potentially spoil our future predictions of points.
Nonetheless, assuming that price is not affected by the points assigned by a taster, it is an indicator of wine’s quality and age. Therefore, we expect it to be an essential feature of the model.
To visualize the relationships of several variables, we will use a density plot where we plot the distribution of target per unique values of a categorical feature. A density plot can be interpreted as the continuous and smoothed version of a histogram which conveys the shape of the distribution easily.
To keep the plots interpretable, we will plot the distribution of points for the countries with more than 100 occurrences.
# make a list of countries that has most occurencescountries = (train_cleaned["country"]             .value_counts())freq_countries = list(    countries[        countries.values > 300]    .index)freq2_countries = list(    countries[        (countries.values <= 300) &        (countries.values >= 100)]    .index)
figsize(20, 10)plt.rcParams['font.size'] = 14# plot points distribution for most frequent countriesplt.subplot(1, 2, 1)plot_distribution(train_cleaned, "points",                  freq_countries, "country")plt.subplot(1, 2, 2)plot_distribution(train_cleaned, "points",                  freq2_countries, "country")
Wine producing market is dominated by the US, France, Italy and Spain which is easily available from the plots. Country is a differentiating feature to determine points of wine. However, there are some exceptions: USA & Portugal or Austria & Germany.
Assuming that the price is an essential determiner for the points, we will explore points and price relationship with country.
# create dataframe of most frequent countriespoints_freq_countries = train_cleaned[    train_cleaned    .country    .isin(freq_countries)]figsize(20, 10)plt.rcParams['font.size'] = 14# plot a scatterplot of Points and Priceplt.subplot(1,2,1)sns.scatterplot(x='price', y='points',                hue='country',                 data=points_freq_countries,                alpha=0.7)plt.xlabel("Price", size=14)plt.ylabel("Points", size=14)plt.title("Points vs Price for Most Frequent Countries",           size=16)# plot a scatterplot of Points and Priceplt.subplot(1,2,2)sns.scatterplot(x='price', y='points',                 data=train_cleaned,                alpha=0.7)plt.xlabel("Price", size=14)plt.ylabel("Points", size=14)plt.title("Points vs Price for all Countries with price range 0-200", size=16)plt.xlim(0,200)
The plot on the left-hand side shows the points and price relationship for the wines produced by the dominant countries in the complete price range. The plot on the right-hand side shows the points and price relationship for all countries zoomed in 0 and 200 price range.
Both plots show that there is a strong positive correlation exists between the points and price, which serves as proof that price is an important predictor. Another valuable insight is, there are two wines from Italy and one from France that received full points from tasters, all of their prices are greater than 200.
Points, represents the score of a particular wine received from a taster. So, a different taster might imply different interpretations of a wine. To investigate this we are going to plot a violin plot.
Violin plot ​is my favourite plot that conveys the distribution of the data shape and summary statistics with the violins outside and slim rectangles inside respectively.
figsize(14, 8)plt.rcParams['font.size'] = 14f = sns.violinplot(data=train_cleaned,                   x="taster_name",                   y="points")f.set_xticklabels(f.get_xticklabels(),                  rotation=90)plt.title("Points from Different Tasters",          size=16)
Each taster’s summary statistics and distribution of points are unique resulting in different violins above. Looking at this picture, we expect it to be the second most important predictor of the model.
We explored points, price, country and the taster_name and relationships between them. In addition to them, variety, region_1 and province will remain in the final feature list since:
variety and region_1 are major elements in differentiating the grapes of the wine so as the wine’s taste and quality
province provides location information when combined with country
Moreover, we observed a normal distribution in the points and identified its mean and variance, which will be useful for us in the Set Evaluation Metric & Establish Baseline. We expect price and taster_name to be the two most important predictors in determining points.
Now, we will move forward with the long-waited and mentioned Feature Engineering & Pre-processing part.
Feature engineering is the process of extracting and transforming information from raw features and potentially create a lot more useful features. We are going to take what we’ve learned about our features in the previous parts and encode that information into new ones. Since our features are mostly categorical, we are going to search for numerical or binary features in the description, variety and title.
Then, we are going to prepare both datasets to be used as an input for our wine rating predictor. This process is called feature pre-processing. This is a necessary step because ML models can only work with numerical data and non-missing values.
Building a good wine rating predictor starts with identifying the most useful features to determine points. At the end of this step, we will identify the most effective feature set, which will provide a good and understandable wine rating predictor in return. We are going to follow feature extraction, categorical feature transformation and missing value filling respectively.
description: It contains information about the wine’s colour, taste and notes (like sweet, dry). Taste and colour-related words are searched and extracted from the description with the function below. If a taste or colour-related word exists in the description, the corresponding feature to that word is assigned with a value of 1, otherwise 0.
title: It contains the production year of the wine. If year information is not available in the title year is assigned to 0.
variety: It contains information about whether different types of grapes are blended. is_blend is assigned to 1 if several grape varieties present, otherwise 0.
Now, using those functions above and search keywords below, we are going to add new features.
# create search terms for new features # to be extracted from descriptionis_red_list = ["red", "Red", "RED",               "noir", "NOIR", "Noir",               "black", "BLACK", "Black"]is_white_list = ["white", "WHITE", "White",                 "blanc", "Blanc", "BLANC",                 "bianco", "Bianco", "BIANCO",                 "blanco", "Blanco", "BLANCO",                 "blanca", "Blanca", "BLANCA"]is_rose_list = ["rose", "ROSE", "Rose",                "rosé", "Rosé", "ROSÉ"]is_sparkling_list = ["sparkling", "SPARKLING", "Sparkling"]is_dry_list = ["dry", "Dry", "DRY",               "dried", "Dried", "DRIED"]is_sweet_list = ["sweet", "Sweet", "SWEET"]desc_extracting_dict = {    "is_red": is_red_list,    "is_white": is_white_list,    "is_rose": is_rose_list,    "is_sparkling": is_sparkling_list,    "is_dry": is_dry_list,    "is_sweet": is_sweet_list}
train_features_added = ExtractFeatures(train_cleaned) test_features_added = ExtractFeatures(test_cleaned)
After the feature engineering, we have 14 features in the train_features_added and test_features_added data frames: country, province, region_1, variety, price, year, taster_name, is_red, is_white, is_rose, is_dry, is_sweet, is_sparkling, is_blend.
As the first part of feature pre-processing steps, categorical features (country, province, region_1, taster_name and variety) will be turned into the numerical features. We selected the ​ordinal encoding ​approach for the transformation, which is the process of assigning positive integers consecutively starting from 1 to each unique value of a particular feature in the order of appearance.
Remember that this is a sample dataset, so we have to consider the possible values that are not present in this sample dataset but may show up in the test dataset. To manage them we are going to use ordinal encoder from the category_encoders library.
When we set handle_unknown parameter to the "value", the ordinal_encoder object will assign the first-time seen values of the test set to -1. Then, we run fit method on the training set, ordinal_encoder object maps the values of the given features to the integers. Then, we run transform method on both datasets, the transformation process is completed according to mapping done during fitting and we get full-numeric datasets in return.
ordinal_encode_cols = ["country", "province", "region_1", "variety",  "taster_name"]data_encoded = EncodeCategoricalData(train_features_added,                                     test_features_added)train_encoded = data_encoded[0]test_encoded = data_encoded[1]
As the second part of the feature pre-processing, we will handle missing values. An easy option is to drop them. However, we already have only 8500 data points to train the model. Therefore, we are going to proceed by filling them. ​Process of filling missing values of features is called imputation. We are going to use the following strategies for different features:
taster_name: Since we expect taster_name to be one of the most important features, it is imputed with a constant value (0 stands for “Unknown taster”). That way, we can maintain each taster’s effect in determining points.
price and year: Imputed with median value since they are numeric columns and the median is not affected by the extreme values.
country, province, region_1 and variety: Imputed with most frequent value since they are categorical and the most frequent values of each feature align. Those are US, California, Napa Valley and Pinot Noir respectively.
One important consideration during imputing is, whatever imputing strategy is selected, both training and test sets are imputed with the same values. For instance, the median of the price in the training dataset is 25 and this value is assigned to missing values of the training dataset and the test dataset to prevent data leakage into the test set.
transformed_data = ImputeMissingValues(train_encoded, test_encoded)train_features = transformed_data[0]train_target = transformed_data[1]test_features = transformed_data[2]test_target = transformed_data[3]
Before moving on to the next article and the section, we are going to reserve some part of the training set as a validation set to provide an unbiased evaluation of a model.
Training dataset will be used for training the machine learning model. Training a machine learning model is the process of the grasping the patterns between the features and the target by our model, and using them for further predictions.
Validation dataset — the held-out part of the training dataset will be used as a measure of how well our model will generalize to a first-time-seen dataset. This measure will be one of the objective selection criteria of the Select an ML Model based on the Evaluation Metric part.
I can hear you thinking why don’t we use the test set for model selection then? Because, if we used the test set for this purpose, we would be selecting a biased model that is trained perfectly fit to our test set. Moreover, the model performance would not be representative.
The test set will come into the play when we are evaluating our fully-specified model at the Train and Evaluate the Model.
In light of the information above, let’s create the validation set by separating 25% of the training set randomly:
X_train, X_valid, y_train, y_valid = train_test_split(    train_features,     train_target,     test_size=0.25,     random_state=42)
Training, validation and test datasets are available under the notebooks/transformed in the repository. They are all ready to be fed into the wine rating predictor!
In this article, we completed the first three steps of a machine learning pipeline. After defining the problem and objectives, we
Cleaned data by removing redundant and collinear features; and duplicate rowsExplored data to have a detailed understanding of the target, features and their relationshipsAdded more useful features (hopefully!), prepared and transformed both datasets
Cleaned data by removing redundant and collinear features; and duplicate rows
Explored data to have a detailed understanding of the target, features and their relationships
Added more useful features (hopefully!), prepared and transformed both datasets
We also completed the crucial step of creating a validation set from the training set.
The second article will show how to select an evaluation metric, establish a baseline, select an ML model and tune its hyperparameters by optimizing the selected model’s evaluation metric (steps 4, 5 and 6).
towardsdatascience.com
The third article will train and evaluate the model with the training and test dataset respectively. It will show how to interpret an ML model’s predictions and draw conclusions (steps 7, 8, and 9).
towardsdatascience.com
The last article will automate this pipeline with Docker and Luigi.
towardsdatascience.com
Thanks for reading 😊 now the article series is complete and you can read the next ones!
For comments or constructive feedback, you can reach out to me on responses, Twitter or Linkedin!
Stay safe and healthy until then 👋