A comprehensive Machine Learning workflow with multiple modelling using caret and caretEnsemble in R | by Gabriel Pierobon | Towards Data Science

I’ll use a very interesting dataset presented in the book Machine Learning with R from Packt Publishing, written by Brett Lantz. My intention is to expand the analysis on this dataset by executing a full supervised machine learning workflow which I’ve been laying out for some time now in order to help me attack any similar problem with a systematic, methodical approach.
If you are thinking this is nothing new, then you’re absolutely right! I’m not coming up with anything new here, just making sure I have all the tools necessary to follow a full process without leaving behind any big detail. Hopefully some of you will find it useful too and be sure you are going to find some judgment errors from my part and/or things you would do differently. Feel free to leave me a comment and help me improve!
Let’s jump ahead and begin to understand what information we are going to work with:
From the book:
“In the field of engineering, it is crucial to have accurate estimates of the performance of building materials. These estimates are required in order to develop safety guidelines governing the materials used in the construction of building, bridges, and roadways.Estimating the strength of concrete is a challenge of particular interest. Although it is used in nearly every construction project, concrete performance varies greatly due to a wide variety of ingredients that interact in complex ways. As a result, it is difficult to accurately predict the strength of the final product. A model that could reliably predict concrete strength given a listing of the composition of the input materials could result in safer construction practices.
For this analysis, we will utilize data on the compressive strength of concrete donated to the UCI Machine Learning Data Repository (http://archive.ics.uci.edu/ml) by I-Cheng Yeh.
According to the website, the concrete dataset contains 1,030 examples of concrete with eight features describing the components used in the mixture. These features are thought to be related to the final compressive strength and they include the amount(in kilograms per cubic meter) of cement, slag, ash, water, superplasticizer, coarse aggregate, and fine aggregate used in the product in addition to the aging time (measured in days).”
I’ve found that splitting the workflow into 6 phases works best for myself. In that sense, I’ll describe this instances as: 1) Setting 2) Exploratory Data Analysis 3) Feature Engineering 4) Data Preparation 5) Modelling 6) Conclusion
In practice, I end up jumping from one to another, many times regardless of order and often in loop. So think of this just as an initial structure and more of a checklist, rather than a step by step guide.
The checklist is quite comprehending and covers most of what you would follow in supervised learning, both for classification and regression problems. In practice, you will most likely end up skipping some of the points from the list because there are things you do for a classification problem that you don’t do for a regression problem, vice versa. Even some of them are sometimes a little redundant (although double checking is always useful in my opinion). I’m sure that with more practice and experience, this list will get tuned and I’m hoping I can also share that again in the future.
I’ll limit myself to short-direct answers, and will expand whenever necessary. Otherwise I’m afraid I might make it too long for anyone to stick around!
You will see that even though I’m including the full checklist, in many cases we don’t have to do absolutely anything. This is because this dataset is pretty straightforward. It feels a little bit like cheating, however, I’ll make sure to mention why we don’t have to do anything and the idea behind that checkpoint.
The last thing I wanted to mention is that when I get to the modelling phase, I will change my approach and showcase some useful modelling tools with R packages caret and caretEnsemble. So instead of going step by step trying out some baseline models and them some more advanced models with hyper-parameter tuning and regularization to reduce overfitting, I’m just going to train a bunch of models at once, since it allows me to make this last phase much concise and straightforward for interpretation. Also because the modelling phase is where one likes to do it’s own thing and get creative by using different approaches. In my opinion, this checklist is very important for phases 1 to 4, but then starting phase 5, it can become less strict.
Enough said, let’s begin:
Find the working files and codes here: https://github.com/gabrielpierobon/ML-workflow-caret
1.1) What are we trying to predict?
We need to accurately estimate the performance of building materials for the field of engineering.
1.2) What type of problem is it? Supervised or Unsupervised Learning? Classification or Regression? Binary or Multi-class? Uni-variate or Multi-variate?
This is a multivariate supervised machine learning problem in which we have to predict numeric outcomes, thus we’ll be using regression techniques.
1.3) What type of data do we have?
Our data is in “csv” format. It presents a header row with the column names. It seems to contain only numeric information.
1.4) Import the dataset
We can simply load it with this code:
concrete <- read.csv(“/Users/Gabriel Pierobon/Desktop/Datasets para cargar/concrete.csv”)#use your own directory!
This is the size of our dataset:
dim(concrete)[1] 1030    9
1,030 rows and 9 columns, one of which is our response/target variable.
We can see that it was imported as a data frame which is the format that we require to work with this data:
class(concrete)[1] "data.frame"
We can also check that the names of our columns were correctly imported.
names(concrete)
From that list, we identify strength as our response/target variable.
We can conclude that our data was correctly imported and thus end our setting phase.
Before we do that, I’d like to require all our libraries altogether. It’s useful to do it all at once at the begging, to improve readability of this work:
library(dplyr)library(ggplot2)library(PerformanceAnalytics)library(ggthemes)library(corrplot)library(car)library(psych)library(caret)library(caretEnsemble)library(doParallel)
With that taken care of, let’s move on to exploratory data analysis:
2.1) View Data (str or dplyr’s glimpse). First look. Anything strange?
The fist way we want to double check what we did in the setting phase is to quickly view our full dataset. We do it like this:
View(concrete)
This will display a window with the dataset. I like to quickly look at it from top to bottom and left to right to reassure the fact that the data was loaded correctly. It’s also a quick and dirty way to detect issues that could be observable at first glance. You don’t want to do this for significantly large datasets.
Next, we are going to take a glimpse at our data and observe just the first few rows of the data frame:
glimpse(concrete)
head(concrete)
2.2) Is it a “tidy” dataset? Need to “gather” or “spread” it? Is it presented in a way we can work with?
We need our data to be presented with individual observations as rows and features as columns. Fortunately for us, this is the case of this dataset, so we won’t have to transform it. Otherwise, we would have to use some sort of function to pivot the data frame to accommodate to our needs (refer to this site for more about that)
2.3) rownames and colnames ok? Should we change them?
We have already checked how these were loaded. Do we need to update/change anything to have clearer understanding of our variables? I don’t think this is the case, we can move on. The idea here is to make sure we move forward comfortably with our feature names, avoiding unnecessary long names or any other sort of confusing situation.
2.4) Check data types. Are they OK? If not, convert
As we have seen, all our variables are of the double type, except for the variable age which is integer. Fortunately, we don’t have to convert anything! It’s very important that we check this in order to avoid some type of error when loading the data. Sometimes a single character in a numeric column can result in the whole column being loaded as character.
2.5) What is our response/target variable? Class imbalance? Study it
Our response variable is strength. Let’s look at some useful statistics about it:
summary(concrete$strength)
We can see that it ranges from 2.3 to 82.6. The median and the mean are really close, but since median is actually smaller, this results in a slight skew of the variable distribution to the right.
We can observe that with a plot using ggplot2:
As we can see, the distribution of the strength variable is not perfectly normal, but we are going to proceed regardless. This shouldn’t be a problem because it’s close. A qqplot also helps with the visual analysis of normality (I leave that to you)
Lastly, since this is a regression problem, we don’t have to worry about class imbalance. In classification, you want to have balanced classes in your response variable (here’s a nice post about class imbalance)
2.6) Rest of the features. Summary statistics. Understand your data
Here we extend the statistical analysis to our other variables/features. We want to pay attention at the minimums and maximums (this is a first check for potential outliers). Also, mean and median difference is something to be concerned about. We would like all our variables to follow the normal distribution as much as possible.
summary(concrete)
We can follow our analysis with a correlation plot. This will present us with a chart showing the correlation between all variables. It will also let us think for the first time if we need all our variables in our model. We don’t want our feature variables to present a high correlation between them. We will take care of this later.
corrplot(cor(concrete), method = “square”)
Another nice way of looking at all of this together is with this correlation plot:
chart.Correlation(concrete)
On one of the diagonals, we can see the distribution of each feature. Not all of them resemble normality, so we will deal with this later.
At this point my first conclusion is that the variable ash has low correlation with our response variable strength and a high correlation with four of the eight other features. It is thus a strong candidate to be removed, we’ll cover this later as well.
2.7) Categorical data/Factors: create count tables to understand different categories. Check all of them.
In this case we are not working with categorical features. You want to make sure you understand your categories, sometimes misspells can introduce problems for factor variables.
We can move on.
2.8) Unnecessary columns? Columns we can quickly understand we don’t need. Drop them
Here we want to look for columns which are totally useless. Anything that would come in the dataset that is really uninformative and we can determine that we should drop. Additional index columns, uninformative string columns, etc. This is not the case for this dataset. You could perfectly do this as soon as you import the dataset, no need to do it specifically at this point.
2.9) Check for missing values. How many? Where? Delete them? Impute them?
Let’s first do an overall check for NAs:
anyNA(concrete)[1] FALSE
Wonderful! no missing values in the whole set! Let me also show you a way we could have detected this for every column:
sapply(concrete, {function(x) any(is.na(x))})
(Some tools for dealing with missing values)
2.10) Check for outliers and other inconsistent data points. Box-plots. DBSCAN for outlier detection?
Outlier detection is more of a craft than anything else, in my opinion. I really like the approach of using DBSCAN clustering for outlier detection but I’m not going to proceed with this so I don’t overextend this analysis. DBSCAN is a clustering algorithm that can detect noise points in the data and not assign them to any cluster. I find it very compelling for outlier detection (more about DBSCAN here)
Instead, I’ll just go ahead with a boxplot and try to work with the points I consider relevant just by sight:
boxplot(concrete[-9], col = “orange”, main = “Features Boxplot”)
We see that there are several potential outliers, however I consider that the age feature could be the most problematic. Let’s look at it isolated:
boxplot(concrete$age, col = “red”)
Are these just 4 outliers? If so, we could just get rid of them. Or shouldn’t we? Let’s find out what this values are and how many of them are there.
age_outliers <- which(concrete$age > 100)concrete[age_outliers, “age”]
Oh my! so there were 62 of them instead of just 4! Obviously this is simply because the same numbers are repeated several times. This makes me think that this age points are actually relevant and we don’t want to get rid of them. 62 data points from our 1,030 dataset seems too high a number to just eliminate (we would be losing plenty of information).
2.11) Check for multicollinearity in numeric data. Variance Inflation Factors (VIF)
We’ve already seen in the correlation plot presented before that there seems to be significant correlation between some features. We want to make sure that multicollinearity is not an issue that prevents us to move forward. In order to do this, we will compute a score called Variance Inflation Factor (VIF) which measures how much the variance of a regression coefficient is inflated due to multicollinearity in the model. If VIF score is more than 10, multicollinearity is strongly suggested and we should try to get rid of the features that are causing it.
First, we generate a simple linear regression model of our target variable explained by each other feature. Afterwards, we call function vif()on the model object and we take a look at the named list output:
simple_lm <- lm(strength ~ ., data = concrete)vif(simple_lm)
Even though there are many variables scoring 5 and higher, none of them surpasses the threshold of 10 so we will consider multicollinearity not to be a big issue. However, some would argue that it could indeed be a problem to have as many features with scores of 7. We will not worry about that at this time.
With this, we consider the Exploratory Data Analysis (EDA) phase over and we move on to Feature Engineering. Feel free to disagree with this being over! I’m sure there still plenty of exploration left to do.
Feature engineering could also be considered a very important craft for a data scientist. It involves the production of new features obtained from the features present in the dataset. This could be as simple as extracting some dates information from string columns, or producing interaction terms. Moreover, it will certainly require some degree of subject matter expertise and domain knowledge, since coming up with new informative features is something intrinsic to the nature of the data and the overall field of application.
Here, we will go super quick, since our data seems to be already very informative and complete (engineers might disagree!).
I wanted to mention that engineering new features will most likely require that we repeat some of the previous analysis we have already done, since we don’t want to introduce new features that add noise to the dataset. So bear in mind that we must always look at our checklist back again and define if we need to analyze things one more time.
3.1) Create new useful features. Interaction Terms. Basic math and statistics. Create categories with if-else structures
Because I’m not a subject matter expert in engineering, I won’t create new features from interaction terms. I’ll just limit myself to verify if any of the variables require any type of transformation.
The first thing that I want to do is to check two variables which seem to have unusual distributions. It is the case of age and superplastic. Let’s plot their original form and also logs below (in red).
par(mfrow = c(2,2))hist(concrete$age)hist(concrete$superplastic)hist(log(concrete$age), col = “red”)hist(log(concrete$superplastic), col = “red”)
While I feel comfortable with converting age to it’s logarithmic form, in the case of superplastic, with as many observations being 0, I’ll have some issues when taking the log of 0, so I’ll take the log and then manually set those to 0.
Below, the code to convert both features:
concrete$age <- log(concrete$age)concrete$superplastic <- log(concrete$superplastic)concrete$superplastic <- ifelse(concrete$superplastic == -Inf, 0,                                  concrete$superplastic)head(concrete)
Note: I spent quite some time at this point trying to create a new superplastic feature by binning the original superplastic feature into 3 numeric categories. However, I didn’t have much success in terms of importance to explain the target variable. I won’t show those attempts, but just know that I indeed tried to work on that for some time unsuccessfully. Failing is also part of learning!
3.2) Create dummies for categorical features. Preferably using One-Hot-Encoding
We are not working with categorical features at this time, so this section will be skipped. However, if you were presented with some factor columns, you will have to make sure your algorithm can work with those, or otherwise proceed with one-hot-encoding them.
3.3) Can we extract some important text from string columns using regular expressions?
We are not working with sting data at this time, so this section will be skipped. This should be a very useful tool when you have some relevant information in character type from which you can create new useful categories.
Thankfully we didn’t have to go through all of that! However, those last 3 sections are a MUST if we are working with that type of information. I’ll make sure to analyze a dataset in which I can show some of that.
Let’s move on to the Data Preparation phase:
4.1) Manual feature selection. Remove noisy, uninformative, highly-correlated, or duplicated features.
Here’s where we take spend time for the second time in this workflow looking at our features and detecting if any of them are uninformative enough or should be dropped because of introducing problems as could be multicollinearity.
As I already had decided, I’ll first drop the ash feature (you might want to leave it and remove it after trying some baseline models, deciding to drop it if it does not help with performance, I already did that, so I’ll drop it now, as was my initial assessment)
concrete$ash <- NULLhead(concrete)
Besides that, I won’t remove anything else.
4.2) Transform data if needed. Scale or normalize if required.
Most of the algorithms we will use require our numeric features to be scaled or normalized (here’s why)
We won’t be doing that in this precise section, but leave it for later, since caret allows us to do some pre-processing within it’s training functionality. This is specifically useful because the algorithm will automatically transform back to it’s original scale when presenting us with predictions and results. If I manually normalized the dataset here, then I would have to manually transform back the predictions.
4.3) Automatic Feature extraction. Dimensionality Reduction (PCA, NMF, t-SNE)
If removing features manually isn’t straightforward enough but we still consider our dataset to contain too many features or too correlated features, we can apply dimensonality reduction techniques. Principal Components Analysis is one of those techniques, and a really useful one to both simplify our dataset, and also remove the issue of multicollinearity for good.
Non-negative Matrix Factorization (NMF) and t-SNE are other two useful dimensionality reduction techniques.
With just 8 features in our dataset and 1 already dropped manually, I don’t consider we require to reduce dimensionality any further.
4.4) Is our dataset randomized?
We are not sure it is randomized, so we will shuffle it just in case:
set.seed(123)concrete_rand <- concrete[sample(1:nrow(concrete)), ]dim(concrete_rand)
4.5) Define an evaluation protocol: how many samples do we have? Hold out method. Cross validation needed?
We have 1,030 samples, so this is definitely a small dataset. We will divide the dataset into train and test sets and make sure we use cross-validation when we train our model. In that way we ensure we are using our few observations as well as we can.
4.6) Split dataset into train & test sets (set seed for replicability)
We first create a set of predictors and a set of our target variable
X = concrete_rand[, -8]y = concrete_rand[, 8]
We check everything is OK:
str(X)
str(y)
We then proceed to split our new X(predictors) and y (target) sets into training and test sets.
Note: you don’t have to separate the sets, and can perfectly go ahead using the formula method. I just prefer to do it this way, because this way I’m sure I align my proceeding to how I’d work with Python’s scikit-learn.
We will use caret’s createDataPartition() function, which generates the partition indexes for us, and we will use them to perform the splits:
set.seed(123)part.index <- createDataPartition(concrete_rand$strength,                                   p = 0.75,                                                           list = FALSE)X_train <- X[part.index, ]X_test <- X[-part.index, ]y_train <- y[part.index]y_test <- y[-part.index]
So now we have 4 sets. Two predictors sets splitted into train and test, and two target sets splitted into train and test as well. All of them using the same index for partitioning.
Once again, let’s check everything worked just fine:
str(X_train)str(X_test)str(y_train)str(y_test)
OK! We are good to go! Now to the modeling phase!
As I mentioned in the introduction, in this phase I will change my approach, and instead of going through a check-list of things to do, I will summarize altogether how we will proceed.
We will use package caretEnsemble in order to train a list of models all at the same time
This will allow us to use the same 5 fold cross-validation for each model, thanks to caret’s functionality
We will allow parallel processing to boost speed
We won’t focus on the nature of the algorithms. We will just use them and comment on the results
We will use a linear model, a support vector machines with radial kernel, a random forest, a gradient boosting tree based model and a gradient boosting linear based model
We won’t do manual hyper-parameter tuning, instead we will allow caret to go through some default tuning in each model
We will compare performance over training and test sets, focusing on RMSE as our metric (root mean squared error)
We will use a very cool functionality from caretEnsemble package and will ensemble the model list and then stack them in order to try to produce an ultimate combination of models to hopefully improve performance even more
So let’s move on.
We first set up parallel processing and cross-validation in trainControl()
registerDoParallel(4)getDoParWorkers()set.seed(123)my_control <- trainControl(method = “cv”, # for “cross-validation”                           number = 5, # number of k-folds                           savePredictions = “final”,                           allowParallel = TRUE)
We then train our list of models using the caretList() function by calling our X_train and y_train sets. We specify trControl with our trainControl object created above, and set methodList to a list of algorithms (check the caret package information to understand what models are available and how to use them).
set.seed(222)model_list <- caretList(X_train,                        y_train,                        trControl = my_control,                        methodList = c(“lm”, “svmRadial”, “rf”,                                        “xgbTree”, “xgbLinear”),                        tuneList = NULL,                        continue_on_fail = FALSE,                         preProcess = c(“center”,”scale”))
I use X_train and y_train, but you can perfectly use y ~ x1 + x2 + ... + xn formula instead
my_control specified the 5-fold cross-validation and activated the parallel processing
tuneList is FALSE because we are not specifying manual hyper-parameter tuning
continue_on_fail is set to FALSE so it stops if something goes wrong
in preProcessing is where we scale the dataset. We choose “center” and “scale”
Now that our caretList was trained, we can take a look at the results. We can access each separate model. Here’s the SVM result:
model_list$svmRadial
Notice caret tries some automatic tuning of the available parameters for the model, and chooses the best model using RMSE as performance metric.
This is the same for each of the other models in our list of models. We won’t go through each one of them. That’s for you to check!
Let’s go straight to our goal, which is finding the model that has the lowest RMSE. We first asses this for the training data.
options(digits = 3)model_results <- data.frame( LM = min(model_list$lm$results$RMSE), SVM = min(model_list$svmRadial$results$RMSE), RF = min(model_list$rf$results$RMSE), XGBT = min(model_list$xgbTree$results$RMSE), XGBL = min(model_list$xgbLinear$results$RMSE) )print(model_results)
In terms of RMSE, the the xgbTree model offers the best result, scoring 4.36 (remember the mean strength was 35.8).
caretEnsemble offers a functionality to resample the performance of this model list and plot it:
resamples <- resamples(model_list)dotplot(resamples, metric = “RMSE”)
We can also see that the xgbTree is also presenting a smaller variance compared to the other models.
Next, we will attempt to create a new model by ensembling our model_list in order to find the best possible model, hopefully a model that takes the best of the 5 we have trained and optimizes performance.
Ideally, we would ensemble models that are low correlated with each other. In this case we will see that some high correlation is present, but we will choose to move on regardless, just for the sake of showcasing this feature:
modelCor(resamples)
Firstly, we train an ensemble of our models using caretEnsemble(), which is going to perform a linear combination with all of them.
set.seed(222)ensemble_1 <- caretEnsemble(model_list,                             metric = “RMSE”,                             trControl = my_control)summary(ensemble_1)
As we can see, we managed to reduce RMSE for the training set to 4.156
Here’s a plot of our ensemble
plot(ensemble_1)
The red dashed line is the ensemble’s RMSE performance.
Next, we can be more specific and try to do an ensemble using other algorithms with caretStack().
Note: I tried some models but wasn’t able to improve performance. I’ll show just one of them which yielded the same performance over the training data. We will use both ensembles regardless, in order to check which one does better with unseen data.
set.seed(222)ensemble_2 <- caretStack(model_list,                          method = “glmnet”,                          metric = “RMSE”,                          trControl = my_control)print(ensemble_2)
Notice RMSE for the best model using glmnet was 4.15, the same as our first ensemble.
Finally, it’s time to evaluate the performance of our models over unseen data, which is in our test set.
We first predict the test set with each model and then compute RMSE:
# PREDICTIONSpred_lm <- predict.train(model_list$lm, newdata = X_test)pred_svm <- predict.train(model_list$svmRadial, newdata = X_test)pred_rf <- predict.train(model_list$rf, newdata = X_test)pred_xgbT <- predict.train(model_list$xgbTree, newdata = X_test)pred_xgbL <- predict.train(model_list$xgbLinear, newdata = X_test)predict_ens1 <- predict(ensemble_1, newdata = X_test)predict_ens2 <- predict(ensemble_2, newdata = X_test)# RMSEpred_RMSE <- data.frame(ensemble_1 = RMSE(predict_ens1, y_test),                        ensemble_2 = RMSE(predict_ens2, y_test),                        LM = RMSE(pred_lm, y_test),                        SVM = RMSE(pred_svm, y_test),                        RF = RMSE(pred_rf, y_test),                        XGBT = RMSE(pred_xgbT, y_test),                        XGBL = RMSE(pred_xgbL, y_test))print(pred_RMSE)
Surprisingly, the xgbLinear model out performs every other model on the test set, including our ensemble_1 and matching the performance of the ensemble_2
We also observe that in general , there is a difference in performance compared to the training set. This is to be expected. We could still try to manually tune hyper-parameters in order to reduce some overfitting, but at this point I believe we have achieved very strong performance over unseen data and I will leave further optimization for a future publication.
The last thing I wanted to show is variable importance. In order to do this, I will calculate our xgbLinear model separately, indicating I want to retain the variable importance and then plot it:
set.seed(123)xgbTree_model <- train(X_train,                       y_train,                       trControl = my_control,                       method = “xgbLinear”,                       metric = “RMSE”,                       preProcess = c(“center”,”scale”),                       importance = TRUE)plot(varImp(xgbTree_model))
Here we can see the high importance of variables age and cement to the prediction of concrete’s strength. This was to be expected since we had already observed a high correlation between them in our initial correlation plot.
Working with the log of age has also allowed us to improve predictability (as verified separately).
Notice some “unimportant” features present in the chart. Should we have dropped them? Can we simplify our model without impacting performance? We could certainly keep working on that if we needed a simpler model for any reason.
This has been quite the journey! We moved along most of the necessary steps in order to execute a complete and careful machine learning workflow. Even though we didn’t have to do a whole lot of changes and transformation to the original data as imported from the csv, we made sure we understood why and what we should have done otherwise.
Moreover, I was able to showcase the interesting work one can do with caret and caretEnsemble in terms of doing multiple modelling, quick and dirty, all at once and being able to quickly compare model performance. The more advanced data scientists and machine learning enthusiasts could possibly take this as a first draft before proceeding with the more advanced algorithms and fine hyper-parameter tuning that will get those extra bits of performance. For the sake of this work, it proved to be really strong even with the basic configuration.
In the book mentioned in the introduction, the author calculates correlation between predictions (using a neural network with 5 hidden layers) and the true values, obtaining a score of 0.924. It also mentions that compared to the original publication (the one his work was based on), this was a significant improvement (original publication achieved 0.885 using a similar neural network)
So how did we do computing the same correlation score as the book’s author?
pred_cor <- data.frame(ensemble_1 = cor(predict_ens1, y_test),                       ensemble_2 = cor(predict_ens2, y_test),                       LM = cor(pred_lm, y_test),                       SVM = cor(pred_svm, y_test),                       RF = cor(pred_rf, y_test),                       XGBT = cor(pred_xgbT, y_test),                       XGBL = cor(pred_xgbL, y_test))print(pred_cor)
Pretty strong performance!