Create predictive models in R with Caret | by Luiz Fonseca | Towards Data Science

Caret is the short for Classification And REgression Training. It is a complete package that covers all the stages of a pipeline for creating a machine learning predictive model. In this tutorial, I will explain the following topics:
How to install caretHow to create a simple modelHow to use cross-validation to avoid overfittingHow to add simple preprocessing to your dataHow to find the best parameters for your chosen modelHow to see the most important features/variables for your modelHow to use your model to predict
How to install caret
How to create a simple model
How to use cross-validation to avoid overfitting
How to add simple preprocessing to your data
How to find the best parameters for your chosen model
How to see the most important features/variables for your model
How to use your model to predict
Installing caret is just as simple as installing any other package in R. Just use the code below. If you’re using RStudio (which is recommended), you can also install it by clicking on “tools” > “Install Packages...” in the toolbar.
install.packages("caret")
We’re gonna do that by using the train() function. The function train() is a core function of caret. As its name suggests, it is used to train a model, that is, to apply an algorithm to a set of data and create a model which represents that dataset.
The train() function has three basic parameters:
FormulaDatasetMethod (or algorithm)
Formula
Dataset
Method (or algorithm)
The formula parameter is where you specify what is your dependent variable (what you want to predict) and independent variables (features). I’ll explain more about how to write your formula below.
The dataset parameter is your data.
The method parameter is a string specifying which classification or regression model to use.
In this tutorial, I’m using the mtcars dataset. It is one of the built-in R datasets. Below, there is an explanation about this dataset:
The data was extracted from the 1974 Motor Trend US magazine, and comprises fuel consumption and 10 aspects of automobile design and performance for 32 automobiles (1973–74 models).
Format A data frame with 32 observations on 11 variables.
mpg: Miles/(US) galloncyl: Number of cylindersdisp: Displacement (cu.in.)hp: Gross horsepowerdrat: Rear axle ratiowt: Weight (1000 lbs)qsec: 1/4 mile timevs: V/Sam: Transmission (0 = automatic, 1 = manual)gear: Number of forward gearscarb: Number of carburetors
mpg: Miles/(US) gallon
cyl: Number of cylinders
disp: Displacement (cu.in.)
hp: Gross horsepower
drat: Rear axle ratio
wt: Weight (1000 lbs)
qsec: 1/4 mile time
vs: V/S
am: Transmission (0 = automatic, 1 = manual)
gear: Number of forward gears
carb: Number of carburetors
let’s take a look at the dataframe.
data(mtcars)    # Load the datasethead(mtcars)
?mtcars         # Get more information about this dataset
Now, let’s create regression models to predict how many miles per gallon (mpg) a car model can reach based on the other attributes.
The formula can be written as “x ~ y, z, w” where x is the dependent variable, mpg, in our case, and y, z and w are independent variables. If you want to pass all attributes you can write it as “x ~ .”.
library(caret)# Simple linear regression model (lm means linear model)model <- train(mpg ~ wt,               data = mtcars,               method = "lm")# Multiple linear regression modelmodel <- train(mpg ~ .,               data = mtcars,               method = "lm")# Ridge regression modelmodel <- train(mpg ~ .,               data = mtcars,               method = "ridge") # Try using "lasso"
That’s how you can use the function train() to create different basic models. Easy, isn’t it?
The function train() has other optional parameters. Let’s learn how to add resampling to our model by adding the parameter trControl (train control) to our train() function.
The resampling process can be done by using K-fold cross-validation, leave-one-out cross-validation or bootstrapping. We are going to use 10-fold cross-validation in this example. To achieve that, we need to use another Caret function, trainControl(). Check the code below.
## 10-fold CV# possible values: boot", "boot632", "cv", "repeatedcv", "LOOCV", "LGOCV"fitControl <- trainControl(method = "repeatedcv",                              number = 10,     # number of folds                           repeats = 10)    # repeated ten timesmodel.cv <- train(mpg ~ .,               data = mtcars,               method = "lasso",  # now we're using the lasso method               trControl = fitControl)  model.cv   ## The lasso ## ## 32 samples## 10 predictors## ## No pre-processing## Resampling: Cross-Validated (10 fold, repeated 10 times) ## Summary of sample sizes: 28, 30, 28, 28, 29, 29, ... ## Resampling results across tuning parameters:## ##   fraction  RMSE      Rsquared   MAE     ##   0.1       4.660950  0.8873870  3.841383##   0.5       2.824138  0.8735388  2.468004##   0.9       3.243825  0.8400985  2.797594## ## RMSE was used to select the optimal model using the smallest value.## The final value used for the model was fraction = 0.5.
The train() function has another optional parameter called preProcess. It’s used to add some pre-processing to your data.
In this example we’re going to use the following pre-processing:
center data (i.e. compute the mean for each column and subtracts it from each respective value);scale data (i.e. put all data on the same scale, e.g. a scale from 0 up to 1)
center data (i.e. compute the mean for each column and subtracts it from each respective value);
scale data (i.e. put all data on the same scale, e.g. a scale from 0 up to 1)
However, there are more pre-processing possibilities such as “BoxCox”, “YeoJohnson”, “expoTrans”, “range”, “knnImpute”, “bagImpute”, “medianImpute”, “pca”, “ica” and “spatialSign”.
model.cv <- train(mpg ~ .,               data = mtcars,               method = "lasso",               trControl = fitControl,               preProcess = c('scale', 'center')) # default: no pre-processing?train    # if you need more information about the train functionmodel.cv## The lasso ## ## 32 samples## 10 predictors## ## Pre-processing: scaled (10), centered (10) ## Resampling: Cross-Validated (10 fold, repeated 10 times) ## Summary of sample sizes: 29, 30, 29, 28, 29, 29, ... ## Resampling results across tuning parameters:## ##   fraction  RMSE      Rsquared   MAE     ##   0.1       4.597673  0.9072492  3.821535##   0.5       2.755728  0.8968303  2.411330##   0.9       3.183950  0.8820037  2.778551## ## RMSE was used to select the optimal model using the smallest value.## The final value used for the model was fraction = 0.5.
We can find the best hyperparameters for our model by using the tuneGrid parameter. This parameter receives A data frame with possible tuning values. The dataframe columns are named the same as the tuning parameters.
To generate the possible values, I am going to use the expand.grid function from the base library. To explain the use of tuneGrid I’m gonna use the ridge regression method.
The ridge method shrinks the coefficients of the predictor variables towards 0, as lambda grows. That shrinking effect decreases the model flexibility, decreasing its variance as well, but increasing bias. The idea of Ridge regression is to find a value for lambda that is a satisfying trade-off between bias and variance.
With the code below we can find the best lambda parameter for ridge regression between 10^-2 up to 10^10.
# Here I generate a dataframe with a column named lambda with 100 values that goes from 10^10 to 10^-2lambdaGrid <- expand.grid(lambda = 10^seq(10, -2, length=100))model.cv <- train(mpg ~ .,               data = mtcars,               method = "ridge",               trControl = fitControl,               preProcess = c('scale', 'center'),               tuneGrid = lambdaGrid,   # Test all the lambda values in the lambdaGrid dataframe               na.action = na.omit)   # Ignore NA valuesmodel.cv## Ridge Regression ## ## 32 samples## 10 predictors## ## Pre-processing: scaled (10), centered (10) ## Resampling: Cross-Validated (10 fold, repeated 10 times) ## Summary of sample sizes: 28, 29, 28, 29, 30, 28, ... ## Resampling results across tuning parameters:## ##   lambda        RMSE       Rsquared   MAE      ##   1.000000e-02   3.133764  0.8542752   2.700711##   1.321941e-02   3.097225  0.8559508   2.670390##   1.747528e-02   3.057315  0.8583961   2.637061##   2.310130e-02   3.015005  0.8621809   2.600386##   3.053856e-02   2.971543  0.8672769   2.562851... # too much output so i cut it out##   1.000000e+10  25.370412  0.8901093  23.047829## ## RMSE was used to select the optimal model using the smallest value.## The final value used for the model was lambda = 0.1629751.
When you call model.cv, you can see the metrics RMSE, Rsquared and MAE for each lambda value that you tested and the model also outputs the best choice for lambda among the values tested. In this case, it was lambda = 0.1629751.
There’s another way of searching for hyper-parameter without passing a list of values to the train() function. We can use search = “random” inside trainControl() and the function will automatically test a range of values.
fitControl <- trainControl(## 10-fold CV                           method = "repeatedcv",                           number = 10,                           repeats = 10,                           search = "random")  # hyper-parameters random search model.cv <- train(mpg ~ .,               data = mtcars,               method = "ridge",               trControl = fitControl,               preProcess = c('scale', 'center'),               na.action = na.omit)model.cv## Ridge Regression ## ## 32 samples## 10 predictors## ## Pre-processing: scaled (10), centered (10) ## Resampling: Cross-Validated (10 fold, repeated 10 times) ## Summary of sample sizes: 28, 29, 29, 29, 28, 28, ... ## Resampling results across tuning parameters:## ##   lambda        RMSE       Rsquared   MAE      ##   7.830203e-05   3.428075  0.8533706   2.950310##   9.214823e-05   3.427668  0.8533953   2.949997##   8.645309e+00  15.314730  0.9024834  13.994384## ## RMSE was used to select the optimal model using the smallest value.## The final value used for the model was lambda = 9.214823e-05.
Now let’s learn how to see which are the most important variables for our model. We can use the Caret function varImp. The return of varImp can be passed to the function ggplot to generate a visualization.
ggplot(varImp(model.cv))
As we can see in the graphic, the displacement variable is them ost important for our predictive model.
At last, we can use the function predict to predict a car’s performance, that is, how many miles it can reach per gallon. I’m gonna pass as argument the same dataframe used to generate the model just to show how the function works.
In a real project you would use a bigger dataframe and separate it into a train set and a test set, but that’s not the purpose here.
predictions <- predict(model.cv, mtcars)predictions##           Mazda RX4       Mazda RX4 Wag          Datsun 710 ##            22.59767            22.11103            26.25381 ##      Hornet 4 Drive   Hornet Sportabout             Valiant ##            21.23520            17.69025            20.38318 ##          Duster 360           Merc 240D            Merc 230 ##            14.38382            22.49686            24.41665 ##            Merc 280           Merc 280C          Merc 450SE ##            18.70104            19.19260            14.17669 ##          Merc 450SL         Merc 450SLC  Cadillac Fleetwood ##            15.60112            15.74344            12.03039 ## Lincoln Continental   Chrysler Imperial            Fiat 128 ##            10.93585            10.49555            27.77480 ##         Honda Civic      Toyota Corolla       Toyota Corona ##            29.89355            29.51135            23.64401 ##    Dodge Challenger         AMC Javelin          Camaro Z28 ##            16.94227            17.73144            13.30775 ##    Pontiac Firebird           Fiat X1-9       Porsche 914-2 ##            16.68855            28.29431            26.15559 ##        Lotus Europa      Ford Pantera L        Ferrari Dino ##            27.63590            18.87023            19.69368 ##       Maserati Bora          Volvo 142E ##            13.93917            24.37227
You can compare the values predicted with the real values. There are some metrics that you can use to see if you’re model is a good predictor, but we are not going to get into that here.
Thank you for reading. I hope it was a useful and pleasurable reading.