Caret vs. tidymodels — create reusable machine learning workflows | by Hannah Roos | Towards Data Science

If you use machine learning models in R, you probably use either caret or tidymodels. Interestingly, both packages were developed by the same author among many others: Max Kuhn. But how do they compare to each other in terms of feasibility and performance?
You may wonder which package you should learn for predictive modelling in R. One thing right ahead: Caret is R’s traditional go-to machine learning package while tidymodels is rather unknown to parts of the R community. Accordingly, it was new to me: a smart medium user recommended tidymodels to me in order to automate the feature pre-processing section that I had shown in my last tutorial. Even though I have been a bit sceptical, I gave it a shot to find out if is any better than caret. Recently, I have challenged myself to predict employee churn on a small simulated dataset with caret — a perfect case to try this with tidymodels, too. It’s a pretty realistic example as well since some of the big players from industry already apply predictive analytics to decrease attrition and increase retention of their valuable talents. Using data mining techniques, historical and labelled employee data can be used to detect features that are associated with turnover (last pay raise, business travel, person-job fit, distance from home etc.). Hence, the algorithm is trained with data from the past to predict the future. This can be a serious problem too, because it allows for discriminatory biases easily. Thus, when a predictive model is applied, it must be constantly evaluated before it is already outdated and makes profound mistakes. Still, if we use them thoughtfully, I am convinced that the use of data analytics can be a powerful tool to improve the workplace for good.
As users slightly differ in their way to make use of the variety of machine learning algorithms available in R, Max Kuhn aimed to develop a uniform machine learning platform which allows for consistent and replicable code. In the R community, the good old caret has already played the role of the state-of-the-art package for a while when Max Kuhn also developed tidymodels in 2020 — the tidyverse version of caret so to speak. Similar to the dplyr-syntax (including %>%, mutate() etc.), tidymodels is based on the idea to structure your code in workflows in which each step is explicitly defined. This is intuitive and allows you to sequentially follow along what your program actually does. A lot of functions are borrowed from several core packages that are all already included in tidymodels, therefore it can be considered a meta- package:
rsample: for sample splitting (e.g. train/test or cross-validation)
recipes: for pre-processing
parsnip: for specifying the model
yardstick: for evaluating the model
dials: for hyperparameter tuning
workflow: for creating ML pipelines
broom: for tidying model outputs
Because tidymodels makes use of these packages, it is easier to learn if you are already familiar with them, but it is not really required.
Please note: If you are interested in the coding part only, you can find complete scripts in my GitHub repository here.
The dataset we will use for our case study is a simulated dataset created by IBM Watson Analytics which can be found on Kaggle. It contains 1470 employee entries and common 38 features (Monthly income, Job satisfaction, Gender etc.) — one of which is our target variable (Employee) Attrition (YES/NO). After having defined an R-project, we can use the neat here-package to set up our paths in a simple way that is more resistant against changes on your local machine. Let’s take a look at our raw data.
Disclaimer: All graphics are made by the author unless specified differently.
None of the observations are missing and the summary the skim function gives us shows some descriptive statistics including mean, standard deviation, percentiles as well as a histogram of each of the variables (26 numeric; 9 characters). In the real world, we would probably never have such a clean and complete dataset, but still we have another problem that is pretty realistic: it appears that 237 employees (16%) left the company while a majority (almost 84%) stayed, so the classes are not equally balanced. Class imbalance requires special treatment because it’s inconvenient to optimize and assess the model’s performance using conventional performance metrics (accuracy, AUC ROC etc.). The reason for this is the following: As accuracy is the proportion of correctly classified cases out of all cases, it would not be a big deal for the algorithm to give us a high score even if it simply classified ALL cases as the majority class (e.g., no) even some of them actually belong to the minority class (e.g., yes). But in our case, we intensely care about the positive cases as it is more detrimental NOT to correctly identify that an employee would leave (e.g., sensitivity or true positive rate) than accidentally predict that an employee would leave if the person actually stayed (false positive rate or 1 — specificity). Thus, we are going to use the F1-score as accuracy metric for training optimization because it assigns more importance to correctly classify positive cases (e.g., employee churn) and boosts the model’s performance in heavily imbalanced datasets.
As a first step, we can calculate any additional features. a payment that is perceived as unfair can influence a person’s intention to leave the job to look for better payment (Harden, Boakye & Ryan, 2018; Sarkar, 2018; Bryant & Allen, 2013). This is why we would like to create another variable that represents the payment competitiveness of each employee’s monthly income — the reasoning behind this is that employees may compare their income against those of their peers who share the same job level. Somebody who perceives his or her payment as fair should be less likely to leave the company compared to a person who gets considerably less for a similar position. To get there, we will use the data.table syntax to first calculate the median compensation by job level and store the appropriate value for each observation. Then we will divide each employee’s monthly income by the median income to get his or her compensation ratio: a measure that directly represents the person’s payment in respect to what would be expected by job level. Thus, a score of 1 means that the employee exactly matches the average payment for this position. A score of 1.2 means that the employee is paid 20% above the average pay and a score of 0.8 means that the person is paid 20% less than what would be expected by the usual payment per job level. As a next step, we remove all variables that are very unlikely to have any predictive power. For example, employee-ID won’t explain any meaningful variation in employee turnover, therefore it should be deleted for now among some other variables. Tidymodels provides us with the opportunity to assign roles to variables, for example such an ID column to retain the identifier while still excluding it from the actual modelling later. But we want to do it manually here to use the same training data for caret (in which we cannot assign a role). Other examples of variables that we want to exclude are those that obviously redundant and therefore could lead to multicollinearity issues (e.g., hourly rate and monthly income). We then save the reduced dataset by properly converting all string variables (e.g., Department) to factors at the same time.
The training data must be used to fit our models and tune our hyperparameters while we save some testing data for our final model evaluation. Why? We could run into the problem to overly fit our model to our sample-specific data so we cannot apply it to new employee data later. A problem that is often referred to as overfitting — a phenomenon that could explain why we sometimes cannot replicate previously found effects anymore. For machine learning models, we often split our data into training and validation/test sets to overcome this issue. The training set is used to train the model. The validation set serves to estimate model performance to tune hyperparameters accordingly. Finally, the test set is saved to challenge the predictive accuracy on data the model has never seen before. For tidymodels, the first rough split can be done using initial_split()which provides a list with indices for splitting which are tied to the main dataset. We use 70% of our data for training and save the remaining 30% for testing. Stratification is used with the target variable. It means that we give each model the chance to be both trained and tested on the same proportion of classes (e.g., 16% yes and 84 % no). Similarly, caret’s createDataPartionining() function automatically uses the outcome variable to balance the class distributions within the splits. By setting list = FALSE, we make sure that the function returns a matrix instead of a list. Thus, we can use the randomly assembled indices on our main dataset using the familiar data.table-syntax. To make the model even more generalizable to new data, we can use another magic trick from our statistical toolbox: cross-validation with various splits. We later apply our trained candidate model(s) to an extra set of observations and repeatedly adjust the parameters to reduce prediction error. These extra observations are several random samples from the training data, leaving out some of the data each time. This makes multiple folds of data to validate the hyperparameters on. We repeat this 3 times and average the performance to receive a preliminary estimation of our model’s performance — a resampling technique thus called 3-fold-cross-validation. Tidymodel’s vfold_cv() does the job for us — we define the number of folds that should be created as well as the number of repeats that should be used for resampling. When we use caret instead, we can use the createFolds() function for the same purpose, but for our comparison we do something better: tidymodels includes a function called rsample2caret() which transforms the folds we have already made in tidymodels to the familiar caret format. Thanks to rsample2caret()and caret2rsample() commands, it’s easy to use identical resamples in whichever package you prefer. This way, the performance metrics we calculate on our folds with each of the frameworks are less biased and more directly comparable.
Now, we use recipes to prepare our data for modelling with tidymodels: the idea is simple — we create an object that contains all the pre-processing steps that serve as ingredients to bake a nicely prepped dataset. This way, our data are ready to be ingested by the model. We need to take care of the order of pre-processing steps here because they are carried out in the order they entered (for example, it does not make much sense to normalize numeric predictors after dummy variables are created because 0 and 1 are treated as numeric values as well). You can think of a recipe as a blueprint that sketches our model formula, variable encodings and the logic of our pre-processing steps before their actual execution. This recipe is later included in a workflow — a structure that can be thought of as a kind of analysis pipeline that bundles your own prespecified feature engineering steps with the model specification.For our study, we want to create dummy variables, remove variables with near to zero variance as well as those that are already highly correlated with a similar variable to deal with multicollinearity. Moreover, we apply random oversampling to deal with class imbalance borrowed from the themis-package. Since there are considerably more negative cases than positive cases in our dataset, we can assume that our models won’t have a good chance to learn how to identify the minority class. The ROSE algorithm can deal with this by generating artificial additional observations with positive outcomes (e.g., leave the company) that mimic our true positives. So, the classes are more nicely balanced. If you are interested in how much class imbalance can change your performance metrics, take a look at my GitHub repo and run bal_log_example.Ras well as imbal_log_example.R.
Caret takes care of all feature engineering steps while training and even automatically handles dummy encoding, so we do not need to specify this in the preProcess argument in our train() call later. A downside of caret’s preProcess option is that has no option to selectively transform specific variables (e.g., certain numeric variables). To make use of the same upsampling technique as in our tidymodels example, we need to use a separate function on the training data imported by the ROSE package. We need to be careful here — for whatever reason, the function swaps the factor levels of our target variable which could affect performance metrics later. To avoid this, we manually assign the factor levels once again in a way that the first level is the positive class.
To create the appropriate models in tidymodels, we first use a recipe that contains a formula as well as the logic of our preprocessing steps. This recipe is later included in a workflow — a structure that can be thought of as a kind of analysis pipeline that bundles your own prespecified feature engineering steps with the model specification. The parsnip package helps us to tell the program which specific model to use. You can also connect an “engine” to the model specification by calling set_engine()to tell the model which underlying algorithm to use (Bayesians would maybe love to hear that you can even use stan). By setting the hyperparameters to tune(), we already make sure they are marked for tuning later. If we already know which setting to use because it worked well in the past, we can also pass it to model specification directly (e.g., if we want to have 200 trees). This step is not necessary if we use caret because we simply pass the model type to the method argument of our train function. Hyperparameters are then tuned with caret’s default tuning range and usually it produces very good results. Let’s say you are interested in which tuning range caret uses for glmnet, you can use getModelInfo(“glmnet")[[1]]$grid. Using tidymodels, we next need to prepare a param object using the parameters function from parsnip. Then, we pass them to the grid function of our choice — here I have applied an irregular grid to find close-to-optimal parameter combinations. In contrast to a grid search for which we go would iterate through all possible parameter combinations and assess their associated model performance, it can quickly become computationally costly and inefficient when the optimal solution is not represented in the grid. Using irregular grids, especially so called “space-filling designs”, chances are higher that we find a good combination more quickly while still covering the full parameter space.
Apparently, the idea of integrating multiple models into one object is not new to tidymodels! After having set up a recipe for pre-processing as well as a model specification, we create a workflow that bundles all the steps we have previously defined for preparing and modelling the data and fit them in one call. But we even go a step further. We create an even richer object that holds multiple workflows. These workflows (e.g., recipe, formula, model) are crossed in a way they can be later all fitted and tuned in a single request. In our case, we try several models, xgboost and glmnet, but this could be extended easily. This workflowset is now our master structure for training multiple models at once. By using option_add, we pass our different custom grids into the option column to our workflowset to modify the default tuning range. We can use the Github code from parsnip to find out which tuning parameters are available and correspond to the parameters we are used to from caret.
Before we start tuning our tidy models, we save a set of performance metrics that should be used to evaluate the goodness of our models. Now we use workflow_map to actually tune different models.
To specify our hyperparameter tuning in caret, we wrap everything including all preprocessing steps in a trainControl-Object.
Unfortunately, it is not possible to use F1 as our metric for optimization: therefore, I needed to write a custom function to make it work.
Unlike in tidymodels, there is no prespecified way to fit and tune different models at once. Thus, we can write our own function to do the job for us: we first create a custom function called train_model as a wrapper for caret’s train but with a placeholder for the method. Then we use lapply()to apply our custom function onto a list of methods. These contain the model types we want to compare with each other. The good thing about this approach is that we can easily adapt or extent the list at any time.
As we can see in the model objects above, we have tuning results for 108 parameter combinations for XGBoost and 9 for glmnet. This is the default that was automatically chosen by caret and the reason why I have specified the same tuning size for tidymodels previously to make a fair comparison between the two. Our tuning results from tidymodels have a similar structure like the nested tuning output from caret shown above, except for the fact that they provide us with more performance metrices. Just call the following:
While experimenting with the two frameworks, I have realized that the tuning process takes much more time in tidymodels compared to caret. Both packages handled a lot of candidate models — 108 combinations * 10 folds * 3 repeats = 3240 iterations for XGBoost and 9 combinations * 10 folds * 3 repeats = 270 iterations for glmnet. This should be very CPU intense anyways, but how do the packages deal with this task? To find out, I have wrapped each training-command into a system.time() to benchmark the timing differences. It turns out that caret just needs a fraction of the time that tidymodels allocates for tuning, just around 2% to be exact. While our tidymodel’s workflow_map() command took more than 6 hours to run through, caret accomplished the same task in less than 8 minutes. At least for my little experiment, tidymodels was 48 times slower than caret! Such a huge difference. Be aware though that this might be partly due to my old machine and could slightly vary each time you track the time. Therefore, I highly encourage you to try the microbenchmark package with the two functions: In case you have the time to evaluate the functions more than once, this should give you a more accurate and reliable result than system.time(). Nevertheless, the issue that tidymodels seems to be much slower than caret overall is not new: Max Kuhn argues that recipes slow down the training process because all pre-processing steps are re-applied within each resample. Another possible explanation comes down to differences in parallel computing between tidymodels and caret.
To find out which method works best on our problem, we need to measure performance for all of them. In our tidymodels example, we have saved all tuning results from the training data in model_race. We can use group by the workflow to collect metrices by model type. The resulting tibble provides us with several performance metrics that were computed for the models that each try a specific hyperparameter configuration from our grid. In our case it provides 10 folds x 3 repeats = 30 results per performance metric for each workflow. Now we can run tidymodel’s autoplot to show ranked workflow performance among candidate models.
So, what is our winning model? The plot shows that among all metrics, logistic regression (in blue) outperforms XGBoost on all candidate models. To compare model performances with caret, we use the resamples-function. It provides us with the range of F1 values across all 3 folds, giving us an estimation on how the model performs on different samples. By visualizing this output with bwplot(resamples), we can see how the performance is distributed across folds and then select the model with the highest average performance or lowest variance in performance. Unlike for tidymodels, we cannot see the performance across training folds for each parameter combination, but we get the range of performance for the best set of hyperparameters. Even if the plot suggests that XGBoost yields a slightly better performance, we will later make predictions with glmnet to make a valid comparison with tidymodels’ logistic regression.
Surprisingly, caret generally provides a much higher F1-score compared to tidymodels even though the F1_Score function from MLmetrics as well as the f_meas() function from yardstick both calculate the harmonic mean between recall and precision without any extra weight put on either of the metrics (balanced F1-score). Thus, the difference in performance is unlikely due to differences in calculation. At best, the difference results from a more targeted optimization process since caret trained our models to keep the F1-score as high as possible.
Still, we are not finished yet: For tidymodels, we need to commit to a model and then finalize it appropriately. In this case, we extract all glmnet candidate models from our model comparison. By running select_best() on the tuning results of our tidymodels logistic regression workflow, we get the performance metrics for the candidate model that ranked highest on the F1-score. To further commit on this model, we finalize our winning workflow: we extract the log_regworkflow from our model race with the set of hyperparameters that worked best numerically. This is how our finalized workflow looks like:
If we however look at caret’s final tuning result, we get other values for the same hyperparameters which could be due to differences in the optimization criteria: Maybe it’s because tidymodel’s glmnet model was not explicitly trained to maximize the F1-value?
print(model_glmnet)
For our tidymodels approach, we use our finalized workflow and fit it to the resamples to calculate performance metrices across folds. For this purpose, we can use fit_resamples()to assess model performance across different folds of the data. In contrast to fitting the finalized workflow to our whole training set, resampling gives us a closer estimation on how the model would perform on new samples. For the final conclusion on its performance, we still must wait for the evaluation on the test data though. As the next step, we use last_fit()to generate the final resampling results that are assessed on the test data. We can take a look at each model’s performance on the training data to confirm this observation.
This last_fit object does not only include the finalized workflow along with our final performance metrics, but it also contains a nice section on our predictions: the predicted classes, class probabilities and actual outcomes. These are nested within the last_fit object, so when we convert them to a data.frame, we can visualize the final predictions.For example, we can plot the density distributions of predicted probabilities coloured by actual attrition outcomes (inspired by a post by Julia Silge).
Because the distributions overlap quite a bit, it seems like there are a lot of cases in which the model predicted both outcomes (yes/no) to be almost equally likely. This also means that active employees and leavers were not that easily separable, but our performance metrics show that the algorithm assign just slightly more probability to the correct outcome in most of the cases. For the positive class, the classifier must have detected the majority of true leavers but also incorrectly left about 30 % of them to the negative class (and vice versa). What happens if we plot the predictions from caret?
Wow, it appears that more true negatives receive an attrition-probability close to zero while the classifier tends to assign a high attrition-probability to employees who actually left. But it also incorrectly classified about a third of true positives as well as true negatives... How does this performance translate to the whole training set? After fitting the model on the training data, we can make predictions using predict().
As hypothesized, about 30% of the data were incorrectly classified within each class which is also reflected in the similar levels of sensitivity (TPR) vs. specificity (TNR). To get similar information for our caret classifier, we can simply call predict.train()and use our model and training data as inputs, then we wrap the predictions in a ConfustionMatrix() command. Now we have a very nice overview of the training data performance.
The output underlines our assumption that caret has been a bit better in disentangling the positive from the negative class. In addition, we get much less false negatives compared to our tidymodels output, suggesting that recall should be relatively enhanced (76 % vs. 71 %).
We have been smart enough to save some of the data for testing purposes — now it is time for the models to actually showcase their prediction skills! The reason for this is that evaluating performance on the same data the classifiers were trained on yields a high risk of overfitting: the model highly depends on the data it has originally trained on, so performance metrics calculated on the same data are by no means independent. To challenge the model’s capacity to handle new observations, we therefore need to test it on data they have never seen before. In order to get performance metrics on the test data for tidymodels, we can call our last_fit object that already carries metrics on the test data for convenience.
It turns out that estimation on the training data would have indeed overestimated the model’s performance since except for specificity, all metrics are lowered a bit on the testing data (e.g., 0.61 vs. 0.71 for sensitivity). This can be further confirmed with our raw confusion matrix on the test data:
In caret, we can simply pass the model along with the target column of our test data to ConfusionMatrix()and also get some performance metrics.
The output suggests that the same methods again perform slightly better compared to tidymodels even though the differences are rather negligible. All in all, it yet again shows how much sense it makes to save some data for testing to avoid becoming overly optimistic about the model’s performance.
We can interpret it like this: The closer the line resembles an 90°-angle, the more specificity comes with high sensitivity. You can think of sensitivity as 1 — FPR, so this is the probability of the model to correctly identify all true positive cases as being positive (or true positive rate). Conversely, specificity is 1 — FNR, so it is the probability of the model to correctly identify all true negative cases as being negative (or true negative rate). Because sensitivity is plotted against 1- specificity, the area under the curve increases when a high sensitivity comes with a high specificity. You can think of the ROC curve as a kind of thought experiment: When we have a sensitivity of 0.75, the best possible value for 1 — specificity or false positive rate would be 0.00. In this case, the detection of true positives would come with all negative cases still identified as such. Imagine we would have a pretty rough model that would simply classify all cases as positives: we would have a perfect sensitivity (TPR), but a miserable specificity or FPR because true negatives (e.g., loyal employees) would also get a positive label. This is bad if you also care about the negative cases and want to differentiate them from the positives. So, the higher the sensitivity, the higher the risk for low specificity and vice versa, represented by the dotted diagonal on the plot. For tidymodels, a sensitivity of 0.75 comes with a specificity of 0.6 which is not a bad trade-off. Thus, as our model becomes smart enough to detect leavers (attrition = yes) while still also identifying active employees (attrition = no), the area under the curve increases.
To make a similar ROC-curve within our caret-framework, we can use MLeval to provide us with a range of ROC-related outputs. Overall, the area under the curve for glmnet appears to be a tiny bit larger compared to our tidymodels output and there is not a huge gap in performance between both frameworks.
Tidymodels comes with a high flexibility because it is based on various modern packages and has a completely customizable workflow structure. This means that you have a lot of degrees of freedom when it comes to creating your very own machine learning project. But because so many steps and objects are included, it can certainly be a bit confusing for beginners. During my own project, I felt like I got a better grasp of what the program does since I have worked my way around a lot of error messages, learning a lot of theory along the way. But this can also be due to the fact that tidymodels is still under development and therefore not as stable yet. If you want a quick and concise solution to your prediction problem instead of setting up a big project, I would recommend caret to you. It is not only faster in terms of runtime, but there are still more resourced and solved issues from experienced users out there. Caret fits a lot of models for you with very little coding work while being as fast as possible thanks to parallel processing. Unlike in tidymodels, the best candidate model across resamples is selected for you automatically which can be neat. One thing is for sure: Max Kuhn certainly did a great job with designing these packages and you are lucky to have two powerful open-source packages to choose from.
My special thanks goes to Robert Lohne, a consultant from Norway, who acted as my intellectual sparring-partner during this project. I really enjoyed our collaborative exchange and I hope that our learning journey won’t stop here!
[1] G. Harden, K. G. Boakye & S. Ryan, Turnover intention of technology professionals: A social exchange theory perspective (2018), Journal of Computer Information Systems, 58(4), 291–300.
[2] J. Sarkar, Linking Compensation and Turnover: Retrospection and Future Directions (2018), IUP Journal of Organizational Behavior, 17(1).
[3] P.C. Bryant & D. G. Allen, Compensation, benefits and employee turnover: HR strategies for retaining top talent (2013), Compensation & Benefits Review, 45(3), 171–175.