Quickly Compare Multiple Models. How to train and evaluate multiple... | by Cole Brendel | Towards Data Science

A question all data scientists must confront when working on any machine learning project is...
Which model architecture is going to best fit my data?
There are several theoretical considerations one should take into account. For example, if your features show strong linear relationships to your dependent variable (target) then a linear model is likely to work best. If the relationships are non-linear, then maybe an SVM or instance-based classifier like K-Nearest Neighbors would be best. If explainability in crucial, a tree based model may be the way to go. Additionally, there are several practical considerations you ought to take into account.
For the purposes of this article, I am going to assume you already know whether you are dealing with a supervised vs. unsupervised problem and whether you need classification or regression.
Unfortunately, there is no clear answer to the question of which model is best. When faced with uncertainty, the scientist in me always resorts to humanity’s most rigorous and sure-fire method for knowledge discovery: the experiment!
In this article, I will show you how to quickly test multiple models on your dataset to find which ones are likely to provide the best performance allowing you to focus your efforts on fine-tuning and optimizing your model.
Before we can start any experimentation, we need a dataset. I am going to assume our problem is a supervised binary classification task. Let’s load in the breast cancer dataset from sklearn to get started.
from sklearn.datasets import load_breast_cancerX, y = data = load_breast_cancer(return_X_y=True)
Next, we need to split the data into a training and test set. I’ve chosen a 75/25 split.
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=8675309)
We are now ready to run some experiments!
We are going to quickly test the fit of 6 different models on this dataset. I have chosen to test:
Logistic Regression: basic linear classifier (good to baseline)Random Forest: ensemble bagging classifierK-Nearest Neighbors: instance based classifierSupport Vector Machines: maximum margin classifierGaussian Naive Bayes: probabilistic classifierXGBoost: ensemble (extreme!) boosting classifier
Logistic Regression: basic linear classifier (good to baseline)
Random Forest: ensemble bagging classifier
K-Nearest Neighbors: instance based classifier
Support Vector Machines: maximum margin classifier
Gaussian Naive Bayes: probabilistic classifier
XGBoost: ensemble (extreme!) boosting classifier
To run the initial experiments, I used the default parameters for each model. To get a more accurate representation of the fit for each of these models, the default parameters would need to be adjusted; however, for the purposes of this article, the general idea is made more clear by not tweaking each model.
from sklearn.linear_model import LogisticRegressionfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.svm import SVCfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.naive_bayes import GaussianNBfrom xgboost import XGBClassifierfrom sklearn import model_selectionfrom sklearn.utils import class_weightfrom sklearn.metrics import classification_reportfrom sklearn.metrics import confusion_matriximport numpy as npimport pandas as pddef run_exps(X_train: pd.DataFrame , y_train: pd.DataFrame, X_test: pd.DataFrame, y_test: pd.DataFrame) -> pd.DataFrame:    '''    Lightweight script to test many models and find winners:param X_train: training split    :param y_train: training target vector    :param X_test: test split    :param y_test: test target vector    :return: DataFrame of predictions    '''        dfs = []models = [          ('LogReg', LogisticRegression()),           ('RF', RandomForestClassifier()),          ('KNN', KNeighborsClassifier()),          ('SVM', SVC()),           ('GNB', GaussianNB()),          ('XGB', XGBClassifier())        ]results = []    names = []    scoring = ['accuracy', 'precision_weighted', 'recall_weighted', 'f1_weighted', 'roc_auc']    target_names = ['malignant', 'benign']for name, model in models:        kfold = model_selection.KFold(n_splits=5, shuffle=True, random_state=90210)        cv_results = model_selection.cross_validate(model, X_train, y_train, cv=kfold, scoring=scoring)        clf = model.fit(X_train, y_train)        y_pred = clf.predict(X_test)        print(name)        print(classification_report(y_test, y_pred, target_names=target_names))results.append(cv_results)        names.append(name)this_df = pd.DataFrame(cv_results)        this_df['model'] = name        dfs.append(this_df)final = pd.concat(dfs, ignore_index=True)return final
There is a lot to unpack in this script. Firstly, we create a variable dfs to hold all of the datasets that will be created from the application of 5-fold cross validation on the training set.
Next, models in a list of tuples holding the name and class for each classifier to be tested. After this, we loop through this list and run 5-fold cross validation. The results of each run are recorded in a pandas dataframe which we append to the dfs list. One must note that the metrics recorded here are weighted mean metrics across both classes. This will not work well for any imbalanced dataset as performance on the majority class will overshadow the minority class. One may wish to tweak the script below to record only the metrics for the classes of interest!
To further aid in evaluation, a classification report on the test set is printed to screen. Finally, we concatenate and return all of our results.
To wrap up our analysis, we are going to analyze the data in the final dataframe returned from the run_exps() script.
To obtain better estimates of the distribution of metrics from each model, I run empirical bootstrapping at 30 samples. Additionally, I partition the data into two sorts: performance metrics and fit-time metrics. The following block of code achieves this.
bootstraps = []for model in list(set(final.model.values)):    model_df = final.loc[final.model == model]    bootstrap = model_df.sample(n=30, replace=True)    bootstraps.append(bootstrap)        bootstrap_df = pd.concat(bootstraps, ignore_index=True)results_long = pd.melt(bootstrap_df,id_vars=['model'],var_name='metrics', value_name='values')time_metrics = ['fit_time','score_time'] # fit time metrics## PERFORMANCE METRICSresults_long_nofit = results_long.loc[~results_long['metrics'].isin(time_metrics)] # get df without fit dataresults_long_nofit = results_long_nofit.sort_values(by='values')## TIME METRICSresults_long_fit = results_long.loc[results_long['metrics'].isin(time_metrics)] # df with fit dataresults_long_fit = results_long_fit.sort_values(by='values')
We now have some robust data to plot and analyze. First, let’s plot our performance metrics from the 5-fold cross validation.
import matplotlib.pyplot as pltimport seaborn as snsplt.figure(figsize=(20, 12))sns.set(font_scale=2.5)g = sns.boxplot(x="model", y="values", hue="metrics", data=results_long_nofit, palette="Set3")plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)plt.title('Comparison of Model by Classification Metric')plt.savefig('./benchmark_models_performance.png',dpi=300)
It is immediately clear that SVMs fit our data rather poorly across all metrics and that the ensemble decision tree models (Random Forest and XGBoost) fit the data very well.
How about the training and scoring times?
plt.figure(figsize=(20, 12))sns.set(font_scale=2.5)g = sns.boxplot(x="model", y="values", hue="metrics", data=results_long_fit, palette="Set3")plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)plt.title('Comparison of Model by Fit and Score Time')plt.savefig('./benchmark_models_time.png',dpi=300)
Wow, SVM just can’t catch a break! The worst performing model and slow to train/score!
It is interesting to note that XGBoost was far and away the slowest to train; however, it was the best performing. We generally see a trade-off in terms of performance and training time. Consider neural networks which require a massive amount of data and time to train, but which generally perform very well.
RandomForest, while relatively slow compared to KNN, GNB, and LogReg had the second-best performance. If I were to continue refining models, I would likely focus most of my effort on RandomForest because it performed nearly identical to XGBoost (their 95% confidence intervals might overlap!) but trained almost 4 times faster!
Should you wish to do more analysis on these models (e.g., calculate confidence intervals around each metric) you will need to access the means and stds for each metric. You will find this information here.
metrics = list(set(results_long_nofit.metrics.values))bootstrap_df.groupby(['model'])[metrics].agg([np.std, np.mean])
time_metrics = list(set(results_long_fit.metrics.values))bootstrap_df.groupby(['model'])[time_metrics].agg([np.std, np.mean])
You now have the tools to rapidly test many different models on your data and to see which architectures might offer the best fit.
I cannot stress enough that such comparisons are not exhaustive!
The above analysis considered only mean precision, recall, etc. In your actual problems, it is unlikely that you will care much about mean precision across your classes, instead, you may be focussed on precision for one class in particular! Additionally, one must tune the hyperparameters of each model to truly evaluate their fit to the data.
Data structures for statistical computing in python, McKinney, Proceedings of the 9th Python in Science Conference, Volume 445, 2010.
@software{reback2020pandas,    author       = {The pandas development team},    title        = {pandas-dev/pandas: Pandas},    month        = feb,    year         = 2020,    publisher    = {Zenodo},    version      = {latest},    doi          = {10.5281/zenodo.3509134},    url          = {https://doi.org/10.5281/zenodo.3509134}}
Harris, C.R., Millman, K.J., van der Walt, S.J. et al. Array programming with NumPy. Nature 585, 357–362 (2020). DOI: 10.1038/s41586–020–2649–2.
Scikit-learn: Machine Learning in Python, Pedregosa et al., JMLR 12, pp. 2825–2830, 2011.
J. D. Hunter, “Matplotlib: A 2D Graphics Environment”, Computing in Science & Engineering, vol. 9, no. 3, pp. 90–95, 2007.
Waskom, M. L., (2021). seaborn: statistical data visualization. Journal of Open Source Software, 6(60), 3021, https://doi.org/10.21105/joss.03021
Chen, T., & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 785–794). New York, NY, USA: ACM. https://doi.org/10.1145/2939672.2939785