Regression with PyCaret: A better machine learning library | by Dario Radečić | Towards Data Science

A couple of days back I’ve covered the basics of the PyCaret library and demonstrated how it defeats well-known Scikit-Learn when it comes to ease of use. Today we’ll get a bit more specific, and cover how regression tasks are handled. Reading the previous article isn’t a prerequisite, but feel free to skim over it if you haven’t used the library before.
Regression problems are everywhere around us — the most obvious being price prediction of any sort. That’s why we must know how to solve regression tasks, and how to solve them efficiently.
If you’re more of a video person, or just want to reinforce your knowledge, feel free to watch our video on the topic. Source code is included:
PyCaret is better than Scikit-Learn for regression tasks — development-time-wise. It all boils down to a couple of function calls. There’s no need to test between several learning algorithms and spending days tweaking the hyperparameters. PyCaret does all of that (and more) for you.
This article is aimed towards someone familiar with the concepts of regression in machine learning. Proficient knowledge isn’t required, as we’ll go over the regression theory briefly.
The article is structured as follows:
High-level regression overviewModel selection, training, and optimizationModel visualizationPredictions and model savingConclusion
High-level regression overview
Model selection, training, and optimization
Model visualization
Predictions and model saving
Conclusion
So without much ado, let’s get started!
I assume you already know what regression is. One paragraph from Investopedia summarizes it far better than I could:
“Regression is a statistical method used in finance, investing, and other disciplines that attempts to determine the strength and character of the relationship between one dependent variable (usually denoted by Y) and a series of other variables (known as independent variables).”
In the most simple terms — we want to fit a line (or hyperplane) through data points to obtain a line of best fit. The algorithm behind aims to find the line which minimizes the cost function — typically MSE or RMSE.
That’s linear regression, but there are other types — like polynomial regression. Unlike its little brother, polynomial regression doesn’t assume linear relationship and can sometimes yield better results. Just don’t go nuts with polynomial degrees — as the model tends to overfit that way.
We also have more sophisticated regression algorithms, like:
Random forests regression
XGBoost regression
In most cases, those will be superior then linear regression, at the cost of model interpretability. There’s no need to think about algorithm selection, as it happens automatically with PyCaret. Let’s see how in the next section.
Now comes the interesting part. We’ll use the real estate dataset from Kaggle. Here’s how it looks:
It needs a bit of cleaning, but nothing drastic. Here’s the code
df.drop([‘No’, ‘X1 transaction date’], axis=1, inplace=True)df.columns = [‘Age’, ‘DistanceToMTR’, ‘NumStores’, ‘Latitude’, ‘Longitude’, ‘UnitPrice’]
The situation is better now, but we still have one “problem”. If I’ve learned anything over the years, it is that pricing datasets are typically skewed. This means the target variable doesn’t follow a normal distribution.
Prediction-wise this is an issue because it affects the model’s performance. Down below is a histogram of the target variable:
Suspicion confirmed. To address this we’ll use a very scientific approach of eyeballing, and remove every record where the UnitPrice is larger than 70. There are better ways to approach this, sure, but I want to start with the modeling ASAP.
df = df[df[‘UnitPrice’] <= 70]
We now have something more normal-looking and can proceed with the modeling process. If you’ve read the previous article, you know this part can’t be any simpler. To start, let’s import the regression module from the PyCaret library and perform basic setup:
from pycaret.regression import *exp_reg = setup(df, target=’UnitPrice’)
After a couple of seconds, you’ll see a success message and a DataFrame with a bunch of information on your data. Feel free to read through it. Next, we can compare performance of a bunch of machine learning models and see which one does the best. It’s as simple as a function call:
compare_models()
Seems like Random forest algorithm did the best overall, so we’ll use it to create the model:
model = create_model(‘rf’)
Model tuning is now also a straightforward process:
tuned_model = tune_model(‘rf’)
Here’s the performance comparison between the original and tuned model:
So yeah, hyperparameter tuning, even though it lasted only a couple of seconds, reduced the RMSE from 6.17 to 5.82. Let’s now make a couple of visualizations from this tuned model.
For starters, let’s plot the residuals of our tuned model:
plot_model(tuned_model)
The performance on the train set is better, but that’s expected. Both train and test set residuals are normally distributed. Next, let’s use SHAP values to explain our model.
SHAP, or SHapley Additive exPlanations, is a way to explain the outputs of a machine learning model. We can use it to see which features are most important by plotting the SHAP values of every feature for every sample.
interpret_model(tuned_model)
Higher distance to MRT (typo on chart, sorry), leads to decrease in unit price. The same goes for the age of the property. The exact opposite is true with location (latitude and longitude). We haven’t visualized this data with geo charts, so it makes no sense for me to provide some reasoning.
Next, let’s make predictions and save our regression model.
Okay, let’s say we’re satisfied with how our model performs, and we want to make a final evaluation on the test set. Here’s how:
predictions = predict_model(tuned_model)
What you probably don’t know is that PyCaret made 70:30 train test split upon dataset loading, so we didn’t have to do this step manually. You can change this percentage, but 70:30 is a good place to start.
We see a bit worse results on the test set, but that’s expected. Before saving the model to a pickle file, we need to finalize it:
finalize_model(tuned_model)
Now the model can be saved for later use, in the environment of your choice:
save_model(tuned_model, ‘realestate_rf’)
And that’s it about regression tasks with PyCaret. Let’s wrap things up in the next section.
This was an interesting article. If you were following along, I hope you’ve spent some additional time preparing the data — and got better results in modeling. It would also be nice to make a geographical plot of this dataset since we have latitude and longitude information. That way we could better explain why the increase in both latitude and longitude increases the price.
Anyway, I hope you’ve managed to get something useful. There’s a lot more to explore, sure, but PyCaret enables us to spend more time on important things — like data preparation and model interpretation, instead of spending weeks training and tuning the model.
Thanks for reading.
Loved the article? Become a Medium member to continue learning without limits. I’ll receive a portion of your membership fee if you use the following link, with no extra cost to you.