Should you Buy or Lease your Next Car? End-to-End Project
By: Luke Newman
Tags: Machine Learning, Regression, Data Visualization, Data Science, Ensemble Learning

Should you Buy or Lease your Next Car? End-to-End Project

Explore the project from collecting data to creating ensemble of strong regressors to answering whether you should buy or lease your next car

Overview

In this post we will walk through an entire data science project from end-to-end and come up with a conclusion to the question––should you buy or lease your next car?

The project will include the following sections (feel free to skip ahead to any section your most interested in):

Frame the Problem and Look at the Big Picture

Get the Data

Explore the Data

Prepare the Data

Select and Train Models

Fine-Tune the System

Analyze Errors of Best Model

Conclusion

To view the project on Github click on the following link here!

Frame the Problem and Look at the Big Picture

The goal of this project is to determine whether car leases are a good deal by comparing the total expense of a lease with the depreciation cost of a car. If the lease is more expensive we can conclude that it is more cost effective to buy the car new and resell it. However, if the lease is less expensive than we conclude you should lease instead of buy.

By using machine learning we predict the price of cars and use it to calculate the depreciation cost with which we compare to the cost of its lease.

The metric we use is Root-Mean-Squared-Error (RMSE) because it gives an idea of how much error the system typically makes in its predictions, with a higher weight for large errors.

So now we need data. Instead of collecting data on hundreds of different cars we only collect data on one of the best rated cars to lease, the Acura TLX. This helps us simplify the problem and if we conclude that you should buy instead of lease this car, we can conclude the same for lower rated cars.

Get the Data

In this project we collect our data using BeautifulSoup’s library. We webscrape Acura TLX car listings off of Truecar.com. The notebook complete with code used for webscraping can be found here.

The code for the rest of the project can be viewed here.

Let’s load in the data we collected by webscraping and take a first initial look into it.

CARS_PATH = os.path.join(PROJECT_ROOT_DIR, "datasets")



def load_car_data(cars_path=CARS_PATH):

csv_path = os.path.join(cars_path, "cars.csv")

return pd.read_csv(csv_path) cars = load_car_data()

Let’s see how complete our dataset is. By looking at the percentage of null values for each column we can get a good idea of how many missing values we are dealing with.

cars.isnull().sum() / len(cars)

Percentage Null of Each Column

We see there are missing values in 6 of our columns. Specifically in our target variable Price and attributes Drive, Engine, Trim, Exterior color, and Interior Color. Since Price is our target variable we will have to drop those instances. Since Drive, Engine, and Trim have a very low missing value percentage it should be okay to drop the instances pertaining to those instances as well. Since Exterior and Interior color are more than 10% null and probably won’t have much predictive power on price we will go ahead and drop those columns as well from the dataset.

cars = cars.drop(columns=["Exterior_color", "Interior_color"])

cars = cars.dropna(axis=0)

cars.reset_index(inplace=True)

Create a Test Set

We use a stratified split here to ensure our test set is as representative of the various ratios in the whole dataset. Since we believe certain features like the year of the car will be very important predictors of price we ensure that they are evenly distributed between the training and test set. By using a stratified split we avoid a sampling bias where the test is not representative of the whole dataset.

The following code uses the pd.cut() function to create a year category attribute with 6 categories (labeled from 0 to 5 and corresponding to each year in the dataset):

cars["Year_cat"] = pd.cut(cars["Year"], bins=[2014.5, 2015.5, 2016.5, 2017.5, 2018.5, 2019.5, 2020.5], labels=[0, 1, 2, 3, 4, 5]

Now we are ready to do stratified sampling based on the year attribute. For this we can use Scikit-Learn’s StratifiedShuffleSplit() class:

from sklearn.model_selection import StratifiedShuffleSplit



split = StratifiedShuffleSplit(n_splits=1, test_size=0.2)

for train_index, test_index in split.split(cars, cars["Year_cat"]):

train_set = cars.loc[train_index]

test_set = cars.loc[test_index]

Now we remove the Year_cat attribute so the data is back to its original state:

for set_ in (test_set, train_set):

set_.drop("Year_cat", axis=1, inplace=True)

Explore the Data

Let us now discover and visualize the data to get a feel of what it looks like and gain insights. We create a copy of the training set so we can play with it without harming the original training set:

cars = train_set.copy()

Below is a table with the columns, data type, and a short description detailing what information they hold.

We will start by looking at the numerical data types and computing the standard correlation coefficient between every pair of attributes using the corr() method:

corr_matrix = cars.corr()

Now let’s look at how much each attribute correlates with the price:

corr_matrix["Price"].sort_values(ascending=False)

Another way to check for correlation between attributes is to use pandas scatter_matrix() function, which plots every numerical attribute against every other numerical attribute.

from pandas.plotting import scatter_matrix



attributes = ["Price", "Year", "Mileage", "Accidents"]

scatter_matrix(cars[attributes], figsize=(12,8))

save_fig("scatter_matrix_plot")

The figure above shows us a couple things. Indeed the attributes year and mileage are going to be very strong predictors of price and show some interaction that we might be able to capture by creating a new feature, miles per year. Also, there is not many instances with 3 accidents. It might be beneficial to group cars with 2 or 3 accidents into one value of 2 or more.

To create the miles per year attribute we need to know how many years old the car is instead of the year. We achieve this through taking the absolute value of the current year subtracted from the year of the car. However, we won’t be able to divide by 0 for 2020 Acura TLXs so we will replace 0s with 1s (in this case the attribute miles_per_year will be the same as mileage).

cars["Years_old"] = abs(cars["Year"] - 2020)

cars["Miles_per_year"] = cars["Mileage"] /

cars["Years_old"].replace(0, 1)

Now let’s look at the correlation of our target price with the new attribute:

corr_matrix = cars.corr()

corr_matrix["Price"].sort_values(ascending=False)

Great! we see it is strongly correlated with the price as well!

One last interesting column I want to look at is Trim. This column contains a lot of information including the type of performance package on the car. It might be beneficial to extract the package and use it as an attribute instead. Let’s observe the correlation between different trims and median price.

cars[["Price","Trim"]].groupby("Trim").median()

Now let’s extract the package and see if there is any obvious correlation between packages.

def extract_package(trims):



packages = ["PMC Edition", "A-Spec", "Advance", "Technology"]



pat = '|'.join(r"\b{}\b".format(x) for x in packages)

packages = pd.Series(trims).str.extract("(" + pat + ")")

packages.fillna("Standard", inplace=True)



return packages cars["Package"] = extract_package(cars["Trim"].values) cars[["Price", "Package"]].groupby("Package").median()

Extracting the package does not seem very helpful. The correlation is weak between different packages and does not follow any trends we expect.

Summary

Now we have a pretty good idea of which features will be the most valuable and have explored relations between them. Some important findings were:

Creating new column miles_per_year adds additional predictive power.

adds additional predictive power. Cars with more than 2 accidents are very sparse in our dataset and it would be better to group together with cars having 2 accidents. We preserve a stronger correlation this way.

Year will be better to use as an ordinal feature. We will subtract the current year of the column to create a column called “years old”.

Extracting the performance package from trim might or might not be useful. There isn’t a clear correlation with price so we can add this as an option in our data preparation phase.

Prepare the Data

It’s time to prepare the data for Machine Learning algorithms. Instead of doing this manually, we write functions for this purpose, for several good reasons:

This will allow us to reproduce these transformations on any dataset (e.g., the next time you get a fresh dataset).

We will gradually build a library of transformation functions that we can reuse in future projects.

This will make it possible for us to easily try various transformations and see which combination of transformations work best.

We need a preparation pipeline for numerical attributes and categorical attributes. Let’s start with the numerical attributes!

Since we are using Scikit-Learn we want our transformers to work seamlessly with Scikit-Learn functionalities (such as pipelines), but thankfully for us all we need to do is create a class and implement three methods: fit() (which returns self), transform(), and fit_transform(). We get the last one for free by simply adding TransformerMixin as a base class.

We also add BaseEstimator as a base class which gives us two extra methods (get_params() and set_params()) that will be useful for automatic hyperparameter tuning.

For example, here is a transformer class that adds the numerical attributes which were discussed earlier and its output when we pass it our training set.

from sklearn.base import BaseEstimator, TransformerMixin



class NumericalAttributesAdder(BaseEstimator, TransformerMixin):

def __init__(self, add_miles_per_year = True):

self.add_miles_per_year = add_miles_per_year

def fit(self, X, y=None):

return self

def transform(self, X):

years_old = abs(X["Year"] - 2020)

accidents_adjusted = X["Accidents"].replace(3, 2)

if self.add_miles_per_year:

miles_per_year = X["Mileage"] / years_old.replace(0, 1)

return pd.concat([X, years_old.rename("Years_old"),

accidents_adjusted.rename("Accidents_adj"),

miles_per_year.rename("Miles_per_year")],

axis=1) else:

return pd.concat([X, years_old.rename("Years_old"),

accidents_adjusted.rename("Accidents_adj")],

axis=1)



num_attr_adder = NumericalAttributesAdder(add_miles_per_year = True)

num_attr_adder.transform(X_train)

Now after we add new attributes we want to select which attributes we use in our machine learning algorithms. We use the same method as above to create a custom transformer that will select the attributes.

class DataFrameSelector(BaseEstimator, TransformerMixin):

def __init__(self, attribute_indices):

self.attribute_indices = attribute_indices

def fit(self, X, y=None):

return self

def transform(self, X):

return X[self.attribute_indices]

As you can see, there are many data transformation steps that need to be executed in the right order. Fortunately, Scikit-Learn provides the Pipeline class to help with sequences of transformations. Here is the pipeline for the numerical attributes:

from sklearn.pipeline import Pipeline

from sklearn.preprocessing import StandardScaler attribs = ["Mileage", "Years_old", "Accidents_adj","Miles_per_year"] num_pipeline = Pipeline([

('num_attribs_adder', NumericalAttributesAdder()),

('select_numeric', DataFrameSelector(attribs)),

('std_scaler', StandardScaler())

])

We used our two custom transformers from above and then Scikit-Learn’s StandardScaler() transformer to standardize all numerical attributes.

Now the next step is to create a pipeline for our categorical attributes. Since these attributes are strings, we need to use OneHotEncoder to transform them into numerical values. But first we create a custom transformer that will extract the package of a car from the Trim attribute we discussed earlier. We initialize add_trim_features=True to give us the option whether to add the new package attribute.

class CategoricalAttributesAdder(BaseEstimator, TransformerMixin):

def __init__(self, add_trim_features = True):

self.add_trim_features = add_trim_features

def fit(self, X, y=None):

return self

def transform(self, X):

if self.add_trim_features:

package = extract_package(X["Trim"])

return pd.concat([X, package.rename("Package")], axis=1)

else:

return X cat_attr_adder = CategoricalAttributesAdder(add_trim_features =True) cat_attr_adder.transform(X_train)

Now the full pipeline for our categorical attributes is ready. We use CategoricalAttributesAdder, DataFrameSelector, and OneHotEncoder to completely transform our categorical attributes into a format suited for Machine Learning. The pipeline is shown below:

attribs = ["Drive", "Engine", "Trim"] cat_pipeline = Pipeline([

("cat_attribs_adder", CategoricalAttributesAdder()),

("select_categoric", DataFrameSelector(attribs)),

("one_hot", OneHotEncoder())

])

So far, we have handled the categorical columns and numerical columns separately. It would be more convenient to have a single transformer able to handle all columns, applying the appropriate transformations to each column. Fortunately, Scikit-Learn introduced the ColumnTransformer for this purpose, and the good news is that it works great with pandas DataFrames. Let’s use it to apply all the transformations to the training set:

from sklearn.compose import ColumnTransformer num_attribs = ["Year", "Mileage", "Accidents"]

cat_attribs = ["Drive", "Engine", "Trim"] full_pipeline = ColumnTransformer([

("num", num_pipeline, num_attribs),

("cat", cat_pipeline, cat_attribs)

])



cars_prepared = full_pipeline.fit_transform(X_train)

In this pipeline, we specify that the numerical columns should be transformed using the num_pipeline that we defined earlier, and the categorical columns should be transformed using the cat_pipeline we defined earlier. Finally, we apply this ColumnTransformer to the training set: it applies each transformer to the appropriate columns and concatenates the outputs along the second axis (the transformers must return the same number of rows).

And that’s it! We have a preprocessing pipeline that takes the full training set and applies the appropriate transformations to each column.

Select and Train Models

At last! We framed the problem, got the data and explored it, sampled a training set and a test set, and we wrote transformation pipelines to clean up and prepare our data for Machine Learning algorithms automatically. We are now ready to select and train Machine Learning models.

Our first base model is a Linear Regression with which we will try to improve from. Using Scikit-Learn’s cross_val_score() allows us to randomly split the training set into 10 distinct folds, and then train and evaluates our model 10 times, picking a different fold for evaluation each time and training on the other 9 folds. The result is an array containing the 10 evaluation scores:

from sklearn.linear_model import LinearRegression

from sklearn.model_selection import cross_val_score lin_reg = LinearRegression()

lin_reg_scores = np.sqrt(-cross_val_score(lin_reg, cars_prepared,

y_train, cv=10, scoring="neg_mean_squared_error"))

Let’s look at the results:

def display_scores(scores):

print("Scores:", scores)

print("Mean:", scores.mean())

print("Standard deviation:", scores.std()) display_scores(lin_reg_scores)

Not bad for our base model! But instead of displaying each model’s scores like this we will plot each model’s scores in a box plot showing the mean, interquartile range, and max/min scores. Remember our metric is RMSE and the lower score the better. Here is a list of the models we try:

Linear Regression

Polynomial Regression

Polynomial Regression with Ridge Regularization

Random Forest Regressor

Support Vector Machine Regressor

Gradient Boosting Regressor

We see that Polynomial Regression did slightly better than plain Linear Regression. Also, Random Forest Regressor and Gradient Boosting Regressor performed significantly better while SVM Regressor performed significantly worse. Remember we are just trying to get a general idea of which Machine Learning algorithms perform the best so all of these algorithms were trained with default settings.

Fine-Tune your System

Now we have a shortlist of promising models. We now need to fine-tune them. Since we trained our models using default hyperparameters we will try and see which hyperparameters work best for our best models, Random Forest Regressor and Gradient Boosting Regressor.

Instead of doing this manually, we get Scikit-Learn’s GridSearchCV to search for us. All we need to do is tell it which hyperparameters we want it to experiment with and what values to try out, and it will use cross-validation to evaluate all possible combinations of hyperparameter values.

Random Forest Regressor

from sklearn.model_selection import GridSearchCV



forest_param_grid = [{'bootstrap': [True],

'max_depth': [12, 14, None],

'max_features': ['auto', 'sqrt'],

'min_samples_leaf': [1, 2],

'min_samples_split': [ 2, 3],

'n_estimators': [100, 200, 300, 400]}

]



forest_reg = RandomForestRegressor()



forest_grid_search = GridSearchCV(forest_reg, forest_param_grid,

cv=5,

scoring="neg_mean_squared_error",

return_train_score=True,

verbose=True,

n_jobs=-1)



forest_grid_search.fit(cars_prepared, y_train)

We can get the best estimator directly:

forest_grid_search.best_estimator_

Gradient Boosting Regressor

gb_param_grid = [{'loss': ['ls'],

'learning_rate': [0.05, 0.1, 0.15],

'n_estimators': [200, 220, 240, 260],

'subsample': [0.8, 0.9],

'min_samples_split': [2, 3],

'min_samples_leaf': [2, 3, 4],

'max_depth': [4, 5, 6, 7],

'max_features': ['sqrt'],

}]



gb_reg = GradientBoostingRegressor()



gb_grid_search = GridSearchCV(gb_reg, gb_param_grid, cv=5,

scoring="neg_mean_squared_error",

return_train_score=True,

verbose=True,

n_jobs=-1)



gb_grid_search.fit(cars_prepared, y_train)

Let’s look at the best estimator:

gb_grid_search.best_estimator_

Ensemble: Random Forest Regressor & Gradient Boosting Regressor

Another way to fine-tune your system is to try and combine the models that perform best. The group (or “ensemble”) will often perform better than the best individual model. Now that we tuned our hyperparameters of our best two models let’s try and combine them to improve our best score even more.

This is relatively easy to do with Scikit-Learn’s VotingRegressor() class which allows us to take the average prediction of our Random Forest Regressor and Gradient Boosting Regressor.

from sklearn.ensemble import VotingRegressor



forest_reg =RandomForestRegressor(**forest_grid_search.best_params_)

gb_reg = GradientBoostingRegressor(**gb_grid_search.best_params_)



forest_gb_reg = VotingRegressor([("forest_reg", forest_reg),

("gb_reg", gb_reg)]) forest_gb_reg_scores = np.sqrt(-cross_val_score(forest_gb_reg,

cars_prepared, y_train, cv=10,

scoring="neg_mean_squared_error")) display_scores(forest_gb_reg_scores)

Look at that it worked! Our mean RMSE is under 1600! Now that we have our best model it is time to analyze its errors. But first let’s make one last box plot comparing the four best models.

Let’s save the best models so that we can come back easily to any of them!

import joblib



MODELS_PATH = os.path.join(PROJECT_ROOT_DIR, "models/")

os.makedirs(MODELS_PATH, exist_ok=True) models = [gb_grid_search.best_estimator_,

forest_grid_search.best_estimator_,

forest_gb_reg]

names = ["gradient_boosting_reg", "forest_reg","ensemble_(RF_&_GB)"] for model, name in zip(models, names):

joblib.dump(model, MODELS_PATH + name)

Analyze Errors of Best Model

We will use cross_val_predict to get a clean prediction on every instance in our training set (“clean” meaning that the prediction is made by a model that never saw the data during training).

from sklearn.model_selection import cross_val_predict



preds = cross_val_predict(forest_gb_reg,cars_prepared,y_train,cv=10)

Let’s plot our predicted price against the actual price. If all of our points were on the diagonal line that would mean our model is perfect and has an error of 0.

This looks pretty good our errors are normally distributed opposed to missing low for one region and high for another. All that’s left is to evaluate on the test set and get one final performance check. Thankfully the test set is easy to prepare using the pipeline we created earlier!

X_test = test_set.drop("Price", axis=1)

y_test = test_set["Price"].copy()

X_test = X_test[cols]

X_test_prepared = full_pipeline.fit_transform(X_test) from sklearn.metrics import mean_squared_error



final_predictions = forest_gb_reg.predict(X_test_prepared)

final_mse = mean_squared_error(y_test, final_predictions)

final_rmse = np.sqrt(final_mse)

final_rmse

Final RMSE: 2207.82

It seems we might’ve overfitted our training set a bit. It might be tempting to go back and try to perform better on the test set but then we risk overfitting the test set. We will leave it be as it is still in the range of scores we got on our training set.

Conclusion

Now back to the original problem. Should you buy or lease your next car?

Now that we have our final model and performance score on the test set we can use it to predict the prices of Acura TLXs with features corresponding to various leases. The leases we use for comparison are from Acura’s website. Let’s look at multiple different lease terms and see what we find out!

The cars we predict have the following features and leases: