Predicting the price of used cars | by Srivathsan G | Towards Data Science

This article aims at analyzing the attributes of Used Cars dataset to perform feature selection and feature engineering before predicting the price of those cars using an XGBoost model. The whole process of data processing, cleaning, feature selection, and modeling is shown in the below sections. The terms “attributes”, “variables” and, “features” are used interchangeably throughout this article and have the same meaning, that is they refer to the columns of the dataset.
Let’s start by understanding what feature selection and feature engineering means.
The process of selecting those input features which are most relevant to the prediction of the output variable is called feature selection. This helps reduce the dimensionality of the data and also makes sure that the model does not learn on those irrelevant features which might decrease its accuracy.
For more information on feature selection click here.
The process of obtaining new features for the given dataset, with or without the use of existing features, which boosts the performance of the model on that dataset is called feature engineering.
For more information on feature engineering click here.
Now let’s dive into the implementation part.
We import the following libraries that are necessary for the prediction process.
import pandas as pdimport matplotlib.pyplot as pltimport numpy as npfrom sklearn.preprocessing import StandardScaler, LabelEncoderfrom sklearn.model_selection import train_test_splitimport mathimport matplotlibimport seaborn as sns%matplotlib inline
The dataset used here is the Used Cars dataset which is loaded. It has 370000 rows of data scraped from eBay and 28 attributes describing each pre-owned car’s details. But a subset of the total dataset is used which has 50000 rows. The content of this data was in German which is translated to English. This translated subset of the original dataset can be found here. A description of the dataset’s attributes is displayed below:
df = pd.read_csv(‘../input/data.csv’, sep=’,’, header=0, encoding=’cp1252')
df.describe()
From the data description, we can see that some numerical variables have values that cannot exist. For example, the yearOfRegistration variable has a min value of 1000 and a max value of 9999 where both values are not possible. Such values will be handled in the upcoming blocks of code.
We then remove the attributes in the dataset which have no meaning for predicting the cost of the automobiles.
df.drop([‘name’,’seller’, ‘offerType’, ‘dateCrawled’, ‘lastSeen’], axis=’columns’, inplace=True)
The rows with attribute values that are not possible(impermissible) are then removed.
df=df[(df.price>=100) & (df.price<=200000) & (df.yearOfRegistration>=1950) & (df.yearOfRegistration<=2019) & (df.monthOfRegistration>0) & (df.monthOfRegistration<=12) & (df.powerPS>10) & (df.powerPS<1000)]df.describe()
We can see from the above data description that the attributes now have only possible values.
We first need to find out which attributes have NA values, i.e fields that are empty.
df.isna().sum()
So the NA values only exist in categorical variables. To fill each of those empty fields(imputation), the mode value of that attribute is used.
var=[‘vehicleType’,’gearbox’,’model’,’fuelType’,’notRepairedDamage’]for i in var:    df[i].fillna(df[i].mode()[0],inplace=True)df.isna().sum()
The features yearOfRegistration, monthOfRegistration, and postalCode are plotted against the price variable.
for i in [‘yearOfRegistration’, ‘monthOfRegistration’,’postalCode’]:    sns.jointplot(x=i, y=’price’,data=df[[i,’price’]],size=7)
The plots along with the Pearson correlation and p-value displayed for each pair of variables can be used as a feature selection metric. The second plot shows that monthOfRegistration has a very low correlation value and a high p-value with the price. So this variable is dropped.
df.drop([‘monthOfRegistration’],axis=’columns’,inplace=True)
This dataset has an attribute dateCreated, which is the date in which the car’s advertisement was created on eBay. By using this attribute and the yearOfRegistration attribute we can create a new feature that tells us the age of the car(ageOfCar).
df[‘ageOfCar’]=pd.DatetimeIndex(df[‘dateCreated’]).year-df[‘yearOfRegistration’]sns.jointplot(x=’ageOfCar’,y=’price’,data=df[[‘ageOfCar’,’price’]],size=7)
We can see that the magnitude of the Pearson correlation from the above plot is the same as that for yearOfRegistration and price, but is negative. This means that with increasing ageOfCar the price decreases which is true in real life too!
Now we drop the variables yearOfRegistration and dateCreated.
df.drop([‘yearOfRegistration’,’dateCreated’],axis=’columns’,inplace=True)
In statistics, an outlier is a data point that differs significantly from other observations. An outlier may be due to variability in the measurement or it may indicate an experimental error; the latter are sometimes excluded from the data set. An outlier can cause serious problems in statistical analyses. (Source: Wikipedia)
So basically outlier is a data point that does not fit the general data distribution. Outliers present in a data can skew the model and disrupt it’s learning process. Hence, now we will detect these outliers in our dataset and remove them using the Boxplot (Inter-Quartile Range) method. To know about the different outlier detection methods click here.
sns.boxplot(x=df[‘price’])
So the outliers for price variable lie after value 25000.
sns.boxplot(x=df[‘ageOfCar’])
sns.boxplot(x=df['powerPS'])
sns.boxplot(x=df['kilometer'])
And similarly, for features ageOfCar, powerPS, kilometer values above 30, above 280, and below 25000 are outliers respectively. These outliers are now removed.
df=df[(df.price<=25000) & (df.ageOfCar>=0) & (df.ageOfCar<=30) & (df.powerPS<=280) & (df.kilometer>=25000)]
Now we need to encode the categorical variables of the dataset so that it can be used by the model for training. There are several encoding techniques for categorical variables. In this, we will be using Label Encoding for variables with 2 classes and Target Encoding for variables with more than 2 classes. Target Encoding is preferred over One-hot Encoding because, the latter results in very high dimensions of data.
x=df.drop([‘price’],axis=’columns’,inplace=False)y=df[‘price’]le1=LabelEncoder().fit(x['gearbox'])x['gearbox'] =le1.transform(x['gearbox'])le2=LabelEncoder().fit(x['notRepairedDamage'])x['notRepairedDamage'] =le2.transform(x['notRepairedDamage'])le3=LabelEncoder().fit(x['abtest'])x['abtest']=le3.transform(x['abtest'])from category_encoders import TargetEncoderte=TargetEncoder(cols=['brand','model','vehicleType','fuelType','postalCode']).fit(x,y)x=te.transform(x)
By using a correlation heat map we can understand the relationship between the variables. Two main observations are made from a heat map(excluding the diagonal elements):
Input variables that have a high correlation with other input variables.
Input variables that have a low correlation with the output variable.
Now let’s generate the heat map.
f, ax = plt.subplots(figsize=(11, 9))cmap = sns.diverging_palette(220, 10, as_cmap=True)x[‘price’]=ysns.heatmap(x.corr(),cmap=cmap,square=True,annot=True)
From the heat map, we can see that there is not much correlation between the input variables but, abtest is poorly correlated with price. So abtest is removed.
x.drop([‘abtest’],axis=’columns’,inplace=True)y=x['price']x.drop(['price'],axis='columns',inplace=True)
We now standardize the variables and split it into train (80%) and validation (20%) sets.
sc=StandardScaler()x=sc.fit_transform(x)X_train, X_val, y_train, y_val = train_test_split(x, y, test_size=0.2, random_state = 42)
We use an XGBoost model to train on the data and predict the output. We also use the GridSearchCV from sklearn module to identify the best parameters for the model. The parameters tuned using GridSearchCV are:
n_estimators: This defines the number of estimators(trees) used by the XGBoost model.
max_depth: This defines the maximum depth allowed by each tree and is used to control overfitting.
eta: This is the learning rate of the model which shrinks the weights of the nodes to make it more robust.
To get an unbiased estimate of the model performance we use the K-fold cross-validation method with cv=5 . The best parameters are identified by using the RMSE as the error metric. Since we need to find the model which produces the least RMSE, we pass the negative of the RMSE to the scoring argument.
import xgboost as xgbfrom sklearn.model_selection import GridSearchCVparam={‘n_estimators’:[10,20,50,100,150,200], ‘max_depth’:range(3,11), ‘eta’:[0.05,0.1,0.15,0.2,0.25,0.3]}xgr=xgb.XGBRegressor()gs=GridSearchCV(estimator=xgr,param_grid=param,scoring=’neg_root_mean_squared_error’,cv=5,verbose=3)gs.fit(X_train,y_train)
After the parameter search is complete we can now view the best parameters identified and its corresponding metric score.
print(gs.best_params_)print(gs.best_score_)
eta=0.05 , n_estimators=200 , max_depth=7 are determined as the parameters which produce the lowest RMSE score. Using these parameters identified by GridSearchCV we build the final model and use it for prediction. For regression problems, such as this one, the error metrics such as RMSE, MSE, MAE, R2, etc. are used. Here we employ RMSE and R2(coefficient of determination) as the error metrics. Thescore() function of XGBoost returns the R2 value for a regression problem.
from sklearn.metrics import mean_squared_errorxgr=xgb.XGBRegressor(eta=0.05,max_depth=7,n_estimators=200)xgr.fit(X_train,y_train)pred=xgr.predict(X_val)print('RMSE: ',mean_squared_error(y_val,pred,squared=False))print('R2Score: ',xgr.score(X_val,y_val))
The RMSE and R2 values show that the results from the model are pretty accurate. Now to evaluate the model’s performance and to also understand how well the distribution of the data is captured we generate plots from the residuals.
In regression analysis, the difference between the observed value of the dependent variable (y) and the predicted value (ŷ) is called the residual (e). Each data point has one residual. (Source: Stattrek)
We use the standardized residuals (residuals divided by their standard deviation) to create scatter and histogram plots.
def residual_plot(y_test, y_pred):    res = y_test — y_pred    fig, axs = plt.subplots(1,2,figsize=(30,10))    std_res = res/np.std(res)    axs[0].title.set_text(‘Scatter Plot of residuals’)    axs[0].set_xlabel(‘True Output’)    axs[0].set_ylabel(‘Residuals’)    axs[0].scatter(y_test,std_res)    axs[1].title.set_text(‘Histogram of residuals’)    sns.distplot(std_res, ax = axs[1]);    plt.show()residual_plot(y_val,pred)
The scatter plot shows that there are some outliers and that the residuals do not have a constant variance. Hence they do not follow a normal distribution. The histogram shows that most of the residuals are concentrated near zero.
To understand the significance of each feature in the output prediction process and derive future inferences we plot the feature importance values directly obtained from the XGBoost model.
feature_important = xgr.feature_importances_values = list(feature_important)data = pd.DataFrame(data=values, index=feat, columns=[“score”]).sort_values(by = “score”)ax=data.plot(kind=’barh’,title=’Feature Importance’,legend=None)ax.set_xlabel(‘Importance value’)ax.set_ylabel(‘Features’)plt.show()
notRepairedDamage, fuelType, and gearBox do not contribute much in predicting the price.
In summary, we have utilized the Used Cars dataset to understand the concepts of data cleaning, imputation, feature selection, and feature engineering. We have also learned how to use the XGBoost model and perform regression analysis on it by plotting the residuals and the feature importance values.
To get the complete code as an IPython Notebook click here.