Build, Develop and Deploy a Machine Learning Model to predict cars price using Gradient Boosting. | by Ayoub RMIDI | Towards Data Science

In this blog post I will cover a Data Science process to build and deploy a Machine Learning Model that can predict a car price, by following the steps below :
Setting the research goalRetrieving DataData Preprocessing & CleansingData Exploration & VisualizationData ModelingModel Deployment
Setting the research goal
Retrieving Data
Data Preprocessing & Cleansing
Data Exploration & Visualization
Data Modeling
Model Deployment
The aim of this work is to get familiarized with a Data Science process as described above by building & deploying a Machine Learning Model that can predict a car price based on its features, by trying 4 regression models and choosing the one with the the highest R2 score and the lowest Root Mean Squared Error.
Retrieving Data is the important step that comes after setting the research goal, for this purpose I used the famous python library specialized in these kinds of tasks which is “BeautifulSoup”. The process for this task is quite simple, first we loop over the ads pages in order to collect the ads URLs by incrementing the page number parameter called oin the base URL which look like this :
basic_url = "https://www.avito.ma/fr/maroc/voitures-à_vendre&o="
once the URLs is collected I set them in a csv file called “ads_urls.csv”, which look like this :
https://www.avito.ma/fr/temara/voitures/peugeot_508_26753561.htmhttps://www.avito.ma/fr/safi/voitures/renualt_kasket_26753552.htmhttps://www.avito.ma/fr/oued_fes/voitures/Citroen_C3_Diesel_26753551.htm....
Each link of these links above, contain data about the car posted in that ad, such as : ‘Model Year’, ‘Mileage’, ‘Fuel Type’, ‘Price’ ..., which seems exactly like this :
then, I read this file containing list of URLs to crawl on ad pages and extract the necessary data, so here is a brief code Snapshot about how it works.
Finally I get the following data schema :
price, year_model, mileage, fuel_type, mark, model, fiscal_power, sector, type, city135 000 DH,Année-Modèle:2013,Kilométrage:160 000–169 999,Type de carburant:Diesel,Marque:Peugeot,Modèle:508,Puissance fiscale:-,-,”Type:Voitures, Offre”, Temara60 000 DH,Année-Modèle:2009,Kilométrage:130 000 - 139 999,Type de carburant:Essence,Marque:Ford,Modèle:Fiesta,Puissance fiscale:7 CV,Secteur:saies,"Type:Voitures, Offre", Fès....
The next step is the most time consuming in every Data Science process which is Data Preprocessing & Cleansing.
The reason behind the Data Preprocessing is to transform our RAW data to a useful one and also to reduce the data size so that it becomes easy to analyse, in this case we need to :
price : remove the “DH” (Dirham currency) character and spaces from the price then transform it to integer.
year_model : remove unwanted string from the year_model column such as : “Année-Modèle:” or “ou plus ancien” then transform it to integer.
mileage : for this field the thing is quite different, we have a range of mileage, for example in each observation we have mileage between 160000 - 169999 KM, so after removing the “KM” and “-” characters I had to render this column with a way to keep only one value and not a range, we may use the min or the max value, to me I chose to use the mean so that we stay in the middle of that range.
fuel_type : remove unwanted string from the fuel_type column like “Type de carburant:”.
mark : remove the “Marque:” string from the mark column.
fiscal_power : for this field I had to handle 2 cases : remove the unwanted string such as “Puissance fiscale: CV” and deal with missimg values, so one of the tips to handle missing values is to fill them with the mean of all fiscal_power columns.
For the Data Exploration and analysis we do not need some columns such as “sector” and “type”, so we end up by dropping them from the data frame.
After the Data Extraction and Data Preprocessing steps, I should now visualize my data set so that I can have more insights about what is happening under the hood, and how my data is distributed.
In the following section, I will run a Q&A session in order to answer many questions based on histograms & plot.
Q : How price is distributed over the year model ?
A : As we can see from the plot above, the cars price increase respectively by years, and more explicitly we can say that the more the car is recently released, the more the price augments, while in the other side the oldest cars still have a low price, and this is totally logical since whenever the cars become kind of old from the date of release, the price starts decreasing.
Q : How is the price distributed over fiscal power ?
A : From the plot above we can notice clearly that there is a huge concentration of points in the range of [2800 DH,800000 DH], and [3 CV, 13 CV], which could be interpreted as first the huge domination price of medium fiscal power cars in the market with correct price and secondly the more the fiscal power increase the price do too.
A : Although weak, it appears that there is a positive relationship between the year_modeland price let’s see what is the actual correlation between price and the other data points. We will confirm it by looking at the Heatmap correlation matrix.
As we can see there is a strong correlation between the price and year_model features with a 0.47 as a correlation score, which validates the strong relationship between the price and year_model columns.
Now we came to the main task in all this process, which is Data Modeling, for this purpose I will use 4 Machine Learning models dedicated for Regression problems, at the end I will do a Benchmarking table to compare each model r2_score and select the best one. The used models are : K Nearest Neighbors regression, Multiple Linear Regression, Decision Tree Regression and Gradient Boosting Regression.
Data Transformation
I Intentionally let this part until the Data Modeling instead of doing it with Data Preprocessing for some visualization purposes.
At the moment I still have 2 categorical features which are the fuel_type and mark , the aim of this section is to preprocess those features in order to make them numerical so that they will fit into our model. In literature there are two famous ways of categorical variable transformations, the first one is label encoding, and the second one is the one hot encoding, for this use case we will use the one hot position and the reason why I choose this kind of data labeling is because I will not need any kind of data normalization later, and also this has the benefit of not weighting a value improperly but does have the downside of adding more columns to the data set.
As shown in the figure above, after transforming the categorical features to numerical ones by using the one hot encoding, we got a wide data frame.
Data Splitting
Usually we split our data into three parts : Training , validation and Testing set, but for simplicity we will use only train and test with 20% in test size, and the rest for training.
Gradient Boosting Regression
Boosting is another ensemble technique for creating collection of powerful predictors, and Gradient Boosting is a technique for producing regression models consisting of collections of regressors.
An ensemble is a collection of predictors whose predictions are combined usually by some sort of weighted average or vote in order to provide an overall prediction that takes its guidance from the collection itself. So boosting is an ensemble technique in which learners are learned sequentially with early learners fitting simple models to the data and then analyzing the data for errors, those errors identify problems or particular instances of the data that are difficult or hard to fit, as a consequence later models focus primarily on those examples trying to get them right.
At the end, all the models contribute with weights and the set is combined into some overall predictors, so boosting is a method of converting a sequence of weak learners into a very complex predictor, it’s a way of increasing the complexity of a particular model initial learners tend to be very simple and then the weighted combination can grow more and more complex as learners are added.
The Math behind Gradient Boosting
This algorithm is an instance of gradient boosting, it’s called gradient boosting because it’s related to a gradient descent sort of procedure.
First we make a set of predictions ŷ(i) for each data point.
We can calculate the error in our predictions, let’s call it J(y, ŷ) , and J just relates the accuracy of ŷ in modelling y.
For mean squared error MSE : J(y, ŷ) = Σ ( y(i) -ŷ(i) )2 .
So now we can try to think about adjusting our prediction ŷ to try to reduce the error above : ŷ(i)= ŷ(i) + α *∇J(y, ŷ) , with : ∇J(y, ŷ) = y(i)- ŷ(i)
Each learner is estimating the gradient of the loss function.
Gradient Descent : take sequence of steps to reduce J .
Sum of predictors, weighted by step size alpha.
Gradient Boosting Regressor — Code Snapshot
Interpreting Residual VS Predicted values
After building the GBR model, it is mostly recommended to plot the error distribution to verify the following assumptions :
Normally distributed
Homoscedastic (The same variance at every X)
Independent
which is already verified in our case as you can check on the notebook for more details, however we need to observe the residual plot to make sure it doesn’t follow a non-linear or Heteroscedasticity distribution.
As we can see from the plot above, the residuals roughly form a “horizontal band” around the 0 line, this suggest that the variances of the error terms are equal, furthermore no one residual “stands out” from the basic random pattern of residuals which involve that there is no outliers.
Models Benchmarking
So after trying many regression models to fit our data set, it’s time to draw a Benchmarking table that will summarize all the results we have got.
╔═══════════════════╦════════════════════╦═══════════════╗║   Model           ║     R^2 score      ║     RMSE      ║╠═══════════════════╬════════════════════╬═══════════════╣║ KNN               ║ 0.56               ║ 37709.67      ║║ Linear Regression ║ 0.62               ║ 34865.07      ║║ Gradient Boosting ║ 0.80               ║ 25176.16      ║║ Decision Tree     ║ 0.63               ║ 34551.17      ║╚═══════════════════╩════════════════════╩═══════════════╝
It appears that the Gradient Boosting model won the battle as it was expected with the lowest RMSE value and the highest R2 score.
Creating the Flask web app
Creating the Flask web app
Flask is a “micro” framework for Python. It is called a micro framework because they want to keep the core simple but expandable. While confusing at first, it is relatively easy to set up a website on Flask using Jinja2 templating.
The flask app consists of 2 main components: the python app (app.py) and the HTML templates, for the app.py here is how it looks like :
So once the app.py is built, we can run it from the Terminal, if everything goes right, we will get the index.html page running on the '/’ route as :http://localhost:8080/ , then we can fill the given form with the right values and get the result as a sweet alert pop-up.
2. Deploy the app to Heroku
For this part, I will need a Heroku account (Free) and the HerokuCLI, I can also use GitHub for this task, but let’s keep things simple.
Set the Procfile: A Procfile is a mechanism for declaring what commands are run by your application’s dynos on the Heroku platform. Create a file called “Procfile” and put the following in it:
web: gunicorn app:app --log-file -
Create the python requirements file by running the following command in your Terminal at the root of your Flask app :
$ pipreqs ./
if you’re working in a python virtual environment, this would be efficient :
$ pip3 freeze > requirements.txt
Create a new app on the Heroku Website by logging into your account.
Once the app is well created we will be redirected to a page showing us the instructions to do for a successful deployment , after being successfully connected to the HerokuCLI, I need to change the directory to our flask App and run the following command :
$ heroku git:clone -a cars-price-prediction$ git add .$ git commit -am "make it better"$ git push heroku master
The app should be now live at my-app-name.herokuapp.com! Check out a working version of the app here.
In this blog post I tried to cover a data science process by building a model that can predict the price of cars based on its features, starting from the Data Collecting to the Data Modeling and comparison between built-in Models, and finishing by Model Deployment as a web app.
Full notebook : https://github.com/PaacMaan/cars-price-predictor/blob/master/cars_price_predictor.ipynb
GitHub repository :
github.com
Web Application link : https://cars-price-prediction.herokuapp.com
Shared with ❤.