Buying a car as a Data Scientist
By: Jonathan Cabreira
Tags: Artificial Intelligence, Application, Machine Learning, Data Science, Recommendation System

Buying a car as a Data Scientist

From a problem statement to a web application

The Application Diagram

After a year living on the Emerald Isle (Ireland), it is time to move back to Brazil — The COVID-19 pandemic turned the whole world upside down.

One interesting (or sad) fact about living in cities such as Rio de Janeiro, or any other Brazilian metropolis, is the fact that having a car is almost mandatory — as long as you can afford one. Based on that, I decided to approach this problem as a good Data Scientist. I built a car recommendation system that searches for cars on a webpage, processes the data and passes it through a machine learning model in order to list and display cars that are more related to what I am looking for — spoiler: The app is helping me a lot

You can check the application and all the code I develop on the webapp and on my Github pages.

The application Score Table

Getting the data

Before this project, I had no practice at all with web scraping. Though, with a little bit of understanding certain modules and how web pages work you can transform the web as an even more powerful source of data. In addition, I can’t forget to mention that I have taken the Mario Filho (one of the references in the field in Brazil ) data science course, which helped me a lot with its content.

I used BeautifulSoup in order to read the page HTML code and get the data from it. The process is “simple”, you open the page you want to get the data from, inspect the item you are interested and store the information you want. To accomplish that, you have to understand the structure of the page and think of an intelligent way of storing and structuring the data you are scraping.

Data Parsing

Web scraping is not an end in itself. In other words, we are always at the mercy of the website design and patterns and because of that, each page has its own way of been scrapped. Besides, we need to parse strings in order to get the information in the format we want.

I got the data in three steps. First, I scraped the search page of the website and stored its link, the car specification, and the carmaker. After that, I parsed each car page in order to get features of the cars. Finally, I used applied a few features engineering on the data so that I could be able to apply a machine learning algorithm.

Getting the Search Page

Parsed data frame

Feature Engineering

In order to avoid sparsity issues, I decided to apply one hot encoding on the dummies variables and select only the top components in terms of frequency. I considered categories with low frequency as noise and that way I was able to reduce a bit the dimensionality of the data without losing too much information. I also converted a collection of raw text to a matrix of TF-IDF features by applying theTfidfVectorizer from scikit learn.

Data labeling

For this project, I decided to label the training data manually. I literally opened the file on a spreadsheet and set 1 for a car I would like to buy or at least see the pictures and zero for cars I would not buy. I could have used active learning techniques for data labeling but I decided, for this project in specific, to do it myself in order to be more accurate as possible.

Model

I started off by building a baseline model with only two features ( price and model). Then I applied different models such as random forest and lightGBM. I also tested different variables for each model so that I could balance the complexity of the model in terms of the number of features.

After all the long process, I decided to create an ensemble model composed of the random forest and the lightGBM.

Data analysis of the model results

Besides the scores the model provides (I used average precision and Roc AUC score as metrics dut to the unbalance data labels), I also build a dashboard that shows information regarding cars with the highest scores.

Every time the database of the application is updated, the dashboard is also updated. So, let’s check a few charts and plots of the current date.

By looking at the charts above I can measure how well the model is going. Well, so far the result is amazing! I am looking for a car of two brands (Volkswagen and Ford) with prices within R$ 17–30 k. In addition to that, Ford Fiesta and Ford ka are the cars that I am most likely to buy. Besides, all of those models are cars that I am interested in. So, the app is working pretty well!

Will the app help me to buy a car?!

The answer to this question is simple: It will definitely help me! Actually it is already helping. I am updating the database on a daily basis in order to check for the best offers. As I am on the verge of arriving in Brazil, I will decide on each one I will buy when I get there.

Conclusion

I built a car recommendation system that works pretty well. All of the recommendations are in accordance with what I am looking for and I am proud of using my data science skills to accomplish this beautiful task.

If you want to know more about the project itself, please check the code on my Github. There is much about it that I didn’t mention in this post such as the use of the container, the Flask, and the Streamlit application.

And if you need any help building your recommendation system, let me know and I will assist you.