Mercari Price Suggestion Challenge — An End-to-End Machine Learning Case Study
By: Chintan Dave
Tags: Kaggle Competition, Feature Engineering, Machine Learning, Exploratory Data Analysis, Deep Learning

5. Feature Engineering

“Coming up with features is difficult, time-consuming, requires expert knowledge. ‘Applied machine learning’ is basically feature engineering.” — Prof. Andrew Ng

We will do the following feature engineering steps on the dataset.

Filling all missing data with some default values

Fill missing brand names

Pre-process text data

Split categories into three parts

Adding Item Description and Item Name word counts

Add cheap or expensive brands feature (binary data)

Vectorizing the text and categorical data

Filling all missing data with some default values

This can simply be done in pandas by using the fillna() function.

filling missing values with a default value

Fill missing brand names

During the EDA, we saw that the sellers are mentioning the brand name of the item in the Name section and Description section. So, I could extract the missing brand name from the either of them. However, I chose to use only the Item Name, because I found that in some descriptions the sellers are comparing the products with other brand names. So chances would be that I would extract the wrong brand name.

It is better to have no data rather than having wrong data.

We will extract the brand name as follows,

Get the list of existing brand names

Create a dictionary of brand_name -> category. This will be used to check whether the “guessed” brand name is of same category as the brand name that we are filling. This is for extra caution.

Pseudo code for filling the missing brands,

for each brand and category in dataframe:

if brand is missing:

for each brand in existing_brands:

if brand exists in item_name_text\

and category == brand_names_categories[brand]:

return brand

Using this I could fill 1,35,274 missing brand names. That is around 27% of originally missing brands.

Pre-process text data

The following steps were done for pre-processing the text.

De-contract the text. For example, don’t -> do not; I’m -> I am; I’ll -> I will

Remove punctuations and extra spaces

Remove stop words

Apply Stemming on the text

Split categories into three parts

It’s pretty easy to split the category into three parts using a single line of code,

Adding Item Name and Item Description word counts

The length of the text can be a deciding factor for the price of the product.

It is always a good idea to standardize the numeric data when we are using it for machine learning tasks. Hence, I also standardized them, so that they lie between -1 and +1.

Add cheap or expensive brands feature

This is totally a new feature that was added which will tell us whether the brand is an expensive brand or not. If you think logically then, expensive brands costs higher than cheaper ones, duh! This feature helped me a lot in predicting the prices. Before showing you the algorithm, let us understand how one aspect of the products helps us a lot in deciding its expensiveness.

Understand that, the expensiveness of a product is decided not just from the price of the product, but also from the category of the product. For example, a T-shirt for Men category of $200 price is very [very] expensive. However, a same priced laptop which is of Electronics category is considered a steal! So, you see, the category of a product shifts the price bracket of the product.

Now talking about our dataset, each category has so many items of various prices. There has to be a price threshold to determine that the product is expensive. What price should be considered for each category? For that, I chose to go with the 95th percentile median price for each category. The 95th percentile was chosen after a doing analysis on how the prices change after each percentile, and 95th percentile seemed to be a decent threshold.

First I created a dictionary with general categories as keys and their respective 95th percentile prices as threshold values.

Creating dictionary which has threshold price for each category

Now I had to create a list of expensive brand names. Keep in mind that this list will be derived from the training dataset. The algorithm to create such list is as follows,

Creating a list of expensive brands

Now we can use this list to create a new feature in the dataset, where we will check the existing brand name, and if that brand name exists in this expensive_brands list, then we will put a value of 1, otherwise 0. This list of expensive_brands is static in nature.

Vectorizing the text and categorical data

A machine does not understand ‘text’ as it is. We need to convert it to some numeric form, so that we can apply the mathematical formulations [models] on top of it. This is called ‘Vectorization’. There are numerous ways to vectorize text data. Here, I have used Tf-idf vectorizer with bi-grams and max_features=1,00,000. Other popular choice of vectorizing text is Countvectorizer. These are also called ‘Bag of Words’ vectorizations. How Tf-idf and Countvectorizer works is beyond the scope of this blog. However, I have attached links in the references section if you want to learn more.

6. Machine Learning Models

The train-cv split was 80:20.

Before discussing about the models and its scores, a quick reminder that, it is very (x10) important to do hyper-parameter tuning on any machine learning model. All the cv scores below are only after doing a thorough hyper-parameter tuning on each model.

Lasso Regressor: This is a Linear Regression model with L1 regularization. Using this model, it gave a score of 0.7016 on the cv set. This score does not look very good. The reason for this may be that, an L1 regularization is a very strong form of regularization by nature. It brings sparsity to the model. Meaning, it has the capability to reduce the importance/weight of a feature down to zero. So, it will totally discard few of the features.

Ridge Regression: This is also a Linear Regression model with L2 regularization. An L2 regularization will surely reduce the weights for less important features, but unlike L1, it will never reduce it to zero. This gave a cv score of 0.4385. This performs fairly good as compared to Lasso.

SGD Regression, is regression model which uses ‘stochastic gradient descent’ approach, unlike the Ridge and Lasso which uses ‘solvers’. This has shown to give better results as compared to the above two regressions. The score on cv was 0.4359.

LGBM, is a light version of Gradient Boosting Model. It is a tree based ensemble model. The cv score using this was 0.4599. This does not provide any better results as compared to the above discussed three models. The reason for that lies in the nature of this model i.e. ‘Tree based model’. Tree based model have shown to not work well on data with very high number of features. Here we have 2,00,000+ features, hence it was bound to not give us better results.

FM FTRL: This model makes use of Factorization Machines (FM) using Follow The Regularized Leader (FTRL) approach. To implement this model a library named WordBatch was used. Here I’ve used 1000 dimensions for the FM. This means, it will take 1000 combinations of the features from the original set of features and create the model using FTRL. If you want to learn how FM FTRL works, then the reference links have been attached in the end. This gave better results than all the above models with a cv score of 0.4265.

Stacking Regressor: This is another type of ensemble model, where we can have several base learners and one meta learner. So basically, the base estimators will be predicted individually, and on top of that, a meta estimator will take those predictions of each base estimator as features and make a prediction. Hence, called Stacking.

I used ridge, sgdr and lgbm as the base estimators and xgb regressor as the meta estimator. This has gave a better result as compared to the other three regression models (except FM_FTRL). I couldn’t include FM_FTRL in the stack, because the FM_FTRL implementation does not have certain elements/functions required for cloning the model’s object during stacking in sklearn. A cv score of 0.4283 was achieved.

Voting Regressor: I have used two types of Voting mechanism.

Sklearn’s voting regressor: Here I used ridge, sgdr, lgbm and the stacking regressor as the base estimators (There is no meta estimator in voting models). This model performs even better than the standalone stacking regressor, giving a cv score of 0.4270.

Manual voting: Here I had to weigh the predictions of more than 1 models manually (best model gets more weight). After trying several combinations, I chose FM_FTRL, stacking regressor and lgbm regressor as the three models and each of them was weighed according to its individual performance. The following weights were used for the three models,

fm_ftrl*0.60 + sstacking*0.18 + lgbm*0.22

Surprisingly LGBM model proved to be very useful here even with the least weight. None of the other three models could take its place in giving better final results, giving a cv score of 0.4210.

CV scores for each model

All these good looking scores, BUT…

Looking at these CV scores I tried to submit the manual voting score - the best model so far (or so it seems) on Kaggle. However, on the Kaggle’s stage 2 test dataset which is of 3.4 million listings, it gave me a final submission score of 0.62143, which was disappointing. After all, what matters is not the cv score, but the final submission score on Kaggle. That reflects the true potential of your model.

7. Deep Learning Models

Finally, I had to resort to the Deep Learning (DL) techniques. Following are the models that I experimented with,

GRU model using Glove word embeddings

1D CNN model using Glove word embeddings

GRU using Fast Text word embeddings

1D CNN model using Fast Text word embeddings

Combination of CNN and GRU in a single model

Simple Multi-Layered Perceptron (MLP) model

Ensemble of best models

Intro to Word Embeddings

We saw earlier that we can vectorize the text data using Bag of Words (BOW). If you see its working, then you will find that when vectorizing the text using BOW, we are not capturing the context of the words. We are just using straight forward methods to calculate and assign a number to each of the words individually. Best we can do is n-grams (like I used bi-grams which takes into consideration 2 words at a time), but vectorizing n-grams is too costly to train and store, because of large number of resultant features. Also if you vectorize the text using a large n [grams], then in-turn it would hurt the model’s performance, because there are too many blind spots in each resultant feature; also referred to as the ‘curse of dimensionality’. (The theory is again beyond the scope of this blog. To learn more, links are attached the end.)

So researchers found a much more efficient way to represent a word which also captures its context. They are called the word embeddings. There are several types of word embeddings like Word2Vec, Glove, Fast Text, etc. Each of the word embeddings are trained separately on humongous data sets (like all the Wikipedia text), using complex deep learning models, which resulted in a certain dimensional representation for each word (like 100D, 200D, 300D and so on). We can easily download the word embeddings file which is in the form of a dictionary, with words as keys and their n-dimensional vector as value.

GRU model using Glove word embeddings

Here, for the text representation I used Glove embedding of 200 dimensions, which will be passed through a GRU layer. For the categorical features, I used an Embedding layer of 32 dimensions. This was a simple model, with not much complexity in it. I just wanted to see how it works out. The full architecture is portrayed as follows,