A Data Science Case Study with Python: Mercari Price Prediction | by Geoffrey Lobo | Towards Data Science

The most effective way to learn data science is by solving data science related problems. Reading, listening and taking notes is valuable, but once you work through a problem, concepts solidify from abstractions into tools you feel confident using. Generally a full cycle data science project includes the following stages:
Data Gathering & WranglingData Analysis & ModelingCommunication & Deployment
Data Gathering & Wrangling
Data Analysis & Modeling
Communication & Deployment
In this case study, we will walk through the Analysis, Modelling and Communication part of the workflow. The general steps involved for solving a data science problem are as follows:
Understand the problem and dataData exploration / data cleaningFeature engineering / feature selectionEstablishing a performance metricModel evaluation and selectionModel optimizationDraw conclusions and document work
Understand the problem and data
Data exploration / data cleaning
Feature engineering / feature selection
Establishing a performance metric
Model evaluation and selection
Model optimization
Draw conclusions and document work
Those of you who are not familiar with the field of Data Science and Python programming language can still follow through the article as it will give an high level overview about how these kind of problems are approached and solved.
While some code snippets are included within the blog, for the full code you can check out this Jupyter Notebook.
Mercari is Japan’s biggest community powered shopping app where people can sell and buy a variety of brand new and used products of different brands, from sweaters to smartphones. Now Mercari would like to suggest the correct prices to the sellers but this is tough because their sellers are enabled to put just about anything, or any bundle of things on Mercari’s marketplace.
So our objective is to build a model that automatically suggests the right product prices to the sellers. We are provided with the following information for each product:
train_id — the id of the product
name — the title of the product
item_condition_id — the condition of the product provided by the sellers
category_name — category of the product
brand_name — the product’s brand name
shipping — 1 if shipping fee is paid by seller and 0 if shipping fee is paid by buyer
item_description — the full description of the product
price — the price that the product was sold for (This is the target variable that we will predict)
This type of problem lies under the category of Supervised Regression Machine Learning:
Supervised: We have access to both the features and the target and our goal is to train a model that can learn a mapping between the two.
Regression: The target variable, price, is a continuous variable.
EDA is an open-ended process where we calculate statistics and make figures to find trends, anomalies, patterns, or relationships within the data. In short, the goal of EDA is to learn what our data can tell us. It generally starts out with a high level overview, then narrows in to specific areas as we find interesting parts of the data. The findings may be interesting in their own right, or they can be used to inform our modeling choices, such as by helping us decide which features to use. In a hurry to get to the machine learning stage, some data scientists either entirely skip the exploratory process or do a very perfunctory job but in reality EDA is one of the most crucial steps in solving a data science related problem.
The data set can be downloaded from Kaggle. To validate the result, we only need the ‘train.tsv’ data file. So let’s get started!
import numpy as npimport pandas as pdimport matplotlib.pyplot as pltimport seaborn as snsfrom sklearn.model_selection import train_test_split
We will split the data into train and test sets in the proportion of 80% and 20% respectively. As a common practice, we will conduct EDA on the train data set only.
In:data = pd.read_csv('train.tsv', sep='\t')train, test = train_test_split(data, test_size=0.2)print(train.shape, test.shape)Out:(1186028, 8) (296507, 8)
There are around 1.18 million data points/rows in the train data and 0.296 million data points/rows in the test data. 8 is the number of columns/features in both the sets. Features and variables mean the same thing here, so they might be used interchangeably within the blog.
#this command displays first few rows of the data settrain.head()
A common problem when dealing with real-world data is missing values. These can arise for many reasons and have to be either filled in or removed before we train a machine learning model. First, let’s get a sense of how many missing values are in each column.
The ‘brand_name’ feature has 42% missing values. Some of the approaches that are usually considered while dealing with missing values are:
Remove the records with the missing values.Remove the feature itself if the number of missing values is higher than some threshold, say 50%.Consider ‘missing values’ as another category of that respective feature itself.
Remove the records with the missing values.
Remove the feature itself if the number of missing values is higher than some threshold, say 50%.
Consider ‘missing values’ as another category of that respective feature itself.
In this project, we will go forward with the third approach.
Now we will start analyzing the features one by one. We will first go through the target variable, Price, and then start analyzing the predictor variables individually and also see how it interacts with the Price variable.
Let’s check the distribution of the Price variable and go through some basic statistics.
train['price'].plot.hist(bins=50, figsize=(10,5), edgecolor='white',range=[0,500])plt.xlabel('Price', fontsize=15)plt.ylabel('Frequency', fontsize=15)plt.tick_params(labelsize=15)plt.title('Price Distribution - Training Set', fontsize=17)plt.show()
train['price'].describe()
The distribution of the Price variable aligns with the above statistics, i.e. the distribution is right skewed.
The median price of the items is $17.
75% of the items have price below $29.
The maximum price of an item from the data is $2009.
In:train['shipping'].value_counts(normalize=True)*100Out:0    55.2676671    44.732333 Name: shipping, dtype: float64
The shipping fee for 55.26% of the items are paid by the buyers. Let’s see how the shipping feature relates with price.
The median price of the products is $14.0 if seller pays for the shipping while the median price of the products is $20.0 if buyer pays for the shipping.
Normally when we buy products online, we need to pay for shipping or delivery of products which are below a certain price. But here the trend is kind of opposite since the median price of items for which the seller pays the shipping fees is lower than the median price of the items for which the buyer pays the shipping fees.
The ‘item_condition_id’ has five unique values ranging from 1 to 5. Lower the number, better the condition of the item.
In:train['item_condition_id'].value_counts(normalize=True, sort=False)*100Out:1    43.194511 2    25.323264 3    29.166934 4     2.153153 5     0.162138 Name: item_condition_id, dtype: float64
43% of the items have item condition ID as 1 while only 0.16% of the items have item condition ID has 5. Let’s see how this feature relates with Price. For comparing and visualizing the relation between a categorical variable and a numerical variable, Box-Plots are very helpful. The code below is used for plotting Box-Plots.
#for easier visualization, we are considering the prices from range of 0-100price_100 = train[train['price']<100]fig, ax = plt.subplots(figsize=(20,7.5))sns.boxplot(x='item_condition_id', y='price', data=price_100, ax=ax)plt.xlabel('Item Condition', fontsize=15)plt.ylabel('Price', fontsize=15)plt.xticks(fontsize=15)plt.yticks(fontsize=15)plt.show()
Though the proportion is low, items with item condition ID as 5 have higher median price when compared with items having better condition.
After going through some of the data in which the item condition ID is given as 5, most of the products with this condition, especially electronic products are being sold for their parts which itself can prove to be valuable. This pretty much explains the higher median price.
There are 1268 unique categories in the ‘category_name’
(train['category_name'].value_counts(normalize=True)*100).head(6)Out:Women/Athletic Apparel/Pants, Tights, Leggings              4.060365Women/Tops & Blouses/T-Shirts                               3.134841 Beauty/Makeup/Face                                          2.321684 Beauty/Makeup/Lips                                          2.025398 Electronics/Video Games & Consoles/Games                    1.798039 Beauty/Makeup/Eyes                                          1.720390 Name: category_name, dtype: float64
As we can see, for each item, there are three sets of categories separated by ‘/’. The categories are arranged from top to bottom with respect to the comprehensiveness. Therefore we can split the categories into three different columns. The three categories will signify main category, first subcategory and second subcategory. While splitting, the missing values will be filled with the string ‘Category Unknown’.
As a common preprocessing practice, we will convert all the textual categorical values to lower case. The reason for doing this will be clear with the following example:
Suppose there are two rows in the data and the product belonging to both the rows is an iPhone and the brand name corresponding to the first entry is ‘APPLE’ and for the second entry it is given as ‘Apple’. So when we featurise the data for applying the machine learning models, any featurisation technique that we will apply will consider ‘APPLE’ and ‘Apple’ has two different values even though they mean the same thing. So, for our model to consider these two values as the same, we will convert all the textual categorical values to lower case. Effectively, ‘APPLE’ and ‘Apple’ will become ‘apple’ and ‘apple’.
After splitting the categories, we have 11 unique main categories, 114 unique first sub-categories, 863 unique second sub-categories.
Let’s see which one of the main categories have the highest number of products in the data and also see how the main categories stack up according to the median price of their respective products.
The maximum number of products, i.e. 44.8% of the total products belong to ‘Women’ category followed by ‘Beauty’ category products which takes up around 14% of the total products while 1.7% of the products, being the minimum, belong to the ‘Sports and Outdoor’ category.
The highest median price of $21 belongs to the items from ‘Men’ category followed by ‘Women’ category having a median price of $19 while the items from ‘Handmade’ category have the lowest median price of $12.
Since there are 114 unique subcategories, it will be difficult to visualize all of them. So, we will take a look at top 10 most popular subcategories and top & bottom 10 subcategories sorted according to the median prices of their respective items.
The most popular subcategory is ‘Athletic Apparel’ which aligns with the previous observation that the most popular main category is ‘Women’ and ‘Athletic Apparel’ comes under both the ‘Women’ and ‘Men’ categories. 9% of the total products comes under the Athletic Apparel category.
From the perspective of the median price of the items, the items from the subcategory of ‘Computers & Tablets’ have the highest median price, with the median price being $40.
The items from the subcategory ‘Paper Goods’ have the lowest median price of $6. ‘Paper Goods’ come under the ‘Handmade’ category. This also validates the previous observation that the items from ‘Handmade’ category have the lowest median price out of all the main categories.
There are 4535 unique brands in the data. Let’s take a look at the top 10 most popular brands.
Nike and Pink are the top two most popular brands and 12.6% of the total products in the data belong to these two brands.
When we had checked for missing values, ‘brand_name’ feature had 42.7% missing values. So to deal with it, we will impute the missing values with the string ‘brand unavailable’.
train[’brand_name’] = train[’brand_name’].fillna(’brand_unavailable’)
Since there were 42% missing values, it will be interesting to see whether the price is impacted when the brand name is given or not for that particular product. For that, we will create a new feature named ‘brand_name_given’ with the values ‘Yes’ and ‘No’ denoting whether the brand name is given or not. Let’s find out!
After looking at the box plots, although there is good amount of overlap, we can say that there is a considerable difference in the prices when the brand is given and when it’s not given. The median price of the product when the brand name is given is $20 and when the brand name is not given, the median price is $14. In a nutshell, this feature will be helpful for the ML model to map some kind of pattern from it.
The ‘item_description’ feature falls under the category of unstructured text data. For this type of data, the preprocessing steps will include the following operations:
Convert all the words to lowercase.Remove common words like ‘or’, ‘and’, ‘is’, ‘the’, ‘was’ etc. We remove them because these kinds of words will be present within the data in high proportion and the ML models won’t be able to generate any meaningful patterns from it. These set of words are known as ‘Stopwords’.Remove special characters like ‘&’, ‘@’, ‘!’, ‘?’ etc.
Convert all the words to lowercase.
Remove common words like ‘or’, ‘and’, ‘is’, ‘the’, ‘was’ etc. We remove them because these kinds of words will be present within the data in high proportion and the ML models won’t be able to generate any meaningful patterns from it. These set of words are known as ‘Stopwords’.
Remove special characters like ‘&’, ‘@’, ‘!’, ‘?’ etc.
For visualization, one of the most effective ways to visualize text data is by plotting ‘Word Clouds’. We will see in the following word clouds about how do the words in the item description compare when the price increases. To check that, we will sort the data according to the prices from low to high and then divide the data into four equivalent parts. By doing this the first quarter will have products with the lowest prices and consequently the fourth quarter will have products with the highest prices.
All the four word clouds are almost identical to each other. Words like ‘brand new’, ‘never used’, ‘good condition’, ‘size’, ‘medium’ etc. are used in a high frequency for most of the products irrespective of the price. This shows that the sellers most of the time try to put in a good word for their product in the product description section, so that they don’t have much trouble in selling off the product quickly.
One interesting thing in the first word cloud are the words ‘description yet’. After a quick glance through the data, there are entries in which the description is not given but just the words ‘No description yet’ and there are 5.57% rows with such words. Just like we did with the ‘brand_name’ feature, we will verify whether the price of the product is impacted when the description is given or not.
There is not much difference to be seen here unlike the ‘brand_name’ feature. The median price of the product when the description is given and when it is not given are almost the same. The 75th percentile value of the product price when the description is given is 5$ more than the 75th percentile value of the product price when the description is not given.
The features that we will select for modelling are ‘brand_name’, ‘brand_name_given’, ‘item_description_id’, ‘shipping’, ‘main_category’, ‘subcategory_1’, ‘subcategory_2’ and ‘item_description’.
A machine learning model can only learn from the data we provide it, so ensuring that data includes all the relevant information for our task is crucial. All the above features that we just analysed will be used for making the model. But a ML model cannot differentiate between 'Nike' and 'Samsung' brand or between ‘Makeup’ and 'Toys' category or with any categorical feature for that matter. So to solve this problem, we need to convert our categorical values to numerical values. We can do this by 'One-Hot Encoding' our categorical values. The concept of ‘One-Hot Encoding' will be clear with the following examples.
Suppose we have a feature called 'Color' and this feature has three unique values: Red, Blue and Green. By applying 'One-Hot Encoding' to this Color feature, three columns will be generated with each column representing the three colors with a binary value of 1 or 0 denoting whether that color is present in that particular row/observation or not.
We can implement ‘One-Hot Encoding’ to the categorical data with the help of Scikit-Learn library in Python. Also, when we apply OHE, we need to apply it to the test data with respect to the train data to avoid data leakage. For example, if there is another color in the test data, ‘Yellow’, which is not present in the train data, then to avoid data leakage, we will ignore ‘Yellow’ while applying OHE to the test data since the test data should always be treated as ‘unseen’.
Now coming onto representing the ‘item_description’ feature into numerical values, we will apply the technique called ‘Bag of Words’. This technique is same as ‘One-Hot Encoding’
In this technique we will build a vocabulary of all the unique words in our dataset, and associate a unique column to each word in the vocabulary. Each sentence is then represented as a list that is as long as the number of distinct words in our vocabulary. At each column in this list, we mark how many times the given word appears in our sentence. This is called a ‘Bag of Words’ model, since it is a representation that completely ignores the order of words in our sentence. The explanation will be clear with the image given below.
Generally, the Root Mean Squared Error (RMSE) metric is used for regression tasks. But as price followed a long-tailed distribution (50% of the products were under $17), in order to make errors on lower price products more relevant than for higher prices, the appropriate metric for this problem would be Root Mean Squared Logarithmic Error (RMSLE). Thus, we will apply the log transformation to the price target variable, to make this assumption available for model training. The formula for RMSLE and what each variable denotes is given in the image below.
The final step to take before getting started with modeling is establishing a naive baseline. This is essentially a guess against which we can compare our results. If the machine learning models do not beat this guess, then we might have to conclude that machine learning is not acceptable for the task or we might need to try a different approach.
For regression problems, a reasonable naive baseline would be; for all the examples in test set, the corresponding price prediction would be the mean value of the price variable of all the examples from the train set. This sets a relatively low bar for any model to surpass. So let’s apply log transformation to and evaluate the baseline model.
#split the target variable and the predictor variablesx_train, x_test = train.drop(['price'], axis=1), test.drop(['price'], axis=1)y_train, y_test = train['price'], test['price']#log transformationy_train_log = np.log1p(y_train)y_test_log = np.log1p(y_test)#for baseline predictiony_train_mean = y_train_log.mean()from sklearn.metrics import mean_squared_errordef rmsle(y_test, y_pred):      result = (np.sqrt(((y_test-y_pred)**2).mean())).round(4)    return result  baseline_result = rmsle(y_test_log,y_train_mean)print(baseline_result)0.7497
In order for us to say that the machine learning algorithm has performed well, the error obtained from the ML model should be less than the baseline error i.e. 0.7497.
There are a ton of machine learning models to choose from and deciding where to start can be intimidating. One of the approaches would be to try out several algorithms and see which one works best! Machine learning is still a field driven primarily by experimental rather than theoretical results and it’s impossible to know ahead of time which model will do best.
We will evaluate four different models covering the complexity spectrum:
Linear Regression
Support Vector Regression
Decision Tree Regression
Extreme Gradient Boosting Regression (XGBoost)
For each of the above models, we can optimize it by tuning their respective hyperparameters. Model hyperparameters are best thought of as settings for a machine learning algorithm that are set by the data scientist before training. Examples would be the number of trees in a random forest or the number of neighbors used in K-nearest neighbors algorithm.
Model parameters are what the model learns during training, such as weights in a linear regression.
Controlling the hyperparameters affects the model performance by altering the balance between underfitting and overfitting in a model. Underfitting is when our model is not complex enough to learn the mapping from features to target. An underfit model has high bias, which we can correct by making our model more complex.
Overfitting is when our model essentially memorizes the training data. An overfit model has high variance, which we can correct by limiting the complexity of the model through regularization. Both an underfit and an overfit model will not be able to generalize well to the testing data.
The problem with choosing the right hyperparameters is that the optimal set will be different for every machine learning problem! Therefore, the only way to find the best settings is to try out a number of them on each new dataset.
The particular hyperparameter tuning technique that we will apply is Random Search with K Fold Cross Validation :
Random Search refers to the technique we will use to select hyperparameters. We define a grid and then randomly sample different combinations and perform K-fold CV, rather than grid search where we exhaustively try out every single combination. Random search performs nearly as well as Grid search with a drastic reduction in run time.
K-Fold CV is where a given data set is split into a K number of sections/folds where each fold is used as a testing set at some point. Let’s take the example of 5-Fold cross validation(K=5). Here, the data set is split into 5 folds. In the first iteration, the first fold is used to test the model and the rest are used to train the model. In the second iteration, 2nd fold is used as the testing set while the rest serve as the training set. This process is repeated until each fold of the 5 folds have been used as the testing set. The test error from all the five iterations are averaged for that specific hyperparamter combination. After the Random Search CV completes, the hyperparameter combination, for which the average test error is the lowest is selected as the optimal hyperparameter combination. This combination will then be used to predict the prices for the examples in the test data.
We will use the Scikit-Learn library in Python for model building. Once you know how to make one model in Scikit-Learn, you can quickly implement a diverse range of algorithms. We can illustrate one example of model creation, training (using .fit ) and testing (using .predict ) with the Decision Tree Regressor along with Random Search CV:
from sklearn.tree import DecisionTreeRegressorfrom sklearn.model_selection import RandomizedSearchCV# Create the model to use for hyperparameter tuningDT = DecisionTreeRegressor(random_state=42)# Minimum number of samples to split a nodemin_samples_split = [25,50,75,100,200]# Maximum depth of each treemax_depth = [5,10,20,30,50]# Define the grid of hyperparameters to searchparams = {'max_depth':max_depth, 'min_samples_split':min_samples_split}# Set up the random search with 4-fold cross validationRS_DT = RandomizedSearchCV(DT, params, scoring='neg_mean_squared_error', cv=4, n_jobs=-1, random_state=42)# Fit on the training dataRS_DT.fit(x_train_final, y_train_log)# this will return the hyperparameters with lowest CV errorRS_DT.best_params_Out:{'max_depth':30, 'min_samples_split':50}
Now we will use the above hyperparameters to evaluate the performance of the model on the test set.
# Create the model with the optimal hyperparametersDT = DecisionTreeRegressor(max_depth=30, min_samples_split=50, random_state=42)# Fit the model on the training dataDT.fit(x_train_final, y_train_log)# Make predictions on the test datay_pred_dt = DT.predict(x_test_final)# Evaluate the modeldt_error = rmsle(y_test_log, y_pred_dt)print(dt_error)Out:0.582
The RMSLE for the Decision Tree model is 0.582 which is 22.3% lower than the baseline model result. Clearly, machine learning is applicable to our problem because of the significant improvement over the baseline! The same procedure is also followed for the rest of the selected ML models. Let’s see how the rest of the models have performed on the test set.
The Linear Regression model gave the lowest RMSLE on the test set. This shows that simpler ML models can also get the job done and that they should not be underestimated.
Also, both the linear models (Linear regression and Support Vector Regression) have performed better than both the non linear models (Decision Tree and XGBoost Regression).
In this article, we’ve walked through a data science case study where we understood the problem statement, did exploratory data analysis, feature transformations and finally selected ML models, did random search along with hyperparameter tuning and evaluated them on the test set and compared the results.
Machine learning is often an iterative rather than linear process. I hope this case study has at least given you an high level overview about how problems related to data science and machine learning are usually approached and solved.
References that helped me write this blog:
https://towardsdatascience.com/a-complete-machine-learning-walk-through-in-python-part-one-c62152f39420https://towardsdatascience.com/a-complete-machine-learning-project-walk-through-in-python-part-two-300f1f8147e2https://towardsdatascience.com/machine-learning-for-retail-price-suggestion-with-python-64531e64186dhttps://medium.com/unstructured/how-i-lost-a-silver-medal-in-kaggles-mercari-price-suggestion-challenge-using-cnns-and-tensorflow-4013660fcded
https://towardsdatascience.com/a-complete-machine-learning-walk-through-in-python-part-one-c62152f39420
https://towardsdatascience.com/a-complete-machine-learning-project-walk-through-in-python-part-two-300f1f8147e2
https://towardsdatascience.com/machine-learning-for-retail-price-suggestion-with-python-64531e64186d
https://medium.com/unstructured/how-i-lost-a-silver-medal-in-kaggles-mercari-price-suggestion-challenge-using-cnns-and-tensorflow-4013660fcded
Thank you for reading!