Price Prediction using Machine Learning Regression — a case study | by Arun Kumar | Towards Data Science

This article is a detailed account of my approach to solving a regression problem, which is also a popular Kaggle competition. Hope you find it useful and enjoy reading it :)
Artificial Intelligence is an integral part of all major e-commerce companies today. With the evolution of the information industry and extensive research in the field of AI in the past two decades, businesses have started to explore the ways to automate various activities using state of the art Machine Learning algorithms and Deep Neural Networks. Many IT giants and start-ups have already taken a big leap in this field and have dedicated teams and resources for research and development of cutting edge AI applications. Online retail platforms today are extensively driven by AI-powered algorithms and applications. Activities ranging from inventory management and quality checking at the warehouse to product recommendation and sales demographics on the website, all employ machine learning at various scales.
Mercari is Japan’s biggest community-powered shopping website. With the aim of realizing a society where global resources are used carefully and where everyone can live richly, the company has developed a flea market application ‘Mercari’ in Japan and the United States that allows individuals to easily and safely buy and sell goods. Mercari’s challenge is to build an algorithm that automatically suggests the right product prices to sellers on its app.
Predicting the price of a product is a tough challenge since very similar products having minute differences such as different brand names, additional specifications, quality, demand of the product, etc. can have very different prices. For example, one of these sweaters cost $335 and the other cost $9.99. Can you guess which one’s which?
Price prediction gets even more difficult when there is a huge range of products, which is common with most of the online shopping platforms. Mercari’s sellers are allowed to list almost anything on the app. It’s highly challenging to predict the price of almost anything that is listed on online platforms.
Let’s look at this problem from Machine Learning perspective.
Mercari has provided user-inputted text descriptions of its products, including details like product category name, brand name, and item condition. Using this data, we have to come up with a model that predicts the price of a product listed on Mercari as accurately as possible. This looks like a standard regression problem.
We must have a yardstick to measure how good or bad our model’s performance is. In machine learning terminology, we call this yardstick performance metric or simply metric. There are various metrics to measure the performance of a regression model, e.g. mean absolute error, mean squared error, mean squared logarithmic error, maximum residual error, median absolute error, coefficient of determination(R2), etc.For this problem, Kaggle uses Root Mean Squared Logarithmic Error(RMSLE). Lesser the RMSLE, better is our model. RMSLE is calculated as
Predicting the price with utmost accuracy is our top business objective.
There are no low latency constraints in this problem. Once a product is listed on the app, we need not suggest its price immediately. However, we can also not take hours or days to predict the price.
18th place solution: The general idea is to train an FM_FTRL model and then a LightGBM model and use an ensemble of both to get the final predictions.Mercari Golf: 0.3875 CV in 75 LOC, 1900s: an ensemble of 4 MLP models, each model having the same architecture but being trained on apparently 2 different datasets.
18th place solution: The general idea is to train an FM_FTRL model and then a LightGBM model and use an ensemble of both to get the final predictions.
Mercari Golf: 0.3875 CV in 75 LOC, 1900s: an ensemble of 4 MLP models, each model having the same architecture but being trained on apparently 2 different datasets.
Having understood the constraints, business objectives and the problem we need to solve , it’s time to get our hands dirty.
The data can be downloaded from the Kaggle competition page. There are two files train.tsv and test.tsv and a Kaggle submission template sample_submission.csv. The total size of the data is 1.03 GB after decompression.
The files consist of product listings. These files are tab-delimited. train.tsv has 1,482,535 rows and test.tsv has 3,460,725 rows. Both train and test files have the following data fields
train_id or test_id - the id of the listing
name - the title of the listing
item_condition_id - the condition of the items provided by the seller
category_name - category of the listing
brand_name
price - the price that the item was sold for. This is the target variable that we will predict. The unit is USD. This column doesn't exist in test.tsv since that is what we will predict.
shipping - 1 if the shipping fee is paid by seller and 0 if paid by buyer
item_description - the full description of the item
A row in a data table is called a data point and a column is called a feature/variable. Going forward in this blog, I will use the words row and data point interchangeably. Same follows with column and feature/variable.
Below is a sample from the train data.
We see that there are some null values (NaN) in the data. We would replace these with ‘missing’. Also, we split the list of three values in the column category_name into three new columnsgencat_name, subcat1_name, subcat2_name. For example, a data point with category_name=[Men, Tops, T-shirts] will have gencat_name=Men, subcat1_name=Tops, and subcat2_name=T-shirts.
Let’s look at the training data from a machine learning engineer’s perspective and see if we can draw some useful inferences. This will give us a fair idea about what our approaches to solving the problem should be.
name
train.name.describe()count     1481661unique    1224596top        Bundlefreq         2232Name: name, dtype: object
There are 1,224,596 unique product names in the raw data(before pre-processing). The most common product name is “Bundle”, which occurs in 2,232 data points.
item_condition_id
The majority of the items are in condition 1. There are very few items which are in condition 5.
brand_name
train.brand_name.describe()count     1481661unique       4808top       missingfreq       632336Name: brand_name, dtype: object
brand_name is missing for a large number (42.68 %) of data points.
gencat_name
The products are distributed across 10 general categories. This column is blank for some of the products, these have been put into a separate category missing.
A huge number of products belong to the category Women. This might be due to the fact that usually, women tend to shop more when compared to men.
gencat_name is missing for 6,314 data points, i.e. 0.43 % of train data.
subcat1_name, subcat2_name
train.subcat1_name.describe()         train.subcat2_name.describe()count              1481661            count                  1481661unique                 114            unique                     871top       Athletic Apparel            top    Pants, Tights, Leggingsfreq                134321            freq                     60152
Further to 10 general categories, there are 114 subcategories of products, which in turn may belong to 871 further subcategories. To put it simply, subcat1_name tells about the category of the item in a broader sense whereas subcat2_name goes one level deeper and gives more detail about what exactly the item is. Recall the example [Men, Tops, T-shirts]The below figures show the top 15 subcategories by number of items.
price
This is the target variable or ‘y’ in our train data. We need to predict this for test data. It has a wide range of values ranging from $3 to $2009. However, most of the items have a price lower than $200, as can be seen in the below graphs and percentile values.
0 percentile value is 3.010 percentile value is 7.020 percentile value is 10.030 percentile value is 12.040 percentile value is 14.050 percentile value is 17.060 percentile value is 20.070 percentile value is 26.080 percentile value is 34.090 percentile value is 51.0100 percentile value is  2009.0
90 percentile value is 51.091 percentile value is 55.092 percentile value is 58.093 percentile value is 62.094 percentile value is 67.095 percentile value is 75.096 percentile value is 85.097 percentile value is 99.098 percentile value is 122.099 percentile value is 170.0100 percentile value is  2009.0
97% of data points have a price less than USD 100.
Very few (only 1%) data points have a price more than USD 170.
Let us explore if there are any trends in the price of the items depending on the values of different columns of data. This will help us decide which columns will be more useful than others in determining the price of an item.
Variation of price with the item condition
I have used simple box plots to see how the price of an item varies with the condition of the item.Note that in a box plot the lower boundary of the box denotes the 25th percentile, upper boundary denotes the 75th percentile, and the line inside the box denotes the 50th percentile or the median.
There is a slight variation of price based on item condition.
Median price decreases as we go from conditions 1 to 4.
Items in condition 5 seem to be having a higher price, probably because they are costly items like electronics.
Variation of price with item category(gencat_name)
Looking at the box plots and mean price of the various categories in the below figure, we can say that there is some variation in the price of items belonging to different product categories.
Variation of price with item subcategory(subcat1_name)
There is a strong variation in the prices of some of the categories of products. For example, items belonging to Computers Tablets, Cameras Photography, Strollers, Musical Instruments, etc. are expensive when compared to the items belonging to Paper Goods, Children, Office Supplies, Trading Cards, etc.
This indicates that categories and subcategories are going to be important features in determining the price of an item.
I have done some basic text pre-processing like removal of non-alphanumeric characters, regular expressions, stopwords, etc. from name and item_description. The code is given below.
This is similar to text pre-processing. I have removed blank spaces and replaced symbol ‘&’(and) with ‘_’(underscore). I have done this cleaning in order to get precise one-hot encoding of categories, which is explained in the featurization section.
Removed 874 rows
A machine understands only numbers, it does not directly understand letters or text that we as humans can read. That means we need to convert our text and categorical data to numbers. This process is called feature extraction or featurization. There are several feature extraction techniques related to different kinds of data, we will see some of these in this section.
Based on my intuition and research on existing approaches, I came up with the following features that I thought would be useful in determining the price of an item. There is no standard rule for using these features, these are purely intuition-based ideas which may vary from problem to problem. Therefore, these are called hand-made features or engineered features. The below table provides the names of the features. description of what each of these features means and how it is calculated.
Let’s check if the above features are really useful.
Uni-variate analysis on some of the engineered features
The below plots have been achieved using seaborn regplot which tries to fit a simple linear model and plots the equation of fitted line/plane over a uni-variate (feature vs target) scatter plot.
example code:sns.regplot(x=’brand_mean_price’, y=’price’, data=train,            scatter_kws={‘alpha’:0.3}, line_kws={‘color’:’orange’})
Features such as brand_mean_price, brand_median price, subcat2_mean_price, subcat2_median_price show strong linear trends. Thus they seem to be useful in determining the price of items.
For the purpose of cross-validation(checking if the trained model is working well on unseen data), I have split our data into train and cv in the ratio of 90:10. I will train our models on train and validate them on cv.Note that the target variable price has been converted to logarithmic scale by using NumPy’s log1p() function. This has been done so that we can use root mean square error as the metric instead of explicitly defining a complex metric RMSLE. (since RMSLE is nothing but RMSE of log values)
Train size: (1332967, 58), CV size: (148108, 58), Test size: (3460725, 58)
One-hot encoding is a scheme of vectorization where each category in a categorical variable is converted into a vector of length equal to the number of data points. The vector contains a value of 1 against each data point that belongs to the category corresponding to the vector and contains 0 otherwise. To understand better, look at the following example.
I have converted all the categorical variables (brand_name, gencat_name, subcat1_name, subcat2_name) to their one-hot encoded vectors. Example code is shown below:
Note that categorical variables item_condition_id and shipping already contain numerical values and there is no need to convert them to vectors.
TF-IDF (term frequency-inverse document frequency) is a statistical measure that evaluates how relevant a word is to a document in a collection of documents. This is done by multiplying two metrics: how many times a word appears in a document, and the inverse document frequency of the word across a set of documents. You can read more about TF-IDF and its mathematical details here.
uni-grams, bi-grams and n-grams: In the fields of computational linguistics and probability, an n-gram is a contiguous sequence of n items from a given sample of text or speech. The items can be syllables, letters, words, etc. depending on the application (words in our case). An n-gram of size 1 is referred to as a ‘uni-gram’, size 2 is a ‘bi-gram’ and so on.
I have encoded name and item_description into TF-IDF vectors of uni-grams, bi-grams and tri-grams. Note that using 1,2,3-grams together would result in a huge number of words in the dictionary of TF-IDF vectorizer and using all of them would result in very high dimensional vectors. To avoid this, I have limited the number of dimensions to 250k for name and 500k for item_description vectors.
The primary purpose of normalization is to scale numeric data from different columns down to an equivalent scale so that the model doesn’t get skewed due to huge variance in a few columns. I have used min-max normalization here(code given below).
We would be feeding our models an input matrix X_train, which contains all the features that we have extracted in the previous section, and an array of corresponding target values, y_train. Therefore, we need to first build X_train by concatenating all the feature vectors side by side.
Train size: (1332967, 755695), CV size: (148108, 755695), Test size: (3460725, 755695)
Now our data is ready to be fed to models. Let’s start modeling.
I will train the following regression models one by one and evaluate their performance on the validation data:
Ridge regressor: linear least squares with l2 regularization
SVM regressor: Support Vector regression with RBF kernel.
RandomForest regressor: a meta estimator that fits a number of decision trees on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting.
LightGBM regressor: a gradient boosting model that uses tree-based learning algorithms.
To know more about these models and read the documentation click on the model name.
Ridge is a linear least squares model with l2 regularization. In other words, it is linear regression with l2 regularizer.Over-fitting or under-fitting of the Ridge model depends on the parameter alpha, which can be tuned to the right value by doing hyper-parameter tuning as shown below.
RMSLE for alpha =  1 is 0.45166601980352833RMSLE for alpha =  2 is 0.44431546233122515RMSLE for alpha =  3 is 0.4424425182737627RMSLE for alpha =  3.5 is 0.44171501703551286RMSLE for alpha =  4 is 0.44154616529145424RMSLE for alpha =  4.5 is 0.4415286167861061RMSLE for alpha =  5 is 0.44161632764828285RMSLE for alpha =  6 is 0.4421832813533032RMSLE for alpha =  7 is 0.44267468278758176
Training the model using best hyper-parameter and testing
Best alpha:  4.5                            Train RMSLE: 0.383449898Cross validation RMSLE:  0.441528616
The models that we would try going further take tremendous amount of time to train when the data is high dimensional. Therefore, I am selecting only top 48,000 features from text TF-IDF vectors and categorical one-hot encoded vectors.
I concatenate these with the numerical features, ridge model’s predictions(y_pred being used as feature)and use them with my models.
Train size: (1332967, 48049), CV size: (148108, 48049), Test size: (3460725, 48049)
SVR is an advanced version of linear regression. It finds a hyperplane in a d-dimensional space that distinctly classifies the data points.
Similar to alpha in linear regression, C in SVR is found by hyper-parameter tuning.
Best C:  0.3SVR(C=0.3, epsilon=0.1, gamma='scale',kernel='rbf', max_iter=200)Train RMSLE: 0.441563037Cross validation RMSLE:  0.457637217
Training RandomForest Regressor with higher values of n_estimators (N) was taking tremendous amount of time without giving any results. Due to this reason, we have trained it with less number of estimators. As you can guess, the results were not satisfactory.
RMSLE for N=10 is 0.487657647	 elapsed time:0:46:21.136700RMSLE for N=20 is 0.472606976	 elapsed time:2:02:11.229545
Light GBM is a fast, distributed, high-performance gradient boosting framework based on decision tree algorithm, used for ranking, classification and many other machine learning tasks.Since it is based on decision tree algorithms, it splits the tree leaf-wise with the best fit whereas other boosting algorithms split the tree depth-wise. So when growing on the same leaf in Light GBM, the leaf-wise algorithm can reduce more loss than the depth-wise algorithm and hence results in much better accuracy. Also, it is surprisingly very fast, hence the word ‘Light’.
Hyperparameter tuning for LightGBM has been done using 3-fold cross-validation using RandomizedSearchCV.
{'colsample_bytree': 0.44583275285359114, 'learning_rate': 0.09997491581800289, 'max_depth': 12, 'min_child_weight': 1.7323522915498704, 'n_estimators': 1323, 'num_leaves': 123}
Train RMSLE: 0.319789825Cross validation RMSLE:  0.424231390
The validation score of LGBM (RMSLE=0.42423) was the best out of all the above models.
I submitted the predictions of Ridge and LGBM to Kaggle. We got the best score with Ridge, RMSLE=0.45444
The score of LGBM (RMSLE=0.45785) was very close to that of Ridge.
The classical machine learning models seem to work decently well but we ought to have much better performance to get a good Kaggle score. Most of the existing approaches have employed some or the other deep learning models such as Convolutional Neural Networks(CNNs), Recurrent Neural Networks(RNNs) or a combination of both. The performance of deep learning models seems to be significantly better than classical ML models, which encouraged me to try a basic deep learning model, MLP.
I have done the following processing on train and test data:
standard text pre-processing (stemming, fill NAs)
concatenating name, brand_name to one column, name.
concatenating name, item_description, category_name to one column, text.
Tfidf 1-gram vectorization of name.
Tfidf 1-gram, 2-gram vectorization of text.
One-hot encoding of categorical features item_condition_id, shipping.
The reasons for choosing MLP over CNN or RNN are:
Fast to train, can afford hidden size 256 instead of 32–64 for RNN or Conv1D.
MLP captures interactions between text and categorical features.
Huge variance gives a strong ensemble with a single model type.
I have trained 4 high variance models of exactly the same architecture and finally taken ensemble of these to get final predictions. This is similar to Bagging in RandomForest. For 2 out of 4 models I have binarized the input data by setting all non-zero values to 1. This is like getting an extra dataset with a binary CountVectorizer instead of TF-IDF.
I used Adam optimizer with learning rate 0.003 and an initial batch size of 512 and trained the model for 2 epochs, doubling the batch size at each epoch. For binarized input, I trained 3 epochs in the same fashion.
With the above model, I got a validation RMSLE=0.3848, which is a great improvement compared to all my previous models.
The final submission score on Kaggle with this model was 0.39446 in the private leaderboard. Although the competition was closed long ago, placing this score on the leaderboard puts me at the 5th position (top 0.2%) in both private and public LB.
I have made some changes in the MLP architecture as well as parameters such as learning rate and batch size to get better results. I have also changed the number of epochs from 3 to 2 for the model on non-binary data, as it starts overfitting from the 3rd epoch.
Instead of taking simple mean, I have taken a weighted average of predictions from 4 models/runs.
For simplicity of the code, and also because I have used Google Colab(training is faster with GPUs than with multi-core CPUs), I have trained the models one after another unlike pool processing in the original kernel.
The validation RMSLE I got was 0.3848 as compared to 0.3875 in the source kernel.
RandomForest was taking too much time to train and hence I had to discard this model.
In the MLP I also tried using dropouts (0.1, 0.2, 0.3, .. 0.5) but the models performed better without dropouts and hence removed them. (I got a validation RMSLE of 0.3872 with dropouts).
I also experimented with different activation units (‘tanh’, ‘sigmoid’, ‘linear’, ‘relu’). ‘relu’ performs significantly better than rest all.
Using deep learning was productive and yielded a very good score on test data. More complex models such as LSTMs and Convolutional Neural Nets can be tried.
We can experiment with more complex MLPs by adding additional layers and larger number of units in hidden layers.
Other vectorization schemes such as Wordbatch can be experimented with ML models.
Regression models like FTRL and FM_FTRL can also be tried.
It was fun as well as a great learning experience doing this case study. I thank you for reading this blog and I hope this has added some value to you. I have included minimal code in this blog. For full code, please refer to the ipython notebooks in my GitHub repository Mercari-Price-Suggestion-Challenge.I would love to hear your reactions, suggestions or queries.You can connect with me on LinkedIn. Here’s my profile.
https://blog.exploratory.io/a-practical-guide-of-exploratory-data-analysis-with-linear-regression-part-1-9f3a182d7a92https://www.kaggle.com/c/mercari-price-suggestion-challenge/overviewhttps://www.kaggle.com/c/mercari-price-suggestion-challenge/discussion/50252https://www.kaggle.com/lopuhin/mercari-golf-0-3875-cv-in-75-loc-1900-shttps://www.appliedaicourse.com/course/11/Applied-Machine-learning-course
https://blog.exploratory.io/a-practical-guide-of-exploratory-data-analysis-with-linear-regression-part-1-9f3a182d7a92
https://www.kaggle.com/c/mercari-price-suggestion-challenge/overview
https://www.kaggle.com/c/mercari-price-suggestion-challenge/discussion/50252
https://www.kaggle.com/lopuhin/mercari-golf-0-3875-cv-in-75-loc-1900-s
https://www.appliedaicourse.com/course/11/Applied-Machine-learning-course
Some rights reserved