Mercari price recommendation for online retail sellers using Machine learning | by Debayan Mitra | Towards Data Science

Business problemUse of Machine learning / Deep learning to solve the business problemEvaluation metric (RMSLE)Exploratory data analysisFeature engineering (Generating 19 new features)Existing solutionsMy Experiments with models for improvementsSummary, results and conclusionsFuture workLink to my profile — Github code and LinkedinReferences
Business problem
Use of Machine learning / Deep learning to solve the business problem
Evaluation metric (RMSLE)
Exploratory data analysis
Feature engineering (Generating 19 new features)
Existing solutions
My Experiments with models for improvements
Summary, results and conclusions
Future work
Link to my profile — Github code and Linkedin
References
This case study is based on a Kaggle competition conducted in 2018 by Mercari which is an online shopping app. Link to the kaggle Competition — https://www.kaggle.com/c/mercari-price-suggestion-challenge.
You can find my entire code in my github profile (link at the end of this blog).
Mercari is an online selling app (quite similar to Quikr in India). Sellers upload used/refurbished products on the store that they want to sell. When they upload the product on the Mercari App, they want to know how much price they should be selling at. This helps the sellers in pricing/valuating their products before actually selling it.
Sellers upload product information like Item name (text format), Item description (text format), Item brand, Item category, Item condition, shipping status. When they upload these product information on the Mercari app, they in return get back the recommended price that would be sold at.
The goal in this case study is to predict this price of a product listing given the product attributes.
This problem can be best solved using Machine learning techniques, which is basically a regression modelling task which basically looks at historical product information/attributes which were similar and also the price in which that was sold and suggests prices accordingly.
Let us look at one row of the training data and understand its fields. (The entire training data contains ~1.4 million item listings)
The dataset comprises of the following fields —
name — This is the item name/product Name of the product that is being listed by the seller. (Text format)item condition id — Contains values one of (1,2,3,4,5). Basically its a number that represents the condition of the Item from a scale of 1–5.Category — Contains the product category taxonomy of the item as a 3-level hierarchy (seperated by ‘/’ delimiter). In the above example, It is electronics as Level 1 , computers & tablets as Level 2, components & parts as Level 3.Brand — The brand of the item being listed. 60% of rows are empty.Shipping — Contains Boolean — 1,0. ‘1’ indicates that the shipping fee is paid by the seller, ‘0’ indicates not paid by the seller.item description — This contains the detailed text description of the item. This contains as much information as possible about the condition, features and all other information relevant to the product. Naturally, we could imagine that this information would be quite relevant to us because price would depend on features, working condition of certain features etc. This would be an interesting Deep NLP task to solve.Price (to be predicted) — This is the price to be predicted in our case. contains float values between 0 and 2009.
name — This is the item name/product Name of the product that is being listed by the seller. (Text format)
item condition id — Contains values one of (1,2,3,4,5). Basically its a number that represents the condition of the Item from a scale of 1–5.
Category — Contains the product category taxonomy of the item as a 3-level hierarchy (seperated by ‘/’ delimiter). In the above example, It is electronics as Level 1 , computers & tablets as Level 2, components & parts as Level 3.
Brand — The brand of the item being listed. 60% of rows are empty.
Shipping — Contains Boolean — 1,0. ‘1’ indicates that the shipping fee is paid by the seller, ‘0’ indicates not paid by the seller.
item description — This contains the detailed text description of the item. This contains as much information as possible about the condition, features and all other information relevant to the product. Naturally, we could imagine that this information would be quite relevant to us because price would depend on features, working condition of certain features etc. This would be an interesting Deep NLP task to solve.
Price (to be predicted) — This is the price to be predicted in our case. contains float values between 0 and 2009.
(Note — This is not RMSE. It is RMSLE)
The evaluation metric that the competition used was basically a score. RMSLE stands for Root Mean Squared Logarithm Error. Here is a great blog post that I found which explains the differences between RMSE and RMSLE and in what cases RMSLE is more relevant in using as an evalation metric for regression tasks — https://medium.com/analytics-vidhya/root-mean-square-log-error-rmse-vs-rmlse-935c6cc1802a
Summarizing a couple of points from the blogpost about why RMSLE is used as a metric here :-
RMSLE penalizes more when you are under-estimating rather than over-estimating — In other words, RMSLE is used in this project ,it is saying that it’s okay if you give a higher price suggestion for certain Items but if you under predict than the actual, it is not acceptable . RMSLE will increase significantly if we under predict as compared to if we over predict by the same amount.I think this makes business sense for this case study, because we would probably be okay to give a higher price estimate for certain listings than actual, but if we underestimate an item’s price, its not okay because the buyer would probably not want to sell in that case.RMSLE can cushion out the effect of Outliers in the Data — Certain items in the dataset which would have a very high price value can actually skew the model if you evaluate on RMSE because of the squared error penalty in RMSE , whereas the RMSLE would only incur a slightly higher penalty for high value cars compared to low value cars because RMSLE has a logarithm term in the metric that cushions this effect.RMSLE considers only relative error and not the absolute error (because of the logarithm term), so a prediction of 9 against 10 and a prediction of 900 against 1000 are both having same relative errors.
RMSLE penalizes more when you are under-estimating rather than over-estimating — In other words, RMSLE is used in this project ,it is saying that it’s okay if you give a higher price suggestion for certain Items but if you under predict than the actual, it is not acceptable . RMSLE will increase significantly if we under predict as compared to if we over predict by the same amount.I think this makes business sense for this case study, because we would probably be okay to give a higher price estimate for certain listings than actual, but if we underestimate an item’s price, its not okay because the buyer would probably not want to sell in that case.
RMSLE can cushion out the effect of Outliers in the Data — Certain items in the dataset which would have a very high price value can actually skew the model if you evaluate on RMSE because of the squared error penalty in RMSE , whereas the RMSLE would only incur a slightly higher penalty for high value cars compared to low value cars because RMSLE has a logarithm term in the metric that cushions this effect.
RMSLE considers only relative error and not the absolute error (because of the logarithm term), so a prediction of 9 against 10 and a prediction of 900 against 1000 are both having same relative errors.
It is always recommended that we first understand the data very well before starting on predictive modelling. This is an extremely crucial task. Let us run through the data and gain some info and insights :
The prices follow a Log Normal Distribution here. From one of the existing Kaggle kernels, I found out that Mercari allows price listings between 3 and 2000 only. So we will filter for these item listings. If you see the graphs below, even the 99.9 percentile value is around 400. The inflection point occurs beyond that. We can build price slabs (according to 5 percentile slabs) to better analyze the distribution of prices of the item listings.
The shipping status is a binary — (1,0). 1 represents shipping fee is paid by seller. 0 represents if shipping fee is not paid by seller. We can check the violin plot plotted below. We can see that ‘0’ has a slightly higher price than ‘1’ (although I expected to be other way around because my assumption was that if the seller is already paying for shipping the item, then the price is expected to be slightly higher). But is the difference statistically significant ?— we can conduct a one way ANOVA test to do this at 5% significance — and this told that the difference in prices in the shipping status is indeed statistically significant.
Similar analysis for item condition has been done below. The differences in prices here are statistically significant too. No missing values have been found in these 2 features.
Categories feature here appear in ‘/’ seperated values . Most categories have 3 level seperators. (See image below). So the nomenclature here is cat1/cat2/cat3 where cat1 is the high level category followed by cat2 & cat3 which are sub level categories.
import seaborn as snssns.set(rc={'figure.figsize':(8,6)}, style = 'whitegrid')sns.barplot(x = "count", y="cat1", data=df_cat1_counts,palette="Blues_d")
Let us look at what types of words occur in this column using a wordcloud. Also, lets analyze price distributions across categories using violin plots.
sns.set(rc={'figure.figsize':(19,7)})sns.violinplot(x="cat1", y="log_price", data = df_train)plt.title('Violin Plots - Price variation in cat1')plt.show()
import matplotlib.pyplot as pltfrom wordcloud import WordCloudwordcloud = WordCloud(collocations=False).generate(text_cat)plt.figure(figsize = (12,6))plt.imshow(wordcloud, interpolation='bilinear')plt.axis("off")plt.title('WordCloud for Category Name')plt.show()
This feature contains ~4800 brands (with 60% of rows are missing). Top brands (by count)are listed below in dataframe. We can see that top 1000 brands (~ 25% top brands) account for about 97% of the product listings. (close to a power law distribution)
Different brands have different price distributions (as expected ). Brand should occur for one of the most relevant features since prices of products are a very heavy function of the brand of the Item. Follows a long tail skewed distribution as we can see below graph.
Let us understand by plotting their wordclouds. (But after some text preprocessing)
# reference - Applied AI Course (Code for Text Preprocessing)import refrom tqdm import tqdm_notebookdef decontracted(phrase):    phrase = re.sub(r"won't", "will not", phrase)    phrase = re.sub(r"can\'t", "can not", phrase)    phrase = re.sub(r"n\'t", " not", phrase)    phrase = re.sub(r"\'re", " are", phrase)    phrase = re.sub(r"\'s", " is", phrase)    phrase = re.sub(r"\'d", " would", phrase)    phrase = re.sub(r"\'ll", " will", phrase)    phrase = re.sub(r"\'t", " not", phrase)    phrase = re.sub(r"\'ve", " have", phrase)    phrase = re.sub(r"\'m", " am", phrase)    return phrasedef text_preprocess(data):    preprocessed = []    for sentance in tqdm_notebook(data):        sent = decontracted(sentance)        sent = sent.replace('\\r', ' ')        sent = sent.replace('\\"', ' ')        sent = sent.replace('\\n', ' ')        sent = re.sub('[^A-Za-z0-9]+', ' ', sent)        # https://gist.github.com/sebleier/554280        sent = ' '.join(e for e in sent.split() if e not in stopwords)        preprocessed.append(sent.lower().strip())    return preprocessed
Above plot looks good — it gives an information about most frequently occuring words.
Let us also do some Topic Modelling using Latent Dirichlet Allocation. Without getting into details, What this does is basically a unsupervised algorithm that finds topics that the sentences are talking about in the entire text corpus. This analysis was inspired from this Kaggle kernel — link.
Let us look at some of the topics generated (by grouping similar words/sentences into topics- which the LDA algorithm does). LDA does a pretty decent job here in our case when applied to the item descriptions feature.
Topic 0 is most likely referring to Jewellery— (necklace,bracelet,chain,gold). Similarly topic 3 is referring to Apparel— (shirt, nike, men,women). Topic 7 is referring to Accessories— (leather,bag,wallet) etc. So we get a good sense of what the item descriptions column is talking about and a summary of the various topics present in it. This feature would be a very important predictor of price as it contains information related to item condition, newness, detailed features of the product etc.
My first hunch intuition is that this would be a Deep NLP task and needs to be solved using some form of LSTM Neural Network.
This was one of the most important parts in the ML system. Inspired by this Kaggle kernel link. Some new features were generated.
The hypotheses here was that the better sentiment an item is conveyed with in item description, the higher the buyers are willing to pay for the Item. I expected a positive co-relation between sentiment scores of item description and price.
from nltk.sentiment.vader import SentimentIntensityAnalyzerdef generate_sentiment_scores(data):    sid = SentimentIntensityAnalyzer()    scores = []    for sentence in tqdm_notebook(data):         sentence_sentiment_score = sid.polarity_scores(sentence)        scores.append(sentence_sentiment_score['compound'])    return scores
I borrowed this code snippet from this Kaggle kernel — link . This basically gets the price statistics by grouping the (Category, Brand, Shipping) features together and generating price statistics — Mean, Median, Std. Deviation, Coefficient of variation, Expected price ranges based on 2 Std. deviations from mean etc. This makes sense because we are basically looking at historical prices of a group of items and feeding them in the feature set to assume that they might be well co-related to the current Prices.
I’m again borrowing this code snippet from the same Kaggle kernel link again as posted above. The code basically creates new features from the item description column like word counts, number of special characters, number of numerics etc.
The most important question is do these new features co-relate well to price — for this we can plot a correlation heatmap.
#ref = https://datatofish.com/correlation-matrix-pandas/# df_corr contains all the newly generated featurescorrMatrix  = df_corr.corr()plt.figure(figsize = (18,9))sns.heatmap(corrMatrix, annot=True)plt.show()
As we can see from the heatmap above, 3 features have a strong co-relation to price output with corelation > 0.5 . The mean, min expected, max expected price features generated seem to be most significant. We shall keep only these 3 new features.
There are many kernels on different approaches, some of them include —
CNN with GloVE for word embeddings — kernel link. This uses CNN model over word embeddings on item name, item description concatenated along with the categorical features to pass it through a Dense MLP. Model gives an RMSLE score of 0.41 (35th place)Sparse MLP — kernel link. This uses a sparse MLP to generate output over Tfidf Vectorizations of text and One Hot Encoding of categorical features. Model gives an RMSLE of 0.38 (1st place)Ridge Model — kernel link. Uses a simple Ridge Regression over Tfidf Text features to generate predictions. Model gives an RMSLE of 0.47LGBM Model — kernel link . Uses LightGBM Regressor to give output score of 0.44.
CNN with GloVE for word embeddings — kernel link. This uses CNN model over word embeddings on item name, item description concatenated along with the categorical features to pass it through a Dense MLP. Model gives an RMSLE score of 0.41 (35th place)
Sparse MLP — kernel link. This uses a sparse MLP to generate output over Tfidf Vectorizations of text and One Hot Encoding of categorical features. Model gives an RMSLE of 0.38 (1st place)
Ridge Model — kernel link. Uses a simple Ridge Regression over Tfidf Text features to generate predictions. Model gives an RMSLE of 0.47
LGBM Model — kernel link . Uses LightGBM Regressor to give output score of 0.44.
We can use the original training data to split into train/test with test data size of 25%.
from sklearn.model_selection import train_test_splitdf_train = pd.read_csv('train.tsv',sep = '\t')df_train_model,df_test_model = train_test_split(df_train,test_size = 0.25)
The input features consisted of categorical and text features, along with the 3 newly generated numeric features. Categorical encodings involved One Hot Encoders, LabelBinarizers. Text encodings involved Tfidf and Count (BOW) Vectorizers. Numeric encodings involved normalization using StandardScaler.
The output variable — price involved a log transform as discussed in EDA section.
import numpy as npy_train  = np.log1p(df_train['price'])
We can start by building a simple linear model with L2 Regularizer which is basically called Ridge Regression. This Model gave a RMSLE of 0.474 on test data. Note that solver = ‘lsqr’ is used for faster training — takes less than a minute to train this model.
%%timefrom sklearn.linear_model import Ridgeridge_model = Ridge(solver = "lsqr", fit_intercept=False)ridge_model.fit(X_train, y_train)preds = np.expm1(ridge_model.predict(X_test))
We can try using GridSearchCV on this model to see if we can improve the score.
from sklearn.model_selection import GridSearchCVparameters = {'alpha':[0.0001,0.001,0.01,0.1,1,10,100,1000,10000],              'fit_intercept' : [False],              'solver' : ['lsqr']}gs_ridge = GridSearchCV(estimator = Ridge(),                        param_grid = parameters,                        cv = 3,                         scoring = 'neg_mean_squared_error',                        verbose = 100,                        return_train_score = True,                        n_jobs = -2)gs_ridge.fit(X_train, y_train)
As we can see below, alpha=10 gives lowest cross validation score when using a 3 fold cross validation. But when using this model. the RMSLE reduced only from 0.474 to 0.472 which is not very significant improvement. Note below in graph, Log Scale of 10 is used since all hyperparameter alpha values were in powers of 10 in conducitng the Grid Search.
lgbm_params_1 = {'n_estimators': 900,                  'learning_rate': 0.15,                 'max_depth': 5,                 'num_leaves': 31,                  'subsample': 0.9,                 'colsample_bytree': 0.8,                 'min_child_samples': 50,                 'n_jobs': -2}
We can improve the Model by Tuning the hyperparameters to slightly overfit by increasing the estimators, reducing learning rate, not limiting max depth, Increasing Num Leaves etc. By tuning these, model improved from 0.50 to 0.47 which is indeed a significant improvement especially in Kaggle competitions and also that scoring is in log scale (RMSLE). Although this model is slightly overfit, we can reduce the model variance by building an ensemble towards the end as we shall see. Main issue with LightGBM is training time — took nearly 2–3 hrs to run on my desktop with 16 GB RAM.
lgbm_params_2 = {'n_estimators': 1500,                 'learning_rate': 0.05,                 'max_depth': -1,                 'num_leaves': 50,                 'subsample': 0.8,                 'colsample_bytree': 0.8,                 'min_child_samples': 50,                 'n_jobs': -1}
This was the first hunch where we can assume safely that the model would work well since most of our data is present in the form of text in item names and item descriptions. Also, all the numeric newly generated 17 features have been passed as input to the numeric features. My hypotheses was that the neural network would learn by itself if these features were relevant or not — so shouldn’t harm the model.
Attached below is the 1st LSTM model — As you can see below, the text and categorical features are first passed through embedding layers and then flattened to concatenate into a Dense NN architecutre.
from tensorflow.keras.utils import plot_modelfrom IPython.display import Imageplot_model(baseline_lstm_model, to_file='baseline_lstm_model.png', show_shapes=True, show_layer_names=True)Image(filename='baseline_lstm_model.png')
This model above gave a RMSLE of 0.48 after 60 epochs and val loss failed to improve significantly when used with adam optimizer. Attached below is the code to generate the above LSTM model 1.
We can select the sentence length by plotting the distribution of sentence lengths.
As we can see from the code, only 0.59% of rows in the item description column have sentence length more than 125. Which means nearly all sentence lengths fall in this category and we can use this for sentence length selection.
>>>select_padding(train_desc,tokenizer_desc,"Item Descriptions",125)Output :Total number of words in the document are  1384190.5927708414700212  % of rows have sentence length >  125
In this model, We have concatenated the texts from name, description, brand, category and created a text feature. I also selected the 3 numeric features which were highly co-related with price as numeric inputs. This model gave an RMSLE of 0.47
The winner’s solution Model used a simple MLP model with sparse input representation of features with only 4 layers . This model gave good results as it was able to learn feature interactions in a better way as compared to LSTM/LGBM models. I added a few layers in my experiment on top of this along with dropouts and BatchNormalization to avoid overfitting. After building 2 such models, We can combine them using some form of ensembling.
As you can see above, the model is as simple as it can be. The 3 MLP models gave RMSLE scores of 0.415, 0.411 respectively.
After building many such models, We can combine them in a way to give better performance. We can use cross validation methods to get the best ensemble model combination. For my final model, However I decided to use only the Ridge Models and MLP models because they were faster to train (< 15 mins), whereas LSTM and LGBM models individually took >3 hrs to train.
What we are doing here in the above method is taking the predictions of the 2 models that we want to ensemble and assign weights to each of them in a way that gives best RMSLE — we can find this using cross validation. When we do this for an ensemble of (2 Ridge Models + 2 MLP Models) -> gives an RMSLE of 0.408 which is significantly lower than all individual models.
preds_final=ensemble_generator(preds_final_ridges,preds_final_mlps)
Roughly 0.1 weightage to Ridge Models and remaining 0.9 weightage to MLP models gives the best Score.
preds_f = 0.1*preds_ridge + 0.9*preds_mlps
Below is the summary of all the models trained and the RMSLE scores obtained. As we can see, the ensemble model is the winner.
Making the final submission on the unseen test dataset in Kaggle gave an RMSLE score of 0.39 which would be in top 1% leaderboard.
As you can see from the graph below, ~55% of points have errors < 5 and ~76% of points have errors < 10. This is one of the way we can visualize the distribution of errors. Here error = (actual — predicted).
Another plot that we can do is plot the Log(Absolute Error) because we have done the price modelling in the log Scale. In the plot below, x-axis is the Log(Abs Error), left y-axis represents the number of points (blue histgoram) against that error. The orange line gives the cumulative points in the right y-axis. We can see that, ~ 99% points have Log(Abs Error) < 4.
We could experiment more on this to further improve predictions —
Using hyperas library to fine tune the neural network architectures to further improve performance
Using other Vectorizations like DictVectorizer() on the text data to generate text features
Use Convolution layers on word embeddings on text data
You can find my complete code here on github link. You can connect me on my linkedin profile link if you want to discuss. You can also connect me on debayanmitra1993@gmail.com.