Mercari Price Suggestion Challenge — A regression case study with missing value imputation.
By: Yash Dixit
Tags: Kaggle, Mercari Price Suggestion, Yash Dixit, Kaggle Competition, Missing Value Imputation

5. Exploratory Data Analysis -

5. 1. Item Condition ID

Mercari : Item Condition ID

After going through Mercari’s website, I found out that the structure of the permalink for each condition id is such that it represents 1 as New, 2 as Like New, 3 as Good, 4 as Fair and 5 Poor Item Condition.

For example if the permalink was “www.mercari.com/search/?categoryIds=2&deliveryType=all&facets=2&itemConditions=1&length=30", here the text written in bold is the filter of products with condition id “New” which is “1” as you can see in the picture above as well.

Bar plot of Frequencies of different Item Condition IDs.

Therefore with respect to train data as uploaded by Mercari, we can see that there are majority of the products which are still in a good state if not the best(more than just Fair condition with condition ID less than 4).

5. 2. Shipping

The Shipping Column denotes whether the item’s shipping charges have been paid by the seller or the buyer.

“1” : This category represents that the shipping charge has to be paid by the seller.

“0” : This category represents that the shipping charge has to be paid by the buyer.

Bar plot of Frequencies of different Shipping Categories.

There was a very close difference between the number of shipping charges paid by the seller(1) and those paid by the buyer(0), although the majority lies within the shipping charges paid by the buyer(0).

Item Condition ID vs. Shipping

Since the products were still New(Item_condition_ID = 1) the shipping charges have been beared by the Seller for most of the listings. For other types of condition IDs the shipping charges were to be paid by the buyer as it had the majority according to the numbers.

This can also suggest that most of the sellers might have an online market on Mercari where they sell brand new products along with free shipping.

5. 3. Brand Name

As we already had a look at details of data given above, there were 632682 out of 1482535 listings which did not have brand names present in them. This summed up to 43% of the total number of the data-points with missing values.

Top Brands before the Missing value Imputation

With this graph we can see that the top brands in terms of frequency are PINK, Nike, Victoria’s Secret and LuLaRoe.

Now, since there were 43% of missing data-points, I performed a text based missing value imputation considering the name and the item description of the listings since there were just too many brands which were 4810 in number. Since 4810 is a really big number, it would practically take a lot of time and resource for the model based missing value imputers to work. Based on my in-depth research with respect to the data, I came up with a robust text-based missing value imputer which worked really well in filling up the missing values.

Firstly, I pre-processed the brands and stored them in a dictionary like a hash table format. Each key in the dictionary would represent the unique brand in its original form and the value of that key would be a list of its pre-processed forms. I have discussed the pre-processing of brands in detail, step by step, in the section “6.2. Missing value Imputation” over here.

This type of missing value imputation worked really well for me in getting the count of missing values from 632682 to 238448 which was now about 16% from a massive 43% in terms of percentage. So, now, based on my observation, it was better to keep the missing value as a separate category since the respective listings didn’t have a specific mention about the brands.

Top Brands after the Missing value Imputation

Thus, a lot had changed after the missing value imputation with respect to the top brands!

Having a look at top 15 brands for every combination of the item condition IDs and Shipping categories, here were some useful derived insights :

Most of the fashion based brands were brand new since the condition id was 1. Therefore, in the category where the shipping charges were paid by the buyer as well as the seller and the item condition id was the best (which was item condition id as 1), I could see a lot of fashion based brands in majority. Similarly, most of the Electronics based brands had the least quality since condition id was 5. Therefore, in the category where the shipping charges were paid by the buyer as well as the seller and the item condition id was the least which is 5, I could see brands such as Apple, Nintendo, Sony and Xbox as a majority.

5. 4. Category Name

There were 1288 unique Category names and 6327 missing values in the category names’ column. It summed up to around 0.42% of the data in this column had missing values.

There were 3 categories per Category Name in the data. The format of the category was main_category/main_sub_category/sub_category and so I had created 3 lists to split them with respect to the punctuation ‘/’.

I performed the same type of missing value imputation on main_category, main_sub_category and sub_category as done in brands.

Horizontal bar plot of Main categories

In the plot above, we can see that there were a majority of Women-Based Listings.

Horizontal Bar plot of Main Sub-Categories

Since there was a majority of Women-based listings, it was pretty obvious that the main sub-categories too had a majority of women-based main sub-categories.

Horizontal Bar plot of Sub-Categories

Therefore it’s predominant from the graph above that there were a majority of attire-based listings and most of them were related to Women of course.

Further on, after having a look at brands, item condition id and shipping categories, these were some useful derived insights :

Women, Kids and Beauty Categories had the majority share in every type of Item Condition ID and Shipping category combinations. But, the interesting insight here was that, listing based on Electronics and Men categories did not have a good condition id which is 5 and this was observed even while working with the analysis with brands as well. Also, I noticed that during the missing value imputation of the brands, the brand ‘PINK’ might have defaulted as a colour in some of the imputations and hence it created a majority in most of the categories.

5. 5. Colour

Mercari : Colour

Since Mercari had an option of colour for filtering the search for a particular category, it turned out to be a useful feature!

After considering those 14 colours given on Mercari’s website, I observed that there were 40% of the data-points(586403 out of 1482535 data-points) which had colours present in them!

Bar plot of Frequency of Colours

Black, pink and red colours had the highest majority amongst the listings based on colours.

These were some useful insights with respect to Categories, Brands, Item condition IDs and Shipping categories:

The products which were in a “New” condition(Item_condition_id=1) had more colours. The Women based products had a lot of colours.

5. 6. Price

Histogram of Price before Variable Transformation

This is the Histogram of the prices without doing any variable tranformation whatsoever. Since the graph was highly right skewed distributed(positively skewed distribution) and also since the prices were just positive numbers, I decided to opt for variable transformation. This was supposed to be done under the intention that since the data was huge and also since there was a wide range of prices, it could aid in speeding up the training process in some way.

Later on I discovered that there were around 874 listings which had a price of $0! I kept those data-points as it is and did not delete them since it could add on to the interpretability. I kept them the way it is because almost 95% of the listings with $0 price had NaN values present in them even after a rigorous missing value imputation.

Since the graph before was highly right skewed distributed(positively skewed distribution) I transformed the prices into its logarithms(to the base 10).

Histogram of Logarithm of Price

The histogram above didn’t seem to be that smooth and it still has some amount of skew to it even after taking the logarithm.

CDF of Logarithm of Price

Also, because this graph had a multi-modal structure, even the CDF wasn’t pretty smooth.

Since this Variable Transform had a minute right skew, I decided to transform it further by taking logarithm of the logarithm of the Prices. Even after doing the transformation further on, I got the same results where both the Histogram as well as the CDF weren’t smooth.

Since the prices were just positive numbers, I decided to consider and have a look at square root of the prices as well!

Histogram of Square root of the Prices

Thus after taking the square root of the prices, the graph turned out to be pretty smooth!

CDF of Square root of the Prices

Even the CDF was better than the Logarithmic Transform! But this Transformation had a lot of a right skew to it! So, I considered taking the Quadraroots as well as the Octaroots of the prices!

Histogram of Octa-root of the Prices

This is the best transformation possible since it has much less disturbance than the Logarithmic transform and also it has somewhat a pretty centred mean and lower variance.

CDF of Octa-root of the Prices

Even the CDF suggests the same thing of having a centred mean and a lower variance and so I considered this transform while modelling as well.

After further analysis I observed that -

There were around 97% of the products with prices within 2 digits. And 2.98% and 0.02% of the products had prices with 3 digits and 4 digits. CM Style Fashion and Gossip Girl were the 2 brands responsible for the listings with price as $0 as their mean prices were $0. Women, Electronics and Men were the top 3 categories which came into the premium category(which were expensive). It was surprising to see that even if the Electronics category had a major share with the least Item condition id(5) they still cost way more. Apple, Louis Vuitton and Chanel had the most number of listings with prices above $750. The listings which had the shipping charges paid by the buyer cost more for every type of category of item condition id. Thus this might prove profitable for the seller since they do not have to pay for the shipping charges. The colours such as Gold and silver had higher prices. Since the colours are manipulated by string comparison it could even happen that Gold and Silver might be actual metals rather than the colours and so they costed more.

5. 7. Item Description and Name

For the two features, Item Description and Item Name, I had considered only the lengths of the text for EDA.

Histogram of Item Description Length

Since there was a massive right skew in the histogram of lengths of item description, I did the same type of variable transformations as done with the variable Price.

Histogram of Octa-Root of Item Description Length

This wasthe best possible smooth curve I could get even if it had a bimodal characteristic. Similarly, with the lengths of Item names, I considered square root as the type of variable transform. As they are just the lengths of the texts I decided not to work much on them after that.

Mercari had mentioned that they had cleaned the data to remove texts that look like prices (e.g. $20) to avoid leakage. Those removed prices were represented as [rm]. So, I plotted whether [rm] was present in both Item Description and the Item name.

Bar plot of presence of [rm] in Item Description

Bar plot of presence of [rm] in Item Name

It turned out that 9.03% of the Item Descriptions had [rm] present in them and 0.50% of the Item names had the special string present. The [rm] features might prove useful since I observed that the mean prices of those listings which had [rm] present in them were higher as compared to the ones which didn’t have them.

After that I decided to find whether there was a relationship between the lengths of the text features and the prices.

GG Plot of Item description length vs. Price

GG Plot of Item Name length vs. Price

These GG Plots gave a pretty useful insight that less the item description lengths were, more were the prices of those listings. Also, more the length of the item name is, more were the prices of the listings.

The conclusions derived from rest of categorical features were:

If the item description length and/or the item name length were more, then the likelihood of the shipping charges paid by the seller was more for all the categories. As the number of colours increased, the length of the item description also increased.

5. 8. Heatmap