Price Prediction using Machine Learning Regression — a case study
By: Arun Kumar
Tags: Artificial Intelligence, E Commerce Solution, Exploratory Data Analysis, Machine Learning, Regression

There is a strong variation in the prices of some of the categories of products. For example, items belonging to Computers Tablets, Cameras Photography, Strollers, Musical Instruments, etc. are expensive when compared to the items belonging to Paper Goods, Children, Office Supplies, Trading Cards, etc.

This indicates that categories and subcategories are going to be important features in determining the price of an item.

3. Cleaning the data: Pre-processing

3.1. Pre-processing text features

I have done some basic text pre-processing like removal of non-alphanumeric characters, regular expressions, stopwords, etc. from name and item_description. The code is given below.

3.2. Pre-processing categories

This is similar to text pre-processing. I have removed blank spaces and replaced symbol ‘&’(and) with ‘_’(underscore). I have done this cleaning in order to get precise one-hot encoding of categories, which is explained in the featurization section.

3.3. Pre-processing price: removing invalid entries

Removed 874 rows

4. Extracting features from data

A machine understands only numbers, it does not directly understand letters or text that we as humans can read. That means we need to convert our text and categorical data to numbers. This process is called feature extraction or featurization. There are several feature extraction techniques related to different kinds of data, we will see some of these in this section.

4.1. Hand made features: Feature Engineering

Based on my intuition and research on existing approaches, I came up with the following features that I thought would be useful in determining the price of an item. There is no standard rule for using these features, these are purely intuition-based ideas which may vary from problem to problem. Therefore, these are called hand-made features or engineered features.

The below table provides the names of the features. description of what each of these features means and how it is calculated.

Let’s check if the above features are really useful.

Uni-variate analysis on some of the engineered features

The below plots have been achieved using seaborn regplot which tries to fit a simple linear model and plots the equation of fitted line/plane over a uni-variate (feature vs target) scatter plot.

example code:

sns.regplot(x=’brand_mean_price’, y=’price’, data=train,

scatter_kws={‘alpha’:0.3}, line_kws={‘color’:’orange’})

Features such as brand_mean_price, brand_median price, subcat2_mean_price, subcat2_median_price show strong linear trends. Thus they seem to be useful in determining the price of items.

4.2. Train, Test split for cross-validation

For the purpose of cross-validation(checking if the trained model is working well on unseen data), I have split our data into train and cv in the ratio of 90:10. I will train our models on train and validate them on cv.

Note that the target variable price has been converted to logarithmic scale by using NumPy’s log1p() function. This has been done so that we can use root mean square error as the metric instead of explicitly defining a complex metric RMSLE. (since RMSLE is nothing but RMSE of log values)

Train size: (1332967, 58), CV size: (148108, 58), Test size: (3460725, 58)

4.3. Converting categorical features to numbers: One-hot encoding

One-hot encoding is a scheme of vectorization where each category in a categorical variable is converted into a vector of length equal to the number of data points. The vector contains a value of 1 against each data point that belongs to the category corresponding to the vector and contains 0 otherwise. To understand better, look at the following example.

Image source: DanB via Kaggle

I have converted all the categorical variables (brand_name, gencat_name, subcat1_name, subcat2_name) to their one-hot encoded vectors. Example code is shown below:

Note that categorical variables item_condition_id and shipping already contain numerical values and there is no need to convert them to vectors.

4.4. Converting text features to numbers: TF-IDF vectorizer

TF-IDF (term frequency-inverse document frequency) is a statistical measure that evaluates how relevant a word is to a document in a collection of documents. This is done by multiplying two metrics: how many times a word appears in a document, and the inverse document frequency of the word across a set of documents. You can read more about TF-IDF and its mathematical details here.

uni-grams, bi-grams and n-grams: In the fields of computational linguistics and probability, an n-gram is a contiguous sequence of n items from a given sample of text or speech. The items can be syllables, letters, words, etc. depending on the application (words in our case). An n-gram of size 1 is referred to as a ‘uni-gram’, size 2 is a ‘bi-gram’ and so on.

I have encoded name and item_description into TF-IDF vectors of uni-grams, bi-grams and tri-grams. Note that using 1,2,3-grams together would result in a huge number of words in the dictionary of TF-IDF vectorizer and using all of them would result in very high dimensional vectors. To avoid this, I have limited the number of dimensions to 250k for name and 500k for item_description vectors.

5. Preparing the data for models

5.1. Column Normalization on numerical features

The primary purpose of normalization is to scale numeric data from different columns down to an equivalent scale so that the model doesn’t get skewed due to huge variance in a few columns. I have used min-max normalization here(code given below).

5.2. Consolidate all features to a sparse matrix

We would be feeding our models an input matrix X_train, which contains all the features that we have extracted in the previous section, and an array of corresponding target values, y_train. Therefore, we need to first build X_train by concatenating all the feature vectors side by side.

Train size: (1332967, 755695), CV size: (148108, 755695), Test size: (3460725, 755695)

Now our data is ready to be fed to models. Let’s start modeling.

6. Modeling: Machine Learning Models

I will train the following regression models one by one and evaluate their performance on the validation data:

Ridge regresso r : linear least squares with l2 regularization

: linear least squares with l2 regularization SVM regresso r : Support Vector regression with RBF kernel.

: Support Vector regression with RBF kernel. RandomForest regresso r : a meta estimator that fits a number of decision trees on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting.

: a meta estimator that fits a number of decision trees on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting. LightGBM regressor: a gradient boosting model that uses tree-based learning algorithms.

To know more about these models and read the documentation click on the model name.

6.1. Ridge regression

Ridge is a linear least squares model with l2 regularization. In other words, it is linear regression with l2 regularizer.

Over-fitting or under-fitting of the Ridge model depends on the parameter alpha, which can be tuned to the right value by doing hyper-parameter tuning as shown below.

RMSLE for alpha = 1 is 0.45166601980352833

RMSLE for alpha = 2 is 0.44431546233122515

RMSLE for alpha = 3 is 0.4424425182737627

RMSLE for alpha = 3.5 is 0.44171501703551286

RMSLE for alpha = 4 is 0.44154616529145424

RMSLE for alpha = 4.5 is 0.4415286167861061

RMSLE for alpha = 5 is 0.44161632764828285

RMSLE for alpha = 6 is 0.4421832813533032

RMSLE for alpha = 7 is 0.44267468278758176

Training the model using best hyper-parameter and testing

Best alpha: 4.5 Train RMSLE: 0.383449898

Cross validation RMSLE: 0.441528616

SelectKBest: Selecting top 48k features from categorical and text features

The models that we would try going further take tremendous amount of time to train when the data is high dimensional. Therefore, I am selecting only top 48,000 features from text TF-IDF vectors and categorical one-hot encoded vectors.

I concatenate these with the numerical features, ridge model’s predictions(y_pred being used as feature)and use them with my models.

Train size: (1332967, 48049), CV size: (148108, 48049), Test size: (3460725, 48049)

6.2. SVM Regression

SVR is an advanced version of linear regression. It finds a hyperplane in a d-dimensional space that distinctly classifies the data points.

Similar to alpha in linear regression, C in SVR is found by hyper-parameter tuning.

Best C: 0.3

SVR(C=0.3, epsilon=0.1, gamma='scale',kernel='rbf', max_iter=200) Train RMSLE: 0.441563037

Cross validation RMSLE: 0.457637217

6.3. RandomForest Regression

Training RandomForest Regressor with higher values of n_estimators (N) was taking tremendous amount of time without giving any results. Due to this reason, we have trained it with less number of estimators. As you can guess, the results were not satisfactory.

RMSLE for N=10 is 0.487657647 elapsed time:0:46:21.136700

RMSLE for N=20 is 0.472606976 elapsed time:2:02:11.229545

6.4. LightGBM Regression

Light GBM is a fast, distributed, high-performance gradient boosting framework based on decision tree algorithm, used for ranking, classification and many other machine learning tasks.

Since it is based on decision tree algorithms, it splits the tree leaf-wise with the best fit whereas other boosting algorithms split the tree depth-wise. So when growing on the same leaf in Light GBM, the leaf-wise algorithm can reduce more loss than the depth-wise algorithm and hence results in much better accuracy. Also, it is surprisingly very fast, hence the word ‘Light’.

Hyperparameter tuning for LightGBM has been done using 3-fold cross-validation using RandomizedSearchCV.

{'colsample_bytree': 0.44583275285359114, 'learning_rate': 0.09997491581800289, 'max_depth': 12, 'min_child_weight': 1.7323522915498704, 'n_estimators': 1323, 'num_leaves': 123}

Train RMSLE: 0.319789825

Cross validation RMSLE: 0.424231390

6.5. Comparing the performance of models

The validation score of LGBM (RMSLE=0.42423) was the best out of all the above models.

I submitted the predictions of Ridge and LGBM to Kaggle. We got the best score with Ridge, RMSLE=0.45444

The score of LGBM (RMSLE=0.45785) was very close to that of Ridge.

7. Final Solution: Regression using Multi-Layered Perceptrons

The classical machine learning models seem to work decently well but we ought to have much better performance to get a good Kaggle score. Most of the existing approaches have employed some or the other deep learning models such as Convolutional Neural Networks(CNNs), Recurrent Neural Networks(RNNs) or a combination of both. The performance of deep learning models seems to be significantly better than classical ML models, which encouraged me to try a basic deep learning model, MLP.

7.1. Preprocessing

I have done the following processing on train and test data:

standard text pre-processing (stemming, fill NAs)

concatenating name , brand_name to one column, name.

, to one column, concatenating name, item_description, category_name to one column, text.

7.2. Vectorization

Tfidf 1-gram vectorization of name .

. Tfidf 1-gram, 2-gram vectorization of text .

. One-hot encoding of categorical features item_condition_id, shipping.

7.3. MLP model architecture

The reasons for choosing MLP over CNN or RNN are:

Fast to train, can aﬀord hidden size 256 instead of 32–64 for RNN or Conv1D.

MLP captures interactions between text and categorical features.

Huge variance gives a strong ensemble with a single model type.

I have trained 4 high variance models of exactly the same architecture and finally taken ensemble of these to get final predictions. This is similar to Bagging in RandomForest. For 2 out of 4 models I have binarized the input data by setting all non-zero values to 1. This is like getting an extra dataset with a binary CountVectorizer instead of TF-IDF.

I used Adam optimizer with learning rate 0.003 and an initial batch size of 512 and trained the model for 2 epochs, doubling the batch size at each epoch. For binarized input, I trained 3 epochs in the same fashion.

7.3. Evaluation of model performance and Kaggle submission

With the above model, I got a validation RMSLE=0.3848, which is a great improvement compared to all my previous models.

The final submission score on Kaggle with this model was 0.39446 in the private leaderboard.

Although the competition was closed long ago, placing this score on the leaderboard puts me at the 5th position (top 0.2%) in both private and public LB.

Improvements to existing approaches

I have made some changes in the MLP architecture as well as parameters such as learning rate and batch size to get better results. I have also changed the number of epochs from 3 to 2 for the model on non-binary data, as it starts overfitting from the 3rd epoch.

Instead of taking simple mean, I have taken a weighted average of predictions from 4 models/runs.

For simplicity of the code, and also because I have used Google Colab(training is faster with GPUs than with multi-core CPUs), I have trained the models one after another unlike pool processing in the original kernel.

The validation RMSLE I got was 0.3848 as compared to 0.3875 in the source kernel.

Things that did not work well

RandomForest was taking too much time to train and hence I had to discard this model.

In the MLP I also tried using dropouts (0.1, 0.2, 0.3, .. 0.5) but the models performed better without dropouts and hence removed them. (I got a validation RMSLE of 0.3872 with dropouts).

I also experimented with different activation units (‘tanh’, ‘sigmoid’, ‘linear’, ‘relu’). ‘relu’ performs significantly better than rest all.

Future Work

Using deep learning was productive and yielded a very good score on test data. More complex models such as LSTMs and Convolutional Neural Nets can be tried.

We can experiment with more complex MLPs by adding additional layers and larger number of units in hidden layers.

Other vectorization schemes such as Wordbatch can be experimented with ML models.

Regression models like FTRL and FM_FTRL can also be tried.

Conclusion

It was fun as well as a great learning experience doing this case study. I thank you for reading this blog and I hope this has added some value to you. I have included minimal code in this blog. For full code, please refer to the ipython notebooks in my GitHub repository Mercari-Price-Suggestion-Challenge.

I would love to hear your reactions, suggestions or queries.

You can connect with me on LinkedIn. Here’s my profile.

References