Where Should You Buy Books Online?
By: Zain Raza
Tags: Supervised Learning, Machine Learning, Scikit Learn, Data Visualization, Data Science

Where Should You Buy Books Online?

Comparing Different Machine Learning Classification Algorithms in Scikit-Learn

Photo by Jonas Jacobsson on Unsplash

Flipkart and Amazon: A Tale of David vs. Goliath

I was inspired to write this after watching this business case study on Flipkart, an e-commerce startup which beat out Amazon as the top online bookseller in India. They were acquired for $18B USD in 2018 by Wal-Mart.

Given their massive success, I began to wonder: how did Flipkart do it? How did they beat out Amazon, a trillion dollar giant?

Presumably, the answer to how a startup wins over a larger company is one thing: differentiation.

How Does Machine Learning Relate to This?

To test the validity of this, I attempted to train a machine learning classifier to answer the following:

Given a book listed online for sale, can you predict which platform it was being sold on?

Learning Objectives

During the process of the experiment I learned a lot about the practical application of machine learning, and I intend to share those findings with you. By the end of this post, please expect to learn about the following:

Tradeoffs Between the Several Popular ML Algorithms, namely: K-Nearest Neighbors, Support Vector Machines, Logistic Regression, Decision Trees, and Random Forest How to Identify Overfitting in the Machine Learning Algorithms mentioned above Tradeoffs Between Standard Scaling and Minmax Scaling

While I have never ran a startup or worked professionally as a data scientist, I tried my hardest in this project to employ the best practices I am learning about at Make School, where I am in the Data Science/Machine Learning concentration. If you spot any errors in my process or have any feedback, I invite you to write a response to this post!

The Environment

Full list of imports from the Jupyter notebook.

With that said, let me first share how I actually conducted this experiment.

I used the following tools:

Python 3.7.6: programming language

Jupyter Notebook: executes Python commands in the browser.

All of the machine learning algorithms you will see are implemented using scikit-learn, a popular framework for implementing ML in Python.

The Dataset

The dataset, titled “Amazon vs. Flipkart Book Prices” on Kaggle, compares how 1,382 of the same books were sold differently between Amazon and Flipkart.

There are two CSV modules: one for each company to display the books they sold, with each sample containing the following features:

book title

author

ISBN number

price

number of reviews (also referred to as “review count”)

average 5-star rating (or more simply, “rating”)

As an example, here is what one section looked like on the “Flipkart” CSV:

A subsection of books listed sale online by Flipkart.

Before going any further, I want to give mandan special acknowledgement for originally posting this dataset online.

Exploratory Data Analysis

I decided to rely only upon price, review count and rating to train the machine learning algorithm, because the other features do not impact a customer’s decision to choose between Flipkart and Amazon. Therefore, these features were the focus of my data analysis.

Comparing Individual Features

Same Distribution, Very Little Variance

For each feature mentioned above, I overlaid histogram plots and created box plots for both classes, in order to easily spot the differences in the distribution across the classes.

At this stage I became very cynical that a high-performance model could be built, since for all three of the feature the classes shared a very similar distribution, and displayed low variance from one another.

For example, here is the histogram generated from plotting both Amazon and Flipkart’s distribution of book prices:

The y-axis shows the percentage of samples that a certain bin “contains” out of the whole dataset. The x-axis is the price of the book, in USD.

A similar story played out for the number of reviews, and the 5-star rating of the book on each platform. I left them out of here for the sake of brevity, but you may please explore the companion slides to see what they look like.

Comparing All Features Together

The next aspect of the dataset I took a look at was the distribution of samples of each class separately.

Since there are only three predictor variables (aka the features) being used, I decided to make 3D-scatter-plots for this data visualization.

Three-dimensional scatter plot of book price, review count, and average rating per Amazon and Flipkart.

As before there was not a lot of variance between features to observe. However the scatterplot on the right does give us a clue on how the classification may be formed: the books sampled from Flipkart are drastically lower than those on Amazon in variance.

Correlation Heat-map

To corroborate what was observed prior, I finally generated a correlation heatmap using seaborn. It again verified that one feature in particular carried a strong corrlelation to the other features, or to the label of the dataset.

Correlation heatmap of the Flipkart vs. Amazon books dataset.

This led me to conclude it was worthwhile to try building a model with all three features used as predictor variables.

Modification of the Dataset

I have to admit that when I began the data analysis, I had a fairly certain presumption that there would not be strong signs of variance or bias in the dataset, which are two of the most fundamental ingredients to a high-performing machine learning classifier. The marketplace of online bookselling is marked by vast competition to begin with, so it makes sense that prices are tight and reviews tend to look the same — there’s just so many titles to choose from, it’s hard to pin down what makes a book different from another with just cold hard numbers.

Fortunately this does not mean we throw up our hands here! As machine learning engineers there are a number of tricks we have at our disposal to help build a better classifier. Here are some of the things I tried out while running my machine learning models in this experiment, to give you an idea:

Removing outliers from the dataset, according to the interquartile range Normalizing the data to a Z-distribution (aka Standard Scaling), or to a range between 0–1 (aka Minmax Scaling) Applying Principal Component Analysis — I reduced the features down to two dimensions, so it would be easier to visually assess the performance of my models

Implementation of Machine Learning Classifiers

K-Nearest Neighbors

Building the Model

One of the first models I attempted to build was with K-Nearest Neighbors, an algorithm based upon the idea that a given sample from the dataset can be classified based upon comparing it to a certain number (represented by the variable k) of its most related samples.

In practice, I trained an instance of the KNNeighborsClassifier in scikit-learn on both versions of my data — all 3 of the features, as well as with just the principal components. In each of these two cases, I also tried training the model with data that had been standard scaled, and minmax scaled.

Results:

The KNN algorithm performed better than I thought possible, reaching a maximum F1-Score of approximately 0.88, with K=1. However due to the naive nature of this algorithm, we should not take this number at face value. There is most probably overfitting occurring.

Additional Commentary

To illustrate, I will provide more context: the model mentioned above was trained on all 3 features, so it is not feasible to visualize here. However, on the left let’s examine the next best KNN model I trained, which was built using 2 principal components of the dataset, and scaled using minmax scaling. It achieved an F1-Score of 0.865.

This model was also trained with K=1. Effectually, this means that the algorithm “learned” how to tell whether or not a book’s seller was Amazon or Flipkart based upon comparing only two samples to each other. Put another way, the model thought it had enough information to tell the two apart based upon a minute fraction of the dataset.

In machine learning, this shortsighted decision-making actually has a special name — overfitting, as I briefly mentioned above. Much like a car mechanic who thinks he knows how rockets fly — despite not knowing anything about aerospace engineering — sometimes we engineer our models to acquire learnings that are overly specialized to the data they’re trained on. Then, they misapply that information to make judgments on testing data, that is not comparable in anyway to the kinds of data, experiences, etc. that the model has seen before.

In a K-Nearest Neighbors model we can easily recognize overfitting, by taking a look at the “jaggedness” of the decision boundary, as seen in the above image. These tend to form when K values are lower, because the model is forced to make decisions looking at the variance and bias between a smaller number of samples — by definition, it overspecializes to those data samples while training. Therefore, a K-Nearest Neighbors model is not advisable to continue for this project, because it is hard to attain an accurate model without running into some overfitting.

Support Vector Machine

Next up I decided to take a go using a Support Vector Classifier model, which is abbreviated below as “SVC”.

Similar to KNN, the SVC model is based upon the idea that by looking at many samples, the model can learn what the classification relationships are by grouping samples that are the similar to each other, in the same class.

One stark contrast between SVC and KNN though, is in the training phase. The SVC takes much longer than the KNN to train, because it inherently needs to look through all the samples in the training data, before it has fully formed into classification function. On the other hand, the KNN could be set to train either longer or faster, based on how high we set the value of K. However, the training phase of the SVC is what allows it to be much more accurate than the KNN, given the same amount of computing power — because by looking at all the samples together, the model is better able to avoid overfitting!

Modifications to the Data

The best results with the Support Vector Classifier were achieved by training the model on two principal components of the dataset, which were both scaled down to a Z-distribution. I then compared results using three types of “kernels”: the linear kernel, the polynomial kernel, and the RBF (radial basis function) kernel. For those new to the term “kernel” (as I sure know I was when I first saw it), all it refers to the “type” of function the SVC is using — e.g. a linear classification boundary, a polynomial of some sort, etc. The RBF kernel refers to the model making circular classification boundaries, aka those that characterized as being “radial.”

Results

With the linear kernel, the model achieved about a 58% accuracy score, and a F1-Score of 0.6933.

With the linear kernel, the model achieved about a 58% accuracy score, and a F1-Score of 0.6933.

I believed this was the least overfitted model, because as you’ll see below the other kernels formed complex boundary lines — on the other hand, this boundary sensibly matches what a “linear” kernel should look like.

In the polynomial kernel, the model achieved roughly the same accuracy of 58%, yet F1-Score jumped up to 0.7154.

In the polynomial kernel, the model achieved roughly the same accuracy of 58%, yet F1-Score jumped up to 0.7154. Closer inspection of my Jupyter notebook shows this due to an increased Recall score (99% versus 91% for the linear kernel above), which mean the polynomial function had less false negative classifications for the “Amazon” label.

In the RBF kernel, the SVC model performed its best by achieving an accuracy of 65%, albeit achieving a slightly less F1-Score of 0.7126.

Due to the nature of the RBF kernel the decision boundary is able to utilize curves much more, in order to capture the distinction between classes. However it still doesn’t really resemble a circle, so I would argue this is still overfitting.

Additional Commentary

All in all when given the choice between an overfitted model and one that is not, we must choose the not-overfitted; therefore I inductively conclude that the Support Vector Machine performed better than the KNN model, and did the best job of doing so in its linear kernel.

Logistic Regression

Next up: what if we want a model that will not only classify our book samples, but provide us with the probability it thinks that classification is actually accurate?

This is the unique capability of the Logistic Regression model, so I decided to try that next!

Modifications to the Data

The Logistic Regression model performed the best when trained on two principal components of the dataset — I evaluated the model after being trained on principal components scaled using Standard Scaling and MinMax scaling as well, just to see how data normalization factored into the results.

Results

When the data was scaled to using MinMax scaling, the Logistic Regression model achieved an accuracy of 62% and an F1-Score of 0.6999.

When the data was scaled to using MinMax scaling, the Logistic Regression model achieved an accuracy of 62% and an F1-Score of 0.6999.

However as you can see to your left, the data points are all bunched up together. This is because MinMax scaling tried to shove all our samples into a distribution between 0–1.

Although the numerical results of this model was on par with that of the Support Vector Classifier, the use of MinMax scaling probably was not the best way to go about normalizing the data. This is because it was made it more difficult to decipher useful information from the graph.

In contrast, when the Logistic Regression model was trained on principal components of the dataset that had been standard scaled, much more of the variance was preserved. Therefore as you can see to the left, we can see the data much more clearly!

To add a cherry to the top, preserving the variance better means the model performs better: it matched the accuracy score of 62%, and obtained an F1-Score of 0.7034.

Decision Tree

All this talk of decision boundaries is fine and dandy, but are there any other ways we can construct a classifier model?

Of course!

One example of this kind of model is building a decision tree, a diagrammatic approach allows us to binarize our classification clearly between Amazon and Flipkart. It is commonly used in the business world to make decisions, since it is easy to follow, and from the hierarchy we will be able to tell which features of the data contribute most to the final decision!

Modifications to the Data

To be completely honest, this was the most challenging part of the whole experiment for me — and I am still looking for better solutions than the approach I ended up taking.

In order to train a Decision Tree classifier in scikit-learn, the features need to represent categorical data, unlike the continuous features of my dataset (which were the book price, its average 5-star rating, and number of reviews as previously mentioned).

Therefore I concluded in order for this process to work I needed to find a way to binarize the distribution of values for each of these features…

Results

To be honest, I will say I never really quite got this part to work. This is because as seen above, there was very little variance for each of these features to begin with. Therefore when I took the common approach of “bucketing” the values of a feature around a certain value, I was guaranteeing that some of the samples were going in the wrong bucket. More detail on this process can be found in my Jupyter notebook.

Despite problems in binarizing the data, using a DecisionTreeClassifier instance from scikit-learn proved to be fairly powerful. The model obtained achieved an accuracy of approximately 58%, and an F1-Score of 0.6962.

All the same, using a DecisionTreeClassifier instance from scikit-learn proved to be fairly powerful. The model obtained achieved an accuracy of approximately 58%, and an F1-Score of 0.6962.

The criterion for this tree was the entropy, or the amount of “uncertainty” that the model calculated for the classification, at each step of the decision-making process (which represents itself here as a depth-first search of the tree).

Looking at this tree, we can conclude the model ranked the review count to be the most telling feature of the dataset when it came to making a classification, since the model placed the review count as the root of the tree (seen at the top).

One more thing, for those of you interested in programming data visualizations like the tree diagram shown above: I implemented this using the pydotplus package. Image was saved using the export_graphviz function from the scikit-learn.tree module.

Random Forest

Last but not least, I would be remissed if I didn’t discuss one of the most powerful classification models: the Random Forest!

The Random Forest classification builds on top of the idea of the decision tree — in fact, all a Random Forest classifier really is a series of multiple decision trees, each of which is trained separately using randomly assigned samples of the dataset. Then each model, which you may also see referred to an “estimator”, makes its own classification about a given testing sample. Since each of the estimators was trained differently, they tend to predict different labels for the sample; and the way the overall model comes to a decision is by taking whichever class more of the models picked!

Explanation of how a Random Forest classifier works. I DO NOT claim ownership of this image. I am using it here only for educational purposes, as is allowed under Fair Use guidelines. Image Source: DataCamp.

Modifications to the Data

This time I didn’t even bother binarizing the data, since bucketing the samples seemed to be of no use when I tried doing it with the singular Decision Tree before. Instead, I made a gut decision to go with 100 estimators for the model, so that even if the models were all screwed up, the “voting” of the masses would produce the accurate result more often than not.

I used all 3 features of the dataset, so that at the end I could do feature ranking to see which contributed most to a final classification.

I used Standard Scaling in order to make the model train faster, reduce bias, and still maintain a lot of the variance.

Results

Fortunately the hypothesis worked out. Although the Random Forest results were sporadic (due to how the data samples are randomly assigned to the estimators), the model was able to achieve an accuracy in the ballpark of 89%, and an F1-Score of 0.9.

Through creating a bar chart of the feature importances, we can also visually rank which features contributed most to the model’s classification:

Like the single Decision Tree, the review count feature was ranked the highest by the model. However, for reasons I have yet been able to explain it ranked the price as the next most important feature, followed by the rating. This is different from the single Decision Tree, which ranked the rating as the second most important feature, and did not care for the price at all.

Like the single Decision Tree, the review count feature was ranked the highest by the model. However, for reasons I have yet been able to explain, it ranked the price as the next most important feature, followed by the rating. This is different from the single Decision Tree, which ranked the rating as the second most important feature, and did not care for the price at all.

Commentary

This is an example of what’s called an ensemble method in machine learning. Derived from the French word for “together”, an ensemble method like the Random Forest is different from the other models discussed so far because it depends on the “wisdom of the crowd”. The idea here is that many weak models working together, should be able to outperform one model working on its own.

Final Conclusion

In the future, my plans are to use one of these models in production. The question is, which one is the best of them all?

K-Fold Cross Validation

A commonly used technique for comparing models to each other is cross-validation. As we saw before with the Random Forest, there can often be drastic differences in how a model performs, based on the kind of training data it is fed (note this idea must only apply to supervised machine learning algorithms).

Therefore, the goal of using cross validation is to decouple the actual performance of the model (i.e. how robust the algorithm is, how well it avoids overfitting, etc.) from the kind of data it was trained on.

Image Source: Stack Abuse

The actual process of cross validation is essentially experimentation in and of itself.

We split up the overall slice of the training data into an arbitrary number of pieces. We build the model using all but one slice of the dataset, and evaluate it on the test data (which becomes whatever slice not used for training).

We can then evaluate the model formed each time as normal; and because we have a classifier, for us that means using metrics from the confusion matrix such as accuracy and F1-Score.

On a side note, the number slices used in this process is often referred to as the variable k; and each slice is actually called a “fold” formally speaking. Hence we have the term “K-Fold Cross Validation”.

So, Which is the Best Model of Them All?

At the end of cross-validating a model, we have a new dataset being the F1-Score/Accuracy/another evaluation metric which we took at each model evaluation (the size is the same as the value of K).

In my case, I decided to perform 5-Fold Cross Validation, each time taking the F1-Score of the model. At the end, I determined the best model would have the least variance in its F1-Score.

When comparing different ML models, we always want to look for which performs the most consistently (i.e. has the least variance) in cross validation.

Therefore for each model I calculated the standard deviation in the F1-Score from the mean of all 5 scores (which the model itself generated through cross validation), and then plotted all five models next to each other on a bar graph

The logistic regression has the least variance, so it wins!

What Differentiates Amazon and Flipkart?

Feature Ranking

One of the critical questions I had when beginning this was to answer how the machine learning model would differentiate books sold on Flipkart, versus those books sold on Amazon.

To recap, the feature ranking of the Random Forest classifier (show above) reveal the three features below were the most impactful, shown from greatest least influential:

Number of Reviews Price Average 5-Star Rating

This suggest the model used the number of reviews a book had received so far to gauge to determine which website it was being sold on. This does make sense, because one may argue that due to Amazon’s gargantuan size (estimated to be around 244 million active users according to The Motley Fool), they generate much more user feedback than Flipkart, which in 2018 was still just beginning to break into the 100’s of millions of active users (according to the Economic Times).

Note however, this is all simply my conjecture. I have not taken into account here whether or not one website’s has users that tend to review more. Also, I am presuming the year the data was taken is 2018, as it was first published on Kaggle in 2019— however the dataset currently doesn’t explicitly mention the data from when these books were inserted into the dataset.

Final Thoughts, Further Improvements

Overall, the best model built so far is a Logistic Regression classifier, wose accuracy was 62%. In other words, this model has a 62% of guessing the bookseller correctly for a given book.

But of course — the learning never ends! To improve this study I would try a few things, namely:

I would go back and see if adding books exclusively sold on either platform, would change the results of the model. For instance, Amazon and Flipkart may have different kinds of deals for certain books. Is it possible those deals could help a classification model? I would improve the performance of the Decision Tree and Random Forest classifiers, since they were the most promising models in these experiments. To deal with the problem of binarizing the data, I would probably look into using feature engineering, a technique which I didn’t even touch upon in this blog post at all!

Resources

There was a lot of material I went over in this article — so no need to fret if you didn’t get it all right away! I encourage you to look through the links below, to further your understanding.