Building a Recommender Engine Part I: Problem Statement & Collecting Data with Selenium
By: Rachel Koenig
Tags: Web Scraping, Selenium, Capstone, Data Science, Recommendation System

Problem

The world of retail is changing rapidly. Many brick and mortar locations are closing and being replaced by online stores, direct-to-consumer brands, and subscription/membership services. However, while the breadth of assortment is something that drives customers to a website, a lot of eCommerce platforms fail to sell through a high percentage of their merchandise. This is often due to poor user browsing experience. Customers can spend hours scrolling through hundreds, sometimes thousands of items of merchandise never finding an item they like. Shoppers need to be provided suggestions based on their likes and needs in order to create a better shopping environment that boosts sales and increases the time spent on a website.

This series will demonstrate a popular solution to this problem, recommender engines!

What is a recommender engine? you may ask. It is an unsupervised learning algorithm (one that does not have a target variable to measure accuracy against) mostly used to aid in consumer decision making. I’m sure you have seen them while online shopping. They also appear in places like streaming apps (aka Netflix and Hulu) to help you select a TV show or movie to watch next and on journalism/media websites like Medium to suggest other articles you may like to read, among many other uses. Obviously many e-retailers like Amazon have already been using recommender algorithms for quite some time, but many smaller or newer sites are still in need. There are different varieties of recommenders that base their predictions on different features. My goal for this project is to create one that is content-based, using elements of the products themselves, and one that is a collaborative item- and user-based using data from amazon reviews. I’ll look at cosine similarity, aiming for a score of 1 or pairwise distance, aiming for a score of 0, as metrics to measure success and check similarity between items and users.

Collecting Data

I started with 278,677 reviews of clothing, shoes, and jewelry from Amazon collected here. One review equals one row of data. After some simple EDA using Python Pandas, I determined there were 23033 unique ASINs (Amazon standard identification number, aka product id) in the dataset and turned them into a list.

Applying the unique( ) function to the end of a Pandas series will return each value in the column only once. Wrapping that code in the Python len( ) function will tell you how many values were returned. If the number matches the number of rows in your data, then you know each row is unique, if it is shorter, than you know that value is in multiple rows. In this case, it tells me each product has an average of 12 reviews.

The set( ) function also creates a non-repeating (unique) collection of items, but since it is unordered and immutable, I save the set as a variable and then turn it into a list with the list( ) function so that it can be sorted alphabetically. The ordered list becomes necessary for consistency in the future when I need to loop through and index the values numerous times.