Network Recommendation for Small Businesses
By: Catharine Wu
Tags: Data Science, Machine Learning, Editors Pick, Programming

Network Recommendation for Small Businesses

Harvard Computational Science and Engineering Capstone Project, Fall 2020

Team members: Catharine Wu, Royce Yap, Zhenru Wang

Disclaimer: The views and opinions expressed in this blog are those of the authors and are not endorsed by Harvard or Alignable.

In partnership with Alignable

Introduction

With over 5 million members spread across more than 30,000 local communities, Alignable aims to help business owners and operators establish their online presence, grow their networks by finding peers with related professional interests, and build valuable relationships through referrals and recommendations.

To achieve this, we want to ensure that businesses on Alignable are able to grow their networks as quickly and effectively as possible. So the key question here is “How do we best recommend potential connections?”, in other words, “Who should connect with whom?” Our solution is to build a recommendation system that would generate ranked lists of businesses so that Alignable can send out referral emails accordingly.

Outline

In this blog, we will walk you through how we built this recommendation system and how effective it is. First, we will introduce the structure of a thoroughly anonymized dataset provided by our partner. Then we’ll take a look at cleaning and preparing the data for model input. Following this, we will explore and deploy two kinds of models on partial data and compare the results. The winner will finally be expanded to the entire dataset. We will also talk about how we evaluate the effectiveness of our model in real life.

Data

For this project, Alignable was able to provide a thoroughly anonymized dataset. It contained basic information for roughly 3 million businesses, including geographical location, business type and scale, and their activities on the Alignable platform.

Broadly, the types of data fell into two categories — the first pertaining to a single business profile, including its industry, community and users that are part of the business. The second category of data sheds light on the relationships between these businesses, in terms of whether they were connected, whether they had provided recommendations or business requests to one another.

Figure 1 below shows a brief description of these datasets and our decision on how to handle the data:

Figure 1 Data Description

Data Unification

We merge the datasets on business_id as the primary key. We generated a single table with useful features for connected businesses. Each row of this table represents one unique source-target business pair, and the columns are different attributes of business profiles and latest activities. For businesses with multiple industry/role values, we selected the ones with the highest confidence. Since we are focusing on recommendations on the business level, we also dropped data on individual users except for their latest sign-in information, as we believe the activity levels of businesses could be relevant in giving us a sense of how active this business may be on the Alignable platform and therefore, its likelihood of connecting with others.

Data Wrangling and Pre-processing

Certain variables had a large proportion of missing values: customer_type (68%), business_scale (66%), industry (46%), and role (74%). We decided to fill those ‘NaN’s with ‘Unknown’s and assign these variables with the lowest weight in the modeling process.

When pre-processing the data, one of the more significant variables we transformed were the industry and role variables. Here, we also developed a scoring system to assign a score of 0 to 1 to industry combinations within business pairs, based on existing connections.

Given that we had 300+ industry and role classes combined, this would help reduce the load of one-hot encoded variables on our model. At the same time, we were also able confirm our hypothesis that businesses within the same industry were more likely to connect with one another.

In terms of connection data on a national scale, we could hardly find any pair of businesses that share a mutual connection. In our data, 25% of businesses only have 1 connection, and ~75% of businesses have 8 connections or fewer. For the purpose of better efficiency and model performance, during the initial modeling process, we made a conscious decision to limit our initial trial to businesses with 10 or more connections. This is similar to the cold-start problem in the field of recommender systems, which we will go into detail in the next section.

Train — Test Split

After cleaning the dataset, now we can split it into training and testing sets. We opted for a temporal split. For each business, we used all the connections made before a timestamp as training, the rest as the test set. We have anonymized and normalized data on when connections were made. For a specific business, we sorted the connections by the time they were created. Then we treated the earlier 75% of the connections as training and the later 25% as the test set. For example, if Business 1 connected with Businesses 2, 3, 4, 5 at times 0.2, 0.3, 0.4, 0.5 respectively, then Business 2, 3, 4 would be the training, and Business 5 would be the test set. This way each business exists in both training and test sets.

Example of Train-Test Split

Modeling Strategies

Our strategy is fitting models on a community level, possibly a different model for each community. We started building our models with the biggest community in our dataset, LA (42,468 businesses). After we are done with model evaluation and have determined the best model, we then run the entire process on the final input table.

Data Preparation

To prepare the data for modeling, we converted boolean columns to integers (0–1), and one-hot-encoded categorical columns, such as types (e.g. B2B) and scales (e.g. Regional). All numerical features are normalized between 0 and 1 to avoid scaling issues.

As we only have data on the pairs that are already connected, we balanced the dataset by choosing some random target businesses that haven’t been connected for each source business. We added a new column indicating their connection status. In the end, each row of the input table is a unique source-target pair, the columns are details about the source and target businesses including type, scale, location, industry, role, online profile, latest activity, as well as their similarities and differences, such as physical distance and mutual connections. The output of the model is whether the pairs are connected, with 1 indicating a connection and 0 otherwise.

Predictive Models

We attempted multiple models that are known for binary classifications, regressions (linear, logistic), SVM, KNN, and trees (decision tree, random forest, boosted trees). In terms of feature importance, the regression models focused on business types and scales, as well as shared connections. Whereas the tree models paid more attention to business activities, e.g. when the businesses joined Alignable and when they were last signed in.

The metric for predictive models is the overall accuracy, and the results are as follows. We can see that Random Forest performed the best.

Predictive Models Accuracy Scores for LA

Collaborative Filtering Models

In parallel to predictive modeling, we explored collaborative filtering models, which is a commonly used recommender model type. Our method belongs to user-based collaborative filtering, namely, collecting existing preferences from other similar users. What is unique in this case is that users (who need recommendations) and items(things being recommended) are both businesses within the Alignable network.

To implement Collaborative Filtering for our dataset, we looked into different matrix factorization techniques, given how sparse the matrix is.

Singular Value Decomposition (SVD)

To handle the scalability and sparsity issue, we leveraged Singular Value Decomposition (SVD) to capture the similarity between users and items, turning the recommendation problem into an optimization problem.

To run the SVD algorithm, we employ the Surprise package, which estimates unknown ratings between certain user-item pairs by performing the minimization of the regularized squared error in the matrix by a standard stochastic gradient descent process.

After running the SVD algorithm, we are able to obtain ratings for each possible pair of businesses and a potential connection. These ratings can then be ranked and the top predictions being shown as potential connections for a single business.

Alternating Least Squares (ALS)

The second matrix factorization algorithm we tried was Alternating Least Squares (ALS). You can find an amazing introduction to ALS in this Blog.

Theoretically, ALS works really well at solving the scalability and sparseness of the rating data. It is implemented in Apache Spark ML, and we are using PySpark so that it runs in a Jupyter notebook. It’s simple and scales well to very large datasets.

Collaborative Filtering Results

The prediction of collaborative filtering models is in the form of a float between 0 and 1. The higher the prediction is, the more likely two businesses are connected. We chose to output numerical numbers, instead of further classifying them into 0 and 1. The goal is to use this prediction to make a ranked list of target business ids, that can be presented to each business as potential recommendations to connect with. We evaluate our models using the Root-Mean-Square-Error (RMSE).

Equation for RMSE

For SVD, using the Surprise package, we obtained a cross-validation RMSE of 0.26.

For ALS, after tuning the algorithm, the best model results in a validation RMSE of 0.28.

This validation RMSE score looks good considering the prediction values are on a scale of 0 to 1.

Model Comparison

To compare the performances of all the models mentioned above on the same scale, we decided to use the unordered Normalized Discounted Cumulative Gain (NDCG) as a metric. The equation is shown in the plot below. It is a really common measure of ranking quality. It is a number between 0 and 1 that represents the similarity of two lists or ranking: 1 being a perfect match and 0 means no match at all. We are using the unordered NDCG here because for those previously-established connections, it is hard to say whether newer or older connections are more important.

NDCG score [1]

We fit the best performing models from predictive models and recommendation models, Random Forest and SVD, on three communities, LA (~40,000 businesses), Midtown East in New York (~25,000), and Downtown Boston (~5,000). We can see from the table below that Random Forest out-performed SVD.

NDCG scores for best-performing models

Model Evaluation and Interpretation

Now that we’ve selected the final model. We fit it on all available data. One issue we encountered was the runtime. For each business we want to recommend, we need to rank all possible target businesses, which could lead to millions of data points for a single community. To mitigate that, we first cluster the target businesses using K-means, and only rank the target businesses that are in the same cluster as the source business. This method speeded up our process drastically and had little effect on the performance. After running the model on all communities, we achieved an 83% overall accuracy on average.

Overall Accuracy among all communities

To see which features contribute the most in the modeling process, we plotted feature importance. We can see that attributes such as when the businesses joined Alignable, time they were recently active, two businesses’ distance and mutual connections have bigger effects.

Feature Importance for Random Forest

We also looked at which communities the model performed the best and worst by inspecting the relationship between accuracy and community size, aka number of businesses per community. Here we show communities with the lowest and highest accuracies and their sizes, and we can see that the best performing communities with thousands of businesses, are significantly larger than the worst performing communities that have less than 50 businesses. It makes sense that our model performs better with larger communities as it has plenty of data to learn from.

Model performs better on larger communities

Final Results

The final deliverable we have are as follows:

A CSV file containing 5 ranked recommendations for each business. To be more specific, each row is a unique business, column 1 thru 5 are the first to 5th most likely recommendations. A single python file that produces this CSV, such that the client can reproduce the result or update it when new data comes in. Our modeling approaches and analysis.

Future Work

After all, the best way of evaluating a model is to deploy it in real-life situations. The CSV file we mentioned in the previous section is currently ready to use for A/B Testing and will be implemented on Alignable’s system.

Reference:

[1]Taifi, M. (2019, Nov 25). MRR vs MAP vs NDCG: Rank-Aware Evaluation Metrics And When To Use Them

Poster: https://drive.google.com/file/d/1EboDthdcQ4ofua9AMPyGM83OdtOw_PfU/view?usp=sharing

A brief video walk-through: https://youtu.be/8lYnGJo8iiY