Recommender Systems: Item-Customer Collaborative Filtering | by Jesko Rehberg | Towards Data Science

Motivation:
Knowing what items our customers should be highly interested in is vital in many businesses. Recapping our FPGrowth data mining, we experienced that quite often products are related to each other. So if one customer buys product A that customer will most likely also purchase product B, because both articles are related (for whatever reason, maybe like Beer and Diapers). Because of this association rule, we can estimate customers to purchase more than just one product within one sales transaction. To add some extra flavor, would it not be awesome to be able to recommend items to our customers, which they do not yet know themselves that they are interested in (but they are)? Let’s take this next step and learn how to set up such a recommending system in Python.
Solution:
We will move out of the market basket (which was on sales transaction id level) into the more overall buying behaviour of our customers (which items our customers bought overall in the past), building a ML recommending system. Our aim is to recommend products to our customers, which they have not yet purchased but which they should be greatly interested in (without necessarily knowing about their desire by themselves yet). The trick here is to look at other similar customers’ shopping habits. Since we do not have any labeled data available which tells us which customers are similar to each other, we will extract similarity between customers simply out of their connection between the same items they bought in the past. In general, similarity is a very interesting metric for lots of analysing purposes, not only linked to item-customer recommendations (see e.g. similarity between words/ sentences in our Chatbot). Looking for similarity in the purchase history, we anticipate that the similar preferences customers had will also extend into the future. So, if customer A purchases products 2,5 and 6 (not necessarily within the same basket) and customer B purchases products 2, 5, 6 and 9 then we can estimate that customer A is also interested in product 9. Please note that it could be that product 9 is very different to the products customer A has ever purchased before, because we are looking at similarity between customers (not between items). ML algorithms that build on this concept are called collaborative filtering (CF) algorithms.
First let us have a look at our data and do the necessary data preparation (data and Jupyter Notebook both stored in my github repository):
import pandas as pddata = pd.read_excel('/content/gdrive/MyDrive/DDDDFolder/DDDD.xlsx')  data.head()
So what we have got is Sales per Sales Day, Customer (e.g. Customer code 0 is one specific customer) and Sales Item (similar to Customer, Sales Item 0 is one specific Sales Item). The other columns are not relevant for our product recommendation, so we will simply drop those:
DataPrep = data[[‘SalesItem’, ‘SalesAmount’, ‘Customer’]] DataPrep.head()
Using Panda’s .info function we see that no NaN (not a number) values exist within our 341422 data rows (which is a good thing):
Now we want to figure out what items have been purchased per customer, using groupby:
DataGrouped = DataPrep.groupby([‘Customer’,‘SalesItem’]).sum().reset_index() DataGrouped.head()
Our data, on which we will build our recommending engine on, has this shape: 36 unique customers (from 0 to 35) and 3751 distinct items. The interesting question is how many different items our customers actually bought? We will use Numpy’s .list function to get the SalesAmount per distinct number of Customers and Items:
import numpy as np customers = list(np.sort(DataGrouped.Customer.unique()))# a list of values, so customers now stores 36 unique customersproducts = list(DataGrouped.SalesItem.unique()) quantity = list(DataGrouped.SalesAmount)
Using Panda’s DataFrame we can converge Numpy’s list back into a Panda’s dataframe:
from pandas import DataFrame DfCustomerUnique = DataFrame(customers,columns=[‘Customer’]) DfCustomerUnique.head()
Now we can build our sparse matrix:
from scipy import sparse from pandas.api.types import CategoricalDtype rows = DataGrouped.Customer.astype(CategoricalDtype(categories=customers)).cat.codes # We have got 35 unique customers, which make up 13837 data rows (index) # Get the associated row indices cols = DataGrouped.SalesItem.astype(CategoricalDtype(categories= products)).cat.codes # We have got unique 3725 SalesItems, making up 13837 data rows (index) # Get the associated column indices #Compressed Sparse Row matrix PurchaseSparse = sparse.csr_matrix((quantity, (rows, cols)), shape=(len(customers), len(products))) #len of customers=35, len of products=3725 #csr_matrix((data, (row_ind, col_ind)), [shape=(M, N)]) #where data, row_ind and col_ind satisfy the relationship a[row_ind[k], col_ind[k]] = data[k]. , see [3]PurchaseSparse
We have 35 customers and 3725 items overall. For these user/item interactions, 13837 of these items actually had a purchase. In terms of sparsity of the matrix, that makes:
MatrixSize = PurchaseSparse.shape[0]*PurchaseSparse.shape[1] # 130375 possible interactions in the matrix (35 unique customers * 3725 unique SalesItems=130375)PurchaseAmount = len(PurchaseSparse.nonzero()[0]) # 13837 SalesItems interacted with;sparsity = 100*(1 - (PurchaseAmount/MatrixSize))sparsity
Since we will use Matrix Factorization for our collaborative filtering it should not be a problem that 89.3% of the interaction matrix is sparse. In plain English, 89,3% in our case means that only 10,7% of our customer-item interactions are already filled, meaning that most items have not been purchased by customers. It is said that collaborative filtering can even work well with even more sparse data. We can prove that it works when checking our decent recommendations in the end. Cosine Similarity is a good measure for sparse data, so we will stick to Cosine (instead of Pearson, Euclidean, Manhattan etc.).
Our Collaborative Filtering will be based on binary data (a set of just two values), which is an important special case of categorical data. For every dataset we will add a 1 as purchased. That means, that this customer has purchased this item, no matter how many the customer actually has purchased in the past. We decided to use this binary data approach for our recommending example. Another approach would be to use the SalesAmount and normalize it, in case you want to treat the Amount of SalesItems purchased as a kind of taste factor, meaning that someone who bought SalesItem x 100 times- while another Customer bought that same SalesItem x only 5 times- does not like it as much. I believe that very often in Sales Recommendations a binary approach makes more sense, but of course that really depends on your data.
def create_DataBinary(DataGrouped):DataBinary = DataGrouped.copy()DataBinary['PurchasedYes'] = 1return DataBinaryDataBinary = create_DataBinary(DataGrouped)DataBinary.head()
Finally, let’s get rid of the column SalesAmount:
purchase_data=DataBinary.drop(['SalesAmount'], axis=1)purchase_data.head()
Trying to conduct collaborative filtering we could now have a look at Yifan Hu, Yehuda Koren and Chris Volinskys`paper ([1]). It’s supposed that also Facebook and Spotify basically use the approach described in this paper. Have you read it? Have you understood it and know how to put this into Python code? Honestly, I can say yes to the first question, but definitely no to the second question. In case you`re answering similarly, I feel honored you to join me through all this step by step, for the sake of better understanding.
CF is one of the most widely researched and implemented recommendation algorithms (according to Aberger [2]). CF is a ML algorithm that collects preferences or taste information from many customers (collaborative, since it is combining collected information) and uses this information to make automatic predictions (filtering out less probable options to find the most probable prediction) about the interests of other customers. Of course, we will only recommend items which our customer has not yet purchased.
..but implicit is better than nothing. Using implicit, we cannot really judge if our customer liked a specific product or not. We can just tell that the customer did bought a specific item. And this information can furthermore be broken down to if the customer purchased a specific item at all (binary, yes or no) or how many times the customer bought this item (as a kind of taste factor: the more often the item was bought the more relevant it seems to be). We will recommend on sales quantity bought by customers on a binary level (purchased yes or no), because I believe that this is most relevant to most of us readers. I think that in real life very often we only have implicit feedback at our fingertips. Implicit feedback like amount of items purchased, clicks on a website, movies watched etc. is not an explicit feedback (“I like” etc). So it could be that even if we purchased a specific book, we do not really like it at all while reading it. Even though this is a disadvantage of implicit feedback very often we just do not have explicit customer feedback, so we have to stick to implicit feedback. Also if the amount of implicit feedback outweighs the amount of explicit feedback we might decide to build our recommending engine on implicit data (sometimes mass of data rules over less data even with better quality). We look for associations between customers, not between products. Therefore, collaborative filtering relies only on observed customer behavior to make recommendations — with no information about the customers or the items required.
In one sentence: our recommendation is based on the assumption that customers who purchased products in a similar quantity share one or more hidden preferences. Due to this shared latent or hidden features customers will likely purchase similar products.
How can we tell how similar one customer is compared to another customer? Or how close one item is to other items? To answer this question it is helpful to state that the approach of similarity is a complement to distance. While the distance between two points can be obtained using the Euclidean distance -by calculating the length of a straight line connecting two points- , similarity is about “how close” the points are. The similarity value is usually expressed as a number between 0 and 1, where 0 means a low similarity, and 1 a high similarity. Let’s break this down into simple Google Sheets/ Microsoft Excel formula (below sheet screenshots used Google Sheets, which you can also find in my Github repository):
imagine our 9 Customers have purchased these 7 Sales items as follows (1 means purchased, 0 means not purchased), then the “PurchaseHistory” Tab looks like this:
The normalized data will look like this in tab “PurchHistNorm”:
..because for eg. customer 1 we have to chekc the magnitude for the sake of normalizing:
So finally the item-item similarity matrix looks like this:
Certainly a product is always highest correlated with itself (Sales Item 1 with Sales Item 1 correlates perfectly scoring a 1). I hope this simple spreadsheet example helps you to get the basic idea behind setting up a Item-Item Similarity.
It is recommended to use Pearson if the data is subject to grade-inflation (different customers may be using different scales). If our data is dense (almost all attributes have non-zero values) and the magnitude of the attribute values is important, use distance measures such as Euclidean or Manhattan. But for our sparse univariate data example we will stick to Cosine Similarity.
Now we will build our Item-Customer Matrix. For better convenience we add I for Item as a prefix before every SalesItem. Otherwise we would only have customer and SalesItem Numbers, which can become a little bit puzzling:
purchase_data[‘SalesItem’] = ‘I’ + purchase_data[‘SalesItem’].astype(str) DfMatrix = pd.pivot_table(purchase_data, values=’PurchasedYes’, index=’Customer’, columns=’SalesItem’)DfMatrix.head()
Since we are only using 1 and 0, we do not need to think about normalization, which is one method of scaling. Normalization reduces the scale of the data to be in a range from 0 to 1:
Xnormalized = X−Xmin / (Xmax−Xmin)
Another form of scaling is standardization, which reduces the scale of the data to have a mean(μ) of 0 and a standard deviation(σ) of 1:
Xstandardized = X−μ / σBut talk is cheap, let`s check to see that even if we would normalize, the result is the same, of course:
DfMatrix=DfMatrix.fillna(0) #NaN values need to get replaced by 0, meaning they have not been purchased yet.DfMatrixNorm3 = (DfMatrix-DfMatrix.min())/(DfMatrix.max()-DfMatrix.min())DfMatrixNorm3.head()
We need to bring our pivot table into the desired format, via reset_index and rename_axis:
DfResetted = DfMatrix.reset_index().rename_axis(None, axis=1) DfResetted.head() #Now each row represents one customer`s buying behaviour: 1 means the customer has purchased, NaN the customer has not yet purchased it
DfMatrix.shape
df=DfResetted 
We need to replace the NaN values with a 0, because our function will not work on NaN values.Please note, that we are only checking if a specific customer bought a specific item, yes or no. That is called binary. If customer bought a specific item, that means 1. If not, then 0. Because of this binary problem there is no use in using any further scaling techniques.
df=df.fillna(0) df.head()
Now we create a dataframe which only includes Sales Items, while Customer is indexed instead.
DfSalesItem = df.drop(‘Customer’, 1) DfSalesItem.head()
So we can now calculate the Item based recommendation. We will normalize dataframe for comparising reasons:
import numpy as npDfSalesItemNorm = DfSalesItem/np.sqrt(np.square(DfSalesItem).sum(axis=0))   DfSalesItemNorm.head()
Calculating with .dot Vectors to compute Cosine Similarities:
ItemItemSim = DfSalesItemNorm.transpose().dot(DfSalesItemNorm) ItemItemSim.head()
Create a placeholder items for closest neighbours to an item
ItemNeighbours = pd.DataFrame(index=ItemItemSim.columns,columns=range(1,10))ItemNeighbours.head()
Loop through our similarity dataframe and fill in neighbouring item names:
for i in range(0,len(ItemItemSim.columns)): ItemNeighbours.iloc[i,:9] = ItemItemSim.iloc[0:,i].sort_values(ascending=False)[:9].index #we only have 9 items, so we can max recommend 9 items (itself included) ItemNeighbours.head()
ItemNeighbours.head().iloc[:11,1:9]#it needs to start at position 1, because position 0 is item itself
# You might want to export this result to Excel nowItemNeighbours.to_excel(“ExportItem-Item-data_neighbours.xlsx”)
Now we will create a customer based recommendation which we need our item similarity matrix for. Then we will have a look which items our customers have bought and get the top N neighbours for each item. Afterwards we calculate the purchase history of the customer for each neighbour and calculate a similarity score for them. So in the end we just have to recommend the items with the highest score. Therefore we will create a place holder matrix for similarities, and fill in the customer column:
CustItemSimilarity = pd.DataFrame(index=df.index,columns=df.columns) CustItemSimilarity.iloc[:,:1] = df.iloc[:,:1] CustItemSimilarity.head()
Finally we will calculate the similarity scores
def getScore(history, similarities):return sum(history*similarities)/sum(similarities)
This takes ages (35 customers * 3725 items), because we now loop through the rows and columns filling in empty spaces with similarity scores (find an improved solution here). Please note that we score items that the customer has already consumed as 0, because there is no point in recommending these again.
from timeit import default_timer as timer #to see how long the computation will take start = timer() for i in range(0,len(CustItemSimilarity.index)): for j in range(1,len(CustItemSimilarity.columns)): user = CustItemSimilarity.index[i] product = CustItemSimilarity.columns[j]if df.loc[i][j] == 1: CustItemSimilarity.loc[i][j] = 0 else: ItemTop = ItemNeighbours.loc[product][1:9] ItemTopSimilarity = ItemItemSim.loc[product].sort_values(ascending=False)[1:9] #here we will use the item dataframe, which we generated during item-item matrix CustomerPurchasings = DfSalesItem.loc[user,ItemTop] CustItemSimilarity.loc[i][j] = getScore(CustomerPurchasings,ItemTopSimilarity) end = timer() print(‘\nRuntime: %0.2fs’ % (end — start)) CustItemSimilarity.head()
Now we will generate a matrix of customer based recommendations as follows:
# Get the top SalesItems CustItemRecommend = pd.DataFrame(index=CustItemSimilarity.index, columns=[‘Customer’,’1',’2',’3',’4',’5',’6']) #Top 1,2..6 CustItemRecommend.head()
Instead of having the matrix filled with similarity scores we want to see the product names. Therefore loop:
for i in range(0,len(CustItemSimilarity.index)): CustItemRecommend.iloc[i,1:] = CustItemSimilarity.iloc[i,:].sort_values(ascending=False).iloc[1:7,].index.transpose() CustItemRecommend.head()
CustItemRecommend.to_excel(“ExportCustomer-Item-CustItemRecommend.xlsx”) 
Congratulations, we have coded a binary recommender engine, which works sufficient on a small data sets. Let us see if we can enhance the performance and scalability using the same dataset in another story. You can find the Jupyter Notebook and Excel example in my Github. Many thanks for reading, hope this was supportive! Any questions, please let me know. You can connect with me on LinkedIn or Twitter.
If you want to recommend using Scikit-Learn or Tensorflow Recommender you might find this post helpful:
towardsdatascience.com
Originally published on my website DAR-Analytics.
Reference:
[1] Yifan Hu, Yehuda Koren and Chris Volinskys, Collaborative Filtering for Implicit Feedback Datasets: http://yifanhu.net/PUB/cf.pdf
[2] Christopher R. Aberger, Recommender: An Analysis of Collaborative Filtering Techniques: https://www.semanticscholar.org/paper/Recommender-%3A-An-Analysis-of-Collaborative-Aberger-caberger/53bd928ceaab4572594590970b877da1278b7a79
[3] scipy.sparse.csr_matrix, https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html