Big Data Market Basket Analysis with Apriori Algorithm on Spark | by Sergen Cansiz | Towards Data Science

The probability of buying a particular product by customers may be based on several factors. Factors such as the customer’s transaction history, demographic characteristics, and interests are examples of these. So, what if we don’t have those factors. It means that the customer has not been to our store before and we don’t have his/her data (not even gender). In such cases, there are methods that we can find the products which customer might buy. Usually, the data used in these methods belong to the other customers who made purchases before in the store. These methods take all transactions, control each item (product) in the transactions, and find the patterns that show which products are mostly bought in the same basket by customers. The methods that find these patterns are called Frequent Pattern Mining or Association Rule Mining [1].
Apriori is one of the algorithms that is used for frequent pattern mining. In this article, I am going to explain how to apply the “Apriori Algorithm” with Spark in Python. It is important to know how the “Apriori Algorithm” works before you start applying the codes below. I am also going to explain the basics of Apriori but if you want to learn it in detail please read my previous article called “What is “Apriori Algorithm” and How It Works”. In this article, will see the following parts,
An example case to apply Apriori
What is Spark and why we should use it
How to prepare transaction data
Apriori terms and basics
Applying Apriori from scratch with PySpark
Short-Cut: You can find the source code of implementation of Apriori algorithm on python PySpark in my GitHub repository and you can use the module created for Spark.
github.com
Suppose there is a greengrocer in New York city. This greengrocer is the biggest greengrocer in New York City and it has millions of customers in a month. The owner of the greengrocer buys tons of products just for one day. However, some days she can’t sell all the products and throws them away because the products’ lifetime expires. So, throwing away products causes lots of loss. Therefore, the owner of the greengrocer decided to innovate their marketing strategy. She wanted to hire a Data Scientist for her Greengrocer. This Data Scientist’s duty is to create an algorithm that can find the products that customers might buy after they get the products into their basket. She was going to offer a discount on the products which the algorithm founds for every particular customer. Thanks to this algorithm, she plans to sell her products just in time. Great idea!Now, think that the data scientist that she hired is you, how would have you started?
Apache Spark [2] is an open-source analytics engine that focuses on speed, ease in use, and distributed system. It has the capability to run machine learning algorithms 100 times faster than undistributed systems [3]. If there is a massive data set Spark would have been the best option to analyze that data set. Okay, let’s go back to our example: You will have to deal with millions of transaction data by considering that the Greengrocer has millions of customers in a month. In order to provide faster data analysis, one of the best options might be Spark.
First, you should do is preparing a data set over the customers’ transactions. But, you shouldn’t forget that one customer might buy one or more than one item from the greengrocer, it means our data-set can’t have a certain number of columns for each row. Therefore, I decided to write transactions line by line and separate each item in the transaction by comma and save as a “txt or CSV” file. Each row represents each transaction with comma-separated items. For example, at the first transaction customer had bought Apple, Mango, and Banana and at the second transaction, the customer had bought Banana and Mango. If you have data-set as CSV you can also use that (It is already comma-separated, but do not forget to remove the header line).
Before starting to implement Apriori on Spark we need to understand the main terms and concepts of it. Apriori algorithm depends on the frequencies of the item set. It creates different tables that include combinations of items. It scans the main dataset that shows all transactions and finds frequencies by considering how many time these combinations occurs in main data-set [4]. These frequencies are called support values in the Apriori algorithm. Also, the number of tables varies according to the maximum item set length. If the maximum item-set length is 5, it means there will be 5 support value tables. However, there is min support value that is used for deciding which item-set should stay in the table. Let’s say min support values have been decided as 2. So, If any item-set occurs in the main data-set less than 2 times, we should remove that item-set from the table.
Let’s inspect the following figure in order to understand how Apriori works. I took a few transaction records in the following example figure in order to make it simple and understandable.
“Dataset” represents our transaction data and each row in the “Dataset” shows each transaction item-set that has been bought at the same time by a customer. There are single item frequencies in Table A. This is the first table that we need to create for the Apriori algorithm. After Table A, there is Table B which includes binary combinations of single items. The most important part is here that the order of items doesn’t make sense. So, “Apple, Mango” and “Mango, Apple” is the same thing. Therefore, while generating combinations of items, replications of item-sets have to be removed. As you can see from Table B, the item-set of “Banana-Coconut” and “Mango-Coconut” whose support values less than 2 has been removed. However, there are triple combinations of single items in Table C and support values that show how many times occur in the main data-set. Item-sets whose support values less than 2 also removed from this table. If you see Table D there is only one item-set whose support value is 0 and it has four single items. This table shouldn’t be taken into account while applying the algorithm. Because there isn’t such a transaction in the main data-set.
So, how can we calculate the probability of buying Mango if a customer bought apple by using these support tables in the above figure? In order to calculate that probability, we need to take the support value of “ Mango ” and “Mango, Apple” together. The support value of “Mango” is 4 and “Mango, Apple” is 3. We can calculate the probability; (3 / 4) * 100 = %75. This probability is called Confidence Value in Apriori. The interpretation of this value is Customers who buy Mango might buy Apple with a %75 confidence value.
Now, we know how to calculate support and confidence value. We are clear to go into the Apriori algorithm with PySpark. In the following section we will see;
Reading data with PySpark
Parsing data to Spark RDD objects
Finding first support values of items with “MapReduce”
Deciding minimum support value
Creating the following support tables with “MapReduce”
Calculating confidence values
Deciding which product customer buys with high confidence
PySpark works on RDDs, it means that we need to convert all transactions records to multiple RDDs. Before doing that we need to be sure SparkContext is created in order to define each record of RDDs and read the CSV file.
Before you start this section please make sure Spark and PySpark are installed on your system.
First, create a python file in your working directory called “apprioriSpark.py” (or any name you want), and then you can easily create a SparkContext by the command below (You should also import SparkContext from PySpark).
After SparkContext has been created we can read the data by using the textFile() method that comes in SparkContext. Please make sure that your data file is in the same directory as your python file.
The default textFile() method reads file line by line, it means every line in our CSV file will be a value in RDD. These RDDs include CSV lines as a single string value (Result at 6th line). That’s why we need to map into every RDD and split these single values by comma in order to obtain every item in lines and split them into an RDD array. In order to do that run;
Now, you can clearly see that every line in the transaction data-set is separated by a comma. And every array in our RDD represents the item lists those are customers’ transactions. We successfully red the file and converted it to a Spark RDD object by considering each transaction that belongs to customers.
Apriori algorithm depends on the frequencies of items. Because of that first, we need to obtain frequencies for every single item. These frequencies will be our first support values in table one (as mentioned in the previous section). In order to that, we need to extract every item in RDDs to whole items of the array. We can do that by using the “flatMap” method.
As you can see from the result; All our transaction items are in a single array. Now we can calculate each unique item’s frequency. Do not forget that these frequencies will be our first supports values. If we would be working on a “NumPy” array, finding frequencies would be easy. But, we work on RDDs and that’s why we need to find a way to obtain frequencies by considering the “MapReduce” approach. Solution; First we can convert every item to a “tuple” object and add “1” as a second item of “tuple”. We can sum these values by using the “reduceByKey” (It is like the groupby method in SQL) method. By summing tuple’s second numbers we can get every unique item’s frequency (how many time occurs on customers’ transactions). We will also need to list unique items in the feature sections. So, we can also obtain unique items by using the “distinct” method.
If you run “supportRdd.collect()” after the above code snippet, you get the first item -support tuples (assume as Table A). As you can see from support values, “Apple” has more frequency than the others. It means the probability of “Apple” purchases is greater than the others. These support values are gotten by considering each item separately. We also need to consider how they occur together in transactions. Thanks to that we can calculate confidence values over the Apriori algorithm. We will see these steps in the next sessions.
In order to decide which item-sets will stay in the support tables, we need to define a min support value. We can choose min support value as the minimum frequency that is in the first support values array (table). If any support values in item set arrays are less than min support value, we should remove that item set from that array. If our data hasn’t got many records, min support might be 1. In such cases, we can define min support as 2 or any value more than 1.
In this example, we have a minimum support value that has been found as 1. To get a more consistent result we can set min support as 2. It means that if any item or item set occurs in a transaction less than 2 times we won’t take it into account. As you can see from line 8 we filtered our first item set table (which is single item support values) according to min support value. We also created a “baseRdd” object. “baseRdd” represents our first support values for every single item. This object will be updated with upcoming support values of combinations. We also need to define “supportRdd” that shows only items without support values. We will use it in the next sections in order to create combinations of items.
We only found each item’s support value (frequencies) until this section. Now, we will create an algorithm that generates combinations of items in a while loop. There will be different support tables that will be created in every loop. This while loop will end when there isn’t any combination whose support value more than min support value. This algorithm will control how many times these combinations of items occur together in transaction data-set in every loop and save it in an RDD.
We already created a “supportRdd” object which only includes the first table’s item set without support values (which only have one item in each row). Now, we will use this RDD in a while loop to combine it with unique items in order to create other support tables. However, this RDD will be updated after every loop. For example;
# Fitst supportRDD([Apple] , [Mango] , [Banana] , [Grapes])# After first loop = supportRDD ([Apple,Mango],[Mango,Banana],[Apple,Banana],[Apple,Grapes] ......)# After second loop = supportRDD([Apple,Mango,Grapes] , [Apple,Banana,Grapes] ........ )
The algorithm will filter every combination table according to min support value. When there is no item set while loop will end. Besides, we also need to define a function that can find replications inside the combined item set. As mentioned before; sets of (Apple, Mango) and (Mango, Apple) are the same thing for the Apriori algorithm. Because of that, we need to find such patterns and remove one of them. As you can see from the below code snippet, there is a function called “removeReplica”. This function removes such duplicated items after a combination and returns only one of them.
Yeap! We are ready to create our while loop. First, we need to define a variable that we can control item-set length in each loop. It is represented as “c” in the above code snippet. This “c” variable starts from 2. Why? Remember that we already created our first support table as “supportRdd”. So, in the while loop support table will start from c=2. It means the first support table that is going to be created in while loop will have item-sets which has 2 items. In order to create combinations of items, we can use the “cartesian” function that comes with PySpark (at line 6). It is created we will remove duplicated items (at line 7). We will use the “c” variable in order to filter combined items more than the “c” variable (at line 9). However, we also use the “distinct” method to get a unique item-set just in case.
As you can see there are two objects created in each loop as “combined” and “combined_2”. Combined already explained above. “Combined_2”, on the other hand, is a combination of the “combined” variable and each line of the whole data set (each transaction). As you can see from line 14, there is a filtering process that controls each item-set whether item-sets of “combined” are in the data-set. If there is not any such item-set in the data-set it removes that item-set. After all, we also get frequencies by using the “reduceByKey” method and filter them according to min support value. At line 21, we add the final support table to “baseRDD” which has our all support values with item-sets (in theory it is called table). However, we also need to update the “supportRdd” variable with “combined_2” without support values. This process will go on until there isn’t left any item-set in “supportRdd”.
Finally, we can calculate the confidence values. You can use the below codes for calculating confidence values for each combination of baseRdd (whole item-set combinations). There is a “Filter” class that can filter data according to confidence calculation. It also includes the method as “calculateConfidence” that calculates confidences.
If you run baseRddConfidence.collect() you can get all confidence values. You can also filter the result which is greater than certain confidence. A few examples from the result is shown below;
[[['Apple'], ['Mango'], 58.333333333333336],  [['Mango'], ['Apple'], 70.0],  [['Apple'], ['Banana'], 41.66666666666667],  [['Apple'], ['Mango', 'Banana'], 33.33333333333333],  [['Mango', 'Apple'], ['Banana'], 57.14285714285714],  [['Apple', 'Banana'], ['Mango'], 80.0],  [['Mango', 'Banana'], ['Apple'], 66.66666666666666],  [['Mango', 'Apple'], ['Raspberry'], 28.57142857142857 [['Raspberry', 'Apple'], ['Mango'], 50.0],  [['Mango', 'Raspberry'], ['Apple'], 50.0],  [['Mango', 'Banana'], ['Raspberry'], 33.33333333333333],  [['Mango', 'Raspberry'], ['Banana'], 50.0],  [['Raspberry', 'Banana'], ['Mango'], 50.0]]
The first item-set in arrays shows the products that customers bought and the second shows customers might buy if they bought products in the first item-set. The last element of the array shows the confidence value for that pattern. For example, a customer who buys Mango might buy Apple with %58 confidence. Another example; customer who buys Mango and Banana might buy Apple with 66.6% confidence. If you look carefully at the first two arrays [“Mango”, “Apple” ] and [“Apple”, “Mango”] have different confidence values. Let’s write the support formula for [“Mango” => “Apple”] (confidence of buying Apple after Mango)
( support(["Mango" , "Apple"]) / support(["Mango"]) ) * 100 = 70
and formula of [“Apple” => “Mango”]
( support(["Mango" , "Apple"]) / support(["Apple"]) ) * 100 = 58
Support of [“Mango”, “Apple”] is 7, support of [“Mango”] is 10, and support of [“Apple”] is 12. Although the frequency (support) of [“Apple” “Mango”] is the same in both calculations, they have different confidence values ​​because the frequency of “Apple” and the frequency of “Mango” are different.
You can also convert the result pandas data-frame by using the below codes;
In this article, we learned what is Frequent Pattern Mining and how to apply it over the Apriori algorithm. We took an example case with a data-set and applied the Apriori algorithm on PySpark from scratch. There are also other FPM methods that you can check out such as FPGrowth, Eclat, and so on. After understanding Apriori you can easily understand how other methods work. If you have any questions please feel free to ask.
Hope it was helpful...
[1] Frequent Pattern Mining https://en.wikipedia.org/wiki/Frequent_pattern_discovery
[2] Apache Sparkhttps://spark.apache.org/
[3] What is Sparkhttps://databricks.com/spark/about
[4] Apriori Algorithm https://en.wikipedia.org/wiki/Apriori_algorithm