Apriori algorithm implementation using optimized approach with pandas | by Karan Kharecha | Towards Data Science

In Big Data, this algorithm is the basic one that is used to find frequent items. Although apriori algorithm is quite slow as it deals with large number of subsets when itemset is big. With more items and less support counts of item, it takes really long to figure out frequent items. Hence, optimisation can be done in programming using few approaches. Each and every algorithm has space complexity and time complexity. To maintain balance between these two can sometimes be quite difficult. Here, the approach that is used for implementing this algorithm focuses on time complexity (i.e., frequent item sets should be returned as quickly as possible).
Firstly, let’s take the sample dataset that this algorithm will be targeting to:sampledata_numbers.csv
Before implementing the algorithm, pre-processing that is to be done in the dataset (not the one above), is assigning a number to each item name. In general explanation of apriori algorithm there is a dataset that shows name of the item.
For example:1 — Milk | 2 — Bread | 3 — Butter | 4 — Ice cream | 5 — Coffee | 6 — Tea
Sets before and after pre-processing would look like the following : {Milk, Bread, Butter} → {1, 2, 3}{Butter, Ice cream} →{3, 4}{Coffee, Tea, Butter, Bread} →{5, 6, 3, 2}{Milk, Bread, Tea, Ice cream} →{1, 2, 6, 4}
The reason behind assigning these numbers, is to speed up the process of discovering subsets. In DataBase Management System (DBMS), this kind of approach is known as normalization.
Now comes the part where coding is to be done in Python using following imports:
from itertools import combinationsfrom operator import itemgetterimport pandas as pd
A function named perform_apriori will take two inputs namely data and support_count:
table = pd.read_csv('../datasets/sampledata_numbers.csv')perform_apriori(data=table, support_count=500)
Before going further, one fact should be known about pandas library. For performing operations on datasets or any collection following sequence should be kept in mind:
→ vectorization → apply with lambda → itterrows → loops
Fastest way to get the job done is vectorization, followed by apply with lambda, itterows and lastly loops. If this algorithm is implemented using loops, it would take days to detect frequent items in large datasets.
Now to get all the single items that count more than or equal to the required support count are obtained by splitting them into different columns and applying sum function on axis 1 along with the filter that returns items that are more than or equal to the support count.
Above data frame is obtained when following line is executed
single_items = (data['items'].str.split(",", expand=True))
Final list of single items is obtained using apply and where methods:
single_items = (data['items'].str.split(",",expand=True))               .apply(pd.value_counts).sum(axis=1)               .where(lambda value: value > support_count).dropna()apriori_data = pd.DataFrame(               {'items': single_items.index.astype(int),                'support_count': single_items.values,               'set_size': 1})
The list that is obtained in single_items is then appended to the main data frame named apriori_data containing all the items/itemsets which count more than or equal to the support count. Following image shows the apriori_data data frame:
Here, this dataframe has three columns where set_size will be used further and even after getting frequent item sets. The support count used here is 500 and hence all the single items are found here initially.
Now comes the step where subsets are to be generated. For doing so, new column named set_size is appended to the existing dataset.
data['set_size'] = data['items'].str.count(",") + 1
In the dataset there are thousands of records, each record has variable size of item set. This size is denoted by the column set_size. Next step is to change the data type of the main column named ‘items’ of the data, data type should be a ‘set of integers’. Following line of code changes the data type:
data['items'] = data['items']                .apply(lambda row: set(map(int, row.split(","))))
To generate subsets, we need a parent set with all the unique items.
single_items_set = set(single_items.index.astype(int))
As the list of all single items is obtained, a loop should be iterated with 2 as initial value:
for length in range(2, len(single_items_set) + 1):    data = data[data['set_size'] >= length]    d = data['items'] \      .apply(lambda st: pd.Series(s if set(s).issubset(st) else None                for s in combinations(single_items_set, length))) \      .apply(lambda col: [col.dropna().unique()[0], col.count()] if                    col.count() >= support_count else None).dropna()    if d.empty:       break        apriori_data = apriori_data.append(pd.DataFrame(                 {'items': list(map(itemgetter(0), d.values)),                 'support_count': list(map(itemgetter(1),d.values)),                 'set_size': length}), ignore_index=True)    return apriori_data
Now, the subsets of length 2 will be generated in the first iteration of the loop. So, records whose set_size is less than 2 will be removed in the first line. Secondly, a method named combinations is used that returns all the subsets of size 2, apply method returns a series for each record of the dataset. This series contains a generated subset s if it is a subset of the item that is in the record, else None. Following image shows the intermediate dataframe of that series.
In the above dataframe, there are total 171 columns, that denotes they were total 171 possible subsets. First column is for subset (32, 33), second column is for subset (32, 2), and so on... Next apply method is in the chain of the first one, this method filter out the records by performing count on all the columns. If the count is ≥ the support count, it is appended to the list.
This image shows the pandas’ Series with list of items (with size 2) and it’s support count. This series is assigned to the variable named d. As this series isn’t empty it will be appended to the apriori_data dataframe. An empty series would denote that, for particular size of the subset/itemset, there are no items whose counts are greater than the support_count and so the loop will break.
Further in the next iteration of the loop, subsets with size 3 will be generated and checked with all the items after filtering the dataset (i.e., first statement of the loop). Loop will continue to iterate till an empty Series is found.
At last, apriori_data dataframe will be returned that will look like the below image.
Following optimization approaches are used here:
Always deal with numeric values rather than string values.Check only those records that contains number of items greater than or equal to the size of subsets that are going to be generated.Use a column set_size to determine itemset’s sizeUse apply with lambda whenever possible, here in this program subsets are checked using this feature only to speed up the execution.
Always deal with numeric values rather than string values.
Check only those records that contains number of items greater than or equal to the size of subsets that are going to be generated.
Use a column set_size to determine itemset’s size
Use apply with lambda whenever possible, here in this program subsets are checked using this feature only to speed up the execution.
NOTE: This program is implemented to get output faster but on the other hand it requires too much memory. Dataframe showed in (Fig. 4) is responsible for it. Initially, there were only 171 columns but it may increase up to thousands, because when subsets with size 3, 4, 5,... are returned by ‘combinations’ method, the size of dataframe starts increasing. It happens when the support count is LOW.
Solution: Rather than running this script on local machine, it is advisable to run it on Google Colab or other cloud Jupyter notebooks.
Link to the entire program: apriori.py