How to Find Closed and Maximal Frequent Itemsets from FP-Growth | by Andrewngai | Towards Data Science

Python Optimization on finding Closed and Maximal Frequent Itemsets
In the last article, I have discussed in detail what is FP-growth, and how does it work to find frequent itemsets. Also, I demonstrated the python implementation from scratch. In this article, I would like to introduce two important concepts in Association Rule Mining, closed, and maximal frequent itemsets. In order to understand the concept, you need to have some basic knowledge of what is an FPtree and what is frequent itemsets. My last article covers all the basics.
Understand and Build FP-Growth Algorithm in Python
Here we quickly review the concept of frequent itemsets. An itemset is whose support is greater than or equal to a minsup(Minimum Support) threshold. Support is the frequency of occurrence of an itemset. For example, given a set of transactions T, we would like to find all itemsets that appear more than 2 times in all transactions. This can be viewed as finding all frequent itemsets with minsup ≥2.
Then what are closed and maximal frequent itemsets?
By definition, An itemset is maximal frequent if none of its immediate supersets is frequent. An itemset is closed if none of its immediate supersets has the same support as the itemset. Let’s use an example and diagram representation to better understand the concept.
Here we have transactions T with 5 transactions, let’s present all the itemsets hierarchy using a tree-type diagram and write down the transactions the itemset appears on top in red.
If we set the minsup to be 2, any itemsets that appear more than twice will be frequent itemsets. And among those frequent itemsets, we can find closed and maximal frequent itemsets by comparing their support(frequency of occurrence) to their supersets.
We can see the maximal itemsets are a subset of closed itemsets. Also, the maximal itemset works as a boundary, anything below the maximal sets is not frequent itemsets(any superset of maximal itemsets are not frequent).
There are many different approaches trying to efficiently find close and maximal frequent itemsets. And this is still a popular research problem in the data mining field. If you are interested, you can find many research articles on different algorithms to solve this problem. I will demonstrate a relatively simple approach to mine the closed and maximal frequent itemsets based on the frequent itemsets found by comparing each items’ support with their supersets. However, this approach can be quite time consuming, considering an O(n2) runtime complexity. To optimize the algorithm when dealing with large databases, we need to take advantage of a python dictionary. By storing all itemsets with the same support count into a dictionary, using support as the key, we can reduce the complexity to O(n). Because we do not need to compare every item since all supersets have ≤ support from their parents. And we only need to compare items with the same support count when finding closed itemsets. The same thing applies when finding maximal itemsets.
For implementation, I used the MLXtend library and fpgrowth function to compute the frequent itemsets first, and write my own function to mine the closed and maximal frequent itemsets from the result of the first step.
#Import all basic librayimport pandas as pdfrom mlxtend.preprocessing import TransactionEncoderimport timefrom mlxtend.frequent_patterns import fpgrowth#Task1 : Compute Frequent Item Set using  mlxtend.frequent_patternste = TransactionEncoder()te_ary = te.fit(dataset).transform(dataset)df = pd.DataFrame(te_ary, columns=te.columns_)start_time = time.time()frequent = fpgrowth(df, min_support=0.001, use_colnames=True)print('Time to find frequent itemset')print("--- %s seconds ---" % (time.time() - start_time))# Task 2&3: Find closed/max frequent itemset using frequent itemset found in task1su = frequent.support.unique()#all unique support count#Dictionay storing itemset with same support count keyfredic = {}for i in range(len(su)):    inset = list(frequent.loc[frequent.support ==su[i]]['itemsets'])    fredic[su[i]] = inset#Dictionay storing itemset with  support count <= keyfredic2 = {}for i in range(len(su)):    inset2 = list(frequent.loc[frequent.support<=su[i]]['itemsets'])    fredic2[su[i]] = inset2#Find Closed frequent itemsetstart_time = time.time()cl = []for index, row in frequent.iterrows():    isclose = True    cli = row['itemsets']    cls = row['support']    checkset = fredic[cls]    for i in checkset:        if (cli!=i):            if(frozenset.issubset(cli,i)):                isclose = False                break        if(isclose):        cl.append(row['itemsets'])print('Time to find Close frequent itemset')print("--- %s seconds ---" % (time.time() - start_time))      #Find Max frequent itemsetstart_time = time.time()ml = []for index, row in frequent.iterrows():    isclose = True    cli = row['itemsets']    cls = row['support']    checkset = fredic2[cls]    for i in checkset:        if (cli!=i):            if(frozenset.issubset(cli,i)):                isclose = False                break        if(isclose):        ml.append(row['itemsets'])print('Time to find Max frequent itemset')print("--- %s seconds ---" % (time.time() - start_time))
Thanks for reading and I am looking forward to hearing your questions and thoughts. If you want to learn more about Data Science and Cloud Computing, you can find me on Linkedin.