Market Basket Analysis — Multiple Support Frequent Item set Mining
By: Srinivas Paturu
Tags: Data Science, Algorithms, Association Rule, Apriori

Market Basket Analysis — Multiple Support Frequent Item set Mining

An improvement over the default MSApriori algorithm for mining itemsets with multiple supports.

Pic credit: Upgrad

Introduction:

The problem of generating Association Rules from a database of transactions (market baskets) is of interest for many retailers. An association rule is of the form 𝑋 → 𝑌 where X, Y are subsets of I (the universal set of all products (or items) and 𝑋 ∩ 𝑌 = ∅.

One can understand the meaning of above association rule as, if the customer has purchased items in the set X then he is likely to purchase items in the set Y. Some well-known examples of association rules can be {Milk} -> {Bread}, {Milk} -> {Bread, Eggs}, {Bread, Butter} -> {Jam} and the humorous {Diapers} -> {Beer}. Please note that association rules are not commutative, .i.e. 𝑋 → 𝑌 does not equal 𝑌 → 𝑋.

The problem of finding association rules for a given transaction database (market baskets) is defined as follows:

a. Given a transaction database of size Z with N distinct items and an input support and confidence, find all the rules 𝑋 → 𝑌 that satisfy the given support and confidence constraints.

b. A support is a threshold that would determine if the items in X are frequent enough to be considered for association rule generation. More specifically if {X}.count / Z >= support, then X is considered a frequent item set.

c. A confidence is a threshold that would determine the conditional probability of buying Y. More specifically if {𝑋⋃𝑌}.count / {X}.count >= confidence, then 𝑋 → 𝑌 is considered a valid association rule.

Suppose if there are ’N’ distinct items/products sold by the retailer, then a brute force approach of counting and generating association rules has the complexity of O(N!).

Apriori algorithm is an improvement over the brute force method based on the observations that a set of items X is a frequent item set only if all the proper subsets of X are also frequent item sets. This would reduce the no of states/rules need to explore for generating all possible association rules. The Apriori algorithm works by this principle and is executed in two steps

a. Find all the frequent item sets in the transaction database of size 1, 2, 3…k.

b. Generate all valid association rules from the frequent item sets 2, 3, 4…k.

c. Stop if no further frequent item sets of size k + 1 can be generated.

The Apriori algorithm requires at least ‘k’ scans of the transaction database.

MSApriori:

MSApriori, which stands for Multiple Support Apriori, is a more constrained definition of the Apriori algorithm for real life scenarios. The Apriori algorithm considers only one support value for all the items in the transactions, irrespective of whether an item is a frequently selling item (such as groceries) or a less frequently selling item (high priced items such as Home decors). So, to establish a weighing scheme for the frequently/rarely sold items, each item is given an individual support. The problem definition for MSApriori algorithm is modified as follows:

a. Given a transaction database and different minimum input support’s (MIS) for each item and confidence, find all the rules 𝑋 → 𝑌 that satisfy the given support and confidence constraints.

b. A support is a threshold that would determine if the items in X are frequent enough to be considered for association rule generation. Because there are different supports for different items in MSApriori, any item set that satisfies the minimum MIS of items in its set is considered frequent. More specifically if

a. |X| = 1,

{X}.count / n >= MIS(X), then X is considered a frequent item set.

b. |X| >= 1,

{X}.count / n >= MIN (MIS(X1), MIS(X2), MIS(X3) … MIS(Xk)); Xi ∈ X, i=1 to k.

c. To discourage very frequently and less frequently items appearing together in any frequent item set, a support difference constraint Φ is enforced on the candidate item set X

a. |X| >= 1,

MAX(Supp(X1), Supp(X2) … Supp(Xk))–MIN(Supp(X1), Supp(X2) … Supp(Xk)) <= Φ

The task of assigning MIS to items must be done in a meticulous manner. For highly moving items such as daily groceries, a high value of MIS is desired. For not so frequently sold items such as high end electronics, a less value of MIS is desired. One such assignment of MIS can be done using the following approximation.

MIS (item) = δ * Supp (item); δ ∈ [0, 1]

The above assignment is discussed in WebMining by Bing Liu et al. An algorithm for solving MSApriori is also proposed in the above book by Bing Liu, let’s call it the default MSApriori algorithm. The default MSApriori algorithm does a scan of transaction database for each candidate item set Ck; k >=2 to compute the frequent item set Fk.

Considering there are ‘Z’ number of total transactions in the database and ’N’ unique items, the default MSApriori algorithm requires calculation of support for all C2; C2 would have N*(N-1)/2 item sets if the MIS of each item is defined as above equation.

Total time complexity = N*(N-1)/2 * Z ~ O(n3).

Similarly, C3, C4… Ck would have corresponding complexities based on the size of item set multiplied by the number of transactions in the database.

Proposed improvement to MSApriori:

The proposed improvement to the default MSApriori is to use a look forward strategy of computing the supports of each Ck in the first pass of the transaction database scan, storing them in a Hash Table (Hk) and retrieving them when required. In the first pass of the transaction database scan, for each transaction of length ‘L’, each Ck (k <= L) is generated locally and the count of its support increased in Hk. So, for C2, the proposed modification would work as follows,

/* modified MSApriori Algorithm form Bing et al */ Modificaiton-1: init-pass() For each item(i) in Transaction T: a. Compute/increment the support of item(i) b. For each item(j) in Transaction T: // (j>i)

Compute the hashcode of itemset(i,j)

Retrieve the support of itemset(i,j) from the hash table(H2)

Initialize/Increment the support of itemset(i,j). Modificaiton-2: CandidateGen2() //Candidate generation function for 2-itemset For each item(i) in list L/C2:

For each item(j) in list L: // (j>i)

Compute the hashcode of itemset(i,j)

Retrieve the support of itemset(i,j) from the hash table(H2)

If support > MIS(item(i)) // and other constrains like Φ

Return itemset(i,j) into F2.

NOTE: CandidateGen2() directly returns the items into F2, no need for scanning the transaction database again!

In the Modificaiton-1, an extra sub-loop is added for each transaction to compute the support for 2-itemsets, assuming the average length of transaction is ‘L’ then this step would add further processing time of L*(L-1)/2 for each transaction in the database. Assuming, there are ‘Z’ transactions in the database, the time required to complete the init-pass() is

Time complexity (init-pass) = L*(L-1)/2 * Z

~= c * Z ~ O(Z) //for all practical reasons L << Z

In the Modificaiton-2, I am adding an extra step for accessing the hash table (H2) which can be done in constant time (TH) in most library implementations. The time required to complete CandidateGen2() is given as

Time complexity = N*(N-1)/2*TH ~ O(n2).

Combining both modifications, total time complexity = O(Z) + O(n2) < O(n3)! (from the default algorithm).

Verification of the result:

The default MSApirioi algorithm is implemented by Phillipe et al in the open-sourced java data mining library SPMF. I have modified the default MSApriori algorithm in SPMF library with my proposed alteration in the program MSAprori_H.java. The results of the execution of the MSApriori and MSApriori_H is as follows:

Dataset considered: retail1.txt

No of distinct items in the dataset: N = 2603.

No of transactions in the dataset: Z = 541909.

A note on the Hash function requirements for this algorithm. We would ideally require a hash function that would give a unique hash value for a set of integers, irrespective of the order of the integers in the given set. Generating such a hash function is definitely not a simple task, so I have relaxed the hash value to be a Java object in my implementation. I have chosen a BitSet object to implement in MSApriori_H.java. Consider a java BitSet object as a long array of Booleans that can hold ON/OFF values in required positions.

Hash function in MSApriori_H.java: