Association Rule Mining — Not Your Typical ML Algorithm
By: Kirk Borne
Tags: Business Analytics, Data Science, Algorithms, Machine Learning, Predictive Analytics

Many mathematical algorithms that we use in data science and machine learning require numeric data. And many algorithms tend to be very complex to implement (such as Support Vector Machines or Local Linear Embedding, which we previously discussed). But, association rule mining is perfect for categorical (non-numeric) data and it involves nothing more than simple counting!

What we have here is a simple algorithm with not so simplistic results! The ratio of actionable insights discovery potential (high) to algorithm complexity (low) is quite large and atypical, IMHO.

Association rule mining is primarily focused on finding frequent co-occurring associations among a collection of items. It is sometimes referred to as “Market Basket Analysis”, since that was the original application area of association mining. The goal is to find associations of items that occur together more often than you would expect from a random sampling of all possibilities.

A Legendary Example

The classic example of market basket analysis is the famous Beer and Diapers association that was frequently mentioned 10–20 years ago in every data mining book — whether the story is real or legend is a subject of debate. The story goes like this: customers who go to the store to buy diapers will also tend to buy beer at the same time with a higher-than-random frequency.

Let us illustrate this with a simple fictitious example. Suppose that a store’s retail transactions database includes the following information:

There are 600,000 transactions in total.

7,500 transactions contain diapers (1.25 percent)

60,000 transactions contain beer (10 percent)

6,000 transactions contain both diapers and beer (1.0 percent)

If there was no association between beer and diapers (i.e., they are statistically independent), then we expect that only 10% of diaper purchasers (=750) will also buy beer (since 10% of all customers buy beer). However, we discover that 80% (=6000/7500) of diaper purchasers also buy beer. This is a factor of 8 increase over what was expected — that is called Lift, which is the ratio of the observed frequency of co-occurrence to the expected frequency. This was determined simply by counting the transactions in the database. So, in this case, the association rule would state that diaper purchasers will also buy beer with a Lift factor of 8.

Source for this example: https://www.youtube.com/watch?v=qMQfUy8ndco

In statistics, Lift is simply estimated by the ratio of the joint probability of two items x and y, divided by the product of their individual probabilities: Lift = P(x,y)/[P(x)P(y)]. If the two items are statistically independent, then P(x,y)=P(x)P(y), corresponding to Lift = 1 in that case. Note that an anti-association between x and y yields Lift values less than 1, which is also an interesting discovery — corresponding to mutually exclusive items that rarely co-occur together.

True Story #1 — Hurricanes

The simple fictitious example shown above was made up, and it is very rare in real world cases to have Lift factors as high as 8. However, there was a case where it did happen. That case was discovered by Walmart in 2004 when a series of hurricanes crossed the state of Florida. After the first hurricane, there were several more hurricanes seen in the Atlantic Ocean heading toward Florida, and so Walmart’s data analysts mined their massive retail transaction database to see what their customers really wanted to buy prior to the arrival of a hurricane.

The analysts found one particular item that increased in sales by a factor of 7 over normal shopping days. That was a huge Lift factor for a real-world case. That one item was not bottled water, or batteries, or beer, or diapers, or flashlights, or generators, or any of the usual things that we might imagine. The item was strawberry pop tarts! One could imagine lots of reasons why this was the most desired product prior to the arrival of a hurricane — pop tarts do not require refrigeration, they do not need to be cooked, they come in individually wrapped portions, they have a long shelf life, they are a snack food, they are a breakfast food, they are a comfort food, kids love them, and we love them. Despite these “obvious” reasons, it was a still a huge surprise! And this discovery provided actionable insights — Walmart stocked their stores with tons of strawberry pop tarts prior to the next hurricanes, and they sold them out. That is a win-win: Walmart wins by making the sell, and customers win by getting the product that they most want. We really want our comfort food during stressful times.

An Exercise For The Student

When I was a Professor at George Mason University, I taught introductory, intermediate, and graduate level data science courses. For the graduate class, I would sometimes give my students an association mining project for the semester. It was open-ended project — I gave them a pointer to a dataset and a pointer to some free association mining software, and then gave them the freedom to explore the dataset for interesting associations (either high lift > 1 or low lift < 1). It was truly unsupervised learning, both algorithmically and pedagogically. I wanted to see how my students would set up their experiments, how they conducted their explorations, what they would find, and how they would communicate their findings to the class.

The software package they used was the WEKA Data Mining & Machine Learning software workbench. And the dataset was the Forest Cover dataset from the UCI KDD Archive. This dataset has over 580,000 point measurements of forest cover types in a large variety of environments. The challenge was to find interesting associations among the different features (tree types, geographic situations) and the environmental variables (soil types, sun exposure, access to water).

It was always a fun exercise, at least for me, because the students had the freedom to explore many different associations in a large dataset. There was no right or wrong answer, though occasionally there were some wrong initial choices that I had to correct before the project went astray. The data storytelling was also an integral part of the exercise.

True Story #2 — More Hurricanes

Another example of association mining was provided to me by a colleague at the university. He is a professor of geoinformation systems and earth science. He and his research team decided to explore the use of association mining to examine the relationship between hurricane strength and the physical characteristics of hurricanes (e.g., internal wind speed, atmospheric temperature, ocean water temperature, humidity, atmospheric pressure in the eye of the hurricane, wind shear, rainfall amounts, direction and propagation speed of the hurricane, etc.). They discovered a strong association between the final strength (hurricane category) of the hurricane when it makes landfall and the values of those different physical characteristics while the hurricane was still far out over the ocean (many days or weeks prior to landfall).

It was a very successful experiment. They were able to predict hurricane intensification and its ultimate strength, and the results proved to be more accurate with association mining than with the standard hurricane model used at that time by the national hurricane center. That was an amazing application of an algorithm that was initially developed for retail store transaction mining.

True Story #3 — Solar Storms and Geo Impacts

There was an equally impressive scientific application that I encountered several years ago when I was working at NASA. Every summer we would have student interns working with us. These students were usually college undergraduates (typically, rising seniors), and they were always very bright. In one of those summers, we had a student who was not yet a senior in high school (4 years younger than his fellow interns). He heard me give a lunchtime talk on data mining, which I presented to the full cohort of NASA summer interns that year. He was working on a project with a NASA space physicist to try to predict when solar energetic particles would reach the earth after the occurrence of a major solar storm on the Sun. There are very serious consequences of such events on our human endeavors. The intern decided to apply association mining to the prediction problem.

What the intern did was very clever. In an approach that is similar to the hurricane example mentioned above, he collected measured physical characteristics of the solar storms on the Sun and for geomagnetic events around the earth (as measured by NASA’s satellites) to look for predictive patterns. But the super-special clever thing that he did was to look at time-shifted data values. For example, he compared events on the Sun with geo events with time lags of 1 hour, 2 hours, 3 hours, etc., up to 12 hours, in order to see when the peak correlation (association!) occurred. He found it — the strongest geomagnetic effects were measured around the earth at approximately 2–3 hours after the solar storm event, and he could tell us which solar parameters were the most predictive features in the multi-satellite observations dataset. His NASA mentor called me into his office to show me the amazing discovery by this high school junior, using the simple techniques that I taught him in my lunchtime seminar. We were all quite impressed!

True Story #4 — Precursor Analytics

My final example of association rule mining is a discovery made many years ago by a major electronics store that sold video cameras (camcorders) and video (VHS) players. The store learned quickly that their upsell offers to customers who bought the player, recommending that they also buy the camcorder on the same day, had very little success. Using a little business domain intuition, the store’s data analysts decided to explore their retail customer transaction database for some customer insights. That turned out to be a smart decision and a great example of analytic thinking!

The analysts discovered that customers who bought a VHS player tend to come back to the store about 3–4 months later to buy the camcorder (probably because the customers decided by then that they wanted to make their own movies). This is an example of precursor analytics. The store then used this information to send discount coupons for camcorders to all of its customers who bought VHS players a few months earlier, in order to entice those customers to return to that store to purchase a camcorder.

Why send a discount coupon if they knew that the customer was going to buy the product anyway? Well, they didn’t want to take the risk that the customer might decide to make their purchase at some other store. After all, this store had the customer intelligence to know when and to whom they should make this very specific targeted product offer. Why waste such valuable customer intelligence! Apparently, this customer engagement program worked! And its success was due to association rule mining.

Actionable Insights Discovery

The above examples illustrate two very useful approaches when exploring your own massive data collections for meaningful (interesting and useful) insights: (1) search for rare and unusual co-occurring associations of non-numeric items (which then makes for powerful insights discovery); and (2) if you have time-labeled metadata for your data (regardless of whether the data items are numeric or nonnumeric), consider introducing a time lag in one or more features in the dataset to see if the strength of association between different features reaches it peak at some particular nonzero value of the time shift​. That latter approach is precursor analytics — finding the early warning signals in one feature that associate with another feature at some future time.

With the massive quantities of data that are now available, and with clever algorithms to explore those data, one can only imagine what surprising and useful associations are waiting to be discovered in your organization’s data that can boost your actionable insights discovery. You too can be a member of the “Data Heroes-Я-us” community.

Start counting!

Follow me on Twitter at @KirkDBorne

Learn more about my freelance consulting / training business: Data Leadership Group LLC

See what we are doing at AI startup DataPrime.ai