Review: SAS Cortex Analytics Simulation Competition
By: Harry Ngo
Tags: Modeling, Predictive Analytics, Machine Learning, Review, Data Science

Review: SAS Cortex Analytics Simulation Competition

Learn predictive analytics with SAS Enterprise Miner

Cortex Analytics Simulation by SAS and HEC Montréal

From late April to early May, I got to participate in the Cortex Analytics Simulation Competition by SAS in collaboration with HEC Montréal. The competition was open to everyone, especially targeted to students, who wanted to explore how predictive analytics is used in a real-world scenario.

I won’t go into too much detail about the game since I don’t want to give away possible answers for future competitions, but I would say it was a good experience learning about predictive analytics using SAS Enterprise Miner.

If you come across the competition and would like to learn more, this article goes through the game scenario, instructions and my personal experience throughout the competition.

Scenario

You’re working on a fundraising campaign for a not-for-profit charitable organisation with 1 million members. The foundation decided to add a direct contact campaign to its list of marketing activities. The objective is to fundraise the most donations given the costs of calling members. This will be done using predictive modeling software (SAS Enterprise Miner) to predict how many and which individuals to target in the campaign.

You will be provided with the dataset of potential donors. You will fit different models based on the previous behavior of donors, scoring donors to predict the donation this year. Using this output, decide how many potential donors to target. Upload your list of IDs which will be ranked from the submissions based on operating surplus: sum of donations - total cost of calling.

I created a table of some of the information given below.

Table 1: some variables listed, Table 2: cost of contacting members

Daily Webinars and Instructions

As it’s an introductory competition, participants are given detailed game instructions and could attend daily 1-hr webinars (provided by SAS).

These webinars included presentations going through the instructions and Q&A sessions. We were given two different methods of modeling:

“1-stage modeling”: a simple model which scores donors based on the predicted amount they will give. “2-stage modeling”: a more complex model which takes into account both the probability of giving (when contacted or not) and the predicted amount.

Of course, the 2-stage modeling process was more effective. It is more commonly known as uplift modeling, where the uplift is calculating by computing the difference between the predicted amounts which take into account the probability of giving when a person is contacted or not.

Competition

The competition was completed via a SAS Virtual Machine/Lab. You have a total of 20 hours over the 5 days to test different models and upload your results. Since it is on a virtual machine in the cloud, you’re required to make a reservation, which takes around 25 minutes to start up. There could also be a limit to how many times you can submit your answers (around 30).

During the competition, your objective is to test different types of machine learning models to predict probabilities or donation amounts. The layout of SAS Enterprise Miner can be quite intimidating at first which is why it’s best to follow the instructions given or the webinar.

SAS Enterprise Miner uses a simple system of drag and drop, where you can add different nodes and connect them to other nodes. This made it easy to learn how to use the software and test new models. Most of the diagram is completed for you and your main task would be to test different models and tweak the properties of them.

Some models include decision trees, regression, neural networks and gradient boosting. They also have high-performance (HP) data mining nodes which use several techniques not used in traditional methods (these may take longer to run). You’ll spend most of your time placing different models and placing the model comparison node to check which model performs the best (the lowest average squared error).

Diagram flow (similar to SAS Enterprise Miner)

The above shows a general diagram in SAS Enterprise Miner I created:

Initial Data Preparation: merges data together. Data Partition: data is split into training and validation sets. Modeling: add different models and modify their properties (here we show regression and decision tree). This stage could take a couple of minutes or even hours if you use complicated models (such as the HP nodes). Model Comparison: compares each model and shows their performance. Data Preparation for Scoring: scores new data based on the model used. SAS Code: exports the predictions to an Excel file.

After receiving your Excel file output, your job is to sort the data, collect how many IDs you want to use and submit your results. You’ll straight away see how well your submitted list of potential donors performs.

What I found particularly useful was seeing how many donors other participants in the competition are targeting so you can get an idea of how much money you could potentially collect. You spend most of your time tinkering around with different models, so it’s best to add different variations of models to find the best performing model. For example, you could use logistic regression instead of linear, or increase the complexity for the decision tree. Other ways you could try are removing or transforming variables.

Conclusion

In the end, I came 3rd overall in the competition, missing out on the top prize of $1000 AUD. I think if you would like to learn more about SAS Enterprise Miner and how it can be used for predictive analytics, joining the SAS Cortex Analytics Simulation Competition could be a great start. Although you have to pay to enter the competition, you’ll be able to share your experience online (or through an interview) and showcase your participant badge on Acclaim if you miss out on first place.