Significance testing survey data with Python | by Geir Freysson | Towards Data Science

Most of the world will make decisions by either guessing or using their gut. They will either be lucky or wrong. — Mixpanel
The opening line of Mixpanel’s $65 million investors’s pitch is arresting. Either you’re using data or you’re a fool. I’d like to add to that catch phrase.
Most of the world don’t think about statistical significance. They will be data-driven, but in the wrong direction. — Geir
Okay, Mixpanel’s was better, but you get the point.
Let’s say we spot a trend in a political poll that can help us decide who to target with our election campaigning. Married people, for example. Do they vote differently than the unattached? Should we be campaigning at singles bars?
The open source library Quantipy can read all major survey platform data types, including SPSS’s .sav files. This one is from the 2019 General Election in the UK.
import quantipy as qpdataset = qp.DataSet('Elections 2019')dataset.read_spss('election survey.sav')
Let’s see what a few select columns from the raw data look like:
dataset[['Gender',          'VoteIntention',          'MaritalStatus',          'weight']].head()
To speed up computing, SPSS stores different answers as numbers and then uses meta-data to map those numbers to answers. (Later, we’ll write about calculating a weight column to improve our results). Let’s stick with voting intention for now:
dataset.meta('VoteIntention')
Comparing responses to the meta-data we see that the first responses were all from Conservative voters, whereas the marital status of those same responses varied. Does it affect how people will vote? Let’s do the calculation:
dataset.crosstab('VoteIntention',                  'MaritalStatus',                  pct=True,        #percentages, not frequencies                 w='weight')      #weigh the results
Aha! So, married people vote CON and single people vote LAB. Maybe it’s an age thing. But what’s going on with people in civil partnerships? They are way more likely to vote Conservative than Labour, CON: 54%, LAB: 16% compared to CON:40%, LAB:33% for the overall population.
Before we start writing our press releases we have to ask ourselves: is this a real thing, or is this simply noise in the data?
Testing survey results for significance (sig-testing) can be a laborious task. The data itself can be complex, and when it’s weighted to better represent the population being surveyed it gets even trickier. Sometimes the so-called effective base should be used for the tests and sometimes not. The market research industry also relies on using proprietary programs like UNICOM Intelligence (aka Dimensions) and Askia, and being able to replicate the results from those applications can be important.
Quantipy helps with all of those challenges. Let’s run a basic sig-test that replicates the Dimensions set-up, which uses a pooled variance and controls for bias introduced by weighting and overlapping samples.
batch = dataset.add_batch(name='vote_intention')batch.add_downbreak('VoteIntention')batch.add_crossbreak('MaritalStatus')batch.weights.append('weight')batch.set_sigtests([0.05])stack = dataset.populate(['vote_intention'])stack.aggregate(['counts', 'cbase'],                 batches=['vote_intention'], verbose=False)stack.add_tests(['vote_intention'])
In the above code, we use Quantipy’s batch mechanism, in case we want to calculate a large quantity of variables to populate our tables or PowerPoint charts. When we run the code, we get a so-called stack that stores all of our results; The calculated percentages, base sizes and the significance matrix. The matrix looks like this:
NaN means there is no significant difference, the stars flag results that have too low effective bases (too few respondents to be significant) and an array of numbers represents a significant difference. In the above, married CON voters have the array [1,4] and are therefore significantly higher than single voters and co-habiting partners. Single voters have the results [2,4] in the Labour column and are therefore significantly more likely to vote Labour than married voters and co-habiting voters, but importantly, the civil partnership column doesn’t have enough responses to be significant.
We can now leverage any python web-app environment (Django, Flask etc.) to show an interactive significance table. Below is a Django app that filters with the drop down menu and sends the relevant sub-set of responses into a sig-test. Exploring the different voting intentions of sub-groups of the data takes on a whole new dimensions when significant testing is applied and visualised.
So, don’t send out that press release on how people in civil partnerships are going to vote, because more data is needed to get a meaningful insight into whether that result is significant, or whether it is just noise in the data trying to lead us astray.
Geir Freysson is co-founder of Datasmoothie, a platform that specialises in the automation of survey data analysis and visualisation. If you’re interested in using open source software for survey data analysis sign up to our newsletter, called Unprompted Awareness.