How To Analyze Survey Data With Python | by Benedikt Droste | Towards Data Science

If you work in market research, you probably also have to deal with survey data. Often these are available as SAV or SPSS files. SPSS is great for statistic analysis of survey data because variables, variable labels, values, and value labels are all integrated in one dataset. With SPSS, categorized variables are easy to analyze:
Unfortunately, SPSS is slow on larger data sets and the macro system for automation is not intuitive and offers just a few options compared to Python. Therefore I would like to show you how to analyze survey data with Python.
First of all, we install the pyreadstat module, which allows us to import SPSS files as DataFrames pip install pyreadstat . If you want to install it manually, you can download the package here:
pypi.org
Now we import the module into a Jupyter notebook and load the dataset:
import numpy as npimport pandas as pdimport pyreadstatdf, meta = pyreadstat.read_sav('...path\\Surveydata.sav')df.head(5)
Let’s take a look at the DataFrame:
We can not read out much from this, because we do not know what the variables and the numerical information mean. The meta container includes all other data, such as the labels and value labels. With meta.column_labels we can print the variable labels:
For the column Sat_overall the matching label is “How satisfied are you overall?”. With a few variables, you can easily assign it from the list. If we had hundreds of variables, the whole thing would be confusing. Therefore, we first create a dictionary so that we can selectively display the correct label for a column if necessary.
meta_dict = dict(zip(meta.column_names, meta.column_labels))
With meta_dict['Sat_overall'] we now get the matching label.
In surveys, we are often interested in the percentage of respondents who chose a specific answer category:
df['Age'].value_counts(normalize=True).sort_index()
From the output we can only read that 29% voted for category 1, 33% for category 2 and almost 38% for category 3. However, in the dictionary meta.value_labels we have all value labels:
df['Age'].map(meta.variable_value_labels['Age']).value_counts(normalize=True)
That looks better! 38% of the respondents are over 50, 33% are between 30 and 50 and the rest is under 30. I personally prefer sorting according to the order of the value labels. Currently, the values ​​are simply sorted by the size of the proportions:
df['Age'].map(meta.variable_value_labels['Age']).value_counts(normalize=True).loc[meta.variable_value_labels['Age'].values()]
Perfect! That’s what we need and the result pretty much matches the output of an SPSS “Frequency”. Frequently, survey data is evaluated according to sociodemographic characteristics. For example, how does satisfaction differ by age group? For this, we can simply use the crosstab function:
pd.crosstab(df['Sat_overall'], df['Age'], dropna=True, normalize='columns')
Again, we cannot do much with the output, so let’s first map the value labels:
pd.crosstab(df['Sat_overall'].\        map(meta.variable_value_labels['Sat_overall']), \        df['Age'].map(meta.variable_value_labels['Age']), \        dropna=True, normalize='columns'). \        loc[meta.variable_value_labels['Sat_overall'].values()]. \        loc[:,meta.variable_value_labels['Age'].values()]*100
Now, we are able to interpret the output. For those over 50, there are the most completely satisfied customers. By contrast, among the under 30s there are the most dissatisfied customers. With the .loc function we can again specify the order here.
In surveys, it is often found out afterwards that the distribution of sociodemographic characteristics does not correspond to the distribution in the customer base. At the beginning we saw that the three groups are almost equally distributed. However, we know that 50% of our customers are under 30, 25% are between 30 and 50 and the rest is over 50. Therefore, we weight our data to reflect this distribution:
weight = np.NaNdf.loc[(df['Age'] == 1), 'weight'] = 0.5/(67/230)df.loc[(df['Age'] == 2), 'weight'] = 0.25/(76/230)df.loc[(df['Age'] == 3), 'weight'] = 0.25/(87/230)
But how can we now take this weight into account in our calculations? For the frequency distribution, we write a small helper function:
def weighted_frequency(x,y):    a = pd.Series(df[[x,y]].groupby(x).sum()[y])/df[y].sum()    b = a.index.map(meta.variable_value_labels[x])    c = a.values    df_temp = pd.DataFrame({'Labels': b, 'Frequency': c})    return df_temp
As a result, we get a DataFrame with the respective labels (in the numerical order from small to large) and the corresponding percentage frequency:
weighted_frequency('Age','weight')
The weighted distribution now corresponds to our customer structure. Let’s take another look at the satisfaction distribution:
weighted_frequency('Sat_overall','weight')
We see that in the weighted distribution, for example, more customers are dissatisfied. We would have underestimated the oddness in our customer base if we had not weighted. With crosstabs a weight can be easily integrated:
pd.crosstab(df['Sat_overall']. \        map(meta.variable_value_labels['Sat_overall']), \        df['Age'].map(meta.variable_value_labels['Age']),         df.weight, aggfunc = sum, dropna=True, \        normalize='columns'). \        loc[meta.variable_value_labels['Sat_overall'].values()]. \        loc[:,meta.variable_value_labels['Age'].values()]*100
All we have to do is adding the parameters for the weight (e.g. df.weight) and the function aggfunc=sum. In this example, the weighted corresponds to the unweighted distribution because the ratio of cases within a group does not change.
In a first step, we have installed pyreadstat, a module with which we can read sav-files into Python and process them further. After that, we looked at how labels and value labels can be assigned and how the analyzes can be presented in a way that is easy to interpret. We saw that Python handles categorized data very well and that it is easy to use.
Respository:https://github.com/bd317/surveydata_with_pythonNotebook:https://github.com/bd317/surveydata_with_python/blob/master/Github_Survey_Data_Upload.ipynbSPSS-Dataset:https://github.com/bd317/surveydata_with_python/blob/master/Surveydata.sav
If you enjoy Medium and Towards Data Science and didn’t sign up yet, feel free to use my referral link to join the community.