Practical Data Analysis Guide with Pandas: Direct Marketing | by Soner Yıldırım | Towards Data Science

Pandas provides numerous functions and methods for data analysis and manipulation. In this post, we will use Pandas to gain insight into a direct marketing dataset.
The dataset is available on Kaggle. It contains relevant data of a marketing campaign done via direct mail. We will explore this dataset and try to understand which customers are likely to spend more money.
Let’s start by reading the dataset into a dataframe.
import numpy as npimport pandas as pddf = pd.read_csv("/content/DirectMarketing.csv")print(df.shape)df.head()
The data consists of 1000 observations (i.e. rows) and 10 features (i.e. columns). The focus is the “AmountSpent” column which indicates how much a customer has spent so far.
Before we start on the analysis, it is best to make sure there are no missing values.
df.isna().sum()Age              0Gender           0OwnHome          0Married          0Location         0Salary           0Children         0History        303Catalogs         0AmountSpent      0dtype: int64
The “History” column contains 303 missing values. These are used to indicate a customer has not made purchase yet. We can check the other values in the column and then decide how to handle the missing values.
df.History.value_counts()High      255Low       230Medium    212Name: History, dtype: int64
Since it is a categorical column, we can replace the missing values with the word “Nothing” that indicates no purchase has been made yet.
df.History.fillna("Nothing", inplace=True)df.isna().sum().sum()0
We start with checking the statistics of the “AmountSpent” column.
print(f'The average money spent is {df.AmountSpent.mean()}. The median is {df.AmountSpent.median()}')The average money spent is 1216.77. The median is 962.0
The mean is much higher than the median which shows that this column does not have a normal distribution. There are outliers with high values. Thus, we expect the “AmountSpent” column to have a right skewed distribution.
Let’s check it with a histogram.
df['AmountSpent'].plot(kind='hist', figsize=(10,6))
It overlaps with the statistics. Most observations are in the low section with a small number of outliers reaching toward the higher end.
Age might be an important feature because direct mail marketing tends to appeal older people. The groupby function will give us an overview.
df[['Age','AmountSpent']].groupby('Age').mean()
The average amount of money spent is much higher for middle and old aged people than for young people.
I also want to see if the campaign is more focused on older people. We can check the number of catalogs sent to each group along with average money spent.
df[['Age','Catalogs','AmountSpent']].groupby('Age').agg(['mean','count'])
The count shows the number of observations in a group. We see that more middle aged people are involved in the campaign. The average number of catalogs sent to young customers are a little less than that of older customers.
As you notice, we can apply multiple aggregations on the group values by passing a list to the aggregate function.
Before calculating the correlations, I will briefly explain what correlation means with regards to variables.
Correlation is a normalization of covariance by the standard deviation of each variable. Covariance is a quantitative measure that represents how much the variations of two variables match each other. In simpler terms, correlation and covariance indicate if the values tend to change in similar ways.
The normalization of covariance cancels out the units. Thus, the correlation value is always between 0 and 1 in case of positive correlation and 0 and -1 in case of negative correlation.
To calculate the correlations between numerical variables, we will use the corr function of pandas and visualize the results with a heat map.
#importing visualization librariesimport matplotlib.pyplot as pltimport seaborn as sns%matplotlib inlineplt.figure(figsize=(10,6))corr = df.corr()sns.heatmap(corr, annot=True)
There is a highly strong correlation between the salary and spent amount. The number of catalogs also has a positive correlation with spent amount.
On the other hand, number of children and spent amount have a small negative correlation.
The correlation matrix shows a negative correlation between the number of children and spent amount. Thus, we expect to see a decreasing spent amount as the number of children increases.
Let’s also check it with the groupby function. I will also include the age column. The groupby function accepts multiple columns and creates a group for each set of categories.
df[['Age','Children','AmountSpent']]\.groupby(['Age','Children']).agg(['mean','count'])
In general, the average money spent decreases as the number of children increases. There are a few exceptions though.
We will check a couple of measures with the gender and marriage columns. The groupby function allows applying different aggregate functions to different columns. Furthermore, we can rename the columns of aggregations.
It will be more clear with the example. I want to create groups based on gender and marriage columns. Then I will calculate:
The average number of catalogs sent to each group (mean)
The total amount of purchases made by each group (sum)
I also want to rename the columns appropriately. Here is the groupby syntax that does this operation:
df[['Gender','Married','AmountSpent','Catalogs']]\.groupby(['Gender','Married'])\.agg(    Average_number_of_catalogs = pd.NamedAgg('Catalogs','mean'),    Total_purchase_amount = pd.NamedAgg('AmountSpent','sum'))
The NamedAgg method is used to rename the columns. Another way to change the column names is the rename function of Pandas.
This retailer should definitely focus on married people.
The location might also be an important indicator. Let’s check the “AmountSpent” column based on the location.
df[['Location','AmountSpent']].groupby('Location')\.agg(['mean','count','sum'])
The number of customers who live close by is more than double the customers that are far. However, they spend more on average than the customers who live close by.
The retailer might want to focus a little more on people who live far.
We can dig more into the dataset and try to gain more insight. We haven’t check some of the features yet. However, the techniques are quite similar.
The main purpose of this post is to introduce different techniques and methods to use in data exploration. There are, of course, more functions and methods that Pandas provides. The best way to learn and master them is to practice a lot.
I suggest to work on sample data sets and try to explore them. There are free resources that you can find sample data sets. For instance, Jeff Hale lists 10 great places to find datasets in this article.
Thank you for reading. Please let me know if you have any feedback.