Why you should be analyzing your marketing campaigns with Python | by Limor Goldhaber | Towards Data Science

Like most young kids one of my favorite birthday games was the piñata. I would eagerly await my turn, crack the biggest smile as I was blindfolded, step up with bat in hand, and swing wildly to break open a side and set free all of the beautiful candy stored within. Obviously trying to hit your target while blindfolded proves to be an almost impossible task, which is incredibly frustrating given the fact that you know for certain there’s something hidden in that piñata.
Trying to build a successful marketing campaign without data is the equivalent of hoping to hit a piñata when you’re blindfolded. Unfortunately, working in marketing can often feel this way. Many marketing teams struggle to secure sufficient analyst resources to successfully employ a data-driven approach in developing growth marketing campaigns: identifying user segments, uncovering growth opportunities, developing churn and retention prediction models, and eventually designing and analyzing the performance & impact. Without data almost any impact derived from the campaigns will be pure luck.
I don’t like to operate blindfolded. I trust data, and so should you.
So I invested the time and energy to learn how to explore and analyze marketing data myself. As a growth marketer who can conduct deep, meaningful data analysis I’m able to identify marketing and growth opportunities, design rapid experimentation, analyze results in a timely manner and respond with additional campaign ideas based on experiments results, without needing to rely on a data scientist or analyst.
It hasn’t been easy. It took me almost 2 years to become proficient with SQL, Python and Statistics at a level that allows me to properly analyze and interpret data for growth marketing. As a marketer, reaching this level of analytical proficiency has been eye opening, and now that I’ve reached this level I would like to help other marketers take the blindfolds off too.
The truth is you don’t have to be a data scientist to effectively analyze marketing campaigns, but you do need to be proficient in Statistics (especially frequentist) in order to design and interpret experiments. If you’re just starting I suggest you first take introductory courses in Statistics. If you’re already proficient in foundational Statistics, invest in learning basic Python functions that will help you analyze marketing campaigns more efficiently and with greater depth.
The most common form of experimentation of marketing campaigns is A/B testing, aka ‘hypothesis testing’. While programs such as Mixpanel or Amplitude offer a handful of ‘off the shelf’ solutions for analyzing campaign performance, using Python at even the most basic level allows you to explore more complex relationships among different segments in your data, identify possible bugs or issues and actually save time through automation.
Let’s take a more granular look at how to start deploying Python for marketing analysis:
Every analysis starts with getting to know your data. This includes (but is not limited to): understanding the way the data is structured, removing unneeded columns, addressing nulls, running summary statistics and exploring possible relationships by creating segments or groups, and finally conducting visual exploration by plotting the data.
As most of the functions you will utilize are included in every EDA you conduct, you can create a template, or use the one I created here, to make exploring your data simple and fast.
Every marketing campaign has specific goals, and majority of campaigns will focus on a conversion metric. Let’s say you’re sending an email blast letting users know about a time limited discount for your product. To properly analyze the impact of your campaign you’ll need to define conversion metrics, such as the conversion rate of users who redeemed the discount of those targeted with the email, as well as the conversion rate of users who continued to use the product after redeeming the discount.
Your campaign will likely convert some segments of users more than others, so you’ll want to repeat the analysis on diverse segments of the population to identify differences in impact.
Using Python you can define a loop that runs the through the different segments of your population and calculates the conversion rate for each. Your segments can be anything from age groups, engagement level, or any other segment of interest. Python allows you to quickly explore conversion on a more granular level, something every successful marketing campaign needs. Without Python you will find yourself spending a significant amount of time repeating similar analyses.
Now let’s put these principles into practice.
Conversion rate is calculated by the total number of users converted divided by the total number of users targeted. In this example we’ll assume that the data is already cleaned for nulls, and organized such that each user is flagged if they were targeted in the campaign, if they converted following the campaign, and / or if they were retained post campaign.
Running these formulas on your data will give you the overall conversion and retention rates. You should compare your results to a control if you’re conducting this analysis as part of an experiment, or compare the results to other benchmarks defined in the campaign design.
Once you have an overall understanding of your campaign performance you’ll want to dive deeper and uncover differences in performance amongst subgroups of your population throughout the campaign duration, as well as compare certain features of your population.
This is where using Python can really expedite your analysis. Using custom functions you can analyze conversion and retention rate for any desired subgroup in your population, such as different languages, dates served or any other relevant feature of your targeted population.
First, create custom functions that take on any dataset and column and return the conversion and retention rates for each segment. You do this by utilizing a “group by” function. You can then calculate the conversion and retention rates similar to the way you calculated it on step 1, and return the outcome as a new dataframe. Note that you may need to unstack the results first. Your functions should look similar to the following:
While you can merge these two functions into a single function, I generally prefer to keep them separate- you may want to explore different features affecting the specific performance of each. For example, in some instances you may find that conversion rates of certain groups are lower, but their retention rates are higher, and you’ll need to further explore the features that may have impacted each rate separately.
Using a table format to explore the impact of different features and segments can start to feel overwhelming. It is often time easier to use plotting functions to visually (and quickly) detect fluctuations and potential relationships impacting the overall results. You can define custom plotting functions to save you time and allow for easier debugging.
There may be several reasons for fluctuations in your data. In some cases you may find that users convert differently on certain days of the week or time of day. In other cases you may notice differences in performance of certain languages, which could be indicative of issues related to translation of the campaign, and whether or not the campaign was served in the correct languages to users. You may even identify the need to use different types of visuals, colors or triggers to better localize the campaign in different markets.
Diving deeper into the data and resolving inconsistencies may require more advanced coding, which is out of scope for this post. I’ll explore how to handle these issues in a separate post in the future. In the meantime, if you identify any possible inconsistencies in your data I recommend consulting with the analysts within your organization. As I was building out my analytical skill set I found input from my Analyst peers to be absolutely invaluable.
As you begin to uncover relationships and conversion rates of subgroups in your target population you’ll need to calculate the lift- this will allow you to understand the rate in which your campaign actually increased conversion. If you designed your campaign in the form of a hypothesis test (aka an A/B test) you’ll first need to split the data into the relevant control and treatment groups, and then calculate the conversion rate of each group separately. From there you will calculate the difference between the treatment group conversion rate and the control conversion rate, divided by the control conversion rate.
Using Python, we can define a function for that:
Finally, the conversion & retention rates and lift you uncover will be meaningless without understanding their statistical significance. To do that you will need to conduct a T-test. There are several types of T-tests, depending on the design of your experiment & campaign. In this case, we’ll look at a two sample T-test calculation. The T-test will provide us with a T statistic and a P-value that will determine the likelihood of getting a result at least as extreme as the result you ended up with. That is, was the outcome of this campaign the result of a random chance, or was the campaign indeed impactful?
In general, a T-statistics of at least 1.96 is considered statistically significant at 95% significance level. Our P-value will need to be less than 0.05 to consider our results statistically significant.
Running a T-test in Python is actually very simple when using the SciPy library. It should look similar to the following:
# t-tetst for two independent samplesfrom scipy.stats import ttest_indt= ttest_ind(control,treatment)print (t)
In order to get the most accurate picture of the groups for whom your campaign was successful It is important to estimate the lift and whether or not the results were statistically significant among subgroups and demographics of your target population. Failing to uncover the lift and statistical significance of the subgroups that account for the targeted population may lead to a false conclusion about the overall impact of your campaign.
If you’d like to learn more about conducting hypothesis testing with Python, as well as analyzing lift and significance levels, you can read more about it in my post “A growth marketer guide to designing A/B tests using python”.
While many marketing teams may lack sufficient analyst resources, every team can still take a more data-driven approach to marketing. Marketers, and specifically those who focus on growth marketing, can greatly benefit by increasing their proficiency in Statistics and coding languages such as Python to develop more meaningful campaigns and properly analyze their impact.
In this post we covered how using Python can help you: (1) quickly uncover the relationships in the data that affect the usage of your product, (2) expedite the analysis of every campaign, which will allow you to experiment even more, and (3) optimize your campaigns to increase conversion and retention rates.
Invest the time in learning the tools you need to be the best possible growth marketer. You’ll find the reward is even better than that old piñata candy.