Bayesian Marketing Mix Modeling in Python via PyMC3 | by Dr. Robert Kübler | Towards Data Science

In this article, I want to combine two concepts that I discussed in earlier posts: Bayesian modeling and marketing mix modeling. Since the chances are high that you are not familiar with both of these topics, let me give you a quick introduction and further readings. I will
motivate what marketing mix modeling is,what Bayesian modeling is, andwhy it makes sense to combine both.
motivate what marketing mix modeling is,
what Bayesian modeling is, and
why it makes sense to combine both.
Then, I will show you how to do it in practice using PyMC3.
If you are an eager reader of my articles (thank you!), you can skip a few sections to get right to the code. Otherwise, please just continue reading.
A fundamental problem for every business is to decide on which channels to spend the marketing budget. You could spend 1.000€ on TV ads, 2.000€ on radio ads, and 3.000€ on web banners each day, just going with your gut feeling. But is this any good?
Maybe the web banner channel is saturated already, and spending only 1.500€ there is just as good as 3.000€. Then you could save 1.500€, or put them into other, better-performing channels for more sales.
Or maybe some channel even has a negative ROI — for each Euro you spend there on ads, you get less than one Euro back in return. We should definitely not waste too much money on such a channel, at least if it is not strategically important from a business perspective.
To answer questions like this, you have to understand how the different media spendings (TV, radio, ...) impact your sales or other KPI of interest.
In marketing mix modeling, you start with a dataset of media spendings. It usually gets extended with some control variables, i.e. further information about anything that might impact the target KPI, such as holidays, the weather, soccer championships, lockdowns, the price of a product, and many more. We will omit the control variables here, though, to keep it concise. Then, of course, you need a KPI you want to predict. This is often sales, number of new customers and such. So, a typical dataset might look like this:
In my older articles, I describe the motivation and how to conduct marketing mix modeling in more detail. In order to understand the rest of this article, please check them both out here:
towardsdatascience.com
towardsdatascience.com
Many estimators and models stem from a maximum likelihood approach. As an example, imagine that you want to estimate the probability p of a coin showing heads. You flip it 10 times and see 8 heads, what do you conclude? A natural estimate for the probability is then p = 8 / 10 = 80%, which also happens to be the maximum likelihood estimate. You could also calculate a confidence interval to see if this estimate is reliable, but we want to take another path.
Imagine that we want to incorporate some prior knowledge about the probability p. If you have drawn the coin randomly from your wallet, for example, there is no reason to think that the coin is biased, so p should not be too far away from 50%, assuming that you are not a magician.
With Bayesian modeling, you can incorporate this prior knowledge to end up with a density estimate of p, i.e. not a single value but an entire distribution. This distribution will probably have a peak somewhere between the maximum likelihood estimate and the prior, maybe 65%.
All in all, Bayesian modeling is about finding a trade-off between prior knowledge and the observed data. In the picture above, this means the following: without any data, we start off with the blue curve. This is just a belief, a gut feeling. Then, we observe data that tells us to move the blue curve more towards the red one. We end up with the yellow hybrid curve, which depicts the so-called posterior distribution.
You can read more about the motivation here:
towardsdatascience.com
Now, understanding the theory is good but we also have to be able to apply it to get things done. Usually, I use the awesome python library PyMC3 for Bayesian modeling. You can see it in action here:
towardsdatascience.com
You can define marketing mix models with a lot of hyperparameters:
the saturation
the carryover strength
the carryover length
...
Then, you can use a hyperparameter optimization method to find the optimal combination. That is what I have done in my other article about marketing mix modeling, An Upgraded Marketing Mix Modeling in Python.
This approach works fine, but there is a thing about it that I don’t like:
The hyperparameter estimates are often unstable.
This means that completely different sets of hyperparameters might yield equally good models. There might be
Model A with a TV carryover strength of 0.4 and a TV saturation of 0.8, and
Model B with a TV carryover strength of 0.9 and a TV saturation of 0.5,
both having the same r2 or MAPE on the test set. From a prediction standpoint, both models are interchangeable, if you stay within the marketing spend boundaries you have seen so far.
However, extrapolating using Model A is completely different from extrapolating with Model B. And this is a rather unsatisfying and troublesome behavior because extrapolation is the thing to do when optimizing media budget. If you used to spend 0€ to 1.000€ per day on TV ads historically so far, for the optimization you have to know what happens when you spend 5.000€ or even 10.000€ as well.
You need a model with excellent extrapolation capabilities.
And usually, you have to choose between more than only two models. In this case, you can proceed in at least two ways
You can pick the first model you create because you are not even aware of the problem. This approach is easy but dangerous.
You can pick a model that seems right to you, some domain experts, or the stakeholders. For some people this is okay but I prefer not baking the output expectations into the model because of the following question:
If somebody knows the answer already, why would I even build a model that just replicates that answer?
There might be ways to do something reasonable from here as well but I want to show you how to circumvent this problem using Bayesian modeling now.
First, let us grab our dataset.
import pandas as pddata = pd.read_csv(   'https://raw.githubusercontent.com/Garve/datasets/4576d323bf2b66c906d5130d686245ad205505cf/mmm.csv',    parse_dates=['Date'],    index_col='Date')X = data.drop(columns=['Sales'])y = data['Sales']
Then, we have to define the saturation and carryover functionality, similar to the last article. In PyMC3 language, it might look like this:
import theano.tensor as ttdef saturate(x, a):    return 1 - tt.exp(-a*x)def carryover(x, strength, length=21):    w = tt.as_tensor_variable(        [tt.power(strength, i) for i in range(length)]    )        x_lags = tt.stack(        [tt.concatenate([            tt.zeros(i),            x[:x.shape[0]-i]        ]) for i in range(length)]    )        return tt.dot(w, x_lags)
The saturation function should be easy to grasp. The carryover, however, is a bit of work. Basically, you can express the carryover transformation as a matrix-vector multiplication. You just have to assemble the matrix x_lags and the vector w first. As an example, we can transform the input vector x = (x1, x2, x3, x4) with a carryover length of 3 via
The carryover function in the code above does exactly this. If we have these functions, we can finally start modeling.
import pymc3 as pmwith pm.Model() as mmm:    channel_contributions = []        for channel in X.columns:        coef = pm.Exponential(f'coef_{channel}', lam=0.0001)        sat = pm.Exponential(f'sat_{channel}', lam=1)        car = pm.Beta(f'car_{channel}', alpha=2, beta=2)                channel_data = X[channel].values        channel_contribution = pm.Deterministic(            f'contribution_{channel}',            coef * saturate(                carryover(                    channel_data,                    car                ),                sat            )        )                channel_contributions.append(channel_contribution)            base = pm.Exponential('base', lam=0.0001)    noise = pm.Exponential('noise', lam=0.0001)        sales = pm.Normal(        'sales',        mu=sum(channel_contributions) + base,        sigma=noise,        observed=y    )        trace = pm.sample(return_inferencedata=True, tune=3000)
We can see all parameters (no hyperparameters anymore!) marked in bold. It is the regression coefficients, saturation strength, carryover strength, the baseline, and the noise.
Note that I did not take the carryover length into account, I rather set it to 21. This is because I could not figure out yet how to create matrices and vectors in the carrover function with variable dimensions in PyMC3. But usually, I would use a Poisson random variable for the carryover length. Leave me a note if you know how to do this properly! 😉 Maybe it is also a good time to give pyro a chance, another probabilistic programming language like PyMC3.
Afterward, we can look at the usual pictures. Let us start with the posterior distributions. Executing
import arviz as azaz.plot_posterior(    trace,    var_names=['~contribution'],    filter_vars='like')
yields
Here, you can see the posteriors for all parameters. They all have a nice unimodal (=one peak) shape. You can also explore how pairs of variables behave together via
az.plot_joint(    trace,    var_names=['coef_TV', 'sat_TV'],)
Here you can see that the saturation strength and the regression coefficient as not independent, but negatively correlated: the higher the coefficient, the lower the saturation parameter tends to be. This makes sense because a higher coefficient can compensate for a slower increasing saturation curve (=lower sat_TV) and vice versa.
Let us look at another one:
az.plot_joint(    trace,    var_names=['car_TV', 'sat_TV'],)
Here, we can see why hyperparameter optimization might have problems. Every dot in this picture is a potential model that you could find with a hyperparameter optimization.
For a truly unique best model, we would rather see a point cloud tightly concentrated around a single point (car_TV_true, sat_TV_true). Here, however, we see that the TV carryover strength can have reasonable values between 0.4 and 0.5, depending on the saturation parameter.
We can also check if the model is any good before we jump to conclusions. Executing
import matplotlib.pyplot as pltwith mmm:    posterior = pm.sample_posterior_predictive(trace)means = posterior['sales'].mean(0)stds = posterior['sales'].std(0)plt.figure(figsize=(20, 8))plt.plot(y.values, linewidth=2, c='r', label='Observations')plt.plot(means, linewidth=1, c='b', label='Mean prediction')plt.fill_between(np.arange(len(y)), means - 2*stds, means + 2*stds, alpha=0.33)plt.legend()
gives us
so it looks like the model picked up something useful. I will not go into detail about how to evaluate the performance of the model any further here, we can do this in the future.
We dealt with distributions so far, but for our favorite channel contributions picture, let us take means to end up with a single value again. Since we introduced some channel contribution variables in the PyMC3 code, we can easily extract them now using a short compute_mean function.
def compute_mean(trace, channel):    return (trace            .posterior[f'contribution_{channel}']            .values            .reshape(4000, 200)            .mean(0)           )channels = ['Banners', 'Radio', 'TV']unadj_contributions = pd.DataFrame(    {'Base': trace.posterior['base'].values.mean()},    index=X.index)for channel in channels:    unadj_contributions[channel] = compute_mean(trace, channel)adj_contributions = (unadj_contributions                     .div(unadj_contributions.sum(axis=1), axis=0)                     .mul(y, axis=0)                    )ax = (adj_contributions      .plot.area(          figsize=(16, 10),          linewidth=1,          title='Predicted Sales and Breakdown',          ylabel='Sales',          xlabel='Date'      )     )    handles, labels = ax.get_legend_handles_labels()ax.legend(    handles[::-1], labels[::-1],    title='Channels', loc="center left",    bbox_to_anchor=(1.01, 0.5))
Looks good!
In this article, we have discussed that marketing mix modeling using a maximum likelihood approach can be difficult because of the hyperparameters estimation. There might be many models that perform well on the test set but have completely different extrapolation behavior — the (hyper)parameters are quite unstable.
However, proper extrapolation is the key to optimization. Therefore, we have designed a basic marketing mix model with saturation and carryover effects in the Bayesian setting. This proved useful because it estimates all the parameters at once, giving us more stable parameter estimates.
We then implemented it using PyMC3 and created a nice contributions chart again to see how much the channel spendings contributed to the sales.
We could now take the Bayesian model that we have created and extend it even further. For example, we can incorporate changing parameters over time. This can be useful if the TV carryover strength started out at 0.8 two years ago but slowly declined to 0.5 over time, which is an example of concept drift. We can easily model this using Gaussian random walks, as shown in my other article about rolling regression:
towardsdatascience.com
In the same way, we can model changing baselines. So far, we have treated the baseline as a single number that is fixed over the whole training time period. But maybe it increased over time as well because of our strategic advertising efforts. This is something that is hard to figure out using our old maximum likelihood models with hyperparameter optimization.
I hope that you learned something new, interesting, and useful today. Thanks for reading!
As the last point, if you
want to support me in writing more about machine learning andplan to get a Medium subscription anyway,
want to support me in writing more about machine learning and
plan to get a Medium subscription anyway,
why not do it via this link? This would help me a lot! 😊
To be transparent, the price for you does not change, but about half of the subscription fees go directly to me.
Thanks a lot, if you consider supporting me!
If you have any questions, write me on LinkedIn!