Paid Search Incrementality. Rethinking your targets and finding the... | by António Lima | Towards Data Science

Many advertisers and digital marketing professionals often work towards achieving a target cost per sale while maximising the number of customers or sales they bring to the business. Today we will challenge the best way to achieve those goals in the paid search world introducing the concept of incrementality. We will do it so by not only explaining the business rational and theory behind it but also proposing actual practical steps to operate in an incrementality framework as well as offering code samples to tackle the challenge in an efficient manner.
Let’s then start by defining incrementality as the impact that a specific action has in changing total output (that being cost, sales, revenue, profit or other). To understand incrementality we have to realise that there is always a baseline situation against which you want to measure the impact of any given action:
Scenario 1 — An action is totally incremental — Ultimately if the baseline scenario is a state of absence of almost everything then all actions are 100% incremental. For instance, having a live website for an e-commerce company is 100% incremental. No sales or revenue can be made without it.
Scenario 2 — An action is partially incremental — Not all actions are as critical as the one above. Consider for example a company that buys advertising space regarding their own brand on Google, it is expectable that if they didn’t do it so then some potential customers would click on their competitors’ paid ads links; however, they cannot say that the revenue originated by people who clicked on their own ads is 100% incremental because some people could have come to their website in any case via the SEO organic link on their own brand, skipping through the competitors’ paid ads. (Note: not advocating for not bidding on your brand, many companies do it because they’re usually cheap clicks and they’re afraid competition will steer away some of their natural customers)
Scenario 3 — An action is partially incremental and you want to find out how much of doing that action is optimal — Imagine a restaurant, for example, trying to figure out how many options they should have available on their menus. Surely the answer is often more than one or two for increased choice and pleasing different subsets of customers, however, at which point the benefit of one extra menu item for increased variability is surpassed by the cost of having to manage extra food inventories, train their chefs to make such recipes and increased difficulty in kitchen coordination to jump between different dishes at one given service? Probably many restaurants would like to have a tool at their disposal that would help them solve this optimisation problem.
Scenario 4 — An action is not incremental at all — A simple example would be giving your employees white notebooks instead of black notebooks: this action is hardly ever incremental on employees’ productivity at any given point. Sometimes, however, non-incremental actions are harder to identify. For example, you show some Facebook ads to your existing customers and they eventually at some point purchase your items again (there’s a correlation). Although, it could be the case that they would buy your items in any case (correlation not implying causation) or even that some of them would eventually buy again from you but they became upset with your company over-communicating and ended up not returning to your website or shop (in this case your action would have been negatively incremental).
A quick note before proceeding: A reasonable expectation regarding the meaning of the expression paid search incrementality could be the pursuit of understanding the incremental value of the channel as a whole (i.e. what would happen if we turned off ppc at all? Would some potential customers still find us via other marketing channels?). Although this can be an immensely insightful study for some companies, this is not what this article will be exploring today. Instead, this article will be focusing on what we described in scenario 3 and its application to the Paid Search world. Hopefully, this could be useful to help you think about your PPC goals and targets, and how to measure what are your optimal bids for achieving them.
So like in the example of restaurants optimising the number of menu items they should have, chances are that some google ads advertising will be beneficial and profitable for your business and that too much google ads advertising starts to become marginally loss making for your company. This article proposes a framework for finding that optimal amount of google ads advertising while bearing an incrementality mindset and steering away from an averages way of thinking. This means making the transition between optimisation maturity 2 and 3 described below (which can be as important and fundamental as making the transition between optimisation maturity 1 and 2):
Optimisation maturity 1 — Company Target — The SEM channel manager ensures that the account is roughly achieving an overall target per conversion and that the return on ad spend is in line with the expectations of upper management or clients. The diversity of campaigns is either limited or the span in actual cost per conversion across campaigns is really big.
Optimisation maturity 2 — Averages — The SEM manager still ensures that the account’s return on ad spend is in line with expectations, but at the same time ensures that it has a robust account structure with a multitude of campaigns achieving similar costs per conversion across all of them. The rationale behind it is that firstly campaigns with an average cost per conversion above target are unprofitable for the company and secondly that volumes of conversions are maximised when all campaigns are on target vs having some campaigns below and others above target (even if at the end of the day both scenarios lead to an overall account on target). As a second step after ensuring the above, the SEM manager will also often ensure that other ways of “slicing” the account other than on a campaign level will also be on par with each other in terms of the average cost per conversion (examples being device type, major locations, ad group types that repeat themselves across many campaigns, etc.). Most companies operate on this level of ppc optimisation maturity.
Optimisation maturity 3 — Incrementality — Companies operating on this framework understand that their average targets and metrics are just the tip of the iceberg. The critical mind shift is that they redefine the question from “what’s my average cost per conversion?” into “what is the maximum I am willing to spend on any given conversion?”. They also understand diminishing returns and that they vary depending on many factors: They know that two campaigns with the same average cost per conversion can have very different incremental costs per conversion and if they have a pound or a dollar more to spend they won’t distribute evenly across these two campaigns but they will instead apply it on the one with lower incremental cost per conversion. However, to put such framework in practice the very first thing they do differently is to generate their own datasets as incrementality metrics are not readily available and this usually involves a lot of cross-team collaboration (and they don’t by any means underestimate the importance of communication, alignment and buy in across all teams involved). Without such an effort, this framework won’t ever be nothing more than a theoretical exercise. Finally, once that’s unlocked, they go from theory to practice, from averages to incrementality, by using their new metrics, reports and marketing business objectives.
But what do we mean by campaigns with the same averages but different incrementality economics? Let’s consider the following example: For both campaigns ABC and XYZ, you have set a target cpa of 6£ (assuming smart bidding here but should be no different if you’re running keyword level manual bidding) and Google is able to get you 100 conversions at that average cost for each campaign. Without taking incrementality into consideration you might have no reason to take different actions when optimising each campaign. However, what might be happening in the background is that campaign ABC would be able to get you 80 conversions at an average price of £5.8 and extra 20 conversions at an average price of £6.8 (average cpa for the total 100 conversion still being £6/conversion) whereas campaign XYZ would be able to get you 80 conversions at an average price of £4 and then finds some slack to get you an extra 20 conversions at an average price of £14 (average cpa for the total conversions still being £6 pounds — and yes this can happen and it does happen all the time in so many occasions!). Basically, in one case you are acquiring extra 20 conversions at a price that’s still reasonable and in the other at a price that probably makes you lose money and this sums up the idea of why averages often hide half of the picture. We finish this section with a chart that illustrates this idea:
At this stage, we have laid the theoretical framework in regards to why PPC Incrementality is important, but we haven’t yet proposed a practical way to operate under this level of optimisation maturity and that is probably the main point of this article. It is harder though than operating on an averages framework hence fewer companies adopt it. The good thing about working with averages is that the data is usually readily available for consumption and it does not require extra steps for generating it. Google ads will provide you with detailed spend and conversion data from which you can easily calculate your average cost per conversion on the account itself or on almost whichever “slice” of the account (per campaign, ad group, device, a combination of device and campaign, location, audience, etc.). If you want to operate on an incrementality framework you will have to figure out which data you want to generate, do generate it and ensure its quality and accuracy.
So, how do you run an incrementality test on your ppc account? And yes I say a test because on the one hand for many people ppc incrementality might have not been a thing they have yet implemented and on the other ppc incrementality does require testing per se. When we earlier gave an example in which out of you 100 conversions you would get at a total cost of £600 you could potentially get 80 of them at a cost of £4 and extra 20 at a cost of £14, we did not mention that there is actually a way to know that is the case (or not) for sure (at least statistically speaking). And the vehicle for that is the very likely known to you google ads’ Drafts and Experiments. This tool will allow you to A/B test any campaign in your account splitting search traffic equally among the control and variant (or original and draft campaign in google ads jargon) and peek into what would have happened to your metrics (spend and conversions) had you have the draft instead of the original in place. The difference between your draft and original campaign will give you your incremental spend and incremental conversions. Careful though, if we don’t equate statistical significance into our test, it will be as informative as doing nothing. Moreover, chances are that you will hardly ever get statistical significance at the campaign level and you will have to get less granular in your test (i.e. consider starting at account/group of campaigns/portfolio level).
Thence it eventually boils down to the following steps (and now we enter into the practical application of the article):
Scenario AnalysisCreate multiple drafts and experiments at scaleMonitor the integrity of the testCollect the data in a suitable manner and get it ready for measurement.Evaluate the TestExploratory AnalysisWrap up and prepare for future incrementality tests.
Scenario Analysis
Create multiple drafts and experiments at scale
Monitor the integrity of the test
Collect the data in a suitable manner and get it ready for measurement.
Evaluate the Test
Exploratory Analysis
Wrap up and prepare for future incrementality tests.
This is planning, playing with numbers and forecasts phase. You should start with a hypothesis as in any good experiment and that hypothesis should be actionable (i.e. its confirmation or not should make you take either action A or action B) and measurable with a good degree of accuracy (i.e. you must be able to generate enough data to reach a statistically significant conclusion and there shouldn’t exist any technical, commercial or business blockers preventing you from generating such data). Let’s get back to our example in which we had a ppc account generating a total of 100 conversions with an average cpc of £6. An example of a good hypothesis, in this case, could be:
You want to find out if your incremental conversions cost you more than £15 (because you breakeven at £15 or because you want a marginal ROAS of at least 2 times to compensate for all other costs you have for example).
If you confirm your hypothesis (incremental conversions > £15 per conv) the action you will take is to decrease your bids up to the same bidding level you tested in this experiment.
If you disprove your hypothesis (incremental conversions < £15 per conv) the action you will take is to maintain your bid levels.
With this in mind you’ll now run some scenarios playing with the variables of the test:
Potential metrics of the test: lower funnel conversions may provide you with the answer you’re most interested in, but upper funnel conversions are more frequent hence meaning shorter test durations. The more suspicious you are in the effect that varying bids might have into the type of visitors that you ppc account attracts the more reason to make the effort to have lower funnel conversions as your test metric, otherwise upper funnel conversions might just do it;
Variation in bidding levels: it’s more interesting to test bidding levels closer to each other like a 10% change, but you might not have enough data to accurately validate such small difference;
Assumptions to what happens to spend and conversions when changing your bids by the levels defined above: if you are decreasing you bids naturally your spending levels will decrease as well, but by how much one might wonder? You should put forward a few assumptions in advance and check what impact in your estimates and confidence levels they will have “if you were to observe such difference at the end of the test”. You want to ensure beforehand that the design of the test is robust enough to withstand different outcomes and still be able to provide you with the answer to your initial question: is my incremental cost per conversion above or below £15? And for that, you will want that your confidence intervals don’t overlap £15.
Time to run the test for: the longer it is the more data you gather and your confidence intervals become smaller, however running a test has always a cost associated with it being it at least number of conversions sacrificed or opportunity cost of not running other tests. The more important is the answer to your question the more willing you will be to extend the test duration.
The more options you consider for the 4 variables described above the more scenarios you will end up with so we want to strike a balance between a comprehensive mix of scenarios and not getting overwhelmed by them. Back to our example, we have created some scenarios ourselves as well. We have considered two metrics; two variations in bidding levels; two different assumptions to what happens to spend and conversions when varying each of the bidding levels; and two test durations. All these combinations resulted in a total of 16 scenarios as per the table below:
The first thing that is important to explain is that you will generate a lot more “spend” data than “conversions” data, hence your spend confidence interval will usually be very close to its point estimate meaning that your test accuracy will mostly depend on your conversions data.
Secondly, the confidence intervals calculation depend not only of your assumptions but also on how much data you will generate (you should look into your account’s historic to have an idea of how much that is in your case). In the “Evaluate the Test” section we dive deeper into confidence intervals calculations which you can then also apply when calculating the expected CIs in the scenario analysis.
So, once we have all these it’s time to make decisions regarding the final design of our experiment and looking at the scenarios in the example it seems that the following are not possible:
A 10% variation in bids doesn’t seem to be an option. Even if we were to run it for 2 months and used leads as the test metric, the confidence interval in the scenario in which we assumed spend to decrease by 15% and conversions by 7% would overlap £15. We could end up in a situation by the end of the test in which we wouldn’t have either confirmation or disproval of our hypothesis hence unsure of whether to take action A or action B.
Running a test based on purchases for the duration of 1 month seems to be out of the table too (in our spend 50% down and conversions 25% down scenario the confidence interval overlaps £15 by a lot — [£8.8 ; £18.8]; and using the other assumptions the confidence interval just touches £15)
Hence our options would be either:
Run the test for 1 month and evaluate it on Leads (both spend/conversions assumptions end up in a confidence interval clear of overlapping £15)
Extend the test duration to two months and turn Purchases into your metric (again CIs clear of overlapping £15)
Which one you end up choosing depends on what’s more important for you in the tradeoff of test duration vs lower funnel conversions.
At this point you have chosen the design of your test and here is where we need to get a bit more technical as we will be introducing code samples to tackle this step. Python being the data language of election will be what will be using, plus we will be interacting with the adwords API. So to start with you will need to connect to your account’s AdWords API. If you haven’t used the API before, the first thing you have to do is to get a Developer Token; a Google Client ID (this is different from the Client Customer Id — the one you actually see in google ads user interface with a number similar to this one: 562–425–3085); Google Client Secret; and Google Refresh Token. There’s very useful documentation in how to get started with the adwords API and generate all of the above in here.
If you have succeeded on the above then the following script should connect you to your adwords API:
The next thing you will probably have to do relates to budgets. If you are not using any shared budgets in your account you can ignore this next bit, but my bet is that at least some of your campaigns are sharing budgets. If that is indeed your account’s case then the next bit sets to address one of the drawbacks of using Drafts and Experiments: you can’t create a draft for a campaign that is using a shared budget. This means that you will have to create individual budgets and assign each of them to your campaigns. Fortunately, you can also do this programmatically, which will save you lots of time when doing it for all the campaigns in your account that you proposed to enter the incrementality test. Once you decide what is a sensible amount for each campaign’s individual budget, the following scripts will help you create them:
Now is a good time to introduce why google has probably named this feature Drafts and Experiments and not juts Drafts or just Experiments, since creating a draft is next on the list. In a nutshell, a draft when created is a duplication of an existing campaign that will always remain linked with its original campaign (i.e. can’t run on its own independently) and that will see some component of it get changed vs the original campaign (in theory you don’t have to but that would seem pointless); whereas an experiment is the settings that will determine in which circumstances and for how long a test between a draft and an original campaign will happen (what is the split of traffic, if this split is made on a cookie or search basis, what’s the start date, etc.). A draft is a campaign, an experiment are settings.
So next on the list is to create the Draft, which can be accomplished by running the script below:
And surely the most important thing to note is the information returned by this create_draft() function as we will need it afterwards to make changes to the draft campaign and also to create the experiment. And that is exactly what we will be doing next: changes to the draft campaign.
As in the examples earlier above we will consider that you are operating on a target_cpa smart bidding strategy and that you want to find out what happens when you drop your targets by 30%. The code below will help you with that.
The code above is doing two things, firstly it creates a target cpa portfolio with a target that is 30% less than the portfolio that is assigned to the original campaign, and secondly, it assigns that newly created portfolio to the draft campaign we created in the step before. There are a few additional things though that we should bear in mind:
If a multitude of your campaigns is assigned to the same portfolio then you wouldn’t want to create a single portfolio for each draft campaign. Instead, a more sensible approach would be to create one portfolio for each group of draft campaigns that share the same target cpa and assign that portfolio to all the drafts with the same target. The exact code to do it would be slightly different but would follow a similar logic.
You might have ad groups within some campaigns that have their own individual targets. If that is the case you will have to ensure that the draft campaign corresponding ad groups will have their targets dropped by 30% too (in this example). Otherwise, you won’t have a fair test as some ad groups won’t be dropped in the same order of magnitude as others. You would need an additional script to do this operation.
Once the above is done, we’re off to the final step of making an experiment live using Drafts and Experiments. Not a complicated one, you just need to grab the draft_id an campaign_id returned by the create_draft() function, then chose an experiment name (ideally something along the lines of a bit that identifies the original campaign and a bit that identifies the whole group of experiments that are part of this test like a suffix: campaign_123-decrease_30_test for example), a split percentage (recommend 50%) and a split_type (recommend Cookie). The following code will then handle the final operation of this section:
When you are running a test there is nothing worse than seeing it being invalidated, particularly for totally unavoidable reasons. This section is about the watchpoints of things that can go wrong if we are not careful and that ultimately would invalidate this test’s data (these incrementality datasets) that is so precious for our end goal. The crux of the matter is that the only setting that should be different between your original and draft campaigns is the bidding levels. So:
If you have any team member or tool that updates ad copy in a systematic way across the account, make sure the same changes are applied to original and draft.
If your landing pages are changing by any reason, make sure that the end of the day original and draft have the same landing pages.
If you’re adding bid modifiers to your campaigns, you need to add those to the drafts too.
As previously mentioned, if your campaigns have ad group targets, they’ll have to be reduced/increased by the same proportion the main campaign targets has been and you need to ensure they keep the same way throughout the test. I have added some example code of how you can quickly make this comparison, in particular, leveraging the adwords API:
I think you got the point already, but for comprehensiveness here are some other settings you might want to bear in mind: Budgets, Location targeting, audiences, networks, ad rotation, conversions, new ad groups or keywords, negative keywords, extensions.
Ultimately it’s a balance of either trying to avoid making many changes to the campaigns that you have created a draft and experiment for; or having a robust system in place that will ensure changes applied to original are also applied to draft.
Finally, one extra note to close this section: Did your portfolios in the draft campaigns went into learning mode and behave «funny» in the first few days? If yes, consider excluding these days from your incrementality dataset.
So you’ve launched your test and you made sure it’s running on fair terms, the next thing you want to do is to gather and prepare the test data in a manner that’s suitable for analysis and evaluation. In line with what we have done so far we will resort to adwords api to tackle this step of the process as well.
But first let me ask you, have you heard about fivetran? If not please do yourself a favour a go implement it straight ahead! Assuming you currently use fivetran or you went on implementing it and came back to the article, we will use the google ads fivetran connector to get ourselves the data we need. The alternative would be to use the AdWords Query Language (AWQL) directly, but fivetran handles all of that for us with just a few clicks.
So, using fivetran what we need to do is to create a new google ads connector and sync a Campaign Performance Report into your database of election (psst... snowflake is the best one) choosing the following fields as per the image below:
AccountDescriptiveName
CampaignId
CampaignName
AdNetworkType2 (Not needed for test analysis but useful for deep dives)
Date
Device (Not needed for test analysis but useful for deep dives)
Clicks (Not needed for test analysis but useful for deep dives)
Conversions
ConversionValue
Cost
Impressions (Not needed for test analysis but useful for deep dives)
SearchImpressionShare (Not needed for test analysis but useful for deep dives)
And finally, once this data has made its way into your database, the following query will organise it in a way that’s suitable for test evaluation (Such query will serve as the backbone for the evaluation of the test):
At this stage, your test is coming to an end or is already finished. You have collected and prepared your data in a way that is now suitable for analysis and that’s exactly what you’re going to do now. But how? There’s not one single right to way to it, but we propose that we use a statistical technique called bootstrap resampling (which is actually not so different from what google itself uses to report their confidence intervals — they use jackknife resampling though). The reason why we need to do this has to due with google only reporting confidence intervals on individual campaigns’ experiments, so if you are running an experiment on a multitude of campaigns you will have to calculate such confidence intervals yourself as if they were one big campaign because they are in fact all part of the same experiment.
The end goal of this stage is not just to find the difference in spend and in conversions between the test and control groups (because that is relatively easy — we just need to look at total spend and conversions on both groups of the experiment) but also to find a confidence interval for that difference hence the resampling. By resampling we will be taking a sample (pass the redundancy) with replacement of n observations of the performance of different campaigns on different days and compare the differences between test and control (n being the total number of rows yielded by the query provided in section 4 — i.e. total combinations of campaigns/days). We will be repeating this process a multitude of times (usually between 1,000 and 10,000) and for each of them comparing the difference between test and control. Having this set of samples we are now able to calculate the point estimate (the mean of the resample’s means) and the variance of performance amongst this set of samples and hence calculating our statistics’ standard deviation as:
σx = ( ∑ ( (xi -x) ^ 2 ) / j) ^ (1⁄2)xi, being the mean of each resamplex,  being the point estimator (the mean of all resamples' means)j,  being the number of resamples
And now having both the variance and the observed difference between control and test we are able to calculate the so much wanted confidence interval as:
CI = x ± 1.64 * ( σx / n ^ (1/2) )x,    being the point estimator (the mean of all resamples' means)σx,   the standard deviation of the samples' meansn,    the size of the sample (i.e. combinations of campaigns/days)1.64, Zα for a 90% Confidence Interval
Finally, we can say with 90% of confidence that the true difference in performance at different bidding levels in our ppc account lies between the lower bound and the upper bound of the confidence interval and make business decisions accordingly — back to our example in the scenario analysis section this means that if the confidence interval is below £15 we will maintain our bidding levels, however, if it is above £15 we will decrease our bids by 30% (as the marginal cost of the marginal conversions would be above our threshold of profitability)
The actual code to do make such calculations using python and a dummy dataset mimicking one prepared accordingly to the methodology proposed in step 4 are as per below:
If you are interested in finding out more about this resampling technique and the theory behind it I really recommend this article.
This is probably one of the most interesting and joyfully stages of this whole journey. You have successfully identified what’s the incremental value of going from bidding level x to bidding level y and gave your business crucial information to make a decision on whether to keep the biding levels or change them depending on the incremental value of your incremental spend. You have done this for the overall account potentially if your doing it for the first time or if you don’t generate a lot of conversion data; or you have gone further and did it for multiple portfolios, groups of campaigns or others (meaning that your test was actually a set of tests and you took actions on each of them accordingly) — for simplicity though, in this section we will assume you did an overall account test.
Nonetheless having a clear hypothesis that we want to answer being the main goal of this venture, a ppc incrementality test should also create many byproducts that you won’t want to ignore. The byproducts might need to be planned (for example if you purposely breakdown your campaigns into targeting locations where it is raining at the moment and locations where it isn’t raining — farfetched in most cases but for instance if you have a windshield business or you sell snow chains this might be important for you) but usually they will be available in on or more of the many adwords report types and in many cases they will even be unintentional and only found after careful post test analysis. So let’s dig more into this second option: I will give a couple of examples in bigger depth and in addition suggest a list of factors that might have an interesting impact into different incrementality levels within your account — your job is to look at all of them and find out which ones are more decisive for your business.
Example 1: Impression share can play a critical role in incrementality. Imagine you sell two types of t-shirts in your website — A blue and a red one (and you have a ppc campaign for each of them). They’re both sold at the same price but the blue T-shirt’s conversion rate is 40% and the red’s is 10%, hence you’re willing to pay 4x more for a blue T-shirt ppc click than a red T-shirt click. In this case you’ll end up with the same cost per conversion on both campaigns but you’ll likely have a higher impression share on the blue campaign (because you’re paying more per click). What’s important about this is that usually, it’s difficult to go after the extra bit of potential customers that you’re not showing ads to when you have 80% impression share than going after a few more of the yet so many available potential customers when your impression share is 20%, hence (rule of thumb) the higher the impression share the higher is your marginal cost per conversion and this is particularly noticeable on both extremes of impression share. Below is a real life based example of the role of impression share in marginal cost per conversion:
Example 2: Your main competitor offers their services both via web and via an app and to attract customers they use both google search ads and universal app campaigns. This means that your competitor will have a wider range of possibilities and a higher probability of competing for the same search keywords in a mobile device than in a computer. The competitive landscape will be more favourable in computer devices and all things equal you would expect your marginal cost per conversion to be higher on mobile devices than in computers.
List of factors potentially playing an important role into incrementality:
Impression Share (and other measures of competitiveness like average position, absolute top of page, etc.)
Device
Location
Audiences (People who have interacted with your website vs. people who haven’t registered any activity with you; Gender; Demographics; etc.)
Campaign Type / Ad group type
Keyword Match Type
Time of Day & Day of Week
Network
Demographic factors like age and gender
Others like weather (mentioned in example above); occurrence of significant event (for newspaper business); etc. As many as your creativity can get you too
Finally, what’s still most important about these analyses is that they should and most likely will inform which tests you will run next. Let’s say again this was the very first test you ran and you had no preconceived idea of which factors would most contribute to incrementality within the account. After you run such exploratory analysis you will be able to formulate a more robust hypothesis and ensure you design a test to statistically prove (or not) that. For example, you might be able to group campaigns into four groups because you observe a pattern and design a test that will ensure you find with precision their incrementality levels and shift spending across them according to the test results; or you might design a test-focused in confirming (or not) that campaigns/ad groups / or devices with higher impression share will be less incremental hence shifting budget to the ones with lower levels of IS.
If you have made it this far then congrats, you’ve done it — you have optimised your paid search account in an incrementality framework. Nevertheless, now it’s also the time to think about the future and how you are going to keep operating in an incrementality world. Below a list of the 3 most important things you want to consider at this wrapping up stage:
We hope that the test has turned out to be conclusive and that has led you to take either one or another action, but even if this is the case these findings might not hold true for long: Auctions’ competitive landscape is ever-evolving and hence marginal cost per conversion will keep changing. You need to plan in order to keep a reasonably constant level of testing inbuilt into your accounts so that you can have accurate and updated answers to guide your optimisation goals.
You need to decide if the next set of tests you will run will be made on the same dimensions as the first one (i.e. account/portfolio/campaigns grouped by a given logic, etc.). If this is the case you still want to sense check if there’s anything you want to change about the test setup (could you run it for a shorter period of time? Given what you now know, should you change the test metric? Etc.). Alternatively, you might have found something interesting in the exploratory analysis that you want now to confirm (statistically speaking). For example, in the next test, you might group your campaigns into low/medium/high impression share and test increasing/decreasing different bidding levels for the different groups; or you might want to test different device multipliers for mobile/tablet/computer instead of changing campaign’s bids.
Finally, at some point, you will also want to consider some sort of automation in terms of implementing and measuring these tests.
This article covered the reasoning behind paid search incrementality, why you should implement it in your marketing optimisation routines, and a practical way to indeed implement it; but instead of using this section to stress any of those points, I will save it to talk about the impact which at the end of the day is what really matters. Hope this will be the extra bit of motivation you need to actually go for it and implement it on your paid search account. So let me end talking about a startup in which we implemented this methodology and what it meant for the company:
Context: This was a company in a high growth phase, hence historically, they were willing to break even on cost per conversion and revenue per conversion (on average). On a given period the company started to feel the need to show signs of profitability so google ads could no longer run at break-even cost (on average). Hence the challenge was defined as: Moving from breaking even on average to breaking even marginally.
The methodology was to run a series of tests from bidding level x to y to z (each of them following the 7 practical steps defined above) to find the point in the bidding level curve where the marginal cost per conversion was the same as the up to the moment average cost.
Results were: 1) For each 10% we saved in total spend in paid search we only sacrificed 4.2% of the total conversions generated by paid search. 2) From breaking even in paid search to a return on ad spend of 233%. 3) No single conversion turning a loss to the company.
Leaving a note on potential alternative ways to tackle this challenge other than the Drafts & Experiments route. The first one is to use google ads bid landscapes — these are basically estimates that google makes about spend, conversions and conversion value were you to have different target cpas or target road. You can use these in the google ads UI for a one-off or programmatically via the API: The main drawback is that is only available on an ad-group level which may not generalise well for your use case and in addition the ad group needs to meet certain criteria. If this becomes available via the API on campaign or portfolio level this can become really powerful though. The second alternative is to use the trafficEstimator API — Haven’t used it myself because I have been working with smart bidding mostly but if you use manual bidding this tool should help you calculate how many clicks you’d generate for different bidding levels(max cpcs in this case) and if you assume a flat conversion rate from click to conversion you will be able to calculate you marginal cost per conversion from bidding level x to bidding level y.