Experiment design is the most critical, and arguably the most difficult, part of the incrementality measurement process. Even the simplest hold out tests can become quite complex when all the unique factors of each platform, data source, and testing environment are considered. If experiments aren’t designed to be statistically significant, have minimal noise, and deliver results that can be trusted, an investment in incrementality measurement is an investment wasted.
In the absence of an in-house data scientist for quality control, marketers should consider an experimentation partner that understands all the nuances of every channel and stays on top of ongoing changes in platforms and policies that will impact what’s required for valid and effective experiment designs.
Know this before designing an experiment
Calculating true contribution of media requires real transaction data. To measure and reveal the direct impact of advertising on business outcomes, experiments need to be anchored on system-of-record information from the business (such as transaction data from commerce platforms like Shopify, BigCommerce, or Salesforce).
Test and control groups do not need to be the same size. No one wants to spend half their budget on an audience that doesn’t actually see their ad. If properly segmented, control groups as small as 10% of the test audience can still be statistically significant.
Not all platforms are the same. Each ad platform has very specific ways to activate audiences and market to them. Experiments have to be designed around these campaign specific levers to ensure all relevant variables are controlled.
The cost and length of time to get valid experiment results depends on what’s being measured, which testing methodologies are required, and the experiment design itself. For example, a simple split test using platform audience targeting may return insights within days, whereas a multivariate geo-experiment typically needs to run a minimum of four to six weeks to get a clean initial read with actionable insights.