Experimentation & Measurement for Search Engine Optimization
By: Brian De Luna
Tags: Causal Inference, Experimentation, SEO, Data, Data Science

We hypothesized that this landing page would increase relevance with its clearer content, and decrease page load time with its lighter code structure, among many other improvements. This would consequently lead to to a higher ranking for our page on search engine results.

But since we can’t know the exact ranking of our pages, we rely on traffic as a proxy for an increase in rankings. That is, when our ranking for our San Francisco search page jumps, we’d expect to see an increase in the traffic to this page from search engines. But how can we measure this effect?

Limitations of A/B Testing

Our Growth team leans heavily on iterative experimentation for nearly every product change to make sure that we can measure effectiveness and to learn as we build. Most data scientists are able to leverage a traditional A/B test at the device- or user-level for all of their experimentation needs. In this setup, users that enter the experiment are randomly bucketed into the treatment groups, and we can directly compare the outcome of the treatment group with that of the control group.

A/B tests have very good power and allow for complete randomization. At Airbnb, they are useful for measuring the treatment effect on metrics pertaining to engagement and conversion. These are events that we log on the Airbnb site, and we can easily measure the incremental uplift in these metrics through a difference-in-means hypothesis test, such as a t-test.

However, in the case of our new “Magic Carpet” page, an A/B test will not allow us to measure an increase in traffic caused by a change in its external search engine ranking. A given page will look different across various search engine bots, and therefore we cannot isolate the effect of Magic Carpet on our rankings.

Therefore, quantifying the impact of such a product change requires a more sophisticated approach.

Leveraging a Market-Level Approach

A key realization is that our search results page isn’t just a single page; in fact, there are many different versions for different cities, towns, and regions. Each of these have what is called a unique “canonical URL”, and we actually have over 100,000 of them that are surfaced on search engines! Therefore, instead of assigning a single visitor to treatment or control, we can set the unit of randomization of our experiment to be a specific canonical URL. We’ll then measure the effect using an approach commonly used in market- or cluster-level experiments.

For example, the San Francisco search results page may be in treatment, and we’d update it with our Magic Carpet design. Meanwhile, the Paris page in the control group will be left the same. This random assignment will then continue to be applied to each of our ~100,000 URLs. This way, when a search engine bot scrapes our site, it will consistently see the same treatment for each page, and the ranking for that page will update accordingly.

However, a more nuanced statistical approach needs to be taken in this case, because we cannot simply make a direct comparison between the traffic of treatment and control URLs. This is because the baseline traffic between different URLs can differ substantially, and many times this difference is larger than the treatment effect we wish to detect. The San Francisco page may have a similar amount of traffic as Paris page, but it probably has something like 100% more traffic than the page of a smaller city, such as New Orleans. This makes it very difficult to measure something like a 2% lift in traffic!

For this reason, we need a mechanism to account for the inherent differences that exist between these URLs by leveraging pre-experiment data, before the change took effect.

Developing a Model: Difference-in-Differences

A difference-in-differences framework is one technique that utilizes pre-experiment data to control for these baseline differences in the absence of any interventions. We can use this method to measure the treatment effect and its statistical significance by using an estimator from a linear model, where for each page i and day t:

Our principle variables include:

traffic_it = number of landing page impressions to page i on day t. We apply a log to this outcome variable to account for its right-skew, and to regularize the heteroskedasticity commonly present in traffic data.

treatmentᵢ = treatment group indicator (equal to 1 if in the treatment group, 0 otherwise)

post_t = pre/post-period indicator (equal to 1 if in the post-period, 0 otherwise)

However, there is still a lot of variation across time and markets in our traffic data that may hinder us from detecting a treatment effect. The difference-in-differences approach allows for an elegant solution to this problem; we can simply add in covariates to our model in order to control for the various effects:

aᵢ = fixed effect (or mean) for a page, to allow for flexible intercepts of each URL

t = time index, to account for overall time trends

dowⱼ = weekday indicators, to account for weekly seasonality

Since we’d like to know the effect of the treatment group in the post-period, the b₂ coefficient reflects the “difference-in-differences”, and therefore the treatment effect that we wish to estimate.

In simpler terms, we’re looking for the incremental effect on the treatment group after the experiment started.