Does your product work?. Randomized controlled trials, imperfect…
By: Naim Kabir
Tags: Product, Statistics, Python, Data Science, Science

Does your product work?

Randomized controlled trials, imperfect compliance, and the counterfactual time machine

We build software to solve human problems. But human problems can be messy, and sometimes it’s not terribly clear whether or not we’ve actually solved them.

Snapchat might tell they’re successful if they see 50% of regular users check out their new dog filter, and Facebook could say they’ve shattered their growth milestones by showing they’ve achieved more than 2.3 billion monthly active users.

But what’s your acceptance criteria when your app is designed to help members cope with anxiety? What metric can you monitor when your software was built to cultivate mindfulness?

Or, in the case of my company—Even — how can we tell if we’re making any impact on the financial health of our members?

We have high engagement with our mobile app, we’ve sent our members more than a billion dollars’ worth of their earned wages, and we’re automatically saving millions for them in rainy day accounts. These are things worth celebrating.

But they aren’t enough.

We’re more directly concerned with if our members’ lives are materially improved because of our product. As a data scientist working to understand our efficacy, that’s basically all I care about.

This is where you go from metric optimization to something more like scientific experimentation. What I need to understand is our Average Treatment Effect — that is, the expected value of the impact we cause in our users’ lives.

To explain how I might do that, I’m going to dive deep into a kind of causal analysis broken down by Judea Pearl in his fantastic book and in this paper (co-written with Alexander Balke).

This analysis is highly relevant for performing experiments involving a product’s effects on its users, provided that you’re part of a data-driven organization where it’s relatively easy to track app usage patterns. Even so, I’ve yet to see even the most academic of studies in tech use the sort of thinking I’m about to dig into.

Quick word of warning: the deep dive is going to involve a bit of scientific method, a lot of probability theory, a bit of Python, and a dash of linear programming. You don’t need to be an expert in any of these topics to understand what I’ll be covering, but it will almost certainly require some time, effort, and maybe a bit of background reading.

I promise you, though, this is cool enough to be worth the challenge.

Let’s start with a science refresher: randomized controlled trials.

Randomized Controlled Trials

Correlation isn’t causation — and what we’re interested in is causation. We want to cause an improvement in our members’ lives. It’s useful to define what “causality” even is before I get all evangelical about it, though. The counterfactual definition is one I personally find to be clearest:

Event A “causes” Event B if and only if, were we to delete Event A from history, Event B would not happen.

To be a cause is to be the domino that sets off the chain reaction, the fuse that blows the dynamite. If you remove the cause, whatever would have come to pass no longer does.

To study causality, then, we can remove something we think is causal and see if its effects still arrive, everything else being equal. This is what a “controlled trial” is trying to do.

We naturally do controlled trials all the time. Think about the last time you tried setting up a new lamp, only to find that it didn’t work. How did you go about solving the problem?

You might’ve had a couple of ideas for what went wrong. The lightbulb could’ve been burnt out. Or maybe the outlet you plugged it into was a dud.

Let’s take one of those ideas and formally call it our hypothesis.

Hypothesis: the outlet is nonfunctional, which means no power can get to the lightbulb, and it can’t light up.

How do we test that hypothesis? Well, we have one trial under our belt where we tried the current outlet, Outlet A. This is our Treatment trial — where we’re treating our system with an input we think caused our ultimate dark and dreary state.

Now we need a Control trial — where we swap out Outlet A with an outlet we know works: Outlet B, which has been faithfully charging our phone for months. Everything in the system has been kept the same except the outlets.

Before we run this trial, let’s frame this experiment as formally as we can. What we’re looking for is the Average Treatment Effect (ATE) of Outlet A on our room’s illumination.

Treatment Effect = The probability of a non-functional lamp given that we use Outlet A, minus the probability of a non-functional lamp given that we use Outlet B.

We already know that P(Darkness|Outlet A) = 1. If it turns out that the room’s still dark when we swap out Outlet A with Outlet B, then P(Darkness|Outlet B) = 1 too and thus ATE = 0. That would mean the outlet was 0% the cause of our problem.

However, if the light instantly turns on when we plug the lamp into Outlet B, then P(Darkness|Outlet B) = 0, and our ATE = 1. Or in other words, Outlet A was 100% the cause of our problem!

So, let’s go ahead and plug the lamp into Outlet B. Bam:

…Surprise! We were talking about a lava lamp this whole time. Now the living room of your imagination is just a tiny bit more whimsical. Source: Warisan Lighting

The lamp lights up; Outlet A was the cause of the darkness.

Problem solved.

Like I said, we do this kind of thing all the time. The complication when it comes to doing a controlled trial on the folks who use our software is that there’s a particularly tricky variable involved: people.

While home appliances are (usually) completely deterministic, people are not. No one person is the same, and the way they use your product and respond to it will often surprise you. Some of them will download it and forget about it, some will log in every day. Some of your members might be helped by the product, but — and this is a scary but necessary thought — there are many of them who might be hurt by it.

How do you do controlled trials of your product when one of the inputs to your experiment can be so variable?

This is where statistics gets involved. We can no longer do a single Control trial and a single Treatment trial like we can with deterministic systems. We need to do hundreds of trials. Thousands.

Each trial will be on a different person. What’s important to enforce is that the criteria for assigning a person to a Control or Treatment condition is completely random. This ensures that the samples of experimental participants in each condition will more or less be identical in aggregate: they’ll have the same proportion of people from each gender, height, income level, disposition, etc. The effect of factors that aren’t the one under scrutiny should sort of just cancel each other out.

We’ve put quite a few pieces together by this point: we’re controlling for a single explanatory variable, and randomizing to mitigate the effect of a bunch of exogenous variables we aren’t studying. We now have a randomized controlled trial (or “RCT”, as the cool kids say)— the gold standard for studying what causes what.

But the story isn’t over yet. We can assign people to the Treatment condition (people assigned to use your product) and Control condition (people who don’t get to use your product), but who’s to say that your participants will actually comply with your assignments?

Who’s to say the participants in your Treatment group will actually download your software and use it, or that people in your Control group won’t find it on the app store and give it a whirl?

You can’t make causal attributions when you don’t know if participants actually got your treatment. So. How do we do that?

Dealing With Imperfect Compliance

Let’s recap our RCT set-up.

We randomly assign participants to either a Treatment or a Control group. They may decide to comply with their assignment, depending on some set of unobservable factors. They’ll respond in some way to your product (or the lack of it), depending on whether they took the treatment, plus some set of unobservable factors.

We can describe how these events interplay with a Bayesian Network, as we do below.

Bayesian Network description of an RCT: This diagram describes conditional dependences of the events we’d expect to see. Z is completely independent of anything else, since it’s done via random coin-flip. U is a catch-all variable for the whole universe of unknowns that affect X and Y, and is also conditionally independent from other variables. X would be conditionally dependent on Z and U. Y would be conditionally dependent on X and U.

We’ll use the Z, X, Y, and U symbols as shorthand for what random variables they represent in this diagram. For example, when we refer to Y, we’re talking about the Outcome variable participants may realize (either a Good outcome or Bad outcome). Using the lowercase, y, refers to an actual realization of that random variable (Good, for example).

This kind of diagram is just a convenient way to describe how we might compute the probability of some joint Z, X, Y, U configuration (for example, “Z=Treatment, X=True, Y=Good, U=some latent factor value” is one such state). In symbolic terms, the diagram is saying that:

Here are some intuitive things you might be able to say based on the diagram/expression:

Knowing X=x changes the probability that Y equals a particular y. Knowing Z=z and/or U=u changes the probability that X is a specific x. Knowing Z=z doesn’t give me any additional information about what u value U might be.

All we’re doing is capturing conditional dependencies here.

Our eventual goal, though, is to be able to calculate an Average Treatment Effect, which is written as:

i.e: The probability of having a good outcome given that we force a participant to take the treatment, vs. the probability of a good outcome given that we force them not to take the treatment. This difference is the ‘goodness’ directly attributable to our intervention

Pay attention to that do() operator, there. It’s saying that we want to know what the probability of a Good outcome is when we force X into a particular state — rather than just passively observing it.

The problem is, we never actually force anyone to do anything in an RCT. We give participants access to the product and then they do whatever they want with it. All we can do is passively observe!

Given this, a naive approach may be to declare that:

But this is a fatal mistake.

We have to keep in mind that different latent factors, represented by U, drove the cases where the participant took treatment vs. when they didn’t. That same U could very well have affected the probability of them having a good outcome, which would muddy our story for how effective the treatment was.

For example, in Even’s case, members who choose to regularly use the app may just be inherently more financially well-off than those who don’t (or vice-versa). In that case we’d see a high ATE, but that says nothing about the effect Even is having on people — it’s just a spurious result of selection bias!

What we want is to be able to compare the chance of a good outcome given treatment vs. the chance of a good outcome without treatment, where all other factors are fixed. That would look more like this:

Here we’re making sure that we’re only capturing the difference in P(Y = Good) between the treated (X = True) and untreated (X = False) groups given the same unobserved condition (the same u), for all possible unobserved conditions (all u ∈ U)

Alright, we’re kind of getting somewhere. We have an expression for how to turn experimental observations into an ATE… but the expression is useless unless we find some way to understand what u states are elements of U, so we can properly control for those states.

But U is this inscrutable random variable that’s supposed to capture the whole universe of unobserved factors that influence why a human being decides to try a product and how they might respond.

How can you even begin to break that down?

Let’s make the problem as simple as possible. We don’t need to model people’s neurophysiology, or the weather, or the stock market, or any of the infinite factors that can influence decision-making on the day-to-day.

Let’s just consider what things can affect the result of our experiment. We know U influences X and Y — i.e, U encompasses peoples’ Compliance behavior (did they use the product or not) as well as their Response behavior (did they get better or not).

We can roll up the myriad factors of the unobserved universe into a few broad types, then: archetypes that determine a participant’s Compliance and Response behaviors.

I enumerate a scheme like this below:

Compliance behavior types

always takers — they manage to access the product no matter how we assign them. compliers — they comply with the group assignment. Folks in the Treatment download the product and use it, folks in the Control do not. deniers — they do the opposite of what they were assigned to do. Folks in the Control find a way to use the product, people in the Treatment group never use it. never takers — they never access the product, regardless of assignment.

Response behavior types

always better — they would have a good outcome regardless of using the product. helped — they would have a good outcome if they use the product. hurt — they get a bad outcome if they use the product, but would’ve been fine without it. never better — they would never have a good outcome, regardless of their use of our product.

All participants in the experiment must have exactly one Compliance behavior type and one Response behavior type, because they’re defined as being mutually exclusive. So you can describe every single person as belonging to one (Compliance, Response) behavior pair, giving us exactly 16 archetypes that a person could possibly be.

I guess it’s kind of like Myers-Briggs — except this framework is actually useful for doing science.

It’s like a horoscope! Except it sorts people into groups that aren’t hot garbage

Great, now we know that U can take on a u value like (Always Taker, Helped) or (Never Taker, Never Better). The reasons why a participant might be of a particular u is unimportant for our analysis — we just need to know what u is. If we know the distribution of U we can actually formulate a far simpler expression for an ATE:

We’re marginalizing the compliance behaviors in this expression — P(U=helped) is actually a union of all P(U=u) where u has a ‘helped’ response type, for example.

But how can we know the distribution of U?

…Ok, so here’s the thing: we’ll never know what that distribution is. That’s impossible to do. How could you ever tell if someone would be “Never Better” in some experimental scenario? That barely even makes sense.

At this point you might be yelling at me through your computer for wasting your time. Why did we just go through all that troublesome deliberation, just to wind up stuck? A lot of studies take shortcuts that don’t involve U or X or anything as complicated as the diagram we’ve set up at all!

For example, a pretty standard practice in medical studies is to formulate an ATE as the following, instead:

This is called an intention-to-treat analysis, and it recognizes its own flaw right there in its name: it’s measuring the effect of an intention to give someone a treatment (an assignment to Treatment or Control), rather than if they actually took it.

You can kind of already tell why I’m not a fan of intention-to-treat, but to make my displeasure truly visceral, I’m going to simulate an experiment for you. Follow along in this notebook:

That was a lot of work we just simulated— we recruited some participants, got informed consent, followed them for some time, measured outcomes, queried our production database to get compliance figures, did some aggregation, etc. etc. At the end we got those 8 beautiful values: P(X,Y|Z=z) for each z in Z (Treatment and Control, in this case).

You know how in documentaries about spider silk and its amazing properties, the narrators will keep mentioning how they had to painstakingly milk 14,000 righteously indignant spiders for three months just to get an ounce of silk?

The true cost of scientific progress. Base image source: Pexels.com.

Our P(X,Y|Z) values are that ounce of silk. They are precious.

Intention-to-treat immediately obliterates them and rolls them up into two coarse numbers: P(Y=Good|Z=Treatment) and P(Y=Good|Z=Control).

What a waste of useful information! It’s like setting our ounce of spider-silk on fire and then also systematically slapping each of our 14,000 already-distressed spiders in the face.

How could you do this

Here is this tragedy of information annihilation in action:

There has to be some use for that rich P(X,Y|Z) information. Right?

The Counterfactual Time Machine

While we can’t ever know what the distribution of U in our participant population is, we do know that P(X,Y|Z) changes conditional on U. Or in other words there is some function U → P(X,Y|Z).

The function is maybe one-to-one or many-to-one — but in either case, knowing what P(X,Y|Z) is should be able to shed light on what U could have possibly been.

Unless it’s a trapdoor function, in which case we’d be hosed.

It’s not though! In fact, U → P(X,Y|Z) is a simple linear function, which you’ll be able to intuit yourself shortly. Let’s think this through:

Take the quantity P(X=True, Y=Good | Z=Treatment). This probability gives us the proportion of people in the Treatment condition who took the treatment and had a good outcome.

But another way we could get that value, assuming we had knowledge of the distribution over U, is by adding up the proportions of people who we know always take the treatment and get better, as well as the proportion who would comply with the treatment assignment and get better.

This lets us assert the following linear relationship:

As it turns out, every single P(X,Y|Z) value we have is a union of some set of P(U=u) values like in the example above, meaning that U is constrained by the data that we can readily observe!

And those aren’t the only constraints. We also have the rules of probability at play: each P(U=u) must lie in the [0,1] interval, and the sum of all P(U=u) must equal 1, as well.

On top of these constraints, we also have an expression involving U we want to learn about: the ATE, our holy grail. Here it is again:

Again, we’re marginalizing over the compliance behaviors in this expression — P(U=hurt) is actually a union of all P(U=u) where u has a ‘hurt’ response type

While we don’t have the tools to solve for exactly what the distribution of U is — what we do have are all the right ingredients for a linear optimization problem.

We can range over the constrained space of what U could have possibly been, and pull out the absolute best and worst-case ATEs. This gives us hard bounds on what the true ATE value actually was.

I set up these constraints and performed this optimization using PuLP, a Python library that lets you frame these kinds of linear programming problems with super readable symbolic expressions:

Pay close attention to what we just did. Using a few assumptions and the structure of our diagram, we were able to go back in time and see what would have happened if we’d lived in the happiest timeline we can imagine, with the highest possible ATE.

Using the same reasoning, we could also go back in time to explore the darkest timeline (if we minimize instead of maximize).

The darkest timeline. Source: Community.

If you were following closely, you might’ve also noticed something pretty alarming.

The intention-to-treat analysis — standard practice for analyzing most experimental results with human participants, remember — declared that our treatment had an ATE of 45%, a moderate positive effect on participant outcomes.

But when we take the treatment variable X into account, we see that the highest possible ATE is -15%.

The treatment hurts people.

Model-driven vs. data-driven thinking

Thinking in terms of a causal model rather than taking the data itself as gospel is the key to unlocking reasoning like this.

Instead of getting stuck with the set of 8 probabilities spat out by the RCT, we were able to reason that something was responsible for how our experimental participants took our treatment and responded to it. With that in mind, we could work out what the results might have been in a different set of circumstances.

I need to level with you here: in most cases, this analysis will get you the same directional results as an intention-to-treat study. I purposefully used a contrived example in which their results were divergent. But it’s important to realize that such a divergence can happen in the first place, and that it can be so dramatic.

I think a lot of people working in applied statistics get used to taking the data we see and throwing it into one of our many ancient and established machines. Or at least, I do. It’s a bad habit.

Contingency table? Throw it in a Chi-squared test and turn the crank. A 2x2 table and super low sample sizes? Throw that into a Fisher’s Exact test and turn the crank. A matrix of features mapping onto a continuous label? Throw it into a linear regression and

Turn.

That.

Crank.

It’s easy to forget that the reduction of data isn’t the end-all be-all of analysis. Understanding and making a mental model of the process that created that data unlocks a whole new set of powers. Powers you might have never realized you had.

Some of those powers let you do causal inference without needing to run an experiment at all.

But that’s a post for later.

For now, I hope I’ve added a sweet new tool to your toolbox — and better, a new way to think about your problems.