What is your sample size? The dangers of pseudo-replication
By: Wicaksono Wijono
Tags: Statistics, Analytics, Machine Learning, Business Intelligence, Data Science

What is your sample size? The dangers of pseudo-replication

Are your uncertainty estimates correct?

Simple, right? In introductory statistics courses, sample size is always given. n = 1000. Or n = 20. But in the real world, you have to figure out n by yourself.

If the data is flat, then n is the number of rows. Easy.

But much of real world data is not flat. What is n? What is love? What does the fox say? What is the meaning of life, the universe, and everything? These are the questions that keep me up at night.

We will start with an overly simplistic example. At first glance, the question “what is n?” sounds absolutely useless, but hopefully towards the end you’ll see that this is a ubiquitous and practical business problem. It is common enough that we have ten thousand rows of data, yet the “sample size” is closer to ten.

The “sample size” plays a direct role in the quantification of uncertainty. A point estimate is not enough; what is the plausible range of values?

Flat Data

Imagine you want to find the average height of students at a school. I’m an oracle and know that the true average height is 167 cm, but you don’t know that.

You secured 10 volunteers, which are representative samples from the normally distributed population (hey, I said imagine). You take out your measuring tape and record, in centimeters:

> 151 180 156 158 147 163 166 161 167 171

In this case, it’s simple. n = 10 and you can build a 95% confidence interval of (155, 169) around 162. 167 is within this CI.

Given the setup, 95% of the random samples you take will result in a 95% CI that includes 167.

Imbalanced Data

You bought your measuring tape from a discount store and you were told it’s cursed: it provides measurements that are slightly off but correct on average.

Not pictured: the Eldritch horror that lies beyond. (source)

It comes with a second measuring tape, for free! The second measuring tape is also cursed in the same way. But you can choose the color!

You chose yellow because you’re a boring person. (source)

As you announce that you will be taking measurements a second time, half of the students said they’ll be late for class and hurriedly left. You used the second tape to record the heights of the first five students, so now your data looks like

> 151 180 156 158 147 163 166 161 167 171

> 153 187 159 157 145

You have 10 students but 15 measurements. This is pseudo-replication — you have collected more data, but the number of sampling units (students) remains unchanged.

How should you estimate the average height? If you average the 15 measurements, then some students get double-counted. Your level if uncertainty will be off because your observations are no longer independent. The measurements of a student depend on the student’s true height.

The correct way is to take the average measurement per student so your data looks like:

>152 183.5 157.5 157.5 146 163 166 161 167 171

And then you take the average of this to get an estimate of 162.4 — average of averages, so to speak. (If you’re not doing this for your A/B tests, you either have an alarming problem or have a better and more advanced method.) But how do you build the confidence interval? You are more certain of the first five values, but are we going to pretend that the latter five carry just as much information?

For that matter, what is n? What is the correct degrees of freedom of the model? We can’t answer that (well, actually, we can, but only for iid Gaussian errors) but it’s better to pretend that n=10 rather than n=15. We’d expect the answer, if it exists, to be somewhere between 10 and 15. We’ll get back to this later.

Balanced Data

After class ended, the remaining five students came to you to get their second measurements taken (hey, they’re nice folks). You maniacally laugh at your now complete dataset, and you can’t wait to unleash its potential in unlocking the secrets of the universe:

> 151 180 156 158 147 163 166 161 167 171

> 153 187 159 157 145 164 163 161 166 172

You take the mean and standard deviation of all observations and construct the 95% CI with n = 20 to get the interval (157.6, 167.1) centered on 162.4.

Pictured: the average scientist after performing a hypothesis test. (source)

Then lightning strikes you because the universe doesn’t want this kind of faulty analysis to see the light of day.

What’s wrong?

Imagine that you repeatedly take measurements three more times and record these observations:

> 151 180 156 158 147 163 166 161 167 171

> 153 187 159 157 145 164 163 161 166 172

> 154 187 156 156 146 163 164 160 168 169

> 152 180 159 155 146 165 162 161 167 170

> 152 183 158 162 145 166 165 158 165 172

Treating these as 50 independent observations (in other words, ignoring pseudo-replication) will result in a very narrow 95% CI of (159.4, 165.1) centered on 162.3. The true value of 167 lies well outside of the CI.

If you repeat the setup with different students, 167 will lie outside the interval much more often than the desired 5% of the time.

The problem is this: you are observing the same 10 students. Why would seeing them over and over again make you that much more certain of the population mean? If that worked, why would people spend so much money collecting data when they can stare at the same thing many many times?

Given that the measurement errors come from the same normal distribution (i.e. the errors are Gaussian, independent of the students’ true heights), the correct approach is to take the average of averages:

> 152.4 183.4 157.6 157.6 145.8 164.2 164.0 160.2 166.6 170.8

Because we only observed 10 students, we construct the 95% CI with n = 10 to get (154.9, 169.7) centered on 162.3.

In reality, very rarely is data ever balanced. Even when you set up a study where the end goal is to get balanced data, participants can and will drop out.

If the bolded error assumption is not reasonable (e.g. binary outcomes) then we cannot do this and have to use more complicated models:

Bayesian machine learning to the rescue!

At its core, Bayesian ML handles pseudo-replication (though it also works for flat data).

This doesn’t sound remarkable until you realize that the overwhelming majority of data suffers from pseudo-replication. Businesses have repeat customers and take (hopefully) multiple measurements from each customer. Studies might want to track the same people over time.

hierarchical structure of data

Remember how we squashed 15 observations into n=10? You can have a dozen stores with thousands of customers each, but once some questions are posed (“What is the average uplift from the promotion?”), you virtually have to squash 10,000 observations into n=12. Either that or you have to pretend that your dependent observations are independent. Not very pretty.

Most of the popular machine learning algorithms (xgboost, random forest, GLM, etc.) do not innately handle pseudo-replication. There are two possible options. You can properly randomize the folds in cross-validation (observations from the same student should never go to different folds!) and hope the results can be generalized. Or, you can aggregate the data (e.g. average the measurements by student) and discard some information.

Here’s the R code I used to generate the data above:

num_students <- 10

num_pseudoreps <- 5 set.seed(12)

height <- rnorm(num_students, 167, 10)

meas <- matrix(0, nrow = num_pseudoreps, ncol = num_students)

for(i in 1:num_pseudoreps){

meas[i,] <- round(height + rnorm(num_students, 0, 2))

}

To get frequentist-like estimates, I will use MCMC with extremely weak / uniform priors to fit a random effects model. In practice these priors should never be used outside of basic examples.

library(rstan) balanced <- data.frame(

student = sort(rep(1:num_students, num_pseudoreps)),

measured = as.numeric(meas)

)

unbalanced <- data.frame(

student = c(1:10, 1:5),

measured = c(meas[1,], meas[2, 1:5])

) model <- stan_model(model_code = '

data {

int<lower=0> N;

int<lower=0> S;

real y[N];

int<lower=1, upper = S> students[N];

}

parameters {

real mu;

vector[S] mu_s;

real<lower=0> pop_sigma;

real<lower=0> meas_sigma;

}

model {

mu ~ normal(0, 10000);

pop_sigma ~ uniform(0, 1000);

meas_sigma ~ uniform(0, 1000);

mu_s ~ normal(mu, pop_sigma);

y ~ normal(mu_s[students], meas_sigma);

}

') set.seed(1)

mcmc <- sampling(

model,

data = list(

N = nrow(balanced),

S = num_students,

y = balanced$measured,

students = balanced$student

),

iter = 5000

)

posterior <- mcmc@sim$samples[[4]]$mu[1001:5000]

Using the balanced data with 50 observations, the posterior distribution has a 95% credible interval of (154.2, 170.2) and mean of 162.3. This is slightly wider than the CI we got earlier, partly because we are estimating one additional nuisance parameter: measurement errors.

The magic happens in the unbalanced case. The 95% credible interval is (154.4, 170.0) and we don’t have to make guesses about what n to use. The benefit is not clear in this example because we’re working with the setup where ignoring pseudo-replication is least harmful.

However, as we drift further away from the comfort of :

* Gaussian distributions (e.g. binary outcomes, heavily skewed population distribution)

* uncorrelated parameters (e.g. loyal customers are likely to visit more and spend more)

* somewhat balanced data (e.g. some customers make a hundred visits a week while others make a single visit)

going with average-of-averages will give progressively worse results. (Check out my GLM article to understand the details of precision-weighted averages.)

The iron is wingardium leviosa’d. (source)

Intuitively, it’s because average-of-averages pretends that all sampling units have equal weight even when they don’t.

Considering that some of the most universal metrics are percentages (e.g. conversion rate) or extremely right-skewed (e.g. average spending), Bayesian machine learning has a place in every organization.

In closing, if you often have to make statistical inferences at work, consider picking up Bayesian machine learning as a core part of your toolkit. They are practically designed to tackle the pseudo-replication problem that’s so prevalent in real-world data.