N Is The Enemy
By: Naim Kabir
Tags: Programming, Data, Tech, Science, Statistics

N Is The Enemy

Big Population + Big Data = Critical Failure

Photo by Joshua Coleman on Unsplash

We’ve been sold a false promise.

Somewhere down the line we tricked ourselves into thinking that truth was a side-effect of volume. “If we collect enough data,” we said, “our overwhelming statistical power will blow a hole in the unknown.”

Instead, we shot ourselves in the foot.

In his article Statistical Paradises & Paradoxes In Big Data, the Harvard statistician (and certifiable genius, as far as I’m concerned) Xiao-li Meng sets down a rigorous proof of just how bad we screw ourselves when we collect data without regard for exactly how it’s collected.

He draws upon mathematics that are elegant but dense — so to get his point across I’m going to resort to the stock-in-trade of the modern technology writer: Javascript simulations.

Javascript simulations that are gross oversimplifications in most respects, but are just correct enough to teach us a couple of lessons.

First, some background.

The Data Defect Index

A fundamental promise of statistics is this: Given a limited sample, we can generalize insights over the greater population.

This is a superpower. It’s why science can occur as a series of small controlled experiments rather than a complete census of the Universe. It makes learning about the world a feasible task.

But not all samples are created equal. Different samples are created by distinct sampling mechanisms: for example, a random telephone dialer, an online survey, etc. Regardless of what form a sampling mechanism takes in practice, we can think of them in the abstract as a binary mask over a population.

Let’s call this mask R and the population data it’s masking G. The mask-value Rᵢ — a 0 or 1 switch — decides whether or not the value Gᵢ is included in our sample.

A sampling mechanism R. The red Rᵢ elements select items of size Gᵢ to be in our sample. We have a sample size of 3.

In the ideal case, sampling mechanisms are completely random, and the correlation between Rᵢ and Gᵢ (CORR[Rᵢ, Gᵢ]) is 0. But oftentimes sampling mechanisms are slightly defective in some way.

To take the absurd and extreme case, imagine a survey designed to gauge a customer population’s eagerness to respond to surveys. If our sampling mechanism (R) is an email blast asking participants to opt-in and let us measure their eagerness to opt-in (G), we can guess that CORR[Rᵢ, Gᵢ] is going to be pretty close to 1.

A DEFECTIVE sampling mechanism R. Since G is sorted by size, an R where we just pick the first items will induce a high CORR[Rᵢ, Gᵢ], since we’re just selecting the biggest items.

That CORR[Rᵢ, Gᵢ] value — or more accurately, 𝔼[CORR[Rᵢ, Gᵢ]²], though it’s not important to be that precise, for now — is something we can refer to as the data defect index or d.d.i. This little number is the wrench in Big Data’s statistical works. As we’ll soon see, it has outsized effects on whether or not we can rely on insights from a large dataset at all.

Bouncing Balls

Let’s investigate the impact of the d.d.i with a toy example.

Say we drop N balls into a space. Some are bigger than the others, but each of them have the same amount of energy. So the big ones bounce slowly while the small ones zip around like excited hedgehogs. Ball size is drawn from a normal distribution.

Pretend you’re a ball-scientist (ballologist?) and that you want to estimate the mean radius of the balls in this little universe. Your plan is to sample n balls, collect a sample mean, and use that as an estimate for the population mean.

There’s a problem though: for whatever contrived reason, you can only sample balls as they fly into a designated sampling zone highlighted in red.

You can play around with the sampling zone below.

Drag slider to resize sampling zone. Sampled balls get painted red. n = 10; N= 75.

What do you notice?

As you slide the sampling zone to be more and more inclusive of the space, the error between the true average radius and the sampled radius goes down. If you slide left and make the sampling zone a smaller fraction of the space, the error goes way up.

There’s a simple reason for this: the smaller the ball, the faster it zips around. If the sampling zone is small, it’s very likely that only the speediest balls will make it there from their starting position. On the other hand, if the sampling zone is large, it’s more likely that we can sample equitably.

This makes the width of the sampling zone a handy analogue for the d.d.i. The lower the width, the higher the d.d.i!

High d.d.i drives up estimate error.

We obtain a fairly obvious finding from this experiment: High d.d.i drives up estimate error.

Sampling Frequency

Let’s extend this a bit to examine the influence of population size and sample size on our estimate’s error.

n = 10; N=750

When we 10x our population size N, our error is consistently worse for each setting of d.d.i. This should be expected. With a larger population of balls, there are more tiny speedy ones that end up sampled quicker than the rest, deflating the average measure of radius.

What happens when we commensurately increase our sample size n to help out our estimate?

n = 100; N=750

A 10x increase in n just gets us back to where we started, as far as error rates go.

When you have a defective sampling mechanism, it’s the sampling frequency that matters, not the sample size.

There’s an important lesson here. When you have a non-zero d.d.i, the absolute size of your sample is worth precisely jack. When you have a defective sampling mechanism, it’s the sampling frequency (f = n/N) that matters, not the sample size.

And if you’re gunning for absurdly high sampling frequencies to cover for a defective sampling mechanism, you might as well do a census.

n=500; N=750; f=2/3.

The Scientist’s Levers

At this point, you’re hopefully convinced that d.d.i can destroy estimates and that its only mitigating factor is an amped-up sampling frequency, f.

Dr. Xiao-li Meng’s proofs uncover this precise expression for the mean squared error of an estimate based on a sampling mechanism R:

where 𝛀² is the variance in G’s distribution, 𝔼[CORR[Rᵢ, Gᵢ]²] is the precise formulation for d.d.i, and f is sampling frequency.

We can’t really do much about 𝛀², so truly the only levers at our disposal here are to:

Decrease d.d.i by having nice randomized sampling mechanisms.

by having nice randomized sampling mechanisms. Increase sampling frequency.

This lines up nicely with our empirical results from before.

N Is The Enemy

An insidious finding falls out of the expression above.

If we try and compare an estimate based on simple random sampling vs. one that depends on a sampling mechanism R, we get a metric called the design effect (Deff). The Deff essentially tells us how much our particular sampling mechanism will screw up vs. the gold standard.

It turns out, with some clever mathematics, that this Deff can be expressed in terms of CORR[Rᵢ, Gᵢ] and our population size N:

Yikes.

To keep Deff constant, the correlation between Rᵢ and Gᵢ needs to shrink by a factor of the square root of N.

This is Xiao-li Meng’s punchline. To keep Deff constant, the correlation between Rᵢ and Gᵢ needs to shrink by a factor of the square root of N as N grows. In other words, CORR[Rᵢ, Gᵢ] must be O(√N⁻¹), or else the sampling mechanism R will produce dumpster-fire results at large N.

To keep Deff constant, the CORR[Rᵢ, Gᵢ] of a selected sampling mechanism R must drop rapidly as a factor of N.

The only sampling mechanisms that behave like this are random. That is to say, for high N, random sampling isn’t just a good idea: it’s a requirement.

Failing to understand this led us to wildly misforecast the 2016 Presidential election. We didn’t think a little response bias could affect our results, even though just a smidge was enough to completely wreck us. Almost inevitably so.

More relevant, now: it’s why our attempts to understand COVID-19’s lethality have been so goddamned scattershot.

N is the enemy, and a high sample size isn’t going to help in the face of it if our sampling mechanisms are defective.

All it’ll do is increase our confidence in a result that is almost certainly wrong.