Advanced Random Sampling in BigQuery SQL
By: Bence Komarniczky
Tags: Statistics, Data Science, Sql, Sampling, Bigquery

Generate some data

There are some great open datasets out there in BigQuery, but they are fairly big so you could easily get charged for querying them if you’re not careful enough. I do encourage you to look at some of these datasets, but here I’ll be generating a tiny dataset just to make sure I don’t get billed for any of the queries (BQ has a 1TB/month free trier limit).

This will give us 2 groups (a and b) with the first group having a uniform random distribution between 5 and 15 and the second group having a uniform random distribution between 2 and 9.

You can use this method to generate a list of numbers from uniform random distribution between any 2 numbers.

We’ll be working with this dataset throughout the tutorial, so I’d recommend you save it as a dataset — I saved it as ds.random_table and this is how I’ll reference it in later queries.

Explanation of query — feel free to skip

If you’re not yet familiar with arrays or the random function in BQ here’s what’s happening in the above query:

We create an array from 1 to 1000.

For group a, we expand this array into one number per row (unnest)

We add a flag group_a and a random number for each of these thousand rows.

and a random number for each of these thousand rows. We repeat the same thing for group_b and add it to the end of the above table with union all .

Naive Sampling

Photo by Djim Loic on Unsplash

There are some SQL tutorials for random sampling online that advocate for using the RAND() function. Let's see how this works and why you should never use it 😉.

We’ll take a random sample of 20 ids and then take the average of the v numbers as well as the counts of group_a and group_b observations.

How do we do this? We generate a random number between 0 and 1 for each item, sort the rows by this random number and then we can take the first 20 rows as our sample.

And now, given this sample table we calculate our statistics:

I get the following results:

Image captured from BigQuery UI

Now let’s run it again:

Image captured from BigQuery UI

Wow, this is different. 😱

Gif from GIPHY.

This is not really ideal as I want to make sure whatever calculations I make in BQ can be reproduced by my data scientist peers. Let’s see how you can do just that.

Reproducible Sampling with Hashes

The problem with RAND() function is that it creates new random numbers on each execution. So row 1 might get 0.2 on the first run but then 0.99 on the second run. What we would like to achieve is a way for us to get the same random numbers for each row.

This is where hashing becomes useful. A hash function takes a string or number and returns a random, but consistent string or number. The easiest hashing function to use in BQ is farm_fingerprint . This function always returns the same integer for the same input and it can take strings as inputs. Let's see how it compares to RAND() which takes no input.

And the results are:

Image captured from BigQuery UI

You can see that our 2 Alberts have the same farm_fingerprint values, while our rand function returned different values. Even better, farm_fingerprint will produce the same number at each run.

Go ahead and try it, the above query costs nothing in BQ as we’re generating all of our data at query time.

Armed with this new shiny function, it’s time to modify our original sampling query and make it fully reproducible.

What’s happening here?

We created an integer column called rn using the row_number function We cast these as strings because farm_fingerprint requires a string or byte input. Then, instead of rand() we’re using the farm_fingerprint function to generate some random numbers that we can order by.

I get the following results, no matter how many times I run the query. (You might get a slightly different result as our original table was generated by rand() so your underlying numbers might be different).

In a real-world scenario, you might already have a row id or user id associated with your data. In which case, feel free to pass that into farm_fingerprint as input. Just watch out for duplicate ids as those duplicate ids will have the same hashes! This might be something you would want in some cases though.

Sampling Ratio with Modulo

We now know how to sample values reproducibly. But for large datasets could be problematic to order the whole table and then take the first n rows. How do we take x% of the table without ordering?

Our hashing function always returns an integer. That integer is sometimes divisible by 10, sometimes not. Because the hash function distributes observations fairly uniformly across integers, we would expect 1 in 10 (10%) of our hashes to be divisible by 10. Similarly, 1% of our numbers — on average — should be divisible by 100. We can use this to our advantage and take the modulo of the hash to decide whether we want to keep a number or not.

To take a 15% sample, we would take the modulo of our hash by 100, then select all numbers that have a modulo less than 15.

There’s one catch here! The BQ modulo operator can return negative numbers for negative integers. So we must take the absolute value first.

The above is a bit more complicated than before. This is because we want to use a where statement and that cannot contain a row_number() function — or anything with an over() part. So we first create the row numbers and then take the sample using farm_fingerprint .

Make a comment in your SQL code when you sample like this as new data scientists might find it difficult to understand the ratio you’re about to sample!

Multiple samples

Photo by Miguel Orós on Unsplash

As our final step, we will take multiple samples and then calculate the sample means per group. This will get interesting, so strap in!

Here’s what we’ll do:

Create an array of length n where is n is the number of times we want to sample. We shall call this number sample_id . Cross join our array of sample_id s to the original dataset to duplicate each item n times. Combine the sample_id and row numbes to get a unique id for each row that we can use with farm_fingerprint . Use the modulo method to take 15% of all observations — on average each sample_id will have 15% of each observations taken. Get the mean value per sample_id and group id.

If you plot the results, you’ll get the following:

Yes, this is from Google sheets. Judge me all you want, I was too lazy to read it into Python. | Source: created by author

This gives us very good estimates — and 2 pretty bell curves — for our 2 different groups. Yes, this is the Central Limit Theorem at work. Go back to the original data generation step and see if this correctly estimates our population means or not!

Remember that BigQuery charges by bytes queried and we only queried our original data once. We just happened to cross join it to a bunch of integers to replicate our sampling 1000 times! 🎉

Now you can use arrays, farm_fingerprint and mod to get a better estimate for your statistics with BigQuery on the cheap!

Conclusion

Thanks for reading to the end! 🙏

Now you know how to do sampling in SQL using some of the handy functions in Google BigQuery. You’ve learned: