Snowflake vs BigQuery: Clash of Titans (Part 2)
By: Joao Marques, Data Beyond Ltd
Tags: Gcp, Snowflake Computing, Bigquery, Snowflake, Google Cloud Platform

Scaling Up

Photo by Volodymyr Hryshchenko on Unsplash

One of the things you expect from any MPP Database is the ability to scale up and down on demand, but:

For technologies with independent storage and compute like Snowflake and BigQuery, you expect one step further: the ability to scale compute “quickly”.

In Snowflake you can scale up and down with a simple DDL command, absolutely seamless and almost instantaneous (couple seconds):

ALTER WAREHOUSE WH_SCALE_UP SET WAREHOUSE_SIZE='XLARGE';

>> QUERY GOES IN HERE;

ALTER WAREHOUSE WH_SCALE_UP SET WAREHOUSE_SIZE='SMALL';

And more, you can scale out your cluster by creating multi-cluster warehouses to avoid query queuing, all at the tips of your fingers with a simple DDL command.

With BigQuery though… a whole different conversation. To enable scalability, you will need to move away from the on-demand and move to the slots model.

Slots are pre-paid computation units, which you need to use the UI or a special console to allocate to projects. Slots require a commitment (you pre-purchase a certain amount of compute), and to move between flat-rate and flex-slots is an admin task done either in the console or through BQ command line. Don’t get me wrong, flex-slots and flat-rate are theoretically great and give you great theoretical flexibility. It’s the implementation that is not that great:

For your typical heterogenous workload, are you really going to monitor the usage every minute, and manually adjust the slots in the BigQuery console? Or issue BQ command line commands?

It is just not practical, which most likely means you will “overkill” the number of slots most of the time, and this is where things can become expensive.

The tempting “on-demand” model where you only pay for what your query uses ends here. As soon as you move to slots you’re paying for allocation/uptime, and as long as slots are allocated and not being used, you’re incurring in idle-cost.

This is easy to control for automated tasks, but automated tasks tend to account for only half the usage in a data warehouse, the other half being user queries and reporting. You can have a quick stroll around this Google Cloud community article to get a feeling for how hard it is to manage slots: Demystifying BigQuery reservations.

You may be thinking “yes, but that is exactly the same with Snowflake, if it’s up you’re paying for it”, and you’re right! But there is one big difference: to stop the compute units (and stop paying) in Snowflake is super simple and can be fully automated (see auto-suspend and alter warehouse set size). The tool helps you reduce wastage, and if you want, you can reduce idle-cost to peanuts.

So my conclusion is pretty simple: Snowflake is a clear winner here. Scaling in Snowflake is totally seamless, effortless, and you can easily reduce idle-costs with auto-shutdown / DDL.

That is definitely not the case with BigQuery…

Scaling Out

Photo by Jørgen Håland on Unsplash

Let’s say you’re doing a new app release, and all of the sudden, everyone is trying to understand how the release is going. Your analysts are doing ad hoc queries to answer CxOs, your data scientists trying to get data to predict user adoption, and your users refreshing reports waiting for the charts to move. Because of this usage spike, you start receiving many more concurrent queries than you usually would: you now have a temporary query queuing problem.

If you’re using Snowflake, you can just enable multi-cluster on your warehouse to manage this workload. It works by analysing the number queries currently in queue, and if the number of queries go above a certain threshold, it adds 1..N more clusters to your warehouse computing power until the queue is handled. As soon as the queries are sorted, it will scale in again, and will auto-suspend after no one needs it. This happens “under the hood” and is 100% seamless to your users.

If you’re on BigQuery, then the first question you’ll need to answer is “how many slots should I buy?”… then, you have a problem: you can’t predict usage… so how will you manage the slots? Well, you’ll just have to choose if you make your users happy during peaks, and pay for a LOT of idle-cost, or if you make their queries slower but keep the idle-cost low.

In BigQuery you need to decide how many slots you buy ahead of time, how many slots you deploy/allocate to each project, and all of that requiring a manual or programmatic access (console/bq cmd line). If your users need the compute to be available to run queries, you need the slots to be allocated, and that means you’re paying for 100% of the time, even if your users are not currently using it.

To “scale in” (and stop paying), you will need to manually de-allocate the slots, and I don’t think your 500 users will give you a call to say when they’re done... If you ask me, this has everything to become very expensive…

Again, a clear win for Snowflake: multi-cluster warehouses were made to handle this, they scale in and out on the fly, seamlessly, and with zero admin.

Goodies Inside The Box — Other Features

Photo by Monique Carrati on Unsplash

Now that we know BigQuery’s cost per unit of work is equivalent to Snowflake’s Enterprise version, let’s talk about some of the main differences in the features they offer (note: this is not an exhaustive list and highlights only the main differences):

Focusing on the differences — BigQuery vs Snowflake Enterprise

Cloud Lock-In Factor

Google BigQuery implies a big “lock-in” with Google Cloud. With Snowflake you can use any of the big-3.

Machine Learning (BigQuery)

I like that Machine Learning is coming more to the DB level, but at the same time, I have always questioned how usable it really is for Companies. Mature data science teams don’t like “black box wizards”, they want to be in control, to be able to use their own models, to tweak them however they fancy, and they like their tools. On top of that, BigQuery ML models will always lag the ones available in the community, which is also not ideal. Given this, even though I like it as a feature, it’s mostly a nice-to-have and most companies will naturally steer away from using it.

Real-time Data Streaming

The use case for real-time (couple of seconds) Data streaming into a Data warehouse is a rare one. Typically you can wait a minute, and if you can’t, you’re probably using the wrong technology to solve your problem. Anyway, if you do require this then BigQuery seems to be a bit easier to set it up. In Snowflake, even though you do it using external tables, it’s not as nice and seamless.