Snowflake Performance Query tuning Using Clustering(Micro batching)
By: Sanjit Khasnobis
Tags: Snowflake, Partitioning, Query Optimization, Clustering, Data Engineering

Magic of snowflake exists in it’s micro partition architecture. In todays topic we will discuss that aspect of snowflake and how that can be used to optimize your select Queries. Each micro-partition contains between 50 MB and 500 MB of uncompressed data (Actual size in snowflake is smaller as data is always stored compressed)

Groups of rows of a table are stored in each micro partitions, organized in a columnar fashion.

The Size and structure allows pruning for very large tables, which can be created with millions or even hundreds of micro partitions.

Micro Partitions(Picture taken from Snowflake Site — https://docs.snowflake.com/en/user-guide/tables-clustering-micropartitions.html)

Today as part of query optimization for snowflake we will go through below steps to showcase Snowflake query optimization and also will show and explain how snowflake does micro partition at the back.

Step 1: We will create one table with Cluster key.

Step 2 : We will insert data from a large sample table to Clustered key table.

Step 3: We will show how clustered table at the back is micro portioned by snowflake after data load completes.

Step 4: We will off the snowflake cache result.

Step 5 : We will query separately both the original table and clustered table to show clustered table returns the data faster than the original table.

Step 1 : We will create one table with Cluster key.

Run the below query in snowflake schema.

CREATE TABLE TEMP_ORDER_CLUSTER

(

O_ORDERKEY NUMBER(38,0),

O_CUSTKEY NUMBER(38,0),

O_ORDERSTATUS VARCHAR(1)

)

CLUSTER BY (O_ORDERSTATUS)

Step 2 : We will insert data from a large sample table to Clustered key table.

For inserting data we are using sample database from snowflake orders table as below.

INSERT INTO TEMP_ORDER_CLUSTER

SELECT O_ORDERKEY,O_CUSTKEY,O_ORDERSTATUS

FROM “SNOWFLAKE_SAMPLE_DATA”.”TPCH_SF10000".”ORDERS”

It will insert 15,000,000,000 records. Which will take around 20/25 minutes in a small/medium virtual warehouse(processing power)

Step 3: We will show how clustered table at the back is micro portioned by snowflake after data load completes

Now, once data is loaded successfully in the above table we will run the below query again and again to check how clustering is happening by snowflake.

SELECT SYSTEM$CLUSTERING_INFORMATION(‘TEMP_ORDER_CLUSTER’,’(O_ORDERSTATUS)’);

This is the first snapshot of the micro partition details of the query.

{

“cluster_by_keys” : “LINEAR(O_ORDERSTATUS)”,

“total_partition_count” : 8832,

“total_constant_partition_count” : 0,

“average_overlaps” : 8831.0,

“average_depth” : 8832.0,

“partition_depth_histogram” : {

“00000” : 0,

“00001” : 0,

“00002” : 0,

“00003” : 0,

“00004” : 0,

“00005” : 0,

“00006” : 0,

“00007” : 0,

“00008” : 0,

“00009” : 0,

“00010” : 0,

“00011” : 0,

“00012” : 0,

“00013” : 0,

“00014” : 0,

“00015” : 0,

“00016” : 0,

“16384” : 8832

}

}

This is the intermediate snapshot of the micro partition details of the query.

{

“cluster_by_keys” : “LINEAR(O_ORDERSTATUS)”,

“total_partition_count” : 9545,

“total_constant_partition_count” : 5710,

“average_overlaps” : 1555.2224,

“average_depth” : 1548.8434,

“partition_depth_histogram” : {

“00000” : 0,

“00001” : 5691,

“00002” : 0,

“00003” : 0,

“00004” : 0,

“00005” : 0,

“00006” : 0,

“00007” : 0,

“00008” : 0,

“00009” : 0,

“00010” : 0,

“00011” : 0,

“00012” : 0,

“00013” : 0,

“00014” : 0,

“00015” : 0,

“00016” : 0,

“04096” : 3854

}

}

This is the final snapshot of the micro partition details of the query -

{

“cluster_by_keys” : “LINEAR(O_ORDERSTATUS)”,

“total_partition_count” : 9993,

“total_constant_partition_count” : 9981,

“average_overlaps” : 0.0198,

“average_depth” : 1.018,

“partition_depth_histogram” : {

“00000” : 0,

“00001” : 9977,

“00002” : 0,

“00003” : 0,

“00004” : 0,

“00005” : 0,

“00006” : 0,

“00007” : 0,

“00008” : 0,

“00009” : 3,

“00010” : 0,

“00011” : 0,

“00012” : 0,

“00013” : 13,

“00014” : 0,

“00015” : 0,

“00016” : 0

}

}

Interesting!! is not it ? You can see with time snowflake is partitioning data in better way. As the average overlaps between data points are reducing so partitioning is becoming more atomic.

Just to give an idea of what is partition depth and overlapping partition I have borrowed the Diagram from snowflake official website -

https://docs.snowflake.com/en/user-guide/tables-clustering-micropartitions.html

I will encourage all of you to go through the above link to get deep understanding of the snowflake micro partitions.

Now , in above diagram if you can see

Case 1 : All 5 partitions are overlapping so

overlapping Micro partitions = 5 and Overall depth = 5

Case 2 : 3 partitions are overlapping so

overlapping Micro partitions = 3 and Overall depth = 3

Case 3 : 3 partitions are overlapping but data line is crossing through only 2 partitions so

overlapping Micro partitions = 3 and but Overall depth = 2

case 4 : None of the partitions are overlapping so

overlapping Micro partitions = 0 and Overall depth = 1

So, When there is no overlap in the range of values across all micro-partitions, the micro-partitions are considered to be in a constant state (i.e. they cannot be improved by clustering).

Step 4: We will off the snowflake cache result.

ALTER SESSION SET USE_CACHED_RESULT = FALSE

Step 5 : We will query separately both the original table and clustered table to show clustered table returns the data faster than the original table.

Query from original Non Clustered table takes — 2.40s

SELECT O_ORDERKEY FROM “SNOWFLAKE_SAMPLE_DATA”.”TPCH_SF10000".”ORDERS”

WHERE O_ORDERSTATUS = ‘P’

Non Clustered table takes more time to fetch the same data

Query from the clustered table takes — 35.69s

SELECT O_ORDERKEY FROM COMMON_LAYERS.STAGING.TEMP_ORDER_CLUSTER

WHERE O_ORDERSTATUS = ‘P’

Clustered table queried in lesser time

So we can see clearly Snowflake Clustered table owns over the Snowflake Non Clustered table.

So, can you find which of the filed can be the best Cluster key in advance in snowflake ? Yes , you can without even creating the Clustered table.

I will let you do investigate on this. One hint the required query is in this blog itself.

Happy Reading !! Happy Coding !!