Minimize Rework, Improve Performance, and Manage Costs of your cloud migrations with Snowflake and Talend
By: John O'Brien
Tags: Snowflake, Cloud Migration, Data Warehouse

Minimize Rework, Improve Performance, and Manage Costs of your cloud migrations with Snowflake and Talend John O'Brien Follow Jul 30 · 8 min read

As today’s cloud architecture capabilities continue to mature, many more organizations are moving away from typical on-premise architecture and adopting a cloud-centric strategy. Along with increasing volumes of data the need for accurate and timely processing of data has increased. Oftentimes data processing challenges directly impact the availability of data and insights to support timely business decisions, leading to missed opportunities. One way our clients gain efficiencies with their data processing is to migrate their workloads to modern cloud solutions.

At Cervello, we have assisted numerous clients in successfully navigating this pathway. One such migration I recently completed was from an existing on-prem Data Warehouse solution to a modern cloud-based architecture. The architecture included Snowflake, a modern built-for-the-cloud data warehouse platform and Talend, a modular development framework with pre-packaged data orchestration capabilities. I’ll use this project as an example to help highlight some best-in-class practices to adhere to if you were to undertake a similar initiative.

During the course of the project, Cervello worked alongside the client team in architecture advisory and lead development roles. Our goal was to train the existing client development team on the new technologies being introduced during the development effort which provided the team with the knowledge and know-how to maintain, develop, and enhance the solution further once our migration was completed.

In this blog post, I’ll highlight important areas that will enable you to minimize rework, improve performance, and manage costs while increasing flexibility with Snowflake and Talend.

So let’s get to it!

Minimize Rework with Talend Templates

While examining our client’s existing on-prem processes we noticed that we could categorize the data ingestion process into three distinct flow types. Utilizing Talend, templates were created for each of these flows and the templates were used to put in place the base framework for the load. This process resulted in only specific changes being required for any new source introduced to the loading process.

Flat File to Snowflake: changes required (source file name, target table name, file schema) Full Table Data Source to Snowflake: changes required (source table, target table, table schema mapping) Incremental Data Source to Snowflake: changes required (source table, target table, table schema mapping)

An example of the template created for a Full Table Data Source to Snowflake Talend Job is shown below. This template was used to create all data ingestion jobs required for a full table migration and only required modification for the SQL_Server_Source _Table and Snowflake_Target_Table objects.

Talend Template: Minimize development time per data source

Utilizing templates made possible in Talend, the development timeframe for ingesting data into the 450 tables within the Snowflake DataWarehouse was dramatically reduced. Moving forward, these templates will reduce development efforts and facilitate the addition of new data sources (flat-file, tables) easily.

Improving Performance with Snowflakes Bulk Loading

Once the processes to ingest data from our required source systems into Snowflake were complete, it was time to run an end to end test and see how the process compared to the existing on-prem solution. In order to take full advantage of Snowflakes architecture the following techniques were implemented, resulting in dramatic performance gains.

Limiting file sizes during data ingestion

During the data ingestion process best practice dictates data to be loaded into Snowflake be broken up into files sized between 10–100Mb.

This allows Snowflake to load data in parallel utilizing multiple threads within each node of the Virtual Data Warehouse. Each thread within a virtual warehouse can process 1 data file in parallel, therefore processing 8 small files provides greater performance than processing 1 large file.

Snowflake: Utilizing Virtual Warehouse Threads for parallel processing

This can be done programmatically within your process, however by using Talend to handle our data ingestion process it will take care of the file splitting automatically. As data is being pulled from the source query, small files are pushed to Snowflake and loaded in smaller chunks resulting in drastic performance improvements.

By utilizing Talend’s architecture to parallelize the loading process, and Snowflake’s performance boost for loading / ingesting data, the data ingestion process was reduced from 16 hours to 4 ½ hours using only an x-small virtual warehouse. Tests were run increasing the warehouse size in attempts to improve performance, however it was quickly determined that the query response time from the source systems was a limiting factor.

Stop using ETL constructs, and focus on adopting an ELT mindset

Another best practice followed is to evolve from using an ETL (Extract-Transform-Load) approach and adopt a process that focuses on ELT (Extract-Load-Transform). By adopting an ELT mindset, data is consumed into Snowflake first, and all Transformations happen within the Snowflake architecture.

The ELT Approach allows us to utilize Snowflakes flexible compute architecture to scale up when needed, and only when needed. In the ETL world, any processing for Transformations would directly impact users / other processes working within the system. By assigning separate virtual warehouses to each process, there is no contention for compute resources, and each process can be scaled accordingly to only use the amount of compute necessary.

The ELT approach also allows us to capitalize on Snowflakes Bulk-Loading capabilities. Within the Transformation process, utilize the following statements :

CREATE WITH SELECT

INSERT INTO WITH SELECT

MERGE INTO

The goal being to minimize single row insert statements and consolidating Insert, Delete, and Update statements as much as possible.

As an added benefit COPY INTO and INSERT INTO are non-blocking statements.

By utilizing Snowflake’s Bulk-Loading capabilities and Talend’s orchestration processes the existing on-prem Transformation process to populate 200+ Dimension and Fact tables was migrated reducing the overall time to completion from 12 hours to 3 ½ hours.

Manage Costs with Snowflake’s Virtual Warehouses

In typical on-prem solutions the architecture usually revolves around the highest peak usage and requires a delicate balance to manage costs while also meeting SLAs set forth by the business. A high watermark for processing power is set and the system is usually at 100% capacity during nightly batch processes, however minimally used throughout daily activities.

With Snowflake, Storage and Compute resources are not tied to each other, this provides flexibility in sizing required compute resources based on usage types (data ingestion vs. reporting) as well as only using larger compute resources when necessary and turning them on and off when not in use.

Separate compute resources for ingestion & reporting

Throughout the day various activities happen within a DataWarehouse environment that require different sets of resources to be successful. Large batch processing can require massive amounts of CPU with little to no CACHE required, where daily reporting requirements may require less CPU, yet benefit greatly by data result sets.

By separating our virtual warehouses into task based assignments, users requesting data for daily analysis are not impeded by any large scale batch processing that may be consuming large amounts of resources.

Depending on the reports/queries being generated, and the business criticality of timely return of a queries results different warehouse sizes can be allocated to the Analysts / Data Scientists to control costs while also meeting the businesses needs.

Suspending & resuming virtual warehouses

The ability to use what you need, when you need it is a huge benefit when it comes to cloud computing, and Snowflake makes this available inside your Data Warehouse environment. When configuring Virtual Data Warehouses for use, it is possible to take advantage of Snowflakes per-second billing to manage costs by suspending / resuming a virtual warehouse when not in use.

Suspending and Resuming warehouses can be done programmatically within your code (best used for scheduled batch processing) or by configuring the Virtual Warehouse to allow Snowflake to Automatically resume a warehouse when needed, and suspend the warehouse once a threshold of inactivity has been reached on the warehouse.

Throughout our migration process we took advantage of Snowflake’s ability allowing you to ‘Pay for what you use’ by using various Warehouse Sizes only for the durations we required an increased server capacity.

Viewing the total duration (start to finish) of data being brought into snowflake from our various sources, to the availability to the end-consumer within our Dimension / Fact tables gives us a view of our credit usage:

X-Small Warehouse — 6h 40m (6.4 credits/day)

Large Warehouse — 1h 20m (10.4 credits/day)

Total Usage = 8h (16.8 credits/day)

We were able to switch warehouses for each individual process by issuing a ‘USE WAREHOUSE <WAREHOUSE_NAME>;’ prior to each execution. This allowed us to bypass the need to scale our warehouse size to our largest process requirement. If we were constrained to one size fits all our credit usage would have looked like the following:

Large Warehouse — 8h (64 credits / day)

That’s a savings of 47.2 credits/day or 73% of our costs each day.

In Conclusion

Although this was only a migration of our client’s existing Data Warehouse platform to a cloud based solution, in doing so, we provided a stable foundation for their evolution to a full end to end cloud architecture. The end result provided faster processing capabilities, enabling next day reporting as opposed to day — 2 reporting and reduced processing costs. In addition, the Development team now has more flexibility to introduce changes on a shortened development and test cycle. Lastly, our client now has a greater confidence in their ability to scale the environment allowing for new data sets to be brought into the process, further enabling their business strategy through enhanced insights. That last one, that is the one we are proudest of.

I hope these takeaways are helpful to you on your migration journey.

About Cervello, a Kearney company

Cervello, is a data and analytics consulting firm and part of Kearney, a leading global management consulting firm. We help our leading clients win by offering unique expertise in data and analytics, and in the challenges associated with connecting data. We focus on performance management, customer and supplier relationships, and data monetization and products, serving functions from sales to finance. We are a Solution Partner of Snowflake due to its unique architecture. Find out more at Cervello.com.

About Snowflake

Thousands of customers deploy Snowflake’s cloud data platform to derive insights from their data by all their business users. Snowflake equips organizations with a single, integrated platform that offers the data warehouse built for the cloud; instant, secure, and governed access to their network of data; and a core architecture to enable many other types of data workloads, such as developing modern data applications. Find out more at Snowflake.com.