Data Materialization in Snowflake Using Confluent Kafka Connect: Part 2
By: Venkat Sekar
Tags: Data, Cloud Computing, Kafka, Open Source, Snowflake

Data Materialization in Snowflake Using Confluent Kafka Connect: Part 2

Making On-Prem Oracle Databases Useable with Snowflake in Azure with Kafka Connect

In Part 1, Data Sync to Snowflake Using Confluent Kafka Connect, we discussed the reasons behind adopting Confluent to ingest data into Snowflake. I also highlighted some of the hurdles that we had faced, and the various ways we ended up solving them.

In this Part 2 post, I’ll review the steps you need to take in order to make the data readily available for consumption once it’s ingested into Snowflake via the Kafka Connector, namely:

Parsing the message payload.

Handling materialized table creation/alterations on the fly.

Applying data conversion as the records get stored in the materialized table.

Upserts/Merging of records in the materialized table.

I’ll walk through the implementation of how each of these was solved, and I hope that by sharing this implementation process, you will get a good starting point if you are looking to use Kafka with Snowflake; in fact, you might come out of it with a more refined approach than what I presented once you’re familiar with the process.

Scenario

To review, in my example, I want to synchronize data from an on-prem database into Snowflake using Kafka.

The software stack is below:

Confluent Kafka

Confluent Connect

JDBC Source connector

Snowflake Kafka connector

Point of View on the JDBC Source Connector

The adoption of the JDBC Kafka connector does have some limitations, as mentioned in Part 1. The JDBC source connector is simplistic and offers a polling-query based approach to retrieve the data from the tables whitelisted:

Unless there are specific columns (a flag column or timestamp column) in the source tables, you cannot identify if the record is an update vs insert.

Only logical deletes can be handled, meaning there is a column indicating a delete.

Table truncates, hard record deletes cannot be determined.

Dropping off a table and recreating the table by the external process can cause issues with the connector.

Between the polling period, if there are multiple updates, only the last committed update will be sent over in the topic.

For these reasons, you have to evaluate if the JDBC source connector is the right choice; this does not mean the connector cannot be used at all.

Here are some situations where I think it can be adopted:

Truncate and load scenario once per day. This is doable in the case of a smaller dataset.

There is not a need for Snowflake to adopt a Slowing Changing Dimension (SCD) implementation.

Let’s say for this discussion that you are ok with doing a snapshot-based load into Snowflake and therefore decided to use the JDBC Kafka Connector.

Limitations

Now, will this solution be suitable for all scenarios? I cannot guarantee that as your situation and mileage will vary. What do I mean by this? Here are a few potential points to think about:

The number of records in the source table.

Load on the source database to query and extract.

Network bandwidth speed for transferring data.

Resource on which Confluent Kafka Connect is running. The number of Kafka Connect instances, memory, etc.

So while what I’ve illustrated is workable, you will need to evaluate according to your environment/situations. So let’s get to the implementation.

Data Acquisition

Though AVRO is the preferred format of the message in Kafka topic, for this implementation I’ll be using the JSON message format.

I chose the JSON message format because the schema, which is table metadata, will be part of each and every message. I’ll use this information to create the final materialized table after the data is ingested into Snowflake.

Confluent Kafka Connect Configuration

Here is the Confluent Kafka Connect configuration which helps us to get a bulk data load of the table into Kafka:

name=orcl-source connector.class=io.confluent.connect.jdbc.JdbcSourceConnector tasks.max=1 connection.url=jdbc:oracle:thin:@localhost:49161:xe connection.user=books_admin connection.password=password dialect.name=OracleDatabaseDialect schema.pattern=APEX_040000 table.whitelist=TEST_DEMO_EMP # mode of operation mode=bulk #convertor definitions topic.prefix=tpc-json- key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter value.converter.schemas.enable=true #To handle data type conversion numeric.mapping=best_fit transforms=Cast transforms.Cast.type=org.apache.kafka.connect.transforms.Cast$Value transforms.Cast.spec=TESTSAL:int32

Data Ingestion

Snowflake Kafka Connect Configuration

Below is a sample of the Kafka Connect configuration:

name=sflksink connector.class=com.snowflake.kafka.connector.SnowflakeSinkConnector tasks.max=1 topics=tpc-json-TEST_DEMO_EMP snowflake.topic2table.map= tpc-json-TEST_DEMO_EMP:CDC_EMPLOYEE buffer.count.records=10000 buffer.flush.time=60 buffer.size.bytes=5000000 snowflake.url.name=XXXXXX.east-us-2.azure.snowflakecomputing.com snowflake.user.name=XXXXXXX snowflake.private.key=MIIFXXXXXXX8w== snowflake.private.key.passphrase=admin snowflake.database.name=DSC_RAW snowflake.schema.name=SOMESCHEMA snowflake.metadata.offset.and.partition=false key.converter=org.apache.kafka.connect.storage.StringConverter value.converter=com.snowflake.kafka.connector.records.SnowflakeJsonConverter

Data Materialization in Snowflake

I’ll kick things off with a summary of key terminology based on the diagram below:

CDC Table: This is the table into which the Kafka-connect will insert the record. In this example, the table name is CDC_EMPLOYEE.

This is the table into which the Kafka-connect will insert the record. In this example, the table name is CDC_EMPLOYEE. Materialized Table: This is the final table which will be used by the user to query the records. The data will be converted to the appropriate data type as reflected by the schema element. In the example the table name is TEST_DEMO_EMP.

This is the final table which will be used by the user to query the records. The data will be converted to the appropriate data type as reflected by the schema element. In the example the table name is TEST_DEMO_EMP. CDC_CTAS_SP: This is the stored-procedure which will help in creating the materialized table.

This is the stored-procedure which will help in creating the materialized table. CDC_DATASYNC_SP: This is the stored-procedure that will ingest the data into the materialized table.

CDC Table

Below is a view of a sample message from the CDC_EMPLOYEE table. The records were inserted by the Snowflake Kafka Connector:

Materialized table

Here is a view of a sample message from the EMPLOYEE table. Using the stored procedures, explained in the following section, we will create the table and insert records into the table, based on the message from the CDC table.