Streaming From Kafka to Snowflake : Part 2— S3 to Snowflake with Snowpipe
By: Avi Paul
Tags: AWS, Snowflake, Kafka Streams

In part 1, we received our data from Kafka to s3. This part contains how to get streaming data from S3 to Snowflake database.

S3 to Snowflake — from snowflake community page

Requirement:

Access to S3 account so that you can get AWS_KEY_ID and AWS_SECRET_KEY Have auto_ingest=true enabled in your snowflake account. If it is not, email snowflake support to enable it.

Step 1: Connect to S3 with snowpipe

Snowpipe — a continuous, straightforward and cost-effective service to load data into Snowflake.

Since our objective is to continuously get the data from our bucket, we will create a pipe called snowpipe to link our bucket. In order to do this, we need to provide the URL of the bucket, and credentials.

create or replace stage snowpipe.public.snowstage

url= 's3://your_s3_bucket'

credentials = (AWS_KEY_ID = 'replace_with_your_aws_key' AWS_SECRET_KEY='replace_with_your_aws_secret');

It is a good idea at this point to check if the stage has been created by doing

show stages;

Step 2: Create a table that will hold JSON files

After we have created the stage, we will need to create the table that will hold the data. We want to use json data for this post just so that we can show that we can process semi structured data. Hence, we are creating a field called jsontext with the data type variant.

create or replace table snowpipe.public.snowtable

(

jsontext variant

);

Step 3: Create your pipe

Now comes the crucial part, creating pipe. Since we are using json file, the file_format type is set as json. And in the create statement, auto_ingest is set to true so that it can ingest data automatically.



copy into snowpipe.public.snowtable

from

file_format = (type = 'JSON'); create or replace pipe snowpipe.public.snowpipe auto_ingest=true ascopy into snowpipe.public.snowtablefrom @snowpipe .public.snowstagefile_format = (type = 'JSON');

The key part here is auto_ingest=true. As of the now, this feature is not enabled by default. If you don’t have it enabled, you need to email snowflake support and ask them to enable it for you

What happens if I get an error that says the feature is unsupported? This means it is not enabled. Hence, you email snowflake support and ask to enable it.

Unsupported feature 'Unknown flag auto_ingest'

Step 4: Copy the arn:aws:sqs from the notification channel

If your auto ingest is enabled, the last step will create an additional “notification channel” when you run show pipes command. If you run this,

show pipes;

You will see something like this where you will see there is a “notification_channel” almost at the end. This channel now has an Amazon Resource Name or arn. Copy this arn and you will need to put it in your s3 bucket properties.

Step 5: Enable Event from S3 bucket

We will now go back to the s3 bucket we have created in step 1. Go to the properties, and in the advanced setting you will see the setting for “events”.

Step 6: Add event details

Once you click on event, you can add a “new notification”. Here we will give it

Name : a name that we like, Event: choose ObjectCreate (all) as the event, Send to: SQS Queue SQS: Add SQS queue ARN SQS Queue ARN: Paste the arn you can copied from Snowflake show pipe command.

Step 7: Test by adding a new json file in the s3 bucket

Now if you add a new json file in S3 bucket it will show up in the snowflake table that you have created earlier. I found that it does not appear right away. So give it 5 to 10 seconds before checking if the json contents is now in your snowtable.

select count(*) from snowpipe.public.snowtable;

select * from snowtable;

Now that you have we have data in S3 bucket, we can proceed to do something more such as doing ETL or applying ML to it.

https://www.servian.com/data-and-analytics/