How to stream real-time data into Snowflake with Amazon Kinesis Firehose | by Abizer Jafferjee (Waterfront Analytics) | Jul, 2020 | Towards Data Science

Businesses today can benefit in real-time from the data they continuously generate at massive scale and speed from various data sources. Whether it is clickstream data from websites, telemetry data from IoT devices or log data from applications, continuously analysing that data can help businesses learn what their customers, applications and products are doing right now and react promptly.
In this article, we want to explore how Amazon Kinesis Firehose can be used to make new data available in a Snowflake data warehouse in real-time rather than using traditional batch processing over long intervals. The motivation is to help businesses understand how they can build the foundation for powerful real-time applications and analytics use cases.
For those that may not be familiar with these technologies, we’ll provide a very quick summary, but our main focus for this post is demonstrating how to code a simple application that sends data to Snowflake. For this demonstration we’ll be using a Twitter API to stream real-time tweets.
Snowflake is a cloud-native, fully relational ANSI SQL data warehouse service available in both AWS and Azure. It provides a consumption-based usage model with unlimited scalability. It’s capable of loading both structured and semi-structured data like JSON, Avro or XML.
Amazon Kinesis Firehose is the easiest way to load streaming data into AWS. It can capture, transform and load streaming data into Amazon Kinesis Analytics, AWS S3, AWS Redshift and AWS Elasticsearch Service. It’s a fully managed service that automatically scales to match the throughput of your data. It can also batch, compress and encrypt the data before loading it.
Tweets are posted on Twitter continuously so it’s a great source for us to use as a real-time stream. For this example, we’ll be using Twitter’s streaming API to get new tweets from Twitter as they are posted. To do this, we must create a persistent connection to the Twitter API and read each connection incrementally, then process each tweet quickly so that our program doesn’t get backed up.
We will build a simple python program to call the Tweets. Before you can do this you must set up your Twitter developer account and create an app. First go to Twitter developer center and create a developer account. Then go to the application console and create a new Twitter application. This will let you get specific credentials for the application to connect to the API.
Next, let’s write a simple app to call the Twitter API with a persistent session. The streaming API is different from a REST API because it pushes messages to a persistent session while a REST API is used to pull data. This allows streaming API to push data in real-time as it becomes available.
To setup our application we will install a python library called Tweepy.
pip install tweepy
Then we set up our call to the API like this:
To authenticate your app with Twitter replace the credential information above with the information from your app credentials tab. To set up our stream we first configure a listener class that tells us how we should handle data. Currently, the listener simply prints the data. We will later on replace that part of the logic to send the data to our AWS Kinesis Firehose. The track variable is a list of keywords for which Twitter will return related Tweets. For this example, I’ve used ‘Basketball’ as a keyword.
The API returns messages that we want to stream and store in a staging area. Typically, Snowflake loads data from S3 so we will be storing our messages as files in S3, staged for Snowflake to ingest.
Before you can set up a Kinesis Firehose and S3 bucket, you’ll need a user with the permissions to create S3 and Kinesis resources. Make sure to also create aws access key and aws secret key for programmatic access as we’ll need those later.
Setting up the Firehose service
Go to the Amazon Kinesis console and create a delivery stream. Choose a stream name and select “Direct PUT or other sources” as your source. This will allow us to directly send the records to our Kinesis stream through an API call. We won’t be setting up any data transformations in this demonstration for simplicity. Click next until you reach the destination section.Choose S3 as your destination, then click on create a new bucket and choose a unique name for your bucket.Next we will choose the buffer size and buffer interval. Kinesis buffers a stream of incoming data until a buffer size or buffer interval is reached. For this demonstration we will be aggressive and set both to a minimum of 1 MB and 60 seconds respectively.For this demonstration we will not set up data compression or encryption. For the IAM role section leave the default choice of allowing AWS to create IAM role with the necessary permissions. The role will allow Kinesis to communicate with S3.Finally, review your setup and create your delivery stream.
Go to the Amazon Kinesis console and create a delivery stream. Choose a stream name and select “Direct PUT or other sources” as your source. This will allow us to directly send the records to our Kinesis stream through an API call. We won’t be setting up any data transformations in this demonstration for simplicity. Click next until you reach the destination section.
Choose S3 as your destination, then click on create a new bucket and choose a unique name for your bucket.
Next we will choose the buffer size and buffer interval. Kinesis buffers a stream of incoming data until a buffer size or buffer interval is reached. For this demonstration we will be aggressive and set both to a minimum of 1 MB and 60 seconds respectively.
For this demonstration we will not set up data compression or encryption. For the IAM role section leave the default choice of allowing AWS to create IAM role with the necessary permissions. The role will allow Kinesis to communicate with S3.
Finally, review your setup and create your delivery stream.
Now that our AWS services are set up, we can modify the listener class in our python application so that each new message is not only printed to standard out but also sent to our Kinesis delivery stream. We can send data to our stream using AWS’s Boto3 library.
pip install boto3
Here’s how we do it:
To authenticate your client with AWS replace your credentials in the credentials variables. Here we simply set up a function that creates Kinesis client using Boto3 in the region our resource is set up and calls the put_record function to write data to our delivery stream name. We also convert the string data type on our data to bytes. To write every message to the delivery stream, we can call our function from the listener class so that it passes new messages to our function as they arrive.
Now you can run your python script and see new messages as Twitter pushes them to your standard out. Also monitor your S3 bucket to see incoming data. As each buffer is complete in our delivery stream, it will write a new file to S3 with the new micro-batch.
Now that we have data coming into S3 in real-time we can set up our Snowflake data warehouse to ingest the data as it’s available. Typically when loading data into Snowflake the preferred approach is to collect large amounts of data into an S3 bucket and load from the external stage via COPY command. However, for loading data continuously, Snowflake has built a data ingestion service called Snowpipe. Snowpipe loads fresh data in micro-batches as soon as it’s available in an external stage.
To set up our Snowflake resources first log into your Snowflake Web UI and switch over to the worksheet tab.
For this demonstration we will need the following Snowflake resources; a warehouse for the compute resource to perform SQL queries, a database to store our tweets, a external stage to load the data into Snowflake and a pipe to load data continuously.
Execute the following SQL to provision these resources
Create a warehouse
Create a warehouse
2. Create a database and table
We use a variant data type because the data we are ingesting is semi-structured.
3. Next we create an external stage to our S3 bucket
4. Create the pipe to copy into the tweets table from the external stage
Now we are almost done. Our pipe will read data from a AWS queue that it created for us. We need to connect this queue to our S3 bucket so that whenever a new file is added to the S3 bucket, the bucket will push it to the queue which will then be read by our pipe. That’s how the pipe knows that a file is ready to be ingested in real-time.
To configure the S3 bucket to the queue, we will need the AWS resource Id for the queue. Run the following SQL:
show pipes
In the results, check the column called notification channel for the pipe you created and copy the resource ID.
Back in AWS, go to S3, click on your bucket for this demonstration, switch to properties and go down to advanced settings. Then click on events and add new event.
Give your event a name. And choose “all object create events”. Select send to SQS queue and add your resource ID in the SQS queue ARN field. Then save your event.
Now you are done. Run your python script and monitor your S3 bucket as it receives new files. These files will now be ingested as new rows in the Snowflake table we created. Confirm by running a SQL select query.
select count(*) from twitter_stream.public.tweets;
Run the query every few minutes and you will notice that the count of rows in your table is increasing. This means that the set up is working correctly. You are receiving data from Twitter directly into Snowflake in real-time.
Run the query below to view the time window for each new batch coming into Snowflake.
Once the data is coming into your data warehouse there are many powerful use cases but that discussion is outside the scope of this article. We’ve shown you a really simple set up to give you a better understanding of the services involved in sending data to your warehouse in real-time.
Apart from Amazon Kinesis there are other stream processing platforms like Apache Kafka. Each has it’s pros and cons and this depends on the solution you want to build. In addition, most use cases have more complex requirements such as multiple data sources, latency specifications and varying data volumes that need to be taken into consideration when designing a solution.
If you have a real-time analytics use case reach out to us at Waterfront Analytics and we can help you ensure high value business outcomes for your use case.