Ingest data from Apache Kafka into Azure Cosmos DB Cassandra API using Kafka Connect
By: Abhishek Gupta
Tags: NoSQL, Cloud, Database, Kafka, Azure

Ingest data from Apache Kafka into Azure Cosmos DB Cassandra API using Kafka Connect

This tutorial is based on a practical scenario along with a re-usable Docker Compose setup to help with iterative development and experimentation Abhishek Gupta Follow Dec 14, 2020 · 11 min read

Azure Cosmos DB Cassandra API is a fully managed cloud service that is compatible with Cassandra Query Language (CQL) v3.11 API. It has no operational overhead and you can benefit from all the underlying Azure Cosmos DB capabilities such as global distribution, automatic scale out partitioning, availability and latency guarantees, encryption at rest, backups etc.

At the time of writing this blog, the Azure Cosmos DB Cassandra API serverless is available in preview mode !

Your existing Cassandra applications can work with the Azure Cosmos DB Cassandra API since it works with CQLv4 compliant drivers (see examples for Java, .Net Core, Node.js, Python etc.) But, you also need to think about integrating with other systems with existing data and bringing that into Azure Cosmos DB. One such system is Apache Kafka, which is a distributed streaming platform. It is used in industries and organizations to solve a wide variety of problems ranging from traditional asynchronous messaging, website activity tracking, log aggregation, real-time fraud detection and much more! It has a rich ecosystem of technologies such as Kafka Streams for stream processing and Kafka Connect for real-time data integration.

Thanks to its scalable design, Apache Kafka often serves as a central component in the overall data architecture with other systems pumping data into it. These could be click stream events, logs, sensor data, orders, database change-events etc. You name it! So, as you can imagine, there is a lot of data in Apache Kafka (topics) but it’s only useful when consumed or ingested into other systems. You could achieve this by writing good old plumbing code using the Kafka Producer/Consumer APIs using a language and client SDK of your choice. But you can do better!

This blog post demonstrates how you can use an open source solution (connector based) to ingest data from Kafka into Azure Cosmos DB Cassandra API. It uses a simple yet practical scenario along with a re-usable setup using Docker Compose to help with iterative development and testing. You will learn about:

Overview of Kafka Connect along with the details of the integration

How to configure and use the connector to work with Azure Cosmos DB

Use the connector to write data to multiple tables from a single Kafka topic

By the end of this blog, you should have a working end to end integration and be able to validate it.

The code and configuration associated with this blog post is available in this GitHub repository — https://github.com/abhirockzz/kafka-cosmosdb-cassandra

Hello, Kafka Connect!

Kafka Connect is a platform to stream data between Apache Kafka and other systems in a scalable and reliable manner. Besides the fact that it only depends on Kafka, the great thing about it is the fact that it provides a suite of ready-to-use connectors. This means that you do not need to write custom integration code to glue systems together; no code, just configuration! In case an existing connector is not available, you can leverage the powerful Kafka Connect framework to build your own connectors.

There are two broad categories of connectors offered by Kafka Connect:

Source connector: It is used to to extract data “from” an external system and send it to Apache Kafka.

Sink connector: It is used to send existing data in Apache Kafka “to” an external system.

In this blog post, we will be using the open source DataStax Apache Kafka connector which is a Sink connector that works on top of Kafka Connect framework to ingest records from a Kafka topic into rows of one or more Cassandra table(s).