Orchestrating change data capture to a data lake
By: Vimarsh Karbhari
Tags: Machine Learning, Data Science, Artificial Intelligence, Data Engineering, Spark

Orchestrating change data capture to a data lake

Building change data capture(CDC) with Spark Streaming SQL

What is change data capture?

If you are a data engineer, CDC will not appear foreign to you. It is an approach to data integration that is based on the checking, capture and delivery of the change to the data source interface. CDC can help loading the source table into your data lake. There is huge amount of data stored in the database or application source and the data team would want to analyze this table. Running queries against the live production database could result in degradation of performance for the external application. The CDC process/pipeline is used to load the table to the external data lake. The apps that need access can use ETL or ad-hoc on the target table stored in the data lake for analysis.

Possible architectural considerations

There are a lot of CDC solutions, including incremental scheduled import jobs or real time jobs. Sqoop is an open source tool that could be used to transport data between Hadoop and relational database. The team can build daily jobs which could be used to load the data into the data lake. This could still create a huge load on the database and affect performance. Hence, the schedule needs to be worked out to make sure application performance is not affected. This scenario could severely limit real time queries and analysis on top of limitations that Sqoop additionally has.

The problem with the previous architecture is the load and bottleneck it creates on the application database. To solve for those issues we can use binlog. The binlog is a set of sequence log files that could then record or insert, update, delete operations. For this streaming CDC pipeline using binlog, first, we can use some out source to us, like JSON or Maxwell to sync binlog to Kafka or some other comparable service. Then the apps downstream can leverage Spark streaming to consume the topic from Kafka sequencing. Sequencing parses up binlog record into the targeted storage system. We could support Insert, Update, Delete, like Kudu or data or HBase. This solution comes with a set of operational challenges if the data is too large especially with Kudu.

Based on our previous architectural consideration, binlog will limit load on application database but it will result in a few other challenges. We can solve for those operational challenges and issues using Spark Streaming SQL. We could build a CDC process/pipeline using Spark Streaming SQL. We can drive Streaming SQL to parse the binlog and merge them into a data lake.

Orchestration for Spark Streaming SQL

SQL is a declarative language. Almost all of the data engineers have SQL skills especially in database and data warehouse like MySQL, HIVESQL or SparkSQL. The advantage of using Stream SQL, even if the developers are not familiar with Spark Streaming or Java or Scala, they can still easily develop streaming processing. Additionally, it is also low cost if you want to migrate from best SQL job to a Streaming SQL job.

As a part of this orchestration, we should have sync the binlog of the table to Kafka using Debezium or other similar products. Binlog is a different format hence, the binlog parser will also be different from a normal parser. We can use Spark Streaming SQL to consume the binlog from Kafka, and parse the binlog using the operation type of this record, like insert, update or delete, then we can merge this past record data to the data lake. Internally, Spark Streaming receives live input data streams and divides the data into batches, which are then processed by the Spark engine to generate the final stream of results in batches.

Step one, we should create two tables, one source which is the Kafka table, and another is target data table. Step two, we create a streaming scan on top of the Kafka table and set some parameters in options clause, like studying offsets, max offset per trigger. Step three would house the major logic of the CDC pipeline. We create a screen to wrap the merge into statements and the job parameters. Step four, we can use Streaming SQL command to launch the SQL file. This command will launch a client mode streaming job. Once that is set and the job runs, we can view the CDC streaming pipeline. At this point, we can check that we can query the data table in the outer link, if there are some data changes in the source database table and they should match. For each batch of the streaming, we call the data’s merge function to merge the parse binlog record into the target data table. That should finish the orchestration.

Post this setup, we will need to setup monitoring and metrics on each step of this process. These metrics should provide us with values to visualize bottlenecks and the flow of data. In addition, we need to setup alerts on the operational metrics. These would allow the team to consume and respond to alerts respectively.

Conclusion

The process above should be able to provide a way to setup a basic CDC pipeline which can handle billions of CDC events relatively well. We can also do the update, delete of mode of ratio on the data table. On top of the solution the team could support schema enforcement and evolution which can provide better data quality and data management. Time travel provides snapshots of data, then we can query any earlier worsening of the data. Only Spark can write data through data, including batch mode and streaming mode, and leverage Presto on how spark can query data from data.