Project: YugabyteDB (App Definition and Development / Database)
File: change-data-capture.md

title: Change data capture (CDC) headerTitle: Change data capture (CDC) linkTitle: Change data capture (CDC) description: Learn how YugabyteDB supports asynchronous replication of data changes (inserts, updates, and deletes) to external databases or applications. techPreview: /preview/releases/versioning/#feature-availability menu: v2.16: parent: architecture-docdb-replication identifier: architecture-docdb-replication-cdc weight: 1160 type: docs Change data capture (CDC) in YugabyteDB provides technology to ensure that any changes in data (inserts, updates, and deletions) are identified, captured, and automatically applied to another data repository instance, or made available for consumption by applications and other tools. Change data capture is useful in a number of scenarios, such as the ones described here. Some microservices require a stream of changes to the data and using CDC in YugabyteDB can provide consumable data changes to CDC subscribers. Remote systems may subscribe to a stream of data changes and then transform and consume the changes. Maintaining separate database instances for transactional and reporting purposes can be used to manage workload performance. Maintaining multiple data centers enables enterprises to provide: High availability (HA) Redundant systems help ensure that your operations virtually never fail. Geo-redundancy Geographically dispersed servers provide resiliency against catastrophic events and natural disasters. Auditing and compliance requirements can require you to use CDC to maintain records of data changes. {{< note title="Note" >}} In the sections below, the terms "data center", "cluster", and "universe" are used interchangeably. We assume here that each YB universe is deployed in a single data center. {{< /note >}} ```goat .-. | Node 1 | | '-' '' | | | YB-Master | | YB-TServer | | CDC Service is stateless CDC Streams metadata | | (Stores CDC | | '-' | | | replicated with Raft | | metadata) | | | CDC Service | | | | .-> | | | | .-. | | <' | | .-. .. | | '-' | | |_ | | v v .-. .-. | Node 2 | | Node 3 | | '-' '' | | '-' '' | | | YB-Master | | YB-TServer | | | | YB-Master | | YB-TServer | | | | (Stores CDC | | '-' | | | | (Stores CDC | | '-' | | | | metadata) | | | CDC Service | | | | | metadata) | | | CDC Service | | | | | | | .-. | | | | | |

| | | .-. .. | | .-. .. | '-' '-' ``` Every YB-TServer has a `CDC service` that is stateless. The main APIs provided by the CDC service are: `createCDCSDKStream` API for creating the stream on the database. `getChangesCDCSDK` API that will be use by the client to get the latest set of changes. Creating a new CDC stream will return a stream UUID. This is facilitated via the tool. To consume the events generated by CDC, we will be using Debezium as the connector. It is an open-source distributed platform that needs to be pointed at the database using the stream ID. Steps to set up Debezium for YugabyteDB CDC are documented on the . Using the Debezium connector for YugabyteDB, we can push the changes from YugabyteDB to a Kafka topic which can then be used by any end user application for the processing and analysis of the records. All data changes for one row, or multiple rows in the same tablet, will be received in the order in which they occur. Due to the distributed nature of the problem, however, there is no guarantee for the order across tablets. For example, let us imagine the following scenario: Two rows are being updated concurrently. These two rows belong to different tablets. The first row `row #1` was updated at time `t1` and the second row `row #2` was updated at time `t2`. In this case, it is possible for CDC to push the later update corresponding to `row #2` change to Kafka before pushing the earlier update, corresponding to `row #1`. Updates for rows will be pushed at least once. With "at-least-once" delivery, you will never lose a message, but might end up being delivered to a CDC consumer more than once. This can happen in case of tablet leader change, where the old leader already pushed changes to Kafka, but the latest pushed `op id` was not updated in the CDC metadata. For example, imagine a CDC client has received changes for a row at times t1 and t3. It is possible for the client to receive those updates again. When you have received a change for a row for timestamp `t`, you will not receive a previously unseen change for that row from an earlier timestamp. This guarantees that receiving any change implies that all earlier changes have been received for a row.