Project: YugabyteDB (App Definition and Development / Database)
File: cdc-confluent-cloud.md

title: Confluent Cloud tutorial for YugabyteDB CDC headerTitle: Confluent Cloud linkTitle: Confluent Cloud description: Confluent Cloud for Change Data Capture in YugabyteDB. headcontent: menu: v2.18: parent: cdc-tutorials identifier: cdc-confluent-cloud weight: 30 type: docs This tutorial describes how to configure a YugabyteDB connector to publish changes to Confluent Cloud, and assumes some familiarity with Docker and YAML files. Set up a Confluent Cloud cluster on the provider of your choice. For more information, refer to in the Confluent documentation. Create an API key for your Confluent cluster and save it. Refer to for instructions. To create a Kafka Connect image with the YugabyteDB connector, start with the . Create a directory which will be used to store all related files. ```sh mkdir kafka-connect-ccloud && cd kafka-connect-ccloud ``` <!-- TODO Vaibhav: Step 2 and 3 can be combined, see why is not working --> Download the YugabyteDB connector jar. ```sh curl -so debezium-connector-yugabytedb-1.9.5.y.22.jar https://github.com/yugabyte/debezium-connector-yugabytedb/releases/download/v1.9.5.y.22/debezium-connector-yugabytedb-1.9.5.y.22.jar ``` Create a `Dockerfile` with the following contents: ```Dockerfile FROM confluentinc/cp-server-connect:7.4.0 ADD debezium-connector-yugabytedb-1.9.5.y.22.jar /usr/share/java/kafka/ USER 1001 ``` To build the image, execute the following command: ```sh docker build . -t custom-connect:latest ``` Create a `docker-compose.yaml` file with the following contents: ```yaml version: '3' services: kafka-CCLOUD-BROKER-ENDPOINT.:latest container_name: kafka-connect-ccloud ports: 8083:8083 environment: CONNECTLOG4JAPPENDERSTDOUTLAYOUT_CONVERSIONPATTERN: "[%d] %p %X{connector.context}%m (%c:%L)%n" CONNECTCUBKAFKA_TIMEOUT: 300 CONNECTBOOTSTRAPSERVERS: "CCLOUD-BROKER-ENDPOINT.confluent.cloud:9092" CONNECTRESTADVERTISEDHOSTNAME: 'kafka-connect-ccloud' CONNECTRESTPORT: 8083 CONNECTGROUPID: kafka-connect-group-01-v04 CONNECTCONFIGSTORAGE_TOPIC: kafka-connect-configs CONNECTOFFSETSTORAGE_TOPIC: kafka-connect-offsets CONNECTSTATUSSTORAGE_TOPIC: kafka-connect-status CONNECTKEYCONVERTER: org.apache.kafka.connect.json.JsonConverter CONNECTVALUECONVERTER: org.apache.kafka.connect.json.JsonConverter CONNECTLOG4JROOT_LOGLEVEL: 'INFO' CONNECTLOG4JLOGGERS: 'org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR' CONNECTCONFIGSTORAGEREPLICATIONFACTOR: '3' CONNECTOFFSETSTORAGEREPLICATIONFACTOR: '3' CONNECTSTATUSSTORAGEREPLICATIONFACTOR: '3' CONNECTPLUGINPATH: '/usr/share/java,/usr/share/confluent-hub-components/,/usr/share/java/kafka/' CONNECTREQUESTTIMEOUT_MS: "20000" CONNECTRETRYBACKOFF_MS: "500" CONNECTSSLENDPOINTIDENTIFICATIONALGORITHM: "https" CONNECTSASLMECHANISM: "PLAIN" CONNECTSECURITYPROTOCOL: "SASL_SSL" CONNECTSASLJAASCONFIG: "org.apache.kafka.common.security.plain.PlainLoginModule required username='CCLOUDUSER' password='CCLOUD_PASSWORD';" CONNECTCONSUMERSECURITYPROTOCOL: "SASLSSL" CONNECTCONSUMERSSLENDPOINTIDENTIFICATION_ALGORITHM: "https" CONNECTCONSUMERSASL_MECHANISM: "PLAIN" CONNECTCONSUMERSASLJAASCONFIG: "org.apache.kafka.common.security.plain.PlainLoginModule required username='CCLOUDUSER' password='CCLOUDPASSWORD';" CONNECTCONSUMERREQUESTTIMEOUTMS: "20000" CONNECTCONSUMERRETRYBACKOFFMS: "500" CONNECTPRODUCERSECURITYPROTOCOL: "SASLSSL" CONNECTPRODUCERSSLENDPOINTIDENTIFICATION_ALGORITHM: "https" CONNECTPRODUCERSASL_MECHANISM: "PLAIN" CONNECTPRODUCERSASLJAASCONFIG: "org.apache.kafka.common.security.plain.PlainLoginModule required username='CCLOUDUSER' password='CCLOUDPASSWORD';" CONNECTPRODUCERREQUESTTIMEOUTMS: "20000" CONNECTPRODUCERRETRYBACKOFFMS: "500" command: bash -c | echo "Launching Kafka Connect worker" /etc/confluent/docker/run & echo "Waiting for Kafka Connect to start listening on localhost:8083 " while : ; do curlstatus=$$(curl -s -o /dev/null -w %{httpcode} http://localhost:8083/connectors) echo -e $$(date) " Kafka Connect listener HTTP state: " $$curl_status " (waiting for 200)" if [ $$curl_status -eq 200 ] ; then break fi sleep 5 done sleep infinity ``` {{< note title="Using Schema Registry" >}} If the configuration is such that it needs schema registry as well, then you need to add the following environment variables to the Docker compose file: ```yaml CONNECTKEYCONVERTER: io.confluent.connect.avro.AvroConverter CONNECTKEYCONVERTERSCHEMAREGISTRY_URL: "https://SCHEMA-RGISTRY-CCLOUD-ENDPOINT.confluent.cloud" CONNECTKEYCONVERTERBASICAUTHCREDENTIALSSOURCE: "USER_INFO" CONNECTKEYCONVERTERSCHEMAREGISTRYBASICAUTHUSERINFO: "SCHEMAREGISTRYUSER:SCHEMAREGISTRYPASSWORD" CONNECTVALUECONVERTER: io.confluent.connect.avro.AvroConverter CONNECTVALUECONVERTERSCHEMAREGISTRY_URL: "https://SCHEMA-REGISTRY-CCLOUD-ENDPOINT.confluent.cloud" CONNECTVALUECONVERTERBASICAUTHCREDENTIALSSOURCE: "USER_INFO" CONNECTVALUECONVERTERSCHEMAREGISTRYBASICAUTHUSERINFO: "SCHEMAREGISTRYUSER:SCHEMAREGISTRYPASSWORD" ``` {{< /note >}} {{< note title="Using authentication and authorization" >}} To configure authentication, follow the instructions in . To configure authorization, follow the instructions in . {{< /note >}} Start the Kafka Connect cluster: ```sh docker-compose up ``` Deploy the connector. For more information, refer to .