Debezium, SMT, Simple Message Transforms, Kafka, Kafka Connect, Postgres, connector
By: Okan Yildirim
Tags: Simple Message Transforms, Kafka, Postgres, Smt, Debezium

Case 1: Ignoring Unnecessary Columns

By default, when there is a change on a row, Debezium sends all columns in the message. However, we don’t need every column in the message. For example, created_date and status columns are not important for our case and we want to send messages without them. In this case, we add column.blacklist to our configuration:

------

"column.blacklist":

"public.settlements.created_date, public.settlements.status"

------

Case 2: Ignoring Tombstone Message

In addition, when delete operation Debezium generates a Kafka record called tombstone message with a value of null. We also may not want tombstone records.

Using the connector option tombstones.on.delete you can control whether upon record deletions a tombstone event should be emitted or not.

-------

“tombstones.on.delete”: “false”

-------

Case 3: Filtering Initial Snapshot

In our case, we need the initial snapshot for past values in the database. But we were interested in only a subset of the records to migrate. Debezium lets us specify a SELECT statement that filters a specific subset of snapshot. When the connector performs a snapshot, it executes this SELECT statement to retrieve specific data from that table.

According to the Debezium documentation you can specify which table rows are included in snapshots by using snapshot.select.statement.overrides configuration as databaseName.tableName. But be careful at this point, for PostgreSQL, Debezium expects you to define this parameter as schemaName.tableName.

---------

"snapshot.select.statement.overrides":"public.settlements", "snapshot.select.statement.overrides.public.settlements":

"SELECT * FROM public.settlements WHERE order_date > '2020-11-06'"

---------

Case 4: Message Filtering — Single Message Transform (SMT)

In many cases, we might be interested in only a subset of the events produced by Debezium. It would be a special business requirement or we might not want to consume unnecessary messages. Debezium delivers every data change event message by default. We could have filtered messages we were interested in on the consumer side. But in our case, the table we wanted to produce has a jsonb column that stores an array of ids with approximately 1–1.5 MB data. When there is an update operation, it generates approximately 3 MB payload with after and before part of the message. Because of the huge payload, we want to produce events with special changes instead of every message. For such problems Debezium provides the single message transforms that use transformation API in Kafka Connect. Confluent defines SMT as:

SMTs transform inbound messages after a source connector has produced them, but before they are written to Kafka.

There are different types of transforms such as filter, reroute, unwrap, outbox, etc. but we will use filter type in our case. The filter SMT supports scripting languages that integrate with JSR 223. The filter SMT and any implementation of the JSR 223 API are not included in the Debezium by default. So we need to add SMT and JSR 223 API implementation by following steps in message filtering debezium documentation.

Firstly, we need to download debezium-scripting-1.4.0.Alpha2.tar.gz for SMT. And also need an implementation for JSR 223 API. In our case, we used Groovy as an expression language. You can also use JavaScript or Javascript with Graal.js.

When we get these jar files, we need to extract them into the Debezium plug-in directories of our Kafka Connect environment. For PostgreSQL connector, extract them into ‘/kafka/connect/debezium-connector-postgres/’.

connect:

container_name: debezium

image: debezium/connect:1.3

ports:

- 8083:8083

links:

- kafka

- postgresql

environment:

- BOOTSTRAP_SERVERS=kafka:9092

- CONFIG_STORAGE_TOPIC=debezium_connect_config

- OFFSET_STORAGE_TOPIC=debezium_connect_offset

- STATUS_STORAGE_TOPIC=debezium_connect_status

- GROUP_ID=settlement-debezium-connect

- CONNECT_PRODUCER_MAX_REQUEST_SIZE=3145728

volumes:

- $PWD/jars:/kafka/connect/debezium-connector-postgres

After your Debezium container is ready, you can check whether jars files moved correctly.

Now that we have added plugins we are ready to use filtering. First of all, we need to configure filter SMT in the connector with the configurations given below.

---------

"transforms": "filter",

"transforms.filter.type": "io.debezium.transforms.Filter"

---------

We also need to specify the regular expression language by configuration given below.

---------

"transforms.filter.language": "jsr223.groovy"

---------

Now we can define our filter conditions by using transforms.filter.condition. We can filter messages by operations and fields. In our example, we need an event when there is an update operation on sent field from false to true.

---------

"transforms.filter.condition": "value.op == 'u' && value.before.sent == false && value.after.sent == true"

---------

When we had completed the configuration we faced a problem that said the value has not ‘op’ field. Then we realized these messages come from Debezium configuration topics. The expression evaluated for every message in the connector including our Debezium config, offset, and status topics. To resolve this issue we need to specify a regular expression with transforms.filter.topic.regex that evaluates the name of the destination topic for an event to determine whether to apply to filter logic. In this way, SMT ignores the event that comes from Debezium configuration topics and filters only messages belong to the target topic.

---------

"transforms.filter.topic.regex":

"settlement-debezium.public.settlements"

---------

Our final curl is given below:

Case 5: Heavy Message Payload

We have mentioned the size of our event message approximately 3 MB. But in kafka connect producer configuration the maximum size of a request is 1048576 byte by default. So we need to change max.request.size of producer config.

Environment variables that start with CONNECT_ will be used to update the Kafka Connect worker configuration file. To update kafka connect configurations we can use CONNECT_PRODUCER_MAX_REQUEST_SIZE as an environment variable.

BOOTSTRAP_SERVERS:”localhost:9092”

CONFIG_STORAGE_TOPIC: “settlement-debezium-config”

OFFSET_STORAGE_TOPIC: “settlement-debezium-offset”

STATUS_STORAGE_TOPIC: “settlement-debezium-status”

GROUP_ID: “settlement-debezium-connect”

CONNECT_PRODUCER_MAX_REQUEST_SIZE: ‘3145728’

It allows you to produce a message but it is not enough. Your topic max.message.bytes configuration must be tuned. You can create a topic as follows:

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 2 --partitions 12 --topic settlement-debezium.public.settlements --config max.message.bytes=3000000

Lastly, on the consumer side, don’t forget to specify configurations such as

FETCH_MAX_BYTES_CONFIG

MAX_PARTITION_FETCH_BYTES_CONFIG

MAX_POLL_RECORDS_CONFIG

that allows you to consume these big messages. Moreover, beware of giving

SESSION_TIMEOUT_MS_CONFIG

MAX_POLL_INTERVAL_MS_CONFIG

because processing these messages may cost too much time.

Conclusion

In this article, we just want to share our experiences on Debezium, what kind of problems we faced, and how we solved these problems by tuning connector configurations. In this way we filter:

Unnecessary Columns

Tombstone Messages

Initial Snapshot

CDC Messages

and handle huge payloads.

If you want to use up-and-coming technologies such as Debezium on production, contribute these works, and develop highly scalable applications join to Trendyol family.