Migration Tool and Tips of Kafka cross-cluster replication: MirrorMaker | by Ning.Zhang | Towards Data Science

This article is about tools and tips that migrate from other cross-cluster Kafka replication tools to the new MirrorMaker (or “MirrorMaker 2”)
There are several advantages of new MirrorMaker. To name a few:
always open-source in Apache Kafka ecosystem
supported by open community with large and diverse userbase
critical features that have not been supported by other tools, e.g. exactly-once semantics, see my previous blog
Recently users from the community have been migrating from Confluent Replicator (an enterprise commercial “cross-cluster” replication tool) to MirrorMaker and they were facing the following problem:
messages which are already replicated by Confluent Replicator is getting replicated again when the MirrorMaker is started on the same topic. This should not happen as messages are getting duplicated at the target cluster.
As Confluent Replicator and MirrorMaker are both built as “connector” of Kafka Connect, or more specifically Source Connector, they maintain an internal “offset topic” to keep track the consumer offsets and are loaded when Confluent Replicator or MirrorMaker restarts.
For Confluent Replicator, offset topic defaults to “connect-offsets”. However the offset topic is named “mm2-offsets.primary.internal” in MirrorMaker.
Another difference is the content format. For Confluent Replicator, the message content looks like:
[“replicator”,{“topic”:”foo”,”partition”:2}] {“offset”:1}
(“replicator” is the default consumer group name of Confluent Replicator)
For MirrorMaker, the message context looks like:
["MirrorSourceConnector",{"cluster":"primary","partition":1,"topic":"foo”}] {"offset":10}
(“MirrorSourceConnector” is the default consumer group name of MirrorMaker)
It may sound feasible to tweak open-source MirrorMaker using different offset topic and adapt to a different message format. But from practice, this requires in-depth knowledge and code change of MirrorMaker.
As a community thread discussed, here is a non-intrusive, neat solution:
For each topic and partition, whenever a new message is replicated a new message with same key but increased offset is produced to the connect-offsets topic, convert the key of this message to Mirror Maker format and produce it in the internal "offset topic" of Mirror Maker.Key : ["replicator-group",{"topic":"TEST","partition":0}]Value: {"offset":24}After posting the message, once the mirror maker is restarted, it will read the internal topic to get the latest offset of that topic for which the message has to be replicated and this way we can ensure no duplicate messages are replicated.Key: ["mm-group",{"cluster":"primary","partition":0,"topic":"TEST"}]Value: {"offset":24}
The following snippet implements the above idea by leveraging native kafka-console-consumer.sh and kafka-console-producer.sh (part of Apache Kafka), leading to several benefits:
no third-party dependency
programming language and OS agnostic
better integrate with Kafka (as kafka-console-*.sh should always work)
easy execution and monitoring, e.g. corn job
The README is here: https://github.com/ning2008wisc/mm2-migration
The idea proposed in the community thread is neat and simple that can be applied to other similar migration scenarios involving Apache Kafka. In future more useful tools and tips will be discussed here. Please stay tuned!