Retry with Avro in Kafka
By: Neelaksh Chauhan
Tags: Tech, Avro, Apache Kafka, Kafka

This is a follow up of our blog series on kafka retries. To know more about how we handle retries/backup with kafka, head over here.

Recently, we wanted to introduce Avro in our kafka infrastructure to enable teams to allow enforcing/sharing of contracts for data publishing and consumption from topics. But we ran into a few challenges with respect to message producing/consuming retries which we are going to discuss here.

Background

We have some service level agreements (SLA’s) for delivery of json based messages in case there are issues in kafka cluster or the consumer:

Producer’s failure retry: Kafka messages that fail when being produced are written to a backup kafka cluster in a common back up topic on a separate cluster and written back to the primary cluster’s original topic after a certain duration. Consumer’s failure retry: Kafka messages that fail to get processed when being consumed are pushed into a common retry topic and retried after a certain amount of delay.

Challenges

a) Handling metadata with Avro messages: In json based messages, we have a “metadata” block inside the data itself. This metadata block allows us to achieve having common backup and retry topics as it can contain information like the groupId and the original topic names in a standard format. We can deserialize this object and simply extract any information from the metadata component.

But this meant we had to enforce a standard format of our kafka messages. Imagine something like this,

{

“metadata”:{

//metadata

},

“payload” :{

//whatever data you want to pass

}

}

We had to find a way to transfer this metadata without requiring deserialization of message because otherwise all the teams who would use this architecture would need to embed this metadata into their Avro schemas as well .

We studied and found that kafka headers¹ could allows us to transfer metadata without making it a part of the core message. This was not utilized for the json based functionality previously because when that was developed we were on a version of kafka that did not have support for kafka headers.

Kafka headers are constructs that allow you to attach data as key value pairs with the message. This gave us the potential of dealing with Avro messages as byte blobs when the need comes and using kafka headers to transfer any necessary metadata.

We plan on eventually moving the json based constructs to kafka headers too now! We’ll discuss more about the exact use case of the data points in headers towards the end.

b) Pushing data to common fallback topics: In case of failures at producer or consumer side, we need to push data to common retry topics which was another challenge.

The issue with having common failure topics for multiple schemas was that as of Confluent 4.0.0 and Kafka 1.1.0, Avro schema names are coupled with the topic names². This has since been updated with more options in newer versions³ but that would require an upgrade which would involve too many changes.

So, we had to devise an approach to get around this. We came up with one as described below:

Serializer end (while producing data) -

Image showing how a message flows when an error is encountered on producer side

We extended the Avro serializer and passed it the original topic’s name via kafka headers. In case something goes wrong while producing message into the primary kafka cluster, this allows us to basically lie to the Avro serializer and make it serialize based on the Avro schema of the original topic’s name while pushing data to the common topic in the backup kafka cluster.

Code for the above Avro serializer:

public class KafkaAvroSerializerWithOriginalTopic extends KafkaAvroSerializer implements

ExtendedSerializer<Object> { @Override

public byte[] serialize(String topic, Headers headers, Object data) {

Header targetTopicHeader = headers.lastHeader(HEADER_NAME_TARGET_TOPIC);

if (targetTopicHeader != null) {

return super.serialize(new String(targetTopicHeader.value()), data);

}

return super.serialize(topic, data);

}

}

Deserializers end ( while consuming data) -

Image showing overall flow if consumer encounters an error while interacting with some further downstream service

We extended the Avro deserializer also to return a wrapped object that would contain,

a) The Avro deserialized data

b) The original data in byte format read by the consumer

Now in case we encounter a failure at consumer’s end, we push the data in b) part to the retry topic. By pushing b) instead of say serializing the data again we avoid any other consumers with different version of schemas unnecessarily generating exceptions in edge cases because we are simply propagating the message we originally received from the producer in this case.

As a side note, for schema compatibility⁴ we decided to go with forward compatibility to allow addition of new required fields without impacting existing consumers.

Code for the above Avro deserializer -

public class KafkaAvroDeserializerWithOriginalData extends KafkaAvroDeserializer { @Override

public Object deserialize(String s, byte[] bytes) {

try {

return new AvroConsumerPayload<>(super.deserialize(s, bytes), bytes, true);

} catch (Exception ex) {

return new AvroConsumerPayload<>(null, bytes, false);

}

} }

So that pretty much covers how data is pushed to these “fallback” topics. But how do we get it back into the original topics?

We have kafka listeners on these “fallback” topics that can parse the json based messages and extract the metadata. But this is not possible for a topic that has multiple Avro messages which we just consider as a bunch of byte blobs that we can’t serialize. This is where kafka headers come in handy. We add metadata like original topic’s name and groupId to kafka headers when pushing data to “fallback” topics. This allows us to push data back to the relevant topic by parsing the information from kafka headers without looking inside the byte blob that is the Avro message.

All this together allowed us to meet an SLA that is equivalent of what we have for json messages.

References:

[1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers

[2] https://docs.confluent.io/current/schema-registry/serdes-develop/index.html#overview

[3] https://github.com/confluentinc/schema-registry/pull/680

[4] https://docs.confluent.io/platform/current/schema-registry/avro.html#summary