Exceptions and Retry Policy in Kafka
By: Victor Alekseev
Tags: Message Queue, Kafka, Messaging, Software Architecture, Retry

General exception management

This chapter will look at the exceptions in terms of message retry, the different strategies for retry execution, and the possible architecture for configuring this aspect of the application.

Classification of exceptions

First, we need to divide possible errors into several groups, each of which requires its specific approach to retry. The meaning of the terms stateful and stateless applied to the exception handling process should be clarified beforehand.

“Stateless” in this case means that it is up to the application to implement a process of sequential processing attempts. We read the message from the topic once and try to process it several times. The state of the datastore (the group’s offset inside the Kafka partition) is not changed in this case.

in this case means that it is up to the application to implement a process of sequential processing attempts. We read the message from the topic once and try to process it several times. The state of the datastore (the group’s offset inside the Kafka partition) is not changed in this case. In contrast, during “Stateful” processing, the implementation of iterations is left to the data store. Over and over, we “discard” a message we have already received, putting it back and forcing the consumer to reread it. Such discarding can be implemented by manipulating the group’s offset in the Kafka partition.

Possible types of exceptions:

Not retryable exceptions (NRE) — are errors that will most likely always be handled with an error. In this case, trying to re-execute the request will not allow us to fix the situation. In the vast majority of cases, we can only log the fact that an exception has been raised, indicating a bug in our application, and send the message into Dead Letter Queue (DLQ) for future analysis. Typical examples of such exceptions are NullPointerException, ClassCastException, NoSuchMethodException, DeserializationException, MessageConversionException, MethodArgumentResolutionException , and so on. This class of exceptions also includes errors in non-idempotent business operations that should not be automatically repeated and cannot be automatically reversed.

— are errors that will most likely always be handled with an error. In this case, trying to re-execute the request will not allow us to fix the situation. In the vast majority of cases, we can only log the fact that an exception has been raised, indicating a bug in our application, and send the message into Dead Letter Queue (DLQ) for future analysis. Typical examples of such exceptions are , and so on. This class of exceptions also includes errors in non-idempotent business operations that should not be automatically repeated and cannot be automatically reversed. Stateless retryable exceptions (SLE) — indicate problems that can be fixed by a re-call and will most likely be fixed soon. Most business level exceptions are of this type. In this case, the critical factor is the expected fixing speed, allowing a sequence of retries to be made within the timeout allocated to process a single message. Until the entire list of message processing attempts is complete, the next request will not be pulled from the partition by the service. After a series of several not successful attempts, such messages are usually sent to DLQ too.

— indicate problems that can be fixed by a re-call and will most likely be fixed soon. Most business level exceptions are of this type. In this case, the critical factor is the expected fixing speed, allowing a sequence of retries to be made within the timeout allocated to process a single message. Until the entire list of message processing attempts is complete, the next request will not be pulled from the partition by the service. After a series of several not successful attempts, such messages are usually sent to DLQ too. Stateful blocking retryable exceptions (SBE) — for which we don’t know how many attempts it will take before the message is appropriately handled for this type of exception. Consequently, we cannot iterate in memory and must periodically reread the message repeatedly from the partition for the next attempt. Simultaneously, the identified problem is quite cardinal for the application and is likely to block the execution of all subsequent requests. Therefore, we have to suspend the flow of incoming messages until we have safely processed this one. Typical examples of such exceptions are: SocketTimeoutException, ConnectException, TransactionTimedOutException, UnknownHostException . In this case, we can never send failed messages to DLQ since the application is undoubtedly inoperable and must stop processing messages in general.

— for which we don’t know how many attempts it will take before the message is appropriately handled for this type of exception. Consequently, we cannot iterate in memory and must periodically reread the message repeatedly from the partition for the next attempt. Simultaneously, the identified problem is quite cardinal for the application and is likely to block the execution of all subsequent requests. Therefore, we have to suspend the flow of incoming messages until we have safely processed this one. Typical examples of such exceptions are: . In this case, we can never send failed messages to DLQ since the application is undoubtedly inoperable and must stop processing messages in general. Stateful not blocking retryable exception (SNBE) — are similar to the previous one, with the difference that the problem detected affects only the processing of this message. Subsequent messages can most likely be handled by the application safely. Usually, this type of situation is related to the fact that the data required to execute the request is not yet available. We have to wait a little while for it to be delivered to the appropriate repository. Whether or not to send messages whose processing causes exceptions of this type to DLQ depends on business requirements and, above all, on the SLA and the specified incident handling policy.

The enum ExceptionType can express this classification with the next members: NRE, SLE, SBE, SNBE.

As a rule, it is not the exception itself used to classify an exception, but the presence of some specific exception (or the successor of some specific exception) in the chain of its causes. Spring provides us with an extremely convenient template for our classifier in the form of the BinaryExceptionClassifier class. It is recommended for convenience to add the following methods to the successor: addException(class, defaultValue) and addException(className, defaultValue). The last one is used when the exception list is externally configured or the type of the necessary exception is available only at runtime.

In some cases, the same class’s exceptions may differ only in some other aspects, such as message text, for example. As an illustration, we can consider working with PostgreSQL when we meet with only one type of exception — PSQLException, containing different text messages. SBE exceptions include only those containing the following message fragments: “canceling statement due to statement timeout”, “canceling statement due to user request”, or “The connection attempt failed”. Therefore, it is also recommended to implement a programmatic exception classification strategy based on an interface ExceptionTypeResolver too. This interface can contain only a single method ExceptionType classify(consumerRecord, consumer, throwable).

We also need to be able to mask some exceptions with others. For example, in some cases, NullPointerException is used by legacy code when an object is not found in some repository. Thus, in fact, we deal not with NRE, but with SNBE. Fortunately, BinaryExceptionClassifier examines the chain of causes and stops at the first exception it knows of. Therefore, if we wrap NullPointerException in StatefulBlockingRetryException, our exception will be classified correctly. Accordingly, it is necessary to have one type of wrapper for each of the above exception types.

Retry strategies

To implement a sequence of execution attempts, Spring offers the standard spring-retry module with the following main utils and annotations:

@EnableRetry on the configuration

on the configuration @Retryable on the methods, which has to be called several times automatically

on the methods, which has to be called several times automatically RetryTemplate for repeating execution some codes fragment

for repeating execution some codes fragment BackOffPolicy for definition parameters of iterations — count of attempts, pauses, and so on

Kafka Spring uses all these tools, but there is one problem: they are all too smart and imply an iteration externally to the code being executed. For NRE and SLE processing, this is enough, but we need to calculate for the last two types of exceptions how long we should delay the next attempt to handle the message. We also need a convenient way to configure this strategy globally or at a specific topic level. It’s best if this would be possible based on a single string value of some configuration parameter.

So we need an interface RetryPolicy with a roughly similar set of methods:

static RetryPolicy parsePolicy(String strArgs)

int getNext(int current)

int getMaxInterval()

void parse(String strArgs)

As practice shows, the following strategies are enough to work:

An unlimited exponential retry policy with parameters: initialDelay, maxDelay, multiplier .

. A limited exponential retry policy with parameters: initialDelay, maxDelay, multiplier, maxDelayCount .

. An arbitrary strategy with direct enumeration of delays: 5000x3, 10000x5, 30000x10, 60000x20.

Configuration

As far as we can see from two different chapters, both aspects of handling messages from various topics can be the subject of the configuration: classification of the possible exceptions and definition retry policy for each of them.

So, the next fragment of the configuration can be applied on the two levels: per topic level and on a global level as a default configuration:

We can hide all aspects of execution related to the configuration behind an ExceptionClassificationService service. It provides an only operation — Pair<ExceptionType,RetryPolicy> classify(records, consumer, exception).

This service can be implemented with the next principal architecture: