Getting started with Kafka and Rust: Part 2
By: Abhishek Gupta
Tags: Messaging, Programming, Kafka, Rust

Switching to Manual commit

By default, the offset commit process is taken care of by the library itself. But we can exercise more control over it by switching to manual mode.

First thing would be do set enable.auto.commit to false - set("enable.auto.commit", "false");

At-least once delivery

To achieve this, we need to make sure we indeed process the message successfully before committing the offset. To simulate this, let’s write a function (named process ) that can fail randomly. We will then use this in our consumer loop and commit only when this functions returns successfully.

We will need to modify our consumer loop

Add manual offset commit based on response from the process function

function Add a label ( 'consumer_thread ) to our thread loop

We call process - this is to simulate processing of each record received by the consumer. In case the processing succeeds (returns Ok ), we commit the record using commit_message.

Note that the commit itself may fail. This should ideally be handled in the commit_callback implementation of ConsumerContext

Run the program

Rename the file src/3_manual_commit.rs to main.rs , and

to , and Execute cargo run

The program output is lengthy, please bear with me.

Output:

produced message with key user-1 in offset 22 of partition 2

produced message with key user-2 in offset 28 of partition 4

post_rebalance callback

rebalanced partition 0

rebalanced partition 1

rebalanced partition 2

rebalanced partition 3

rebalanced partition 4

rebalanced partition 5

received key user-5 with value User { id: 5, email: "user-5@foobar.com" } in offset 52 from partition 3

SUCCESSFULLY processed User info User { id: 5, email: "user-5@foobar.com" }

committed offset Offset(53) in partition 3

received key user-2 with value User { id: 2, email: "user-2@foobar.com" } in offset 28 from partition 4

SUCCESSFULLY processed User info User { id: 2, email: "user-2@foobar.com" }

produced message with key user-3 in offset 35 of partition 0

committed offset Offset(29) in partition 4

received key user-1 with value User { id: 1, email: "user-1@foobar.com" } in offset 22 from partition 2

FAILED to process User info User { id: 1, email: "user-1@foobar.com" }

loop encountered processing error. closing consumer...

post_rebalance callback

ALL partitions have been REVOKED

Notice these logs messages when process returns successfully:

received key user-5 with value User { id: 5, email: “user-5@foobar.com” } in offset 52 from partition 3 SUCCESSFULLY processed User info User { id: 5, email: “user-5@foobar.com” } committed offset Offset(52) in partition 3

For a failure scenario:

received key user-1 with value User { id: 1, email: “user-1@foobar.com” } in offset 22 from partition 2 FAILED to process User info User { id: 1, email: “user-1@foobar.com” } loop encountered processing error. closing consumer…

We ended up stopping the consumer when processing failed? The question here is:

How to handle messages that did not get processed?

Note that failure could happen due to many reasons. A couple of them are:

Processing failed (this is what we simulated in this example), or,

Processing was successful, but the commit failed

If we continue with our consumer loop after a failed message, we could end up losing messages (data loss). Why? It’s because the commit_message method also marks smaller offsets (less that the one being handled) as committed. For e.g. if you had a scenario where offset 20 from partition 5 failed to get processed (and committed), you continue processing and offset 21 from partition 5 was processed and committed successfully, you will end up missing data from offset 20 - this is because committing offset 21 will also commit offsets 20 and below. Even after you re-start the application, this will not be detected.

To prevent this…

You can either: