Repository: JasperFx/wolverine (2026-06-19T00:30:31Z)

Issue:
Kafka: apply the contiguous OffsetWatermark to the Store & PerMessage commit strategies (out-of-order completion under parallelism)
Follow-up from the Kafka re-evaluation epic #3134 and @mbar10's review in https://github.com/JasperFx/wolverine/issues/3134#issuecomment-4746794622.

## Problem

Kafka listeners default to `EndpointMode.BufferedInMemory` with `MaxDegreeOfParallelism = max(ProcessorCount, 5)`, so handlers run concurrently and **can complete out of order**.

The `KafkaOffsetCommitter.OffsetWatermark` already solves exactly this — it tracks the highest *contiguous* (gap-free) completed offset so a commit never moves the committed position past an offset whose predecessors are still in flight. But today that watermark is only wired into the **`Batch`** strategy.

The default **`StoreThenAutoFlush` (`Store`)** and **`PerMessage`** strategies call `StoreOffset` / `Commit(offset + 1)` per message with **no contiguity guard**. Under parallel, out-of-order completion they can advance the committed position past an in-flight predecessor. On a crash, that predecessor's offset has effectively been committed and won't be redelivered — a loss window in buffered mode.

## Scope / mitigations

- The no-loss guarantee for Wolverine has always come from **Durable mode + the inbox**, not offset bookkeeping, and the by-key concurrency path (#3140) commits at inbox-persist time in consumption order — so that path is already safe.
- This change is specifically about tightening **buffered-mode crash recovery** under parallelism for the `Store` and `PerMessage` strategies.

## Proposed change

Route the `Store` and `PerMessage` strategies through the same per-partition `OffsetWatermark` contiguous-watermark logic the `Batch` strategy already uses, so that:

- `Store` stores the contiguous watermark (`contiguous + 1`) rather than each message's own `offset + 1`.
- `PerMessage` commits the contiguous watermark rather than each individual offset.

so out-of-order completion can never store/commit ahead of the lowest in-flight offset.

## Acceptance

- A buffered, parallel listener that completes offsets out of order never commits/stores past an in-flight predecessor (unit test against `KafkaOffsetCommitter` covering all three strategies with shuffled completion order).
- No behavior change for strictly in-order completion.

Note: this is the offset-correctness half of @mbar10's review. The other half — fully native (DB-free) exactly-once via Kafka transactional EOS (`SendOffsetsToTransaction`/`CommitTransaction`) — is the existing outbox-vs-EOS gap and is tracked separately under #3134, not here.



Discussion:

Patch:
diff --git a/docs/guide/messaging/transports/kafka.md b/docs/guide/messaging/transports/kafka.md
index 8a7e7913b..378a14443 100644
--- a/docs/guide/messaging/transports/kafka.md
+++ b/docs/guide/messaging/transports/kafka.md
@@ -261,6 +261,16 @@ If you explicitly set `EnableAutoCommit = true` via `ConfigureConsumer`, Wolveri
 commits and leaves offset management entirely to the Kafka client. Pending/stored offsets are flushed on a
 graceful shutdown so progress is not lost.
 
+::: tip In-flight–safe under concurrency
+All three manual commit strategies (`StoreThenAutoFlush`, `PerMessage`, and the batch modes) route through a
+per-partition watermark, so when a listener processes messages concurrently (the default buffered mode runs
+up to `MaxDegreeOfParallelism` handlers at once) and a later offset finishes before an earlier one, the
+committed/stored position **never advances past a message that is still in flight**. The watermark also makes
+no assumption that offsets are contiguous, so it behaves correctly on compacted topics and on transactional
+topics read with `read_committed`, where the broker hands out offset gaps. As always, the strongest
+crash-safety still comes from the durable inbox (see _Idempotency &amp; Exactly-Once_ below).
+:::
+
 ### Recommended Configuration by Use Case
 
 **At-least-once delivery** (recommended for most use cases):
diff --git a/src/Transports/Kafka/Wolverine.Kafka/Internals/KafkaListener.cs b/src/Transports/Kafka/Wolverine.Kafka/Internals/KafkaListener.cs
index b42ca1ed0..b7fd17255 100644
--- a/src/Transports/Kafka/Wolverine.Kafka/Internals/KafkaListener.cs
+++ b/src/Transports/Kafka/Wolverine.Kafka/Internals/KafkaListener.cs
@@ -48,6 +48,10 @@ public KafkaListener(KafkaTopic topic, ConsumerConfig config,
                         result = _consumer.Consume(_cancellation.Token);
                         var message = result.Message;
 
+                        // Seed the offset watermark in consume (in-order) order so out-of-order handler
+                        // completion can never commit past this still-in-flight offset (GH-3161).
+                        _committer.Track(result.Topic, result.Partition.Value, result.Offset.Value);
+
                         // Non-blocking retry-tier topic (GH-3148): wait out the fixed delay relative to
                         // when the record was produced before reprocessing. Records in a tier are
                         // time-ordered, so the head record gates the rest.
diff --git a/src/Transports/Kafka/Wolverine.Kafka/Internals/KafkaOffsetCommitter.cs b/src/Transports/Kafka/Wolverine.Kafka/Internals/KafkaOffsetCommitter.cs
index 524a3320e..cdcebc22e 100644
--- a/src/Transports/Kafka/Wolverine.Kafka/Internals/KafkaOffsetCommitter.cs
+++ b/src/Transports/Kafka/Wolverine.Kafka/Internals/KafkaOffsetCommitter.cs
@@ -7,9 +7,17 @@ namespace Wolverine.Kafka.Internals;
 /// <summary>
 /// Encapsulates the offset-commit strategy for a single Kafka consumer (GH-3150). Shared by the
 /// single-topic and grouped listeners. Commits the *specific* TopicPartitionOffset of each completed
-/// message (never the consumer's global position), batches with a contiguous per-partition watermark
-/// so it never commits ahead of the lowest in-flight offset, and respects the user's
+/// message (never the consumer's global position) and respects the user's
 /// EnableAutoCommit/EnableAutoOffsetStore configuration.
+///
+/// All three manual strategies (PerMessage, Store, Batch) route through a per-partition
+/// <see cref="OffsetWatermark"/> so that under concurrent, out-of-order handler completion the
+/// committed/stored position can never move past an offset that is still in flight (GH-3161). The
+/// watermark is seeded by <see cref="Track"/> at consume time (offsets arrive in order per partition)
+/// and advanced by <see cref="Complete"/> as each message finishes; the committable position is the
+/// lowest still-in-flight delivered offset, or the high-water consumed offset + 1 when nothing is in
+/// flight — which also tolerates the offset gaps a compacted or transactional (read_committed) topic
+/// produces.
 /// </summary>
 internal sealed class KafkaOffsetCommitter
 {
@@ -107,11 +115,12 @@ internal static void ApplyTo(ConsumerConfig config, CommitMode mode)
     }
 
     /// <summary>
-    /// Record successful processing of a message at the given coordinates and commit/store as the
-    /// strategy dictates. The committed offset is always <paramref name="offset"/> + 1 (the next
-    /// offset to resume from), per Kafka convention.
+    /// Record that a message has been *delivered* to a handler (read off the partition, in order) so the
+    /// watermark knows it is in flight. Called from the consume loop before dispatch so the committable
+    /// position can never advance past a delivered-but-incomplete offset, even when an earlier message
+    /// finishes after a later one (GH-3161).
     /// </summary>
-    public void Complete(string topic, int partition, long offset)
+    public void Track(string topic, int partition, long offset)
     {
         if (_strategy == Strategy.Automatic)
         {
@@ -119,37 +128,62 @@ public void Complete(string topic, int partition, long offset)
         }
 
         var tp = new TopicPartition(topic, new Partition(partition));
-
-        if (_strategy == Strategy.PerMessage)
+        lock (_lock)
         {
-            CommitOffsets([new TopicPartitionOffset(tp, new Offset(offset + 1))]);
-            return;
+            WatermarkFor(tp).Track(offset);
         }
+    }
 
-        if (_strategy == Strategy.Store)
+    /// <summary>
+    /// Record successful processing of a message at the given coordinates and commit/store as the
+    /// strategy dictates. The committed offset is the lowest still-in-flight delivered offset (or the
+    /// high-water consumed offset + 1 when nothing is in flight) — the next offset to resume from, per
+    /// Kafka convention — so out-of-order completion never advances the position past in-flight work.
+    /// </summary>
+    public void Complete(string topic, int partition, long offset)
+    {
+        if (_strategy == Strategy.Automatic)
         {
-            StoreOffsets([new TopicPartitionOffset(tp, new Offset(offset + 1))]);
             return;
         }
 
-        // Strategy.Batch — advance the per-partition contiguous watermark and commit on threshold.
+        var tp = new TopicPartition(topic, new Partition(partition));
+
         List<TopicPartitionOffset>? toCommit = null;
+        List<TopicPartitionOffset>? toStore = null;
         lock (_lock)
         {
-            if (!_watermarks.TryGetValue(tp, out var watermark))
-            {
-                watermark = new OffsetWatermark();
-                _watermarks[tp] = watermark;
-            }
-
-            watermark.Register(offset);
-            _sinceLastCommit++;
+            var watermark = WatermarkFor(tp);
+            watermark.Complete(offset);
 
-            if (ShouldCommitBatch())
+            switch (_strategy)
             {
-                toCommit = PendingWatermarkOffsets();
-                _sinceLastCommit = 0;
-                _lastCommitTimestamp = Stopwatch.GetTimestamp();
+                case Strategy.PerMessage:
+                    if (watermark.TryTakeCommittable(out var perMessage))
+                    {
+                        toCommit = [new TopicPartitionOffset(tp, new Offset(perMessage))];
+                    }
+
+                    break;
+
+                case Strategy.Store:
+                    if (watermark.TryTakeCommittable(out var store))
+                    {
+                        toStore = [new TopicPartitionOffset(tp, new Offset(store))];
+                    }
+
+                    break;
+
+                case Strategy.Batch:
+                    _sinceLastCommit++;
+                    if (ShouldCommitBatch())
+                    {
+                        toCommit = PendingWatermarkOffsets();
+                        _sinceLastCommit = 0;
+                        _lastCommitTimestamp = Stopwatch.GetTimestamp();
+                    }
+
+                    break;
             }
         }
 
@@ -157,6 +191,22 @@ public void Complete(string topic, int partition, long offset)
         {
             CommitOffsets(toCommit);
         }
+
+        if (toStore is { Count: > 0 })
+        {
+            StoreOffsets(toStore);
+        }
+    }
+
+    private OffsetWatermark WatermarkFor(TopicPartition tp)
+    {
+        if (!_watermarks.TryGetValue(tp, out var watermark))
+        {
+            watermark = new OffsetWatermark();
+            _watermarks[tp] = watermark;
+        }
+
+        return watermark;
     }
 
     private bool ShouldCommitBatch()
@@ -261,50 +311,74 @@ private void StoreOffsets(List<TopicPartitionOffset> offsets)
     }
 
     /// <summary>
-    /// Tracks the highest contiguous (gap-free) completed offset for a single partition so a batch
-    /// commit never moves the committed position past an offset whose predecessors are still in
-    /// flight. Offsets within a Kafka partition are contiguous, so a simple "advance while the next
-    /// offset is present" loop yields the correct watermark even under out-of-order completion.
+    /// Tracks the safe commit position for a single partition so the committed offset never moves past
+    /// an offset that is still in flight, even under concurrent out-of-order completion (GH-3161).
+    ///
+    /// <para><see cref="Track"/> records each delivered offset (consume order) as in-flight;
+    /// <see cref="Complete"/> clears it on success. The committable position is the lowest still
+    /// in-flight delivered offset, or the high-water consumed offset + 1 once nothing is in flight. This
+    /// makes no assumption that offsets are gap-free, so it also behaves correctly on compacted or
+    /// transactional (read_committed) topics where the broker hands out non-contiguous offsets, and it
+    /// self-heals across a rebalance because the in-flight set drains as handlers finish.</para>
+    ///
+    /// <para>The reported position is monotonic: a re-seek that re-delivers an already-passed offset
+    /// will not move the committed position backwards.</para>
     /// </summary>
     internal sealed class OffsetWatermark
     {
-        private long _contiguous = -1; // highest contiguous completed offset, or -1 before seeding
-        private bool _seeded;
-        private bool _dirty;
-        private readonly SortedSet<long> _ahead = new();
+        private readonly SortedSet<long> _inflight = new(); // delivered but not yet completed
+        private long _highWater = -1; // highest delivered offset seen, or -1 before anything is seen
+        private bool _seen;
+        private long _lastCommittable = -1; // last next-offset reported, to keep the position monotonic
 
-        public void Register(long offset)
+        public void Track(long offset)
         {
-            if (!_seeded)
-            {
-                _contiguous = offset - 1;
-                _seeded = true;
-            }
+            _inflight.Add(offset);
+            Observe(offset);
+        }
 
-            if (offset <= _contiguous)
+        public void Complete(long offset)
+        {
+            _inflight.Remove(offset);
+            // Tolerate a Complete without a prior Track (defensive) by still advancing the high water mark.
+            Observe(offset);
+        }
+
+        private void Observe(long offset)
+        {
+            if (!_seen)
             {
-                return; // already covered
+                _highWater = offset;
+                // Seed the baseline resume position at the first delivered offset so we only ever
+                // commit/store once progress advances past it — never a redundant baseline write.
+                _lastCommittable = offset;
+                _seen = true;
+                return;
             }
 
-            _ahead.Add(offset);
-
-            while (_ahead.Remove(_contiguous + 1))
+            if (offset > _highWater)
             {
-                _contiguous++;
-                _dirty = true;
+                _highWater = offset;
             }
         }
 
         /// <summary>
-        /// If new contiguous progress has been made since the last take, return the offset to commit
-        /// (contiguous + 1, i.e. the next offset to resume from) and reset the dirty flag.
+        /// If the safe commit position has advanced since the last take, return the next offset to
+        /// commit/store (the lowest in-flight offset, or high-water + 1 when nothing is in flight).
         /// </summary>
         public bool TryTakeCommittable(out long nextOffset)
         {
-            if (_dirty)
+            if (!_seen)
+            {
+                nextOffset = default;
+                return false;
+            }
+
+            var candidate = _inflight.Count == 0 ? _highWater + 1 : _inflight.Min;
+            if (candidate > _lastCommittable)
             {
-                nextOffset = _contiguous + 1;
-                _dirty = false;
+                _lastCommittable = candidate;
+                nextOffset = candidate;
                 return true;
             }
 
diff --git a/src/Transports/Kafka/Wolverine.Kafka/Internals/KafkaTopicGroupListener.cs b/src/Transports/Kafka/Wolverine.Kafka/Internals/KafkaTopicGroupListener.cs
index 55ca8aa63..605920bb0 100644
--- a/src/Transports/Kafka/Wolverine.Kafka/Internals/KafkaTopicGroupListener.cs
+++ b/src/Transports/Kafka/Wolverine.Kafka/Internals/KafkaTopicGroupListener.cs
@@ -47,6 +47,10 @@ public KafkaTopicGroupListener(KafkaTopicGroup endpoint, ConsumerConfig config,
                         result = _consumer.Consume(_cancellation.Token);
                         var message = result.Message;
 
+                        // Seed the offset watermark in consume (in-order) order so out-of-order handler
+                        // completion can never commit past this still-in-flight offset (GH-3161).
+                        _committer.Track(result.Topic, result.Partition.Value, result.Offset.Value);
+
                         var envelope = mapper!.CreateEnvelope(result.Topic, message);
                         envelope.TopicName = result.Topic;
                         envelope.Offset = result.Offset.Value;


Test patch:
diff --git a/src/Persistence/RavenDbTests/Internal/Generated/WolverineHandlers/StartRavenDiagSagaHandler956550714.cs b/src/Persistence/RavenDbTests/Internal/Generated/WolverineHandlers/StartRavenDiagSagaHandler956550714.cs
new file mode 100644
index 000000000..ef7127b8e
--- /dev/null
+++ b/src/Persistence/RavenDbTests/Internal/Generated/WolverineHandlers/StartRavenDiagSagaHandler956550714.cs
@@ -0,0 +1,53 @@
+// <auto-generated/>
+#pragma warning disable
+using Raven.Client.Documents;
+
+namespace Internal.Generated.WolverineHandlers
+{
+    // START: StartRavenDiagSagaHandler956550714
+    [global::System.CodeDom.Compiler.GeneratedCode("JasperFx", "1.0.0")]
+    public sealed class StartRavenDiagSagaHandler956550714 : Wolverine.Runtime.Handlers.MessageHandler
+    {
+        private readonly Raven.Client.Documents.IDocumentStore _documentStore;
+
+        public StartRavenDiagSagaHandler956550714(Raven.Client.Documents.IDocumentStore documentStore)
+        {
+            _documentStore = documentStore;
+        }
+
+
+
+        public override async System.Threading.Tasks.Task HandleAsync(Wolverine.Runtime.MessageContext context, System.Threading.CancellationToken cancellation)
+        {
+            using var asyncDocumentSession = _documentStore.OpenAsyncSession();
+            // The actual message body
+            var startRavenDiagSaga = (RavenDbTests.StartRavenDiagSaga)context.Envelope.Message;
+
+            // Application-specific Open Telemetry auditing
+            System.Diagnostics.Activity.Current?.SetTag("Id", startRavenDiagSaga.Id);
+            var ravenDiagSaga = new RavenDbTests.RavenDiagSaga();
+            
+            // The actual message execution
+            ravenDiagSaga.Start(startRavenDiagSaga);
+
+            context.SetSagaId(startRavenDiagSaga.Id);
+            System.Diagnostics.Activity.Current?.SetTag("wolverine.saga.id", startRavenDiagSaga.Id.ToString());
+            System.Diagnostics.Activity.Current?.SetTag("wolverine.saga.type", "RavenDbTests.RavenDiagSaga");
+            if (!ravenDiagSaga.IsCompleted())
+            {
+                await asyncDocumentSession.StoreAsync(ravenDiagSaga, cancellation).ConfigureAwait(false);
+            }
+
+            
+            // Commit all pending changes
+            await asyncDocumentSession.SaveChangesAsync(cancellation).ConfigureAwait(false);
+
+        }
+
+    }
+
+    // END: StartRavenDiagSagaHandler956550714
+    
+    
+}
+
diff --git a/src/Transports/Kafka/Wolverine.Kafka.Tests/kafka_offset_committer.cs b/src/Transports/Kafka/Wolverine.Kafka.Tests/kafka_offset_committer.cs
index 5b78cb19a..bc1ec2884 100644
--- a/src/Transports/Kafka/Wolverine.Kafka.Tests/kafka_offset_committer.cs
+++ b/src/Transports/Kafka/Wolverine.Kafka.Tests/kafka_offset_committer.cs
@@ -117,22 +117,28 @@ public void batch_count_commits_watermark_after_threshold()
     }
 
     [Fact]
-    public void batch_count_never_commits_ahead_of_a_gap()
+    public void batch_count_never_commits_ahead_of_an_in_flight_offset()
     {
         var committer = committerFor(CommitMode.BatchCount, batchCount: 3);
 
+        // Delivered in consume order; 101 will stay in flight while later offsets finish first.
+        committer.Track("t", 0, 100);
+        committer.Track("t", 0, 101);
+        committer.Track("t", 0, 102);
+        committer.Track("t", 0, 103);
+
         // Out-of-order completion: 100, 102, 103 — 101 is still in flight.
         committer.Complete("t", 0, 100);
         committer.Complete("t", 0, 102);
-        committer.Complete("t", 0, 103); // threshold reached, but 101 is missing
+        committer.Complete("t", 0, 103); // threshold reached, but 101 is still in flight
 
-        // Watermark may only advance to 100+1 = 101 (everything below the gap), never past it.
+        // The committed position may only advance to the lowest in-flight offset (101), never past it.
         theConsumer.Received(1).Commit(Arg.Is<IEnumerable<TopicPartitionOffset>>(list =>
             list.Single().Offset.Value == 101));
 
         theConsumer.ClearReceivedCalls();
 
-        // Now 101 completes -> the gap fills and the watermark jumps to 103+1 = 104 on flush.
+        // Now 101 completes -> nothing in flight -> the position jumps to the high-water 103 + 1 = 104.
         committer.Complete("t", 0, 101);
         committer.Flush();
 
@@ -140,6 +146,79 @@ public void batch_count_never_commits_ahead_of_a_gap()
             list.Single().Offset.Value == 104));
     }
 
+    [Fact]
+    public void per_message_never_commits_ahead_of_an_in_flight_offset()
+    {
+        var committer = committerFor(CommitMode.PerMessage);
+
+        // Delivered in order 0,1,2; a later message completes first under concurrency.
+        committer.Track("t", 0, 0);
+        committer.Track("t", 0, 1);
+        committer.Track("t", 0, 2);
+
+        // 2 finishes while 0 and 1 are still in flight -> nothing may be committed yet.
+        committer.Complete("t", 0, 2);
+        theConsumer.DidNotReceive().Commit(Arg.Any<IEnumerable<TopicPartitionOffset>>());
+
+        // 0 finishes -> safe to resume from the next in-flight offset (1).
+        committer.Complete("t", 0, 0);
+        theConsumer.Received(1).Commit(Arg.Is<IEnumerable<TopicPartitionOffset>>(list =>
+            list.Single().Offset.Value == 1));
+
+        theConsumer.ClearReceivedCalls();
+
+        // 1 finishes -> nothing in flight -> resume past the highest delivered (2 + 1 = 3).
+        committer.Complete("t", 0, 1);
+        theConsumer.Received(1).Commit(Arg.Is<IEnumerable<TopicPartitionOffset>>(list =>
+            list.Single().Offset.Value == 3));
+    }
+
+    [Fact]
+    public void store_never_stores_ahead_of_an_in_flight_offset()
+    {
+        var committer = committerFor(CommitMode.StoreThenAutoFlush);
+
+        // Delivered in order 0,1,2; out-of-order completion under concurrency.
+        committer.Track("t", 0, 0);
+        committer.Track("t", 0, 1);
+        committer.Track("t", 0, 2);
+
+        // 2 finishes while 0 and 1 are still in flight -> nothing may be stored yet.
+        committer.Complete("t", 0, 2);
+        theConsumer.DidNotReceive().StoreOffset(Arg.Any<TopicPartitionOffset>());
+
+        // 0 finishes -> safe to store the next in-flight offset (1).
+        committer.Complete("t", 0, 0);
+        theConsumer.Received(1).StoreOffset(Arg.Is<TopicPartitionOffset>(t => t.Offset.Value == 1));
+
+        theConsumer.ClearReceivedCalls();
+
+        // 1 finishes -> nothing in flight -> store past the highest delivered (2 + 1 = 3).
+        committer.Complete("t", 0, 1);
+        theConsumer.Received(1).StoreOffset(Arg.Is<TopicPartitionOffset>(t => t.Offset.Value == 3));
+    }
+
+    [Fact]
+    public void per_message_tolerates_non_contiguous_offsets_from_a_compacted_topic()
+    {
+        var committer = committerFor(CommitMode.PerMessage);
+
+        // A compacted / transactional topic hands out gaps (1 and 2 were compacted or were control
+        // records) — the watermark must not stall waiting for offsets that will never be delivered.
+        committer.Track("t", 0, 0);
+        committer.Track("t", 0, 3);
+
+        committer.Complete("t", 0, 0);
+        theConsumer.Received(1).Commit(Arg.Is<IEnumerable<TopicPartitionOffset>>(list =>
+            list.Single().Offset.Value == 3)); // resume from the next in-flight delivered offset
+
+        theConsumer.ClearReceivedCalls();
+
+        committer.Complete("t", 0, 3);
+        theConsumer.Received(1).Commit(Arg.Is<IEnumerable<TopicPartitionOffset>>(list =>
+            list.Single().Offset.Value == 4)); // high-water + 1, gaps skipped
+    }
+
     [Fact]
     public void flush_commits_pending_batch_watermark()
     {