Repository: nats-io/nats-server
Pull request #6792: [FIXED] Only deliver replicated message after quorum
Author: login: MauriceVanVeen
Created: 2025-04-11T17:01:52Z | Merged: 2025-04-17T16:11:17Z

Related to https://github.com/nats-io/nats-server/pull/6469, about the following code:
```go
	// Update delivered first.
	o.updateDelivered(dseq, seq, dc, ts)

	// Send message.
	o.outq.send(pmsg)
```

`o.updateDelivered` requires proposing delivered state through Raft, and even if proposing fails, we immediately sent the message to the client. This is great for performance, but really bad for properly replicating this piece of data. Before the before-mentioned PR there would be a bunch of nasty side-effects of stuck consumer, perceived data loss through missed redeliveries, etc. Because clients could get messages that a new leader wouldn't know about if proposals failed.

The core issue is that we should only send the message AFTER we had quorum on updating delivered state. Otherwise the following could happen: message gets sent to the client, `updateDelivered` proposal fails, leader changes, `AckSync` will now timeout indefinitely even with retries because the new leader doesn't know this message was even delivered.

Signed-off-by: Maurice van Veen <github@mauricevanveen.com>


Diff:
diff --git a/server/consumer.go b/server/consumer.go
index 4852c9b59fa..f9a1beca73f 100644
--- a/server/consumer.go
+++ b/server/consumer.go
@@ -436,6 +436,7 @@ type consumer struct {
 	rdqi              avl.SequenceSet
 	rdc               map[uint64]uint64
 	replies           map[uint64]string
+	pendingDeliveries map[uint64]*jsPubMsg // Messages that can be delivered after achieving quorum.
 	maxdc             uint64
 	waiting           *waitQueue
 	cfg               ConsumerConfig
@@ -1533,6 +1534,7 @@ func (o *consumer) setLeader(isLeader bool) {
 		o.rdq = nil
 		o.rdqi.Empty()
 		o.pending = nil
+		o.resetPendingDeliveries()
 		// ok if they are nil, we protect inside unsubscribe()
 		o.unsubscribe(o.ackSub)
 		o.unsubscribe(o.reqSub)
@@ -2170,6 +2172,11 @@ func (o *consumer) updateConfig(cfg *ConsumerConfig) error {
 	if cfg.MaxAckPending != o.cfg.MaxAckPending {
 		o.maxp = cfg.MaxAckPending
 		o.signalNewMessages()
+		// If MaxAckPending is lowered, we could have allocated a pending deliveries map of larger size.
+		// Reset it here, so we can shrink the map.
+		if cfg.MaxAckPending < o.cfg.MaxAckPending {
+			o.resetPendingDeliveries()
+		}
 	}
 	// AckWait
 	if cfg.AckWait != o.cfg.AckWait {
@@ -2529,6 +2536,16 @@ func (o *consumer) addAckReply(sseq uint64, reply string) {
 	o.replies[sseq] = reply
 }
 
+// Used to remember messages that need to be sent for a replicated consumer, after delivered quorum.
+// Lock should be held.
+func (o *consumer) addReplicatedQueuedMsg(pmsg *jsPubMsg) {
+	// Is not explicitly limited in size, but will at maximum hold maximum ack pending.
+	if o.pendingDeliveries == nil {
+		o.pendingDeliveries = make(map[uint64]*jsPubMsg)
+	}
+	o.pendingDeliveries[pmsg.seq] = pmsg
+}
+
 // Lock should be held.
 func (o *consumer) updateAcks(dseq, sseq uint64, reply string) {
 	if o.node != nil {
@@ -3089,7 +3106,6 @@ func (o *consumer) processAckMsg(sseq, dseq, dc uint64, reply string, doSample b
 	}
 
 	// Check if this ack is above the current pointer to our next to deliver.
-	// This could happen on a cooperative takeover with high speed deliveries.
 	if sseq >= o.sseq {
 		// Let's make sure this is valid.
 		// This is only received on the consumer leader, so should never be higher
@@ -4842,7 +4858,14 @@ func (o *consumer) deliverMsg(dsubj, ackReply string, pmsg *jsPubMsg, dc uint64,
 	}
 
 	// Send message.
-	o.outq.send(pmsg)
+	// If we're replicated we MUST only send the message AFTER we've got quorum for updating
+	// delivered state. Otherwise, we could be in an invalid state after a leader change.
+	// We can send immediately if not replicated, not using acks, or using flow control (incompatible).
+	if o.node == nil || ap == AckNone || o.cfg.FlowControl {
+		o.outq.send(pmsg)
+	} else {
+		o.addReplicatedQueuedMsg(pmsg)
+	}
 
 	// Flow control.
 	if o.maxpb > 0 && o.needFlowControl(psz) {
@@ -6104,3 +6127,10 @@ func (o *consumer) stopAndClearPtmr() {
 	stopAndClearTimer(&o.ptmr)
 	o.ptmrEnd = time.Time{}
 }
+
+func (o *consumer) resetPendingDeliveries() {
+	for _, pmsg := range o.pendingDeliveries {
+		pmsg.returnToPool()
+	}
+	o.pendingDeliveries = nil
+}
diff --git a/server/jetstream_cluster.go b/server/jetstream_cluster.go
index ea157180cb2..79bcad8c369 100644
--- a/server/jetstream_cluster.go
+++ b/server/jetstream_cluster.go
@@ -5043,6 +5043,11 @@ func (js *jetStream) applyConsumerEntries(o *consumer, ce *CommittedEntry, isLea
 				o.mu.Lock()
 				err = o.store.UpdateDelivered(dseq, sseq, dc, ts)
 				o.ldt = time.Now()
+				// Need to send message to the client, since we have quorum to do so now.
+				if pmsg, ok := o.pendingDeliveries[sseq]; ok {
+					o.outq.send(pmsg)
+					delete(o.pendingDeliveries, sseq)
+				}
 				o.mu.Unlock()
 				if err != nil {
 					panic(err.Error())
diff --git a/server/jetstream_cluster_1_test.go b/server/jetstream_cluster_1_test.go
index 77bf4e08730..5be0e01d6b0 100644
--- a/server/jetstream_cluster_1_test.go
+++ b/server/jetstream_cluster_1_test.go
@@ -8118,6 +8118,130 @@ func TestJetStreamClusterInvalidJSACKOverRoute(t *testing.T) {
 	}
 }
 
+func TestJetStreamClusterConsumerOnlyDeliverMsgAfterQuorum(t *testing.T) {
+	c := createJetStreamClusterExplicit(t, "R3S", 3)
+	defer c.shutdown()
+
+	nc, js := jsClientConnect(t, c.randomServer())
+	defer nc.Close()
+
+	_, err := js.AddStream(&nats.StreamConfig{
+		Name:      "TEST",
+		Subjects:  []string{"foo"},
+		Retention: nats.LimitsPolicy,
+		Replicas:  3,
+	})
+	require_NoError(t, err)
+
+	_, err = js.AddConsumer("TEST", &nats.ConsumerConfig{
+		Durable:   "CONSUMER",
+		AckPolicy: nats.AckExplicitPolicy,
+		Replicas:  3,
+		AckWait:   2 * time.Second,
+	})
+	require_NoError(t, err)
+
+	_, err = js.Publish("foo", nil)
+	require_NoError(t, err)
+
+	checkFor(t, time.Second, 100*time.Millisecond, func() error {
+		return checkState(t, c, globalAccountName, "TEST")
+	})
+
+	cl := c.consumerLeader(globalAccountName, "TEST", "CONSUMER")
+	acc, err := cl.lookupAccount(globalAccountName)
+	require_NoError(t, err)
+	mset, err := acc.lookupStream("TEST")
+	require_NoError(t, err)
+	o := mset.lookupConsumer("CONSUMER")
+	require_NotNil(t, o)
+	rn := o.raftNode().(*raft)
+
+	// Force the leader to not be able to make proposals.
+	rn.Lock()
+	rn.werr = errors.New("block proposals")
+	rn.Unlock()
+
+	sub, err := js.PullSubscribe("foo", "CONSUMER")
+	require_NoError(t, err)
+	defer sub.Unsubscribe()
+
+	// We must only receive a message AFTER quorum was met for updating delivered state.
+	// This should time out since proposals are blocked.
+	msgs, err := sub.Fetch(1, nats.MaxWait(2*time.Second))
+	require_Error(t, err, nats.ErrTimeout)
+	require_Len(t, len(msgs), 0)
+
+	// Allow proposals to be made again.
+	rn.Lock()
+	rn.werr = nil
+	rn.Unlock()
+
+	// Now it should pass.
+	msgs, err = sub.Fetch(1, nats.MaxWait(2*time.Second))
+	require_NoError(t, err)
+	require_Len(t, len(msgs), 1)
+	require_NoError(t, msgs[0].AckSync())
+}
+
+func TestJetStreamClusterConsumerResetPendingDeliveriesOnMaxAckPendingUpdate(t *testing.T) {
+	c := createJetStreamClusterExplicit(t, "R3S", 3)
+	defer c.shutdown()
+
+	nc, js := jsClientConnect(t, c.randomServer())
+	defer nc.Close()
+
+	_, err := js.AddStream(&nats.StreamConfig{
+		Name:      "TEST",
+		Subjects:  []string{"foo"},
+		Retention: nats.LimitsPolicy,
+		Replicas:  3,
+	})
+	require_NoError(t, err)
+
+	cfg := &nats.ConsumerConfig{
+		Durable:       "CONSUMER",
+		AckPolicy:     nats.AckExplicitPolicy,
+		Replicas:      3,
+		AckWait:       2 * time.Second,
+		MaxAckPending: 100,
+	}
+	_, err = js.AddConsumer("TEST", cfg)
+	require_NoError(t, err)
+
+	cl := c.consumerLeader(globalAccountName, "TEST", "CONSUMER")
+	acc, err := cl.lookupAccount(globalAccountName)
+	require_NoError(t, err)
+	mset, err := acc.lookupStream("TEST")
+	require_NoError(t, err)
+	o := mset.lookupConsumer("CONSUMER")
+	require_NotNil(t, o)
+
+	o.mu.Lock()
+	o.pendingDeliveries = map[uint64]*jsPubMsg{0: getJSPubMsgFromPool()}
+	o.mu.Unlock()
+
+	// Increasing does not reset pending deliveries.
+	cfg.MaxAckPending++
+	_, err = js.UpdateConsumer("TEST", cfg)
+	require_NoError(t, err)
+
+	o.mu.Lock()
+	l := len(o.pendingDeliveries)
+	o.mu.Unlock()
+	require_Equal(t, l, 1)
+
+	// Decreasing does reset pending deliveries, so we can shrink the map.
+	cfg.MaxAckPending--
+	_, err = js.UpdateConsumer("TEST", cfg)
+	require_NoError(t, err)
+
+	o.mu.Lock()
+	l = len(o.pendingDeliveries)
+	o.mu.Unlock()
+	require_Equal(t, l, 0)
+}
+
 //
 // DO NOT ADD NEW TESTS IN THIS FILE (unless to balance test times)
 // Add at the end of jetstream_cluster_<n>_test.go, with <n> being the highest value.
diff --git a/server/norace_1_test.go b/server/norace_1_test.go
index 189e8de2fc1..b3612660c52 100644
--- a/server/norace_1_test.go
+++ b/server/norace_1_test.go
@@ -5078,7 +5078,7 @@ func TestNoRaceJetStreamPullConsumersAndInteriorDeletes(t *testing.T) {
 		Durable:       "foo",
 		FilterSubject: "foo",
 		MaxAckPending: 20000,
-		AckWait:       time.Minute,
+		AckWait:       5 * time.Second,
 		AckPolicy:     nats.AckExplicitPolicy,
 	})
 	require_NoError(t, err)
@@ -5147,7 +5147,7 @@ func TestNoRaceJetStreamPullConsumersAndInteriorDeletes(t *testing.T) {
 	case <-ch:
 		// OK
 	case <-time.After(30 * time.Second):
-		t.Fatalf("Consumers took too long to consumer all messages")
+		t.Fatalf("Consumers took too long to consume all messages")
 	}
 }
 


Reviews:
user: login: MauriceVanVeen
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-11T17:03:03Z
user: login: neilalexander
body: Overall I think this is a good change and I strongly believe prioritising correctness is the right thing to do here. I do see that it may potentially reduce performance on some replicated consumers but that will depend heavily on the usage pattern, and also need to keep in mind that some replicated streams with interest and WQ retention policies will not be able to avoid this penalty by scaling consumers.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-14T14:54:07Z
user: login: neilalexander
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-14T14:59:32Z
user: login: MauriceVanVeen
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-14T15:05:04Z
user: login: neilalexander
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-14T15:08:39Z
user: login: MauriceVanVeen
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-14T15:29:38Z
user: login: neilalexander
body: LGTM but think this also needs @derekcollison's approval too.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-14T16:04:37Z
user: login: derekcollison
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-17T10:56:12Z
user: login: MauriceVanVeen
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-17T10:59:52Z
user: login: MauriceVanVeen
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-17T11:00:25Z
user: login: derekcollison
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-17T11:03:51Z
user: login: MauriceVanVeen
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-17T11:15:27Z
user: login: derekcollison
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-17T13:57:50Z
user: login: MauriceVanVeen
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-17T14:42:47Z
user: login: derekcollison
body: LGTM
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-17T15:19:04Z

Review comments:
diff_hunk: @@ -5078,7 +5078,7 @@ func TestNoRaceJetStreamPullConsumersAndInteriorDeletes(t *testing.T) {
 		Durable:       "foo",
 		FilterSubject: "foo",
 		MaxAckPending: 20000,
-		AckWait:       time.Minute,
+		AckWait:       5 * time.Second,
path: server/norace_1_test.go
user: login: MauriceVanVeen
body: Needed to be lowered, because now the new leader after stepdown actually knows a message was delivered and adheres to the `AckWait`.

Before, it only sent the message quickly due to this bug, because the new leader did not know the previous leader already sent the message.
created_at: 2025-04-11T17:03:03Z
author_association: MEMBER
diff_hunk: @@ -5043,6 +5043,11 @@ func (js *jetStream) applyConsumerEntries(o *consumer, ce *CommittedEntry, isLea
 				o.mu.Lock()
 				err = o.store.UpdateDelivered(dseq, sseq, dc, ts)
 				o.ldt = time.Now()
+				// Need to send message to the client, since we have quorum to do so now.
+				if pmsg, ok := o.pendingDeliveries[sseq]; ok {
+					o.outq.send(pmsg)
+					delete(o.pendingDeliveries, sseq)
path: server/jetstream_cluster.go
user: login: neilalexander
body: We may want to check `len(o.pendingDeliveries)` vs `cap(o.pendingDeliveries)` here to ensure that we shrink & repopulate the map if the usage pattern changes, otherwise we could hold onto memory until the leader changes. Maybe if the length drops to below one quarter of the capacity or so. WDYT?
created_at: 2025-04-14T14:53:45Z
author_association: MEMBER
diff_hunk: @@ -5043,6 +5043,11 @@ func (js *jetStream) applyConsumerEntries(o *consumer, ce *CommittedEntry, isLea
 				o.mu.Lock()
 				err = o.store.UpdateDelivered(dseq, sseq, dc, ts)
 				o.ldt = time.Now()
+				// Need to send message to the client, since we have quorum to do so now.
+				if pmsg, ok := o.pendingDeliveries[sseq]; ok {
+					o.outq.send(pmsg)
+					delete(o.pendingDeliveries, sseq)
path: server/jetstream_cluster.go
user: login: neilalexander
body: Or maybe only if the capacity goes over a certain threshold...
created_at: 2025-04-14T14:59:32Z
author_association: MEMBER
diff_hunk: @@ -5043,6 +5043,11 @@ func (js *jetStream) applyConsumerEntries(o *consumer, ce *CommittedEntry, isLea
 				o.mu.Lock()
 				err = o.store.UpdateDelivered(dseq, sseq, dc, ts)
 				o.ldt = time.Now()
+				// Need to send message to the client, since we have quorum to do so now.
+				if pmsg, ok := o.pendingDeliveries[sseq]; ok {
+					o.outq.send(pmsg)
+					delete(o.pendingDeliveries, sseq)
path: server/jetstream_cluster.go
user: login: MauriceVanVeen
body: > here to ensure that we shrink & repopulate the map if the usage pattern changes

Good point, we should at least do that check when `MaxAckPending` changes. As that should probably shrink it.

Am a bit wary of constantly shrinking and resizing the map. When you do fetch N, where N<MaxAckPending, the `o.pendingDeliveries` will be max N in size. Then all messages will be sent to the client, and it could technically shrink. But if the client then fetches again, we'll need to allocate again. Hence why I'm thinking about only resizing based on `MaxAckPending` being lowered.

Also, not entirely sure, but believe `cap(..)` can't be used on a map?
created_at: 2025-04-14T15:05:04Z
author_association: MEMBER
diff_hunk: @@ -5043,6 +5043,11 @@ func (js *jetStream) applyConsumerEntries(o *consumer, ce *CommittedEntry, isLea
 				o.mu.Lock()
 				err = o.store.UpdateDelivered(dseq, sseq, dc, ts)
 				o.ldt = time.Now()
+				// Need to send message to the client, since we have quorum to do so now.
+				if pmsg, ok := o.pendingDeliveries[sseq]; ok {
+					o.outq.send(pmsg)
+					delete(o.pendingDeliveries, sseq)
path: server/jetstream_cluster.go
user: login: neilalexander
body: Ah, indeed it can't, well let's do so on `MaxAckPending` updates at least and we can watch how it goes.
created_at: 2025-04-14T15:08:39Z
author_association: MEMBER
diff_hunk: @@ -5043,6 +5043,11 @@ func (js *jetStream) applyConsumerEntries(o *consumer, ce *CommittedEntry, isLea
 				o.mu.Lock()
 				err = o.store.UpdateDelivered(dseq, sseq, dc, ts)
 				o.ldt = time.Now()
+				// Need to send message to the client, since we have quorum to do so now.
+				if pmsg, ok := o.pendingDeliveries[sseq]; ok {
+					o.outq.send(pmsg)
+					delete(o.pendingDeliveries, sseq)
path: server/jetstream_cluster.go
user: login: MauriceVanVeen
body: Done, it now gets reset if `MaxAckPending` gets lowered.
created_at: 2025-04-14T15:29:37Z
author_association: MEMBER
diff_hunk: @@ -436,6 +436,7 @@ type consumer struct {
 	rdqi              avl.SequenceSet
 	rdc               map[uint64]uint64
 	replies           map[uint64]string
+	pendingDeliveries map[uint64]*jsPubMsg // Messages that can be delivered after achieving quorum.
path: server/consumer.go
user: login: derekcollison
body: Can this grow unbounded?
created_at: 2025-04-17T10:54:42Z
author_association: MEMBER
diff_hunk: @@ -2529,6 +2534,16 @@ func (o *consumer) addAckReply(sseq uint64, reply string) {
 	o.replies[sseq] = reply
 }
 
+// Used to remember messages that need to be sent for a replicated consumer, after delivered quorum.
+// Lock should be held.
+func (o *consumer) addReplicatedQueuedMsg(pmsg *jsPubMsg) {
+	// Is not explicitly limited in size, but will at maximum hold maximum ack pending.
path: server/consumer.go
user: login: derekcollison
body: Are we sure this comment is true?
created_at: 2025-04-17T10:55:17Z
author_association: MEMBER
diff_hunk: @@ -2529,6 +2534,16 @@ func (o *consumer) addAckReply(sseq uint64, reply string) {
 	o.replies[sseq] = reply
 }
 
+// Used to remember messages that need to be sent for a replicated consumer, after delivered quorum.
+// Lock should be held.
+func (o *consumer) addReplicatedQueuedMsg(pmsg *jsPubMsg) {
+	// Is not explicitly limited in size, but will at maximum hold maximum ack pending.
path: server/consumer.go
user: login: MauriceVanVeen
body: The stream sequence is used as a key, which is the same for `o.pending`. That means it can only ever become as large as max length of `o.pending`, i.e. `MaxAckPending`.
created_at: 2025-04-17T10:59:51Z
author_association: MEMBER
diff_hunk: @@ -436,6 +436,7 @@ type consumer struct {
 	rdqi              avl.SequenceSet
 	rdc               map[uint64]uint64
 	replies           map[uint64]string
+	pendingDeliveries map[uint64]*jsPubMsg // Messages that can be delivered after achieving quorum.
path: server/consumer.go
user: login: MauriceVanVeen
body: Can grow up to `MaxAckPending`, just like `o.pending`. See also [this comment](https://github.com/nats-io/nats-server/pull/6792/files/b26bcbbc78f0d8d3997f2a4f46bad4036d14bd25#r2048716554).
created_at: 2025-04-17T11:00:25Z
author_association: MEMBER
diff_hunk: @@ -2529,6 +2534,16 @@ func (o *consumer) addAckReply(sseq uint64, reply string) {
 	o.replies[sseq] = reply
 }
 
+// Used to remember messages that need to be sent for a replicated consumer, after delivered quorum.
+// Lock should be held.
+func (o *consumer) addReplicatedQueuedMsg(pmsg *jsPubMsg) {
+	// Is not explicitly limited in size, but will at maximum hold maximum ack pending.
path: server/consumer.go
user: login: derekcollison
body: ok so when new message comes into stream and we kick the consumer we just bail because we hit max pending yes?
created_at: 2025-04-17T11:03:51Z
author_association: MEMBER
diff_hunk: @@ -2529,6 +2534,16 @@ func (o *consumer) addAckReply(sseq uint64, reply string) {
 	o.replies[sseq] = reply
 }
 
+// Used to remember messages that need to be sent for a replicated consumer, after delivered quorum.
+// Lock should be held.
+func (o *consumer) addReplicatedQueuedMsg(pmsg *jsPubMsg) {
+	// Is not explicitly limited in size, but will at maximum hold maximum ack pending.
path: server/consumer.go
user: login: MauriceVanVeen
body: `o.getNextMsg` bails based on hitting max pending, indeed.
created_at: 2025-04-17T11:15:27Z
author_association: MEMBER
diff_hunk: @@ -2529,6 +2534,16 @@ func (o *consumer) addAckReply(sseq uint64, reply string) {
 	o.replies[sseq] = reply
 }
 
+// Used to remember messages that need to be sent for a replicated consumer, after delivered quorum.
+// Lock should be held.
+func (o *consumer) addReplicatedQueuedMsg(pmsg *jsPubMsg) {
+	// Is not explicitly limited in size, but will at maximum hold maximum ack pending.
path: server/consumer.go
user: login: derekcollison
body: Understood but I am wondering if we should short circuit in stream's signalConsumers()? WDYT?
created_at: 2025-04-17T13:57:50Z
author_association: MEMBER
diff_hunk: @@ -2529,6 +2534,16 @@ func (o *consumer) addAckReply(sseq uint64, reply string) {
 	o.replies[sseq] = reply
 }
 
+// Used to remember messages that need to be sent for a replicated consumer, after delivered quorum.
+// Lock should be held.
+func (o *consumer) addReplicatedQueuedMsg(pmsg *jsPubMsg) {
+	// Is not explicitly limited in size, but will at maximum hold maximum ack pending.
path: server/consumer.go
user: login: MauriceVanVeen
body: Could potentially be an additional optimization, although can't say how large the gain would be.
Either way I think it should be in a different PR.
created_at: 2025-04-17T14:42:47Z
author_association: MEMBER
diff_hunk: @@ -2529,6 +2534,16 @@ func (o *consumer) addAckReply(sseq uint64, reply string) {
 	o.replies[sseq] = reply
 }
 
+// Used to remember messages that need to be sent for a replicated consumer, after delivered quorum.
+// Lock should be held.
+func (o *consumer) addReplicatedQueuedMsg(pmsg *jsPubMsg) {
+	// Is not explicitly limited in size, but will at maximum hold maximum ack pending.
path: server/consumer.go
user: login: derekcollison
body: Let's do that PR after this one that short circuits.
created_at: 2025-04-17T15:18:42Z
author_association: MEMBER

PR comments:
user: login: MauriceVanVeen
created_at: 2025-04-11T17:04:27Z
body: Keeping this PR as draft for a little bit, to still do some more testing using both our internal tooling and Antithesis.
But it can be reviewed already, just holding off on merging for now.
author_association: MEMBER
user: login: derekcollison
created_at: 2025-04-11T17:17:38Z
body: This will most surely tank performance of ordered consumers for quick consumption..
author_association: MEMBER
user: login: MauriceVanVeen
created_at: 2025-04-11T17:18:13Z
body: > This will most surely tank performance of ordered consumers for quick consumption..

Ordered non-replicated consumers are not impacted. Only replicated consumers, those need proper replication.
author_association: MEMBER
user: login: derekcollison
created_at: 2025-04-11T17:18:48Z
body: With any replicated consumer with even low RTTs this will limit performance. In general we want to be able to consumer faster than you can ingest (in general). So let's do some benchmarks with real RTT between servers and discuss.
author_association: MEMBER
user: login: derekcollison
created_at: 2025-04-11T17:19:31Z
body: That is why I would send periodic consumer snapshots to sync replicas. 
author_association: MEMBER
user: login: MauriceVanVeen
created_at: 2025-04-14T13:41:44Z
body: Tested and did some benchmarking. Antithesis testing was not able to run into the 'infinite timing out of AckSync' anymore :tada: 
Benchmarking did of course show lower performance (because now the data is actually replicated), but if the workload actually does some work (and not simply ack immediately), the app processing time starts canceling out the added RTT. Especially when using the new JetStream API, since that sends fetch requests early when using `Consume`.

This PR ensures data correctness and allows 'exactly once' consumption. The latter was not always true during/after leader changes prior to this PR. Regarding the former, there were data correctness issues with the message delivered counter not monotonically increasing for every delivery due to sending the message to the client first and only replicating after, which could fail and then the next delivery would report incorrect delivery count.

Also reiterating that this is only about replicated consumers. Ordered/AckNone/FlowControled/non-replicated consumers remain the same.

author_association: MEMBER
user: login: neilalexander
created_at: 2025-04-17T15:30:33Z
body: @MauriceVanVeen Happy to rebase on top of latest main? Then I'll merge.
author_association: MEMBER