Repository: JasperFx/wolverine (2026-07-28T19:39:43Z)

Issue:
CircuitBreakingTests: durable_and_not_parallel.the_circuit_breaker_should_trip_and_restart is unreliable on CI
`CircuitBreakingTests.RabbitMq.durable_and_not_parallel.the_circuit_breaker_should_trip_and_restart` fails intermittently on `CICircuitBreaking`, always with the same shape:

```
System.TimeoutException : Listener did not process the expected message count 1200
in the time allowed. Only 1190 unique messages received. 10 of 1200 published never made it.
```

Seen most recently on [#3678](https://github.com/JasperFx/wolverine/pull/3678) (a docs/tests-only PR that cannot touch this code path), where the parent commit had passed the same job 31/31. `CICircuitBreaking` has also been red on #3656.

**Skipped for now** via a `[Fact(Skip = ...)]` override on the `durable_and_not_parallel` subclass, so unrelated PRs are not blocked. The sibling variants (buffered, inline, local) and the other assertion in this class (`everything_is_wonderful_..._do_not_ever_trip`) still run, so circuit-breaker trip/restart behavior remains covered.

## Why this needs a real fix, not a longer timeout

This is the same test that GH-3137 already touched twice — first with `ProcessingBudget` raised to 3 minutes, then with `RequiredProcessedCountOnTrip` made overridable so buffered variants could require fewer than 1200. **The lesson recorded there was that a "flake" in this test turned out to be a genuine message-loss bug**, so the small-N loss should be explained before it is tuned away.

The durable variant is exactly the one that should *not* lose messages: buffered mode acks to the broker as soon as a message lands in the in-memory buffer (documented tradeoff, hence its lower required count), but durable persists before acking. Losing 10 of 1200 under a durable inbox is either:

1. a real gap in the requeue/circuit-pause interaction — messages requeued around the moment the breaker trips and the listener tears down, or
2. a test-harness accounting problem — `MessageRecorder.TrackPublished` racing the trip, or the waiter resolving against a count that double-counts requeues.

## Suggested work

- [ ] Determine which of the two it is. Instrument the run to dump the 10 missing message ids and their last known state (published / received / requeued / in the inbox) rather than only the count.
- [ ] If (1): fix the loss and un-skip. This would be a real durability bug worth its own issue.
- [ ] If (2): make the harness deterministic — gate on listener lifecycle transitions rather than a wall-clock budget over a fixed count, in the same spirit as `waitForListenerToResumeAsync` — and un-skip.
- [ ] Either way, remove the `Skip` and reference this issue in the commit.

## Repro

```bash
docker compose up -d rabbitmq postgresql
dotnet test src/Transports/RabbitMQ/CircuitBreakingTests/CircuitBreakingTests.csproj \
  --filter "FullyQualifiedName~durable_and_not_parallel"
```

Loads the runner (it is a 1200-message churn test), so it reproduces most readily on a contended machine.


Discussion:

Patch:
diff --git a/src/Wolverine/Persistence/Durability/ListenerInboxRecovery.cs b/src/Wolverine/Persistence/Durability/ListenerInboxRecovery.cs
index 5c82047ea..ed3ee7787 100644
--- a/src/Wolverine/Persistence/Durability/ListenerInboxRecovery.cs
+++ b/src/Wolverine/Persistence/Durability/ListenerInboxRecovery.cs
@@ -111,7 +111,34 @@ private async Task<int> recoverFromStoreAsync(IMessageStore store, CancellationT
             }
 
             await store.ReassignIncomingAsync(_runtime.DurabilitySettings.AssignedNodeNumber, envelopes);
-            await _circuit.EnqueueDirectlyAsync(envelopes);
+
+            // GH-3680. See the matching comment in RecoverIncomingMessagesCommand: once these rows are
+            // owned by this live node, only this node can ever give them back. A failed hand-off into the
+            // listener (e.g. the circuit tripped between the status check above and here) would otherwise
+            // strand them permanently.
+            try
+            {
+                await _circuit.EnqueueDirectlyAsync(envelopes);
+            }
+            catch (Exception e)
+            {
+                _logger.LogError(e,
+                    "Error trying to enqueue {Count} recovered messages into single node listener {Listener}. Releasing them back to any node so that they are recovered on a later sweep",
+                    envelopes.Count, destination);
+
+                try
+                {
+                    await store.ReassignIncomingAsync(TransportConstants.AnyNode, envelopes);
+                }
+                catch (Exception releaseFailure)
+                {
+                    _logger.LogError(releaseFailure,
+                        "Error trying to release {Count} un-enqueued messages for {Listener} back to any node",
+                        envelopes.Count, destination);
+                }
+
+                break;
+            }
 
             _logger.RecoveredIncoming(envelopes);
             _logger.LogInformation(
diff --git a/src/Wolverine/Persistence/Durability/RecoverIncomingMessagesCommand.cs b/src/Wolverine/Persistence/Durability/RecoverIncomingMessagesCommand.cs
index a3f8fe924..65a8e0dc0 100644
--- a/src/Wolverine/Persistence/Durability/RecoverIncomingMessagesCommand.cs
+++ b/src/Wolverine/Persistence/Durability/RecoverIncomingMessagesCommand.cs
@@ -51,7 +51,26 @@ public async Task<AgentCommands> ExecuteAsync(IWolverineRuntime runtime,
 
         await _store.ReassignIncomingAsync(_settings.AssignedNodeNumber, envelopes);
 
-        await _circuit.EnqueueDirectlyAsync(envelopes);
+        // GH-3680. The rows above are now owned by *this* node, and orphan recovery only ever releases
+        // rows owned by a node it has proven to be dead. If the hand-off into the listener fails -- and it
+        // absolutely can, because a circuit breaker trip between DeterminePageSize() and here nulls out the
+        // agent's receiver -- then nothing on any node will ever release them again and the messages are
+        // lost for the life of the process. Hand ownership back so the next sweep can retry.
+        try
+        {
+            await _circuit.EnqueueDirectlyAsync(envelopes);
+        }
+        catch (Exception e)
+        {
+            _logger.LogError(e,
+                "Error trying to enqueue {Count} recovered messages into the listener for {Destination}. Releasing them back to any node so that they are recovered on a later sweep",
+                envelopes.Count, _count.Destination);
+
+            await releaseAsync(envelopes);
+
+            return AgentCommands.Empty;
+        }
+
         _logger.RecoveredIncoming(envelopes);
 
         _logger.LogInformation("Successfully recovered {Count} messages from the inbox for listener {Listener}",
@@ -67,6 +86,20 @@ public async Task<AgentCommands> ExecuteAsync(IWolverineRuntime runtime,
         return AgentCommands.Empty;
     }
 
+    private async Task releaseAsync(IReadOnlyList<Envelope> envelopes)
+    {
+        try
+        {
+            await _store.ReassignIncomingAsync(TransportConstants.AnyNode, envelopes);
+        }
+        catch (Exception e)
+        {
+            _logger.LogError(e,
+                "Error trying to release {Count} un-enqueued messages for {Destination} back to any node",
+                envelopes.Count, _count.Destination);
+        }
+    }
+
     public virtual int DeterminePageSize(IListenerCircuit listener, IncomingCount count,
         DurabilitySettings durabilitySettings)
     {
diff --git a/src/Wolverine/Runtime/WorkerQueues/DurableReceiver.cs b/src/Wolverine/Runtime/WorkerQueues/DurableReceiver.cs
index 9c0f787d5..03a6af4ce 100644
--- a/src/Wolverine/Runtime/WorkerQueues/DurableReceiver.cs
+++ b/src/Wolverine/Runtime/WorkerQueues/DurableReceiver.cs
@@ -422,6 +422,17 @@ private async Task receiveOneAsync(Envelope envelope)
                 await executeWithRetriesAsync(async () =>
                 {
                     envelope.OwnerId = TransportConstants.AnyNode;
+
+                    // GH-3680 defense in depth. Never write an inbox row that no recovery sweep can see.
+                    // Status defaults to 'Outgoing' and received_at to null on an envelope that never went
+                    // through MarkReceived, and both are filter columns for inbox recovery.
+                    if (envelope.Status == EnvelopeStatus.Outgoing)
+                    {
+                        envelope.Status = EnvelopeStatus.Incoming;
+                    }
+
+                    envelope.Destination ??= Uri;
+
                     assignAncillaryStoreIfNeeded(envelope);
                     try
                     {
@@ -612,6 +623,17 @@ public async ValueTask ProcessReceivedMessagesAsync(DateTimeOffset now, IListene
             throw new OperationCanceledException();
         }
 
+        // GH-3680. MarkReceived has to happen BEFORE the latch check, not after it. It is what stamps
+        // Status = Incoming, Destination (persisted as received_at) and Listener onto the envelope. The
+        // latched branch below persists the envelope to the inbox as a safety net, so skipping MarkReceived
+        // wrote rows with the default Status of 'Outgoing' and a null received_at -- invisible to
+        // CheckRecoverableIncomingMessagesOperation (which filters on Status = 'Incoming') and to
+        // LoadPageOfGloballyOwnedIncomingAsync (which filters on received_at), i.e. permanently orphaned.
+        // A null Listener also meant the latched branch silently skipped the defer back to the broker, so
+        // nothing else was going to redeliver them either. That is real message loss under a *durable*
+        // inbox whenever a circuit breaker trip latches the receiver mid-flight.
+        foreach (var envelope in envelopes) envelope.MarkReceived(listener, now, _settings, _endpoint.WireTap);
+
         // A latched receiver must apply the full single-envelope semantics (persist as
         // owner 0 + defer back to the listener), so route through the one-at-a-time path
         // which already implements them (GH-3492).
@@ -621,8 +643,6 @@ public async ValueTask ProcessReceivedMessagesAsync(DateTimeOffset now, IListene
             return;
         }
 
-        foreach (var envelope in envelopes) envelope.MarkReceived(listener, now, _settings, _endpoint.WireTap);
-
         // Per-envelope guards that the single-envelope path (receiveOneAsync) has always
         // applied but this batch path historically skipped (GH-3492): serializer unwrap
         // (MassTransit-style interop), missing id/message-type dead-lettering, and expiry.


Test patch:
diff --git a/src/Transports/RabbitMQ/CircuitBreakingTests/RabbitMq/durable_and_not_parallel.cs b/src/Transports/RabbitMQ/CircuitBreakingTests/RabbitMq/durable_and_not_parallel.cs
index 98e1a4ba1..327d169ff 100644
--- a/src/Transports/RabbitMQ/CircuitBreakingTests/RabbitMq/durable_and_not_parallel.cs
+++ b/src/Transports/RabbitMQ/CircuitBreakingTests/RabbitMq/durable_and_not_parallel.cs
@@ -41,7 +41,7 @@ protected override void configureListener(WolverineOptions opts)
     // this test can be a genuine message-loss bug -- so it is skipped rather than tuned away until
     // the missing messages are actually accounted for. Every sibling variant, and the
     // "do not ever trip" assertion in this class, still run.
-    [Fact(Skip = "Unreliable on CI, see https://github.com/JasperFx/wolverine/issues/3680")]
+    [Fact]
     public override Task the_circuit_breaker_should_trip_and_restart()
     {
         return base.the_circuit_breaker_should_trip_and_restart();