Repository: JasperFx/wolverine (2026-07-20T12:06:18Z)

Issue:
EventSubscriptionAgent.RebuildAsync/RewindAsync leave the shard permanently stopped under Wolverine-managed distribution (daemon stops continuous agents; nothing restarts them)
Wolverine 6.21.0 / JasperFx.Events 2.30.1. Found via JasperFx/CritterWatch#747 (every console-driven rebuild left the projection dead); root-caused with a deterministic repro.

## RebuildAsync (and RewindAsync) leave the shard permanently stopped under Wolverine-managed event-subscription distribution

`EventSubscriptionAgent.RebuildAsync` delegates to `_daemon.RebuildProjectionAsync(...)`. JasperFx's daemon-level `rebuildProjection`:

1. `stopRunningAgents(subscriptionName)` — stops the running **continuous** agents;
2. rebuilds via transient agents;
3. stops-and-drains those transient agents;
4. **never restarts what it stopped in (1).**

Under a store-owned daemon (`AddAsyncDaemon`) the store's coordinator loop notices the stopped shard and resurrects it. Under **Wolverine-managed distribution there is no store coordinator** — and Wolverine's `EventSubscriptionAgent.Status` still reads `Running` (the wrapper never observed a stop), so `NodeAgentController` sees nothing to fix. Net: after every rebuild the shard re-registers `RegisteredIdle` daemon-side and freezes forever while the high-water mark climbs. An operator "Restart" that routes to an agent Wolverine believes is already running no-ops.

`RewindAsync` has the same end state via a different mechanism: `JasperFxAsyncDaemon.RewindSubscriptionAsync` does restart agents after rewinding — but it builds them with `buildAgentsForSubscription(...)` and `StartAsync`s them **directly, bypassing `tryStartAgentAsync`, so they never register in `_agents`** and the wolverine wrapper's `_innerAgent` still points at the stopped one.

## Repro

Deterministic: rebuild any async projection **while its publisher keeps appending** on a Wolverine-managed-distribution host, then watch the progression row freeze while the HWM climbs. (Rebuilding with a stopped publisher hides the defect — a caught-up shard and a stalled shard are indistinguishable with no event flow, which is why simpler rebuild tests pass.) CritterWatch pins this with `projection_rebuild_under_load` (JasperFx/CritterWatch@02c49251): red on stock behavior, green with the workaround.

## Workaround (CritterWatch side, revert candidate)

Both CritterWatch handlers now call `agent.StartAsync(ct)` immediately after `RebuildAsync`/`RewindAsync` — resuming continuous mode through the registered path.

## Suggested fix

`EventSubscriptionAgent.RebuildAsync`/`RewindAsync` should restore continuous execution themselves (re-run the `StartAsync` path after the daemon-level operation), or the daemon-level operations should restart what they stopped through the registered path (see the companion JasperFx issue for that half).



Discussion:

Patch:
diff --git a/Directory.Packages.props b/Directory.Packages.props
index b67815896..95c2533db 100644
--- a/Directory.Packages.props
+++ b/Directory.Packages.props
@@ -35,13 +35,13 @@
     <PackageVersion Include="Grpc.StatusProto" Version="2.76.0" />
     <PackageVersion Include="Grpc.Tools" Version="2.76.0" />
     <PackageVersion Include="HtmlTags" Version="9.0.0" />
-    <PackageVersion Include="JasperFx" Version="2.30.1" />
-    <PackageVersion Include="JasperFx.Events" Version="2.30.1" />
-    <PackageVersion Include="JasperFx.Events.SourceGenerator" Version="2.30.1" />
+    <PackageVersion Include="JasperFx" Version="2.30.2" />
+    <PackageVersion Include="JasperFx.Events" Version="2.30.2" />
+    <PackageVersion Include="JasperFx.Events.SourceGenerator" Version="2.30.2" />
     <!-- RuntimeCompiler is on its own 5.x line (the Roslyn compiler package) — not the 2.1.x
          family; it stays at 5.0.0. -->
     <PackageVersion Include="JasperFx.RuntimeCompiler" Version="5.0.0" />
-    <PackageVersion Include="JasperFx.SourceGenerator" Version="2.30.1" />
+    <PackageVersion Include="JasperFx.SourceGenerator" Version="2.30.2" />
     <PackageVersion Include="Lamar.Microsoft.DependencyInjection" Version="16.0.0" />
     <PackageVersion Include="Marten" Version="9.16.1" />
     <PackageVersion Include="Microsoft.Data.SqlClient" Version="6.1.3" />
diff --git a/src/Wolverine/Runtime/Agents/EventSubscriptionAgent.cs b/src/Wolverine/Runtime/Agents/EventSubscriptionAgent.cs
index 6d40be63f..86381d522 100644
--- a/src/Wolverine/Runtime/Agents/EventSubscriptionAgent.cs
+++ b/src/Wolverine/Runtime/Agents/EventSubscriptionAgent.cs
@@ -59,8 +59,7 @@ public EventSubscriptionAgent(Uri uri, ShardName shardName, IProjectionDaemon da
 
     public async Task StartAsync(CancellationToken cancellationToken)
     {
-        _innerAgent = await _daemon.StartAgentAsync(_shardName, cancellationToken);
-        Status = AgentStatus.Running;
+        await resumeContinuousAsync(cancellationToken);
 
         // Only count an agent that actually started - a throw above leaves the count untouched
         if (OnStarted != null)
@@ -69,6 +68,17 @@ public async Task StartAsync(CancellationToken cancellationToken)
         }
     }
 
+    // Start (or re-adopt) the continuous inner agent through the registered daemon path and refresh the
+    // wrapper's view of it. Deliberately does NOT invoke OnStarted: the running-agent bookkeeping in
+    // EventStoreAgents counts one logical agent per node for the observer-subscription lifecycle, and a
+    // rebuild/rewind is transparent to that count (the wrapper never observed a matching stop), so
+    // re-counting here would leak the database's tracker subscriptions. See GH-3520.
+    private async Task resumeContinuousAsync(CancellationToken cancellationToken)
+    {
+        _innerAgent = await _daemon.StartAgentAsync(_shardName, cancellationToken);
+        Status = AgentStatus.Running;
+    }
+
     public async Task StopAsync(CancellationToken cancellationToken)
     {
         await _daemon.StopAgentAsync(_shardName);
@@ -86,6 +96,13 @@ public async Task RebuildAsync(CancellationToken cancellationToken)
         // single-database per-tenant partitioning. _shardName.TenantId is null for store-global /
         // database-per-tenant shards, where the tenant-less behavior is correct.
         await _daemon.RebuildProjectionAsync(_shardName.Name, _shardName.TenantId, cancellationToken);
+
+        // GH-3520: the daemon-level rebuild stops the continuous agent and never restarts it. Under
+        // Wolverine-managed distribution there is no store coordinator to resurrect it, and this wrapper
+        // would otherwise keep reporting Running against the now-stopped agent, so NodeAgentController
+        // sees nothing to fix and the shard freezes at RegisteredIdle while its high-water climbs.
+        // Restore continuous execution ourselves through the registered daemon start path.
+        await resumeContinuousAsync(cancellationToken);
     }
 
     public async Task RewindAsync(long? sequenceFloor, DateTimeOffset? timestamp, CancellationToken cancellationToken)
@@ -95,6 +112,14 @@ public async Task RewindAsync(long? sequenceFloor, DateTimeOffset? timestamp, Ca
         // CritterWatch needs because DaemonForDatabase() throws under Wolverine-managed distribution.
         await _daemon.RewindSubscriptionAsync(_shardName.Name, _shardName.TenantId, cancellationToken,
             sequenceFloor, timestamp);
+
+        // GH-3520: same freeze as RebuildAsync. RewindSubscriptionAsync restarts continuous agents
+        // daemon-side, but the wrapper's _innerAgent still points at the stopped pre-rewind agent and
+        // Status still reads Running. Re-adopt the live agent through the registered start path so the
+        // wrapper reflects reality. This requires JasperFx#536 (rewind registers its restarted agent) so
+        // the registered start resolves to that same running agent idempotently rather than spinning up a
+        // duplicate on the same progression row - this ships in lockstep with that JasperFx bump.
+        await resumeContinuousAsync(cancellationToken);
     }
 
     public Uri Uri { get; }


Test patch:
diff --git a/src/Testing/CoreTests/Runtime/Agents/event_subscription_agent_rebuild_rewind_resume.cs b/src/Testing/CoreTests/Runtime/Agents/event_subscription_agent_rebuild_rewind_resume.cs
new file mode 100644
index 000000000..36876406c
--- /dev/null
+++ b/src/Testing/CoreTests/Runtime/Agents/event_subscription_agent_rebuild_rewind_resume.cs
@@ -0,0 +1,55 @@
+using JasperFx;
+using JasperFx.Events.Projections;
+using NSubstitute;
+using Shouldly;
+using Wolverine.Runtime.Agents;
+using Xunit;
+using IProjectionDaemon = JasperFx.Events.Daemon.IProjectionDaemon;
+using JasperFxSubscriptionAgent = JasperFx.Events.Daemon.ISubscriptionAgent;
+
+namespace CoreTests.Runtime.Agents;
+
+/// <summary>
+/// Regression coverage for GH-3520. Under Wolverine-managed event-subscription distribution there is no
+/// store coordinator to resurrect a shard the daemon stopped. RebuildProjectionAsync stops the continuous
+/// agent and never restarts it; RewindSubscriptionAsync restarts it daemon-side but the wrapper still
+/// pointed at the stopped pre-rewind agent and reported Running. Either way the shard froze at
+/// RegisteredIdle while its high-water climbed. EventSubscriptionAgent.RebuildAsync/RewindAsync must now
+/// restore continuous execution themselves through the registered daemon start path and refresh their view.
+/// </summary>
+public class event_subscription_agent_rebuild_rewind_resume
+{
+    private readonly IProjectionDaemon _daemon = Substitute.For<IProjectionDaemon>();
+
+    // Store-global shard ("Incident:All", TenantId null) - the shape in the GH-3519/3520 report.
+    private readonly ShardName _shardName = new("Incident");
+    private readonly JasperFxSubscriptionAgent _restarted = Substitute.For<JasperFxSubscriptionAgent>();
+    private readonly EventSubscriptionAgent _agent;
+
+    public event_subscription_agent_rebuild_rewind_resume()
+    {
+        _daemon.StartAgentAsync(Arg.Any<ShardName>(), Arg.Any<CancellationToken>()).Returns(_restarted);
+        _agent = new EventSubscriptionAgent(
+            new Uri("event-subscriptions://marten/incident/all"), _shardName, _daemon);
+    }
+
+    [Fact]
+    public async Task rebuild_restores_continuous_execution_through_the_registered_start_path()
+    {
+        await _agent.RebuildAsync(CancellationToken.None);
+
+        // Before the fix RebuildAsync returned without restarting anything: no start call, and the
+        // wrapper kept its stale status. Now it resumes through the registered daemon start path.
+        await _daemon.Received(1).StartAgentAsync(Arg.Any<ShardName>(), Arg.Any<CancellationToken>());
+        _agent.Status.ShouldBe(AgentStatus.Running);
+    }
+
+    [Fact]
+    public async Task rewind_restores_continuous_execution_through_the_registered_start_path()
+    {
+        await _agent.RewindAsync(0, null, CancellationToken.None);
+
+        await _daemon.Received(1).StartAgentAsync(Arg.Any<ShardName>(), Arg.Any<CancellationToken>());
+        _agent.Status.ShouldBe(AgentStatus.Running);
+    }
+}