Repository: JasperFx/wolverine (2026-07-22T07:22:39Z)

Issue:
EventSubscriptionAgent health check compares a per-tenant agent's position against the database-wide high water mark — idle tenants report thousands of 'events behind' and get auto-restarted
### Setup

- Marten sharded multi-tenant store, `Events.UseTenantPartitionedEvents = true` (each tenant draws its own event sequence)
- `UseWolverineManagedEventSubscriptionDistribution = true` with per-tenant agent fan-out (#3280)

### The bug

`EventSubscriptionAgent.CheckHealthAsync` computes its behind-by gap as:

```csharp
var tracker = _daemon.Tracker;
var highWaterMark = tracker.HighWaterMark;          // database-wide mark
var currentSequence = _innerAgent?.Position ?? 0;   // per-tenant sequence

var behindCount = highWaterMark - currentSequence;
```

For a per-tenant agent, `Position` is a position in **that tenant's own sequence** (per-tenant partitioning gives every tenant an independent sequence — the reason the per-tenant fan-out exists at all, per #3280). But `tracker.HighWaterMark` is the **database-wide** mark. The two are not comparable, so the health check measures one tenant's progress against other tenants' event volume.

### Observed

A tenant with **zero events** (no `mt_event_progression` row at all, nothing to process) reports:

```
Projection event-subscriptions://marten/main/<database>/invoicejournalentries/all/v4/<tenant> is 8900 events behind (critical threshold: 5000)
```

where 8,900 is the database's high-water mark driven entirely by *other* tenants. Every low-traffic tenant in a busy database trips the same false Unhealthy/Degraded as soon as the database as a whole passes the thresholds — the more successful the busiest tenant, the sicker every idle tenant looks.

### Knock-on: stall detector + auto-restart

The same mismatched comparison feeds the stall path:

```csharp
else if (currentSequence < highWaterMark &&
         DateTimeOffset.UtcNow - _lastAdvancedAt > StallTimeout)
```

`currentSequence < highWaterMark` is permanently true for almost every per-tenant agent (their tenant sequence is always below the database mark), so any agent whose tenant simply has no new events for 60s starts counting "consecutive stalls" and is auto-restarted after 3 health checks — a stop/start cycle for perfectly healthy, idle agents. (Agents more than `criticalThreshold` "behind" return early at the behind-check and never reach it, which only masks the restart churn for the emptiest tenants.)

### Expected

For a shard with `_shardName.TenantId != null`, the gap should be computed against the **tenant's own** high-water mark (the `HighWaterMark:{tenant}` progression row / per-tenant tracker state), not the database-wide `tracker.HighWaterMark`. A tenant with no per-tenant mark has nothing to be behind — the check should report Healthy, and the stall detector should not treat "tenant sequence below database mark" as pending work.



Discussion:
Related (same environment, independent bug): JasperFx/jasperfx#553 — the extended-progression heartbeat writes from the same per-tenant agent fleet exhausted the database server's connection slots.



Patch:
diff --git a/src/Wolverine/Runtime/Agents/EventSubscriptionAgent.cs b/src/Wolverine/Runtime/Agents/EventSubscriptionAgent.cs
index 2398b27a8..59603ed51 100644
--- a/src/Wolverine/Runtime/Agents/EventSubscriptionAgent.cs
+++ b/src/Wolverine/Runtime/Agents/EventSubscriptionAgent.cs
@@ -161,8 +161,18 @@ public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context,
             LoadThresholds();
         }
 
-        var tracker = _daemon.Tracker;
-        var highWaterMark = tracker.HighWaterMark;
+        // GH-3580: a per-tenant agent's Position lives in its OWN tenant's event sequence (per-tenant
+        // event partitioning gives every tenant an independent sequence -- the reason the per-tenant
+        // fan-out exists, GH-3280), so it must be measured against that tenant's high-water mark, not
+        // the database-wide tracker mark. The inner JasperFx agent already carries the tenant-scoped
+        // mark: it is seeded from the per-tenant ceiling at start and only ever raised by the tenanted
+        // high-water coordinator's per-tenant routing (marten#4717). Comparing against the database-wide
+        // mark made every quiet tenant in a busy database read as thousands of "events behind" (a tenant
+        // with zero events reported the full database mark) and fed the stall detector the permanently
+        // true "currentSequence < highWaterMark", auto-restarting healthy idle agents.
+        var highWaterMark = _shardName.TenantId != null
+            ? _innerAgent?.HighWaterMark ?? 0
+            : _daemon.Tracker.HighWaterMark;
         var currentSequence = _innerAgent?.Position ?? 0;
 
         var warningThreshold = _warningThreshold ?? DefaultWarningThreshold;


Test patch:
diff --git a/src/Testing/CoreTests/Runtime/Agents/event_subscription_agent_health_check_uses_tenant_scoped_high_water.cs b/src/Testing/CoreTests/Runtime/Agents/event_subscription_agent_health_check_uses_tenant_scoped_high_water.cs
new file mode 100644
index 000000000..fa488e0f5
--- /dev/null
+++ b/src/Testing/CoreTests/Runtime/Agents/event_subscription_agent_health_check_uses_tenant_scoped_high_water.cs
@@ -0,0 +1,124 @@
+using JasperFx;
+using JasperFx.Events.Daemon;
+using JasperFx.Events.Projections;
+using Microsoft.Extensions.Diagnostics.HealthChecks;
+using Microsoft.Extensions.Logging.Abstractions;
+using NSubstitute;
+using Shouldly;
+using Wolverine.Runtime.Agents;
+using Xunit;
+using IProjectionDaemon = JasperFx.Events.Daemon.IProjectionDaemon;
+using JasperFxSubscriptionAgent = JasperFx.Events.Daemon.ISubscriptionAgent;
+
+namespace CoreTests.Runtime.Agents;
+
+/// <summary>
+/// Regression coverage for GH-3580. A per-tenant subscription agent's Position lives in its own
+/// tenant's event sequence (per-tenant event partitioning gives every tenant an independent sequence,
+/// GH-3280), but CheckHealthAsync measured it against the database-wide ShardStateTracker.HighWaterMark.
+/// Every quiet tenant in a busy database read as thousands of "events behind" — a tenant with zero
+/// events reported the full database mark as its lag — and the stall detector's permanently true
+/// "currentSequence &lt; highWaterMark" auto-restarted healthy idle agents. The health check must use
+/// the inner agent's tenant-scoped high-water mark (seeded and routed per tenant since marten#4717)
+/// for tenant-scoped shards.
+/// </summary>
+public class event_subscription_agent_health_check_uses_tenant_scoped_high_water
+{
+    private readonly IProjectionDaemon _daemon = Substitute.For<IProjectionDaemon>();
+    private readonly ShardStateTracker _tracker = new(NullLogger.Instance);
+    private readonly JasperFxSubscriptionAgent _inner = Substitute.For<JasperFxSubscriptionAgent>();
+
+    public event_subscription_agent_health_check_uses_tenant_scoped_high_water()
+    {
+        _daemon.Tracker.Returns(_tracker);
+        _daemon.StartAgentAsync(Arg.Any<ShardName>(), Arg.Any<CancellationToken>()).Returns(_inner);
+        _inner.Status.Returns(AgentStatus.Running);
+    }
+
+    private async Task<EventSubscriptionAgent> startedAgentFor(ShardName shardName, Uri uri)
+    {
+        var agent = new EventSubscriptionAgent(uri, shardName, _daemon);
+        await agent.StartAsync(CancellationToken.None);
+        return agent;
+    }
+
+    [Fact]
+    public async Task a_tenant_with_no_events_is_healthy_even_when_the_database_mark_is_far_ahead()
+    {
+        // The database-wide mark is way past the critical threshold, driven entirely by other tenants
+        await _tracker.MarkHighWaterAsync(8900);
+
+        // This tenant has never appended an event: no per-tenant high-water, no progression
+        _inner.HighWaterMark.Returns(0);
+        _inner.Position.Returns(0);
+
+        var agent = await startedAgentFor(
+            new ShardName("invoicejournalentries", "all", 4, "98123456"),
+            new Uri("event-subscriptions://marten/main/db/invoicejournalentries/all/v4/98123456"));
+
+        var result = await agent.CheckHealthAsync(new HealthCheckContext());
+
+        // Before the fix: "Projection ... is 8900 events behind (critical threshold: 5000)"
+        result.Status.ShouldBe(HealthStatus.Healthy);
+    }
+
+    [Fact]
+    public async Task a_tenant_genuinely_behind_its_own_mark_is_still_reported()
+    {
+        await _tracker.MarkHighWaterAsync(100_000);
+
+        // Behind by 5900 against its OWN tenant's sequence — past the critical threshold of 5000
+        _inner.HighWaterMark.Returns(6000);
+        _inner.Position.Returns(100);
+
+        var agent = await startedAgentFor(
+            new ShardName("invoicejournalentries", "all", 4, "98123456"),
+            new Uri("event-subscriptions://marten/main/db/invoicejournalentries/all/v4/98123456"));
+
+        var result = await agent.CheckHealthAsync(new HealthCheckContext());
+
+        result.Status.ShouldBe(HealthStatus.Unhealthy);
+        result.Description!.ShouldContain("5900 events behind");
+    }
+
+    [Fact]
+    public async Task a_caught_up_tenant_does_not_trip_the_stall_detector_on_repeated_checks()
+    {
+        await _tracker.MarkHighWaterAsync(8900);
+
+        // Fully caught up within its own tenant's sequence, idle (no new events)
+        _inner.HighWaterMark.Returns(120);
+        _inner.Position.Returns(120);
+
+        var agent = await startedAgentFor(
+            new ShardName("invoicejournalentries", "all", 4, "98123456"),
+            new Uri("event-subscriptions://marten/main/db/invoicejournalentries/all/v4/98123456"));
+
+        // Before the fix, "Position < database mark" read as pending work forever, so an idle tenant
+        // racked up consecutive stalls and got auto-restarted. Repeated checks must stay healthy.
+        for (var i = 0; i < 5; i++)
+        {
+            var result = await agent.CheckHealthAsync(new HealthCheckContext());
+            result.Status.ShouldBe(HealthStatus.Healthy);
+        }
+    }
+
+    [Fact]
+    public async Task a_store_global_agent_still_measures_against_the_database_wide_mark()
+    {
+        await _tracker.MarkHighWaterAsync(8900);
+
+        // Store-global shard (TenantId null): the database-wide tracker mark is the right ceiling,
+        // and the inner agent's default HighWaterMark of 0 must NOT suppress the alert
+        _inner.Position.Returns(100);
+
+        var agent = await startedAgentFor(
+            new ShardName("invoicejournalentries"),
+            new Uri("event-subscriptions://marten/main/db/invoicejournalentries/all/v4"));
+
+        var result = await agent.CheckHealthAsync(new HealthCheckContext());
+
+        result.Status.ShouldBe(HealthStatus.Unhealthy);
+        result.Description!.ShouldContain("8800 events behind");
+    }
+}