Repository: elsa-workflows/elsa-core (2026-07-30T00:03:31Z)

Issue:
Thread-safety: DefaultTenantService.DeactivateTenantsAsync mutates the tenant dictionary without _refreshLock (intermittent OverflowException at teardown)
## Description

`DefaultTenantService` keeps its live tenant set in a plain, non-thread-safe `IDictionary<string, Tenant>` (`_tenantsDictionary`), but its mutators are **inconsistently synchronized**:

- `RefreshAsync` (and `ActivateTenantsAsync`, which calls it) acquire the internal `_refreshLock` `SemaphoreSlim`.
- `DeactivateTenantsAsync` takes **no lock at all** — it reads `dictionary.Values.ToArray()` and then unregisters each tenant, entirely unsynchronized.

Because `DeactivateTenantsAsync` shares no lock with the `Refresh`/`Activate` path, the two can mutate the same `Dictionary` concurrently. Concurrent mutation of a non-thread-safe `Dictionary` can corrupt its internal counters; once the value count transiently reads negative, `Values.ToArray()` attempts `new Tenant[negativeCount]`, which the CLR reports as:

```
System.OverflowException : Arithmetic operation resulted in an overflow.
```

This most commonly surfaces at **host shutdown**, when `ActivateTenants.StopAsync` calls `DeactivateTenantsAsync` while a tenant refresh is still in flight.

Relevant current source (`src/modules/Elsa.Common/Multitenancy/Implementations/DefaultTenantService.cs`, `main`):

```csharp
public async Task DeactivateTenantsAsync(CancellationToken cancellationToken = default)
{
    var dictionary = await GetTenantsDictionaryAsync(cancellationToken);
    var tenants = dictionary.Values.ToArray();   // no _refreshLock held
    foreach (var tenant in tenants)
        await UnregisterTenantAsync(tenant, false, cancellationToken);
}

public async Task RefreshAsync(CancellationToken cancellationToken = default)
{
    await _refreshLock.WaitAsync(cancellationToken);   // Refresh *does* lock
    try
    {
        // ... mutates _tenantsDictionary (Register/UnregisterTenantAsync) ...
    }
    finally
    {
        _refreshLock.Release();
    }
}
```

## Steps to Reproduce

The defect is a data race, so it reproduces deterministically by driving the two mutators concurrently against a real `DefaultTenantService`:

1. Resolve `ITenantService` (`DefaultTenantService`).
2. From multiple threads/tasks, call `ActivateTenantsAsync` / `RefreshAsync` and `DeactivateTenantsAsync` concurrently in a tight loop (e.g. a `Parallel.For` / a set of `Task.Run` hammering both for a few seconds).
3. Observe intermittent `System.OverflowException` from `DeactivateTenantsAsync` (`Values.ToArray()`), and/or other `Dictionary`-corruption exceptions.

A realistic (non-synthetic) trigger also exists without any manual hammering: `ConfigurationTenantsProvider.OnOptionsChanged` is an `async void` handler invoked by `IOptionsMonitor.OnChange` on an arbitrary thread pool thread; it calls `RefreshAsync`. If a configuration/reload token fires while the host is stopping (`ActivateTenants.StopAsync` → `DeactivateTenantsAsync`), `Refresh` and `Deactivate` race the same dictionary.

**Reproduction Rate:** Intermittent under normal host start/stop (timing dependent — e.g. teardown of integration-test hosts). Deterministic (fails within seconds) when the two mutators are driven concurrently in a loop.

## Expected Behavior

Tenant activation, deactivation, and refresh serialize against each other so the internal tenant dictionary is never mutated by two callers at once. Host shutdown never throws an `OverflowException` from `DeactivateTenantsAsync`.

## Actual Behavior

`DeactivateTenantsAsync` runs unsynchronized against `RefreshAsync`/`ActivateTenantsAsync`, corrupting the shared `Dictionary` and intermittently throwing `System.OverflowException` at `Values.ToArray()`, typically during host teardown.

## Environment

- **Elsa Package Version:** Observed on **3.7.1**; the affected `DefaultTenantService` code is line-identical through **3.8.0-preview1** and on the current **`main`** branch, so it is not fixed in any published version.
- **.NET:** Reproduced on **.NET 10** (also observed under a .NET 11 preview). Not runtime-specific — it is a concurrency defect.
- **Operating System:** Not OS-specific (reproduced on Linux; the race is platform-independent).

## Log Output

Sanitized stack (framework + Elsa frames):

```
System.OverflowException : Arithmetic operation resulted in an overflow.
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Elsa.Common.Multitenancy.DefaultTenantService.DeactivateTenantsAsync(CancellationToken cancellationToken)
   at Elsa.Common.Multitenancy.HostedServices.ActivateTenants.StopAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory`1.DisposeAsync()
```

## Troubleshooting Attempts

- Confirmed the exception is a teardown fault, not an assertion/logic failure — the tenant operations themselves succeed.
- Confirmed it is not runtime/BCL-specific by reproducing on stable .NET with the real `DefaultTenantService`.
- Verified the `DeactivateTenantsAsync` source is unchanged across 3.7.x → `main`, so no version bump resolves it.
- **Workaround** (generic, in case it helps others): wrap `ITenantService` in a decorator that serializes the three mutators (`ActivateTenantsAsync` / `DeactivateTenantsAsync` / `RefreshAsync`) behind a single `SemaphoreSlim`, passing reads through ungated. Note: such a decorator should **not** dispose that gate, since `DeactivateTenantsAsync` runs during host shutdown and would otherwise hit `ObjectDisposedException` at teardown.

## Suggested Fix

Acquire `_refreshLock` in `DeactivateTenantsAsync` (and any other method that mutates `_tenantsDictionary`) so all mutators share the same gate as `RefreshAsync` — or back the tenant set with a thread-safe collection.

## Additional Context

The negative-count `OverflowException` requires two concurrent writers, so serializing the mutators fully resolves it. Reads are not the cause (a reader alone cannot drive the count negative), though gating reads as well would risk deadlocking a read issued during activation.

## Related Issues

None found matching this specific thread-safety defect at the time of filing.



Discussion:

Patch:
diff --git a/src/modules/Elsa.Common/Multitenancy/Implementations/DefaultTenantService.cs b/src/modules/Elsa.Common/Multitenancy/Implementations/DefaultTenantService.cs
index 5dc7f0c05d..95a336aa95 100644
--- a/src/modules/Elsa.Common/Multitenancy/Implementations/DefaultTenantService.cs
+++ b/src/modules/Elsa.Common/Multitenancy/Implementations/DefaultTenantService.cs
@@ -7,14 +7,13 @@ public class DefaultTenantService(IServiceScopeFactory scopeFactory, ITenantScop
 {
     private readonly AsyncServiceScope _serviceScope = scopeFactory.CreateAsyncScope();
     private readonly SemaphoreSlim _initializationLock = new(1, 1);
-    private readonly SemaphoreSlim _refreshLock = new(1, 1);
+    private readonly SemaphoreSlim _tenantMutationLock = new(1, 1);
     private IDictionary<string, Tenant>? _tenantsDictionary;
     private IDictionary<Tenant, TenantScope>? _tenantScopesDictionary;
 
     public async ValueTask DisposeAsync()
     {
         await _serviceScope.DisposeAsync();
-        _initializationLock.Dispose();
     }
 
     public async Task<Tenant?> FindAsync(string id, CancellationToken cancellationToken = default)
@@ -60,22 +59,33 @@ public async Task ActivateTenantsAsync(CancellationToken cancellationToken = def
 
     public async Task DeactivateTenantsAsync(CancellationToken cancellationToken = default)
     {
-        var dictionary = await GetTenantsDictionaryAsync(cancellationToken);
-        var tenants = dictionary.Values.ToArray();
+        var dictionary = await GetTenantsDictionaryForMutationAsync(cancellationToken);
+        await _tenantMutationLock.WaitAsync(cancellationToken);
+
+        try
+        {
+            var tenants = dictionary.Values.ToArray();
 
-        foreach (var tenant in tenants)
-            await UnregisterTenantAsync(tenant, false, cancellationToken);
+            foreach (var tenant in tenants)
+            {
+                await UnregisterTenantAsync(tenant, false, cancellationToken);
+            }
+        }
+        finally
+        {
+            _tenantMutationLock.Release();
+        }
     }
 
     public async Task RefreshAsync(CancellationToken cancellationToken = default)
     {
-        await _refreshLock.WaitAsync(cancellationToken);
+        var currentTenants = await GetTenantsDictionaryForMutationAsync(cancellationToken);
+        await _tenantMutationLock.WaitAsync(cancellationToken);
 
         try
         {
             await using var scope = scopeFactory.CreateAsyncScope();
             var tenantsProvider = scope.ServiceProvider.GetRequiredService<ITenantsProvider>();
-            var currentTenants = await GetTenantsDictionaryAsync(cancellationToken);
             var currentTenantIds = currentTenants.Keys;
             var tenantsFromProvider = (await tenantsProvider.ListAsync(cancellationToken)).ToList();
             var newTenants = tenantsFromProvider.Count == 0
@@ -99,10 +109,22 @@ public async Task RefreshAsync(CancellationToken cancellationToken = default)
         }
         finally
         {
-            _refreshLock.Release();
+            _tenantMutationLock.Release();
         }
     }
 
+    private async Task<IDictionary<string, Tenant>> GetTenantsDictionaryForMutationAsync(CancellationToken cancellationToken)
+    {
+        var dictionary = await GetTenantsDictionaryAsync(cancellationToken);
+
+        // The dictionary is published before initialization completes so lifecycle event handlers can read it.
+        // Wait for any concurrent initializer before allowing a mutation to proceed.
+        await _initializationLock.WaitAsync(cancellationToken);
+        _initializationLock.Release();
+
+        return dictionary;
+    }
+
     private async Task<IDictionary<string, Tenant>> GetTenantsDictionaryAsync(CancellationToken cancellationToken)
     {
         if (_tenantsDictionary == null)
@@ -157,4 +179,4 @@ private async Task UnregisterTenantAsync(Tenant tenant, bool isDeleted, Cancella
             }
         }
     }
-}
\ No newline at end of file
+}


Test patch:
diff --git a/test/unit/Elsa.Common.UnitTests/Multitenancy/DefaultTenantServiceTests.cs b/test/unit/Elsa.Common.UnitTests/Multitenancy/DefaultTenantServiceTests.cs
index 89685a0d58..13566e714b 100644
--- a/test/unit/Elsa.Common.UnitTests/Multitenancy/DefaultTenantServiceTests.cs
+++ b/test/unit/Elsa.Common.UnitTests/Multitenancy/DefaultTenantServiceTests.cs
@@ -119,6 +119,147 @@ public async Task RefreshAsync_WhenProviderChangesFromTenantsToEmpty_KeepsDefaul
         }
     }
 
+    [Fact]
+    public async Task DeactivateTenantsAsync_WhenRefreshIsInProgress_WaitsForRefreshBeforeDeactivating()
+    {
+        var tenant = new Tenant { Id = "tenant-1", Name = "Tenant 1" };
+        var timeout = TimeSpan.FromSeconds(5);
+        var refreshStarted = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
+        var completeRefresh = new TaskCompletionSource<IEnumerable<Tenant>>(TaskCreationOptions.RunContinuationsAsynchronously);
+        var listRequests = 0;
+        var tenantsProvider = Substitute.For<ITenantsProvider>();
+        tenantsProvider.ListAsync(Arg.Any<CancellationToken>()).Returns(async _ =>
+        {
+            if (Interlocked.Increment(ref listRequests) == 1)
+            {
+                return new[] { tenant };
+            }
+
+            refreshStarted.TrySetResult();
+            return await completeRefresh.Task;
+        });
+
+        var (tenantService, serviceProvider) = await CreateTenantServiceAsync(tenantsProvider);
+        Task? refreshTask = null;
+        Task? deactivateTask = null;
+
+        try
+        {
+            await tenantService.ListAsync();
+
+            refreshTask = tenantService.RefreshAsync();
+            await refreshStarted.Task.WaitAsync(timeout);
+
+            deactivateTask = tenantService.DeactivateTenantsAsync();
+
+            Assert.False(deactivateTask.IsCompleted);
+
+            completeRefresh.SetResult([tenant]);
+            await refreshTask.WaitAsync(timeout);
+            await deactivateTask.WaitAsync(timeout);
+
+            Assert.Empty(await tenantService.ListAsync());
+        }
+        finally
+        {
+            completeRefresh.TrySetResult([tenant]);
+
+            if (refreshTask != null)
+            {
+                await refreshTask.WaitAsync(timeout);
+            }
+
+            if (deactivateTask != null)
+            {
+                await deactivateTask.WaitAsync(timeout);
+            }
+
+            if (tenantService is IAsyncDisposable disposable)
+            {
+                await disposable.DisposeAsync();
+            }
+
+            await serviceProvider.DisposeAsync();
+        }
+    }
+
+    [Fact]
+    public async Task DeactivateTenantsAsync_WhenInitializationIsInProgress_WaitsForInitializationBeforeDeactivating()
+    {
+        var tenant = new Tenant { Id = "tenant-1", Name = "Tenant 1" };
+        var timeout = TimeSpan.FromSeconds(5);
+        var initializationStarted = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
+        var completeInitialization = new TaskCompletionSource<IEnumerable<Tenant>>(TaskCreationOptions.RunContinuationsAsynchronously);
+        var tenantsProvider = Substitute.For<ITenantsProvider>();
+        tenantsProvider.ListAsync(Arg.Any<CancellationToken>()).Returns(async _ =>
+        {
+            initializationStarted.TrySetResult();
+            return await completeInitialization.Task;
+        });
+
+        var (tenantService, serviceProvider) = await CreateTenantServiceAsync(tenantsProvider);
+        Task<IEnumerable<Tenant>>? initializationTask = null;
+        Task? deactivateTask = null;
+
+        try
+        {
+            initializationTask = tenantService.ListAsync();
+            await initializationStarted.Task.WaitAsync(timeout);
+
+            deactivateTask = tenantService.DeactivateTenantsAsync();
+
+            Assert.False(deactivateTask.IsCompleted);
+
+            completeInitialization.SetResult([tenant]);
+            await initializationTask.WaitAsync(timeout);
+            await deactivateTask.WaitAsync(timeout);
+
+            Assert.Empty(await tenantService.ListAsync());
+        }
+        finally
+        {
+            completeInitialization.TrySetResult([tenant]);
+
+            if (initializationTask != null)
+            {
+                await initializationTask.WaitAsync(timeout);
+            }
+
+            if (deactivateTask != null)
+            {
+                await deactivateTask.WaitAsync(timeout);
+            }
+
+            if (tenantService is IAsyncDisposable disposable)
+            {
+                await disposable.DisposeAsync();
+            }
+
+            await serviceProvider.DisposeAsync();
+        }
+    }
+
+    [Fact]
+    public async Task DeactivateTenantsAsync_AfterDisposeAsync_DoesNotUseDisposedSynchronizationPrimitives()
+    {
+        var tenant = new Tenant { Id = "tenant-1", Name = "Tenant 1" };
+        var (tenantService, serviceProvider) = await CreateTenantServiceAsync([tenant]);
+
+        try
+        {
+            await tenantService.ListAsync();
+            await ((IAsyncDisposable)tenantService).DisposeAsync();
+
+            await tenantService.DeactivateTenantsAsync();
+
+            Assert.Empty(await tenantService.ListAsync());
+        }
+        finally
+        {
+            await serviceProvider.DisposeAsync();
+        }
+    }
+
     private static Task<(ITenantService TenantService, ServiceProvider ServiceProvider)> CreateTenantServiceAsync(IEnumerable<Tenant> tenants, Func<List<Tenant>>? tenantsFactory = null)
     {
         var tenantList = tenants.ToList();
@@ -127,6 +268,11 @@ public async Task RefreshAsync_WhenProviderChangesFromTenantsToEmpty_KeepsDefaul
         var tenantsProvider = Substitute.For<ITenantsProvider>();
         tenantsProvider.ListAsync(Arg.Any<CancellationToken>()).Returns(_ => getTenants());
 
+        return CreateTenantServiceAsync(tenantsProvider);
+    }
+
+    private static Task<(ITenantService TenantService, ServiceProvider ServiceProvider)> CreateTenantServiceAsync(ITenantsProvider tenantsProvider)
+    {
         var services = new ServiceCollection();
         services.AddSingleton(_ => tenantsProvider);
         services.AddSingleton<ITenantScopeFactory, DefaultTenantScopeFactory>();