Repository: App-vNext/Polly

Issue:
ObjectDisposedException thrown if a request is inflight when configuration is reloaded
Testing out Polly `8.0.0-beta.1` in an internal production application, we have observed the following exception thrown when using a resilience strategy occur if the application's configuration is reloaded while the policy is in use.

```
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'CircuitStateController'.
   at Polly.CircuitBreaker.CircuitStateController`1.EnsureNotDisposed()
   at Polly.CircuitBreaker.CircuitStateController`1.OnActionSuccessAsync(Outcome`1 outcome, ResilienceContext context)
   at Polly.CircuitBreaker.CircuitBreakerResilienceStrategy`1.ExecuteCore[TState](Func`3 callback, ResilienceContext context, TState state)
   at Polly.Utils.Pipeline.BridgeComponent`1.<ConvertValueTask>g__ConvertValueTaskAsync|5_0[TTo](ValueTask`1 valueTask, ResilienceContext resilienceContext)
   at Polly.Utils.Pipeline.CompositeComponent.ExecuteCoreWithTelemetry[TResult,TState](Func`3 callback, ResilienceContext context, TState state)
   at Polly.Utils.StrategyHelper.<ExecuteCallbackSafeAsync>g__AwaitTask|0_0[TResult,TState,T](ValueTask`1 task, Boolean continueOnCapturedContext)
   at Polly.Outcome`1.GetResultOrRethrow()
   at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, ResilienceContext context)
   at MyApplication.Providers.ApiClient.ExecuteAsync[TResult](String rateLimitToken, String operationKey, ClientExecuteOptions`1 clientExecuteOptions, Func`3 action, CancellationToken cancellationToken) in /_/src/MyApplication/Providers/ApiClient.cs:line 186
   at MyApplication.Providers.ApiClient.ExecuteAsync[T](String rateLimitToken, String operationKey, Func`2 func, ClientExecuteOptions`1 clientExecuteOptions, CancellationToken cancellationToken) in /_/src/MyApplication/Providers/ApiClient.cs:line 125
   at MyApplication.Providers.InflightClient.GetByOrderIdAsync(String tenant, String orderId, CancellationToken cancellationToken, Boolean handleExecutionFaults) in /_/src/MyApplication/Providers/InflightClient.cs:line 74
   at MyApplication.Services.StatusService.GetOrderStatusAsync(String tenant, String id, CancellationToken cancellationToken) in /_/src/MyApplication/Services/StatusService.cs:line 109
   at MyApplication.Controllers.StatusController.GetStatusAsync(String tenant, String id, CancellationToken cancellationToken) in /_/src/MyApplication/Controllers/StatusController.cs:line 143
   at MyApplication.Controllers.StatusController.GetStatusByOrderIdAuthorized(String tenant, String id, CancellationToken cancellationToken) in /_/src/MyApplication/Controllers/StatusController.cs:line 96
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
```

I can't share the application code, but the overall structure is very similar to https://github.com/martincostello/polly-sandbox/pull/1.

Over a pool of 6 EC2 instances handling ~8,500 requests per minute between them at the time, we had 2 occurrences, so imagine there's an element of chance to whether or not it happens when the application is under load when the configuration is reloaded.



Hints:
Just found a second cluster of 3 of the same exception in another environment with a similar load and number of instances in service.
I was worried this might happen. The solution is to keep track of active executions in circuit breaker and delay the disposal until the pipeline serves no more executions.

Patch:
diff --git a/src/Polly.Core/Registry/RegistryPipelineComponentBuilder.cs b/src/Polly.Core/Registry/RegistryPipelineComponentBuilder.cs
index e29c89560da..4493631f8cd 100644
--- a/src/Polly.Core/Registry/RegistryPipelineComponentBuilder.cs
+++ b/src/Polly.Core/Registry/RegistryPipelineComponentBuilder.cs
@@ -57,14 +57,17 @@ private Builder CreateBuilder()
         builder.InstanceName = _instanceName;
         _configure(builder, context);
 
+        var timeProvider = builder.TimeProvider;
         var telemetry = new ResilienceStrategyTelemetry(
             new ResilienceTelemetrySource(builder.Name, builder.InstanceName, null),
             builder.TelemetryListener);
 
         return new(
-            () => PipelineComponentFactory.WithDisposableCallbacks(
-                    builder.BuildPipelineComponent(),
-                    context.DisposeCallbacks),
+            () =>
+            {
+                var innerComponent = PipelineComponentFactory.WithDisposableCallbacks(builder.BuildPipelineComponent(), context.DisposeCallbacks);
+                return PipelineComponentFactory.WithExecutionTracking(innerComponent, timeProvider);
+            },
             context.ReloadTokens,
             telemetry);
     }
diff --git a/src/Polly.Core/Utils/Pipeline/ExecutionTrackingComponent.cs b/src/Polly.Core/Utils/Pipeline/ExecutionTrackingComponent.cs
new file mode 100644
index 00000000000..5a03e7f94fc
--- /dev/null
+++ b/src/Polly.Core/Utils/Pipeline/ExecutionTrackingComponent.cs
@@ -0,0 +1,59 @@
+﻿namespace Polly.Utils.Pipeline;
+
+internal sealed class ExecutionTrackingComponent : PipelineComponent
+{
+    public static readonly TimeSpan Timeout = TimeSpan.FromSeconds(30);
+
+    public static readonly TimeSpan SleepDelay = TimeSpan.FromSeconds(1);
+
+    private readonly TimeProvider _timeProvider;
+    private int _pendingExecutions;
+
+    public ExecutionTrackingComponent(PipelineComponent component, TimeProvider timeProvider)
+    {
+        Component = component;
+        _timeProvider = timeProvider;
+    }
+
+    public PipelineComponent Component { get; }
+
+    public bool HasPendingExecutions => Interlocked.CompareExchange(ref _pendingExecutions, 0, 0) > 0;
+
+    internal override async ValueTask<Outcome<TResult>> ExecuteCore<TResult, TState>(
+        Func<ResilienceContext, TState, ValueTask<Outcome<TResult>>> callback,
+        ResilienceContext context,
+        TState state)
+    {
+        Interlocked.Increment(ref _pendingExecutions);
+
+        try
+        {
+            return await Component.ExecuteCore(callback, context, state).ConfigureAwait(context.ContinueOnCapturedContext);
+        }
+        finally
+        {
+            Interlocked.Decrement(ref _pendingExecutions);
+        }
+    }
+
+    public override async ValueTask DisposeAsync()
+    {
+        var start = _timeProvider.GetTimestamp();
+        var stopwatch = Stopwatch.StartNew();
+
+        // We don't want to introduce locks or any synchronization primitives to main execution path
+        // so we will do "dummy" retries until there are no more executions.
+        while (HasPendingExecutions)
+        {
+            await _timeProvider.Delay(SleepDelay).ConfigureAwait(false);
+
+            // stryker disable once equality : no means to test this
+            if (_timeProvider.GetElapsedTime(start) > Timeout)
+            {
+                break;
+            }
+        }
+
+        await Component.DisposeAsync().ConfigureAwait(false);
+    }
+}
diff --git a/src/Polly.Core/Utils/Pipeline/PipelineComponentFactory.cs b/src/Polly.Core/Utils/Pipeline/PipelineComponentFactory.cs
index 125f742a103..5c992295fc8 100644
--- a/src/Polly.Core/Utils/Pipeline/PipelineComponentFactory.cs
+++ b/src/Polly.Core/Utils/Pipeline/PipelineComponentFactory.cs
@@ -24,6 +24,8 @@ public static PipelineComponent WithDisposableCallbacks(PipelineComponent compon
         return new ComponentWithDisposeCallbacks(component, callbacks.ToList());
     }
 
+    public static PipelineComponent WithExecutionTracking(PipelineComponent component, TimeProvider timeProvider) => new ExecutionTrackingComponent(component, timeProvider);
+
     public static PipelineComponent CreateComposite(
         IReadOnlyList<PipelineComponent> components,
         ResilienceStrategyTelemetry telemetry,
diff --git a/src/Polly.Testing/ResiliencePipelineExtensions.cs b/src/Polly.Testing/ResiliencePipelineExtensions.cs
index 19692182d30..feea4ad59d3 100644
--- a/src/Polly.Testing/ResiliencePipelineExtensions.cs
+++ b/src/Polly.Testing/ResiliencePipelineExtensions.cs
@@ -40,10 +40,10 @@ private static ResiliencePipelineDescriptor GetPipelineDescriptorCore<T>(Pipelin
         var components = new List<PipelineComponent>();
         component.ExpandComponents(components);
 
-        var descriptors = components.Select(s => new ResilienceStrategyDescriptor(s.Options, GetStrategyInstance<T>(s))).ToList();
+        var descriptors = components.OfType<BridgeComponentBase>().Select(s => new ResilienceStrategyDescriptor(s.Options, GetStrategyInstance<T>(s))).ToList().AsReadOnly();
 
         return new ResiliencePipelineDescriptor(
-            descriptors.Where(s => !ShouldSkip(s.StrategyInstance)).ToList().AsReadOnly(),
+            descriptors,
             isReloadable: components.Exists(s => s is ReloadableComponent));
     }
 
@@ -54,16 +54,9 @@ private static object GetStrategyInstance<T>(PipelineComponent component)
             return reactiveBridge.Strategy;
         }
 
-        if (component is BridgeComponent nonReactiveBridge)
-        {
-            return nonReactiveBridge.Strategy;
-        }
-
-        return component;
+        return ((BridgeComponent)component).Strategy;
     }
 
-    private static bool ShouldSkip(object instance) => instance is ReloadableComponent || instance is ComponentWithDisposeCallbacks;
-
     private static void ExpandComponents(this PipelineComponent component, List<PipelineComponent> components)
     {
         if (component is CompositeComponent pipeline)
@@ -78,6 +71,11 @@ private static void ExpandComponents(this PipelineComponent component, List<Pipe
             components.Add(reloadable);
             ExpandComponents(reloadable.Component, components);
         }
+        else if (component is ExecutionTrackingComponent tracking)
+        {
+            components.Add(tracking);
+            ExpandComponents(tracking.Component, components);
+        }
         else if (component is ComponentWithDisposeCallbacks callbacks)
         {
             ExpandComponents(callbacks.Component, components);


Test patch:
diff --git a/test/Polly.Core.Tests/CircuitBreaker/Controller/ScheduledTaskExecutorTests.cs b/test/Polly.Core.Tests/CircuitBreaker/Controller/ScheduledTaskExecutorTests.cs
index 4b85dd4fc99..f24b8791324 100644
--- a/test/Polly.Core.Tests/CircuitBreaker/Controller/ScheduledTaskExecutorTests.cs
+++ b/test/Polly.Core.Tests/CircuitBreaker/Controller/ScheduledTaskExecutorTests.cs
@@ -123,11 +123,11 @@ public void Dispose_WhenScheduledTaskExecuting()
             ResilienceContextPool.Shared.Get(),
             out var task);
 
-        ready.WaitOne(TimeSpan.FromSeconds(2)).Should().BeTrue();
+        ready.WaitOne(TimeSpan.FromSeconds(10)).Should().BeTrue();
         scheduler.Dispose();
         disposed.Set();
 
-        scheduler.ProcessingTask.Wait(TimeSpan.FromSeconds(2)).Should().BeTrue();
+        scheduler.ProcessingTask.Wait(TimeSpan.FromSeconds(10)).Should().BeTrue();
     }
 
     [Fact]
diff --git a/test/Polly.Core.Tests/Registry/ResiliencePipelineRegistryTests.cs b/test/Polly.Core.Tests/Registry/ResiliencePipelineRegistryTests.cs
index 547f1111cb7..23f1d5032dc 100644
--- a/test/Polly.Core.Tests/Registry/ResiliencePipelineRegistryTests.cs
+++ b/test/Polly.Core.Tests/Registry/ResiliencePipelineRegistryTests.cs
@@ -4,6 +4,7 @@
 using Polly.Testing;
 using Polly.Timeout;
 using Polly.Utils;
+using Polly.Utils.Pipeline;
 
 namespace Polly.Core.Tests.Registry;
 
@@ -380,6 +381,20 @@ public void GetOrAddPipeline_Ok()
         called.Should().Be(1);
     }
 
+    [Fact]
+    public void GetOrAddPipeline_EnsureCorrectComponents()
+    {
+        var id = new StrategyId(typeof(string), "A");
+
+        using var registry = CreateRegistry();
+
+        var pipeline = registry.GetOrAddPipeline(id, builder => builder.AddTimeout(TimeSpan.FromSeconds(1)));
+        pipeline.Component.Should().BeOfType<ExecutionTrackingComponent>().Subject.Component.Should().BeOfType<CompositeComponent>();
+
+        var genericPipeline = registry.GetOrAddPipeline<string>(id, builder => builder.AddTimeout(TimeSpan.FromSeconds(1)));
+        pipeline.Component.Should().BeOfType<ExecutionTrackingComponent>().Subject.Component.Should().BeOfType<CompositeComponent>();
+    }
+
     [Fact]
     public void GetOrAddPipeline_Generic_Ok()
     {
diff --git a/test/Polly.Core.Tests/Utils/Pipeline/ExecutionTrackingComponentTests.cs b/test/Polly.Core.Tests/Utils/Pipeline/ExecutionTrackingComponentTests.cs
new file mode 100644
index 00000000000..2cebbffc5e3
--- /dev/null
+++ b/test/Polly.Core.Tests/Utils/Pipeline/ExecutionTrackingComponentTests.cs
@@ -0,0 +1,168 @@
+﻿using System;
+using System.Threading.Tasks;
+using Microsoft.Extensions.Time.Testing;
+using Polly.Utils.Pipeline;
+
+namespace Polly.Core.Tests.Utils.Pipeline;
+
+public class ExecutionTrackingComponentTests
+{
+    private readonly FakeTimeProvider _timeProvider = new();
+
+    [Fact]
+    public async Task DisposeAsync_PendingOperations_Delayed()
+    {
+        using var assert = new ManualResetEvent(false);
+        using var executing = new ManualResetEvent(false);
+
+        await using var inner = new Inner
+        {
+            OnExecute = () =>
+            {
+                executing.Set();
+                assert.WaitOne();
+            }
+        };
+
+        var component = new ExecutionTrackingComponent(inner, _timeProvider);
+        var execution = Task.Run(() => new ResiliencePipeline(component, Polly.Utils.DisposeBehavior.Allow).Execute(() => { }));
+        executing.WaitOne();
+
+        var disposeTask = component.DisposeAsync().AsTask();
+        _timeProvider.Advance(ExecutionTrackingComponent.SleepDelay);
+        inner.Disposed.Should().BeFalse();
+        assert.Set();
+
+        _timeProvider.Advance(ExecutionTrackingComponent.SleepDelay);
+        await execution;
+
+        _timeProvider.Advance(ExecutionTrackingComponent.SleepDelay);
+        await disposeTask;
+
+        inner.Disposed.Should().BeTrue();
+    }
+
+    [Fact]
+    public async Task HasPendingExecutions_Ok()
+    {
+        using var assert = new ManualResetEvent(false);
+        using var executing = new ManualResetEvent(false);
+
+        await using var inner = new Inner
+        {
+            OnExecute = () =>
+            {
+                executing.Set();
+                assert.WaitOne();
+            }
+        };
+
+        await using var component = new ExecutionTrackingComponent(inner, _timeProvider);
+        var execution = Task.Run(() => new ResiliencePipeline(component, Polly.Utils.DisposeBehavior.Allow).Execute(() => { }));
+        executing.WaitOne();
+
+        component.HasPendingExecutions.Should().BeTrue();
+        assert.Set();
+        await execution;
+
+        component.HasPendingExecutions.Should().BeFalse();
+    }
+
+    [Fact]
+    public async Task DisposeAsync_Timeout_Ok()
+    {
+        using var assert = new ManualResetEvent(false);
+        using var executing = new ManualResetEvent(false);
+
+        await using var inner = new Inner
+        {
+            OnExecute = () =>
+            {
+                executing.Set();
+                assert.WaitOne();
+            }
+        };
+
+        var component = new ExecutionTrackingComponent(inner, _timeProvider);
+        var execution = Task.Run(() => new ResiliencePipeline(component, Polly.Utils.DisposeBehavior.Allow).Execute(() => { }));
+        executing.WaitOne();
+
+        var disposeTask = component.DisposeAsync().AsTask();
+        inner.Disposed.Should().BeFalse();
+        _timeProvider.Advance(ExecutionTrackingComponent.Timeout - TimeSpan.FromSeconds(1));
+        inner.Disposed.Should().BeFalse();
+        _timeProvider.Advance(TimeSpan.FromSeconds(1));
+        _timeProvider.Advance(TimeSpan.FromSeconds(1));
+        await disposeTask;
+        inner.Disposed.Should().BeTrue();
+
+        assert.Set();
+        await execution;
+    }
+
+    [Fact]
+    public async Task DisposeAsync_WhenRunningMultipleTasks_Ok()
+    {
+        var tasks = new ConcurrentQueue<ManualResetEvent>();
+        await using var inner = new Inner
+        {
+            OnExecute = () =>
+            {
+                var ev = new ManualResetEvent(false);
+                tasks.Enqueue(ev);
+                ev.WaitOne();
+            }
+        };
+
+        var component = new ExecutionTrackingComponent(inner, TimeProvider.System);
+        var pipeline = new ResiliencePipeline(component, Polly.Utils.DisposeBehavior.Allow);
+
+        for (int i = 0; i < 10; i++)
+        {
+            _ = Task.Run(() => pipeline.Execute(() => { }));
+        }
+
+        while (tasks.Count != 10)
+        {
+            await Task.Delay(1);
+        }
+
+        var disposeTask = component.DisposeAsync().AsTask();
+
+        while (tasks.Count > 1)
+        {
+            tasks.TryDequeue(out var ev).Should().BeTrue();
+            ev!.Set();
+            ev.Dispose();
+            disposeTask.Wait(1).Should().BeFalse();
+            inner.Disposed.Should().BeFalse();
+        }
+
+        // last one
+        tasks.TryDequeue(out var last).Should().BeTrue();
+        last!.Set();
+        last.Dispose();
+        await disposeTask;
+        inner.Disposed.Should().BeTrue();
+    }
+
+    private class Inner : PipelineComponent
+    {
+        public bool Disposed { get; private set; }
+
+        public override ValueTask DisposeAsync()
+        {
+            Disposed = true;
+            return default;
+        }
+
+        public Action OnExecute { get; set; } = () => { };
+
+        internal override async ValueTask<Outcome<TResult>> ExecuteCore<TResult, TState>(Func<ResilienceContext, TState, ValueTask<Outcome<TResult>>> callback, ResilienceContext context, TState state)
+        {
+            OnExecute();
+
+            return await callback(context, state);
+        }
+    }
+}