Repository: temporalio/sdk-java
Pull request #2433: :boom: [Breaking] Asyncify slot suppliers
Author: login: Sushisource
Created: 2025-02-28T03:03:36Z | Merged: 2025-03-24T16:44:26Z

<!--- Note to EXTERNAL Contributors -->
<!-- Thanks for opening a PR! 
If it is a significant code change, please **make sure there is an open issue** for this. 
We work best with you when we have accepted the idea first before you code. -->

<!--- For ALL Contributors 👇 -->

## What was changed
Made the `reserveSlot` method on `SlotSupplier` interface async.

Note that, as it stands, this can actually slightly _increase_ the number of threads used because the slot suppliers have been changed in a way that they will not block the caller (which is the whole point) but in order to do that, the resource based one at least needs a couple threads that it didn't before. 

For the fixed size supplier, I think I've managed to come up with something that should always be non-blocking.

## Why?
This is to support https://github.com/temporalio/sdk-java/issues/1456 where the pollers themselves will be made async, and thus will be able to take advantage of async slot reservation as well.

## Checklist
<!--- add/delete as needed --->

1. Closes <!-- add issue number here -->

2. How was this tested:
Existing tests

3. Any docs updates needed?
<!--- update README if applicable
      or point out where to update docs.temporal.io -->



Diff:
diff --git a/temporal-sdk/src/main/java/io/temporal/internal/worker/ActivityPollTask.java b/temporal-sdk/src/main/java/io/temporal/internal/worker/ActivityPollTask.java
index 0327bc6f33..0bd6873f5e 100644
--- a/temporal-sdk/src/main/java/io/temporal/internal/worker/ActivityPollTask.java
+++ b/temporal-sdk/src/main/java/io/temporal/internal/worker/ActivityPollTask.java
@@ -95,22 +95,21 @@ public ActivityTask poll() {
     }
     PollActivityTaskQueueResponse response;
     SlotPermit permit;
+    SlotSupplierFuture future;
     boolean isSuccessful = false;
-
     try {
-      permit =
+      future =
           slotSupplier.reserveSlot(
               new SlotReservationData(
                   pollRequest.getTaskQueue().getName(),
                   pollRequest.getIdentity(),
                   pollRequest.getWorkerVersionCapabilities().getBuildId()));
-    } catch (InterruptedException e) {
-      Thread.currentThread().interrupt();
-      return null;
     } catch (Exception e) {
       log.warn("Error while trying to reserve a slot for an activity", e.getCause());
       return null;
     }
+    permit = Poller.getSlotPermitAndHandleInterrupts(future, slotSupplier);
+    if (permit == null) return null;
 
     try {
       response =
diff --git a/temporal-sdk/src/main/java/io/temporal/internal/worker/LocalActivitySlotSupplierQueue.java b/temporal-sdk/src/main/java/io/temporal/internal/worker/LocalActivitySlotSupplierQueue.java
index b573677f68..74cec7d872 100644
--- a/temporal-sdk/src/main/java/io/temporal/internal/worker/LocalActivitySlotSupplierQueue.java
+++ b/temporal-sdk/src/main/java/io/temporal/internal/worker/LocalActivitySlotSupplierQueue.java
@@ -23,6 +23,7 @@
 import io.temporal.worker.tuning.LocalActivitySlotInfo;
 import io.temporal.worker.tuning.SlotPermit;
 import io.temporal.worker.tuning.SlotReleaseReason;
+import io.temporal.worker.tuning.SlotSupplierFuture;
 import io.temporal.workflow.Functions;
 import java.util.concurrent.*;
 import javax.annotation.Nullable;
@@ -83,22 +84,30 @@ private void processQueue() {
       QueuedLARequest request = null;
       try {
         request = requestQueue.take();
+
+        SlotSupplierFuture future = slotSupplier.reserveSlot(request.data);
         try {
-          slotPermit = slotSupplier.reserveSlot(request.data);
+          slotPermit = future.get();
         } catch (InterruptedException e) {
+          SlotPermit maybePermitAnyway = future.abortReservation();
+          if (maybePermitAnyway != null) {
+            slotSupplier.releaseSlot(SlotReleaseReason.neverUsed(), maybePermitAnyway);
+          }
           Thread.currentThread().interrupt();
           return;
-        } catch (Exception e) {
+        } catch (ExecutionException e) {
           log.error(
               "Error reserving local activity slot, dropped activity id {}",
               request.task.getActivityId(),
               e);
           continue;
         }
+
         request.task.getExecutionContext().setPermit(slotPermit);
         afterReservedCallback.apply(request.task);
       } catch (InterruptedException e) {
         Thread.currentThread().interrupt();
+        return;
       } catch (Throwable e) {
         // Fail the workflow task if something went wrong executing the local activity (at the
         // executor level, otherwise, the LA handler itself should be handling errors)
@@ -112,6 +121,11 @@ private void processQueue() {
               LocalActivityResult.processingFailed(
                   executionContext.getActivityId(), request.task.getAttemptTask().getAttempt(), e));
         }
+        if (e.getCause() instanceof InterruptedException) {
+          // It's possible the interrupt happens inside the callback, so check that as well.
+          Thread.currentThread().interrupt();
+          return;
+        }
       }
     }
   }
@@ -162,11 +176,9 @@ public boolean isTerminated() {
   @Override
   public CompletableFuture<Void> shutdown(ShutdownManager shutdownManager, boolean interruptTasks) {
     running = false;
-    if (requestQueue.isEmpty()) {
-      // Just interrupt the thread, so that if we're waiting on blocking take the thread will
-      // be interrupted and exit. Otherwise the loop will exit once the queue is empty.
-      queueThreadService.shutdownNow();
-    }
+    // Always interrupt. This won't cause any *tasks* to be interrupted, since the queue thread is
+    // only responsible for handing them out.
+    queueThreadService.shutdownNow();
 
     return interruptTasks
         ? shutdownManager.shutdownExecutorNowUntimed(
@@ -182,6 +194,7 @@ public void awaitTermination(long timeout, TimeUnit unit) {
       // timeout duration if no task was ever submitted.
       return;
     }
+
     ShutdownManager.awaitTermination(queueThreadService, unit.toMillis(timeout));
   }
 }
diff --git a/temporal-sdk/src/main/java/io/temporal/internal/worker/NexusPollTask.java b/temporal-sdk/src/main/java/io/temporal/internal/worker/NexusPollTask.java
index 2fc47c0cd6..b1b62b6bad 100644
--- a/temporal-sdk/src/main/java/io/temporal/internal/worker/NexusPollTask.java
+++ b/temporal-sdk/src/main/java/io/temporal/internal/worker/NexusPollTask.java
@@ -85,22 +85,21 @@ public NexusTask poll() {
     }
     PollNexusTaskQueueResponse response;
     SlotPermit permit;
+    SlotSupplierFuture future;
     boolean isSuccessful = false;
-
     try {
-      permit =
+      future =
           slotSupplier.reserveSlot(
               new SlotReservationData(
                   pollRequest.getTaskQueue().getName(),
                   pollRequest.getIdentity(),
                   pollRequest.getWorkerVersionCapabilities().getBuildId()));
-    } catch (InterruptedException e) {
-      Thread.currentThread().interrupt();
-      return null;
     } catch (Exception e) {
       log.warn("Error while trying to reserve a slot for a nexus task", e.getCause());
       return null;
     }
+    permit = Poller.getSlotPermitAndHandleInterrupts(future, slotSupplier);
+    if (permit == null) return null;
 
     try {
       response =
diff --git a/temporal-sdk/src/main/java/io/temporal/internal/worker/Poller.java b/temporal-sdk/src/main/java/io/temporal/internal/worker/Poller.java
index 33d630e50e..b55e6611bf 100644
--- a/temporal-sdk/src/main/java/io/temporal/internal/worker/Poller.java
+++ b/temporal-sdk/src/main/java/io/temporal/internal/worker/Poller.java
@@ -27,6 +27,9 @@
 import io.temporal.internal.common.GrpcUtils;
 import io.temporal.internal.task.VirtualThreadDelegate;
 import io.temporal.worker.MetricsType;
+import io.temporal.worker.tuning.SlotPermit;
+import io.temporal.worker.tuning.SlotReleaseReason;
+import io.temporal.worker.tuning.SlotSupplierFuture;
 import java.time.Duration;
 import java.util.Objects;
 import java.util.concurrent.*;
@@ -222,6 +225,25 @@ public WorkerLifecycleState getLifecycleState() {
     return WorkerLifecycleState.ACTIVE;
   }
 
+  static SlotPermit getSlotPermitAndHandleInterrupts(
+      SlotSupplierFuture future, TrackingSlotSupplier<?> slotSupplier) {
+    SlotPermit permit;
+    try {
+      permit = future.get();
+    } catch (InterruptedException e) {
+      SlotPermit maybePermitAnyway = future.abortReservation();
+      if (maybePermitAnyway != null) {
+        slotSupplier.releaseSlot(SlotReleaseReason.neverUsed(), maybePermitAnyway);
+      }
+      Thread.currentThread().interrupt();
+      return null;
+    } catch (ExecutionException e) {
+      log.warn("Error while trying to reserve a slot", e.getCause());
+      return null;
+    }
+    return permit;
+  }
+
   @Override
   public String toString() {
     // TODO using pollThreadNamePrefix here is ugly. We should consider introducing some concept of
diff --git a/temporal-sdk/src/main/java/io/temporal/internal/worker/TrackingSlotSupplier.java b/temporal-sdk/src/main/java/io/temporal/internal/worker/TrackingSlotSupplier.java
index 6ee36d6398..ca91b247fe 100644
--- a/temporal-sdk/src/main/java/io/temporal/internal/worker/TrackingSlotSupplier.java
+++ b/temporal-sdk/src/main/java/io/temporal/internal/worker/TrackingSlotSupplier.java
@@ -26,7 +26,7 @@
 import java.util.Collections;
 import java.util.Map;
 import java.util.Optional;
-import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.*;
 import java.util.concurrent.atomic.AtomicInteger;
 
 /**
@@ -48,14 +48,20 @@ public TrackingSlotSupplier(SlotSupplier<SI> inner, Scope metricsScope) {
     publishSlotsMetric();
   }
 
-  public SlotPermit reserveSlot(SlotReservationData dat) throws InterruptedException {
-    SlotPermit p = inner.reserveSlot(createCtx(dat));
-    issuedSlots.incrementAndGet();
-    return p;
+  public SlotSupplierFuture reserveSlot(SlotReservationData data) {
+    final SlotSupplierFuture future;
+    try {
+      future = inner.reserveSlot(createCtx(data));
+    } catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+
+    future.thenRun(issuedSlots::incrementAndGet);
+    return future;
   }
 
-  public Optional<SlotPermit> tryReserveSlot(SlotReservationData dat) {
-    Optional<SlotPermit> p = inner.tryReserveSlot(createCtx(dat));
+  public Optional<SlotPermit> tryReserveSlot(SlotReservationData data) {
+    Optional<SlotPermit> p = inner.tryReserveSlot(createCtx(data));
     if (p.isPresent()) {
       issuedSlots.incrementAndGet();
     }
diff --git a/temporal-sdk/src/main/java/io/temporal/internal/worker/WorkflowPollTask.java b/temporal-sdk/src/main/java/io/temporal/internal/worker/WorkflowPollTask.java
index e41cb6d0b5..3746b04df1 100644
--- a/temporal-sdk/src/main/java/io/temporal/internal/worker/WorkflowPollTask.java
+++ b/temporal-sdk/src/main/java/io/temporal/internal/worker/WorkflowPollTask.java
@@ -35,7 +35,10 @@
 import io.temporal.serviceclient.MetricsTag;
 import io.temporal.serviceclient.WorkflowServiceStubs;
 import io.temporal.worker.MetricsType;
-import io.temporal.worker.tuning.*;
+import io.temporal.worker.tuning.SlotPermit;
+import io.temporal.worker.tuning.SlotReleaseReason;
+import io.temporal.worker.tuning.SlotSupplierFuture;
+import io.temporal.worker.tuning.WorkflowSlotInfo;
 import java.util.Objects;
 import java.util.function.Supplier;
 import javax.annotation.Nonnull;
@@ -121,23 +124,24 @@ public WorkflowPollTask(
   @Override
   @SuppressWarnings("deprecation")
   public WorkflowTask poll() {
-    boolean isSuccessful = false;
     SlotPermit permit;
+    SlotSupplierFuture future;
+    boolean isSuccessful = false;
     try {
-      permit =
+      future =
           slotSupplier.reserveSlot(
               new SlotReservationData(
                   pollRequest.getTaskQueue().getName(),
                   pollRequest.getIdentity(),
                   pollRequest.getWorkerVersionCapabilities().getBuildId()));
-    } catch (InterruptedException e) {
-      Thread.currentThread().interrupt();
-      return null;
     } catch (Exception e) {
-      log.warn("Error while trying to reserve a slot for workflow task", e.getCause());
+      log.warn("Error while trying to reserve a slot for a workflow", e.getCause());
       return null;
     }
 
+    permit = Poller.getSlotPermitAndHandleInterrupts(future, slotSupplier);
+    if (permit == null) return null;
+
     TaskQueueKind taskQueueKind = stickyQueueBalancer.makePoll();
     boolean isSticky = TaskQueueKind.TASK_QUEUE_KIND_STICKY.equals(taskQueueKind);
     PollWorkflowTaskQueueRequest request = isSticky ? stickyPollRequest : pollRequest;
diff --git a/temporal-sdk/src/main/java/io/temporal/worker/tuning/FixedSizeSlotSupplier.java b/temporal-sdk/src/main/java/io/temporal/worker/tuning/FixedSizeSlotSupplier.java
index e9b08d3441..8e24e4a627 100644
--- a/temporal-sdk/src/main/java/io/temporal/worker/tuning/FixedSizeSlotSupplier.java
+++ b/temporal-sdk/src/main/java/io/temporal/worker/tuning/FixedSizeSlotSupplier.java
@@ -21,8 +21,11 @@
 package io.temporal.worker.tuning;
 
 import com.google.common.base.Preconditions;
+import java.util.ArrayDeque;
 import java.util.Optional;
-import java.util.concurrent.*;
+import java.util.Queue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.locks.ReentrantLock;
 
 /**
  * This implementation of {@link SlotSupplier} provides a fixed number of slots backed by a
@@ -32,18 +35,89 @@
  */
 public class FixedSizeSlotSupplier<SI extends SlotInfo> implements SlotSupplier<SI> {
   private final int numSlots;
-  private final Semaphore executorSlotsSemaphore;
+  private final AsyncSemaphore executorSlotsSemaphore;
+
+  /**
+   * A simple version of an async semaphore. Unfortunately there's not any readily available
+   * properly licensed library I could find for this which is a bit shocking, but this
+   * implementation should be suitable for our needs
+   */
+  static class AsyncSemaphore {
+    private final ReentrantLock lock = new ReentrantLock();
+    private final Queue<CompletableFuture<Void>> waiters = new ArrayDeque<>();
+    private int permits;
+
+    AsyncSemaphore(int initialPermits) {
+      this.permits = initialPermits;
+    }
+
+    /**
+     * Acquire a permit asynchronously. If a permit is available, returns a completed future,
+     * otherwise returns a future that will be completed when a permit is released.
+     */
+    public CompletableFuture<Void> acquire() {
+      lock.lock();
+      try {
+        if (permits > 0) {
+          permits--;
+          return CompletableFuture.completedFuture(null);
+        } else {
+          CompletableFuture<Void> waiter = new CompletableFuture<>();
+          waiters.add(waiter);
+          return waiter;
+        }
+      } finally {
+        lock.unlock();
+      }
+    }
+
+    public boolean tryAcquire() {
+      lock.lock();
+      try {
+        if (permits > 0) {
+          permits--;
+          return true;
+        }
+        return false;
+      } finally {
+        lock.unlock();
+      }
+    }
+
+    /**
+     * Release a permit. If there are waiting futures, completes the next one instead of
+     * incrementing the permit count.
+     */
+    public void release() {
+      lock.lock();
+      try {
+        CompletableFuture<Void> waiter = waiters.poll();
+        if (waiter != null) {
+          if (!waiter.complete(null) && waiter.isCancelled()) {
+            // If this waiter was cancelled, we need to release another permit, since this waiter
+            // is now useless
+            release();
+          }
+        } else {
+          permits++;
+        }
+      } finally {
+        lock.unlock();
+      }
+    }
+  }
 
   public FixedSizeSlotSupplier(int numSlots) {
     Preconditions.checkArgument(numSlots > 0, "FixedSizeSlotSupplier must have at least one slot");
     this.numSlots = numSlots;
-    executorSlotsSemaphore = new Semaphore(numSlots);
+    executorSlotsSemaphore = new AsyncSemaphore(numSlots);
   }
 
   @Override
-  public SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException {
-    executorSlotsSemaphore.acquire();
-    return new SlotPermit();
+  public SlotSupplierFuture reserveSlot(SlotReserveContext<SI> ctx) throws Exception {
+    CompletableFuture<Void> slotFuture = executorSlotsSemaphore.acquire();
+    return SlotSupplierFuture.fromCompletableFuture(
+        slotFuture.thenApply(ignored -> new SlotPermit()), () -> slotFuture.cancel(true));
   }
 
   @Override
diff --git a/temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java b/temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java
index db671b5867..34891bef75 100644
--- a/temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java
+++ b/temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java
@@ -24,6 +24,7 @@
 import java.time.Duration;
 import java.time.Instant;
 import java.util.Optional;
+import java.util.concurrent.*;
 
 /** Implements a {@link SlotSupplier} based on resource usage for a particular slot type. */
 @Experimental
@@ -32,6 +33,9 @@ public class ResourceBasedSlotSupplier<SI extends SlotInfo> implements SlotSuppl
   private final ResourceBasedController resourceController;
   private final ResourceBasedSlotOptions options;
   private Instant lastSlotIssuedAt = Instant.EPOCH;
+  // For slot reservations that are waiting to re-check resource usage
+  private final ScheduledExecutorService scheduler;
+  private static ScheduledExecutorService defaultScheduler;
 
   /**
    * Construct a slot supplier for workflow tasks with the given resource controller and options.
@@ -42,7 +46,20 @@ public class ResourceBasedSlotSupplier<SI extends SlotInfo> implements SlotSuppl
   public static ResourceBasedSlotSupplier<WorkflowSlotInfo> createForWorkflow(
       ResourceBasedController resourceBasedController, ResourceBasedSlotOptions options) {
     return new ResourceBasedSlotSupplier<>(
-        WorkflowSlotInfo.class, resourceBasedController, options);
+        WorkflowSlotInfo.class, resourceBasedController, options, null);
+  }
+
+  /**
+   * As {@link #createForWorkflow(ResourceBasedController, ResourceBasedSlotOptions)}, but allows
+   * overriding the internal thread pool. It is recommended to share the same executor across all
+   * resource based slot suppliers in a worker.
+   */
+  public static ResourceBasedSlotSupplier<WorkflowSlotInfo> createForWorkflow(
+      ResourceBasedController resourceBasedController,
+      ResourceBasedSlotOptions options,
+      ScheduledExecutorService scheduler) {
+    return new ResourceBasedSlotSupplier<>(
+        WorkflowSlotInfo.class, resourceBasedController, options, scheduler);
   }
 
   /**
@@ -54,7 +71,20 @@ public static ResourceBasedSlotSupplier<WorkflowSlotInfo> createForWorkflow(
   public static ResourceBasedSlotSupplier<ActivitySlotInfo> createForActivity(
       ResourceBasedController resourceBasedController, ResourceBasedSlotOptions options) {
     return new ResourceBasedSlotSupplier<>(
-        ActivitySlotInfo.class, resourceBasedController, options);
+        ActivitySlotInfo.class, resourceBasedController, options, null);
+  }
+
+  /**
+   * As {@link #createForActivity(ResourceBasedController, ResourceBasedSlotOptions)}, but allows
+   * overriding the internal thread pool. It is recommended to share the same executor across all
+   * resource based slot suppliers in a worker.
+   */
+  public static ResourceBasedSlotSupplier<ActivitySlotInfo> createForActivity(
+      ResourceBasedController resourceBasedController,
+      ResourceBasedSlotOptions options,
+      ScheduledExecutorService scheduler) {
+    return new ResourceBasedSlotSupplier<>(
+        ActivitySlotInfo.class, resourceBasedController, options, scheduler);
   }
 
   /**
@@ -66,7 +96,20 @@ public static ResourceBasedSlotSupplier<ActivitySlotInfo> createForActivity(
   public static ResourceBasedSlotSupplier<LocalActivitySlotInfo> createForLocalActivity(
       ResourceBasedController resourceBasedController, ResourceBasedSlotOptions options) {
     return new ResourceBasedSlotSupplier<>(
-        LocalActivitySlotInfo.class, resourceBasedController, options);
+        LocalActivitySlotInfo.class, resourceBasedController, options, null);
+  }
+
+  /**
+   * As {@link #createForLocalActivity(ResourceBasedController, ResourceBasedSlotOptions)}, but
+   * allows overriding the internal thread pool. It is recommended to share the same executor across
+   * all resource based slot suppliers in a worker.
+   */
+  public static ResourceBasedSlotSupplier<LocalActivitySlotInfo> createForLocalActivity(
+      ResourceBasedController resourceBasedController,
+      ResourceBasedSlotOptions options,
+      ScheduledExecutorService scheduler) {
+    return new ResourceBasedSlotSupplier<>(
+        LocalActivitySlotInfo.class, resourceBasedController, options, scheduler);
   }
 
   /**
@@ -77,14 +120,34 @@ public static ResourceBasedSlotSupplier<LocalActivitySlotInfo> createForLocalAct
    */
   public static ResourceBasedSlotSupplier<NexusSlotInfo> createForNexus(
       ResourceBasedController resourceBasedController, ResourceBasedSlotOptions options) {
-    return new ResourceBasedSlotSupplier<>(NexusSlotInfo.class, resourceBasedController, options);
+    return new ResourceBasedSlotSupplier<>(
+        NexusSlotInfo.class, resourceBasedController, options, null);
+  }
+
+  /**
+   * As {@link #createForNexus(ResourceBasedController, ResourceBasedSlotOptions)}, but allows
+   * overriding the internal thread pool. It is recommended to share the same executor across all
+   * resource based slot suppliers in a worker.
+   */
+  public static ResourceBasedSlotSupplier<NexusSlotInfo> createForNexus(
+      ResourceBasedController resourceBasedController,
+      ResourceBasedSlotOptions options,
+      ScheduledExecutorService scheduler) {
+    return new ResourceBasedSlotSupplier<>(
+        NexusSlotInfo.class, resourceBasedController, options, scheduler);
   }
 
   private ResourceBasedSlotSupplier(
       Class<SI> clazz,
       ResourceBasedController resourceBasedController,
-      ResourceBasedSlotOptions options) {
+      ResourceBasedSlotOptions options,
+      ScheduledExecutorService scheduler) {
     this.resourceController = resourceBasedController;
+    if (scheduler == null) {
+      this.scheduler = getDefaultScheduler();
+    } else {
+      this.scheduler = scheduler;
+    }
     // Merge default options for any unset fields
     if (WorkflowSlotInfo.class.isAssignableFrom(clazz)) {
       this.options =
@@ -139,29 +202,45 @@ private ResourceBasedSlotSupplier(
   }
 
   @Override
-  public SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException {
-    while (true) {
-      if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
-        return new SlotPermit();
-      } else {
-        Duration mustWaitFor;
-        try {
-          mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
-        } catch (ArithmeticException e) {
-          mustWaitFor = Duration.ZERO;
-        }
-        if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
-          Thread.sleep(mustWaitFor.toMillis());
-        }
-
-        Optional<SlotPermit> permit = tryReserveSlot(ctx);
-        if (permit.isPresent()) {
-          return permit.get();
-        } else {
-          Thread.sleep(10);
-        }
-      }
+  public SlotSupplierFuture reserveSlot(SlotReserveContext<SI> ctx) throws Exception {
+    if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
+      return SlotSupplierFuture.completedFuture(new SlotPermit());
+    }
+    return tryReserveSlot(ctx)
+        .map(SlotSupplierFuture::completedFuture)
+        .orElseGet(() -> scheduleSlotAcquisition(ctx));
+  }
+
+  private SlotSupplierFuture scheduleSlotAcquisition(SlotReserveContext<SI> ctx) {
+    Duration mustWaitFor;
+    try {
+      mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
+    } catch (ArithmeticException e) {
+      mustWaitFor = Duration.ZERO;
+    }
+
+    CompletableFuture<Void> permitFuture;
+    if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
+      permitFuture =
+          CompletableFuture.supplyAsync(() -> null, delayedExecutor(mustWaitFor.toMillis()));
+    } else {
+      permitFuture = CompletableFuture.completedFuture(null);
     }
+
+    // After the delay, try to reserve the slot
+    return SlotSupplierFuture.fromCompletableFuture(
+        permitFuture.thenCompose(
+            ignored -> {
+              Optional<SlotPermit> permit = tryReserveSlot(ctx);
+              // If we couldn't get a slot this time, delay for a short period and try again
+              return permit
+                  .map(CompletableFuture::completedFuture)
+                  .orElseGet(
+                      () ->
+                          CompletableFuture.supplyAsync(() -> null, delayedExecutor(10))
+                              .thenCompose(ig -> scheduleSlotAcquisition(ctx)));
+            }),
+        () -> permitFuture.cancel(true));
   }
 
   @Override
@@ -190,4 +269,28 @@ public ResourceBasedController getResourceController() {
   private Duration timeSinceLastSlotIssued() {
     return Duration.between(lastSlotIssuedAt, Instant.now());
   }
+
+  // Polyfill for Java 9 delayedExecutor
+  private Executor delayedExecutor(long delay) {
+    return r -> scheduler.schedule(() -> scheduler.execute(r), delay, TimeUnit.MILLISECONDS);
+  }
+
+  private static ScheduledExecutorService getDefaultScheduler() {
+    synchronized (ResourceBasedSlotSupplier.class) {
+      if (defaultScheduler == null) {
+        defaultScheduler =
+            Executors.newScheduledThreadPool(
+                // Two threads seem needed here, so that reading PID decisions doesn't interfere
+                // overly with firing off scheduled tasks or one another.
+                2,
+                r -> {
+                  Thread t = new Thread(r);
+                  t.setName("ResourceBasedSlotSupplier.scheduler");
+                  t.setDaemon(true);
+                  return t;
+                });
+      }
+      return defaultScheduler;
+    }
+  }
 }
diff --git a/temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedTuner.java b/temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedTuner.java
index 77573f93f9..c3e7f7df3c 100644
--- a/temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedTuner.java
+++ b/temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedTuner.java
@@ -22,6 +22,7 @@
 
 import io.temporal.common.Experimental;
 import java.time.Duration;
+import java.util.concurrent.ScheduledExecutorService;
 import javax.annotation.Nonnull;
 
 /** A {@link WorkerTuner} that attempts to allocate slots based on available system resources. */
@@ -51,6 +52,7 @@ public class ResourceBasedTuner implements WorkerTuner {
   private final ResourceBasedSlotOptions activitySlotOptions;
   private final ResourceBasedSlotOptions localActivitySlotOptions;
   private final ResourceBasedSlotOptions nexusSlotOptions;
+  private final ScheduledExecutorService executor;
 
   public static Builder newBuilder() {
     return new Builder();
@@ -63,6 +65,7 @@ public static final class Builder {
     private @Nonnull ResourceBasedSlotOptions localActivitySlotOptions =
         DEFAULT_ACTIVITY_SLOT_OPTIONS;
     private @Nonnull ResourceBasedSlotOptions nexusSlotOptions = DEFAULT_NEXUS_SLOT_OPTIONS;
+    private @Nonnull ScheduledExecutorService executor;
 
     private Builder() {}
 
@@ -115,13 +118,23 @@ public Builder setNexusSlotOptions(@Nonnull ResourceBasedSlotOptions nexusSlotOp
       return this;
     }
 
+    /**
+     * Set the executor used for checking resource usage periodically. Defaults to a two-thread
+     * pool.
+     */
+    public Builder setExecutor(@Nonnull ScheduledExecutorService executor) {
+      this.executor = executor;
+      return this;
+    }
+
     public ResourceBasedTuner build() {
       return new ResourceBasedTuner(
           controllerOptions,
           workflowSlotOptions,
           activitySlotOptions,
           localActivitySlotOptions,
-          nexusSlotOptions);
+          nexusSlotOptions,
+          executor);
     }
   }
 
@@ -133,35 +146,38 @@ public ResourceBasedTuner(
       ResourceBasedSlotOptions workflowSlotOptions,
       ResourceBasedSlotOptions activitySlotOptions,
       ResourceBasedSlotOptions localActivitySlotOptions,
-      ResourceBasedSlotOptions nexusSlotOptions) {
+      ResourceBasedSlotOptions nexusSlotOptions,
+      ScheduledExecutorService executor) {
     this.controller = ResourceBasedController.newSystemInfoController(controllerOptions);
     this.workflowSlotOptions = workflowSlotOptions;
     this.activitySlotOptions = activitySlotOptions;
     this.localActivitySlotOptions = localActivitySlotOptions;
     this.nexusSlotOptions = nexusSlotOptions;
+    this.executor = executor;
   }
 
   @Nonnull
   @Override
   public SlotSupplier<WorkflowSlotInfo> getWorkflowTaskSlotSupplier() {
-    return ResourceBasedSlotSupplier.createForWorkflow(controller, workflowSlotOptions);
+    return ResourceBasedSlotSupplier.createForWorkflow(controller, workflowSlotOptions, executor);
   }
 
   @Nonnull
   @Override
   public SlotSupplier<ActivitySlotInfo> getActivityTaskSlotSupplier() {
-    return ResourceBasedSlotSupplier.createForActivity(controller, activitySlotOptions);
+    return ResourceBasedSlotSupplier.createForActivity(controller, activitySlotOptions, executor);
   }
 
   @Nonnull
   @Override
   public SlotSupplier<LocalActivitySlotInfo> getLocalActivitySlotSupplier() {
-    return ResourceBasedSlotSupplier.createForLocalActivity(controller, localActivitySlotOptions);
+    return ResourceBasedSlotSupplier.createForLocalActivity(
+        controller, localActivitySlotOptions, executor);
   }
 
   @Nonnull
   @Override
   public SlotSupplier<NexusSlotInfo> getNexusSlotSupplier() {
-    return ResourceBasedSlotSupplier.createForNexus(controller, nexusSlotOptions);
+    return ResourceBasedSlotSupplier.createForNexus(controller, nexusSlotOptions, executor);
   }
 }
diff --git a/temporal-sdk/src/main/java/io/temporal/worker/tuning/SlotSupplier.java b/temporal-sdk/src/main/java/io/temporal/worker/tuning/SlotSupplier.java
index fa584d443e..226d5aea51 100644
--- a/temporal-sdk/src/main/java/io/temporal/worker/tuning/SlotSupplier.java
+++ b/temporal-sdk/src/main/java/io/temporal/worker/tuning/SlotSupplier.java
@@ -36,16 +36,19 @@
 @Experimental
 public interface SlotSupplier<SI extends SlotInfo> {
   /**
-   * This function is called before polling for new tasks. Your implementation should block until a
-   * slot is available then return a permit to use that slot.
+   * This function is called before polling for new tasks. Your implementation should return a
+   * future that is completed with a {@link SlotPermit} when one becomes available.
+   *
+   * <p>These futures may be cancelled if the worker is shutting down or otherwise abandons the
+   * reservation. This can cause an {@link InterruptedException} to be thrown, in the thread running
+   * your implementation. You may want to catch it to perform any necessary cleanup, and then you
+   * should rethrow the exception. Other thrown exceptions will be logged.
    *
    * @param ctx The context for slot reservation.
-   * @return A permit to use the slot which may be populated with your own data.
-   * @throws InterruptedException The worker may choose to interrupt the thread in order to cancel
-   *     the reservation, or during shutdown. You may perform cleanup, and then should rethrow the
-   *     exception.
+   * @return A future that will be completed with a permit to use the slot when one becomes
+   *     available. Never return null, or complete the future with null.
    */
-  SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException;
+  SlotSupplierFuture reserveSlot(SlotReserveContext<SI> ctx) throws Exception;
 
   /**
    * This function is called when trying to reserve slots for "eager" workflow and activity tasks.
diff --git a/temporal-sdk/src/main/java/io/temporal/worker/tuning/SlotSupplierFuture.java b/temporal-sdk/src/main/java/io/temporal/worker/tuning/SlotSupplierFuture.java
new file mode 100644
index 0000000000..b42a6f6f48
--- /dev/null
+++ b/temporal-sdk/src/main/java/io/temporal/worker/tuning/SlotSupplierFuture.java
@@ -0,0 +1,106 @@
+/*
+ * Copyright (C) 2022 Temporal Technologies, Inc. All Rights Reserved.
+ *
+ * Copyright (C) 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Modifications copyright (C) 2017 Uber Technologies, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this material except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.temporal.worker.tuning;
+
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CompletableFuture;
+import javax.annotation.CheckReturnValue;
+import javax.annotation.Nullable;
+
+/**
+ * Represents a future that will be completed with a {@link SlotPermit} when a slot is available.
+ *
+ * <p>This class exists to provide a reliable cancellation mechanism, since {@link
+ * CompletableFuture} does not provide cancellations that properly propagate up the chain.
+ */
+public abstract class SlotSupplierFuture extends CompletableFuture<SlotPermit> {
+  /**
+   * Abort the reservation attempt. Direct implementations should cancel or interrupt any underlying
+   * processes that are attempting to reserve a slot.
+   */
+  @Nullable
+  @CheckReturnValue
+  public abstract SlotPermit abortReservation();
+
+  @Override
+  public boolean cancel(boolean mayInterruptIfRunning) {
+    throw new UnsupportedOperationException(
+        "Do not call cancel on SlotSupplierFuture, use abortReservation");
+  }
+
+  /** See {@link CompletableFuture#completedFuture(Object)} */
+  public static SlotSupplierFuture completedFuture(SlotPermit permit) {
+    return new SlotSupplierFuture() {
+      @Override
+      public SlotPermit abortReservation() {
+        return permit;
+      }
+
+      {
+        complete(permit);
+      }
+    };
+  }
+
+  /**
+   * Create a new {@link SlotSupplierFuture} from a {@link CompletableFuture}
+   *
+   * @param abortHandler The handler to call when the reservation is aborted. This should abort the
+   *     furthest-upstream future, or call being waited on, in order to properly propagate
+   *     cancellation downstream.
+   */
+  public static SlotSupplierFuture fromCompletableFuture(
+      CompletableFuture<SlotPermit> future, Runnable abortHandler) {
+    SlotSupplierFuture wrapper =
+        new SlotSupplierFuture() {
+          @Override
+          public SlotPermit abortReservation() {
+            // Try to force the future into an exceptional state.
+            // completeExceptionally returns true only if it successfully transitions.
+            boolean abortedNow =
+                this.completeExceptionally(new CancellationException("Reservation aborted"));
+            if (abortedNow) {
+              abortHandler.run();
+              future.cancel(true);
+              return null;
+            } else {
+              // The future has already completed normally so return the permit.
+              return this.join();
+            }
+          }
+        };
+
+    // Propagate the delegate future’s outcome to our wrapper
+    future.whenComplete(
+        (result, throwable) -> {
+          // If our wrapper isn’t already completed (via abortReservation), complete it.
+          if (!wrapper.isDone()) {
+            if (throwable != null) {
+              wrapper.completeExceptionally(throwable);
+            } else {
+              wrapper.complete(result);
+            }
+          }
+        });
+
+    return wrapper;
+  }
+}
diff --git a/temporal-sdk/src/test/java/io/temporal/internal/worker/SlotSupplierTest.java b/temporal-sdk/src/test/java/io/temporal/internal/worker/SlotSupplierTest.java
index 8a393bd7ed..1cdf8daf50 100644
--- a/temporal-sdk/src/test/java/io/temporal/internal/worker/SlotSupplierTest.java
+++ b/temporal-sdk/src/test/java/io/temporal/internal/worker/SlotSupplierTest.java
@@ -37,7 +37,6 @@
 import io.temporal.serviceclient.WorkflowServiceStubs;
 import io.temporal.worker.tuning.*;
 import java.util.Objects;
-import java.util.concurrent.TimeoutException;
 import java.util.concurrent.atomic.AtomicInteger;
 import org.junit.Test;
 import org.junit.runner.RunWith;
@@ -60,7 +59,7 @@ public static Object[] data() {
   }
 
   @Test
-  public void supplierIsCalledAppropriately() throws InterruptedException, TimeoutException {
+  public void supplierIsCalledAppropriately() {
     WorkflowServiceStubs client = mock(WorkflowServiceStubs.class);
     when(client.getServerCapabilities())
         .thenReturn(() -> GetSystemInfoResponse.Capabilities.newBuilder().build());
@@ -71,13 +70,17 @@ public void supplierIsCalledAppropriately() throws InterruptedException, Timeout
 
     SlotSupplier<WorkflowSlotInfo> mockSupplier = mock(SlotSupplier.class);
     AtomicInteger usedSlotsWhenCalled = new AtomicInteger(-1);
-    when(mockSupplier.reserveSlot(
-            argThat(
-                src -> {
-                  usedSlotsWhenCalled.set(src.getUsedSlots().size());
-                  return true;
-                })))
-        .thenReturn(new SlotPermit());
+    try {
+      when(mockSupplier.reserveSlot(
+              argThat(
+                  src -> {
+                    usedSlotsWhenCalled.set(src.getUsedSlots().size());
+                    return true;
+                  })))
+          .thenReturn(SlotSupplierFuture.completedFuture(new SlotPermit()));
+    } catch (Exception e) {
+      throw new RuntimeException(e);
+    }
 
     StickyQueueBalancer stickyQueueBalancer = new StickyQueueBalancer(5, true);
     Scope metricsScope =
@@ -119,7 +122,11 @@ public void supplierIsCalledAppropriately() throws InterruptedException, Timeout
 
     if (throwOnPoll) {
       assertThrows(RuntimeException.class, poller::poll);
-      verify(mockSupplier, times(1)).reserveSlot(any());
+      try {
+        verify(mockSupplier, times(1)).reserveSlot(any());
+      } catch (Exception e) {
+        throw new RuntimeException(e);
+      }
       verify(mockSupplier, times(1)).releaseSlot(any());
       assertEquals(0, trackingSS.getUsedSlots().size());
     } else {
@@ -128,8 +135,12 @@ public void supplierIsCalledAppropriately() throws InterruptedException, Timeout
       // We can't test this in the verifier, since it will get an up-to-date reference to the map
       // where the slot *is* used.
       assertEquals(0, usedSlotsWhenCalled.get());
-      verify(mockSupplier, times(1))
-          .reserveSlot(argThat(arg -> Objects.equals(arg.getTaskQueue(), TASK_QUEUE)));
+      try {
+        verify(mockSupplier, times(1))
+            .reserveSlot(argThat(arg -> Objects.equals(arg.getTaskQueue(), TASK_QUEUE)));
+      } catch (Exception e) {
+        throw new RuntimeException(e);
+      }
       verify(mockSupplier, times(0)).releaseSlot(any());
       assertEquals(1, trackingSS.getUsedSlots().size());
     }
diff --git a/temporal-sdk/src/test/java/io/temporal/internal/worker/WorkflowSlotsSmallSizeTests.java b/temporal-sdk/src/test/java/io/temporal/internal/worker/WorkflowSlotsSmallSizeTests.java
index 7020e37b3a..b4bec7654c 100644
--- a/temporal-sdk/src/test/java/io/temporal/internal/worker/WorkflowSlotsSmallSizeTests.java
+++ b/temporal-sdk/src/test/java/io/temporal/internal/worker/WorkflowSlotsSmallSizeTests.java
@@ -22,7 +22,6 @@
 
 import static org.junit.Assert.assertEquals;
 
-import com.uber.m3.util.ImmutableMap;
 import io.temporal.activity.ActivityInterface;
 import io.temporal.activity.ActivityMethod;
 import io.temporal.activity.ActivityOptions;
@@ -38,7 +37,6 @@
 import java.time.Duration;
 import java.util.ArrayList;
 import java.util.List;
-import java.util.Map;
 import java.util.concurrent.Semaphore;
 import java.util.concurrent.TimeUnit;
 import org.junit.After;
@@ -181,20 +179,9 @@ public String activity(String input) {
     }
   }
 
-  private Map<String, String> getWorkerTags(String workerType) {
-    return ImmutableMap.of(
-        "worker_type",
-        workerType,
-        "task_queue",
-        testWorkflowRule.getTaskQueue(),
-        "namespace",
-        "UnitTest");
-  }
-
   private void assertIntraWFTSlotCount(int allowedToRun) {
     int runningLAs = activitiesAreLocal ? allowedToRun : 0;
     int runningAs = activitiesAreLocal ? 0 : allowedToRun;
-    int runningWFTs = activitiesAreLocal ? 1 : 0;
     assertCurrentUsedCount(runningAs, runningLAs);
   }
 
diff --git a/temporal-sdk/src/test/java/io/temporal/testUtils/CountingSlotSupplier.java b/temporal-sdk/src/test/java/io/temporal/testUtils/CountingSlotSupplier.java
index a9d0045e05..eed0e9c85a 100644
--- a/temporal-sdk/src/test/java/io/temporal/testUtils/CountingSlotSupplier.java
+++ b/temporal-sdk/src/test/java/io/temporal/testUtils/CountingSlotSupplier.java
@@ -22,7 +22,7 @@
 
 import io.temporal.worker.tuning.*;
 import java.util.Optional;
-import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.*;
 import java.util.concurrent.atomic.AtomicInteger;
 
 public class CountingSlotSupplier<SI extends SlotInfo> extends FixedSizeSlotSupplier<SI> {
@@ -37,9 +37,9 @@ public CountingSlotSupplier(int numSlots) {
   }
 
   @Override
-  public SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException {
-    SlotPermit p = super.reserveSlot(ctx);
-    reservedCount.incrementAndGet();
+  public SlotSupplierFuture reserveSlot(SlotReserveContext<SI> ctx) throws Exception {
+    SlotSupplierFuture p = super.reserveSlot(ctx);
+    p.thenRun(reservedCount::incrementAndGet);
     return p;
   }
 
diff --git a/temporal-sdk/src/test/java/io/temporal/worker/ResourceBasedTunerTests.java b/temporal-sdk/src/test/java/io/temporal/worker/ResourceBasedTunerTests.java
index ca61f73713..0e65065e38 100644
--- a/temporal-sdk/src/test/java/io/temporal/worker/ResourceBasedTunerTests.java
+++ b/temporal-sdk/src/test/java/io/temporal/worker/ResourceBasedTunerTests.java
@@ -27,6 +27,8 @@
 import io.temporal.activity.ActivityInterface;
 import io.temporal.activity.ActivityOptions;
 import io.temporal.activity.LocalActivityOptions;
+import io.temporal.client.WorkflowClient;
+import io.temporal.client.WorkflowOptions;
 import io.temporal.common.reporter.TestStatsReporter;
 import io.temporal.serviceclient.MetricsTag;
 import io.temporal.testing.internal.SDKTestWorkflowRule;
@@ -36,6 +38,7 @@
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
+import java.util.concurrent.TimeUnit;
 import org.junit.Rule;
 import org.junit.Test;
 import org.junit.experimental.categories.Category;
@@ -99,13 +102,37 @@ public void canRunHeavyMemoryWithResourceBasedTuner() {
     workflow.execute(50, 50, 30000000);
   }
 
+  @Test(timeout = 30 * 1000)
+  public void canShutdownInTheMiddle() throws InterruptedException {
+    WorkflowClient client = testWorkflowRule.getWorkflowClient();
+    ResourceTunerWorkflow workflow =
+        client.newWorkflowStub(
+            ResourceTunerWorkflow.class,
+            WorkflowOptions.newBuilder()
+                .setTaskQueue(testWorkflowRule.getTaskQueue())
+                .validateBuildWithDefaults());
+    WorkflowClient.start(workflow::execute, 10, 10, 1000);
+    workflow.activitiesStarted();
+    testWorkflowRule.getTestEnvironment().getWorkerFactory().shutdownNow();
+    testWorkflowRule.getTestEnvironment().getWorkerFactory().awaitTermination(3, TimeUnit.SECONDS);
+    reporter.assertGauge(MetricsType.WORKER_TASK_SLOTS_USED, getWorkerTags("WorkflowWorker"), 0);
+    reporter.assertGauge(MetricsType.WORKER_TASK_SLOTS_USED, getWorkerTags("ActivityWorker"), 0);
+    reporter.assertGauge(
+        MetricsType.WORKER_TASK_SLOTS_USED, getWorkerTags("LocalActivityWorker"), 0);
+  }
+
   @WorkflowInterface
   public interface ResourceTunerWorkflow {
     @WorkflowMethod
     String execute(int numActivities, int localActivities, int memCeiling);
+
+    @UpdateMethod
+    void activitiesStarted();
   }
 
   public static class ResourceTunerWorkflowImpl implements ResourceTunerWorkflow {
+    private boolean activitiesStarted = false;
+
     @Override
     public String execute(int numActivities, int localActivities, int memCeiling) {
       SleepActivity activity =
@@ -133,12 +160,19 @@ public String execute(int numActivities, int localActivities, int memCeiling) {
         promises.add(promise);
       }
 
+      activitiesStarted = true;
+
       for (Promise<Void> promise : promises) {
         promise.get();
       }
 
       return "I'm done";
     }
+
+    @Override
+    public void activitiesStarted() {
+      Workflow.await(() -> activitiesStarted);
+    }
   }
 
   @ActivityInterface
diff --git a/temporal-sdk/src/test/java/io/temporal/worker/tuning/FixedSizeSlotSupplierTest.java b/temporal-sdk/src/test/java/io/temporal/worker/tuning/FixedSizeSlotSupplierTest.java
new file mode 100644
index 0000000000..711dde45dd
--- /dev/null
+++ b/temporal-sdk/src/test/java/io/temporal/worker/tuning/FixedSizeSlotSupplierTest.java
@@ -0,0 +1,56 @@
+/*
+ * Copyright (C) 2022 Temporal Technologies, Inc. All Rights Reserved.
+ *
+ * Copyright (C) 2012-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Modifications copyright (C) 2017 Uber Technologies, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this material except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.temporal.worker.tuning;
+
+import static org.junit.Assert.*;
+
+import com.uber.m3.tally.NoopScope;
+import io.temporal.internal.worker.SlotReservationData;
+import io.temporal.internal.worker.TrackingSlotSupplier;
+import org.junit.Test;
+
+public class FixedSizeSlotSupplierTest {
+
+  @Test
+  public void ensureInterruptingReservationWorksWhenWaitingOnSemaphoreInQueue() throws Exception {
+    FixedSizeSlotSupplier<WorkflowSlotInfo> supplier = new FixedSizeSlotSupplier<>(1);
+    TrackingSlotSupplier<WorkflowSlotInfo> trackingSS =
+        new TrackingSlotSupplier<>(supplier, new NoopScope());
+    // Reserve one slot and don't release
+    SlotPermit firstPermit =
+        trackingSS.reserveSlot(new SlotReservationData("bla", "blah", "bla")).get();
+
+    // Try to reserve another slot
+    SlotSupplierFuture secondSlotFuture =
+        trackingSS.reserveSlot(new SlotReservationData("bla", "blah", "bla"));
+    // Try to reserve a third
+    SlotSupplierFuture thirdSlotFuture =
+        trackingSS.reserveSlot(new SlotReservationData("bla", "blah", "bla"));
+
+    // Cancel second reservation & release first permit, which should allow third to be acquired
+    SlotPermit maybePermit = secondSlotFuture.abortReservation();
+    assertNull(maybePermit);
+    trackingSS.releaseSlot(SlotReleaseReason.neverUsed(), firstPermit);
+
+    SlotPermit p = thirdSlotFuture.get();
+    assertNotNull(p);
+  }
+}


Reviews:
user: login: cretz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T14:21:04Z
user: login: Sushisource
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-28T19:14:42Z
user: login: Sushisource
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-28T19:15:45Z
user: login: cretz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T19:42:56Z
user: login: cretz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T19:44:48Z
user: login: Sushisource
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-28T21:42:52Z
user: login: Quinn-With-Two-Ns
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T05:59:04Z
user: login: Quinn-With-Two-Ns
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T06:00:34Z
user: login: Quinn-With-Two-Ns
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T06:02:53Z
user: login: Quinn-With-Two-Ns
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T06:07:29Z
user: login: cretz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T11:55:43Z
user: login: cretz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T11:58:17Z
user: login: Quinn-With-Two-Ns
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T14:45:26Z
user: login: Sushisource
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T16:57:07Z
user: login: cretz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T17:11:39Z
user: login: Sushisource
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T17:26:14Z
user: login: cretz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T19:36:56Z
user: login: cretz
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T19:37:37Z
user: login: Quinn-With-Two-Ns
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T18:43:31Z
user: login: Quinn-With-Two-Ns
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T18:43:58Z
user: login: Quinn-With-Two-Ns
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T05:48:56Z
user: login: Quinn-With-Two-Ns
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T05:50:22Z

Review comments:
diff_hunk: @@ -36,16 +37,19 @@
 @Experimental
 public interface SlotSupplier<SI extends SlotInfo> {
   /**
-   * This function is called before polling for new tasks. Your implementation should block until a
-   * slot is available then return a permit to use that slot.
+   * This function is called before polling for new tasks. Your implementation should return a
+   * Promise that is completed with a {@link SlotPermit} when one becomes available.
+   *
+   * <p>These futures may be cancelled if the worker is shutting down or otherwise abandons the
+   * reservation. This can cause an {@link InterruptedException} to be thrown, in the thread running
+   * your implementation. You may want to catch it to perform any necessary cleanup, and then you
+   * should rethrow the exception.
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/SlotSupplier.java
user: login: cretz
body: > and then you should rethrow the exception

They can't rethrow without wrapping because the checked exception was removed from this interface. We should add back `throws InterruptedException`, though could consider `throws Exception` as a way of saying we have code to handle any exception.
created_at: 2025-02-28T14:00:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -32,6 +33,18 @@ public class ResourceBasedSlotSupplier<SI extends SlotInfo> implements SlotSuppl
   private final ResourceBasedController resourceController;
   private final ResourceBasedSlotOptions options;
   private Instant lastSlotIssuedAt = Instant.EPOCH;
+  // For slot reservations that are waiting to re-check resource usage
+  private static final ScheduledExecutorService scheduler =
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java
user: login: cretz
body: I think you should consider accepting this in the constructor. Can have a default, but we should allow people to control thread creation if they want.
created_at: 2025-02-28T14:10:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -139,29 +152,43 @@ private ResourceBasedSlotSupplier(
   }
 
   @Override
-  public SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException {
-    while (true) {
-      if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
-        return new SlotPermit();
-      } else {
-        Duration mustWaitFor;
-        try {
-          mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
-        } catch (ArithmeticException e) {
-          mustWaitFor = Duration.ZERO;
-        }
-        if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
-          Thread.sleep(mustWaitFor.toMillis());
-        }
-
-        Optional<SlotPermit> permit = tryReserveSlot(ctx);
-        if (permit.isPresent()) {
-          return permit.get();
-        } else {
-          Thread.sleep(10);
-        }
-      }
+  public CompletableFuture<SlotPermit> reserveSlot(SlotReserveContext<SI> ctx) {
+    if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
+      return CompletableFuture.completedFuture(new SlotPermit());
     }
+    return tryReserveSlot(ctx)
+        .map(CompletableFuture::completedFuture)
+        .orElseGet(() -> scheduleSlotAcquisition(ctx));
+  }
+
+  private CompletableFuture<SlotPermit> scheduleSlotAcquisition(SlotReserveContext<SI> ctx) {
+    Duration mustWaitFor;
+    try {
+      mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
+    } catch (ArithmeticException e) {
+      mustWaitFor = Duration.ZERO;
+    }
+
+    CompletableFuture<Void> permitFuture;
+    if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
+      permitFuture =
+          CompletableFuture.supplyAsync(() -> null, delayedExecutor(mustWaitFor.toMillis()));
+    } else {
+      permitFuture = CompletableFuture.completedFuture(null);
+    }
+
+    // After the delay, try to reserve the slot
+    return permitFuture.thenCompose(
+        ignored -> {
+          Optional<SlotPermit> permit = tryReserveSlot(ctx);
+          // If we couldn't get a slot this time, delay for a short period and try again
+          return permit
+              .map(CompletableFuture::completedFuture)
+              .orElseGet(
+                  () ->
+                      CompletableFuture.supplyAsync(() -> null, delayedExecutor(10))
+                          .thenCompose(ig -> scheduleSlotAcquisition(ctx)));
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java
user: login: cretz
body: ChatGPT is telling me that cancellation does not propagate across `orElseGet` or `thenCompose` operators, but that doesn't seem right and I can't find this in the docs at https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html. May need a test somehow confirming cancel _after_ this `orElseGet` does actually prevent the delayed call from being invoked. If it doesn't work, I can make some suggestions I think.
created_at: 2025-02-28T14:17:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -139,29 +152,43 @@ private ResourceBasedSlotSupplier(
   }
 
   @Override
-  public SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException {
-    while (true) {
-      if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
-        return new SlotPermit();
-      } else {
-        Duration mustWaitFor;
-        try {
-          mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
-        } catch (ArithmeticException e) {
-          mustWaitFor = Duration.ZERO;
-        }
-        if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
-          Thread.sleep(mustWaitFor.toMillis());
-        }
-
-        Optional<SlotPermit> permit = tryReserveSlot(ctx);
-        if (permit.isPresent()) {
-          return permit.get();
-        } else {
-          Thread.sleep(10);
-        }
-      }
+  public CompletableFuture<SlotPermit> reserveSlot(SlotReserveContext<SI> ctx) {
+    if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
+      return CompletableFuture.completedFuture(new SlotPermit());
     }
+    return tryReserveSlot(ctx)
+        .map(CompletableFuture::completedFuture)
+        .orElseGet(() -> scheduleSlotAcquisition(ctx));
+  }
+
+  private CompletableFuture<SlotPermit> scheduleSlotAcquisition(SlotReserveContext<SI> ctx) {
+    Duration mustWaitFor;
+    try {
+      mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
+    } catch (ArithmeticException e) {
+      mustWaitFor = Duration.ZERO;
+    }
+
+    CompletableFuture<Void> permitFuture;
+    if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
+      permitFuture =
+          CompletableFuture.supplyAsync(() -> null, delayedExecutor(mustWaitFor.toMillis()));
+    } else {
+      permitFuture = CompletableFuture.completedFuture(null);
+    }
+
+    // After the delay, try to reserve the slot
+    return permitFuture.thenCompose(
+        ignored -> {
+          Optional<SlotPermit> permit = tryReserveSlot(ctx);
+          // If we couldn't get a slot this time, delay for a short period and try again
+          return permit
+              .map(CompletableFuture::completedFuture)
+              .orElseGet(
+                  () ->
+                      CompletableFuture.supplyAsync(() -> null, delayedExecutor(10))
+                          .thenCompose(ig -> scheduleSlotAcquisition(ctx)));
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java
user: login: Sushisource
body: I think this works fine because the returned future is `permitFuture` which is `thenCompose`d with the future from here. Added a test that confirms.
created_at: 2025-02-28T19:14:42Z
author_association: MEMBER
diff_hunk: @@ -36,16 +37,19 @@
 @Experimental
 public interface SlotSupplier<SI extends SlotInfo> {
   /**
-   * This function is called before polling for new tasks. Your implementation should block until a
-   * slot is available then return a permit to use that slot.
+   * This function is called before polling for new tasks. Your implementation should return a
+   * Promise that is completed with a {@link SlotPermit} when one becomes available.
+   *
+   * <p>These futures may be cancelled if the worker is shutting down or otherwise abandons the
+   * reservation. This can cause an {@link InterruptedException} to be thrown, in the thread running
+   * your implementation. You may want to catch it to perform any necessary cleanup, and then you
+   * should rethrow the exception.
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/SlotSupplier.java
user: login: Sushisource
body: Added Exception
created_at: 2025-02-28T19:15:45Z
author_association: MEMBER
diff_hunk: @@ -139,29 +152,43 @@ private ResourceBasedSlotSupplier(
   }
 
   @Override
-  public SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException {
-    while (true) {
-      if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
-        return new SlotPermit();
-      } else {
-        Duration mustWaitFor;
-        try {
-          mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
-        } catch (ArithmeticException e) {
-          mustWaitFor = Duration.ZERO;
-        }
-        if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
-          Thread.sleep(mustWaitFor.toMillis());
-        }
-
-        Optional<SlotPermit> permit = tryReserveSlot(ctx);
-        if (permit.isPresent()) {
-          return permit.get();
-        } else {
-          Thread.sleep(10);
-        }
-      }
+  public CompletableFuture<SlotPermit> reserveSlot(SlotReserveContext<SI> ctx) {
+    if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
+      return CompletableFuture.completedFuture(new SlotPermit());
     }
+    return tryReserveSlot(ctx)
+        .map(CompletableFuture::completedFuture)
+        .orElseGet(() -> scheduleSlotAcquisition(ctx));
+  }
+
+  private CompletableFuture<SlotPermit> scheduleSlotAcquisition(SlotReserveContext<SI> ctx) {
+    Duration mustWaitFor;
+    try {
+      mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
+    } catch (ArithmeticException e) {
+      mustWaitFor = Duration.ZERO;
+    }
+
+    CompletableFuture<Void> permitFuture;
+    if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
+      permitFuture =
+          CompletableFuture.supplyAsync(() -> null, delayedExecutor(mustWaitFor.toMillis()));
+    } else {
+      permitFuture = CompletableFuture.completedFuture(null);
+    }
+
+    // After the delay, try to reserve the slot
+    return permitFuture.thenCompose(
+        ignored -> {
+          Optional<SlotPermit> permit = tryReserveSlot(ctx);
+          // If we couldn't get a slot this time, delay for a short period and try again
+          return permit
+              .map(CompletableFuture::completedFuture)
+              .orElseGet(
+                  () ->
+                      CompletableFuture.supplyAsync(() -> null, delayedExecutor(10))
+                          .thenCompose(ig -> scheduleSlotAcquisition(ctx)));
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java
user: login: cretz
body: My concern is that ChatGPT says that a cancel of the outer future does not propagate the cancel to the `thenCompose`d future. Not sure I believe it because I haven't tested, but it does concern me. I would have to test Java behavior of cancel when the `thenCompose` has already run to create a new future. I may set aside some time to do this.

But I fear reading https://stackoverflow.com/questions/25417881/canceling-a-completablefuture-chain and poking around, there may need to be some other mechanism to make sure the outer completable future cancel propagates to cancelling the delayed executor. I couldn't tell if the test was doing that. I wouldn't be surprised if cancel is not hierarchical.
created_at: 2025-02-28T19:42:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,16 +37,19 @@
 @Experimental
 public interface SlotSupplier<SI extends SlotInfo> {
   /**
-   * This function is called before polling for new tasks. Your implementation should block until a
-   * slot is available then return a permit to use that slot.
+   * This function is called before polling for new tasks. Your implementation should return a
+   * Promise that is completed with a {@link SlotPermit} when one becomes available.
+   *
+   * <p>These futures may be cancelled if the worker is shutting down or otherwise abandons the
+   * reservation. This can cause an {@link InterruptedException} to be thrown, in the thread running
+   * your implementation. You may want to catch it to perform any necessary cleanup, and then you
+   * should rethrow the exception. Other thrown exceptions will be logged.
    *
    * @param ctx The context for slot reservation.
-   * @return A permit to use the slot which may be populated with your own data.
-   * @throws InterruptedException The worker may choose to interrupt the thread in order to cancel
-   *     the reservation, or during shutdown. You may perform cleanup, and then should rethrow the
-   *     exception.
+   * @return A future that will be completed with a permit to use the slot when one becomes
+   *     available. Never return null, or complete the future with null.
    */
-  SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException;
+  CompletableFuture<SlotPermit> reserveSlot(SlotReserveContext<SI> ctx) throws Exception;
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/SlotSupplier.java
user: login: cretz
body: ```suggestion
  Future<SlotPermit> reserveSlot(SlotReserveContext<SI> ctx) throws Exception;
```

I wonder if this is more flexible. That some implementations happen to use `CompletableFuture` doesn't affect us right? Or do we need `CompletableFuture` utilities besides tests?
created_at: 2025-02-28T19:44:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,16 +37,19 @@
 @Experimental
 public interface SlotSupplier<SI extends SlotInfo> {
   /**
-   * This function is called before polling for new tasks. Your implementation should block until a
-   * slot is available then return a permit to use that slot.
+   * This function is called before polling for new tasks. Your implementation should return a
+   * Promise that is completed with a {@link SlotPermit} when one becomes available.
+   *
+   * <p>These futures may be cancelled if the worker is shutting down or otherwise abandons the
+   * reservation. This can cause an {@link InterruptedException} to be thrown, in the thread running
+   * your implementation. You may want to catch it to perform any necessary cleanup, and then you
+   * should rethrow the exception. Other thrown exceptions will be logged.
    *
    * @param ctx The context for slot reservation.
-   * @return A permit to use the slot which may be populated with your own data.
-   * @throws InterruptedException The worker may choose to interrupt the thread in order to cancel
-   *     the reservation, or during shutdown. You may perform cleanup, and then should rethrow the
-   *     exception.
+   * @return A future that will be completed with a permit to use the slot when one becomes
+   *     available. Never return null, or complete the future with null.
    */
-  SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException;
+  CompletableFuture<SlotPermit> reserveSlot(SlotReserveContext<SI> ctx) throws Exception;
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/SlotSupplier.java
user: login: Sushisource
body: I feel like it's less flexible. You can't compose raw Futures or really do much of anything with them besides `get` them
created_at: 2025-02-28T21:42:52Z
author_association: MEMBER
diff_hunk: @@ -36,16 +37,19 @@
 @Experimental
 public interface SlotSupplier<SI extends SlotInfo> {
   /**
-   * This function is called before polling for new tasks. Your implementation should block until a
-   * slot is available then return a permit to use that slot.
+   * This function is called before polling for new tasks. Your implementation should return a
+   * Promise that is completed with a {@link SlotPermit} when one becomes available.
+   *
+   * <p>These futures may be cancelled if the worker is shutting down or otherwise abandons the
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/SlotSupplier.java
user: login: Quinn-With-Two-Ns
body: >These futures may be cancelled if the worker is shutting down or otherwise abandons the

You are aware `CompletableFuture` cancellation does not propagate upstream correct? 
created_at: 2025-03-04T05:59:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -139,29 +152,43 @@ private ResourceBasedSlotSupplier(
   }
 
   @Override
-  public SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException {
-    while (true) {
-      if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
-        return new SlotPermit();
-      } else {
-        Duration mustWaitFor;
-        try {
-          mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
-        } catch (ArithmeticException e) {
-          mustWaitFor = Duration.ZERO;
-        }
-        if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
-          Thread.sleep(mustWaitFor.toMillis());
-        }
-
-        Optional<SlotPermit> permit = tryReserveSlot(ctx);
-        if (permit.isPresent()) {
-          return permit.get();
-        } else {
-          Thread.sleep(10);
-        }
-      }
+  public CompletableFuture<SlotPermit> reserveSlot(SlotReserveContext<SI> ctx) {
+    if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
+      return CompletableFuture.completedFuture(new SlotPermit());
     }
+    return tryReserveSlot(ctx)
+        .map(CompletableFuture::completedFuture)
+        .orElseGet(() -> scheduleSlotAcquisition(ctx));
+  }
+
+  private CompletableFuture<SlotPermit> scheduleSlotAcquisition(SlotReserveContext<SI> ctx) {
+    Duration mustWaitFor;
+    try {
+      mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
+    } catch (ArithmeticException e) {
+      mustWaitFor = Duration.ZERO;
+    }
+
+    CompletableFuture<Void> permitFuture;
+    if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
+      permitFuture =
+          CompletableFuture.supplyAsync(() -> null, delayedExecutor(mustWaitFor.toMillis()));
+    } else {
+      permitFuture = CompletableFuture.completedFuture(null);
+    }
+
+    // After the delay, try to reserve the slot
+    return permitFuture.thenCompose(
+        ignored -> {
+          Optional<SlotPermit> permit = tryReserveSlot(ctx);
+          // If we couldn't get a slot this time, delay for a short period and try again
+          return permit
+              .map(CompletableFuture::completedFuture)
+              .orElseGet(
+                  () ->
+                      CompletableFuture.supplyAsync(() -> null, delayedExecutor(10))
+                          .thenCompose(ig -> scheduleSlotAcquisition(ctx)));
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java
user: login: Quinn-With-Two-Ns
body: Canceling a completable future is not a good way to tell the producer the result is no longer needed, it is to tell the consumer.
created_at: 2025-03-04T06:00:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -139,29 +152,43 @@ private ResourceBasedSlotSupplier(
   }
 
   @Override
-  public SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException {
-    while (true) {
-      if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
-        return new SlotPermit();
-      } else {
-        Duration mustWaitFor;
-        try {
-          mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
-        } catch (ArithmeticException e) {
-          mustWaitFor = Duration.ZERO;
-        }
-        if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
-          Thread.sleep(mustWaitFor.toMillis());
-        }
-
-        Optional<SlotPermit> permit = tryReserveSlot(ctx);
-        if (permit.isPresent()) {
-          return permit.get();
-        } else {
-          Thread.sleep(10);
-        }
-      }
+  public CompletableFuture<SlotPermit> reserveSlot(SlotReserveContext<SI> ctx) {
+    if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
+      return CompletableFuture.completedFuture(new SlotPermit());
     }
+    return tryReserveSlot(ctx)
+        .map(CompletableFuture::completedFuture)
+        .orElseGet(() -> scheduleSlotAcquisition(ctx));
+  }
+
+  private CompletableFuture<SlotPermit> scheduleSlotAcquisition(SlotReserveContext<SI> ctx) {
+    Duration mustWaitFor;
+    try {
+      mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
+    } catch (ArithmeticException e) {
+      mustWaitFor = Duration.ZERO;
+    }
+
+    CompletableFuture<Void> permitFuture;
+    if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
+      permitFuture =
+          CompletableFuture.supplyAsync(() -> null, delayedExecutor(mustWaitFor.toMillis()));
+    } else {
+      permitFuture = CompletableFuture.completedFuture(null);
+    }
+
+    // After the delay, try to reserve the slot
+    return permitFuture.thenCompose(
+        ignored -> {
+          Optional<SlotPermit> permit = tryReserveSlot(ctx);
+          // If we couldn't get a slot this time, delay for a short period and try again
+          return permit
+              .map(CompletableFuture::completedFuture)
+              .orElseGet(
+                  () ->
+                      CompletableFuture.supplyAsync(() -> null, delayedExecutor(10))
+                          .thenCompose(ig -> scheduleSlotAcquisition(ctx)));
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java
user: login: Quinn-With-Two-Ns
body: https://stackoverflow.com/questions/23320407/how-to-cancel-java-8-completable-future
created_at: 2025-03-04T06:02:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -96,18 +98,19 @@ public ActivityTask poll() {
     PollActivityTaskQueueResponse response;
     SlotPermit permit;
     boolean isSuccessful = false;
-
+    CompletableFuture<SlotPermit> future =
path: temporal-sdk/src/main/java/io/temporal/internal/worker/ActivityPollTask.java
user: login: Quinn-With-Two-Ns
body: `reserveSlot` is a user defined function so I would except any use case to be under a `try` block so exceptions don't leek out and crash the worker or cause undesirable behaviour 
created_at: 2025-03-04T06:07:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,16 +37,19 @@
 @Experimental
 public interface SlotSupplier<SI extends SlotInfo> {
   /**
-   * This function is called before polling for new tasks. Your implementation should block until a
-   * slot is available then return a permit to use that slot.
+   * This function is called before polling for new tasks. Your implementation should return a
+   * Promise that is completed with a {@link SlotPermit} when one becomes available.
+   *
+   * <p>These futures may be cancelled if the worker is shutting down or otherwise abandons the
+   * reservation. This can cause an {@link InterruptedException} to be thrown, in the thread running
+   * your implementation. You may want to catch it to perform any necessary cleanup, and then you
+   * should rethrow the exception. Other thrown exceptions will be logged.
    *
    * @param ctx The context for slot reservation.
-   * @return A permit to use the slot which may be populated with your own data.
-   * @throws InterruptedException The worker may choose to interrupt the thread in order to cancel
-   *     the reservation, or during shutdown. You may perform cleanup, and then should rethrow the
-   *     exception.
+   * @return A future that will be completed with a permit to use the slot when one becomes
+   *     available. Never return null, or complete the future with null.
    */
-  SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException;
+  CompletableFuture<SlotPermit> reserveSlot(SlotReserveContext<SI> ctx) throws Exception;
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/SlotSupplier.java
user: login: cretz
body: I mean more flexible for users. Obviously it is less flexible for the caller (us) of these to get a more abstract type, but if a user only has a `Future`, forcing a `CompletableFuture` on them is less flexible. But it's not a big deal if _we_ need this response as a completable future.
created_at: 2025-03-07T11:55:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -139,29 +152,43 @@ private ResourceBasedSlotSupplier(
   }
 
   @Override
-  public SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException {
-    while (true) {
-      if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
-        return new SlotPermit();
-      } else {
-        Duration mustWaitFor;
-        try {
-          mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
-        } catch (ArithmeticException e) {
-          mustWaitFor = Duration.ZERO;
-        }
-        if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
-          Thread.sleep(mustWaitFor.toMillis());
-        }
-
-        Optional<SlotPermit> permit = tryReserveSlot(ctx);
-        if (permit.isPresent()) {
-          return permit.get();
-        } else {
-          Thread.sleep(10);
-        }
-      }
+  public CompletableFuture<SlotPermit> reserveSlot(SlotReserveContext<SI> ctx) {
+    if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
+      return CompletableFuture.completedFuture(new SlotPermit());
     }
+    return tryReserveSlot(ctx)
+        .map(CompletableFuture::completedFuture)
+        .orElseGet(() -> scheduleSlotAcquisition(ctx));
+  }
+
+  private CompletableFuture<SlotPermit> scheduleSlotAcquisition(SlotReserveContext<SI> ctx) {
+    Duration mustWaitFor;
+    try {
+      mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
+    } catch (ArithmeticException e) {
+      mustWaitFor = Duration.ZERO;
+    }
+
+    CompletableFuture<Void> permitFuture;
+    if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
+      permitFuture =
+          CompletableFuture.supplyAsync(() -> null, delayedExecutor(mustWaitFor.toMillis()));
+    } else {
+      permitFuture = CompletableFuture.completedFuture(null);
+    }
+
+    // After the delay, try to reserve the slot
+    return permitFuture.thenCompose(
+        ignored -> {
+          Optional<SlotPermit> permit = tryReserveSlot(ctx);
+          // If we couldn't get a slot this time, delay for a short period and try again
+          return permit
+              .map(CompletableFuture::completedFuture)
+              .orElseGet(
+                  () ->
+                      CompletableFuture.supplyAsync(() -> null, delayedExecutor(10))
+                          .thenCompose(ig -> scheduleSlotAcquisition(ctx)));
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java
user: login: cretz
body: :+1: So I do not know of what a good way in Java to tell the implementer of something async that it is cancelled? I am guessing we use thread interruption for synchronous activity code today and disallow async/future-returning activity code? Any ideas here?
created_at: 2025-03-07T11:58:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -139,29 +152,43 @@ private ResourceBasedSlotSupplier(
   }
 
   @Override
-  public SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException {
-    while (true) {
-      if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
-        return new SlotPermit();
-      } else {
-        Duration mustWaitFor;
-        try {
-          mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
-        } catch (ArithmeticException e) {
-          mustWaitFor = Duration.ZERO;
-        }
-        if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
-          Thread.sleep(mustWaitFor.toMillis());
-        }
-
-        Optional<SlotPermit> permit = tryReserveSlot(ctx);
-        if (permit.isPresent()) {
-          return permit.get();
-        } else {
-          Thread.sleep(10);
-        }
-      }
+  public CompletableFuture<SlotPermit> reserveSlot(SlotReserveContext<SI> ctx) {
+    if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
+      return CompletableFuture.completedFuture(new SlotPermit());
     }
+    return tryReserveSlot(ctx)
+        .map(CompletableFuture::completedFuture)
+        .orElseGet(() -> scheduleSlotAcquisition(ctx));
+  }
+
+  private CompletableFuture<SlotPermit> scheduleSlotAcquisition(SlotReserveContext<SI> ctx) {
+    Duration mustWaitFor;
+    try {
+      mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
+    } catch (ArithmeticException e) {
+      mustWaitFor = Duration.ZERO;
+    }
+
+    CompletableFuture<Void> permitFuture;
+    if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
+      permitFuture =
+          CompletableFuture.supplyAsync(() -> null, delayedExecutor(mustWaitFor.toMillis()));
+    } else {
+      permitFuture = CompletableFuture.completedFuture(null);
+    }
+
+    // After the delay, try to reserve the slot
+    return permitFuture.thenCompose(
+        ignored -> {
+          Optional<SlotPermit> permit = tryReserveSlot(ctx);
+          // If we couldn't get a slot this time, delay for a short period and try again
+          return permit
+              .map(CompletableFuture::completedFuture)
+              .orElseGet(
+                  () ->
+                      CompletableFuture.supplyAsync(() -> null, delayedExecutor(10))
+                          .thenCompose(ig -> scheduleSlotAcquisition(ctx)));
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java
user: login: Quinn-With-Two-Ns
body: So the standard library would use `Future` for this like here https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ScheduledFuture.html. The other option is to return a special interface like `SlotSupplierFuture` interface that implements the  `CompletableFuture ` interface and another different "cancel" method that actually does what you want 
created_at: 2025-03-10T14:45:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -139,29 +152,43 @@ private ResourceBasedSlotSupplier(
   }
 
   @Override
-  public SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException {
-    while (true) {
-      if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
-        return new SlotPermit();
-      } else {
-        Duration mustWaitFor;
-        try {
-          mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
-        } catch (ArithmeticException e) {
-          mustWaitFor = Duration.ZERO;
-        }
-        if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
-          Thread.sleep(mustWaitFor.toMillis());
-        }
-
-        Optional<SlotPermit> permit = tryReserveSlot(ctx);
-        if (permit.isPresent()) {
-          return permit.get();
-        } else {
-          Thread.sleep(10);
-        }
-      }
+  public CompletableFuture<SlotPermit> reserveSlot(SlotReserveContext<SI> ctx) {
+    if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
+      return CompletableFuture.completedFuture(new SlotPermit());
     }
+    return tryReserveSlot(ctx)
+        .map(CompletableFuture::completedFuture)
+        .orElseGet(() -> scheduleSlotAcquisition(ctx));
+  }
+
+  private CompletableFuture<SlotPermit> scheduleSlotAcquisition(SlotReserveContext<SI> ctx) {
+    Duration mustWaitFor;
+    try {
+      mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
+    } catch (ArithmeticException e) {
+      mustWaitFor = Duration.ZERO;
+    }
+
+    CompletableFuture<Void> permitFuture;
+    if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
+      permitFuture =
+          CompletableFuture.supplyAsync(() -> null, delayedExecutor(mustWaitFor.toMillis()));
+    } else {
+      permitFuture = CompletableFuture.completedFuture(null);
+    }
+
+    // After the delay, try to reserve the slot
+    return permitFuture.thenCompose(
+        ignored -> {
+          Optional<SlotPermit> permit = tryReserveSlot(ctx);
+          // If we couldn't get a slot this time, delay for a short period and try again
+          return permit
+              .map(CompletableFuture::completedFuture)
+              .orElseGet(
+                  () ->
+                      CompletableFuture.supplyAsync(() -> null, delayedExecutor(10))
+                          .thenCompose(ig -> scheduleSlotAcquisition(ctx)));
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java
user: login: Sushisource
body: I've dealt with this by returning future and using an anonymous class in the one spot (two counting test code) I actually need another stage. It works fine since the "callback" (incrementing the counter in the tracking slot supplier) only ever needs to be called if `get` is actually invoked, and it always is anyway.
created_at: 2025-03-18T16:57:06Z
author_association: MEMBER
diff_hunk: @@ -139,29 +152,43 @@ private ResourceBasedSlotSupplier(
   }
 
   @Override
-  public SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException {
-    while (true) {
-      if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
-        return new SlotPermit();
-      } else {
-        Duration mustWaitFor;
-        try {
-          mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
-        } catch (ArithmeticException e) {
-          mustWaitFor = Duration.ZERO;
-        }
-        if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
-          Thread.sleep(mustWaitFor.toMillis());
-        }
-
-        Optional<SlotPermit> permit = tryReserveSlot(ctx);
-        if (permit.isPresent()) {
-          return permit.get();
-        } else {
-          Thread.sleep(10);
-        }
-      }
+  public CompletableFuture<SlotPermit> reserveSlot(SlotReserveContext<SI> ctx) {
+    if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
+      return CompletableFuture.completedFuture(new SlotPermit());
     }
+    return tryReserveSlot(ctx)
+        .map(CompletableFuture::completedFuture)
+        .orElseGet(() -> scheduleSlotAcquisition(ctx));
+  }
+
+  private CompletableFuture<SlotPermit> scheduleSlotAcquisition(SlotReserveContext<SI> ctx) {
+    Duration mustWaitFor;
+    try {
+      mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
+    } catch (ArithmeticException e) {
+      mustWaitFor = Duration.ZERO;
+    }
+
+    CompletableFuture<Void> permitFuture;
+    if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
+      permitFuture =
+          CompletableFuture.supplyAsync(() -> null, delayedExecutor(mustWaitFor.toMillis()));
+    } else {
+      permitFuture = CompletableFuture.completedFuture(null);
+    }
+
+    // After the delay, try to reserve the slot
+    return permitFuture.thenCompose(
+        ignored -> {
+          Optional<SlotPermit> permit = tryReserveSlot(ctx);
+          // If we couldn't get a slot this time, delay for a short period and try again
+          return permit
+              .map(CompletableFuture::completedFuture)
+              .orElseGet(
+                  () ->
+                      CompletableFuture.supplyAsync(() -> null, delayedExecutor(10))
+                          .thenCompose(ig -> scheduleSlotAcquisition(ctx)));
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java
user: login: cretz
body: Can there be an example of how a user who is implementing a slot supplier can react to reservation cancellation? Or maybe that one already is. My fear is they'll use naive futures and it'll hang shutdown. Which of course is their fault, I just want to make sure we clarify (maybe via test but for sure via docs) that if they want to handle cancellation of reserve slot, they may need provide their own future implementation that reacts to the `cancel` call properly.
created_at: 2025-03-18T17:11:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -139,29 +152,43 @@ private ResourceBasedSlotSupplier(
   }
 
   @Override
-  public SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException {
-    while (true) {
-      if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
-        return new SlotPermit();
-      } else {
-        Duration mustWaitFor;
-        try {
-          mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
-        } catch (ArithmeticException e) {
-          mustWaitFor = Duration.ZERO;
-        }
-        if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
-          Thread.sleep(mustWaitFor.toMillis());
-        }
-
-        Optional<SlotPermit> permit = tryReserveSlot(ctx);
-        if (permit.isPresent()) {
-          return permit.get();
-        } else {
-          Thread.sleep(10);
-        }
-      }
+  public CompletableFuture<SlotPermit> reserveSlot(SlotReserveContext<SI> ctx) {
+    if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
+      return CompletableFuture.completedFuture(new SlotPermit());
     }
+    return tryReserveSlot(ctx)
+        .map(CompletableFuture::completedFuture)
+        .orElseGet(() -> scheduleSlotAcquisition(ctx));
+  }
+
+  private CompletableFuture<SlotPermit> scheduleSlotAcquisition(SlotReserveContext<SI> ctx) {
+    Duration mustWaitFor;
+    try {
+      mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
+    } catch (ArithmeticException e) {
+      mustWaitFor = Duration.ZERO;
+    }
+
+    CompletableFuture<Void> permitFuture;
+    if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
+      permitFuture =
+          CompletableFuture.supplyAsync(() -> null, delayedExecutor(mustWaitFor.toMillis()));
+    } else {
+      permitFuture = CompletableFuture.completedFuture(null);
+    }
+
+    // After the delay, try to reserve the slot
+    return permitFuture.thenCompose(
+        ignored -> {
+          Optional<SlotPermit> permit = tryReserveSlot(ctx);
+          // If we couldn't get a slot this time, delay for a short period and try again
+          return permit
+              .map(CompletableFuture::completedFuture)
+              .orElseGet(
+                  () ->
+                      CompletableFuture.supplyAsync(() -> null, delayedExecutor(10))
+                          .thenCompose(ig -> scheduleSlotAcquisition(ctx)));
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java
user: login: Sushisource
body: I think the existing docstring is reasonably clear on that front. I could add a warning about using CompletableFuture specifically, and if they do that they maybe want to be aware of how that can go wrong (which wasn't obvious to me at least so maybe that's a good idea) but I suppose otherwise it's like any other Future returning Java API
created_at: 2025-03-18T17:26:14Z
author_association: MEMBER
diff_hunk: @@ -139,29 +152,43 @@ private ResourceBasedSlotSupplier(
   }
 
   @Override
-  public SlotPermit reserveSlot(SlotReserveContext<SI> ctx) throws InterruptedException {
-    while (true) {
-      if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
-        return new SlotPermit();
-      } else {
-        Duration mustWaitFor;
-        try {
-          mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
-        } catch (ArithmeticException e) {
-          mustWaitFor = Duration.ZERO;
-        }
-        if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
-          Thread.sleep(mustWaitFor.toMillis());
-        }
-
-        Optional<SlotPermit> permit = tryReserveSlot(ctx);
-        if (permit.isPresent()) {
-          return permit.get();
-        } else {
-          Thread.sleep(10);
-        }
-      }
+  public CompletableFuture<SlotPermit> reserveSlot(SlotReserveContext<SI> ctx) {
+    if (ctx.getNumIssuedSlots() < options.getMinimumSlots()) {
+      return CompletableFuture.completedFuture(new SlotPermit());
     }
+    return tryReserveSlot(ctx)
+        .map(CompletableFuture::completedFuture)
+        .orElseGet(() -> scheduleSlotAcquisition(ctx));
+  }
+
+  private CompletableFuture<SlotPermit> scheduleSlotAcquisition(SlotReserveContext<SI> ctx) {
+    Duration mustWaitFor;
+    try {
+      mustWaitFor = options.getRampThrottle().minus(timeSinceLastSlotIssued());
+    } catch (ArithmeticException e) {
+      mustWaitFor = Duration.ZERO;
+    }
+
+    CompletableFuture<Void> permitFuture;
+    if (mustWaitFor.compareTo(Duration.ZERO) > 0) {
+      permitFuture =
+          CompletableFuture.supplyAsync(() -> null, delayedExecutor(mustWaitFor.toMillis()));
+    } else {
+      permitFuture = CompletableFuture.completedFuture(null);
+    }
+
+    // After the delay, try to reserve the slot
+    return permitFuture.thenCompose(
+        ignored -> {
+          Optional<SlotPermit> permit = tryReserveSlot(ctx);
+          // If we couldn't get a slot this time, delay for a short period and try again
+          return permit
+              .map(CompletableFuture::completedFuture)
+              .orElseGet(
+                  () ->
+                      CompletableFuture.supplyAsync(() -> null, delayedExecutor(10))
+                          .thenCompose(ig -> scheduleSlotAcquisition(ctx)));
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/ResourceBasedSlotSupplier.java
user: login: cretz
body: > I suppose otherwise it's like any other Future returning Java API

Yup but still a bit confusing because most don't manually implement cancel. I propose we alter the javadoc to change from:

> These futures may be cancelled if the worker is shutting down or otherwise abandons the reservation. This can cause an {@link InterruptedException} to be thrown, in the thread running your implementation. You may want to catch it to perform any necessary cleanup, and then you should rethrow the exception.

To:

> These futures may get `cancel` invoked on them if the worker is shutting down or otherwise abandons the reservation. Users need to make sure they handle `cancel` on the resulting future. By default in Java, futures created by users or composed from other futures will not properly propagate `cancel` so users need to make sure this is handled properly.

Can still add note about the exception if we do handle that special. Regardless, non-blocking.
created_at: 2025-03-18T19:36:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -48,10 +49,52 @@ public TrackingSlotSupplier(SlotSupplier<SI> inner, Scope metricsScope) {
     publishSlotsMetric();
   }
 
-  public SlotPermit reserveSlot(SlotReservationData dat) throws InterruptedException {
-    SlotPermit p = inner.reserveSlot(createCtx(dat));
-    issuedSlots.incrementAndGet();
-    return p;
+  public Future<SlotPermit> reserveSlot(SlotReservationData dat) {
path: temporal-sdk/src/main/java/io/temporal/internal/worker/TrackingSlotSupplier.java
user: login: Quinn-With-Two-Ns
body: ```suggestion
  public Future<SlotPermit> reserveSlot(SlotReservationData data) {
```
created_at: 2025-03-19T18:43:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -48,10 +49,52 @@ public TrackingSlotSupplier(SlotSupplier<SI> inner, Scope metricsScope) {
     publishSlotsMetric();
   }
 
-  public SlotPermit reserveSlot(SlotReservationData dat) throws InterruptedException {
-    SlotPermit p = inner.reserveSlot(createCtx(dat));
-    issuedSlots.incrementAndGet();
-    return p;
+  public Future<SlotPermit> reserveSlot(SlotReservationData dat) {
+    final Future<SlotPermit> originalFuture;
+    try {
+      originalFuture = inner.reserveSlot(createCtx(dat));
+    } catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+
+    return new Future<SlotPermit>() {
+      private final AtomicBoolean callbackInvoked = new AtomicBoolean(false);
+
+      private SlotPermit executeCallbackIfNeeded(SlotPermit permit) {
+        if (callbackInvoked.compareAndSet(false, true)) {
+          issuedSlots.incrementAndGet();
+        }
+        return permit;
+      }
+
+      @Override
+      public boolean cancel(boolean mayInterruptIfRunning) {
+        return originalFuture.cancel(mayInterruptIfRunning);
+      }
+
+      @Override
+      public boolean isCancelled() {
+        return originalFuture.isCancelled();
+      }
+
+      @Override
+      public boolean isDone() {
+        return originalFuture.isDone();
+      }
+
+      @Override
+      public SlotPermit get() throws InterruptedException, ExecutionException {
+        SlotPermit permit = originalFuture.get();
+        return executeCallbackIfNeeded(permit);
+      }
+
+      @Override
+      public SlotPermit get(long timeout, TimeUnit unit)
+          throws InterruptedException, ExecutionException, TimeoutException {
+        SlotPermit permit = originalFuture.get(timeout, unit);
+        return executeCallbackIfNeeded(permit);
+      }
+    };
   }
 
   public Optional<SlotPermit> tryReserveSlot(SlotReservationData dat) {
path: temporal-sdk/src/main/java/io/temporal/internal/worker/TrackingSlotSupplier.java
user: login: Quinn-With-Two-Ns
body: ```suggestion
  public Optional<SlotPermit> tryReserveSlot(SlotReservationData data) {
```
created_at: 2025-03-19T18:43:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,16 +36,19 @@
 @Experimental
 public interface SlotSupplier<SI extends SlotInfo> {
   /**
-   * This function is called before polling for new tasks. Your implementation should block until a
-   * slot is available then return a permit to use that slot.
+   * This function is called before polling for new tasks. Your implementation should return a
+   * Promise that is completed with a {@link SlotPermit} when one becomes available.
path: temporal-sdk/src/main/java/io/temporal/worker/tuning/SlotSupplier.java
user: login: Quinn-With-Two-Ns
body: ```suggestion
   * Future that is completed with a {@link SlotPermit} when one becomes available.
```
created_at: 2025-03-24T05:48:56Z
author_association: CONTRIBUTOR

PR comments:
user: login: Sushisource
created_at: 2025-03-20T17:50:02Z
body: Looks like there can be some reserve/release mismatches. Investigating that.
author_association: MEMBER
user: login: Sushisource
created_at: 2025-03-20T18:16:53Z
body: Actually I've just realized this ends up being a huge mess because anything like `thenApply` you call on this type returns a `CompleteableFuture` instead of a `SlotSupplierFuture`, and there's, AFAICT, no good way to fix that without manually overriding every single method which seems awful.
author_association: MEMBER
user: login: Quinn-With-Two-Ns
created_at: 2025-03-20T18:21:03Z
body: >thenApply you call on this type returns a CompleteableFuture instead of a SlotSupplierFuture

Why is that an issue?
author_association: CONTRIBUTOR
user: login: Sushisource
created_at: 2025-03-20T23:30:11Z
body: > > thenApply you call on this type returns a CompleteableFuture instead of a SlotSupplierFuture
> 
> Why is that an issue?

I suppose it's not really
author_association: MEMBER