Repository: PrefectHQ/prefect

Issue:
Allow users to set delay transition time for Secure Task Concurrency Slots
### First check

- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar request and didn't find it.
- [X] I searched the Prefect documentation for this feature.

### Prefect Version

2.x

### Describe the current behavior

Currently, the delay transition time for secure  task concurrency slots is hardcoded to 30 seconds.

https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/orchestration/core_policy.py#L140-L143

### Describe the proposed behavior

Introduce a new prefect configuration/setting that allows the user to set the value for the transition delay.

### Example Use

Example usage similar as documented in https://docs.prefect.io/2.11.4/guides/settings/

### Additional context

Our flows are using mapped tasks that are limited by task concurrency tags. The task by themselves take roughly 1 seconds. However, because the transition delay is hard coded to 30, after each task is "completed", the worker is idle and waits for 30 seconds. It would be great if this delay could be configured by the user.


Discussion:
Hello @WillRaphaelson 
Any updates on this?
Thanks @OptimeeringBigya ive upper our internal priority on this. When we pick it up, we can decide on the best implementation. I think this should be a setting on the worker. Thoughts?
@WillRaphaelson I am unsure about the internal working within Prefect. 

I would think that the value in the server is the default value for all the workers and the value in a specific worker works as an override for that worker.

If this is too complicated, I think it is fine to just have it in worker only.
We have several PRs on our end blocked by this feature request that I would rather have something than nothing. :)
Any updates on when this is going live? This is causing impact for us as well

Resolving patch:
diff --git a/src/prefect/server/orchestration/core_policy.py b/src/prefect/server/orchestration/core_policy.py
--- a/src/prefect/server/orchestration/core_policy.py
+++ b/src/prefect/server/orchestration/core_policy.py
@@ -30,6 +30,7 @@
 )
 from prefect.server.schemas import core, filters, states
 from prefect.server.schemas.states import StateType
+from prefect.settings import PREFECT_TASK_RUN_TAG_CONCURRENCY_SLOT_WAIT_SECONDS
 from prefect.utilities.math import clamped_poisson_interval
 
 
@@ -92,7 +93,8 @@ class SecureTaskConcurrencySlots(BaseOrchestrationRule):
     This rule checks if concurrency limits have been set on the tags associated with a
     TaskRun. If so, a concurrency slot will be secured against each concurrency limit
     before being allowed to transition into a running state. If a concurrency limit has
-    been reached, the client will be instructed to delay the transition for 30 seconds
+    been reached, the client will be instructed to delay the transition for the duration
+    specified by the "PREFECT_TASK_RUN_TAG_CONCURRENCY_SLOT_WAIT_SECONDS" setting
     before trying again. If the concurrency limit set on a tag is 0, the transition will
     be aborted to prevent deadlocks.
     """
@@ -138,7 +140,7 @@ async def before_transition(
                     stale_limit.active_slots = list(active_slots)
 
                 await self.delay_transition(
-                    30,
+                    PREFECT_TASK_RUN_TAG_CONCURRENCY_SLOT_WAIT_SECONDS.value(),
                     f"Concurrency limit for the {tag} tag has been reached",
                 )
             else:
diff --git a/src/prefect/settings.py b/src/prefect/settings.py
--- a/src/prefect/settings.py
+++ b/src/prefect/settings.py
@@ -760,6 +760,12 @@ def default_cloud_ui_url(settings, value):
 This value does not overwrite individually set retry delay seconds
 """
 
+PREFECT_TASK_RUN_TAG_CONCURRENCY_SLOT_WAIT_SECONDS = Setting(int, default=30)
+"""
+The number of seconds to wait before retrying when a task run
+cannot secure a concurrency slot from the server.
+"""
+
 PREFECT_LOCAL_STORAGE_PATH = Setting(
     Path,
     default=Path("${PREFECT_HOME}") / "storage",


Test patch: