Repository: PrefectHQ/prefect

Issue:
Better rate limit retries
<!-- Thanks for contributing to Prefect Core! 🎉-->

## Summary
<!-- A sentence summarizing the PR -->
During certain large burst events coming from a highly parallelized workflow, API rate limits can be encountered (represented as 429 status codes).  The default retry logic for bad status codes is not well suited for the longer durations over which rate limits are computed, so this PR separates out 429 client handling logic into its own separate custom code path.


## Changes
<!-- What does this PR change? -->
Adds generous sleeps and retries when the client encounters a 429 status code, maximizing the chances for success when this (rare) situation is encountered.


## Importance
<!-- Why is this PR important? -->
Better UX and workflow stability. 


## Checklist
<!-- PRs will not be reviewed unless these boxes are checked -->

This PR:

- [x] adds new tests (if appropriate)
- [x] adds a change file in the `changes/` directory (if appropriate)
- [x] updates docstrings for any new functions or function arguments, including `docs/outline.toml` for API reference docs (if appropriate)


Discussion:

Resolving patch:
diff --git a/src/prefect/client/client.py b/src/prefect/client/client.py
--- a/src/prefect/client/client.py
+++ b/src/prefect/client/client.py
@@ -573,6 +573,7 @@ def _send_request(
         url: str,
         params: Dict[str, JSONLike] = None,
         headers: dict = None,
+        rate_limit_counter: int = 1,
     ) -> "requests.models.Response":
         import requests
 
@@ -616,6 +617,27 @@ def _send_request(
                 f"Request duration: {round(end_time - start_time, 4)} seconds"
             )
 
+        # custom logic when encountering an API rate limit:
+        # each time we encounter a rate limit, we sleep for
+        # 3 minutes + random amount, where the random amount
+        # is uniformly sampled from (0, 10 * 2 ** rate_limit_counter)
+        # up to (0, 640), at which point an error is raised if the limit
+        # is still being hit
+        rate_limited = response.status_code == 429
+        if rate_limited and rate_limit_counter <= 6:
+            jitter = random.random() * 10 * (2 ** rate_limit_counter)
+            naptime = 3 * 60 + jitter  # 180 second sleep + increasing jitter
+            self.logger.debug(f"Rate limit encountered; sleeping for {naptime}s...")
+            time.sleep(naptime)
+            response = self._send_request(
+                session=session,
+                method=method,
+                url=url,
+                params=params,
+                headers=headers,
+                rate_limit_counter=rate_limit_counter + 1,
+            )
+
         # Check if request returned a successful status
         try:
             response.raise_for_status()
@@ -711,28 +733,6 @@ def _request(
         response = self._send_request(
             session=session, method=method, url=url, params=params, headers=headers
         )
-
-        # custom logic when encountering an API rate limit:
-        # each time we encounter a rate limit, we sleep for
-        # 3 minutes + random amount, where the random amount
-        # is uniformly sampled from (0, 10 * 2 ** rate_limit_counter)
-        # up to (0, 640), at which point an error is raised if the limit
-        # is still being hit
-        rate_limited = response.status_code == 429
-        iter_count = 1
-        while rate_limited:
-            jitter = random.random() * 10 * (2 ** iter_count)
-            naptime = 3 * 60 + jitter  # 180 second sleep + increasing jitter
-            self.logger.debug(f"Rate limit encountered; sleeping for {naptime}s...")
-            time.sleep(naptime)
-            response = self._send_request(
-                session=session, method=method, url=url, params=params, headers=headers
-            )
-            rate_limited = response.status_code == 429
-            iter_count += 1
-            if iter_count > 6:
-                raise ClientError("Rate limit encountered during execution.")
-
         # parse the response
         try:
             json_resp = response.json()


Test patch: