Repository: dwavesystems/dwave-cloud-client

Issue:
Http request retry `backoff_max` not working with `urllib3>=2`
### Problem

Prior to `requests==2.30.0` (May 2023), `urllib3` was upper-bounded to [`<1.27`](https://github.com/psf/requests/blob/87d63de8739263bbe17034fba2285c79780da7e8/setup.py#L64C6-L64C27), but in `2.30.0` Requests added support for urllib3, v2.

In `urllib3==2`, way of specifying `backoff_max` time for request retries (`urllib3.Retry`) changed. Now they finally support it as a construction-time parameter, but before we had to override `Retry.BACKOFF_MAX` and in later versions `Retry.DEFAULT_BACKOFF_MAX` attributes either before or after construction.

Note that we don't use urllib3 directly in the cloud client, but via Requests, which accepts (passes) urllib3's spec for retries, [`max_retries`](https://github.com/dwavesystems/dwave-cloud-client/blob/d0f7e6935f3069ffe4d1e04b548427f0413a4013/dwave/cloud/client/base.py#L643).

### Impact

For users with `urllib3>=2` installed (fresh installs after May 3, 2023):
- minimal for `dwave.cloud.Client` usage (primarily multipart upload requests), and
- minimal for `dwave.cloud.api` clients (`Region` requests),

i.e. `backoff_max` would not be set to 60 sec (our default), but to 120 sec (urllib3's default).

Only if user explicitly wanted to modify the `backoff_max` time, via our `http_retry_backoff_max` config parameter, they would notice somewhat significant impact of this issue.

Discussion:

Patch:
diff --git a/dwave/cloud/api/client.py b/dwave/cloud/api/client.py
index 470f6e0..d330eda 100644
--- a/dwave/cloud/api/client.py
+++ b/dwave/cloud/api/client.py
@@ -320,9 +320,10 @@ class DWaveAPIClient:
 
         retry = urllib3.Retry(**kwargs)
 
-        # note: `Retry.BACKOFF_MAX` can't be set on construction
+        # note: prior to `urllib3==2`, backoff_max had to be set manually on object
         if backoff_max is not None:
-            retry.BACKOFF_MAX = backoff_max
+            # handle `urllib3>=1.21.1,<1.27` AND `urllib3>=1.21.1,<3`
+            retry.BACKOFF_MAX = retry.backoff_max = backoff_max
 
         return retry
 
diff --git a/dwave/cloud/client/base.py b/dwave/cloud/client/base.py
index 28a9e9a..43f797e 100644
--- a/dwave/cloud/client/base.py
+++ b/dwave/cloud/client/base.py
@@ -622,11 +622,7 @@ class Client(object):
 
         # create http idempotent Retry config
         def get_retry_conf():
-            # need a subclass to override the backoff_max
-            class Retry(urllib3.Retry):
-                BACKOFF_MAX = self.http_retry_backoff_max
-
-            return Retry(
+            retry = urllib3.Retry(
                 total=self.http_retry_total,
                 connect=self.http_retry_connect,
                 read=self.http_retry_read,
@@ -637,6 +633,12 @@ class Client(object):
                 raise_on_status=True,
                 respect_retry_after_header=True)
 
+            if self.http_retry_backoff_max is not None:
+                # handle `urllib3>=1.21.1,<1.27` AND `urllib3>=1.21.1,<3`
+                retry.BACKOFF_MAX = retry.backoff_max = self.http_retry_backoff_max
+
+            return retry
+
         session = BaseUrlSession(base_url=endpoint)
         session.mount('http://',
             TimeoutingHTTPAdapter(timeout=self.request_timeout,
diff --git a/releasenotes/notes/support-urllib3-v2-5c5a2cb29a47b43d.yaml b/releasenotes/notes/support-urllib3-v2-5c5a2cb29a47b43d.yaml
new file mode 100644
index 0000000..c8e7dad
--- /dev/null
+++ b/releasenotes/notes/support-urllib3-v2-5c5a2cb29a47b43d.yaml
@@ -0,0 +1,6 @@
+---
+fixes:
+  - |
+    Correctly set `backoff_max` time for retried requests when ``urllib3>=2.0``
+    is used.
+    See `#566 <https://github.com/dwavesystems/dwave-cloud-client/issues/566>`_.
\ No newline at end of file


Test patch:
diff --git a/tests/test_client.py b/tests/test_client.py
index 5b72294..e59df11 100644
--- a/tests/test_client.py
+++ b/tests/test_client.py
@@ -555,7 +555,8 @@ class ClientConstruction(unittest.TestCase):
         self.assertEqual(retry.redirect, opts['http_retry_redirect'])
         self.assertEqual(retry.status, opts['http_retry_status'])
         self.assertEqual(retry.backoff_factor, opts['http_retry_backoff_factor'])
-        self.assertEqual(retry.BACKOFF_MAX, opts['http_retry_backoff_max'])
+        backoff_max = getattr(retry, 'backoff_max', getattr(retry, 'BACKOFF_MAX'))
+        self.assertEqual(backoff_max, opts['http_retry_backoff_max'])
 
     def test_http_retry_params_from_config(self):
         retry_opts = {