Repository: dask/distributed

Issue:
`OverflowError` in `Cluster._sync_cluster_info()`
When `_sync_cluster_info` fails over an extended period, it raises an `OverflowError`:

```
Task exception was never retrieved
future: <Task finished name='Task-128673' coro=<Cluster._sync_cluster_info() done, defined at /opt/homebrew/Caskroom/mambaforge/base/envs/benchmarking/lib/python3.10/site-packages/distributed/deploy/cluster.py:144> exception=OverflowError(34, 'Result too large')>
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/benchmarking/lib/python3.10/site-packages/distributed/deploy/cluster.py", line 178, in _sync_cluster_info
    interval = min(max_interval, self._sync_interval * 1.5**err_count)
OverflowError: (34, 'Result too large')
Task exception was never retrieved
future: <Task finished name='Task-186646' coro=<Cluster._sync_cluster_info() done, defined at /opt/homebrew/Caskroom/mambaforge/base/envs/benchmarking/lib/python3.10/site-packages/distributed/deploy/cluster.py:144> exception=OverflowError(34, 'Result too large')>
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/benchmarking/lib/python3.10/site-packages/distributed/deploy/cluster.py", line 178, in _sync_cluster_info
    interval = min(max_interval, self._sync_interval * 1.5**err_count)
OverflowError: (34, 'Result too large')
Task exception was never retrieved
future: <Task finished name='Task-187847' coro=<Cluster._sync_cluster_info() done, defined at /opt/homebrew/Caskroom/mambaforge/base/envs/benchmarking/lib/python3.10/site-packages/distributed/deploy/cluster.py:144> exception=OverflowError(34, 'Result too large')>
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/benchmarking/lib/python3.10/site-packages/distributed/deploy/cluster.py", line 178, in _sync_cluster_info
    interval = min(max_interval, self._sync_interval * 1.5**err_count)
OverflowError: (34, 'Result too large')
```



Discussion:

Resolving patch:
diff --git a/distributed/deploy/cluster.py b/distributed/deploy/cluster.py
--- a/distributed/deploy/cluster.py
+++ b/distributed/deploy/cluster.py
@@ -175,7 +175,9 @@ async def _sync_cluster_info(self):
                         exc_info=True,
                     )
             # Sleep, with error backoff
-            interval = min(max_interval, self._sync_interval * 1.5**err_count)
+            interval = _exponential_backoff(
+                err_count, self._sync_interval, 1.5, max_interval
+            )
             await asyncio.sleep(interval)
 
     async def _close(self):
@@ -579,3 +581,15 @@ def __eq__(self, other):
 
     def __hash__(self):
         return id(self)
+
+
+def _exponential_backoff(
+    attempt: int, multiplier: float, exponential_base: float, max_interval: float
+) -> float:
+    """Calculate the duration of an exponential backoff"""
+    try:
+        interval = multiplier * exponential_base**attempt
+    except OverflowError:
+        return max_interval
+
+    return min(max_interval, interval)


Test patch:
diff --git a/distributed/deploy/tests/test_cluster.py b/distributed/deploy/tests/test_cluster.py
--- a/distributed/deploy/tests/test_cluster.py
+++ b/distributed/deploy/tests/test_cluster.py
@@ -3,7 +3,7 @@
 import pytest
 from tornado.ioloop import IOLoop
 
-from distributed.deploy.cluster import Cluster
+from distributed.deploy.cluster import Cluster, _exponential_backoff
 from distributed.utils_test import gen_test
 
 
@@ -52,3 +52,12 @@ def __init__(self, *args, **kwargs):
     assert [(w.category, *w.message.args) for w in warninfo] == [
         (DeprecationWarning, "setting the loop property is deprecated")
     ]
+
+
+def test_exponential_backoff():
+    assert _exponential_backoff(0, 1.5, 3, 20) == 1.5
+    assert _exponential_backoff(1, 1.5, 3, 20) == 4.5
+    assert _exponential_backoff(2, 1.5, 3, 20) == 13.5
+    assert _exponential_backoff(5, 1.5, 3, 20) == 20
+    # avoid overflow
+    assert _exponential_backoff(1000, 1.5, 3, 20) == 20