Repository: twisted/twisted

Issue:
t.a.i.backoffPolicy fails after a certain number of retries (builtins.OverflowError)
|[<img alt="wiml's avatar" src="https://avatars.githubusercontent.com/u/156891?s=50" width="50" height="50">](https://github.com/wiml)| @wiml reported|
|-|-|
|Trac ID|trac#9476|
|Type|defect|
|Created|2018-07-02 21:05:08Z|

I have some connections which can go down for a while, but I want to keep checking (once per minute, e.g.) until they become reachable again. Supposedly, I can do this with the `maxDelay` parameter of `backoffPolicy`.

After a while, though, instead of continuing to re-try, `policy` will start raising an exception:

```
Jul 02 12:59:22 [8534]: File ".../python3.6/site-packages/automat/_methodical.py", line 135, in _connectionFailed
Jul 02 12:59:22 [8534]: value = output(oself, *args, **kwargs)
Jul 02 12:59:22 [8534]: File ".../python3.6/site-packages/automat/_methodical.py", line 169, in __call__
Jul 02 12:59:22 [8534]: return self.method(oself, *args, **kwargs)
Jul 02 12:59:22 [8534]: File ".../python3.6/site-packages/twisted/application/internet.py", line 740, in _ignoreAndWait
Jul 02 12:59:22 [8534]: return self._doWait()
Jul 02 12:59:22 [8534]: File ".../python3.6/site-packages/twisted/application/internet.py", line 744, in _doWait
Jul 02 12:59:22 [8534]: delay = self._timeoutForAttempt(self._failedAttempts)
Jul 02 12:59:22 [8534]: File ".../python3.6/site-packages/twisted/application/internet.py", line 520, in policy
Jul 02 12:59:22 [8534]: return min(initialDelay * (factor ** attempt), maxDelay) + jitter()
Jul 02 12:59:22 [8534]: builtins.OverflowError: (34, 'Numerical result out of range')
```

(Some pathnames sanitized.) A little investigation shows that once the `attempt` count exceeds 1750, the intermediate result is too large to represent in a `float`, and so even though it will be discarded and replaced by `maxDelay` this will cause retries to fail:

```
>>> factor = 1.5
>>> (factor ** 1750)
1.4444527745742028e+308
>>> (factor ** 1751)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: (34, 'Numerical result out of range')
```

This will take only a couple of days at `maxDelay=60.0`: less than a weekend.

Versions:
* Twisted==18.4.0
* Python 3.6.3 (default, Jan  4 2018, 16:40:53) 
* [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux
* CentOS Linux release 7.4.1708 (Core)


<details><summary>Searchable metadata</summary>

```
trac-id__9476 9476
type__defect defect
reporter__wiml wiml
priority__normal normal
milestone__None None
branch__ 
branch_author__ 
status__closed closed
resolution__fixed fixed
component__core core
keywords__None None
time__1530565508694370 1530565508694370
changetime__1558292011336444 1558292011336444
version__None None
owner__jandelgado jandelgado

```
</details>



Discussion:

Resolving patch:
diff --git a/src/twisted/application/internet.py b/src/twisted/application/internet.py
--- a/src/twisted/application/internet.py
+++ b/src/twisted/application/internet.py
@@ -517,9 +517,15 @@ def backoffPolicy(initialDelay=1.0, maxDelay=60.0, factor=1.5,
     @rtype: see L{ClientService.__init__}'s C{retryPolicy} argument.
     """
     def policy(attempt):
-        return min(initialDelay * (factor ** attempt), maxDelay) + jitter()
+        try:
+            delay = min(initialDelay * (factor ** min(100, attempt)), maxDelay)
+        except OverflowError:
+            delay = maxDelay
+        return delay + jitter()
     return policy
 
+
+
 _defaultPolicy = backoffPolicy()
 
 


Test patch:
diff --git a/src/twisted/test/test_application.py b/src/twisted/test/test_application.py
--- a/src/twisted/test/test_application.py
+++ b/src/twisted/test/test_application.py
@@ -18,6 +18,7 @@
     asyncio = None
 
 from twisted.application import service, internet, app, reactors
+from twisted.application.internet import backoffPolicy
 from twisted.internet import interfaces, defer, protocol, reactor
 from twisted.persisted import sob
 from twisted.plugins import twisted_reactors
@@ -967,3 +968,32 @@ def getReactorTypes():
         self.assertRaises(SystemExit, options.opt_help_reactors)
         message = options.messageOutput.getvalue()
         self.assertNotIn("reactors not available", message)
+
+
+
+class BackoffPolicyTests(unittest.TestCase):
+    """Tests of L{twisted.application.internet.backoffPolicy}"""
+    def test_calculates_correct_values(self):
+        """Test that backoffPolicy() calculates expected values"""
+        pol = backoffPolicy(1.0, 60.0, 1.5, jitter=lambda: 1)
+        self.assertAlmostEqual(pol(0), 2)
+        self.assertAlmostEqual(pol(1), 2.5)
+        self.assertAlmostEqual(pol(10), 58.6650390625)
+        self.assertEqual(pol(20), 61)
+        self.assertEqual(pol(100), 61)
+
+    def test_does_not_overflow_on_high_attempts(self):
+        """Test makes sure backoffPolicy() does not fail for large values of
+           the attempt parameter. In previous versions, this test failed
+           when attempt was larger than 1750.
+           See https://twistedmatrix.com/trac/ticket/9476"""
+        pol = backoffPolicy(1.0, 60.0, 1.5, jitter=lambda: 1)
+        self.assertEqual(pol(1751), 61)
+        self.assertEqual(pol(1000000), 61)
+
+    def test_does_not_overflow_with_large_factor_value(self):
+        """Test makes sure that even with unusual parameters, any
+           OverflowError within backoffPolicy() will be caught and
+           maxDelay will be returned instead"""
+        pol = backoffPolicy(1.0, 60.0, 1E10, jitter=lambda: 1)
+        self.assertEqual(pol(1751), 61)