Repository: urllib3/urllib3

Issue:
Add `other` counter to Retry for fully fine-grained retry configuration
```python
from urllib3 import PoolManager                                                   
from urllib3.util.retry import Retry                                              
from urllib3.util.timeout import Timeout                                          
                                                                                  
                                                                                  
def main(**kwargs):                                                               
    url = 'https://78.155.216.172'                                                
    mgr = PoolManager()                                                           
    retry_opts = {                                                                
        'total': None,                                                            
        'redirect': 3,                                                            
        'raise_on_redirect': True,                                                
        'connect': 1,                                                             
        'read': 1,                                                                
    }                                                                             
    res = mgr.request(                                                            
        'GET',                                                                    
        url,                                                                      
        retries=Retry(                                                            
            **retry_opts,                                                         
        ),                                                                        
        timeout=Timeout(                                                          
            connect=3,                                                            
            read=5,                                                               
        ),                                                                        
    )
```
Retry option `connect=1` is ignored. Urllib tries to connect again and again.
Console output:
```
Starting new HTTPS connection (1): 78.155.216.172:443
Incremented Retry for (url='/'): Retry(total=None, connect=1, read=1, redirect=3, status=None)
Retrying (Retry(total=None, connect=1, read=1, redirect=3, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),)': /
Starting new HTTPS connection (2): 78.155.216.172:443
Incremented Retry for (url='/'): Retry(total=None, connect=1, read=1, redirect=3, status=None)
Retrying (Retry(total=None, connect=1, read=1, redirect=3, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),)': /
Starting new HTTPS connection (3): 78.155.216.172:443
Incremented Retry for (url='/'): Retry(total=None, connect=1, read=1, redirect=3, status=None)
Retrying (Retry(total=None, connect=1, read=1, redirect=3, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),)': /
Starting new HTTPS connection (4): 78.155.216.172:443
Incremented Retry for (url='/'): Retry(total=None, connect=1, read=1, redirect=3, status=None)
Retrying (Retry(total=None, connect=1, read=1, redirect=3, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),)': /
Starting new HTTPS connection (5): 78.155.216.172:443
Incremented Retry for (url='/'): Retry(total=None, connect=1, read=1, redirect=3, status=None)
Retrying (Retry(total=None, connect=1, read=1, redirect=3, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),)': /
Starting new HTTPS connection (6): 78.155.216.172:443

... AND SO ON
```


Discussion:
The exception raised here is an [`SSLError`](https://urllib3.readthedocs.io/en/latest/reference/index.html#urllib3.exceptions.SSLError), which subclasses `HTTPError`. In other words, it's not considered to be a connection error, and only `total` can be used here to avoid infinite retries.

The [docs about `total`](https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.retry.Retry) say:

> Set to None to remove this constraint and fall back on other counts. It’s a good idea to set this to some sensibly-high value to account for unexpected edge cases and avoid infinite retry loops.

Your `SSLError` is such an "unexpected edge-case", and it's not covered by other counts. So you need to define a "sensibly-high value" here. If you set `total` to `6` (redirect + connect + read + 1), then your other values will take precedence, but urllib3 won't retry more than six times.

Sorry, using timeouts correctly is hard, but this issue is not a bug in urllib3, so I'll close this issue for now. Feel free to provide more information, if there's a bug somewhere we'll be happy to reopen.
@pquentin 
Docs say "total (int) – Total number of retries to allow. *Takes precedence over other counts*."
You say "So you need to define a "sensibly-high value" here. If you set total to 6 (redirect + connect + read + 1), then your *other values will take precedence*"

Are not these mutually exclusive statements?
Well, anyway it is a bug in design of retrying mechanism because I can't:
* set total to low value like 0 or zero
* AND set redirect to higher value like 3. In this case total will take precedence (according source code and simple test).

In other words, I do not need sensibly-high value for edge cases, I need zero. But I can't use zero because it will ruin other retry counters.
> Are not these mutually exclusive statements?

No, because "other counts" does not include all possible exceptions, while "total" does.

> Well, anyway it is a bug in design of retrying mechanism because I can't [...]

I guess one option would be to add an "other" count, to make sure that all edge cases are covered. You would then set both `total` and `other` to zero. Would that work for you?
Well, if you mean this `other` counter will work for any error not covered by (connect + read + redirect) then, yes, it would work.
In that case I will set `other` to zero and `total` to `None`
Yes, this is what I mean. And yes, set `total` to `None`, not zero. Okay, let's reopen this now that it's a feature request. :)

Resolving patch:
diff --git a/src/urllib3/util/retry.py b/src/urllib3/util/retry.py
--- a/src/urllib3/util/retry.py
+++ b/src/urllib3/util/retry.py
@@ -54,8 +54,7 @@ class Retry(object):
         Total number of retries to allow. Takes precedence over other counts.
 
         Set to ``None`` to remove this constraint and fall back on other
-        counts. It's a good idea to set this to some sensibly-high value to
-        account for unexpected edge cases and avoid infinite retry loops.
+        counts.
 
         Set to ``0`` to fail on the first retry.
 
@@ -96,6 +95,18 @@ class Retry(object):
 
         Set to ``0`` to fail on the first retry of this type.
 
+    :param int other:
+        How many times to retry on other errors.
+
+        Other errors are errors that are not connect, read, redirect or status errors.
+        These errors might be raised after the request was sent to the server, so the
+        request might have side-effects.
+
+        Set to ``0`` to fail on the first retry of this type.
+
+        If ``total`` is not set, it's a good idea to set this to 0 to account
+        for unexpected edge cases and avoid infinite retry loops.
+
     :param iterable method_whitelist:
         Set of uppercased HTTP method verbs that we should retry on.
 
@@ -166,6 +177,7 @@ def __init__(
         read=None,
         redirect=None,
         status=None,
+        other=None,
         method_whitelist=DEFAULT_METHOD_WHITELIST,
         status_forcelist=None,
         backoff_factor=0,
@@ -180,6 +192,7 @@ def __init__(
         self.connect = connect
         self.read = read
         self.status = status
+        self.other = other
 
         if redirect is False or total is False:
             redirect = 0
@@ -204,6 +217,7 @@ def new(self, **kw):
             read=self.read,
             redirect=self.redirect,
             status=self.status,
+            other=self.other,
             method_whitelist=self.method_whitelist,
             status_forcelist=self.status_forcelist,
             backoff_factor=self.backoff_factor,
@@ -348,7 +362,14 @@ def is_retry(self, method, status_code, has_retry_after=False):
 
     def is_exhausted(self):
         """ Are we out of retries? """
-        retry_counts = (self.total, self.connect, self.read, self.redirect, self.status)
+        retry_counts = (
+            self.total,
+            self.connect,
+            self.read,
+            self.redirect,
+            self.status,
+            self.other,
+        )
         retry_counts = list(filter(None, retry_counts))
         if not retry_counts:
             return False
@@ -386,6 +407,7 @@ def increment(
         read = self.read
         redirect = self.redirect
         status_count = self.status
+        other = self.other
         cause = "unknown"
         status = None
         redirect_location = None
@@ -404,6 +426,11 @@ def increment(
             elif read is not None:
                 read -= 1
 
+        elif error:
+            # Other retry?
+            if other is not None:
+                other -= 1
+
         elif response and response.get_redirect_location():
             # Redirect retry?
             if redirect is not None:
@@ -432,6 +459,7 @@ def increment(
             read=read,
             redirect=redirect,
             status=status_count,
+            other=other,
             history=history,
         )
 


Test patch:
diff --git a/test/test_retry.py b/test/test_retry.py
--- a/test/test_retry.py
+++ b/test/test_retry.py
@@ -13,6 +13,7 @@
     MaxRetryError,
     ReadTimeoutError,
     ResponseError,
+    SSLError,
 )
 
 
@@ -83,6 +84,7 @@ def test_retry_default(self):
         assert retry.connect is None
         assert retry.read is None
         assert retry.redirect is None
+        assert retry.other is None
 
         error = ConnectTimeoutError()
         retry = Retry(connect=1)
@@ -97,6 +99,20 @@ def test_retry_default(self):
         assert Retry(0).raise_on_redirect
         assert not Retry(False).raise_on_redirect
 
+    def test_retry_other(self):
+        """ If an unexpected error is raised, should retry other times """
+        other_error = SSLError()
+        retry = Retry(connect=1)
+        retry = retry.increment(error=other_error)
+        retry = retry.increment(error=other_error)
+        assert not retry.is_exhausted()
+
+        retry = Retry(other=1)
+        retry = retry.increment(error=other_error)
+        with pytest.raises(MaxRetryError) as e:
+            retry.increment(error=other_error)
+        assert e.value.reason == other_error
+
     def test_retry_read_zero(self):
         """ No second chances on read timeouts, by default """
         error = ReadTimeoutError(None, "/", "read timed out")