shazow/urllib3 — Issue #32: Provide a way for callers to see HTTP exceptions
Author: brandon-rhodes
Created: 2012-01-21 01:59:43+00:00
State: open

Whether a `request()` or `urlopen()` call is given `retries=0` or `retries=`, the exception returned on most connection and network failures is a `MaxRetryError`. This prevents client code from seeing the “real error” that killed the connection and, therefore, clients based on `urllib3` cannot, say, adjust their behavior based on whether the problem is any of the various errors at the HTTP level, or whether the problem is some specific socket error.

At least three solutions are possible, and I defer to the maintainers to decide which is the most `urllib3`-ish.

First, it could be judged a design error to have introduced `MaxRetryError` in the first place, and the final attempt's exception could be allowed to make it through to the client (some would want to throw in a `retry_count` attribute to add a bit of information about what happened, but I am not sure that that is advisable). The `except` clause near line 350 of `connectionpool.py` would then look something like:
```python
except (HTTPException, SocketError), e:
# Connection broken, discard. It will be replaced next _get_conn().
conn = None
# If this was the last try, let the caller see the real error.
if not retries:
raise
```
Other adjustments to the code might accomplish the same thing — a larger refactoring could eliminate even making the final attempt inside of an `except:`, for example — but this is the simplest approach I could find for this first approach.

Second, the `MaxRetryError` could be marked up with a list of failures. This would be a bit tricky, since there are finicky problems with keeping stack traces around, but at least the exception object from each failure could be kept around — shorn of its stack trace — and delivered back with the `MaxRetryError`. This would even make it clear whether the problem was an actual network death of the HTTP protocol, or whether something like a redirect loop had kept the request going through too many attempts — and thus much more information would be available for the client to determine what had happened.

Finally, `retries=None` could be a special signal that the bare exception should be returned. This would maintain (so far as I can see) 100% compatibility with the current code base. The `ConnectionPool.urlopen()` function needs two or three lines adjusted so that the `None` value avoids offending the various `if` statements, but with minimal intrusion the change lets someone like me — whose application, alas, cares quite deeply what mode of failure an HTTP request encounters — take advantage of `urllib3` but without breaking existing code that might always expect the `MaxRetryError` exception.

Of course, other approaches might also be possible that I have not thought of.

I very much like the quality of code that I am seeing in `urllib3` and, if I can only get the actual exceptions back that it encounters, these already-written connection pools are going to save me a lot of work. Thanks!