socket.timeout is thrown instead of requests.exceptions.Timeout

When _timeout_ parameter is used, socket operations can thrown _socket.timeout_ exception, which is not converted into _requests.exceptions.Timeout_. This causes users to handle two different types of exceptions, which essentially represent the same error.

Here is corresponding traceback:

```
>>> r=requests.get('http://files.animeblog.ru/manga/Bakuman/Bakuman_vol01.rar', timeout=0.5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/u/vpython/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/Users/u/vpython/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/u/vpython/lib/python2.7/site-packages/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/Users/u/vpython/lib/python2.7/site-packages/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/Users/u/vpython/lib/python2.7/site-packages/requests/adapters.py", line 222, in send
    r.content
  File "/Users/u/vpython/lib/python2.7/site-packages/requests/models.py", line 550, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/Users/u/vpython/lib/python2.7/site-packages/requests/models.py", line 496, in generate
    chunk = self.raw.read(chunk_size)
  File "/Users/u/vpython/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 148, in read
    return self._fp.read(amt)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 561, in read
    s = self.fp.read(amt)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
socket.timeout: timed out
```


Good spot, I'll take a look at this. =)


We don't catch this and urllib3 doesn't try to catch and re-throw this error. If anything, the patch should be submitted upstream to urllib3 and then we won't need changes here.


I believe this creates a leaky abstraction: requests intentionally or unintentionally exposes internal implementation details, which erode the value of abstraction layer on top of urllib3.

From practical standpoint this means that one cannot rely on requests providing consistent and well-defined interface. In particular one would have to handle exceptions that can be thrown by urllib3 and below in addition to requests exceptions, which creates unnecessary dependencies that requests library was meant to abstract.

Regardless of how urllib3 deals socket timeout it would be best for requests to not require users to handle two different exceptions for basically the same error condition. After all requests is HTTP library for human beings.


Yep, @dmalenko is right. We're getting this exact issue too (with the same traceback). I _don't_ think it's simply urllib3's fault, because it's requests that's calling `urllib3.Response.read()` and not handling errors around that call.

Arguably urllib3 should handle socket.timeout or other errors too, and convert them to a urllib3.TimeoutError. But even if it did that, requests would still need to catch that and re-raise a `requests.Timeout`, otherwise the user of requests would still have to know to catch `urllib3.TimeoutError`.

So I think requests should simply catch the socket.timeout / socket.error in `models.Response.iter_content()`. It could also catch `urllib3.TimeoutError` and do the same thing, in case urllib3 fixes their part of this issue.


@benhoyt We're in total agreement. I had this assigned to me but haven't had time to do anything about it, and won't do for a few weeks. We're open to a Pull Request if you'd like to write one, otherwise this is unlikely to happen in the next couple of weeks.


I looked into this a bit. I didn't get too far, but I'll note my observations for the benefit of others. [The documentation](http://docs.python-requests.org/en/latest/user/quickstart/#timeouts) says:

> timeout only effects the connection process itself, not the downloading of the response body.

As seen here, it's not true. In @dmalenko's case, the connection is opened and the headers are sent fast enough, but the server sends data so slowly that recv timeouts. The same thing happens with `iter_content`.

This happens because urllib3 sets the timeout for the underlying httplib socket to catch connection timeouts. After the connection is established, the timeout stays set and affects the recv calls, but the errors aren't catched anymore.


Thanks, @miikka. I notice that part of the documentation is inside a "Note:" -- to me it looks like that's basically an implementation detail. I'd think when I specify a timeout I want it to be a timeout on the whole thing, so maybe in this case the documentation can simply be changed to match reality?

Note that I don't think the above affects the issue in any way -- we still want this to be a `requests.Timeout`.


Yeah @miikka, you're right, it isn't true. Sometimes.

We confusingly use `timeout` in two different ways. For non-chunked requests, we use `timeout` as a parameter on the underlying HTTP connection, which applies it just as httplib would. For chunked uploads, we use it as a timeout on getting the connection from the connection pool, and then apply no timeout to the connection itself (see #1422). I'll look into getting this mess sorted out, but if you want to try to tackle it yourselves please do. =)


I am still having this problem. Is this because it has not been corrected or because I am using an old version of urllib3 or requests library?


Which exact problem are you seeing?


```
  File "run_script.py", line 152, in <module>
    j = requests.get(my_url, params = mbc_params, timeout=10.00)
  File "/home/nhssr/.PythonVirtualEnvironment/lib/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/home/nhssr/.PythonVirtualEnvironment/lib/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/nhssr/.PythonVirtualEnvironment/lib/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/nhssr/.PythonVirtualEnvironment/lib/requests/sessions.py", line 596, in send
    r.content
  File "/home/nhssr/.PythonVirtualEnvironment/lib/requests/models.py", line 694, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/home/nhssr/.PythonVirtualEnvironment/lib/requests/models.py", line 627, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/home/nhssr/.PythonVirtualEnvironment/lib/requests/packages/urllib3/response.py", line 240, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/home/nhssr/.PythonVirtualEnvironment/lib/requests/packages/urllib3/response.py", line 187, in read
    data = self._fp.read(amt)
  File "/apollo/package/local_1/Linux-2.6c2.5-x86_64/Python27/Python27-309.0-0/lib/python2.7/httplib.py", line 543, in read
    return self._read_chunked(amt)
  File "/apollo/package/local_1/Linux-2.6c2.5-x86_64/Python27/Python27-309.0-0/lib/python2.7/httplib.py", line 585, in _read_chunked
    line = self.fp.readline(_MAXLINE + 1)
  File "/apollo/package/local_1/Linux-2.6c2.5-x86_64/Python27/Python27-309.0-0/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
socket.timeout: timed out
```


I am catching this:
  except requests.exceptions.Timeout:


But it does not seem to be enough. I am not using any of the low level libraries explicitly. I was hoping I could simple use the requests library to handle all my HTTP requests. Now I am trying to see how I can catch that exception. I guess I am gonna have to import the socket library as well or maybe even urllib. But it would seem to be little bit contrary to the intent of the requests library to be a wrapper for those other libraries that I have to call on them to handle this exception.


You seem to have misunderstood this bug. It is still open because it remains broken.


I was wondering if that's the case. Thank your for your response. I'll try 

import socket
..
except socket.timeout:
..


Thanks @miikka for the explanation that let me reliably reproduce this error using HTTPBin's Drip endpoint:

```
Python 2.7.6 (v2.7.6:3a1db0d2747e, Nov 10 2013, 00:42:54) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__version__
'2.2.1'
>>> r = requests.get('http://httpbin.org/drip?numbytes=5&duration=5', timeout=0.2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/adapters.py", line 394, in send
    r.content
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/models.py", line 679, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/models.py", line 616, in generate
    decode_content=True):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 236, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 183, in read
    data = self._fp.read(amt)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 543, in read
    return self._read_chunked(amt)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 585, in _read_chunked
    line = self.fp.readline(_MAXLINE + 1)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
socket.timeout: timed out
```

I'm working around as @kaisarea suggested, catching both the `RequestException` exception and the `socket.timeout`.


this is fixed? we can trust requests won't throw socket.timeout or any other exceptions besides http://docs.python-requests.org/en/latest/user/quickstart/#errors-and-exceptions?


No, you cannot trust that. For example, requests may throw IOError, or ValueError, or a number of other things that are not in that list.

However, requests _should not_ throw naked versions of exceptions from urllib3, httplib, or socket. If it does, we consider it a bug. I cannot 100% promise that it doesn't, because we've missed points in the past and probably will again. Bugs happen. =(


Ok @Lukasa , thanks! I need to implement a "retry" logic and I can't use urllib3 retries because I need more control. So I needed to know which exceptions should I catch, but I'll catch `Exception` anyway and catch others to do custom stuff for each one.


We also see `socket.error` connection reset by peer from the iter content call 


@sbv-csis What version of requests, and where did you install it from?


@Lukasa We use 2.8.1 from https://pypi.python.org/pypi/requests


@sbv-csis Can you post any code that will reproduce this? And the traceback?


In essence it's

```
with open(filename, 'wb') as f:
    r = request.get(amazons3url, stream=True)
    for chunk in r.iter_content(128 * 1024):
        f.write(chunk)
```

The traceback is:

```
    File "/opt/python2.7/lib/python2.7/site-packages/saws/s3.py", line 185, in download
      for block in response.iter_content(block_size):
    File "/opt/python2.7/lib/python2.7/site-packages/requests/models.py", line 657, in generate
      for chunk in self.raw.stream(chunk_size, decode_content=True):
    File "/opt/python2.7/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 326, in stream
      data = self.read(amt=amt, decode_content=decode_content)
    File "/opt/python2.7/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 282, in read
      data = self._fp.read(amt)
    File "/opt/python2.7/lib/python2.7/httplib.py", line 612, in read
      s = self.fp.read(amt)
    File "/opt/python2.7/lib/python2.7/socket.py", line 384, in read
      data = self._sock.recv(left)
    File "/opt/python2.7/lib/python2.7/ssl.py", line 734, in recv
      return self.read(buflen)
    File "/opt/python2.7/lib/python2.7/ssl.py", line 621, in read
      v = self._sslobj.read(len or 1024)
  socket.error: [Errno 104] Connection reset by peer
```


Yup, so this is a urllib3 bug. Thanks for spotting it, I've raised it as shazow/urllib3#756.


cheers! Thanks for the response.