requests.head redirect timeout (on only linux environments?)

On OSX I can run this fine:

``` python
# python2.6
Python 2.6.7 (r267:88850, Jul 31 2011, 19:30:54) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.head("http://feedproxy.google.com/~r/Techcrunch/~3/1P88-kHZWGs/", allow_redirects=True, timeout=10)
>>> r
<Response [200]>
>>> r.history
[<Response [301]>]
>>> requests.__version__
'0.12.1'
```

However, the same request on an Ubuntu box times out - in multiple network environments:

``` python
# python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.head("http://feedproxy.google.com/~r/Techcrunch/~3/1P88-kHZWGs/", allow_redirects=True, timeout=10)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/envs/rdb/lib/python2.6/site-packages/requests/api.py", line 74, in head
    return request('head', url, **kwargs)
  File "/home/ubuntu/envs/rdb/lib/python2.6/site-packages/requests/api.py", line 40, in request
    return s.request(method=method, url=url, **kwargs)
  File "/home/ubuntu/envs/rdb/lib/python2.6/site-packages/requests/sessions.py", line 229, in request
    r.send(prefetch=prefetch)
  File "/home/ubuntu/envs/rdb/lib/python2.6/site-packages/requests/models.py", line 624, in send
    self._build_response(r)
  File "/home/ubuntu/envs/rdb/lib/python2.6/site-packages/requests/models.py", line 227, in _build_response
    r.content  # Consume socket so it can be released
  File "/home/ubuntu/envs/rdb/lib/python2.6/site-packages/requests/models.py", line 782, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/home/ubuntu/envs/rdb/lib/python2.6/site-packages/requests/models.py", line 728, in generate
    chunk = self.raw.read(chunk_size)
  File "/home/ubuntu/envs/rdb/lib/python2.6/site-packages/requests/packages/urllib3/response.py", line 146, in read
    return self._fp.read(amt)
  File "/usr/lib/python2.6/httplib.py", line 518, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.6/httplib.py", line 551, in _read_chunked
    line = self.fp.readline()
  File "/usr/lib/python2.6/socket.py", line 397, in readline
    data = recv(1)
socket.timeout: timed out
>>> requests.__version__
'0.12.1'
```

Looks like some sort of chunk reading issue?


Hmm, interesting. Thanks for the URL, we'll have to dive in further.


DNR for me in Fedora with Python 2.7.3 and 3.2.1. Interesting.


DNR in Ubuntu Precise with Python 2.7.3 as well.


Here's a quick workaround for anyone else who happens to encounter this before a fix:

``` python

import requests
from requests.utils import requote_uri
from urlparse import urlparse, urljoin

def unroll_url(url, max_redirects=30, timeout=None, history=None):
    if history is None:
        history = set()

    if max_redirects <= 0:
        raise requests.exceptions.TooManyRedirects()

    if url in history:
        raise requests.exceptions.RequestException("Redirect loop detected.")

    history.add(url)

    r = requests.head(url, timeout=timeout, allow_redirects=False)
    if r.status_code in set([301, 302, 303, 307]) and 'location' in r.headers:
        redirect_url = r.headers['location']

        # Handle redirection without scheme (see: RFC 1808 Section 4)
        if redirect_url.startswith('//'):
            parsed_original_url = urlparse(url)
            redirect_url = '%s:%s' % (parsed_original_url.scheme, url)

        # Facilitate non-RFC2616-compliant 'location' headers
        # (e.g. '/path/to/resource' instead of 'http://domain.tld/path/to/resource')
        if not urlparse(redirect_url).netloc:
            redirect_url = urljoin(url, requote_uri(redirect_url))

        return unroll_url(redirect_url,
                          max_redirects = max_redirects - 1,
                          timeout = timeout,
                          history = history)

    return r.url
```

[Edit: Updated this to include non-absolute URL support, infinite loop redirect checking.]


I was able to repro on Ubuntu 10.04

``` python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.head("http://feedproxy.google.com/~r/Techcrunch/~3/1P88-kHZWGs/", allow_redirects=True, timeout=10)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/david/.virtualenvs/testenv/lib/python2.6/site-packages/requests/api.py", line 74, in head
  return request('head', url, **kwargs)
File "/home/david/.virtualenvs/testenv/lib/python2.6/site-packages/requests/api.py", line 40, in request
  return s.request(method=method, url=url, **kwargs)
File "/home/david/.virtualenvs/testenv/lib/python2.6/site-packages/requests/sessions.py", line 229, in request
  r.send(prefetch=prefetch)
File "/home/david/.virtualenvs/testenv/lib/python2.6/site-packages/requests/models.py", line 624, in send
  self._build_response(r)
  File "/home/david/.virtualenvs/testenv/lib/python2.6/site-packages/requests/models.py", line 227, in _build_response
  r.content  # Consume socket so it can be released
File "/home/david/.virtualenvs/testenv/lib/python2.6/site-packages/requests/models.py", line 782, in content
  self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
File "/home/david/.virtualenvs/testenv/lib/python2.6/site-packages/requests/models.py", line 728, in generate
  chunk = self.raw.read(chunk_size)
File "/home/david/.virtualenvs/testenv/lib/python2.6/site-packages/requests/packages/urllib3/response.py", line 146, in read
  return self._fp.read(amt)
File "/usr/lib/python2.6/httplib.py", line 518, in read
  return self._read_chunked(amt)
File "/usr/lib/python2.6/httplib.py", line 551, in _read_chunked
  line = self.fp.readline()
File "/usr/lib/python2.6/socket.py", line 397, in readline
  data = recv(1)
  socket.timeout: timed out
  >>> requests.__version__
  '0.12.1'
```


DNR for me in Ubuntu Lucid (10.04) with Python 2.6.7:

```

$ python
Python 2.6.7 (r267:88850, Dec  2 2011, 20:27:26) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.head("http://feedproxy.google.com/~r/Techcrunch/~3/1P88-kHZWGs/", allow_redirects=True, timeout=10)
>>> r.status_code
200
>>> requests.__version__
'0.12.1'
```

I'm starting to believe it's specific to 2.6.5.


(I guess Lucid officially packages 2.6.5, and I have 2.6.7 via a separate package repository.)