Fixed a bug that caused infinite redirections if the response contains a...

... Location header pointing to itself (try it against http://google.com/robots.txt to test it)


Thanks for this @MarioVilas! I agree that this would fix a bug if this happens, though I can't reproduce this with Google's robots.txt. I have to admit that this is an unlikely bug though: surely the associated page would just never work for any browser?


I've only seen it with Google's robots.txt, but only if I try to access it
using a wrong URL (notice the missing "www." in the beginning). I don't
know if Google is sending the wrong headers, or if Requests (or urllib3) is
keeping the old Location header for some reason.

To be honest I didn't look much into the cause of the bug, I needed
something fixed for tomorrow and this did the trick :) but I thought you
should know anyhow.

On Tue, Aug 20, 2013 at 2:45 PM, Cory Benfield notifications@github.comwrote:

> Thanks for this @MarioVilas https://github.com/MarioVilas! I agree that
> this would fix a bug if this happens, though I can't reproduce this with
> Google's robots.txt. I have to admit that this is an unlikely bug though:
> surely the associated page would just never work for any browser?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/pull/1544#issuecomment-22942116
> .

## 

“There's a reason we separate military and the police: one fights the enemy
of the state, the other serves and protects the people. When the military
becomes both, then the enemies of the state tend to become the people.”


@MarioVilas See, that's my point, I can't reproduce it with `robots.txt` regardless of which URL I use. What version of Requests are you using? (`requests.__version__` will tell you)


Actually, doubling back to our code, Requests should throw a `TooManyRedirects` exception. I think that's the correct behaviour here (it's in line with what browsers do).


I've tried an old one and the most recent one (checked out of github), in both I see the bug.

This small script reproduces the issue on my machine:

from requests import Session
session = Session()
resp = session.request(
    method  = "GET",
    url     = "http://google.com/robots.txt",
    headers = {'host': 'google.com'},
    verify  = False,
    stream  = True,
    timeout = 10.0,
    allow_redirects = True,
)


You're right. In fact I don't think Google is sending a malformed response
at all, my patch just hides the real cause of the bug:

$ curl -v http://google.com/robots.txt > /dev/null
- About to connect() to google.com port 80 (#0)
-   Trying 173.194.41.238...
- connected
- Connected to google.com (173.194.41.238) port 80 (#0)
  > GET /robots.txt HTTP/1.1
  > User-Agent: curl/7.28.1
  > Host: google.com
  > Accept: _/_
  >
  < HTTP/1.1 301 Moved Permanently
  < Location: http://www.google.com/robots.txt
  < Content-Type: text/html; charset=UTF-8
  < X-Content-Type-Options: nosniff
  < Date: Sat, 17 Aug 2013 13:28:09 GMT
  < Expires: Mon, 16 Sep 2013 13:28:09 GMT
  < Cache-Control: public, max-age=2592000
  < Server: sffe
  < Content-Length: 229
  < X-XSS-Protection: 1; mode=block
  < Alternate-Protocol: 80:quic

$ curl -v http://www.google.com/robots.txt > /dev/null
- About to connect() to www.google.com port 80 (#0)
-   Trying 173.194.41.240...
- connected
- Connected to www.google.com (173.194.41.240) port 80 (#0)
  > GET /robots.txt HTTP/1.1
  > User-Agent: curl/7.28.1
  > Host: www.google.com
  > Accept: _/_
  >
  < HTTP/1.1 200 OK
  < Vary: Accept-Encoding
  < Content-Type: text/plain
  < Last-Modified: Fri, 09 Aug 2013 11:32:07 GMT
  < Date: Sat, 17 Aug 2013 13:28:23 GMT
  < Expires: Sat, 17 Aug 2013 13:28:23 GMT
  < Cache-Control: private, max-age=0
  < X-Content-Type-Options: nosniff
  < Server: sffe
  < X-XSS-Protection: 1; mode=block
  < Alternate-Protocol: 80:quic
  < Transfer-Encoding: chunked
  <


So are you actually getting a redirect loop here? If not, I think there's no problem.


Why are you setting the Host header yourself?


There is no redirect loop, Google works correctly. This only happens with Requests when the Host header is set and the redirection takes you to another host.

That's how I got to reproduce the problem. In my full program I've got a bunch more headers set. Now that I know without it the bug can't be triggered, I'll just filter the host header before calling Requests.


Closing the pull request because the patch is wrong, the problem is somewhere else.


FWIW, Requests should set the Host header correctly itself, you shouldn't need to do it. =)