redirected request with a wrong host header

for example:
r = requests.get(url, allow_redirects=True, headers={"Host":"www.baidu.com"})

first response is:
302 with location: http://www.abc.com/index.html

then the followed request is also with the Host of www.baidu.com, for 302 location is a abs Request-URI, should not the request with the host of www.abc.com?


Hi @wttang 

For the most part, we actively discourage users from specifying they're own `Host` header. Because we discourage users from handling their own `Host` header, we do not remove it on redirect because 99% of the time there's no need for us to do so. By specifying this header yourself and not handling redirects yourself, we continue using the `Host` header.

We very well could just always attempt to remove the `Host` header but I'm -0.5 on this matter because it will almost certainly be removed by someone in the future regardless of how well we comment the surrounding lines of code. If you're making requests that require you to override things like this from requests, you should be prepared to also be handling redirects yourself.


I don't think we can remove the host header on redirect. If the user specifies the host header they're suggesting that they know more than we do, and in that situation we shouldn't override it.


Yeah. I agree with @Lukasa. By specifying a custom header like this, you're basically asserting that you know what you want to happen and if something unexpected happens then you have to remedy it, even if that means having to manually handle redirects


the behavior of requests is not the same with a modern broswer.  just try www2.baidu.com,
I think the behavior of chrome or firefox is more reasonable


@wttang It's not at all clear to me what you mean by that. www2.baidu.com works fine. Chrome or Firefox don't let you choose what host header you set, so they don't have _any_ behaviour in this situation.


@Lukasa ok, i got it.  you could try with curl -H"Host: xxx.yyy.com" url.  (curl/7.19.7)
the behavior of curl is also use new-host header, if a 302 return, not user set one.


@wttang Yeah, here we have a philosophical difference with cURL.

When you override our default headers we assume that you know better than us. This affects all kinds of things. For example, when you set the `Host` header we change the way we handle cookies. However, the API doesn't have the flexibility for you to be able to set the boundary for your pre-existing knowledge. You can't say "the Host header should be this value, but only if we aren't redirected off-host, and use the old host header for cookie values, but use the new host header to validate the TLS certificate". That's just too complicated to specify.

So instead we just pick which of those we think will apply and _always_ apply it. This means you can easily work out what a redirect will look like from your original command line, we don't mutate it under your feet to remove the special knowledge. I value that kind of transparency.

The other option is to say "sure, you know more than us, but the second you get redirected you don't". That will work fine too. However, it pushes the problem onto different users. Users who like our current behaviour ("leave Host set as-is") will need to change their code, while you won't.

I don't think either position is correct or right (though if @bagder has time I'd love his input on why cURL chose its behaviour), and in that sort of situation I will always prefer the status quo.

I think this is "working as intended".


First out, we (curl) picked our behavior many moons ago so this is something that has basically worked like this "forever" (since Sep 2004 to be exact). And to be clear: curl actually checks if the redirect is to a new host name and only then it disables the custom one, so if the redirect goes to the same host name as the original one it still allows the custom Host:.

The reason I went with this is that after a redirect that you allow curl to follow, you can basically not get the custom header done correctly if it goes to another host name, and it is causing harm more often than good to allow the custom one after a redirect-to-other-host.

In general in curl land, we've opted to "sensible defaults" and keeping it fairly simple when it comes to behaviors on followed redirects since we also make it very easy for users to do the entire redirect following "manually" and then they can opt to do whatever they please in all requests.


If i request http://a.com/ with Host header 'a.com' and receive a redirect to http://www.a.com/, requests then issues a request to the server hosting www.a.com while specifying the host header 'a.com'. This creates an infinite redirect.

In what circumstance does this behavior work as intended? I can't see it solving anything if it maintains the Host header during a redirect to an entirely different domain. I've never seen another library/program which does this. This behavior makes no sense.


@dieselmachine it's working as intended because users are not supposed to set the Host header and if they do then they need to understand that there are going to be big differences in behaviour in certain cases.


Curl's behavior described in comment https://github.com/kennethreitz/requests/issues/2366#issuecomment-67457623 is very sensible thus what I would expect from _HTTP library for humans_. 


For reference, this is how easy it is to solve the problem with urllib2.

```
req = urllib2.Request(url)
req.add_unredirected_header('Host', host)
rsp = urllib2.urlopen(req)
```

It is impossible to properly handle this situation in requests without disabling redirects and manually processing them by reading the location header and then issuing another request.

When would a person ever want to send the original Host header to the host contained in the redirect location? Web browsers (made "for humans") will not send the same Host header after a redirect, because it would break things. I would expect a library "for humans" to handle things similarly.

When would the default behavior NOT break something, is what I'm wondering. I can't see a single situation where this behavior would make things work.


I'm increasingly inclined to agree with this logic. I'm struggling to see what pain we'd introduce by doing this for 3.0.0.