requests sends HTTP auth details in Host header

I noticed that the Host header of my http requests generated by `requests` have the HTTP auth credentials, like this:

```
Host: admin:viG7Xs@google.com
```

(This is when I specified them in the URL.)

Is this standard? It looks weird to me.


Wow, that's wrong! I don't have my laptop right now so I can't investigate, but can anyone repro with the latest version of requests?


``` pycon
>>> import requests
>>> r = requests.get('https://username:foo@httpbin.org/get')
>>> r.json()['headers']
{'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'python-requests/2.4.1 CPython/3.3.2 Darwin/13.3.0', 'Authorization': 'Basic dXNlcm5hbWU6Zm9v', 'X-Request-Id': 'fb06edf3-14c8-42ad-b3c5-fefa801ddd9a', 'Connection': 'close', 'Host': 'httpbin.org'}
>>> r.request.headers
{'Authorization': 'Basic dXNlcm5hbWU6Zm9v', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive', 'User-Agent': 'python-requests/2.4.1 CPython/3.3.2 Darwin/13.3.0'}
```

Note that this doesn't seem to be reproducable. @cool-RR can you share the code and requests version that caused this?


Ian, can you please check using something like Fiddler the true Host of the
request?

On Thu, Sep 25, 2014 at 6:44 PM, Ian Cordasco notifications@github.com
wrote:

> > > > import requests>>> r = requests.get('https://username:foo@httpbin.org/get')>>> r.json()['headers']{'Accept': '_/_', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'python-requests/2.4.1 CPython/3.3.2 Darwin/13.3.0', 'Authorization': 'Basic dXNlcm5hbWU6Zm9v', 'X-Request-Id': 'fb06edf3-14c8-42ad-b3c5-fefa801ddd9a', 'Connection': 'close', 'Host': 'httpbin.org'}>>> r.request.headers{'Authorization': 'Basic dXNlcm5hbWU6Zm9v', 'Accept': '_/_', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive', 'User-Agent': 'python-requests/2.4.1 CPython/3.3.2 Darwin/13.3.0'}
> 
> Note that this doesn't seem to be reproducable. @cool-RR
> https://github.com/cool-RR can you share the code and requests version
> that caused this?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2247#issuecomment-56838438
> .


@cool-RR I'll do that when you can give me the version and the code. The true host received by httpbin is what is printed on the 4th line. Also, it would look as though we're stripping the auth out and creating an Authorization header as we should be, before we even pass the URL on to urllib3.


Python 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit (AMD64)]
on win32
Type "copyright", "credits" or "license()" for more information.
DreamPie 1.2.1

> > > import requests
> > > requests.**version**
> > > 0: '2.4.1'
> > > requests.get('http://yo:pw@google.com')
> > > 1: <Response [400]>

Request in Fiddler raw view:

GET http://yo:pw@google.com/ HTTP/1.1
Host: yo:pw@google.com
Connection: keep-alive
Accept: _/_
Accept-Encoding: gzip, deflate
Authorization: Basic eW86cHc=
User-Agent: python-requests/2.4.1 CPython/2.7.6 Windows/7

Also Fiddler shows another warning about the port not being specified. Also
should the credentials show in the first line too?

On Thu, Sep 25, 2014 at 6:47 PM, Ian Cordasco notifications@github.com
wrote:

> @cool-RR https://github.com/cool-RR I'll do that when you can give me
> the version and the code. The true host received by httpbin is what is
> printed on the 4th line. Also, it would look as though we're stripping the
> auth out and creating an Authorization header as we should be, before we
> even pass the URL on to urllib3.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2247#issuecomment-56838948
> .


This bug is quite clearly proxy specific. =) Our generating of Host headers and Request URLs is clearly wrong there. 


Sorry, I'm confused. Why is this clearly proxy-specific?

On Thu, Sep 25, 2014 at 7:15 PM, Cory Benfield notifications@github.com
wrote:

> This bug is quite clearly proxy specific. =) Our generating of Host
> headers and Request URLs is clearly wrong there.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2247#issuecomment-56843117
> .


The full request URL is only sent when sent to a proxy. I presume you have your `$HTTP_PROXY` environment variable set to Fiddler. We've detected that and used our proxy-specific handling. This is why @sigmavirus24 couldn't reproduce your behaviour. 


I'm seeing this with mitmproxy (and using explicity proxy settings). If I use netcat then I see this:

![screen shot 2014-09-25 at 11 26 45 am](https://cloud.githubusercontent.com/assets/240830/4408147/d623579a-44d0-11e4-9147-0a0a7b700612.png)

So I can confirm this is a proxy-specific issue.


I suggest comparing the output from the proxy-specific request to the
normal one to find any other diversions (like maybe port number missing or
anything else)

On Thu, Sep 25, 2014 at 7:27 PM, Ian Cordasco notifications@github.com
wrote:

> I'm seeing this with mitmproxy (and using explicity proxy settings). If I
> use netcat then I see this:
> 
> [image: screen shot 2014-09-25 at 11 26 45 am]
> https://cloud.githubusercontent.com/assets/240830/4408147/d623579a-44d0-11e4-9147-0a0a7b700612.png
> 
> So I can confirm this is a proxy-specific issue.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2247#issuecomment-56844956
> .


I blame https://github.com/shazow/urllib3/blob/f28732e73e9af6656d32584221ccc8b4bdb83920/urllib3/poolmanager.py#L242
(Investigating, there was a bugreport about this in the past)


That explains why it's in the Host header, but not why it's in the request URL.

Both are likely to be my fault though, that's code I wrote.


Na, I was wrong, I mixed up the auth info and the port


https://hg.python.org/cpython/file/bfdb995e8d7d/Lib/http/client.py#l1061
`urlsplit` from the stdlib includes the auth stuff in the netloc.
We should probably remove the auth stuff from the url before passing the URL down to httplib.


We have to remove the auth info here:
https://github.com/kennethreitz/requests/blob/a718a81d273503bd2ffae8e6cb036a8516eb426a/requests/adapters.py#L273


Did we decide if/where the bug was in urllib3? I'd like to have it fixed in urllib3 also.


I assumed, urllib3 expected users to not include auth stuff in URLs but set them in the headers


Mmm, I guess that's one interpretation. Not sure I'd expect that, at least with `proxy_from_url`.

But either way, I'm still confused about how auth was getting into `Url.netloc`? Was it all because of non-`://` schemas?


It was in the URL which was passed as is down to httplib which used it in the request line and Host header


To clarify: It was never in Url.netloc, this was a mistake on my part. Urllib3 never touches the URL in the codepath used by requests


Ah k, thanks.


Probably redundant, but here's an example where I faced the problem: http://stackoverflow.com/questions/26109264/pip-proxy-authentication-and-not-supported-proxy-scheme