"407 Proxy Authentication Required" after redirect with load balanced authenticated proxies

I'm having a problem I traced down to requests as the equivalent curl command works. I am running HAProxy on `localhost:5555` to load balance requests to a pool of authenticated backend http proxies as explained [here](http://serverfault.com/questions/386431/authenticated-proxies-on-haproxy-load-balancer). It works fine for fetching urls without redirection, as shown below using [httpie](https://github.com/jakubroztocil/httpie) (same behaviour observed using requests directly from the shell):

```
$ http 'http://httpbin.org/get' --proxy http:http://localhost:5555 --headers --debug 
HTTPie 0.9.2
HTTPie data: /home/gsakkis/.httpie
Requests 2.5.3
Pygments 2.0.2
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] linux2

>>> requests.request({'allow_redirects': False,
 'auth': None,
 'cert': None,
 'data': OrderedDict(),
 'files': DataDict(),
 'headers': {'User-Agent': 'HTTPie/0.9.2'},
 'method': 'get',
 'params': ParamsDict(),
 'proxies': {u'http': u'http://localhost:5555'},
 'stream': True,
 'timeout': 30,
 'url': u'http://httpbin.org/get',
 'verify': True})

HTTP/1.0 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 269
Content-Type: application/json
Date: Tue, 10 Mar 2015 10:38:37 GMT
Server: nginx
Via: 1.0 fl291 (squid/3.1.10)
X-Cache: MISS from fl291
X-Cache-Lookup: MISS from fl291:80
```

If I fetch a url that redirects without actually redirecting it still works, returning 302:

```
$ http 'http://httpbin.org/redirect-to?url=http://example.com/' --proxy http:http://localhost:5555 --headers --debug
HTTPie 0.9.2
HTTPie data: /home/gsakkis/.httpie
Requests 2.5.3
Pygments 2.0.2
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] linux2

>>> requests.request({'allow_redirects': False,
 'auth': None,
 'cert': None,
 'data': OrderedDict(),
 'files': DataDict(),
 'headers': {'User-Agent': 'HTTPie/0.9.2'},
 'method': 'get',
 'params': ParamsDict(),
 'proxies': {u'http': u'http://localhost:5555'},
 'stream': True,
 'timeout': 30,
 'url': u'http://httpbin.org/redirect-to?url=http://example.com/',
 'verify': True})

HTTP/1.0 302 Moved Temporarily
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Tue, 10 Mar 2015 10:41:00 GMT
Location: http://example.com/
Server: nginx
Via: 1.0 mf5 (squid/3.1.10)
X-Cache: MISS from mf5
X-Cache-Lookup: MISS from mf5:80
```

However attempting to follow the redirect fails with 407:

```
$ http 'http://httpbin.org/redirect-to?url=http://example.com/' --proxy http:http://localhost:5555 --headers --follow --debug
HTTPie 0.9.2
HTTPie data: /home/gsakkis/.httpie
Requests 2.5.3
Pygments 2.0.2
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] linux2

>>> requests.request({'allow_redirects': True,
 'auth': None,
 'cert': None,
 'data': OrderedDict(),
 'files': DataDict(),
 'headers': {'User-Agent': 'HTTPie/0.9.2'},
 'method': 'get',
 'params': ParamsDict(),
 'proxies': {u'http': u'http://localhost:5555'},
 'stream': True,
 'timeout': 30,
 'url': u'http://httpbin.org/redirect-to?url=http://example.com/',
 'verify': True})

HTTP/1.0 407 Proxy Authentication Required
Connection: keep-alive
Content-Length: 4
Content-Type: text/html
Date: Tue, 10 Mar 2015 10:42:02 GMT
Mime-Version: 1.0
Proxy-Authenticate: Basic realm="login"
Server: squid/3.1.10
Via: 1.0 mf5 (squid/3.1.10)
X-Cache: MISS from mf5
X-Cache-Lookup: NONE from mf5:80
X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
```

The equivalent curl command seems to work fine though:

```
$ curl 'http://httpbin.org/redirect-to?url=http://example.com/' -s -L -x localhost:5555 -v -o /dev/null
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5555 (#0)
> GET http://httpbin.org/redirect-to?url=http://example.com/ HTTP/1.1
> User-Agent: curl/7.35.0
> Host: httpbin.org
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 302 Moved Temporarily
< Server: nginx
< Date: Tue, 10 Mar 2015 10:44:56 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 0
< Location: http://example.com/
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< X-Cache: MISS from fl291
< X-Cache-Lookup: MISS from fl291:80
< Via: 1.0 fl291 (squid/3.1.10)
* HTTP/1.0 connection set to keep alive!
< Connection: keep-alive
< 
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://example.com/'
* Hostname was found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5555 (#1)
> GET http://example.com/ HTTP/1.0
> User-Agent: curl/7.35.0
> Host: example.com
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Accept-Ranges: bytes
< Cache-Control: max-age=604800
< Content-Type: text/html
< Date: Tue, 10 Mar 2015 10:44:57 GMT
< ETag: "359670651"
< Expires: Tue, 17 Mar 2015 10:44:57 GMT
< Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
< Server: ECS (cpm/F9D5)
< X-Cache: HIT
< x-ec-custom-error: 1
< Content-Length: 1270
< X-Cache: MISS from bv400
< X-Cache-Lookup: MISS from bv400:80
< Via: 1.0 bv400 (squid/3.1.10)
* HTTP/1.0 connection set to keep alive!
< Connection: keep-alive
< 
{ [data not shown]
* Connection #1 to host localhost left intact
```


Can you use tcpdump to show the HTTP packets flowing between your HAProxy instance and requests? Is this something you're familiar with doing?


I have played with it occasionally but I'm far from an expert. Any particular command line I could try?


Try `tcpdump -nnvvXSs 1514 -i lo port 5555` will be a good start, assuming you have a `lo` interface.


Thanks; there you go: http://pastebin.com/raw.php?i=dRiFnf2c


So that leaves us with the following message flow:

```
GET http://httpbin.org/redirect-to?url=http://example.com/ HTTP/1.1
Host: httpbin.org
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: HTTPie/0.9.2


HTTP/1.0 302 Moved Temporarily
Server: nginx
Date: Tue, 10 Mar 2015 11:40:51 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Location: http://example.com/
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Cache: MISS from hv453
X-Cache-Lookup: MISS from hv453:80
Via: 1.0 hv453 (squid/3.1.10)
Connection: keep-alive


GET http://example.com/ HTTP/1.1
Host: example.com
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: HTTPie/0.9.2


HTTP/1.0 407 Proxy Authentication Required
Server: squid/3.1.10
Mime-Version: 1.0
Date: Tue, 10 Mar 2015 04:43:35 GMT
Content-Type: text/html
Content-Length: 4
X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
Proxy-Authenticate: Basic realm="login"
X-Cache: MISS from hv453
X-Cache-Lookup: NONE from hv453:80
Via: 1.0 hv453 (squid/3.1.10)
Connection: keep-alive

407.
```

Can you get me tcpdump of cURL following that redirect as well? I have a theory.


Sure, pasted at http://pastebin.com/raw.php?i=T4npM2XM


We have a winner! HAProxy is at fault.

In your HAProxy configuration you add the Proxy-Authorization header automatically so that you don't have to manually authorize your requests. That's all well and good, however it seems that HAProxy is only adding that header to the first request on a connection. That's why cURL works: it opens a new TCP connection for the second request.

This is problematic, because HAProxy is explicitly telling us we can keep the connection alive, as you can see by the `Connection: keep-alive` header that requests sees. As RFC 7230 says:

> A proxy or gateway MUST parse a received Connection header field before a message is forwarded [...] and then remove the Connection header field itself (or replace it with the intermediary's own connection options for the forwarded message).

_NB: It's an interesting aside to note that cURL is sending the Proxy-Connection header despite an explicit admonition [not to do so from RFC 7230 section A.1.2](https://tools.ietf.org/html/rfc7230#appendix-A.1.2)._

In the short term, you can try adding `Connection: close` to your headers and see if that resolves the problem. Longer term, we should investigate why HAProxy doesn't add the header on subsequent requests.


Initial reading suggests that HAProxy is doing this on purpose. From the docs:

> By default HAProxy operates in a tunnel-like mode with regards to persistent connections: for each connection it processes the first request and forwards everything else (including additional requests) to selected server.


Even more compelling (emphasis mine):

> In HTTP mode, it is possible to rewrite, add or delete some of the request and response headers based on regular expressions. [..] But there is a limitation to this : **since HAProxy's HTTP engine does not support keep-alive, only headers passed during the first request of a TCP session will be seen**. All subsequent headers will be considered data only and not analyzed.


@gsakkis Another option is to try passing the username and password to the proxy URL, and hope that HAProxy's HTTP engine is too stupid to strip the Proxy-Authorization header off. That is, change your command line to:

``` bash
$ http 'http://httpbin.org/redirect-to?url=http://example.com/' --proxy http:http://username:password@localhost:5555 --headers --follow --debug
HTTPie 0.9.2
```


The fact that curl closes the connection between the requests is a regression and a mistake... I'll take that.

The Proxy-Connection: header we use was added once upon the time because it was necessary and I have not found a good reason to remove it again, even if RFC7230 now has arrived saying so...!


@bagder Thanks for looking in. =) I suspected that was going to be the reasoning for Proxy-Connection, but did just want to clarify it. I doubt we'll add it.

Glad we were able to help find a regression, even if that regression made it look like you were working and we weren't! ;)

Given that feedback, I'm closing this issue because it's not a requests bug, it's a problem with @gsakkis' HAProxy usage/configuration. @gsakkis, please note that it's definitely possible to configure HAProxy to do what you want it to do, you just need to make some configuration changes or change the way you make the requests.


@Lukasa passing credentials to the proxy url does work indeed! I'd never figure it out on my own, thank you so much for the investigation!

PS: I linked to this ticket on the selected ServerFault answer in case someone stumbles on it too.


@gsakkis Happy to help, I'm glad we were able to resolve the situation. =)