get fails for "http://www.merrickbank.com/"

I try requests.get("http://www.merrickbank.com/"), but it does not work. It takes minutes for it to just fail, freezing up quite a bit.


Weirdly curl gets back a `Content-Length` header but requests gets `chunked`.


So if you announce compression it chunks.


So if they send chunked responses (when doing gzip) they send broken data. Especially:

``` http
X-Frame-Options: SAMEORIGIN
Set-Cookie: mb-CookieP=; HttpOnly; 
Secure
Set-Cookie: mb-CookieP=; HttpOnly; Secure
Transfer-Encoding: chunked
```

That is a newline there in the `Set-Cookie`-header which is simply invalid.
We then miss the chunked encoding and break.
This has to be fixed on their side.
You can work around it with

``` py
requests.get("http://www.merrickbank.com/", headers={'Accept-Encoding': None})
```


@t-8ch has diagnosed this correctly. Sadly there isn't much we can do about this on our side beyond what has been described here.


That's a bit of a cop-out.  Chrome, Firefox, and Internet Explorer (Internet Explorer, for God's sake) can all successfully fetch that page.  This glitch in their webserver can be worked around.  The requests lib ought to be a little more forgiving.


@mgdelmonte this isn't our fault. This is httplib's fault. Raise the bug there and eventually if it's ever fixed it'll trickle down to here.


@mgdelmonte That's extremely disingenuous, and I think you know that.

Chrome, Firefox, and Internet Explorer are browsers whose development is backed by the paid, full-time efforts of a team of developers. They own the entirety of their software stack, do not have backwards compatibility concerns with other libraries, and are beholden only to themselves for release schedules.

Requests is a tool whose development is funded by zero people. Three core developers devote more than 40 cumulative hours a week to this project, _for free_, with ancillary and extremely important contributions from people like @t-8ch which total probably another 10 hours, also _for free_. We do not control our entire software stack: many of the tools we build on top are are in the python standard library, like `httplib`, `cookielib`, `ssl`, and more. The fact that this project exists _at all_ is entirely down to the goodwill of a series of developers whose names you will never know and who received exactly no compensation for their work other than, if they are extremely lucky, the thanks of someone else.

As @sigmavirus24 points out, the bug is in httplib, which we rely on to do our HTTP parsing. Even then, I wouldn't call it a bug: there is a very clear specification for HTTP headers which is being violated by the server in question. At some point we need to stop tolerating the mistakes of others and say that something is simply not HTTP. This is particularly true of [Merrick Bank](http://www.merrickbank.com), a presumably financial institution that actually _did_ pay someone to build their website.

If you would like to get this fixed, please file a bug against httplib. More than that, please submit a patch. The last time we had one of these bugs was in #1804. For that bug I actually did provide a fix for `httplib`, which was merged more than one year after it was proposed. For that reason, I do not have particularly high hopes for this being fixed any time soon.

@mgdelmonte Please consider the way you act towards purely volunteer run projects. We are doing the best we can, but we need to pick our battles. If you'd like to pay one of us some money to address this bug, I'm sure we can come to some arrangement. Otherwise, we will pursue this in terms of the priorities of the requests project, and in those cases we promise nothing when the bug is upstream of us.


@Lukasa my sincere apologies, and I do appreciate the value of the requests lib.  My point was more to mock Internet Explorer (which frankly surprises me if it handles anything less than perfectly clean data) than to deride your efforts.

I'll file a bug report with httplib, which appears to be the source.  It's hard to foresee someone breaking spec like Merrick has, but nevertheless it seems like an easy accommodation to make with no penalties elsewhere, to ignore a broken HTTP header line.


@mgdelmonte Sincerely, thankyou for the apology. =) Miscommunications on the Internet are easy, and I'm sure you meant no ill-will. Your apology was extremely graceful.

When you file your httplib bug, please link it here. I'll make sure to keep an eye on it, and if I find time tomorrow I may even try to flesh out a patch. We'll see if this one merges quicker. ;)


@Lukasa no problem, we're all pulling in the same direction (usually).  I submitted an issue for httplib here

http://bugs.python.org/issue24363

This is my first time submitting an issue through that system so I was a little mystified and wasn't sure how to submit a patch, so just recommended my fix.  I'm amazed there aren't more problems with readheaders() as it seems to completely ignore the requirement of a blank line to terminate the header.  Yes, you shouldn't have malformed header lines, either; but certainly it's easier to skip them and accommodate the mistakes of others than to terminate parsing altogether, especially when the specification has a clear and (mostly) unambiguous terminator.

Anyway, hopefully this clears it up for good.  Famous last words, right?