snoyberg/http-conduit — pull request #131 by redneb (2013-06-03 06:17:59+00:00)
Title: A couple of fixes related to chunked transfer encoding

I discovered a couple of issues related to chunked transfer encoding:

1. When the length of a chunk is empty this is treated as zero (instead of an error) which signals the end of the chunked body.
2. When the length of a chunk contains junk, the junk is skipped. Only chunk extensions should be skipped (they start with a ';'). If the length contains only junk, then the length is treated as being empty which in turn is treated as being zero (like 1 above).
3. The most serious one: The last chunk is a regular chunk, it just has length zero. This means that it must be terminated by a CRLF (aside from the CRLF that terminates the length). Because of that, it is possible that http-conduit will think that the extra CRLF is part of the next response and reject the next response even thought there's nothing wrong with it. For some weird reason, this didn't happen most of the time in my tests. It seems that http-conduit discards all data left it has received after is has parsed the response and before it sends the next request. Am I right about this guess? If so, this is a **bad** idea. There should never be garbage between two consecutive responses. Any such garbage possibly indicates bugs in http-conduit.

I also took the liberty to rename the error from InvalidChunkHeaders to InvalidChunkedData to better reflect the fact that this is a problem related to the chunked encoding and not to any header issues.

Finally, I also inlcuded a commit about relaying downstream the CRLF that terminates each chunk when ```sendHeaders``` is true. I am not sure about this one as I am not sure about the purpose of ```sendHeaders```, but I think this is the right behavior.