Chunkyparse

Hopefully this does the job. Here are the details.

This version doesn't handle line-folding (CRLF followed by space or horizontal tab) in HTTP input. Hopefully, not too many websites are this evil. To handle line-folding using a parser-generator, I think we need a Context Free Grammar (because the grammar is recursive), and not just a Deterministic Finite Automaton, which is much easier to construct. Furthermore, we need our parser to be interactive, because our input doesn't come in one shot and we don't know when the input ends for us to parse the input in bulk. In fact, if we _did_ know when the input ends, we wouldn't be writing this parser in the first place.

An interactive DFA seems really hard to build (http://flex.sourceforge.net/manual/How-can-I-have-multiple-input-sources-feed-into-the-same-scanner-at-the-same-time_003f.html). I wrote a version which processes input in bulk (https://github.com/anirudhSK/chunkinatr/commits/master) using flex. Unfortunately, I found it very hard to convert it into an interactive parser. Given that it's so hard to make an interactive DFA, it's probably much harder to make an interactive CFG (which is actually required to handle line-folding). So, I decided to punt on line-folding altogether. Now, this makes the problem easy enough that we don't need special tools any more, and we can simply hand code the parser :(. I guess you do have to throw one away.

Anyway, this is my cleanest attempt at hand coding the chunky parser for our needs. As I said, it doesn't handle line-folding at all. However it does handle arbitrary spaces (not tabs) in the chunk header, which some web servers seem to need (nytimes.com, for instance, puts some spaces in their chunk header, just to trip us us). One way to handle line-folding is to pre-process all the incoming data to replace folded lines with a single whitespace (which is allowed and recommended by RFC2616, page 15). This will get us closer to the real HTTP spec. This pre-processing will let us confirm to the RFC without intrusive modifications to our code base.

Anirudh