oalders/www-mechanize-cached — pull request #4 by kentfredric (2013-06-28 06:52:27+00:00)
Title: Allow caching w/ Content-Encoding == chunked + no size header

Attached is patch for issue #3 , which I've inadvertently been patching in myself to all WWW:M:C installs I have which need to work with MCPAN Api.

---

Content-Encoding = chunked means its easier to determine transfer
corruption because each chunk is prefixed by size.

This renders any header-based size information useless, and means
anything that can't determine file-size before sending == uncacheable.

This is the case with any streamed content, or any content that is
inherently streamed and not buffered by the server.

This includes nginx w/ gzip, because gzip documents are /streamed/.

ie:

1. backend receives UA request
2. backend begins sending response
3. gzip begins
4. first byte of gzipped data is transmitted
5. ???
6. last byte of data is gzipped
7. last byte of data is transmitted

and there's no way to know in advance how big it will be, without
changing the flow to this:

1. backend receives UA request
2. backend begins sending response
3. gzip begins
4. gzipped data pools somewhere ( memory? disk? )
5. ???
6. last byte of gzipped data is gzipped
7. content-length header is sent
8. first byte of gzipped data is sent
9. ...
10. last byte of gzipped data is sent

In case #1, the user receives the file basically 'as soon as possible'

And in case #2, the user has to wait for the whole file to be rendered
and gzipped before they even see

...

So #1 greatly improves user experience, while #2 greatly diminishes it.

^^-> for effiency, chunked encoding w/ no content-length header is key.

So then it is permissible that a document that was transferred with
transfer-encoding = chunked , may be considered "complete", in a
reliable fashion, and thus cached, regardless of the content-length
header.