mikeal/request — pull request #660 by mikeal (2013-09-13 08:41:09+00:00)
Title: Work in Progress: Request re-write for new streams.

This is nowhere near finished and most of the tests still don't pass.

This is full re-write of request from the ground up. It is built on top of new style streams (streams2) and there is a concerted effort to do more of the work in dependencies.

### Deprecations

* piping to a destination w/ a callback parameter forces buffering.

`request(url, function (e, resp) { }).pipe(fs.createWriteStream(file)` will now have the side effect of also buffering the entire response in to memory. Because new style streams can be piped well after nextTick() we don't have a good way to know if we should or should not buffer the response other than the callback parameter. we still will not ever buffer the response when a callback is not given.

* JSON.stringify(request(url)) probably breaks.

Deprecating the old hacky getSafe() function will dramatically improve performance and, for the most part, stringifying options with sockets attached is a bad idea.

* all signing, cookies, and other non-default options will require the user explicitly install their dependency.

The dep list is growing exponentially and because of the way cookies and signing work in particular it isn't possible for request to support these features without adding code to request itself. Rather than continue to let the dep list grow forever in to the future, and with the understanding that one day we may even want a feature that requires a compiled dependency, I'm going to move to a model where we check for the dep when you use the feature and throw if you didn't install it.

This will not be the case for default options which will still have their deps included.

### Features

* NEW STYLE STREAMS!

* gzip support, enabled by default.

We now default to including the Accept-Encoding header for gzip and transparently decompressing the stream in the underlying HTTP Duplex stream.

* less memory copies when using binary response buffering.

Basically we just moved to @rvagg's `bl` module which we pipe ourself to in order to buffer the response. `bl` gives use a compatible buffer API but actually stores all the chunks in an array so we save on the big memcopy we used to do.

The code is also much cleaner, which is always the case when you re-write a module that hasn't had a full re-write since node.js had promises built in :)