segmentio/batch-stream — pull request #2 by brianc (2014-05-22 18:38:48+00:00)
Title: Add tests & add highWaterMark support

I ran into an issue where I was doing something like this:

```js
queryStream.pipe(batchStream).pipe(slowWriter)
```

The query stream was streaming 20 million rows into a slow writer. Because batch-stream doesn't set a highWaterMark it ends up slurping so many rows into memory that the process eventually hits the v8 heap limit and starts thrashing on GC really, really hard. :frowning:

Another way to reproduce this is like so:

```js
var Readable = require('stream').Readable
var Writable = require('stream').Writable
var Batch = require('batch-stream')

var readable = new Readable({objectMode: true})
var val = 0

readable._read = function() {
var value = val++
console.log('read', value)
this.push(value)
}

var sink = new Writable({objectMode: true})

sink._write = function(chunk, enc, cb) {
console.log('write', chunk)
setTimeout(cb, 1000)
}

var batch = new Batch({size: 1000, highWaterMark: 1})

readable.pipe(batch).pipe(sink)
```

What I would expect from the code sample above is for at _most_ 2000 reads to happen before the slow writer receives a row, but if you run it it will read over 1,000,000 rows from the reader before hitting the writer. :eyes:

This commit adds support for back pressure by allowing the user to optionally specify a `highWaterMark` on the options which will be passed to the underlying Transform stream. It's backwards compatible too! :clap:

I also added some tests because...tests are cool. :sunglasses: