rvagg/node-levelup — pull request #165 by pgte (2013-07-23 21:11:45+00:00)
Title: WIP: write stream redo using batch-write-stream

For discussion:

Rewrote [write_stream](https://github.com/rvagg/node-levelup/blob/writestream-rewrite/lib/write-stream.js) based on [batch-write-stream](https://github.com/pgte/batch-write-stream).

(Unfortunately I can't extend `readable-stream.Writable` or stream.Writable to make it write in batches)

## Benchmarks

Running the existing stream benchmark locally yielded very similar results to writestream-rewrite:

### write-stream-optimization-benchmark-fix branch:

```
→ node stream-bench.js
--------------------------------------------------------------
Filled source! Took 15220ms, streaming to destination...
Reading took 7477ms,
Done! Took 7479ms, 49% of batch time
```

### writestream-rewrite branch:

```
→ node stream-bench.js
--------------------------------------------------------------
Filled source! Took 15468ms, streaming to destination...
Read finished! Took 7946ms
Done! Took 7947ms, 51% of batch time
```

Still have to rerun a set of benchmarks I have and update the results page.

## Magic values

All the magic values are in options:

* [flushWait](https://github.com/rvagg/node-levelup/blob/writestream-rewrite/lib/write-stream.js#L20)
* [maxConcurrentBatches](https://github.com/rvagg/node-levelup/blob/writestream-rewrite/lib/write-stream.js#L18)

These default values were empirically set: they simply were the values that yielded the most local performance.

As said, will rerun some benchmarks to extract some more significant numbers from this version.
My feeling is that if we want to optimize, as @dominictarr hinted, that must be done in run-time, since the numbers we'll extract will only be relevant in the local system for the `db.copy` pattern.