rvagg/node-levelup — pull request #153 by rvagg (2013-07-03 22:31:55+00:00)
Title: Streaming replication is pathologically slow

Streaming one level instance into another is much slower than expected. @dominictarr was looking into this with me at Nodeconf using this gist https://gist.github.com/brycebaril/5893248

The gist has two files, one that creates a small 1e6 record leveldb instance (~24MB on disk) and another which will attempt to use `db1.createReadStream().pipe(db2.createWriteStream())` to clone the instance to another level instance.

What we were seeing is that once the initial batch is sent, each buffered batch will typically be only a single record, significantly slowing down the transfer. In our case it was taking ~75 seconds.