pouchdb/pouchdb — pull request #1963 by ig3 (2014-04-16 19:48:56+00:00)
Title: WIP - change batching for replication

I still can't get a clean test run for this in node (haven't tried the browser yet). It is close to what I had intended, but I'm stymied by a surprising number of leveldb bugs. I'm taking a break now, but thought you might like to consider this.

With all the interference, I have lost track a bit, but I believe this puts bounds on memory for both changes and replicate (i.e. it should work even for very large changes feeds). It throttles changes to limit the queue of replicate batches. It signals uptodate, but a bit prematurely. The event should be deferred until the documents are written. I know what I want to do about this, but must get back to other work now.

There is a new option for replication: opts.batchs_limit, which is an upper bound on the number of batches queued.

The algorithm is somewhat as Dale and Calvin have discussed, but different in some details.

npm test was running fairly reliably with batch_size = batches_limit = 1 (after fixing a few bugs elsewhere), but is not yet clean with batch_size = batches_limit = 100.

The latest problem appears to be a fault in allDocs on leveldb (#1961). It it can't be sorted soon I'll remove the revision-1 stuff pending a fix, to get this going and stable.

It may be just my bad luck, but at the moment I am feeling there may be a bit too much change and too little testing going on. It sure seems easy for me to find bugs... Anyway, time for a nap and I'll look at it with fresh eyes later.

Comments and suggestions welcome.