kdkeck/elasticsearch-river-mongodb — pull request #1 by cheald (2014-04-18 22:35:59+00:00)
Title: Fix multi-op support and add support for oplog refs

I've expanded on my previous fixes and spent a fair amount of time with my head down in tokumx's oplog. Rather than flattening oplog entries, we treat all oplog entries as potential multi-op entries. When a ref op is encountered, the appropriate refs are found and iterated (using the same technique as the tokumx server's replication code). This results in full support for tokumx transactions and multi-ops, while still maintaining compatibility with the vanilla Mongo oplog format.

Additionally, the inner loop issue you previous mentioned has been resolved; checking the river status in the inner loop allows for dashboard interrupts of the river without requiring continual creation of new cursors.

The biggest outstanding issue is the oplog.refs lookup. The semantics are kinda fuzzy, but the general idea is that we will have one or more refs with monotonically-increasing seq numbers. The oplog.refs _id format is `{oid: ObjectId(...), seq: num}`, but an index only exists on `_id`. The way the server finds entries is via:

LOCK_REASON(lockReason, "repl: finding oplog.refs entry to apply");
Client::ReadContext ctx(rsOplogRefs, lockReason);
const BSONObj query = BSON("_id" << BSON("$gt" << BSON("oid" << oid << "seq" << seq)));
if (!Collection::findOne(rsOplogRefs, query, entry, true)) {
break;
}

The only thing it does here that I haven't been able to replicate is the fourth parameter in findOne, which is a `requireIndex` boolean, which appears to cause the query to be executed in index-only mode. I haven't found a way to replicate that with the driver yet, and explain() shows that the query is not executed in indexOnly mode. Improving that should substantially improve performance, since the _id index is clustered.

I've not yet written tests, but since you're working on that, I figured this code might be useful in the course of testing full compatibility.