richardwilly98/elasticsearch-river-mongodb — pull request #433 by ankon (2014-12-12 18:32:12+00:00)
Title: River silently loses documents on restarts

When the river is running, and then stopped (either using an administrative command, or by shutting down the elasticsearch nodes) it remembers the timestamp of the last slurped operation.

When it is restarted afterwards however it will not import operations that happened between that timestamp and the shard's latest oplog timestamp (which could be for a completely different database FWIW!).

The commits in this PR fix that, and clean up the start procedure and collection slurper slightly while there.

The basic issue was that the river correctly calculated the last timestamp that should be the start of the import, but then only gave it to the collection slurper (which ignores non-initial imports), while starting the tailer with the oplog timestamp.

Note: this patch set is on top of the current master, and will conflict in small ways with #407. I can rebase it to that if needed, but in this way it should be easier to review them independently. As the river cannot be restarted easily without #407 (the status checker is stopped) the test uses some reflection with Elasticsearch's RiversService to trigger a restart.