10gen-labs/mongo-connector — pull request #48 by adgaudio (2013-10-23 17:33:57+00:00)
Title: ENH: add a --no-dump option to disable dump_collection() if no timestamp is defined in oplog config

Hi,

This PR adds a --no-dump option to mongo connector. I'd love your feedback!

Readme additions:

```
`--no-dump` tells mongo connector to begin scanning from the the top of the
oplog rather than dump the entire namespace, if and only if the --oplog-ts
file doesn't already have a record for the oplog. Why might you want this?
By default, if there is no timestamp for an oplog (ie if the --oplog-ts file
is empty), mongo connector will dump the entire namespace. You may want
to disable that functionality.
```

Here's the logic that let to the PR:

- I didn't want to dump the entire collection into my target database (which in this case is Neo4j). I only wanted documents in the oplog.
- It seemed like the dump_collection() operation should be optional, particularly when dealing with large sharded collections.
- It seemed like more work to get the latest timestamp in each oplog for each shard and set the config entry appropriately.