share/igor — pull request #3 by lefnire (2013-07-02 04:31:31+00:00)
Title: allow batch size as `-b 1000` for collections too large for `.toArray() | async.map` in memory

I've got a situation trying to migrate my database with Igor: not all the mongo documents end up getting the OT data applied, which results in a "Invalid snapshot" error. When I was looking through my collections, I noticed it was consistently only on one collection: `usersPrivate`, which houses the largest data in the store (a single user can be multiple MB).

I *think* what's going wrong then is `cursor.toArray()` -> `async.map` is crapping out, since it's too much to handle. This PR adds an option for passing a batch size as `-b 500` or whatever.

Note:
1. I'm not sure this is the best way to handle batches, so feel free to scrap the PR if you have a better idea
1. It still doesn't seem to fix the issue. I still have documents without the OT data applied. Fewer, so it *may* have helped - but they're there still. Any ideas?