fluent/fluent-plugin-mongo — pull request #27 by shunwen (2013-11-06 08:03:30+00:00)
Title: Rollback ignore duplicate key error

Sorry I made a mistake. Ignoring the exception raised by `collection.insert(records, INSERT_ARGUMENT)` without further handling will drop the rest of records in the same chunk. Please help to roll back my commits, thanks.

#### Bug reason
What I did was rescue and ignore `E11000`, which means duplicate key error, so **out_mongo** won't keep retrying and get stuck. Taking a further look at **mongo-ruby-driver**'s code, the [`insert_documents`](https://github.com/mongodb/mongo-ruby-driver/blob/1.9.2/lib/mongo/collection.rb#L1168..1173) method puts `documents` into a `message` and send to [`insert_batch`](https://github.com/mongodb/mongo-ruby-driver/blob/1.9.2/lib/mongo/collection.rb#L1109..1116) to perform the insertion. `insert_batch` will raise exception as soon as it hits an `E11000` or any other error unless the option `continue_on_error` is set, and leave the rest of records un-inserted.

#### The other solution
No, so far I didn't come up with a good way to just ignore `E11000`. I will submit an issue to mongodb to ask for this feature...

Currently **out_mongo** will be blocked when there's an error which requires human interception to fix. I thought to use the `continue_on_error` flag. By setting it, `insert_documents` will append error from each `insert_batch` call into array `errors`, move on to the rest of documents, and raise `errors.last` at last. But the problem is that there might be multiple calls of `insert_batch` and therefore multiple errors, if `errors.last` is `E11000` then the rest of errors will be ignored by `out_mongo` as well. Still `documents` wrapped in the same `message` of the `insert_batch` call will be dropped.