mongoid/moped — pull request #320 by dawid-sklodowski (2014-09-22 14:54:13+00:00)
Title: Fix retries and failover

Pull-request that fixes failover and retry mechanism.

Changes in details:

* Refactoring: move ``` with_retry ``` method to ``` Cluster ``` -- it belongs there as it operates on cluster.
* Introduce retries on write operations -- it makes sense, because:
* Update is idempotent
* Delete is idempotent (deletes rows which are matching query)
* Insert -- in worse case scenario we could end up with duplicated data, however given that moped is used by mongoid, which inserts rows always with _id already present, therefore such duplicated insert will raise unique index on _id violation, which is fine.
* Fixes failover mechanism -- Node#ensure_connected was written in a way that failover didn't take into account messages after executing operations ( processing of DB replies by callbacks was outside of begin-rescue block), therefore failover mechanism was not exercised in most cases it was meant for.
* Removes ``` Reconfigure ``` failover mechanism -- it was raising new exceptions but not retrying -- it should be good enough to just retry.
* Refactoring: Move recognition mechanism for some errors from ``` Errors ``` class to ``` Reply ``` class, so errors recognition is in one place.
* Fixes refresh mechanism -- if node was successfully refreshed it isn't down any more.

Outcome of those changes is that you can kill / restart mongo replica-set nodes in whatever order and as often as you like. You can even stop all of them for couple of seconds (driven by ``` retry_count ``` and ``` retry_interval ```) and application will be able to recover without loosing any operations or throwing errors.