mongoid/mongoid — pull request #3884 by niedfelj (2014-12-12 17:45:44+00:00)
Title: Logging exceptions that happen during a command/query

We spent weeks trying to track down networking issues between Heroku and Mongolab because we were not seeing answers from commands like:

MOPED: 127.0.0.1:27017 COMMAND database=admin command={:ismaster=>1} runtime: 3.2830ms

They just kept bouncing between nodes asking "ismaster". The main underlying cause was connection pool saturation, which was compounded by the fact that every uncaught exception (Timeout::Error) ends in the Failure class trying to Disconnect (which in turn tries to check out another connection to disconnect that isn't even the offending connection and of course doesn't make a lot of sense with pool saturation, Moped PR coming for that).

I've added a few lines to the log_subscriber so that if the payload has an exception, it shows it along with the command or query etc that it was TRYING to run, instead of making it look like it completed successfully. In most cases, Mongoid/Moped will still handle the exception/retry if it needs to, but surfacing these exceptions is extremely useful for optimizing our cluster and figuring out when and where we are having any sort of issue (connectivity, node failure)