NimbusAPI/Nimbus — pull request #121 by michaelnoonan (2014-07-17 03:33:02+00:00)
Title: Logging for LongRunningTasks + Integration Tests

We've been finding a lot of `MessageLockLostExceptions` in our error logs. The logging is a bit sparse around the `LongLivedTaskWrapper` and this commit seeks to provide the diagnostics required to better understand why these exceptions are being thrown.

There are two primary cases we're hunting down:

1. The Message isn't even Dispatched to the Handler before the MessageLock expires, so it's dead in the water before the dispatch task is even started.
2. The Message is Dispatched in time, but the `LongLivedTaskWrapper` watcher task isn't running so it never renews the MessageLock.

The smoking gun looks like ThreadPool starvation under high load conditions.