Particular/NServiceBus — pull request #2234 by synhershko (2014-07-15 12:22:28+00:00)
Title: Fixes #2133: Stop losing timeouts

Simplify querying for timeouts, and introduce cleaning-up abilities to guarantee we don't skip any timeouts

1. One simple query is getting executed with a 1024 page size and with a projection to make sure we minimize traffic while still getting one stable, sorted page of results

2. nextTimeToRunQuery is only used to control subsequent executions. So we set it to immediate if we know the page of results is not the last one (we have more than 1024 results, or the results were stale). When new messages come in this wait time is invalidated and query is run again (and again, until results are not stale or we found no results at all).

3. We don't care about duplicates since TryRemove is getting called by the dispatcher later, and it does a lookup by ID. This provides us the deduplication that we need.

4. While this simple query, without paging, should provide assurance that no timeouts are being skipped while querying, in some rare edge cases this can still happen. We can either change `TimeoutPersisterReceiver` to take Eventually Consistent stores into account and invalidate the startSlice when needed, or have each individual such store take this up upon itself. For this PR we chose the latter, and use periodical cleanup process to make sure no timeouts are missed in the long run.

The defaults I wired to the gap and timing auto-props may require a bit more thinking, but overall I'm happy with this change now.

Also pending thorough discussion on timeouts SLA - do we rather fire timeouts late than miss firing any (taking into account the previous implementation was firing missed timeouts a restart).

/cc @andreasohlund @SimonCropp @johnsimons @udidahan @dannycohen