Allow last_seq through doc_ids check

BugzID: 17898
BugzID: 17840
Send message asynchronusly from event manager

The replicator_manager was blocking for too long responding to
resume_scan events and causing general breakage.

BugzID: 17907
Handle last_seq returns from failed filters

Fabric changes will now return a last sequence when the continuous request has timed out but docs
have failed to pass the filter. This enables more accurate tracking of progress.

BugzID: 17709
Increase dt between heartbeat and inactive timeouts

The current theory is that on occasion the inactivity timeout in the
ibrowse client fires before the replicator receives a heartbeat line.
This patch replaces the 5 second delta with the "div 3" logic that
results in a 20 second delta by default.  This is the logic that we used
back when the replicator used heartbeats instead of timeouts to control
the frequency of heartbeat lines.

As an aside, the timeout value we're adjusting here actually serves a
dual purpose.  When a _changes response is streaming this parameter
controls the frequency of heartbeat newlines, useful e.g. for highly
selective filtered feeds.  In between streams of _changes it controls
how long the _changes coordinator will wait for updates to the DB before
terminating the response with a last_seq entry.

BugzID: 17709
Cancel replication jobs on every node, not just owner node

It is safe to attempt to cancel a job owned by another node, just in
case ownership changed at the same time as the document was deleted.

BugzID: 19518
Update to use new couch_event application
Fix replication checkpoint frequency

The patch to track and checkpoint last_seq values seen in the _changes
feed resulted in high frequency updates to the checkpoint document. This
is because the comparison made for "needs a checkpoint" was including
the replicator's incrementing counter which the changes reader was
bumping each time it saw last_seq even if it was the same last_seq as
the previous time.

This just compares the new last_seq value to the old last_seq value and
if they're equal then skips notifying the replication manager so that no
superfluous checkpoint is made.
Make replication request timeout configurable

This is intended as a defensive measure against failures in code that's out
of our control, such as ibrowse.

BugzID: 20550
Fix a non-tail-recursive call that leaks memory

By recursing in the protected body of a try/catch we weren't properly
using tail recursion when restarting the changes feed. This leads to
mailbox accumulation of messages which slowly eats all RAM on the node.

BugzId: 20601
Move changes_reader to a dedicated module

Also, export read_changes for better upgrade handling.

BugzID: 24294
Extract code that queues items into its own fun

BugzID: 24294
Provide an abstract interface to replicator stats

Switch to an ordered_dict() under the hood.

BugzID: 24298
Remove old code_change, set module version to 1
Report a correct value for changes_pending

This uses the work done in BugzId: 24236 that provides a new pending
field in _changes feeds. The source _changes is polled with a
`?since=seq&limit=0` query string to get the remaning changes to be
processed for a given sequence.

BugzId: 26015
Fix messages leaking from streaming HTTP responses

When we cancel a streaming HTTP response (ie, Transfer-Encoding:
chunked) its possible to leave ibrowse messages in the mailbox. For
processes like the changes reader that will continually retry HTTP
requests these messages can build up and cause the process to slow to a
crawl which in turns causes replications to seemingly get stuck.

This patch changes the HTTP request functions to be wholly contained in
a try/catch/after clause. Using the after clause we can be sure that
we're removing all ibrowse related messages before returning to the
calling function. We also release the worker from the same after clause
and inform the release if its a changes feed worker which is important
as these workers are not managed by a pool.

BugzId: 25341
Only set an ibrowse request timeout for non-_changes requests

BugzID: 25246
Rate limit the retrieval of pending changes

This limits the update of the changes_pending field to once every
`connection_timeout` milliseconds.

BugzId: 26015
Remove progress field from replication tasks

The progress field doesn't make much sense in the context of a
replication. The new changes_pending field should be the correct gauge
of how far behind a replication is.

BugzId: 26015
Remove unused unpack_seqs function

This is no longer needed now that we're not calculating progress based
off update_seq values.

BugzId: 26015
Fallback to using source seq for Apache CouchDB

If the update_seq is a number we assume that the source server is Apache
CouchDB so we fall back to comparing the current update_seq against the
update_seq value in the database info blob.

BugzId: 26015