messenger: add the shell of a system to delay incoming Message delivery

When ms_inject_delay_type matches that of the incoming Connection,
the Pipe sets up a delay queue that it shuttles all Messages through.
This lets us check cleanup and some notification code but doesn't
actually generate any delays.

Signed-off-by: Greg Farnum <greg@inktank.com>
msgr: move the delay queue initialization into start_reader

The Pipe doesn't know the peer type in the constructor. It
doesn't always know in start_reader either, so this needs more work,
but at least it knows more frequently than it did.

Signed-off-by: Greg Farnum <greg@inktank.com>
msgr: clear out the delay queue when stop()ing

After some brief thought, I believe deleting any messages in the
delay queue is correct -- we are trying to simulate line delays
in delivery and so anything still in the queue has supposedly
not arrived yet. So delete them when we stop the Pipe for
any reason.

Signed-off-by: Greg Farnum <greg@inktank.com>
msgr: add a delay_until queue that is used to delay deliveries.

Its life-cycle matches that of delay_queue, and the delayed_delivery
function respects it. For now queue_received is just setting it to
delay everything by 1 second.

Signed-off-by: Greg Farnum <greg@inktank.com>
msg/Pipe: refactor msgr delays

- move all delay state into a single class
- create thread once and only once per Pipe
- adjust debug levels
- discard messages at the appropriate times

Signed-off-by: Sage Weil <sage@inktank.com>
msg/Pipe: drop queue helpers

There is a single caller; these only obfuscate.

Signed-off-by: Sage Weil <sage@inktank.com>
msg/Pipe: start delay thread *after* we know peer type

At end of connect(), or end of accept().

Signed-off-by: Sage Weil <sage@inktank.com>
msg/Pipe: release dispatch throttle on delayed queue discard

This avoids leaking into the throttle and deadlocking.

Signed-off-by: Sage Weil <sage@inktank.com>
msg/Pipe: flush delayed messages when stealing/failing pipes

If we are failing a pipe, flush the incoming messages before we try to
reconnect.  Similarly, flush queued messages on an existing pipe beore we
replace it.  This ensures that when we get a socket failure and reconnect
the delayed messages are handled in the normal fashion.

Specifically, it fixes a situation like:

 - read msg, update in_seq etc.
 - delay msg
 - pipe faults
 - peer reconnects, we replace existing pipe, discard delayed msgs
 - peer resends msgs
 - we discard, because they are < in_seq

Signed-off-by: Sage Weil <sage@inktank.com>
mds: don't create bloom filter for incomplete dir

Creating bloom filter for incomplete dir that was added by log
replay will confuse subsequent dir lookup and can create null
dentry for existing file. The erroneous null dentry confuses the
fragstat accounting and causes undeletable empty directory.

The fix is check if the dir is complete before creating the bloom
filter. For the MDCache::trim_non_auth{,_subtree} cases, just do
not call CDir::add_to_bloom because bloom filter is useless for
replica.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: journal remote inode's projected parent

Server::_rename_prepare() adds remote inode's parent instead of
projected parent to the journal. So during journal replay, the
journal entry for the rename operation will wrongly revert the
remote inode's projected rename. This issue can be reproduced by:

 touch file1
 ln file1 file2
 rm file1
 mv file2 file3

After journal replay, file1 reappears and directory's fragstat
gets corrupted.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Merge branch 'wip-msgr-delay-queue' into next