OSD: add config option for peering_wq batch size

Large peering_wq batch sizes may excessively delay
peering messages resulting in unreasonably long
peering.  This may speed up peering.

Backport: cuttlefish
Related: #5084
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 39e5a2a406b77fa82e9a78c267b679d49927e3c3)
FileStore: add global replay guard for split, collection_rename

In the event of a split or collection rename, we need to ensure that
we don't replay any operations on objects within those collections
prior to that point.  Thus, we mark a global replay guard on the
collection after doing a syncfs and make sure to check that in
_check_replay_guard() for all object operations.

Fixes: #5154
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit f3f92fe21061e21c8b259df5ef283a61782a44db)

Conflicts:

	src/os/FileStore.cc
ReplicatedPG: replace clean_up_local with a debug check

Stray objects should have been cleaned up in the merge_log
transactions.  Only on the primary have those operations
necessarily been flushed at activate().

Fixes: 5084
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 278c7b59228f614addf830cb0afff4988c9bc8cb)
PG: start flush on primary only after we process the master log

Once we start serving reads, stray objects must have already
been removed.  Therefore, we have to flush all operations
up to the transaction writing out the authoritative log.
On replicas, we flush in Stray() if we will not eventually
be activated and in ReplicaActive if we are in the acting
set.  This way a replica won't serve a replica read until
the store is consistent.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit b41f1ba48563d1d3fd17c2f62d10103b5d63f305)
PG, ReplicatedPG: pass a transaction down to ReplicatedPG::on_change

Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 9f56a7b8bfcb63cb4fbbc0c9b8ff01de9e518c57)
ReplicatedPG: track temp collection contents, clear during on_change

We also assert in on_flushed() that the temp collection is actually
empty.

Fixes: #5670
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 47516d9c4b7f023f3a16e166749fa7b1c7b3b24c)

Conflicts:

	src/osd/ReplicatedPG.cc
msgr: clear_pipe inside pipe_lock on mark_down_all

Observed a segfault in rebind -> mark_down_all -> clear_pipe -> put that
may have been due to a racing thread clearing the connection_state pointer.
Do the clear_pipe() call under the protection of pipe_lock, as we do in
all other contexts.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 5fc1dabfb3b2cbffdee3214d24d7769d6e440e45)

Conflicts:

	src/msg/SimpleMessenger.cc
msgr: clear con->pipe inside pipe_lock on mark_down

We need to do this under protection of the pipe_lock.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit afafb87e8402242d3897069f4b94ba46ffe0c413)
msg/Pipe: goto fail_unlocked on early failures in accept()

Instead of duplicating an incomplete cleanup sequence (that does not
clear_pipe()), goto fail_unlocked and do the cleanup in a generic way.
s/rc/r/ while we are here.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit ec612a5bda119cea52bbac9b2a49ecf1e83b08e5)
msgr: reaper: make sure pipe has been cleared (under pipe_lock)

All paths to pipe shutdown should have cleared the con->pipe reference
already.  Assert as much.

Also, do it under pipe_lock!

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 9586305a2317c7d6bbf31c9cf5b67dc93ccab50d)
msg/Pipe: fix RECONNECT_SEQ behavior

Calling handle_ack() here has no effect because we have already
spliced sent messages back into our out queue.  Instead, pull them out
of there and discard.  Add a few assertions along the way.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 495ee108dbb39d63e44cd3d4938a6ec7d11b12e3)
msg/Pipe: be a bit more explicit about encoding outgoing messages

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 4282971d47b90484e681ff1a71ae29569dbd1d32)
msg/Pipe: assert lock is held in various helpers

These all require that we hold pipe_lock.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 579d858aabbe5df88543d096ef4dbddcfc023cca)
msg/Pipe: avoid creating empty out_q entry

We need to maintain the invariant that all sub queues in out_q are never
empty.  Fix discard_requeued_up_to() to avoid creating an entry unless we
know it is already present.

This bug leads to an incorrect reconnect attempt when

 - we accept a pipe (lossless peer)
 - they send some stuff, maybe
 - fault
 - we initiate reconnect, even tho we have nothing queued

In particular, we shouldn't reconnect because we aren't checking for
resets, and the fact that our out_seq is 0 while the peer's might be
something else entirely will trigger asserts later.

This fixes at least one source of #5626, and possibly #5517.

Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 9f1c27261811733f40acf759a72958c3689c8516)
msg/Pipe: unlock msgr->lock earlier in accept()

Small cleanup.  Nothing needs msgr->lock for the previously larger
window.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit ad548e72fd94b4a16717abd3b3f1d1be4a3476cf)
msgr: mark_down_all() after, not before, rebind

If we are shutting down all old connections and binding to new ports,
we want to avoid a sequence like:

 - close all prevoius connections
 - new connection comes in on old port
 - rebind to new ports
 -> connection from old port leaks through

As a first step, close all connections after we shut down the old
accepter and before we start the new one.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 07a0860a1899c7353bb506e33de72fdd22b857dd)

Conflicts:

	src/msg/SimpleMessenger.cc
msgr: adjust nonce on rebind()

We can have a situation where:

 - we have a pipe to a peer
 - pipe goes to standby (on peer)
 - we rebind to a new port
 - ....
 - we rebind again to the same old port
 - we connect to peer

and get reattached to the ancient pipe from two instances back.  Avoid that
by picking a new nonce each time we rebind.

Add 1,000,000 each time so that the port is still legible in the printed
output.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 994e2bf224ab7b7d5b832485ee14de05354d2ddf)

Conflicts:

	src/msg/Accepter.cc
msgr: maintain list of accepting pipes

New pipes exist in a sort of limbo before we know who the peer is and
add them to rank_pipe.  Keep a list of them in accepting_pipes for that
period.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit dd4addef2d5b457cc9a58782fe42af6b13c68b81)
msgr: close accepting_pipes from mark_down_all()

We need to catch these pipes too, particularly when doing a rebind(),
to avoid them leaking through.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 687fe888b32ac9d41595348dfc82111c8dbf2fcb)
msgr: fix a typo/goto-cross from dd4addef2d

We didn't build or review carefully enough!

Signed-off-by: Greg Farnum <greg@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 1a84411209b13084b3edb87897d5d678937e3299)