ReplicatedPG,PG: move duplicate FlushedEvt logic info on_flushed()

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG,ReplicatedPG: allow multiple flushes to be in progress

Shortly, we won't know precisely when to flush, so Strays
may need to flush twice.

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: call start_flush in Active() instead of using do_pending_flush()

This way, we don't block an OpWQ thread while waiting on the filestore.

Signed-off-by: Samuel Just <sam.just@inktank.com>
ReplicatedPG/PGBackend: block all ops other than Pull prior to active

Previously, it was guarranteed that prior to activation, flushed would
be false on a replica.  Now, there may be a period where flushed is true
due to the flush in Stray completing prior to activation and flushed
being false again.  This is necessary since shortly it won't be possible
to determine from the osdmap whether a stray will be activated in a
particular interval.

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: don't requeue waiting_for_active unless flushed and active

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: remove unused Peering::flushed

This has been unused for quite some time!

Signed-off-by: Samuel Just <sam.just@inktank.com>
osd: Simple dout() fix

Signed-off-by: David Zafman <david.zafman@inktank.com>
osd: Backfill peers should not be included in the acting set

Create actingbackfill in choose_acting()
Use first backfill target as previously
Add asserts to catch inappropriate use of actingbackfill
Use is_acting() in proc_replica_info() because this is before actingbackfill set
Remove backfill_targets from stray_set to prevent purge_strays from removing collection
Can't check is_replica() anymore for backfill operations since a backfill isn't
a replica due to acting set change.

fixes: #5855

Signed-off-by: David Zafman <david.zafman@inktank.com>