ReplicatedPG,PG: move duplicate FlushedEvt logic info on_flushed()

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG,ReplicatedPG: allow multiple flushes to be in progress

Shortly, we won't know precisely when to flush, so Strays
may need to flush twice.

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: call start_flush in Active() instead of using do_pending_flush()

This way, we don't block an OpWQ thread while waiting on the filestore.

Signed-off-by: Samuel Just <sam.just@inktank.com>
ReplicatedPG/PGBackend: block all ops other than Pull prior to active

Previously, it was guarranteed that prior to activation, flushed would
be false on a replica.  Now, there may be a period where flushed is true
due to the flush in Stray completing prior to activation and flushed
being false again.  This is necessary since shortly it won't be possible
to determine from the osdmap whether a stray will be activated in a
particular interval.

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: don't requeue waiting_for_active unless flushed and active

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: remove unused Peering::flushed

This has been unused for quite some time!

Signed-off-by: Samuel Just <sam.just@inktank.com>