test/system/st_rados_watch: create object before watching

Signed-off-by: Samuel Just <sam.just@inktank.com>
test/librados/watch_notify: create foo before watching

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: do not serve requests until replicas have activated

There are two problems:
1) We choose the min last_update amoung peers with the max local-les
value as an upper bound on requests which could have been reported to
the client as committed.  We then, for ec pools, roll back to that point
to ensure that we don't inadvertently commit to an update which fewer
than K replicas actually saw.  If the primary sets local-les, accepts an
update from a client, and there is a new interval before any of the
replicas have been activated, we will end up being forced to use that
update which no other replica has seen as the new last_update.  This
will cause the object to become unfound.  We don't have this problem as
long as all active replicas agree on last_update before we accept IO.

2) Even for replicated pools, we would then immediately respond to the
request which created the primary-only update with a commit since it is
in the log and we have no outstanding repops.  If we then lose that
primary before any of the replicas in the new interval record the new
log, we will not only lose the object, but also the log entry recording
it, which will result in a lost write.

For these reasons, it seems like we need to wait for the replicas to
activate before we can process new requests essentially because whatever
update we select as last_update is essentially regarded as committed as
soon as we accept IO.

Fixes: #7649
Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: do not wait for flushed before activation

This should reduce the sting of the previous commit somewhat.  We wait
for the activation transactions to clear prior to accepting IO anyway,
so we can go ahead and get that process started without waiting for the
flush.

Signed-off-by: Samuel Just <sam.just@inktank.com>
Merge remote-tracking branches 'origin/wip-sam-testing2', 'origin/wip-7649', 'origin/wip-7671', 'origin/wip-7681' and 'origin/wip-7696' into wip-sam-testing