doc: improve incomplete state documentation

Signed-off-by: Samuel Just <sam.just@inktank.com>
doc: add documentation for undersized

Signed-off-by: Samuel Just <sam.just@inktank.com>
osd/: add state PG_STATE_PEERED

We need a state to represent the pg state where we are below
min_size, but trying to recover.

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: remove very obsolete comment about backfill_pos

Signed-off-by: Samuel Just <sam.just@inktank.com>
PG: break waiting_for_peered out of waiting_for_active

waiting_for_peered now holds ops until peering completes (activation,
not necessarily state active).  waiting_for_active now holds
specifically MOSDOp blocked on:
- scrub
- replay
- state active

Signed-off-by: Samuel Just <sam.just@inktank.com>
osd_types: add last_peered, last_became_peered

Signed-off-by: Samuel Just <sam.just@inktank.com>
osd/: s/backfill_read/recovery_read for obc locking

We are going to expand our usage of this for normal
replica recovery.

Signed-off-by: Samuel Just <sjust@redhat.com>
PGBackend: rename clear_state to clear_recovery_state

This is called when we complete recovery, and does not clear
all state.  clear_recovery_state is clearer.

Signed-off-by: Samuel Just <sjust@redhat.com>
ReplicatedPG: always take recovery_read_lock for recovery

Signed-off-by: Samuel Just <sjust@redhat.com>
PG::proc_master_log: assume les and history.les from log source

Otherwise, we might later go peered (not active) and not distribute
a non-0 last_epoch_started.  This should be safe as the log recipient
will have a last_update reflecting that interval as well.

Signed-off-by: Samuel Just <sjust@redhat.com>
PG: move most of all_activated_and_commited into the AllReplicasActivated handler

It's confusing to have that logic split up so much.

Signed-off-by: Samuel Just <sjust@redhat.com>
osd/: update peer_missing and local missing if we write a degraded object

Signed-off-by: Samuel Just <sjust@redhat.com>
osd/: don't block writes on degraded objects

Signed-off-by: Samuel Just <sjust@redhat.com>
PG::purge_strays: purge peer_missing as well

Signed-off-by: Samuel Just <sjust@redhat.com>
osd/: go "peered" instead of "active" when < min_size

In the case of a replicated pool, the pg will transition to "peered"
rather than "active", where it can perform backfill bringing itself up
to min_size peers.  To that end, most operations now require
is_peered rather than is_active (OSDOps being the primary exception).
Also, rather than using the query_epoch on the activation message as the
activation epoch (for last_epoch_started) on the replica, we instead
use the last_epoch_started in the info sent by the primary.  This
allows the primary to not advance last_epoch_started past the last
known actual activation.  This will prevent later peering epochs from
requiring the last_update from a peered epoch to go active (might be
divergent).  We do allow scrubbing while peered.

Fixes: #7862
Signed-off-by: Samuel Just <sam.just@inktank.com>
print peered
PG: fix activate_committed epoch
revove extra +peered
fix error in Active::react with info assert
PG::append_log: remove deleted missing objects