Merge branch 'next' into t
osd: PG::Interval -> pg_interval_t

Signed-off-by: Sage Weil <sage@newdream.net>
osd: fix check for whether to recalculate past_intervals

We may not recalculate all the way back to last_interval_clean due to
the oldest_map floor.  Figure out what we want and could calculate before
deciding whether what we have is insufficient.

Also, print something if we discard and recalculate so it is clear what is
happening and why.

Signed-off-by: Sage Weil <sage@newdream.net>
osd: include past_intervals in pg debug printout

Signed-off-by: Sage Weil <sage@newdream.net>
osd: only generate missing intervals in generate_past_intervals

We can (currently) get into a situation where we don't have the full
history back to last_epoch_clean because non-primaries record past
intervals but don't initially have the full history, resulting in a partial
recent history.

If this happens, only fill in what's missing; no need to rebuild the recent
parts too.

Signed-off-by: Sage Weil <sage@newdream.net>
osd: share past intervals with notifies

Send past_intervals along with pg_info_t on every notify.  The reasoning
here is as follows:

 - we already have the state in memory
 - if we don't send it, and the primary doesn't have it, it will
   recalculate it by reading/decoding many previous maps from disk
 - for a highly-tortured cluster, i see past_intervals on the order of
   ~6 KB, times 600 pgs means ~2.5 MB sent for every activate_map(). for
   comparison, the same cluster would need to read and decode ~1 GB of
   maps to recalculate the same info.
 - for healthy clusters, the data is small, and costs little.
 - for unhealthy clusters, the data is large, but most useful.

In theory we could set a threshold so that we don't send it if it is
large, but allow the primary to query it explicitly.  I doubt it's worth
the complexity.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
osd: drop useless PG::fulfill_info()

There is a nice symmetry there with fulfill_log(), but it is a short
function with a single caller that mostly just forces us to copy a bunch
of data structures around unnecessarily.  Drop it.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
osd: fill in past intervals during advance_map

If ceph-osd is way behind, we will advance through past maps before we
mark ourselves up.  This avoids the slow recalculation once we are up, and
the ensuing badness.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
osd: set dirty_info in generate_past_intervals

This ensures that we save our work.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
osd: always share past_intervals

Share past intervals when starting up new replicas.  This can happen via
an MOSDPGInfo or an MOSDPGLog message.

Fix up get_or_create_pg() so the past_intervals arg is required (and a ref,
like the other args). Fix doxygen comment.

Now the only time generate_past_intervals() should do any work is when
upgrading old clusters, during pg creation, and (possibly) during pg
split (when that is fully implemented).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
osd: fix dirty_info check for advance/activate paths

Previously we would check and write dirty_info *without the pg lock* after
doing the advance and activate map calls.  This was unlikely to race with
anything because the queues were drained, but definitely not right.

Instead, do the write in activate_map, or explicitly if activate_map is
not called (so that we record our progress after handling maps when we are
not up).

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
osd: skip scrub scheduling if we aren't up

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
osd: write dirty info on handle info, notify, log

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
osd: dirty_info if history.merge updated anything

In proc_replica_info and proc_primary_info, we may or may not update
the pg_info_t.  If we do, set dirty_info, so that it will be recorded.
Same goes for when the primary pushes out updated stats to us.

Also, do not write a purged_snaps() update directory; rely on the caller
to write out dirty info.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
osd: do not merge history on query

We shouldn't modify the local notion of the history without recording it to
disk.  And we (probably) also don't need to do that at all on query.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
osd: use PG::write_if_dirty() helper

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
osd: fix nested transaction in all_activated_and_committed()

all_activated_and_committed() is called from _activate_committed(), called
from a objectstore completion, and also from the state machine, which is
part of a larger transaction.

Instead, set dirty_info, and build/apply a transaction in the caller
(the completion) as needed.  Fixes part of #2360.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
osd: use dirty flags in activate(), merge_log()

These are all called from within the state machine, so we can simply set
the dirty flags.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
osd: set dirty flags on rewind_divergent_log

Make sure we record any rewind_divergent_log.  In the activate case, this
will happen anyway, but mark it dirty here for correctness/completeness.

The merge_log case might be a bug.

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>
mon: drop obsolete osd/PG.h #includes

Signed-off-by: Sage Weil <sage.weil@dreamhost.com>