ReplicatedPG:start_flush send a second delete

Suppose we start with the following in the cache pool:

30:[29,21,20,15,10,4]:[22(21), 15(15,10), 4(4)]+head

The object doesn't exist at 29 or 20.

First, we flush 4 leaving the backing pool with:

3:[]+head

Then, we begin to flush 15 with a delete with snapc 4:[4] leaving the
backing pool with:

4:[4]:[4(4)]

Then, we finish flushing 15 with snapc 9:[4] with leaving the backing
pool with:

9:[4]:[4(4)]+head

Next, snaps 10 and 15 are removed causing clone 10 to be removed leaving
the cache with:

30:[29,21,20,4]:[22(21),4(4)]+head

We next begin to flush 22 by sending a delete with snapc 4(4) since
prev_snapc is 4 <---------- here is the bug

The backing pool ignores this request since 4 < 9 (ORDERSNAP) leaving it
with:

9:[4]:[4(4)]

Then, we complete flushing 22 with snapc 19:[4] leaving the backing pool
with:

19:[4]:[4(4)]+head

Then, we begin to flush head by deleting with snapc 22:[21,20,4] leaving
the backing pool with:

22[21,20,4]:[22(21,20), 4(4)]

Finally, we flush head leaving the backing pool with:

30:[29,21,20,4]:[22(21*,20*),4(4)]+head

When we go to flush clone 22, all we know is that 22 is dirty, has snaps
[21], and 4 is clean. As part of flushing 22, we need to do two things:
1) Ensure that the current head is cloned as cloneid 4 with snaps [4] by
sending a delete at snapc 4:[4].
2) Flush the data at snap sequence < 21 by sending a copyfrom with snapc
20:[20,4].

Unfortunately, it is possible that 1, 1&2, or 1 and part of the flush
process for some other now non-existent clone have already been
performed.  Because of that, between 1) and 2), we need to send
a second delete ensuring that the object does not exist at 20.

Fixes: #9054
Backport: firefly
Signed-off-by: Samuel Just <sam.just@inktank.com>
Merge remote-tracking branch 'gh/next'
Add a new field 'expected_num_objects' to pg_pool_t which denotes the expected number of objects on this pool.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
Add a new monitor command to let user specify the expected number of objects during pool creation.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
Add a new transaction OP (collection hint) to ObjectStore.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
Implement the collection hint transaction, add a new transation type as expected number of objects.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
Add tests for the collection hint OP: 1) Store Test 2) Idempotent Test.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
mon/OSDMonitor: reweight-by-pg

This is just like reweight-by-utilization, but looks purely at the PG to
OSD mapping, not at the number of bytes used on the target disks.  This
allows the reweighting to be done before any data is written into the
cluster, when no data will need to migrate as a result of the reweight.

Signed-off-by: Sage Weil <sage@redhat.com>
qa/workunits/cephtool/test.sh: test reweight-by-pg

Signed-off-by: Sage Weil <sage@redhat.com>
mon/OSDMonitor: adjust weights up, when possible

Note when OSDs are underloaded, as well.  If that is the case, adjust the
OSD reweight value if, if possible.  (It won't always be possible since
weights are capped at 1.)

Note that we set the underload threshold to the average, as we want to
aggressively adjust weights up (back to 1.0) whenever possible.  This gets
us a more efficient mapping calculation and reduces the amount of "noise"
in the weights.

Signed-off-by: Sage Weil <sage@redhat.com>
mon/OSDMonitor: reweight-by-pg for pool(s)

Allow the reweight-by-pg to look at a specific set of pools.  If the list
is ommitted, use PGs from all pools.  This allows you to focus on a
specific pool (the one that will dominate data usage).  Otherwise things
may not be quite right because other pools may have PGs that contain
much less data.

Signed-off-by: Sage Weil <sage@redhat.com>
mon/OSDMonitor: respect CRUSH weights for reweight-by-pg

Do not assume that all OSDs are weighted equally for reweight-by-pg.

Note that reweight-by-utilization already reweights based on the size of
the OSD volume; we presume that this is already reflected by the CRUSH
weights.

Signed-off-by: Sage Weil <sage@redhat.com>
Merge pull request #2199 from ceph/wip-reweight

mon: allow reweighting of osds by pg (isntead of bytes used)

Reviewed-by: Guang Yang <yguang@yahoo-inc.com>
Merge remote-tracking branches 'upstream/wip-9144', 'origin/wip-9054' and 'guangyy/wip-pg-splitting' into wip-sam-testing