qa/workunits/hadoop-wordcount: use -x

Signed-off-by: Sage Weil <sage@redhat.com>
qa/workunits/hadoop-wordcount: fix/use -rmr command

-rm -r -f ... doesn't seem to work; use -rmr instead.

Signed-off-by: Sage Weil <sage@redhat.com>
qa/workunits/hadoop: move all hadoop tests into a hadoop/ dir

Signed-off-by: Sage Weil <sage@redhat.com>
Add a new field 'expected_num_objects' to pg_pool_t which denotes the expected number of objects on this pool.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
Add a new monitor command to let user specify the expected number of objects during pool creation.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
Add a new transaction OP (collection hint) to ObjectStore.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
Implement the collection hint transaction, add a new transation type as expected number of objects.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
Add tests for the collection hint OP: 1) Store Test 2) Idempotent Test.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)
mon/OSDMonitor: reweight-by-pg

This is just like reweight-by-utilization, but looks purely at the PG to
OSD mapping, not at the number of bytes used on the target disks.  This
allows the reweighting to be done before any data is written into the
cluster, when no data will need to migrate as a result of the reweight.

Signed-off-by: Sage Weil <sage@redhat.com>
qa/workunits/cephtool/test.sh: test reweight-by-pg

Signed-off-by: Sage Weil <sage@redhat.com>
mon/OSDMonitor: adjust weights up, when possible

Note when OSDs are underloaded, as well.  If that is the case, adjust the
OSD reweight value if, if possible.  (It won't always be possible since
weights are capped at 1.)

Note that we set the underload threshold to the average, as we want to
aggressively adjust weights up (back to 1.0) whenever possible.  This gets
us a more efficient mapping calculation and reduces the amount of "noise"
in the weights.

Signed-off-by: Sage Weil <sage@redhat.com>
mon/OSDMonitor: reweight-by-pg for pool(s)

Allow the reweight-by-pg to look at a specific set of pools.  If the list
is ommitted, use PGs from all pools.  This allows you to focus on a
specific pool (the one that will dominate data usage).  Otherwise things
may not be quite right because other pools may have PGs that contain
much less data.

Signed-off-by: Sage Weil <sage@redhat.com>
mon/OSDMonitor: respect CRUSH weights for reweight-by-pg

Do not assume that all OSDs are weighted equally for reweight-by-pg.

Note that reweight-by-utilization already reweights based on the size of
the OSD volume; we presume that this is already reflected by the CRUSH
weights.

Signed-off-by: Sage Weil <sage@redhat.com>
Merge pull request #2199 from ceph/wip-reweight

mon: allow reweighting of osds by pg (isntead of bytes used)

Reviewed-by: Guang Yang <yguang@yahoo-inc.com>
mon: make reweight-by-* sanity limits configurable

Also drop the somewhat redundant osd_sum.kb check; the main thing we care
about here is

Signed-off-by: Sage Weil <sage@redhat.com>
Merge pull request #2279 from ceph/wip-hadoop

fix and reorg hadoop workunits

Reviewed-by: Greg Farnum <greg@inktank.com>
Merge pull request #2287 from ceph/wip-reweight-tunables

mon: make reweight-by-* sanity limits configurable

Reviewed-by: Greg Farnum <greg@inktank.com>
mon: fix signed/unsigned warnings

Signed-off-by: Sage Weil <sage@redhat.com>
Merge pull request #2043 from guangyy/wip-pg-splitting

Support 'expected_num_objects' parameter when creating pool for pg folder splitting

Reviewed-by: Samuel Just <sam.just@inktank.com>
erasure-code: preload the jerasure plugin

Load the jerasure plugin when ceph-osd starts to avoid the following
scenario:

* ceph-osd-v1 is running but did not load jerasure

* ceph-osd-v2 is installed being installed but takes time : the files
  are installed before ceph-osd is restarted

* ceph-osd-v1 is required to handle an erasure coded placement group and
  loads jerasure (the v2 version which is not API compatible)

* ceph-osd-v1 calls the v2 jerasure plugin and does not reference the
  expected part of the code and crashes

Although this problem shows in the context of teuthology, it is unlikely
to happen on a real cluster because it involves upgrading immediately
after installing and running an OSD. Once it is backported to firefly,
it will not even happen in teuthology tests because the upgrade from
firefly to master will use the firefly version including this fix.

While it would be possible to walk the plugin directory and preload
whatever it contains, that would not work for plugins such as jerasure
that load other plugins depending on the CPU features, or even plugins
such as isa which only work on specific CPU.

http://tracker.ceph.com/issues/9153 Fixes: #9153

Backport: firefly
Signed-off-by: Loic Dachary <loic-201408@dachary.org>