tests: qa/workunits/cephtool/test.sh ! and -e

From the bash man page:

   set -e exit immediately ... The shell does not exit ... if the
   command's return value is being inverted with !

Add an explicit exit 1 where appropriate.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
tests: check osd health in cephtool/test.sh

Add a trivial osd health test at the beginning of each group of
tests. When facing an intermittent failure, it is difficult to diagnose
if the cluster appears to be missing an OSD but there is no indication
as to when the OSDs were last up.

The tests are now only run after all OSDs are up.

These checks can be disabled with --no-sanity-check to allow running
some tests that have less requirements than running all the tests.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
documentation: revise placement group number guide

When a cluster has few OSDs (less than 50) propose a preselection of
values: as long as the number of placement groups is not too small nor
too large, it won't make much of a difference anyway.

Users of small clusters tend to blindly apply the (OSD*100)/(pool size)
formula and worry about chosing a wrong value because they do not
understand the tradeoffs. The preselection will hopefully save them from
this uncertainty.

Add an explanation of how placement groups relate to OSDs, CRUSH and
pools to help understand the tradeoffs. Explain the
tradeoffs (durability, distribution and resource usages) with examples.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>
Reviewed-by: Gerben Meijer <infernix@gmail.com>
Reviewed-by: Laurent Guerby <laurent@guerby.net>
Merge pull request #2417 from dachary/wip-placement-group

doc: revise placement group number guide.

Reviewed-by: John Wilkins <jowilkin@redhat.com>
Merge pull request #2414 from dachary/wip-cephtool-test

tests: make qa/workunits/cephtool/test.sh more robust

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
librados: Add rados_pool_get_tiers call

Fixes: #8188
Signed-off-by: Adam Crume <adamcrume@gmail.com>