librbd: fix DiffIterateStress test

If we write to an interval that didn't previously exist and then discard
it so that it again doesn't exist, all during the same interval, then we
should not include it in the 'written' set (or exists set, obviously).

Similarly, when we got to look at a merged diff, we can ignore extents
that were written (and possibly zeroed) if they neither existed before nor
after.

Bump up the iteration count to get more confidence that this is actually
correct.

Signed-off-by: Sage Weil <sage@inktank.com>
msgr: add second per-message throttler to message policy

We already have a throttler that lets of limit the amount of memory
consumed by messages from a given source.  Currently this is based only
on the size of the message payload.  Add a second throttler that limits
the number of messages so that we can effectively throttle small requests
as well.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: throttle client messages by count, not just by bytes

This lets us put a cap on outstanding client IOs.  This is particularly
important for clients issuing lots of small IOs.

Fixes: #4579
Signed-off-by: Sage Weil <sage@inktank.com>
librbd: fix DiffIterateStress again

- fix seed
- the array indices are points in time; no need to subtract one from i!
- pick a random seed and print it to stdout

I ran this with several different seeds without failure, so I am confident
we are in good shape.  And if we ever get a future failure, we'll have the
seed to reproduce.

Signed-off-by: Sage Weil <sage@inktank.com>
mon: make 'osd crush move ...' idempotent

If we don't need to move the item, return success.

Signed-off-by: Sage Weil <sage@inktank.com>
global: flush log before stopping/starting on daemonize

Ensure that we push log data out before we restart logging.  This may not
be strictly necessary, but it avoids a whole class of possible pitfalls.

Signed-off-by: Sage Weil <sage@inktank.com>
global: call config observers on global_init (and start logging!)

Currently we don't start logging on daemon startup unless the log_file
parameter was adjusted by ceph.conf.  Instead, we should call all config
observers so that the logging subsystem is fully configured and we log
even prior to the daemonize and common_init_finish (when we call observers
again).  This fixes logging for the initial period before we daemonize.
For some of the daemons (osd, mon), this includes significant work.  It
also fixes the problem where users don't see the 'ceph version ...' banner
on daemon start.

Backport: bobtail
Signed-off-by: Sage Weil <sage@inktank.com>
rbd.py: remove unnecessary semicolons

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
pybind: add docstrings for make_ex()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
pybind: unify make_ex() code in cephfs.py and rados.py

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
rados.py: chmod -x

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
rados.py: add some docstrings

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
rados.py: fix create_pool()

Call rados_pool_create_with_all() only if auid and crush_rule
are set properly. In case only crush_rule is set call
rados_pool_create_with_crush_rule() on librados, not the other
way around.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
cephfs.py: add docstring for shutdown() and version()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
rados.py: add docstrings for *Error classes and others

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
rados.py: add more docstrings

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
rados.py: add docstrings for Ioctx::__aio_*_cb()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
md_config_t: change from class to struct as the name indicates

Change md_config_t from class to struct to fix some issues
from clang. The name indicates it's a struct so change it to
struct and unify the usaged type.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
mds: fix journaler to set temp_fetch_len appropriately and read the requested amount

The _prefetch() function which intereprets temp_fetch_len interprets
it as the amount of data we need from read_pos, which is the beginning
of read_buf. So by setting it to the amount *more* we needed, we were
getting stuck forever if we actually hit this condition. Fix it by
setting temp_fetch_len based on the amount of data we need in aggregate.

Furthermore, we were previously rounding *down* the requested amount in
order to read only full log segments. Round up instead!

Fixes #4618

Signed-off-by: Greg Farnum <greg@inktank.com>
Merge pull request #203 from dalgaaf/wip-da-pybind-pylint-v2

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>