msgr: add second per-message throttler to message policy

We already have a throttler that lets of limit the amount of memory
consumed by messages from a given source.  Currently this is based only
on the size of the message payload.  Add a second throttler that limits
the number of messages so that we can effectively throttle small requests
as well.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: throttle client messages by count, not just by bytes

This lets us put a cap on outstanding client IOs.  This is particularly
important for clients issuing lots of small IOs.

Fixes: #4579
Signed-off-by: Sage Weil <sage@inktank.com>
global: flush log before stopping/starting on daemonize

Ensure that we push log data out before we restart logging.  This may not
be strictly necessary, but it avoids a whole class of possible pitfalls.

Signed-off-by: Sage Weil <sage@inktank.com>
global: call config observers on global_init (and start logging!)

Currently we don't start logging on daemon startup unless the log_file
parameter was adjusted by ceph.conf.  Instead, we should call all config
observers so that the logging subsystem is fully configured and we log
even prior to the daemonize and common_init_finish (when we call observers
again).  This fixes logging for the initial period before we daemonize.
For some of the daemons (osd, mon), this includes significant work.  It
also fixes the problem where users don't see the 'ceph version ...' banner
on daemon start.

Backport: bobtail
Signed-off-by: Sage Weil <sage@inktank.com>
rbd.py: remove unnecessary semicolons

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
pybind: add docstrings for make_ex()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
pybind: unify make_ex() code in cephfs.py and rados.py

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
rados.py: chmod -x

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
rados.py: add some docstrings

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
rados.py: fix create_pool()

Call rados_pool_create_with_all() only if auid and crush_rule
are set properly. In case only crush_rule is set call
rados_pool_create_with_crush_rule() on librados, not the other
way around.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
cephfs.py: add docstring for shutdown() and version()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
rados.py: add docstrings for *Error classes and others

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
rados.py: add more docstrings

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
rados.py: add docstrings for Ioctx::__aio_*_cb()

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
md_config_t: change from class to struct as the name indicates

Change md_config_t from class to struct to fix some issues
from clang. The name indicates it's a struct so change it to
struct and unify the usaged type.

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
mds: fix journaler to set temp_fetch_len appropriately and read the requested amount

The _prefetch() function which intereprets temp_fetch_len interprets
it as the amount of data we need from read_pos, which is the beginning
of read_buf. So by setting it to the amount *more* we needed, we were
getting stuck forever if we actually hit this condition. Fix it by
setting temp_fetch_len based on the amount of data we need in aggregate.

Furthermore, we were previously rounding *down* the requested amount in
order to read only full log segments. Round up instead!

Fixes #4618

Signed-off-by: Greg Farnum <greg@inktank.com>
Merge pull request #203 from dalgaaf/wip-da-pybind-pylint-v2

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
mds: Keep LogSegment ref for openc backtrace

The MDRequest is destroyed once the client reply is sent, but
we need the reference to the LogSegment for updating the backtrace, so
store a temporary ref to the LogSegment for later.

Fixes #4660.
Signed-off-by: Sam Lang <sam.lang@inktank.com>
Merge pull request #206 from ceph/wip-4660

mds: Keep LogSegment ref for openc backtrace

Reviewed-by: Greg Farnum <greg@inktank.com>
journaler: remove the unused prefetch_from member variable

Signed-off-by: Greg Farnum <greg@inktank.com>