msgr: add second per-message throttler to message policy

We already have a throttler that lets of limit the amount of memory
consumed by messages from a given source.  Currently this is based only
on the size of the message payload.  Add a second throttler that limits
the number of messages so that we can effectively throttle small requests
as well.

Signed-off-by: Sage Weil <sage@inktank.com>
osd: throttle client messages by count, not just by bytes

This lets us put a cap on outstanding client IOs.  This is particularly
important for clients issuing lots of small IOs.

Fixes: #4579
Signed-off-by: Sage Weil <sage@inktank.com>
global: flush log before stopping/starting on daemonize

Ensure that we push log data out before we restart logging.  This may not
be strictly necessary, but it avoids a whole class of possible pitfalls.

Signed-off-by: Sage Weil <sage@inktank.com>
global: call config observers on global_init (and start logging!)

Currently we don't start logging on daemon startup unless the log_file
parameter was adjusted by ceph.conf.  Instead, we should call all config
observers so that the logging subsystem is fully configured and we log
even prior to the daemonize and common_init_finish (when we call observers
again).  This fixes logging for the initial period before we daemonize.
For some of the daemons (osd, mon), this includes significant work.  It
also fixes the problem where users don't see the 'ceph version ...' banner
on daemon start.

Backport: bobtail
Signed-off-by: Sage Weil <sage@inktank.com>
mds: fix journaler to set temp_fetch_len appropriately and read the requested amount

The _prefetch() function which intereprets temp_fetch_len interprets
it as the amount of data we need from read_pos, which is the beginning
of read_buf. So by setting it to the amount *more* we needed, we were
getting stuck forever if we actually hit this condition. Fix it by
setting temp_fetch_len based on the amount of data we need in aggregate.

Furthermore, we were previously rounding *down* the requested amount in
order to read only full log segments. Round up instead!

Fixes #4618

Signed-off-by: Greg Farnum <greg@inktank.com>
mds: Keep LogSegment ref for openc backtrace

The MDRequest is destroyed once the client reply is sent, but
we need the reference to the LogSegment for updating the backtrace, so
store a temporary ref to the LogSegment for later.

Fixes #4660.
Signed-off-by: Sam Lang <sam.lang@inktank.com>
Merge pull request #206 from ceph/wip-4660

mds: Keep LogSegment ref for openc backtrace

Reviewed-by: Greg Farnum <greg@inktank.com>
journaler: remove the unused prefetch_from member variable

Signed-off-by: Greg Farnum <greg@inktank.com>
Merge pull request #202 from ceph/wip-log-boot

Fixes #4676.

Reviewed-by: Greg Farnum <greg@inktank.com>
Pipe: call discard_requeued_up_to under pipe_lock

Fixes: #4627
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
ceph_argparse: add _daemon versions of argparse calls

mon needs to call argparse for a couple of -- options, and the
argparse_witharg routines were attempting to cerr/exit on missing
arguments.  This is appropriate for the CLI usage, but not the daemon
usage.  Add a 'cli' flag that can be set false for the daemon usage
(and cause the parsing routine to return false instead of exit).

The daemon's parsing code due for a rewrite soon.

Signed-off-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Use _daemon version of argparse functions

Allow argparse functions to fail if no argument given by using
special versions that avoid the default CLI behavior of "cerr/exit"

Fixes: #4678
Signed-off-by: Dan Mick <dan.mick@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Revert "global: call config observers on global_init (and start logging!)"

This reverts commit a30917746614275baeb718e902133f06ef44fba6. This commit
includes calls that involve Mutexes, Lockers, and lockdep -- which isn't
yet set up, so things break horribly. A more subtle approach is required.

Signed-off-by: Greg Farnum <greg@inktank.com>
FileJournal: clarify meaning of start_seq and fix initialization

Second guessing the first sequence number from the FileStore
was silly and broke tests which had the temerity to start at
1 instead of 2...

Fixes: #4687
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
Merge remote-tracking branch 'origin/wip-osd-throttle2' into next

Reviewed-by: Greg Farnum <greg@inktank.com>
config: fix osd_client_message_cap comment

Signed-off-by: Greg Farnum <greg@inktank.com>
Merge branch 'wip-journaler-4618' into next

Reviewed-by: Sam Lang <sam.lang@inktank.com>
test: update rbd formatted-output for progress changes

Progress output now goes to stderr instead of stdout.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Merge branch 'next'