osd: leave osd_lock locked in shutdown()

No callers expect the lock to be dropped.

Fixes: #3816
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 98a763123240803741ac9f67846b8f405f1b005b)
FileJournal::wrap_read_bl: adjust pos before returning

Otherwise, we may feed an offset past the end of the journal to
check_header in read_entry and incorrectly determine that the entry is
corrupt.

Fixes: 4296
Backport: bobtail
Backport: argonaut
Reviewed-by: Sage Weil <sage@inktank.com>
Signed-off-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 5d54ab154ca790688a6a1a2ad5f869c17a23980a)
rados: obey op_size for 'get'

Otherwise we try to read the whole object in one go, which doesn't bode
well for large objects (either non-optimal or simply broken).

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit 234becd3447a679a919af458440bc31c8bd6b84f)
rados: remove unused "check_stdio" parameter

Signed-off-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit bb860e49a7faeaf552538a9492ef0ba738c99760)
ObjectCacher: fix debug log level in split

Level 0 should never be used for this kind of debugging.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
Reviewed-by: Dan Mick <dan.mick@inktank.com>
(cherry picked from commit cb3ee33532fb60665f39f6ccb1d69d67279fd5e1)
rgw: don't iterate through all objects when in namespace

Fixes: #4363
Backport: argonaut, bobtail
When listing objects in namespace don't iterate through all the
objects, only go though the ones that starts with the namespace
prefix

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit 6669e73fa50e3908ec825ee030c31a6dbede6ac0)
formatter: add the ability to dump attrs in xml entities

xml entities may have attrs assigned to them. Add the ability
to set them. A usage example:

formatter->open_array_section_with_attrs("container",
     FormatterAttrs("name", "foo", NULL));

This will generate the following xml entity:
<container name="foo">

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 7cb6ee28073824591d8132a87ea09a11c44efd66)

Conflicts:
	src/common/Formatter.cc
rgw: set attrs on various list bucket xml results (swift)

Fixes: #4247
The list buckets operation was missing some attrs on the different
xml result entities. This fixes it.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
(cherry picked from commit 4384e59ad046afc9ec53a2d2f1fff6a86e645505)
deb:  Add ceph-coverage to ceph-test deb package

Teuthology uses the ceph-coverage script extensively
and expects it to be installed by the ceph task.  Add
the script to the ceph-test debian package so that it
gets installed for that use case.

Signed-off-by: Sam Lang <sam.lang@inktank.com>
(cherry picked from commit 376cca2d4d4f548ce6b00b4fc2928d2e6d41038f)
Fix output of 'ceph osd tree --format=json'

Signed-off-by: Tyler Brekke <tyler.brekke@inktank.com>
(cherry picked from commit 9bcba944c6586ad5f007c0a30e69c6b5a886510b)
msgr: drop messages on cons with CLOSED Pipes

Back in commit 6339c5d43974f4b495f15d199e01a141e74235f5, we tried to make
this deal with a race between a faulting pipe and new messages being
queued.  The sequence is

- fault starts on pipe
- fault drops pipe_lock to unregister the pipe
- user (objecter) queues new message on the con
- submit_message reopens a Pipe (due to this bug)
- the message managed to make it out over the wire
- fault finishes faulting, calls ms_reset
- user (objecter) closes the con
- user (objecter) resends everything

It appears as though the previous patch *meant* to drop *m on the floor in
this case, which is what this patch does.  And that fixes the crash I am
hitting; see #4271.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 0f42eddef5da6c1babe9ed51ceaa3212a42c2ec4)
osd: allow pg log trim during (non-classic) scrub

Chunky (and deep) scrub do not care about PG log trimming.  Classic scrub
still does.

Deep scrub can take a long time, so not trimming the log during that period
may eat lots of RAM; avoid that!

Might fix: #4179
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 0ba8db6b664205348d5499937759916eac0997bf)
osd: restructure calc_trim

No functional change, except that we log more debug, yay!

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 86df164d04f6e31a0f20bbb94dbce0599c0e8b3d)
osd: allow (some) log trim when degraded, but not during recovery

We allow some trim during degraded, although we keep more entries around to
improve our chances of a restarting OSD of doing log-based recovery.

Still disallow during recovery...

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 6d89b34e5608c71b49ef33ab58340e90bd8da6e4)
common: reduce default in-memory logs for non-daemons

The default of 100000 can result in hundreds of MBs of extra memory
used. This was most obvious when using librbd with caching enabled,
since there was a dout(0) accidentally left in the ObjectCacher.

refs: #4352
backport: bobtail
Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
(cherry picked from commit 7c208d2f8e3f28f4055a4ae51eceae892dcef1dc)
Fix radosgw actually reloading after rotating logs.

The --signal argument to Debian's start-stop-daemon doesn't
make it send a signal, but defines which signal should be send
when --stop is specified.

Signed-off-by: Jan Harkes <jaharkes@cs.cmu.edu>
(cherry picked from commit 44f1cc5bc42f9bb6d5a386037408d2de17dc5413)
log: drop default 'log max recent' from 100k -> 10k

Use less memory.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit c021c5ccf0c063cccd7314964420405cea6406de)
osd/PG: rename require_same_or_newer_map -> is_same_or_newer_map

This avoids confusion with the OSD method of the same name, and better
matches what the function tests (and does not do).

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit ba7e815a18cad110525f228db1b3fe39e011409e)

Conflicts:

	src/osd/ReplicatedPG.cc
osd: mark down connections from old peers

Close out any connection with an old peer.  This avoids a race like:

- peer marked down
- we get map, mark down the con
- they reconnect and try to send us some stuff
- we share our map to tell them they are old and dead, but leave the con
  open
...
- peer marks itself up a few times, eventually reuses the same port
- sends messages on their fresh con
- we discard because of our old con

This could cause a tight reconnect loop, but it is better than wrong
behavior.

Other possible fixes:
 - make addr nonce truly unique (augment pid in nonce)
 - make a smarter 'disposable' msgr state (bleh)

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 881e9d850c6762290f8be24da9e74b9dc112f1c9)
rgw: set up curl with CURL_NOSIGNAL

Fixes: #4425
Backport: bobtail
Apparently, libcurl needs that in order to be thread safe. Side
effect is that if libcurl is not compiled with c-ares support,
domain name lookups are not going to time out.
Issue affected keystone.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 88725316ddcfa02ff110e659f7a8131dc1ea2cfc)