crush: prevent integer overflow on reweight

Disallow setting OSD weights to a value over 10,000 and cap bucket weight
at 10,000,000 in a CRUSH map. Addresses issue #2101.

Signed-off-by: caleb miles <caleb.miles@inktank.com>
crush: Set maximum device/bucket weights.

Signed-off-by: caleb miles <caleb.miles@inktank.com>
crush: sum and check quantized weights for bucket

Sum the quantized weights for each bucket, and check that for overflow.

This could change the results of a compile marginally if the map is using
non-divisible weight values that quantize funny.  The old code might
calculate a bucket sum that is not the actual sum of the quantized weights.

Signed-off-by: Sage Weil <sage@inktank.com>
debian: fix ceph-fs-common-dbg depends

Signed-off-by: James Page <james.page@ubuntu.com>
rgw: don't override subuser perm mask if perm not specified

Bug #2650. We were overriding subuser perm mask whenever subuser
was modified, even if perm mask was not passed.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Merge remote-tracking branch 'gh/wip-2101'
ReplicatedPG,PG: dump recovery/backfill state on pg query

Signed-off-by: Samuel Just <sam.just@inktank.com>
test_librbd: fix warnings

test/test_librbd.cc: In member function ‘virtual void LibRBD_TestClone_Test::TestBody()’:
warning: test/test_librbd.cc:1040:111: format ‘%ld’ expects argument of type ‘long int’, but argument 2 has type ‘uint64_t {aka long long unsigned int}’ [-Wformat]
warning: test/test_librbd.cc:1040:111: format ‘%ld’ expects argument of type ‘long int’, but argument 3 has type ‘uint64_t {aka long long unsigned int}’ [-Wformat]
warning: test/test_librbd.cc:1040:111: format ‘%ld’ expects argument of type ‘long int’, but argument 4 has type ‘int64_t {aka long long int}’ [-Wformat]

Signed-off-by: Sage Weil <sage@inktank.com>
ReplicatedPG: don't mark repop done until apply completes

Consider the following sequence:
1. issue, apply repop
2. replicas and primary commit
  Here, repop->waitfor_(ack|disk) are empty, so we mark
  repop->done and remove_repop.
3. interval change, repops still in queue are marked aborted
4. activate, last_update_applied = last_update
5. the repop from one enters apply_repop, is not aborted,
   and finds that last_update_applied has passed it by.

Fixes #2749

Signed-off-by: Samuel Just <sam.just@inktank.com>
mon: purge removed osds from [near]full sets

The [near]full sets are volatile state.  Remove removed (or created)
osds from the set when we process a map.

Fixes: #2779
Signed-off-by: Sage Weil <sage@inktank.com>
mon: use single helper for [near]full sets

Use a single helper to add/remove osds from the [near]full sets.  This
keeps the logic in a single place, and simplifies the code somewhat.

Signed-off-by: Sage Weil <sage@inktank.com>
qa: download tests from specified branch

These python tests aren't installed, so they need to be downloaded

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
PG: merge_log always use stats from authoritative replica

If the osd recieving the log has divergent entries, it will
also have a "divergent" stat structure.  In general, it suffices
to simply trust the stat structure shipped with the authoritative
log and info since merge_log is only used to merge an authoritative
log.

Probably fixes #2769.

In cases like #2769, this bug can result in a primary with a stat
structure which double counts an operation: once for the
divergent operation, and once for the replay.  It turned up
in a regression suite run as a scrub stat mismatch.

Signed-off-by: Samuel Just <sam.just@inktank.com>
OSD: send_still_alive when we get a reply if we reported failure

When we get a ping reply, remove the peer from the failure_queue
and send a still alive message if the peer is in the failure_pending
map.

Otherwise, the monitor could slowly accumulate sporadic failure reports
leading to an osd being incorrectly marked out.

This bug may have been contributing to the wrongly-marked-down
thrashing observed on some systems.

Signed-off-by: Samuel Just <sam.just@inktank.com>
qa: download tests from specified branch

These python tests aren't installed, so they need to be downloaded

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
doc: cookbooks issue resolved, so changed 'ceph-cookbooks' back to 'ceph.'

Signed-off-by: John Wilkins <john.wilkins@inktank.com>
doc: remove last reference to ceph-cookbooks.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>
crushtool: allow information generated during testing to be dumped
to a set of CSV files for off-line analysis.

Signed-off-by: caleb miles <caleb.miles@inktank.com>
OSD: add config options to fake missed pings

In order to test monitor and osd failure detection and false
positive correction, this patch adds the following options:

 1. osd_debug_drop_ping_probability: probability of dropping
    a string of pings from a client upon ping recipt.
 2. osd_debug_drop_ping_duration: number of pings to drop in
    a row.

This should help with replicating some wrongly-marked-down
thrashing cases.

Signed-off-by: Samuel Just <sam.just@inktank.com>
vstart: use absolute path for keyring

Stores absolute path to the generated keyring so that tests running in
other directories (e.g. src/java/test) can simply reference the
generated ceph.conf.

Signed-off-by: Noah Watkins <jawhawk@cs.ucsc.edu>