osd: initialize new_state field when we use it

If we use operator[] on a new int field its value is undefined; avoid
reading it or using |= et al until we initialize it.

Fixes: #4967
Backport: cuttlefish, bobtail
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: David Zafman <david.zafman@inktank.com>
(cherry picked from commit 50ac8917f175d1b107c18ecb025af1a7b103d634)
osd: do not assume head obc object exists when getting snapdir

For a list-snaps operation on the snapdir, do not assume that the obc for the
head means the object exists.  This fixes a race between a head deletion and
a list-snaps that wrongly returns ENOENT, triggered by the DiffItersateStress
test when thrashing OSDs.

Fixes: #5183
Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Samuel Just <sam.just@inktank.com>
(cherry picked from commit 29e4e7e316fe3f3028e6930bb5987cfe3a5e59ab)
mon: Monitor: backup monmap using all ceph features instead of quorum's

When a monitor is freshly created and for some reason its initial sync is
aborted, it will end up with an incorrect backup monmap.  This monmap is
incorrect in the sense that it will not contain the monitor's names as
it will expect on the next run.

This results from us being using the quorum features to encode the monmap
when backing it up, instead of CEPH_FEATURES_ALL.

Fixes: #5203

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
(cherry picked from commit 626de387e617db457d6d431c16327c275b0e8a34)
debian: sync up postinst and prerm with latest

- do not use invoke-rc.d for upstart
- do not stop daemons on upgrade
- misc other cleanups

This corresponds to the state of master as of cf9aa7a.

Signed-off-by: Sage Weil <sage@inktank.com>
rgw: don't copy object idtag when copying object

Fixes: #5204
When copying object we ended up also copying the original
object idtag which overrode the newly generated one. When
refcount put is called with the wrong idtag the count
does't go down.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit b1312f94edc016e604f1d05ccfe2c788677f51d1)
rgw: only append prefetched data if reading from head

Fixes: #5209
Backport: bobtail, cuttlefish
If the head object wrongfully contains data, but according to the
manifest we don't read from the head, we shouldn't copy the prefetched
data. Also fix the length calculation for that data.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit c5fc52ae0fc851444226abd54a202af227d7cf17)
OSDMonitor: skip new pools in update_pools_status() and get_pools_health()

New pools won't be full.  mon->pgmon()->pg_map.pg_pool_sum[poolid] will
implicitly create an entry for poolid causing register_new_pgs() to assume that
the newly created pgs in the new pool are in fact a result of a split
preventing MOSDPGCreate messages from being sent out.

Fixes: #4813
Backport: cuttlefish
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 0289c445be0269157fa46bbf187c92639a13db46)
upstart: handle upper case in cluster name and id

Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>
(cherry picked from commit 851619ab6645967e5d7659d9b0eea63d5c402b15)
PG: don't write out pg map epoch every handle_activate_map

We don't actually need to write out the pg map epoch on every
activate_map as long as:
a) the osd does not trim past the oldest pg map persisted
b) the pg does update the persisted map epoch from time
to time.

To that end, we now keep a reference to the last map persisted.
The OSD already does not trim past the oldest live OSDMapRef.
Second, handle_activate_map will trim if the difference between
the current map and the last_persisted_map is large enough.

Fixes: #4731
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
(cherry picked from commit 2c5a9f0e178843e7ed514708bab137def840ab89)

Conflicts:

	src/common/config_opts.h
	src/osd/PG.cc
	- last_persisted_osdmap_ref gets set in the non-static
	  PG::write_info
PGLog: only add entry to caller_ops in add() if reqid_is_indexed()

Fixes: #5216
Signed-off-by: Samuel Just <sam.just@inktank.com>
Reviewed-by: Sage Weil <sage@inktank.com>
mon: fix uninitialized fields in MMonHealth

Backport: cuttlefish
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit d7e2ab1451e284cd4273cca47eec75e1d323f113)
qa: rsync test: exclude /usr/local

Some plana have non-world-readable crap in /usr/local/samba.  Avoid
/usr/local entirely for that and any similar landmines.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 82211f2197241c4f3d3135fd5d7f0aa776eaeeb6)
os/LevelDBStore: do compact_prefix() work asynchronously

We generally do not want to block while compacting a range of leveldb.
Push the blocking+waiting off to a separate thread.  (leveldb will do what
it can to avoid blocking internally; no reason for us to wait explicitly.)

This addresses part of #5176.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 4af917d4478ec07734a69447420280880d775fa2)
os/LevelDBStore: allow compaction of key ranges

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit e20c9a3f79ccfeb816ed634ca25de29fc5975ea8)
mon/MonitorDBStore: allow compaction of ranges

Allow a transaction to describe the compaction of a range of keys.  Do this
in a backward compatible say, such that older code will interpret the
compaction of a prefix + range as compaction of the entire prefix.  This
allows us to avoid introducing any new feature bits.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit ab09f1e5c1305a64482ebbb5a6156a0bb12a63a4)

Conflicts:

	src/mon/MonitorDBStore.h
mon: compact trimmed range, not entire prefix

This will reduce the work that leveldb is asked to do by only triggering
compaction of the keys that were just trimmed.

We ma want to further reduce the work by compacting less frequently, but
this is at least a step in that direction.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 6da4b20ca53fc8161485c8a99a6b333e23ace30e)
os/LevelDBStore: merge adjacent ranges in compactionqueue

If we get behind and multiple adjacent ranges end up in the queue, merge
them so that we fire off compaction on larger ranges.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit f628dd0e4a5ace079568773edfab29d9f764d4f0)
mon: make compaction bounds overlap

When we trim items N to M, compact over range (N-1) to M so that the
items in the queue will share bounds and get merged.  There is no harm in
compacting over a larger range here when the lower bound is a key that
doesn't exist anyway.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit a47ca583980523ee0108774b466718b303bd3f46)
os/LevelDBStore: add perfcounters

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 7802292e0a49be607d7ba139b44d5ea1f98e07e6)
mon: return instead of exit(3) via preforker

This lets us run all the locally-scoped dtors so that leak checking will
work.

Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit 3c5706163b72245768958155d767abf561e6d96d)