upstart/ceph-osd: make crush update on start optional

By default, we will do some (minimal) CRUSH tree adjustment.  However,
we make this update optional.

Fixes: #3993
Signed-off-by: Sage Weil <sage@inktank.com>
ceph-disk-activate: specify full path for blkid, initctl, service

/sbin apparently isn't in the path when udev runs us.

Signed-off-by: Sage Weil <sage@inktank.com>
ceph-disk-activate: auto detect init system

Look for an option 'init' in ceph.conf.  Otherwise, check if we're ubuntu.
If so, use upstart.  Otherwise, use sysvinit.

Signed-off-by: Sage Weil <sage@inktank.com>
udev: trigger ceph-disk-activate directly from udev

There is no need to depend on upstart for this.

Signed-off-by: Sage Weil <sage@inktank.com>
ceph-disk-activate: catch daemon start errors

Signed-off-by: Sage Weil <sage@inktank.com>
debian: start/stop ceph-all event on install/uninstall

This helps us avoid the confusing situation with upstart where an individual
daemon job is running (like ceph-osd id=2) but the container jobs ceph-osd-all
and ceph-all are not.

Signed-off-by: Sage Weil <sage@inktank.com>
ceph-disk-prepare: do partprobe after setting final partition type

This is necessary to kick udev into processing the updated partition and
running its rules.

Signed-off-by: Sage Weil <sage@inktank.com>
ceph-disk-activate: use full paths for everything

We are run from udev, which doesn't get a decent PATH.

Signed-off-by: Sage Weil <sage@inktank.com>
ceph-disk-activate: pull mount options from ceph.conf

Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>
ceph-disk-prepare: add initial support for dm-crypt

Keep keys in /etc/ceph/dmcrypt-keys.

Identify partition instances by the partition UUID.  Identify encrypted
partitions by a parallel set of type UUIDs.

Signed-off-by: Alexandre Marangone <alexandre.maragone@inktank.com>
Signed-off-by: Sage Weil <sage@inktank.com>
udev: trigger on dmcrypted osd partitions

Automatically map encrypted journal partitions.

For encrypted OSD partitions, map them, wait for the mapped device to
appear, and then ceph-disk-activate.

This is much simpler than doing the work in ceph-disk-activate.

Signed-off-by: Sage Weil <sage@inktank.com>
ceph-disk-prepare: always force mkfs.xfs

Signed-off-by: Alexandre Marangone <alexandre.marangone@inktank.com>
Merge remote-tracking branch 'gh/wip-deploy'

Reviewed-by: Gary Lowell <gary.lowell@inktank.com>
fix operator>=(bufferlist& l, bufferlist& r)

  bufferlist a;
  a.append("A");
  bufferlist ab;
  ab.append("AB");

a >= ab failed, throwing an instance of 'ceph::buffer::end_of_buffer'
because it tried to access a[1]. All comparison operators should be
tested using a lexicographic sort like strcmp or memcmp (-1, 0, 1).
In the meantime, the missing test is added:

  if (l.length() == p && r.length() > p) return false;

A set of unit tests demonstrating the problem and covering all comparison
operators are added to show that the proposed fix works as expected.

http://tracker.ceph.com/issues/4157 refs #4157

Signed-off-by: Loic Dachary <loic@dachary.org>
unit tests for src/common/buffer.{cc,h}

The following static constructors are implemented by opaque classes
defined in buffer.cc ( buffer::raw_char, buffer::raw_posix_aligned
etc. ). Testing the implementation of these classes is done by
variations of the calls to the static constructors.

    copy(const char *c, unsigned len);
    create(unsigned len);
    claim_char(unsigned len, char *buf);
    create_malloc(unsigned len);
    claim_malloc(unsigned len, char *buf);
    create_static(unsigned len, char *buf);
    create_page_aligned(unsigned len);

The raw_mmap_pages class cannot be tested because it is commented out in
raw_posix_aligned. The raw_hack_aligned class is only tested under Cygwin.
The raw_posix_aligned class is not tested under Cygwin.

The unittest_bufferlist.sh script calls unittest_bufferlist with the
CEPH_BUFFER_TRACK=true environment variable to enable the code
tracking the memory usage. It cannot be done within the bufferlist.cc
file itself because it relies on the initialization of a global
variable  ( buffer_track_alloc ).

When raw_posix_aligned is called on DARWIN, the data is not aligned
on CEPH_PAGE_SIZE because it calls valloc(size) which is the equivalent of
memalign(sysconf(_SC_PAGESIZE),size) and not memalign(CEPH_PAGE_SIZE,size).
For this reason the alignment test is de-activated on DARWIN.

The tests are grouped in

TEST(BufferPtr, ... ) for buffer::ptr
TEST(BufferListIterator, ...) for buffer::list::iterator
TEST(BufferList, ...) for buffer::list
TEST(BufferHash, ...) for buffer::hash

and each method ( and all variations of the prototype ) are
included into a single TEST() function.

Although most aspects of the methods are tested, including exceptions
and border cases, inconsistencies are not highlighted . For
instance

    buffer::list::iterator i;
    i.advance(1);

would dereference a buffer::raw NULL pointer although

    buffer::ptr p;
    p.wasted()

asserts instead of dereferencing the buffer::raw NULL pointer. It
would be better to always assert in case a NULL pointer is about to be
used. But this is a minor inconsistency that is probably not worth a
test.

The following buffer::list methods

    ssize_t read_fd(int fd, size_t len);
    int write_fd(int fd) const;

are not fully tested because the border cases cannot be reliably
reproduced. Going thru a pointer indirection when calling the ::writev
or safe_read functions would allow the test to create mockups to synthetize
the conditions for border cases.

tracker.ceph.com/issues/4066 refs #4066

Signed-off-by: Loic Dachary <loic@dachary.org>