osdc/ObjectCacher: match reads with their original rx buffers

Consider a sequence like:

 1- start read on 100~200
       100~200 state rx
 2- truncate to 200
       100~100 state rx
 3- start read on 200~200
       100~100 state rx
       200~200 state rx
 4- get 100~200 read result

Currently this makes us crash on

osdc/ObjectCacher.cc: 738: FAILED assert(bh->length() <= start+(loff_t)length-opos)

when processing the second 200~200 bufferhead (it is too big).  The
larger issue, though, is that we should not be looking at this data at
all; it has been truncated away.

Fix this by marking each rx buffer with the read request that is sent to
fill it, and only fill it from that read request.  Then the first reply
will fill the first 100~100 extend but not touch the other extent; the
second read will do that.

Signed-off-by: Sage Weil <sage@inktank.com>
osdc/ObjectCacher: do not merge rx buffers

We do not try to merge rx buffers currently.  Make that explicit and
documented in the code that it is not supported.  (Otherwise the
last_read_tid values will get lost and read results won't get applied
to the cache properly.)

Signed-off-by: Sage Weil <sage@inktank.com>
mon/PGMap: OSD byte counts 4x too large (conversion to bytes overzealous)

Fixes: #6049
Signed-off-by: Dan Mick <dan.mick@inktank.com>
pybind: fix Rados.conf_parse_env test

This happens after we connect, which means we get ENOSYS always.
Instead, parse_env inside the normal setup method, which had the added
benefit of being able to debug these tests.

Backport: dumpling
Signed-off-by: Sage Weil <sage@inktank.com>
mon/Paxos: cleanup: use do_refresh from handle_commit

This avoid duplicated code by using the helper created exactly for this
purpose.

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon/Paxos: return whether store_state stored anything

Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon/Paxos: always refresh after any store_state

If we store any new state, we need to refresh the services, even if we
are still in the midst of Paxos recovery.  This is because the
subscription path will share any committed state even when paxos is
still recovering.  This prevents a race like:

 - we have maps 10..20
 - we drop out of quorum
 - we are elected leader, paxos recovery starts
 - we get one LAST with committed states that trim maps 10..15
 - we get a subscribe for map 10..20
   - we crash because 10 is no longer on disk because the PaxosService
     is out of sync with the on-disk state.

Fixes: #6045
Backport: dumpling
Signed-off-by: Sage Weil <sage@inktank.com>
Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
Merge pull request #517 from dmick/wip-6049

mon/PGMap: OSD byte counts 4x too large (conversion to bytes overzealous)

Reviewed-by: Joao Eduardo Luis <joao.luis@inktank.com>
gtest: add build-aux/test-driver to .gitignore

Signed-off-by: Sage Weil <sage@inktank.com>
json_spirit: remove unused typedef

In file included from json_spirit/json_spirit_writer.cpp:7:0:
json_spirit/json_spirit_writer_template.h: In function 'String_type json_spirit::non_printable_to_string(unsigned int)':
json_spirit/json_spirit_writer_template.h:37:50: warning: typedef 'Char_type' locally defined but not used [-Wunused-local-typedefs]
         typedef typename String_type::value_type Char_type;

(Also, ha ha, this file uses \r\n.)

Signed-off-by: Sage Weil <sage@inktank.com>
fuse: fix warning when compiled against old fuse versions

client/fuse_ll.cc: In function 'void invalidate_cb(void*, vinodeno_t, int64_t, int64_t)':
warning: client/fuse_ll.cc:540: unused variable 'fino'

Signed-off-by: Sage Weil <sage@inktank.com>
.gitignore: ignore test-driver

Signed-off-by: Sage Weil <sage@inktank.com>
Merge remote-tracking branch 'gh/wip-6004' into next

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Reviewed-by: Yan, Zheng <zheng.z.yan@intel.com>
Merge remote-tracking branch 'gh/next'
erasure code abstract interface

The erasure coded pool relies on this abstract interface to encode and
decode the chunks stored in the OSD. It has been designed to be
generic enough to accomodate the libraries and algorithms that are
most likely to be used. It does not claim to be universal.

http://tracker.ceph.com/issues/5878 refs #5878

Signed-off-by: Loic Dachary <loic@dachary.org>
default erasure code implementation : M=2 K=1

This is the default erasure code implementation. Although it is fully
functional and could be used on actual data, it is mainly provided for
testing purposes. It splits data in two, computes an XOR parity and
can sustain the loss of one chunk.

http://tracker.ceph.com/issues/5878 refs #5878

Signed-off-by: Loic Dachary <loic@dachary.org>
erasure code plugin system

An erasure coded pool relies on encoding/decoding functions
implemented by a plugin and loaded with something like:

    parameters["directory"] = "/usr/lib/ceph/erasure-code"
    ErasureCodeInterfaceRef erasure_code;
    ErasureCodePlugin::factory(&erasure_code, "example", parameters)

It will then use it thru the ErasureCodeInterface methods:

    virtual int encode(const set<int> &want_to_encode,
		       const bufferlist &in,
		       map<int, bufferlist> *encoded) = 0;

    virtual int decode(const set<int> &want_to_read,
		       const map<int, bufferlist> &chunks,
		       map<int, bufferlist> *decoded) = 0;

For a given plugin, there will be as many ErasureCodeInterface
instances as there are erasure pools, each of them presumably using
a different set of parameters (i.e. M=10, K=5 or M=20, K=5 ).

A ErasureCodePluginRegistry singleton holds all erasure plugin objects
and dlopen(2) handles for the lifetime of the OSD and is cleaned up by
the destructor.

An erasure code plugin must provide the following entry point:

  void __erasure_code_init(char *plugin_name)

which is responsible for registering an object derived from
ErasureCodePlugin in the registry singleton :

  registry.plugins[plugin_name] = new ErasureCodePluginExample();

The ErasureCodePlugin derived object must provide a factory method
from which the concrete implementation of the ErasureCodeInterface
object can be generated:

  virtual int factory(ErasureCodeInterfaceRef *erasure_code,
                      const map<std::string,std::string> &parameters) {
    *erasure_code = ErasureCodeInterfaceRef(new ErasureCodeExample(parameters));
    return 0;
  }

The plugin loader uses the same assumptions as ClassHandler.{cc,h} and
will eventually have to be modified in the same way for better
portability.

A plugin is loaded the first time it is requested so that an OSD which
is not used for erasure coded pool does not load any.

An example plugin based on ErasureCodeExample is provided and used in
the associated unit test.

http://tracker.ceph.com/issues/5878 refs #5878

Signed-off-by: Loic Dachary <loic@dachary.org>