update hadoop-wordcount test to be able to run on hadoop 2.x. The hadoop and mapreduce library are no longer hard coded so they can be specified to point to the right path. The relative paths hdfs are changed to absolute paths.
A sample command to run the test on hadoop 2.x is
TESTDIR=/home/test HADOOP_HOME=/usr/lib/hadoop HADOOP_MR_HOME=/usr/lib/hadoop-mapreduce sh workunits/hadoop-wordcount/test.sh starting hadoop-wordcount test

Signed-off-by: rootfs <hchen@redhat.com>
utf8: export encode_utf8() and decode_utf8()

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
rgw: improve delmited listing of bucket

If found a prefix, calculate a string greater than that so that next
request we can skip to that. This is still not the most efficient way to
do it. It'll be better to push it down to the objclass, but that'll
require a much bigger change.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
rgw: add NextMarker param for bucket listing

Partially fixes #8858.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
rgw: account common prefixes for MaxKeys in bucket listing

To be more in line with the S3 api. Beforehand we didn't account the
common prefixes towards the MaxKeys (a single common prefix counts as a
single key). Also need to adjust the marker now if it is pointing at a
common prefix.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
rgw: list extra objects to set truncation flag correctly

Otherwise we end up returning wrong truncated value, and no data on the
next iteration.

Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
rgw: dump prefix unconditionally

As part of issue #8858, and to be more in line with S3, dump the Prefix
field when listing bucket even if bucket is empty.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
osd/MemStore: handle collection_move_rename within the same collection

Signed-off-by: Sage Weil <sage@redhat.com>
os/MemStore: fix wrlock ordering checks

We can't compare the shared_ptrs themselves; we need to compare the
addresses of the actual objects.

Signed-off-by: Sage Weil <sage@redhat.com>
os/MemStore: copy attrs on clone

Backport: firefly
Signed-off-by: Sage Weil <sage@redhat.com>
ceph_test_objectstore: test memstore

Signed-off-by: Sage Weil <sage@redhat.com>
Reduce ObjectCacher flush overhead

Flush op in ObjectCacher will iterate the whole active object set, each
dirty object also may own several BufferHead. If the object set is large,
it will consume too much time.

Use dirty_bh instead to reduce overhead. Now only dirty BufferHead will
be checked.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Add rbdcache max dirty object option

Librbd will calculate max dirty object according to rbd_cache_max_size, it
doesn't suitable for every case. If user set image order 24, the calculating
result is too small for reality. It will increase the overhead of trim call
which is called each read/write op.

Now we make it as option for tunning, by default this value is calculated.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Wait tx state buffer in flush_set

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
ObjectCacher: fix dirty_or_tx_bh logic in bh_set_state()

The else-if chain here was wrong. Handling dirty or tx buffers and
errors should be in independent conditions.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
ObjectCacher: fix bh_{add,remove} dirty_or_tx_bh accounting

tx buffers need to go on the bh_lru_rest as well, and removing erases
(not inserts) them into dirty_or_tx_bh.

Signed-off-by: Josh Durgin <josh.durgin@inktank.com>
mon: AuthMonitor: always encode full regardless of keyserver having keys

On clusters without cephx, assuming an admin never added a key to the
cluster, the monitors have empty key servers.  A previous patch had the
AuthMonitor not encoding an empty keyserver as a full version.

As such, whenever the monitor restarts we will have to read the whole
state from disk in the form of incrementals.  This poses a problem upon
trimming, as we do every now and then: whenever we start the monitor, it
will start with an empty keyserver, waiting to be populated from whatever
we have on disk.  This is performed in update_from_paxos(), and the
AuthMonitor's will rely on the keyserver version to decide which
incrementals we care about -- basically, all versions > keyserver version.

Although we started with an empty keyserver (version 0) and are expecting
to read state from disk, in this case it means we will attempt to read
version 1 first.  If the cluster has been running for a while now, and
even if no keys have been added, it's fair to assume that version is
greater than 0 (or even 1), as the AuthMonitor also deals and keeps track
of auth global ids.  As such, we expect to read version 1, then version 2,
and so on.  If we trim at some point however this will not be possible,
as version 1 will not exist -- and we will assert because of that.

This is fixed by ensuring the AuthMonitor keeps track of full versions
of the key server, even if it's of an empty key server -- it will still
keep track of the key server's version, which is incremented each time
we update from paxos even if it is empty.

Fixes: #8851
Backport: dumpling, firefly

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
powerdns: RADOS Gateway backend for bucket directioning

This backend can be used to create one global namespace for multiple
RGW regions.

Using a CNAME DNS response the traffic is directed towards the RGW region
without using HTTP redirects.
uncomment cleanup command
Merge pull request #2105 from rootfs/wip-qa-hadoop-wordcount

update hadoop-wordcount test to be able to run on hadoop 2.x. 

Reviewed-by: Sage Weil <sage@redhat.com>