unit tests for FileStore::_detect_fs when running on ext4

unit tests are added in test/filestore/store_test.cc for the
FileStore::_detect_fs method, when using ext4. It tests the following
situations:

 * without user_xattr, ext4 fails
 * mounted with user_xattr, ext4 fails if filestore_xattr_use_omap is false
 * mounted with user_xattr, ext4 succeeds if filestore_xattr_use_omap is true

The tests are grouped in an ext4 dedicated class

  TEST(EXT4StoreTest, _detect_fs)

The qa/workunits/filestore/filestore.sh script is added to prepare the
environment required for the unit tests ( create an image file,
formats it with ext4 etc.). It runs ceph_test_filestore with a sudo
to allow it to mount(2) and umount(2) the ext4 file system. It is
called with

  ceph_test_filestore --gtest_filter=EXT4StoreTest.*

to only run the ext4 dependent tests.

The filestore.sh script is meant to be used as part of teuthology
in order to increase the code coverage for src/os/FileStore.cc. It is
self tested and can be checked from the source directory with

  CEPH_TEST_FILESTORE=../../../src/ceph_test_filestore filestore.sh TEST

http://tracker.ceph.com/issues/4321 refs #4321

Signed-off-by: Loic Dachary <loic@dachary.org>
mds: Locker needs to remember requested max_size changes from clients

Previously, if we received an MClientCaps request containing a change
in the inode's max size, and _do_cap_update() was unable to process
the request immediately (due to a locking issue), we would wait-list
the request by adding a call to check_inode_max_size() once the lock
became stable. However, we then tossed out the message without in any
way propagating the new max size which had been requested!

Handle this by extending check_inode_max_size to also accept parameters
for increasing the max size, and by storing all the parameters explicitly
in the C_MDL_CheckMaxSize Context instead of relying on defaults. That
gets us to the point where we *can* notice we need to increase the max. To
actually do so, we now pass calc_new_client_ranges() the requested max
size instead of the actual size if we're doing an update.

Notice that as a side effect of this, all clients get to see the max size
increase instead of just the requester. This should be okay, but it is
chattier than in the optimal case (where we don't get stuck on a lock).

Fixes #3637

Signed-off-by: Greg Farnum <greg@inktank.com>
mds: only go through the max_size change rigamarole if the client requested it

The previous patch was forcing a new size change even if we were
doing it as part of our regular optimistic settings; we don't much
want to do that. This is a small optimization, but Sage asked for
it and it's very easy.

Signed-off-by: Greg Farnum <greg@inktank.com>
os: bring leveldbstore options up to date

LevelDB has a lot of options which we don't implement right now. Add
an options struct to the LevelDBStore which users can access as they
wish in order to set values different from the defaults.
This will let us set various size values, as well as turning on
caching or bloom filter read optimizations.

Signed-off-by: Jim Schutt <jaschut@sandia.gov>
Signed-off-by: Greg Farnum <greg@inktank.com>
config: provide settings for the LevelDB stores we use

Now that we can set up the LevelDB options internally, provide
config options on the OSD and the Monitor. We leave the OSD values
at the defaults for now as they're performance-sensitive, but we
set new values on the Monitor so that it can scale to large PGMaps.
(Previously there were issues with large PGMaps taking forever to write;
these changes to the use of compression and the default block and
write buffers counteract them.)

Since we pass these variables through, users who are interested in
doing so now can test and tune them more appropriately.

Reported-by: Jim Schutt <jaschut@sandia.gov>
Signed-off-by: Greg Farnum <greg@inktank.com>
Merge pull request #215 from ceph/wip-leveldb-config

os: bring leveldbstore options up to date

Reviewed-by: Sage Weil <sage@inktank.com>
osd/PG.cc: initialize PG::flushed in constructor

Initialize PG::flushed in constructor with false as
described in doc/dev/osd_internals/pg.rst .

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
(cherry picked from commit fb840c8ff75b0c66dfeed48e8558542fe3da4c24)
mon: PaxosService: add helper function to check if a given version exists

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mon: Monitor: convert osdmap_full as well

Store conversion wasn't converting the osdmap_full/ versions, only the
incrementals under osdmap/ and the latest full version stashed.  This
would lead to some serious problems during OSDMonitor's update_from_paxos
when the latest stashed didn't correspond to the first available
incremental.

Fixes: #4521

Signed-off-by: Joao Eduardo Luis <joao.luis@inktank.com>
mds: pass proper mask to CInode::get_caps_issued

There is a total of 22 cap bits and file lock uses 8 cap bits.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Greg Farnum <greg@inktank.com>
mds: change XLOCK/XLOCKDONE's next state to LOCK

For simplelock and filelock, XLOCK/XLOCKDONE's next state is SYNC.
But filelock in XLOCK/XLOCKDONE state allow Fb caps, filelock in
SYNC state does not. So filelock can be stuck in XLOCK/XLOCKDONE
state forever if there are Fb caps issued.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
Reviewed-by: Sage Weil <sage@inktank.com>
Merge remote-tracking branch 'gh/wip-4521-fix' into next

Reviewed-by: Sage Weil <sage@inktank.com>
leveldbstore: handle old versions of leveldb

The filter_policy (bloom filter) stuff is fairly new in LevelDB's life,
and it turns out that precise's version is too old for it. Add conditional
compilation for those members in order to build and work properly.

Signed-off-by: Greg Farnum <greg@inktank.com>
rbd: add --no-progress switch

Disable progress output to stderr.t

Signed-off-by: Sage Weil <sage@inktank.com>
Merge pull request #219 from ceph/wip-rbd-progress

Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
client: disable invalidate callbacks :(

See #4746; it deadlocks right now.

Signed-off-by: Greg Farnum <greg@inktank.com>
rbd: Only allow shrinking an image when --allow-shrink flag is passed

Signed-off-by: Wido den Hollander <wido@widodh.nl>
Reviewed-by: Sage Weil <sage@inktank.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
doc: Removed fragmented logging info. Consolidated into one doc.

Logging was variously described in the ceph configuration document,
a configuration reference, and a section in operations. Since
logging and debugging are generally used with troubleshooting,
I consolidated the docs and placed them in the troubleshooting
section. Also fixed the example and provided additional detail.

fixes: #3804

Signed-off-by: John Wilkins <john.wilkins@inktank.com>
doc: Updated hyperlink.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>
doc: Removed. Not in toc, and otherwise generates a warning.

Signed-off-by: John Wilkins <john.wilkins@inktank.com>