test/librados/cmd.cc: tolerate thrashing on pg_command tests

We may get ENXIO (osd down) or ENOENT (pg dne (yet) on the target osd) if
there is thrashing going on.

Fixes: #6122
Signed-off-by: Sage Weil <sage@inktank.com>
mds: start internal MDS request for fragmentating directory

Start internal MDS request for fragmentating directory operation. With
MDS request, we can easily acquire locks required by the fragmentating
directory operation. (The old way to get locks is 'try lock' style,
which is not reliable)

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: delete orphan dirfrags after fragmentating directory

delete old dirfrags after the EFragment::OP_COMMIT event is logged.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: journal original dirfrags for rollback

Fragments with different 'bits' can be merged into one fragment. So we can't
use 'basefrag' and 'bits' to calculate the original fragments when rolling
back a merge operation.

We also can't rely on MDCache::adjust_dir_fragments() to restore the original
fragments' statistics during rolling back. This is because fragments are not
always complete in the rollback case.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: delete orphan dirfrags during MDS recovers

This patch make the MDS use following steps to fragmentate directory.

1. freeze the old dirfrags
2. journal EFragment::OP_PREPARE
3. store the new dirfrags
4. journal EFragment::OP_COMMIT
5. delete the old dirfrags
6. journal EFragment::OP_FINISH

The newly introduced event EFragment::OP_FINISH indicates that all orphan
frags have been deleted. The new process guarantees that orphan frags can
be properly deleted if the MDS crashes while fragmentating directory.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: trim log segment after finishing uncommitted fragments

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: properly store fragmenting dirfrags

fragmenting dirfrag does not exist on the object store. So all non-null
dentries should be included when committing a fragmenting dirfrag.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: properly update rstat/fragstat when fragmentating directory

The stale rstat/dirstat check in CDir::merge() is wrong. dirfrag's
rstat/fragstat is stale if the accounted rstat/fragstat version isn't
equal to inode's rstat/dirstat version.

For CDir::split(), no need to worry about if the rstat/fragstat of the
origin dirfrag is stale. If it's stale, the rstat/fragstat of the
resulting dirfrags are stale too.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: don't fragmentate stray directories

the code that prepares and purges stray dentry assumes that we never
freeze stray directories. So disable fragmentating stray directories
temporarily.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: fix MDCache::merge_dir()

fragment 'bits' should be negative for the merging case.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: fix CInode::get_dirfrags_under()

make sure it return true when all dirfrags under the given frag_t
are found.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: handle dirfrag mismatch when processing readdir request

If client has outdated dirfrags information, readdir request from
it may specify a non-existing dirfrag. Current method to handle this
case is reply -EAGAIN and let client retry. When client receives the
the -EAGAIN reply, it need to refresh its dirfrags information first,
then re-send the readdir request.

A better way to handle client request that specify a non-existing
dirfrag is: MDS chooses a approximate dirfrag, then handle the request
like normal. When client receives the readdir reply, it will also
update its dirfrags information.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
client: use dir_result_t::END as flag

So we don't lose the latest readdir frag and offset after marking
end of readdir.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
client: handle dirfrag mismatch when processing readdir reply

If client has outdated directory fragments information, it may request
readdir an non-existent directory fragment. In this case, the MDS finds
an approximate directory fragment and sends its contents back to the
client. When receiving a reply with fragment that is different than the
requested one, the client need to reset the 'readdir offset'.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
fragtree_t: fix get_leaves_under()

If fragtree is (*^1, 1*^1) and we want leaves under frag 000*.
get_leaves_under() return frag 0*, frag 10* and frag 11*. This is
obviously wrong.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: don't scatter gather update bare-bones dirfrags

avoid adding bare-bones dirfrag that has not yet been fetched from
the disk to the journal.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: stop propagating rstat if freezing/frozen dirfrag is encountered.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: fix bloom filter leaks

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: reduce verbosity of dir fragment logging

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
qa/workunits: Add large number of files in a directory test script

Test the directory fragments feature

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>