mds: journal original dirfrags for rollback

Fragments with different 'bits' can be merged into one fragment. So we can't
use 'basefrag' and 'bits' to calculate the original fragments when rolling
back a merge operation.

We also can't rely on MDCache::adjust_dir_fragments() to restore the original
fragments' statistics during rolling back. This is because fragments are not
always complete in the rollback case.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: delete orphan dirfrags during MDS recovers

This patch make the MDS use following steps to fragmentate directory.

1. freeze the old dirfrags
2. journal EFragment::OP_PREPARE
3. store the new dirfrags
4. journal EFragment::OP_COMMIT
5. delete the old dirfrags
6. journal EFragment::OP_FINISH

The newly introduced event EFragment::OP_FINISH indicates that all orphan
frags have been deleted. The new process guarantees that orphan frags can
be properly deleted if the MDS crashes while fragmentating directory.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: trim log segment after finishing uncommitted fragments

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: properly store fragmenting dirfrags

fragmenting dirfrag does not exist on the object store. So all non-null
dentries should be included when committing a fragmenting dirfrag.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: properly update rstat/fragstat when fragmentating directory

The stale rstat/dirstat check in CDir::merge() is wrong. dirfrag's
rstat/fragstat is stale if the accounted rstat/fragstat version isn't
equal to inode's rstat/dirstat version.

For CDir::split(), no need to worry about if the rstat/fragstat of the
origin dirfrag is stale. If it's stale, the rstat/fragstat of the
resulting dirfrags are stale too.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: don't fragmentate stray directories

the code that prepares and purges stray dentry assumes that we never
freeze stray directories. So disable fragmentating stray directories
temporarily.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: fix MDCache::merge_dir()

fragment 'bits' should be negative for the merging case.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: fix CInode::get_dirfrags_under()

make sure it return true when all dirfrags under the given frag_t
are found.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: handle dirfrag mismatch when processing readdir request

If client has outdated dirfrags information, readdir request from
it may specify a non-existing dirfrag. Current method to handle this
case is reply -EAGAIN and let client retry. When client receives the
the -EAGAIN reply, it need to refresh its dirfrags information first,
then re-send the readdir request.

A better way to handle client request that specify a non-existing
dirfrag is: MDS chooses a approximate dirfrag, then handle the request
like normal. When client receives the readdir reply, it will also
update its dirfrags information.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
client: use dir_result_t::END as flag

So we don't lose the latest readdir frag and offset after marking
end of readdir.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
client: handle dirfrag mismatch when processing readdir reply

If client has outdated directory fragments information, it may request
readdir an non-existent directory fragment. In this case, the MDS finds
an approximate directory fragment and sends its contents back to the
client. When receiving a reply with fragment that is different than the
requested one, the client need to reset the 'readdir offset'.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
fragtree_t: fix get_leaves_under()

If fragtree is (*^1, 1*^1) and we want leaves under frag 000*.
get_leaves_under() return frag 0*, frag 10* and frag 11*. This is
obviously wrong.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: don't scatter gather update bare-bones dirfrags

avoid adding bare-bones dirfrag that has not yet been fetched from
the disk to the journal.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: stop propagating rstat if freezing/frozen dirfrag is encountered.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: fix bloom filter leaks

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: reduce verbosity of dir fragment logging

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
qa/workunits: Add large number of files in a directory test script

Test the directory fragments feature

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>