LU-3336 lfsck: regenerate lost layout EA

To find out orphan OST-objects, the LFSCK on OST side maintains
two bitmaps in RAM for the OST-object accessed during the LFSCK.
After the first cycle system scanning, the LFSCK got the bitmap
for the known OST-objects, and got another bitmap for which OST
objects have been referenced by MDT-objects. Then the LFSCK can
know which OST-objects are not referenced by any MDT-object via
comparing the two bitmaps.

If the MDT-object exists, then check the layout EA. If related
layout EA is empty, then fill the layout EA slot with the given
OST-object stripe information. If the given OST-object stripe
index exceeds current layout EA size, then extend the layout EA
and fill the gap slot(s) as empty, which can be filled by others.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ie542289b397576be457b04792e845324d6926836
Reviewed-on: http://review.whamcloud.com/7810
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3336 lfsck: namespace visible lost+found directory

The LFSCK will create the lost MDT-objects under a namespace visible
directory, then the admin can future handle those orphans with human
knowledge later.

The directory is named as "lost+found" under ".lustre", which is not
the same as backend (for ldiskfs case) "/lost+found". The latter one
is invisible from Lustre namespace. Under ".lustre/lost+found/", the
LFSCK will create sub-directory "MDTxxxx" for each MDT. (xxxx is the
MDT number)

The ".lustre/lost+found" uses special FID:

enum dot_lustre_oid {
        FID_OID_DOT_LUSTRE      = 1UL,
        FID_OID_DOT_LUSTRE_OBF  = 2UL,
        FID_OID_DOT_LUSTRE_LPF  = 3UL,
};

const struct lu_fid LU_LPF_FID = { .f_seq = FID_SEQ_DOT_LUSTRE,
                                   .f_oid = FID_OID_DOT_LUSTRE_LPF,
                                   .f_ver = 0x0000000000000000 };

Both ".lustre" and its child "lost+found" reside on the MDT0. As for
the "lost+found"'s children "MDTxxxx" uses normal FID and resides on
the corresponding MDT. The orphans will be linked under the "MDTxxx"
by the LFSCK on such MDT when it find the orphans.

The ".lustre/lost+found" is created by the MDT0 when mdd_prepare().
The ".lustre/lost+found/MDTxxxx" is created by the LFSCK on related
MDT when it needs to process orphans. (create on demand)

The permission for ".lustre/lost+found" is "r-x------" for root user.
Only the LFSCK can create sub-directories "MDTxxxx" under it. All the
other modifications are denied.

The permission for ".lustre/lost+found/MDTxxxx" is "rwx------" for
root user. Any modification is the same as operated on normal file.

The lost+found mechanism is not only for LFSCK phase II but also for
other LFSCK orphan handling. Since it is visible to Lustre namespace,
the admin can operate the orphans as other normal files easily.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Idd37c87c2517175022127a2a605ebb3ddf1f223a
Reviewed-on: http://review.whamcloud.com/8305
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3336 lfsck: recreate the lost MDT-object

If the MDT-object does not exist but related OST-obejct still
alive, then when handle the orphan OST-object, the MDT-object
will be re-created under /.lustre/lost+found/MDTxxxx with the
given OST-object stripe and owner information. The other part
of the MDT-obejct layout EA will be empty which may be filled
by other orphan OST-object(s) as the LFSCK processing.

The name for the above re-created MDT-object will be named as
${MOUNT_POINT}/.lustre/lost+found/MDTxxxx/N-${FID}, the "FID"
is the MDT-object's fid. If the OST-object has zero PFID (for
some reason), then LFSCK will assign new FID to the parent.

If the MDT-object exists, but related layout EA slot is occupied
by other OST-object which may be created by former LFSCK running
for repairing dangling referenced MDT-object or may be not.

For the latter case, the LFSCK will create a new MDT-object with
the given OST-object under /.lustre/lost+found/MDTxxxx with name
"C-${FID1}-${FID2}-${idx}":
FID1: the new MDT-object's fid.
FID2: the original MDT-object's fid.
idx: index in the layout EA.

For the former one, then means the former LFSCK guess was wrong.
If such new created OST-object has never been modified which can
be indicated with lu_attr::la_ctime is zero and lu_attr::la_mode
has no "S_ISUID" set, then the LFSCK will destroy it and replace
it with the orphan OST-object; otherwise the LFSCK will keep the
new data, and create a new MDT-object to reference the orphan as
describe above with name "C-${FID1}-${FID2}-${idx}".

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ia7700f7b2e4e09d6b576d025295fdb5c10ca90e0
Reviewed-on: http://review.whamcloud.com/7811
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4604 lfsck: LFSCK async updates RPC flow control

Control the max in flight async updates RPCs count in OSP layer to
avoid too much in flight RPCs causes memory issue and OST overload.

Currently, we use a semaphore for that which is initialized as 512.

It also contains the layout LFSCK performance tests:
1) lfsck_layout performance under single MDS mode with kinds of
   stripe counts and kinds of file sets, for routine check case
   and repairing case.

2) lfsck_layout performance under DNE mode with kinds of file
   sets, for routine check case and repairing case.

3) lfsck_layout impact on create performance.

This patch also fixes some bugs related with lu_device_type
operations smooth the lfsck performance test.

There was no protection when inc/dec lu_device_type::ldt_device_nr,
which may caused the ldt_device_nr to be wrong and trigger assert.
This patch redefine lu_device_type::ldt_device_nr as atomic type.

There was no protection when add/del lu_device_type::ldt_linkage
into/from the global lu_device_types list, which may caused bad
address accessing. This patch uses the existing obd_types_lock
to protect related operations.

We do NOT need lu_types_stop() any longer. Such function scans
the global lu_device_types list, and for each type item on it
which has zerod lu_device_type::ldt_device_nr, call its stop()
method. In fact, the lu_device_type::ldt_device_nr only will be
zero when the last lu_device_fini() is called, and at that time,
inside the lu_device_fini(), its stop() method will be called.
So it is unnecessary to call the stop() again via lu_types_stop().

Test-Parameters: mdtcount=2 ostcount=2 envdefinitions=SLOW=yes testlist=lfsck-performance
Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ib9bae6ad0446e8705fa2767d080150e82a495e2f
Reviewed-on: http://review.whamcloud.com/8694
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3539 protocol: Change UPDATE_OBJ RPC format

This patch updates struct update for DNE phase II.  See the DNE phase
II HLD for more information.

The comment describing the UPDATE_OBJ RPC format is already a bit out
of date.  This patch simplies and updates it so that the overall
structure is easier to see while the details are commented in-line.

The related type names are modified to be more consistent with the
terminology used in the HLD and the comment.

Change OBJ_XXX-->OUT_XXX, and refine a few update structures to
object_update_xxx. And there are no logic changes in this patch.

Change-Id: I4355b4c863abb448713115f17b80402ec96009f8
Signed-off-by: Di Wang <di.wang@intel.com>
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/7128
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>