LU-5727 ldlm: revert changes to ldlm_cancel_aged_policy()

The changes to ldlm_cancel_aged_policy() intrdouced from LU-4300
(bfae5a) was incorrect. This patch revert this part of changes.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Ic04f894c8bf54dbbdfb0e3bada5f6e4a367711a5
Reviewed-on: http://review.whamcloud.com/12448
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5517 lfsck: repair invalid nlink count

If the namespace LFSCK has verified all the known name entries during
the first-stage scanning, then the MDT-object's linkEA is trustable.
So if the non-directory MDT-object's nlink attribute does not match
the MDT-object linkEA entries count, then the LFSCK will repair the
MDT-object's nlink attribute according to its linkEA entries count.

One exception is that: the linkEA space is limited, if there are too
much hard links on the MDT-object and exceeds the object's linkEA
space limitation, then some name entries cannot be recorded in the
linkEA. Under such case, we will add some flags (LLF_SKIP_NLINK)
in the LFSCK tracing file for related MDT-objects. Then the LFSCK
can skip the nlink attribute verification for the marked MDT-objects
during the second-stage scanning.

This patch also cleanup the LFSCK environment when current LFSCK
scanning exits (completed/stopped/failed) to avoid some stale to
misguide the next LFSCK scanning.

This patch also makes some code adjustment for the former landed
LFSCK patches according to the inspection feedback.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Iedc676e8cc06a52f55e82372e6dc8b30008e20f4
Reviewed-on: http://review.whamcloud.com/11516
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5518 lfsck: recover orphans from backend lost+found

Some local filesystem consistency verification tools, such as
e2fsck for ldiskfs, will add the orphan objects under backend
special /lost+found directory. Such directory is invisible to
clients. The namespace LFSCK will scan such directory, and for
the objects that were visible to clients originally (according
to the FID), the namespace LFSCK will move them back to the
normal namespace (according to the linkEA) or to the global
visible .lustre/lost+found/MDTxxxx/ directory.

The namespace LFSCK will insert the orphan (that is under the
backend /lost+found directory) FID into the namespace LFSCK
tracing file for further processing (via the subsequent namespace
LFSCK second-stage scanning). At the same time, remove the orphan
name entry from backend /lost+found directory. There is an interval
between the orphan name entry removed from the backend /lost+found
directory and the orphan FID in the LFSCK tracing file handled. In
such interval, the LFSCK can be reset, then all the FIDs recorded
in the namespace LFSCK tracing file will be dropped. To guarantee
that the orphans can be found when LFSCK run next time without
e2fsck again, when remove the orphan name entry, the LFSCK will
set the orphan's ctime attribute as 1. Since normal applications
cannot change the object's ctime attribute as 1. Then when LFSCK
run next time, it can record the object (that ctime is 1) in the
namespace LFSCK tracing file during the first-stage scanning. Once
the FID in the LFSCK tracing file has been handled successfully,
then the object's ctime attribute will be changed to normal time.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I2f8e23801a868f2c99c630face4bbfe08ece8844
Reviewed-on: http://review.whamcloud.com/11536
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5519 lfsck: LFSCK code framework adjustment (2)

Enhance LFSCK code framework for striped directory verification,
including:

1) New lfsck component APIs: lfsck_operations::lfsck_open_dir()
   and lfsck_operations::lfsck_close_dir().
1.1) The namespace LFSCK uses lfsck_open_dir() to initialize the
     LMV EA verification environment.
1.2) The namespace LFSCK uses lfsck_close_dir() to indicate the
     end of the scanning the striped directory.

2) When create orphan parent, the LMV EA should be considered.

3) Data structure definition and extention.

4) Other code cleanup.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I01e14645796c3fa3269c5f3cdcbb755ccb455d7e
Reviewed-on: http://review.whamcloud.com/11845
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5675 quota: correctly set II_FL_NONUNQ in dt_index_read()

Fix regression introduced by 60e07b972114df24105a3a1bfa7365892f72a4a7
II_FL_NONUNQ should be set in the index information only when the
backend index supports DT_IND_NONUNQ.
This mistake triggers the following warning in the qsd code:

Lustre: 25029:0:(qsd_reint.c:237:qsd_reint_index()) lustre-MDT0000:
II_FL_NONUNQ is set on index transfer for fid
[0x200000005:0x1011:0x0], it shouldn't be

Signed-off-by: Johann Lombardi <johann.lombardi@intel.com>
Change-Id: I42a9c70666c741064b356ffd253b49e5ed7b3e93
Reviewed-on: http://review.whamcloud.com/12074
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5626 ldiskfs: update non-htree dotdot in rename

In 2.4+, when renaming a directory, its old dotdot entry will
be removed firstly, then the new dotdot entry is inserted, and
ldiskfs tries to append FID-in-dirent to the new entry.
But the space for dotdot entry may not be enough to hold
the new dotdot with FID-in-dirent, such as an MDT device
restored from file-level backup, or a device upgraded from 1.8.

In that case, for non-HTree directories, the ".." entry
will be written in the next available space in the directory
block.  This is invalid, as the ".." entry must be the
second entry in the block.

The same bug was fixed for HTree directories in LU-2638.
As Fan Yong said then: we do not want to introduce
complex logic to handle directory data moving, instead, in
such case, ignore the FID-in-dirent for the new dotdot entry,
and just insert the new dotdot entry.

There is one known flaw: This patch, like the one for
LU-2638, skips the entire data section rather than just
the FID.  This could cause trouble if something else ever
uses this section with ".." entries.

Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: I57fc492e694973f5020191e4e2c79c74c7c4f18c
Reviewed-on: http://review.whamcloud.com/11939
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
LU-4942 at: per-export lock callback timeout

The lock callback timeout is calculated as an average per namespace.
This does not reflect individual client behavior.
Instead, we should calculate it on a per-export basis.

Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Change-Id: I12e3fc5f8d261cce252fcf13f22193273dc054ee
Tested-by: Elena Gryaznova <Elena_Gryaznova@xyratex.com>
Reviewed-by: Andriy Skulysh <Andriy_Skulysh@xyratex.com>
Reviewed-by: Alexey Lyashkov <Alexey_Lyashkov@xyratex.com>
Xyratex-bug-id: MRP-417
Reviewed-on: http://review.whamcloud.com/9336
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
LU-5674 test: print spl debug info

Print /proc/spl files in case of tests fail

Signed-off-by: Minh Diep <minh.diep@intel.com>
Change-Id: I33890eedb9e5a2f87ae94cddd1db346c3c17c06d
Reviewed-on: http://review.whamcloud.com/11580
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Reviewed-by: Isaac Huang <he.huang@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5287 export: hold exp_lock when modify exp_flags

There are few places that changing the exp_flags without holding
exp_lock, that could overwrite other concurrent exp_flags updating.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I9c0421ffeacd6bf6feaa31dca9f9fc0e7963bc8b
Reviewed-on: http://review.whamcloud.com/11871
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5641 tests: ensure user daemon is in group bin

sanity test_103a assumes user daemon is a member of group bin.
This addition makes that so even in el7 where it's
not true by default.

Test-Parameters: clientdistro=el7 mdsfilesystemtype=ldiskfs mdtfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I180479e80d4da5f1bfec98c25676feaae85c29d7
Reviewed-on: http://review.whamcloud.com/12044
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
LU-5743 build: Update to zfs/spl 0.6.3-1.1

Update ZFS and SPL version to the latest released patch version of
0.6.3-1.1

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I19ec6da5c431b41e741b469e43f5373502d43130
Reviewed-on: http://review.whamcloud.com/12273
Tested-by: Maloo <hpdd-maloo@intel.com>
Tested-by: Jenkins
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5756 hsm: add missing return code in llapi_hsm_copytool_register

llapi_hsm_copytool_register() returns a bogus error code on one error
path.

Removed a never true "if" case.

If the LL_IOC_HSM_CT_START fails, then we have to go to out_kuc,
not out_err.

Change-Id: I738078392f98d4616f6451beece4f0d97bc8051d
Signed-off-by: frank zago <fzago@cray.com>
Reviewed-on: http://review.whamcloud.com/12314
Reviewed-by: Henri Doreau <henri.doreau@cea.fr>
Tested-by: Jenkins
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5568 lnet: fix kernel crash when network failed to start

When loading Lustre modules without proper network configuration,
it always hit the following kernel panic:

LNetError: 105-4: Error -100 starting up LNI tcp
LNetError: 2145:0:(api-ni.c:823:lnet_unprepare())
 ASSERTION( list_empty(&the_lnet.ln_nis) ) failed:
LNetError: 2145:0:(api-ni.c:823:lnet_unprepare()) LBUG
Pid: 2145, comm: modprobe
x0aCall Trace:
[<ffffffffa044f853>] libcfs_debug_dumpstack+0x53/0x80 [libcfs]
[<ffffffffa044fdf5>] lbug_with_loc+0x45/0xc0 [libcfs]
[<ffffffffa04f3267>] lnet_unprepare+0x297/0x340 [lnet]
[<ffffffffa04f3b5c>] LNetNIInit+0x25c/0x3e0 [lnet]
[<ffffffff81061bc6>] ? put_online_cpus+0x56/0x80
[<ffffffffa0983000>] ? init_module+0x0/0x1000 [ptlrpc]
[<ffffffffa081310c>] ptlrpc_ni_init+0x2c/0x1a0 [ptlrpc]
[<ffffffffa0983000>] ? init_module+0x0/0x1000 [ptlrpc]
[<ffffffffa0813291>] ptlrpc_init_portals+0x11/0xf0 [ptlrpc]
[<ffffffffa0983000>] ? init_module+0x0/0x1000 [ptlrpc]
[<ffffffffa09831c4>] init_module+0x1c4/0x1000 [ptlrpc]
[<ffffffff810020e2>] do_one_initcall+0xe2/0x190
[<ffffffff810ca7fb>] load_module+0x129b/0x1a90
[<ffffffff812da590>] ? ddebug_dyndbg_module_param_cb+0x0/0x60
[<ffffffff810c7133>] ? copy_module_from_fd.isra.43+0x53/0x150
[<ffffffff810cb1a6>] SyS_finit_module+0xa6/0xd0
[<ffffffff815f2119>] system_call_fastpath+0x16/0x1b
...

This is because in lnet_startup_lndnis(), we may add list items to
@the_lnet.ln_nis and @the_lnet.ln_nis_cpt before it failed. But in
lnet_startup_lndis() failure path,it did not cleanup list thus
causing assertion in lnet_unprepare().

Fix this problem by:
1) move lnet_shutdown_lndnis() back to lnet_startup_lndnis() so
that lnet_startup_lndnis() will cleanup itself.

2) move codes in lnet_startup_lndnis() that starts a single
NI into a new function called lnet_startup_lndni().

3)make lnet_dyn_add_ni() call lnet_startup_lndni() instead of
lnet_startup_lndnis().

This patch also fix problem LU-5734 addressed since they are
closely related.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: I1082361626881e798fca49981fe92b4082769ecf
Reviewed-on: http://review.whamcloud.com/11718
Tested-by: Jenkins
Reviewed-by: Amir Shehata <amir.shehata@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Revert "LU-5568 lnet: fix kernel crash when network failed to start"

Isaac has found many issues with the patch and advise to revert it.

This reverts commit 8fab48a8ef0bad6961c2ca1e2959726252ba43ac.

Change-Id: I9d8422e018d9dc809eed53dcfe376cb5154181af
Reviewed-on: http://review.whamcloud.com/12502
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5584 llite: ensure all data flush out when umount

Write out all extents when clear inode. Otherwise we
may lose data while umount.

Change-Id: I199217c96a7ba49c2f3577a65750e7956a9af63a
Signed-off-by: Yang Sheng <yang.sheng@intel.com>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-on: http://review.whamcloud.com/12103
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>