LU-3489 osd-zfs: varsize index support

LFSCK needs indices with keys/values of different size.
add a support for this to osd-zfs. key/value size can't
vary within a single index object so far.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I6faa730ce1f778563c1fcf43d3b50c5b62c899d9
Reviewed-on: http://review.whamcloud.com/7152
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4334 utils: Only set a single property for nodes

For servicenode, failnode and mgsnode only set a single property but
concatenate data.  ZFS can only set a single property for a given
name so this prevents erroneously overwritting previous entries.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ia0316e138b0d68dcda1d14811e43db3bbed64457
Reviewed-on: http://review.whamcloud.com/11161
Tested-by: Jenkins
Reviewed-by: Isaac Huang <he.huang@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3660 acl: support mount option "noacl" for zfs

zfs parse mount option in zpl layer, but osd-zfs calls dmu
directly, since ldiskfs parse "noacl" at mount time, osd-zfs
should do the same in osd_mount().

Similar to ldiskfs, osd-zfs should check whether ACL is enabled
for ACL operations, if not -EOPNOTSUPP is returned.

add sanity test 103b for "noacl" mount option.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I11075a9de8759fde956e7ae8b715840a56d0428e
Reviewed-on: http://review.whamcloud.com/11157
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Li Wei <wei.g.li@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5150 acl: 'cp -rp' set empty default ACL

'cp -rp' will set an empty default ACL on target, MDD should treat
this as removing specified ACL.

add sanity 103c to verify this.

Signed-off-by: Lai Siayo <lai.siyao@intel.com>
Change-Id: I2b6947e241675c291a43999e6324bbb295d3c88d
Reviewed-on: http://review.whamcloud.com/11158
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Li Wei <wei.g.li@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5434 mdd: disregard empty POSIX ACLs

Some files may have a POSIX ACL consisting of only a struct
posix_acl_xattr_header, resulting in unexpected permission errors.
Update mdd_check_acl() to return -EAGAIN when such ACLs are
encountered so the caller falls back to standard UNIX
permissions.

These empty ACLs originate from tools like cp. In some cases (e.g.
'cp -rp') just POSIX_ACL_XATTR_VERSION is given as the value of the
system.posix_acl_default extended attribute.  Lustre accepts and
stores this value on disk, and the bad default ACL then propagates
from directories to newly created directory entries.  Separate
patches will prevent the storage of these bad ACLs.  This patch
mitigates problems caused by the ones already stored.

Other minor updates:
- Use the kernel function posix_acl_xattr_count() to count the
  number of ACL entries.
- Convert remainder of mdd_check_acl() to TAB indentation.

Signed-off-by: Ned Bass <bass6@llnl.gov>
Change-Id: Ib4f7b3874696b7645b9de3f1f10ef45dc4105646
Reviewed-on: http://review.whamcloud.com/11300
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5612 osd: typo in osd_declare_write()

Some bits are lost when porting the fix of LU-4611
3902ff4c54925b2f1fcb732a32ed7ee5428e9f77

That could result in insufficient credits in certain situation.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I828a75c674978278b5adaf210586baa177ea0dc9
Reviewed-on: http://review.whamcloud.com/11889
Tested-by: Jenkins
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5600 kernel: kernel update RHEL6.5 [2.6.32-431.29.2.el6]

Update RHEL6.5 kernel to 2.6.32-431.29.2.el6

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I4832cf12cb49eebd1c81fc4e07a53d0a1315500d
Reviewed-on: http://review.whamcloud.com/11838
Tested-by: Jenkins
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5610 tests: Handle quoted module options

When test-framework.sh translates module options to environment
variables for remote nodes, quotes should be escaped to the subshell.

This patch is back-ported from the following one:
Lustre-commit: 4597789df0b6b9fcc7bcbe5529adab3522f0e6ac
Lustre-change: http://review.whamcloud.com/11887

Signed-off-by: Stephen Champion <schamp@sgi.com>
Change-Id: I02daf5ca36170ea901f6b0e4e5152cf4835945dc
Signed-off-by: Jian Yu <jian.yu@intel.com>
Reviewed-on: http://review.whamcloud.com/11907
Tested-by: Jenkins
Reviewed-by: Stephen Champion <that.schamp@gmail.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5349 include: use __vmalloc() to avoid __GFP_FS default

When possible, try to Direct use of __vmalloc() which allows
for protection flag specification, and particularly to not
set __GFP_FS, which can cause some deadlock situations in our
code.

This patch is the b2_5 back-port of :
Lustre-commit: 635421641bfa8a412edebd4ffbbdf05fd0c47410
Lustre-change: http://review.whamcloud.com/11190/

Also corrects a typo in __OBD_VMALLOC_VERBOSE name.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I588ebf27d1ff7538745b732d174696c379107ea3
Reviewed-on: http://review.whamcloud.com/11739
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Revert "LU-3489 osd-zfs: varsize index support"

This seems to be causing LU-5649

This reverts commit d9e627c09ccc5c22aa10704020fb9f45bcf26ed0.
LU-5040 osd: fix osd declare credit for quota

osd_attr_set() always calls ll_vfs_dq_init() to initialize dquot for
the inode, while in some cases osd_declare_attr_set() does not reserve
credit for it.

This patch fixes this issue. This patch also corrects the quota credit
accounting in osd_declare_qid() by judging whether the inode's
i_dquot[quota_type] has been allocated, while old code judgement is
based on the file uid/gid existence, which is not correct.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I16555cb1097e1a3e75cdcb4852a2c5e1772ddd88
Reviewed-on: http://review.whamcloud.com/11097
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5114 utils: Fix autoconf warning

Fix mount_osd_ldiskfs_LDADD autoconf warning.  This corrects a fix
left out of commit 0d77156572c2b77581f01d960c0e5738b369de6a.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I2815f093951c215a8c92678348444cbd18297e25
Reviewed-on: http://review.whamcloud.com/12005
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4725 mdt: child-parent lock ordering in rename

change rename so that it always has parent-child lock ordering,
otherwise it may deadlock with other operations.

Lustre-commit: 4e308ef74f271ec7e19917e3c0f88586537582c3
Lustre-change: http://review.whamcloud.com/9538/

LU-4725 obd: lu_object_find_at hung

lu_object_find_at hungs if called 2 times and object is removed
in between. It makes mdt_rename_sanity not workable for rename.
Change mdt_rename_sanity to work on existing object.

Lustre-commit: b7c72ec1ddeda2cf94ea151f974d3f94e3a7d1ed
Lustre-change: http://review.whamcloud.com/10484/

Test-Parameters: alwaysuploadlogs \
envdefinitions=SLOW=yes,ENABLE_QUOTA=yes,ONLY=54 \
ossjob=lustre-b2_4 mdsjob=lustre-b2_4 ossbuildno=73 mdsbuildno=73 \
testlist=sanityn

Signed-off-by: Rahul Deshmukh <rahul_deshmukh@xyratex.com>
Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Change-Id: Ic9ce52bfcd8788834347fba155cc8c6be674dcd8
Reviewed-on: http://review.whamcloud.com/11615
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5144 mdt: rename vs link deadlock

it may happen that tgt child for rename is a directory, which is used
as tgt dir for link in parallel, as the result rename locks src child,
tgt child what may deadlock with tgt dir, src file locking of link.

As not dir cannot replace dir, check it before locking the tgt child.

Lustre-commit: b2c2cfc24fa461b3e54cd204a697eeee9a89d49c
Lustre-change: http://review.whamcloud.com/10570/

Test-Parameters: alwaysuploadlogs \
envdefinitions=SLOW=yes,ENABLE_QUOTA=yes,ONLY=55 \
ossjob=lustre-b2_4 mdsjob=lustre-b2_4 ossbuildno=73 mdsbuildno=73 \
testlist=sanityn

Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Signed-off-by: Rahul Deshmukh <rahul_deshmukh@xyratex.com>
Change-Id: I09e96b132d9069d4818e8dc2628c903eff38e53d
Reviewed-on: http://review.whamcloud.com/10918
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4855 fld: refer to MDT0 for fld lookup in some cases

It is possible that when fld client is trying to lookup seq
on one of MDT, but the connection between the client and the MDT
is not being initialized yet, especially during striped dir creation,
because client will only send create req to the master MDT, then
master MDT will distribute the operation to all of other MDT, instead
of client distributing these requests, which will usually trigger
the connection.

In this case, we will send the fld request to MDT0, since it has
all of location information.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I577ada2a782ecbdc863d05e8d007c0914a1e7837
Reviewed-on: http://review.whamcloud.com/9877
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/11780
LU-4933 osc: Automatically increase the max_dirty_mb

When RPC size or the max RPCs in flight is increased, the actual
limit might be max_dirty_mb. This patch automatically increases
the max_dirty_mb value at connection time and when the related
values are tuned manually by proc file system.

This patch also changes the unit of "cl_dirty" and "cl_dirty_max"
in client_obd from byte to page.

Lustre-change: http://review.whamcloud.com/10446/
Lustre-commit: bdc5bb52c55470cf8020933f80e327c397810603

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Change-Id: I8480122b7370247b17de81d731c2f2b5f67892ce
Reviewed-on: http://review.whamcloud.com/10937
Tested-by: Jenkins
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
LU-5405 llog: add newly opened llog at tail of handle list

Add newly opened llog handle at the tail of handle list to increase
lookup speed, especially for cancel operation, because the canceled
log will be removed from the list, and lookup is from the beginning.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Signed-off-by: Li Xi <lixi@ddn.com>
Change-Id: I4c5c22901ea5818a8ee50ef97d6dabe4839b9a74
Reviewed-on: http://review.whamcloud.com/11575
Tested-by: Jenkins
Reviewed-by: Ian Costello <costello.ian@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

Conflicts:
	lustre/obdclass/llog_cat.c
LU-5428 libcfs: false alarm of libcfs watchdog

lc_watchdog_disable will not delete kernel timer, benefit of this
is no overhead of del_timer in each serivce loop, however, there
is a corner case that lc_watchdog_touch can race with lcw_cb when
service thread is actually idle:

1. service thread armed a timer before processing request, after
   it handled request, it called lc_watchdog_disable() and set
   lc_watchdog as disabled, but kernel timer is still outstanding.
2. service thread slept for many seconds because there is no
   request to handle
3. it's waken up by incoming request
4. it called lc_watchdog_touch and set watchdog status to enabled.
   now because timer is still alive, if timer is expired right
   before cfs_timer_arm(), then lcw_cb will set a disabled watchdog
   as "expired", which is wrong.
5. the next call of lc_watchdog_disable will complain for expired
   watchdog.

There are two options to fix this issue:
1. always del_timer in lc_watchdog_disable, however, it may increase
   system overhead when service threads are busy.
2. set watchdog as "enabled" after cfs_timer_arm, so there is no
   window for lcw_cb to see false status of watchdog.

This patch chooses the second approach to fix the problem

Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Change-Id: Iac24082e2d63de8330285cf243ed585da6524ab9
Reviewed-on: http://review.whamcloud.com/11415
Tested-by: Jenkins
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-1279 kernel: Fix concurrent module loading deadlocks

Concurrently starting multiple OSTs on a single OSS frequently
triggers 30s deadlocks on module_mutex.  This RHEL 6 kernel bug
applies to any module that results in additional request_module()
calls in its init callback.  In Lustre, at least ptlrpc and libcfs are
affected.  (RHEL 7 should have enough fixes in this area, but testing
needs to be done.)  This patch adds a fix adapted from a number of
upstream commits to the RHEL 6 kernel patch series.

Change-Id: Ibdd384fd7622a0b4fcbf4cb5fdb864de87fcc25e
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/11229
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-5299 obdclass: avoid race during Server device start

Handle concurrent starts for same device (multiple mounts, ...).
But allows for separate nosvc and nomgs case.
Also add a specific test of concurent MDT/OST start with an
artificial delay to verify.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I442819a5b865ed3e98477f9d2602efc4d09d7860
Reviewed-on: http://review.whamcloud.com/11139
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>