LU-5200 llite: Cmp of unsigned value against 0 is always true

Comparison of unsigned value against 0 is always true.
desc->bd_iov_count declared as int and atomic_sub_return() have
first argument int also.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I89cedc4991d1a117f824ce6b5d8cf045595fdc0f
Reviewed-on: http://review.whamcloud.com/11217
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4993 osd-ldiskfs: Support 3.14 kernel changes to bio api

In the 3.14 kernel code base several data fields in
struct bio were moved into a new structure called
bvec_iter. This patch updates osd-ldiskfs to handle
this api change.

Change-Id: I849a1d62462c58a79766c176060b27c621627646
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/10995
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3270 statahead: statahead thread wait for RPCs to finish

Statahead thread should wait for inflight stat RPCs to finish in
case statahead RPC callback may access data allocated in statahead
thread context.

ll_sa_entry_fini() should keep old entry if stat RPC is not
finished yet.

Simplify sai refcounting:
* newly allocated sai will hold one refcount, and it will put it
  after starting statahead thread.
* statahead thread holds one refcount.
* agl thread holds one refcount.
* stat process calls do_statahead_enter() which will try to get
  sai, and if it's valid, it will revalidate from statahead cache,
  and put refcount after use.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I55a4fe66a5f6c04595d3bc84f0cd3750f20e0ee4
Reviewed-on: http://review.whamcloud.com/9663
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4334 utils: Only set a single property for nodes

For servicenode, failnode and mgsnode only set a single property but
concatenate data.  ZFS can only set a single property for a given
name so this prevents erroneously overwritting previous entries.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ia0316e138b0d68dcda1d14811e43db3bbed64457
Reviewed-on: http://review.whamcloud.com/11161
Tested-by: Jenkins
Reviewed-by: Isaac Huang <he.huang@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5275 libcfs: cleanup the proc hash and cfs wrappers

Remove the non seq file version of the cfs_hash functions.
Remove the basic cfs wrappers for procfs definations.

Change-Id: Ie4996656267998b78e44cdedfbf972f358e04d50
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/11743
Tested-by: Jenkins
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3676 llite: to configure max_cached_mb correctly

If there exists MGS conf_param to reduce the memory cache
max_cached_mb it will fail because dt_exp is not initialized
yet.

It should just go ahead and configure it because certainly it
have enough free LRU slot to deduct ccc_lru_left.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I317e556413bc63086378dec294d8ba2792afca52
Reviewed-on: http://review.whamcloud.com/11783
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-4416 osd-ldiskfs: limit trans size to match reservations

Linux commit 8f7d89f36 (jbd2: transaction reservation support)
shrunk the limit of a transaction size to be
j_max_transaction_buffers / 2 to accomodate the reservations.
This can have a perforamce impact so we limit this change to
3.10+ kernels. The same commit that changed this is the same
one that changed the number of arguments to ext4_journal_start
function so we use that test for this. This patch adjusts the
limit in osd_trans_start to match depending on the kernel
version.

Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: I10a65474dbaf7b67fe7cd59cc2544759d1e3270d
Reviewed-on: http://review.whamcloud.com/10376
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-3192 osc: split different type of IO

Do not allow different type of pages at the same rpc.

Signed-off-by: Alexander Boyko <alexander_boyko@xyratex.com>
Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Xyratex-bug-id: MRP-859
Change-Id: Ic595a29f685757faf4d9c3de9c2d2ae8fd039baf
Reviewed-on: http://review.whamcloud.com/10930
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5592 ofd: fix incorrect check for NULL

Fix incorrect check for NULL in ofd_version_get_check()

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: Ia671f86ab051eafa5336ba18f0378cdac846f0c5
Reviewed-on: http://review.whamcloud.com/11773
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5496 ldlm: reconstruct proper flags on enqueue resend

otherwise, waiting lock may get granted as no BLOCKED_GRANTED
flag is returned

Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Xyratex-bug-id: MRP-1944
Change-Id: I5e938ff0454d5e8694b09f9fff3c1f82d746360d
Reviewed-on: http://review.whamcloud.com/11644
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5520 ldlm: resend AST callbacks

While clients will resend client->server RPCs, servers would not
resend server->client RPCs such as LDLM callbacks (blocking
or completion callbacks/ASTs).  This could result in clients being
evicted from the server if blocking callbacks were dropped by the
network (a failed router or lossy network) and the client did not
cancel the requested lock in time.
In order to fix this problem, this patch adds the ability to resend
LDLM callbacks from the server and give the client a chance to
respond within the timeout period before it is evicted:

- resend BL AST within lock callback timeout period;
- still do not resend CANCEL_ON_BLOCK;
- regular resend for CP AST without BL AST embedded;
- prolong lock callback timeout on resend;

some fixes:
- recovery-small test_10 to actually evict the client
  with dropped BL AST;
- ETIMEDOUT to be returned if send limit is expired;

Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Change-Id: Ie65fac94ea68defffd1769cbbb0f74381c11262c
Tested-by: Elena Gryaznova <Elena_Gryaznova@xyratex.com>
Reviewed-by: Alexey Lyashkov <Alexey_Lyashkov@xyratex.com>
Reviewed-by: Andriy Skulysh <Andriy_Skulysh@xyratex.com>
Xyratex-bug-id: MRP-417
Reviewed-on: http://review.whamcloud.com/9335
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5600 kernel: kernel update RHEL6.5 [2.6.32-431.29.2.el6]

Update RHEL6.5 kernel to 2.6.32-431.29.2.el6

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I4832cf12cb49eebd1c81fc4e07a53d0a1315500d
Reviewed-on: http://review.whamcloud.com/11837
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Revert "LU-5261 osc: use wait_for_completion_killable() instead"

This is causing LU-5446

This reverts commit 2b3663dda896f669c87feb49e7f3c7d85a89cefe.

Change-Id: I8bd254137ad0d402bad5f5aac85aa52cd3d47f63
Reviewed-on: http://review.whamcloud.com/11892
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>
LU-1669 llite: Replace write mutex with range lock

Testing has shown the ll_inode_inode's lli_write_mutex to be a
limiting factor with single shared file write performance, when using
many writing threads on a single machine. Even if each thread is
writing to a unique portion of the file, the lli_write_mutex will
prevent no more than a single thread to ever write to the file
simultaneously.

This change attempts to remove this bottle neck, by replacing this
mutex with a range lock. This should allow multiple threads to write
to a single file simultaneously iff the threads are writing to unique
regions of the file.

Performance testing shows this change to garner a significant
performance boost to write bandwidth. Using FIO on a single machine
with 32 GB of RAM, write performance tests were run with and without
this change applied; the results are below:

    +---------+-----------+---------+--------+--------+--------+
    |     fio v2.0.13     |        Write Bandwidth (KB/s)      |
    +---------+-----------+---------+--------+--------+--------+
    | # Tasks | GB / Task | Test 1  | Test 2 | Test 3 | Test 4 |
    +---------+-----------+---------+--------+--------+--------+
    |    1    |    64     |  452446 | 454623 | 457653 | 463737 |
    |    2    |    32     |  850318 | 565373 | 602498 | 733027 |
    |    4    |    16     | 1058900 | 463546 | 529107 | 976284 |
    |    8    |     8     | 1026300 | 468190 | 576451 | 963404 |
    |   16    |     4     | 1065500 | 503160 | 462902 | 830065 |
    |   32    |     2     | 1068600 | 462228 | 466963 | 749733 |
    |   64    |     1     |  991830 | 556618 | 557863 | 710912 |
    +---------+-----------+---------+--------+--------+--------+

 * Test 1: Lustre client running 04ec54f. File per process write
           workload. This test was used as a baseline for what we
           _could_ achieve in the single shared file tests if the
           bottle necks were removed.

 * Test 2: Lustre client running 04ec54f. Single shared file
           workload, each task writing to a unique region.

 * Test 3: Lustre client running 04ec54f + I0023132b. Single shared
           file workload, each task writing to a unique region.

 * Test 4: Lustre client running 04ec54f + this patch.
           Single shared file workload, each task writing to a unique
           region.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: I71e060c190065d87a20dc8df3104f898883d0583
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-on: http://review.whamcloud.com/6320
Tested-by: Jenkins
Reviewed-by: Hiroya Nozaki <nozaki.hiroya@jp.fujitsu.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-1669 vvp: Use lockless __generic_file_aio_write

Testing multi-threaded single shard file write performance has shown
the inode mutex to be a limiting factor when using the
generic_file_aio_write function. To work around this bottle neck, this
change replaces the locked version of that call with the lock less
version, specifically, __generic_file_aio_write.

In order to maintain posix consistency, Lustre must now employ it's
own locking mechanism in the higher layers. Currently writes are
protected using the lli_write_mutex in the ll_inode_info structure.
To protect against simultaneous write and truncate operations, since
we no longer take the inode mutex during writes, we must down the
lli_trunc_sem semaphore.

Unfortunately, this change by itself does not garner any performance
benefits. Using FIO on a single machine with 32 GB of RAM, write
performance tests were ran with and without this change applied; the
results are below:

    +---------+-----------+---------+--------+--------+
    |     fio v2.0.13     |   Write Bandwidth (KB/s)  |
    +---------+-----------+---------+--------+--------+
    | # Tasks | GB / Task | Test 1  | Test 2 | Test 3 |
    +---------+-----------+---------+--------+--------+
    |    1    |    64     |  452446 | 454623 | 457653 |
    |    2    |    32     |  850318 | 565373 | 602498 |
    |    4    |    16     | 1058900 | 463546 | 529107 |
    |    8    |     8     | 1026300 | 468190 | 576451 |
    |   16    |     4     | 1065500 | 503160 | 462902 |
    |   32    |     2     | 1068600 | 462228 | 466963 |
    |   64    |     1     |  991830 | 556618 | 557863 |
    +---------+-----------+---------+--------+--------+

 * Test 1: Lustre client running 04ec54f. File per process write
           workload. This test was used as a baseline for what we
           _could_ achieve in the single shared file tests if the
           bottle necks were removed.

 * Test 2: Lustre client running 04ec54f. Single shared file
           workload, each task writing to a unique region.

 * Test 3: Lustre client running 04ec54f + this patch. Single shared
           file workload, each task writing to a unique region.

In order to garner any real performance benefits out of a single
shared file workload, the lli_write_mutex needs to be broken up into a
range lock. That would allow write operations to unique regions of a
file to be executed concurrently. This work is left to be done in a
follow up patch.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: I0023132b5d941b3304f39f015f95106542998072
Reviewed-on: http://review.whamcloud.com/6672
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5499 tests: keep /sbin/mount.lustre until cleanup

Don't unmount /sbin/mount.lustre in the middle of running tests
on a local test system if it is not doing final cleanup.  Otherwise,
later mounts may fail.

The current /sbin/mount.lustre mountpoint is an empty stub that
returns success (0) if executed, but doesn't mount the filesystem.
Instead, create a mountpoint that prints an message if executed and
returns an error to the caller, so it is easier to debug problems.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Ied7b69f536bad87333cf5c543384723412500c1e
Reviewed-on: http://review.whamcloud.com/11259
Reviewed-by: frank zago <fzago@cray.com>
Tested-by: Jenkins
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4746 libcfs: Use Linux kernel current_umask() function

Lustre not using kernel current_umask() function breaks GRSecurity
umask handling. This is also needed for the linux api cleanup.

Replaces current->fs->umask with more secure current_umask() function

Change-Id: Ide0b83eb3e6c69e1e2178ede37ce708227f1c107
Signed-off-by: Andrew Prout <ajprout@hotmail.com>
Signed-off-by: Cliff White <cliffwhi@intel.com>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/11642
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
LU-5596 lnet: Remove obsolete LNET variable

For Lustre version 2.6.50 and later, the variable
session_features is defined as "LST_FEATS_MASK". For
earlier versions of Lustre, the same variable is defined
as "LST_FEATS_EMPTY.

Since Lustre master is at 2.6.52 or later, the second
definition of "session_features" can be removed.

Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: I52e7a914880509cfcd4961032ab7775bbaf626a8
Reviewed-on: http://review.whamcloud.com/11825
Tested-by: Jenkins
Reviewed-by: Isaac Huang <he.huang@intel.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5610 tests: Handle quoted module options

When test-framework.sh translates module options to environment
variables for remote nodes, quotes sould be escaped to the subshell.

Signed-off-by: Stephen Champion <schamp@sgi.com>
Change-Id: I937cc28b96b54ea75082c7d8789c762b4db16c5f
Reviewed-on: http://review.whamcloud.com/11887
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5508 osp: RPC adjustment for remote transaction

1) For remote transaction, the set_attr/set_xattr RPC should not be
   prepared in declare phase. According to our current transaction/
   dt_object_lock framework, the transaction sponsor will start the
   transaction firstly, then try to acquire related dt_object_lock
   if needed. That is a general rule, and the LFSCK needs to follow
   such rule when repair inconsistent linkEA, in spite of local or
   remote MDT-object.

   For linkEA repairing case, before the LFSCK thread obtained the
   dt_object_lock on the target MDT-object, it cannot know whether
   the MDT-object has linkEA or not, neither invalid or not.

   Since the LFSCK cannot hold dt_object_lock before the (remote)
   transaction start (otherwise there will be potential deadlock),
   it cannot prepare related RPC for repairing during the declare
   phase as other normal transactions do.

   To resolve the trouble, we will make OSP to prepare related RPC
   (set_attr/set_xattr/del_xattr) after remote transaction started,
   and trigger the remote updating (RPC sending) when trans_stop.
   Then the up layer users, such as LFSCK, can follow the general
   rule to handle trans_start/dt_object_lock for repairing linkEA
   inconsistency without distinguishing remote MDT-object.

2) Some adjustment for OSP object attributes cache maintainig to make
   the logic more clear and reasonable.
2.1) Update the cached attribute in osp_attr_set(), but not in the
     osp_declare_attr_set().
2.2) Update the cached extended attribute in osp_xattr_set(), but not
     in the osp_declare_xattr_set().
2.3) Drop the cached extended attribute in osp_xattr_del().

3) Typo fixing and code cleanup.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I43c88a8fd3b184c91a4b3cbd4104e35f9915ee24
Reviewed-on: http://review.whamcloud.com/11382
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>