LU-3333 ptlrpc: Protect request buffer changing

*_enlarge_reqbuf class of functions can change request body location
for a request that's already in replay list, as such a parallel
traverser of the list (after_reply -> ptlrpc_free_committed) might
access freed and scrambled memory causing assertion.

Since all such users only can get to this request under imp_lock, take
imp_lock to protect against them in *_enlarge_reqbuf

Change-Id: I0fa690aabd8696a9f05b94c66e06e30eefb5c759
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/10074
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-4808 tests: sanity cleanup to work on Xeon Phi

* Made the test_74c robust to different platforms.
* Fix an issue with values more than 2^32 (for example 9.36473e+09).
* Fix an integer comparison in case of empty variable.
* Check for tools presents (rsync and getfattr).
* Check for CLIENTONLY mode.
* Remove "-x" options from grep (it's not always supported).
* Coding style cleanup.

Lustre-commit: e217648d50da55366ef819e78bb8d7601a109077
Lustre-change: http://review.whamcloud.com/9766

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: I8e553da96afd78e96c9c534898db7173aa268223
Reviewed-on: http://review.whamcloud.com/10086
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
LU-4351 tests: skip loop test in SLES11 clients

Avoid known failure in test 54c due to SLES specific
incompatible implementation flaw in kernel loop devices.

Lustre-commit: f5683b151be0e8ba0d41e85a8f99850c7d00e55c
Lustre-change: http://review.whamcloud.com/9955

Test-Parameters: clientdistro=sles11sp3 testlist=sanity
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: If39aa85e8c1f468df532aa93610d3eb7cd8d32f5
Reviewed-on: http://review.whamcloud.com/10085
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
LU-4257 clio: replace semaphore with mutex

According https://www.kernel.org/doc/Documentation/mutex-design.txt:
 - the mutex subsystem is slightly faster and has better scalability
   for contended workloads. In terms of 'ops per CPU cycle', the
   semaphore kernel performed 551 ops/sec per 1% of CPU time used,
   while the mutex kernel performed 3825 ops/sec per 1% of CPU time
   used - it was 6.9 times more efficient.
 - there are no fastpath tradeoffs, the mutex fastpath is just as
   tight as the semaphore fastpath. On x86, the locking fastpath is
   2 instructions.
 - 'struct mutex' semantics are well-defined and are enforced if
   CONFIG_DEBUG_MUTEXES is turned on. Semaphores on the other hand
   have virtually no debugging code or instrumentation.

One more benefit of mutex is optimistic spinning. It try to spin for
acquisition when there are no pending waiters and the lock owner is
currently running on a (different) CPU. The rationale is that if the
lock owner is running, it is likely to release the lock soon.

This significantly reduce amount of context switches when locked
region is small and we have high contention.

Lustre commit: 96f482b509867bb70cc26e94e60903df82b09f30
Lustre change: http://review.whamcloud.com/9095

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Signed-off-by: Shuichi Ihara <sihara@ddn.com>
Change-Id: I29cb202468610636ebd45d49281c4e503930bde6
Reviewed-on: http://review.whamcloud.com/9943
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4719 mdt: Don't try to print non-existent objects

mdt_dump_lmm() needs to make sure it's not trying to print
objects for a default striping from a directory, since there
are not objects in this case.

Return early if the D_INFO debug mask is disenabled, so it
doesn't iterate through the LOV/LMV layout at all.  That is
just a waste of effort.

Lustre-change: http://review.whamcloud.com/#/c/9520
Lustre-commit: ebfc6f4b54961e5e0762af7d2555f2dc5d1e1ec3

Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Change-Id: Ifbda7c0bfdec74c5ee4c097ea24613a4ce692572
Reviewed-on: http://review.whamcloud.com/10021
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Matt Ezell <ezellma@ornl.gov>
Tested-by: Maloo <hpdd-maloo@intel.com>
LU-4008 mdt: Shrink default LOVEA reply buffers

Instead of allocating maximum possible LOVEA for intent requests,
allocate a sane default to fit up to 100 stripes, should the size
be bigger, the buffer reallocation code will kick in.

Additionally, since quite awhile ago we no longer send
unlink cookies or EA data to the clients from unlinks/renames,
so we no longer need to allocate reply buffer for them in
mdt_close() and other such places.
Also clean up unused fields related to these buffers in
md_attr structure.

Lustre-commit: 00260466f9c84f21e6db55d5dd6e87e8a9906438
Lustre-change: http://review.whamcloud.com/9322

Change-Id: I06b9f5665ec2616cd9a5d483872369d710400d22
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/9942
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
LU-4008 mdt: Use actual size for layout lock intents

The intent for layout looks up the size of the lov xattr. That size
should be used for the reply buffer instead of the maximum size ever
seen.

Lustre-commit: 586a609c9d5a53e75174b66269c685f594ded7c2
Lustre-change: http://review.whamcloud.com/#/c/9571

Signed-off-by: Matt Ezell <ezellma@ornl.gov>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: Id5d560001df3d058d189339ede33347f8353990a
Reviewed-on: http://review.whamcloud.com/10018
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4708 ldiskfs: do not copy NUL terminator from direntry

Do not try to copy a NUL terminator from the source dirent if there
is no dir_data for that record.  This can happen with 1.x filesystems
that do not have a dirdata field in the dirent to store the FID.

The target buffer is expected to be NUL terminated, and this is
handled by using kzalloc() to allocate it and zero out the buffer,
but the source buffer may not have a trailing NUL byte if it is a
multiple of 4 bytes in length.

If the dirent is aligned at the end of the directory block/page this
can cause an oops by accessing beyond the end of the page, or will
otherwise copy a garbage byte to the target buffer.

Lustre commit: a101a95a3f58cde3ed912d074dc283b4083672ff
Lustre change: http://review.whamcloud.com/9510

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Iab52f916d7571b3d2587255280c72a52c60399c9
Reviewed-on: http://review.whamcloud.com/9510
Tested-by: Jenkins
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/9662
LU-4791 lod: subtract xattr overhead when calculate max EA size.

1. It needs to subtract the xattr overhead when calculate the max
EA size, otherwise, it might get oversize stripe_count, which cause
ENOSPC error in the following xattr_set.
2. Release stripe information, if xattr set fails.

Lustre-change: http://review.whamcloud.com/9835
Lustre-commit: a11c69ad81ca8071b9d97d5032539dd7b36873ee

Change-Id: I4a4fe0681f2cd9dd6f242a03c75326acde02ae05
Signed-off-by: wang di <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/10020
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5048 ldiskfs: SLES ext4-mballoc-extra-checks patch fix

Original SLES patch version, from LU-1812, was missing a latest hunk.
This caused OSTs to be put in read-only mode due to wrong
inconsistencies detected.

This patch is back-ported from the following one:
Lustre-commit: 62993e6261f552baddb1035ffda5a037427bbcc8
Lustre-change: http://review.whamcloud.com/10294

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: Ic7de0e7f957da2d0a8ebecf7a3f15f39bbec3db5
Reviewed-on: http://review.whamcloud.com/10304
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5025 kernel: kernel update [RHEL6.5 2.6.32-431.17.1.el6]

update RHEL6.5 kernel to 2.6.32-431.17.1.el6

Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I0c89b09f40cae082dd03a1effb80fe36c3ea0897
Reviewed-on: http://review.whamcloud.com/10283
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Jenkins
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2099 osd: clear iobuf up on I/O completion

it was a bad idea to clear iobuf on buffer release because the
same buffers can be used few times in case of transient ENOSPC.

Change-Id: Ic1d2ddaa0794f2f6f7d1ba71620e06b4ea7a99d2
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
LU-4783 mgs: start params log during mgs initialization

Params log was introduced by LU-3155 to set parameters permanently
with lctl "set_param -P" instead of "conf_param". But in a newly
mounted filesystem, the log won't be created automatically until "lctl
set_param -P" is called. This causes that the other servers can't make
a local copy successfully during their setup, and the misleading
console error message "LustreError: 13a-8: Failed to get MGS log
params and no local copy." is printed.
This patch is to start params log during mgs intialization so that the
error message is quieted.
Also, test_76b is added to conf-sanity.sh to verify params log setup.

Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: Ifc3abc1e35ef20230a1be1cdf1c4cc8f57b2788e
Reviewed-on: http://review.whamcloud.com/9737
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5005 utils: handle ROOT path correctly

MDT replies path '\0' to the request of fid2path on ROOT,
client utils should handle it properly.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Ic7064866b0e8ef33fc0a17e273a120d6d6dae4fe
Reviewed-on: http://review.whamcloud.com/10211
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4743 osp: continue llog processing

when unexpected record type is met

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I3dd45f501192bc6dd3bfddb550fd455ae319b923
Reviewed-on: http://review.whamcloud.com/9574
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
LU-4778 llog: update catlog when plain log destroyed

When the empty plain log is destroyed while processing
catlog, we should call llog_cat_cleanup() to close the
plain log and update in catlog.

Intel-bug-id: INTL-64

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I97b1d5cc59e84fdb4888274b48f223a063293647
Reviewed-on: http://review.whamcloud.com/9702
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3254 llite: access layout version under a lock

We used to access layout version under the protection of ldlm
lock, this introduces extra overhead for dlm lock matching.

In this patch, lli_layout_lock is introduced to access the layout
version. Also, when a layout lock is losing, we should tear down
mmap of the correspoding inode to avoid stale data accessing in the
future.

This is part of technical verification of replication.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I657c6a8b27f2cdaf468dc208c43c1d92f3843705
Reviewed-on: http://review.whamcloud.com/8689
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4065 mdt: missing NL in CERROR format

Add a missing 'n' at the end of the CERROR format parameter to
avoid annoying "format at mdt_coordinator.c:977:mdt_hsm_cdt_start
doesn't end in newline" msg in syslog and the consequence of 2
merged lines/traces in Lustre debug-log.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: Ic8baf81886b5be3de32ee595be99e20428729f6d
Reviewed-on: http://review.whamcloud.com/9597
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
LU-4799 llite: Do not rate limit dirty page discard warning

Messages which are printed by ll_dirty_page_discard_warn() should not
be rate limited.  If they are rate limited, some files which may be
corrupted on client eviction will not be reported to the user.

This patch changes the CWARN to a CDEBUG to disable console message
rate limiting for this message.  The dirty page discard warnings are
already limited on a per-file basis by the function vvp_vmpage_error
which calls ll_dirty_page_discard_warn only if the ccc_object's
cob_discard_page_warned == 0.

Signed-off-by: Ryan Haasken <haasken@cray.com>
Change-Id: Ia8a13c7add89c890014170254e462ae861de0fc5
Reviewed-on: http://review.whamcloud.com/9752
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Cory Spitz <spitzcor@cray.com>
Reviewed-by: Ann Koehler <amk@cray.com>
Reviewed-by: Chris Horn <hornc@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4397 utils: fix lfs_df loop for disconnected client

If the client is disconnected from the MDT on which "lfs df" is
run (typically MDT0 for the mountpoint), then the file open can
fail with -ESHUTDOWN, causing mntdf() to loop forever trying to
find other MDTs or OSTs to get statfs data from.

The caller of llapi_obd_statfs() cannot tell the difference between
an error from open() and ioctl(IOC_OBD_STATFS), so llapi_obd_statfs()
should return a fatal error if the open failed since all of the
later opens will fail as well.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I05d48d5b6680c382c9e05c02fd82a9a200500c1e
Reviewed-on: http://review.whamcloud.com/8949
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>