LU-744 obdclass: revise cl_page refcount

By holding a refcount for radix tree, we can remove a couple of
spin lock and atomic operation for cl_page.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Signed-off-by: Gregoire Pichon <gregoire.pichon@bull.net>
Change-Id: I4d8dc04b9b4ef066b54a90be3cfbd0c09d96c320
Reviewed-on: http://review.whamcloud.com/5429
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2855 kuc: error management in KUC broadcast delivery

In KUC broadcast groups, the message delivery fails if
a userland process terminates without closing its kuc
channel properly.
This patch improves the behavior of KUC broadcast groups,
to make it more adapted for broadcast usage: the message
delivery is successful if at least 1 userland process is
present in this group and receives
the message successfully.

Signed-off-by: Thomas Leibovici <thomas.leibovici@cea.fr>
Signed-off-by: JC Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Change-Id: I2a121c7cfcaa6c2ee5ac48b721668bf2f254d848
Reviewed-on: http://review.whamcloud.com/5521
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2523 tests: disable racer setstripe until fixed

Disable the use of "lfs setstripe" in racer file_create.sh until
the LU-2523 and LU-2789 are fixed, so that racer can be added to
the "review" test workload instead of only in the "full" workload.

Test-Parameters: testlist=racer,racer,racer
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I8b0814ef7864efd817ff585f715183fbf73ebbe5
Reviewed-on: http://review.whamcloud.com/5424
Reviewed-by: John Hammond <johnlockwoodhammond@gmail.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2523 mdt: handle -ENOENT && !MDS_OPEN_CREAT in reint open

If mdt_open_by_fid_lock() returns -ENOENT and MDS_OPEN_CREAT is not
set in create flags then bail out and return -ENOENT from
mdt_reint_open().  In mdt_open_by_fid_lock() if -ENOENT is returned
then ensure that DISP_IT_EXECD is set in the reply disposition.  Add
sanity test_27B to call the LL_IOC_LOV_SETSTRIPE ioctl() on an open
unlinked file, a situation which triggers the first case.  In sanityn
test 30, remote the requirement that opening an unlinked file via
/proc/PID/exe return -ESTALE.  In racer's file_create.sh, reinstate
the call to 'lfs setstripe' before dd is started, effectively
reverting commit 0be1c87fe2d4ffddaca9e568cc137518b7368b2d.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I74e2297c4210ff18acbf60efdf51049d9a88cbea
Reviewed-on: http://review.whamcloud.com/5417
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Hudson
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2840 tests: Clean the environment for test.

Use unique directory for sanityn test_21. Avoid it
impacted by previous test.

Signed-off-by: yang sheng <yang.sheng@intel.com>
Change-Id: I36a06605732bb0048c25c622d4c8f1192d761a88
Reviewed-on: http://review.whamcloud.com/5556
Tested-by: Hudson
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2694 test: fix is_empty_fs in t-f

The original is_empty_fs is incorrect, which can cause unexpected
error when running lfsck.sh, because lfsck sets the max error
level based on whether filesystem is empty.

Test-Parameters: testlist=lfsck
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I2269cc41744e3c9fe228898323a1508a03616efe
Reviewed-on: http://review.whamcloud.com/5576
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Tested-by: Hudson
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2694 test: improve lfsck.sh

The test directory of lfsck.sh contains some files referencing
same object, which could cause error when removing the directory
on test cleanup.

Test-Parameters: testlist=lfsck
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I334ff2b7b5f77498eed940f009e4bc18728bb5da
Reviewed-on: http://review.whamcloud.com/5606
Tested-by: Hudson
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2852 script: sanity 27u should cleanup test dir

sanity 27u should cleanup test dir before test in case there
are remaining files from previous tests.

remove 27u from ALWAYS_EXCEPT since it can pass now.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: Id21589c1e1ecf0f4143fad3574cd779c73adb7aa
Reviewed-on: http://review.whamcloud.com/5670
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2523 racer: don't clobber DURATION

Don't clobber racer's DURATION if SLOW is not set to no.

Test-Parameters: envdefinitions=SLOW=yes \
		 testlist=racer,racer,racer,racer,racer,racer

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I86c7f54e3fb7a06388d7083001a80b91d09ac27d
Reviewed-on: http://review.whamcloud.com/5741
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
LU-2730 mdt: fix erroneous LASSERT in mdt_reint_opcode

Only set return code err_serious(-EFAULT) when neccessary and do not
run unknown opcode through err_serious.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Iea616d8afd676ee5ccda52cf09b398198f38f992
Reviewed-on: http://review.whamcloud.com/5416
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-2816 llite: Set RAS_INCREASE_STEP correctly

RAS_INCREASE_STEP is by pages instead of by bytes.

However, I found it caused performance loss by setting it to be 4MB so
I set it back to 1MB. After 4MB RPC is enabled by default, more work
should be done to pick up a right value of max_read_ahead_mb to
maximize performance gain of 4MB RPC.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ic000f266cfefc827e112e03f76cf467c73ba88ad
Reviewed-on: http://review.whamcloud.com/5691
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2735 test: disable LRU to avoid page deletion

In sanity.sh, subtest 151 checks whether the page written previously
is in cache or not, this patch disable the LRU of objects in OST to
avoid the object and its content(pages) to be dropped.

Change-Id: Ie481d13215dac599e0b7e122fcc7e9819a053af5
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: http://review.whamcloud.com/5475
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2910 clio: skip iov update when tot_nrsegs is zero

When tot_nrsegs is zero, we should skip the iov update too.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I52d4d52a65802f3967dd6a96ad46ec40fd4ef355
Reviewed-on: http://review.whamcloud.com/5747
Tested-by: Hudson
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2712 tests: enable sanity/sanityn SLOW tests

Enable a number of subtests in sanity.sh and sanityn.sh that are
currently skipped for normal "review" builds because of "SLOW=no".
Checking the maloo test history for these tests, they are verified
as always passing for "full" builds in the SLOW=yes case.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I340ccdf1323d14a34dc87a5217b3256c8f62cab0
Reviewed-on: http://review.whamcloud.com/5218
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2194 test: avoid wrong eviction in recovery_small

in subtest 19a and 19b of recovery_small, the locks from OST/MDT
should be cancelled before "drop_ldlm_cancel" to avoid wrong eviction
from OST/MDT for it will also drop its lock cancellation request.

Change-Id: I49d1d189f6791224a3f132dc8cf2dd8b3d51d43e
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Reviewed-on: http://review.whamcloud.com/5679
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Hudson
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2571 lfsck: run lfsck on server node

Usually we run e2fsck on OST and MDS nodes, and run lfsck on clients,
but it's not necessary to install e2fsprogs rpm on clients.
So, if lfsck is not found on client, we will try server node instead.

Test-Parameters: testlist=lfsck
Signed-off-by: Liu Ying <emoly.liu@intel.com>
Change-Id: I3979dc0236e81163f3283eac3148c36c8ddccf63
Reviewed-on: http://review.whamcloud.com/5139
Tested-by: Hudson
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-1897 test: replay-single test_70b dbench not found

dbench is taking too long to start. It was observed
that a 12 seconds sleep was not long enough to start
dbench on all clients.

I added a loop to rally all clients before starting
the testing and improved the error messages.

Signed-off-by: Keith Mannthey <keith.mannthey@intel.com>
Change-Id: I7b0fce4c39c003c4db23686b295fb5e45a6f8a92
Reviewed-on: http://review.whamcloud.com/5761
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-951 test: waiting import state in fail().

Anyway, There still has a rare chance that the request meet
a invalid import after fail() return. So we should waiting
import restore to a certain state and doing next operation.
Add wait_import_state_mount() to check import state while
client has a mount point.

Signed-off-by: yang sheng <yang.sheng@intel.com>
Change-Id: I55ccb0fd30d69eae651978804ef1e303d9939a71
Reviewed-on: http://review.whamcloud.com/5531
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
LU-3002 tests: Fix an OST index issue in replay-single

When run with 16 OSTs, replay-single 90 happened to pick ost14 as
ostfail and complained:

  lfs find does not report the affected lustre-OST000d_UUID for fd

The "d" in "fd" is a hexadecimal OST index, taken from
"lustre-OST000d_UUID".  Because the actual file names use decimal OST
indices, the test failed to match the file "f13" with "fd".  Moreover,
the test assumes the OST index is a single-digit number, which may not
be case with more than 15 OSTs.  This patch changes the test to simply
calculate the index from the facet name.

Test-Parameters: ostcount=32 testlist=replay-single envdefinitions=ONLY=90
Change-Id: I0c95d28f83c41cabf4ed8e5ed618897d6000da3b
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/5796
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2622 obdclass: Remove the global cl_env list

New cl_env structures are allocated using a SLAB cache specifically
created for allocating and freeing these structure. Without this patch,
when a thread is finished with a cl_env structure it places it on a
global list instead of freeing it back to the SLAB cache. With this
patch, this global list is completely removed, and cl_env structures are
released immediately back to the SLAB cache.

The motivation for this change essentially boils down to this secondary
global list cache being completely unnecessary, and only serving to
serialize any calls to cl_env_get and cl_env_put. This has proven to
cause a severe performance impact on large core count systems,
specifically during memory reclamation (i.e. ll_releasepage).

For example, on BG/Q Sequoia IO nodes, we've experienced nearly all
68 cores of a machine spinning on the lock protecting this global
list. Some example stack traces showcasing this problem were gathered
using sysrq-l, and are displayed below:

    CPU56:
    Call Trace:
    [c00000000fe3bb30] [c000000000008d1c] .show_stack+0x7c/0x184 (unreliable)
    [c00000000fe3bbe0] [c00000000027604c] .showacpu+0x64/0x94
    [c00000000fe3bc70] [c000000000068b30] .generic_smp_call_function_interrupt+0x10c/0x230
    [c00000000fe3bd40] [c00000000001d11c] .smp_message_recv+0x34/0x78
    [c00000000fe3bdc0] [c00000000002526c] .bgq_ipi_dispatch+0x118/0x18c
    [c00000000fe3be50] [c00000000007b20c] .handle_IRQ_event+0x88/0x18c
    [c00000000fe3bf00] [c00000000007dc90] .handle_percpu_irq+0x8c/0x100
    [c00000000fe3bf90] [c00000000001b808] .call_handle_irq+0x1c/0x2c
    [c0000003e1c4a4c0] [c0000000000059f0] .do_IRQ+0x154/0x1e0
    [c0000003e1c4a570] [c0000000000144dc] exc_external_input_book3e+0x110/0x114
    --- Exception: 501 at ._raw_spin_lock+0xd8/0x1a8
        LR = ._raw_spin_lock+0x104/0x1a8
    [c0000003e1c4a860] [8000000000b04f38] libcfs_nidstrings+0x2acc/0xfffffffffffe5824 [libcfs] (unreliable)
    [c0000003e1c4a910] [c00000000042d4cc] ._spin_lock+0x10/0x24
    [c0000003e1c4a980] [80000000024c2f4c] .cl_env_get+0xec/0x480 [obdclass]
    [c0000003e1c4aa60] [80000000024c336c] .cl_env_nested_get+0x8c/0xf0 [obdclass]
    [c0000003e1c4aaf0] [800000000692070c] .ll_releasepage+0xbc/0x200 [lustre]
    [c0000003e1c4aba0] [c000000000094110] .try_to_release_page+0x68/0x8c
    [c0000003e1c4ac10] [c0000000000a4190] .shrink_page_list.clone.0+0x3d8/0x63c
    [c0000003e1c4adc0] [c0000000000a47d8] .shrink_inactive_list+0x3e4/0x690
    [c0000003e1c4af90] [c0000000000a4f54] .shrink_zone+0x4d0/0x4d4
    [c0000003e1c4b0c0] [c0000000000a5a68] .try_to_free_pages+0x204/0x3d0
    [c0000003e1c4b220] [c00000000009d044] .__alloc_pages_nodemask+0x460/0x738
    [c0000003e1c4b3a0] [c000000000095af4] .grab_cache_page_write_begin+0x7c/0xec
    [c0000003e1c4b450] [8000000006920964] .ll_write_begin+0x94/0x270 [lustre]
    [c0000003e1c4b520] [c0000000000968c8] .generic_file_buffered_write+0x148/0x374
    [c0000003e1c4b660] [c000000000097050] .__generic_file_aio_write+0x374/0x3d8
    [c0000003e1c4b760] [c00000000009712c] .generic_file_aio_write+0x78/0xe8
    [c0000003e1c4b810] [800000000693ed4c] .vvp_io_write_start+0xfc/0x3e0 [lustre]
    [c0000003e1c4b8e0] [80000000024d9c6c] .cl_io_start+0xcc/0x220 [obdclass]
    [c0000003e1c4b980] [80000000024e1a84] .cl_io_loop+0x194/0x2c0 [obdclass]
    [c0000003e1c4ba30] [80000000068ba1d8] .ll_file_io_generic+0x498/0x670 [lustre]
    [c0000003e1c4bb30] [80000000068ba834] .ll_file_aio_write+0x1d4/0x3a0 [lustre]
    [c0000003e1c4bc00] [80000000068bab50] .ll_file_write+0x150/0x320 [lustre]
    [c0000003e1c4bce0] [c0000000000d1ba8] .vfs_write+0xd0/0x1c4
    [c0000003e1c4bd80] [c0000000000d1d98] .SyS_write+0x54/0x98
    [c0000003e1c4be30] [c000000000000580] syscall_exit+0x0/0x2c

    CPU63:
    Call Trace:
    [c00000000fe03b30] [c000000000008d1c] .show_stack+0x7c/0x184 (unreliable)
    [c00000000fe03be0] [c00000000027604c] .showacpu+0x64/0x94
    [c00000000fe03c70] [c000000000068b30] .generic_smp_call_function_interrupt+0x10c/0x230
    [c00000000fe03d40] [c00000000001d11c] .smp_message_recv+0x34/0x78
    [c00000000fe03dc0] [c00000000002526c] .bgq_ipi_dispatch+0x118/0x18c
    [c00000000fe03e50] [c00000000007b20c] .handle_IRQ_event+0x88/0x18c
    [c00000000fe03f00] [c00000000007dc90] .handle_percpu_irq+0x8c/0x100
    [c00000000fe03f90] [c00000000001b808] .call_handle_irq+0x1c/0x2c
    [c0000003c4f0a510] [c0000000000059f0] .do_IRQ+0x154/0x1e0
    [c0000003c4f0a5c0] [c0000000000144dc] exc_external_input_book3e+0x110/0x114
    --- Exception: 501 at ._raw_spin_lock+0xdc/0x1a8
        LR = ._raw_spin_lock+0x104/0x1a8
    [c0000003c4f0a8b0] [800000000697a578] msgdata.87439+0x20/0xfffffffffffccf88 [lustre] (unreliable)
    [c0000003c4f0a960] [c00000000042d4cc] ._spin_lock+0x10/0x24
    [c0000003c4f0a9d0] [80000000024c17e8] .cl_env_put+0x178/0x420 [obdclass]
    [c0000003c4f0aa70] [80000000024c1ab0] .cl_env_nested_put+0x20/0x40 [obdclass]
    [c0000003c4f0aaf0] [8000000006920794] .ll_releasepage+0x144/0x200 [lustre]
    [c0000003c4f0aba0] [c000000000094110] .try_to_release_page+0x68/0x8c
    [c0000003c4f0ac10] [c0000000000a4190] .shrink_page_list.clone.0+0x3d8/0x63c
    [c0000003c4f0adc0] [c0000000000a47d8] .shrink_inactive_list+0x3e4/0x690
    [c0000003c4f0af90] [c0000000000a4f54] .shrink_zone+0x4d0/0x4d4
    [c0000003c4f0b0c0] [c0000000000a5a68] .try_to_free_pages+0x204/0x3d0
    [c0000003c4f0b220] [c00000000009d044] .__alloc_pages_nodemask+0x460/0x738
    [c0000003c4f0b3a0] [c000000000095af4] .grab_cache_page_write_begin+0x7c/0xec
    [c0000003c4f0b450] [8000000006920964] .ll_write_begin+0x94/0x270 [lustre]
    [c0000003c4f0b520] [c0000000000968c8] .generic_file_buffered_write+0x148/0x374
    [c0000003c4f0b660] [c000000000097050] .__generic_file_aio_write+0x374/0x3d8
    [c0000003c4f0b760] [c00000000009712c] .generic_file_aio_write+0x78/0xe8
    [c0000003c4f0b810] [800000000693ed4c] .vvp_io_write_start+0xfc/0x3e0 [lustre]
    [c0000003c4f0b8e0] [80000000024d9c6c] .cl_io_start+0xcc/0x220 [obdclass]
    [c0000003c4f0b980] [80000000024e1a84] .cl_io_loop+0x194/0x2c0 [obdclass]
    [c0000003c4f0ba30] [80000000068ba1d8] .ll_file_io_generic+0x498/0x670 [lustre]
    [c0000003c4f0bb30] [80000000068ba834] .ll_file_aio_write+0x1d4/0x3a0 [lustre]
    [c0000003c4f0bc00] [80000000068bab50] .ll_file_write+0x150/0x320 [lustre]
    [c0000003c4f0bce0] [c0000000000d1ba8] .vfs_write+0xd0/0x1c4
    [c0000003c4f0bd80] [c0000000000d1d98] .SyS_write+0x54/0x98
    [c0000003c4f0be30] [c000000000000580] syscall_exit+0x0/0x2c

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: Ief4b524784e07d7677ecb8a9ce97a7b54ccc6f75
Reviewed-on: http://review.whamcloud.com/5204
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>