LU-2640 osd: enhance transaction debug to trace rollback

Some modifications may need to rollback because of failures during
the transactions. The original transaction trace mechanism did not
consider rollback cases, so cannot correctly record sub-operations
and the rollback cases.

In this patch, not only the specified sub-operation will be traced,
but also its rollback operation (if may rollback) will be recorded
also. For example:

When declare for ref_add, it will record the ref_add sub-operation
by "osd_thandle::ot_declare_ops[OSD_OT_REF_ADD]"; when consume the
ref_add credit, the possible rollback will be recorded through the
"osd_thandle::ot_declare_ops_rb[OSD_OT_REF_ADD]"; and when need to
rollback the ref_add (by ref_del) in subsequent processing, it can
be traced by the "osd_thandle::ot_declare_ops_rb[OSD_OT_REF_ADD]";
and once rollback starts, only "osd_thandle::ot_declare_ops_rb[X]"
can be consumed.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Iade214eff468c2298f84920b997040e06f1a204f
Reviewed-on: http://review.whamcloud.com/5138
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-2663 tests: don't remove objects by debugfs

In lfsck.sh, we shouldn't use debugfs to remove objects, that'll
cause quota usage inconsistence warning in e2fsck.

Fix improper LBUG() in mdd_create_data(), MDS_OPEN_HAS_OBJS is
still used by lfsck.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I93c6f5c40e359d0d8402868be548088d6e2ac5da
Reviewed-on: http://review.whamcloud.com/5186
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>
LU-2576 osc: various fix in osc_enter_cache()

- It should not go to sleep if there isn't any inflight write
  RPC, otherwise, it could not be waked up for a long time,
  until dirty flush triggered on other OSCs or other objects.

- If the page can't be granted due to too many dirty pages
  (> obd_max_dirty_pages), osc_enter_cache() should return
  -EDQUOT.

- It should pass NULL osc object to  osc_io_unplug(), otherwise
  dirty flush can't be triggered if the passed object is
  clean.

- It should not wait in a loop of "while (cli->cl_dirty > 0)",
  trigger the dirty flush once is enough. If there are too
  many threads that consumed grants immediately, then we
  should turn to sync write, but not trigger flush and wait
  for grant again (that'll block io process on grant).

- Refuse setting max_dirty_mb as 0 via proc file.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I449cba07bd427749ab023c249d9e200aba1b406a
Reviewed-on: http://review.whamcloud.com/4963
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Prakash Surya <surya1@llnl.gov>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-2010 tests: replay-single/44a,b,45 find only tested mdc

If multiple filesystems are mounted, then test_45 would have found
mdcdevs for each filesystem.  This corrects that to only look for the
filesystem that is being tested against.

Also correctly delete old test files.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I0297cf9b21e9c97ee468cd81d5d4ac409c59d1e0
Reviewed-on: http://review.whamcloud.com/4724
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-2101 clio: cl_sync_io_wait() need a barrier

cl_sync_io_note() has a small window between cfs_atomic_dec_and_test
and cfs_waitq_broadcast, another thread might have destroyed anchor
between this window, then cl_sync_io_note() is calling wakeup on a
poisoned waitq and crash.

Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Change-Id: I04bfc738c8c5bec3a1e8b51131fae4199ef5ec9a
Reviewed-on: http://review.whamcloud.com/5199
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Tested-by: Hudson
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-1897 test: replay-single test_70b dbench not found

dbench is exiting before test_70b completes is work.

dbench is being used as a background load in this test.
In one observed case test was taking a little under 90s but
dbench was exiting at 62s. Duration was set at 90.

Presently dbench starts for 60s then the system sleeps
for 12s, then we start a new 60s timer in which we expect
dbench to be running.  There is a large window in which
this test will fail.

The main issue is the use of Duration for use as both the
timer of the real test and the timer of the background load.

I raised the real Duration and started the timer for the real
test before starting dbench such that the real test should
finish before dbench does or about the same time.

Signed-off-by: Keith Mannthey <keith.mannthey@intel.com>
Change-Id: Iba243fe3132a5fa677e5a7bcd09e491727ba092a
Reviewed-on: http://review.whamcloud.com/4973
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Hudson
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-2709 build: fix 'memory corruption' errors

Fix 'memory corruption' defects found by Coverity version 6.5.0:
Out-of-bounds access (OVERRUN_STATIC)
Overrunning static array by passing it to a
function which indexes it at a higher byte position.
Unbounded source buffer (STRING_SIZE)
Passing string of unknown size to a function that expects
a string of a particular size.

Signed-off-by: Sebastien Buisson <sebastien.buisson@bull.net>
Change-Id: Iabc9bd7609d285a3709737748f2100b90769d795
Reviewed-on: http://review.whamcloud.com/5210
Tested-by: Hudson
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-2697 osp: remove trailing newline on thread names

The trailing newline on the thread name causes confusion with some
utilities that examine this string.

Signed-off-by: chas williams - CONTRACTOR <chas@cmf.nrl.navy.mil>
Change-Id: I031aafd2541b1829bc721ebcb9f52cb5e73101b3
Reviewed-on: http://review.whamcloud.com/5189
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-2693 tests: check Lustre version in conf-sanity test 66

Lustre starts supporting the "replace nids" feature
(commit 6aa30ff) since version 2.3.59. We need add
a Lustre version check into conf-sanity test 66 to
make sure the test does not run on unsupported
Lustre version.

Test-Parameters: envdefinitions=SLOW=yes \
serverjob=lustre-b2_1 serverbuildno=164 \
testlist=conf-sanity

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: I7eee38664e66059bbc028c3eb7d9741b1351b883
Reviewed-on: http://review.whamcloud.com/5198
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-2592 tests: do not remove $TMP/*active in rpc.sh

The $TMP/*active files record the current active facets info
under failover test configuration. They are removed from
init_test_env() initially. If failover tests use do_rpc_nodes()
which performs rpc.sh and in which init_test_env() is used,
then the active facets info will be lost during the testing.

This patch fixes the above issue by introducing an RPC_MODE
variable which controls that the $TMP/*active files will not
be removed from init_test_env() in rpc.sh.

Test-Parameters: envdefinitions=SLOW=yes,ENABLE_QUOTA=yes clientcount=4 osscount=2 mdscount=2 austeroptions=-R failover=true useiscsi=true testlist=replay-vbr
Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: Iab464449e7c41ffd76256e9ec60b48b326f3ea9e
Reviewed-on: http://review.whamcloud.com/5127
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Wei Liu <wei3.liu@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-2221 ptlrpc: kerberos support for kernel>=2.6.24

Since kernel 2.6.24 the scatterlist struct has no field "page".
Then related Lustre kerberos code cannot work anymore.

So do not access scatterlist::page directly, instead, use the
scatterlist functions sg_set_page and sg_assign_page.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Signed-off-by: Thomas Stibor <thomas@stibor.net>
Change-Id: I446925bb42c1e018a55a69948383c8f71976f1fa
Reviewed-on: http://review.whamcloud.com/4394
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
LU-1282 lprocfs: reduce lprocfs stats memory use

* Move percpu counter common data out, do not need to store them
  redundantly in percpu counter.
* LPROCFS_STATS_FLAG_IRQ_SAFE flag implies three things:
  1. lprocfs_counter needs lc_sum_irq.
  2. when a stats is a percpu counter stat, all its percpu counter
     gets allocated along with the stats itself.
  3. when a stats is a non-percpu stat, lprocfs_stats_lock() needs
     disable irq.
* change lprocfs_counter to make non-irq-safe stats counter do not
  use lc_sum_irq counter, which can save memory.
* Right now, only obd_memory stats use LPROCFS_STATS_FLAG_IRQ_SAFE
  flag.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I2eb037f9dcda983844857fc068c428c4fa387e7a
Reviewed-on: http://review.whamcloud.com/3246
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-1876 hsm: bugfix for deadlock in glimpse and layout change

This dead lock is as follows: glimpse calls ccc_inode_lsm_get()
inside glimpse IO, it will be blocked at grabbing lo_type_guard
sem, because another thread is changing layout which holds
lo_type_guard but waiting for all active IO to complete.

The basic rule: any attempt to grab lo_type_guard inside an IO is
generically risky.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ib41c03f0a73e2ef7cd37e7689c2d55f7fd0484bf
Reviewed-on: http://review.whamcloud.com/5158
Reviewed-by: jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
LU-1876 hsm: Avoid deadlock on i_mutex and layout change

The deadlock is as follows:
Proc 1 (write): start an IO -> try to acquire i_mutex in
            generic_file_aio_write()
Proc 2 (setattr): Acquire i_mutex in ll_setattr_raw -> initialise IO
            -> ll_layout_refresh() -> wait for IO above.

It's actually not necessary to acquire i_mutex in ll_setattr_raw()
until setattr really starts.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I1988d67eab8bcac2ee63b9ec0ecab06d0b7cfc66
Reviewed-on: http://review.whamcloud.com/5159
Tested-by: Hudson
Reviewed-by: jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
LU-2700 mdt: serializing the rename

BFL is needed in the rename process even though the source and
target are under the same directory, because we still need
serialize the locking of these two objects.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I61593e3902f1f81ac282c020d912fdf28ce2cc2c
Reviewed-on: http://review.whamcloud.com/5203
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Tested-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
LU-2466 lnet: Use Hash Table for Remote Route List

This change updates the remote route list (the_lnet.ln_remote_nets)
to be a hash table (the_lnet.ln_remote_nets_hash) to speed up
access when dealing with a large number of routes (over 1000).

Signed-off-by: Doug Oucharek <doug.s.oucharek@intel.com>
Change-Id: I2dac8075ff38d29bd38905b5b1b002603c5a56b6
Reviewed-on: http://review.whamcloud.com/5023
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Tested-by: Hudson
Reviewed-by: John Hammond <john.hammond@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1187 osp: add osp_md_object for remote directory.

In declare phase, the update for remote object will be added
in the remote list of the transaction, which will then being
sent to the remote MDT during trans_start.

Add osp_md_object for remote directory, any updates for
remote operation will be packed into RPC by api in
the osp_md_object

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I5fac24e21e6663dca0e2d2fa5d90d4a07a002136
Reviewed-on: http://review.whamcloud.com/4927
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1187 mdt: add out handler for object update

Add object update handler on remote MDT. Because the RPC
only includes object updates, (no metadata operation), the
out_handler will skip MDD layer and call osd layer directly.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I9d553145cedfffcbb26d839ed8c7d9cc0887ed86
Reviewed-on: http://review.whamcloud.com/4928
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1187 mdt: Add pre_cleanup phase in MDT stack cleanup

Add pre_cleanup phase in MDT stack cleanup, so MD OSP can be
disconnected before cleanup MDT stack.

Signed-off-by: wang di <di.wang@whamcloud.com>
Change-Id: Id1a56c393c22b4fc526c81c05b11bfb656a80559
Reviewed-on: http://review.whamcloud.com/4929
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1187 lod: prepare default stripe EA for remote directory

During remote directory create, it should prepare the default
stripe EA in declare, so the default EA can be set correctly
on the remote directory.

Change-Id: Ief93e72cc033f3553865f2abba53e81df0f3fa89
Signed-off-by: Wang Di <di.wang@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/4335
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>