LU-4624 statahead: Improve statahead debug messages

The statahead debug messages include the pid of the current
process in their body.  This is both redudant (because all
lustre log messages contain the pid), and sometimes downright
misleading.  For instance the messages would say something like
"stopping statahead thread 3446".  One would probably think
that 3446 is the pid of the process that is being stopped,
but in fact it was the pid of the caller issuing the stop signal.

We remove all superfluous pids from the messages.

Next we have the ll_statahead_thread() and the ll_agl_thread() record
their respective pids in their respective ptlrpc_thread structures.
This allows to print the pid of the thread that we are trying to
stop (which is actually useful info) from other threads, such as those
calling ll_stop_statahead().

Change-Id: Id4def179e02e9097f3f22048192d45bdf15b0aa8
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Reviewed-on: http://review.whamcloud.com/9360
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4611 osd: improve credits calculation

- llog catalog do not declare records twice (for old and new objects)
  this might be an issue on ZFS with full debug enabled..
- llog to specify append by pos=-1, so OSD can take this into account
- object create/destroy should not include OI, this is calculated yet
- EA declaration to improve few specific cases
- osd_declare_write() to recognize overwrite optimistically,
  using inode size and i_blocks
- osd_declare_write() to optimize very specific cases, like legacy
  blockmaps with small offsets and allocated indirects
- index delete modify just a single block

preliminary testing on a local setup with 7 OSTs:
1360 credits before and 436 credits after.

llog declarations and index inserts (part of llog object creation)
still consume about 70% (7 OSTs):
   create: 7/28, destroy: 1/4
   attr_set: 2/2, xattr_set: 8/21
   write: 37/179, punch: 14/56, quota 2/2
   insert: 8/135, delete: 2/5
   ref_add: 1/1, ref_del: 3/3

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: Icb5c79df1f8ba248509b9d2561ac8843bb01f6af
Reviewed-on: http://review.whamcloud.com/9258
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4775 lfsck: dump all visibile lfsck parameter correctly

We missed to handle "LPF_BROADCAST" and "LPF_ORPHAN" when dump
the lfsck parameters and caused the output via /proc interface
incomplete and bad formatted.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I462c79e80be5b384a291b30980a95698a8dd9d19
Reviewed-on: http://review.whamcloud.com/9690
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
LU-4777 lfsck: redefine LFSCK_RBTREE_BITMAP_MASK

It should be the rbtree bitmap bits size, not the bytes size.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I0c090702f3f2f5037926d6384b605751a0a9c948
Reviewed-on: http://review.whamcloud.com/9703
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
LU-4108 tests: decrease load for zfs in performance-sanity/4

Given the poor performance of metadata on ZFS (cf LU-2600), this patch
decreases the number of files created (similar to what happens in
perforamnce-scale.sh).

Test-Parameters: mdtfilesystemtype=zfs ostfilesystemtype=zfs  mdsfilesystemtype=zfs testlist=performance-sanity
Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I998c8525bf218dacad07d98ac842f4c12102cd9a
Reviewed-on: http://review.whamcloud.com/9725
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
LU-4585 out: remove OBJECT_UPDATE_PER_RPC_MAX limit.

Extend tx_args if the update inside 1 RPC is more than
the count of tx_args count in tgt_thread_info.

And also for create remote object, the remote RPC might
happen in trans stop, so we need to add return value
for mdd_trans_stop to track the error.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Ibb87181d87dfb6c3ed7e6757592bccd4a836a98a
Reviewed-on: http://review.whamcloud.com/9590
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
LU-4167 tests: update version check to conf-sanity test_32

This patch updates version check codes to skip running LFSCK on an
OST for versions >= 2.5.0 in conf-sanity.sh test_32.

Test-Parameters: testlist=conf-sanity envdefinitions=SLOW=yes, ONLY=32 ossjob=lustre-b2_4 mdsjob=lustre-b2_4 ossbuildno=71 mdsbuildno=71 clientdistro=el6 ossdistro=el6 mdsdistro=el6 clientarch=x86_64 ossarch=x86_64 mdsarch=x86_64
Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: Ibc0c34a0c15f4882a0dd06dae75d1f1fe10b6f16
Reviewed-on: http://review.whamcloud.com/8132
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-3373 osd-ldiskfs: readdir replace by iterate

Use iterate instead of readdir callback in iop.

Signed-off-by: yang sheng <yang.sheng@intel.com>
Change-Id: Icb08292009c965ca693814e854ae2e77b3e7a4f0
Reviewed-on: http://review.whamcloud.com/8231
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Peng Tao <bergwolf@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4446 tests: block on layout lock fix up on sanityn:51c

Fixed two problems: set fail_loc on MDS and block at layout lock
by writing the test file.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I8028e855aad5882981a313fb28ec5b87f593832b
Reviewed-on: http://review.whamcloud.com/8789
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-1538 tests: unmount /sbin/mount.lustre bind mount

If the lustre/utils/mount.lustre file has changed since it was bind
mounted on /sbin/mount.lustre, unmount it and mount the new one
there, to avoid strange problems when testing from a local tree

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Ie5360dae9f33553acb914c8ce549e848c1500c1e
Reviewed-on: http://review.whamcloud.com/8912
Tested-by: Jenkins
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3254 llite: access layout version under a lock

We used to access layout version under the protection of ldlm
lock, this introduces extra overhead for dlm lock matching.

In this patch, lli_layout_lock is introduced to access the layout
version. Also, when a layout lock is losing, we should tear down
mmap of the correspoding inode to avoid stale data accessing in the
future.

This is part of technical verification of replication.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I657c6a8b27f2cdaf468dc208c43c1d92f3843705
Reviewed-on: http://review.whamcloud.com/8689
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4596 lprocfs: mdt/*/exports/*/uuid is empty after remount

need to assign exp_nid_stats even if old stats are present

Change-Id: I9b0ac4c7ccee6cbd457827b20123bf40d91aaf33
Xyratex-bug-id: MRP-1641
Signed-off-by: Andriy Skulysh <Andriy_Skulysh@xyratex.com>
Reviewed-on: http://review.whamcloud.com/9171
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3059 lu: shrink lu_object_header by 8 bytes on x86_64

Locate the loh_flags and loh_ref fields together in lu_object_header
to avoid holes and shrink the structure by 8 bytes.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I36b6132a819b6557d6d1688adb3e1f7d282d22b5
Reviewed-on: http://review.whamcloud.com/9185
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4614 llite: remove lustre_generic_file_{read,write}

It looks like lustre_generic_file_{read,write} are a holdover from
2.6.19 where generic_file_aio_read() replaced generic_file_readv()
and cross-kernel interoperability was required for some period of
time. Lustre has since removed support for those older kernels, but
it looks like the wrappers were not deleted at that time. This patch
will delete them.

Pass &iocb->ki_pos as the last argument for these functions instead
of crw_pos, since this is the convention for other callers.  Verify
that this is the same as the current crw_pos argument.  This code can
likely be cleaned up further in a later patch.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Id19edacf93ee2538634feb95580ab885f6069ee7
Reviewed-on: http://review.whamcloud.com/9230
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
LU-4496 build: enable RPMs build with --disable-modules

Enable the following RPMs build:
  ./configure --disable-modules
  make rpms

Addition to commit c92afbcd4a8234994affa444d59645b80fa19ec5
(Reviewed-on: http://review.whamcloud.com/6020)

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I7b26fac928404a75e974b414085bf1a584ad8bb9
Reviewed-on: http://review.whamcloud.com/9306
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Christopher J. Morrone <chris.morrone.llnl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4629 ldlm: fix NULL pointer dereference

Pointer '*exp' returned from call to function 'class_conn2export'
at line 523 may be NULL and may be dereferenced at line 543.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I084ae2ae76861aef4fd93477551bc9d0dd5f4478
Reviewed-on: http://review.whamcloud.com/9323
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4629 lnet: fix issues found by Klocwork Insight tool

Null pointer 'cp' that comes from line 2544 may be dereferenced
at line 2618.

Pointer 'ni' checked for NULL at line 1569 may be passed to
function and may be dereferenced there by passing argument 1 to
function 'lnet_ni_notify_locked' at line 1621.

Null pointer 'best_iface' that comes from line 802 may be
dereferenced at line 832.

Buffer overflow of string buffer due to non null terminated string.

sscanf format specification '%lx' expects type 'unsigned long int *'
for 'x', but parameter 4 has a different type 'long int*'.

Pointer 'tsc' returned from call to function 'sfw_find_test_case'
at line 571 may be NULL and will be dereferenced at line 572.

sscanf format specification '%i' expects type 'int *' for 'i',
but parameter 3 has a different type 'unsigned int*'.

Local variable 'hash' is never used.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I2a942735cd93906453b92dcc711a956b7adf5615
Reviewed-on: http://review.whamcloud.com/9386
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Isaac Huang <he.huang@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4629 mdc: fix issue found by Klocwork Insight tool

Pointer 'mod' checked for NULL at line 160 may be dereferenced at line 208.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I130de2f5b2066b8797e2454d4a603f53b3ffcbef
Reviewed-on: http://review.whamcloud.com/9387
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4629 obdclass: fix use after release

Pointer 'type->typ_name' checked for NULL at line 195 will be passed
to function and may be dereferenced there by passing argument 1 to
function 'lprocfs_try_remove_proc_entry' at line 246.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I697ffab97a5bd1ab55339722d87a3dfca5c1e173
Reviewed-on: http://review.whamcloud.com/9388
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4629 libcfs: fix issues found by Klocwork Insight tool

sscanf format specification '%u' expects type 'unsigned int *'
for 'u', but parameter 3 has a different type 'int*'.

Pointer 'cfs_crypto_hash_type(d->hd_hash->ha_id)' returned from
call to function 'cfs_crypto_hash_type' at line 261 may be NULL
and will be dereferenced at line 261.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I9945a7e5d44329aa6417946106212cb269395290
Reviewed-on: http://review.whamcloud.com/9400
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>