LU-4345 osp: store valid bits in setattr record

Backport a series of 4 patches to resolve the random ID on OST
objects problem.

1: store valid bits in setattr record (LU-4345)

We'd store LA_UID/LA_GID bit along with the UID/GID in the setattr
llog record, otherwise, osp could set a random uid/gid to the OST
object.

Lustre-commit: 80f90fcde73e2faff8b7b0ffc7c19bc52982e027
Lustre-change: http://review.whamcloud.com/10223

2: Correctly check for invalid setattr record (LU-5188)

Patch for LU-4345 (commit 80f90fcde73e2faff8b7b0ffc7c19bc52982e027 )
has a correct comment about lsr_valid member being either 0 or
having UID and GID fields set, but the check has a typo causing
it to check for lsr_valid to be both 0 and have the bits set which
is impossible.

The osp_sync_new_setattr_job() should return 0 for invalid record,
so that sync thread can continue processing on other records.

Lustre-commit: 79dd530f1352e6b57fcb870a1e0f2c2a05a0648d
Lustre-change: http://review.whamcloud.com/10706

3: don't skip attr_set for osp objects (LU-5296)

The lsr_valid handling in osp_sync_add_rec() was got a problem in
commit 80f90fcde73e because it stored all of the passed attr flags
in struct llog_setattr64_rec, even though there are only fields for
storing the UID and GID.  Since the time stamps do not need to be
propagated to the OSTs (the MDT values are good enough), this is
not a problem.  Also, osp_sync_add_rec stored LA_UID/LA_GID flags
on disk, but they are not part of lustre_idl.h or lustre_disk.h and
are not guaranteed to be constant over time.  Instead, use the flags
OBD_MD_FLUID/OBD_MD_FLGID that are in lustre_idl.h.

Lustre-commit: 2d5a5e81660adbcd05f14a10bc6c246d6f108d10
Lustre-change: http://review.whamcloud.com/10989

4: return 1 if osp_sync_xxx_job issue RPC (LU-5188)

Return 1 if osp_sync_new_xxx_job() issue RPC, so
sp_sync_process_record() can decrease opd_syn_rpc_in_flight
and opd_syn_rpc_in_progress correctly if RPC is not
being sent, otherwise the opd_sync_thread will not be
stopped, caused LBUG (see LU-5244)

Lustre-commit: 73f47cdda42305df0be57e0ea6252eb6fd36db55
Lustre-change: http://review.whamcloud.com/10828

Test-Parameters: alwaysuploadlogs \
envdefinitions=SLOW=yes,ENABLE_QUOTA=yes,ONLY=34 \
ossjob=lustre-b2_4 mdsjob=lustre-b2_4 ossbuildno=73 mdsbuildno=73 \
testlist=sanity-quota

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Ie231465b274627561e7eb58f30b868472751bd71
Signed-off-by: Jian Yu <jian.yu@intel.com>
Reviewed-on: http://review.whamcloud.com/11435
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4579 ldlm: Properly display time on lock timeout

When lock timeout happens, need to printtime since last lock
activity, not since when the lock was granted.
As such set l_last_activity to current time when sending callbacks too.

Lustre-change: http://review.whamcloud.com/10601
Lustre-commit: f3864905b26a90ed3a9bb6f99357264fa40015c4

Change-Id: I31d0de6d9add67865cfcb16af903647f7cb932a0
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/10601
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-on: http://review.whamcloud.com/10997
LU-4090 kernel: a kernel patch for jbd2 hung

patch get from linux vanilla kernel commit
0ef54180e0187117062939202b96faf04c8673bc (v3.10-rc2)

jbd2:drop checkpoint mutex when waiting in __jbd2_log_wait_for_space()

While trying to debug an an issue under extreme I/O loading
on preempt-rt kernels, the following backtrace was observed
via SysRQ output:

rm              D ffff8802203afbc0  4600  4878   4748 0x00000000
ffff8802217bfb78 0000000000000082 ffff88021fc2bb80 ffff88021fc2bb80
ffff88021fc2bb80 ffff8802217bffd8 ffff8802217bffd8 ffff8802217bffd8
ffff88021f1d4c80 ffff88021fc2bb80 ffff8802217bfb88 ffff88022437b000
Call Trace:
[<ffffffff8172dc34>] schedule+0x24/0x70
[<ffffffff81225b5d>] jbd2_log_wait_commit+0xbd/0x140
[<ffffffff81060390>] ? __init_waitqueue_head+0x50/0x50
[<ffffffff81223635>] jbd2_log_do_checkpoint+0xf5/0x520
[<ffffffff81223b09>] __jbd2_log_wait_for_space+0xa9/0x1f0
[<ffffffff8121dc40>] start_this_handle.isra.10+0x2e0/0x530
[<ffffffff81060390>] ? __init_waitqueue_head+0x50/0x50
[<ffffffff8121e0a3>] jbd2__journal_start+0xc3/0x110
[<ffffffff811de7ce>] ? ext4_rmdir+0x6e/0x230
[<ffffffff8121e0fe>] jbd2_journal_start+0xe/0x10
[<ffffffff811f308b>] ext4_journal_start_sb+0x5b/0x160
[<ffffffff811de7ce>] ext4_rmdir+0x6e/0x230
[<ffffffff811435c5>] vfs_rmdir+0xd5/0x140
[<ffffffff8114370f>] do_rmdir+0xdf/0x120
[<ffffffff8105c6b4>] ? task_work_run+0x44/0x80
[<ffffffff81002889>] ? do_notify_resume+0x89/0x100
[<ffffffff817361ae>] ? int_signal+0x12/0x17
[<ffffffff81145d85>] sys_unlinkat+0x25/0x40
[<ffffffff81735f22>] system_call_fastpath+0x16/0x1b

What is interesting here, is that we call log_wait_commit, from
within wait_for_space, but we are still holding the checkpoint_mutex
as it surrounds mostly the whole of wait_for_space.  And then, as we
are waiting, journal_commit_transaction can run, and if the
JBD2_FLUSHED bit is set, then we will also try to take the same
checkpoint_mutex.

It seems that we need to drop the checkpoint_mutex while sitting in
jbd2_log_wait_commit, if we want to guarantee that progress can be
made by jbd2_journal_commit_transaction().  There does not seem to be
anything preempt-rt specific about this, other then perhaps increasing
the odds of it happening.

This patch is back-ported from the following one:
Lustre-commit: f8e80f01d6883c3e55841528ac16811f6588e260
Lustre-change: http://review.whamcloud.com/11041

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I9adc35d316d7ea0379d0df2d38276297f682f8f7
Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Jian Yu <jian.yu@intel.com>
Reviewed-on: http://review.whamcloud.com/11492
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5252 hsm: llapi_hsm_event_fd is local, so make it static.

This is a backport to b2_5.

Lustre-change: http://review.whamcloud.com/10808
Lustre-commit: 9feb80c65b077204f7091e90f314098266fb7e82

Change-Id: I9905d0251702a11a9c2546ba4bd78fd00658d28a
Signed-off-by: Frank Zago <fzago@cray.com>
Signed-off-by: James Nunez <james.a.nunez@intel.com>
Signed-off-by: Jian Yu <jian.yu@intel.com>
Reviewed-on: http://review.whamcloud.com/11362
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>