LU-572 ptlrpc: make ptlrpc_recover_import() async

Add an async argument for ptlrpc_recover_import() and with async nonzero,
ptlrpc_recover_import() will return immediately after sending connecting
RPC.

This is needed for imperative recovery since it may recover multiple
targets meanwhile.

Change-Id: Iccc40fca4d3d5d3fb0b2965e62beda23e2fb3b4c
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1191
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
ORNL-10: Basic IR implementation

To support imperative recovery, there is a target status table defined for
each file system defined on the MGS. When a target registers itself to the
MGS, the MGS will change this table correspondingly.

In the status table, one important field is target NID. This NID information
is used by clients locating server where target lives. By transferring this NID
to clients, clients can know the restarting of targets earlier. This is
so-called imperative recovery - the MGS notifies clients to do recovery
imperatively instead of timeout based standard recovery.

To implement imperative recovery, clients are asked to cache a NID table, which
contains the location information of all servers. Clients need to hold a read
mode ldlm plain lock - recover lock - to cache this table. Whenever the MGS
wants to change this table, it will enqueue an EXCL recover lock so that all
clients will be notified for this change. Clients will request for a new read
recover lock and then query for the MGS for NID table updates.

Change-Id: I3b38ba142b810df507805b71972feeb1bade1ac2
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1217
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
ORNL-13: NonIR client support

In this task, NonIR support is added. NONIR clients means those clients who
doesn't know the protocol of imperative recovery. This means they won't be
notified for the restarting of target.

To support NonIR clients, the MGS has to record how many NonIR clients per
file system, and track those clients. In this way, if there are NonIR clients
for a specific file system, the MGS should tell the restarting target to disable
imperative recovery; otherwise, these `old' clients are to be evicted easily.

Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Change-Id: I3725c66b74d702aa213644ee9a6f89d59b8a8083
Reviewed-on: http://review.whamcloud.com/1218
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
ORNL-14: Configuration and Unit test cases

In this patch, some procfs and test cases are added for imperative recovery.

Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Change-Id: Ied49b622208157a74e8547e4609d61ee5041f624
Reviewed-on: http://review.whamcloud.com/1219
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Yu Jian <yujian@whamcloud.com>
ORNL-28: Solve reconnecting race between IR and SR

if there is a connecting request on the fly when client import is
notified by IR, it will set the corresponding conn uuid to a higher
prio and set imp_force_verify so that it will do reconnection
immediately in case RPC timeout happens.

Change-Id: I77e799e1f12b49f3c0271585c1ce812d16dc1ef6
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1291
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
ORNL-28: Set recovery timeout correctly

make sure recovery window uses timeout value from lustre config;
in current implementation this piece of code is totally wrong since
it just disregards timeout configuration.

Conflicts: with MRP-451, adjusted to the MRP-451 version landed
to mainstream.

Change-Id: I0cb0d777569cccd96f30da11834c6e333a673816
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1292
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
ORNL-28: remove lock contention in reconnecting

target_handle_connect() used to grab obd::obd_recovery_task_lock to update
obd_next_recovery_transno and obd_connected_clients. In this patch, I revised
this piece of code by:
- modify obd_connected_clients to cfs_atomic_t
- grab obd_recovery_task_lock only if ocd_transno is less than
  obd_next_recovery_transno

In this way, target_handle_connect() don't grab any global lock.

Change-Id: I971897de82566e239fe750cf95f2c3f1646325bc
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1293
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mikhail Pershin <tappro@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
ORNL-28 recovery: rework extend_recovery_timer()

extend_recovery_timer() is used to adjust timeout value of a recovering
target. In the original implementation, there was a problem it stopped
the target from firing a timer again for version recovery case.

Change-Id: I815a15fb5d3104e52a189eed1529c58d7a8d03b9
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1620
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mikhail Pershin <tappro@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
ORNL-8 IR: minor fixes to IR patches

Summary of fixes:
1. ORNL-13: add comments to mgs_handle_fslog_hack();
2. ORNL-14: typo and accurate error/debug messages;
            minor fixes in recovery-small.sh;

Change-Id: I6317067eb6250faf5df21be719c82de17b0f4cc9
Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/1582
Tested-by: Hudson
Reviewed-by: Yu Jian <yujian@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-887 mgc: prevent client IR with old server

Prevent IR enabled client start IR handling with IR unawared MGS
server.

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I9773eafa437358dbf4988e0fddd490b0daf59358
Reviewed-on: http://review.whamcloud.com/1798
Tested-by: Hudson
Reviewed-by: Jinshan Xiong <jinshan.xiong@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-924 test: sync client data before reboot server

Lustre doesn't write client data synchronously (to avoid flooding sync
writes when there are many clients connecting, see exp_need_sync), so
if the server reboots before client data reachs disk, the client data
will be lost and client will be evicted after recovery, which is not
what we expected.

Signed-off-by: Niu Yawei <niu@whamcloud.com>
Change-Id: I21bcafe1f2630285c9108c0528467eb177e3449b
Reviewed-on: http://review.whamcloud.com/1888
Tested-by: Hudson
Reviewed-by: Chris Gearing <chris@whamcloud.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-868 ptlrpc: Fix the timeout for waiting next replay

During recovery, when setting the timeout for waiting the next
replay, it should consider netlatency(added into timeout) and
early reply as well, so if server sends the early reply for
the request, the client might extend the timeout according to
current estimate service time.

Signed-off-by: Wang di <di.wang@whamcloud.com>
Change-Id: I23ebf1dc3f525f78573890be26474b2c79c65a6d
Reviewed-on: http://review.whamcloud.com/1716
Tested-by: Hudson
Reviewed-by: Jinshan Xiong <jinshan.xiong@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-889 recovery: rework extend_recovery_timer()

Commit 8da33c6cc5192303fcd18f45892e1f115004e662 is buggy where
the time used to rearm the timer is wrong. In this patch, I
will reset all changes and use the same scheme as 2.1 release.

Signed-off-by: Jinshan Xiong <jinshan.xiong@whamcloud.com>
Change-Id: Icd887687262644e1a1cfffd6f34eacc5bc418f84
Reviewed-on: http://review.whamcloud.com/1722
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <tappro@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
ORNL-27: Cancel on completion lock on the MGS

We should cancel the recover/config LCK_EX lock immediately
when they are granted to accelerate enqueue process.

Also, it doesn't make sense to add mgc recover/config lock into
LRU list because these kinds of lock would never be canceled
voluntarily. Restore LDLM_FL_NO_LRU flag and apply it for mgc
lock.

Signed-off-by: Jinshan Xiong <jay@whamcloud.com>
Change-Id: I369b57ca4780b0bfa07d33b4423b468481263ade
Reviewed-on: http://review.whamcloud.com/1261
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1259 mgs: avoid cancelling IR lock twice

There is a race that if IR shared lock is revoked fastly the
completion ast can be called twice with the lock granted. The
IR lock should be cancelled only once in this case.

Signed-off-by: Jinshan Xiong <jinshan.xiong@whamcloud.com>
Change-Id: Ie1b2742d436fb6f549cfcb25dc50fbbabcd1f008
Reviewed-on: http://review.whamcloud.com/2390
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1252 recovery: reduce reconnect time for IR

Two problems fixed in this patch:
1. network latency can be increased to incredible large after all
   connections have been tried. Limited it to not be over
   CONNECTION_SWITCH_MAX;
2. reconnect the failing target ASAP when the previous connecting
   RPC failed, if the client has already been notified by IR.
With the above two fix, it can reduce the recovery time if there was a
connecting RPC in flight when the client is notified by IR.

Signed-off-by: Jinshan Xiong <jinshan.xiong@whamcloud.com>
Change-Id: If43e93037e418b7a775228ca7abbe8b337e44e9a
Reviewed-on: http://review.whamcloud.com/2371
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1252 recovery: don't always swap nidtbl entries

Swap nidtbl entries only when server and client are using different
byte order.

Signed-off-by: Jinshan Xiong <jinshan.xiong@whamcloud.com>
Change-Id: I58ebe98e81040ec187f4ecd75020b7b84c7d8904
Reviewed-on: http://review.whamcloud.com/2410
Tested-by: Hudson
Reviewed-by: Johann Lombardi <johann@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1095: debug: Clarify a IR message about support.

During testing of Imperative Recovery we turned off IR but in the
logs we saw a IR:ON message which is confusing since we set it to
disable on the MGS. The IR:ON message is only to make aware that
IR functionality exist. A better message would be IR: [DIS|EN]ABLED.

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: I5990b26b326ce912faf6c5d556cb92334748322c
Reviewed-on: http://review.whamcloud.com/2853
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@whamcloud.com>
LU-1582 tests: do not force sync before failover

Commit 17a69cf25ed0991e04d85c259f4294dc59734e1e forced sync on every
target on a facet before it was failed, in order to fix failures in
recovery-small.sh test_105() due to initial client connect failing.
This was causing local-node testing to hang in replay-ost-single.sh
test_8b() because the client mountpoint was being sync'd and would
never finish due to a fail_loc preventing any RPCs from completing.

Syncing the targets before failover also defeats the purpose of many
recovery tests.  Instead, recovery-small test_105 creates a file on
the to-be-failed OST and writes to it, to trigger the exp_need_sync
and commit the new client export, without incorrectly forcing every
OST to be sync'd for every test that is using facet_failover().

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ied5f2ecf1f6523d4163916b24e5c4281902d500c
Reviewed-on: http://review.whamcloud.com/3239
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@whamcloud.com>
Reviewed-by: Niu Yawei <niu@whamcloud.com>
LU-1634 test: Check client only for version matching

For reliability, we should check nidtbl versions on the client only
to make sure the client is not notified if IR on the MGS is disabled.
Otherwise, if the MGS can't respond while OST is restarting, the
nidtbl version won't be updated and this will fail the test.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I903add4261f5fb133ae4ac170b10c0783c226e29
Reviewed-on: http://review.whamcloud.com/3511
Tested-by: Hudson
Reviewed-by: Yu Jian <yujian@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>