[HBASE-8221] [0.89-fb] Ensure that we flush/close the regions before RS stops answering requests

Author: aaiyer

Summary:
Ensure that we will first close the regions, and then stop taking
    client requests.

    This will reduce the number of errors seen by the client.

Test Plan:
    MR unit tests
    TestHRegionCloseRetry is the only test that fails.

Reviewers: kannan, liyintang, kranganathan, rshroff

Reviewed By: rshroff

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D729486

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1462831 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-8223] Fixes race condition between append and close in HLog

Author: rshroff

Summary:
The RegionServer was performing a shutdown as it got
YouAreDeadException. It closed the HLog and was waiting to close all the
regions where it got stuck.

The reason region close is stuck because it's trying to obtain a lock
which is occupied by the put op, which in turn is waiting for append to
HLog to complete.

There is race condition here is between append/sync and HLog.close()
work flow.

Scenario:

Thread 1 => doing the append
Thread 2 => doing HregionServer shutdown

Timeline:

t1> 1: Verifies that LogSyncer is not shutting down and HLog is not
closed and calls sync()
t2> 2: HRegionServer issued HLog.close()
t3> 2: In HLog.close(), it joins the LogSyncer thread, which signals all
the threads waiting on syncDone and exits.
t4> 1: In sync, it sees that the sync has not complete until its txd,
hence adds itself to the syncDone.await queue.

Note: at t4, it does not check whether the LogSyncer Thread is alive or
not, which caused this hang.

The fix is to check whether the LogSyncer thread is closing or not
before going to await for the sync to finish.

Test Plan: mr

Reviewers: liyintang, aaiyer

Reviewed By: aaiyer

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D756413

Task ID: 2222578

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1462832 13f79535-47bb-0310-9956-ffa450edef68