[private] Small change to build hbase against alligator.

Author: pritam

Summary:
The constructor for LocalFileSystem was changed in
https://phabricator.fb.com/D401312 to throw an IOException. Making a
small change to hbase so that we can compile against alligator HDFS.

Test Plan:
1) Build hbase with alligator HDFS, hadoop-hive and Titan
hadoop.

Reviewers: kannan

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D476652

Task ID: 774638

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378275 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-5355] RPC compression HTable interface

Author: aurickq

Summary:
1. ability to change RPC compression on the fly
2. put an option to enable RPC tx/rx compression in loadtest tool
3. bugfix: call returnCompressor/returnDecompressor to return codecs to CodecPool

Test Plan: ran loadtest

Reviewers: kannan, kranganathan, aaiyer, nspiegelberg

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D540377

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378276 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-6423] Fix Config strings for throwing RegionOverloadedExceptions

Author: aaiyer

Summary:
Noticed that the config strings were not set correctly for the HBase-6423
diff. Fix the config strings for HBASE-6423.

Test Plan:
MR tests. Test on the unicorn test cluster for blocking
memstore exceptions.

Reviewers: kannan, kranganathan

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D546688

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378277 13f79535-47bb-0310-9956-ffa450edef68
Revert D530335 [HBASE-6235] Reset the max and min for the time varying metrics in HBase

Author: liyintang

Summary:
Revert: D530335 ([HBASE-6235] Reset the max and min for the time varying metrics in HBase ) in hbase trunk as well.

Because the D482541 hasn't committed in the alligator branch and we want to build hbase-trunk with alligator for shadow testing.

Test Plan: Not tested

Reviewers: kannan, hkuang, pritam, aaiyer

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D544220

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378320 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-6235] Reset the max and min for HBaseRPCMetrics

Author: liyintang

Summary: Reset the max and min for HBaseRPCMetrics

Test Plan: tested in the ods shadow

Reviewers: kannan, aaiyer

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D546829

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378321 13f79535-47bb-0310-9956-ffa450edef68
[master] Fix for trunk -> saturn rpc incompat: client stuck on /hbase/root-region-server-complete

Author: kranganathan

Summary: If we don't find /hbase/root-region-server-complete, also try the legacy location (/hbase/root-region-server). Once all clients are upgraded, we can remove this code.

Test Plan:
Tested by running the load tester against a table:
bin/hbase org.apache.hadoop.hbase.util.LoadTestTool -num_keys 50 -write 1:50:20 -tn kannan_interop_test -compression NONE -zk hbasectrl047.ash3.facebook.com

Reviewers: kannan, mycnyc, aaiyer

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D547972

Task ID: 1373721

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378323 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-6583] Create a CF if its not present in the load test tool

Author: kranganathan

Summary: The load test tool used to error out with an NPE if the CF is not present. This patch will add the CF.

Test Plan: Tested against a table on the new shadow

Reviewers: kannan

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D548066

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378324 13f79535-47bb-0310-9956-ffa450edef68
[master] Fix TestHLog failure.

Author: pritam

Test Plan: Run the test.

Reviewers: mbautin, hkuang, kannan, kranganathan

Reviewed By: hkuang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D539685

Task ID: 1297831

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378325 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-6508] [89-fb] Handle backwards compatibility for clients

Author: avf

Summary:
HServerInfo serialization was changed in HBASE-6508. This broke
compatibility with HBase clients that aretrieve cluster status. This
patch allows clients and server to disable sending the last flushed
sequence id field of serverInfo when it is not necessary, restoring
backwards compatibility between clients and servers.

Test Plan:
1. Test communication between new client and old server
2. Test communication between old client and new server

Revert Plan:

Reviewers: kannan, aaiyer

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D548575

Task ID: 1373422

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378328 13f79535-47bb-0310-9956-ffa450edef68
[master] Disable/Enable the load balancer from hbase shell

Author: liyintang

Summary:
Disable/Enable the load balancer from hbase shell
For example:
./bin/hbase shell
hbase(main):001:0> enable_loadbalancer
LoadBalacner disabled: false

hbase(main):002:0> disable_loadbalancer
LoadBalacner disabled: true

hbase(main):003:0> get_loadbalancer
LoadBalacner disabled: true

Test Plan: tested on dev cluster

Reviewers: kannan

Reviewed By: kannan

CC: hbase-eng@, hbase-ops@lists, paultuckfield

Differential Revision: https://phabricator.fb.com/D548720

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378330 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-6590] Assign SequenceNumber to BulkLoaded files.

Author: aaiyer

Summary:
StoreFiles within a store are sorted based on the sequenceId. SequenceId is a monotonically increasing number that accompanies every edit written to the WAL. For entries that update the same cell, we would like the latter edit to win. This comparision is accomplished using memstoreTS, at the KV level; and sequenceId at the StoreFile level (to order scanners in the KeyValueHeap).

BulkLoaded files are generated outside of HBase/RegionServer, so they do not have a sequenceId written in the file.  This causes HBase to lose track of the point in time, when the BulkLoaded file was imported to HBase. Resulting in a behavior, that **only** supports viewing bulkLoaded files as files back-filling data from the begining of time.

By assigning a sequence number to the file, we can allow the bulk loaded file to fit in where we want. Either at the "current time" or the "begining of time". The latter is the default, to maintain backward compatibility.

Design approach:
Store files keep track of the sequence Id in the trailer. Since we do not wish to edit/rewrite the bulk loaded file upon import, we will encode the assigned sequenceId into the fileName. The filename RegEx is updated for this regard. If the sequenceId is encoded in the filename, the sequenceId will be used as the sequenceId for the file. If none is found, the sequenceId will be considered 0 (as per the default, backward-compatible behavior).

To enable clients to request pre-existing behavior, the command line utility allows for 2 ways to import BulkLoaded Files: to assign or not assign a sequence Number.
- If a sequence Number is assigned, the imporeted file will be imported with the "current sequence Id".
- if the sequence Number is not assigned, it will be as if it was backfilling old data, from the begining of time.

Compaction behavior:
- With the current compaction algorithm, bulk loaded files -- that backfill data, to the begining of time -- can cause a compaction storm, converting every minor compaction to a major compaction. To address this, these files are excluded from minor compaction, based on a config param. (enabled for the messages use case).
- Since, bulk loaded files that are not back-filling data do not cause this issue, they will not be ignored during minor compactions based on the config parameter. This is also required to ensure that there are no holes in the set of files selected for compaction -- this is necessary to preserve the order of KV's comparision before and after comparision.

Test Plan:
mr tests

extend test bulk import and test both behaviors.

Reviewers: kannan, kranganathan, gqchen, nspiegelberg, mbautin, liyintang

Reviewed By: kannan

CC: HBase Diffs Facebook Group

Differential Revision: https://reviews.facebook.net/D3789

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378331 13f79535-47bb-0310-9956-ffa450edef68
[master] [89-fb] HMaster shouldn't read /hbase/root-region-server-complete every second

Author: avf

Summary:
After the new master failover design, LegacyRootZNodeUpdater in
HMaster would read /hbase/root-region-server-complete every
second. This is not necessary, as the only place where it is updated
is in RegionManager: simply reading it once at start and then waking
the LegacyRootZNodeUpdater thread up only if RegionManager updates
/hbase/root-region-server-complete should suffice.

Test Plan:
Unit test, test master failover

Revert Plan:

Reviewers: kannan, kranganathan, mbautin, pkhemani

Reviewed By: kannan

CC: hbase-eng@, aaiyer

Differential Revision: https://phabricator.fb.com/D549793

Task ID: 1383655

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378332 13f79535-47bb-0310-9956-ffa450edef68
[master] Add support to call setBatch for a scan in ThriftServerRunner

Author: jeremyfein

Summary:
Extends the thrift interface to include new field batchSize on TScan when
opening a scanner using TScan. When set, this field triggers a call to
"setBatch" on the scan object to limit number of values reurned.

Test Plan: Need to deploy on test cluster

Reviewers: nzhang, chaoyc, mbautin, aaiyer

Reviewed By: aaiyer

CC: hbase-eng@, davejwatson

Differential Revision: https://phabricator.fb.com/D551420

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378335 13f79535-47bb-0310-9956-ffa450edef68
Reduce overhead of maintaing get/next size metric [89-fb] [HBASE-6217]

Author: mycnyc

Summary:
Accumulate increments to the metrics and apply them all at
once. [89-fb] [HBASE-6217]

Test Plan: unit tests

Reviewers: kannan, liyintang

Reviewed By: kannan

CC: hbase-eng@, michalgr

Differential Revision: https://phabricator.fb.com/D552282

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378336 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-6629] [0.89-fb] Fast fail, without retries, if the client has not been able to contact a server in a while

Author: aaiyer

Summary:
New way to improve client side waiting for a unresponsive regionserver.
 1) keep count of repeated connect failures to a server.
 2) throw RepeatedConnectException instead of ConnectException.
 This contains more deatils such as
 - # of failures
 - time of first failure
 show rpcRetryTimetout and current time

Test Plan:
Run unit tests on MR to test for existing stuff. -- All pass.

 Deploy to dev cluster.
   - run a load tester job,
   - kill one of the RS during the load testing job
   - see that the RepeatedConnectExceptions get thrown
   - see the # of ops should be better with the diff.
 Graph for #ops and latency (ms) uploaded to
https://www.facebook.com/pxlcld/8hnw

https://www.facebook.com/pxlcld/8hnh

Reviewers: kranganathan, kannan, avf

Reviewed By: kranganathan

CC: hbase-eng@, riyer, anshumansingh26, liujiakai

Differential Revision: https://phabricator.fb.com/D535439

Task ID: 1285514

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378337 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-6423] [0.89-fb] change hbase block multiplier to support float.

Author: aaiyer

Summary:
Currently this is set to long. So, we have to set it to 2.0 at least.
Changing it to float will allow us better control.

Test Plan: mr unit test.

Reviewers: kannan, kranganathan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D546696

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378339 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-6590] [0.89-fb] Fix TestStoreFile.

Author: aaiyer

Summary:
fix TestStoreFile to reflect the fact that bulk load
can also have sequence id

Test Plan: run the unit test

Reviewers: kannan, kranganathan

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D554005

Blame Revision: https://reviews.facebook.net/D3789

Task ID: 1405145

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378341 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-6640] [0.89-fb] Allow multiple regions to be opened simultaneously, from a thread pool

Author: aaiyer

Summary:
Although master can assign multiple regions to the RS simultaneously,
  regionservers currently only process this one by one.

  This causes the time taken for region opening to be ~5-10 mins during a
  rack failure.

  A simple change to parallize region opens. This should help us cut down this
  time.

Test Plan:
   test on mr
   do a rack failure test and check the time taken to assign+open regions

Reviewers: kannan, kranganathan, avf

Reviewed By: avf

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D545919

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378342 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-6371] Level Based Compaction

Author: liyintang

Summary: We've already discussed all features here. We'll fill in a detailed summary later.

Test Plan: New tests added for size based levelAssignment. Also tests for Recent first level traversal and vice versa.

Reviewers: liyintang, kannan

Reviewed By: liyintang

CC: hbase-eng@, mbautin

Differential Revision: https://phabricator.fb.com/D548547

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378348 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-5898] Consider double-checked locking for block cache lock

Author: liyintang

Summary:
I understand the intension of IdLock is to avoid duplicatedly loading data to cache.
However, I believe it is more reasonable to acquire this IdLock ONLY when  the cache misses (assuming the cache is enabled).
The sequence may be:
1) check block has been cached or not
2) If cached, just return the cached data.
3) If not, acquire the IdLock
4) Acquired the IdLock and check the block cache again
5) If cache misses, load the data into cache and return the data
6) If cache hits, return the cached data.

In this way,  we don't need to pay this lock overhead if this particular block has already been cached.

Test Plan: tested on the ODS shadow and production cluster for a while

Reviewers: kannan, kranganathan, michalgr

Reviewed By: kannan

CC: hbase-eng@, sdong

Differential Revision: https://phabricator.fb.com/D552051

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1378349 13f79535-47bb-0310-9956-ffa450edef68