[HBASE-5519][jira] Incorrect warning in splitlogmanager

Summary: fix warning

Test Plan: none

Reviewers: mbautin

Reviewed By: mbautin

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D433604

Task ID: 955713

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1305808 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-5469] [89-fb] Add baseline compression efficiency to DataBlockEncodingTool

Summary: Fixing DataBlockEncodingTool. This is the 89-fb version of
https://reviews.facebook.net/D2409.

Test Plan:
Run unit tests
Run DataBlockEncodingTool on a few files

Reviewers: kannan, aaiyer, liyintang, kranganathan, pkhemani

Reviewed By: kannan

Differential Revision: https://reviews.facebook.net/D2463

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1306182 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-5679] [89-fb] Port load test tool and related unit tests from trunk to 89-fb

Summary: When open-sourcing LoadTestTool that originated in 89-fb, numerous
improvements to the tool were made, and unit tests based on the tool were
created as part of HBASE-4908. These improvements need to be ported back to
89-fb.

Test Plan:
Unit tests
Run LoadTestTool on cluster

Reviewers: Kannan, Karthik, Liyin, nspiegelberg

Reviewed By: Kannan

Differential Revision: https://reviews.facebook.net/D2535

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1307631 13f79535-47bb-0310-9956-ffa450edef68
[master] Make log splitting and master failover tests more stable

Summary:
I have noticed flakiness in TestLogSplitOnMasterFailover, and while addressing
it, made some more improvements to simulated regionserver and master killing and
daemon isolation in unit test:

* Making TestLogSplitOnMasterFailover itself more stable by waiting until all
regions are online before proceeding with the second half of data loading.
Making behavior more consistent by waiting until the master starts splitting
dead regionserver's logs before killing the master itself. In this case, the
backup master always has to split a log directory with a "-splitting" suffix.
* Making HBaseTestingUtility.waitUntilAllRegionsAssigned more robust by
contacting regionservers and verifying that they have opened the regions META
says they have opened.
* Adding an option to use a separate DFS client in each master and regionserver,
so that datanodes marked as bad in one client do not affect others.
* Shutting down the log syncer thread on a simulated regionserver kill.

Test Plan:
Run all unit tests once.
Run all tests whose name contains "LogSplit" in a loop overnight.

Reviewers: pkhemani, liyintang, kranganathan, pritamdamania

Reviewed By: pkhemani

Differential Revision: https://reviews.facebook.net/D2325

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1308651 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-5700] [89-fb] Fix TestMiniClusterLoad* test failures

Summary: Porting TestMiniClusterLoad* tests to 89-fb in HBASE-5679 uncovered
certain problems with mini-cluster setup in 89-fb that need to be fixed. The
mini ZooKeeper cluster was shut down and not instantiated the next time a mini
cluster was started. Also, 89-fb requires separate "namespaces" for
ZooKeeperWrapper when running multiple independent tests in the same unit test.

Test Plan:
Run all unit tests.
Run TestMiniClusterLoad* tests multiple times each.

Reviewers: kannan, pkhemani, liyintang

Reviewed By: kannan

Differential Revision: https://reviews.facebook.net/D2583

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1309066 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-5731] [89-fb] Make max line length 100 in linter

Summary: We have switched to 100 characters per line in our Java files. Making
the change in the linter.

Test Plan: arc lint

Reviewers: JIRA, nspiegelberg, Kannan, tedyu, stack

Reviewed By: tedyu

Differential Revision: https://reviews.facebook.net/D2625

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1310000 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-5355] [89-fb] Compressed RPC's for HBase

Summary:
Needed by a lot of folks who want to do large batch inserts and large remote
cluster reads where network is the bottleneck. This is done at the RPC layer.

Does the following:

- Configuration to control of RPC's are compressed (set "hbase.rpc.compression"
to compression type, eg - "gz")
- Version is bumped from 3 to 4, but is fully backward compatible. If
compression is set to none (default), the the older version number is used.
- Adds a compression type field to each Call object on both the client and
server side

Test Plan:
Wrote a unit test and ran all the unit tests.

Reviewers: kannan, todd, stack, pkhemani

Reviewed By: kannan

CC: nzhang, nspiegelberg, mbautin, Liyin, gqchen, dhruba, khemani, Karthik,
Kannan

Differential Revision: https://reviews.facebook.net/D1671

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1310606 13f79535-47bb-0310-9956-ffa450edef68
[master] Assignment plan improvement and region assignment verification report

Summary:
The previous diff D407485, Integrate the favoredNodes into the new assignment manager, enabled the Master to assign regions based on the assignment plan. This diff is to improvement the assignment plan and to provide a tool to verify the region assignment.

1) Improvement the assignment plan for the new created tables:
Each new created table has one assignment domain. Currently each new table will have its own assignment domain by shuffling all the region servers based on the table name.

The assignment domain will be easy to extend for multi tenancy.

The primary region server is placed in the round robin way, fairly across different racks and hosts. Also the secondary and tertiary region server will be placed by some randomness. The randomness can make sure that the regions, shared with the same primary region server, have a very high probability to run with different secondary region server and tertiary region server.

2) Improvement the assignment plan for the existing tables:
The region placement tool can take a snapshot for the current region assignment situation by scanning META and also can get a locality report from the file system. Based on these two information, the tool can place the primary region server for EACH TABLE to achieve the maximum table level locality, the minimum region movement and slot-based balancing.

In addition, the tool use the same way to place the secondary and tertiary region server as the new created table case.

3) Region assignment verification report
The region placement tool can also generate the region assignment report for each table. Here is the sample report from my dev cluster.

Test Plan:
1) run all the unit tests
2) tested on dev cluster
3) tested on migrate002 cluster

Reviewers: kannan, kranganathan

Reviewed By: kranganathan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D438425

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1310607 13f79535-47bb-0310-9956-ffa450edef68
[master] Fix bug of closed region transitioning from CLOSED to CLOSING

Summary:
Fixed the situation of a transtion from CLOSED to CLOSING in znode for a region.
This happens when Master sends another Close Region RPC right before the Region Server closes the region.
The message gets there right after the close happens, and the regionserver changes the state from CLOSE to CLOSING.

Fixed this by ignoring any close region requests that are marked as already closed

Test Plan: mvn -Dtest=TestHRegionDoubleClose test

Reviewers: madhuvaidya, pkhemani, kranganathan, mbautin

Reviewed By: kranganathan

CC: kranganathan, hbase-eng@, kannan, mbautin

Differential Revision: https://phabricator.fb.com/D435024

Task ID: 982325

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1310608 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-5606][jira] SplitLogManger async delete node hangs log splitting when ZK connection is lost

Summary: the timeout monitor thread shouldn't delete task nodes

Test Plan: SplitLogManager unit tests pass

Reviewers: kannan, kranganathan, mbautin, liyintang

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D441755

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1310610 13f79535-47bb-0310-9956-ffa450edef68
[JIRA] [HBASE-5618] SplitLogManager - prevent unnecessary attempts to resubmits

Summary: update heartbeat time as soon as possible - w/o first waiting to read zk node (1) when zk watch fires (2) when resubmit fails with bad-version

Test Plan: TestSplitLogMangaer passes.

Reviewers: kannan, kranganathan, liyintang, mbautin

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D441769

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1310611 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-5708] [89-fb] Improving robustness of map-reduce-related and other unit tests

Summary:
Some map-reduce-based tests are failing when executed concurrently in 89-fb
because mini-map-reduce cluster uses /tmp/hadoop-<username> for temporary
data.

Test Plan:
Run unit tests
Run map-reduce-based unit tests 50 times each

Reviewers: kannan, kranganathan, liyintang, pkhemani

Reviewed By: liyintang

CC: stack

Differential Revision: https://reviews.facebook.net/D2601

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1310612 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-5730] [89-fb] Make HRegionThriftServer's thread pool bounded

Summary: This JIRA is for a quick fix in 89-fb to reuse TBoundedThreadPoolServer
in HRegionThriftServer. We will address whatever problems HRegionThriftServer
still has in trunk in HBASE-5703.

Test Plan:
- Run unit tests.
- Enable embedded thrift server and use the native HBase client as a load test.
Monitor the number of thrift worker threads created.

Reviewers: kannan, alevchuk, schen

Reviewed By: kannan

Differential Revision: https://reviews.facebook.net/D2643

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1310613 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-5744] [89-fb] Thrift server metrics should be long instead of int

Summary: As we measure our Thrift call latencies in nanoseconds, we need to make
latencies long instead of int everywhere. There is a bug where we truncate a
nanosecond latency to int, which is a problem with RPCs that take more than
2.147483647 seconds to process.

Test Plan:
TestThriftServer is updated to test for the failure case (an RPC is artificially
made to take 3 seconds). The new test case fails without the fix.

Re-run all unit tests.

Reviewers: schen, dhruba, kannan, liyintang

Reviewed By: schen

CC: stack

Differential Revision: https://reviews.facebook.net/D2679

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1310973 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-5335] Dynamic Schema Config

Summary: Ability to add config options on a per-table & per-cf basis

Test Plan: - mvn test

Reviewers:  kannan, liyintang, mbautin
Reviewed By: mbautin

Differential Revision: https://reviews.facebook.net/D2247

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1311668 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-5727] Fix TestHeapSize, broken by HBASE-5335

Summary: Added 'baseConf' reference in HBASE-5335 and forgot to update
FIXED_OVERHEAD

Test Plan:
 - mvn test -Dtest=TestHeapSize

Reviewers: kannan, stack, mbautin, liyintang
Reviewed By: mbautin

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1324690 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-5104] [89-fb] Intra-row pagination by adding row offset per CF.

Summary: Added row offset per CF to Get and Scan.

Test Plan: Added unit test.

Reviewers: kannan, mbautin

Reviewed By: mbautin

Differential Revision: https://reviews.facebook.net/D2385

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1326043 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-5335] Fix TestShell error introduced by Dynamic Config

Summary: Error introduced because of an admin.rb conflict when
cherry-picking from apache89 to internal.

Test Plan:
 - mvn test -Dtest=TestShell

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1326044 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-5684] [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust

Summary:
Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We
need it to start a process-based HDFS cluster as well. We also need to make the
whole thing more stable so we can use it in unit tests.

Also all logs of local HBase cluster daemons are now tailed to the primary log
of the unit test with the appropriate prefixes to make debugging easier.

Test Plan:
Run the new unit test multiple times (10x or 50x).
Run all unit tests.

Reviewers: kannan, liyintang, pkhemani, kranganathan, madhuvaidya, amirshim, tedyu

Reviewed By: liyintang

CC: tedyu

Differential Revision: https://reviews.facebook.net/D2709

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1326045 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-5763] [89-fb] Fix random failures in TestFSErrorsExposed

Summary: TestFSErrorsExposed frequently fails due to unclean mini-cluster
shutdown. Bringing datanodes back up, waiting for some time, and preemptively
killing all regionservers and the master before shutdown.

Test Plan: Run TestFSErrorsExposed 100 times

Reviewers: liyintang, kannan, pritamdamania, amirshim

Reviewed By: amirshim

CC: stack

Differential Revision: https://reviews.facebook.net/D2739

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1326046 13f79535-47bb-0310-9956-ffa450edef68