[0.89-fb] [HBASE-6680] Allow multiple SplitLogWorkers per RS

Author: aaiyer

Summary:
Currently SplitLogs is the slowest part of the pipeline in
processing dead servers. Based on rough back of envelop calculations,
  for rack switch reboot, it takes about 2-3 rounds of log assignments ( 20 *
      8 logs / (80 - delta) remaining servers) to get through this phase.

  One low hanging fruit is to increase the #threads for split logs in each RS.
  Setting the default to 3, because 2 may be cutting it too close.

Test Plan:
run MR tests.
  - only failure (TestMasterTransitions) fails on master as well.

  push to dev cluster. kil one RS. Check master logs for split log messages.
     output shows multiple threads per RS working in parallel to grab and process logs.

Reviewers: kannan, kranganathan, avf

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D560200

Task ID: 1417129

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1379539 13f79535-47bb-0310-9956-ffa450edef68
support for bin/hadoop classpath bin/hbase classpath

Author: pkhemani

Summary: as in title

Test Plan: ran changed bin/hadoop and bin/hbase scripts in the source directories and that worked fine.

Reviewers: mbautin, sdong

Reviewed By: mbautin

CC: hbase-eng@, hdfs-dev@, pritam

Differential Revision: https://phabricator.fb.com/D559998

Task ID: 1409831

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1379540 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-6681] [89-fb] Propagate RegionOverloadedException to the Thrift client

Author: mbautin

Summary: To correctly propagate RegionOverloadedException (added in HBASE-6423) to the Thrift client we need to make it a Thrift exception. One way to do that is to inherit from IOError and add the desired client-side delay in milliseconds to IOError.

Test Plan: Run a Thrift-based load test with small memstore that would result in blocking writes and verify that IOErrors indicating that the regionserver has been overloaded are being propagated to the client.

Reviewers: aaiyer, kranganathan, nzhang

Reviewed By: aaiyer

Differential Revision: https://reviews.facebook.net/D4983

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1379541 13f79535-47bb-0310-9956-ffa450edef68
[0.89-fb] [HBASE-6680]  Separate serverName and workerName for SplitLogWorkers

Author: aaiyer

Summary:
 With regards to Prakash's comments on the original diff
 https://phabricator.fb.com/D560200

1) move worker id logic inside splitLogWorker
2) change SplitLogManager to rely on the serverName instead of workerName to
handle server failures.

Test Plan:
additional testing:
   - deploy to dev cluster
   - initiate split logs by killing RS
   - Kill another RS during split logs and verify that the master handles the
   event correctly.

Reviewers: kannan, pkhemani, kranganathan

Reviewed By: pkhemani

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D561386

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1379542 13f79535-47bb-0310-9956-ffa450edef68
[0.89-fb] [HBASE-6605] Allow bulk load to go past failures.

Author: aaiyer

Summary:
Bulk load currently fails immediately whenever one file in the
bulk load directory is not successfully bulk loaded.

Allow for this behavior to be controlled through a conf change

Test Plan: test out manually on dev cluster

Reviewers: madhuvaidya, kannan, kranganathan

Reviewed By: kannan

CC: ruifang, hbase-eng@

Differential Revision: https://phabricator.fb.com/D548896

Task ID: 660885

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1379543 13f79535-47bb-0310-9956-ffa450edef68
Remove src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java.
This class is a duplicate of another one in the regionserver package and
is not being used.

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1379578 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-6704] Improve KV usage in StoreScanner#next

Author: rvadali

Summary:
The KV passed to ScanQueryMatcher#match should be used throughout
StoreScanner#next

Test Plan:
tbd

Revert Plan:

Tags:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Reviewers: kranganathan, kannan, aaiyer

Reviewed By: aaiyer

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D554638

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1379755 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-6705] [89-fb] Refactor duplicate CF parsing code in ThriftServerRunner

Author: mbautin

Summary: We have a lot of repetitive code conditional on famAndQf.length <= 1. Refactoring that into a function.

Test Plan: Unit tests

Reviewers: kannan, aaiyer, kranganathan

Reviewed By: aaiyer

Differential Revision: https://reviews.facebook.net/D5091

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1380850 13f79535-47bb-0310-9956-ffa450edef68
[0.89-fb] [master] Remove region's last flushed sequence id _after_ closing

Author: avf

Summary:
When we close a region during normal operation, we flush this region's memstore,
updating last flushed sequence id. This means last flushed sequence id continues
to be reported along with the rest of HServerInfo.

This revision removes the last flushed sequence id _after_ the region
is closed to handle this scenario correctly.

Test Plan:
Test on DL cluster: gracefully move a region from one node to another and
verify that the "last flushed sequence id that is less" is no longer printed.

Revert Plan:

Reviewers: kannan, aaiyer

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D565614

Task ID: 1426639

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1381162 13f79535-47bb-0310-9956-ffa450edef68
[master] Better log message on receiving ZK unassigned events

Author: mbautin

Summary: We are trying to diagnose a bug when a NodeCreated event caused by the OFFLINE node being created in the unassigned region ZK directory triggers a region state transition event, and then a NodeDataChanged event triggers a duplicate instance of the same event. This requires better logging of the ZK event that triggered a region state transition event.

Test Plan: Failure testing on a cluster

Reviewers: aaiyer

Reviewed By: aaiyer

Differential Revision: https://reviews.facebook.net/D5115

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1381584 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-5898] Fix TestFromClientSide#testCacheOnWriteEvictOnClose

Author: michalgr

Summary: Because of double-checked locking we miss twice when block is not present in cache. Test expects only one miss.

Test Plan: mvn -Dtest=TestFromClientSide#testCacheOnWriteEvictOnClose test

Reviewers: liyintang, kannan

Reviewed By: kannan

CC: Karthik, gqchen, aaiyer, mbautin

Differential Revision: https://reviews.facebook.net/D4977

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1381999 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-6741] [0.89-fb] Detect duplicate assignments more accurately.

Author: aaiyer

Summary:
We have seen cases where multiple notifications from ZK about a
region being OPENED at a particular RS, results in the master
believing that the region is duplicately assigned -- even though
both the notifications mention the same RS (in the data).

There are two ways to fix this:
 - detect and fix why we are seeing multiple notifications, and/or
 - fix the duplicate assignment logic to handle multiple notifications
 from the same server.

 This diff does the latter. Whenever a region is opened, we cache the location
of where the region is opened. If we get a second notification for region
being opened at the same server, then we will not consider it to be a
duplicate notification.

Test Plan:
Add a test using injection handler framework to duplicate the
region opened events.

Reviewers: kannan, kranganathan, mbautin

Reviewed By: mbautin

CC: hbase-eng@, pkhemani

Differential Revision: https://phabricator.fb.com/D565749

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1382294 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-6735] [89-fb] Remove reflection from HBaseClient

Author: michalgr

Summary:
Every time we read response in HBaseClient we create instance of valueClass using reflection and Hadoop APIs. valueClass is set in a constructor of HBaseClient to one of parameters. The parameter is always HbaseObjectWritable.

We can remove reflection by hardcoding HbaseObjectWritable.

Test Plan:
Unit tests. 3 fails:
* TestDistributedLogSplitAtStartup
* TestLogSplitOnMasterFailover
* TestDistributedLogSplitting

All 3 fails when run against trunk.

Reviewers: kannan

Reviewed By: kannan

CC: Karthik, mbautin, Liyin, aaiyer

Differential Revision: https://reviews.facebook.net/D5271

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1383389 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-6764] [89-fb] Organize Thrift API methods, make more consistent across repositories

Author: mbautin

Summary: Sorting all Thrift method definitions in Hbase.thrift alphabetically to make diffing easier. We have a very different copy of hbase.thrift in fbcode, and this a step towards reconciling the two. Also adding the attributes argument, which currently does nothing.

Test Plan: Unit tests

Reviewers: kranganathan, aaiyer

Reviewed By: kranganathan

CC: hbase-eng@, davejwatson

Differential Revision: https://phabricator.fb.com/D571612

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1383697 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-6629] [0.89-fb] Fix NullPointerException in HConnectionManager

Author: michalgr

Summary: There is NullPointerException that is thrown from time to time in HConnectionManager

Test Plan:
TestMasterTransitions#testRegionCloseWhenNoMetaHBase2428 was failing with NullPointerException every time I run it locally, and occasionally when run in map reduce. Now it passes.

I will run all tests in map reduce to see if it fixes more tests.

Reviewers: aaiyer

Reviewed By: aaiyer

CC: Karthik

Differential Revision: https://reviews.facebook.net/D5349

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1384311 13f79535-47bb-0310-9956-ffa450edef68
[0.89-fb] [HBASE-6680]  Fix split log worker names

Author: aaiyer

Summary:
 SplitLogWorker is using a human readable worker name that contains
 space to identify itself to SplitLogManager.

 eg:
  <machine name> Worker #X.

 SplitLogWorker uses <machine name>-worker-#number as its temporary directory.
 While this is guaranteed to be unique, it appears that SplitLogManager tries
 to discover the same directory using the "worker name" given by
 SplitLogWorker. This is required to move the split recovered edits files to
 the appropriate place.

 Making both the names same, so that SplitLogManager can discover the
 directory correctly.

Test Plan:
  deploy to test cluster.
  kill a rs.
  ensure that the split log manager discovers the destination correctly, and
  moves the split files.

Reviewers: pkhemani, kannan

Reviewed By: kannan

CC: hbase-eng@, michalgr

Differential Revision: https://phabricator.fb.com/D569555

Task ID: 1452354

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1384672 13f79535-47bb-0310-9956-ffa450edef68
[master] [89-fb] Fix NPE in setOpen

Author: mbautin

Summary: Noticed this in TestRegionStateOnMasterFailure. rHBASEEIGHTNINEFBBRANCH1382294 introduced the line where the NPE happens. Apparently setOpen can sometimes be called when the server name is still null (that should still be investigated).

Test Plan: Run unit tests

Reviewers: aaiyer

Reviewed By: aaiyer

Differential Revision: https://reviews.facebook.net/D5397

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1384673 13f79535-47bb-0310-9956-ffa450edef68
[0.89-fb] add debug message to account for split log timing

Summary: Instrumenting split log methods to find out the bottleneck

Test Plan:
trivial change. deploy to dev cluster to see logs.
  will deploy to dark launch and collect the stats.

Reviewers: kannan, kranganathan, avf, pkhemani

Reviewed By: kranganathan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D561619

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1384679 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-6747] [0.89-fb] Allow the HBase server to limit the size of results returned.

Author: aaiyer

Summary:
We have seen issues where fetching large rows can get the server
into a bad state. This results in Full GC issues, and/or the server
getting OutOfMemory Errors, that can be fatal.

We will henceforth bail quickly, and not pursue a request if it
tries to fetch more than (a configurable) number of bytes.

Test Plan: add a test to TestClientSide to test this behavior.

Reviewers: kannan, kranganathan

Differential Revision: https://phabricator.fb.com/D569695

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1385057 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-6728] [89-fb] bound responseQueue to prevent OOM

Author: michalgr

Summary:
In HBaseServer, if connection is not able to send responses as fast as they are produced, responses are queued in per connection queue. The queue is unbounded and this can cause OOM.

This patch introduces size-based boundary, a configurable limit on how much memory can be occupied by responses in all responsequeues. If this limit is exceeded handler threads have to wait to submit a response.

Test Plan: Unit tests + I will try to observe impact on latency in performance test i used to compare different HBase changes.

Reviewers: kannan

Reviewed By: kannan

CC: Karthik, mbautin, avf, Liyin, gqchen, lhofhansl

Differential Revision: https://reviews.facebook.net/D5337

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1385058 13f79535-47bb-0310-9956-ffa450edef68