HBASE-4884 allow commands to have their own shell settings

Summary:
for example, we might want a smaller heap for thrift compared to the
regionserver.

This might help with an OOM situation we have been hitting on ERIS and HASHOUT,
but even if not it's a useful knob to have.

Test Plan:
* tested with file missing.  no problem, used default conf/hbase-env.sh
settings.

* tested with conf/hbase-env-thrift.sh file with heap size of 1234.  ps axww
showed -Xmx1234m

* started regionserver using bin/hadoop-daemon.sh

* started hbase shell

* ran some jruby scripts via bin/hbase org.jruby.Main

Reviewers: dhruba, kranganathan, mbautin, paultuckfield

Reviewed By: paultuckfield

CC: hbase-ops@lists, dsp, hbase-eng@lists, paultuckfield, dhruba

Differential Revision: 364007

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1212709 13f79535-47bb-0310-9956-ffa450edef68
[master] FSUtils can return per-host locality fraction

Summary:
HBase-based block placment benefits from having the fractional locality
of each region on each server. This changes adds a way to output that
from FSUtils. The process is the same as is used to get the best host
per region, but saves the locality in a map.

Test Plan:
The HBase locality check tool outputs the same results as before,
fractional locality output inspected manually.

Reviewers: kranganathan, kannan

Reviewed By: kannan

CC: hbase-eng@lists, kannan, cgist

Differential Revision: 364427

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1212710 13f79535-47bb-0310-9956-ffa450edef68
HBASE-4863 Fix thread leaks in the HBase thread pool server

Summary: This is part of a fix to solve Thrift server problems observed on Eris
and Hashout clusters. A recent jstack captured by @rthiessen on Eris at the time
of a Thrift server crash had over 15,000 Thrift worker threads. We need to bound
that number with a configuration knob and set it reasonably.

Test Plan: Run Thrift server on a 5-node cluster, open a lot of connections,
and monitor the number of worker threads using jstack.

Reviewers: kannan, kranganathan, rthiessen, nspiegelberg

Reviewed By: kannan

CC: hbase-eng@lists, alasla, davejwatson, nspiegelberg, kannan, kranganathan,
mbautin, cgthayer, pkhemani, gqchen

Differential Revision: 364343

Revert Plan: OK

Task ID: 762593

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1212711 13f79535-47bb-0310-9956-ffa450edef68
D519 HBASE-4823 [jira] long running scans lose benefit of bloomfilters and timerange hints

Revision Status:  Accepted · Next step: arc commit --revision 519
Author: aaiyer
Reviewed by:  kannan
Test Plan: mvn test on the MR

Unit: ★ No Unit Test Coverage
Lint: ★ No Linters Available
Host: dev1396.snc6.facebook.com
Path: /data/users/aaiyer/external-89/0.89-fb/ (svn)
  Arcanist Project: hbase
  Apply Patch:  arc patch D519
  Export Patch: arc export --revision 51

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1212712 13f79535-47bb-0310-9956-ffa450edef68
[master] Regions can have favored nodes which are passed to DFSClient

Summary:
HBase-based block placement depends on being able to place each block of
a single region on a specific set of hosts. This change adds a favored
node field to HRegions, and passes those nodes to the filesystem
whenever a store file is written.

Test Plan:
New test suite TestRegionFavoredNodes. It uses hostname and port taken
directly from the datanodes, whereas practical usage will likely use
only the hostname of the region server. However, the HDFS test case
TestFavoredNodes#testPartiallySpecifiedFavoredNodes covers the case
where port is unspecified.

Reviewers: kranganathan, kannan

Reviewed By: kranganathan

CC: hbase-eng@lists, kranganathan, cgist, nspiegelberg

Differential Revision: 368588

Task ID: 735408

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1212713 13f79535-47bb-0310-9956-ffa450edef68
[master] Refactor preferred assignments in RegionManager

Summary:
Preferred assignments was refactored to encapsulate the management of
assigning regions to particular servers, to support future enhancement
of this assignment logic. RegionManager now supports having regions
sticky to some servers, with a new type of persistent preferred
assignment. Transient perfered assignments behave the same as before.

Test Plan: mvn test; move_region in shell, which uses preferred assignments

Reviewers: kranganathan, kannan

Reviewed By: kranganathan

CC: hbase-eng@lists, kranganathan

Differential Revision: 371583

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1212714 13f79535-47bb-0310-9956-ffa450edef68
BaseScanner looks for favorednodes hints in META

Summary:
** 89 master only **
The favored nodes used for explicit placement of regions are persisted
as a column in the .META. table. This change reads those columns in the
base scanner and updates the persistent preferred assignments in the
RegionManager as necessary.

Test Plan:
manually enter hints into META and verify that they are passed to the
region manager by inspecting log output

Reviewers: kranganathan, kannan

Reviewed By: kranganathan

CC: hbase-eng@lists, kranganathan

Differential Revision: 375699

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1225312 13f79535-47bb-0310-9956-ffa450edef68
[master] Randomize regions in RegionServer report

Summary:
The regions passed in the RegionServer report are used as the candidates
for load balancing. The addition of load balancing regions by explicit
placements will require that candidate regions are not always the so
that any regions without explicit placements will eventually be
considered for the default load balancing.

The regions intended to be included in the report are the most loaded,
but were simply just the first N. This change randomly chooses N regions
to offer as load balancing candidates for each report.

Test Plan: Verify that a cluster can load balance its regions

Reviewers: kranganathan, kannan

Reviewed By: kranganathan

CC: hbase-eng@lists, kranganathan

Differential Revision: 375841

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1225314 13f79535-47bb-0310-9956-ffa450edef68
[master] Munkres' optimal assignment algorithm

Summary:
Explicit placement of regions by locality requires an algorithm for
assigning regions to servers given the locality of each region on each
server. This is an example of the assignment problem which aims to
minimize the total cost (negative locality) of all assignments.

Munkres' algorithm is an optimal algorithm for solving the assignment
problem.  It runs in O(n^3) time and requires O(n^2) space. This may
make it prohibitively costly for large clusters, but remains an option
for moderately sized clusters.  This implementation took ~15s to run for
size n=2000 with real-world cost values.

Test Plan: mvn -Dtest=TestMunkresAssignment test

Reviewers: kranganathan, kannan

Reviewed By: kranganathan

CC: hbase-eng@lists, kranganathan

Differential Revision: 371585

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1225315 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-4997] Pull in HBASE-4897 and HBASE-4997

Summary:
This fixes 3 issues

1. It fixes double counting of batch.installed. In createTaskIfAbsent(), if the
newly constructed task is not eventually used then we have to remember to
decrement batch.installed

2. There was an unhandled race between the zk-callback thread updating a task
without a batch and the another thread installing a batch into that task.

3. Made sure that batch.installed is being accessed by only one thread. Easier
to argue correctness that way.

Test Plan: TestSplitLogManager passes

Reviewers: kannan, kranganathan, mbautin

Reviewed By: kannan

CC: hbase-eng@lists, pkhemani, kannan

Differential Revision: 375764

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1225316 13f79535-47bb-0310-9956-ffa450edef68
[master] upgrade the INCREMENT and GET SIZE metrics as schema metrics

Summary:
The INCREMENT and GET SIZE metrics are still using the old style of per column
family metrics. So upgrade these 2 metrics as schema metrics.

Tag this diff as [master] since this problem has been solved in open source
trunk.

Test Plan:
Run all the unit tests. The following 4 unit tests are failed with and without
this change.

TestScanner
TestZKBasedReopenRegion
TestHeapSize
TestRegionStateOnMasterFailure

Reviewers: kannan, mbautin

Reviewed By: kannan

CC: hbase-eng@lists, nspiegelberg, kannan

Differential Revision: 377941

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1225317 13f79535-47bb-0310-9956-ffa450edef68
[master] Fix TestZooKeeper failure

Summary:
TestZooKeeper failed in a non deterministic  way caused by the thread
interleaving.

Fixed the problem by increasing the number of retry.

Test Plan: Passed TestZooKeeper 20 times continuously.

Reviewers: kannan, pkhemani

Reviewed By: kannan

CC: hbase-eng@lists, nspiegelberg, kannan

Differential Revision: 377055

Task ID: 841280

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1225318 13f79535-47bb-0310-9956-ffa450edef68
[jira] [HBASE-5031] [89-fb] Remove hard-coded non-existent host name from TestScanner

Summary: TestScanner is failing on 0.89-fb because it has a hard-coded fake host
name that it is trying to look up. Replacing this with 127.0.0.1:<random_port>
instead.

Reviewed at https://reviews.facebook.net/D867.

Test Plan: Run TestScanner

Reviewers: Liyin, Kannan, JIRA

Reviewed By: Liyin

CC: Liyin

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1227389 13f79535-47bb-0310-9956-ffa450edef68
[master] Increase the sleep time for TestZooKeeper

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1227390 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-5014] PutSortReducer should adhere to memory limits

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1227391 13f79535-47bb-0310-9956-ffa450edef68
[master] Preferred assignments should ignore servers with no load info

Summary:
Before a server becomes dead or its info is removed from the collection
of server infos, it may be in the process of shutting down. Regions with
preferred assignments should not go to servers which are in such a
state, as identified by the absence of load information, which is among
the first things to be removed when a server shuts down.

Before, regions might get transient assignments to a server which is in
the process of shutting down, then that assignment would have to timeout
before the region could be assigned to another server. This change
should prevent that.

Test Plan:
Gracefully stop a region server which is the primary favored node for
some regions. Observe that those regions move to secondary or tertiary
nodes immediately.

Reviewers: kranganathan, kannan, liyintang

Reviewed By: liyintang

CC: hbase-eng@lists, kranganathan, liyintang, kannan

Differential Revision: 376982

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1227392 13f79535-47bb-0310-9956-ffa450edef68
[master] Average load should compensate for ignored servers

Summary:
When computing the average load across region servers for load balancing
purposes, ServerManager ignores all regions hosted by servers which have
regions being held for them (preferred assignments). However, the number
of servers over which the load was being averaged was not adjusted,
causing the average load to change significantly (and incorrectly)
whenever a preferred assignment was made.

This change adjusts the number of servers when computing the average
load, to compensate for the servers which are ignored.

Test Plan:
mvn -Dtest=TestRegionRebalancing test
Use the shell's move_region command on a balanced cluster and observe
that there are no extraneous region movements.

Reviewers: kranganathan, kannan, liyintang

Reviewed By: liyintang

CC: hbase-eng@lists, kranganathan, liyintang

Differential Revision: 375176

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1227393 13f79535-47bb-0310-9956-ffa450edef68
[master] Regions should not have multiple transient preferred assignments

Summary:
Transient preferred assignments affect the way regions are load balanced
across servers, and those assignments should not exist longer than
necessary. If a server has a transient preferred assignment for a region
which is not in transition, that assignment will prevent the region from
receiving other regions without preferred assignments.

Some unexpected behavior which was observed was a region would have
multiple transient preferred assignments, only one of which could be
honored. The unhonored assignment would cause that server to ignore
assignments of other regions without preferred assignments. If that
server was the least-loaded, the ignored regions would not be assigned
to other servers and would remain unassigned until the preferred
assignment timed out.

This change causes any attempt to add a transient preferred assignment
for a region which already has an assignment to be ignored.

Test Plan:
On a cluster with a mix of regions with and without preferred
assignments, observe from log output that duplicate transient preferred
assignments are prevented and that load balancing does not cause some
regions to be unassigned for extended durations.

Reviewers: kranganathan, kannan, nspiegelberg

Reviewed By: nspiegelberg

CC: hbase-eng@lists, kranganathan, nspiegelberg, cgist

Differential Revision: 376980

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1227394 13f79535-47bb-0310-9956-ffa450edef68
[master] Master passes favored nodes to region servers

Summary:
Region servers can specify favored nodes when writing files with the
DFSClient, but they need to know the favored nodes in the first place.
Since the master already knows the favored nodes from when it scans
META, it can pass them to the region servers in the REGION_OPEN
messages. The ThrottledRegionReopener is also used to reopen regions
when the favored nodes change so that the region server can pick up the
changes.

Test Plan:
Run on a cluster, enter favored nodes into META, observe that all blocks
for any region are placed on the favored nodes.

Reviewers: kranganathan, liyintang, kannan

Reviewed By: kranganathan

CC: hbase-eng@lists, kranganathan, cgist

Differential Revision: 379441

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1227395 13f79535-47bb-0310-9956-ffa450edef68
[master] Refactor LoadBalancer, add AssignmentLoadBalancer

Summary:
The existing load balancer only considers the number of regions when
balancing regions across servers. The addition of explicit region
placements requires that explicitly placed regions are not moved to
different servers by the load balancer.

This change abstracts the load balancer functionality such that multiple
different load balancer implementations can be run, and adds the
AssignmentLoadBalancer which moves regions to the favored nodes. The
default load balancer now ignores any region with an explicit placement.

Test Plan:
- verify that regions without hints are balanced as before
- enter favored node hints into META
- check that regions are assigned to the favored nodes
- kill/stop a region server
- check that the regions on that server move to secondary/tertiary
- start the offline region server
- check that primary regions for that server are assigned to it

Reviewers: kranganathan, kannan, liyintang

Reviewed By: liyintang

CC: hbase-eng@lists, kannan, liyintang, kranganathan, cgist, nspiegelberg

Differential Revision: 376983

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1227396 13f79535-47bb-0310-9956-ffa450edef68