[HBASE-7655] Separates pread latencies for compactions vs normal read ops

Author: rshroff

Summary:
When we read a block, we already know whether the read operation is for
compaction or a normal operation.

This change adds new metrics to separately measures this read operations
to measure the performance impact.

Test Plan:
Test it on titanshadow.
http://fburl.com/9680926

Reviewers: liyintang

Reviewed By: liyintang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D686418

Task ID: 2043991

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459451 13f79535-47bb-0310-9956-ffa450edef68
[0.89-fb] [ HBASE-7517] fix PFFE issue when ROOT moves around

Author: aaiyer

Summary:
Fix the issue in TITAN018, that gets appservers
to be stuck throwing PFFE.

It looks like the appservers that are having this trouble
are app servers that have the old-notion of ROOT. This seems
to not get updated, due to the fast fail mechanisms.

Ensure that we always get/refresh the knowledge from ZK, if
the root region server is expected to be inFastFail mode.

Test Plan:
Repro and test.
See if this can be programmtically done as well.

Reviewers: liyintang

Reviewed By: liyintang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D686758

Task ID: 1966872

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459452 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7655] Fixing a change

Author: rshroff

Summary: The if clause got added during git rebase. Fixing it.

Test Plan: Build

Reviewers: liyintang, adela

Reviewed By: adela

Differential Revision: https://phabricator.fb.com/D687333

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459453 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7707][89-fb] A simple tool which will COPY or create a smaller table out of another table

Author: adela

Summary:
as in title, 2 possibilities:
1) user can bulkupload the old table to a new table
2) can create smaller table out of a big table by specifying the number of regions (and KVs) which will be copied.
--- HOW TO RUN THIS---
> ./bin/hbase org.apache.hadoop.hbase.mapreduce.TableCopyUtils -bl
Name of the old table: test_table
Name of the new table: abcb
Do you want to set start and stop row for the old table scanner? [y/n] n

(this will create a full copy of table test_table and the new table will be called abcb)

Test Plan:
ran this locally on a very small table. Planning to run this
on tsh25 to create a small table out of cell5_v1 for compaction hook testing.

Reviewers: liyintang

Reviewed By: liyintang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D686059

Task ID: 2051427

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459454 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7707][89-fb] Improved copy table user interface

Author: adela

Summary:
Deleted the functionality of creating a smaller table by scan
and put. Improved the interface for the users

Test Plan: tested on my machine

Reviewers: liyintang

Reviewed By: liyintang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D693127

Task ID: 2051427

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459455 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7732] Honors the hfile.io.bytes.per.checksum parameter for HFile Writer

Author: rshroff

Summary:
The HFile Writer used to take the default value
DEFAULT_BYTES_PER_CHECKSUM for bytesPerChecksum. The
hfile.io.bytes.per.checksum was added to provide an option to change
this setting. The HFile Writer create() was not reading this
configuration option. Changed it to read it on every create() call.

Test Plan:
Added a small unit test to make sure that the getBytesPerChecksum
is called and the value from the conf is used.
More suggestions welcomed.

MR:
http://hbaseproxy001.prn1.facebook.com/mr_jt/hbasebackup003-ash3/jobdetails.jsp?jobid=job_201209211223_47382&refresh=30

Reviewers: liyintang

Reviewed By: liyintang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D694332

Task ID: 2064279

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459456 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7734] Fix TestFromClientSide#testFilterAcrossMultipleRegions in trunk.

Author: manukranthk

Summary: In RegionScanner.nextRows, checking filter's done value to ensure that null, which was the expected value by the client, was returned. This check was removed when ScanPrefetcher was introduced.

Test Plan: Ensure Unit tests pass.

Reviewers: liyintang, kranganathan, aaiyer

Reviewed By: liyintang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D695147

Task ID: 1980553

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459457 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7737] Normalizes stringValue for HServerAddress

Author: rshroff

Summary: Always sets the stringValue to host_ip_addr:port.

Test Plan:
Added a small unit test.
MR:
http://hbaseproxy001.prn1.facebook.com/mr_jt/hbasebackup003-ash3/jobdetails.jsp?jobid=job_201209211223_47720&refresh=30

Reviewers: liyintang

Reviewed By: liyintang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D695632

Task ID: 2068631

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459458 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7276] Fix TestRegionServerMetrics failing test one more time

Author: adela

Summary:
Can't see how we get negative numbers there. I am putting the
test case in a separate class let's see if that will fix the problem

Test Plan: tested on my machine

Reviewers: liyintang, rshroff, manukranthk

Reviewed By: rshroff

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D695499

Task ID: 1981461

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459459 13f79535-47bb-0310-9956-ffa450edef68
[89-fb] [HBASE-7509] Export quorum metrics from HRegionServer

Author: aaiyer

Summary:
DFSClient metrics are disbled to save CPU usage.
This diff will expoet the required quorum Read Metrics
from HRegionServer

Test Plan:
deploy to dev cluster and ensure that the
metrics are seen via ODS counter

part of the output from hbasedev131.ash3:
"hadoop.regionserver_quorumreadsdone": 23,
"hadoop.regionserver_quorumreadsexecutedincurthread": 0,
"hadoop.regionserver_quorumreadwins": 0,

Reviewers: liyintang, rshroff

Reviewed By: rshroff

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D697700

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459460 13f79535-47bb-0310-9956-ffa450edef68
[0.89-fb] [master] Fix the order of handling MSG_REPORT_EXITING

Author: aaiyer

Summary:
Before this fix, we process all the region closes, before marking
the RS as dead. This causes an issue where all the transient region
assignments are placed on the exiting server (if it happens to be the
    primary). This would not happen, if the server were in the list
of dead servers.

But, because we add the server to the list of dead servers, after
processing the region closes, the server seems "alive and active"
during the assignment process. Fixing the order, so that the
region will be assigned to a different RS, (considering the exiting
    RS as dead.)

Test Plan:
test suite. only failure TestHRegionCloseRetry, also fails
without the diff

Push to TSH025 master and make sure that the regions are not assigned
back to the exiting server.

4 /tmp/2013-02-06_14_39_23_hbase-trunk_fixRSExitingOrder-mrFailures  Failed on
MR:  TestHRegionClose TestHRegionCloseRetry TestRegionServerMetrics
TestThriftServer
1 /tmp/2013-02-06_14_39_23_hbase-trunk_fixRSExitingOrder-localFailures  Failed
on Local:  TestHRegionCloseRetry
mv target/surefire-reports
mr_unit/2013-02-06_14_39_23_hbase-trunk_fixRSExitingOrder

Reviewers: liyintang

Reviewed By: liyintang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D701100

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459461 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7532] Enable the FavoredNodes for the HFileOutputFormat

Author: manukranthk

Summary: HFileOutputFormat has a writer embedded but it is not configured to use the FavoredNodes. This revision is to add that support to the HFileOutputFormat while it is used as an OutputFormat either the Mapper or the Reducer for a MapReduce task.

Test Plan: Unit Testing.

Reviewers: liyintang, kannan

Reviewed By: liyintang

CC: hbase-eng@, astewart

Differential Revision: https://phabricator.fb.com/D675950

Task ID: 1964673

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459462 13f79535-47bb-0310-9956-ffa450edef68
[master] Fast fail META and ROOT during rack reboots. Do not wait for 10 mins

Author: aaiyer

Summary:
Having .META. unavailable during rack failures can block the
client threads trying to fetch information from meta.

In case of a rack reboot. we will not wait for 10 minutes for these
servers.

Test Plan:
MR unit tests.
Deploy to dev cluster, and kill the rack with .META. or ROOT.
  - verify that the recovery takes off immediately
kill a normal rack.
  - verify that we do still wait for 10 mins.

Reviewers: kannan, kranganathan, liyintang

Reviewed By: liyintang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D544346

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459463 13f79535-47bb-0310-9956-ffa450edef68
[0.89-fb] [HBASE-7517] Improve the PFFE mechanism such that (i) Ensure that if a thread gets the AtomicBoolean for a single server, it will not be stopped to get the AtomicBoolean for a different server. (ii) PFFE is only thrown for threads that are not a

Author: aaiyer

Summary:
Improvements to the PFFE mechanism, so as to let the
retrying threads to handle errors as if they were not in PFFE mode.

Threads that are allowed to go through (1 per server) will follow the
same logic as normal threads -- when not in fast fail mode. (The only
    exception being that all errors will be logged as LOG.error).
These threads will never throw a PFFE.

Threads that are not allowed to go through, will throw PFFE Exception.

Test Plan:
Add unit test to debug.
Test it out on tsh025.

Reviewers: liyintang, kannan, kranganathan, adela

Reviewed By: liyintang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D690778

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459464 13f79535-47bb-0310-9956-ffa450edef68
[0.89-fb] [HBASE-7852] Ensure that the command line options are respected.

Author: aaiyer

Summary:
LoadIncrementalHFiles should extract the config options after
Tool.run() is called, to ensure that the command line overrides
are respected.

Test Plan:
Test out on the dev cluster, with command line overrides.
Ensure that the command line override for configuration settings is
respected

Reviewers: adela, liyintang

Reviewed By: liyintang

CC: hbase-eng@, ryandm

Differential Revision: https://phabricator.fb.com/D693206

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459465 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-5032] Add Delete Column into the delete bloom filter

Author: adela

Summary:
https://issues.apache.org/jira/browse/HBASE-5032

Previously, the delete family bloom filter only contains the row key
which has the delete family. It helps us to avoid the top-row seek
operation.

This jira attempts to add the delete column into this delete bloom
filter as well (rename the delete family bloom filter as delete bloom
filter).

The motivation is to save seek ops for scan time-range queries if we
know there is no delete column for this row/column.
We can seek directly to the exact timestamp we are interested in,
instead of seeking to the latest timestamp and keep skipping to find
out whether there is any delete column before the interested timestamp.

Test Plan: write unit tests

Reviewers: kannan, liyintang, kranganathan

Reviewed By: liyintang

CC: erling, hbase-eng@

Differential Revision: https://phabricator.fb.com/D622271

Task ID: 849012, 1980066

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459466 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7903] Add the partial rows, max response size, prefetch options to streaming

Author: rshroff

Summary:
Adds the partial rows, max response size, batchsize and prefetch options
to the TableInputFormat for HBase streaming.

Test Plan:
Test it on SH004. Scanned a 1.3G row.
http://hbase185.frc1.facebook.com.:50060/tasklog?taskid=attempt_201302121607_3232_m_000000_0&all=true

O/P file: https://phabricator.fb.com/P1997817

Reviewers: liyintang, pritam

Reviewed By: liyintang

CC: nzhang, hbase-eng@

Differential Revision: https://phabricator.fb.com/D715153

Task ID: 2120016

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459467 13f79535-47bb-0310-9956-ffa450edef68
[0.89-fb] [HBASE-7852] fix TestLoadIncrementalHFiles

Author: aaiyer

Summary:
TestIncrementalHFiles  does not use the Tool.run() mechanism to
do the bulk load.

In order to fix the test, ensure that we pick up the default options in
the constructor. If Tool.run() is called, the default options are overridden.

Test Plan: TestLoadIncrementalHFiles

Reviewers: liyintang

Reviewed By: liyintang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D718323

Blame Revision: https://phabricator.fb.com/D693206

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459468 13f79535-47bb-0310-9956-ffa450edef68
[0.89-fb] 	[HBASE-5783] Fix a possible ConcurrentModificationException

Author: aaiyer

Summary: Fix possible ConcurrentModificationException

Test Plan: unit tests

Reviewers: adela, liyintang

Reviewed By: adela

CC: hbase-eng@, manukranthk

Differential Revision: https://phabricator.fb.com/D719424

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459469 13f79535-47bb-0310-9956-ffa450edef68
[Master]Optimize HRegionServer.updateOutboundMsgs and HRegionServer.addOutboundMsgs

Author: adela

Summary: Optimizing the two methods to execute in O(n).

Test Plan: will run unittests

Reviewers: aaiyer, liyintang, kannan

Reviewed By: aaiyer

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D715356

Task ID: 2124466

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1459470 13f79535-47bb-0310-9956-ffa450edef68