[HBASE-7276]  Update the readRequestCount for getRegionInfo only when called from RPC

Author: liyintang

Summary: D645831 increased the readRequestCount for every call of getRegionInfo and actually we shall only do this if the function is called from RPC level. So fix this problem in this diff.

Test Plan: Tested on one of the RS in the titan shadow025 and seems like the request cnt back to normal.

Reviewers: pritam, kannan, adela

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D653435

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1420337 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7348] Add some statistics from DFSClient to RegionServerMetrics

Author: liyintang

Summary: DFSClient actually collected a number of useful statistics such as bytesLocalRead, bytesLocalRackRead and so on. So this diff is going to merge these metrics into the RegionServerMetrics.

Test Plan: Going to test it on the titan shadow cluster in order to investigate the non-local read.

Reviewers: hkuang, kannan, adela, sdong

Reviewed By: sdong

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D657336

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1422051 13f79535-47bb-0310-9956-ffa450edef68
[Master] Additional assignment plan for secondary and tertiary for existing table

Author: adela

Summary:
Created additional assignment plan for placing the secondary and tertiary. The plan works as following:

For each primaryRegion we get the region server. We are trying to pick the servers for secondary and teriary randomly (very similar to placeSecondaryAndTertiaryRS, but we check if this region is already sharing the server for the primary, if so we skip this server. If we have skipped all servers on a rack we go to another rack until our condition is satisfied). In case we reached the last rack we will place the regions there.

Also added printing baseline and projected dispersion when running --diff

Test Plan: tested on shadow clusterr]

Reviewers: liyintang, kannan

Reviewed By: liyintang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D650484

Task ID: 1943777

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1422052 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7358] Adding Compaction write size metric

Author: manukranthk

Summary:
Adding metric to capture the amount of data(in bytes) that was written during a compaction. This along with the volume of data flushed might give better estimates on the usage patterns. (There was some problem with my
git commit messages so, I was not able to commit to the earlier revision(D653175), hence adding a new revision.)

Test Plan: Adding unit tests.

Reviewers: kannan, liyintang

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D657388

Task ID: 1944049

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1422319 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7366] Make old logs directory have a subdirectory structure

Author: amirshim

Summary:
To speed up replication, it is useful to have each regionserver's old logs in a separate directory, so we don't have to do LS on a 300,000 file directory (namenode doesn't like that).  So we're changing the oldlogs directory structure to contain subdirectories of the format: "10.159.11.45%3A60020", for example:
/TITAN029-ASH3-HBASE/.oldlogs/10.159.11.45%3A60020/10.159.11.45%3A60020.1355452542203

Test Plan:
Unit tests - passed.

Run on cluster, and make sure that files are being moved to the correct subdirectories, and being deleted after the retention time has elapsed.

Reviewers: liyintang, nspiegelberg, mycnyc

Reviewed By: liyintang

CC: hbase-eng@, davejwatson

Differential Revision: https://phabricator.fb.com/D659154

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1423101 13f79535-47bb-0310-9956-ffa450edef68
[master] Add support in hbase-daemon for regionserver stop-for-restart

Author: mbm

Summary:
Since hadoopctl wants (and kind of needs) to own the stopping and starting
of its daemons for restarts, this exposes an interface to the the jruby used
in doing the localised restart of the regionserver. hadoopctl can pick this
up and use this during a restart.

Test Plan:
This has been tested successfully on hbase430.frc1 (though it gets reverted
by fbdeploy - or something similar - periodically). Basically - the tests
for the positive situation and negative situations with hadoopctl (other
types of stop, hbase-daemon.sh that doesn't support stop-for-restart etc)
all now pass.

Reviewers: arjen, aleksandr, paultuckfield, rthiessen, kannan, kranganathan, liyintang

Reviewed By: arjen

CC: hbase-prodeng@, hbase-eng@

Differential Revision: https://phabricator.fb.com/D645234

Task ID: 906907

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1424044 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7408] Removes the pread/seek+read option and defaults it to pread

Author: rshroff

Summary:
The change removes the pread/seek+read option and
defaults the option to always do pread.

During the compaction phase, HBase was using sequential reads to load the HDFS blocks via DFSCLient. Sequential reads causes HDFS to perform prefetching of the data which might not be used necessarily used by HBase. The reasong being that the HFile Block might be already present in the cache or it might need to skip a few blocks in the HFile(Bloom filter blocks), etc. This leads to un necessary iops on disks as Hbase might have to seek to a new location to fetch more data which might not have been prefetched.
Doing preads will prevent this prefetching of the data and hence reduce the iops on disks.

The actual performance numbers will be obtained by deploying this change on one of the shadow clusters.

Test Plan:
Execute the unit tests

http://hbasectrl056.ash3.facebook.com:50030/jobdetails.jsp?jobid=job_201209211223_31395

TestHFileWriterV2 is failing, looking into it.
After the review, will deploy it on shadow cluster to measure the
performance.

Reviewers: liyintang

Reviewed By: liyintang

CC: hbase-eng@, kannan

Differential Revision: https://phabricator.fb.com/D661385

Task ID: 1964373

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1425100 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7416] augments tier compaction with fixed boundary option

Author: liyintang

Summary: Currently, in tier compaction the aged-based algorithm considers HFile's age in disk relative to the current time, thus the tiers are actually shifting along the time. In order to best use our prior information about how applications consume the data, it needs another feature to perceive the tiers relative to a fixed time point. For more details, please refer to https://our.intern.facebook.com/intern/wiki/index.php/HBase/Boundary_Based_Tier_Compaction

Test Plan: Modify the TestTierCompaction.java to accommodate the new feature.

Reviewers: aaiyer, kannan, cjin

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D619488

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1425101 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7276] Fix TestRegionServerMetrics.testNumReadsAndWrites to work with the recent changes in HRegion metrics.

Author: adela

Summary:
-removed read counter increase in close();
-changed increment of read operation during getScanner from HRegion to HRegionServer

Test Plan: unit test is passing now

Reviewers: liyintang, andrewl

Reviewed By: liyintang

CC: hbase-eng@, kannan

Differential Revision: https://phabricator.fb.com/D666035

Task ID: 1981461

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1426297 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7444] Change the default user name in the HBase unit test

Author: liyintang

Summary: After moving to baboon,  a large set of HBase unit tests are failed due to the previous hbase user name format.

Test Plan: tested on previous failed unit tests such as org.apache.hadoop.hbase.TestGlobalMemStoreSize

Reviewers: adela, pritam

Reviewed By: adela

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D667427

Task ID: 1984003

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1426610 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7276] Keep track of the num of rows being read and updated in RegionServer

Author: liyintang

Summary:
There was some confusions before whether RS shall maintain the num of read/write rpc call ops or the num of rows being read/updated. After some discussions with Adela, we felt like there is useless to maintain the rpc call ops since region server had already kept track of the multiput, next and get ops anyway.

So this diff is to keep track of the num of rows being read and updated in RS. In addition, I have removed the per-region level read/write cnt since it is not cose-effective. (Expensive to maintain it but not much value gained)

Test Plan: going to test it on the dev cluster and shadow cluster

Reviewers: adela

Reviewed By: adela

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D667725

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1426612 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7276] Keep track of total requests number more actively

Author: liyintang

Summary: Previously, RS generates the total number of requests once the metrics has been pushed, which is usually 1 min interval. In order to get the fine-grained data in the master UI, RS shall keep track of the delta by itself.

Test Plan: tested on my dev clusters

Reviewers: adela

Reviewed By: adela

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D668267

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1426810 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-2376] Remove injection from compaction codepath

Author: pritam

Test Plan: Run the unit test.

Reviewers: kannan, liyintang

Reviewed By: kannan

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D669948

Task ID: 1944096

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1428557 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7491] testBatchedUpload test cases more deterministic

Author: rshroff

Summary:
Instead of randomly killing any region server, the test case now kills the
region server to which the next put operation is going to be performed.

It also fixes a bug in BatchedUpload code path, where it now updates the
map with the new location of the region incase it was moved.

Test Plan: Ran TestBatchedUpload test. MR Unit Test suite in progress

Reviewers: aaiyer, liyintang

Reviewed By: liyintang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D669544

Task ID: 1981445

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1429042 13f79535-47bb-0310-9956-ffa450edef68
[89-fb] [HBASE-7509] Regionserver support to control quorum reads

Author: aaiyer

Summary:
It will be good to have the ability to control the
paramaters for quorum reads while the regionserver is still running.

We want to control:
  1) the timeout that we wait for, before initiating the second read.
  2) number of threads allocated for quorum reads
     - setting this to 0 will disable quorum reads

 Depends on the quorum diff in HDFS to add the DFSClient calls.
https://phabricator.fb.com/D615354

 Will not commit this until alligator is updated. But, putting out
 for review together.

 Notes for Amit: (Will not be part of the commit)
         b398400e7b11 remove src/main/java/org/apache/hadoop/hbase/client/ParallelHDFSReads.java
         0ecb342447e2 lint errors
         4e609ef08613 add a ruby script to manage quorum reads
         d3e5cb243c06 control timeout as well
         20bd6254ad8a revert HMasterInterface
         7378e5e30541 rename function in setNumQuorumReadThreadsForHDFS -> setNumHDFSQuorumReadThreads
         cf443b6c65f7 regionserver support

Test Plan: deploy to dev cluster and set the params on the fly

Reviewers: liyintang, kannan, kranganathan

Reviewed By: liyintang

CC: hbase-eng@, sdong, hkuang, hdfs-dev@

Differential Revision: https://phabricator.fb.com/D635914

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1429973 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7511] Export preads latency metrics from HRegionServer metrics

Author: rshroff

Summary:
HBase already populates the preads latencies at HFile level but does not
export it via jmx. This change exports the fsPreadlatency metrics.

It also gets rid of the fsReadLatency metrics because these are not
populated as HBase now does only preads.

Test Plan:
Tested on hbase817.ash3 with custom build. After RS restart the metrics
were getting populated.

http://fburl.com/9226339 (check 3:45pm onwards)
http://fburl.com/9226350 (")

Reviewers: liyintang

Reviewed By: liyintang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D671587

Task ID: 1996587

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1432704 13f79535-47bb-0310-9956-ffa450edef68
[Master] Print locality and dispersion for the current assignment plan in a specific format

Author: adela

Summary:
added a method which will allow printing locality and
dispersion per table in the same format like in HBaseLocalityCheck

Test Plan:
Ran RegionPlacement -ld
tested on tsh25, sample output of 1 table:
13/01/03 15:10:47 INFO master.RegionPlacement: For Table: MailBox_cell5_v1 ; #Total Regions: 1760 ; The average locality for primary is 69.59913 %
13/01/03 15:10:47 INFO master.RegionPlacement: For Table: MailBox_cell5_v1 ; #Total Regions: 1760 ; The average locality for secondary is 4.718978 %
13/01/03 15:10:47 INFO master.RegionPlacement: For Table: MailBox_cell5_v1 ; #Total Regions: 1760 ; The average locality for tertiary is 4.8160424 %
13/01/03 15:10:47 INFO master.RegionPlacement: For Table: MailBox_cell5_v1 ; #Total Regions: 1760 ; The average dispersion score is 1.0

Reviewers: liyintang

Reviewed By: liyintang

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D670270

Task ID: 1979096

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1432705 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-6368] Remove use of deprecated guava functions

Author: amirshim

Summary:
It would be nice to be able to use Guava version 14, for some of the new classes, e.g. Range classes.  So first we need to make sure we don't use any of the deprecated functions.

Test Plan: Compile and run tests.

Reviewers: nspiegelberg

Reviewed By: nspiegelberg

CC: HBase Eng@

Differential Revision: https://phabricator.fb.com/D672447

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1432706 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7514] Fix the TestLogSplitOnMasterFailover.testWithDistributedLogSplittingAndErrors

Author: manukranthk

Summary: The Test is polling for the .META. table and failing after a certain number of retries with a PreemptiveFastFailException. Catching that exception in the Test since it only knows about RetriesExhaustedException and catches that.

Test Plan: Ensuring that the Unit Tests pass consistently.

Reviewers: liyintang, aaiyer

Reviewed By: aaiyer

CC: hbase-eng@

Differential Revision: https://phabricator.fb.com/D672827

Task ID: 1997870

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1432707 13f79535-47bb-0310-9956-ffa450edef68
[HBASE-7424] Enable the DeltaEncoding for the HFileOutputFormat

Author: manukranthk

Summary: HFileOutputFormat has a writer embedded but it is not configured to use the DeltaEncoding and FavoredNodes. This revision is to add that support to the HFileOutputFormat while it is used as an OutputFormat either the Mapper or the Reducer for a MapReduce task.

Test Plan: Adding Unit Tests to test the working of the HFileOutputFormat

Reviewers: liyintang, kannan

CC: hbase-eng@, astewart

Differential Revision: https://phabricator.fb.com/D669389

Task ID: 1964673

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89-fb@1432708 13f79535-47bb-0310-9956-ffa450edef68