Have NSFRE messages also display the row associated with the region request.

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181552 13f79535-47bb-0310-9956-ffa450edef68
Ability to Discard Bad HTable Puts

Summary:
While debugging an application consistency issue, we noticed
that a single, synchronous Put request threw a
NoServerForRegionException but eventually succeeded 90 seconds later.
The problem is that failed put requests are not actually removed from
the HTable's writeBuffer. This makes sense for asynchronous puts using
setAutoFlush(false) but don't make sense for the default case where we
expect synchronous operation. We should discard all failed puts for the
synchronous case and provide an API so asynchronous requests can have
their failed puts cleared.

Test Plan:
- test with appserver

Reviewed By: kannan
Reviewers: kannan, jgray
CC: , kannan, nspiegelberg, hbase@lists, pkhemani
Differential Revision: 264990

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181553 13f79535-47bb-0310-9956-ffa450edef68
Unified block format for the HFile format version 2

Summary:
In the HFile format version 2 we use unified-format blocks for almost
everything stored within an HFile. A block consists of an 8-bit "magic"
signature identifying the block type, 4-byte compressed size, 4-byte
uncompressed size, and 8-byte offset of the previous block of the same type in
the file, allowing to seek to the previous block efficiently without using
block
index. This diff contains the infrastructure necessary for writing and reading
these new-format blocks in both compressed and uncompressed mode. I have tried
to minimize read operations and fall back to two reads (header + data part) if
the on-disk block size is unknown, but to improve this further we could e.g.
always read the header of the next block just in case, as Karthik suggested.
Also, I have tried to minimize byte array copy operations, even though they are
sometimes necessary to achieve abstraction through writing to an in-memory data
stream. I am planning to profile the complete system before trying to further
refine this part.

Test Plan:
Unit test. Load test as part of larger HFile format version 2 testing.

Reviewed By: kannan
Reviewers: kannan, liyintang, kranganathan, gqchen, aaiyer, jgray
Commenters: liyintang, nspiegelberg, kranganathan, jgray
CC: hbase@lists, , liyintang, nspiegelberg, kranganathan,
mbautin, jgray, kannan, kenny
Revert Plan:
These are new files so they can be checked in safely. However, the whole HFIle
format v2 change is non-reverse-compatible.

Differential Revision: 250060

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181554 13f79535-47bb-0310-9956-ffa450edef68
HFile format version 2: HFile reader/writer hierarchy, and an updated fixed file trailer

Summary:
With the introduction of HFile format version 2, we still want to support
reading the old format for the purpose of automatic migration. To better ensure
backwards-compatibility, we keep the version 1 writer as well, and utilize
existing unit tests to ensure our version 1 reader/writer implementations are
consistent. The version 1 reader/writer are updated to significantly reuse
version 2's functionality, and the block cache always contains version 2 blocks.
As a result, we get a natural class hierarchy of HFile readers and writers for
the two versions, and break the original monolithic HFile class into a class
hierarchy. HFile block index has been moved out of HFile into its own class, which will be
submitted separately. The new FixedFileTrailer implementation supports both v1
and v2 trailers, and is aware of the fields that are valid in each version.

Test Plan:
Unit tests. Load testing using HBaseTest. New backwards-compatibility unit
tests will be added. Will test in dark launch.

Reviewed By: nspiegelberg
Reviewers: kannan, liyintang, kranganathan, gqchen, aaiyer, nspiegelberg, jgray
Commenters: jgray, kannan
CC: hbase@lists, , mbautin, jgray, kenny, kannan, nspiegelberg
Revert Plan:
The HFile format v2 is a backwards-compatible, but non-reverse-compatible
feature, and has to be tested thoroughly even before being deployed to dark
launch.

Differential Revision: 251875

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181555 13f79535-47bb-0310-9956-ffa450edef68
Multi-level block index

Summary:
This is a version of Liyin's original implementation of multi-level block
indexes with a significant amount of refactoring, documentation, and bug fixes
to the point that it can run smoothly in HBaseTest for at least a few days. I
also added a unit test, which currently only tests one- and two-level block
indexes. There is a discussion of multi-level block indexes at
https://issues.apache.org/jira/browse/HBASE-3857, and the relevant JIRA these
changes will be part of is https://issues.apache.org/jira/browse/HBASE-3856.

Test Plan:
Unit tests. Load test (HBaseTest). Dark launch.

Reviewed By: kannan
Reviewers: liyintang, kranganathan, kannan, nspiegelberg, gqchen, aaiyer
Commenters: jgray, liyintang
CC: hbase@lists, , mbautin, jgray, kenny, liyintang, kannan
Revert Plan:
The new block index implementation is capable of reading version 1 HFile block
indexes and does that as part of HFile reader v1 in D251875. Howver, multi-level
block indexes introduce non-reverse-compatible changes to the block index
format.

Differential Revision: 251949

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181556 13f79535-47bb-0310-9956-ffa450edef68
Compound Bloom filters

Summary:
This is a port of D237675 to be consistent with other HFile format version 2
changes. Instead of breaking an existing ByteBloomFilter's bit array into
blocks, we accumulate keys into a Bloom filter chunk as an HFile is being
written, and flush this Bloom filter to disk as soon as a fixed number of keys
is reached. At read time, the appropriate Bloom filter block is determined using
binary search on the key, loaded, cached, and queried.

New configuration options:
io.storefile.bloom.compound - a boolean option specifying whether compound
Bloom filters are turned on
io.storefile.bloom.block.size - the target size of a single Bloom filter chunk
(only takes effect when io.storefile.bloom.compound=true)
io.storefile.bloom.cacheonwrite - allows to enable cache-on-write for Bloom
chunks

The TestCompoundBloomFilter unit test validates the false positive rate to
ensure it closely matches what is expected. Compound Bloom filters do not rely
on an estimate of how many keys will be added to the Bloom filter, so they can
hit the target false positive rate much more precisely. There are still reasons
for the discrepancy between the theoretical and observed false positive rate,
and they are controlled for in the unit test:
- Previously, the number of bits in the Bloom was computed as if the number of
hash functions could be an arbitrary number, while it must be an integer. In
this implementation, the number of bits is adjusted to ensure the given target
error rate as soon as the precise (rounded) number of hash functions is known.
- The last chunk of a compound Bloom filter may be undersized, even if it
undergoes folding, so the observed error rate may be lower than expected.
Therefore, the last chunk of a multi-chunk compound Bloom filter is excluded
when checking the lower bound of the false positive rate.
- We use combinatorial hashing, which gives a slightly higher false positive
rate than a set of truly independent hash functions (thanks to Nicolas for
pointing this out). However, we can test the false positive rate as though we
had independent hash functions, by looking up random bits instead of the bits
the hash functions point to. This technique is referred to as "fake lookups" in
the code and is obviously only valid for the unit test. That said, it does yield
the false positive rate that is closer to the expectation than that of our
production Bloom filters.

While most of the Bloom filter logic is kept out of HFile, there is only one
version of StoreFile (i.e. there is no hierarchy of StoreFile implementations
similar to that introduced for StoreFile in D251875). Therefore, HFile needs to
know where to store Bloom filter metadata, even though it does not need to know
anything about how that metadata is structured. In version 2, we store Bloom
filter metadata as part of the "load-on-open" section of the HFile.

Finally, Bloom filter code has been split into a writer interface and Bloom
filter itself (which one might call reader) to avoid keeping around fields and
code that is only used when writing at read time. Note
that CompoundBloomFilterWriter implements InlineBlockWriter, an interface
introduced in D250060 for writing "inline" blocks into the data section of a
version 2 HFile.

Test Plan:
Unit tests. Load test (HBaseTest).
Still need to improve HBaseTest to allow testing ROWCOL Bloom filters.

Reviewed By: nspiegelberg
Reviewers: nspiegelberg, kannan, kranganathan, gqchen, jgray, aaiyer
Commenters: jgray
CC: hbase-hdfs@lists, , mbautin, jgray, nspiegelberg, kenny
Revert Plan:
HFile format version 2 introduces backwards-compatible but
non-reverse-compatible changes.

Differential Revision: 251932

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181557 13f79535-47bb-0310-9956-ffa450edef68
"IdLock" synchronization primitive that will be used to synchronize on block offsets in HFileReaderV2

Summary:
HFile block read operations are currently synchronized on the block index key
object for the particular block. This is done so that only one client goes to
HDFS to read the block and caches it, and all other clients interested in the
block pick it up from the cache. However, with HFile format v2, the data block
index might be multi-level, so there is not a canonical object to use for
synchronizing clients. The IdLock synchronization primitive allows to
synchronize on arbitrary long numbers, creating temporary objects for them as
necessary.

Test Plan:
New unit test. Load testing as part of HFile format v2 testing.

Reviewed By: kannan
Reviewers: kannan, kranganathan, liyintang
Commenters: nspiegelberg, jgray
CC: , hbase@lists, kannan, nspiegelberg, jgray
Revert Plan:
OK

Differential Revision: 248448

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181558 13f79535-47bb-0310-9956-ffa450edef68
Unit test changes, HFile reader/writer API call changes, and utility functions for HFile format version 2

Summary:
This is the sixth diff for the HFile version 2 / compound Bloom filters /
multi-level block indexes project. It bundles the following types of changes:
- API calls to create HFile readers/writes now have to go through the
appropriate factory methods.
- Unit test changes to enable testing both HFile format version 1 and 2. The
set of tests to be made parameterized was selected in an ad-hoc way during
development. Certain unit tests had to be converted to JUnit 4, but still extend
HBaseTestCase, to minimize code changes. Now that the default HFile format for
this feature has been flipped to version 2, we might be able to avoid a lot of
these unit test changes by running two rounds of test with the default HFile
format version specified through e.g. a system property. Also,
testHeapSizeForBlockIndex was moved from TestHFile to block index's own unit
test, TestHFileBlockIndex.
- New utility functions required for other parts of HFile v2 functionality
(e.g. changes to Bytes and TestBytes).
- A fix for a typo in Hash.java javadoc.

Test Plan:
Unit tests. Load testing with HBase test. Will push to dark launch along with
other HFile format v2 changes.

Reviewed By: nspiegelberg
Reviewers: kannan, liyintang, nspiegelberg, kranganathan, gqchen, aaiyer
CC: hbase@lists, , mbautin, nspiegelberg
Revert Plan:
HFile format v2 changes are backwards-compatible but not reverse-compatible.

Differential Revision: 252110

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181559 13f79535-47bb-0310-9956-ffa450edef68
Removing DynamicByteBloomFilter. This should have been part of D251932.

Summary:
See D251932 for details. DynamicByteBloomFilter is being removed as
part of HFile format v2 changes, as we are transitioning to compound
Bloom filters. Committing this without a separate Differential
revision and review, because this was already reviewed as part of
D251932.

Test Plan: Unit tests, dark launch.
Reviewed By: nspiegelberg
Reviewers: nspiegelberg, kannan, kranganathan, gqchen, jgray, aaiyer
Revert plan:
OK
Commenters: jgray
CC: hbase-hdfs@lists, , mbautin, jgray, nspiegelberg,
Revert Plan:
HFile format version 2 introduces backwards-compatible but
non-reverse-compatible changes.

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181560 13f79535-47bb-0310-9956-ffa450edef68
Fix RegionServerDynamicMetrics to push NumOps metrics for every interval

Summary:
RegionServerDynamicMetrics uses the MBean
RegionServerDynamicStatistics to push out metrics. This MBean did not
have the functionality of dynamically updating the metrics list it uses.

Test Plan:
Verify metrics in dev cluster

Reviewed By: kannan
Reviewers: kannan, nspiegelberg, dhruba, hkuang, dms
Commenters: nspiegelberg, dhruba
CC: lshepard, mattwkelly, brent, pritam, kannan, ,
hbase@lists, nspiegelberg, dhruba
Revert Plan:
Tags:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Differential Revision: 255866

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181561 13f79535-47bb-0310-9956-ffa450edef68
Adding limit / row / CF clause to Get (and Scan)

Summary:
Use setMaxResultsPerColumnFamily to limit the number of values returned by Get
per CF.

Test Plan:
Added unit tests.

Reviewed By: kannan
Reviewers: kannan, gqchen, jfan
Commenters: gqchen, jfan
CC: madhuvaidya, kannan, gqchen, liyintang, jfan, hbase@lists, anirudht
Differential Revision: 250900

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181562 13f79535-47bb-0310-9956-ffa450edef68
HBASE-3987: Fix a Bloom Filter NPE on a failure to load Bloom filter data

Summary:
This is a fix for an NPE that happens in passesBloomFilter. The meta block
fails to load, and the catch block on lines 1066-1068 sets the Bloom filter to
null. Then all other threads waiting on the Bloom filter to load get a chance to
try to load the meta block, and one of them eventually succeeds and proceeds to
query the Bloom filter in StoreFile.passesBloomFilter, but bloomFilter has been
already set to null.

The fix (Nicolas's idea) is to cache the bloomFilter variable as a local
variable in passesBloomFilter.

A similar fix for the "earth" branch is D266546.

Test Plan:
Unit test (TestStoreFile)

Reviewed By: kannan
Reviewers: nspiegelberg, kannan
CC: hbase@lists, , kannan
Revert Plan:
OK

Differential Revision: 266571

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181563 13f79535-47bb-0310-9956-ffa450edef68
Fix major compaction bug for a table at a per column family level.

Summary:
Major compaction at a per column family level for a table was being
incorrectly downgraded to a minor compaction. This is a longer term fix.

Test Plan:
1) Test in dev cluster.
2) Run relevant tests in TestFromClientSide.

Reviewed By: kannan
Reviewers: kannan
CC: kannan, hbase@lists
Revert Plan:
Tags:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Differential Revision: 267161

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181564 13f79535-47bb-0310-9956-ffa450edef68
Enhance region_status.rb to report status of a single table.

Summary:
For the purpose of Snapshot/Backup we would wish to wait for
all regions of a table to come up before we verify the backed up table.
For this purpose the region_status.rb script needs to be modified. This
functionality should be of great use in other places as well.

Test Plan:
1) Test on dev cluster.

Reviewed By: nspiegelberg
Reviewers: nspiegelberg, madhuvaidya, aaiyer
CC: hbase@lists, , nspiegelberg
Revert Plan:
Tags:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Differential Revision: 268472

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181565 13f79535-47bb-0310-9956-ffa450edef68
add BloomFilter and timestamprange for bulk import

Summary:
add BloomFilter and timestamprange for bulk import.

This is the version updated for hfile v2.

Test Plan:
unit tests.

Reviewed By: kannan
Reviewers: nspiegelberg, kannan, kranganathan, mbautin
Commenters: kranganathan, nspiegelberg, mbautin
CC: hbase@lists, kranganathan, nspiegelberg, gqchen, ,
mbautin, kannan
Differential Revision: 237110

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181566 13f79535-47bb-0310-9956-ffa450edef68
Correctly specifying pread and isCompaction in block read API calls

Summary:
This fixes incorrectly specified pread (positional-read) and isCompaction
(determining whether compaction or regular counters will be incremented)
parameters when calling block reader API functions. These bugs are specific to
HFile format v2, and were found during compaction workload testing with the
HFileReadWriteTest tool (266598).

Also refactoring HFile scanner hierarchy to prevent accidental modifications of
pread and isCompaction flags.

Test Plan:
Run the HFileReadWriteTest tool (266598), and make sure that the number of
preads is 0 and the number of seek + read operations per second is the same as
the number of blocks read per second, as reported by the tool.

Reviewed By: kannan
Reviewers: kannan, liyintang, pritam
CC: hbase@lists, , kannan, mbautin
Revert Plan:
OK

Differential Revision: 267302

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181567 13f79535-47bb-0310-9956-ffa450edef68
Adding RowMutation for being able to do both Delete and Put in MR jobs

Summary:
I have added a union-style class called RowMutation to encapsulate either a
Delete or a Put in MapReduce jobs.
This is going to be useful in gathering up mutations for batch import jobs that
also require deletes to be made available.

I have also added a matching RowMutationSortReducer modified from the initial
PutSortReducer so as to know how to sort these elements in a MR job.

Finally, I added a line for selecting the proper reducer in the
HFileOutputFormatter, due to the rather hacky nature of checking for instances
of input...

Test Plan:
I've implemented a small unit test which creates a MR job with a mapper that
statically outputs a Put and a Delete. The new reducer properly sorts them and
outputs them and the test wil dump to a file using HFileOutputFormat.

Reviewed By: kannan
Reviewers: kannan, jfan, nspiegelberg
Commenters: nspiegelberg, gqchen
CC: kannan, nspiegelberg, gqchen, bogdan
Differential Revision: 267252

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181568 13f79535-47bb-0310-9956-ffa450edef68
A unit test to read a pre-existing version 1 binary HFile

Summary:
To make sure the version of HBase with HFile v2 changes stays compatible with
the legacy HFile format, we need to have a pre-existing binary HFile in the v1
format and a unit test that reads that binary file using the new HFile v1 reader
code.

Test Plan:
Run the unit test.

Reviewed By: kannan
Reviewers: kannan, gqchen
CC: hbase@lists, , kannan
Revert Plan:
OK

Differential Revision: 269117

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181569 13f79535-47bb-0310-9956-ffa450edef68
Assign regions based on block locality and print out the region locality

Summary:
reate a command line tools to print out the region locality info
Example: running this command line in my dev cluster: 210
====================
 ./bin/hbase localityck

Host :230. has 13 / 13 blocks for the test1 :
2d59bb40ffba29ea04fe11b27d997646
Host :250. has 12 / 12 blocks for the test1 :
c60bd76550b9c6d5348740f4a41fc9bf
Host :230. has 12 / 12 blocks for the test1 :
6fb75bb06213d19171abf816791b2423
Host :230. has 4 / 4 blocks for the -ROOT- : 70236052
Host :230. has 12 / 12 blocks for the test1 :
01c048a618024b38a5f0192614a9660a
Host :230. has 12 / 12 blocks for the test1 :
54dd7d26a69c9474f98f54cbdc35cb98
Host :290. has 4 / 4 blocks for the .META. :
1028785192
Host :230. has 12 / 12 blocks for the test1 :
36dbc39af1465227cb6118fb17da4a20
Host :230. has 12 / 12 blocks for the test1 :
4d3958e8ad30bcbe39c037ceb94f3c10
Host :230. has 12 / 12 blocks for the test1 :
ced427fa96da9eb3cd9c351ca074a208
Host :230. has 3 / 3 blocks for the test2 :
a638e6d34311e2995154679dea73bb8c
Host :230. has 12 / 12 blocks for the test1 :
5035d560a22b2a65389c4b6acde9cbf8
Host :230. has 12 / 12 blocks for the test1 :
bd55c6af492a684adc573380267bf188
Host :250. has 12 / 12 blocks for the test1 :
59534b14cff211e712151be628445aec
Host :250. has 12 / 12 blocks for the test1 :
9ec644b196af2d551b3decfc7409ef9b
Host :230. has 12 / 12 blocks for the test1 :
0e9416af3be05fb3844d4cac0c4b5198
Host :250. has 12 / 12 blocks for the test1 :
59865cdd283215cc2250423d2584d034
Host :250. has 12 / 12 blocks for the test1 :
e511fb3efa475866d7b27ac17aea81e4

Test Plan:
It has been tested on dev cluster. I will continue working on more test cases
to stablize the code.

Reviewed By: kannan
Reviewers: kannan, kranganathan, gqchen, nspiegelberg, bogdan
Commenters: dhruba, mbautin
CC: , hbase@lists, kannan, dhruba, liyintang, mbautin, achao
Differential Revision: 257279

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181570 13f79535-47bb-0310-9956-ffa450edef68
Restore interoperability with old clients/servers after HFile v2 changes

Summary:
HFile v2 changes added three new region load metrics (rootIndexSizeKB,
totalStaticIndexSizeKB, and totalStaticBloomSizeKB) and resulted in a client
that was incompatible with the old server, and a server that was incompatible
with the old client, because the serialization format of HServerLoad changed. I
am now removing these new fields from the serialized format. They will still be
accessible through the region server web UI, but won't show up in the output of
the "status 'detailed'" command in the hbase shell.

Thanks to Madhu for catching this bug.

Test Plan:
Tested on a five-node cluster. Still need to do tests with real old clients and
servers.

Reviewed By: kannan
Reviewers: madhuvaidya, kannan
CC: hbase@lists, , kannan
Revert Plan:
OK. This actually fixes a backwards-compatibility issue.

Differential Revision: 270343

git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.89@1181571 13f79535-47bb-0310-9956-ffa450edef68