HDFS: Backward compabile SendBlock constructor

Summary: Previous patch "Inline Checksum (Part 1): Block read logic" changed constructor of SendBlock called by BlockReconstructor. This change compromises old interface so it can be two way compatible. Will remove it later.

Test Plan: N/A

Reviewers: weiyan, hkuang

Reviewed By: weiyan
Command line option to start avatarnodes with NULL zookeeper.

Summary:
At times even when the zookeeper node is NULL, we need the
functionality to still start the primary/standby avatar. One usecase is
while upgrading small HBase clusters instead of failing over to a
standby we can simply restart the primary and do a rolling upgrade.

Test Plan:
1) All avatar unit tests
2) Unit tests added.

Reviewers: tomasz, hkuang

Reviewed By: tomasz
Add setReuseAddress to free ports in MiniAvatarCluster

Summary:
I found that this should help in situations where we restart nodes,
where sockets can be in TIME_WAIT state.

Example:

http://svn.apache.org/repos/asf/directory/sandbox/jvermillard/mina/java/org/apache/mina/util/AvailablePortFinder.java

Test Plan: ant test

Reviewers: pritam

Reviewed By: pritam
Fix TestStandbySafeMode.

Summary:
TestStandbySafeMode had a bug where it didn't wait for
clearPrimary but instead directly asserted that after failover
ClearPrimary should have been processed. We need it to wait for a while
before asserting this.

Test Plan: Run the unit test

Reviewers: tomasz, hkuang

Reviewed By: tomasz
HDFS DataNode: avoid two string concatenation when ClientTraceLog is not enabled

Summary:
Currently, we always calculate localAddress and remoteAddress even if ClientTraceLog is not enabled. Both of them are string concatenations. Remove them when ClientTraceLog is not enabled. Another place uses remoteAddress is thread name. Replace that with just addresses without ports. I think that's good enough for thread name.

The performance improvement is minor but it won't be worse.

Test Plan: N/A

Reviewers: weiyan, hkuang

Reviewed By: hkuang
Allow Namenode to exclude/deprioritize datanodes which do not heartbeat with the namenode

Test Plan:
added a unit test.

push to dev cluster and see the number of failures reduce during rack failure test.

Reviewers: kannan, hkuang

Reviewed By: hkuang
Do not update replication queues in removeStoredBlock when replication queues are not initialized.

Summary:
There was a bug that the Standby hit because even though the
Standby never intializes replication queues, it was incorrectly adding
blocks to its under replication queue. This caused a continuously
increasing missing block count on the Standby.

Test Plan: All unit tests.

Reviewers: hkuang, tomasz, weiyan, sdong

Reviewed By: tomasz

Task ID: 1742688
Add retries for connecting zk

Summary:
-This patch adds retries when talking to zk for test cases.
-changes the max client connections to 500

Test Plan:
running multiple runs of avatara tests.
Now I didn't see any restarts caused by zookeeper, but still see
problems with binding ports

Reviewers: pritam

Reviewed By: pritam
BlockDecompressorStream not to return -1 on IOExeption until EOFException

Summary:
Currently BlockDecompressorStream.read() return -1 on whatever IOException thrown by HDFS, which is wrong.
This patch changes the bahavior. (This behavior is introduced since this class is introduced when LZO is added to Hadoop).

Test Plan: run all lzo tests.

Reviewers: hkuang, aching, weiyan, lif

Reviewed By: lif

CC: mattwkelly, guanghao, sambavim, kevinwilfong, gang, bmandhani
Make sure remote job tracker exits the proces

Summary: Call System.exit() explicitly.

Test Plan: deploy

Reviewers: aching

Reviewed By: aching
Fix build error
Fix problem with upgrade

Summary:
I found that when upgrading old image to the new layout, if the edits
are empty, we have conflict with the loaded and saved image digest,
since the saved image is different (e.g., layout version). During
upgrade we need to clean the digest, so it doesn't conflict.

Test Plan: Manually

Reviewers: pritam, hkuang

Reviewed By: hkuang
Gracefully recovery from runtime exceptions in logSync

Summary:
Previously when a runtime exception (e.g., NullPointerException)
occured in FSEditLog.logSync(bool), notifyAll() on this object would
not be called, leading to deadlock under certain conditions brought up
in BookKeeperJournalManager testing.

This revision puts notifyAll() in a finally block and also adds
additional logging when editLogStream is set to NULL.

Test Plan:
Run all unit tests for the core and highavailability modules

Revert Plan:

Reviewers: tomasz, hkuang

Reviewed By: tomasz

Task ID: 883632
savedLocalDirs should be null, not ""

Summary: This caused a null localDirs and all tasks to fail

Test Plan: tried on mgraph in prod

Reviewers: rvadali, stupaq

Reviewed By: stupaq

CC: digraph@, mattwkelly

Task ID: 1758538
Send IP address as part of resource grant

Summary:
When CoronaTaskTracker create appInfo, it will use the local host IPaddress instead of name.
TrackerClientCache will check if it is ipaddress. If yes, it will create InetAddress from the
ipaddress. Otherwise, it will use hostname.

Test Plan: Test with the sleep sample test

Reviewers: rvadali

Reviewed By: rvadali

Task ID: 1741421
Improve task launching logic in corona job tracker

Summary:
CoronataskLauncher will have a pool of workqueues, which are based on TaskTrackers. Each thread
will service a single queue at a time.

Test Plan:
Tested with the sleep test, TestMiniCoronaRunJob, and some unit test cases under
VENDOR/hadoop-0.20/src/contrib/corona/src/test/org/apache/hadoop

Reviewers: rvadali

Reviewed By: rvadali

Task ID: 1447990
Enhance image/edits config validator for supporting uris

Summary:
It seems there was a problem with the edits/image pathg validation when
we were specifying uris in configuration. This patch ensures that the
uris are converted to names.

In the future, we will have to add some new logic for handling
bookkeeper or other shared devices.

Test Plan: amended test cases

Reviewers: hkuang, pritam, avf
Fix saveVersion for svn checkouts

Summary:
It seems that no svn info fails, because it looks for .svn directory in
hadoop-0.20, which is placed two directories up in the trunk.

This patch should fix it, it runs svn info, and if this fails it assumes
that it's a git-svn repo.

Test Plan: deploy

Reviewers: pritam, hkuang

Reviewed By: pritam
Revert "Enhance image/edits config validator for supporting uris"

This reverts commit 9ce913101a5d8ebae04a3474d449236705761caf.
Enhance image/edits config validator for supporting uris

Summary:
It seems there was a problem with the edits/image pathg validation when
we were specifying uris in configuration. This patch ensures that the
uris are converted to names.

In the future, we will have to add some new logic for handling
bookkeeper or other shared devices.

Test Plan: amended test cases

Reviewers: hkuang, pritam, avf