Prevent unnecessary DNS lookups

Summary:
1. Do not use InetSocketAddress as the key for the cache: the constructor does DNS lookups.
   Just use a string of the form host:port as the key
2. CoronaJobTrakcer.getTaskTrackerClient should call
   TrackerClientCache.getClient, not TrackerClientCache.createClient

Test Plan:
unit tests

Revert Plan:

Reviewers: dms, aching, pyang

Reviewed By: dms

Task ID: 1148515
[raid] make the DistributedRaidFileSystem rename the parity file together with source file.

Summary: Codec should be initialized correctly.

Reviewers: weiyan, hkuang

Reviewed By: hkuang
Fix a bug in safe block counting for RBWs.

Summary:
There was a bug in the safe block count while processing RBWs.
If an RBW block was reported twice by the same datanode while the NN was
in safemode, the NN would incorrectly increment the safe block count
twice.

Test Plan:
1) Unit test added
2) All unit tests.

Revert Plan:

Reviewers: hkuang, tomasz, weiyan

Reviewed By: hkuang

Task ID: 1144182
DFSOutputStream.closeThreads now throws InterruptedIOException

Summary:
Changed the DFSOutputStream.closeThreads method to throw an
InterruptedIOException instead of an IOException.

Test Plan: By running "ant test-hdfs".

Reviewers: sdong, hkuang, pkhemani

Reviewed By: pkhemani

CC: pkhemani

Task ID: 1109322
Adding an ODS counter for FSCK operations

Summary:
Added in NamenodeMetrics a new TimeVaryingLongMetric
for each fsck() method call.

Test Plan:
Running "ant test-dfs" (still in progress).
Updating this section after more testing will be done

Reviewers: sdong, hkuang

Reviewed By: sdong

Task ID: 1152886
DFSClient doesn't retry block recovery on "block already committed exception"

Summary: Block recovery will never successful if a block is already committed. It's just a waste of time for clients to retry. By passing this information from namenodes all the way to clients, clients can avoid some retries.

Test Plan: ant test

Reviewers: hkuang, pkhemani

Reviewed By: hkuang

Task ID: 1135383
Add a raidshell command to raid a list of paths

Summary:
Add a command in this format:
-distRaid <raid_policy_name> <path1> ... <pathn>

that submit a map/reduce job to raid the given list of paths. A path could be a file or a directory. if it is a directory, only its children files are going to be raided.

Test Plan: Add a unit test in TestRaidShell

Reviewers: weiyan, dikang

Reviewed By: weiyan
HDFS: recounting INode count and disk space before exiting safe mode

Summary:
Standby Namenode's disk space quota is not accurate, as it misses addAdditionalBlock operation compared to primary. So the number needs to be recount before it becomes primary.

One thing I'm not sure is the cost of the method. Shall we complicate the codes a little bit to avoid one unnecessary recounting? Or we can do it as a follow-up.

Test Plan: Deploy in cluster.

Reviewers: hkuang, tomasz

Reviewed By: hkuang

Task ID: 1127167
Fix FastCopy bug regarding aligning datanodes for locality.

Summary:
FastCopy had a bug where it sorted both datanode lists for
locality. This would cause problems in cases where we had src = {b, a
  , c} and dst = {b, c, d }. The final two lists would be :

a, b, c and
b, c, d

instead of something like

b, c, a
b, c, d

So the copies would incorrectly be performed from a->b, b->c and c->d.

Test Plan:
Unit tests added.

Revert Plan:

Reviewers: hkuang, sdong

Reviewed By: hkuang
Attempts for a task should go to different threads.

Summary:
The assignment of task attempt -> worker thread in CoronaTaskLauncher
should be such that different attempts of the task go to different
threads.

Test Plan:
unit tests

Revert Plan:

Reviewers: dms, aching, pyang

Reviewed By: dms

Task ID: 1148515
Prevent CM -> Job Tracker timeouts.

Summary:
The cluster manager has a timeout on getting a response from the job
tracker on thrift calls. If the job tracker is doing something time
intensive and is not able to complete the thrift call, it can be timed
out.

This patch tries to prevent this situation from occuring. We add a
SessionDriverService.Iface implementation that simply queues up the
calls from the Cluster Manager during the thrift call. This will
guarantee quick response times to the cluster manager. There is a
separate thread to take the calls from the queue and execute them
in-order.

Keeping the calls in the same queue is important since that will reflect
the sequence of calls that the Cluster Manager makes.

Test Plan:
unit tests

Revert Plan:

Reviewers: dms, aching, pyang

Reviewed By: dms
NioServerSocketChannelFactory needs the maximum number of worker threads.

Summary: The worker thread factory had enough maximum threads, but netty assumes number of cores * 2.  This is likely not adequate for our clusters.

Test Plan: Passed all corona unittests.

Reviewers: dms, rvadali, pyang

Reviewed By: rvadali

CC: mattwkelly

Task ID: 1152668
Make FastCopy work with Avatar.

Summary:
FastCopy was broken even with regular Avatar (no failovers)
  since it relied on the uri from the filesystem interface to determine
  the namenode address. In case of Avatar this uri was the logical name
  and not the real physical address of the machine. I've changed
  FastCopy such that it works with AvatarNodes before and after a
  failover. FastCopy still doesn't work during a failover, but I feel
  this is not a necessity right now since FastCopy is used as part of MR
  jobs which can retry failed jobs during a failover and then correctly
  run FastCopy again after a failover is done.

Test Plan:
1) TestFastCopy*
2) TestAvatarFastCopy

Revert Plan:

Reviewers: hkuang, sdong

Reviewed By: hkuang

Task ID: 784627
Change exception type from Runtime to IOException in Standby

Summary:
It looks like in the testDoubleFailover, the rollFSImage is sometimes
succeeding, but since we are performing failover, the responder might
already by down. It's safer to thhrow IOException in the diff'ed line,
since this will not result in standby being shutdown.

Test Plan: ant test TestAvatarTxIds

Reviewers: hkuang, pritam

Reviewed By: pritam
Allow -distRaid command to raid all files in the subtree

Summary:
- distRaid raid all files in the subtree rooted at the given directory
- add a test case for this
- validate input paths

Test Plan: TestRaidShell

Reviewers: weiyan

Reviewed By: weiyan
For tests, reduce netty threadpool size

Summary: Use 10 threads per netty server. Otherwise we are getting OOM

Test Plan:
unit tests

Revert Plan:

Reviewers: dikang

Reviewed By: dikang
Fix build and use static constant

Summary: Fix build and use static constant

Test Plan:
ran MiniCoronaRunJob

Revert Plan:

Reviewers: dikang

Reviewed By: dikang
Fix bug in detecting dead job tracker.

Summary:
The job tracker reporter thread in the corona task tracker should
periodically attempt a heartbeat to the corona job tracker to detect
a dead job tracker. The job tracker reporter has logic to heartbeat
every 3 minutes even if there are no tasks running for that job, but
there is a bug in that logic which is fixed by this diff.

Also:
1. Set a name for the job tracker reporter thread. This will help in
   debugging.
1. prefix log messages from job tracker reporter with the reporter name.

Test Plan:
unit tests

Revert Plan:

Reviewers: dms, aching, pyang

Reviewed By: dms
Make bin/hadoop/job -killJob work

Summary:
Make CoronaJobTracker.killJob() handle job IDs that belong to some other
job. We can then use CoronaClient to kill the session.
While testing this I found that we always get a session ID when we
create a CoronaJobTracker object, even if all we do is kill another job.
So in this change, I also changed the logic so that getNewJobID() does
the work of getting a new session ID.

Test Plan:
ran unit tests, tested bin/hadoop/job -killJob on a sleep job

Revert Plan:

Reviewers: dms, aching, pyang

Reviewed By: dms

Task ID: 1140268
Remove synchronization from getListenerAddress.

Summary:
The primary avatar had a bug where it would enter a deadlock
since Server.waitForHandlers() is synchronized and would take the lock
and wait for all handlers and then a RPC handler could call
getListenerAddress() and wait for the lock on the Server object. Looks
like getListenerAddress() should not be synchronized since it does not
modify any state.

Test Plan: 1) All unit tests.

Reviewers: hkuang, weiyan, tomasz

Reviewed By: hkuang

Task ID: 1155907