Inline Checksum (Part 1): Block read logic

Summary:
This is the first part of inline checksum patch. Refactor the blockSender logic to make it possible to allow two kind of logics.
BlockWithChecksumFileReader implements exactly the same logic as before. No behavior change. Codes were directly copied as much as possible.
BlockInlineChecksumReader implements the logic for inline checksum format. The codes are not enabled. So there should have not any function change after the patch.

Test Plan: ant test

Reviewers: hkuang

Reviewed By: hkuang

CC: mattwkelly, security-diffs@lists
Inline Checksum (Part 2): Block Write Logic

Summary:
This is the second part of inline checksum patch. Refactor the blockReceiver logic to make it possible to allow two kind of logics.
BlockWithChecksumFileWriter implements exactly the same block receiving logic as before. No behavior change. Codes were directly copied as much as possible.
BlockInlineChecksumWriter implements the logic for inline checksum format. The codes are not enabled. So there should have not any function change after the patch.

This is the version for trunk that removed src/hdfs/org/apache/hadoop/hdfs/server/datanode/BlockInlineChecksumWriter.java from the patch

Test Plan: N/A
HDFS: Data Transfer to allow 0 size block to be transferred if meta file has 0 size.

Summary: Now 0-size meta file is allowed if the block size is 0 in block recovery and client reading. This patch expands it to block scanner and block transfer

Test Plan: add TestTransferBlock.

Reviewers: hkuang, pritam, weiyan

Reviewed By: hkuang

Task ID: 1360138
Decouple failover logic from DistributedAvatarFileSystem.

Summary:
For protocols other than ClientProtocol, we need to use the
same failover logic. For example for the balancer we might need the same
piece of logic for NamenodeProtocol to do a failover. For this purpose
I'm decoupling the failover logic so that it can be reused easily.

I also removed some zookeeper watcher stuff that was not being used.

Test Plan:
All avatar unit tests.

Revert Plan:

Tags:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Reviewers: hkuang, weiyan, sdong, tomasz

Reviewed By: hkuang

Task ID: 1142725
Kill the JVM of the task reported done

Summary:
This fixes the problem of the JVMs that are running even after
the task itself is reported is done.
Specifically it was happening because there were threads that are
non-daemon and they keep on running even after the main() part of the
taks finishes.

This way we create a 'kill' command to be run X seconds later and kill
the JVM is the task is reported as done. We were already doing that but
only when the jvm shell execution exited, this also does it when the
java process is still running and not returning, but the task is
reported as done.

Test Plan:
I tested it manually in my single node setup. Having a job
that simulates the problem (launching threads that won't die after the
map task is done).

The Child processes were successfully destroyed.

Reviewers: aching, rvadali

Reviewed By: aching

Task ID: 1349122
Optimize rpc class/method name serialization

Summary:

I did some progiling of the RPC layer, and want to introduce some
performance fixes, especially for places when we write class and method
names, as well as members of some very frequent classes, which for sure
are ascii.

1) Writable class name for respones
2) Method name for invocations

3) If the return type is string, also the instance is speculated to be
ascii, and is written using the optimized way. This only affects places
where a rpc call returns Stringi or String[], NOT a complex object with String
members.

4) Couple of places that I observed introduce overhead when writing
responses (e.g., DatanodeID, DatanodeInof, BlocksWithLocations,
HdfsFileStatus) - in all those places we are writing ascii strings
(e.g., datanode ids), and we can do it much faster with this patch.

Alltogether this cuts several percent on average for rpc calls when I do
experiments in my dev server (including the call itself),
by cutting up to 50% the time to set up the reponse for clients.

I speculate only for strings shorter than 1000 bytes, which will
definitely catch all class names (with packages) and method names.

---------------------------------------------

Test Plan:

manually, added test case to make sure the optimized writes fall back to
normal writes for non-ascii characters.

Reviewers: hkuang
Fix race condition in Datanode refresh namenodes.

Summary:
This is similar to Pritam's fix for avatardatanode. This race condition
here makes the unit TestRefreshNamenodes fail sometimes. Please see
D543807.

Test Plan: ant test

Reviewers: hkuang, pritam, weiyan

Reviewed By: hkuang

Task ID: 1375482
Revert "Optimize rpc class/method name serialization"

This reverts commit 293e8227a621c79181194e1ad242e5dff8d2433b.
Namespace Notifier initial version

Summary:
Initial version of the namespace notifier, with most of the
functionality in place: reading from the edit log,
notifying one or multiple clients, detection of a failed
client by the server, detection of a failed server by a client
(the heartbeat system), saving notifications in the server history,
reconnecting by the client when a server comes back up.

Test Plan: Manual testing + unit tests for all the server side components.

Reviewers: sdong, hkuang, tomasz, jaxlaw

Reviewed By: sdong
Minor APITrace improvements.

Summary: Four changes: (1) do log collapsing for non-positional reads like we do for writes; (2) stop tracing getPos; (3) delete unused method, setStreamTracer(); and (4) trace PIDs.  The first two changes help us avoid tracing 100K's of messages for each MapReduce task.

Test Plan: Compiled classes for APITrace.java and APITraceFileSystem.java.  Added the resulting classes and inner classes to the distribution JAR for MRDEV.  Replaced hadoop-0.20-core.jar on a single datanode in MRDEV with my .jar and changed configs on that machine to use APITraceFileSystem.  Run hive queries that cause logs to be collected.

Reviewers: sdong, dhruba

Reviewed By: sdong
Make sure build_all.sh adds generated.pom to ops/common/hadoop-0.20

Summary:
To push the alligator jars to a maven repository we need to
have a generated.pom file alongwith it so that we can take care of all
dependencies as well.

Test Plan: Run build_all.sh and verify generated.pom is added.

Reviewers: mbautin, liyintang, avf, hkuang

Reviewed By: liyintang
Remove log clutter in Configuration

Summary: Yeah, not sure why this is here, but it clutters up the log with exception traces when it is not really an issue.

Test Plan: NA

Reviewers: aching, dms

Reviewed By: aching

CC: mattwkelly
Optimize rpc class/method name serialization

Summary:
Resubmitted D547333

This patch cuts the time for setting up rpc responses by up to 50%, but
optimizing the way the object names are serialized to the stream.

1) Class names for respones
2) method names for invokations
3) Class members that do not exceed 1000 chars, are speculated to be
ascii as well

Altogether this cust several percent on average for rpc calls when I do
experiments in my dev server.

Test Plan: ant test + manually

Reviewers: hkuang

Reviewed By: hkuang
Allow corona to run without pools file

Summary:
This handles the case when corona is configured with json and
no pools file.
We do need that sometimes since we are setting up one-off clusters with
no need for pools, but the corona cluster fails to start now.

Test Plan: Peter is setting up a cluster now that triggered the issue.

Reviewers: aching, rvadali, pyang

Reviewed By: aching

CC: pknowles
HDFS Hardlink Part-II

Summary:
In the second part of hardlink diff, it supports the DU operation, setModification time, setPermission, setUser, setGroup, setAccessTime and the quota verification for the hardlink files

Please the details about these operations at

Test Plan: TestFileHardLink and other unit tests

Reviewers: hkuang, sdong, pritam, kannan
Retrying zookeeper initialization for Avatar clients.

Summary:
At times due to DNS issues some clients may not be able to
initialize their zk connection due to UnknownHostExceptions. To
alleviate this problem I've added retries around this initialization.

Test Plan: All avatar unit tests.

Reviewers: hkuang, sdong, tomasz, weiyan

Reviewed By: hkuang

Task ID: 1374448
HDFS: pread local read to avoid close the same file and reopen it.

Summary:
I didn't try to generalize the patch as it looks like that we can't generalize it to satisfiy HBase's requirement without big efforts. So instead I narrow down the patch to only working with pread local read case.
I also manged to remove two setting variables in the patch.
I want to first hear from people whether they are comfortable with the approach.

Test Plan: Will add unit tests

Reviewers: hkuang, dhruba, weiyan

Reviewed By: hkuang
DFSClient Minor change: DFSInputStream's initial dead node size is set to be smaller

Summary: Usually there is no dead node through life cycle when reading a file. Set the initial size to be as small as possible.

Test Plan: N/A

Reviewers: weiyan, hkuang

Reviewed By: weiyan
DFSClient: Change from NetUtils.isLocalAddress() to cached version in two places

Summary: Two more places need to change from NetUtils.isLocalAddress() to cached version for performance

Test Plan: N/A

Reviewers: weiyan, hkuang

Reviewed By: hkuang
Make standalone JT configurable

Summary:
Right now the configuration that is used to launch a remote
jobtracker is set cluster-wide. This is undesirable since we have large
jobs that need more memory and potentially other command line things
that we want to configure per job.

This patch makes it so that the configuration of the job is applied to
the configuration object used when constructing the
CoronaJobTrackerRunner

Test Plan: Just tested in the single node hadoop

Reviewers: rvadali, aching

Reviewed By: rvadali

Task ID: 1405185