Logging fix

Summary: Make resource limit messages debug level

Test Plan:
compile

Revert Plan:

Reviewers: dms, aching, pyang

Reviewed By: aching
Add recovery from NFS ingest failure

Summary:
I still observe at time that the edits.new cannot be seen by the
standby. This potentially creates a regression from the federation
branch.

In this patch, after upload of the checkpoint is done, and we see that
there is no ingest, we can instantiate ingest for edits -> since after
rolling the image edits.new was turned into edits. This basically will
mean that if we can instantiate edits.new after saving namespace, it's
going to help. If we fail there, we will do this after uploading. By
doing so, in the worst case we fallback to the federation behaviour.

-added check for instantiateIngest, in which if the file does not exist,
we don't change the state -> otherwise standby sets the state to
ingesting edits.new, ingest is not null, and standby thinks that ingest
is working, where in fact
it's not, and this is not a correct behavior -> ingest won't be
recreated until next checkpoint !!!

-ported retries for edits.new to trunk, since they were only committed
to alligator, increased the number of retries to 30

-added test case to simulate edits.new failure

-added some logging for failover, fixed some comment issues

Test Plan: all avatar tests

Reviewers: hkuang

Reviewed By: hkuang

Task ID: 1039092
Fix configured pools to fail jobs without pool settings and invalid pools, change default pool group to default from defaultpoolgroup

Test Plan:
passed all unittests
Verified changes on the local corona instance

Reviewers: rvadali, dms, pyang

Reviewed By: rvadali

Task ID: 1120186
Print speculation info

Summary:
When the task can be speculated it would be good to print the
information about why we are speculating it

Test Plan: I expect to see the message in the logs when I run it.

Reviewers: rvadali, aching, pyang
Fix memory reporting

Test Plan: running in prod already

Reviewers: rvadali, aching

Reviewed By: rvadali
Only add the pool stats to the all pools row.  Ignore pool group stats

Summary: Get rid of double counting.

Test Plan: local corona instance

Reviewers: dms, rvadali, pyang

Reviewed By: rvadali

Task ID: 1120910
Fix the failed OEV test

Summary: This diff fixed a bug where XMLTokenizer does not handle the offset of an operator.

Test Plan: TestOfflineEditsViewer

Reviewers: tomasz

Reviewed By: tomasz
Add a log message when datanode received an unexpected command

Summary: When ADN receives a command from standby that should come from the primary, log this event.

Test Plan: compile

Reviewers: tomasz

Reviewed By: tomasz

Task ID: 774638
recoverLease should return true if the file has no blocks

Summary: This makes recoverLease return true if the file is found to have no blocks and is finalized as part of the recoverLease call. Previously recoverLease would return false and the client would have to retry in such case.

Test Plan: Run on dev cluster with HBase workload

Reviewers: tomasz, hkuang

Reviewed By: hkuang

Task ID: 774638
[Raid Dir-Raid] BlockPlacementPolicyRaid will choose the correct stripe in Dir-Raid.

Summary:
In Dir-Raid, given a block, BlockPlacementPolicyRaid will choose the
stripe correctly.

Test Plan: ant test

Reviewers: weiyan, hkuang, rvadali, schen

Reviewed By: weiyan

Task ID: 1078029
[Raid Corona BugFix] Catch throwable in JobClient and RaidNode and include corona libs in the raidnode classpath

Summary:
After I push the corona release, raidnode fails to submit raid jobs but no error is
shown in the log. After investigating, I realize the problem happens at getClass
and a throwable is thrown (java.lang.NoClassDefFoundError: org/apache/thrift/TFieldIdEnum)
which is not caught by either JobClient and RaidNode

Include the Corona lib in the bin/hadoop and make JobClient and RaidNode catches throwable

Test Plan: deploy

Reviewers: dms, aching, pyang, rvadali

Reviewed By: rvadali
[Raid Dir-Raid] TriggerMonitor, PurgeMonitor changes and disable har

Summary:
PurgeMonitor scans the parity files and check if source directories or files exist
secondly, it checks if there are parity files for higher priority codes (we don't consider
a simulated code could delete a normal code even its priority is higher)
Modify trigger monitor to scan leaf directory.

Test Plan:
ant test
TestRaidPurge

Reviewers: dikang, hkuang

Reviewed By: hkuang

Task ID: 1078000, 1078264
Fix deadlock in node manager.

Summary:
We should not try to acquire a node lock while holding its node
container lock. This is because there are other code paths where the
node lock is taken first and then the node container lock.

Fix the deadlock by having a separate data structure to hold the
runnable nodes of a type. This removes the need for iterating over the
node containers.

Test Plan:
test cluster

Revert Plan:

Reviewers: dms, aching, pyang

Reviewed By: dms

Task ID: 1126283
Pool level aggregation of job counters

Summary: Pool level aggregation of job counters

Test Plan:
jconsole + single node setup

Revert Plan:

Reviewers: dms, aching, pyang

Reviewed By: dms

Task ID: 987904
Remove interrupting and joining the emptier

Summary:
We need to remove the interruption and joining of the emptier, it's
using the client, and we can't be sure if the interruption would not be
swallowed anywhere, and I have seen the failover hanging on emptier
join().

-we are cleanly shutting down rpc threads so this should be enough, and
the emptier thread is a daemon, which will be exited anyways on the JVM
exit.

Test Plan: deploy

Reviewers: hkuang, pritam

Reviewed By: hkuang
Make log lines that are not prefixed with LOG information debug only in PoolCalculator.

Test Plan: looked at local logs

Reviewers: rvadali, dms, pyang

Reviewed By: rvadali
Log node name for bad resources.

Summary: Log node name for bad resources.

Test Plan:
compile

Revert Plan:

Reviewers: dms, aching, pyang

Reviewed By: dms
Report free resources instead of used.

Summary:
Modify ClusterNodeInfo to specify free resources instead of used.

This helps simplify the reporting of free disk space when there are
multiple disks. We just report the largest free space on any one disk.

Test Plan:
unit tests.

Revert Plan:

Reviewers: dms, aching, pyang

Reviewed By: dms
DFSClient to reduce DNS check when determine whether an address is local

Summary: DFSClient to cache all known local addresses so future checking will be cheaper.

Test Plan: Add test testIsLocalAddress

Reviewers: hkuang, dhruba

Reviewed By: hkuang

Task ID: 1048754
[HDFS UI] Improve performance of dfsclusterhealth.jsp when connects to the namenode remotely and make it work for non-avatar federation cluster

Test Plan: manual test. it works for both non-avatar federation cluster and avatar federation cluster

Reviewers: hkuang, tomasz, sdong, pritam

Reviewed By: hkuang

Task ID: 840547