Parse "True" and "1" as "true", "False" and "0" as "false"

Summary: Parse "True" and "1" as "true", "False" and "0" as "false" in Configuration.getBoolean.

Test Plan: Unit tests

Reviewers: pritam, tjackson, hkuang

Reviewed By: hkuang
Simplify standby backup logic

Summary:
I think that the best model for standby backup is when we keep up to
maximum number of backups, and we simply not allow for the the standby
to start if there is more backups. This will force manual intervention,
but will prevent the standby from starting up over and over again.

I think this is the best solution.

Test Plan:
added tests which ensure that the retentions manager is doing its job
properly

Reviewers: hkuang, pritam

Reviewed By: hkuang
Disable standby backup limits for tests

Summary:
Disabling standby limits so the test do not interfere with each other,
between test cases.

Test Plan:
ant test

ant test

Reviewers: pritam, hkuang

Reviewed By: pritam
TestFileStatusExtended failure

Summary:
It turns out that in about 2-3% cases the way we do sampling is going to
return less than 5 out of 100 files with threshold 0.1.
I increase the number of files to 200

Test Plan: manually tested probabilities in java

Reviewers: pritam

Reviewed By: pritam
Fix Date Time Format

Summary:
Fix the date format in corona Web UIs so that the dates are
shown as yyyy/MM/dd HH:mm:ss like in cm.jsp and remove superfluous
imports.

Test Plan:
checked each endpoint while running a job and after for
history endpoints to make sure the date format is consistent and changed
to yyyy/MM/dd HH:mm:ss

Reviewers: rvadali

Reviewed By: rvadali

Task ID: 1792844
add seqNo between Corona cluster manager and CoronaJobTracker

Summary:
seqNos are added to cluster manager and corona jobtracker transactions. heatbeat from JT will not change the seqNo. CM will
save the seqNo from JT, and expect that will be the next value from JT for any transaction other than heartbeta. if the seqNo
doens't match, CM will kill the JT. JT will save the seqNo from CM, and expect that will be the next value from CM. If the seqNo
doesn't match, JT will kill itself.

Test Plan:
the sleep sample(safeMode and normal mode), TestSessionDriver, TestLoadManager, TestClusterManager, TestPreemption,
TestResourceCheck

Reviewers: rvadali, aching

Reviewed By: rvadali

Task ID: 1794271
Another source of mismatched generation stamp replicas

Summary: Looks that NN accepts a replica which has higher generation stamp than the one in memory if it is not the last block of an under-construction file. This diff changes the behavior to reject such a replica. Another option to make datanode to replicate and provision a replica with higher generation stamp.

Test Plan: TestCorruptBlocks

Reviewers: tomasz, sdong

Reviewed By: tomasz

Task ID: 1783644
Fix OfferService ZK interaction

Summary:
1) check if failover happened only one in checkFailover and mode
DNA_REGISTER check to proces commands

2) more serious !isnotValidStandbyCommand - for every command on the
primary, which is not a valid standby command we talk to ZK. We need to
check the cached value, instead of talking to zk.

Test Plan: ant test

Reviewers: pritam

Reviewed By: pritam
Fix bug where datanode couldn't register and continuously sent block reports

Summary:
I saw an issue where a datanode couldn't process the
DNA_REGISTER command and continuously failed registering and as a result
sent Incremental Block Reports at a very high rate to the NameNode and
the incremental block report would fail each time due to the datanode
not being registered. In such a case, now this patch restarts the
offerservices.

Test Plan: All avatar unit tests.

Reviewers: sdong, tomasz, hkuang

Reviewed By: tomasz

Task ID: 1817915
Inline checksums - fix problem with block report

Summary:
I discovered that with inline checksums, the block reports will crash
the offer service whenever there are blocks scheduled for deletion.
Block.isInlineChecksumBlockFilename() was returning TRUE for
toDelete.blk_... blocks - scheduled for deletion, and then parsing would
crash of course. This is serious, since every time we have block
scheduled for deletion and a block report, this would kill OfferService
- also on startup it would be the same

This patch adds a variety of testcases to assert the filenames, whether
they are returning correct values for various names: old format, new
format, toDelete blocks, etc. Also it checks wheteher a directory
containing new/old/toDelete/sth else is listed correctly for block
reports

Test Plan: added test case

Reviewers: hkuang, sdong, pritam

Reviewed By: sdong
Make the injections to be the last statements of doCheckpoint

Summary:
TestAvatarCheckpointingFailures failed because the testcase was notified
earlier than the checkpoint failed than handleCheckpointFailure function
was called. This diff makes the injections to be the last statements of
checkpoint to make sure all actions have already taken place when the
checkpoint notifies testing handlers that it completed.

Make the runtime volatile in FSEditLog as well to make sure it's not
cached across threads.

Test Plan: ant test TestAvatarCheckpointingFailures

Reviewers: hkuang, pritam

Reviewed By: hkuang
Fix TestAvatarCleanShutdown conflict with host file name

Summary:
When running in parallel there is a conflict with host file name in this
test. This patch adds a timestampt to the host file name.

Test Plan: mrunit 50 times

Reviewers: hkuang, pritam

Reviewed By: hkuang
Enable private actions always coronajobdetails.jsp

Summary:
There was a diff earlier for coronataskdetails.jsp, but
coronajobdetails.jsp was left out.

Test Plan: grep for webinterface.private.actions

Reviewers: aching, jeanxu
Fix the free disk space api.

Summary:
Do not use a flag for free/total.

Test Plan: unit tests

Reviewers:  jeanxu

CC:

Task ID: #

Blame Rev:
Fixes for java7 - batch 3

Summary:
-TestOfflineEditsViewer, fixed try finally cluster shutdown
-TestTrash - cleanup test directory
-TestReplication - needed to add some static before class setup

Test Plan: ant tests on jdk7

Reviewers: hkuang, pritam, weiyan, avf, dikang, sdong

Reviewed By: hkuang
Fix broken link for failed tasks generated by corona job tracker

Summary:
Corona job tracker generates links for failed tasks, and users can get more information by clicking the links. Currently for failed tasks, the generated links point to corona job tracker URLs. This is changed in this diff so that they point to proxy job tracker URLs.

Trac Bug: #1747099

Blame Rev:

Reviewed By:
rvadali

Test Plan:
- Deployed Corona on my dev server
- Submitted a sleep job
- Killed a map task manually before the job completed
- Visited cluster manager web page, and the link of the failed task pointed to the correct URL.

Revert Plan:

Database Impact:

Memcache Impact:

Other Notes:

EImportant:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -
Fix TestAvatarCheckpointing

Summary:
For TestAVatarCheckpoinitng:
-assert that test checkpoint failed by checking the appropriate event,
instead of the signature.

-For DatanodeMxBean, fix try finally

Test Plan: ant test both

Reviewers: hkuang, pritam, avf, dikang, weiyan

Reviewed By: dikang
[DistCp Tos] support the TOS setting in DistCp

Summary:
Set the TOS value to 0x4 by default, also make it to be user
configurable.

Test Plan: test run on dev server, ant test -Dtestcase=TestCopyFiles.

Reviewers: hkuang, pknowles

Reviewed By: pknowles

Task ID: 1814473
[HDFS Namenode BugFix] process over replictaed block asynly

Summary:
After discussion with Hairong, it seems it doesn't make sense to process the over replicated block under the write lock
we should let OverReplicationMonitor do that. This could avoid the expensive blockPlacementPolicyRaid operation.

It seems the setReplicationInternal is doing the same thing

Test Plan: running test, probably break some tests

Reviewers: hkuang, tomasz, sdong, dikang, pritam

Reviewed By: tomasz
Fix reporting of received blocks when they are rejected

Summary:
When a block is rejected for ibr by addStoredBlock, it is still reported
as received. This patch fixes this and simplifies the reporting logic:

-now we won't report the IGNORE, deleted, nor rejected blocks

Test Plan: ant test

Reviewers: hkuang, pritam, sdong

Reviewed By: hkuang