[Raid bugfix] Fix the concurrent issue in the Codec initialization.

Summary: The Codec initialization will fail in concurrent calls.

Test Plan: ant test

Reviewers: sdong, weiyan

Reviewed By: weiyan
[Raid] refactor the TestCodec.testMultiThreadCreation

Summary:
Use the Callable instead of Runnable, it will fail in both IOException
and other RuntimeException cases.

Test Plan: ant test.

Reviewers: sdong, weiyan

Reviewed By: sdong
[Raid Perf] Merge the ErasureCode speed up changes.

Test Plan: unit tests,

Reviewers: weiyan, smuralid, dhruba, hkuang

Reviewed By: weiyan

Task ID: 1124534
[Raid Perf] Improve the encoder speed.

Reviewers: weiyan, hkuang, smuralid

Reviewed By: weiyan

Task ID: 1124534
[Raid fix] fix the ErasureCode bug

Summary: We should cleanup the write buffer before do the R-S calculation.

Test Plan: ant test

Reviewers: weiyan, smuralid

Reviewed By: weiyan
[Raid Dir-Raid] Log raided/fixed files in scribe

Summary:
We want to record the files raided or fixed by the raidnode in offline to figure out
affected files if raid bugs are discovered in production.
Because offline encoding and block fixing are done by the mapper tasks, it's better to
log these information to scribe in each mapper. But then we introduce a dependancy
on the scribe and we should closely monitor the availabiliy of scribe logger.

Reviewers: dikang, hkuang

Reviewed By: dikang

Task ID: 1079204
Additional logging for the isInitialized command.

Summary:
bin/hadoop avatarshell -isInitialized did not return any
success or failure message on the terminal when it was run. We need to
add this in so that ops can easily determine whether the command
succeeded or not.

Test Plan: TestAvatarShell

Reviewers: tomasz, weiyan, sdong

Reviewed By: tomasz

Task ID: 1147638
Make thriftfs use thrift 0.7

Summary: Using the internal thrift is causing thrift compatibility problems in production.

Test Plan: Made the existing unit test compile and run

Reviewers: pyang, aching

Reviewed By: pyang

Task ID: 1145733
Put the libs for cluster manager in their own directory.

Summary: This will avoid polluting the common lib directory

Test Plan:
tested that the modified deployment could start with a prism
configuration

Reviewers: aching, pyang

Reviewed By: aching

CC: platform-diffs@lists
Allow customizing classpath when launching tasks

Summary:
We want the corona task tracker to use apache thrift, but should not
force the tasks to get this classpath onto the tasks launched. This diff
provides a configurable way of removing parts of the task tracker
classpath.

Test Plan:
ran the single node setup with the exclude pattern set and
observed that the thrift jar was getting excluded

Reviewers: aching, pyang

Reviewed By: aching

CC: dms
[Raid Bugfix] doFileRaid and doDirRaid could return false and we should consider it a failure

Summary:
In the scribe encoding log, we only consider the encoding fails when it throws exception.
But actually the encoding could fail because of other reasons.

Test Plan: deploy

Reviewers: dikang, hkuang

Reviewed By: dikang
Add an option to avatarzk to wait for the transaction id znode.

Summary:
When we use the command bin/hadoop avatarzk -shutdownAvatar,
  before we try to failover to the Standby it would be helpful to make
  sure that the Primary has synced its last txid to zookeeper. This can
  be done by checking whether the transaction id and session id znode
  both have the same session id. This adds a command line parameter
  -waittxid to avatarzk to do this.

Test Plan:
1) Unit tests added.

Revert Plan:

Reviewers: hkuang, tomasz, weiyan

Reviewed By: weiyan
[Raid Dir-Raid] Raid fsck

Summary:
Raid fsck returns the unrecoverable files under given path.
For directory raid, a file may have multiple stripes, and stripes could be
related to multiple files.
To determine if a corrupt file is recoverable or not, we first locate all related
stripes to the file and collect the number of corrupt blocks for each related
stripe. Then we look at the missing blocks in the file and check if they could be
recoverable. If the stripe contains the missing block has too many corrupt blocks,
we consider the file is unrecoverable.

Test Plan: add a test TestDirectoryRaidShellFsck

Reviewers: hkuang, dikang

Reviewed By: dikang

Task ID: 1079196
Major speed improvements to the corona UI (Chrome 3.3x, Firefox 7.171x)

Summary:
* Factored out some common UI methods to new class WebUtils.java
* All the fetches for the pool table, active table, and retired table are done through AJAX calls to external JSPs to get back JSON
* Pulled in themeswitchertool to the static directory to speed up the load
* New page by default is cm.jsp, preserved the old page as cm_old.jsp and cm_old.js in case of issues

Reviewers: rvadali, pyang, dms

Reviewed By: rvadali

CC: kevinwilfong, rmurthy, pknowles, davidrecordon, achao

Task ID: 1186467
Changes to collect a new metrics related to total time taken to fix a file after it is detected to be corrupt.

Summary:
(1) Added a new RaidNodeMetric "numFilesToFixDropped". This stands for the number of files which are detected to be corrupt in the current wake cycle of the corruptionWorker, but that are not submitted for reconstruction since number of pending jobs exceeds the limit.
(2) Logging the "File_Fix_WaitTime" metric. This is the time delay between the detection of a corrupt file and the time when the reconstruction job starts execution.

Test Plan:
For (1) unit test : TestRaidShellFsck_CorruptCounter. The new subtest testRaidNodeMetricsNumFilesToFixDropped() should pass.
For (2) Run TestBlockFixer and observe the mapreduce logs generated in the test output. There will be log entries of the form: RaidMetrics: {"maxPendingJobs":100,"maxFilesPerTask":2,"fileFixWaitTime":6723,"type":"FILE_FIX_WAITTIME"}

Reviewers: dikang, weiyan, hkuang

Reviewed By: weiyan
Make a task tracker profile all tasks on it

Summary:
Provide a reconfigurable property to make a task tracker turn on
profiling for all the tasks that run on it.

Test Plan: tested with a sleep job on a single node setup.

Reviewers: dms, aching, pyang

Reviewed By: aching