Include dependencies with Hadoop when publishing jar to maven

Summary: As external dependencies are added to Hadoop, we also have to add them to HBase, because the Hadoop .pom file we publish to the local Maven repository does not contain dependencies. Here I am trying to use the makepom target to auto-generate those dependencies from ivy. Also, I am setting Hadoop version to 0.20. I will double-check that this works correctly with build_all.sh in the warehouse branch.

Test Plan:
In Hadoop directory:
ant clean
copy-hdfs-jars-to-maven.sh

In HBase directory:
Build HBase

In the warehouse branch checkout:
./build_all.sh --hadoop=true

Reviewers: pritam, avf, weiyan, sdong, dms

Reviewed By: pritam
isConsistent() always returns true when the namenode is out of safemode.

Summary:
Once the NN leaves safemode, the safemode's isConsistent()
function can return false and cause assertion failures since the safe
block count can go below the total block count due to deletions.

Test Plan: All unit tests.

Reviewers: mbautin, hkuang

Reviewed By: mbautin

Task ID: 1207899
Adding a readme to corona

Summary: This is a brief overview of the corona project

Test Plan: N/A

Reviewers: aching, rvadali, pyang

Reviewed By: rvadali
Adding default datanode to refreshNamenodes command

Summary:
When using the DFSAdmin's "-refreshNamenodes" command, if no
"datanodehost:port" is given, then it will default to
"localhost:defaultconfport", where the "defaultconfport" is the port
value taken from the "dfs.datanode.ipc.address" configuration property.

Test Plan:
Running "ant test-hdfs" at the moment. Also tested that by
running "hadoop dfsadmin -refreshNamenodes" I get
"localhost:defaultconfport", and by running
"hadoop dfsadmin -refreshNamenodes somehost:someport", I indeed get
"somehost:someport".

Reviewers: hkuang, sdong

Reviewed By: sdong

Task ID: 1153147
HDFS: Minor bug. Fix a line that makes no sense

Summary:
This codes make no sense and throw NullPointerException in scanner thread. I fixed the codes to what it intended to be (I think).
The end result should be the same as previous.

Test Plan: ant test

Reviewers: weiyan, tomasz, hkuang

Reviewed By: hkuang
Update pom.xml for hbase to include version 2.9 of the Surefire plugin

Summary:
Some of the hbase tests were failing with the following error:

[INFO] Error while executing forked tests.; nested exception is
org.apache.maven.surefire.booter.shade.org.codehaus.plexus.util.cli.CommandLineException:
Error inside systemErr parser

The above error was fixed in Surefire 2.7
http://jira.codehaus.org/browse/SUREFIRE-640

In addition, the surefire plugin wasn't quitting when the tests were finished.

Test Plan: Run tests and see if they are more reliable
Reviewed By: mbautin
Reviewers: mbautin, liyintang
CC: nspiegelberg, mbautin
Fix copy-hdfs-jars-to-maven.sh to unbreak Jenkins HBase builds

Summary:
Apparently the COMMANDER_WORKSPACE variable is not set in the Jenkins
environment, so using the username to determine whether we are running
as part of a Jenkins build.

Simple change, not reviewed.

Test Plan:
Tested on a build server.
Adding time breakdown for launch task

Summary:
This will help us diagnose cases when the tasks are not
launching quickly enough.

Test Plan:
test cluster, since this is a logging change only - there's
not that much testing required.

Reviewers: aching, rvadali

Reviewed By: rvadali
Fix namenode deadlock

Summary:
This diff
1. changed synchornized methods in FSNamesystem to use either read or write lock;
2. Relaxed a lot of methods in NameNodeSafeModeInfo and StandbySafeMode not to be synchronized; If they have to be synchronized, acquire a namesystem lock first.

Test Plan: DFSTEST

Reviewers: pritam, tomasz

Reviewed By: pritam

Task ID: 1233053
Unbreak the build

Summary: Oops.

Test Plan: ant jar

Reviewers: rvadali, hkuang

Reviewed By: rvadali
Fix stuck NameNode caused by a missing edit stream

Summary: This diff fixed NPE caused by a missing edit stream in SyncThread by moving fetching the number of streams closer to where it is used. It fixed the way it handles edit stream error. It also adds a try-catch to catch Throwable that will print the runtime exception and for NameNode to exit when RE happens.

Test Plan: dfstest

Reviewers: tomasz

Reviewed By: tomasz

Task ID: 1236623
Add outstanding heartbeats, reports to the NN webui.

Summary:
During failover, we might need to know which datanodes have not
yet reported back to us.

Test Plan: 1) Check webUI on dev cluster.

Reviewers: hkuang, tomasz, weiyan

Reviewed By: tomasz
Scheduler runtime metric change

Summary:
Currently we only store the runtime of the last run of the
scheduler. The metric makes it hard to set an alarm on when the
scheduler is being slow.
Instead whenever the scheduler is 'running' we should return the time
spent in this run.

Test Plan: Will test that this metric works locally

Reviewers: rvadali, aching

Reviewed By: rvadali

CC: pknowles

Task ID: 1140807
Trace File System (Streams)

Summary: Created a file-system layer that traces important calls when enabled.

Test Plan:
1) Replace client's core.jar with the new build/hadoop-${hadoop.version}-core.jar.

2) Update client's config.

In core-site.xml, using the following key=>val properties:
fs.hdfs.impl                  => org.apache.hadoop.hdfs.APITraceFileSystem
fs.apitrace.underlyingfs.impl => org.apache.hadoop.hdfs.DistributedRaidFileSystem
fs.raid.underlyingfs.impl     => org.apache.hadoop.hdfs.DistributedAvatarFileSystem

In core-site-custom.xml, make sure fs.hdfs.impl is not overridden.

Add the following line to log4j.properties:
log4j.logger.org.apache.hadoop.metrics.APITrace=DEBUG

3) Run './bin/hadoop dfs -<cmd> <args>' for various commands on a client communicating with a test cluster.  Besides producing the normal output for HDFS, there should be encoded log statements printed to stdout.  The expectations for various commands are listed below:

list:  N/A (we only do stream tracing)
put:   N/A (uses copyFromLocalFile, which we currently do not trace)
get:   FileSystem.open, at least one FSOuptutStream.read, and FSOutputStream.close
cat:   FileSystem.open, at least one FSOuptutStream.read, and FSOutputStream.close
tail:  FileSystem.open, FSOutputStream.seek, at least one FSOuptutStream.read, FSOutputStream.getPos, and FSOutputStream.close
move:  N/A (uses rename, which we don't trace until the next diff)
copy:  FileSystem.open(src)=1, FileSystem.create(dst)=2, at least one FSOutputStream[1].read, at least one collapsed write for [2], FSOutputStream[2].close, FS least one FSOuptutStream.read, and FSInputStream[1].close
rm:    N/A (uses delete, which we don't trace until the next diff)
mkdir: N/A (uses mkdirs, which we don't trace until the next diff)

Reviewers: sdong, dhruba

Reviewed By: sdong
Trace File System (NameNode)

Summary: Trace of non-stream calls.  See D525644 for stream tracing.

Test Plan:
1) Replace client's core.jar with the new build/hadoop-${hadoop.version}-core.jar.

2) Update client's config.

In core-site.xml, using the following key=>val properties:
fs.hdfs.impl                  => org.apache.hadoop.hdfs.APITraceFileSystem
fs.apitrace.underlyingfs.impl => org.apache.hadoop.hdfs.DistributedRaidFileSystem
fs.raid.underlyingfs.impl     => org.apache.hadoop.hdfs.DistributedAvatarFileSystem

In core-site-custom.xml, make sure fs.hdfs.impl is not overridden.

Add the following line to log4j.properties:
log4j.logger.org.apache.hadoop.metrics.APITrace=DEBUG

3) Run './bin/hadoop dfs -<cmd> <args>' for various commands on a client communicating with a test cluster.  Besides producing the normal output for HDFS, there should be encoded log statements printed to stdout.  The expectations for various commands are listed below:

list:  FileSystem.getFileStatus(/), FileSystem.listStatus, FileSystem.close
put:   FileSystem.close (uses copyFromLocalFile, which we currently do not trace)
get:   FileSystem.setVerifyChecksum, three calls to FileSystem.getFileStatus, FileSystem.open, at least one FSInputStream.read, FSInputStream.close, FileSystem.close
cat:   FileSystem.setVerifyChecksum, two calls to FileSystem.getFileStatus, FileSystem.open, at least one FSInputStream.read, FSInputStream.close, FileSystem.close
tail:  FileSystem.setVerifyChecksum, two calls to FileSystem.getFileStatus, FileSystem.open, FSInputStream.seek, at least one FSInputStream.read, FSInputStream.getPos, FSInputStream.close, FileSystem.close
move:  three calls to FileSystem.getFileStatus, FileSystem.rename, FileSystem.close
copy:  two calls to FileSystem.getFileStatus, FileSystem.open, FileSystem.getDefaultReplication, FileSystem.getDefaultBlockSize, FileSystem.create, at least one FSInputStream.read, at least one collapsed write, FSOutputStream.close, FSInputStream.close, FileSystem.close
rm:    three calls to FileSystem.getFileStatus, FileSystem.mkdirs(.Trash), two calls to FileSystem.getFileStatus, FIleSystem.rename (move to .Trash), FileSyste.close
mkdir: FileSystem.mkdirs, FileSystem.close

Reviewers: sdong, dhruba

Reviewed By: sdong
Changes for making perf testing easier

Summary:
This diff lets us make a task tracker usable for profiling tasks.

1. Allow specifying a the command (in place of $JAVA_HOME/bin/java) that
will be used to start the child process. This can be used to start a command
that does the profiling and starts the child. This will allow us to use the
Studio Analyzer collect command for profiling tasks that run on a task tracker.

2. Make corona task tracker get the number of CPUs on the machine through
configuration. This, along with a suitable cpu->resource mapping on the cluster manager
will let us control the number of map slots on the machine.

Test Plan: tested on a single node setup.

Reviewers: dms, pyang

Reviewed By: dms

CC: binu
HDFS: Add a way to disable DFSClientMetrics from being updated

Summary: We realised that the costs of DFSClientMetrics is not small compared to total client costs (sometimes can be 45%) but most of the time, we don't use it (Actually I didn't find where it is used in either data warehouse clusters or HBase). Add an option to disable the metrics to save the costs for cost sensitive applications.

Test Plan: Manually turn it off and make sure normal operations works as normal

Reviewers: hkuang, tomasz, weiyan

Reviewed By: tomasz

CC: dhruba
Fix bug in the Standby Avatar, where it never expired datanodes.

Summary:
Since the standby avatar is in safemode, its heartbeat checker
never ran and as a result it never marked datanodes as dead. This is
dangerous since it can cause an incorrect safe block count on the
standby avatar.

Test Plan:
1) Reproduced in unit test.
2) All unit tests.

Revert Plan:

Tags:

- begin *PUBLIC* platform impact section -
Bugzilla: #
- end platform impact -

Reviewers: hkuang, tomasz, weiyan, sdong

Reviewed By: hkuang

Task ID: 1246886
Fix TestDFSClientMetrics

Summary: Fix the testcase broken by 530788

Test Plan: ant test

Reviewers: sdong, dikang

Reviewed By: dikang
Enable r/w permissions for everyone for ZK cache files.

Summary:
We need to allow all users to read and write to the cache files
so that we can have only a single cache file on the machine which is
shared by everyone and not a per user cache file.

Test Plan:
1) Run TestAvatarFailoverCaching,
     TestAvatarContinuousFailover, TestCachingAvatarZooKeeperClient (all
         use the cache)

Reviewers: hkuang, tomasz, weiyan, sdong, dms

Reviewed By: dms

Task ID: 1252127