MAPREDUCE-1302. TrackerDistributedCacheManager can delete file asynchronously

Description: With the help of AsyncDiskService from MAPREDUCE-1213, we should be able to delete
files from distributed cache asynchronously. That will help make task initialization faster, because task initialization calls the code that
localizes files into the cache and may delete some other files.
The deletion can slow down the task initialization speed.
Reason: Performance improvement
Author: Zheng Shao
Ref: CDH-495
MAPREDUCE-1644. Remove Sqoop from Apache Hadoop (moving to github)

Description: Sqoop is moving to github! All code for sqoop is already live at
http://github.com/cloudera/sqoop - this issue removes the duplicate code from the Apache Hadoop
repository. CDH users should install the separate 'sqoop' package for this functionality.
Reason: Moving to a separate package
Author: Aaron Kimball
Ref: CDH-1404
CLOUDERA-BUILD. Update hadoop-config.sh to reflect new jar version.
HDFS-909. Fix race in edit log rolling

Description: Fixes a race condition when rolling edit logs that can corrupt
             the logs.
Reason: Potential namenode metadata corruption bug.
Author: Todd Lipcon
Ref: CDH-1174
HDFS-200. Support append and sync for hadoop 0.20 branch.

Description: Provides basic support for append and sync on 0.20
Reason: Append and sync required for durable HBase and many other
        applications.
Author: Dhruba Borthakur
Ref: CDH-659
HDFS-101. Datanodes should continue to forward acks until client stops pipeline.

Description: When one node in the pipeline dies, the datanodes in between the client
             and the dead node should stay alive and continue to forward acks until
             the client stops the pipeline. This fixes an issue where the client
             would incorrectly determine that the local DN had failed when in fact
             another DN in the pipeline was at fault.
Reason: Common source of failed pipeline recovery in cluster fault testing
Author: Hairong Kuang, Todd Lipcon
Ref: CDH-693
HDFS-988. Fix bug where savenameSpace can corrupt edits log.

Description: Fixes several synchronization errors in the NameNode and ensures
             that all edits have been synced to the edits log before
             the namespace is saved.
Reason: Fixes potential data corruption bug.
Author: Todd Lipcon
Ref: CDH-1436
HDFS-826. Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline. (dhruba)

Description: Adds an API in DFSOutputStream to determine the current length
             of the write pipeline.
Reason: Necessary for better reliability of HBase write-ahead logs.
Author: Dhruba Borthakur
Ref: CDH-931
HDFS-142. Blocks that are being written by a client are stored in the blocksBeingWritten directory.

git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append@953482 13f79535-47bb-0310-9956-ffa450edef68

Description: Moves blocks being written by clients into a different
             directory in dfs.data.dir. Also fixes several other bugs
             in the datanode and namenode to support various error
             conditions related to append and sync.
Reason: Necessary for proper recovery of synced data in several error conditions.
Author: Dhruba Borthakur, Nicolas Spiegelberg, Todd Lipcon
Ref: CDH-659
HDFS-1054. remove sleep before retry for allocating a block.

Description: When the write pipeline fails to allocate a new block,
             it previously slept for hard-coded 6 seconds before
             retrying. This sleep has little reasoning behind it,
             so is removed.
Reason: Improve failure recovery performance for interactive applications
        like HBase.
Author: Todd Lipcon
Ref: CDH-931
HDFS-894. Fix ipcPort tracking in Datanode registration. TODO: add the test case from JIRA

Description: Fixes the NameNode to properly reregister datanodes when they
             crash and restart with a different IPC port (eg when IPC port
             is configured to be ephemeral)
Reason: Fixes errors on clusters with ephemeral ports.
Author: Todd Lipcon
Ref: CDH-659
HDFS-1055. Improve thread naming in DN Xceiver

Description: Names the threads created by the DataNode based on the action
             they are performing.
Reason: Eases diagnosis of datanode performance/lock contention issues.
Author: Todd Lipcon
Ref: CDH-659
HDFS-611. Move block deletions to an async thread. Applying this to make the HDFS-142 patch apply cleanly

Description: Moves the deletion of blocks in the datanode into a thread
             pool. Substantially improves datanode heartbeat consistency
             for workloads with heavy deletes and/or lots of disks.
Reason: Substantially reduces frequency of "could not complete block"
        errors and needless re-replication on clusters with lots of disks
        or heavy deletes.
Author: Zheng Shao
Ref: CDH-659
HDFS-1056. Fix possible multinode deadlocks during block recovery when using ephemeral dataxceiver ports

Description: Fixes the logic by which datanodes identify local RPC targets
             during block recovery for the case when the datanode
             is configured with an ephemeral data transceiver port.
Reason: Potential internode deadlock for clusters using ephemeral ports
Author: Todd Lipcon
Ref: CDH-659
HDFS-1211. BlockReceiver logs too much at INFO level when using sync()

Description: Reduces the log level from INFO to DEBUG for a common message
             in the datanode log when using the sync feature.
Reason: Substantially reduces DN log chattiness for syncing clients.
Author: Todd Lipcon
Ref: CDH-659
HDFS-895. Allow hflush/sync to operate in parallel with other writers

Description: Modifies synchronization of the DFSOutputStream sync feature
             such that multiple threads can sync the same stream
             concurrently and each will wait only the minimal amount
             of time. Also allows further writes to continue past the
             sync point while the sync waits.
Reason: Substantial performance improvement for durable HBase
Author: Todd Lipcon
Ref: CDH-659
HDFS-561. Fix read timeouts in write pipeline to stage correctly

Description: Previously, the read timeout on the write pipeline was
             incorrectly calculated. This caused the client to detect
             the wrong failed datanode when a datanode's network
             failed or froze for another reason.
Reason: Fix recovery behavior for frozen datanodes
Author: Kan Zhang
Ref: CDH-659
HDFS-1203. Add small sleep to prevent DN flooding NN in error cases

Description: If the datanode experiences an error in sending its block
             reports to the name node, it previously would loop retrying
             with no delay between attempts. In the case that the DN
             is sending an invalid report, this will flood the NN with
             RPCs. This patch adds a short sleep before the retry.
Reason: Prevents possible flood of RPCs to the NameNode in DN error
        conditions.
Author: Todd Lipcon
Ref: CDH-659
HADOOP-6722. Workaround a TCP spec quirk by not allowing NetUtils.connect to connect to itself

Description: TCP's ephemeral port assignment results in the possibility
             that a client can connect back to its own outgoing socket,
             resulting in failed RPCs or datanode transfers.
Reason: Fixes intermittent errors in cluster testing with ephemeral
        IPC/transceiver ports on datanodes.
Author: Todd Lipcon
Ref: CDH-659
HDFS-1057. Fixes for concurrent readers behind an appended file

Description: Allows a client to read a file while it is still being
             written by a writer, so long as the writer has called
             sync().
Reason: Used by HBase replication, and useful for other "tail"-like
        applications.
Author: Sam Rash
Ref: CDH-659