SPARK-1612: Fix potential resource leaks

JIRA: https://issues.apache.org/jira/browse/SPARK-1612

Move the "close" statements into a "finally" block.

Author: zsxwing <zsxwing@gmail.com>

Closes #535 from zsxwing/SPARK-1612 and squashes the following commits:

ae52f50 [zsxwing] Update to follow the code style
549ba13 [zsxwing] SPARK-1612: Fix potential resource leaks
[SPARK-2379] Fix the bug that streaming's receiver may fall into a dead loop

Author: joyyoj <sunshch@gmail.com>

Closes #1694 from joyyoj/SPARK-2379 and squashes the following commits:

d73790d [joyyoj] SPARK-2379 Fix the bug that streaming's receiver may fall into a dead loop
22e7821 [joyyoj] Merge remote-tracking branch 'apache/master'
3f4a602 [joyyoj] Merge remote-tracking branch 'remotes/apache/master'
f4660c5 [joyyoj] [SPARK-1998] SparkFlumeEvent with body bigger than 1020 bytes are not read properly
SPARK-2791: Fix committing, reverting and state tracking in shuffle file consolidation

All changes from this PR are by mridulm and are drawn from his work in #1609. This patch is intended to fix all major issues related to shuffle file consolidation that mridulm found, while minimizing changes to the code, with the hope that it may be more easily merged into 1.1.

This patch is **not** intended as a replacement for #1609, which provides many additional benefits, including fixes to ExternalAppendOnlyMap, improvements to DiskBlockObjectWriter's API, and several new unit tests.

If it is feasible to merge #1609 for the 1.1 deadline, that is a preferable option.

Author: Aaron Davidson <aaron@databricks.com>

Closes #1678 from aarondav/consol and squashes the following commits:

53b3f6d [Aaron Davidson] Correct behavior when writing unopened file
701d045 [Aaron Davidson] Rebase with sort-based shuffle
9160149 [Aaron Davidson] SPARK-2532: Minimal shuffle consolidation fixes
[SPARK-2786][mllib] Python correlations

Author: Doris Xin <doris.s.xin@gmail.com>

Closes #1713 from dorx/pythonCorrelation and squashes the following commits:

5f1e60c [Doris Xin] reviewer comments.
46ff6eb [Doris Xin] reviewer comments.
ad44085 [Doris Xin] style fix
e69d446 [Doris Xin] fixed missed conflicts.
eb5bf56 [Doris Xin] merge master
cc9f725 [Doris Xin] units passed.
9141a63 [Doris Xin] WIP2
d199f1f [Doris Xin] Moved correlation names into a public object
cd163d6 [Doris Xin] WIP
update pom.xml for hadoop-2.3-cdh50.0 and hbase 0.96.1.1
Mesos workaround
SPARK-2638 MapOutputTracker concurrency improvement

Rolled back files not intended for checkin
Updated concurrency fix for using same monitor on the synchronized and wait logic
Manually revert custom changes to master
Revert "update pom.xml for hadoop-2.3-cdh50.0 and hbase 0.96.1.1"

This reverts commit a91f6a39af4e799617fb6f8cb490aaedfbb2af4e.
Revert "SPARK-2638 MapOutputTracker concurrency improvement"

This reverts commit 31dcd4fbad565e9dca06307a516a682d86dcfc09.
Manually revert custom changes to master
Merge branch 'master' of https://github.com/javadba/spark