SPARK-2532: Minimal shuffle consolidation fixes

All changes from this PR are by @mridulm and are drawn from his work in #1609. This patch is intended to fix all major issues related to shuffle file consolidation that @mridulm found, while minimizing changes to the code, with the hope that it may be more easily merged into 1.1.

This patch is **not** intended as a replacement for #1609, which provides many additional benefits, including fixes to ExternalAppendOnlyMap, improvements to DiskBlockObjectWriter's API, and several new unit tests.

If it is feasible to merge #1609 for the 1.1 deadline, that is a preferable option.
I meant the former case : close on a writer fails with an exception; while earlier streams succeeded.
So now we have some writers which have committed data (which is not removed by subsequent revert) while others are reverted.

On the face of it, I agree, it should not cause issues : but then since the expectation from this class is never enforced; and so can silently fail.