SPARK-2532: Minimal shuffle consolidation fixes

All changes from this PR are by @mridulm and are drawn from his work in #1609. This patch is intended to fix all major issues related to shuffle file consolidation that @mridulm found, while minimizing changes to the code, with the hope that it may be more easily merged into 1.1.

This patch is **not** intended as a replacement for #1609, which provides many additional benefits, including fixes to ExternalAppendOnlyMap, improvements to DiskBlockObjectWriter's API, and several new unit tests.

If it is feasible to merge #1609 for the 1.1 deadline, that is a preferable option.
I'm not certain I understand. The situation I am imagining is that we commit to the first Writer, then the second one fails. In HashShuffleWriter, we will then call revertPartialWritesAndClose() on all Writers, causing us to revert all the changes back to "initialPosition", which should revert even the committed data.