[SPARK-2532] Consolidated shuffle fixes

Status of the PR
- [X] Cherry pick and merge changes from internal branch to spark master
- [X] Remove WIP comments and 2G branch references.
- [X] Tests for BlockObjectWriter
- [x] Tests for ExternalAppendOnlyMap
- [x] Tests for ShuffleBlockManager



This is just more defensive cleanup compared to earlier and setting batchIndex to unusable value.
To ensure that any close related exceptions do not result in unclosed files (which impacts ulimit of spark process)