[SPARK-2532] Consolidated shuffle fixes

Status of the PR
- [X] Cherry pick and merge changes from internal branch to spark master
- [X] Remove WIP comments and 2G branch references.
- [X] Tests for BlockObjectWriter
- [x] Tests for ExternalAppendOnlyMap
- [x] Tests for ShuffleBlockManager



As I mentioned above, we did fix a dcl bug, bug that did not seem sufficient.
I agree this is a rare condition, and the 'fix' is a hack to workaround the problem : but pending identifying root cause, this is the best we have unfortunately.
Any thoughts ?