[SPARK-2532] Consolidated shuffle fixes

Status of the PR
- [X] Cherry pick and merge changes from internal branch to spark master
- [X] Remove WIP comments and 2G branch references.
- [X] Tests for BlockObjectWriter
- [x] Tests for ExternalAppendOnlyMap
- [x] Tests for ShuffleBlockManager



The reason for change to var from def is perhaps subtle.
Consider the case of :

add for mapIdToIndex with mapId 0
add for mapIdToIndex with mapId 1
add for mapIdToIndex with mapId 0 (on re-execution)
add for mapIdToIndex with mapId 1 (on re-execution)

Now both 0 and 1 will end up with the same index assigned (since it was based on mapIdToIndex.size).