SPARK-2792. Fix reading too much or too little data from each stream in ExternalMap / Sorter

All these changes are from @mridulm's work in #1609, but extracted here to fix this specific issue and make it easier to merge not 1.1. This particular set of changes is to make sure that we read exactly the right range of bytes from each spill file in EAOM: some serializers can write bytes after the last object (e.g. the TC_RESET flag in Java serialization) and that would confuse the previous code into reading it as part of the next batch. There are also improvements to cleanup to make sure files are closed.

In addition to bringing in the changes to ExternalAppendOnlyMap, I also copied them to the corresponding code in ExternalSorter and updated its test suite to test for the same issues.
This is just paranoid stuff, you can remove the catch IOException part actually
The reason it exists is cos we have a Ulimit'ed FileInputStream - which enforces a ulimit of 8k on spark (which is the fd limit in our clusters).
For large reducers, this prevents task from getting killed.
So this is an attempt to ensure that the stream is truely closed.

I did not realize this had leaked out.