SPARK-2711. Create a ShuffleMemoryManager to track memory for all spilling collections

This tracks memory properly if there are multiple spilling collections in the same task (which was a problem before), and also implements an algorithm that lets each thread grow up to 1 / 2N of the memory pool (where N is the number of threads) before spilling, which avoids an inefficiency with small spills we had before (some threads would spill many times at 0-1 MB because the pool was allocated elsewhere).
Looks like this case exists because a new thread may join when all other threads have collectively occupied all of the memory. Maybe add a comment here that describes this scenario?

(At first I thought this could never happen because each thread is guaranteed to not go over 1/N, and hence there is always 1/2N memory available. However, this assumes the number of threads is static, which is not true here)