I would rather have a hidden config param which can be tuned if required (however rarely), as opposed to have magic values which actually have no experimental basis.
The impact of this PR is not just in memory usage, but also in the IO characteristics of spark.

In particular, this is going to have significant performance impact if there are a large number of small files : which are fairly common - and in a lot of times, like in our experiments, aggressively aimed towards for the performance characterstics we are shooting for.