Another way to think about it is this. This code path is really just to protect users who are doing something that already leads to very poor performance - shuffles where they use way too many partitions and the files are really tiny. The best thing a user can do there is to coalesce the number of mappers or reducers to get much better peformance. If they can figure that out, there is no reason to ever tune this.