Hi @andrewor14 and @aarondav , thanks for your great work. Is it possible to create files under `spark.local.dir` rather than tmp dir to let block manager to manage all these spill files. 
1. Normally multiple disks will be configured in `spark.local.dir`, compared to only 1 disk for tmp dir. It can gain better throughput of multiple disks when concurrently spilling to disks.
2. Block manager can delete files on exit or exceed the cleanup time.
3. It might be easy be measure the spill size or others.