From what I can tell here `doCleanupRDD` is not blocking and the other two are blocking. Is there any ways these could all be non-blocking?

Otherwise if there is, e.g. GC on one of the slaves, or one of the shuffles takes a long time to delete all of the files, then you could end up in a situation where you are sitting in this loop for a long time. That doesn't seem super desirable.