[SPARK-3801] More efficient app dir cleanup

The newly merged more conservative app directory cleanup can be more efficient. Since it is not needed to store newer files, using `exists` to check if newer files are there can be more efficient because redundant items are skipped.