[SPARK-3970] Remove duplicate removal of local dirs

The shutdown hook of `DiskBlockManager` would remove localDirs. So do not need to register them with `Utils.registerShutdownDeleteDir`. It causes duplicate removal of these local dirs and corresponding exceptions.
Adding a check to see if the dir exits in `listFilesSafely` helps. I add another commit for that. However, the duplicate delete logic is undesirable and should not be there too.