Avoid a listdir call when cleaning up directories.