Gracefully handle empty directories when setting permissions.