Only check for permissions if directory exists