Don't spit errors if directories to be cleaned are missing

#5126