More fixes to removing directory, prevent directories from being denied writes as files