Fix issue where directories that were not suppose to be removed (with files that were not suppose to be removed) were removed