Fix issue where directories that were not suppose to be removed (with files that were not suppose to be removed) were removed.
Also added tests to verify.