Repository: iterative/dvc

Issue:
external outputs: broken if pipeline output doesn't exist during stage initialization
# Bug Report

## Description

S3 external outputs are broken for pipelines since https://github.com/iterative/dvc/commit/7211bd02eda74d5434f1b7996647f7027e6e83b0 because of a bug in s3fs (and probably in other filesystems). They will only break if running a stage for which an output doesn't already exist. When initializing the stage, DVC will try to remove the nonexistent output and raise a `FileNotFound` error.

### Reproduce

`dvc repro` will break if there is an external output and that output does not exist yet.

In a new repo, using some `<s3_path>` that doesn't exist yet, do this:

```
$ echo 'foo' > foo
$ dvc stage add --external -n foo -d foo -O <s3_path> 'aws s3 cp params.yaml <s3_path>'
$ dvc repro -v
```

### Expected

`dvc repro` shouldn't fail while removing outputs. In this case, it fails because of what seems like a bug or at least inconsistent behavior in fsspec. Like mentioned in https://github.com/iterative/dvc/issues/5961#issuecomment-1365822275, [`output.remove`](https://github.com/iterative/dvc/blob/main/dvc/output.py#L901) for s3fs and other async filesystems calls [_expand_path](https://github.com/fsspec/filesystem_spec/blob/master/fsspec/asyn.py#L719). When the path doesn't exist and `recursive=True`, `_expand_path` raises `FileNotFoundError`. When `recursive=False`, it returns the path. It also returns the path for the `LocalFileSystem` regardless of whether `recursive=True`, so not sure if it was intended to raise an error only for this specific scenario.


Discussion:
Not sure if this should be a new issue, but I'm seeing the same problem now because we have `recursive=True` in https://github.com/iterative/dvc/blob/8f768addf956b56820b325bd717613bef7f0c141/dvc/output.py#L968.
@daavoo Do you recall from https://github.com/iterative/dvc/pull/8825 if we need `recursive=True`?
> @daavoo Do you recall from #8825 if we need `recursive=True`?

I didn't test its removal but it was supposed to be a legacy option no longer needed https://github.com/iterative/dvc-objects/pull/176#discussion_r1068110883

Resolving patch:
diff --git a/dvc/output.py b/dvc/output.py
--- a/dvc/output.py
+++ b/dvc/output.py
@@ -965,7 +965,10 @@ def relative_update(self, inc: int = 1) -> None:
         return added, False if added else modified
 
     def remove(self, ignore_remove=False):
-        self.fs.remove(self.fs_path, recursive=True)
+        try:
+            self.fs.remove(self.fs_path, recursive=True)
+        except FileNotFoundError:
+            pass
         if self.protocol != Schemes.LOCAL:
             return
 


Test patch:
diff --git a/tests/func/repro/test_repro.py b/tests/func/repro/test_repro.py
--- a/tests/func/repro/test_repro.py
+++ b/tests/func/repro/test_repro.py
@@ -1273,3 +1273,12 @@ def test_repro_missing_lock_info(tmp_dir, dvc, copy_script):
 
     stages = dvc.reproduce(stage.addressing)
     assert len(stages) == 1
+
+
+def test_repro_rm_recursive(tmp_dir, dvc):
+    # check that dir output recursively removes files in the dir
+    tmp_dir.gen({"dir": {"foo": "foo"}})
+    dvc.stage.add(name="dir", cmd="mkdir dir", outs=["dir"])
+    dvc.reproduce()
+    assert (tmp_dir / "dir").exists()
+    assert not (tmp_dir / "dir" / "foo").exists()