Repository: iterative/dvc

Issue:
`dvc checkout` creates empty directories 
After #9444, `index.checkout` creates empty directories that were not created with `hashfile.checkout`.

This is causing that after a `dvc checkout`, dependencies previously reported as `deleted` are being reported as `modified`

Reproduction with https://github.com/iterative/example-get-started-experiments:

- Before #9444

```console
$ dvc exp run
Reproducing experiment 'myoid-lich'                                   
Verifying data sources in stage: 'data/pool_data.dvc'                                                                                               
ERROR: failed to reproduce 'data/pool_data.dvc': missing data 'source': data/pool_data
```

- After #9444

```console
$  dvc exp run
Reproducing experiment 'peart-gest'                                   
Verifying data sources in stage: 'data/pool_data.dvc'                                                                                               
WARNING: 'data/pool_data' is empty.                                                                                                                 
WARNING: 'data/pool_data' is empty.     
Running stage 'data_split':                                                                                                                         
```

We should not try to run `data_split`, which depending on the user code might fail or continue silently when having an empty dir as dependency



Discussion:
I found this because it is causing `exp run --allow-missing` to not work as expected because `exp run` internally calls `dvc checkout`.

`--allow-missing` expects missing dependencies to report a `deleted` status.  Those missing dependencies are now being reported as `modified` and the pipeline tries to execute the stages using the empty directories as input.

cc @efiop @dberenbaum I am unsure if this creation of empty dirs should be considered a `--allow-missing` problem only
Actually just realized that this is kind of p0 because it causes plain `exp run` to also behave unexpectedly and try to run the pipeline on empty dependencies.

Updated description

@daavoo Looks like the code snippet is not from the same repository? Trying to reproduce the way it is described.
> @daavoo Looks like the code snippet is not from the same repository? Trying to reproduce the way it is described.

I forgot to comment that the lock files in the repo are 3.0 so you need to remove some lines otherwise the 2.X exp run doesn't work. But it is from the same repository. You should still be able to reproduce the 3.0 behavior, which is the unexpected one, just cloning the repo

```bash
git clone https://github.com/iterative/example-get-started-experiments
cd example-get-started-experiments
python3 -m venv .venv

sed -i "/\bhash: md5\b/d" dvc.lock
sed -i "/\bhash: md5\b/d" data/pool_data.dvc

source .venv/bin/activate
pip install 'dvc<3'

echo "2.X"
dvc exp run

cd ..
rm -rf example-get-started-experiments

git clone https://github.com/iterative/example-get-started-experiments
cd example-get-started-experiments
python3 -m venv .venv
source .venv/bin/activate
pip install 'dvc>=3'
echo "3.0"
dvc exp run

cd ..
rm -rf example-get-started-experiments
```

```
2.X
Reproducing experiment 'yummy-shah'
Verifying data sources in stage: 'data/pool_data.dvc'
ERROR: failed to reproduce 'data/pool_data.dvc': missing data 'source': data/pool_data
```

```
3.0
Reproducing experiment 'gouty-reif'
Verifying data sources in stage: 'data/pool_data.dvc'

Running stage 'data_split':
> python src/data_split.py
Traceback (most recent call last):
  File "/home/daavoo/reproduce-dvc-issues/9618/example-get-started-experiments/src/data_split.py", line 4, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'
```
@daavoo Ah, I'm really sorry, I just `cd example-get-started` accidentally instead of `example-get-started-experiments`. Can reproduce now, taking a look...
Strictly speaking it is doing what it is supposed to - create a directory and allow for missing files. It knows the structure of the directory from retrieving it from remote automatically, but doesn't do that for files. Thus creating a directory skeleton without files. There are two ways we could go here:

* make it use remote files as well (I remember people asking for this and it does make sense)
* make checkout error-out if no files at all were created and delete (or not create in the first place) the directories

Both make sense to me and probably both should be implemented, but in the mean time I gravitate to the second option, as it will be needed for the first one anyway (e.g. imagine we can't access remote files). Taking a look.
A similarly strange situation might occur if you happen to have some files (but not all) in your cache and `allow_missing` will create a directory with some files present, so I'm not sure we should use `allow_missing` during `checkout` in experiments. Seems like what we are really trying to achieve there is to try to strictly checkout dependencies and error out if we don't have something, but use `allow_missing` for the output(s) that our stage is going to try to reproduce.

Overall that reminds me of (requested in the past) `dvc checkout` for "data sources"(true initial dependencies).
So two situations:

1) `dvc exp run` uses `checkout --allow-missing` internally for both dependencies and outputs of the stage. Here we should not use `--allow-missing` for deps, but use it for `outputs`.

2) `dvc exp run --allow-missing` on top of 1) should use `checkout --allow-missing` on deps as well.


@pmrowla 's point is that `allow-missing` should be output level and not file level. And maybe we need something like `--allow-partial` in addition?

Previous behaviour was relying on `allow-missing` on output level, but `pull`'s `allow-missing` is file level. So we have a mixup now. Maybe we could have two different flags.
So the conclusion is that `--allow-missing` should be output-level thing. The file-level was likely my misinterpretation during index checkout implementation.
Out of scope for this (we said allow-partial is not needed for now).

Still, some allow-partial functionality might be valuable, but imo it will probably not be useful without some more granular control like providing a subpath or glob inside a dep dir - where we guarantee all files exist there but "allow missing" outside of it (but still in the dep)
Thinking about this some more, there is a lot of confusion in what --allow-missing means and it is once again mixing up data management and pipeline management.

I think it makes sense for checkout (similar to other data commands) to do by default as much as possible even if it means that some stuff will be partial. This should be the behavior without any flags at all, just by default. At this point for that --allow-missing is redundant and can be removed, as user will just get a partial result + an error by default, which he can then decide to delete himself.

Now when it comes to pipelines (e.g. in dvc exp run case), the aim here is reproducibility and what we want here is to make sure we have the dependencies that we've recorded in the same state, so that we could run the pipeline. So the question here is "do we have all the files in order to restore the workspace to the state in which we can reproduce/run this pipeline?". And this is not a checkout problem, but an index/cache analysis problem that needs to be done before we even try to checkout anything. This also reminds me of the previously discussed need to be able to restore "virtual workspace" from the run-cache (currently we can't say if run-cache will work without checking out files in the workspace), which has a similar problem of needing to validate that it has cache needed.

So what I propose is:

1) For both pull and checkout, make `--alow-missing` just not make it error out if some files are missing. Partial state is totally justified here. I would maybe even consider deleting `--allow-missing` flag completely, but it is already there so we could keep it. We are already here pretty much today.

2) For `exp run`, stop using `dvc checkout --allow-missing` internally, as it is not what we want. Instead, always verify that we have all the dependencies we need intact, but if some are not `--allow-missing` flag would tell us that it is okay to skip those deps/outputs as a whole. The naming is not suitable here though, it should be something like `--allow-missing-deps` or something else, because it is clear that it is not the same `--allow-missing`. Even current descriptions for `--allow-missing` in `repro` and in `pull` have  very different meanings:

In pull:
```
--allow-missing - allows the command to succeed even if some files or directories are missing.
```

In repro:
```
--allow-missing - skip stages with no other changes than missing data.
```

In case of repro it seems like a very unfortunate oversight that we didn't pick a better name 🙁 
Thanks @efiop. It makes sense except I'm still not convinced about the creation of empty directories and how we report their status.

> This is causing that after a `dvc checkout`, dependencies previously reported as `deleted` are being reported as `modified`

So a checkout of a `deleted` directory will always change its status to `modified`, right? Do users really expect this change in status even if none of the files are available? Do they want to generate this empty directory structure? It feels more like an annoyance than a feature and not the same as being able to partially pull some files in a dataset.

> In case of repro it seems like a very unfortunate oversight that we didn't pick a better name 🙁

Can we add an alias and start directing people towards it? Do you have a suggestion? `--skip-missing`?
@dberenbaum It seems like we are mixing up checkout, status and `exp run` again there.

Regarding `dvc checkout` creating an empty directory/skeleton if no files are available - we are on the same page that it makes sense if there is at least 1 file created in there, right? And no files is an edge case that to me seems to make sense by induction logic from 1 file case. If we don't agree on that, then I guess alternative approach of only creating parent dirs if we can create a file is what we can agree on?

If you checkout and it is only able to create the skeleton or 1 file, but not complete directory - it should be reported as `modified`, this makes sense to me and is expected, because in the current form status just blindly compares what you have recorded in dvc file to what you actually have in the workspace. The only way to be smarter about it is going into the virtual directory territory and we are not there just yet (but it makes sense in the future, why not). `status` is also not granular, hence the `modified`, even though it is really `deleted` for all missing files in reality.

But the reason why we are discussing this is `dvc exp run` and @daavoo 's use case, where `dvc exp run` runs `dvc checkout --allow-missing` blindly which results in the pipeline trying to be ran with incomplete dependencies and that is a separate problem that should be solved by validating that our cache for those dependencies is complete and fully available.

Regarding the name, I don't have a particular suggestion. `--skip-missing` sounds better to me though.


> Regarding `dvc checkout` creating an empty directory/skeleton if no files are available - we are on the same page that it makes sense if there is at least 1 file created in there, right? And no files is an edge case that to me seems to make sense by induction logic from 1 file case. If we don't agree on that, then I guess alternative approach of only creating parent dirs if we can create a file is what we can agree on?

Right, I'm questioning the no files case specifically.
I'll take a final look, maybe we could just bring back old behavior one way or another (i think we can and that would improve the stats too, just going through stages of acceptance here 😄 ). Def want to have something around tomorrow.

Resolving patch:
diff --git a/dvc/repo/checkout.py b/dvc/repo/checkout.py
--- a/dvc/repo/checkout.py
+++ b/dvc/repo/checkout.py
@@ -161,19 +161,23 @@ def onerror(target, exc):
     if not force:
         _check_can_delete(diff.files_delete, new, self.root_dir, self.fs)
 
-    failed = []
+    failed = set()
+    out_paths = [out.fs_path for out in view.outs if out.use_cache and out.is_in_repo]
 
     def checkout_onerror(src_path, dest_path, _exc):
         logger.debug(
             "failed to create {%s} from {%s}", dest_path, src_path, exc_info=True
         )
-        failed.append(dest_path)
+
+        for out_path in out_paths:
+            if self.fs.path.isin_or_eq(dest_path, out_path):
+                failed.add(out_path)
 
     with Callback.as_tqdm_callback(
         unit="file",
         desc="Checkout",
     ) as cb:
-        changes = apply(
+        apply(
             diff,
             self.root_dir,
             self.fs,
@@ -184,15 +188,19 @@ def checkout_onerror(src_path, dest_path, _exc):
             **kwargs,
         )
 
-    out_changes = _build_out_changes(view, changes)
+    out_changes = _build_out_changes(view, diff.changes)
 
     typ_map = {ADD: "added", DELETE: "deleted", MODIFY: "modified"}
     for key, typ in out_changes.items():
         out_path = self.fs.path.join(self.root_dir, *key)
-        self.state.save_link(out_path, self.fs)
-        stats[typ_map[typ]].append(_fspath_dir(out_path))
+
+        if out_path in failed:
+            self.fs.remove(out_path, recursive=True)
+        else:
+            self.state.save_link(out_path, self.fs)
+            stats[typ_map[typ]].append(_fspath_dir(out_path))
 
     if failed and not allow_missing:
-        raise CheckoutError(failed, stats)
+        raise CheckoutError([relpath(out_path) for out_path in failed], stats)
 
     return stats
diff --git a/dvc/repo/worktree.py b/dvc/repo/worktree.py
--- a/dvc/repo/worktree.py
+++ b/dvc/repo/worktree.py
@@ -158,7 +158,7 @@ def push_worktree(
         ) as cb:
             cb.set_size(total)
             try:
-                stats = apply(
+                apply(
                     diff,
                     remote_obj.path,
                     remote_obj.fs,
@@ -166,7 +166,7 @@ def push_worktree(
                     latest_only=remote_obj.worktree,
                     jobs=jobs,
                 )
-                pushed += sum(len(changes) for changes in stats.values())
+                pushed += sum(len(changes) for changes in diff.changes.values())
             except VersioningNotSupported:
                 logger.exception("")
                 raise DvcException(


Test patch: