Repository: iterative/dvc

Issue:
cloud versioning: Can't update stage created with `import-url --version-aware <directory>` 
```console
$ dvc import-url --version-aware s3://diglesia-bucket/data
Importing 's3://diglesia-bucket/data' -> 'data'                       
                                                                                                                  
To track the changes with git, run:                                                                               

        git add data.dvc

To enable auto staging, run:

        dvc config core.autostage true
$ cat data.dvc
md5: 2fdd3fb9f27b4d264814b96d2f25e8b4
frozen: true
deps:
- md5: a042035f7fb17fc84a42e33787ae4b6a.dir
  size: 13
  nfiles: 3
  path: s3://diglesia-bucket/data
  files:
  - size: 5
    version_id: zMz0GJcKV7joJPsRsKQxQ_7THawD67Id
    etag: 042967ff088f7f436015fbebdcad1f28
    relpath: bar
  - size: 4
    version_id: 5fieW.ptGcMsBY4QRdF6jlebNNGp0wGB
    etag: d3b07384d113edec49eaa6238ad5ff00
    relpath: foo
  - size: 4
    version_id: tq_sYdOm8tSXXLAiyxm0syJwHqSvw_NR
    etag: 7361528e901ca2f2f1952a68ad242d79
    relpath: goo
outs:
- md5: b5c587798f2c0f599ef935244b290bf1.dir
  size: 13
  nfiles: 3
  path: data
  push: false
```

Calling `dvc update` makes it lose the cloud versioning info.

```console
$ dvc update data.dvc
$ cat data.dvc
md5: f2e472bf6cd15efaf23e1268e6843d06
frozen: true
deps:
- md5: a042035f7fb17fc84a42e33787ae4b6a.dir
  size: 13
  nfiles: 3
  path: s3://diglesia-bucket/data
outs:
- md5: b5c587798f2c0f599ef935244b290bf1.dir
  size: 13
  nfiles: 3
  path: data
  push: false
```

It looks like `dvc update` only applies worktree updates for non-imported stages  (i.e. `dvc add data && dvc push`):
https://github.com/iterative/dvc/blob/f1c9e9365b1886758099c9f8e604817a33d767e4/dvc/repo/update.py#L48-L52



Discussion:
@dberenbaum Not sure if this ever worked. What would be the correct workflow to get updates for a stage created with `import-url --version-aware`?


Looks like a bug to me. `dvc update` should "remember" if it's a version-aware import.
> It looks like dvc update only applies worktree updates for non-imported stages (i.e. dvc add data && dvc push):

It's not supposed to apply a worktree update in this case. `import_stages` is supposed to handle the version-aware case for import-url stages.

I know `import-url`+`update` worked for individual files, but I don't remember if we tested `update` with directories after the `files` in deps change (https://github.com/iterative/dvc/pull/8528). The issue is probably that we aren't setting `dep.fs.version_aware` when we re-load an import stage from the .dvc file.

For files it should work as-is because the deps url will have a version ID in the URL (and we automatically enable `fs.version_aware` based on the versioned URL). But for dirs `s3://diglesia-bucket/data` will get handled as an non-version-aware filesystem.

`Output.__init__()` probably needs to be updated so that we explicitly set `version_aware=True` in the `fs_config` when `files` is set (the bug here is for deps, but this really applies to outs as well)
https://github.com/iterative/dvc/blob/f1c9e9365b1886758099c9f8e604817a33d767e4/dvc/output.py#L303

I think after that fix the existing `update` behavior should work properly.
> Output.__init__() probably needs to be updated so that we explicitly set version_aware=True in the fs_config when files is set (the bug here is for deps, but this really applies to outs as well)

Thinking about it more, it should be specific to `Dependency.__init__()`. For outputs it doesn't work because `out.fs` is local/repo FS (and should not be version aware) even when we have `files` (from pushing `out` to a remote)

Resolving patch:
diff --git a/dvc/output.py b/dvc/output.py
--- a/dvc/output.py
+++ b/dvc/output.py
@@ -327,7 +327,9 @@ def __init__(  # noqa: C901
         meta = Meta.from_dict(info or {})
         # NOTE: when version_aware is not passed into get_cloud_fs, it will be
         # set based on whether or not path is versioned
-        fs_kwargs = {"version_aware": True} if meta.version_id else {}
+        fs_kwargs = {}
+        if meta.version_id or files:
+            fs_kwargs["version_aware"] = True
 
         if fs_config is not None:
             fs_kwargs.update(**fs_config)


Test patch:
diff --git a/tests/unit/output/test_output.py b/tests/unit/output/test_output.py
--- a/tests/unit/output/test_output.py
+++ b/tests/unit/output/test_output.py
@@ -176,3 +176,30 @@ def test_dumpd_cloud_versioning_dir(mocker):
 
     dumpd = out.dumpd()
     assert dumpd == {"path": "path", "files": files}
+
+
+def test_version_aware_is_set_based_on_files(mocker):
+    import dvc.fs as dvc_fs
+
+    get_fs_config = mocker.spy(dvc_fs, "get_fs_config")
+
+    stage = mocker.MagicMock()
+    stage.repo.fs.version_aware = False
+    stage.repo.fs.PARAM_CHECKSUM = "etag"
+    files = [
+        {
+            "size": 3,
+            "version_id": "WYRG4BglP7pD.gEoJP6a4AqOhl.FRA.h",
+            "etag": "acbd18db4cc2f85cedef654fccc4a4d8",
+            "md5": "acbd18db4cc2f85cedef654fccc4a4d8",
+            "relpath": "bar",
+        }
+    ]
+    Output(stage, "path", files=files)
+    # version_aware is passed as `True` if `files` is present`.
+    # This will be intentionally ignored in filesystems that don't handle it
+    # in `_prepare_credentials`.
+    assert get_fs_config.call_args_list[0][1] == {
+        "url": "path",
+        "version_aware": True,
+    }