Repository: iterative/dvc

Issue:
'.../plots/metrics/metrics' - file type error
Using the `dvclive.lightning.DVCLiveLogger` with `dir="data/train/dvclive"`, I can see the plots of the metrics in the "Plots" view of the VS Code DVC extension. But at the top where it shows the selected experiments, it displays an error with message:

> 'data/train/dvclive/plots/metrics/metrics' - file type error
Only JSON, YAML, CSV and TSV formats are supported.

The summary metrics are stored in `data/train/dvclive/metrics.json` and the `.tsv` files of the logged metrics are in `data/train/dvclive/plots/metrics/`. There exists no file named `metrics` without extension inside this folder.


Discussion:
@aschuh-hf can you please post a screenshot of the issue. Does `plots diff --json` return any errors for those revisions? What version of DVC are you using?
After working more on my repo, some PRs pushed and merged, etc. I don't actually see this error any  longer. But also because I changed the `train` stage in my `dvc.yaml` and updated the DVC extension.

I am using DVC 3.5.1 and was using the latest DVC extension as of two days ago (just now updated to v1.0.19.

As I may not be able to help reproduce this error, we could close this issue for now until it pops up again.
I deleted the `train` stage from my `dvc.lock` file and removed the stage outputs from my working directory. After running `dvc exp run train` anew with the modified stage in `dvc.yaml`, I see the error again.

<img width="617" alt="Screen Shot 2023-07-18 at 12 41 54 AM" src="https://github.com/iterative/vscode-dvc/assets/77496589/e49f10bd-6d97-4b8c-85fe-c13110d95848">

```
DVC version: 3.5.1 (conda)
--------------------------
Platform: Python 3.10.6 on Linux-3.10.0-1127.8.2.el7.x86_64-x86_64-with-glibc2.17
Subprojects:
        dvc_data = 2.5.0
        dvc_objects = 0.23.0
        dvc_render = 0.5.3
        dvc_task = 0.3.0
        scmrepo = 1.0.4
Supports:
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2023.6.0, boto3 = 1.26.76)
Config:
        Global: /home/aschuh/.config/dvc
        System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: xfs on /dev/sda1
Caches: local
Remotes: s3, s3
Workspace directory: xfs on /dev/sda1
Repo: dvc (subdir), git
Repo.site_cache_dir: /var/tmp/dvc/repo/764b1d03f8b416f032df6c50c3809e7f
```
Screenshot of DVCLiveLogger output files under `data/train/...`:

<img width="279" alt="Screen Shot 2023-07-18 at 12 44 59 AM" src="https://github.com/iterative/vscode-dvc/assets/77496589/e3e61a26-f150-442d-8f87-4e2c3cad6d21">

@aschuh-hf do you mean the `dvc.yaml`? Direct changes to the `dvc.lock` are not advised.
I edited the `dvc.lock` to remove the stage. I thought one choice for YAML and the way DVC works was to make it quite straightforward to edit these files manually (e.g., I never used `dvc stage add`, `dvc run`, etc. in my few years of using DVC).
Once `dvc exp run` completed (now), the `dvc.lock` file was updated with the new `train` stage entry and the error is gone.
But I see no reason why it would be displayed for a fresh project when `dvc.lock` would not even exist yet.
You should remove the stage from the `dvc.yaml` if you do not want it to run again. The `dvc.lock` is automatically maintained by dvc and manually making changes will result in unexpected errors.

Take a look at these docs: 
- https://dvc.org/doc/user-guide/project-structure/dvcyaml-files
- https://dvc.org/doc/user-guide/project-structure/dvcyaml-files#dvclock-file
The error I am reporting is unlikely caused by me editing the `dvc.lock` file because I would get the exact same behavior if I would simply use `git restore dvc.lock` after running `dvc exp run` for the first time (with `core.autostage=false`). All I did is revert the change `dvc exp run` does after completion.
`git restore dvc.lock` is equivalent to manually updating the `dvc.lock` file.
> git restore dvc.lock is equivalent to manually updating the dvc.lock file.

Does `git checkout` not also touch the `dvc.lock` file? The file is tracked by Git, not DVC, so I should be allowed to use `git` commands such as `git restore` to modify its contents. Besides, there is no "undo `dvc exp run`" command in the list of DVC commands. Because IMHO that command is `git restore` / `git reset`.

Would it make sense to focus on the error rather than how it was elicited? Clearly the error message refers to a file path that never existed (`data/train/plots/metrics/metrics`), neither on the file system nor in my `dvc.yaml` file (I also did not make any such change to `dvc.lock`). That should hint that there is indeed something awry.
@aschuh-hf If you run `dvc plots diff`, do you get the same error?
Yes, `dvc plots diff` also reports this error. Though both in the CLI output and VS Code Extension "Plots" view the path is now "data/train/plots/metrics" (which is the correct path) after I merged some code and run some trial experiments to test my DVC + Ray + AWS setup.

```
$ dvc plots diff --json
DVC failed to load some plots for following revisions: 'workspace, HEAD'.                                                                                                                                                                                                                                                                                                                         
{
  "errors": [
    {
      "name": "data/train/plots/metrics",
      "rev": "workspace",
      "source": "data/train/plots/metrics",
      "type": "PlotMetricTypeError",
      "msg": "'data/train/plots/metrics' - file type error\nOnly JSON, YAML, CSV and TSV formats are supported."
    },
    {
      "name": "data/train/plots/metrics",
      "rev": "HEAD",
      "source": "data/train/plots/metrics",
      "type": "PlotMetricTypeError",
      "msg": "'data/train/plots/metrics' - file type error\nOnly JSON, YAML, CSV and TSV formats are supported."
    }
  ],
  "data": {
    "data/train/plots/metrics": []
  }
}
```

This was after I started a new run with `dvc exp run` (using the extension's "Run Experiment" button). My base commit has no valid `train` stage output as I just set up the experiment pipeline with DVCLive.
Thanks @aschuh-hf! I'm going to transfer this to DVC then since it doesn't seem like the issue is specific to the VS Code extension. Do you know how to reproduce the error by any chance?
I have a `dvc.yaml` file with a `train` stage with a `plots` entry.

```
stages:
  train:
    cmd: python train.py
    params:
    - data
    - loss
    - model
    - train
    deps:
    - train.py
    outs:
    - data/train/state
    metrics:
    - data/train/metrics.json
    plots:
    - data/train/plots/metrics:
        x: step
```

This entry is returned by `dvc.repo.plots._collect_plots` with key `fs_path` equal to `data/train/plots/metrics` as this is the folder specified in my `dvc.yaml:train.plots` entry. This `fs_path` key is checked against the expected file name extensions in the `dvc.repo.plots.parse` function which raises this error.

<img width="346" alt="Screen Shot 2023-07-18 at 11 08 04 PM" src="https://github.com/iterative/dvc/assets/77496589/87c0e310-a5e4-4e6b-9e01-60407ba92737">

https://github.com/iterative/dvc/blob/ea5f7d40c131aa949f307e31e14e5d99024c20da/dvc/repo/plots/__init__.py#L323-L341

https://github.com/iterative/dvc/blob/ea5f7d40c131aa949f307e31e14e5d99024c20da/dvc/repo/plots/__init__.py#L182-L188
Output of `dvc.repo.plots.Plots.collect()` when `data/train/plots/metrics` directory does not exist:

```
[{'workspace': {'definitions': {'data': {'': {'data': {'data/train/plots/metrics': {'x': 'step'}}}}},
   'sources': {'data': {'data/train/plots/metrics': {'props': {'x': 'step'},
      'data_source': functools.partial(<function error_handler.<locals>.wrapper at 0x7ff81e2a9000>, <dvc.fs.dvc.DVCFileSystem object at 0x7ff834102cb0>, 'data/train/plots/metrics', props={'x': 'step'}, onerror=None)}}}}}]
```

Once the directory exists (e.g., written by DVCLive), the `Plots.collect()` method lists individual `.tsv` files instead.

Maybe this `if`-`else` in `unpack_if_dir` has something to do with the issue:
https://github.com/iterative/dvc/blob/ea5f7d40c131aa949f307e31e14e5d99024c20da/dvc/repo/plots/__init__.py#L536-L539
Thanks @aschuh-hf! I can confirm that with the `dvc.yaml` you shared, I get the same error from `dvc plots show/diff`. It looks like for some reason it duplicates the basename of the path when a plot stage output points to a directory path.
It only happens when `dvc.lock` is missing, in which case `dvc_data.fs.DataFileSystem.ls(data/train/plots/metrics)` does not know if `data/train/plots/metrics` is a directory or file [here](https://github.com/iterative/dvc-data/blob/efa543a2b600eb93461ad55234d500d0ba0df370/src/dvc_data/fs.py#L97-L102). It assumes it is a file and `dvc_data.fs.DataFileSystem.ls("data/train/plots/metrics")` returns itself. `LocalFileSystem.ls(path)` only returns subpaths and fails if `path` is a file, so not sure if it's expected that `dvc_data.fs.DataFileSystem.ls(path)` returns `path` itself. cc @efiop 

Resolving patch:
diff --git a/dvc/fs/dvc.py b/dvc/fs/dvc.py
--- a/dvc/fs/dvc.py
+++ b/dvc/fs/dvc.py
@@ -288,7 +288,7 @@ def ls(  # pylint: disable=arguments-differ # noqa: C901
         dvc_infos = {}
         if dvc_fs:
             dvc_path = _get_dvc_path(dvc_fs, subkey)
-            with suppress(FileNotFoundError):
+            with suppress(FileNotFoundError, NotADirectoryError):
                 for info in dvc_fs.ls(dvc_path, detail=True):
                     dvc_infos[dvc_fs.path.name(info["name"])] = info
             dvc_exists = bool(dvc_infos) or dvc_fs.exists(dvc_path)


Test patch:
diff --git a/tests/unit/fs/test_dvc.py b/tests/unit/fs/test_dvc.py
--- a/tests/unit/fs/test_dvc.py
+++ b/tests/unit/fs/test_dvc.py
@@ -162,6 +162,16 @@ def test_isdir_mixed(tmp_dir, dvc):
     assert not fs.isfile("dir")
 
 
+def test_ls_dirty(tmp_dir, dvc):
+    tmp_dir.dvc_gen({"data": "data"})
+    (tmp_dir / "data").unlink()
+
+    tmp_dir.gen({"data": {"foo": "foo", "bar": "bar"}})
+
+    fs = DVCFileSystem(repo=dvc)
+    assert set(fs.ls("data")) == {"data/foo", "data/bar"}
+
+
 @pytest.mark.parametrize(
     "dvcfiles,extra_expected",
     [