Repository: iterative/dvc

Issue:
`dvc.api.DVCFileSystem` not working with `rev=...`
# Bug Report
## Description
I am unable to use the `dvc.api.DVCFileSystem(".", rev="da6..")` if I set `ref=...`.


### Reproduce

```bash
git init
dvc init

echo "something" > deps.txt
dvc stage add -n mystage -d deps.txt -o outs.txt "echo '$(date)' > outs.txt"
dvc repro
git add .
git commit -m "..."

echo " else" >> deps.txt
dvc repro
git add .
git commit -m "..."
```

then use the following script `main.py`:

```python
import dvc.api
import sys

rev = sys.argv[1] if len(sys.argv) == 2 else None

fs = dvc.api.DVCFileSystem(".", rev=rev)
for file in fs.find("/", detail=False, dvc_only=True):
    print(file)
    print(fs.read_text(file))
```

`python main.py` works fine for me but `python main.py <any revision from git log>` yields:

```python
/outs.txt
Traceback (most recent call last):
  File "/ssd/fzills/Examples/dvc_fs_bug/main.py", line 11, in <module>
    print(fs.read_text(file))
  File "/tikhome/fzills/miniconda3/envs/znlib/lib/python3.10/site-packages/fsspec/spec.py", line 674, in read_text
    with self.open(
  File "/tikhome/fzills/miniconda3/envs/znlib/lib/python3.10/site-packages/fsspec/spec.py", line 1094, in open
    self.open(
  File "/tikhome/fzills/miniconda3/envs/znlib/lib/python3.10/site-packages/fsspec/spec.py", line 1106, in open
    f = self._open(
  File "/tikhome/fzills/miniconda3/envs/znlib/lib/python3.10/site-packages/dvc/fs/dvc.py", line 264, in _open
    return dvc_fs.open(dvc_path, mode=mode)
  File "/tikhome/fzills/miniconda3/envs/znlib/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 197, in open
    return self.fs.open(path, mode=mode, **kwargs)
  File "/tikhome/fzills/miniconda3/envs/znlib/lib/python3.10/site-packages/dvc_data/fs.py", line 70, in open
    fs, fspath = self._get_fs_path(path, **kwargs)
  File "/tikhome/fzills/miniconda3/envs/znlib/lib/python3.10/site-packages/dvc_data/fs.py", line 62, in _get_fs_path
    raise FileNotFoundError
FileNotFoundError

```

### Environment information

<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc doctor`:**

```yaml
$ dvc doctor
DVC version: 2.38.1 (pip)
---------------------------------
Platform: Python 3.10.6 on Linux-5.4.0-135-generic-x86_64-with-glibc2.31
Subprojects:
        dvc_data = 0.28.4
        dvc_objects = 0.14.0
        dvc_render = 0.0.15
        dvc_task = 0.1.8
        dvclive = 1.2.2
        scmrepo = 0.1.4
Supports:
        http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2022.11.0, boto3 = 1.24.59)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda6
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/sda6
Repo: dvc, git
```



Discussion:
Hi, @rlamy I would like to work on this issue. I have set up everything and reproduced the error. So far I think it is a file full path-related problem. 
@karanjakhar I'm investigating this as well, but feel free to send a PR, as I probably won't have time to fix this before the holidays.

So the issue happens in [`dvc_data.fs.DataFileSystem._get_fs_path()`](https://github.com/iterative/dvc-data/blob/e0d19abd5d25525d8d4bc0068c9ac748f3c2aad6/src/dvc_data/fs.py#L56). In that function, `entry.odb.path` has the wrong value `/.dvc/cache`, which is interpreted as an absolute path (note the leading slash) and causes the cache lookup to fail.

That wrong value comes from `Repo.__init__` and specifically the [initialisation of the ODBManager](https://github.com/iterative/dvc/blob/6fc0a6ff98aa41e4a0fcc2120413132ab2ba76a1/dvc/odbmgr.py#L27). When the `rev` parameter is used, the repo uses `GitFileSystem` and there is a confusion between `repo.config["cache"]["dir"]` being `/.dvc/cache` (which seems correct for gitfs) and `ODBManager` using that value _verbatim_ as the path to the cache, even though it uses localfs. I'm not sure where exactly is the error in that process, but, in any case, the following code is enough to demonstrate the issue:
```python
from dvc.repo import Repo
repo = Repo(rev="HEAD")
odb = repo.odb._odb["repo"]
print(odb.fs, odb.path)
```
Result:
```
<dvc_objects.fs.local.LocalFileSystem object at 0x102bdf0a0> /.dvc/cache
```
@rlamy why `/.dvc/cache` is correct for gitfs?
@rlamy so far I can't find any bug or problem with the process, I think we need to pass the full path in the initialization of ODBManger in case the rev parameter is used. 
@rlamy Should FileSystem be gitfs or path should be absolute?
@karanjakhar sorry for the delay, got lost in notifications.

> @rlamy why /.dvc/cache is correct for gitfs?

He meant that it is "formally" correct, because the root of the repo in gitfs is `/`. But obviously that is not correct at the same time because we can't have cache right in the gitfs.

> @rlamy so far I can't find any bug or problem with the process, I think we need to pass the full path in the initialization of ODBManger in case the rev parameter is used.

That sounds correct.

> @rlamy Should FileSystem be gitfs or path should be absolute?

gitfs. From DvcFileSystem perspective it's root is `/` and it is completely detached from the local filesystem that it resides on.
@efiop So when "rev" is passed. odb.fs should be gitfs right?
@karanjakhar Ah, looks like I misunderstood you. odb.fs should be localfs, not gitfs and odb.path should be an absolute local path, not gitfs path.
As odb is renamed to cache. 
https://github.com/iterative/dvc/pull/8987#issue-1576763859

To reproduce this issue now:
```python
from dvc.repo import Repo
repo = Repo(rev="HEAD")
odb = repo.cache._odb["repo"]
print(odb.fs, odb.path)
```

Resolving patch:
diff --git a/dvc/cachemgr.py b/dvc/cachemgr.py
--- a/dvc/cachemgr.py
+++ b/dvc/cachemgr.py
@@ -1,3 +1,5 @@
+import os
+
 from dvc.fs import GitFileSystem, Schemes
 from dvc_data.hashfile.db import get_odb
 
@@ -28,16 +30,34 @@ def __init__(self, repo):
         self.config = config = repo.config["cache"]
         self._odb = {}
 
+        default = None
+        if repo and repo.dvc_dir:
+            if isinstance(repo.fs, GitFileSystem):
+                relparts = ()
+                if repo.root_dir != "/":
+                    # subrepo
+                    relparts = repo.fs.path.relparts(repo.root_dir, "/")
+                dvc_dir = os.path.join(
+                    repo.scm.root_dir,
+                    *relparts,
+                    repo.DVC_DIR,
+                )
+                if os.path.exists(dvc_dir):
+                    default = os.path.join(dvc_dir, self.CACHE_DIR)
+            else:
+                default = repo.fs.path.join(repo.dvc_dir, self.CACHE_DIR)
+
         local = config.get("local")
 
         if local:
             settings = {"name": local}
-        elif "dir" not in config:
+        elif "dir" not in config and not default:
             settings = None
         else:
             from dvc.config_schema import LOCAL_COMMON
 
-            settings = {"url": config["dir"]}
+            url = config.get("dir") or default
+            settings = {"url": url}
             for opt in LOCAL_COMMON:
                 if opt in config:
                     settings[str(opt)] = config.get(opt)
diff --git a/dvc/commands/cache.py b/dvc/commands/cache.py
--- a/dvc/commands/cache.py
+++ b/dvc/commands/cache.py
@@ -1,4 +1,5 @@
 import argparse
+import os
 
 from dvc.cli import completion
 from dvc.cli.utils import append_doc_link, fix_subparsers
@@ -9,13 +10,21 @@
 class CmdCacheDir(CmdConfig):
     def run(self):
         if self.args.value is None and not self.args.unset:
+            from dvc.config import ConfigError
+
             if self.args.level:
                 conf = self.config.read(level=self.args.level)
             else:
                 # Use merged config with default values
                 conf = self.config
-            self._check(conf, False, "cache", "dir")
-            ui.write(conf["cache"]["dir"])
+            try:
+                self._check(conf, False, "cache", "dir")
+                path = conf["cache"]["dir"]
+            except ConfigError:
+                if not self.config.dvc_dir or self.args.level:
+                    raise
+                path = os.path.join(self.config.dvc_dir, "cache")
+            ui.write(path)
             return 0
         with self.config.edit(level=self.args.level) as conf:
             if self.args.unset:
diff --git a/dvc/config.py b/dvc/config.py
--- a/dvc/config.py
+++ b/dvc/config.py
@@ -170,10 +170,6 @@ def load(self, validate: bool = True, config: Optional["DictStrAny"] = None):
         self.clear()
         self.update(conf)
 
-        # Add resolved default cache.dir
-        if not self["cache"].get("dir") and self.dvc_dir:
-            self["cache"]["dir"] = os.path.join(self.dvc_dir, "cache")
-
     def _get_fs(self, level):
         # NOTE: this might be a Gitfs, which doesn't see things outside of
         # the repo.


Test patch: