Repository: iterative/dvc

Issue:
Hitting `assert self.is_in_repo` in dvc push after 3.0 upgrade
# Bug Report

When trying to push a new file after upgradeing to 3.0 I'm hitting an unxexpected error, and the logs give no indication of what's going on. It's possible I've done something wrong, but at the very least the error message should be improved. I'm creating a PR to make a start on this.

The issue might be due to the dvc cache living in an external directory. I'm not exactly sure.  The traceback I'm getting with a new improved error message is:

```
2023-06-16 16:08:22,858 ERROR: unexpected error - Unexpected error
self.def_path='Drop7-MedianNoWinter10GSD_landcover_invar_scratch_split6_V65_epoch5_stepNone.pt'
self.repo=Repo: '/data/projects/smart/smart_expt_dvc'
self.repo.root_dir='/data/projects/smart/smart_expt_dvc'
self.fs_path='/data/projects/smart/smart_expt_dvc/models/fusion/Drop7-MedianNoWinter10GSD/packages/Drop7-MedianNoWinter10GSD_landcover_invar_scratch_split6_V65/Drop7-MedianNoWinter10GSD_landcover_invar_scratch_split6_V65_epoch5_stepNone.pt'
self.fs.path.realpath(self.fs_path)='/data/dvc-caches/smart_expt_dvc_cache/files/md5/c5/e09b8697020d86f077b0a24425f4cc'
Traceback (most recent call last):
  File "/home/local/KHQ/jon.crall/code/dvc/dvc/cli/__init__.py", line 209, in main
    ret = cmd.do_run()
  File "/home/local/KHQ/jon.crall/code/dvc/dvc/cli/command.py", line 26, in do_run
    return self.run()
  File "/home/local/KHQ/jon.crall/code/dvc/dvc/commands/data_sync.py", line 60, in run
    processed_files_count = self.repo.push(
  File "/home/local/KHQ/jon.crall/code/dvc/dvc/repo/__init__.py", line 64, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/local/KHQ/jon.crall/code/dvc/dvc/repo/push.py", line 63, in push
    used = self.used_objs(
  File "/home/local/KHQ/jon.crall/code/dvc/dvc/repo/__init__.py", line 491, in used_objs
    for odb, objs in self.index.used_objs(
  File "/home/local/KHQ/jon.crall/code/dvc/dvc/repo/index.py", line 480, in used_objs
    for odb, objs in stage.get_used_objs(
  File "/home/local/KHQ/jon.crall/code/dvc/dvc/stage/__init__.py", line 743, in get_used_objs
    for odb, objs in out.get_used_objs(*args, **kwargs).items():
  File "/home/local/KHQ/jon.crall/code/dvc/dvc/output.py", line 1154, in get_used_objs
    obj = self.get_obj(filter_info=kwargs.get("filter_info"))
  File "/home/local/KHQ/jon.crall/code/dvc/dvc/output.py", line 912, in get_obj
    obj = oload(self.cache, self.hash_info)
  File "/home/local/KHQ/jon.crall/code/dvc/dvc/output.py", line 529, in cache
    raise AssertionError('\n'.join([
AssertionError: Unexpected error
self.def_path='Drop7-MedianNoWinter10GSD_landcover_invar_scratch_split6_V65_epoch5_stepNone.pt'
self.repo=Repo: '/data/projects/smart/smart_expt_dvc'
self.repo.root_dir='/data/projects/smart/smart_expt_dvc'
self.fs_path='/data/projects/smart/smart_expt_dvc/models/fusion/Drop7-MedianNoWinter10GSD/packages/Drop7-MedianNoWinter10GSD_landcover_invar_scratch_split6_V65/Drop7-MedianNoWinter10GSD_landcover_invar_scratch_split6_V65_epoch5_stepNone.pt'
self.fs.path.realpath(self.fs_path)='/data/dvc-caches/smart_expt_dvc_cache/files/md5/c5/e09b8697020d86f077b0a24425f4cc'
```

This is indicating that `Drop7-MedianNoWinter10GSD_landcover_invar_scratch_split6_V65_epoch5_stepNone.pt'` is not in the repo path because the cache lives outside of the repo?

I'm not sure if this is a bug or if I messed something up somehow. I'll note that have found a workaround. If I unprotect the file that my error message reports, it seems to be able to get to the next one. I had to do this for several files (although not every file in the repo, thankfully), but as eventually able to get a push to work. 

### Reproduce
I'm not sure how to reproduce the bug at this time. 

### Environment information

<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc doctor`:**

```console
DVC version: 3.0.1.dev8+gfd73b389
---------------------------------
Platform: Python 3.10.5 on Linux-5.15.0-69-generic-x86_64-with-glibc2.31
Subprojects:
	dvc_data = 1.11.0
	dvc_objects = 0.23.0
	dvc_render = 0.5.3
	dvc_task = 0.3.0
	scmrepo = 1.0.3
Supports:
	azure (adlfs = 2023.4.0, knack = 0.9.0, azure-identity = 1.10.0),
	gdrive (pydrive2 = 1.15.4),
	gs (gcsfs = 2023.6.0),
	hdfs (fsspec = 2023.6.0, pyarrow = 8.0.0),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.5.2),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.5.2),
	oss (ossfs = 2021.8.0),
	s3 (s3fs = 2023.6.0, boto3 = 1.26.76),
	ssh (sshfs = 2023.4.1),
	webdav (webdav4 = 0.9.7),
	webdavs (webdav4 = 0.9.7),
	webhdfs (fsspec = 2023.6.0)
Config:
	Global: /home/local/KHQ/jon.crall/.config/dvc
	System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/md0
Caches: local
Remotes: s3, ssh
Workspace directory: ext4 on /dev/md0
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/a0457bedea4e28e1518387be0506369e
```



Discussion:
I think this is likely a bug with out-of-repo cache directories (i.e. shared caches between multiple different DVC repos on the same machine). I moved my external cache into `<repo>/.dvc/cache`, and then symlinked the old location to the new location, and the errors went away. 
Yeah, looks like it is a symlink that confuses `is_in_repo` implementation. Looks like if we don't `realpath` it - it works fine. Preparing a patch to see if nothing else will get triggered.

Resolving patch:
diff --git a/dvc/output.py b/dvc/output.py
--- a/dvc/output.py
+++ b/dvc/output.py
@@ -510,7 +510,7 @@ def is_in_repo(self):
             return False
 
         return self.repo and self.fs.path.isin(
-            self.fs.path.realpath(self.fs_path),
+            self.fs_path,
             self.repo.root_dir,
         )
 


Test patch: