Repository: iterative/dvc

Issue:
Allow to customize `site_cache_dir`
TLDR. In some scenarios, like using DVC inside a Kubernetes pod, it is convenient/required.

More Context: https://discord.com/channels/485586884165107732/485596304961962003/1089965621837582426

> I want to use dvc in a Kubernetes cluster, where the created pods get a read-only /var partition. Therefore I am getting the ERROR: unexpected error - [Errno 30] Read-only file system: '/var/tmp/dvc'

Also related: https://discord.com/channels/485586884165107732/563406153334128681/1090392828888236105

> We ran into the /var/tmp/dvc permissions issue with v2.49 which apparently is fixed with v2.50+. (Just waiting for somebody else to turn up so that I can ask them to test it.) But this has tickled my curiosity -- what is the "site cache"? And what does it let us do? 


Discussion:
cc @efiop 
We ran into #9184 with v2.49 which apparently is fixed with v2.50+. (Just waiting for somebody else to turn up so that I can ask them to test it.) 
But this has tickled my curiosity -- what is the "site cache"? And what does it let us do? 
I couldn't find anything in the docs ..
I might add that cleaning up `/var/tmp/dvc` created by multiple team-members on multiple nodes of a HPC (each with their own `/var`) without sudo rights has been a pain in the proverbial. 

Having the ability to specify a path on a shared volume is essential in this context.
@johnyaku 2.50+ will create that directory with correct rights so that other users can access it too.

Do not put it on a shared volume, this is meant to be put on a high-performance normal filesystem and putting it on nfs will make everything slower and might even hang depending on what shared fs you are using (e.g. nfs v3 has locking messed up).
Regarding customizing site_cache_dir, I think we should just fallback to `/tmp` (or maybe some other location) instead if we catch EROFS and not provide a config option for now. We still could provide a config option, but the goal is to not have to do any manual configuration in normal cases and have normal scenarios work out of the box.
> We still could provide a config option, but the goal is to not have to do any manual configuration in normal cases and have normal scenarios work out of the box.

I think the current request is one of those cases. I vote for just adding it so at least people outside those normal scenarios have a workaround. People in "normal" scenarios would just not notice it.

We should probably extend https://github.com/iterative/dvc.org/issues/4385 to not only drop the old configs but also include a minimal explanation of the new `site_cache_dir`.
Thanks @efiop and @daavoo. I await the cliff notes to figure out how to best apply it to our situation. For the record our file system is `panfs`, which is reasonably high performance but I'm not sure if it would still be subject to locking.

Not having sudo rights and the fact that the `.db` files are read-only makes the clean up a bit of work, but we'll get there! Thanks for the quick response as always!
@johnyaku You shouldn't need sudo rights. The shared directory will be created with 777 rights, so that everyone has access to it.
We were running v2.49 for about a week, which used a different umask. Mostly people were working on different nodes of the cluster, and so the permissions didn't collide until a couple of days ago.

I _could_ just get everybody to `chmod 0777` everything, but it is complicated by the fact that my default umask allows group members to write to my directories, so recursive `chmod` runs into ownership issues. I figured it would be easier to nuke `/var/tmp/dvc` and start again. Just a pity that there is one on every node of the cluster ...
@johnyaku chmod or nuke will work equally well. Sorry for the inconvenience 🙁 
@daavoo Did some additional research and having `/var` read-only is indeed pretty uncommon and judging from the discord thread it is just their particular configuration being overly strict with read-only stuff and it sounds like they can easilly adjust it. It might be easier to adjust that than to bother configuring a special dvc option. Waiting for a response from the OP for now.
re:
> Regarding customizing site_cache_dir, I think we should just fallback to /tmp (or maybe some other location) instead if we catch EROFS and not provide a config option for now. We still could provide a config option, but the goal is to not have to do any manual configuration in normal cases and have normal scenarios work out of the box.

While `tmp` seems like a reasonable place to fallback to, isn't giving a config/ escape-hatch for users to configure this directly simpler, safer and covers all cases?

Falling back to `/tmp` is still very opinionated, might not work for everyone (spread files outside the workspace), `/tmp` can be memory mapped, often wiped on reboot for example, or shared between containers... it might matter or not.
But there's still an inherent risk in such a choice, especially outside the workspace. Not sure if there's a one-size-fits-all here


@omesser I'm not against adding an option per se. It would still be useful for debugging at the very least. But if we can make this work out of the box with a similar level of effort - we should.

At this point I'm just trying to get more info, because having a read-only `/var` is pretty unusual and will probably mess with a lot of other tools and services.
> But if we can make this work out of the box with a similar level of effort - we should.

Completely agree on that.
I'm only suggesting that if this is a long process (collecting more inputs? waiting for more users to hit this?) then an escape hatch will give quick relief to suffering users in the meantime

I am assuming here that it's a very small and safe effort of course. let me know if it's not the case
@omesser Yeah, no problem adding the option. I was about to do that myself when initially saw this issue. Might still end up doing that, as having it just in case is useful (but I would not recommend it in docs as if you need it - you are probably doing something wrong). So far we are down to 1 user having problems with it. 
@johnyaku Regarding panfs, it is a network-attached storage and `site_cache_dir` should not reside on it. Please do not share that dir between nodes, it is specifically meant to be site-specific for optimal performance. 
I've been following along as I ran into this issue in my organization as well, however it seems to be an outgrowth of a persistent issue of sharing repo's between users not being very well supported. Being able to configure an alternative `site_cache_dir` would help me facilitate this practice (though I will likely need to continue adding hooks to `chmod g+w` to everything so that tmp dbs don't cause DVC operations to blow up between users. If there is a broader feature request/discussion open somewhere about supporting git-like `core.share=group` configuration in DVC I would add this issue to it. (I went looking but didn't find anything that matched, but I might have just missed it. I'm happy to start such a request/discussion if seems like a good idea)
@joegrover How do you work with it so far? You have a single git/dvc repo that multiple users use at the same time? How is that working out for you so far? We've seen this before and usually it gets very messy even with git alone and so I personally tend to recommend each user having their own repo but use a shared cache or even better shared local remote as that is easiest to manage for multiple users with no risk that one user will do something bad to a shared repo.
@efiop In this specific use case we are tightly controlling the way users are interacting with both git and dvc, with the idea being that the end user isn't even aware that they are interacting with a version control system at all. This also allows us to feel confident in blowing away and re-cloning a repo if it gets into a funky local state. It definitely has it's drawbacks, but we are also small enough that coordinating work between a few people is easier than trying to build for a potentially infinite set of users.
@joegrover Makes sense! Sounds like us making those internal files accessible by all users (e.g. 0o777 on those internal sqlite databases) should fix it for you and continue working without headaches. Or am I missing something in your explanation?

The fix could be, as you've suggested, for us to just add a sticky bit, but I would rather just apply desired permissions to the files we create (good behaviour would be to indeed have some kind of option).

Or we could take into account uid when creating cache keys, so that for a particular repo those internal cached files are not shared between users at all. This is very simple and easy to make work by default without any new configuration. If you don't have thousands of users accessing the same repository and you are fine with a trade-off that those cached files would need to be built once for every user - I would recommend going with this.
@efiop Oh yeah, DBs with 0o777 should definitely do the trick. Is that the proposed solution or already in place? Sticky-bit stuff always gives me a headache so I'm happy to not have to deal with it. I like the user specific cache idea, though some of the files we are tracking are in the 20+GB range and that's probably not going to slow down. I don't _think_ that you are saying the cache would contain a copy of the tracked data file per user, but I just wanted to make sure.
@joegrover By cache I mean lightweight internal files that don't contain your data at all. Your dvc cache stays in `.dvc/cache` or wherever you've configured it to be. `site_cache_dir` is used for optimization, for internal files (we basically cache the structure of your project without data, so that we can quickly navigate it in `dvc ls` and other commands and not have to recollect it all the time). Similar to `.dvc/tmp`.

Alight, if user-specific cache works for you, then that's the easiest way to go about it without having to introduce any new options and not complicating any stuff. Just need to include `uid` when generating the token in https://github.com/iterative/dvc/blob/a6c89cac8d82b6836acd068d0d02c26b79c851f5/dvc/repo/__init__.py#L586
Thanks all for the consideration that has gone into this. And thanks also for the insight into what the `site_cache_dir` actually does!

For clarity, `/var` is writeable on all nodes for us, but the `.db` files created by v2.49 were not. If v2.50+ creates these files with 0777 then I think we will be fine once we find and clean up the old ones.

Now that I have a bit of an idea about what `site_cache_dir` does I suspect that have multiple copies over multiple nodes is not really going to be an issue. Sure, there might be some redundancy but if the idea is to improve performace with fast, local maps of the directory structure then I think is worth it.

Point about `panfs` noted. Thanks @efiop 
@johnyaku 

> For clarity, /var is writeable on all nodes for us, but the .db files created by v2.49 were not. If v2.50+ creates these files with 0777 then I think we will be fine once we find and clean up the old ones.

It doesn't create 0o777 db files, but only parent directories. The bug was that the dirs (e.g. /var/tmp/dvc) was not 0o777. Are db files also a problem for you? If so, then suggested fix with including `uid` should also be a solution for you.


We haven't directly experienced problems with _using_ the `.db` files. The problem was deleting them. Not impossible, just an extra hoop to herd cats thru.
@johnyaku Could you elaborate why you need to delete them manually?
I think the suggestion somewhere was to delete `/var/tmp/dvc`. We couldn't do this with read-only `.db` files in there. 

I know the alternative was to `chmod 0777` recursively. But my umask allows for others to write to directories that I create, and so there were subdirectories in there that I didn't own, and so couldn't `chmod`. I could have dealt with this one directory at a time, but it seemed easier to just nuke it all and start again.
@johnyaku I think in your case we are mixing up different topics. You seem to be simply affected by the fixed bug and nuking `/var/tmp/dvc` should fix it with newer dvc versions.

Resolving patch:
diff --git a/dvc/config_schema.py b/dvc/config_schema.py
--- a/dvc/config_schema.py
+++ b/dvc/config_schema.py
@@ -150,6 +150,7 @@ def __call__(self, data):
         Optional("autostage", default=False): Bool,
         Optional("experiments"): Bool,  # obsoleted
         Optional("check_update", default=True): Bool,
+        "site_cache_dir": str,
         "machine": Lower,
     },
     "cache": {
diff --git a/dvc/repo/__init__.py b/dvc/repo/__init__.py
--- a/dvc/repo/__init__.py
+++ b/dvc/repo/__init__.py
@@ -558,13 +558,15 @@ def dvcfs(self) -> "DVCFileSystem":
 
     @cached_property
     def site_cache_dir(self) -> str:
+        import getpass
         import hashlib
 
         import platformdirs
 
         from dvc.fs import GitFileSystem
 
-        cache_dir = platformdirs.site_cache_dir("dvc", "iterative", opinion=True)
+        default = platformdirs.site_cache_dir("dvc", "iterative", opinion=True)
+        cache_dir = self.config["core"].get("site_cache_dir") or default
 
         if isinstance(self.fs, GitFileSystem):
             relparts = ()
@@ -583,7 +585,9 @@ def site_cache_dir(self) -> str:
         finally:
             os.umask(umask)
 
-        repo_token = hashlib.md5(os.fsencode(root_dir)).hexdigest()  # noqa: S324
+        repo_token = hashlib.md5(  # noqa: S324
+            str((root_dir, getpass.getuser())).encode()
+        ).hexdigest()
 
         return os.path.join(repos_dir, repo_token)
 


Test patch: