Repository: huggingface/huggingface_hub

Issue:
Permission on local cached files should be configurable
**Is your feature request related to a problem? Please describe.**
If multiple users attempt to share the same models cache (e.g., on a company internal shared file system), anytime model files are downloaded, their permissions are 0600 (on linux) regardless of the umask. This is because file_download is using `tempfile.NamedTemporaryFile` (see https://github.com/huggingface/huggingface_hub/blob/60077db55b81f2add3fd3bf774d7dc0f7acc4857/src/huggingface_hub/file_download.py#L727 and https://stackoverflow.com/questions/10541760/can-i-set-the-umask-for-tempfile-namedtemporaryfile-in-python), so only the user who downloaded them can read those files. As a result, users can't share a local cache, which is a waste of resources and time for users.

**Describe the solution you'd like**
huggingface_hub should `chmod` the downloaded files (for example, using the current `os.umask`, or using a new configuration option) after the file is downloaded and before it is renamed so it can be shared across users.

**Describe alternatives you've considered**
Separate caches per user is a workaround but is a massive waste of disk space and time. Each user would have to wait for new downloads even if the files have been downloaded by another user already.

**Additional context**
Trying to use the HuggingFace diffusers/transformers library in an enterprise context.



Discussion:
Hi @antoche , thanks for opening this issue. This seems a very legit use case to me. So what we could do is to `chmod` the temporary file just before moving it to the cache [here](https://github.com/huggingface/huggingface_hub/blob/60077db55b81f2add3fd3bf774d7dc0f7acc4857/src/huggingface_hub/file_download.py#L745) and [here](https://github.com/huggingface/huggingface_hub/blob/60077db55b81f2add3fd3bf774d7dc0f7acc4857/src/huggingface_hub/file_download.py#L1244). 
I would set the file permissions to match the same as the cache directory itself. Something like: (not tested)

```py
import os
import stat

cache_dir_mode = os.stat(cache_dir).st_mode
# https://docs.python.org/3/library/stat.html#stat.S_IMODE
os.chmod(temp_file.name, stat.S_IMODE(cache_dir_mode))
```

Would you like to try that and open a PR for it ?


That sounds like a sensible solution. I'd love to contribute but I have quite a few hurdles in my way due to corporate security&legal limitations. I already have another change in the wings related to proxy/certificate settings that I'm trying get over the fence to you, I'm not sure when I'll get that done. I'll add this one on my plate, but if anyone else has some time to address it, feel free to go ahead.
Ok thanks for letting us know @antoche. Not sure yet how we will prioritize this so when and if you have time, feel free to go ahead. If I start to work on it, I'll assigned the issue to myself to let you know.
@Wauplin, sent me here from a similar Issue I have just filed https://github.com/huggingface/huggingface_hub/issues/1215

In group environment usually users must set `umask` to support shared files. So you could refer to that setting to know which perms to set.

In my issue I also linked to https://github.com/huggingface/datasets/pull/2157 where `datasets` deals with a similar issue if it helps.

Resolving patch:
diff --git a/src/huggingface_hub/file_download.py b/src/huggingface_hub/file_download.py
--- a/src/huggingface_hub/file_download.py
+++ b/src/huggingface_hub/file_download.py
@@ -5,7 +5,9 @@
 import os
 import re
 import shutil
+import stat
 import tempfile
+import uuid
 import warnings
 from contextlib import contextmanager
 from dataclasses import dataclass
@@ -747,7 +749,7 @@ def _resumable_file_manager() -> Generator[io.BufferedWriter, None, None]:
             )
 
         logger.info("storing %s in cache at %s", url, cache_path)
-        os.replace(temp_file.name, cache_path)
+        _chmod_and_replace(temp_file.name, cache_path)
 
         if force_filename is None:
             logger.info("creating metadata file for %s", cache_path)
@@ -1249,7 +1251,7 @@ def _resumable_file_manager() -> Generator[io.BufferedWriter, None, None]:
             )
 
         logger.info("storing %s in cache at %s", url, blob_path)
-        os.replace(temp_file.name, blob_path)
+        _chmod_and_replace(temp_file.name, blob_path)
 
         logger.info("creating pointer to %s from %s", blob_path, pointer_path)
         _create_relative_symlink(blob_path, pointer_path, new_blob=True)
@@ -1401,3 +1403,28 @@ def _int_or_none(value: Optional[str]) -> Optional[int]:
         return int(value)  # type: ignore
     except (TypeError, ValueError):
         return None
+
+
+def _chmod_and_replace(src: str, dst: str) -> None:
+    """Set correct permission before moving a blob from tmp directory to cache dir.
+
+    Do not take into account the `umask` from the process as there is no convenient way
+    to get it that is thread-safe.
+
+    See:
+    - About umask: https://docs.python.org/3/library/os.html#os.umask
+    - Thread-safety: https://stackoverflow.com/a/70343066
+    - About solution: https://github.com/huggingface/huggingface_hub/pull/1220#issuecomment-1326211591
+    - Fix issue: https://github.com/huggingface/huggingface_hub/issues/1141
+    - Fix issue: https://github.com/huggingface/huggingface_hub/issues/1215
+    """
+    # Get umask by creating a temporary file in the cached repo folder.
+    tmp_file = Path(dst).parent.parent / f"tmp_{uuid.uuid4()}"
+    try:
+        tmp_file.touch()
+        cache_dir_mode = Path(tmp_file).stat().st_mode
+        os.chmod(src, stat.S_IMODE(cache_dir_mode))
+    finally:
+        tmp_file.unlink()
+
+    os.replace(src, dst)


Test patch:
diff --git a/tests/test_file_download.py b/tests/test_file_download.py
--- a/tests/test_file_download.py
+++ b/tests/test_file_download.py
@@ -13,6 +13,7 @@
 # limitations under the License.
 import os
 import re
+import stat
 import unittest
 from pathlib import Path
 from tempfile import TemporaryDirectory
@@ -242,6 +243,25 @@ def test_dataset_lfs_object(self):
             (url, '"95aa6a52d5d6a735563366753ca50492a658031da74f301ac5238b03966972c9"'),
         )
 
+    def test_hf_hub_download_custom_cache_permission(self):
+        """Checks `hf_hub_download` respect the cache dir permission.
+
+        Regression test for #1141 #1215.
+        https://github.com/huggingface/huggingface_hub/issues/1141
+        https://github.com/huggingface/huggingface_hub/issues/1215
+        """
+        with TemporaryDirectory() as tmpdir:
+            # Equivalent to umask u=rwx,g=r,o=
+            previous_umask = os.umask(0o037)
+            try:
+                filepath = hf_hub_download(
+                    DUMMY_RENAMED_OLD_MODEL_ID, "config.json", cache_dir=tmpdir
+                )
+                # Permissions are honored (640: u=rw,g=r,o=)
+                self.assertEqual(stat.S_IMODE(os.stat(filepath).st_mode), 0o640)
+            finally:
+                os.umask(previous_umask)
+
     def test_download_from_a_renamed_repo_with_hf_hub_download(self):
         """Checks `hf_hub_download` works also on a renamed repo.