Repository: huggingface/huggingface_hub
Pull request #2932: Adding Token Refresh Xet Tests
Author: login: rajatarya
Created: 2025-03-17T23:43:43Z | Merged: 2025-03-24T16:00:13Z

Two e2e tests that test refresh token scenario.

1. does normal upload/download with shimmed `xet_get` call to verify that
   token is refreshed in download path.

2. Directly calling `hfxet.download_files()` with `token_refresher` callback
   to ensure that hfxet calls the token refresher as expected.

Also, added download verification for `upload_large_folder` (using `snapshot_download`) for more end-to-end coverage.


Diff:
diff --git a/.github/workflows/python-tests.yml b/.github/workflows/python-tests.yml
index a007803044..f1c026826b 100644
--- a/.github/workflows/python-tests.yml
+++ b/.github/workflows/python-tests.yml
@@ -28,7 +28,6 @@ jobs:
             "Everything else",
             "Inference only",
             "Xet only"
-            
           ]
         include:
           - python-version: "3.13" # LFS not ran on 3.8
@@ -135,7 +134,6 @@ jobs:
               eval "RUN_GIT_LFS_TESTS=1 $PYTEST ../tests -k 'HfLargefilesTest'"
             ;;
 
-
             fastai)
               eval "$PYTEST ../tests/test_fastai*"
             ;;
diff --git a/tests/test_xet_upload.py b/tests/test_xet_upload.py
index c4ded9d059..9afaa4f3db 100644
--- a/tests/test_xet_upload.py
+++ b/tests/test_xet_upload.py
@@ -15,13 +15,21 @@
 from contextlib import contextmanager
 from io import BytesIO
 from pathlib import Path
-from unittest.mock import patch
+from typing import Tuple
+from unittest.mock import MagicMock, patch
 
 import pytest
 
 from huggingface_hub import HfApi, RepoUrl
 from huggingface_hub._commit_api import _upload_lfs_files, _upload_xet_files
-from huggingface_hub.file_download import get_hf_file_metadata, hf_hub_download, hf_hub_url
+from huggingface_hub.file_download import (
+    _get_metadata_or_catch_error,
+    get_hf_file_metadata,
+    hf_hub_download,
+    hf_hub_url,
+)
+from huggingface_hub.utils import build_hf_headers
+from huggingface_hub.utils._xet import refresh_xet_metadata
 
 from .testing_constants import ENDPOINT_STAGING, TOKEN
 from .testing_utils import repo_name, requires
@@ -227,6 +235,12 @@ def test_upload_large_folder(self, api, tmp_path, repo_url: RepoUrl) -> None:
 
         # Check all files have been uploaded
         uploaded_files = api.list_repo_files(repo_id=repo_id)
+
+        # Download and verify content
+        local_dir = Path(tmp_path) / "snapshot"
+        local_dir.mkdir()
+        api.snapshot_download(repo_id=repo_id, local_dir=local_dir, cache_dir=None)
+
         for i in range(N_FILES_PER_FOLDER):
             for j in range(N_FILES_PER_FOLDER):
                 assert f"subfolder_{i}/file_xet_{i}_{j}.bin" in uploaded_files
@@ -240,3 +254,93 @@ def test_upload_large_folder(self, api, tmp_path, repo_url: RepoUrl) -> None:
             metadata = get_hf_file_metadata(url)
             xet_metadata = metadata.xet_metadata
             assert xet_metadata is not None
+
+            # Verify xet files
+            xet_file = local_dir / f"subfolder_{i}/file_xet_{i}_{j}.bin"
+            assert xet_file.read_bytes() == f"content_lfs_{i}_{j}".encode()
+
+            # Verify regular files
+            regular_file = local_dir / f"subfolder_{i}/file_regular_{i}_{j}.txt"
+            assert regular_file.read_bytes() == f"content_regular_{i}_{j}".encode()
+
+
+@requires("hf_xet")
+class TestXetE2E(TestXetUpload):
+    def test_hf_xet_with_token_refresher(self, api, tmp_path, repo_url):
+        """
+        Test the hf_xet.download_files function with a token refresher.
+
+        This test manually calls the hf_xet.download_files function with a token refresher
+        function to verify that the token refresh mechanism works as expected. It aims to
+        identify regressions in the hf_xet.download_files function.
+
+        * Define a token refresher function that issues a token refresh by returning a new
+           access token and expiration time.
+        * Mock the token refresher function.
+        * Construct the necessary headers and metadata for the file to be downloaded.
+        * Call the download_files function with the token refresher, forcing a token refresh.
+        * Assert that the token refresher function was called as expected.
+
+        This test ensures that the downloaded file is the same as the uploaded file.
+        """
+        from hf_xet import PyPointerFile, download_files
+
+        filename_in_repo = "binary_file.bin"
+        repo_id = repo_url.repo_id
+
+        # Upload a file
+        api.upload_file(
+            path_or_fileobj=self.bin_file,
+            path_in_repo=filename_in_repo,
+            repo_id=repo_id,
+        )
+
+        # manually construct parameters to hf_xet.download_files and use a locally defined token_refresher function
+        # to verify that token refresh works as expected.
+        def token_refresher() -> Tuple[str, int]:
+            # Issue a token refresh by returning a new access token and expiration time
+            new_xet_metadata = refresh_xet_metadata(xet_metadata=xet_metadata, headers=headers)
+            return new_xet_metadata.access_token, new_xet_metadata.expiration_unix_epoch
+
+        mock_token_refresher = MagicMock(side_effect=token_refresher)
+
+        # headers
+        headers = build_hf_headers(token=TOKEN)
+
+        # metadata for url
+        (url_to_download, etag, commit_hash, expected_size, xet_metadata, head_call_error) = (
+            _get_metadata_or_catch_error(
+                repo_id=repo_id,
+                filename=filename_in_repo,
+                revision="main",
+                repo_type="model",
+                headers=headers,
+                endpoint=api.endpoint,
+                token=TOKEN,
+                proxies=None,
+                etag_timeout=None,
+                local_files_only=False,
+            )
+        )
+
+        incomplete_path = Path(tmp_path) / "file.bin.incomplete"
+        py_file = [
+            PyPointerFile(path=str(incomplete_path.absolute()), hash=xet_metadata.file_hash, filesize=expected_size)
+        ]
+
+        # Call the download_files function with the token refresher, set expiration to 0 forcing a refresh
+        download_files(
+            py_file,
+            endpoint=xet_metadata.endpoint,
+            token_info=(xet_metadata.access_token, 0),
+            token_refresher=mock_token_refresher,
+            progress_updater=None,
+        )
+
+        # assert that our local token_refresher function was called by hfxet as expected.
+        mock_token_refresher.assert_called_once()
+
+        # Check that the downloaded file is the same as the uploaded file
+        with open(incomplete_path, "rb") as f:
+            downloaded_content = f.read()
+        assert downloaded_content == self.bin_content


Reviews:
user: login: hanouticelina
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T10:22:58Z
user: login: hanouticelina
body: thanks @rajatarya for taking care of this! I left a couple of comments, let me know what you think 🙂 
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T15:13:41Z
user: login: rajatarya
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T16:43:23Z
user: login: rajatarya
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T21:29:50Z
user: login: rajatarya
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T21:30:02Z
user: login: rajatarya
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T21:32:07Z
user: login: rajatarya
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T21:32:21Z
user: login: Wauplin
body: Thanks for iterating and for the detailed tests! LGTM except the last test where I'm unsure what is supposed to be tested. Also let's wait for @hanouticelina 's approval first 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T09:02:38Z
user: login: hanouticelina
body: thank you @rajatarya for the iteration! I moved the tests to `test_xet_upload.py ` as we have everything setup there for uploading into a temp repo in staging. Also, let's avoid using `unittest.TestCase` and use only pytest instead. 
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T11:26:32Z
user: login: rajatarya
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T17:54:27Z
user: login: rajatarya
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T18:01:21Z
user: login: rajatarya
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T18:04:51Z
user: login: hanouticelina
body: Looks good, thank you @rajatarya! I pushed [5310054](https://github.com/huggingface/huggingface_hub/pull/2932/commits/5310054da9d83f5a39da489524927028da4373d9) to simplify a bit the `test_upload_large_folder` test to just call `snapshot_download` and then check that the content matches.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T10:16:04Z

Review comments:
diff_hunk: @@ -33,6 +33,8 @@ jobs:
         include:
           - python-version: "3.13" # LFS not ran on 3.8
             test_name: "lfs"
+          - python-version: "3.11"
path: .github/workflows/python-tests.yml
user: login: hanouticelina
body: is xet not supported on python 3.8? it should be the case as we already run xet tests on this python version.
created_at: 2025-03-18T10:05:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -135,6 +141,9 @@ jobs:
               eval "RUN_GIT_LFS_TESTS=1 $PYTEST ../tests -k 'HfLargefilesTest'"
             ;;
 
+            xet)
path: .github/workflows/python-tests.yml
user: login: hanouticelina
body: this is a duplicate, we already isolated xet tests here: https://github.com/huggingface/huggingface_hub/blob/e4e2a049e469167f1f29f33280ebf10087f6f9ce/.github/workflows/python-tests.yml#L99
created_at: 2025-03-18T10:06:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -116,6 +116,15 @@ def get_fastapi_version() -> str:
     return _get_version("fastapi")
 
 
+# HF Xet
+def is_hf_xet_available() -> bool:
+    return is_package_available("hf_xet")
+
+
+def get_hf_xet_version() -> str:
+    return _get_version("hf_xet")
+
+
path: src/huggingface_hub/utils/_runtime.py
user: login: hanouticelina
body: this is a duplicate, it's already implemented here: https://github.com/huggingface/huggingface_hub/blob/de81c60411ccfdf38112b51dca05236b663182c7/src/huggingface_hub/utils/_runtime.py#L165
created_at: 2025-03-18T10:08:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,325 @@
+import os
+from datetime import datetime
+from pathlib import Path
+from typing import Dict, Optional, Tuple
+from unittest import TestCase, skip
+from unittest.mock import MagicMock, patch
+
+from huggingface_hub import HfApi
+from huggingface_hub.constants import ENDPOINT
+from huggingface_hub.file_download import _get_metadata_or_catch_error
+from huggingface_hub.file_download import xet_get as original_xet_get
+from huggingface_hub.utils import SoftTemporaryDirectory, XetMetadata, build_hf_headers, tqdm
+from huggingface_hub.utils._xet import refresh_xet_metadata
+
+from .testing_constants import TOKEN
+from .testing_utils import repo_name
+
+
+WORKING_REPO_SUBDIR = f"fixtures/working_repo_{__name__.split('.')[-1]}"
+WORKING_REPO_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), WORKING_REPO_SUBDIR)
+
+
+def is_hf_xet_available():
+    """
+    Checks if the `hf_xet` module is available and can be imported. Used to skip tests that require `hf_xet` when it is
+    not available.
+    Returns:
+        bool: True if the `hf_xet` module is available and can be imported, False otherwise.
+    """
+    try:
+        from hf_xet import PyPointerFile
+
+        _p = PyPointerFile("path", "hash", 100)
+    except ImportError:
+        return False
+    return True
+
+
+def require_hf_xet(test_case):
+    """
+    Decorator marking a test that requires hf_xet.
+    These tests are skipped when hf_xet is not installed.
+    """
+    if not is_hf_xet_available():
+        return skip("Test requires hf_xet")(test_case)
+    else:
+        return test_case
path: tests/test_xet_integration.py
user: login: hanouticelina
body: no need for this, you can use `@requires("hf_xet")`:
```python
from .testing_utils import requires

@requires("hf_xet")
class TestHfXet(TestCase):
         ...
```
created_at: 2025-03-18T14:56:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,325 @@
+import os
+from datetime import datetime
+from pathlib import Path
+from typing import Dict, Optional, Tuple
+from unittest import TestCase, skip
+from unittest.mock import MagicMock, patch
+
+from huggingface_hub import HfApi
+from huggingface_hub.constants import ENDPOINT
+from huggingface_hub.file_download import _get_metadata_or_catch_error
+from huggingface_hub.file_download import xet_get as original_xet_get
+from huggingface_hub.utils import SoftTemporaryDirectory, XetMetadata, build_hf_headers, tqdm
+from huggingface_hub.utils._xet import refresh_xet_metadata
+
+from .testing_constants import TOKEN
+from .testing_utils import repo_name
+
+
+WORKING_REPO_SUBDIR = f"fixtures/working_repo_{__name__.split('.')[-1]}"
+WORKING_REPO_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), WORKING_REPO_SUBDIR)
+
+
+def is_hf_xet_available():
+    """
+    Checks if the `hf_xet` module is available and can be imported. Used to skip tests that require `hf_xet` when it is
+    not available.
+    Returns:
+        bool: True if the `hf_xet` module is available and can be imported, False otherwise.
+    """
+    try:
+        from hf_xet import PyPointerFile
+
+        _p = PyPointerFile("path", "hash", 100)
+    except ImportError:
+        return False
+    return True
path: tests/test_xet_integration.py
user: login: hanouticelina
body: no need to define this helper, there is one defined in here: https://github.com/huggingface/huggingface_hub/blob/de81c60411ccfdf38112b51dca05236b663182c7/src/huggingface_hub/utils/_runtime.py#L165
and you can import it like this:
```python
from .utils._runtime import is_xet_available
```
created_at: 2025-03-18T14:59:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,325 @@
+import os
+from datetime import datetime
+from pathlib import Path
+from typing import Dict, Optional, Tuple
+from unittest import TestCase, skip
+from unittest.mock import MagicMock, patch
+
+from huggingface_hub import HfApi
+from huggingface_hub.constants import ENDPOINT
+from huggingface_hub.file_download import _get_metadata_or_catch_error
+from huggingface_hub.file_download import xet_get as original_xet_get
+from huggingface_hub.utils import SoftTemporaryDirectory, XetMetadata, build_hf_headers, tqdm
+from huggingface_hub.utils._xet import refresh_xet_metadata
+
+from .testing_constants import TOKEN
+from .testing_utils import repo_name
+
+
+WORKING_REPO_SUBDIR = f"fixtures/working_repo_{__name__.split('.')[-1]}"
+WORKING_REPO_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), WORKING_REPO_SUBDIR)
+
+
+def is_hf_xet_available():
+    """
+    Checks if the `hf_xet` module is available and can be imported. Used to skip tests that require `hf_xet` when it is
+    not available.
+    Returns:
+        bool: True if the `hf_xet` module is available and can be imported, False otherwise.
+    """
+    try:
+        from hf_xet import PyPointerFile
+
+        _p = PyPointerFile("path", "hash", 100)
+    except ImportError:
+        return False
+    return True
+
+
+def require_hf_xet(test_case):
+    """
+    Decorator marking a test that requires hf_xet.
+    These tests are skipped when hf_xet is not installed.
+    """
+    if not is_hf_xet_available():
+        return skip("Test requires hf_xet")(test_case)
+    else:
+        return test_case
+
+
+@require_hf_xet
+class TestHfXet(TestCase):
path: tests/test_xet_integration.py
user: login: hanouticelina
body: I think we can move `test_force_refresh_token` and `test_hfxet_direct_download_files_token_refresher` to [tests/test_xet_download.py](https://github.com/huggingface/huggingface_hub/blob/xet-integration/tests/test_xet_download.py)

Also, I just realized that `test__xet_available`, `test_upload_and_download_with_hf_xet`, `test_upload_folder_download_snapshot_with_hf_xet` and `test_upload_large_folder_download_snapshot_with_hf_xet` might be unnecessary, we already have some e2e flow tested in [tests/test_xet_upload.py](https://github.com/huggingface/huggingface_hub/blob/xet-integration/tests/test_xet_upload.py). 

My suggestion would be to enhance the existing tests in `test_xet_upload.py` and/or `test_xet_download.py` to cover any missing scenarios, wdyt? 
created_at: 2025-03-18T15:13:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,325 @@
+import os
+from datetime import datetime
+from pathlib import Path
+from typing import Dict, Optional, Tuple
+from unittest import TestCase, skip
+from unittest.mock import MagicMock, patch
+
+from huggingface_hub import HfApi
+from huggingface_hub.constants import ENDPOINT
+from huggingface_hub.file_download import _get_metadata_or_catch_error
+from huggingface_hub.file_download import xet_get as original_xet_get
+from huggingface_hub.utils import SoftTemporaryDirectory, XetMetadata, build_hf_headers, tqdm
+from huggingface_hub.utils._xet import refresh_xet_metadata
+
+from .testing_constants import TOKEN
+from .testing_utils import repo_name
+
+
+WORKING_REPO_SUBDIR = f"fixtures/working_repo_{__name__.split('.')[-1]}"
+WORKING_REPO_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), WORKING_REPO_SUBDIR)
+
+
+def is_hf_xet_available():
+    """
+    Checks if the `hf_xet` module is available and can be imported. Used to skip tests that require `hf_xet` when it is
+    not available.
+    Returns:
+        bool: True if the `hf_xet` module is available and can be imported, False otherwise.
+    """
+    try:
+        from hf_xet import PyPointerFile
+
+        _p = PyPointerFile("path", "hash", 100)
+    except ImportError:
+        return False
+    return True
+
+
+def require_hf_xet(test_case):
+    """
+    Decorator marking a test that requires hf_xet.
+    These tests are skipped when hf_xet is not installed.
+    """
+    if not is_hf_xet_available():
+        return skip("Test requires hf_xet")(test_case)
+    else:
+        return test_case
+
+
+@require_hf_xet
+class TestHfXet(TestCase):
path: tests/test_xet_integration.py
user: login: rajatarya
body: Sounds good, will do.
created_at: 2025-03-19T16:43:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,325 @@
+import os
+from datetime import datetime
+from pathlib import Path
+from typing import Dict, Optional, Tuple
+from unittest import TestCase, skip
+from unittest.mock import MagicMock, patch
+
+from huggingface_hub import HfApi
+from huggingface_hub.constants import ENDPOINT
+from huggingface_hub.file_download import _get_metadata_or_catch_error
+from huggingface_hub.file_download import xet_get as original_xet_get
+from huggingface_hub.utils import SoftTemporaryDirectory, XetMetadata, build_hf_headers, tqdm
+from huggingface_hub.utils._xet import refresh_xet_metadata
+
+from .testing_constants import TOKEN
+from .testing_utils import repo_name
+
+
+WORKING_REPO_SUBDIR = f"fixtures/working_repo_{__name__.split('.')[-1]}"
+WORKING_REPO_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), WORKING_REPO_SUBDIR)
+
+
+def is_hf_xet_available():
+    """
+    Checks if the `hf_xet` module is available and can be imported. Used to skip tests that require `hf_xet` when it is
+    not available.
+    Returns:
+        bool: True if the `hf_xet` module is available and can be imported, False otherwise.
+    """
+    try:
+        from hf_xet import PyPointerFile
+
+        _p = PyPointerFile("path", "hash", 100)
+    except ImportError:
+        return False
+    return True
path: tests/test_xet_integration.py
user: login: rajatarya
body: Removed test_xet_integration.py.
created_at: 2025-03-19T21:29:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,325 @@
+import os
+from datetime import datetime
+from pathlib import Path
+from typing import Dict, Optional, Tuple
+from unittest import TestCase, skip
+from unittest.mock import MagicMock, patch
+
+from huggingface_hub import HfApi
+from huggingface_hub.constants import ENDPOINT
+from huggingface_hub.file_download import _get_metadata_or_catch_error
+from huggingface_hub.file_download import xet_get as original_xet_get
+from huggingface_hub.utils import SoftTemporaryDirectory, XetMetadata, build_hf_headers, tqdm
+from huggingface_hub.utils._xet import refresh_xet_metadata
+
+from .testing_constants import TOKEN
+from .testing_utils import repo_name
+
+
+WORKING_REPO_SUBDIR = f"fixtures/working_repo_{__name__.split('.')[-1]}"
+WORKING_REPO_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), WORKING_REPO_SUBDIR)
+
+
+def is_hf_xet_available():
+    """
+    Checks if the `hf_xet` module is available and can be imported. Used to skip tests that require `hf_xet` when it is
+    not available.
+    Returns:
+        bool: True if the `hf_xet` module is available and can be imported, False otherwise.
+    """
+    try:
+        from hf_xet import PyPointerFile
+
+        _p = PyPointerFile("path", "hash", 100)
+    except ImportError:
+        return False
+    return True
+
+
+def require_hf_xet(test_case):
+    """
+    Decorator marking a test that requires hf_xet.
+    These tests are skipped when hf_xet is not installed.
+    """
+    if not is_hf_xet_available():
+        return skip("Test requires hf_xet")(test_case)
+    else:
+        return test_case
path: tests/test_xet_integration.py
user: login: rajatarya
body: removed test_xet_integration.py
created_at: 2025-03-19T21:30:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -33,6 +33,8 @@ jobs:
         include:
           - python-version: "3.13" # LFS not ran on 3.8
             test_name: "lfs"
+          - python-version: "3.11"
path: .github/workflows/python-tests.yml
user: login: rajatarya
body: Not sure how this got in there, removing.
created_at: 2025-03-19T21:32:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -135,6 +141,9 @@ jobs:
               eval "RUN_GIT_LFS_TESTS=1 $PYTEST ../tests -k 'HfLargefilesTest'"
             ;;
 
+            xet)
path: .github/workflows/python-tests.yml
user: login: rajatarya
body: Good catch, removing.
created_at: 2025-03-19T21:32:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -240,3 +240,15 @@ def test_upload_large_folder(self, api, tmp_path, repo_url: RepoUrl) -> None:
             metadata = get_hf_file_metadata(url)
             xet_metadata = metadata.xet_metadata
             assert xet_metadata is not None
+
+        # Download and verify content
+        local_dir = Path(tmp_path) / "snapshot"
+        local_dir.mkdir()
+        api.snapshot_download(repo_id=repo_id, local_dir=local_dir, cache_dir=None)
+
+        for i in range(N_FILES_PER_FOLDER):
+            for j in range(N_FILES_PER_FOLDER):
+                for filename in [f"subfolder_{i}/file_xet_{i}_{j}.bin", f"subfolder_{i}/file_regular_{i}_{j}.txt"]:
+                    local_file = Path(folder) / filename
+                    downloaded_file = hf_hub_download(repo_id=repo_id, filename=filename, cache_dir=None)
path: tests/test_xet_upload.py
user: login: Wauplin
body: Just to be sure to understand, is this test supposed to check the downloaded content is the same as the uploaded one? or is it to check that both "download to cache" and "download to local_dir" are working and retrieving the same content? 

To check that download works correctly, I'd just do the `snapshot_download` as you did and then iterate on `i/j`  and check the content is `f"content_lfs_{i}_{j}"`/`f"content_regular_{i}_{j}"`
created_at: 2025-03-20T09:01:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -240,3 +250,153 @@ def test_upload_large_folder(self, api, tmp_path, repo_url: RepoUrl) -> None:
             metadata = get_hf_file_metadata(url)
             xet_metadata = metadata.xet_metadata
             assert xet_metadata is not None
+
+        # Download and verify content
+        local_dir = Path(tmp_path) / "snapshot"
+        local_dir.mkdir()
+        api.snapshot_download(repo_id=repo_id, local_dir=local_dir, cache_dir=None)
+
+        for i in range(N_FILES_PER_FOLDER):
+            for j in range(N_FILES_PER_FOLDER):
+                for filename in [f"subfolder_{i}/file_xet_{i}_{j}.bin", f"subfolder_{i}/file_regular_{i}_{j}.txt"]:
+                    local_file = Path(folder) / filename
+                    downloaded_file = hf_hub_download(repo_id=repo_id, filename=filename, cache_dir=None)
+                    assert Path(local_file).read_bytes() == Path(downloaded_file).read_bytes()
+
+
+@requires("hf_xet")
+class TestXetE2E(TestXetUpload):
+    def test_force_refresh_token(self, api, repo_url, tmp_path):
path: tests/test_xet_upload.py
user: login: hanouticelina
body: I'm not sure I understand how this tests the token refresh process, I don't see a mock of the `token_refresher` function as it's done in `test_hf_xet_with_token_refresher`. is it by checking that `xet_get` is called?
created_at: 2025-03-20T11:22:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -240,3 +240,15 @@ def test_upload_large_folder(self, api, tmp_path, repo_url: RepoUrl) -> None:
             metadata = get_hf_file_metadata(url)
             xet_metadata = metadata.xet_metadata
             assert xet_metadata is not None
+
+        # Download and verify content
+        local_dir = Path(tmp_path) / "snapshot"
+        local_dir.mkdir()
+        api.snapshot_download(repo_id=repo_id, local_dir=local_dir, cache_dir=None)
+
+        for i in range(N_FILES_PER_FOLDER):
+            for j in range(N_FILES_PER_FOLDER):
+                for filename in [f"subfolder_{i}/file_xet_{i}_{j}.bin", f"subfolder_{i}/file_regular_{i}_{j}.txt"]:
+                    local_file = Path(folder) / filename
+                    downloaded_file = hf_hub_download(repo_id=repo_id, filename=filename, cache_dir=None)
path: tests/test_xet_upload.py
user: login: rajatarya
body: The intent is to verify that the bytes are downloaded (no cache used) and contain the same contents as they exist locally. I'll change it to check the contents directly instead of comparing to the local files.
created_at: 2025-03-20T17:54:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -240,3 +250,153 @@ def test_upload_large_folder(self, api, tmp_path, repo_url: RepoUrl) -> None:
             metadata = get_hf_file_metadata(url)
             xet_metadata = metadata.xet_metadata
             assert xet_metadata is not None
+
+        # Download and verify content
+        local_dir = Path(tmp_path) / "snapshot"
+        local_dir.mkdir()
+        api.snapshot_download(repo_id=repo_id, local_dir=local_dir, cache_dir=None)
+
+        for i in range(N_FILES_PER_FOLDER):
+            for j in range(N_FILES_PER_FOLDER):
+                for filename in [f"subfolder_{i}/file_xet_{i}_{j}.bin", f"subfolder_{i}/file_regular_{i}_{j}.txt"]:
+                    local_file = Path(folder) / filename
+                    downloaded_file = hf_hub_download(repo_id=repo_id, filename=filename, cache_dir=None)
+                    assert Path(local_file).read_bytes() == Path(downloaded_file).read_bytes()
+
+
+@requires("hf_xet")
+class TestXetE2E(TestXetUpload):
+    def test_force_refresh_token(self, api, repo_url, tmp_path):
path: tests/test_xet_upload.py
user: login: rajatarya
body: This test simply changes the expiration time of the token to now(), ensuring the refresh should happen immediately. However, by not mocking the token_refresh function itself it doesn't add anything to the overall test coverage.

I'll drop this test, I don't think it is adding anything over the `test_hf_xet_with_token_refresher` test.
created_at: 2025-03-20T18:01:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -240,3 +250,153 @@ def test_upload_large_folder(self, api, tmp_path, repo_url: RepoUrl) -> None:
             metadata = get_hf_file_metadata(url)
             xet_metadata = metadata.xet_metadata
             assert xet_metadata is not None
+
+        # Download and verify content
+        local_dir = Path(tmp_path) / "snapshot"
+        local_dir.mkdir()
+        api.snapshot_download(repo_id=repo_id, local_dir=local_dir, cache_dir=None)
+
+        for i in range(N_FILES_PER_FOLDER):
+            for j in range(N_FILES_PER_FOLDER):
+                for filename in [f"subfolder_{i}/file_xet_{i}_{j}.bin", f"subfolder_{i}/file_regular_{i}_{j}.txt"]:
+                    local_file = Path(folder) / filename
+                    downloaded_file = hf_hub_download(repo_id=repo_id, filename=filename, cache_dir=None)
+                    assert Path(local_file).read_bytes() == Path(downloaded_file).read_bytes()
+
+
+@requires("hf_xet")
+class TestXetE2E(TestXetUpload):
+    def test_force_refresh_token(self, api, repo_url, tmp_path):
path: tests/test_xet_upload.py
user: login: rajatarya
body: The only thing this test covers that isn't covered by `test_hf_xet_with_token_refresher` is that it starts from the user function `hf_hub_download` so is more of a an end-to-end test.

`test_hf_xet_with_token_refresher` is more of a lower-level regression test that ensures `hf_xet` doesn't change its interface. It calls `download_files` directly, which isn't something a user can do. 

That is why I originally had both tests, but I don't think it is worth maintaining both tests given how subtle the coverage point is.
created_at: 2025-03-20T18:04:51Z
author_association: CONTRIBUTOR

PR comments:
user: login: HuggingFaceDocBuilderDev
created_at: 2025-03-17T23:47:19Z
body: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/huggingface_hub/pr_2932). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
author_association: NONE
user: login: rajatarya
created_at: 2025-03-19T22:01:02Z
body: I believe this PR is ready for final review. I believe I have addressed all the feedback (thanks @hanouticelina, great feedback!)

My apologies - I should have better tracked the tests in the `test_xet_upload.py` and `test_xet_download.py` - they are better organized, better structured, cleaner, and much more thorough than the tests I wrote. And as those tests were merged they had already set up the GitHub Action workflow changes for xet so this PR was duplicating (poorly) a bunch of that.

Now this PR is simply adding two token refresh tests on to `test_xet_download.py`, and adding download + byte verification of `upload_large_folder` using `snapshot_download` in `test_xet_upload.py`.
author_association: CONTRIBUTOR
user: login: rajatarya
created_at: 2025-03-20T18:23:44Z
body: OK now simplified to only one new test for token refresh. Thanks for all the great feedback!
author_association: CONTRIBUTOR
user: login: rajatarya
created_at: 2025-03-21T21:44:45Z
body: > OK now simplified to only one new test for token refresh. Thanks for all the great feedback!

Am I good to merge? @bpronan @hanouticelina can you approve if you think ready for merging?
author_association: CONTRIBUTOR