Repository: huggingface/huggingface_hub
Pull request #2875: Xet download workflow
Author: login: hanouticelina
Created: 2025-02-18T17:35:57Z | Merged: 2025-03-04T13:48:22Z


Partially resolves #2713.

This PR adds the Xet download workflow implemented in [xetpoc_huggingface_hub](https://github.com/huggingface-internal/xetpoc_huggingface_hub) (internal). The upload one will be integrated in a separate PR

The main branch for xet storage integration is [xet-integration](https://github.com/huggingface/huggingface_hub/tree/xet-integration).

Main changes:

- [x] Make `hf_xet` available as an optional dependency via `pip install huggingface_hub[hf_xet]` 
**Note**: since it's a common part for download and upload, this has been pushed directly  into [xet-integration](https://github.com/huggingface/huggingface_hub/tree/xet-integration) branch.
- [x] Integrate changes from the xet poc for the download workflow only.
- [x] Add tests.
- [ ] Add documentation.

to try it in from this branch:
```bash
pip install -e ".[dev,hf_xet]"
export HF_DEBUG=1 #  if you want to set huggingface_hub logger to debug level
huggingface-cli download huggingface/distilbert-base-uncased-xet
```

cc @bpronan @assafvayner @rajatarya 

Linked issue #2713: Xet Storage Integration
This issue tracks the integration of xet storage support into `huggingface_hub`.

The current plan is to make `hf_xet` available as an optional dependency via `pip install huggingface_hub[hf_xet]`, similar to what we already have with `hf_transfer`. 
Mid-term, the plan is to integrate `hf_xet` as a required dependency.

## Implementation Plan

The integration will be completed in phases on a dedicated xet branch. As a first step, we need to have the staging environment (https://hub-ci.huggingface.co/) ready for xet testing. cc @coyotte508 maybe.

On `huggingface_hub` side, we will split the work into parallel tasks for the upload and download workflows, each handled through separate PRs:

- Integrate changes from the xet poc. A full diff between the poc and the main huggingface_hub can be found [here](https://github.com/huggingface-internal/xetpoc_huggingface_hub/compare/70ff78947fd0817b2794e47d5a813ac4d3853fde...main) (private link).
- Add tests and provide documentation for each part.

cc @julien-c @LysandreJik @bpronan @Pierrci 

Linked issue comments: user: login: Wauplin
created_at: 2025-10-01T09:44:15Z
body: done as part of https://github.com/huggingface/huggingface_hub/pull/2887 and https://github.com/huggingface/huggingface_hub/pull/2875 (and many smaller ones :) )

🎉 
author_association: CONTRIBUTOR

Diff:
diff --git a/src/huggingface_hub/file_download.py b/src/huggingface_hub/file_download.py
index e9f3d9fba7..f31e859b21 100644
--- a/src/huggingface_hub/file_download.py
+++ b/src/huggingface_hub/file_download.py
@@ -1,4 +1,3 @@
-import contextlib
 import copy
 import errno
 import inspect
@@ -38,6 +37,7 @@
     OfflineModeIsEnabled,
     SoftTemporaryDirectory,
     WeakFileLock,
+    XetMetadata,
     build_hf_headers,
     get_fastai_version,  # noqa: F401 # for backward compatibility
     get_fastcore_version,  # noqa: F401 # for backward compatibility
@@ -56,15 +56,17 @@
     is_tf_available,  # noqa: F401 # for backward compatibility
     is_torch_available,  # noqa: F401 # for backward compatibility
     logging,
+    parse_xet_headers,
+    refresh_xet_metadata,
     reset_sessions,
     tqdm,
     validate_hf_hub_args,
 )
 from .utils._http import _adjust_range_header
-from .utils._runtime import _PY_VERSION  # noqa: F401 # for backward compatibility
+from .utils._runtime import _PY_VERSION, is_xet_available  # noqa: F401 # for backward compatibility
 from .utils._typing import HTTP_METHOD_T
 from .utils.sha import sha_fileobj
-from .utils.tqdm import is_tqdm_disabled
+from .utils.tqdm import _get_progress_bar_context
 
 
 logger = logging.get_logger(__name__)
@@ -160,12 +162,15 @@ class HfFileMetadata:
         size (`size`):
             Size of the file. In case of an LFS file, contains the size of the actual
             LFS file, not the pointer.
+        xet_metadata (`XetMetadata`, *optional*):
+            Xet metadata for the file. This is only set if the file is stored using Xet storage.
     """
 
     commit_hash: Optional[str]
     etag: Optional[str]
     location: str
     size: Optional[int]
+    xet_metadata: Optional[XetMetadata] = None
 
 
 @validate_hf_hub_args
@@ -396,23 +401,13 @@ def http_get(
         f" {{actual_size}} ({displayed_filename}).\nThis is usually due to network issues while downloading the file."
         " Please retry with `force_download=True`."
     )
-
-    # Stream file to buffer
-    progress_cm: tqdm = (
-        tqdm(  # type: ignore[assignment]
-            unit="B",
-            unit_scale=True,
-            total=total,
-            initial=resume_size,
-            desc=displayed_filename,
-            disable=is_tqdm_disabled(logger.getEffectiveLevel()),
-            name="huggingface_hub.http_get",
-        )
-        if _tqdm_bar is None
-        else contextlib.nullcontext(_tqdm_bar)
-        # ^ `contextlib.nullcontext` mimics a context manager that does nothing
-        #   Makes it easier to use the same code path for both cases but in the later
-        #   case, the progress bar is not closed when exiting the context manager.
+    progress_cm = _get_progress_bar_context(
+        desc=displayed_filename,
+        log_level=logger.getEffectiveLevel(),
+        total=total,
+        initial=resume_size,
+        name="huggingface_hub.http_get",
+        _tqdm_bar=_tqdm_bar,
     )
 
     with progress_cm as progress:
@@ -487,6 +482,108 @@ def http_get(
         )
 
 
+def xet_get(
+    *,
+    incomplete_path: Path,
+    xet_metadata: XetMetadata,
+    headers: Dict[str, str],
+    expected_size: Optional[int] = None,
+    displayed_filename: Optional[str] = None,
+    _tqdm_bar: Optional[tqdm] = None,
+) -> None:
+    """
+    Download a file using Xet storage service.
+
+    Args:
+        incomplete_path (`Path`):
+            The path to the file to download.
+        xet_metadata (`XetMetadata`):
+            The metadata needed to make the request to the xet storage service.
+        headers (`Dict[str, str]`):
+            The headers to send to the xet storage service.
+        expected_size (`int`, *optional*):
+            The expected size of the file to download. If set, the download will raise an error if the size of the
+            received content is different from the expected one.
+        displayed_filename (`str`, *optional*):
+            The filename of the file that is being downloaded. Value is used only to display a nice progress bar. If
+            not set, the filename is guessed from the URL or the `Content-Disposition` header.
+
+    **How it works:**
+        The file download system uses Xet storage, which is a content-addressable storage system that breaks files into chunks
+        for efficient storage and transfer.
+
+        `hf_xet.download_files` manages downloading files by:
+        - Taking a list of files to download (each with its unique content hash)
+        - Connecting to a storage server (CAS server) that knows how files are chunked
+        - Using authentication to ensure secure access
+        - Providing progress updates during download
+
+        Authentication works by regularly refreshing access tokens through `refresh_xet_metadata` to maintain a valid
+        connection to the storage server.
+
+        The download process works like this:
+        1. Create a local cache folder at `~/.cache/huggingface/xet/chunk-cache` to store reusable file chunks
+        2. Download files in parallel:
+            2.1. Prepare to write the file to disk
+            2.2. Ask the server "how is this file split into chunks?" using the file's unique hash
+                The server responds with:
+                - Which chunks make up the complete file
+                - Where each chunk can be downloaded from
+            2.3. For each needed chunk:
+                - Checks if we already have it in our local cache
+                - If not, download it from cloud storage (S3)
+                - Save it to cache for future use
+                - Assemble the chunks in order to recreate the original file
+
+    """
+    try:
+        from hf_xet import PyPointerFile, download_files  # type: ignore[no-redef]
+    except ImportError:
+        raise ValueError(
+            "To use optimized download using Xet storage, you need to install the hf_xet package. "
+            "Try `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`."
+        )
+
+    def token_refresher() -> Tuple[str, int]:
+        new_xet_metadata = refresh_xet_metadata(xet_metadata=xet_metadata, headers=headers)
+        if new_xet_metadata is None:
+            raise ValueError("Failed to refresh token using xet metadata.")
+        return new_xet_metadata.access_token, new_xet_metadata.expiration_unix_epoch
+
+    pointer_files = [
+        PyPointerFile(path=str(incomplete_path.absolute()), hash=xet_metadata.file_hash, filesize=expected_size)
+    ]
+
+    if not displayed_filename:
+        displayed_filename = incomplete_path.name
+
+    # Truncate filename if too long to display
+    if len(displayed_filename) > 40:
+        displayed_filename = f"{displayed_filename[:40]}(…)"
+
+    progress_cm = _get_progress_bar_context(
+        desc=displayed_filename,
+        log_level=logger.getEffectiveLevel(),
+        total=expected_size,
+        initial=0,
+        name="huggingface_hub.xet_get",
+        _tqdm_bar=_tqdm_bar,
+    )
+
+    with progress_cm as progress:
+
+        def progress_updater(progress_bytes: float):
+            progress.update(progress_bytes)
+
+        download_files(
+            pointer_files,
+            endpoint=xet_metadata.endpoint,
+            token_info=(xet_metadata.access_token, xet_metadata.expiration_unix_epoch),
+            token_refresher=token_refresher,
+            progress_updater=[progress_updater],
+        )
+
+
 def _normalize_etag(etag: Optional[str]) -> Optional[str]:
     """Normalize ETag HTTP header, so it can be used to create nice filepaths.
 
@@ -922,7 +1019,7 @@ def _hf_hub_download_to_cache_dir(
 
     # Try to get metadata (etag, commit_hash, url, size) from the server.
     # If we can't, a HEAD request error is returned.
-    (url_to_download, etag, commit_hash, expected_size, head_call_error) = _get_metadata_or_catch_error(
+    (url_to_download, etag, commit_hash, expected_size, xet_metadata, head_call_error) = _get_metadata_or_catch_error(
         repo_id=repo_id,
         filename=filename,
         repo_type=repo_type,
@@ -1017,6 +1114,8 @@ def _hf_hub_download_to_cache_dir(
             expected_size=expected_size,
             filename=filename,
             force_download=force_download,
+            etag=etag,
+            xet_metadata=xet_metadata,
         )
         if not os.path.exists(pointer_path):
             _create_symlink(blob_path, pointer_path, new_blob=True)
@@ -1067,7 +1166,7 @@ def _hf_hub_download_to_local_dir(
         return str(paths.file_path)
 
     # Local file doesn't exist or commit_hash doesn't match => we need the etag
-    (url_to_download, etag, commit_hash, expected_size, head_call_error) = _get_metadata_or_catch_error(
+    (url_to_download, etag, commit_hash, expected_size, xet_metadata, head_call_error) = _get_metadata_or_catch_error(
         repo_id=repo_id,
         filename=filename,
         repo_type=repo_type,
@@ -1144,6 +1243,8 @@ def _hf_hub_download_to_local_dir(
             expected_size=expected_size,
             filename=filename,
             force_download=force_download,
+            etag=etag,
+            xet_metadata=xet_metadata,
         )
 
     write_download_metadata(local_dir=local_dir, filename=filename, commit_hash=commit_hash, etag=etag)
@@ -1317,6 +1418,7 @@ def get_hf_file_metadata(
         size=_int_or_none(
             r.headers.get(constants.HUGGINGFACE_HEADER_X_LINKED_SIZE) or r.headers.get("Content-Length")
         ),
+        xet_metadata=parse_xet_headers(r.headers),  # type: ignore
     )
 
 
@@ -1336,10 +1438,10 @@ def _get_metadata_or_catch_error(
     storage_folder: Optional[str] = None,  # only used to store `.no_exists` in cache
 ) -> Union[
     # Either an exception is caught and returned
-    Tuple[None, None, None, None, Exception],
+    Tuple[None, None, None, None, None, Exception],
     # Or the metadata is returned as
     # `(url_to_download, etag, commit_hash, expected_size, None)`
-    Tuple[str, str, str, int, None],
+    Tuple[str, str, str, int, Optional[XetMetadata], None],
 ]:
     """Get metadata for a file on the Hub, safely handling network issues.
 
@@ -1356,6 +1458,7 @@ def _get_metadata_or_catch_error(
             None,
             None,
             None,
+            None,
             OfflineModeIsEnabled(
                 f"Cannot access file since 'local_files_only=True' as been set. (repo_id: {repo_id}, repo_type: {repo_type}, revision: {revision}, filename: {filename})"
             ),
@@ -1367,6 +1470,7 @@ def _get_metadata_or_catch_error(
     commit_hash: Optional[str] = None
     expected_size: Optional[int] = None
     head_error_call: Optional[Exception] = None
+    xet_metadata: Optional[XetMetadata] = None
 
     # Try to get metadata from the server.
     # Do not raise yet if the file is not found or not accessible.
@@ -1414,13 +1518,15 @@ def _get_metadata_or_catch_error(
             if expected_size is None:
                 raise FileMetadataError("Distant resource does not have a Content-Length.")
 
+            xet_metadata = metadata.xet_metadata
+
             # In case of a redirect, save an extra redirect on the request.get call,
             # and ensure we download the exact atomic version even if it changed
             # between the HEAD and the GET (unlikely, but hey).
             #
             # If url domain is different => we are downloading from a CDN => url is signed => don't send auth
             # If url domain is the same => redirect due to repo rename AND downloading a regular file => keep auth
-            if url != metadata.location:
+            if xet_metadata is not None and url != metadata.location:
                 url_to_download = metadata.location
                 if urlparse(url).netloc != urlparse(metadata.location).netloc:
                     # Remove authorization header when downloading a LFS blob
@@ -1458,7 +1564,7 @@ def _get_metadata_or_catch_error(
     if not (local_files_only or etag is not None or head_error_call is not None):
         raise RuntimeError("etag is empty due to uncovered problems")
 
-    return (url_to_download, etag, commit_hash, expected_size, head_error_call)  # type: ignore [return-value]
+    return (url_to_download, etag, commit_hash, expected_size, xet_metadata, head_error_call)  # type: ignore [return-value]
 
 
 def _raise_on_head_call_error(head_call_error: Exception, force_download: bool, local_files_only: bool) -> NoReturn:
@@ -1502,6 +1608,8 @@ def _download_to_tmp_and_move(
     expected_size: Optional[int],
     filename: str,
     force_download: bool,
+    etag: Optional[str],
+    xet_metadata: Optional[XetMetadata],
 ) -> None:
     """Download content from a URL to a destination path.
 
@@ -1544,14 +1652,31 @@ def _download_to_tmp_and_move(
             _check_disk_space(expected_size, incomplete_path.parent)
             _check_disk_space(expected_size, destination_path.parent)
 
-        http_get(
-            url_to_download,
-            f,
-            proxies=proxies,
-            resume_size=resume_size,
-            headers=headers,
-            expected_size=expected_size,
-        )
+        is_xet_enabled = xet_metadata is not None and xet_metadata.file_hash is not None
+        if is_xet_enabled and is_xet_available():
+            logger.info("Xet Storage is enabled for this repo. Downloading file from Xet Storage..")
+            xet_get(
+                incomplete_path=incomplete_path,
+                xet_metadata=xet_metadata,  # type: ignore
+                headers=headers,
+                expected_size=expected_size,
+                displayed_filename=filename,
+            )
+        else:
+            if is_xet_enabled:
+                logger.warning(
+                    "Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. "
+                    "Falling back to regular HTTP download. "
+                    "For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`"
+                )
+            http_get(
+                url_to_download,
+                f,
+                proxies=proxies,
+                resume_size=resume_size,
+                headers=headers,
+                expected_size=expected_size,
+            )
 
     logger.info(f"Download complete. Moving file to {destination_path}")
     _chmod_and_move(incomplete_path, destination_path)
diff --git a/src/huggingface_hub/utils/__init__.py b/src/huggingface_hub/utils/__init__.py
index b9715dc0ad..1b6f4b2543 100644
--- a/src/huggingface_hub/utils/__init__.py
+++ b/src/huggingface_hub/utils/__init__.py
@@ -107,4 +107,5 @@
 from ._telemetry import send_telemetry
 from ._typing import is_jsonable, is_simple_optional_type, unwrap_simple_optional_type
 from ._validators import smoothly_deprecate_use_auth_token, validate_hf_hub_args, validate_repo_id
+from ._xet import XetMetadata, parse_xet_headers, refresh_xet_metadata
 from .tqdm import are_progress_bars_disabled, disable_progress_bars, enable_progress_bars, tqdm, tqdm_stream_file
diff --git a/src/huggingface_hub/utils/_xet.py b/src/huggingface_hub/utils/_xet.py
new file mode 100644
index 0000000000..3db72f7792
--- /dev/null
+++ b/src/huggingface_hub/utils/_xet.py
@@ -0,0 +1,104 @@
+from dataclasses import dataclass
+from typing import Dict, Optional
+
+from .. import constants
+from . import get_session, validate_hf_hub_args
+
+
+@dataclass(frozen=True)
+class XetMetadata:
+    endpoint: str
+    access_token: str
+    expiration_unix_epoch: int
+    refresh_route: Optional[str] = None
+    file_hash: Optional[str] = None
+
+
+def parse_xet_headers(headers: Dict[str, str]) -> Optional[XetMetadata]:
+    """
+    Parse XET metadata from the HTTP headers or return None if not found.
+
+    Args:
+        headers (`Dict`):
+           HTTP headers to extract the XET metadata from.
+    Returns:
+        `XetMetadata` or `None`:
+            The metadata needed to make the request to the xet storage service.
+            Returns `None` if the headers do not contain the XET metadata.
+    """
+    # endpoint, access_token and expiration are required
+    try:
+        endpoint = headers[constants.HUGGINGFACE_HEADER_X_XET_ENDPOINT]
+        access_token = headers[constants.HUGGINGFACE_HEADER_X_XET_ACCESS_TOKEN]
+        expiration_unix_epoch = int(headers[constants.HUGGINGFACE_HEADER_X_XET_EXPIRATION])
+    except (KeyError, ValueError, TypeError):
+        return None
+
+    return XetMetadata(
+        endpoint=endpoint,
+        access_token=access_token,
+        expiration_unix_epoch=expiration_unix_epoch,
+        refresh_route=headers.get(constants.HUGGINGFACE_HEADER_X_XET_REFRESH_ROUTE),
+        file_hash=headers.get(constants.HUGGINGFACE_HEADER_X_XET_HASH),
+    )
+
+
+@validate_hf_hub_args
+def refresh_xet_metadata(
+    *,
+    xet_metadata: XetMetadata,
+    headers: Dict[str, str],
+    endpoint: Optional[str] = None,
+) -> XetMetadata:
+    """
+    Utilizes the information in the parsed metadata to request the Hub xet access token.
+
+    Args:
+        xet_metadata: (`XetMetadata`):
+            The xet metadata provided by the Hub API.
+        headers (`Dict[str, str]`):
+            Headers to use for the request, including authorization headers and user agent.
+        endpoint (`str`, `optional`):
+            The endpoint to use for the request. Defaults to the Hub endpoint.
+    Returns:
+        `XetMetadata`: The metadata needed to make the request to the xet storage service.
+    Raises:
+        [`~utils.HfHubHTTPError`]
+            If the Hub API returned an error.
+        [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError)
+            If the Hub API response is improperly formatted.
+    """
+    if xet_metadata.refresh_route is None:
+        raise ValueError("The provided xet metadata does not contain a refresh endpoint.")
+    endpoint = endpoint if endpoint is not None else constants.ENDPOINT
+    url = f"{endpoint}{xet_metadata.refresh_route}"
+    return _fetch_xet_metadata_with_url(url, headers)
+
+
+@validate_hf_hub_args
+def _fetch_xet_metadata_with_url(
+    url: str,
+    headers: Dict[str, str],
+) -> XetMetadata:
+    """
+    Requests the xet access token from the supplied URL.
+
+    Args:
+        url: (`str`):
+            The access token endpoint URL.
+        headers (`Dict[str, str]`):
+            Headers to use for the request, including authorization headers and user agent.
+    Returns:
+        `XetMetadata`:
+            The metadata needed to make the request to the xet storage service.
+    Raises:
+        [`~utils.HfHubHTTPError`]
+            If the Hub API returned an error.
+        [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError)
+            If the Hub API response is improperly formatted.
+    """
+    resp = get_session().get(headers=headers, url=url)
+    metadata = parse_xet_headers(resp.headers)  # type: ignore
+    if metadata is None:
+        raise ValueError("Xet headers have not been correctly set by the server.")
+    return metadata
diff --git a/src/huggingface_hub/utils/tqdm.py b/src/huggingface_hub/utils/tqdm.py
index b22b797c32..4c1fcef4be 100644
--- a/src/huggingface_hub/utils/tqdm.py
+++ b/src/huggingface_hub/utils/tqdm.py
@@ -84,9 +84,9 @@
 import logging
 import os
 import warnings
-from contextlib import contextmanager
+from contextlib import contextmanager, nullcontext
 from pathlib import Path
-from typing import Dict, Iterator, Optional, Union
+from typing import ContextManager, Dict, Iterator, Optional, Union
 
 from tqdm.auto import tqdm as old_tqdm
 
@@ -277,3 +277,31 @@ def _inner_read(size: Optional[int] = -1) -> bytes:
         yield f
 
         pbar.close()
+
+
+def _get_progress_bar_context(
+    *,
+    desc: str,
+    log_level: int,
+    total: Optional[int] = None,
+    initial: int = 0,
+    unit: str = "B",
+    unit_scale: bool = True,
+    name: Optional[str] = None,
+    _tqdm_bar: Optional[tqdm] = None,
+) -> ContextManager[tqdm]:
+    if _tqdm_bar is not None:
+        return nullcontext(_tqdm_bar)
+        # ^ `contextlib.nullcontext` mimics a context manager that does nothing
+        #   Makes it easier to use the same code path for both cases but in the later
+        #   case, the progress bar is not closed when exiting the context manager.
+
+    return tqdm(
+        unit=unit,
+        unit_scale=unit_scale,
+        total=total,
+        initial=initial,
+        desc=desc,
+        disable=is_tqdm_disabled(log_level=log_level),
+        name=name,
+    )
diff --git a/tests/test_xet_download.py b/tests/test_xet_download.py
new file mode 100644
index 0000000000..79b6e9a728
--- /dev/null
+++ b/tests/test_xet_download.py
@@ -0,0 +1,279 @@
+import os
+from contextlib import contextmanager
+from pathlib import Path
+from unittest.mock import DEFAULT, Mock, patch
+
+from huggingface_hub import snapshot_download
+from huggingface_hub.file_download import (
+    HfFileMetadata,
+    get_hf_file_metadata,
+    hf_hub_download,
+    hf_hub_url,
+    try_to_load_from_cache,
+    xet_get,
+)
+from huggingface_hub.utils import XetMetadata
+
+from .testing_utils import (
+    DUMMY_XET_FILE,
+    DUMMY_XET_MODEL_ID,
+    requires,
+    with_production_testing,
+)
+
+
+@requires("hf_xet")
+@with_production_testing
+class TestXetFileDownload:
+    @contextmanager
+    def _patch_xet_file_metadata(self, with_xet_metadata: bool):
+        xet_metadata = (
+            XetMetadata(
+                endpoint="mock_endpoint",
+                access_token="mock_token",
+                expiration_unix_epoch=9999999999,
+                file_hash="mock_hash",
+            )
+            if with_xet_metadata
+            else None
+        )
+        patcher = patch("huggingface_hub.file_download.get_hf_file_metadata")
+        mock_metadata = patcher.start()
+        mock_metadata.return_value = HfFileMetadata(
+            commit_hash="mock_commit",
+            etag="mock_etag",
+            location="mock_location",
+            size=1024,
+            xet_metadata=xet_metadata,
+        )
+        try:
+            yield mock_metadata
+        finally:
+            patcher.stop()
+
+    def test_xet_get_called_when_xet_metadata_present(self, tmp_path):
+        """Test that xet_get is called when xet metadata is present."""
+        with self._patch_xet_file_metadata(with_xet_metadata=True) as mock_metadata:
+            with patch("huggingface_hub.file_download.xet_get") as mock_xet_get:
+                with patch("huggingface_hub.file_download._create_symlink"):
+                    hf_hub_download(
+                        DUMMY_XET_MODEL_ID,
+                        filename=DUMMY_XET_FILE,
+                        cache_dir=tmp_path,
+                        force_download=True,
+                    )
+
+                    # Verify xet_get was called with correct parameters
+                    mock_xet_get.assert_called_once()
+                    _, kwargs = mock_xet_get.call_args
+                    assert "xet_metadata" in kwargs
+                    assert kwargs["xet_metadata"] == mock_metadata.return_value.xet_metadata
+
+    def test_backward_compatibility_no_xet_metadata(self, tmp_path):
+        """Test backward compatibility when response has no xet metadata."""
+        with self._patch_xet_file_metadata(with_xet_metadata=False):
+            with patch("huggingface_hub.file_download.http_get") as mock_http_get:
+                with patch("huggingface_hub.file_download._create_symlink"):
+                    hf_hub_download(
+                        DUMMY_XET_MODEL_ID,
+                        filename=DUMMY_XET_FILE,
+                        cache_dir=tmp_path,
+                        force_download=True,
+                    )
+
+                    # Verify http_get was called
+                    mock_http_get.assert_called_once()
+
+    def test_get_xet_file_metadata_basic(self) -> None:
+        """Test getting metadata from a file on the Hub."""
+        url = hf_hub_url(
+            repo_id=DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+        )
+        metadata = get_hf_file_metadata(url)
+        xet_metadata = metadata.xet_metadata
+        assert xet_metadata is not None
+        assert xet_metadata.endpoint is not None
+        assert xet_metadata.access_token is not None
+        assert isinstance(xet_metadata.expiration_unix_epoch, int)
+        assert xet_metadata.file_hash is not None
+        assert xet_metadata.refresh_route is not None
+
+    def test_basic_download(self, tmp_path):
+        # Make sure that xet_get is called
+        with patch("huggingface_hub.file_download.xet_get", wraps=xet_get) as _xet_get:
+            filepath = hf_hub_download(
+                DUMMY_XET_MODEL_ID,
+                filename=DUMMY_XET_FILE,
+                cache_dir=tmp_path,
+            )
+
+            assert os.path.exists(filepath)
+            assert os.path.getsize(filepath) > 0
+
+            _xet_get.assert_called_once()
+
+    def test_try_to_load_from_cache(self, tmp_path):
+        cached_path = try_to_load_from_cache(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+        assert cached_path is None
+
+        downloaded_path = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+
+        # Now should find it in cache
+        cached_path = try_to_load_from_cache(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+        assert cached_path == downloaded_path
+
+    def test_cache_reuse(self, tmp_path):
+        path1 = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+
+        assert os.path.exists(path1)
+
+        with patch("huggingface_hub.file_download._download_to_tmp_and_move") as mock:
+            # Second download should use cache
+            path2 = hf_hub_download(
+                DUMMY_XET_MODEL_ID,
+                filename=DUMMY_XET_FILE,
+                cache_dir=tmp_path,
+            )
+
+            assert path1 == path2
+            mock.assert_not_called()
+
+    def test_download_to_local_dir(self, tmp_path):
+        local_dir = tmp_path / "local_dir"
+        local_dir.mkdir(exist_ok=True, parents=True)
+
+        cache_dir = tmp_path / "cache"
+        cache_dir.mkdir(exist_ok=True, parents=True)
+        # Download to local dir
+        returned_path = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            local_dir=local_dir,
+            cache_dir=cache_dir,
+        )
+        assert local_dir in Path(returned_path).parents
+
+        for path in cache_dir.glob("**/blobs/**"):
+            assert not path.is_file()
+        for path in cache_dir.glob("**/snapshots/**"):
+            assert not path.is_file()
+
+    def test_force_download(self, tmp_path):
+        # First download
+        path1 = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+
+        # Force download should re-download even if in cache
+        with patch("huggingface_hub.file_download.xet_get") as mock_xet_get:
+            path2 = hf_hub_download(
+                DUMMY_XET_MODEL_ID,
+                filename=DUMMY_XET_FILE,
+                cache_dir=tmp_path,
+                force_download=True,
+            )
+
+            assert path1 == path2
+            mock_xet_get.assert_called_once()
+
+    def test_fallback_to_http_when_xet_not_available(self, tmp_path):
+        """Test that http_get is used when hf_xet is not available."""
+        with self._patch_xet_file_metadata(with_xet_metadata=True):
+            # Mock is_xet_available to return False
+            with patch.multiple(
+                "huggingface_hub.file_download",
+                is_xet_available=Mock(return_value=False),
+                http_get=DEFAULT,
+                xet_get=DEFAULT,
+                _create_symlink=DEFAULT,
+            ) as mocks:
+                hf_hub_download(
+                    DUMMY_XET_MODEL_ID,
+                    filename=DUMMY_XET_FILE,
+                    cache_dir=tmp_path,
+                    force_download=True,
+                )
+
+                # Verify http_get was called and xet_get was not
+                mocks["http_get"].assert_called_once()
+                mocks["xet_get"].assert_not_called()
+
+    def test_use_xet_when_available(self, tmp_path):
+        """Test that xet_get is used when hf_xet is available."""
+        with self._patch_xet_file_metadata(with_xet_metadata=True):
+            with patch.multiple(
+                "huggingface_hub.file_download",
+                is_xet_available=Mock(return_value=True),
+                http_get=DEFAULT,
+                xet_get=DEFAULT,
+                _create_symlink=DEFAULT,
+            ) as mocks:
+                hf_hub_download(
+                    DUMMY_XET_MODEL_ID,
+                    filename=DUMMY_XET_FILE,
+                    cache_dir=tmp_path,
+                    force_download=True,
+                )
+
+                # Verify xet_get was called and http_get was not
+                mocks["xet_get"].assert_called_once()
+                mocks["http_get"].assert_not_called()
+
+
+@requires("hf_xet")
+@with_production_testing
+class TestXetSnapshotDownload:
+    def test_download_model(self, tmp_path):
+        """Test that snapshot_download works with Xet storage."""
+        storage_folder = snapshot_download(
+            DUMMY_XET_MODEL_ID,
+            cache_dir=tmp_path,
+        )
+
+        assert os.path.exists(storage_folder)
+        assert os.path.isdir(storage_folder)
+        assert os.path.exists(os.path.join(storage_folder, DUMMY_XET_FILE))
+
+        with open(os.path.join(storage_folder, DUMMY_XET_FILE), "rb") as f:
+            content = f.read()
+            assert len(content) > 0
+
+    def test_snapshot_download_cache_reuse(self, tmp_path):
+        """Test that snapshot_download reuses cached files."""
+        # First download
+        storage_folder1 = snapshot_download(
+            DUMMY_XET_MODEL_ID,
+            cache_dir=tmp_path,
+        )
+
+        with patch("huggingface_hub.file_download.xet_get") as mock_xet_get:
+            # Second download should use cache
+            storage_folder2 = snapshot_download(
+                DUMMY_XET_MODEL_ID,
+                cache_dir=tmp_path,
+            )
+
+            # Verify same folder is returned
+            assert storage_folder1 == storage_folder2
+
+            # Verify xet_get was not called (files were cached)
+            mock_xet_get.assert_not_called()
diff --git a/tests/test_xet_utils.py b/tests/test_xet_utils.py
new file mode 100644
index 0000000000..6bb262ad31
--- /dev/null
+++ b/tests/test_xet_utils.py
@@ -0,0 +1,222 @@
+from unittest.mock import MagicMock
+
+import pytest
+
+from huggingface_hub import constants
+from huggingface_hub.utils._xet import (
+    XetMetadata,
+    _fetch_xet_metadata_with_url,
+    parse_xet_headers,
+    refresh_xet_metadata,
+)
+
+
+def test_parse_valid_headers_minimal() -> None:
+    headers = {
+        "X-Xet-Cas-Url": "https://xet.example.com",
+        "X-Xet-Access-Token": "xet_token_abc",
+        "X-Xet-Token-Expiration": "1234567890",
+    }
+
+    metadata = parse_xet_headers(headers)
+
+    assert metadata is not None
+    assert metadata.endpoint == "https://xet.example.com"
+    assert metadata.access_token == "xet_token_abc"
+    assert metadata.expiration_unix_epoch == 1234567890
+    assert metadata.refresh_route is None
+    assert metadata.file_hash is None
+
+
+def test_parse_valid_headers_full() -> None:
+    headers = {
+        "X-Xet-Cas-Url": "https://xet.example.com",
+        "X-Xet-Access-Token": "xet_token_abc",
+        "X-Xet-Token-Expiration": "1234567890",
+        "X-Xet-Refresh-Route": "/api/refresh",
+        "X-Xet-Hash": "sha256:abcdef",
+    }
+
+    metadata = parse_xet_headers(headers)
+
+    assert metadata is not None
+    assert metadata.endpoint == "https://xet.example.com"
+    assert metadata.access_token == "xet_token_abc"
+    assert metadata.expiration_unix_epoch == 1234567890
+    assert metadata.refresh_route == "/api/refresh"
+    assert metadata.file_hash == "sha256:abcdef"
+
+
+@pytest.mark.parametrize(
+    "missing_key",
+    [
+        "X-Xet-Cas-Url",
+        "X-Xet-Access-Token",
+        "X-Xet-Token-Expiration",
+    ],
+)
+def test_parse_missing_required_header(missing_key: str) -> None:
+    headers = {
+        "X-Xet-Cas-Url": "https://xet.example.com",
+        "X-Xet-Access-Token": "xet_token_abc",
+        "X-Xet-Token-Expiration": "1234567890",
+    }
+
+    # Remove the key to test
+    headers.pop(missing_key)
+
+    metadata = parse_xet_headers(headers)
+    assert metadata is None
+
+
+def test_parse_invalid_expiration() -> None:
+    """Test parsing headers with invalid expiration format returns None."""
+    headers = {
+        "X-Xet-Cas-Url": "https://xet.example.com",
+        "X-Xet-Access-Token": "xet_token_abc",
+        "X-Xet-Token-Expiration": "not-a-number",
+    }
+
+    metadata = parse_xet_headers(headers)
+    assert metadata is None
+
+
+def test_refresh_metadata_success(mocker) -> None:
+    initial_metadata = XetMetadata(
+        endpoint="https://example.xethub.hf.co",
+        access_token="old_token",
+        expiration_unix_epoch=1234567890,
+        refresh_route="/api/models/username/repo_name/xet-read-token/token",
+    )
+
+    # Mock headers for the refreshed response
+    mock_response = MagicMock()
+    mock_response.headers = {
+        "X-Xet-Cas-Url": "https://example.xethub.hf.co",
+        "X-Xet-Access-Token": "new_token",
+        "X-Xet-Token-Expiration": "1234599999",
+        "X-Xet-Refresh-Route": "/api/models/username/repo_name/xet-read-token/token",
+    }
+
+    mock_session = MagicMock()
+    mock_session.get.return_value = mock_response
+    mocker.patch("huggingface_hub.utils._xet.get_session", return_value=mock_session)
+
+    headers = {"user-agent": "user-agent-example"}
+    refreshed_metadata = refresh_xet_metadata(
+        xet_metadata=initial_metadata,
+        headers=headers,
+    )
+
+    # Verify the request
+    expected_url = f"{constants.ENDPOINT}/api/models/username/repo_name/xet-read-token/token"
+    mock_session.get.assert_called_once_with(
+        headers=headers,
+        url=expected_url,
+    )
+
+    assert refreshed_metadata.endpoint == "https://example.xethub.hf.co"
+    assert refreshed_metadata.access_token == "new_token"
+    assert refreshed_metadata.expiration_unix_epoch == 1234599999
+
+
+def test_refresh_metadata_custom_endpoint(mocker) -> None:
+    initial_metadata = XetMetadata(
+        endpoint="https://example.xethub.hf.co",
+        access_token="old_token",
+        expiration_unix_epoch=1234567890,
+        refresh_route="/api/models/username/repo_name/xet-read-token/token",
+    )
+
+    custom_endpoint = "https://custom.xethub.hf.co"
+
+    # Mock headers for the refreshed response
+    mock_response = MagicMock()
+    mock_response.headers = {
+        "X-Xet-Cas-Url": "https://custom.xethub.hf.co",
+        "X-Xet-Access-Token": "new_token",
+        "X-Xet-Token-Expiration": "1234599999",
+    }
+
+    mock_session = MagicMock()
+    mock_session.get.return_value = mock_response
+    mocker.patch("huggingface_hub.utils._xet.get_session", return_value=mock_session)
+
+    headers = {"user-agent": "user-agent-example"}
+    refresh_xet_metadata(
+        xet_metadata=initial_metadata,
+        headers=headers,
+        endpoint=custom_endpoint,
+    )
+
+    # Verify the request used the custom endpoint
+    expected_url = f"{custom_endpoint}/api/models/username/repo_name/xet-read-token/token"
+    mock_session.get.assert_called_once_with(
+        headers=headers,
+        url=expected_url,
+    )
+
+
+def test_refresh_metadata_missing_refresh_route() -> None:
+    # Create metadata without refresh_route
+    metadata = XetMetadata(
+        endpoint="https://example.xethub.hf.co",
+        access_token="token123",
+        expiration_unix_epoch=1234567890,
+        # No refresh_route
+    )
+
+    headers = {"user-agent": "user-agent-example"}
+
+    # Verify it raises ValueError
+    with pytest.raises(ValueError, match="The provided xet metadata does not contain a refresh endpoint."):
+        refresh_xet_metadata(
+            xet_metadata=metadata,
+            headers=headers,
+        )
+
+
+def test_fetch_xet_metadata_with_url(mocker) -> None:
+    mock_response = MagicMock()
+    mock_response.headers = {
+        "X-Xet-Cas-Url": "https://example.xethub.hf.co",
+        "X-Xet-Access-Token": "xet_token123",
+        "X-Xet-Token-Expiration": "1234567890",
+    }
+
+    # Mock the session.get method
+    mock_session = MagicMock()
+    mock_session.get.return_value = mock_response
+    mocker.patch("huggingface_hub.utils._xet.get_session", return_value=mock_session)
+
+    # Call the function
+    url = "https://example.xethub.hf.co/api/models/username/repo_name/xet-read-token/token"
+    headers = {"user-agent": "user-agent-example"}
+    metadata = _fetch_xet_metadata_with_url(url=url, headers=headers)
+
+    # Verify the request
+    mock_session.get.assert_called_once_with(
+        headers=headers,
+        url=url,
+    )
+
+    # Verify returned metadata
+    assert metadata.endpoint == "https://example.xethub.hf.co"
+    assert metadata.access_token == "xet_token123"
+    assert metadata.expiration_unix_epoch == 1234567890
+
+
+def test_fetch_xet_metadata_with_url_invalid_response(mocker) -> None:
+    mock_response = MagicMock()
+    mock_response.headers = {"Content-Type": "application/json"}  # No XET headers
+
+    # Mock the session.get method
+    mock_session = MagicMock()
+    mock_session.get.return_value = mock_response
+    mocker.patch("huggingface_hub.utils._xet.get_session", return_value=mock_session)
+
+    url = "https://example.xethub.hf.co/api/models/username/repo_name/xet-read-token/token"
+    headers = {"user-agent": "user-agent-example"}
+
+    with pytest.raises(ValueError, match="Xet headers have not been correctly set by the server."):
+        _fetch_xet_metadata_with_url(url=url, headers=headers)
diff --git a/tests/testing_utils.py b/tests/testing_utils.py
index 3a18ca1706..27ce7b6389 100644
--- a/tests/testing_utils.py
+++ b/tests/testing_utils.py
@@ -50,6 +50,12 @@
 NO = ("n", "no", "f", "false", "off", "0")
 
 
+# Xet testing
+DUMMY_XET_MODEL_ID = "celinah/dummy-xet-testing"
+DUMMY_XET_FILE = "dummy.safetensors"
+DUMMY_XET_REGULAR_FILE = "dummy.txt"
+
+
 def repo_name(id: Optional[str] = None, prefix: str = "repo") -> str:
     """
     Return a readable pseudo-unique repository name for tests.


Reviews:
user: login: Wauplin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T10:01:02Z
user: login: Wauplin
body: Left some comments but overall looks in pretty good shape!

state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T17:36:52Z
user: login: bpronan
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-28T19:18:50Z
user: login: bpronan
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-28T19:24:01Z
user: login: bpronan
body: Looks great. The test coverage is fantastic.

The condition checking for `hf_xet` installation is the main blocker.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-28T19:47:18Z
user: login: hanouticelina
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T09:27:23Z
user: login: hanouticelina
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T10:30:02Z
user: login: hanouticelina
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T10:52:23Z
user: login: Wauplin
body: Yay! Approving the PR since I've left only cosmetic/refactoring comments. Very nice job on the test side :) 


(is it still a draft PR btw?)
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T11:46:30Z
user: login: hanouticelina
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T13:42:48Z
user: login: hanouticelina
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T13:43:22Z
user: login: hanouticelina
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T13:56:25Z
user: login: hanouticelina
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T13:56:38Z
user: login: Wauplin
body: Final approval! :heavy_check_mark: 

From the PR description, there is still a documentation bullet point but let's handle it in a separate PR. Ok to merge once we get a last review from Xet team :)
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T14:41:27Z
user: login: bpronan
body: Looks great!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-03T19:23:04Z
user: login: rajatarya
body: Love it!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T20:25:12Z

Review comments:
diff_hunk: @@ -0,0 +1,77 @@
+import os
+from unittest.mock import patch
+
+import pytest
+
+from huggingface_hub.file_download import get_hf_file_metadata, hf_hub_download, hf_hub_url, try_to_load_from_cache
+
+from .testing_utils import (
+    DUMMY_XET_MODEL_ID,
+    requires,
+    with_production_testing,
+)
+
+
+@requires("hf_xet")
+@with_production_testing
+class TestXetDownload:
+    file_name = "dummy.txt"
+    xet_file_name = "dummy.safetensors"
path: tests/test_xet_download.py
user: login: Wauplin
body: since `DUMMY_XET_MODEL_ID` is a constant, I'd make them some constants as well (`DUMMY_XET_FILE` / `DUMMY_XET_REGULAR_FILE` ?)
created_at: 2025-02-27T10:01:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -487,6 +493,118 @@ def http_get(
         )
 
 
+def xet_get(
+    incomplete_path: Path,
path: src/huggingface_hub/file_download.py
user: login: Wauplin
body: ```suggestion
    *,
    incomplete_path: Path,
```

(nit) let's force keyword argument, easier to change things in the future
created_at: 2025-02-28T17:15:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -487,6 +493,118 @@ def http_get(
         )
 
 
+def xet_get(
+    incomplete_path: Path,
+    xet_metadata: XetMetadata,
+    headers: Dict[str, str],
+    expected_size: Optional[int] = None,
+    displayed_filename: Optional[str] = None,
+    _tqdm_bar: Optional[tqdm] = None,
+) -> None:
+    """
+    Download a file using Xet storage service.
+
+    Args:
+        incomplete_path (`Path`):
+            The path to the file to download.
+        xet_metadata (`XetMetadata`):
+            The metadata needed to make the request to the xet storage service.
+        headers (`Dict[str, str]`):
+            The headers to send to the xet storage service.
+        expected_size (`int`, *optional*):
+            The expected size of the file to download. If set, the download will raise an error if the size of the
+            received content is different from the expected one.
+        displayed_filename (`str`, *optional*):
+            The filename of the file that is being downloaded. Value is used only to display a nice progress bar. If
+            not set, the filename is guessed from the URL or the `Content-Disposition` header.
+
+    **How it works:**
+        The file download system uses Xet storage, which is a content-addressable storage system that breaks files into chunks
+        for efficient storage and transfer.
+
+        `hf_xet.download_files` manages downloading files by:
+        - Taking a list of files to download (each with its unique content hash)
+        - Connecting to a storage server (CAS server) that knows how files are chunked
+        - Using authentication to ensure secure access
+        - Providing progress updates during download
+
+        Authentication works by regularly refreshing access tokens through `refresh_xet_metadata` to maintain a valid
+        connection to the storage server.
+
+        The download process works like this:
+        1. Creates a local cache folder at `~/.cache/huggingface/xet/chunk-cache` to store reusable file chunks
+        2. Downloads files in parallel:
+            2.1. Prepares to write the file to disk
+            2.2. Asks the server "how is this file split into chunks?" using the file's unique hash
+                The server responds with:
+                - Which chunks make up the complete file
+                - Where each chunk can be downloaded from
+            2.3. For each needed chunk:
+                - Checks if we already have it in our local cache
+                - If not, downloads it from cloud storage (S3)
+                - Saves it to cache for future use
+                - Assembles the chunks in order to recreate the original file
path: src/huggingface_hub/file_download.py
user: login: Wauplin
body: ```suggestion
        1. Create a local cache folder at `~/.cache/huggingface/xet/chunk-cache` to store reusable file chunks
        2. Download files in parallel:
            2.1. Prepare to write the file to disk
            2.2. Ask the server "how is this file split into chunks?" using the file's unique hash
                The server responds with:
                - Which chunks make up the complete file
                - Where each chunk can be downloaded from
            2.3. For each needed chunk:
                - Check if we already have it in our local cache
                - If not, download it from cloud storage (S3)
                - Save it to cache for future use
                - Assemble the chunks in order to recreate the original file
```

(personal preference for "instructions")
created_at: 2025-02-28T17:19:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -487,6 +493,118 @@ def http_get(
         )
 
 
+def xet_get(
+    incomplete_path: Path,
+    xet_metadata: XetMetadata,
+    headers: Dict[str, str],
+    expected_size: Optional[int] = None,
+    displayed_filename: Optional[str] = None,
+    _tqdm_bar: Optional[tqdm] = None,
+) -> None:
+    """
+    Download a file using Xet storage service.
+
+    Args:
+        incomplete_path (`Path`):
+            The path to the file to download.
+        xet_metadata (`XetMetadata`):
+            The metadata needed to make the request to the xet storage service.
+        headers (`Dict[str, str]`):
+            The headers to send to the xet storage service.
+        expected_size (`int`, *optional*):
+            The expected size of the file to download. If set, the download will raise an error if the size of the
+            received content is different from the expected one.
+        displayed_filename (`str`, *optional*):
+            The filename of the file that is being downloaded. Value is used only to display a nice progress bar. If
+            not set, the filename is guessed from the URL or the `Content-Disposition` header.
+
+    **How it works:**
+        The file download system uses Xet storage, which is a content-addressable storage system that breaks files into chunks
+        for efficient storage and transfer.
+
+        `hf_xet.download_files` manages downloading files by:
+        - Taking a list of files to download (each with its unique content hash)
+        - Connecting to a storage server (CAS server) that knows how files are chunked
+        - Using authentication to ensure secure access
+        - Providing progress updates during download
+
+        Authentication works by regularly refreshing access tokens through `refresh_xet_metadata` to maintain a valid
+        connection to the storage server.
+
+        The download process works like this:
+        1. Creates a local cache folder at `~/.cache/huggingface/xet/chunk-cache` to store reusable file chunks
+        2. Downloads files in parallel:
+            2.1. Prepares to write the file to disk
+            2.2. Asks the server "how is this file split into chunks?" using the file's unique hash
+                The server responds with:
+                - Which chunks make up the complete file
+                - Where each chunk can be downloaded from
+            2.3. For each needed chunk:
+                - Checks if we already have it in our local cache
+                - If not, downloads it from cloud storage (S3)
+                - Saves it to cache for future use
+                - Assembles the chunks in order to recreate the original file
+
+    """
+    try:
+        from hf_xet import PyPointerFile, download_files  # type: ignore[no-redef]
+    except ImportError:
+        raise ValueError(
+            "To use optimized download using Xet storage, you need to install the hf_xet package. "
+            "Try `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`."
+        )
+
+    def token_refresher() -> Tuple[str, int]:
+        new_xet_metadata = refresh_xet_metadata(xet_metadata=xet_metadata, headers=headers)
+        if new_xet_metadata is None:
+            raise ValueError("Failed to refresh token using xet metadata.")
+        return new_xet_metadata.access_token, new_xet_metadata.expiration_unix_epoch
+
+    pointer_files = [
+        PyPointerFile(path=str(incomplete_path.absolute()), hash=xet_metadata.file_hash, filesize=expected_size)
+    ]
+
+    if not displayed_filename:
+        displayed_filename = incomplete_path.name
+
+    # Truncate filename if too long to display
+    if len(displayed_filename) > 40:
+        displayed_filename = f"{displayed_filename[:40]}(…)"
+
+    # Stream file to buffer
+    progress_cm: tqdm = (
+        tqdm(  # type: ignore[assignment]
+            unit="B",
+            unit_scale=True,
+            total=expected_size,
+            initial=0,
+            desc=displayed_filename,
+            disable=True if (logger.getEffectiveLevel() == logging.NOTSET) else None,
+            # ^ set `disable=None` rather than `disable=False` by default to disable progress bar when no TTY attached
+            # see https://github.com/huggingface/huggingface_hub/pull/2000
+            name="huggingface_hub.xet_get",
+        )
+        if _tqdm_bar is None
+        else contextlib.nullcontext(_tqdm_bar)
+        # ^ `contextlib.nullcontext` mimics a context manager that does nothing
+        #   Makes it easier to use the same code path for both cases but in the later
+        #   case, the progress bar is not closed when exiting the context manager.
+    )
path: src/huggingface_hub/file_download.py
user: login: Wauplin
body: This could be abstracted in an utility somewhere (same as `http_get`)
created_at: 2025-02-28T17:21:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1544,14 +1673,29 @@ def _download_to_tmp_and_move(
             _check_disk_space(expected_size, incomplete_path.parent)
             _check_disk_space(expected_size, destination_path.parent)
 
-        http_get(
-            url_to_download,
-            f,
-            proxies=proxies,
-            resume_size=resume_size,
-            headers=headers,
-            expected_size=expected_size,
-        )
+        if xet_metadata is not None and xet_metadata.file_hash is not None:
path: src/huggingface_hub/file_download.py
user: login: Wauplin
body: I think we should either:
1. make `hf_xet` a default dependency
2. or use the xet-path only if `hf_xet` is installed.

For now, I'd go with 2. so it's an opt-in process. The problem with current workflow (AFAIU) is that if `hf_xet` is not installed and a repo becomes "xet-enabled" then users won't be able to download things, even though they haven't changed anything to their setup.

Feel free to ignore if I misunderstood it :grimacing: 
created_at: 2025-02-28T17:25:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -107,4 +107,9 @@
 from ._telemetry import send_telemetry
 from ._typing import is_jsonable, is_simple_optional_type, unwrap_simple_optional_type
 from ._validators import smoothly_deprecate_use_auth_token, validate_hf_hub_args, validate_repo_id
+from ._xet import (
+    XetMetadata,
+    parse_xet_headers,
+    refresh_xet_metadata,
+)
path: src/huggingface_hub/utils/__init__.py
user: login: Wauplin
body: ```suggestion
from ._xet import XetMetadata, parse_xet_headers, refresh_xet_metadata
```
(nit)
created_at: 2025-02-28T17:27:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,222 @@
+from unittest.mock import MagicMock
+
+import pytest
+
+from huggingface_hub import constants
+from huggingface_hub.utils._xet import (
+    XetMetadata,
+    _fetch_xet_metadata_with_url,
+    parse_xet_headers,
+    refresh_xet_metadata,
+)
+
+
+def test_parse_valid_headers_minimal() -> None:
+    headers = {
+        constants.HUGGINGFACE_HEADER_X_XET_ENDPOINT: "https://xet.example.com",
+        constants.HUGGINGFACE_HEADER_X_XET_ACCESS_TOKEN: "xet_token_abc",
+        constants.HUGGINGFACE_HEADER_X_XET_EXPIRATION: "1234567890",
path: tests/test_xet_utils.py
user: login: Wauplin
body: Let's not use constants in this test module to make the expected headers more readable (and also detect issue if constant value is updated without a good reason)
created_at: 2025-02-28T17:30:41Z
author_association: CONTRIBUTOR
path: tests/test_xet_utils.py
user: login: Wauplin
body: Nice self-contained tests!
created_at: 2025-02-28T17:31:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,226 @@
+import os
+from pathlib import Path
+from unittest.mock import patch
+
+from huggingface_hub import snapshot_download
+from huggingface_hub.file_download import (
+    HfFileMetadata,
+    get_hf_file_metadata,
+    hf_hub_download,
+    hf_hub_url,
+    try_to_load_from_cache,
+)
+from huggingface_hub.utils import XetMetadata
+
+from .testing_utils import (
+    DUMMY_XET_FILE,
+    DUMMY_XET_MODEL_ID,
+    requires,
+    with_production_testing,
+)
+
+
+@requires("hf_xet")
+@with_production_testing
+class TestXetFileDownload:
+    def test_xet_get_called_when_xet_metadata_present(self, tmp_path):
+        """Test that xet_get is called when xet metadata is present."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            xet_metadata = XetMetadata(
+                endpoint="mock_endpoint",
+                access_token="mock_token",
+                expiration_unix_epoch=9999999999,
+                file_hash="mock_hash",
+            )
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=xet_metadata,
+            )
+
+            with patch("huggingface_hub.file_download.xet_get") as mock_xet_get:
+                with patch("huggingface_hub.file_download._create_symlink"):
+                    hf_hub_download(
+                        DUMMY_XET_MODEL_ID,
+                        filename=DUMMY_XET_FILE,
+                        cache_dir=tmp_path,
+                        force_download=True,
+                    )
+
+                    # Verify xet_get was called with correct parameters
+                    mock_xet_get.assert_called_once()
+                    _, kwargs = mock_xet_get.call_args
+                    assert "xet_metadata" in kwargs
+                    assert kwargs["xet_metadata"] == xet_metadata
+
+    def test_backward_compatibility_no_xet_metadata(self, tmp_path):
+        """Test backward compatibility when response has no xet metadata."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=None,  # No xet_metadata
+            )
+
+            with patch("huggingface_hub.file_download.http_get") as mock_http_get:
+                with patch("huggingface_hub.file_download._create_symlink"):
+                    hf_hub_download(
+                        DUMMY_XET_MODEL_ID,
+                        filename=DUMMY_XET_FILE,
+                        cache_dir=tmp_path,
+                        force_download=True,
+                    )
+
+                    # Verify http_get was called
+                    mock_http_get.assert_called_once()
+
+    def test_get_xet_file_metadata_basic(self) -> None:
+        """Test getting metadata from a file on the Hub."""
+        url = hf_hub_url(
+            repo_id=DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+        )
+        metadata = get_hf_file_metadata(url)
+        xet_metadata = metadata.xet_metadata
+        assert xet_metadata is not None
+        assert xet_metadata.endpoint is not None
+        assert xet_metadata.access_token is not None
+        assert isinstance(xet_metadata.expiration_unix_epoch, int)
+        assert xet_metadata.file_hash is not None
+        assert xet_metadata.refresh_route is not None
+
+    def test_basic_download(self, tmp_path):
path: tests/test_xet_download.py
user: login: Wauplin
body: is it possible to test file has been downloaded using Xet? With something like
```
with patch("huggingface_hub.file_download.get_hf_file_metadata", side_effect=huggingface_hub.file_download.get_hf_file_metadata)
```
this way should should have the best of both worlds: test that xet is used (as done above but without mocked values) + test that download worked
created_at: 2025-02-28T17:36:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1544,14 +1673,29 @@ def _download_to_tmp_and_move(
             _check_disk_space(expected_size, incomplete_path.parent)
             _check_disk_space(expected_size, destination_path.parent)
 
-        http_get(
-            url_to_download,
-            f,
-            proxies=proxies,
-            resume_size=resume_size,
-            headers=headers,
-            expected_size=expected_size,
-        )
+        if xet_metadata is not None and xet_metadata.file_hash is not None:
path: src/huggingface_hub/file_download.py
user: login: bpronan
body: +1 for using the xet-path only if `hf_xet` is installed.

If we ensure that it's installed and the repo becomes "xet-enabled", the user can still download the content even without `hf_xet` installed. We have backwards compatibility built into our service to support these cases. The service streams the entire file for download if you hit that endpoint.
created_at: 2025-02-28T19:18:49Z
author_association: COLLABORATOR
diff_hunk: @@ -1544,14 +1673,29 @@ def _download_to_tmp_and_move(
             _check_disk_space(expected_size, incomplete_path.parent)
             _check_disk_space(expected_size, destination_path.parent)
 
-        http_get(
-            url_to_download,
-            f,
-            proxies=proxies,
-            resume_size=resume_size,
-            headers=headers,
-            expected_size=expected_size,
-        )
+        if xet_metadata is not None and xet_metadata.file_hash is not None:
+            logger.info("Xet Storage is enabled for this repo. Downloading file from Xet Storage..")
+            xet_get(
+                incomplete_path,
+                xet_metadata=xet_metadata,
+                headers=headers,
+                expected_size=expected_size,
+                displayed_filename=filename,
+            )
+
+            # TODO: xetpoc - the http_get path is building this out, so we're replicating that logic here
path: src/huggingface_hub/file_download.py
user: login: bpronan
body: I would prefer if we could either address this TODO or remove it if the logic fits in the greater download action.
created_at: 2025-02-28T19:24:01Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,226 @@
+import os
+from pathlib import Path
+from unittest.mock import patch
+
+from huggingface_hub import snapshot_download
+from huggingface_hub.file_download import (
+    HfFileMetadata,
+    get_hf_file_metadata,
+    hf_hub_download,
+    hf_hub_url,
+    try_to_load_from_cache,
+)
+from huggingface_hub.utils import XetMetadata
+
+from .testing_utils import (
+    DUMMY_XET_FILE,
+    DUMMY_XET_MODEL_ID,
+    requires,
+    with_production_testing,
+)
+
+
+@requires("hf_xet")
+@with_production_testing
+class TestXetFileDownload:
+    def test_xet_get_called_when_xet_metadata_present(self, tmp_path):
+        """Test that xet_get is called when xet metadata is present."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            xet_metadata = XetMetadata(
+                endpoint="mock_endpoint",
+                access_token="mock_token",
+                expiration_unix_epoch=9999999999,
+                file_hash="mock_hash",
+            )
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=xet_metadata,
+            )
+
+            with patch("huggingface_hub.file_download.xet_get") as mock_xet_get:
+                with patch("huggingface_hub.file_download._create_symlink"):
+                    hf_hub_download(
+                        DUMMY_XET_MODEL_ID,
+                        filename=DUMMY_XET_FILE,
+                        cache_dir=tmp_path,
+                        force_download=True,
+                    )
+
+                    # Verify xet_get was called with correct parameters
+                    mock_xet_get.assert_called_once()
+                    _, kwargs = mock_xet_get.call_args
+                    assert "xet_metadata" in kwargs
+                    assert kwargs["xet_metadata"] == xet_metadata
+
+    def test_backward_compatibility_no_xet_metadata(self, tmp_path):
+        """Test backward compatibility when response has no xet metadata."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=None,  # No xet_metadata
+            )
+
+            with patch("huggingface_hub.file_download.http_get") as mock_http_get:
+                with patch("huggingface_hub.file_download._create_symlink"):
+                    hf_hub_download(
+                        DUMMY_XET_MODEL_ID,
+                        filename=DUMMY_XET_FILE,
+                        cache_dir=tmp_path,
+                        force_download=True,
+                    )
+
+                    # Verify http_get was called
+                    mock_http_get.assert_called_once()
+
+    def test_get_xet_file_metadata_basic(self) -> None:
+        """Test getting metadata from a file on the Hub."""
+        url = hf_hub_url(
+            repo_id=DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+        )
+        metadata = get_hf_file_metadata(url)
+        xet_metadata = metadata.xet_metadata
+        assert xet_metadata is not None
+        assert xet_metadata.endpoint is not None
+        assert xet_metadata.access_token is not None
+        assert isinstance(xet_metadata.expiration_unix_epoch, int)
+        assert xet_metadata.file_hash is not None
+        assert xet_metadata.refresh_route is not None
+
+    def test_basic_download(self, tmp_path):
path: tests/test_xet_download.py
user: login: hanouticelina
body: indeed, nice suggestion! in [c6c601c](https://github.com/huggingface/huggingface_hub/pull/2875/commits/c6c601c7f929880c4bcac4afbb2189d3edb08f7a), I added :
```python
with patch("huggingface_hub.file_download.xet_get", wraps=xet_get) as _xet_get:
```
to make sure that the (original) xet_get function is called here.
created_at: 2025-03-03T09:27:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1544,14 +1673,29 @@ def _download_to_tmp_and_move(
             _check_disk_space(expected_size, incomplete_path.parent)
             _check_disk_space(expected_size, destination_path.parent)
 
-        http_get(
-            url_to_download,
-            f,
-            proxies=proxies,
-            resume_size=resume_size,
-            headers=headers,
-            expected_size=expected_size,
-        )
+        if xet_metadata is not None and xet_metadata.file_hash is not None:
path: src/huggingface_hub/file_download.py
user: login: hanouticelina
body: addressed in [314130e](https://github.com/huggingface/huggingface_hub/pull/2875/commits/314130e2ad34c435dcfc79ae1e1cf413af1c0e0f). I added another level of if-else to be able to warn the user in that case and added tests for that.
created_at: 2025-03-03T10:30:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1544,14 +1673,29 @@ def _download_to_tmp_and_move(
             _check_disk_space(expected_size, incomplete_path.parent)
             _check_disk_space(expected_size, destination_path.parent)
 
-        http_get(
-            url_to_download,
-            f,
-            proxies=proxies,
-            resume_size=resume_size,
-            headers=headers,
-            expected_size=expected_size,
-        )
+        if xet_metadata is not None and xet_metadata.file_hash is not None:
+            logger.info("Xet Storage is enabled for this repo. Downloading file from Xet Storage..")
+            xet_get(
+                incomplete_path,
+                xet_metadata=xet_metadata,
+                headers=headers,
+                expected_size=expected_size,
+                displayed_filename=filename,
+            )
+
+            # TODO: xetpoc - the http_get path is building this out, so we're replicating that logic here
path: src/huggingface_hub/file_download.py
user: login: hanouticelina
body: I think this explicit directory creation is likely unnecessary since we're calling `_chmod_and_move` that uses `shutil.move()` which will automatically create the destination directory if needed when copying files (right? @Wauplin). anyways, i removed it for now
created_at: 2025-03-03T10:52:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1544,14 +1652,39 @@ def _download_to_tmp_and_move(
             _check_disk_space(expected_size, incomplete_path.parent)
             _check_disk_space(expected_size, destination_path.parent)
 
-        http_get(
-            url_to_download,
-            f,
-            proxies=proxies,
-            resume_size=resume_size,
-            headers=headers,
-            expected_size=expected_size,
-        )
+        if xet_metadata is not None and xet_metadata.file_hash is not None:
+            if is_xet_available():
+                logger.info("Xet Storage is enabled for this repo. Downloading file from Xet Storage..")
+                xet_get(
+                    incomplete_path=incomplete_path,
+                    xet_metadata=xet_metadata,
+                    headers=headers,
+                    expected_size=expected_size,
+                    displayed_filename=filename,
+                )
+            else:
+                logger.warning(
+                    "Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. "
+                    "Falling back to regular HTTP download. "
+                    "For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`"
+                )
+                http_get(
+                    url_to_download,
+                    f,
+                    proxies=proxies,
+                    resume_size=resume_size,
+                    headers=headers,
+                    expected_size=expected_size,
+                )
+        else:
+            http_get(
+                url_to_download,
+                f,
+                proxies=proxies,
+                resume_size=resume_size,
+                headers=headers,
+                expected_size=expected_size,
+            )
path: src/huggingface_hub/file_download.py
user: login: Wauplin
body: (nit) let's collate "http_get" calls into one

```suggestion
    is_xet_enabled = xet_metadata is not None and xet_metadata.file_hash is not None
    if is_xet_enabled and is_xet_available():
        logger.info("Xet Storage is enabled for this repo. Downloading file from Xet Storage..")
        xet_get(
            incomplete_path=incomplete_path,
            xet_metadata=xet_metadata,
            headers=headers,
            expected_size=expected_size,
            displayed_filename=filename,
        )
    else:
        if is_xet_enabled:
            logger.warning(
                "Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. "
                "Falling back to regular HTTP download. "
                "For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`"
            )
        http_get(
            url_to_download,
            f,
            proxies=proxies,
            resume_size=resume_size,
            headers=headers,
            expected_size=expected_size,
        )
```
created_at: 2025-03-03T11:38:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -277,3 +277,28 @@ def _inner_read(size: Optional[int] = -1) -> bytes:
         yield f
 
         pbar.close()
+
+
+def _get_progress_bar_context(
+    *,
+    desc: str,
+    log_level: int,
+    total: Optional[int] = None,
+    initial: int = 0,
+    unit: str = "B",
+    unit_scale: bool = True,
+    name: Optional[str] = None,
+    _tqdm_bar: Optional[tqdm] = None,
+) -> ContextManager[tqdm]:
+    if _tqdm_bar is not None:
+        return nullcontext(_tqdm_bar)
path: src/huggingface_hub/utils/tqdm.py
user: login: Wauplin
body: let's keep existing comment just in case

```suggestion
        return nullcontext(_tqdm_bar)
        # ^ `contextlib.nullcontext` mimics a context manager that does nothing
        #   Makes it easier to use the same code path for both cases but in the later
        #   case, the progress bar is not closed when exiting the context manager.
```
created_at: 2025-03-03T11:39:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,300 @@
+import os
+from pathlib import Path
+from unittest.mock import DEFAULT, Mock, patch
+
+from huggingface_hub import snapshot_download
+from huggingface_hub.file_download import (
+    HfFileMetadata,
+    get_hf_file_metadata,
+    hf_hub_download,
+    hf_hub_url,
+    try_to_load_from_cache,
+    xet_get,
+)
+from huggingface_hub.utils import XetMetadata
+
+from .testing_utils import (
+    DUMMY_XET_FILE,
+    DUMMY_XET_MODEL_ID,
+    requires,
+    with_production_testing,
+)
+
+
+@requires("hf_xet")
+@with_production_testing
+class TestXetFileDownload:
+    def test_xet_get_called_when_xet_metadata_present(self, tmp_path):
+        """Test that xet_get is called when xet metadata is present."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            xet_metadata = XetMetadata(
+                endpoint="mock_endpoint",
+                access_token="mock_token",
+                expiration_unix_epoch=9999999999,
+                file_hash="mock_hash",
+            )
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=xet_metadata,
+            )
+
+            with patch("huggingface_hub.file_download.xet_get") as mock_xet_get:
+                with patch("huggingface_hub.file_download._create_symlink"):
+                    hf_hub_download(
+                        DUMMY_XET_MODEL_ID,
+                        filename=DUMMY_XET_FILE,
+                        cache_dir=tmp_path,
+                        force_download=True,
+                    )
+
+                    # Verify xet_get was called with correct parameters
+                    mock_xet_get.assert_called_once()
+                    _, kwargs = mock_xet_get.call_args
+                    assert "xet_metadata" in kwargs
+                    assert kwargs["xet_metadata"] == xet_metadata
+
+    def test_backward_compatibility_no_xet_metadata(self, tmp_path):
+        """Test backward compatibility when response has no xet metadata."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=None,  # No xet_metadata
+            )
+
+            with patch("huggingface_hub.file_download.http_get") as mock_http_get:
+                with patch("huggingface_hub.file_download._create_symlink"):
+                    hf_hub_download(
+                        DUMMY_XET_MODEL_ID,
+                        filename=DUMMY_XET_FILE,
+                        cache_dir=tmp_path,
+                        force_download=True,
+                    )
+
+                    # Verify http_get was called
+                    mock_http_get.assert_called_once()
+
+    def test_get_xet_file_metadata_basic(self) -> None:
+        """Test getting metadata from a file on the Hub."""
+        url = hf_hub_url(
+            repo_id=DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+        )
+        metadata = get_hf_file_metadata(url)
+        xet_metadata = metadata.xet_metadata
+        assert xet_metadata is not None
+        assert xet_metadata.endpoint is not None
+        assert xet_metadata.access_token is not None
+        assert isinstance(xet_metadata.expiration_unix_epoch, int)
+        assert xet_metadata.file_hash is not None
+        assert xet_metadata.refresh_route is not None
+
+    def test_basic_download(self, tmp_path):
+        # Make sure that xet_get is called
+        with patch("huggingface_hub.file_download.xet_get", wraps=xet_get) as _xet_get:
+            filepath = hf_hub_download(
+                DUMMY_XET_MODEL_ID,
+                filename=DUMMY_XET_FILE,
+                cache_dir=tmp_path,
+            )
+
+            assert os.path.exists(filepath)
+            assert os.path.getsize(filepath) > 0
+
+            _xet_get.assert_called_once()
+
+    def test_try_to_load_from_cache(self, tmp_path):
+        cached_path = try_to_load_from_cache(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+        assert cached_path is None
+
+        downloaded_path = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+
+        # Now should find it in cache
+        cached_path = try_to_load_from_cache(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+        assert cached_path == downloaded_path
+
+    def test_cache_reuse(self, tmp_path):
+        path1 = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+
+        assert os.path.exists(path1)
+
+        with patch("huggingface_hub.file_download._download_to_tmp_and_move") as mock:
+            # Second download should use cache
+            path2 = hf_hub_download(
+                DUMMY_XET_MODEL_ID,
+                filename=DUMMY_XET_FILE,
+                cache_dir=tmp_path,
+            )
+
+            assert path1 == path2
+            mock.assert_not_called()
+
+    def test_download_to_local_dir(self, tmp_path):
+        local_dir = tmp_path / "local_dir"
+        local_dir.mkdir(exist_ok=True, parents=True)
+
+        cache_dir = tmp_path / "cache"
+        cache_dir.mkdir(exist_ok=True, parents=True)
+        # Download to local dir
+        returned_path = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            local_dir=local_dir,
+            cache_dir=cache_dir,
+        )
+        assert local_dir in Path(returned_path).parents
+
+        for path in cache_dir.glob("**/blobs/**"):
+            assert not path.is_file()
+        for path in cache_dir.glob("**/snapshots/**"):
+            assert not path.is_file()
+
+    def test_force_download(self, tmp_path):
+        # First download
+        path1 = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+
+        # Force download should re-download even if in cache
+        with patch("huggingface_hub.file_download.xet_get") as mock_xet_get:
+            path2 = hf_hub_download(
+                DUMMY_XET_MODEL_ID,
+                filename=DUMMY_XET_FILE,
+                cache_dir=tmp_path,
+                force_download=True,
+            )
+
+            assert path1 == path2
+            mock_xet_get.assert_called_once()
+
+    def test_fallback_to_http_when_xet_not_available(self, tmp_path):
+        """Test that http_get is used when hf_xet is not available."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=XetMetadata(
+                    endpoint="mock_endpoint",
+                    access_token="mock_token",
+                    expiration_unix_epoch=9999999999,
+                    file_hash="mock_hash",
+                ),
+            )
+
+            # Mock is_xet_available to return False
+            with patch.multiple(
+                "huggingface_hub.file_download",
+                is_xet_available=Mock(return_value=False),
+                http_get=DEFAULT,
+                xet_get=DEFAULT,
+                _create_symlink=DEFAULT,
+            ) as mocks:
+                hf_hub_download(
+                    DUMMY_XET_MODEL_ID,
+                    filename=DUMMY_XET_FILE,
+                    cache_dir=tmp_path,
+                    force_download=True,
+                )
+
+                # Verify http_get was called and xet_get was not
+                mocks["http_get"].assert_called_once()
+                mocks["xet_get"].assert_not_called()
path: tests/test_xet_download.py
user: login: Wauplin
body: :astonished: 
created_at: 2025-03-03T11:41:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,300 @@
+import os
+from pathlib import Path
+from unittest.mock import DEFAULT, Mock, patch
+
+from huggingface_hub import snapshot_download
+from huggingface_hub.file_download import (
+    HfFileMetadata,
+    get_hf_file_metadata,
+    hf_hub_download,
+    hf_hub_url,
+    try_to_load_from_cache,
+    xet_get,
+)
+from huggingface_hub.utils import XetMetadata
+
+from .testing_utils import (
+    DUMMY_XET_FILE,
+    DUMMY_XET_MODEL_ID,
+    requires,
+    with_production_testing,
+)
+
+
+@requires("hf_xet")
+@with_production_testing
+class TestXetFileDownload:
+    def test_xet_get_called_when_xet_metadata_present(self, tmp_path):
+        """Test that xet_get is called when xet metadata is present."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            xet_metadata = XetMetadata(
+                endpoint="mock_endpoint",
+                access_token="mock_token",
+                expiration_unix_epoch=9999999999,
+                file_hash="mock_hash",
+            )
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=xet_metadata,
+            )
+
+            with patch("huggingface_hub.file_download.xet_get") as mock_xet_get:
+                with patch("huggingface_hub.file_download._create_symlink"):
+                    hf_hub_download(
+                        DUMMY_XET_MODEL_ID,
+                        filename=DUMMY_XET_FILE,
+                        cache_dir=tmp_path,
+                        force_download=True,
+                    )
+
+                    # Verify xet_get was called with correct parameters
+                    mock_xet_get.assert_called_once()
+                    _, kwargs = mock_xet_get.call_args
+                    assert "xet_metadata" in kwargs
+                    assert kwargs["xet_metadata"] == xet_metadata
+
+    def test_backward_compatibility_no_xet_metadata(self, tmp_path):
+        """Test backward compatibility when response has no xet metadata."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=None,  # No xet_metadata
+            )
+
+            with patch("huggingface_hub.file_download.http_get") as mock_http_get:
+                with patch("huggingface_hub.file_download._create_symlink"):
+                    hf_hub_download(
+                        DUMMY_XET_MODEL_ID,
+                        filename=DUMMY_XET_FILE,
+                        cache_dir=tmp_path,
+                        force_download=True,
+                    )
+
+                    # Verify http_get was called
+                    mock_http_get.assert_called_once()
+
+    def test_get_xet_file_metadata_basic(self) -> None:
+        """Test getting metadata from a file on the Hub."""
+        url = hf_hub_url(
+            repo_id=DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+        )
+        metadata = get_hf_file_metadata(url)
+        xet_metadata = metadata.xet_metadata
+        assert xet_metadata is not None
+        assert xet_metadata.endpoint is not None
+        assert xet_metadata.access_token is not None
+        assert isinstance(xet_metadata.expiration_unix_epoch, int)
+        assert xet_metadata.file_hash is not None
+        assert xet_metadata.refresh_route is not None
+
+    def test_basic_download(self, tmp_path):
+        # Make sure that xet_get is called
+        with patch("huggingface_hub.file_download.xet_get", wraps=xet_get) as _xet_get:
+            filepath = hf_hub_download(
+                DUMMY_XET_MODEL_ID,
+                filename=DUMMY_XET_FILE,
+                cache_dir=tmp_path,
+            )
+
+            assert os.path.exists(filepath)
+            assert os.path.getsize(filepath) > 0
+
+            _xet_get.assert_called_once()
+
+    def test_try_to_load_from_cache(self, tmp_path):
+        cached_path = try_to_load_from_cache(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+        assert cached_path is None
+
+        downloaded_path = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+
+        # Now should find it in cache
+        cached_path = try_to_load_from_cache(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+        assert cached_path == downloaded_path
+
+    def test_cache_reuse(self, tmp_path):
+        path1 = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+
+        assert os.path.exists(path1)
+
+        with patch("huggingface_hub.file_download._download_to_tmp_and_move") as mock:
+            # Second download should use cache
+            path2 = hf_hub_download(
+                DUMMY_XET_MODEL_ID,
+                filename=DUMMY_XET_FILE,
+                cache_dir=tmp_path,
+            )
+
+            assert path1 == path2
+            mock.assert_not_called()
+
+    def test_download_to_local_dir(self, tmp_path):
+        local_dir = tmp_path / "local_dir"
+        local_dir.mkdir(exist_ok=True, parents=True)
+
+        cache_dir = tmp_path / "cache"
+        cache_dir.mkdir(exist_ok=True, parents=True)
+        # Download to local dir
+        returned_path = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            local_dir=local_dir,
+            cache_dir=cache_dir,
+        )
+        assert local_dir in Path(returned_path).parents
+
+        for path in cache_dir.glob("**/blobs/**"):
+            assert not path.is_file()
+        for path in cache_dir.glob("**/snapshots/**"):
+            assert not path.is_file()
+
+    def test_force_download(self, tmp_path):
+        # First download
+        path1 = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+
+        # Force download should re-download even if in cache
+        with patch("huggingface_hub.file_download.xet_get") as mock_xet_get:
+            path2 = hf_hub_download(
+                DUMMY_XET_MODEL_ID,
+                filename=DUMMY_XET_FILE,
+                cache_dir=tmp_path,
+                force_download=True,
+            )
+
+            assert path1 == path2
+            mock_xet_get.assert_called_once()
+
+    def test_fallback_to_http_when_xet_not_available(self, tmp_path):
+        """Test that http_get is used when hf_xet is not available."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=XetMetadata(
+                    endpoint="mock_endpoint",
+                    access_token="mock_token",
+                    expiration_unix_epoch=9999999999,
+                    file_hash="mock_hash",
+                ),
+            )
+
+            # Mock is_xet_available to return False
+            with patch.multiple(
+                "huggingface_hub.file_download",
+                is_xet_available=Mock(return_value=False),
+                http_get=DEFAULT,
+                xet_get=DEFAULT,
+                _create_symlink=DEFAULT,
+            ) as mocks:
+                hf_hub_download(
+                    DUMMY_XET_MODEL_ID,
+                    filename=DUMMY_XET_FILE,
+                    cache_dir=tmp_path,
+                    force_download=True,
+                )
+
+                # Verify http_get was called and xet_get was not
+                mocks["http_get"].assert_called_once()
+                mocks["xet_get"].assert_not_called()
+
+    def test_use_xet_when_available(self, tmp_path):
+        """Test that xet_get is used when hf_xet is available."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=XetMetadata(
+                    endpoint="mock_endpoint",
+                    access_token="mock_token",
+                    expiration_unix_epoch=9999999999,
+                    file_hash="mock_hash",
+                ),
+            )
path: tests/test_xet_download.py
user: login: Wauplin
body: I feel this could be an helper

```py
with _patch_xet_file_metadata():
    ...
```

(used 3-4 times in this module, right?)
created_at: 2025-03-03T11:43:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1544,14 +1673,29 @@ def _download_to_tmp_and_move(
             _check_disk_space(expected_size, incomplete_path.parent)
             _check_disk_space(expected_size, destination_path.parent)
 
-        http_get(
-            url_to_download,
-            f,
-            proxies=proxies,
-            resume_size=resume_size,
-            headers=headers,
-            expected_size=expected_size,
-        )
+        if xet_metadata is not None and xet_metadata.file_hash is not None:
+            logger.info("Xet Storage is enabled for this repo. Downloading file from Xet Storage..")
+            xet_get(
+                incomplete_path,
+                xet_metadata=xet_metadata,
+                headers=headers,
+                expected_size=expected_size,
+                displayed_filename=filename,
+            )
+
+            # TODO: xetpoc - the http_get path is building this out, so we're replicating that logic here
path: src/huggingface_hub/file_download.py
user: login: Wauplin
body: should be enough yes :+1: 
created_at: 2025-03-03T11:45:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1544,14 +1652,39 @@ def _download_to_tmp_and_move(
             _check_disk_space(expected_size, incomplete_path.parent)
             _check_disk_space(expected_size, destination_path.parent)
 
-        http_get(
-            url_to_download,
-            f,
-            proxies=proxies,
-            resume_size=resume_size,
-            headers=headers,
-            expected_size=expected_size,
-        )
+        if xet_metadata is not None and xet_metadata.file_hash is not None:
+            if is_xet_available():
+                logger.info("Xet Storage is enabled for this repo. Downloading file from Xet Storage..")
+                xet_get(
+                    incomplete_path=incomplete_path,
+                    xet_metadata=xet_metadata,
+                    headers=headers,
+                    expected_size=expected_size,
+                    displayed_filename=filename,
+                )
+            else:
+                logger.warning(
+                    "Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. "
+                    "Falling back to regular HTTP download. "
+                    "For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`"
+                )
+                http_get(
+                    url_to_download,
+                    f,
+                    proxies=proxies,
+                    resume_size=resume_size,
+                    headers=headers,
+                    expected_size=expected_size,
+                )
+        else:
+            http_get(
+                url_to_download,
+                f,
+                proxies=proxies,
+                resume_size=resume_size,
+                headers=headers,
+                expected_size=expected_size,
+            )
path: src/huggingface_hub/file_download.py
user: login: hanouticelina
body: much better! thanks
created_at: 2025-03-03T13:42:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,300 @@
+import os
+from pathlib import Path
+from unittest.mock import DEFAULT, Mock, patch
+
+from huggingface_hub import snapshot_download
+from huggingface_hub.file_download import (
+    HfFileMetadata,
+    get_hf_file_metadata,
+    hf_hub_download,
+    hf_hub_url,
+    try_to_load_from_cache,
+    xet_get,
+)
+from huggingface_hub.utils import XetMetadata
+
+from .testing_utils import (
+    DUMMY_XET_FILE,
+    DUMMY_XET_MODEL_ID,
+    requires,
+    with_production_testing,
+)
+
+
+@requires("hf_xet")
+@with_production_testing
+class TestXetFileDownload:
+    def test_xet_get_called_when_xet_metadata_present(self, tmp_path):
+        """Test that xet_get is called when xet metadata is present."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            xet_metadata = XetMetadata(
+                endpoint="mock_endpoint",
+                access_token="mock_token",
+                expiration_unix_epoch=9999999999,
+                file_hash="mock_hash",
+            )
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=xet_metadata,
+            )
+
+            with patch("huggingface_hub.file_download.xet_get") as mock_xet_get:
+                with patch("huggingface_hub.file_download._create_symlink"):
+                    hf_hub_download(
+                        DUMMY_XET_MODEL_ID,
+                        filename=DUMMY_XET_FILE,
+                        cache_dir=tmp_path,
+                        force_download=True,
+                    )
+
+                    # Verify xet_get was called with correct parameters
+                    mock_xet_get.assert_called_once()
+                    _, kwargs = mock_xet_get.call_args
+                    assert "xet_metadata" in kwargs
+                    assert kwargs["xet_metadata"] == xet_metadata
+
+    def test_backward_compatibility_no_xet_metadata(self, tmp_path):
+        """Test backward compatibility when response has no xet metadata."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=None,  # No xet_metadata
+            )
+
+            with patch("huggingface_hub.file_download.http_get") as mock_http_get:
+                with patch("huggingface_hub.file_download._create_symlink"):
+                    hf_hub_download(
+                        DUMMY_XET_MODEL_ID,
+                        filename=DUMMY_XET_FILE,
+                        cache_dir=tmp_path,
+                        force_download=True,
+                    )
+
+                    # Verify http_get was called
+                    mock_http_get.assert_called_once()
+
+    def test_get_xet_file_metadata_basic(self) -> None:
+        """Test getting metadata from a file on the Hub."""
+        url = hf_hub_url(
+            repo_id=DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+        )
+        metadata = get_hf_file_metadata(url)
+        xet_metadata = metadata.xet_metadata
+        assert xet_metadata is not None
+        assert xet_metadata.endpoint is not None
+        assert xet_metadata.access_token is not None
+        assert isinstance(xet_metadata.expiration_unix_epoch, int)
+        assert xet_metadata.file_hash is not None
+        assert xet_metadata.refresh_route is not None
+
+    def test_basic_download(self, tmp_path):
+        # Make sure that xet_get is called
+        with patch("huggingface_hub.file_download.xet_get", wraps=xet_get) as _xet_get:
+            filepath = hf_hub_download(
+                DUMMY_XET_MODEL_ID,
+                filename=DUMMY_XET_FILE,
+                cache_dir=tmp_path,
+            )
+
+            assert os.path.exists(filepath)
+            assert os.path.getsize(filepath) > 0
+
+            _xet_get.assert_called_once()
+
+    def test_try_to_load_from_cache(self, tmp_path):
+        cached_path = try_to_load_from_cache(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+        assert cached_path is None
+
+        downloaded_path = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+
+        # Now should find it in cache
+        cached_path = try_to_load_from_cache(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+        assert cached_path == downloaded_path
+
+    def test_cache_reuse(self, tmp_path):
+        path1 = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+
+        assert os.path.exists(path1)
+
+        with patch("huggingface_hub.file_download._download_to_tmp_and_move") as mock:
+            # Second download should use cache
+            path2 = hf_hub_download(
+                DUMMY_XET_MODEL_ID,
+                filename=DUMMY_XET_FILE,
+                cache_dir=tmp_path,
+            )
+
+            assert path1 == path2
+            mock.assert_not_called()
+
+    def test_download_to_local_dir(self, tmp_path):
+        local_dir = tmp_path / "local_dir"
+        local_dir.mkdir(exist_ok=True, parents=True)
+
+        cache_dir = tmp_path / "cache"
+        cache_dir.mkdir(exist_ok=True, parents=True)
+        # Download to local dir
+        returned_path = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            local_dir=local_dir,
+            cache_dir=cache_dir,
+        )
+        assert local_dir in Path(returned_path).parents
+
+        for path in cache_dir.glob("**/blobs/**"):
+            assert not path.is_file()
+        for path in cache_dir.glob("**/snapshots/**"):
+            assert not path.is_file()
+
+    def test_force_download(self, tmp_path):
+        # First download
+        path1 = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+
+        # Force download should re-download even if in cache
+        with patch("huggingface_hub.file_download.xet_get") as mock_xet_get:
+            path2 = hf_hub_download(
+                DUMMY_XET_MODEL_ID,
+                filename=DUMMY_XET_FILE,
+                cache_dir=tmp_path,
+                force_download=True,
+            )
+
+            assert path1 == path2
+            mock_xet_get.assert_called_once()
+
+    def test_fallback_to_http_when_xet_not_available(self, tmp_path):
+        """Test that http_get is used when hf_xet is not available."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=XetMetadata(
+                    endpoint="mock_endpoint",
+                    access_token="mock_token",
+                    expiration_unix_epoch=9999999999,
+                    file_hash="mock_hash",
+                ),
+            )
+
+            # Mock is_xet_available to return False
+            with patch.multiple(
+                "huggingface_hub.file_download",
+                is_xet_available=Mock(return_value=False),
+                http_get=DEFAULT,
+                xet_get=DEFAULT,
+                _create_symlink=DEFAULT,
+            ) as mocks:
+                hf_hub_download(
+                    DUMMY_XET_MODEL_ID,
+                    filename=DUMMY_XET_FILE,
+                    cache_dir=tmp_path,
+                    force_download=True,
+                )
+
+                # Verify http_get was called and xet_get was not
+                mocks["http_get"].assert_called_once()
+                mocks["xet_get"].assert_not_called()
+
+    def test_use_xet_when_available(self, tmp_path):
+        """Test that xet_get is used when hf_xet is available."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=XetMetadata(
+                    endpoint="mock_endpoint",
+                    access_token="mock_token",
+                    expiration_unix_epoch=9999999999,
+                    file_hash="mock_hash",
+                ),
+            )
path: tests/test_xet_download.py
user: login: hanouticelina
body: yep!
created_at: 2025-03-03T13:43:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,300 @@
+import os
+from pathlib import Path
+from unittest.mock import DEFAULT, Mock, patch
+
+from huggingface_hub import snapshot_download
+from huggingface_hub.file_download import (
+    HfFileMetadata,
+    get_hf_file_metadata,
+    hf_hub_download,
+    hf_hub_url,
+    try_to_load_from_cache,
+    xet_get,
+)
+from huggingface_hub.utils import XetMetadata
+
+from .testing_utils import (
+    DUMMY_XET_FILE,
+    DUMMY_XET_MODEL_ID,
+    requires,
+    with_production_testing,
+)
+
+
+@requires("hf_xet")
+@with_production_testing
+class TestXetFileDownload:
+    def test_xet_get_called_when_xet_metadata_present(self, tmp_path):
+        """Test that xet_get is called when xet metadata is present."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            xet_metadata = XetMetadata(
+                endpoint="mock_endpoint",
+                access_token="mock_token",
+                expiration_unix_epoch=9999999999,
+                file_hash="mock_hash",
+            )
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=xet_metadata,
+            )
+
+            with patch("huggingface_hub.file_download.xet_get") as mock_xet_get:
+                with patch("huggingface_hub.file_download._create_symlink"):
+                    hf_hub_download(
+                        DUMMY_XET_MODEL_ID,
+                        filename=DUMMY_XET_FILE,
+                        cache_dir=tmp_path,
+                        force_download=True,
+                    )
+
+                    # Verify xet_get was called with correct parameters
+                    mock_xet_get.assert_called_once()
+                    _, kwargs = mock_xet_get.call_args
+                    assert "xet_metadata" in kwargs
+                    assert kwargs["xet_metadata"] == xet_metadata
+
+    def test_backward_compatibility_no_xet_metadata(self, tmp_path):
+        """Test backward compatibility when response has no xet metadata."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=None,  # No xet_metadata
+            )
+
+            with patch("huggingface_hub.file_download.http_get") as mock_http_get:
+                with patch("huggingface_hub.file_download._create_symlink"):
+                    hf_hub_download(
+                        DUMMY_XET_MODEL_ID,
+                        filename=DUMMY_XET_FILE,
+                        cache_dir=tmp_path,
+                        force_download=True,
+                    )
+
+                    # Verify http_get was called
+                    mock_http_get.assert_called_once()
+
+    def test_get_xet_file_metadata_basic(self) -> None:
+        """Test getting metadata from a file on the Hub."""
+        url = hf_hub_url(
+            repo_id=DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+        )
+        metadata = get_hf_file_metadata(url)
+        xet_metadata = metadata.xet_metadata
+        assert xet_metadata is not None
+        assert xet_metadata.endpoint is not None
+        assert xet_metadata.access_token is not None
+        assert isinstance(xet_metadata.expiration_unix_epoch, int)
+        assert xet_metadata.file_hash is not None
+        assert xet_metadata.refresh_route is not None
+
+    def test_basic_download(self, tmp_path):
+        # Make sure that xet_get is called
+        with patch("huggingface_hub.file_download.xet_get", wraps=xet_get) as _xet_get:
+            filepath = hf_hub_download(
+                DUMMY_XET_MODEL_ID,
+                filename=DUMMY_XET_FILE,
+                cache_dir=tmp_path,
+            )
+
+            assert os.path.exists(filepath)
+            assert os.path.getsize(filepath) > 0
+
+            _xet_get.assert_called_once()
+
+    def test_try_to_load_from_cache(self, tmp_path):
+        cached_path = try_to_load_from_cache(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+        assert cached_path is None
+
+        downloaded_path = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+
+        # Now should find it in cache
+        cached_path = try_to_load_from_cache(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+        assert cached_path == downloaded_path
+
+    def test_cache_reuse(self, tmp_path):
+        path1 = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+
+        assert os.path.exists(path1)
+
+        with patch("huggingface_hub.file_download._download_to_tmp_and_move") as mock:
+            # Second download should use cache
+            path2 = hf_hub_download(
+                DUMMY_XET_MODEL_ID,
+                filename=DUMMY_XET_FILE,
+                cache_dir=tmp_path,
+            )
+
+            assert path1 == path2
+            mock.assert_not_called()
+
+    def test_download_to_local_dir(self, tmp_path):
+        local_dir = tmp_path / "local_dir"
+        local_dir.mkdir(exist_ok=True, parents=True)
+
+        cache_dir = tmp_path / "cache"
+        cache_dir.mkdir(exist_ok=True, parents=True)
+        # Download to local dir
+        returned_path = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            local_dir=local_dir,
+            cache_dir=cache_dir,
+        )
+        assert local_dir in Path(returned_path).parents
+
+        for path in cache_dir.glob("**/blobs/**"):
+            assert not path.is_file()
+        for path in cache_dir.glob("**/snapshots/**"):
+            assert not path.is_file()
+
+    def test_force_download(self, tmp_path):
+        # First download
+        path1 = hf_hub_download(
+            DUMMY_XET_MODEL_ID,
+            filename=DUMMY_XET_FILE,
+            cache_dir=tmp_path,
+        )
+
+        # Force download should re-download even if in cache
+        with patch("huggingface_hub.file_download.xet_get") as mock_xet_get:
+            path2 = hf_hub_download(
+                DUMMY_XET_MODEL_ID,
+                filename=DUMMY_XET_FILE,
+                cache_dir=tmp_path,
+                force_download=True,
+            )
+
+            assert path1 == path2
+            mock_xet_get.assert_called_once()
+
+    def test_fallback_to_http_when_xet_not_available(self, tmp_path):
+        """Test that http_get is used when hf_xet is not available."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=XetMetadata(
+                    endpoint="mock_endpoint",
+                    access_token="mock_token",
+                    expiration_unix_epoch=9999999999,
+                    file_hash="mock_hash",
+                ),
+            )
+
+            # Mock is_xet_available to return False
+            with patch.multiple(
+                "huggingface_hub.file_download",
+                is_xet_available=Mock(return_value=False),
+                http_get=DEFAULT,
+                xet_get=DEFAULT,
+                _create_symlink=DEFAULT,
+            ) as mocks:
+                hf_hub_download(
+                    DUMMY_XET_MODEL_ID,
+                    filename=DUMMY_XET_FILE,
+                    cache_dir=tmp_path,
+                    force_download=True,
+                )
+
+                # Verify http_get was called and xet_get was not
+                mocks["http_get"].assert_called_once()
+                mocks["xet_get"].assert_not_called()
+
+    def test_use_xet_when_available(self, tmp_path):
+        """Test that xet_get is used when hf_xet is available."""
+        with patch("huggingface_hub.file_download.get_hf_file_metadata") as mock_metadata:
+            mock_metadata.return_value = HfFileMetadata(
+                commit_hash="mock_commit",
+                etag="mock_etag",
+                location="mock_location",
+                size=1024,
+                xet_metadata=XetMetadata(
+                    endpoint="mock_endpoint",
+                    access_token="mock_token",
+                    expiration_unix_epoch=9999999999,
+                    file_hash="mock_hash",
+                ),
+            )
path: tests/test_xet_download.py
user: login: hanouticelina
body: addressed in [9cdf99d](https://github.com/huggingface/huggingface_hub/pull/2875/commits/9cdf99decf7da657d93e78d064f0a66723ec76bc)
created_at: 2025-03-03T13:56:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1544,14 +1652,39 @@ def _download_to_tmp_and_move(
             _check_disk_space(expected_size, incomplete_path.parent)
             _check_disk_space(expected_size, destination_path.parent)
 
-        http_get(
-            url_to_download,
-            f,
-            proxies=proxies,
-            resume_size=resume_size,
-            headers=headers,
-            expected_size=expected_size,
-        )
+        if xet_metadata is not None and xet_metadata.file_hash is not None:
+            if is_xet_available():
+                logger.info("Xet Storage is enabled for this repo. Downloading file from Xet Storage..")
+                xet_get(
+                    incomplete_path=incomplete_path,
+                    xet_metadata=xet_metadata,
+                    headers=headers,
+                    expected_size=expected_size,
+                    displayed_filename=filename,
+                )
+            else:
+                logger.warning(
+                    "Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. "
+                    "Falling back to regular HTTP download. "
+                    "For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`"
+                )
+                http_get(
+                    url_to_download,
+                    f,
+                    proxies=proxies,
+                    resume_size=resume_size,
+                    headers=headers,
+                    expected_size=expected_size,
+                )
+        else:
+            http_get(
+                url_to_download,
+                f,
+                proxies=proxies,
+                resume_size=resume_size,
+                headers=headers,
+                expected_size=expected_size,
+            )
path: src/huggingface_hub/file_download.py
user: login: hanouticelina
body: Added in [9cdf99d](https://github.com/huggingface/huggingface_hub/pull/2875/commits/9cdf99decf7da657d93e78d064f0a66723ec76bc)
created_at: 2025-03-03T13:56:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,279 @@
+import os
+from contextlib import contextmanager
+from pathlib import Path
+from unittest.mock import DEFAULT, Mock, patch
+
+from huggingface_hub import snapshot_download
+from huggingface_hub.file_download import (
+    HfFileMetadata,
+    get_hf_file_metadata,
+    hf_hub_download,
+    hf_hub_url,
+    try_to_load_from_cache,
+    xet_get,
+)
+from huggingface_hub.utils import XetMetadata
+
+from .testing_utils import (
+    DUMMY_XET_FILE,
+    DUMMY_XET_MODEL_ID,
+    requires,
+    with_production_testing,
+)
+
+
+@requires("hf_xet")
+@with_production_testing
+class TestXetFileDownload:
+    @contextmanager
+    def _patch_xet_file_metadata(self, with_xet_metadata=True):
path: tests/test_xet_download.py
user: login: Wauplin
body: ```suggestion
    def _patch_xet_file_metadata(self, with_xet_metadata: bool):
```

(making it required for readability in the tests + it's already used like this)
created_at: 2025-03-03T14:40:20Z
author_association: CONTRIBUTOR

PR comments:
user: login: HuggingFaceDocBuilderDev
created_at: 2025-02-18T17:39:45Z
body: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/huggingface_hub/pr_2875). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
author_association: NONE
user: login: hanouticelina
created_at: 2025-03-03T14:48:13Z
body: thanks @Wauplin for the review ❤️  let's wait for a final one from @bpronan or/and @rajatarya.  
agree about having a separate PR for the documentation! 
author_association: CONTRIBUTOR
user: login: rajatarya
created_at: 2025-03-03T17:36:02Z
body: > thanks @Wauplin for the review ❤️ let's wait for a final one from @bpronan or/and @rajatarya. agree about having a separate PR for the documentation!

I have docs written and ready in Notion, am in the process of porting them over to repo right now. Will have a PR up for this today. I worked with @jsulz on these docs.
author_association: CONTRIBUTOR
user: login: Wauplin
created_at: 2025-03-04T15:44:59Z
body: :tada: 
author_association: CONTRIBUTOR