Repository: huggingface/huggingface_hub
Pull request #1038 (2022-09-07 12:47:10)

Handle backoff on HTTP 503 error when pushing repeatedly
Fix https://github.com/huggingface/huggingface_hub/issues/1027.

This PR adds a separate helper to perform HTTP requests with a backoff strategy. Retries can be made on specific exceptions or specific status codes. By default, we mainly target Connection timeout and HTTP 503 Service Unavailable. It is a slight variation of the existing backoff implementation in `_request_wrapper` but separated so that we can unit tests and document it properly.

This PR adds backoff usage by default when pushing a LFS file to the Hub, which is the initial purpose of https://github.com/huggingface/huggingface_hub/issues/1027. At the moment sleep times will increase as such: 1s, 2s, 4s, 8s, 8s, 8s... 

I did not add the helper in the documentation since I'm not exactly sure yet if that's something we want to do at the moment. I think other HTTP-related helpers will come as user-agent handler (https://github.com/huggingface/huggingface_hub/issues/1018) so maybe we can update documentation at that time. Docstring for `http_backoff` is already ready.

(for the record the excellent [backoff package](https://github.com/litl/backoff) exists but is not used here to avoid an extra external dependency)

HuggingFaceDocBuilderDev: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/huggingface_hub/pr_1038). All of your documentation changes will be reflected on that endpoint.
codecov[bot]: # [Codecov](https://codecov.io/gh/huggingface/huggingface_hub/pull/1038?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface) Report
> Merging [#1038](https://codecov.io/gh/huggingface/huggingface_hub/pull/1038?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface) (7f3cfe8) into [main](https://codecov.io/gh/huggingface/huggingface_hub/commit/8e5f219ba3fb7dda2249ab018f652ae04457d07f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface) (8e5f219) will **increase** coverage by `0.18%`.
> The diff coverage is `97.61%`.

```diff
@@            Coverage Diff             @@
##             main    #1038      +/-   ##
==========================================
+ Coverage   83.01%   83.20%   +0.18%     
==========================================
  Files          36       37       +1     
  Lines        3822     3847      +25     
==========================================
+ Hits         3173     3201      +28     
+ Misses        649      646       -3     
```


| [Impacted Files](https://codecov.io/gh/huggingface/huggingface_hub/pull/1038?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface) | Coverage Δ | |
|---|---|---|
| [src/huggingface\_hub/lfs.py](https://codecov.io/gh/huggingface/huggingface_hub/pull/1038/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface#diff-c3JjL2h1Z2dpbmdmYWNlX2h1Yi9sZnMucHk=) | `74.84% <66.66%> (ø)` | |
| [src/huggingface\_hub/file\_download.py](https://codecov.io/gh/huggingface/huggingface_hub/pull/1038/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface#diff-c3JjL2h1Z2dpbmdmYWNlX2h1Yi9maWxlX2Rvd25sb2FkLnB5) | `85.79% <100.00%> (+0.34%)` | :arrow_up: |
| [src/huggingface\_hub/utils/\_\_init\_\_.py](https://codecov.io/gh/huggingface/huggingface_hub/pull/1038/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface#diff-c3JjL2h1Z2dpbmdmYWNlX2h1Yi91dGlscy9fX2luaXRfXy5weQ==) | `100.00% <100.00%> (ø)` | |
| [src/huggingface\_hub/utils/\_http.py](https://codecov.io/gh/huggingface/huggingface_hub/pull/1038/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface#diff-c3JjL2h1Z2dpbmdmYWNlX2h1Yi91dGlscy9faHR0cC5weQ==) | `100.00% <100.00%> (ø)` | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface)

Wauplin: Thanks @SBrandeis and @merveenoyan for the review ! I'm merging once CI is happy :)

Patch:
diff --git a/docs/source/how-to-cache.mdx b/docs/source/how-to-cache.mdx
index 1c2ca83e..a0651765 100644
--- a/docs/source/how-to-cache.mdx
+++ b/docs/source/how-to-cache.mdx
@@ -238,3 +238,56 @@ HFCacheInfo(
     ],
 )
 ```
+
+## Clean your cache
+
+Scanning your cache is interesting but what you really want to do next is usually to
+delete some portions to free up some space on your drive. This is possible using the
+[`~HFCacheInfo.delete_revisions`] helper from [`HFCacheInfo`] object returned when
+scanning the cache.
+
+Pass a list of revisions to delete and the tool will define a strategy to free up the
+space. It returns a [`DeleteCacheStrategy`] object that describes which files and
+folders will be deleted and the expected freed space. Once you agree with the deletion,
+you must execute it to make the deletion effective. In order to avoid discrepancies, you
+cannot edit a strategy manually.
+
+The strategy to delete revisions is the following:
+
+- the `snapshot` folder containing the revision symlinks is deleted.
+- blobs files that are targeted only by revisions to be deleted are deleted as well.
+- if a revision is linked to 1 or more `refs`, references are deleted.
+- if all revisions from a repo are deleted, the entire cached repository is deleted.
+
+Here is a simple usage example. See reference for details.
+
+```py
+>>> from huggingface_hub import scan_cache_dir
+
+>>> delete_strategy = scan_cache_dir().delete_revisions(
+...     "81fd1d6e7847c99f5862c9fb81387956d99ec7aa"
+...     "e2983b237dccf3ab4937c97fa717319a9ca1a96d",
+...     "6c0e6080953db56375760c0471a8c5f2929baf11",
+... )
+>>> print("Will free " + delete_strategy.expected_freed_size_str)
+Will free 8.6G
+
+>>> delete_strategy.execute()
+Cache deletion done. Saved 8.6G.
+```
+
+<Tip>
+
+Revision hashes are unique across all repositories. This means you don't need to
+provide any `repo_id` or `repo_type` when removing revisions.
+
+</Tip>
+
+<Tip warning={true}>
+
+If a revision is not found in the cache, it will be silently ignored. Besides, if a file
+or folder cannot be found while trying to delete it, a warning will be logged but no
+error is thrown. The deletion continues for other paths contained in the
+[`DeleteCacheStrategy`] object.
+
+</Tip>
diff --git a/docs/source/package_reference/cache.mdx b/docs/source/package_reference/cache.mdx
index 7f55bbcf..a2da98b8 100644
--- a/docs/source/package_reference/cache.mdx
+++ b/docs/source/package_reference/cache.mdx
@@ -35,6 +35,11 @@ All structures are built and returned by [`scan_cache_dir`] and are immutable.
 [[autodoc]] huggingface_hub.CachedFileInfo
     - size_on_disk_str
 
+### DeleteCacheStrategy
+
+[[autodoc]] huggingface_hub.DeleteCacheStrategy
+    - expected_freed_size_str
+
 ## Exceptions
 
 ### CorruptedCacheException
diff --git a/src/huggingface_hub/__init__.py b/src/huggingface_hub/__init__.py
index 55b9ff72..b6a641b9 100644
--- a/src/huggingface_hub/__init__.py
+++ b/src/huggingface_hub/__init__.py
@@ -205,6 +205,7 @@ __getattr__, __dir__, __all__ = _attach(
             "CachedRepoInfo",
             "CachedRevisionInfo",
             "CorruptedCacheException",
+            "DeleteCacheStrategy",
             "HFCacheInfo",
             "scan_cache_dir",
         ],
diff --git a/src/huggingface_hub/file_download.py b/src/huggingface_hub/file_download.py
index f3fd28ea..13ae7d23 100644
--- a/src/huggingface_hub/file_download.py
+++ b/src/huggingface_hub/file_download.py
@@ -6,7 +6,6 @@ import os
 import re
 import sys
 import tempfile
-import time
 import warnings
 from contextlib import contextmanager
 from functools import partial
@@ -20,6 +19,7 @@ import packaging.version
 import requests
 from filelock import FileLock
 from huggingface_hub import constants
+from requests.exceptions import ConnectTimeout, ProxyError
 
 from . import __version__
 from .constants import (
@@ -37,6 +37,7 @@ from .utils import (
     EntryNotFoundError,
     LocalEntryNotFoundError,
     hf_raise_for_status,
+    http_backoff,
     logging,
     tqdm,
     validate_hf_hub_args,
@@ -478,30 +479,17 @@ def _request_wrapper(
         return response
 
     # 3. Exponential backoff
-    tries, success = 0, False
-    while not success:
-        tries += 1
-        try:
-            response = requests.request(
-                method=method.upper(), url=url, timeout=timeout, **params
-            )
-            success = True
-        except (
-            requests.exceptions.ConnectTimeout,
-            requests.exceptions.ProxyError,
-        ) as err:
-            if tries > max_retries:
-                raise err
-            else:
-                logger.info(
-                    f"{method} request to {url} timed out, retrying..."
-                    f" [{tries/max_retries}]"
-                )
-                sleep_time = min(
-                    max_wait_time, base_wait_time * 2 ** (tries - 1)
-                )  # Exponential backoff
-                time.sleep(sleep_time)
-    return response
+    return http_backoff(
+        method=method,
+        url=url,
+        max_retries=max_retries,
+        base_wait_time=base_wait_time,
+        max_wait_time=max_wait_time,
+        retry_on_exceptions=(ConnectTimeout, ProxyError),
+        retry_on_status_codes=(),
+        timeout=timeout,
+        **params,
+    )
 
 
 def _request_with_retry(*args, **kwargs) -> requests.Response:
diff --git a/src/huggingface_hub/lfs.py b/src/huggingface_hub/lfs.py
index 9b1cf8af..c1558308 100644
--- a/src/huggingface_hub/lfs.py
+++ b/src/huggingface_hub/lfs.py
@@ -28,7 +28,7 @@ import requests
 from huggingface_hub.constants import ENDPOINT, REPO_TYPES_URL_PREFIXES
 from requests.auth import HTTPBasicAuth
 
-from .utils import hf_raise_for_status, validate_hf_hub_args
+from .utils import hf_raise_for_status, http_backoff, validate_hf_hub_args
 from .utils.sha import sha256, sha_fileobj
 
 
@@ -308,7 +308,7 @@ def _upload_single_part(upload_url: str, fileobj: BinaryIO):
 
     Raises: `requests.HTTPError` if the upload resulted in an error
     """
-    upload_res = requests.put(upload_url, data=fileobj)
+    upload_res = http_backoff("PUT", upload_url, data=fileobj)
     hf_raise_for_status(upload_res)
     return upload_res
 
@@ -376,7 +376,7 @@ def _upload_multi_part(
             seek_from=chunk_size * part_idx,
             read_limit=chunk_size,
         ) as fileobj_slice:
-            part_upload_res = requests.put(part_upload_url, data=fileobj_slice)
+            part_upload_res = http_backoff("PUT", part_upload_url, data=fileobj_slice)
             hf_raise_for_status(part_upload_res)
             etag = part_upload_res.headers.get("etag")
             if etag is None or etag == "":
diff --git a/src/huggingface_hub/utils/__init__.py b/src/huggingface_hub/utils/__init__.py
index 08353836..cf517637 100644
--- a/src/huggingface_hub/utils/__init__.py
+++ b/src/huggingface_hub/utils/__init__.py
@@ -21,6 +21,7 @@ from ._cache_manager import (
     CachedRepoInfo,
     CachedRevisionInfo,
     CorruptedCacheException,
+    DeleteCacheStrategy,
     HFCacheInfo,
     scan_cache_dir,
 )
@@ -34,6 +35,7 @@ from ._errors import (
     RevisionNotFoundError,
     hf_raise_for_status,
 )
+from ._http import http_backoff
 from ._paths import filter_repo_objects
 from ._subprocess import run_subprocess
 from ._validators import HFValidationError, validate_hf_hub_args, validate_repo_id
diff --git a/src/huggingface_hub/utils/_cache_manager.py b/src/huggingface_hub/utils/_cache_manager.py
index aef5eec5..58f15b33 100644
--- a/src/huggingface_hub/utils/_cache_manager.py
+++ b/src/huggingface_hub/utils/_cache_manager.py
@@ -13,14 +13,20 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 """Contains utilities to manage the HF cache directory."""
+import os
+import shutil
+from collections import defaultdict
 from dataclasses import dataclass
 from pathlib import Path
 from typing import Dict, FrozenSet, List, Optional, Set, Union
 
 from ..constants import HUGGINGFACE_HUB_CACHE
+from . import logging
 from ._typing import Literal
 
 
+logger = logging.get_logger(__name__)
+
 REPO_TYPE_T = Literal["model", "dataset", "space"]
 
 
@@ -166,6 +172,82 @@ class CachedRepoInfo:
         return {ref: revision for revision in self.revisions for ref in revision.refs}
 
 
+@dataclass(frozen=True)
+class DeleteCacheStrategy:
+    """Frozen data structure holding the strategy to delete cached revisions.
+
+    This object is not meant to be instantiated programmatically but to be returned by
+    [`~utils.HFCacheInfo.delete_revisions`]. See documentation for usage example.
+
+    Args:
+        expected_freed_size (`float`):
+            Expected freed size once strategy is executed.
+        blobs (`FrozenSet[Path]`):
+            Set of blob file paths to be deleted.
+        refs (`FrozenSet[Path]`):
+            Set of reference file paths to be deleted.
+        repos (`FrozenSet[Path]`):
+            Set of entire repo paths to be deleted.
+        snapshots (`FrozenSet[Path]`):
+            Set of snapshots to be deleted (directory of symlinks).
+    """
+
+    expected_freed_size: int
+    blobs: FrozenSet[Path]
+    refs: FrozenSet[Path]
+    repos: FrozenSet[Path]
+    snapshots: FrozenSet[Path]
+
+    @property
+    def expected_freed_size_str(self) -> str:
+        """
+        (property) Expected size that will be freed as a human-readable string.
+
+        Example: "42.2K".
+        """
+        return _format_size(self.expected_freed_size)
+
+    def execute(self) -> None:
+        """Execute the defined strategy.
+
+        <Tip warning={true}>
+
+        If this method is interrupted, the cache might get corrupted. Deletion order is
+        implemented so that references and symlinks are deleted before the actual blob
+        files.
+
+        </Tip>
+
+        <Tip warning={true}>
+
+        This method is irreversible. If executed, cached files are erased and must be
+        downloaded again.
+
+        </Tip>
+        """
+        # Deletion order matters. Blobs are deleted in last so that the user can't end
+        # up in a state where a `ref`` refers to a missing snapshot or a snapshot
+        # symlink refers to a deleted blob.
+
+        # Delete entire repos
+        for path in self.repos:
+            _try_delete_path(path, path_type="repo")
+
+        # Delete snapshot directories
+        for path in self.snapshots:
+            _try_delete_path(path, path_type="snapshot")
+
+        # Delete refs files
+        for path in self.refs:
+            _try_delete_path(path, path_type="ref")
+
+        # Delete blob files
+        for path in self.blobs:
+            _try_delete_path(path, path_type="blob")
+
+        logger.info(f"Cache deletion done. Saved {self.expected_freed_size_str}.")
+
+
 @dataclass(frozen=True)
 class HFCacheInfo:
     """Frozen data structure holding information about the entire cache-system.
@@ -205,6 +287,114 @@ class HFCacheInfo:
         """
         return _format_size(self.size_on_disk)
 
+    def delete_revisions(self, *revisions: str) -> DeleteCacheStrategy:
+        """Prepare the strategy to delete one or more revisions cached locally.
+
+        Input revisions can be any revision hash. If a revision hash is not found in the
+        local cache, a warning is thrown but no error is raised. Revisions can be from
+        different cached repos since hashes are unique across repos,
+
+        Examples:
+        ```py
+        >>> from huggingface_hub import scan_cache_dir
+        >>> cache_info = scan_cache_dir()
+        >>> delete_strategy = cache_info.delete_revisions(
+        ...     "81fd1d6e7847c99f5862c9fb81387956d99ec7aa"
+        ... )
+        >>> print(f"Will free {delete_strategy.expected_freed_size_str}.")
+        Will free 7.9K.
+        >>> delete_strategy.execute()
+        Cache deletion done. Saved 7.9K.
+        ```
+
+        ```py
+        >>> from huggingface_hub import scan_cache_dir
+        >>> scan_cache_dir().delete_revisions(
+        ...     "81fd1d6e7847c99f5862c9fb81387956d99ec7aa",
+        ...     "e2983b237dccf3ab4937c97fa717319a9ca1a96d",
+        ...     "6c0e6080953db56375760c0471a8c5f2929baf11",
+        ... ).execute()
+        Cache deletion done. Saved 8.6G.
+        ```
+
+        <Tip warning={true}>
+
+        `delete_revisions` returns a [`~utils.DeleteCacheStrategy`] object that needs to
+        be executed. The [`~utils.DeleteCacheStrategy`] is not meant to be modified but
+        allows having a dry run before actually executing the deletion.
+
+        </Tip>
+        """
+        hashes_to_delete: Set[str] = set(revisions)
+
+        repos_with_revisions: Dict[
+            CachedRepoInfo, Set[CachedRevisionInfo]
+        ] = defaultdict(set)
+
+        for repo in self.repos:
+            for revision in repo.revisions:
+                if revision.commit_hash in hashes_to_delete:
+                    repos_with_revisions[repo].add(revision)
+                    hashes_to_delete.remove(revision.commit_hash)
+
+        if len(hashes_to_delete) > 0:
+            logger.warning(
+                "Revision(s) not found - cannot delete them:"
+                f" {', '.join(hashes_to_delete)}"
+            )
+
+        delete_strategy_blobs: Set[Path] = set()
+        delete_strategy_refs: Set[Path] = set()
+        delete_strategy_repos: Set[Path] = set()
+        delete_strategy_snapshots: Set[Path] = set()
+        delete_strategy_expected_freed_size = 0
+
+        for affected_repo, revisions_to_delete in repos_with_revisions.items():
+            other_revisions = affected_repo.revisions - revisions_to_delete
+
+            # If no other revisions, it means all revisions are deleted
+            # -> delete the entire cached repo
+            if len(other_revisions) == 0:
+                delete_strategy_repos.add(affected_repo.repo_path)
+                delete_strategy_expected_freed_size += affected_repo.size_on_disk
+                continue
+
+            # Some revisions of the repo will be deleted but not all. We need to filter
+            # which blob files will not be linked anymore.
+            for revision_to_delete in revisions_to_delete:
+                # Snapshot dir
+                delete_strategy_snapshots.add(revision_to_delete.snapshot_path)
+
+                # Refs dir
+                for ref in revision_to_delete.refs:
+                    delete_strategy_refs.add(affected_repo.repo_path / "refs" / ref)
+
+                # Blobs dir
+                for file in revision_to_delete.files:
+                    if file.blob_path not in delete_strategy_blobs:
+                        is_file_alone = True
+                        for revision in other_revisions:
+                            for rev_file in revision.files:
+                                if file.blob_path == rev_file.blob_path:
+                                    is_file_alone = False
+                                    break
+                            if not is_file_alone:
+                                break
+
+                        # Blob file not referenced by remaining revisions -> delete
+                        if is_file_alone:
+                            delete_strategy_blobs.add(file.blob_path)
+                            delete_strategy_expected_freed_size += file.size_on_disk
+
+        # Return the strategy instead of executing it.
+        return DeleteCacheStrategy(
+            blobs=frozenset(delete_strategy_blobs),
+            refs=frozenset(delete_strategy_refs),
+            repos=frozenset(delete_strategy_repos),
+            snapshots=frozenset(delete_strategy_snapshots),
+            expected_freed_size=delete_strategy_expected_freed_size,
+        )
+
 
 def scan_cache_dir(cache_dir: Optional[Union[str, Path]] = None) -> HFCacheInfo:
     """Scan the entire HF cache-system and return a [`~HFCacheInfo`] structure.
@@ -356,7 +546,9 @@ def _scan_cached_repo(repo_path: Path) -> CachedRepoInfo:
         )
 
     # Scan over `refs` directory
-    refs_by_hash: Dict[str, Set[str]] = {}  # key is revision hash, value is set of refs
+
+    # key is revision hash, value is set of refs
+    refs_by_hash: Dict[str, Set[str]] = defaultdict(set)
     if refs_path.exists():
         # Example of `refs` directory
         # ── refs
@@ -378,10 +570,7 @@ def _scan_cached_repo(repo_path: Path) -> CachedRepoInfo:
             with ref_path.open() as f:
                 commit_hash = f.read()
 
-            if commit_hash in refs_by_hash:
-                refs_by_hash[commit_hash].add(ref_name)
-            else:
-                refs_by_hash[commit_hash] = {ref_name}
+            refs_by_hash[commit_hash].add(ref_name)
 
     # Scan snapshots directory
     cached_revisions: Set[CachedRevisionInfo] = set()
@@ -442,7 +631,7 @@ def _scan_cached_repo(repo_path: Path) -> CachedRepoInfo:
     if len(refs_by_hash) > 0:
         raise CorruptedCacheException(
             "Reference(s) refer to missing commit hashes:"
-            f" {refs_by_hash} ({repo_path})."
+            f" {dict(refs_by_hash)} ({repo_path})."
         )
 
     # Build and return frozen structure
@@ -467,3 +656,30 @@ def _format_size(num: int) -> str:
             return f"{num_f:3.1f}{unit}"
         num_f /= 1000.0
     return f"{num_f:.1f}Y"
+
+
+def _try_delete_path(path: Path, path_type: str) -> None:
+    """Try to delete a local file or folder.
+
+    If the path does not exists, error is logged as a warning and then ignored.
+
+    Args:
+        path (`Path`)
+            Path to delete. Can be a file or a folder.
+        path_type (`str`)
+            What path are we deleting ? Only for logging purposes. Example: "snapshot".
+    """
+    logger.info(f"Delete {path_type}: {path}")
+    try:
+        if path.is_file():
+            os.remove(path)
+        else:
+            shutil.rmtree(path)
+    except FileNotFoundError:
+        logger.warning(
+            f"Couldn't delete {path_type}: file not found ({path})", exc_info=True
+        )
+    except PermissionError:
+        logger.warning(
+            f"Couldn't delete {path_type}: permission denied ({path})", exc_info=True
+        )
diff --git a/src/huggingface_hub/utils/_http.py b/src/huggingface_hub/utils/_http.py
new file mode 100644
index 00000000..57d2c787
--- /dev/null
+++ b/src/huggingface_hub/utils/_http.py
@@ -0,0 +1,131 @@
+# coding=utf-8
+# Copyright 2022-present, the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Contains utilities to handle HTTP requests in Huggingface Hub."""
+import time
+from http import HTTPStatus
+from typing import Tuple, Type, Union
+
+import requests
+from requests import Response
+from requests.exceptions import ConnectTimeout, ProxyError
+
+from . import logging
+from ._typing import Literal
+
+
+logger = logging.get_logger(__name__)
+
+HTTP_METHOD_T = Literal["GET", "OPTIONS", "HEAD", "POST", "PUT", "PATCH", "DELETE"]
+
+
+def http_backoff(
+    method: HTTP_METHOD_T,
+    url: str,
+    *,
+    max_retries: int = 5,
+    base_wait_time: float = 1,
+    max_wait_time: float = 8,
+    retry_on_exceptions: Union[Type[Exception], Tuple[Type[Exception], ...]] = (
+        ConnectTimeout,
+        ProxyError,
+    ),
+    retry_on_status_codes: Union[int, Tuple[int, ...]] = HTTPStatus.SERVICE_UNAVAILABLE,
+    **kwargs,
+) -> Response:
+    """Wrapper around requests to retry calls on an endpoint, with exponential backoff.
+
+    Endpoint call is retried on exceptions (ex: connection timeout, proxy error,...)
+    and/or on specific status codes (ex: service unavailable). If the call failed more
+    than `max_retries`, the exception is thrown or `raise_for_status` is called on the
+    response object.
+
+    Re-implement mechanisms from the `backoff` library to avoid adding an external
+    dependencies to `hugging_face_hub`. See https://github.com/litl/backoff.
+
+    Args:
+        method (`Literal["GET", "OPTIONS", "HEAD", "POST", "PUT", "PATCH", "DELETE"]`):
+            HTTP method to perform.
+        url (`str`):
+            The URL of the resource to fetch.
+        max_retries (`int`, *optional*, defaults to `5`):
+            Maximum number of retries, defaults to 5 (no retries).
+        base_wait_time (`float`, *optional*, defaults to `1`):
+            Duration (in seconds) to wait before retrying the first time.
+            Wait time between retries then grows exponentially, capped by
+            `max_wait_time`.
+        max_wait_time (`float`, *optional*, defaults to `8`):
+            Maximum duration (in seconds) to wait before retrying.
+        retry_on_exceptions (`Type[Exception]` or `Tuple[Type[Exception]]`, *optional*, defaults to `(ConnectTimeout, ProxyError,)`):
+            Define which exceptions must be caught to retry the request. Can be a single
+            type or a tuple of types.
+            By default, retry on `ConnectTimeout` and `ProxyError`.
+        retry_on_status_codes (`int` or `Tuple[int]`, *optional*, defaults to `503`):
+            Define on which status codes the request must be retried. By default, only
+            HTTP 503 Service Unavailable is retried.
+        **kwargs (`dict`, *optional*):
+            kwargs to pass to `requests.request`.
+
+    Example:
+    ```
+    >>> from huggingface_hub.utils import http_backoff
+
+    # Same usage as "requests.request".
+    >>> response = http_backoff("GET", "https://www.google.com")
+    >>> response.raise_for_status()
+
+    # If you expect a Gateway Timeout from time to time
+    >>> http_backoff("PUT", upload_url, data=data, retry_on_status_codes=504)
+    >>> response.raise_for_status()
+    ```
+    """
+    if isinstance(retry_on_exceptions, type):  # Tuple from single exception type
+        retry_on_exceptions = (retry_on_exceptions,)
+
+    if isinstance(retry_on_status_codes, int):  # Tuple from single status code
+        retry_on_status_codes = (retry_on_status_codes,)
+
+    nb_tries = 0
+    sleep_time = base_wait_time
+    while True:
+        nb_tries += 1
+        try:
+            # Perform request and return if status_code is not in the retry list.
+            response = requests.request(method=method, url=url, **kwargs)
+            if response.status_code not in retry_on_status_codes:
+                return response
+
+            # Wrong status code returned (HTTP 503 for instance)
+            logger.warning(
+                f"HTTP Error {response.status_code} thrown while requesting"
+                f" {method} {url}"
+            )
+            if nb_tries > max_retries:
+                response.raise_for_status()  # Will raise uncaught exception
+                # We return response to avoid infinite loop in the corner case where the
+                # user ask for retry on a status code that doesn't raise_for_status.
+                return response
+
+        except retry_on_exceptions as err:
+            logger.warning(f"'{err}' thrown while requesting {method} {url}")
+
+            if nb_tries > max_retries:
+                raise err
+
+        # Sleep for X seconds
+        logger.warning(f"Retrying in {sleep_time}s [{nb_tries/max_retries}].")
+        time.sleep(sleep_time)
+
+        # Update sleep time for next retry
+        sleep_time = min(max_wait_time, sleep_time * 2)  # Exponential backoff


Test patch:
diff --git a/tests/test_utils_cache.py b/tests/test_utils_cache.py
index 1f8009bf..d1110916 100644
--- a/tests/test_utils_cache.py
+++ b/tests/test_utils_cache.py
@@ -13,7 +13,8 @@ import pytest
 from _pytest.fixtures import SubRequest
 from huggingface_hub._snapshot_download import snapshot_download
 from huggingface_hub.commands.cache import ScanCacheCommand
-from huggingface_hub.utils import scan_cache_dir
+from huggingface_hub.utils import DeleteCacheStrategy, HFCacheInfo, scan_cache_dir
+from huggingface_hub.utils._cache_manager import _try_delete_path
 
 from .testing_constants import TOKEN
 
@@ -345,3 +346,306 @@ class TestCorruptedCacheUtils(unittest.TestCase):
             " {'revision_hash_that_does_not_exist': {'not_main'}} "
             + f"({self.repo_path }).",
         )
+
+
+class TestDeleteRevisionsDryRun(unittest.TestCase):
+    cache_info: Mock  # Mocked HFCacheInfo
+
+    def setUp(self) -> None:
+        """Set up fake cache scan report."""
+        repo_A_path = Path("repo_A")
+        blobs_path = repo_A_path / "blobs"
+        snapshots_path = repo_A_path / "snapshots_path"
+
+        # Define blob files
+        main_only_file = Mock()
+        main_only_file.blob_path = blobs_path / "main_only_hash"
+        main_only_file.size_on_disk = 1
+
+        detached_only_file = Mock()
+        detached_only_file.blob_path = blobs_path / "detached_only_hash"
+        detached_only_file.size_on_disk = 10
+
+        pr_1_only_file = Mock()
+        pr_1_only_file.blob_path = blobs_path / "pr_1_only_hash"
+        pr_1_only_file.size_on_disk = 100
+
+        detached_and_pr_1_only_file = Mock()
+        detached_and_pr_1_only_file.blob_path = (
+            blobs_path / "detached_and_pr_1_only_hash"
+        )
+        detached_and_pr_1_only_file.size_on_disk = 1000
+
+        shared_file = Mock()
+        shared_file.blob_path = blobs_path / "shared_file_hash"
+        shared_file.size_on_disk = 10000
+
+        # Define revisions
+        repo_A_rev_main = Mock()
+        repo_A_rev_main.commit_hash = "repo_A_rev_main"
+        repo_A_rev_main.snapshot_path = snapshots_path / "repo_A_rev_main"
+        repo_A_rev_main.files = {main_only_file, shared_file}
+        repo_A_rev_main.refs = {"main"}
+
+        repo_A_rev_detached = Mock()
+        repo_A_rev_detached.commit_hash = "repo_A_rev_detached"
+        repo_A_rev_detached.snapshot_path = snapshots_path / "repo_A_rev_detached"
+        repo_A_rev_detached.files = {
+            detached_only_file,
+            detached_and_pr_1_only_file,
+            shared_file,
+        }
+        repo_A_rev_detached.refs = {}
+
+        repo_A_rev_pr_1 = Mock()
+        repo_A_rev_pr_1.commit_hash = "repo_A_rev_pr_1"
+        repo_A_rev_pr_1.snapshot_path = snapshots_path / "repo_A_rev_pr_1"
+        repo_A_rev_pr_1.files = {
+            pr_1_only_file,
+            detached_and_pr_1_only_file,
+            shared_file,
+        }
+        repo_A_rev_pr_1.refs = {"refs/pr/1"}
+
+        # Define repo
+        repo_A = Mock()
+        repo_A.repo_path = Path("repo_A")
+        repo_A.size_on_disk = 4444
+        repo_A.revisions = {repo_A_rev_main, repo_A_rev_detached, repo_A_rev_pr_1}
+
+        # Define cache
+        cache_info = Mock()
+        cache_info.repos = [repo_A]
+        self.cache_info = cache_info
+
+    def test_delete_detached_revision(self) -> None:
+        strategy = HFCacheInfo.delete_revisions(self.cache_info, "repo_A_rev_detached")
+        expected = DeleteCacheStrategy(
+            expected_freed_size=10,
+            blobs={
+                # "shared_file_hash" and "detached_and_pr_1_only_hash" are not deleted
+                Path("repo_A/blobs/detached_only_hash"),
+            },
+            refs=set(),  # No ref deleted since detached
+            repos=set(),  # No repo deleted as other revisions exist
+            snapshots={Path("repo_A/snapshots_path/repo_A_rev_detached")},
+        )
+        self.assertEqual(strategy, expected)
+
+    def test_delete_pr_1_revision(self) -> None:
+        strategy = HFCacheInfo.delete_revisions(self.cache_info, "repo_A_rev_pr_1")
+        expected = DeleteCacheStrategy(
+            expected_freed_size=100,
+            blobs={
+                # "shared_file_hash" and "detached_and_pr_1_only_hash" are not deleted
+                Path("repo_A/blobs/pr_1_only_hash")
+            },
+            refs={Path("repo_A/refs/refs/pr/1")},  # Ref is deleted !
+            repos=set(),  # No repo deleted as other revisions exist
+            snapshots={Path("repo_A/snapshots_path/repo_A_rev_pr_1")},
+        )
+        self.assertEqual(strategy, expected)
+
+    def test_delete_pr_1_and_detached(self) -> None:
+        strategy = HFCacheInfo.delete_revisions(
+            self.cache_info, "repo_A_rev_detached", "repo_A_rev_pr_1"
+        )
+        expected = DeleteCacheStrategy(
+            expected_freed_size=1110,
+            blobs={
+                Path("repo_A/blobs/detached_only_hash"),
+                Path("repo_A/blobs/pr_1_only_hash"),
+                # blob shared in both revisions and only those two
+                Path("repo_A/blobs/detached_and_pr_1_only_hash"),
+            },
+            refs={Path("repo_A/refs/refs/pr/1")},
+            repos=set(),
+            snapshots={
+                Path("repo_A/snapshots_path/repo_A_rev_detached"),
+                Path("repo_A/snapshots_path/repo_A_rev_pr_1"),
+            },
+        )
+        self.assertEqual(strategy, expected)
+
+    def test_delete_all_revisions(self) -> None:
+        strategy = HFCacheInfo.delete_revisions(
+            self.cache_info, "repo_A_rev_detached", "repo_A_rev_pr_1", "repo_A_rev_main"
+        )
+        expected = DeleteCacheStrategy(
+            expected_freed_size=4444,
+            blobs=set(),
+            refs=set(),
+            repos={Path("repo_A")},  # No remaining revisions: full repo is deleted
+            snapshots=set(),
+        )
+        self.assertEqual(strategy, expected)
+
+    def test_delete_unknown_revision(self) -> None:
+        with self.assertLogs() as captured:
+            strategy = HFCacheInfo.delete_revisions(
+                self.cache_info, "repo_A_rev_detached", "abcdef123456789"
+            )
+
+        # Expected is same strategy as without "abcdef123456789"
+        expected = HFCacheInfo.delete_revisions(self.cache_info, "repo_A_rev_detached")
+        self.assertEqual(strategy, expected)
+
+        # Expect a warning message
+        self.assertEqual(len(captured.records), 1)
+        self.assertEqual(captured.records[0].levelname, "WARNING")
+        self.assertEqual(
+            captured.records[0].message,
+            "Revision(s) not found - cannot delete them: abcdef123456789",
+        )
+
+
+@pytest.mark.usefixtures("fx_cache_dir")
+class TestDeleteStrategyExecute(unittest.TestCase):
+    cache_dir: Path
+
+    def test_execute(self) -> None:
+        # Repo folders
+        repo_A_path = self.cache_dir / "repo_A"
+        repo_A_path.mkdir()
+        repo_B_path = self.cache_dir / "repo_B"
+        repo_B_path.mkdir()
+
+        # Refs files in repo_B
+        refs_main_path = repo_B_path / "refs" / "main"
+        refs_main_path.parent.mkdir(parents=True)
+        refs_main_path.touch()
+        refs_pr_1_path = repo_B_path / "refs" / "refs" / "pr" / "1"
+        refs_pr_1_path.parent.mkdir(parents=True)
+        refs_pr_1_path.touch()
+
+        # Blobs files in repo_B
+        (repo_B_path / "blobs").mkdir()
+        blob_1 = repo_B_path / "blobs" / "blob_1"
+        blob_2 = repo_B_path / "blobs" / "blob_2"
+        blob_3 = repo_B_path / "blobs" / "blob_3"
+        blob_1.touch()
+        blob_2.touch()
+        blob_3.touch()
+
+        # Snapshot folders in repo_B
+        snapshot_1 = repo_B_path / "snapshots" / "snapshot_1"
+        snapshot_2 = repo_B_path / "snapshots" / "snapshot_2"
+
+        snapshot_1.mkdir(parents=True)
+        snapshot_2.mkdir()
+
+        # Execute deletion
+        # Delete repo_A + keep only blob_1, main ref and snapshot_1 in repo_B.
+        DeleteCacheStrategy(
+            expected_freed_size=123456,
+            blobs={blob_2, blob_3},
+            refs={refs_pr_1_path},
+            repos={repo_A_path},
+            snapshots={snapshot_2},
+        ).execute()
+
+        # Repo A deleted
+        self.assertFalse(repo_A_path.exists())
+        self.assertTrue(repo_B_path.exists())
+
+        # Only `blob` 1 remains
+        self.assertTrue(blob_1.exists())
+        self.assertFalse(blob_2.exists())
+        self.assertFalse(blob_3.exists())
+
+        # Only ref `main` remains
+        self.assertTrue(refs_main_path.exists())
+        self.assertFalse(refs_pr_1_path.exists())
+
+        # Only `snapshot_1` remains
+        self.assertTrue(snapshot_1.exists())
+        self.assertFalse(snapshot_2.exists())
+
+
+@pytest.mark.usefixtures("fx_cache_dir")
+class TestTryDeletePath(unittest.TestCase):
+    cache_dir: Path
+
+    def test_delete_path_on_file_success(self) -> None:
+        """Successfully delete a local file."""
+        file_path = self.cache_dir / "file.txt"
+        file_path.touch()
+        _try_delete_path(file_path, path_type="TYPE")
+        self.assertFalse(file_path.exists())
+
+    def test_delete_path_on_folder_success(self) -> None:
+        """Successfully delete a local folder."""
+        dir_path = self.cache_dir / "something"
+        subdir_path = dir_path / "bar"
+        subdir_path.mkdir(parents=True)  # subfolder
+
+        file_path_1 = dir_path / "file.txt"  # file at root
+        file_path_1.touch()
+
+        file_path_2 = subdir_path / "config.json"  # file in subfolder
+        file_path_2.touch()
+
+        _try_delete_path(dir_path, path_type="TYPE")
+
+        self.assertFalse(dir_path.exists())
+        self.assertFalse(subdir_path.exists())
+        self.assertFalse(file_path_1.exists())
+        self.assertFalse(file_path_2.exists())
+
+    def test_delete_path_on_missing_file(self) -> None:
+        """Try delete a missing file."""
+        file_path = self.cache_dir / "file.txt"
+
+        with self.assertLogs() as captured:
+            _try_delete_path(file_path, path_type="TYPE")
+
+        # Assert warning message with traceback for debug purposes
+        self.assertEquals(len(captured.output), 1)
+        self.assertTrue(
+            captured.output[0].startswith(
+                "WARNING:huggingface_hub.utils._cache_manager:Couldn't delete TYPE:"
+                f" file not found ({file_path})\nTraceback (most recent call last):"
+            )
+        )
+
+    def test_delete_path_on_missing_folder(self) -> None:
+        """Try delete a missing folder."""
+        dir_path = self.cache_dir / "folder"
+
+        with self.assertLogs() as captured:
+            _try_delete_path(dir_path, path_type="TYPE")
+
+        # Assert warning message with traceback for debug purposes
+        self.assertEquals(len(captured.output), 1)
+        self.assertTrue(
+            captured.output[0].startswith(
+                "WARNING:huggingface_hub.utils._cache_manager:Couldn't delete TYPE:"
+                f" file not found ({dir_path})\nTraceback (most recent call last):"
+            )
+        )
+
+    def test_delete_path_on_local_folder_with_wrong_permission(self) -> None:
+        """Try delete a local folder that is protected."""
+        dir_path = self.cache_dir / "something"
+        dir_path.mkdir()
+        file_path_1 = dir_path / "file.txt"  # file at root
+        file_path_1.touch()
+        dir_path.chmod(444)  # Read-only folder
+
+        with self.assertLogs() as captured:
+            _try_delete_path(dir_path, path_type="TYPE")
+
+        # Folder still exists (couldn't be deleted)
+        self.assertTrue(dir_path.is_dir())
+
+        # Assert warning message with traceback for debug purposes
+        self.assertEquals(len(captured.output), 1)
+        self.assertTrue(
+            captured.output[0].startswith(
+                "WARNING:huggingface_hub.utils._cache_manager:Couldn't delete TYPE:"
+                f" permission denied ({dir_path})\nTraceback (most recent call last):"
+            )
+        )
+
+        # For proper cleanup
+        dir_path.chmod(509)
diff --git a/tests/test_utils_http.py b/tests/test_utils_http.py
new file mode 100644
index 00000000..69fb0fff
--- /dev/null
+++ b/tests/test_utils_http.py
@@ -0,0 +1,127 @@
+import time
+import unittest
+from typing import Generator
+from unittest.mock import Mock, call, patch
+
+from huggingface_hub.utils._http import http_backoff
+from requests import ConnectTimeout, HTTPError
+
+
+URL = "https://www.google.com"
+
+
+@patch("huggingface_hub.utils._http.requests.request")
+class TestHttpBackoff(unittest.TestCase):
+    def test_backoff_no_errors(self, mock_request: Mock) -> None:
+        """Test normal usage of `http_backoff`."""
+        data_mock = Mock()
+        response = http_backoff("GET", URL, data=data_mock)
+        mock_request.assert_called_once_with(method="GET", url=URL, data=data_mock)
+        self.assertIs(response, mock_request())
+
+    def test_backoff_3_calls(self, mock_request: Mock) -> None:
+        """Test `http_backoff` with 2 fails."""
+        response_mock = Mock()
+        mock_request.side_effect = (ValueError(), ValueError(), response_mock)
+        response = http_backoff(  # retry on ValueError, instant retry
+            "GET", URL, retry_on_exceptions=ValueError, base_wait_time=0.0
+        )
+        self.assertEqual(mock_request.call_count, 3)
+        mock_request.assert_has_calls(
+            calls=[
+                call(method="GET", url=URL),
+                call(method="GET", url=URL),
+                call(method="GET", url=URL),
+            ]
+        )
+        self.assertIs(response, response_mock)
+
+    def test_backoff_on_exception_until_max(self, mock_request: Mock) -> None:
+        """Test `http_backoff` until max limit is reached with exceptions."""
+        mock_request.side_effect = ConnectTimeout()
+
+        with self.assertRaises(ConnectTimeout):
+            http_backoff("GET", URL, base_wait_time=0.0, max_retries=3)
+
+        self.assertEqual(mock_request.call_count, 4)
+
+    def test_backoff_on_status_code_until_max(self, mock_request: Mock) -> None:
+        """Test `http_backoff` until max limit is reached with status codes."""
+        mock_503 = Mock()
+        mock_503.status_code = 503
+        mock_504 = Mock()
+        mock_504.status_code = 504
+        mock_504.raise_for_status.side_effect = HTTPError()
+        mock_request.side_effect = (mock_503, mock_504, mock_503, mock_504)
+
+        with self.assertRaises(HTTPError):
+            http_backoff(
+                "GET",
+                URL,
+                base_wait_time=0.0,
+                max_retries=3,
+                retry_on_status_codes=(503, 504),
+            )
+
+        self.assertEqual(mock_request.call_count, 4)
+
+    def test_backoff_on_exceptions_and_status_codes(self, mock_request: Mock) -> None:
+        """Test `http_backoff` until max limit with status codes and exceptions."""
+        mock_503 = Mock()
+        mock_503.status_code = 503
+        mock_request.side_effect = (mock_503, ConnectTimeout())
+
+        with self.assertRaises(ConnectTimeout):
+            http_backoff("GET", URL, base_wait_time=0.0, max_retries=1)
+
+        self.assertEqual(mock_request.call_count, 2)
+
+    def test_backoff_on_valid_status_code(self, mock_request: Mock) -> None:
+        """Test `http_backoff` until max limit with a valid status code.
+
+        Quite a corner case: the user wants to retry is status code is 200. Requests are
+        retried but in the end, the HTTP 200 response is returned if the server returned
+        only 200 responses.
+        """
+        mock_200 = Mock()
+        mock_200.status_code = 200
+        mock_request.side_effect = (mock_200, mock_200, mock_200, mock_200)
+
+        response = http_backoff(
+            "GET", URL, base_wait_time=0.0, max_retries=3, retry_on_status_codes=200
+        )
+
+        self.assertEqual(mock_request.call_count, 4)
+        self.assertIs(response, mock_200)
+
+    def test_backoff_sleep_time(self, mock_request: Mock) -> None:
+        """Test `http_backoff` sleep time goes exponential until max limit.
+
+        Since timing between 2 requests is sleep duration + some other stuff, this test
+        can be unstable. However, sleep durations between 10ms and 50ms should be enough
+        to make the approximation that measured durations are the "sleep time" waited by
+        `http_backoff`. If this is not the case, just increase `base_wait_time`,
+        `max_wait_time` and `expected_sleep_times` with bigger values.
+        """
+        sleep_times = []
+
+        def _side_effect_timer() -> Generator[ConnectTimeout, None, None]:
+            t0 = time.time()
+            while True:
+                yield ConnectTimeout()
+                t1 = time.time()
+                sleep_times.append(round(t1 - t0, 2))
+                t0 = t1
+
+        mock_request.side_effect = _side_effect_timer()
+
+        with self.assertRaises(ConnectTimeout):
+            http_backoff(
+                "GET", URL, base_wait_time=0.01, max_wait_time=0.05, max_retries=5
+            )
+
+        self.assertEqual(mock_request.call_count, 6)
+
+        # Assert sleep times are exponential until plateau
+        expected_sleep_times = [0.01, 0.02, 0.04, 0.05, 0.05]
+        self.assertListEqual(sleep_times, expected_sleep_times)
diff --git a/tests/testing_utils.py b/tests/testing_utils.py
index 51b48ca1..7912c285 100644
--- a/tests/testing_utils.py
+++ b/tests/testing_utils.py
@@ -128,7 +128,7 @@ def offline(mode=OfflineSimulationMode.CONNECTION_FAILS, timeout=1e-16):
     """
     Simulate offline mode.
 
-    There are three offline simulatiom modes:
+    There are three offline simulation modes:
 
     CONNECTION_FAILS (default mode): a ConnectionError is raised for each network call.
         Connection errors are created by mocking socket.socket
@@ -136,7 +136,7 @@ def offline(mode=OfflineSimulationMode.CONNECTION_FAILS, timeout=1e-16):
         The default timeout value is low (1e-16) to speed up the tests.
         Timeout errors are created by mocking requests.request
     HF_HUB_OFFLINE_SET_TO_1: the HF_HUB_OFFLINE_SET_TO_1 environment variable is set to 1.
-        This makes the http/ftp calls of the library instantly fail and raise an OfflineModeEmabled error.
+        This makes the http/ftp calls of the library instantly fail and raise an OfflineModeEnabled error.
     """
     import socket