Repository: huggingface/huggingface_hub
Pull request #876 (2022-05-17 09:50:20)

ENH Removed history writing in Keras model card
This PR solves #869. It's a broken down version of https://github.com/huggingface/huggingface_hub/pull/872.


Patch:
diff --git a/codecov.yml b/codecov.yml
new file mode 100644
index 00000000..69cb7601
--- /dev/null
+++ b/codecov.yml
@@ -0,0 +1,1 @@
+comment: false
diff --git a/docs/source/package_reference/file_download.mdx b/docs/source/package_reference/file_download.mdx
index 79d1063c..1a05dfdf 100644
--- a/docs/source/package_reference/file_download.mdx
+++ b/docs/source/package_reference/file_download.mdx
@@ -8,3 +8,109 @@
 
 [[autodoc]] huggingface_hub.hf_hub_url
 
+## Caching
+
+The methods displayed above are designed to work with a caching system that prevents re-downloading files.
+The caching system was updated in v0.8.0 to allow directory structure and file sharing across
+libraries that depend on the hub.
+
+The caching system is designed as follows:
+
+```
+<CACHE_DIR>
+├─ <MODELS>
+├─ <DATASETS>
+├─ <SPACES>
+```
+
+The `<CACHE_DIR>` is usually your user's home directory. However, it is customizable with the
+`cache_dir` argument on all methods, or by specifying the `HF_HOME` environment variable.
+
+Models, datasets and spaces share a common root. Each of these repositories contains the namespace
+(organization, username) if it exists, alongside the repository name:
+
+```
+<CACHE_DIR>
+├─ models--julien-c--EsperBERTo-small
+├─ models--lysandrejik--arxiv-nlp
+├─ models--bert-base-cased
+├─ datasets--glue
+├─ datasets--huggingface--DataMeasurementsFiles
+├─ spaces--dalle-mini--dalle-mini
+```
+
+It is within these folders that all files will now be downloaded from the hub. Caching ensures that
+a file isn't downloaded twice if it already exists and wasn't updated; but if it was updated,
+and you're asking for the latest file, then it will download the latest file (while keeping
+the previous file intact in case you need it again).
+
+In order to achieve this, all folders contain the same skeleton:
+
+```
+<CACHE_DIR>
+├─ datasets--glue
+│  ├─ refs
+│  ├─ blobs
+│  ├─ snapshots
+...
+```
+
+Each folder is designed to contain the following:
+
+### Refs
+
+The `refs` folder contains files which indicates the latest revision of the given reference. For example,
+if we have previously fetched a file from the `main` branch of a repository, the `refs`
+folder will contain a file named `main`, which will itself contain the commit identifier of the current head.
+
+If the latest commit of `main` has `aaaaaa` as identifier, then it will contain `aaaaaa`.
+
+If that same branch gets updated with a new commit, that has `bbbbbb` as an identifier, then
+redownloading a file from that reference will update the `refs/main` file to contain `bbbbbb`.
+
+### Blobs
+
+The `blobs` folder contains the actual files that we have downloaded. The name of each file is their hash.
+
+### Snapshots
+
+The `snapshots` folder contains symlinks to the blobs mentioned above. It is itself made up of several folders:
+one per known revision!
+
+In the explanation above, we had initially fetched a file from the `aaaaaa` revision, before fetching a file from
+the `bbbbbb` revision. In this situation, we would now have two folders in the `snapshots` folder: `aaaaaa`
+and `bbbbbb`.
+
+In each of these folders, live symlinks that have the names of the files that we have downloaded. For example,
+if we had downloaded the `READMD.md` file at revision `aaaaaa`, we would have the following path:
+
+```
+<CACHE_DIR>/<REPO_NAME>/snapshots/aaaaaa/README.md
+```
+
+That `README.md` file is actually a symlink linking to the blob that has the hash of the file.
+
+Creating the skeleton this way means opens up the mechanism to file sharing: if the same file was fetched in
+revision `bbbbbb`, it would have the same hash and the file would not need to be redownloaded.
+
+### In practice
+
+In practice, it should look like the following tree in your cache:
+
+```
+    [  96]  .
+    └── [ 160]  models--julien-c--EsperBERTo-small
+        ├── [ 160]  blobs
+        │   ├── [321M]  403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
+        │   ├── [ 398]  7cb18dc9bafbfcf74629a4b760af1b160957a83e
+        │   └── [1.4K]  d7edf6bd2a681fb0175f7735299831ee1b22b812
+        ├── [  96]  refs
+        │   └── [  40]  main
+        └── [ 128]  snapshots
+            ├── [ 128]  2439f60ef33a0d46d85da5001d52aeda5b00ce9f
+            │   ├── [  52]  README.md -> ../../blobs/d7edf6bd2a681fb0175f7735299831ee1b22b812
+            │   └── [  76]  pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
+            └── [ 128]  bbc77c8132af1cc5cf678da3f1ddf2de43606d48
+                ├── [  52]  README.md -> ../../blobs/7cb18dc9bafbfcf74629a4b760af1b160957a83e
+                └── [  76]  pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
+```
\ No newline at end of file
diff --git a/setup.py b/setup.py
index 748f39d1..9a2eae17 100644
--- a/setup.py
+++ b/setup.py
@@ -37,6 +37,7 @@ extras["tensorflow"] = ["tensorflow", "pydot", "graphviz"]
 
 extras["testing"] = [
     "pytest",
+    "pytest-cov",
     "datasets",
     "soundfile",
 ]
@@ -58,7 +59,8 @@ setup(
     author="Hugging Face, Inc.",
     author_email="julien@huggingface.co",
     description=(
-        "Client library to download and publish models on the huggingface.co hub"
+        "Client library to download and publish models, datasets and other repos on the"
+        " huggingface.co hub"
     ),
     long_description=open("README.md", "r", encoding="utf-8").read(),
     long_description_content_type="text/markdown",
diff --git a/src/huggingface_hub/__init__.py b/src/huggingface_hub/__init__.py
index f7f8827e..ea376f78 100644
--- a/src/huggingface_hub/__init__.py
+++ b/src/huggingface_hub/__init__.py
@@ -16,70 +16,177 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-__version__ = "0.7.0.dev0"
-
-from .commands.user import notebook_login
-from .constants import (
-    CONFIG_NAME,
-    FLAX_WEIGHTS_NAME,
-    HUGGINGFACE_CO_URL_HOME,
-    HUGGINGFACE_CO_URL_TEMPLATE,
-    PYTORCH_WEIGHTS_NAME,
-    REPO_TYPE_DATASET,
-    REPO_TYPE_MODEL,
-    REPO_TYPE_SPACE,
-    TF2_WEIGHTS_NAME,
-    TF_WEIGHTS_NAME,
-)
-from .fastai_utils import (
-    _save_pretrained_fastai,
-    from_pretrained_fastai,
-    push_to_hub_fastai,
-)
-from .file_download import cached_download, hf_hub_download, hf_hub_url
-from .hf_api import (
-    DatasetSearchArguments,
-    HfApi,
-    HfFolder,
-    ModelSearchArguments,
-    create_repo,
-    dataset_info,
-    delete_file,
-    delete_repo,
-    get_dataset_tags,
-    get_full_repo_name,
-    get_model_tags,
-    list_datasets,
-    list_metrics,
-    list_models,
-    list_repo_files,
-    login,
-    logout,
-    model_info,
-    move_repo,
-    repo_type_and_id_from_hf_id,
-    set_access_token,
-    space_info,
-    unset_access_token,
-    update_repo_visibility,
-    upload_file,
-    whoami,
-)
-from .hub_mixin import ModelHubMixin, PyTorchModelHubMixin
-from .inference_api import InferenceApi
-from .keras_mixin import (
-    KerasModelHubMixin,
-    from_pretrained_keras,
-    push_to_hub_keras,
-    save_pretrained_keras,
-)
-from .repocard import (
-    metadata_eval_result,
-    metadata_load,
-    metadata_save,
-    metadata_update,
+# ***********
+# vendored from https://github.com/scientific-python/lazy_loader
+import importlib
+import importlib.util
+import inspect
+import os
+import sys
+import types
+import warnings
+
+
+class _LazyImportWarning(Warning):
+    pass
+
+
+def _attach(package_name, submodules=None, submod_attrs=None):
+    """Attach lazily loaded submodules, functions, or other attributes.
+
+    Typically, modules import submodules and attributes as follows:
+
+    ```py
+    import mysubmodule
+    import anothersubmodule
+
+    from .foo import someattr
+    ```
+
+    The idea is to replace a package's `__getattr__`, `__dir__`, and
+    `__all__`, such that all imports work exactly the way they would
+    with normal imports, except that the import occurs upon first use.
+
+    The typical way to call this function, replacing the above imports, is:
+
+    ```python
+    __getattr__, __dir__, __all__ = lazy.attach(
+        __name__,
+        ['mysubmodule', 'anothersubmodule'],
+        {'foo': ['someattr']}
+    )
+    ```
+    This functionality requires Python 3.7 or higher.
+
+    Args:
+        package_name (`str`):
+            Typically use `__name__`.
+        submodules (`set`):
+            List of submodules to attach.
+        submod_attrs (`dict`):
+            Dictionary of submodule -> list of attributes / functions.
+            These attributes are imported as they are used.
+
+    Returns:
+        __getattr__, __dir__, __all__
+
+    """
+    if submod_attrs is None:
+        submod_attrs = {}
+
+    if submodules is None:
+        submodules = set()
+    else:
+        submodules = set(submodules)
+
+    attr_to_modules = {
+        attr: mod for mod, attrs in submod_attrs.items() for attr in attrs
+    }
+
+    __all__ = list(submodules | attr_to_modules.keys())
+
+    def __getattr__(name):
+        if name in submodules:
+            return importlib.import_module(f"{package_name}.{name}")
+        elif name in attr_to_modules:
+            submod_path = f"{package_name}.{attr_to_modules[name]}"
+            submod = importlib.import_module(submod_path)
+            attr = getattr(submod, name)
+
+            # If the attribute lives in a file (module) with the same
+            # name as the attribute, ensure that the attribute and *not*
+            # the module is accessible on the package.
+            if name == attr_to_modules[name]:
+                pkg = sys.modules[package_name]
+                pkg.__dict__[name] = attr
+
+            return attr
+        else:
+            raise AttributeError(f"No {package_name} attribute {name}")
+
+    def __dir__():
+        return __all__
+
+    if os.environ.get("EAGER_IMPORT", ""):
+        for attr in set(attr_to_modules.keys()) | submodules:
+            __getattr__(attr)
+
+    return __getattr__, __dir__, list(__all__)
+
+
+# ************
+
+__version__ = "0.8.0.dev0"
+
+
+__getattr__, __dir__, __all__ = _attach(
+    __name__,
+    submodules=[],
+    submod_attrs={
+        "commands.user": ["notebook_login"],
+        "constants": [
+            "CONFIG_NAME",
+            "FLAX_WEIGHTS_NAME",
+            "HUGGINGFACE_CO_URL_HOME",
+            "HUGGINGFACE_CO_URL_TEMPLATE",
+            "PYTORCH_WEIGHTS_NAME",
+            "REPO_TYPE_DATASET",
+            "REPO_TYPE_MODEL",
+            "REPO_TYPE_SPACE",
+            "TF2_WEIGHTS_NAME",
+            "TF_WEIGHTS_NAME",
+        ],
+        "fastai_utils": [
+            "_save_pretrained_fastai",
+            "from_pretrained_fastai",
+            "push_to_hub_fastai",
+        ],
+        "file_download": ["cached_download", "hf_hub_download", "hf_hub_url"],
+        "hf_api": [
+            "DatasetSearchArguments",
+            "HfApi",
+            "HfFolder",
+            "ModelSearchArguments",
+            "create_repo",
+            "dataset_info",
+            "delete_file",
+            "delete_repo",
+            "get_dataset_tags",
+            "get_full_repo_name",
+            "get_model_tags",
+            "list_datasets",
+            "list_metrics",
+            "list_models",
+            "list_repo_files",
+            "login",
+            "logout",
+            "model_info",
+            "move_repo",
+            "repo_type_and_id_from_hf_id",
+            "set_access_token",
+            "space_info",
+            "unset_access_token",
+            "update_repo_visibility",
+            "upload_file",
+            "whoami",
+        ],
+        "hub_mixin": ["ModelHubMixin", "PyTorchModelHubMixin"],
+        "inference_api": ["InferenceApi"],
+        "keras_mixin": [
+            "KerasModelHubMixin",
+            "from_pretrained_keras",
+            "push_to_hub_keras",
+            "save_pretrained_keras",
+        ],
+        "repository": ["Repository"],
+        "_snapshot_download": ["snapshot_download"],
+        "utils": ["logging"],
+        "utils.endpoint_helpers": ["DatasetFilter", "ModelFilter"],
+        "repocard": [
+            "metadata_eval_result",
+            "metadata_load",
+            "metadata_save",
+            "metadata_update",
+        ],
+    },
 )
-from .repository import Repository
-from .snapshot_download import snapshot_download
-from .utils import logging
-from .utils.endpoint_helpers import DatasetFilter, ModelFilter
diff --git a/src/huggingface_hub/_snapshot_download.py b/src/huggingface_hub/_snapshot_download.py
new file mode 100644
index 00000000..df65e4f2
--- /dev/null
+++ b/src/huggingface_hub/_snapshot_download.py
@@ -0,0 +1,223 @@
+import os
+from fnmatch import fnmatch
+from pathlib import Path
+from typing import Dict, List, Optional, Union
+
+from .constants import DEFAULT_REVISION, HUGGINGFACE_HUB_CACHE, REPO_TYPES
+from .file_download import REGEX_COMMIT_HASH, hf_hub_download, repo_folder_name
+from .hf_api import HfApi, HfFolder
+from .utils import logging
+from .utils._deprecation import _deprecate_positional_args
+
+
+logger = logging.get_logger(__name__)
+
+
+def _filter_repo_files(
+    *,
+    repo_files: List[str],
+    allow_regex: Optional[Union[List[str], str]] = None,
+    ignore_regex: Optional[Union[List[str], str]] = None,
+) -> List[str]:
+    allow_regex = [allow_regex] if isinstance(allow_regex, str) else allow_regex
+    ignore_regex = [ignore_regex] if isinstance(ignore_regex, str) else ignore_regex
+    filtered_files = []
+    for repo_file in repo_files:
+        # if there's an allowlist, skip download if file does not match any regex
+        if allow_regex is not None and not any(
+            fnmatch(repo_file, r) for r in allow_regex
+        ):
+            continue
+
+        # if there's a denylist, skip download if file does matches any regex
+        if ignore_regex is not None and any(
+            fnmatch(repo_file, r) for r in ignore_regex
+        ):
+            continue
+
+        filtered_files.append(repo_file)
+    return filtered_files
+
+
+@_deprecate_positional_args
+def snapshot_download(
+    repo_id: str,
+    *,
+    revision: Optional[str] = None,
+    repo_type: Optional[str] = None,
+    cache_dir: Union[str, Path, None] = None,
+    library_name: Optional[str] = None,
+    library_version: Optional[str] = None,
+    user_agent: Optional[Union[Dict, str]] = None,
+    proxies: Optional[Dict] = None,
+    etag_timeout: Optional[float] = 10,
+    resume_download: Optional[bool] = False,
+    use_auth_token: Optional[Union[bool, str]] = None,
+    local_files_only: Optional[bool] = False,
+    allow_regex: Optional[Union[List[str], str]] = None,
+    ignore_regex: Optional[Union[List[str], str]] = None,
+) -> str:
+    """Download all files of a repo.
+
+    Downloads a whole snapshot of a repo's files at the specified revision. This
+    is useful when you want all files from a repo, because you don't know which
+    ones you will need a priori. All files are nested inside a folder in order
+    to keep their actual filename relative to that folder.
+
+    An alternative would be to just clone a repo but this would require that the
+    user always has git and git-lfs installed, and properly configured.
+
+    Args:
+        repo_id (`str`):
+            A user or an organization name and a repo name separated by a `/`.
+        revision (`str`, *optional*):
+            An optional Git revision id which can be a branch name, a tag, or a
+            commit hash.
+        repo_type (`str`, *optional*):
+            Set to `"dataset"` or `"space"` if uploading to a dataset or space,
+            `None` or `"model"` if uploading to a model. Default is `None`.
+        cache_dir (`str`, `Path`, *optional*):
+            Path to the folder where cached files are stored.
+        library_name (`str`, *optional*):
+            The name of the library to which the object corresponds.
+        library_version (`str`, *optional*):
+            The version of the library.
+        user_agent (`str`, `dict`, *optional*):
+            The user-agent info in the form of a dictionary or a string.
+        proxies (`dict`, *optional*):
+            Dictionary mapping protocol to the URL of the proxy passed to
+            `requests.request`.
+        etag_timeout (`float`, *optional*, defaults to `10`):
+            When fetching ETag, how many seconds to wait for the server to send
+            data before giving up which is passed to `requests.request`.
+        resume_download (`bool`, *optional*, defaults to `False):
+            If `True`, resume a previously interrupted download.
+        use_auth_token (`str`, `bool`, *optional*):
+            A token to be used for the download.
+                - If `True`, the token is read from the HuggingFace config
+                  folder.
+                - If a string, it's used as the authentication token.
+        local_files_only (`bool`, *optional*, defaults to `False`):
+            If `True`, avoid downloading the file and return the path to the
+            local cached file if it exists.
+        allow_regex (`list of str`, `str`, *optional*):
+            If provided, only files matching this regex are downloaded.
+        ignore_regex (`list of str`, `str`, *optional*):
+            If provided, files matching this regex are not downloaded.
+
+    Returns:
+        Local folder path (string) of repo snapshot
+
+    <Tip>
+
+    Raises the following errors:
+
+    - [`EnvironmentError`](https://docs.python.org/3/library/exceptions.html#EnvironmentError)
+      if `use_auth_token=True` and the token cannot be found.
+    - [`OSError`](https://docs.python.org/3/library/exceptions.html#OSError) if
+      ETag cannot be determined.
+    - [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError)
+      if some parameter value is invalid
+
+    </Tip>
+    """
+
+    if cache_dir is None:
+        cache_dir = HUGGINGFACE_HUB_CACHE
+    if revision is None:
+        revision = DEFAULT_REVISION
+    if isinstance(cache_dir, Path):
+        cache_dir = str(cache_dir)
+
+    if isinstance(use_auth_token, str):
+        token = use_auth_token
+    elif use_auth_token:
+        token = HfFolder.get_token()
+        if token is None:
+            raise EnvironmentError(
+                "You specified use_auth_token=True, but a Hugging Face token was not"
+                " found."
+            )
+    else:
+        token = None
+
+    if repo_type is None:
+        repo_type = "model"
+    if repo_type not in REPO_TYPES:
+        raise ValueError(
+            f"Invalid repo type: {repo_type}. Accepted repo types are:"
+            f" {str(REPO_TYPES)}"
+        )
+
+    storage_folder = os.path.join(
+        cache_dir, repo_folder_name(repo_id=repo_id, repo_type=repo_type)
+    )
+
+    # if we have no internet connection we will look for an
+    # appropriate folder in the cache
+    # If the specified revision is a commit hash, look inside "snapshots".
+    # If the specified revision is a branch or tag, look inside "refs".
+    if local_files_only:
+
+        if REGEX_COMMIT_HASH.match(revision):
+            commit_hash = revision
+        else:
+            # retrieve commit_hash from file
+            ref_path = os.path.join(storage_folder, "refs", revision)
+            with open(ref_path) as f:
+                commit_hash = f.read()
+
+        snapshot_folder = os.path.join(storage_folder, "snapshots", commit_hash)
+
+        if os.path.exists(snapshot_folder):
+            return snapshot_folder
+
+        raise ValueError(
+            "Cannot find an appropriate cached snapshot folder for the specified"
+            " revision on the local disk and outgoing traffic has been disabled. To"
+            " enable repo look-ups and downloads online, set 'local_files_only' to"
+            " False."
+        )
+
+    # if we have internet connection we retrieve the correct folder name from the huggingface api
+    _api = HfApi()
+    repo_info = _api.repo_info(
+        repo_id=repo_id, repo_type=repo_type, revision=revision, token=token
+    )
+    filtered_repo_files = _filter_repo_files(
+        repo_files=[f.rfilename for f in repo_info.siblings],
+        allow_regex=allow_regex,
+        ignore_regex=ignore_regex,
+    )
+    commit_hash = repo_info.sha
+    snapshot_folder = os.path.join(storage_folder, "snapshots", commit_hash)
+    # if passed revision is not identical to commit_hash
+    # then revision has to be a branch name or tag name.
+    # In that case store a ref.
+    if revision != commit_hash:
+        ref_path = os.path.join(storage_folder, "refs", revision)
+        os.makedirs(os.path.dirname(ref_path), exist_ok=True)
+        with open(ref_path, "w") as f:
+            f.write(commit_hash)
+
+    # we pass the commit_hash to hf_hub_download
+    # so no network call happens if we already
+    # have the file locally.
+
+    for repo_file in filtered_repo_files:
+        _ = hf_hub_download(
+            repo_id,
+            filename=repo_file,
+            repo_type=repo_type,
+            revision=commit_hash,
+            cache_dir=cache_dir,
+            library_name=library_name,
+            library_version=library_version,
+            user_agent=user_agent,
+            proxies=proxies,
+            etag_timeout=etag_timeout,
+            resume_download=resume_download,
+            use_auth_token=use_auth_token,
+        )
+
+    return snapshot_folder
diff --git a/src/huggingface_hub/constants.py b/src/huggingface_hub/constants.py
index 91f930d8..728f45ac 100644
--- a/src/huggingface_hub/constants.py
+++ b/src/huggingface_hub/constants.py
@@ -29,6 +29,13 @@ ENDPOINT = os.getenv("HF_ENDPOINT") or (
 )
 
 HUGGINGFACE_CO_URL_TEMPLATE = ENDPOINT + "/{repo_id}/resolve/{revision}/{filename}"
+HUGGINGFACE_HEADER_X_REPO_COMMIT = "X-Repo-Commit"
+HUGGINGFACE_HEADER_X_LINKED_ETAG = "X-Linked-Etag"
+
+REPO_ID_SEPARATOR = "--"
+# ^ this substring is not allowed in repo_ids on hf.co
+# and is the canonical one we use for serialization of repo ids elsewhere.
+
 
 REPO_TYPE_DATASET = "dataset"
 REPO_TYPE_SPACE = "space"
diff --git a/src/huggingface_hub/fastai_utils.py b/src/huggingface_hub/fastai_utils.py
index eb16c2bd..7fee0f6c 100644
--- a/src/huggingface_hub/fastai_utils.py
+++ b/src/huggingface_hub/fastai_utils.py
@@ -6,6 +6,7 @@ from typing import Any, Dict, Optional
 
 from packaging import version
 
+from huggingface_hub import snapshot_download
 from huggingface_hub.constants import CONFIG_NAME
 from huggingface_hub.file_download import (
     _PY_VERSION,
@@ -14,7 +15,6 @@ from huggingface_hub.file_download import (
 )
 from huggingface_hub.hf_api import HfApi, HfFolder
 from huggingface_hub.repository import Repository
-from huggingface_hub.snapshot_download import snapshot_download
 from huggingface_hub.utils import logging
 
 
diff --git a/src/huggingface_hub/file_download.py b/src/huggingface_hub/file_download.py
index 17200360..c3e602f8 100644
--- a/src/huggingface_hub/file_download.py
+++ b/src/huggingface_hub/file_download.py
@@ -3,9 +3,11 @@ import fnmatch
 import io
 import json
 import os
+import re
 import sys
 import tempfile
 import time
+import warnings
 from contextlib import contextmanager
 from functools import partial
 from hashlib import sha256
@@ -23,7 +25,10 @@ from . import __version__
 from .constants import (
     DEFAULT_REVISION,
     HUGGINGFACE_CO_URL_TEMPLATE,
+    HUGGINGFACE_HEADER_X_LINKED_ETAG,
+    HUGGINGFACE_HEADER_X_REPO_COMMIT,
     HUGGINGFACE_HUB_CACHE,
+    REPO_ID_SEPARATOR,
     REPO_TYPES,
     REPO_TYPES_URL_PREFIXES,
 )
@@ -142,6 +147,9 @@ def get_fastcore_version():
     return _fastcore_version
 
 
+REGEX_COMMIT_HASH = re.compile(r"^[0-9a-f]{40}$")
+
+
 @_deprecate_positional_args
 def hf_hub_url(
     repo_id: str,
@@ -255,11 +263,31 @@ def url_to_filename(url: str, etag: Optional[str] = None) -> str:
     return filename
 
 
-def filename_to_url(filename, cache_dir=None) -> Tuple[str, str]:
+def filename_to_url(
+    filename,
+    cache_dir: Optional[str] = None,
+    legacy_cache_layout: Optional[bool] = False,
+) -> Tuple[str, str]:
     """
     Return the url and etag (which may be `None`) stored for `filename`. Raise
     `EnvironmentError` if `filename` or its stored metadata do not exist.
+
+    Args:
+        filename (`str`):
+            The name of the file
+        cache_dir (`str`, *optional*):
+            The cache directory to use instead of the default one.
+        legacy_cache_layout (`bool`, *optional*, defaults to `False`):
+            If `True`, uses the legacy file cache layout i.e. just call `hf_hub_url`
+            then `cached_download`. This is deprecated as the new cache layout is
+            more powerful.
     """
+    if not legacy_cache_layout:
+        warnings.warn(
+            "`filename_to_url` uses the legacy way cache file layout",
+            FutureWarning,
+        )
+
     if cache_dir is None:
         cache_dir = HUGGINGFACE_HUB_CACHE
     if isinstance(cache_dir, Path):
@@ -455,6 +483,7 @@ def cached_download(
     resume_download: Optional[bool] = False,
     use_auth_token: Union[bool, str, None] = None,
     local_files_only: Optional[bool] = False,
+    legacy_cache_layout: Optional[bool] = False,
 ) -> Optional[str]:  # pragma: no cover
     """
     Download from a given URL and cache it if it's not already present in the
@@ -496,6 +525,11 @@ def cached_download(
         local_files_only (`bool`, *optional*, defaults to `False`):
             If `True`, avoid downloading the file and return the path to the
             local cached file if it exists.
+        legacy_cache_layout (`bool`, *optional*, defaults to `False`):
+            Set this parameter to `True` to mention that you'd like to continue
+            the old cache layout. Putting this to `True` manually will not raise
+            any warning when using `cached_download`. We recommend using
+            `hf_hub_download` to take advantage of the new cache.
 
     Returns:
         Local path (string) of file or if networking is off, last version of
@@ -514,6 +548,13 @@ def cached_download(
 
     </Tip>
     """
+    if not legacy_cache_layout:
+        warnings.warn(
+            "`cached_download` is the legacy way to download files from the HF hub,"
+            " please consider upgrading to `hf_hub_download`",
+            FutureWarning,
+        )
+
     if cache_dir is None:
         cache_dir = HUGGINGFACE_HUB_CACHE
     if isinstance(cache_dir, Path):
@@ -689,6 +730,66 @@ def cached_download(
     return cache_path
 
 
+def _normalize_etag(etag: str) -> str:
+    """Normalize ETag HTTP header, so it can be used to create nice filepaths.
+
+    The HTTP spec allows two forms of ETag:
+      ETag: W/"<etag_value>"
+      ETag: "<etag_value>"
+
+    The hf.co hub guarantees to only send the second form.
+
+    Args:
+        etag (`str`): HTTP header
+
+    Returns:
+        `str`: string that can be used as a nice directory name.
+    """
+    return etag.strip('"')
+
+
+def _create_relative_symlink(src: str, dst: str) -> None:
+    """Create a symbolic link named dst pointing to src as a relative path to dst.
+
+    The relative part is mostly because it seems more elegant to the author.
+
+    The result layout looks something like
+        └── [ 128]  snapshots
+            ├── [ 128]  2439f60ef33a0d46d85da5001d52aeda5b00ce9f
+            │   ├── [  52]  README.md -> ../../blobs/d7edf6bd2a681fb0175f7735299831ee1b22b812
+            │   └── [  76]  pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
+    """
+    relative_src = os.path.relpath(src, start=os.path.dirname(dst))
+    try:
+        os.remove(dst)
+    except OSError:
+        pass
+    try:
+        os.symlink(relative_src, dst)
+    except OSError:
+        # Likely running on Windows
+        if os.name == "nt":
+            raise OSError(
+                "Windows requires Developer Mode to be activated, or to run Python as "
+                "an administrator, in order to create symlinks.\nIn order to "
+                "activate Developer Mode, see this article: "
+                "https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development"
+            )
+        else:
+            raise
+
+
+def repo_folder_name(*, repo_id: str, repo_type: str) -> str:
+    """Return a serialized version of a hf.co repo name and type, safe for disk storage
+    as a single non-nested folder.
+
+    Example: models--julien-c--EsperBERTo-small
+    """
+    # remove all `/` occurrences to correctly convert repo to directory name
+    parts = [f"{repo_type}s", *repo_id.split("/")]
+    return REPO_ID_SEPARATOR.join(parts)
+
+
 @_deprecate_positional_args
 def hf_hub_download(
     repo_id: str,
@@ -708,9 +809,36 @@ def hf_hub_download(
     resume_download: Optional[bool] = False,
     use_auth_token: Union[bool, str, None] = None,
     local_files_only: Optional[bool] = False,
+    legacy_cache_layout: Optional[bool] = False,
 ):
     """Download a given file if it's not already present in the local cache.
 
+    The new cache file layout looks like this:
+    - The cache directory contains one subfolder per repo_id (namespaced by repo type)
+    - inside each repo folder:
+        - refs is a list of the latest known revision => commit_hash pairs
+        - blobs contains the actual file blobs (identified by their git-sha or sha256, depending on
+          whether they're LFS files or not)
+        - snapshots contains one subfolder per commit, each "commit" contains the subset of the files
+          that have been resolved at that particular commit. Each filename is a symlink to the blob
+          at that particular commit.
+
+    [  96]  .
+    └── [ 160]  models--julien-c--EsperBERTo-small
+        ├── [ 160]  blobs
+        │   ├── [321M]  403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
+        │   ├── [ 398]  7cb18dc9bafbfcf74629a4b760af1b160957a83e
+        │   └── [1.4K]  d7edf6bd2a681fb0175f7735299831ee1b22b812
+        ├── [  96]  refs
+        │   └── [  40]  main
+        └── [ 128]  snapshots
+            ├── [ 128]  2439f60ef33a0d46d85da5001d52aeda5b00ce9f
+            │   ├── [  52]  README.md -> ../../blobs/d7edf6bd2a681fb0175f7735299831ee1b22b812
+            │   └── [  76]  pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
+            └── [ 128]  bbc77c8132af1cc5cf678da3f1ddf2de43606d48
+                ├── [  52]  README.md -> ../../blobs/7cb18dc9bafbfcf74629a4b760af1b160957a83e
+                └── [  76]  pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd
+
     Args:
         repo_id (`str`):
             A user or an organization name and a repo name separated by a `/`.
@@ -735,8 +863,6 @@ def hf_hub_download(
         force_download (`bool`, *optional*, defaults to `False`):
             Whether the file should be downloaded even if it already exists in
             the local cache.
-        force_filename (`str`, *optional*):
-            Use this name instead of a generated file name.
         proxies (`dict`, *optional*):
             Dictionary mapping protocol to the URL of the proxy passed to
             `requests.request`.
@@ -753,6 +879,10 @@ def hf_hub_download(
         local_files_only (`bool`, *optional*, defaults to `False`):
             If `True`, avoid downloading the file and return the path to the
             local cached file if it exists.
+        legacy_cache_layout (`bool`, *optional*, defaults to `False`):
+            If `True`, uses the legacy file cache layout i.e. just call [`hf_hub_url`]
+            then `cached_download`. This is deprecated as the new cache layout is
+            more powerful.
 
     Returns:
         Local path (string) of file or if networking is off, last version of
@@ -771,21 +901,271 @@ def hf_hub_download(
 
     </Tip>
     """
-    url = hf_hub_url(
-        repo_id, filename, subfolder=subfolder, repo_type=repo_type, revision=revision
+    if force_filename is not None:
+        warnings.warn(
+            "The `force_filename` parameter is deprecated as a new caching system, "
+            "which keeps the filenames as they are on the Hub, is now in place.",
+            FutureWarning,
+        )
+        legacy_cache_layout = True
+
+    if legacy_cache_layout:
+        url = hf_hub_url(
+            repo_id,
+            filename,
+            subfolder=subfolder,
+            repo_type=repo_type,
+            revision=revision,
+        )
+
+        return cached_download(
+            url,
+            library_name=library_name,
+            library_version=library_version,
+            cache_dir=cache_dir,
+            user_agent=user_agent,
+            force_download=force_download,
+            force_filename=force_filename,
+            proxies=proxies,
+            etag_timeout=etag_timeout,
+            resume_download=resume_download,
+            use_auth_token=use_auth_token,
+            local_files_only=local_files_only,
+            legacy_cache_layout=legacy_cache_layout,
+        )
+
+    if cache_dir is None:
+        cache_dir = HUGGINGFACE_HUB_CACHE
+    if revision is None:
+        revision = DEFAULT_REVISION
+    if isinstance(cache_dir, Path):
+        cache_dir = str(cache_dir)
+
+    if subfolder is not None:
+        # This is used to create a URL, and not a local path, hence the forward slash.
+        filename = f"{subfolder}/{filename}"
+
+    if repo_type is None:
+        repo_type = "model"
+    if repo_type not in REPO_TYPES:
+        raise ValueError(
+            f"Invalid repo type: {repo_type}. Accepted repo types are:"
+            f" {str(REPO_TYPES)}"
+        )
+
+    storage_folder = os.path.join(
+        cache_dir, repo_folder_name(repo_id=repo_id, repo_type=repo_type)
     )
+    os.makedirs(storage_folder, exist_ok=True)
 
-    return cached_download(
-        url,
-        library_name=library_name,
-        library_version=library_version,
-        cache_dir=cache_dir,
-        user_agent=user_agent,
-        force_download=force_download,
-        force_filename=force_filename,
-        proxies=proxies,
-        etag_timeout=etag_timeout,
-        resume_download=resume_download,
-        use_auth_token=use_auth_token,
-        local_files_only=local_files_only,
+    # cross platform transcription of filename, to be used as a local file path.
+    relative_filename = os.path.join(*filename.split("/"))
+
+    # if user provides a commit_hash and they already have the file on disk,
+    # shortcut everything.
+    if REGEX_COMMIT_HASH.match(revision):
+        pointer_path = os.path.join(
+            storage_folder, "snapshots", revision, relative_filename
+        )
+        if os.path.exists(pointer_path):
+            return pointer_path
+
+    url = hf_hub_url(repo_id, filename, repo_type=repo_type, revision=revision)
+
+    headers = {
+        "user-agent": http_user_agent(
+            library_name=library_name,
+            library_version=library_version,
+            user_agent=user_agent,
+        )
+    }
+    if isinstance(use_auth_token, str):
+        headers["authorization"] = f"Bearer {use_auth_token}"
+    elif use_auth_token:
+        token = HfFolder.get_token()
+        if token is None:
+            raise EnvironmentError(
+                "You specified use_auth_token=True, but a huggingface token was not"
+                " found."
+            )
+        headers["authorization"] = f"Bearer {token}"
+
+    url_to_download = url
+    etag = None
+    commit_hash = None
+    if not local_files_only:
+        try:
+            r = _request_with_retry(
+                method="HEAD",
+                url=url,
+                headers=headers,
+                allow_redirects=False,
+                proxies=proxies,
+                timeout=etag_timeout,
+            )
+            r.raise_for_status()
+            commit_hash = r.headers[HUGGINGFACE_HEADER_X_REPO_COMMIT]
+            if commit_hash is None:
+                raise OSError(
+                    "Distant resource does not seem to be on huggingface.co (missing"
+                    " commit header)."
+                )
+            etag = r.headers.get(HUGGINGFACE_HEADER_X_LINKED_ETAG) or r.headers.get(
+                "ETag"
+            )
+            # We favor a custom header indicating the etag of the linked resource, and
+            # we fallback to the regular etag header.
+            # If we don't have any of those, raise an error.
+            if etag is None:
+                raise OSError(
+                    "Distant resource does not have an ETag, we won't be able to"
+                    " reliably ensure reproducibility."
+                )
+            etag = _normalize_etag(etag)
+            # In case of a redirect,
+            # save an extra redirect on the request.get call,
+            # and ensure we download the exact atomic version even if it changed
+            # between the HEAD and the GET (unlikely, but hey).
+            if 300 <= r.status_code <= 399:
+                url_to_download = r.headers["Location"]
+        except (requests.exceptions.SSLError, requests.exceptions.ProxyError):
+            # Actually raise for those subclasses of ConnectionError
+            raise
+        except (
+            requests.exceptions.ConnectionError,
+            requests.exceptions.Timeout,
+            OfflineModeIsEnabled,
+        ):
+            # Otherwise, our Internet connection is down.
+            # etag is None
+            pass
+
+    # etag is None == we don't have a connection or we passed local_files_only.
+    # try to get the last downloaded one from the specified revision.
+    # If the specified revision is a commit hash, look inside "snapshots".
+    # If the specified revision is a branch or tag, look inside "refs".
+    if etag is None:
+        # In those cases, we cannot force download.
+        if force_download:
+            raise ValueError(
+                "We have no connection or you passed local_files_only, so"
+                " force_download is not an accepted option."
+            )
+        commit_hash = revision
+        if not REGEX_COMMIT_HASH.match(revision):
+            ref_path = os.path.join(storage_folder, "refs", revision)
+            with open(ref_path) as f:
+                commit_hash = f.read()
+
+        pointer_path = os.path.join(
+            storage_folder, "snapshots", commit_hash, relative_filename
+        )
+        if os.path.exists(pointer_path):
+            return pointer_path
+
+        # If we couldn't find an appropriate file on disk,
+        # raise an error.
+        # If files cannot be found and local_files_only=True,
+        # the models might've been found if local_files_only=False
+        # Notify the user about that
+        if local_files_only:
+            raise ValueError(
+                "Cannot find the requested files in the disk cache and"
+                " outgoing traffic has been disabled. To enable hf.co look-ups"
+                " and downloads online, set 'local_files_only' to False."
+            )
+        else:
+            raise ValueError(
+                "Connection error, and we cannot find the requested files in"
+                " the disk cache. Please try again or make sure your Internet"
+                " connection is on."
+            )
+
+    # From now on, etag and commit_hash are not None.
+    blob_path = os.path.join(storage_folder, "blobs", etag)
+    pointer_path = os.path.join(
+        storage_folder, "snapshots", commit_hash, relative_filename
     )
+
+    os.makedirs(os.path.dirname(blob_path), exist_ok=True)
+    os.makedirs(os.path.dirname(pointer_path), exist_ok=True)
+    # if passed revision is not identical to commit_hash
+    # then revision has to be a branch name or tag name.
+    # In that case store a ref.
+    if revision != commit_hash:
+        ref_path = os.path.join(storage_folder, "refs", revision)
+        os.makedirs(os.path.dirname(ref_path), exist_ok=True)
+        with open(ref_path, "w") as f:
+            f.write(commit_hash)
+
+    if os.path.exists(pointer_path) and not force_download:
+        return pointer_path
+
+    if os.path.exists(blob_path) and not force_download:
+        # we have the blob already, but not the pointer
+        logger.info("creating pointer to %s from %s", blob_path, pointer_path)
+        _create_relative_symlink(blob_path, pointer_path)
+        return pointer_path
+
+    # Prevent parallel downloads of the same file with a lock.
+    lock_path = blob_path + ".lock"
+
+    # Some Windows versions do not allow for paths longer than 255 characters.
+    # In this case, we must specify it is an extended path by using the "\\?\" prefix.
+    if os.name == "nt" and len(os.path.abspath(lock_path)) > 255:
+        lock_path = "\\\\?\\" + os.path.abspath(lock_path)
+
+    if os.name == "nt" and len(os.path.abspath(blob_path)) > 255:
+        blob_path = "\\\\?\\" + os.path.abspath(blob_path)
+
+    with FileLock(lock_path):
+
+        # If the download just completed while the lock was activated.
+        if os.path.exists(pointer_path) and not force_download:
+            # Even if returning early like here, the lock will be released.
+            return pointer_path
+
+        if resume_download:
+            incomplete_path = blob_path + ".incomplete"
+
+            @contextmanager
+            def _resumable_file_manager() -> "io.BufferedWriter":
+                with open(incomplete_path, "ab") as f:
+                    yield f
+
+            temp_file_manager = _resumable_file_manager
+            if os.path.exists(incomplete_path):
+                resume_size = os.stat(incomplete_path).st_size
+            else:
+                resume_size = 0
+        else:
+            temp_file_manager = partial(
+                tempfile.NamedTemporaryFile, mode="wb", dir=cache_dir, delete=False
+            )
+            resume_size = 0
+
+        # Download to temporary file, then copy to cache dir once finished.
+        # Otherwise you get corrupt cache entries if the download gets interrupted.
+        with temp_file_manager() as temp_file:
+            logger.info("downloading %s to %s", url, temp_file.name)
+
+            http_get(
+                url_to_download,
+                temp_file,
+                proxies=proxies,
+                resume_size=resume_size,
+                headers=headers,
+            )
+
+        logger.info("storing %s in cache at %s", url, blob_path)
+        os.replace(temp_file.name, blob_path)
+
+        logger.info("creating pointer to %s from %s", blob_path, pointer_path)
+        _create_relative_symlink(blob_path, pointer_path)
+
+    try:
+        os.remove(lock_path)
+    except OSError:
+        pass
+
+    return pointer_path
diff --git a/src/huggingface_hub/hf_api.py b/src/huggingface_hub/hf_api.py
index c9f9f1f9..626a23cc 100644
--- a/src/huggingface_hub/hf_api.py
+++ b/src/huggingface_hub/hf_api.py
@@ -55,7 +55,7 @@ USERNAME_PLACEHOLDER = "hf_user"
 logger = logging.get_logger(__name__)
 
 
-# TODO: remove after deprecation period is over (v0.7)
+# TODO: remove after deprecation period is over (v0.8)
 def _validate_repo_id_deprecation(repo_id, name, organization):
     """Returns (name, organization) from the input."""
     if repo_id and not name and organization:
@@ -78,7 +78,7 @@ def _validate_repo_id_deprecation(repo_id, name, organization):
     elif name or organization:
         warnings.warn(
             "`name` and `organization` input arguments are deprecated and "
-            "will be removed in v0.7. Pass `repo_id` instead.",
+            "will be removed in v0.8. Pass `repo_id` instead.",
             FutureWarning,
         )
     else:
@@ -89,7 +89,9 @@ def _validate_repo_id_deprecation(repo_id, name, organization):
     return name, organization
 
 
-def repo_type_and_id_from_hf_id(hf_id: str, hub_url: Optional[str] = None):
+def repo_type_and_id_from_hf_id(
+    hf_id: str, hub_url: Optional[str] = None
+) -> Tuple[Optional[str], Optional[str], str]:
     """
     Returns the repo type and ID from a huggingface.co URL linking to a
     repository
@@ -558,7 +560,7 @@ class HfApi:
 
         <Tip>
 
-        Warning: Deprecated, will be removed in v0.7. Please use
+        Warning: Deprecated, will be removed in v0.8. Please use
         [`HfApi.set_access_token`] instead.
 
         </Tip>
@@ -583,7 +585,7 @@ class HfApi:
         """
         warnings.warn(
             "HfApi.login: This method is deprecated in favor of `set_access_token`"
-            " and will be removed in v0.7.",
+            " and will be removed in v0.8.",
             FutureWarning,
         )
         path = f"{self.endpoint}/api/login"
@@ -651,7 +653,7 @@ class HfApi:
             token (``str``, `optional`):
                 Hugging Face token. Will default to the locally saved token if not provided.
             name (``str``, `optional`):
-                Name of the repository. This is deprecated in favor of repo_id and will be removed in v0.7.
+                Name of the repository. This is deprecated in favor of repo_id and will be removed in v0.8.
             function_name (``str``, `optional`):
                 If _validate_or_retrieve_token is called from a function, name of that function to be passed inside deprecation warning.
         Returns:
@@ -690,7 +692,7 @@ class HfApi:
 
         <Tip>
 
-        Warning: Deprecated, will be removed in v0.7. Please use
+        Warning: Deprecated, will be removed in v0.8. Please use
         [`HfApi.unset_access_token`] instead.
 
         </Tip>
@@ -702,7 +704,7 @@ class HfApi:
         """
         warnings.warn(
             "HfApi.logout: This method is deprecated in favor of `unset_access_token` "
-            "and will be removed in v0.7.",
+            "and will be removed in v0.8.",
             FutureWarning,
         )
         if token is None:
diff --git a/src/huggingface_hub/keras_mixin.py b/src/huggingface_hub/keras_mixin.py
index 2cc1565c..9d54135e 100644
--- a/src/huggingface_hub/keras_mixin.py
+++ b/src/huggingface_hub/keras_mixin.py
@@ -6,14 +6,13 @@ from shutil import copytree, rmtree
 from typing import Any, Dict, Optional, Union
 
 import yaml
-from huggingface_hub import ModelHubMixin
+from huggingface_hub import ModelHubMixin, snapshot_download
 from huggingface_hub.file_download import (
     get_tf_version,
     is_graphviz_available,
     is_pydot_available,
     is_tf_available,
 )
-from huggingface_hub.snapshot_download import snapshot_download
 
 from .constants import CONFIG_NAME
 from .hf_api import HfApi, HfFolder
@@ -27,45 +26,28 @@ if is_tf_available():
     import tensorflow as tf
 
 
-def _extract_hyperparameters_from_keras(model):
+def _create_hyperparameter_table(model):
+    """Parse hyperparameter dictionary into a markdown table."""
     if model.optimizer is not None:
-        hyperparameters = dict()
-        hyperparameters["optimizer"] = model.optimizer.get_config()
-        hyperparameters[
+        optimizer_params = model.optimizer.get_config()
+        optimizer_params[
             "training_precision"
         ] = tf.keras.mixed_precision.global_policy().name
+        table = "|"
+        for key in optimizer_params.keys():
+            table += f" {key} |"
+
+        table += "\n|"
+        for key in optimizer_params.keys():
+            table += "-" * len(key)
+            table += "|"
+
+        table += "\n|"
+        for key in optimizer_params.keys():
+            table += f"{optimizer_params[key]}|"
     else:
-        hyperparameters = None
-    return hyperparameters
-
-
-def _parse_model_history(model, save_directory):
-    lines = None
-    if model.history is not None:
-        if model.history.history != {}:
-            path = os.path.join(save_directory, "history.json")
-            with open(path, "w", encoding="utf-8") as f:
-                json.dump(model.history.history, f, indent=2, sort_keys=True)
-            lines = []
-            logs = model.history.history
-            num_epochs = len(logs["loss"])
-
-            for value in range(num_epochs):
-                epoch_dict = {
-                    log_key: log_value_list[value]
-                    for log_key, log_value_list in logs.items()
-                }
-                values = dict()
-                for k, v in epoch_dict.items():
-                    if k.startswith("val_"):
-                        k = "validation_" + k[4:]
-                    elif k != "epoch":
-                        k = "train_" + k
-                    splits = k.split("_")
-                    name = " ".join([part.capitalize() for part in splits])
-                    values[name] = v
-                lines.append(values)
-    return lines
+        table = None
+    return table
 
 
 def _plot_network(model, save_directory):
@@ -82,26 +64,6 @@ def _plot_network(model, save_directory):
     )
 
 
-def _write_metrics(model, model_card, save_directory):
-    lines = _parse_model_history(model, save_directory)
-    if lines is not None:
-        model_card += "\n| Epochs |"
-
-        for i in lines[0].keys():
-            model_card += f" {i} |"
-        model_card += "\n |"
-        for i in range(len(lines[0].keys()) + 1):
-            model_card += "--- |"  # add header of table
-        for line in lines:
-            model_card += f"\n| {lines.index(line) + 1}|"  # add values
-            for key in line:
-                value = round(line[key], 3)
-                model_card += f" {value}| "
-    else:
-        model_card += "Model history needed"
-    return model_card
-
-
 def _create_model_card(
     model,
     repo_dir: Path,
@@ -111,7 +73,7 @@ def _create_model_card(
     """
     Creates a model card for the repository.
     """
-    hyperparameters = _extract_hyperparameters_from_keras(model)
+    hyperparameters = _create_hyperparameter_table(model)
     if plot_model and is_graphviz_available() and is_pydot_available():
         _plot_network(model, repo_dir)
     readme_path = f"{repo_dir}/README.md"
@@ -125,13 +87,9 @@ def _create_model_card(
     if hyperparameters is not None:
         model_card += "\n## Training procedure\n"
         model_card += "\n### Training hyperparameters\n"
-        model_card += "\nThe following hyperparameters were used during training:\n"
-        model_card += "\n".join(
-            [f"- {name}: {value}" for name, value in hyperparameters.items()]
-        )
+        model_card += "\nThe following hyperparameters were used during training:\n\n"
+        model_card += hyperparameters
         model_card += "\n"
-    model_card += "\n ## Training Metrics\n"
-    model_card = _write_metrics(model, model_card, repo_dir)
     if plot_model and os.path.exists(f"{repo_dir}/model.png"):
         model_card += "\n ## Model Plot\n"
         model_card += "\n<details>"
@@ -223,6 +181,18 @@ def save_pretrained_keras(
         else:
             metadata["tags"] = [task_name]
 
+    if model.history is not None:
+        if model.history.history != {}:
+            path = os.path.join(save_directory, "history.json")
+            if os.path.exists(path):
+                warnings.warn(
+                    "`history.json` file already exists, it will be overwritten by the"
+                    " history of this version.",
+                    UserWarning,
+                )
+            with open(path, "w", encoding="utf-8") as f:
+                json.dump(model.history.history, f, indent=2, sort_keys=True)
+
     _create_model_card(model, save_directory, plot_model, metadata)
     tf.keras.models.save_model(
         model, save_directory, include_optimizer=include_optimizer, **model_save_kwargs
diff --git a/src/huggingface_hub/snapshot_download.py b/src/huggingface_hub/snapshot_download.py
index 5cd0a5fa..888fcc1d 100644
--- a/src/huggingface_hub/snapshot_download.py
+++ b/src/huggingface_hub/snapshot_download.py
@@ -1,252 +1,14 @@
-import os
-from fnmatch import fnmatch
-from glob import glob
-from pathlib import Path
-from typing import Dict, List, Optional, Union
+# TODO: remove in 0.11
 
-from .constants import DEFAULT_REVISION, HUGGINGFACE_HUB_CACHE
-from .file_download import cached_download, hf_hub_url
-from .hf_api import HfApi, HfFolder
-from .utils import logging
-from .utils._deprecation import _deprecate_positional_args
+import warnings
 
 
-REPO_ID_SEPARATOR = "--"
-# ^ this substring is not allowed in repo_ids on hf.co
-# and is the canonical one we use for serialization of repo ids elsewhere.
+warnings.warn(
+    "snapshot_download.py has been made private and will no longer be available from"
+    " version 0.11. Please use `from huggingface_hub import snapshot_download` to"
+    " import the only public function in this module. Other members of the file may be"
+    " changed without a deprecation notice.",
+    FutureWarning,
+)
 
-
-logger = logging.get_logger(__name__)
-
-
-@_deprecate_positional_args
-def snapshot_download(
-    repo_id: str,
-    *,
-    revision: Optional[str] = None,
-    cache_dir: Union[str, Path, None] = None,
-    library_name: Optional[str] = None,
-    library_version: Optional[str] = None,
-    user_agent: Optional[Union[Dict, str]] = None,
-    proxies: Optional[Dict] = None,
-    etag_timeout: Optional[float] = 10,
-    resume_download: Optional[bool] = False,
-    use_auth_token: Optional[Union[bool, str]] = None,
-    local_files_only: Optional[bool] = False,
-    allow_regex: Optional[Union[List[str], str]] = None,
-    ignore_regex: Optional[Union[List[str], str]] = None,
-) -> str:
-    """Download all files of a repo.
-
-    Downloads a whole snapshot of a repo's files at the specified revision. This
-    is useful when you want all files from a repo, because you don't know which
-    ones you will need a priori. All files are nested inside a folder in order
-    to keep their actual filename relative to that folder.
-
-    An alternative would be to just clone a repo but this would require that the
-    user always has git and git-lfs installed, and properly configured.
-
-    Args:
-        repo_id (`str`):
-            A user or an organization name and a repo name separated by a `/`.
-        revision (`str`, *optional*):
-            An optional Git revision id which can be a branch name, a tag, or a
-            commit hash.
-        cache_dir (`str`, `Path`, *optional*):
-            Path to the folder where cached files are stored.
-        library_name (`str`, *optional*):
-            The name of the library to which the object corresponds.
-        library_version (`str`, *optional*):
-            The version of the library.
-        user_agent (`str`, `dict`, *optional*):
-            The user-agent info in the form of a dictionary or a string.
-        proxies (`dict`, *optional*):
-            Dictionary mapping protocol to the URL of the proxy passed to
-            `requests.request`.
-        etag_timeout (`float`, *optional*, defaults to `10`):
-            When fetching ETag, how many seconds to wait for the server to send
-            data before giving up which is passed to `requests.request`.
-        resume_download (`bool`, *optional*, defaults to `False):
-            If `True`, resume a previously interrupted download.
-        use_auth_token (`str`, `bool`, *optional*):
-            A token to be used for the download.
-                - If `True`, the token is read from the HuggingFace config
-                  folder.
-                - If a string, it's used as the authentication token.
-        local_files_only (`bool`, *optional*, defaults to `False`):
-            If `True`, avoid downloading the file and return the path to the
-            local cached file if it exists.
-        allow_regex (`list of str`, `str`, *optional*):
-            If provided, only files matching this regex are downloaded.
-        ignore_regex (`list of str`, `str`, *optional*):
-            If provided, files matching this regex are not downloaded.
-
-    Returns:
-        Local folder path (string) of repo snapshot
-
-    <Tip>
-
-    Raises the following errors:
-
-    - [`EnvironmentError`](https://docs.python.org/3/library/exceptions.html#EnvironmentError)
-      if `use_auth_token=True` and the token cannot be found.
-    - [`OSError`](https://docs.python.org/3/library/exceptions.html#OSError) if
-      ETag cannot be determined.
-    - [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError)
-      if some parameter value is invalid
-
-    </Tip>
-    """
-    # Note: at some point maybe this format of storage should actually replace
-    # the flat storage structure we've used so far (initially from allennlp
-    # if I remember correctly).
-
-    if cache_dir is None:
-        cache_dir = HUGGINGFACE_HUB_CACHE
-    if revision is None:
-        revision = DEFAULT_REVISION
-    if isinstance(cache_dir, Path):
-        cache_dir = str(cache_dir)
-
-    if isinstance(use_auth_token, str):
-        token = use_auth_token
-    elif use_auth_token:
-        token = HfFolder.get_token()
-        if token is None:
-            raise EnvironmentError(
-                "You specified use_auth_token=True, but a Hugging Face token was not"
-                " found."
-            )
-    else:
-        token = None
-
-    # remove all `/` occurrences to correctly convert repo to directory name
-    repo_id_flattened = repo_id.replace("/", REPO_ID_SEPARATOR)
-
-    # if we have no internet connection we will look for the
-    # last modified folder in the cache
-    if local_files_only:
-        # possible repos have <path/to/cache_dir>/<flatten_repo_id> prefix
-        repo_folders_prefix = os.path.join(cache_dir, repo_id_flattened)
-
-        # list all possible folders that can correspond to the repo_id
-        # and are of the format <flattened-repo-id>.<revision>.<commit-sha>
-        # now let's list all cached repos that have to be included in the revision.
-        # There are 3 cases that we have to consider.
-
-        # 1) cached repos of format <repo_id>.{revision}.<any-hash>
-        # -> in this case {revision} has to be a branch
-        repo_folders_branch = glob(repo_folders_prefix + "." + revision + ".*")
-
-        # 2) cached repos of format <repo_id>.{revision}
-        # -> in this case {revision} has to be a commit sha
-        repo_folders_commit_only = glob(repo_folders_prefix + "." + revision)
-
-        # 3) cached repos of format <repo_id>.<any-branch>.{revision}
-        # -> in this case {revision} also has to be a commit sha
-        repo_folders_branch_commit = glob(repo_folders_prefix + ".*." + revision)
-
-        # combine all possible fetched cached repos
-        repo_folders = (
-            repo_folders_branch + repo_folders_commit_only + repo_folders_branch_commit
-        )
-
-        if len(repo_folders) == 0:
-            raise ValueError(
-                "Cannot find the requested files in the cached path and outgoing"
-                " traffic has been disabled. To enable model look-ups and downloads"
-                " online, set 'local_files_only' to False."
-            )
-
-        # check if repo id was previously cached from a commit sha revision
-        # and passed {revision} is not a commit sha
-        # in this case snapshotting repos locally might lead to unexpected
-        # behavior the user should be warned about
-
-        # get all folders that were cached with just a sha commit revision
-        all_repo_folders_from_sha = set(glob(repo_folders_prefix + ".*")) - set(
-            glob(repo_folders_prefix + ".*.*")
-        )
-        # 1) is there any repo id that was previously cached from a commit sha?
-        has_a_sha_revision_been_cached = len(all_repo_folders_from_sha) > 0
-        # 2) is the passed {revision} is a branch
-        is_revision_a_branch = (
-            len(repo_folders_commit_only + repo_folders_branch_commit) == 0
-        )
-
-        if has_a_sha_revision_been_cached and is_revision_a_branch:
-            # -> in this case let's warn the user
-            logger.warn(
-                f"The repo {repo_id} was previously downloaded from a commit hash"
-                " revision and has created the following cached directories"
-                f" {all_repo_folders_from_sha}. In this case, trying to load a repo"
-                f" from the branch {revision} in offline mode might lead to unexpected"
-                " behavior by not taking into account the latest commits."
-            )
-
-        # find last modified folder
-        storage_folder = max(repo_folders, key=os.path.getmtime)
-
-        # get commit sha
-        repo_id_sha = storage_folder.split(".")[-1]
-        model_files = os.listdir(storage_folder)
-    else:
-        # if we have internet connection we retrieve the correct folder name from the huggingface api
-        _api = HfApi()
-        model_info = _api.model_info(repo_id=repo_id, revision=revision, token=token)
-
-        storage_folder = os.path.join(cache_dir, repo_id_flattened + "." + revision)
-
-        # if passed revision is not identical to the commit sha
-        # then revision has to be a branch name, e.g. "main"
-        # in this case make sure that the branch name is included
-        # cached storage folder name
-        if revision != model_info.sha:
-            storage_folder += f".{model_info.sha}"
-
-        repo_id_sha = model_info.sha
-        model_files = [f.rfilename for f in model_info.siblings]
-
-    allow_regex = [allow_regex] if isinstance(allow_regex, str) else allow_regex
-    ignore_regex = [ignore_regex] if isinstance(ignore_regex, str) else ignore_regex
-
-    for model_file in model_files:
-        # if there's an allowlist, skip download if file does not match any regex
-        if allow_regex is not None and not any(
-            fnmatch(model_file, r) for r in allow_regex
-        ):
-            continue
-
-        # if there's a denylist, skip download if file does matches any regex
-        if ignore_regex is not None and any(
-            fnmatch(model_file, r) for r in ignore_regex
-        ):
-            continue
-
-        url = hf_hub_url(repo_id, filename=model_file, revision=repo_id_sha)
-        relative_filepath = os.path.join(*model_file.split("/"))
-
-        # Create potential nested dir
-        nested_dirname = os.path.dirname(
-            os.path.join(storage_folder, relative_filepath)
-        )
-        os.makedirs(nested_dirname, exist_ok=True)
-
-        path = cached_download(
-            url,
-            cache_dir=storage_folder,
-            force_filename=relative_filepath,
-            library_name=library_name,
-            library_version=library_version,
-            user_agent=user_agent,
-            proxies=proxies,
-            etag_timeout=etag_timeout,
-            resume_download=resume_download,
-            use_auth_token=use_auth_token,
-            local_files_only=local_files_only,
-        )
-
-        if os.path.exists(path + ".lock"):
-            os.remove(path + ".lock")
-
-    return storage_folder
+from ._snapshot_download import *  # noqa
diff --git a/src/huggingface_hub/utils/_deprecation.py b/src/huggingface_hub/utils/_deprecation.py
index d77fc0d3..cf4bce1c 100644
--- a/src/huggingface_hub/utils/_deprecation.py
+++ b/src/huggingface_hub/utils/_deprecation.py
@@ -3,7 +3,7 @@ from functools import wraps
 from inspect import Parameter, signature
 
 
-def _deprecate_positional_args(func=None, *, version="0.7"):
+def _deprecate_positional_args(func=None, *, version="0.8"):
     """Decorator for methods that issues warnings for positional arguments.
     Using the keyword-only argument syntax in pep 3102, arguments after the
     * will issue a warning when passed as a positional argument.
@@ -11,7 +11,7 @@ def _deprecate_positional_args(func=None, *, version="0.7"):
     Args:
         func (``Callable``, `optional`):
             Function to check arguments on.
-        version (``Callable``, `optional`, defaults to ``"0.7"``):
+        version (``Callable``, `optional`, defaults to ``"0.8"``):
             The version when positional arguments will result in error.
     """
 


Test patch:
diff --git a/.github/workflows/python-tests.yml b/.github/workflows/python-tests.yml
index d286f14b..f398918d 100644
--- a/.github/workflows/python-tests.yml
+++ b/.github/workflows/python-tests.yml
@@ -42,12 +42,15 @@ jobs:
       - run: |
           if [[ "${{ matrix.test_repository }}" == "Repository only" ]]
           then
-            pytest -Werror::FutureWarning --log-cli-level=INFO -sv ./tests/test_repository.py -k RepositoryTest
-          
+            pytest --cov -Werror::FutureWarning --log-cli-level=INFO -sv ./tests/test_repository.py -k RepositoryTest
+
           else
-            pytest -Werror::FutureWarning --log-cli-level=INFO -sv ./tests/ -k 'not RepositoryTest'
+            pytest --cov -Werror::FutureWarning --log-cli-level=INFO -sv ./tests/ -k 'not RepositoryTest'
           fi
 
+      - name: Upload coverage reports to Codecov with GitHub Action
+        uses: codecov/codecov-action@v3
+
   build_pytorch:
     runs-on: ubuntu-latest
     strategy:
@@ -67,7 +70,10 @@ jobs:
         pip install --upgrade pip
         pip install .[testing,torch]
 
-    - run: pytest -Werror::FutureWarning -sv ./tests/test_hubmixin*
+    - run: pytest --cov -Werror::FutureWarning -sv ./tests/test_hubmixin*
+
+    - name: Upload coverage reports to Codecov with GitHub Action
+      uses: codecov/codecov-action@v3
 
   build_tensorflow:
     runs-on: ubuntu-latest
@@ -89,8 +95,11 @@ jobs:
         pip install --upgrade pip
         pip install .[testing,tensorflow]
 
-    - run: pytest -Werror::FutureWarning -sv ./tests/test_keras*
-  
+    - run: pytest --cov -Werror::FutureWarning -sv ./tests/test_keras*
+
+    - name: Upload coverage reports to Codecov with GitHub Action
+      uses: codecov/codecov-action@v3
+
   build_fastai:
     runs-on: ubuntu-latest
     strategy:
@@ -110,7 +119,10 @@ jobs:
         pip install --upgrade pip
         pip install .[testing,fastai]
 
-    - run: pytest -Werror::FutureWarning -sv ./tests/test_fastai*
+    - run: pytest --cov -Werror::FutureWarning -sv ./tests/test_fastai*
+
+    - name: Upload coverage reports to Codecov with GitHub Action
+      uses: codecov/codecov-action@v3
 
   tests_lfs:
     runs-on: ubuntu-latest
@@ -132,4 +144,7 @@ jobs:
         pip install --upgrade pip
         pip install .[testing]
 
-    - run: RUN_GIT_LFS_TESTS=1 pytest -Werror::FutureWarning -sv ./tests/ -k "HfLargefilesTest"
+    - run: RUN_GIT_LFS_TESTS=1 pytest --cov -Werror::FutureWarning -sv ./tests/ -k "HfLargefilesTest"
+
+    - name: Upload coverage reports to Codecov with GitHub Action
+      uses: codecov/codecov-action@v3
diff --git a/tests/test_cache_layout.py b/tests/test_cache_layout.py
new file mode 100644
index 00000000..442fa6a5
--- /dev/null
+++ b/tests/test_cache_layout.py
@@ -0,0 +1,390 @@
+import os
+import tempfile
+import time
+import unittest
+import uuid
+from io import BytesIO
+
+from huggingface_hub import (
+    HfApi,
+    create_repo,
+    delete_repo,
+    hf_hub_download,
+    snapshot_download,
+    upload_file,
+)
+from huggingface_hub.utils import logging
+
+from .testing_constants import ENDPOINT_STAGING, TOKEN, USER
+from .testing_utils import with_production_testing
+
+
+logger = logging.get_logger(__name__)
+MODEL_IDENTIFIER = "hf-internal-testing/hfh-cache-layout"
+
+
+def repo_name(id=uuid.uuid4().hex[:6]):
+    return "repo-{0}-{1}".format(id, int(time.time() * 10e3))
+
+
+def get_file_contents(path):
+    with open(path) as f:
+        content = f.read()
+
+    return content
+
+
+@with_production_testing
+class CacheFileLayoutHfHubDownload(unittest.TestCase):
+    def test_file_downloaded_in_cache(self):
+        with tempfile.TemporaryDirectory() as cache:
+            hf_hub_download(MODEL_IDENTIFIER, "file_0.txt", cache_dir=cache)
+
+            expected_directory_name = f'models--{MODEL_IDENTIFIER.replace("/", "--")}'
+            expected_path = os.path.join(cache, expected_directory_name)
+
+            refs = os.listdir(os.path.join(expected_path, "refs"))
+            snapshots = os.listdir(os.path.join(expected_path, "snapshots"))
+
+            expected_reference = "main"
+
+            # Only reference should be `main`.
+            self.assertListEqual(refs, [expected_reference])
+
+            with open(os.path.join(expected_path, "refs", expected_reference)) as f:
+                snapshot_name = f.readline().strip()
+
+            # The `main` reference should point to the only snapshot we have downloaded
+            self.assertListEqual(snapshots, [snapshot_name])
+
+            snapshot_path = os.path.join(expected_path, "snapshots", snapshot_name)
+            snapshot_content = os.listdir(snapshot_path)
+
+            # Only a single file in the snapshot
+            self.assertEqual(len(snapshot_content), 1)
+
+            snapshot_content_path = os.path.join(snapshot_path, snapshot_content[0])
+
+            # The snapshot content should link to a blob
+            self.assertTrue(os.path.islink(snapshot_content_path))
+
+            resolved_blob_relative = os.readlink(snapshot_content_path)
+            resolved_blob_absolute = os.path.normpath(
+                os.path.join(snapshot_path, resolved_blob_relative)
+            )
+
+            with open(resolved_blob_absolute) as f:
+                blob_contents = f.read().strip()
+
+            # The contents of the file should be 'File 0'.
+            self.assertEqual(blob_contents, "File 0")
+
+    def test_file_downloaded_in_cache_with_revision(self):
+        with tempfile.TemporaryDirectory() as cache:
+            hf_hub_download(
+                MODEL_IDENTIFIER, "file_0.txt", cache_dir=cache, revision="file-2"
+            )
+
+            expected_directory_name = f'models--{MODEL_IDENTIFIER.replace("/", "--")}'
+            expected_path = os.path.join(cache, expected_directory_name)
+
+            refs = os.listdir(os.path.join(expected_path, "refs"))
+            snapshots = os.listdir(os.path.join(expected_path, "snapshots"))
+
+            expected_reference = "file-2"
+
+            # Only reference should be `file-2`.
+            self.assertListEqual(refs, [expected_reference])
+
+            with open(os.path.join(expected_path, "refs", expected_reference)) as f:
+                snapshot_name = f.read().strip()
+
+            # The `main` reference should point to the only snapshot we have downloaded
+            self.assertListEqual(snapshots, [snapshot_name])
+
+            snapshot_path = os.path.join(expected_path, "snapshots", snapshot_name)
+            snapshot_content = os.listdir(snapshot_path)
+
+            # Only a single file in the snapshot
+            self.assertEqual(len(snapshot_content), 1)
+
+            snapshot_content_path = os.path.join(snapshot_path, snapshot_content[0])
+
+            # The snapshot content should link to a blob
+            self.assertTrue(os.path.islink(snapshot_content_path))
+
+            resolved_blob_relative = os.readlink(snapshot_content_path)
+            resolved_blob_absolute = os.path.normpath(
+                os.path.join(snapshot_path, resolved_blob_relative)
+            )
+
+            with open(resolved_blob_absolute) as f:
+                blob_contents = f.readline().strip()
+
+            # The contents of the file should be 'File 0'.
+            self.assertEqual(blob_contents, "File 0")
+
+    def test_file_download_happens_once(self):
+        # Tests that a file is only downloaded once if it's not updated.
+
+        with tempfile.TemporaryDirectory() as cache:
+            path = hf_hub_download(MODEL_IDENTIFIER, "file_0.txt", cache_dir=cache)
+            creation_time_0 = os.path.getmtime(path)
+
+            time.sleep(2)
+
+            path = hf_hub_download(MODEL_IDENTIFIER, "file_0.txt", cache_dir=cache)
+            creation_time_1 = os.path.getmtime(path)
+
+            self.assertEqual(creation_time_0, creation_time_1)
+
+    def test_file_download_happens_once_intra_revision(self):
+        # Tests that a file is only downloaded once if it's not updated, even across different revisions.
+
+        with tempfile.TemporaryDirectory() as cache:
+            path = hf_hub_download(MODEL_IDENTIFIER, "file_0.txt", cache_dir=cache)
+            creation_time_0 = os.path.getmtime(path)
+
+            time.sleep(2)
+
+            path = hf_hub_download(
+                MODEL_IDENTIFIER, "file_0.txt", cache_dir=cache, revision="file-2"
+            )
+            creation_time_1 = os.path.getmtime(path)
+
+            self.assertEqual(creation_time_0, creation_time_1)
+
+    def test_multiple_refs_for_same_file(self):
+        with tempfile.TemporaryDirectory() as cache:
+            hf_hub_download(MODEL_IDENTIFIER, "file_0.txt", cache_dir=cache)
+            hf_hub_download(
+                MODEL_IDENTIFIER, "file_0.txt", cache_dir=cache, revision="file-2"
+            )
+
+            expected_directory_name = f'models--{MODEL_IDENTIFIER.replace("/", "--")}'
+            expected_path = os.path.join(cache, expected_directory_name)
+
+            refs = os.listdir(os.path.join(expected_path, "refs"))
+            refs.sort()
+
+            snapshots = os.listdir(os.path.join(expected_path, "snapshots"))
+            snapshots.sort()
+
+            # Directory should contain two revisions
+            self.assertListEqual(refs, ["file-2", "main"])
+
+            refs_contents = [
+                get_file_contents(os.path.join(expected_path, "refs", f)) for f in refs
+            ]
+            refs_contents.sort()
+
+            # snapshots directory should contain two snapshots
+            self.assertListEqual(refs_contents, snapshots)
+
+            snapshot_links = [
+                os.readlink(
+                    os.path.join(expected_path, "snapshots", filename, "file_0.txt")
+                )
+                for filename in snapshots
+            ]
+
+            # All snapshot links should point to the same file.
+            self.assertEqual(*snapshot_links)
+
+
+@with_production_testing
+class CacheFileLayoutSnapshotDownload(unittest.TestCase):
+    def test_file_downloaded_in_cache(self):
+        with tempfile.TemporaryDirectory() as cache:
+            snapshot_download(MODEL_IDENTIFIER, cache_dir=cache)
+
+            expected_directory_name = f'models--{MODEL_IDENTIFIER.replace("/", "--")}'
+            expected_path = os.path.join(cache, expected_directory_name)
+
+            refs = os.listdir(os.path.join(expected_path, "refs"))
+
+            snapshots = os.listdir(os.path.join(expected_path, "snapshots"))
+            snapshots.sort()
+
+            # Directory should contain two revisions
+            self.assertListEqual(refs, ["main"])
+
+            ref_content = get_file_contents(
+                os.path.join(expected_path, "refs", refs[0])
+            )
+
+            # snapshots directory should contain two snapshots
+            self.assertListEqual([ref_content], snapshots)
+
+            snapshot_path = os.path.join(expected_path, "snapshots", snapshots[0])
+
+            files_in_snapshot = os.listdir(snapshot_path)
+
+            snapshot_links = [
+                os.readlink(os.path.join(snapshot_path, filename))
+                for filename in files_in_snapshot
+            ]
+
+            resolved_snapshot_links = [
+                os.path.normpath(os.path.join(snapshot_path, link))
+                for link in snapshot_links
+            ]
+
+            self.assertTrue(all([os.path.isfile(l) for l in resolved_snapshot_links]))
+
+    def test_file_downloaded_in_cache_several_revisions(self):
+        with tempfile.TemporaryDirectory() as cache:
+            snapshot_download(MODEL_IDENTIFIER, cache_dir=cache, revision="file-3")
+            snapshot_download(MODEL_IDENTIFIER, cache_dir=cache, revision="file-2")
+
+            expected_directory_name = f'models--{MODEL_IDENTIFIER.replace("/", "--")}'
+            expected_path = os.path.join(cache, expected_directory_name)
+
+            refs = os.listdir(os.path.join(expected_path, "refs"))
+            refs.sort()
+
+            snapshots = os.listdir(os.path.join(expected_path, "snapshots"))
+            snapshots.sort()
+
+            # Directory should contain two revisions
+            self.assertListEqual(refs, ["file-2", "file-3"])
+
+            refs_content = [
+                get_file_contents(os.path.join(expected_path, "refs", ref))
+                for ref in refs
+            ]
+            refs_content.sort()
+
+            # snapshots directory should contain two snapshots
+            self.assertListEqual(refs_content, snapshots)
+
+            snapshots_paths = [
+                os.path.join(expected_path, "snapshots", s) for s in snapshots
+            ]
+
+            files_in_snapshots = {s: os.listdir(s) for s in snapshots_paths}
+            links_in_snapshots = {
+                k: [os.readlink(os.path.join(k, _v)) for _v in v]
+                for k, v in files_in_snapshots.items()
+            }
+
+            resolved_snapshots_links = {
+                k: [os.path.normpath(os.path.join(k, link)) for link in v]
+                for k, v in links_in_snapshots.items()
+            }
+
+            all_links = [b for a in resolved_snapshots_links.values() for b in a]
+            all_unique_links = set(all_links)
+
+            # [ 100]  .
+            # ├── [ 140]  blobs
+            # │   ├── [   7]  4475433e279a71203927cbe80125208a3b5db560
+            # │   ├── [   7]  50fcd26d6ce3000f9d5f12904e80eccdc5685dd1
+            # │   ├── [   7]  80146afc836c60e70ba67933fec439ab05b478f6
+            # │   ├── [   7]  8cf9e18f080becb674b31c21642538269fe886a4
+            # │   └── [1.1K]  ac481c8eb05e4d2496fbe076a38a7b4835dd733d
+            # ├── [  80]  refs
+            # │   ├── [  40]  file-2
+            # │   └── [  40]  file-3
+            # └── [  80]  snapshots
+            #     ├── [ 120]  5e23cb3ae7f904919a442e1b27dcddae6c6bc292
+            #     │   ├── [  52]  file_0.txt -> ../../blobs/80146afc836c60e70ba67933fec439ab05b478f6
+            #     │   ├── [  52]  file_1.txt -> ../../blobs/50fcd26d6ce3000f9d5f12904e80eccdc5685dd1
+            #     │   ├── [  52]  file_2.txt -> ../../blobs/4475433e279a71203927cbe80125208a3b5db560
+            #     │   └── [  52]  .gitattributes -> ../../blobs/ac481c8eb05e4d2496fbe076a38a7b4835dd733d
+            #     └── [ 120]  78aa2ebdb60bba086496a8792ba506e58e587b4c
+            #         ├── [  52]  file_0.txt -> ../../blobs/80146afc836c60e70ba67933fec439ab05b478f6
+            #         ├── [  52]  file_1.txt -> ../../blobs/50fcd26d6ce3000f9d5f12904e80eccdc5685dd1
+            #         ├── [  52]  file_3.txt -> ../../blobs/8cf9e18f080becb674b31c21642538269fe886a4
+            #         └── [  52]  .gitattributes -> ../../blobs/ac481c8eb05e4d2496fbe076a38a7b4835dd733d
+
+            # Across the two revisions, there should be 8 total links
+            self.assertEqual(len(all_links), 8)
+
+            # Across the two revisions, there should only be 5 unique files.
+            self.assertEqual(len(all_unique_links), 5)
+
+
+class ReferenceUpdates(unittest.TestCase):
+    _api = HfApi(endpoint=ENDPOINT_STAGING)
+
+    @classmethod
+    def setUpClass(cls):
+        """
+        Share this valid token in all tests below.
+        """
+        cls._token = TOKEN
+        cls._api.set_access_token(TOKEN)
+
+    def test_update_reference(self):
+        repo_id = f"{USER}/{repo_name()}"
+        create_repo(repo_id, token=self._token, exist_ok=True)
+
+        try:
+            upload_file(
+                path_or_fileobj=BytesIO(b"Some string"),
+                path_in_repo="file.txt",
+                repo_id=repo_id,
+                token=self._token,
+            )
+
+            with tempfile.TemporaryDirectory() as cache:
+                hf_hub_download(repo_id, "file.txt", cache_dir=cache)
+
+                expected_directory_name = f'models--{repo_id.replace("/", "--")}'
+                expected_path = os.path.join(cache, expected_directory_name)
+
+                refs = os.listdir(os.path.join(expected_path, "refs"))
+
+                # Directory should contain two revisions
+                self.assertListEqual(refs, ["main"])
+
+                initial_ref_content = get_file_contents(
+                    os.path.join(expected_path, "refs", refs[0])
+                )
+
+                # Upload a new file on the same branch
+                upload_file(
+                    path_or_fileobj=BytesIO(b"Some new string"),
+                    path_in_repo="file.txt",
+                    repo_id=repo_id,
+                    token=self._token,
+                )
+
+                hf_hub_download(repo_id, "file.txt", cache_dir=cache)
+
+                final_ref_content = get_file_contents(
+                    os.path.join(expected_path, "refs", refs[0])
+                )
+
+                # The `main` reference should point to two different, but existing snapshots which contain
+                # a 'file.txt'
+                self.assertNotEqual(initial_ref_content, final_ref_content)
+                self.assertTrue(
+                    os.path.isdir(
+                        os.path.join(expected_path, "snapshots", initial_ref_content)
+                    )
+                )
+                self.assertTrue(
+                    os.path.isfile(
+                        os.path.join(
+                            expected_path, "snapshots", initial_ref_content, "file.txt"
+                        )
+                    )
+                )
+                self.assertTrue(
+                    os.path.isdir(
+                        os.path.join(expected_path, "snapshots", final_ref_content)
+                    )
+                )
+                self.assertTrue(
+                    os.path.isfile(
+                        os.path.join(
+                            expected_path, "snapshots", final_ref_content, "file.txt"
+                        )
+                    )
+                )
+        except Exception:
+            raise
+        finally:
+            delete_repo(repo_id, token=self._token)
diff --git a/tests/test_file_download.py b/tests/test_file_download.py
index 05499e38..071478c0 100644
--- a/tests/test_file_download.py
+++ b/tests/test_file_download.py
@@ -57,7 +57,7 @@ class CachedDownloadTests(unittest.TestCase):
     def test_bogus_url(self):
         url = "https://bogus"
         with self.assertRaisesRegex(ValueError, "Connection error"):
-            _ = cached_download(url)
+            _ = cached_download(url, legacy_cache_layout=True)
 
     def test_no_connection(self):
         invalid_url = hf_hub_url(
@@ -68,20 +68,26 @@ class CachedDownloadTests(unittest.TestCase):
         valid_url = hf_hub_url(
             DUMMY_MODEL_ID, filename=CONFIG_NAME, revision=REVISION_ID_DEFAULT
         )
-        self.assertIsNotNone(cached_download(valid_url, force_download=True))
+        self.assertIsNotNone(
+            cached_download(valid_url, force_download=True, legacy_cache_layout=True)
+        )
         for offline_mode in OfflineSimulationMode:
             with offline(mode=offline_mode):
                 with self.assertRaisesRegex(ValueError, "Connection error"):
-                    _ = cached_download(invalid_url)
+                    _ = cached_download(invalid_url, legacy_cache_layout=True)
                 with self.assertRaisesRegex(ValueError, "Connection error"):
-                    _ = cached_download(valid_url, force_download=True)
-                self.assertIsNotNone(cached_download(valid_url))
+                    _ = cached_download(
+                        valid_url, force_download=True, legacy_cache_layout=True
+                    )
+                self.assertIsNotNone(
+                    cached_download(valid_url, legacy_cache_layout=True)
+                )
 
     def test_file_not_found(self):
         # Valid revision (None) but missing file.
         url = hf_hub_url(DUMMY_MODEL_ID, filename="missing.bin")
         with self.assertRaisesRegex(requests.exceptions.HTTPError, "404 Client Error"):
-            _ = cached_download(url)
+            _ = cached_download(url, legacy_cache_layout=True)
 
     def test_revision_not_found(self):
         # Valid file but missing revision
@@ -91,14 +97,14 @@ class CachedDownloadTests(unittest.TestCase):
             revision=DUMMY_MODEL_ID_REVISION_INVALID,
         )
         with self.assertRaisesRegex(requests.exceptions.HTTPError, "404 Client Error"):
-            _ = cached_download(url)
+            _ = cached_download(url, legacy_cache_layout=True)
 
     def test_standard_object(self):
         url = hf_hub_url(
             DUMMY_MODEL_ID, filename=CONFIG_NAME, revision=REVISION_ID_DEFAULT
         )
-        filepath = cached_download(url, force_download=True)
-        metadata = filename_to_url(filepath)
+        filepath = cached_download(url, force_download=True, legacy_cache_layout=True)
+        metadata = filename_to_url(filepath, legacy_cache_layout=True)
         self.assertEqual(metadata, (url, f'"{DUMMY_MODEL_ID_PINNED_SHA1}"'))
 
     def test_standard_object_rev(self):
@@ -108,8 +114,8 @@ class CachedDownloadTests(unittest.TestCase):
             filename=CONFIG_NAME,
             revision=DUMMY_MODEL_ID_REVISION_ONE_SPECIFIC_COMMIT,
         )
-        filepath = cached_download(url, force_download=True)
-        metadata = filename_to_url(filepath)
+        filepath = cached_download(url, force_download=True, legacy_cache_layout=True)
+        metadata = filename_to_url(filepath, legacy_cache_layout=True)
         self.assertNotEqual(metadata[1], f'"{DUMMY_MODEL_ID_PINNED_SHA1}"')
         # Caution: check that the etag is *not* equal to the one from `test_standard_object`
 
@@ -117,8 +123,8 @@ class CachedDownloadTests(unittest.TestCase):
         url = hf_hub_url(
             DUMMY_MODEL_ID, filename=PYTORCH_WEIGHTS_NAME, revision=REVISION_ID_DEFAULT
         )
-        filepath = cached_download(url, force_download=True)
-        metadata = filename_to_url(filepath)
+        filepath = cached_download(url, force_download=True, legacy_cache_layout=True)
+        metadata = filename_to_url(filepath, legacy_cache_layout=True)
         self.assertEqual(metadata, (url, f'"{DUMMY_MODEL_ID_PINNED_SHA256}"'))
 
     def test_dataset_standard_object_rev(self):
@@ -136,8 +142,8 @@ class CachedDownloadTests(unittest.TestCase):
         )
         self.assertEqual(url, url2)
         # now let's download
-        filepath = cached_download(url, force_download=True)
-        metadata = filename_to_url(filepath)
+        filepath = cached_download(url, force_download=True, legacy_cache_layout=True)
+        metadata = filename_to_url(filepath, legacy_cache_layout=True)
         self.assertNotEqual(metadata[1], f'"{DUMMY_MODEL_ID_PINNED_SHA1}"')
 
     def test_dataset_lfs_object(self):
@@ -147,19 +153,20 @@ class CachedDownloadTests(unittest.TestCase):
             repo_type=REPO_TYPE_DATASET,
             revision=DATASET_REVISION_ID_ONE_SPECIFIC_COMMIT,
         )
-        filepath = cached_download(url, force_download=True)
-        metadata = filename_to_url(filepath)
+        filepath = cached_download(url, force_download=True, legacy_cache_layout=True)
+        metadata = filename_to_url(filepath, legacy_cache_layout=True)
         self.assertEqual(
             metadata,
             (url, '"95aa6a52d5d6a735563366753ca50492a658031da74f301ac5238b03966972c9"'),
         )
 
-    def test_hf_hub_download(self):
+    def test_hf_hub_download_legacy(self):
         filepath = hf_hub_download(
             DUMMY_MODEL_ID,
             filename=CONFIG_NAME,
             revision=REVISION_ID_DEFAULT,
             force_download=True,
+            legacy_cache_layout=True,
         )
-        metadata = filename_to_url(filepath)
+        metadata = filename_to_url(filepath, legacy_cache_layout=True)
         self.assertEqual(metadata[1], f'"{DUMMY_MODEL_ID_PINNED_SHA1}"')
diff --git a/tests/test_hf_api.py b/tests/test_hf_api.py
index 4f5ecfb2..c64cf6c4 100644
--- a/tests/test_hf_api.py
+++ b/tests/test_hf_api.py
@@ -161,7 +161,7 @@ class HfApiLoginTest(HfApiCommonTest):
         with pytest.warns(
             FutureWarning,
             match=r"HfApi.login: This method is deprecated in favor of "
-            r"`set_access_token` and will be removed in v0.7.",
+            r"`set_access_token` and will be removed in v0.8.",
         ):
             self._api.login(username=USER, password=PASS)
 
@@ -169,7 +169,7 @@ class HfApiLoginTest(HfApiCommonTest):
         with pytest.warns(
             FutureWarning,
             match=r"HfApi.logout: This method is deprecated in favor of "
-            r"`unset_access_token` and will be removed in v0.7.",
+            r"`unset_access_token` and will be removed in v0.8.",
         ):
             try:
                 self._api.logout()
@@ -511,7 +511,9 @@ class HfApiUploadFileTest(HfApiCommonTestWithLogin):
                 user=USER,
                 repo=REPO_NAME,
             )
-            filepath = cached_download(url, force_download=True)
+            filepath = cached_download(
+                url, force_download=True, legacy_cache_layout=True
+            )
             with open(filepath) as downloaded_file:
                 content = downloaded_file.read()
             self.assertEqual(content, self.tmp_file_content)
@@ -538,7 +540,9 @@ class HfApiUploadFileTest(HfApiCommonTestWithLogin):
                 user=USER,
                 repo=REPO_NAME,
             )
-            filepath = cached_download(url, force_download=True)
+            filepath = cached_download(
+                url, force_download=True, legacy_cache_layout=True
+            )
             with open(filepath) as downloaded_file:
                 content = downloaded_file.read()
             self.assertEqual(content, self.tmp_file_content)
@@ -565,7 +569,9 @@ class HfApiUploadFileTest(HfApiCommonTestWithLogin):
                 user=USER,
                 repo=REPO_NAME,
             )
-            filepath = cached_download(url, force_download=True)
+            filepath = cached_download(
+                url, force_download=True, legacy_cache_layout=True
+            )
             with open(filepath) as downloaded_file:
                 content = downloaded_file.read()
             self.assertEqual(content, filecontent.getvalue().decode())
@@ -648,7 +654,9 @@ class HfApiUploadFileTest(HfApiCommonTestWithLogin):
                 user=USER,
                 repo=REPO_NAME,
             )
-            filepath = cached_download(url, force_download=True)
+            filepath = cached_download(
+                url, force_download=True, legacy_cache_layout=True
+            )
             with open(filepath) as downloaded_file:
                 content = downloaded_file.read()
             self.assertEqual(content, self.tmp_file_content)
diff --git a/tests/test_keras_integration.py b/tests/test_keras_integration.py
index 6e816fc7..8a8634fc 100644
--- a/tests/test_keras_integration.py
+++ b/tests/test_keras_integration.py
@@ -1,3 +1,4 @@
+import json
 import os
 import shutil
 import tempfile
@@ -226,8 +227,49 @@ class HubKerasSequentialTest(HubMixingTestKeras):
         self.assertIn("model.png", files)
         self.assertIn("README.md", files)
         self.assertIn("history.json", files)
+        with open(f"{WORKING_REPO_DIR}/{REPO_NAME}/history.json") as f:
+            history = json.load(f)
+
+        self.assertEqual(history, model.history.history)
         self.assertEqual(len(files), 7)
 
+    def test_save_model_card_history_removal(self):
+        REPO_NAME = repo_name("save")
+        model = self.model_init()
+        model = self.model_fit(model)
+        with tempfile.TemporaryDirectory() as tmpdirname:
+            os.makedirs(f"{tmpdirname}/{WORKING_REPO_DIR}/{REPO_NAME}")
+            with open(
+                f"{tmpdirname}/{WORKING_REPO_DIR}/{REPO_NAME}/history.json", "w+"
+            ) as fp:
+                fp.write("Keras FTW")
+
+            with pytest.warns(
+                UserWarning, match="`history.json` file already exists, *"
+            ):
+                save_pretrained_keras(
+                    model,
+                    f"{tmpdirname}/{WORKING_REPO_DIR}/{REPO_NAME}",
+                )
+                # assert that it's not the same as old history file and it's overridden
+                with open(
+                    f"{tmpdirname}/{WORKING_REPO_DIR}/{REPO_NAME}/history.json", "r"
+                ) as f:
+                    history_content = f.read()
+                    self.assertNotEqual("Keras FTW", history_content)
+
+            # Check the history is saved as a json in the repository.
+            files = os.listdir(f"{tmpdirname}/{WORKING_REPO_DIR}/{REPO_NAME}")
+            self.assertIn("history.json", files)
+
+            # Check that there is no "Training Metrics" section in the model card.
+            # This was done in an older version.
+            with open(
+                f"{tmpdirname}/{WORKING_REPO_DIR}/{REPO_NAME}/README.md", "r"
+            ) as file:
+                data = file.read()
+            self.assertNotIn(data, "Training Metrics")
+
     def test_save_pretrained_optimizer_state(self):
         REPO_NAME = repo_name("save")
         model = self.model_init()
diff --git a/tests/test_snapshot_download.py b/tests/test_snapshot_download.py
index c2f193ba..8e856479 100644
--- a/tests/test_snapshot_download.py
+++ b/tests/test_snapshot_download.py
@@ -4,10 +4,11 @@ import tempfile
 import time
 import unittest
 
+import pytest
+
 import requests
-from huggingface_hub import HfApi, Repository
+from huggingface_hub import HfApi, Repository, snapshot_download
 from huggingface_hub.hf_api import HfFolder
-from huggingface_hub.snapshot_download import snapshot_download
 from huggingface_hub.utils import logging
 
 from .testing_constants import ENDPOINT_STAGING, TOKEN, USER
@@ -276,14 +277,6 @@ class SnapshotDownloadTests(unittest.TestCase):
                 cache_dir=tmpdirname,
             )
 
-            # now load from cache and make sure warning to be raised
-            with self.assertWarns(Warning):
-                snapshot_download(
-                    f"{USER}/{REPO_NAME}",
-                    cache_dir=tmpdirname,
-                    local_files_only=True,
-                )
-
         # cache multiple commits and make sure correct commit is taken
         with tempfile.TemporaryDirectory() as tmpdirname:
             # first download folder to cache it
@@ -358,3 +351,10 @@ class SnapshotDownloadTests(unittest.TestCase):
 
     def test_download_model_with_ignore_regex_list(self):
         self.check_download_model_with_regex(["*.git*", "*.pt"], allow=False)
+
+
+def test_snapshot_download_import():
+    with pytest.warns(FutureWarning, match="has been made private"):
+        from huggingface_hub.snapshot_download import snapshot_download as x  # noqa
+
+    assert x is snapshot_download