Repository: huggingface/huggingface_hub
Pull request #2167 (2024-03-28 13:18:15)

Fix URL when uploading to proxy
Follow-up PR after https://github.com/huggingface/huggingface_hub/pull/2155 to definitively fix https://github.com/huggingface/huggingface_hub/issues/2152. Thanks @livegoplayer for reporting the remaining issue in https://github.com/huggingface/huggingface_hub/issues/2152#issuecomment-2024984796. I made a mistake when trying the fix and my machine was still able to reach hf.co. This PR should settle the problem for good.

@livegoplayer would you mind trying again on your side? You can test it by installing `huggingface_hub` from this branch:

```
pip install git+https://github.com/huggingface/huggingface_hub@fix-url-for-good-when-using-procy
```

You testing scripts from https://github.com/huggingface/huggingface_hub/issues/2152#issuecomment-2024984796 looks good so normally upload should correctly work. Please let me know how it goes!

cc @padeoe vor visibility

HuggingFaceDocBuilderDev: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/huggingface_hub/pr_2167). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
livegoplayer: 我又测试了一遍
![image](https://github.com/huggingface/huggingface_hub/assets/31812698/89737a0e-b00b-4992-ac2c-ccc4f4f41926)
![image](https://github.com/huggingface/huggingface_hub/assets/31812698/86c9d5b0-3705-4a0f-ba07-f6ae23b9f9c3)

![image](https://github.com/huggingface/huggingface_hub/assets/31812698/53d57be2-b154-495e-8e93-846340c5ecef)

livegoplayer: 这是我的脚本地址
https://github.com/livegoplayer/scripts/blob/main/huggingface_io/test.py
livegoplayer: > The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/huggingface_hub/pr_2167). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Wauplin: Hi @livegoplayer I just pushed a (last?) fix: https://github.com/huggingface/huggingface_hub/pull/2167/commits/64abc0e735dcc315fbde1af6650e616f41617bdd. This was an oversight from my side. Just for the record:
- in https://github.com/huggingface/huggingface_hub/pull/2155 I fixed the return value when creating a repo
- which fixed uploading a regular file to the Hub
- in https://github.com/huggingface/huggingface_hub/pull/2167, I initially fixed the upload of a small LFS file
- finally, in https://github.com/huggingface/huggingface_hub/pull/2167/commits/64abc0e735dcc315fbde1af6650e616f41617bdd I fixed the upload of multipart LFS files

I previously forgot to distinguish the "single part" and "multi part" LFS workflows, that's why you got an error while I didn't. I only tried to upload a 1kb LFS file which has 1 step less than your 150MB file (since the 150Mb is split in chunks of 16MB). Anyway, I hope all issues are now fix. @livegoplayer could you retry and confirm? Thank you in advance :pray:  
livegoplayer: > 你好@livegoplayer我刚刚推送了一个（最后一个？）修复：[64abc0e](https://github.com/huggingface/huggingface_hub/commit/64abc0e735dcc315fbde1af6650e616f41617bdd)。这是我的疏忽。仅供记录：
> 
> * 在[修复 HF_ENDPOINT 未正确处理 #2155](https://github.com/huggingface/huggingface_hub/pull/2155)中，我修复了创建存储库时的返回值
> * 修复了将常规文件上传到集线器的问题
> * 在[上传到代理时修复 URL  #2167](https://github.com/huggingface/huggingface_hub/pull/2167)中，我最初修复了一个小 LFS 文件的上传
> * 最后，在[64abc0e](https://github.com/huggingface/huggingface_hub/commit/64abc0e735dcc315fbde1af6650e616f41617bdd)中我修复了分段 LFS 文件的上传
> 
> 我之前忘记区分“单部分”和“多部分”LFS 工作流程，这就是为什么你收到错误而我没有。我只尝试上传一个 1kb LFS 文件，该文件比您的 150MB 文件少 1 步（因为 150Mb 被分成 16MB 的块）。无论如何，我希望所有问题现在都得到解决。@livegoplayer您可以重试并确认吗？提前谢谢🙏

我测试成功了，我上传了一个很多g的文件，并且成功了
Wauplin: Yay! :tada: 
Thanks for confirming! We should be good to go then :) 

Patch:
diff --git a/src/huggingface_hub/_commit_api.py b/src/huggingface_hub/_commit_api.py
index c7333e0e..b807106b 100644
--- a/src/huggingface_hub/_commit_api.py
+++ b/src/huggingface_hub/_commit_api.py
@@ -399,7 +399,7 @@ def _upload_lfs_files(
     def _wrapped_lfs_upload(batch_action) -> None:
         try:
             operation = oid2addop[batch_action["oid"]]
-            lfs_upload(operation=operation, lfs_batch_action=batch_action, headers=headers)
+            lfs_upload(operation=operation, lfs_batch_action=batch_action, headers=headers, endpoint=endpoint)
         except Exception as exc:
             raise RuntimeError(f"Error while uploading '{operation.path_in_repo}' to the Hub.") from exc
 
diff --git a/src/huggingface_hub/hf_api.py b/src/huggingface_hub/hf_api.py
index dbc5bb25..1d9c8b9c 100644
--- a/src/huggingface_hub/hf_api.py
+++ b/src/huggingface_hub/hf_api.py
@@ -82,8 +82,6 @@ from .community import (
     deserialize_event,
 )
 from .constants import (
-    _HF_DEFAULT_ENDPOINT,
-    _HF_DEFAULT_STAGING_ENDPOINT,
     DEFAULT_ETAG_TIMEOUT,
     DEFAULT_REQUEST_TIMEOUT,
     DEFAULT_REVISION,
@@ -123,6 +121,7 @@ from .utils import (  # noqa: F401 # imported for backward compatibility
     build_hf_headers,
     experimental,
     filter_repo_objects,
+    fix_hf_endpoint_in_url,
     get_session,
     hf_raise_for_status,
     logging,
@@ -431,14 +430,7 @@ class RepoUrl(str):
     """
 
     def __new__(cls, url: Any, endpoint: Optional[str] = None):
-        # check if a proxy has been set => if yes, update the returned URL to use the proxy
-        if ENDPOINT not in (_HF_DEFAULT_ENDPOINT, _HF_DEFAULT_STAGING_ENDPOINT):
-            url = url.replace(_HF_DEFAULT_ENDPOINT, ENDPOINT)
-            url = url.replace(_HF_DEFAULT_STAGING_ENDPOINT, ENDPOINT)
-        if endpoint not in (None, _HF_DEFAULT_ENDPOINT, _HF_DEFAULT_STAGING_ENDPOINT):
-            url = url.replace(_HF_DEFAULT_ENDPOINT, endpoint)
-            url = url.replace(_HF_DEFAULT_STAGING_ENDPOINT, endpoint)
-
+        url = fix_hf_endpoint_in_url(url, endpoint=endpoint)
         return super(RepoUrl, cls).__new__(cls, url)
 
     def __init__(self, url: Any, endpoint: Optional[str] = None) -> None:
diff --git a/src/huggingface_hub/lfs.py b/src/huggingface_hub/lfs.py
index a6be3eb1..d6fb9e21 100644
--- a/src/huggingface_hub/lfs.py
+++ b/src/huggingface_hub/lfs.py
@@ -31,6 +31,7 @@ from huggingface_hub.constants import ENDPOINT, HF_HUB_ENABLE_HF_TRANSFER, REPO_
 
 from .utils import (
     build_hf_headers,
+    fix_hf_endpoint_in_url,
     get_session,
     hf_raise_for_status,
     http_backoff,
@@ -193,6 +194,7 @@ def lfs_upload(
     lfs_batch_action: Dict,
     token: Optional[str] = None,
     headers: Optional[Dict[str, str]] = None,
+    endpoint: Optional[str] = None,
 ) -> None:
     """
     Handles uploading a given object to the Hub with the LFS protocol.
@@ -230,6 +232,7 @@ def lfs_upload(
     # 2. Upload file (either single part or multi-part)
     header = upload_action.get("header", {})
     chunk_size = header.get("chunk_size")
+    upload_url = fix_hf_endpoint_in_url(upload_action["href"], endpoint=endpoint)
     if chunk_size is not None:
         try:
             chunk_size = int(chunk_size)
@@ -237,15 +240,16 @@ def lfs_upload(
             raise ValueError(
                 f"Malformed response from LFS batch endpoint: `chunk_size` should be an integer. Got '{chunk_size}'."
             )
-        _upload_multi_part(operation=operation, header=header, chunk_size=chunk_size, upload_url=upload_action["href"])
+        _upload_multi_part(operation=operation, header=header, chunk_size=chunk_size, upload_url=upload_url)
     else:
-        _upload_single_part(operation=operation, upload_url=upload_action["href"])
+        _upload_single_part(operation=operation, upload_url=upload_url)
 
     # 3. Verify upload went well
     if verify_action is not None:
         _validate_lfs_action(verify_action)
+        verify_url = fix_hf_endpoint_in_url(verify_action["href"], endpoint)
         verify_resp = get_session().post(
-            verify_action["href"],
+            verify_url,
             headers=build_hf_headers(token=token, headers=headers),
             json={"oid": operation.upload_info.sha256.hex(), "size": operation.upload_info.size},
         )
diff --git a/src/huggingface_hub/utils/__init__.py b/src/huggingface_hub/utils/__init__.py
index 0a3a0a32..97c236ec 100644
--- a/src/huggingface_hub/utils/__init__.py
+++ b/src/huggingface_hub/utils/__init__.py
@@ -47,7 +47,14 @@ from ._fixes import SoftTemporaryDirectory, WeakFileLock, yaml_dump
 from ._git_credential import list_credential_helpers, set_git_credential, unset_git_credential
 from ._headers import LocalTokenNotFoundError, build_hf_headers, get_token_to_send
 from ._hf_folder import HfFolder
-from ._http import OfflineModeIsEnabled, configure_http_backend, get_session, http_backoff, reset_sessions
+from ._http import (
+    OfflineModeIsEnabled,
+    configure_http_backend,
+    fix_hf_endpoint_in_url,
+    get_session,
+    http_backoff,
+    reset_sessions,
+)
 from ._pagination import paginate
 from ._paths import IGNORE_GIT_FOLDER_PATTERNS, filter_repo_objects
 from ._runtime import (
diff --git a/src/huggingface_hub/utils/_http.py b/src/huggingface_hub/utils/_http.py
index 591bd837..081d84a4 100644
--- a/src/huggingface_hub/utils/_http.py
+++ b/src/huggingface_hub/utils/_http.py
@@ -21,7 +21,7 @@ import time
 import uuid
 from functools import lru_cache
 from http import HTTPStatus
-from typing import Callable, Tuple, Type, Union
+from typing import Callable, Optional, Tuple, Type, Union
 
 import requests
 from requests import Response
@@ -306,3 +306,16 @@ def http_backoff(
 
         # Update sleep time for next retry
         sleep_time = min(max_wait_time, sleep_time * 2)  # Exponential backoff
+
+
+def fix_hf_endpoint_in_url(url: str, endpoint: Optional[str]) -> str:
+    """Replace the default endpoint in a URL by a custom one.
+
+    This is useful when using a proxy and the Hugging Face Hub returns a URL with the default endpoint.
+    """
+    endpoint = endpoint or constants.ENDPOINT
+    # check if a proxy has been set => if yes, update the returned URL to use the proxy
+    if endpoint not in (None, constants._HF_DEFAULT_ENDPOINT, constants._HF_DEFAULT_STAGING_ENDPOINT):
+        url = url.replace(constants._HF_DEFAULT_ENDPOINT, endpoint)
+        url = url.replace(constants._HF_DEFAULT_STAGING_ENDPOINT, endpoint)
+    return url


Test patch:
diff --git a/tests/test_utils_http.py b/tests/test_utils_http.py
index f82a7a17..be5ceaea 100644
--- a/tests/test_utils_http.py
+++ b/tests/test_utils_http.py
@@ -3,10 +3,11 @@ import threading
 import time
 import unittest
 from multiprocessing import Process, Queue
-from typing import Generator
+from typing import Generator, Optional
 from unittest.mock import Mock, call, patch
 from uuid import UUID
 
+import pytest
 import requests
 from requests import ConnectTimeout, HTTPError
 
@@ -14,6 +15,7 @@ from huggingface_hub.constants import ENDPOINT
 from huggingface_hub.utils._http import (
     OfflineModeIsEnabled,
     configure_http_backend,
+    fix_hf_endpoint_in_url,
     get_session,
     http_backoff,
     reset_sessions,
@@ -292,3 +294,20 @@ def _is_uuid(string: str) -> bool:
     except ValueError:
         return False
     return str(uuid_obj) == string
+
+
+@pytest.mark.parametrize(
+    ("base_url", "endpoint", "expected_url"),
+    [
+        # Staging url => unchanged
+        ("https://hub-ci.huggingface.co/resolve/...", None, "https://hub-ci.huggingface.co/resolve/..."),
+        # Prod url => unchanged
+        ("https://huggingface.co/resolve/...", None, "https://huggingface.co/resolve/..."),
+        # Custom endpoint + staging url => fixed
+        ("https://hub-ci.huggingface.co/api/models", "https://mirror.co", "https://mirror.co/api/models"),
+        # Custom endpoint + prod url => fixed
+        ("https://huggingface.co/api/models", "https://mirror.co", "https://mirror.co/api/models"),
+    ],
+)
+def test_fix_hf_endpoint_in_url(base_url: str, endpoint: Optional[str], expected_url: str) -> None:
+    assert fix_hf_endpoint_in_url(base_url, endpoint) == expected_url