Repository: huggingface/huggingface_hub
Pull request #1506 (2023-06-12 16:35:42)

Encourage to overwrite CommitScheduler.push_to_hub
Demo Space for working implementation examples: https://huggingface.co/spaces/Wauplin/space_to_dataset_saver

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/huggingface/huggingface_hub/pull/1506?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface) Report
Patch coverage: **`100.00`**% and project coverage change: **`+3.34`** :tada:
> Comparison is base [(`b74a44e`)](https://app.codecov.io/gh/huggingface/huggingface_hub/commit/b74a44e265dd8ef48595c04c7f27f45b523ed409?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface) 79.23% compared to head [(`1d0026f`)](https://app.codecov.io/gh/huggingface/huggingface_hub/pull/1506?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface) 82.58%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #1506      +/-   ##
==========================================
+ Coverage   79.23%   82.58%   +3.34%     
==========================================
  Files          56       56              
  Lines        6049     6051       +2     
==========================================
+ Hits         4793     4997     +204     
+ Misses       1256     1054     -202     
```


| [Impacted Files](https://app.codecov.io/gh/huggingface/huggingface_hub/pull/1506?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface) | Coverage Δ | |
|---|---|---|
| [src/huggingface\_hub/\_commit\_scheduler.py](https://app.codecov.io/gh/huggingface/huggingface_hub/pull/1506?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface#diff-c3JjL2h1Z2dpbmdmYWNlX2h1Yi9fY29tbWl0X3NjaGVkdWxlci5weQ==) | `96.46% <100.00%> (+0.06%)` | :arrow_up: |

... and [5 files with indirect coverage changes](https://app.codecov.io/gh/huggingface/huggingface_hub/pull/1506/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/huggingface/huggingface_hub/pull/1506?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface).

HuggingFaceDocBuilderDev: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/huggingface_hub/pr_1506). All of your documentation changes will be reflected on that endpoint.

Patch:
diff --git a/docs/source/guides/inference.mdx b/docs/source/guides/inference.mdx
index 5b7c9aea..5cb59e30 100644
--- a/docs/source/guides/inference.mdx
+++ b/docs/source/guides/inference.mdx
@@ -198,7 +198,7 @@ You can catch it and handle it in your code:
 Some tasks require binary inputs, for example, when dealing with images or audio files. In this case, [`InferenceClient`]
 tries to be as permissive as possible and accept different types:
 - raw `bytes`
-- a file-like object, opened as binary (`with open("audio.wav", "rb") as f: ...`)
+- a file-like object, opened as binary (`with open("audio.flac", "rb") as f: ...`)
 - a path (`str` or `Path`) pointing to a local file
 - a URL (`str`) pointing to a remote file (e.g. `https://...`). In this case, the file will be downloaded locally before
 sending it to the Inference API.
diff --git a/docs/source/guides/upload.mdx b/docs/source/guides/upload.mdx
index 79cdf248..a20c85d7 100644
--- a/docs/source/guides/upload.mdx
+++ b/docs/source/guides/upload.mdx
@@ -33,13 +33,7 @@ Whenever you want to upload files to the Hub, you need to log in to your Hugging
   best practice to load the token from a secure vault instead of saving it in plain in
   your Colaboratory notebook.
 
-## Push files without Git
-
-If you don't have Git installed on your system, use [`create_commit`] to push your files to the Hub. [`create_commit`] uses the HTTP protocol to upload files to the Hub.
-
-However, [`create_commit`] is a low-level API for working at a commit level. The [`upload_file`] and [`upload_folder`] functions are higher-level APIs that use [`create_commit`] under the hood and are generally more convenient. We recommend trying these functions first if you don't need to work at a lower level.
-
-### Upload a file
+## Upload a file
 
 Once you've created a repository with [`create_repo`], you can upload a file to your repository using [`upload_file`].
 
@@ -56,7 +50,7 @@ Specify the path of the file to upload, where you want to upload the file to in
 ... )
 ```
 
-### Upload a folder
+## Upload a folder
 
 Use the [`upload_folder`] function to upload a local folder to an existing repository. Specify the path of the local folder
 to upload, where you want to upload the folder to in the repository, and the name of the repository you want to add the
@@ -108,7 +102,7 @@ but before that, all previous logs on the repo on deleted. All of this in a sing
 ... )
 ```
 
-### Non-blocking upload
+## Non-blocking upload
 
 In some cases, you want to push data without blocking your main thread. This is particularly useful to upload logs and
 artifacts while continuing a training. To do so, you can use the `run_as_future` argument in both [`upload_file`] and
@@ -156,7 +150,7 @@ Future(...)
 Future(...)
 ```
 
-### Upload a folder by chunks
+## Upload a folder by chunks
 
 [`upload_folder`] makes it easy to upload an entire folder to the Hub. However, for large folders (thousands of files or
 hundreds of GB), it can still be challenging. If you have a folder with a lot of files, you might want to upload
@@ -195,7 +189,7 @@ notice.
 
 </Tip>
 
-### Scheduled uploads
+## Scheduled uploads
 
 The Hugging Face Hub makes it easy to save and version data. However, there are some limitations when updating the same file thousands of times. For instance, you might want to save logs of a training process or user
 feedback on a deployed Space. In these cases, uploading the data as a dataset on the Hub makes sense, but it can be hard to do properly. The main reason is that you don't want to version every update of your data because it'll make the git repository unusable. The [`CommitScheduler`] class offers a solution to this problem.
@@ -264,9 +258,62 @@ For more details about the [`CommitScheduler`], here is what you need to know:
     `scheduler.lock` lock to ensure thread-safety. The lock is blocked only when the scheduler scans the folder for
     changes, not when it uploads data. You can safely assume that it will not affect the user experience on your Space.
 
-### create_commit
+### Space persistence demo
+
+Persisting data from a Space to a Dataset on the Hub is the main use case for [`CommitScheduler`]. Depending on the use
+case, you might want to structure your data differently. The structure has to be robust to concurrent users and
+restarts which often implies generating UUIDs. Besides robustness, you should upload data in a format readable by the 🤗 Datasets library for later reuse. We created a [Space](https://huggingface.co/spaces/Wauplin/space_to_dataset_saver)
+that demonstrates how to save several different data formats (you may need to adapt it for your own specific needs).
 
-If you want to work at a commit-level, use the [`create_commit`] function directly. There are two types of operations supported by [`create_commit`]:
+### Custom uploads
+
+[`CommitScheduler`] assumes your data is append-only and should be uploading "as is". However, you
+might want to customize the way data is uploaded. You can do that by creating a class inheriting from [`CommitScheduler`]
+and overwrite the `push_to_hub` method (feel free to overwrite it any way you want). You are guaranteed it will
+be called every `every` minutes in a background thread. You don't have to worry about concurrency and errors but you
+must be careful about other aspects, such as pushing empty commits or duplicated data.
+
+In the (simplified) example below, we overwrite `push_to_hub` to zip all PNG files in a single archive to avoid
+overloading the repo on the Hub:
+
+```py
+class ZipScheduler(CommitScheduler):
+    def push_to_hub(self):
+        # 1. List PNG files
+          png_files = list(self.folder_path.glob("*.png"))
+          if len(png_files) == 0:
+              return None  # return early if nothing to commit
+
+        # 2. Zip png files in a single archive
+        with tempfile.TemporaryDirectory() as tmpdir:
+            archive_path = Path(tmpdir) / "train.zip"
+            with zipfile.ZipFile(archive_path, "w", zipfile.ZIP_DEFLATED) as zip:
+                for png_file in png_files:
+                    zip.write(filename=png_file, arcname=png_file.name)
+
+            # 3. Upload archive
+            self.api.upload_file(..., path_or_fileobj=archive_path)
+
+        # 4. Delete local png files to avoid re-uploading them later
+        for png_file in png_files:
+            png_file.unlink()
+```
+
+You can find a complete code in [this Gradio app](https://huggingface.co/spaces/Wauplin/space_to_dataset_saver/blob/main/app_1M_image.py).
+
+When you overwrite `push_to_hub`, you have access to the attributes of [`CommitScheduler`] and especially:
+- [`HfApi`] client: `api`
+- Folder parameters: `folder_path` and `path_in_repo`
+- Repo parameters: `repo_id`, `repo_type`, `revision`
+- The thread lock: `lock`
+
+## create_commit
+
+The [`upload_file`] and [`upload_folder`] functions are high-level APIs that are generally convenient to use. We recommend
+trying these functions first if you don't need to work at a lower level. However, if you want to work at a commit-level,
+you can use the [`create_commit`] function directly.
+
+There are two types of operations supported by [`create_commit`]:
 
 - [`CommitOperationAdd`] uploads a file to the Hub. If the file already exists, the file contents are overwritten. This operation accepts two arguments:
 
@@ -317,6 +364,9 @@ For more detailed information, take a look at the [`HfApi`] reference.
 
 ## Push files with Git LFS
 
+All the methods described above use the Hub's API to upload files. This is the recommended way to upload files to the Hub.
+However we also provide [`Repository`], a wrapper around the git tool to manage a local repository.
+
 Git LFS automatically handles files larger than 10MB. But for very large files (>5GB), you need to install a custom transfer agent for Git LFS:
 
 ```bash
diff --git a/src/huggingface_hub/_commit_scheduler.py b/src/huggingface_hub/_commit_scheduler.py
index 28c23af8..037e0d86 100644
--- a/src/huggingface_hub/_commit_scheduler.py
+++ b/src/huggingface_hub/_commit_scheduler.py
@@ -162,7 +162,26 @@ class CommitScheduler:
             return None
 
         logger.info("(Background) scheduled commit triggered.")
+        return self.push_to_hub()
 
+    def push_to_hub(self) -> Optional[CommitInfo]:
+        """
+        Push folder to the Hub and return the commit info.
+
+        <Tip warning={true}>
+
+        This method is not meant to be called directly. It is run in the background by the scheduler, respecting a
+        queue mechanism to avoid concurrent commits. Making a direct call to the method might lead to concurrency
+        issues.
+
+        </Tip>
+
+        The default behavior of `push_to_hub` is to assume an append-only folder. It lists all files in the folder and
+        uploads only changed files. If no changes are found, the method returns without committing anything. If you want
+        to change this behavior, you can inherit from [`CommitScheduler`] and override this method. This can be useful
+        for example to compress data together in a single file before committing. For more details and examples, check
+        out our [integration guide](https://huggingface.co/docs/huggingface_hub/main/en/guides/upload#scheduled-uploads).
+        """
         # Check files to upload (with lock)
         with self.lock:
             logger.debug("Listing files to upload for scheduled commit.")
diff --git a/src/huggingface_hub/_inference.py b/src/huggingface_hub/_inference.py
index 4aab6568..5fa3a471 100644
--- a/src/huggingface_hub/_inference.py
+++ b/src/huggingface_hub/_inference.py
@@ -44,6 +44,7 @@ from pathlib import Path
 from typing import TYPE_CHECKING, Any, BinaryIO, ContextManager, Dict, Generator, List, Optional, Union, overload
 
 from requests import HTTPError, Response
+from requests.structures import CaseInsensitiveDict
 
 from ._inference_types import ClassificationOutput, ConversationalOutput, ImageSegmentationOutput
 from .constants import INFERENCE_ENDPOINT
@@ -96,17 +97,31 @@ class InferenceClient:
             or a URL to a deployed Inference Endpoint. Defaults to None, in which case a recommended model is
             automatically selected for the task.
         token (`str`, *optional*):
-            Hugging Face token. Will default to the locally saved token.
+            Hugging Face token. Will default to the locally saved token. Pass `token=False` if you don't want to send
+            your token to the server.
         timeout (`float`, `optional`):
             The maximum number of seconds to wait for a response from the server. Loading a new model in Inference
             API can take up to several minutes. Defaults to None, meaning it will loop until the server is available.
+        headers (`Dict[str, str]`, `optional`):
+            Additional headers to send to the server. By default only the authorization and user-agent headers are sent.
+            Values in this dictionary will override the default values.
+        cookies (`Dict[str, str]`, `optional`):
+            Additional cookies to send to the server.
     """
 
     def __init__(
-        self, model: Optional[str] = None, token: Optional[str] = None, timeout: Optional[float] = None
+        self,
+        model: Optional[str] = None,
+        token: Union[str, bool, None] = None,
+        timeout: Optional[float] = None,
+        headers: Optional[Dict[str, str]] = None,
+        cookies: Optional[Dict[str, str]] = None,
     ) -> None:
         self.model: Optional[str] = model
-        self.headers = build_hf_headers(token=token)
+        self.headers = CaseInsensitiveDict(build_hf_headers(token=token))  # contains 'authorization' + 'user-agent'
+        if headers is not None:
+            self.headers.update(headers)
+        self.cookies = cookies
         self.timeout = timeout
 
     def __repr__(self):
@@ -157,7 +172,12 @@ class InferenceClient:
             with _open_as_binary(data) as data_as_binary:
                 try:
                     response = get_session().post(
-                        url, json=json, data=data_as_binary, headers=self.headers, timeout=self.timeout
+                        url,
+                        json=json,
+                        data=data_as_binary,
+                        headers=self.headers,
+                        cookies=self.cookies,
+                        timeout=self.timeout,
                     )
                 except TimeoutError as error:
                     # Convert any `TimeoutError` to a `InferenceTimeoutError`
@@ -214,7 +234,7 @@ class InferenceClient:
         ```py
         >>> from huggingface_hub import InferenceClient
         >>> client = InferenceClient()
-        >>> client.audio_classification("audio.wav")
+        >>> client.audio_classification("audio.flac")
         [{'score': 0.4976358711719513, 'label': 'hap'}, {'score': 0.3677836060523987, 'label': 'neu'},...]
         ```
         """
@@ -250,7 +270,7 @@ class InferenceClient:
         ```py
         >>> from huggingface_hub import InferenceClient
         >>> client = InferenceClient()
-        >>> client.automatic_speech_recognition("hello_world.wav")
+        >>> client.automatic_speech_recognition("hello_world.flac")
         "hello world"
         ```
         """
@@ -760,7 +780,7 @@ class InferenceClient:
         >>> client = InferenceClient()
 
         >>> audio = client.text_to_speech("Hello world")
-        >>> Path("hello_world.wav").write_bytes(audio)
+        >>> Path("hello_world.flac").write_bytes(audio)
         ```
         """
         response = self.post(json={"inputs": text}, model=model, task="text-to-speech")


Test patch:
diff --git a/tests/test_commit_scheduler.py b/tests/test_commit_scheduler.py
index 477563e4..2996e1b6 100644
--- a/tests/test_commit_scheduler.py
+++ b/tests/test_commit_scheduler.py
@@ -2,7 +2,7 @@ import time
 import unittest
 from io import SEEK_END
 from pathlib import Path
-from unittest.mock import MagicMock, call, patch
+from unittest.mock import MagicMock, patch
 
 import pytest
 
@@ -32,7 +32,7 @@ class TestCommitScheduler(unittest.TestCase):
         except Exception:
             pass
 
-    @patch("huggingface_hub._commit_scheduler.CommitScheduler._push_to_hub")
+    @patch("huggingface_hub._commit_scheduler.CommitScheduler.push_to_hub")
     def test_mocked_push_to_hub(self, push_to_hub_mock: MagicMock) -> None:
         self.scheduler = CommitScheduler(
             folder_path=self.cache_dir,
@@ -40,10 +40,10 @@ class TestCommitScheduler(unittest.TestCase):
             every=1 / 60 / 10,  # every 0.1s
             hf_api=self.api,
         )
-        time.sleep(0.25)
+        time.sleep(0.3)
 
-        # Triggered 3 times (at 0.0s, 0.1s and 0.2s) with empty args
-        push_to_hub_mock.assert_has_calls([call(), call(), call()])
+        # Triggered at least twice times (at 0.0s and then 0.1s, 0.2s,...)
+        self.assertGreater(len(push_to_hub_mock.call_args_list), 2)
 
         # Can get the last upload result
         self.assertEqual(self.scheduler.last_future.result(), push_to_hub_mock.return_value)
diff --git a/tests/test_inference_client.py b/tests/test_inference_client.py
index 7bf57d81..8b0b45b7 100644
--- a/tests/test_inference_client.py
+++ b/tests/test_inference_client.py
@@ -14,6 +14,7 @@
 import io
 import unittest
 from pathlib import Path
+from unittest.mock import MagicMock, patch
 
 import numpy as np
 import pytest
@@ -21,6 +22,7 @@ from PIL import Image
 
 from huggingface_hub import InferenceClient, hf_hub_download
 from huggingface_hub._inference import _open_as_binary
+from huggingface_hub.utils import build_hf_headers
 
 from .testing_utils import with_production_testing
 
@@ -251,3 +253,37 @@ class TestResolveURL(InferenceClientTest):
     def test_unsupported_task(self) -> None:
         with self.assertRaises(NotImplementedError):
             InferenceClient()._resolve_url(task="unknown-task")
+
+
+class TestHeadersAndCookies(unittest.TestCase):
+    def test_headers_and_cookies(self) -> None:
+        client = InferenceClient(headers={"X-My-Header": "foo"}, cookies={"my-cookie": "bar"})
+        self.assertEqual(client.headers["X-My-Header"], "foo")
+        self.assertEqual(client.cookies["my-cookie"], "bar")
+
+    def test_headers_overwrite(self) -> None:
+        # Default user agent
+        self.assertTrue(InferenceClient().headers["user-agent"].startswith("unknown/None;"))
+
+        # Overwritten user-agent
+        self.assertEqual(InferenceClient(headers={"user-agent": "bar"}).headers["user-agent"], "bar")
+
+        # Case-insensitive overwrite
+        self.assertEqual(InferenceClient(headers={"USER-agent": "bar"}).headers["user-agent"], "bar")
+
+    @patch("huggingface_hub._inference.get_session")
+    def test_mocked_post(self, get_session_mock: MagicMock) -> None:
+        """Test that headers and cookies are correctly passed to the request."""
+        client = InferenceClient(headers={"X-My-Header": "foo"}, cookies={"my-cookie": "bar"})
+        response = client.post(data=b"content", model="username/repo_name")
+        self.assertEqual(response, get_session_mock().post.return_value)
+
+        expected_user_agent = build_hf_headers()["user-agent"]
+        get_session_mock().post.assert_called_once_with(
+            "https://api-inference.huggingface.co/models/username/repo_name",
+            json=None,
+            data=b"content",
+            headers={"user-agent": expected_user_agent, "X-My-Header": "foo"},
+            cookies={"my-cookie": "bar"},
+            timeout=None,
+        )