Repository: huggingface/huggingface_hub
Pull request #1717 (2023-10-08 17:49:03)

Add `token` parameter to `HfApi`'s `snapshot_download` and `hf_hub_download`
Adds a `token` parameter to `HfApi`'s `snapshot_download`/`hf_hub_download` to be consistent with the rest of the methods.

(It would come in handy in https://github.com/huggingface/datasets/pull/6269 🙂)

HuggingFaceDocBuilderDev: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/huggingface_hub/pr_1717). All of your documentation changes will be reflected on that endpoint.
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/huggingface/huggingface_hub/pull/1717?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface) Report
All modified lines are covered by tests :white_check_mark:
> Comparison is base [(`f752932`)](https://app.codecov.io/gh/huggingface/huggingface_hub/commit/f7529325e0739a0c818f84d50b9642b30918407f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface) 82.51% compared to head [(`5871d42`)](https://app.codecov.io/gh/huggingface/huggingface_hub/pull/1717?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface) 82.47%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #1717      +/-   ##
==========================================
- Coverage   82.51%   82.47%   -0.05%     
==========================================
  Files          62       62              
  Lines        7262     7262              
==========================================
- Hits         5992     5989       -3     
- Misses       1270     1273       +3     
```


| [Files](https://app.codecov.io/gh/huggingface/huggingface_hub/pull/1717?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface) | Coverage Δ | |
|---|---|---|
| [src/huggingface\_hub/hf\_api.py](https://app.codecov.io/gh/huggingface/huggingface_hub/pull/1717?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface#diff-c3JjL2h1Z2dpbmdmYWNlX2h1Yi9oZl9hcGkucHk=) | `86.55% <ø> (ø)` | |

... and [2 files with indirect coverage changes](https://app.codecov.io/gh/huggingface/huggingface_hub/pull/1717/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/huggingface/huggingface_hub/pull/1717?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=huggingface).


Patch:
diff --git a/src/huggingface_hub/hf_api.py b/src/huggingface_hub/hf_api.py
index b8e793c6..23e55589 100644
--- a/src/huggingface_hub/hf_api.py
+++ b/src/huggingface_hub/hf_api.py
@@ -4237,6 +4237,7 @@ class HfApi:
         proxies: Optional[Dict] = None,
         etag_timeout: float = 10,
         resume_download: bool = False,
+        token: Optional[Union[str, bool]] = None,
         local_files_only: bool = False,
         legacy_cache_layout: bool = False,
     ) -> str:
@@ -4322,6 +4323,11 @@ class HfApi:
                 data before giving up which is passed to `requests.request`.
             resume_download (`bool`, *optional*, defaults to `False`):
                 If `True`, resume a previously interrupted download.
+            token (`bool` or `str`, *optional*):
+                A valid authentication token (see https://huggingface.co/settings/token).
+                If `None` or `True` and machine is logged in (through `huggingface-cli login`
+                or [`~huggingface_hub.login`]), token will be retrieved from the cache.
+                If `False`, token is not sent in the request header.
             local_files_only (`bool`, *optional*, defaults to `False`):
                 If `True`, avoid downloading the file and return the path to the
                 local cached file if it exists.
@@ -4358,6 +4364,10 @@ class HfApi:
         """
         from .file_download import hf_hub_download
 
+        if token is None:
+            # Cannot do `token = token or self.token` as token can be `False`.
+            token = self.token
+
         return hf_hub_download(
             repo_id=repo_id,
             filename=filename,
@@ -4376,7 +4386,7 @@ class HfApi:
             proxies=proxies,
             etag_timeout=etag_timeout,
             resume_download=resume_download,
-            token=self.token,
+            token=token,
             local_files_only=local_files_only,
             legacy_cache_layout=legacy_cache_layout,
         )
@@ -4395,6 +4405,7 @@ class HfApi:
         etag_timeout: float = 10,
         resume_download: bool = False,
         force_download: bool = False,
+        token: Optional[Union[str, bool]] = None,
         local_files_only: bool = False,
         allow_patterns: Optional[Union[List[str], str]] = None,
         ignore_patterns: Optional[Union[List[str], str]] = None,
@@ -4455,6 +4466,11 @@ class HfApi:
                 If `True`, resume a previously interrupted download.
             force_download (`bool`, *optional*, defaults to `False`):
                 Whether the file should be downloaded even if it already exists in the local cache.
+            token (`bool` or `str`, *optional*):
+                A valid authentication token (see https://huggingface.co/settings/token).
+                If `None` or `True` and machine is logged in (through `huggingface-cli login`
+                or [`~huggingface_hub.login`]), token will be retrieved from the cache.
+                If `False`, token is not sent in the request header.
             local_files_only (`bool`, *optional*, defaults to `False`):
                 If `True`, avoid downloading the file and return the path to the
                 local cached file if it exists.
@@ -4490,6 +4506,10 @@ class HfApi:
         """
         from ._snapshot_download import snapshot_download
 
+        if token is None:
+            # Cannot do `token = token or self.token` as token can be `False`.
+            token = self.token
+
         return snapshot_download(
             repo_id=repo_id,
             repo_type=repo_type,
@@ -4505,7 +4525,7 @@ class HfApi:
             etag_timeout=etag_timeout,
             resume_download=resume_download,
             force_download=force_download,
-            token=self.token,
+            token=token,
             local_files_only=local_files_only,
             allow_patterns=allow_patterns,
             ignore_patterns=ignore_patterns,
diff --git a/src/huggingface_hub/templates/datasetcard_template.md b/src/huggingface_hub/templates/datasetcard_template.md
index 6d9281f9..f8cb4c80 100644
--- a/src/huggingface_hub/templates/datasetcard_template.md
+++ b/src/huggingface_hub/templates/datasetcard_template.md
@@ -1,103 +1,143 @@
 ---
-# For reference on model card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/datasetcard.md?plain=1
+# For reference on dataset card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/datasetcard.md?plain=1
 # Doc / guide: https://huggingface.co/docs/hub/datasets-cards
 {{ card_data }}
 ---
 
 # Dataset Card for {{ pretty_name | default("Dataset Name", true) }}
 
-## Dataset Description
+<!-- Provide a quick summary of the dataset. -->
 
-- **Homepage:** {{ homepage_url | default("", true)}}
-- **Repository:** {{ repo_url | default("", true)}}
-- **Paper:** {{ paper_url | default("", true)}}
-- **Leaderboard:** {{ leaderboard_url | default("", true)}}
-- **Point of Contact:** {{ point_of_contact | default("", true)}}
+{{ dataset_summary | default("", true) }}
 
-### Dataset Summary
+## Dataset Details
 
-{{ dataset_summary | default("[More Information Needed]", true)}}
+### Dataset Description
 
-### Supported Tasks and Leaderboards
+<!-- Provide a longer summary of what this dataset is. -->
 
-{{ supported_tasks_and_leaderboards_section | default("[More Information Needed]", true)}}
+{{ dataset_description | default("", true) }}
 
-### Languages
+- **Curated by:** {{ curators | default("[More Information Needed]", true)}}
+- **Funded by [optional]:** {{ funded_by | default("[More Information Needed]", true)}}
+- **Shared by [optional]:** {{ shared_by | default("[More Information Needed]", true)}}
+- **Language(s) (NLP):** {{ language | default("[More Information Needed]", true)}}
+- **License:** {{ license | default("[More Information Needed]", true)}}
 
-{{ languages_section | default("[More Information Needed]", true)}}
+### Dataset Sources [optional]
 
-## Dataset Structure
+<!-- Provide the basic links for the dataset. -->
+
+- **Repository:** {{ repo | default("[More Information Needed]", true)}}
+- **Paper [optional]:** {{ paper | default("[More Information Needed]", true)}}
+- **Demo [optional]:** {{ demo | default("[More Information Needed]", true)}}
+
+## Uses
 
-### Data Instances
+<!-- Address questions around how the dataset is intended to be used. -->
 
-{{ data_instances_section | default("[More Information Needed]", true)}}
+### Direct Use
 
-### Data Fields
+<!-- This section describes suitable use cases for the dataset. -->
 
-{{ data_fields_section | default("[More Information Needed]", true)}}
+{{ direct_use | default("[More Information Needed]", true)}}
 
-### Data Splits
+### Out-of-Scope Use
 
-{{ data_splits_section | default("[More Information Needed]", true)}}
+<!-- This section addresses misuse, malicious use, and uses that the dataset will not work well for. -->
+
+{{ out_of_scope_use | default("[More Information Needed]", true)}}
+
+## Dataset Structure
+
+<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->
+
+{{ dataset_structure | default("[More Information Needed]", true)}}
 
 ## Dataset Creation
 
 ### Curation Rationale
 
+<!-- Motivation for the creation of this dataset. -->
+
 {{ curation_rationale_section | default("[More Information Needed]", true)}}
 
 ### Source Data
 
-#### Initial Data Collection and Normalization
+<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->
+
+#### Data Collection and Processing
+
+<!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. -->
 
-{{ data_collection_section | default("[More Information Needed]", true)}}
+{{ data_collection_and_processing_section | default("[More Information Needed]", true)}}
 
-#### Who are the source language producers?
+#### Who are the source data producers?
 
-{{ source_language_producers_section | default("[More Information Needed]", true)}}
+<!-- This section describes the people or systems who originally created the data. It should also include self-reported demographic or identity information for the source data creators if this information is available. -->
 
-### Annotations
+{{ source_data_producers_section | default("[More Information Needed]", true)}}
+
+### Annotations [optional]
+
+<!-- If the dataset contains annotations which are not part of the initial data collection, use this section to describe them. -->
 
 #### Annotation process
 
+<!-- This section describes the annotation process such as annotation tools used in the process, the amount of data annotated, annotation guidelines provided to the annotators, interannotator statistics, annotation validation, etc. -->
+
 {{ annotation_process_section | default("[More Information Needed]", true)}}
 
 #### Who are the annotators?
 
+<!-- This section describes the people or systems who created the annotations. -->
+
 {{ who_are_annotators_section | default("[More Information Needed]", true)}}
 
-### Personal and Sensitive Information
+#### Personal and Sensitive Information
+
+<!-- State whether the dataset contains data that might be considered personal, sensitive, or private (e.g., data that reveals addresses, uniquely identifiable names or aliases, racial or ethnic origins, sexual orientations, religious beliefs, political opinions, financial or health data, etc.). If efforts were made to anonymize the data, describe the anonymization process. -->
+
+{{ personal_and_sensitive_information | default("[More Information Needed]", true)}}
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+{{ bias_risks_limitations | default("[More Information Needed]", true)}}
+
+### Recommendations
 
-{{ personal_and_sensitive_information_section | default("[More Information Needed]", true)}}
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
-## Considerations for Using the Data
+{{ bias_recommendations | default("Users should be made aware of the risks, biases and limitations of the dataset. More information needed for further recommendations.", true)}}
 
-### Social Impact of Dataset
+## Citation [optional]
 
-{{ social_impact_section | default("[More Information Needed]", true)}}
+<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->
 
-### Discussion of Biases
+**BibTeX:**
 
-{{ discussion_of_biases_section | default("[More Information Needed]", true)}}
+{{ citation_bibtex | default("[More Information Needed]", true)}}
 
-### Other Known Limitations
+**APA:**
 
-{{ known_limitations_section | default("[More Information Needed]", true)}}
+{{ citation_apa | default("[More Information Needed]", true)}}
 
-## Additional Information
+## Glossary [optional]
 
-### Dataset Curators
+<!-- If relevant, include terms and calculations in this section that can help readers understand the dataset or dataset card. -->
 
-{{ dataset_curators_section | default("[More Information Needed]", true)}}
+{{ glossary | default("[More Information Needed]", true)}}
 
-### Licensing Information
+## More Information [optional]
 
-{{ licensing_information_section | default("[More Information Needed]", true)}}
+{{ more_information | default("[More Information Needed]", true)}}
 
-### Citation Information
+## Dataset Card Authors [optional]
 
-{{ citation_information_section | default("[More Information Needed]", true)}}
+{{ dataset_card_authors | default("[More Information Needed]", true)}}
 
-### Contributions
+## Dataset Card Contact
 
-{{ contributions_section | default("[More Information Needed]", true)}}
+{{ dataset_card_contact | default("[More Information Needed]", true)}}
\ No newline at end of file
diff --git a/src/huggingface_hub/templates/modelcard_template.md b/src/huggingface_hub/templates/modelcard_template.md
index ec2d18d4..8c9243fb 100644
--- a/src/huggingface_hub/templates/modelcard_template.md
+++ b/src/huggingface_hub/templates/modelcard_template.md
@@ -19,6 +19,7 @@
 {{ model_description | default("", true) }}
 
 - **Developed by:** {{ developers | default("[More Information Needed]", true)}}
+- **Funded by [optional]:** {{ funded_by | default("[More Information Needed]", true)}}
 - **Shared by [optional]:** {{ shared_by | default("[More Information Needed]", true)}}
 - **Model type:** {{ model_type | default("[More Information Needed]", true)}}
 - **Language(s) (NLP):** {{ language | default("[More Information Needed]", true)}}
@@ -77,7 +78,7 @@ Use the code below to get started with the model.
 
 ### Training Data
 
-<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
 {{ training_data | default("[More Information Needed]", true)}}
 
@@ -108,7 +109,7 @@ Use the code below to get started with the model.
 
 #### Testing Data
 
-<!-- This should link to a Data Card if possible. -->
+<!-- This should link to a Dataset Card if possible. -->
 
 {{ testing_data | default("[More Information Needed]", true)}}
 


Test patch:
diff --git a/tests/test_repocard.py b/tests/test_repocard.py
index 454d3df3..8cef6ba0 100644
--- a/tests/test_repocard.py
+++ b/tests/test_repocard.py
@@ -886,10 +886,8 @@ class DatasetCardTest(TestCaseWithCapLog):
         # Here we pass the card data as kwargs as well so template picks up pretty_name.
         card = DatasetCard.from_template(
             card_data,
-            homepage_url="https://huggingface.co",
-            repo_url="https://github.com/huggingface/huggingface_hub",
-            paper_url="https://arxiv.org/pdf/1910.03771.pdf",
-            point_of_contact="https://huggingface.co/nateraw",
+            repo="https://github.com/huggingface/huggingface_hub",
+            paper="https://arxiv.org/pdf/1910.03771.pdf",
             dataset_summary=(
                 "This is a test dataset card to check if the template variables "
                 "in the dataset card template are working."
@@ -898,8 +896,8 @@ class DatasetCardTest(TestCaseWithCapLog):
         self.assertTrue(card.text.strip().startswith("# Dataset Card for My Cool Dataset"))
         self.assertIsInstance(card, DatasetCard)
 
-        matches = re.findall(r"Homepage:\*\* https:\/\/huggingface\.co", str(card))
-        self.assertEqual(matches[0], "Homepage:** https://huggingface.co")
+        matches = re.findall(r"Repository:\*\* https://github\.com/huggingface/huggingface_hub", str(card))
+        self.assertEqual(matches[0], "Repository:** https://github.com/huggingface/huggingface_hub")
 
     @require_jinja
     def test_dataset_card_from_custom_template(self):