Repository: huggingface/datasets
Pull request: Fix push_to_hub by not calling create_branch if PR branch
Fix push_to_hub by not calling create_branch if PR branch (e.g. `refs/pr/1`).

Note that currently create_branch raises a 400 Bad Request error if the user passes a PR branch (e.g. `refs/pr/1`).

EDIT:
~~Fix push_to_hub by not calling create_branch if branch exists.~~

Note that currently create_branch raises a 403 Forbidden error even if all these conditions are met:
- exist_ok is passed
- the branch already exists
- the user does not have WRITE permission

Fix #7067.

Related issue:
- https://github.com/huggingface/huggingface_hub/issues/2419

Problem statement:
Convert_to_parquet fails for datasets with multiple configs
If the dataset has multiple configs, when using the `datasets-cli convert_to_parquet` command to avoid issues with the data viewer caused by loading scripts, the conversion process only successfully converts the data corresponding to the first config. When it starts converting the second config, it throws an error:

```
Traceback (most recent call last):
  File "/opt/anaconda3/envs/dl/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/datasets/commands/datasets_cli.py", line 41, in main
    service.run()
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/datasets/commands/convert_to_parquet.py", line 83, in run
    dataset.push_to_hub(
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/datasets/dataset_dict.py", line 1713, in push_to_hub
    api.create_branch(repo_id, branch=revision, token=token, repo_type="dataset", exist_ok=True)
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 5503, in create_branch
    hf_raise_for_status(response)
  File "/opt/anaconda3/envs/dl/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 358, in hf_raise_for_status
    raise BadRequestError(message, response=response) from e
huggingface_hub.utils._errors.BadRequestError:  (Request ID: Root=1-669fc665-7c2e80d75f4337496ee95402;731fcdc7-0950-4eec-99cf-ce047b8d003f)

Bad request:
Invalid reference for a branch: refs/pr/1
```

Hints from issue discussion:
Many users have encountered the same issue, which has caused inconvenience.

https://discuss.huggingface.co/t/convert-to-parquet-fails-for-datasets-with-multiple-configs/86733

Thanks for reporting.

I will make the code more robust.
I have opened an issue in the huggingface-hub repo:
- https://github.com/huggingface/huggingface_hub/issues/2419

I am opening a PR to avoid calling `create_branch` if the branch already exists.

Commit message: Reword comment

Patch to review:
diff --git a/src/datasets/arrow_dataset.py b/src/datasets/arrow_dataset.py
index 779091a75af..c24345d1e1e 100644
--- a/src/datasets/arrow_dataset.py
+++ b/src/datasets/arrow_dataset.py
@@ -5620,7 +5620,8 @@ def push_to_hub(
         )
         repo_id = repo_url.repo_id
 
-        if revision is not None:
+        if revision is not None and not api.revision_exists(repo_id, revision, repo_type="dataset", token=token):
+            # We check revision_exists because create_branch raises 400/403 error even if branch exists and exist_ok
             api.create_branch(repo_id, branch=revision, token=token, repo_type="dataset", exist_ok=True)
 
         if not data_dir:
diff --git a/src/datasets/dataset_dict.py b/src/datasets/dataset_dict.py
index 9e3e8543b77..0621f2cd127 100644
--- a/src/datasets/dataset_dict.py
+++ b/src/datasets/dataset_dict.py
@@ -1709,7 +1709,8 @@ def push_to_hub(
         )
         repo_id = repo_url.repo_id
 
-        if revision is not None:
+        if revision is not None and not api.revision_exists(repo_id, revision, repo_type="dataset", token=token):
+            # We check revision_exists because create_branch raises 400/403 error even if branch exists and exist_ok
             api.create_branch(repo_id, branch=revision, token=token, repo_type="dataset", exist_ok=True)
 
         if not data_dir:


Code review comments:
diff_hunk: @@ -5620,7 +5620,8 @@ def push_to_hub(
         )
         repo_id = repo_url.repo_id
 
-        if revision is not None:
+        if revision is not None and not api.revision_exists(repo_id, revision, repo_type="dataset", token=token):
path: src/datasets/arrow_dataset.py
text: @author:
Alternatively:
```suggestion
        if revision is not None and not revision.startswith("refs/pr/"):
```

@author:
I think this modification would be more efficient:
- The modification above will fix the 400 Bad Request error without the additional network call `revision_exists`.
- In the case of a 403 Forbidden error, anyway we will raise an error some code lines below when trying to push to the branch if the user does not have write permission.

@author:
On the other hand, it is true that the call to `revision_exists` is a more general solution.