Unable to push dataset - `create_pr` problem
Opened by agombert (closed)

### Describe the bug

I can't upload to the hub the dataset I manually created locally (Image dataset). I have a problem when using the method `.push_to_hub` which asks for a `create_pr` attribute which is not compatible.

### Steps to reproduce the bug

here what I have:

```python
dataset.push_to_hub("agomberto/FrenchCensus-handwritten-texts")
```
Output:
```python
Pushing split train to the Hub.
Pushing dataset shards to the dataset hub:   0%|                                                                                                                                      | 0/2 [00:00<?, ?it/s]
Creating parquet from Arrow format:   0%|                                                                                                                                             | 0/3 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 12.70ba/s]
Pushing dataset shards to the dataset hub:   0%|                                                                                                                                      | 0/2 [00:01<?, ?it/s]

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py:259, in hf_raise_for_status(response, endpoint_name)
    258 try:
--> 259     response.raise_for_status()
    260 except HTTPError as e:

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/requests/models.py:1021, in Response.raise_for_status(self)
   1020 if http_error_msg:
-> 1021     raise HTTPError(http_error_msg, response=self)

HTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/datasets/agomberto/FrenchCensus-handwritten-texts/commit/main

The above exception was the direct cause of the following exception:

HfHubHTTPError                            Traceback (most recent call last)
Cell In[7], line 1
----> 1 dataset.push_to_hub("agomberto/FrenchCensus-handwritten-texts")

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/datasets/dataset_dict.py:1583, in DatasetDict.push_to_hub(self, repo_id, private, token, branch, max_shard_size, num_shards, embed_external_files)
   1581 logger.warning(f"Pushing split {split} to the Hub.")
   1582 # The split=key needs to be removed before merging
-> 1583 repo_id, split, uploaded_size, dataset_nbytes, _, _ = self[split]._push_parquet_shards_to_hub(
   1584     repo_id,
   1585     split=split,
   1586     private=private,
   1587     token=token,
   1588     branch=branch,
   1589     max_shard_size=max_shard_size,
   1590     num_shards=num_shards.get(split),
   1591     embed_external_files=embed_external_files,
   1592 )
   1593 total_uploaded_size += uploaded_size
   1594 total_dataset_nbytes += dataset_nbytes

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/datasets/arrow_dataset.py:5275, in Dataset._push_parquet_shards_to_hub(self, repo_id, split, private, token, branch, max_shard_size, num_shards, embed_external_files)
   5273         shard.to_parquet(buffer)
   5274         uploaded_size += buffer.tell()
-> 5275         _retry(
   5276             api.upload_file,
   5277             func_kwargs={
   5278                 "path_or_fileobj": buffer.getvalue(),
   5279                 "path_in_repo": shard_path_in_repo,
   5280                 "repo_id": repo_id,
   5281                 "token": token,
   5282                 "repo_type": "dataset",
   5283                 "revision": branch,
   5284             },
   5285             exceptions=HTTPError,
   5286             status_codes=[504],
   5287             base_wait_time=2.0,
   5288             max_retries=5,
   5289             max_wait_time=20.0,
   5290         )
   5291     shards_path_in_repo.append(shard_path_in_repo)
   5293 # Cleanup to remove unused files

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/datasets/utils/file_utils.py:285, in _retry(func, func_args, func_kwargs, exceptions, status_codes, max_retries, base_wait_time, max_wait_time)
    283 except exceptions as err:
    284     if retry >= max_retries or (status_codes and err.response.status_code not in status_codes):
--> 285         raise err
    286     else:
    287         sleep_time = min(max_wait_time, base_wait_time * 2**retry)  # Exponential backoff

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/datasets/utils/file_utils.py:282, in _retry(func, func_args, func_kwargs, exceptions, status_codes, max_retries, base_wait_time, max_wait_time)
    280 while True:
    281     try:
--> 282         return func(*func_args, **func_kwargs)
    283     except exceptions as err:
    284         if retry >= max_retries or (status_codes and err.response.status_code not in status_codes):

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py:120, in validate_hf_hub_args.<locals>._inner_fn(*args, **kwargs)
    117 if check_use_auth_token:
    118     kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.__name__, has_token=has_token, kwargs=kwargs)
--> 120 return fn(*args, **kwargs)

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/huggingface_hub/hf_api.py:2998, in HfApi.upload_file(self, path_or_fileobj, path_in_repo, repo_id, token, repo_type, revision, commit_message, commit_description, create_pr, parent_commit)
   2990 commit_message = (
   2991     commit_message if commit_message is not None else f"Upload {path_in_repo} with huggingface_hub"
   2992 )
   2993 operation = CommitOperationAdd(
   2994     path_or_fileobj=path_or_fileobj,
   2995     path_in_repo=path_in_repo,
   2996 )
-> 2998 commit_info = self.create_commit(
   2999     repo_id=repo_id,
   3000     repo_type=repo_type,
   3001     operations=[operation],
   3002     commit_message=commit_message,
   3003     commit_description=commit_description,
   3004     token=token,
   3005     revision=revision,
   3006     create_pr=create_pr,
   3007     parent_commit=parent_commit,
   3008 )
   3010 if commit_info.pr_url is not None:
   3011     revision = quote(_parse_revision_from_pr_url(commit_info.pr_url), safe="")

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py:120, in validate_hf_hub_args.<locals>._inner_fn(*args, **kwargs)
    117 if check_use_auth_token:
    118     kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.__name__, has_token=has_token, kwargs=kwargs)
--> 120 return fn(*args, **kwargs)

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/huggingface_hub/hf_api.py:2548, in HfApi.create_commit(self, repo_id, operations, commit_message, commit_description, token, repo_type, revision, create_pr, num_threads, parent_commit)
   2546 try:
   2547     commit_resp = get_session().post(url=commit_url, headers=headers, data=data, params=params)
-> 2548     hf_raise_for_status(commit_resp, endpoint_name="commit")
   2549 except RepositoryNotFoundError as e:
   2550     e.append_to_message(_CREATE_COMMIT_NO_REPO_ERROR_MESSAGE)

File ~/miniconda3/envs/hwocr/lib/python3.8/site-packages/huggingface_hub/utils/_errors.py:301, in hf_raise_for_status(response, endpoint_name)
    297     raise BadRequestError(message, response=response) from e
    299 # Convert `HTTPError` into a `HfHubHTTPError` to display request information
    300 # as well (request id and/or server error message)
--> 301 raise HfHubHTTPError(str(e), response=response) from e

HfHubHTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/datasets/agomberto/FrenchCensus-handwritten-texts/commit/main (Request ID: Root=1-645a66bf-255ad91602a6404e6cb70fba)

Forbidden: pass `create_pr=1` as a query parameter to create a Pull Request
```

And then when I do

```python
dataset.push_to_hub("agomberto/FrenchCensus-handwritten-texts", create_pr=1)
```

I get 

```python
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[8], line 1
----> 1 dataset.push_to_hub("agomberto/FrenchCensus-handwritten-texts", create_pr=1)

TypeError: push_to_hub() got an unexpected keyword argument 'create_pr'
```

### Expected behavior

I would like to have the dataset updloaded [here](https://huggingface.co/datasets/agomberto/FrenchCensus-handwritten-texts).

### Environment info

```bash
- `datasets` version: 2.12.0
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.8.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 1.5.3
```

Comments:
Thanks for reporting, @agombert.

In this case, I think the root issue is authentication: before pushing to Hub, you should authenticate. See our docs: https://huggingface.co/docs/datasets/upload_dataset#upload-with-python
> 2. To upload a dataset on the Hub in Python, you need to log in to your Hugging Face account:
   ```
   huggingface-cli login
   ```

Hey @albertvillanova well I actually did :D 

<img width="1079" alt="Capture d’écran 2023-05-09 à 18 02 58" src="https://github.com/huggingface/datasets/assets/17645711/e091aa20-06b1-4dd3-bfdb-35e832c66f8d">


That is weird that you get a Forbidden error if you are properly authenticated...

Today we had a big outage issue affecting the Hugging Face Hub. Could you please retry to push_to_hub your dataset? Maybe that was the cause...

Yes I've just tried again and same error 403 :/

Login successful but also got this error "Forbidden: pass `create_pr=1` as a query parameter to create a Pull Request"

Make sure your API token has a `write` role. I had the same issue as you with the `read` token. Creating a `write` token and using that solved the issue.

> Make sure your API token has a `write` role. I had the same issue as you with the `read` token. Creating a `write` token and using that solved the issue.

I generate a token with write role. It works! thank you so much.

@dmitrijsk amazing thanks so much ! 
The error should be clearer when the token is read-only – I wasted a lot of time there..

Based on the number of reactions (https://github.com/huggingface/datasets/issues/5833#issuecomment-1586521001), many users have issues debugging this. @Wauplin Maybe a more informative error can be thrown in `hfh` if a token's role is insufficient for an op. WDYT?

Yes indeed. I created an issue some time ago about it: https://github.com/huggingface/huggingface_hub/issues/1653. I'll prioritize it more and let you know. Thanks for the ping.

Hey everyone :wave: The error message has been fixed to be more informative. As mentioned in https://github.com/huggingface/datasets/issues/5833#issuecomment-1586521001, the n°1 reason why this is happening is that a `read` token has been used instead of `write`. The fix has being shipped on the server meaning that you don't need to update any client library. The new error message looks like this: 

```
huggingface_hub.utils._errors.HfHubHTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/models/Wauplin/test_recovered/preupload/main (Request ID: Root=1-6532752e-1ee492b070d9e3020e68bddc;25ca3387-44bc-433e-b49f-6e290305ed10)

Forbidden: you must use a write token to upload to a repository.
```

---

cc @mariosasko I let you close this issue if you feel it's completely solved

@Wauplin Reopening it. Indeed, the above error message is thrown if pushing to an **existing** repo with a `read` token. However, if the repo does not exist and needs to be created (by calling `create_repo` in `push_to_hub`), then passing a `read` token will raise the following:
```python
HTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/repos/create

The above exception was the direct cause of the following exception:

HfHubHTTPError                            Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_errors.py](https://localhost:8080/#) in hf_raise_for_status(response, endpoint_name)
    318         # Convert `HTTPError` into a `HfHubHTTPError` to display request information
    319         # as well (request id and/or server error message)
--> 320         raise HfHubHTTPError(str(e), response=response) from e
    321 
    322 

HfHubHTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/repos/create (Request ID: Root=1-653291a1-0e9cc2b16c049ff510834d47;7edeb8a3-4bf1-4d9b-8f7f-06d430da9ee7)

You don't have the rights to create a dataset under this namespace
```

I think this error message should also be more informative!

> However, if the repo does not exist and needs to be created (by calling create_repo in push_to_hub), then passing a read token will raise the following:

This seems to be a different issue than the one reported above right? Still agree that a more informative message would be nice. Can you open an issue on moon-landing for it please? (not sure I can open a PR myself for this one :grimacing:  )

@Wauplin Done :)