Title: HfHubHTTPError: 500 Server Error: Internal Server Error for url:
State: closed

### Describe the bug

newly uploaded datasets, since yesterday, yields an error.

old datasets, works fine.

Seems like the datasets api server returns a 500

I'm getting the same error, when I invoke `load_dataset` with my dataset.

Long discussion about it here, but I'm not sure anyone from huggingface have seen it.
https://discuss.huggingface.co/t/hfhubhttperror-500-server-error-internal-server-error-for-url/99580/1

### Steps to reproduce the bug

this api url: 

https://huggingface.co/api/datasets/neoneye/simon-arc-shape-v4-rev3


respond with:

```
{"error":"Internal Error - We're working hard to fix this as soon as possible!"}
```


### Expected behavior

return no error with newer datasets.

With older datasets I can load the datasets fine.

### Environment info

# Browser
When I access the api in the browser:
https://huggingface.co/api/datasets/neoneye/simon-arc-shape-v4-rev3

```
{"error":"Internal Error - We're working hard to fix this as soon as possible!"}
```

### Request headers

```
Accept	text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Encoding	gzip, deflate, br, zstd
Accept-Language	en-US,en;q=0.5
Connection	keep-alive
Host	huggingface.co
Priority	u=1
Sec-Fetch-Dest	document
Sec-Fetch-Mode	navigate
Sec-Fetch-Site	cross-site
Upgrade-Insecure-Requests	1
User-Agent	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:127.0) Gecko/20100101 Firefox/127.0
```

### Response headers

```
X-Firefox-Spdy	h2
access-control-allow-origin	https://huggingface.co
access-control-expose-headers	X-Repo-Commit,X-Request-Id,X-Error-Code,X-Error-Message,X-Total-Count,ETag,Link,Accept-Ranges,Content-Range
content-length	80
content-type	application/json; charset=utf-8
cross-origin-opener-policy	same-origin
date	Fri, 26 Jul 2024 19:09:45 GMT
etag	W/"50-9qrwU+BNI4SD0Fe32p/nofkmv0c"
referrer-policy	strict-origin-when-cross-origin
vary	Origin
via	1.1 1624c79cd07e6098196697a6a7907e4a.cloudfront.net (CloudFront)
x-amz-cf-id	SP8E7n5qRaP6i9c9G83dNAiOzJBU4GXSrDRAcVNTomY895K35H0nJQ==
x-amz-cf-pop	CPH50-C1
x-cache	Error from cloudfront
x-error-message	Internal Error - We're working hard to fix this as soon as possible!
x-powered-by	huggingface-moon
x-request-id	Root=1-66a3f479-026417465ef42f49349fdca1
```

Comments:
same issue here. @albertvillanova @lhoestq 

Also impacted by this issue in many of my datasets (though not all) - in my case, this also seems to affect datasets that have been updated recently. Git cloning and the web interface still work:
- https://huggingface.co/api/datasets/acmc/cheat_reduced
- https://huggingface.co/api/datasets/acmc/ghostbuster_reuter_reduced
- https://huggingface.co/api/datasets/acmc/ghostbuster_wp_reduced
- https://huggingface.co/api/datasets/acmc/ghostbuster_essay_reduced

Oddly enough, the system status looks good: https://status.huggingface.co/

Hey how to download these datasets using git cloning?

Also reported here
https://github.com/huggingface/huggingface_hub/issues/2425

I have been getting the same error for the past 8 hours as well

Same error since yesterday, fails on any new dataset created

Same here. I cannot download the HelpSteer2 dataset: https://huggingface.co/datasets/nvidia/HelpSteer2 which has been uploaded about a month ago

> Hey how to download these datasets using git cloning?

You'll find a guide [here](https://huggingface.co/docs/hub/en/datasets-downloading) 👍🏻

Same here for imdb dataset

It also happens with this dataset: https://huggingface.co/datasets/ylacombe/jenny-tts-6h-tagged

same here for all datsets in the sentence-tramsformers repo and related collections.

same issue with dataset that i recently uploaded on my repo.
seems that the upload date of the datset is not relevat (getting this issue with both old datasets and newer ones)

for some reason, i was able to get the dataset by turning it private and accessing it with the id token (accessing it as public while providing the token doesn not work)..... but i can say if that is just a random coincidence.

seems not much deterministic, for a specific dataset (sentence-transformer nq ) , that was "down" since some hours , worked for like 5-10 minutes, then stopped again

now even this dataset (that worked since some min ago, and that i'm in the middle of processing steps) stopped working: _https://huggingface.co/datasets/bobox/msmarco-bm25-EduScore/_

as already pointed out, there are no updates on **_https://status.huggingface.co/_**

\n
\n

an example of the whole error message:
``` 
HfHubHTTPError  

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, token, use_auth_token, task, streaming, num_proc, storage_options, trust_remote_code, **config_kwargs)
   2592 
   2593     # Create a dataset builder
-> 2594     builder_instance = load_dataset_builder(
   2595         path=path,
   2596         name=name,

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, token, use_auth_token, storage_options, trust_remote_code, _require_default_config_name, **config_kwargs)
   2264         download_config = download_config.copy() if download_config else DownloadConfig()
   2265         download_config.storage_options.update(storage_options)
-> 2266     dataset_module = dataset_module_factory(
   2267         path,
   2268         revision=revision,

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, cache_dir, trust_remote_code, _require_default_config_name, _require_custom_configs, **download_kwargs)
   1912                         f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
   1913                     ) from None
-> 1914                 raise e1 from None
   1915     else:
   1916         raise FileNotFoundError(

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, cache_dir, trust_remote_code, _require_default_config_name, _require_custom_configs, **download_kwargs)
   1832             hf_api = HfApi(config.HF_ENDPOINT)
   1833             try:
-> 1834                 dataset_info = hf_api.dataset_info(
   1835                     repo_id=path,
   1836                     revision=revision,

[/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py](https://localhost:8080/#) in _inner_fn(*args, **kwargs)
    112             kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.__name__, has_token=has_token, kwargs=kwargs)
    113 
--> 114         return fn(*args, **kwargs)
    115 
    116     return _inner_fn  # type: ignore

[/usr/local/lib/python3.10/dist-packages/huggingface_hub/hf_api.py](https://localhost:8080/#) in dataset_info(self, repo_id, revision, timeout, files_metadata, token)
   2362 
   2363         r = get_session().get(path, headers=headers, timeout=timeout, params=params)
-> 2364         hf_raise_for_status(r)
   2365         data = r.json()
   2366         return DatasetInfo(**data)

[/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_errors.py](https://localhost:8080/#) in hf_raise_for_status(response, endpoint_name)
    369         # Convert `HTTPError` into a `HfHubHTTPError` to display request information
    370         # as well (request id and/or server error message)
--> 371         raise HfHubHTTPError(str(e), response=response) from e
    372 
    373 

HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/api/datasets/bobox/xSum-processed (Request ID: Root=1-66a527f0-756cfbc35cc466f075382289;7d5dc06a-37e9-4c22-874d-92b0b1023276)

Internal Error - We're working hard to fix this as soon as possible!
``` 

we're working on a fix !

We fixed the issue, you can load datasets again, sorry for the inconvenience !

I can confirm, it's working now. I can load the dataset, yay. Thank you @lhoestq 

@lhoestq thank you so much! 

Hi I'm getting the same error with this [dataset](https://huggingface.co/datasets/huggan/smithsonian_butterflies_subset) 
Working on the course of stable diffusion , trying to run this [notebook](https://colab.research.google.com/github/huggingface/diffusion-models-class/blob/main/unit1/01_introduction_to_diffusers.ipynb#scrollTo=-yX-MZhSsxwp) 
this is the error: 
`HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/datasets/huggan/smithsonian_butterflies_subset/resolve/3cdedf844922ab40393d46d4c7f81c596e1c6d45/data/train-00000-of-00001.parquet (Request ID: Root=1-66ed3481-3393f4ab268b711440d31e02;c3ca2a7d-ae7b-4ba3-9947-9426711946a8)

Internal Error - We're working hard to fix this as soon as possible!`



Thanks for reporting, we are investigating !