HfHubHTTPError: 429 Client Error: Too Many Requests for URL when trying to access Fineweb-10BT on 4A100 GPUs using SLURM
State: open

### Describe the bug

I am trying to run some finetunings on 4 A100 GPUs using SLURM using axolotl training framework which in turn uses Huggingface's Trainer and Accelerate on [Fineweb-10BT](https://huggingface.co/datasets/HuggingFaceFW/fineweb), but I end up running into 429 Client Error: Too Many Requests for URL error when I call next(dataloader_iter). Funny is, that I can run some test fine tuning (for just 200 training steps) in 1 A100 GPU using SLURM. Is there any rate limiter set for querying dataset? I could run the fine tuning with the same settings (4 A100 GPUs in SLURM) last month.

### Steps to reproduce the bug

You would need a server installed with SLURM

1. Create conda environment
1.1 conda create -n example_env -c conda-forge gxx=11 python=3.10
1.2 conda activate example_env
1.3 pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124
1.4 conda install nvidia/label/cuda-12.4.0::cuda-toolkit
1.5 Download flash_attn-2.7.4.post1+cu12torch2.5cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
1.6 pip3 install packaging
1.7 pip3 install ninja
1.8 pip3 install mlflow
1.9 Clone https://github.com/calvintanama/axolotl.git
1.10 `cd` to `axolotl`
1.11 pip3 install -e '.[deepspeed]'

2. Run the training
2.1. Create a folder called `config_run` in axolotl directory
2.2. Copy `config/phi3_pruned_extra_pretrain_22_29_bottleneck_residual_8_a100_4.yaml` to `config_run`
2.3. Change yaml file in the `config_run` accordingly
2.4. Change directory and conda environment name in `jobs/train_phi3_22_29_bottleneck_residual_8_a100_4_temp.sh`
2.5. `jobs/train_phi3_22_29_bottleneck_residual_8_a100_4_temp.sh`

### Expected behavior

This should not cause any error, but gotten

```
File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/accelerate/data_loader.py", line 552, in __iter__
[rank3]:     current_batch = next(dataloader_iter)
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 701, in __next__
[rank3]:     data = self._next_data()
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 757, in _next_data
[rank3]:     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 33, in fetch
[rank3]:     data.append(next(self.dataset_iter))
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/accelerate/data_loader.py", line 338, in __iter__
[rank3]:     for element in self.dataset:
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 2266, in __iter__
[rank3]:     for key, example in ex_iterable:
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1866, in __iter__
[rank3]:     for key, example in self.ex_iterable:
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1084, in __iter__
[rank3]:     yield from self._iter()
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1263, in _iter
[rank3]:     for key, transformed_example in outputs:
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1258, in <genexpr>
[rank3]:     outputs = (
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1244, in iter_outputs
[rank3]:     for i, key_example in inputs_iterator:
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1106, in iter_batched_inputs
[rank3]:     for key, example in iterator:
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1866, in __iter__
[rank3]:     for key, example in self.ex_iterable:
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1535, in __iter__
[rank3]:     for x in self.ex_iterable:
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 374, in __iter__
[rank3]:     for key, pa_table in self.generate_tables_fn(**gen_kwags):
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 90, in _generate_tables
[rank3]:     if parquet_fragment.row_groups:
[rank3]:   File "pyarrow/_dataset_parquet.pyx", line 386, in pyarrow._dataset_parquet.ParquetFileFragment.row_groups.__get__
[rank3]:   File "pyarrow/_dataset_parquet.pyx", line 393, in pyarrow._dataset_parquet.ParquetFileFragment.metadata.__get__
[rank3]:   File "pyarrow/_dataset_parquet.pyx", line 382, in pyarrow._dataset_parquet.ParquetFileFragment.ensure_complete_metadata
[rank3]:   File "pyarrow/error.pxi", line 89, in pyarrow.lib.check_status
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 827, in read_with_retries
[rank3]:     out = read(*args, **kwargs)
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/huggingface_hub/hf_file_system.py", line 1013, in read
[rank3]:     return super().read(length)
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/fsspec/spec.py", line 1941, in read
[rank3]:     out = self.cache._fetch(self.loc, self.loc + length)
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/fsspec/caching.py", line 234, in _fetch
[rank3]:     self.cache = self.fetcher(start, end)  # new block replaces old
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/huggingface_hub/hf_file_system.py", line 976, in _fetch_range
[rank3]:     hf_raise_for_status(r)
[rank3]:   File "/home/hk-project-test-p0023745/cd7437/miniconda3/envs/llmpruning_train_temp/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 482, in hf_raise_for_status
[rank3]:     raise _format(HfHubHTTPError, str(e), response) from e
[rank3]: huggingface_hub.errors.HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/datasets/HuggingFaceFW/fineweb/resolve/0f039043b23fe1d4eed300b504aa4b4a68f1c7ba/sample/10BT/006_00000.parquet
```

### Environment info

- datasets 3.5.0
- torch 2.5.1
- transformers 4.46.2

Hi ! make sure to be logged in with your HF account (e.g. using `huggingface-cli login` or passing `token=` to `load_dataset()`), otherwise you'll get rate limited at one point

Hey @calvintanama! Just building on what @lhoestq mentioned above — I ran into similar issues in multi-GPU SLURM setups and here’s what worked for me...

This 429 Client Error: Too Many Requests comes from the Hugging Face Hub’s rate limiting, which restricts unauthenticated or high-volume access (especially in multi-GPU/distributed setups like SLURM).

As @lhoestq mentioned, the solution is to make sure you are authenticated with the Hugging Face Hub in every process (especially on each GPU/worker node). You can do this by:

Running huggingface-cli login (interactive)

Or passing your token explicitly:

```python
load_dataset("HuggingFaceFW/fineweb", token="hf_your_token_here")
# If you’re using a SLURM cluster, ensure every node/process receives access to the token via env var:
```

```bash
export HF_TOKEN=hf_your_token_here
```

and then in Python:
```python
from datasets import load_dataset
load_dataset("HuggingFaceFW/fineweb", token=os.environ["HF_TOKEN"])
```
Also consider downloading the dataset beforehand with load_dataset(..., streaming=False) and storing it locally if you're repeatedly training with it.