AutoProcessor.from_pretrained retries HTTP 429 responses multiple times for every config file when using HF_ENDPOINT
Author: peacefulotter
Pull request: no
Labels: name: bug

### System Info

(uv run transformers env fails with name « CompletionCreateParamsStreaming » is not defined, but this is a separate issue) 

- transformers 5.12.1
- windows 11 

### Who can help?

@Cyrilvallez 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When `AutoProcessor.from_pretrained()` is used with a proxy configured through the `HF_ENDPOINT` environment variable, the method retries every request that receives an HTTP 429 (Too Many Requests) response several times before moving on.

For example, it attempts up to 5 GET requests for each of the following files:
- processor_config.json
- tokenizer_config.json
- tokenizer.json
and several other optional configuration files

Eventually, from_pretrained() returns but the repeated retries significantly increase the loading time and flood the logs with 429 errors.

I'm not sure whether this is intended behavior, but retrying multiple times on every 429 for optional files seems excessive. It makes me wonder whether the retry logic is being applied correctly / too broadly.

```py
import os

os.environ["HF_ENDPOINT"] = "https://<your-proxy>"

from transformers import AutoProcessor

AutoProcessor.from_pretrained(
    "facebook/dinov3-vits16-pretrain-lvd1689m"
)
```

### Expected behavior

Either avoid retrying optional files after repeated 429 responses, or
Retry more selectively, so that model loading does not incur dozens of failed requests before returning.

Comments:
I've been looking into this and would love to contribute a fix! I reproduced the behavior locally with a mock server and traced it to the interaction between Transformers' optional-file probing and `huggingface_hub`'s retry logic.

Sharing my findings below. If the approach looks good to maintainers, I'm happy to open a PR in the appropriate repository.

## Investigation & Reproduction

### What's happening

When `AutoProcessor.from_pretrained()` is called, Transformers may sequentially probe several processor and tokenizer configuration files, including:

- `config.json`
- `processor_config.json`
- `tokenizer_config.json`
- `tokenizer.json`
- `special_tokens_map.json`
- `vocab.json`

Each probe eventually calls `hf_hub_download()` from `huggingface_hub`.

In `huggingface_hub >= 1.x`, the internal `http_backoff()` function includes HTTP `429` in its default retry status-code list:

```python
# huggingface_hub/utils/_http.py
_DEFAULT_RETRY_ON_STATUS_CODES = (408, 429, 500, 502, 503, 504)
```

The current implementation supports Hugging Face's `RateLimit` headers and can use the reset value provided there. However, it does not appear to support the standard `Retry-After` header.

As a result, when a custom `HF_ENDPOINT` proxy returns:

```http
HTTP/1.1 429 Too Many Requests
Retry-After: 1
```

without Hugging Face-style `RateLimit` headers, `http_backoff()` falls back to its exponential retry schedule:

```text
1s -> 2s -> 4s -> 8s -> 8s
```

This produces roughly 23 seconds of waiting for each independent file probe.

Because the optional files are requested independently and sequentially, the delay can multiply across the loading process:

```text
6 files x 23 seconds = approximately 138 seconds
```

These are primarily metadata `HEAD` requests, so the client may spend more than two minutes retrying without downloading any file content.

### Reproduction

I used a local mock server that returned the following response for every request:

```http
HTTP/1.1 429 Too Many Requests
Retry-After: 1
```

Representative output for one file:

```text
--- Fetching config.json ---

HEAD /facebook/dinov2-base/resolve/main/config.json

HTTP Error 429 thrown while requesting HEAD .../config.json
Retrying in 1s [Retry 1/5].

HEAD /facebook/dinov2-base/resolve/main/config.json
Retrying in 2s [Retry 2/5].

HEAD /facebook/dinov2-base/resolve/main/config.json
Retrying in 4s [Retry 3/5].

HEAD /facebook/dinov2-base/resolve/main/config.json
Retrying in 8s [Retry 4/5].

HEAD /facebook/dinov2-base/resolve/main/config.json
Retrying in 8s [Retry 5/5].

HEAD /facebook/dinov2-base/resolve/main/config.json
-> LocalEntryNotFoundError after approximately 23 seconds
```

The same retry pattern occurred for each independently probed file:

| File | Approximate delay |
|---|---:|
| `config.json` | 23s |
| `processor_config.json` | 23s |
| `tokenizer_config.json` | 23s |
| `tokenizer.json` | 23s |
| `special_tokens_map.json` | 23s |
| `vocab.json` | 23s |

Final result:

```text
Total wall-clock time: approximately 138 seconds
```

The server returned `Retry-After: 1` on every response, but because it did not return the Hugging Face-specific `RateLimit` headers, the client used exponential backoff instead.

## Root Cause

I found two related behaviors.

### 1. `Retry-After` is not used by `http_backoff()`

`http_backoff()` can use the reset value from Hugging Face's `RateLimit` response headers, but it does not currently fall back to the standard `Retry-After` header.

This is already tracked in:

- huggingface/huggingface_hub#2360 - `http_backoff` does not respect `Retry-After` headers on 429 requests

That issue has been open since July 2024 and does not currently appear to have an associated PR. A maintainer suggested an adapter-based implementation there at the time.

### 2. Callers cannot distinguish a 429 through a typed exception

After the retry sequence is exhausted, `hf_raise_for_status()` raises an `HfHubHTTPError` whose message includes the `429 Too Many Requests` response.

However, there is no dedicated `RateLimitError` subclass.

This means higher-level callers such as Transformers' `cached_files()` cannot catch rate limiting separately from other hub HTTP failures. The loading process can therefore continue to probe the remaining optional files, with each file independently going through the same retry sequence.

## Possible Fixes

### Option A - Honor `Retry-After` in `http_backoff()`

This is already tracked in huggingface/huggingface_hub#2360.

Supporting `Retry-After` would improve the waiting behavior for custom endpoints and proxies that use the standard header rather than Hugging Face's `RateLimit` headers.

In the reproduction above, respecting `Retry-After: 1` would reduce the retry delay for each file from approximately 23 seconds to approximately 5 seconds.

I'm happy to pick up that existing issue if maintainers agree with the current direction.

### Option B - Add a typed `RateLimitError`

A dedicated exception could look conceptually like:

```python
class RateLimitError(HfHubHTTPError):
    pass
```

and be raised for HTTP `429` responses after the configured retry sequence is exhausted.

Transformers could then distinguish rate limiting from an ordinary missing optional file and stop probing the remaining files after the first file's retry sequence fails.

This would address the fan-out problem, although it would likely require coordinated changes in both `huggingface_hub` and Transformers.

### Option C - Implement both

My initial preference would be:

1. Add standard `Retry-After` support in `huggingface_hub`.
2. Expose exhausted HTTP `429` responses through a typed exception.
3. Allow Transformers to stop probing additional optional files once rate limiting has been confirmed.

The first change reduces the cost of each retry sequence, while the typed exception allows higher-level callers to avoid repeating that sequence across multiple optional files.

## Questions Before Opening a PR

1. Should the initial work happen in `huggingface_hub`, beginning with the existing `Retry-After` issue?
2. Would maintainers be open to a dedicated `RateLimitError` subclass for exhausted `429` responses?
3. If such an exception is added, should Transformers propagate it immediately rather than continuing to probe optional files?
4. Would you prefer these to be handled as separate PRs, with the `huggingface_hub` behavior addressed first?

I'm happy to open a PR in the appropriate repository based on the preferred scope.


Awesome findings, thanks. 

Additionally, I have discovered that after reloading my ipython kernel and running the code, `hugginface_hub` initally returns a 404 error after looking for `https://my_proxy/.../facebook/dino../processor_config.json`. 

There is indeed no `processor_config.json` for dinov3, but I feel like hf shouldn't yield an error here. 

Running the same code again hits me with the original 429 issue.

Might be two separate issues here.. 

@Cyrilvallez any news ? 

I have noticed a similar weird behavior happens when force_download = true, something is definitely wrong in how arguments are processed 

cc @Wauplin maybe? To be fair, I did not read all the super long ai comments, so may be more Transformers related

> 1. Retry-After is not used by http_backoff()

we've recently shipped https://github.com/huggingface/huggingface_hub/pull/4460 in latest release (1.22.0) which I think should mitigate this issue. If the server rate limits the client, we try our best to retry in a correct way.

Note that the best solution so far is to not use proxy at all, and pass a token to be authenticated with the Hub (I know it's not always possible but worth mentioning)