Inference API takes forever and output: "Model ... is currently loading" [couldn't reopen #23793]
Author: cassiasamp
Pull request: no
Labels: name: bug

### System Info

It wasn't possible to reopen issue #23793 ..so I've created a new one. I'd like to know if there's a fix for it or any explanation, workaround or estimative for model availability.

Unfortunately, it seems we are still getting the same error response that many others reported for more than one model. 



### Who can help?

@ArthurZucker

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Execute a curl to a model
```
curl https://api-inference.huggingface.co/models/meta-llama/llama-2-13b-chat \
        -X POST \
        -d '{"inputs": "Deploying my first endpoint was not an amazing experience."}' \
        -H "Authorization: Bearer <insert-hf-token-here>"
```
2. Get this error response
```
{"error":"Model meta-llama/Llama-2-13b-chat is currently loading","estimated_time":20.0}
```

### Expected behavior

Status 200 and llamas response...currently even in the HF website, I've just gotten a time out
<img width="634" alt="image" src="https://github.com/user-attachments/assets/56972306-72b5-4990-b831-6e1a6c993673">


Comments:
cc @Narsil too 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I'm getting this too. This model worked the first couple times I tested it today, but for about an hour I've been getting the "currently loading" 503 error every time.

`estimated_time` is always the same and it never actually loads:

```
OpenAI HTTP Error (spotted in ruby-openai 7.1.0): {"error"=>"Model LoneStriker/aya-23-8B-6.0bpw-h6-exl2 is currently loading", "estimated_time"=>357.9726867675781}

// ... several minutes later

OpenAI HTTP Error (spotted in ruby-openai 7.1.0): {"error"=>"Model LoneStriker/aya-23-8B-6.0bpw-h6-exl2 is currently loading", "estimated_time"=>357.9726867675781}
```

I have the same error for few days for some models.

is there any new information regarding this? 

cc @Wauplin if you have seen this recently

I looked into it and the message seems wrong indeed. When trying on `meta-llama/llama-2-13b-chat` I got 

```
{"error":"Model meta-llama/Llama-2-13b-chat is currently loading","estimated_time":20.0}
```

while we should get something like `"Inference API is not available for model meta-llama/Llama-2-13b-chat"`

In fact, [Llama-2-13b-chat](https://huggingface.co/meta-llama/Llama-2-13b-chat) is not compatible with `transformers` not `TGI`. The model you might be looking at is [Llama-2-13b-hf](https://huggingface.co/meta-llama/Llama-2-13b-hf) which is the same model in a different format. Though if you try it, you'll get an error 

```
The model meta-llama/Llama-2-13b-hf is too large to be loaded automatically (26GB > 10GB). Please use Spaces (https://huggingface.co/spaces) or Inference Endpoints (https://huggingface.co/inference-endpoints).
```
(which is self explanatory)

If I'm not mistaken, I also got the "estimated_time" error  or "too large to be loaded" message when I actually needed to get permission to run the model..not only when the model wasn't available for some reason..or wasn't compatible.

I don't know if it's possible, but It could be nice to have different error messages depending on the situation.

> If I'm not mistaken, I also got the "estimated_time" error or "too large to be loaded" message when I actually needed to get permission to run the model..

I'd be interested if you remember the model id. You don't need permissions to run a model that is public (even gated ones).   

Got it, I will try to remember :/ it's been months and I really needed the application to work...so I ended up testing a bunch of models we could use at the time. What stuck with me was what I mentioned..and I also found the #23793 issue at the time, 
that's why I made the suggestion

It's possible that the problem regarding gated models on inference API has been fixed over the last months. Regarding [this problem](https://github.com/huggingface/transformers/issues/32665#issuecomment-2402620254) it's identified and we'll indeed check to improve that.