Inference API takes forever and output: "Model ... is currently loading"
Author: AIEngKenn
Pull request: no

### System Info

I'm currently working on my final project (Fine-tuning the "Helsinki-NLP/opus-mt-en-zh" and "Helsinki-NLP/opus-mt-zh-en" models for Translation English-Chinese), have already trained the model, and deployed it to the Hub. The issue is that I'm not able to consume the API properly: it's slow, it seems not to work properly, and it takes forever to load. 

I'm trying to use the Inference API connected to my Nextjs (Frontend) app, but I'm getting this error message: 

```
{
    "error": "Model ... is currently loading",
    "estimated_time": 20
}
```

Sometimes, It works, and then It stops working... 

Any help, please? 

Please, give me all the possible suggestions, would love to explore.

Thanks

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
async function translateEnglishtoChinese(text: string) {
  const response = await fetch(
    "https://api-inference.huggingface.co/models/KennStack01/Helsinki-NLP-opus-mt-en-zh",
    {
      headers: {
        Authorization: `Bearer ${process.env.NEXT_PUBLIC_HF_TOKEN}`,
      },
      method: "POST",
      body: JSON.stringify({
        inputs: text,
      }),
    }
  );
  const result = await response.json();
  console.log("English to Chinese", result[0]?.translation_text);
  return result;
}

async function translateChinesetoEnglish(text: string) {
  const response = await fetch(
    "https://api-inference.huggingface.co/models/KennStack01/Helsinki-NLP-opus-mt-zh-en",
    {
      headers: {
        Authorization: `Bearer ${process.env.NEXT_PUBLIC_HF_TOKEN}`,
      },
      method: "POST",
      body: JSON.stringify({
        inputs: text,
      }),
    }
  );
  const result = await response.json();
  console.log("Chinese to English", result[0]?.translation_text);
  return result;
}
```

### Expected behavior

Expecting to see a valid translation_text. Simple as that. Thanks :)

Comments:
cc @Narsil 

Any update? :( 

Still unsolved. [The same problem in the Discord channel.](https://discord.com/channels/879548962464493619/1112875103311630366)

Hi, things should be back to normal, can you confirm ?

This model and some others were still running very old code, and some internal changes have made it crash, unfortunately silently for us, since everything was still 200 status codes somehow.

We merely updated everything.
Thanks @oOraph for the fix.

I will check now, also a question: will there be additional parameters added to the voice recognition, something besides file (e.g. wait_for_model or output_language)
[About additional parameters](https://discord.com/channels/879548962464493619/1115750221226446889)

@Narsil

![image](https://github.com/huggingface/transformers/assets/100136305/08decc4c-a1fc-4654-a6ed-cd9f732f8d87)



openai/whisper-large-v2 worked right away, but still in English, although the source was in Russian.
![image](https://github.com/huggingface/transformers/assets/100136305/c393e9fa-2616-4d0d-bc77-c741eb798a31)


Still getting this error:

`{
    "error": "Model *** is currently loading",
    "estimated_time": 20
}`

Any help?

Hi @KennStack01 :), just made the test, if you haven't requested your model for a while it 's offline so you get this message (flagged as an error so that it cannot be confused with an answer to your prompt but it's not really an error in the sense your model won't work). I just tested your model and it got online in 20-22 seconds the two times I tried. So you get this message for sth like 20 s (sometimes more for bigger models but yours should be really fast to load given its size). Once it's online your prompt gets correctly answered :) (unless I misunderstood what you're saying and you're saying sometimes it just does not load at all, which I did not observe but would still be possible). And once online, it will stay online for a while, especially if you're requesting it regularly. But at some point it will get offline again and be preempted by others. More or less quickly, depending on several factors, essentially the whole cluster's load, the last time it was requested and the resources it consumes, sometimes only a few minutes later: this could explain your test @Zapzatron. Because from what I understand, in your test, you requested openai/whisper-large once and took the estimated time in response to know how long to sleep (which makes total sense). But from the tests I made, the estimated_time provided in answer was bad and it actually got online in less than 247s (in sth like 70s). Since you did not request it, it could actually have been already brought offline by the time you ended your sleep and requested it again. I would suggest that you actually poll the api a bit more frequently, sth like every 20 seconds after the first message, and would not pay too much attention to the estimated_time to know how long to sleep :)

[I tried every 20 seconds, but that's a lot of requests](https://discord.com/channels/879548962464493619/1112875103311630366)
![image](https://github.com/huggingface/transformers/assets/100136305/72cc7f6d-5ad3-4a16-b89c-8be5c19794a7)



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

It's more than 10 minutes I am getting:

is currently loading","estimated_time":300.8891906738281}

```
async function query(data) {
	const response = await fetch(
		"https://api-inference.huggingface.co/models/ZeroWw/Mistral-7B-Instruct-v0.3.int8",
		{
			headers: { Authorization: "Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" },
			method: "POST",
			body: JSON.stringify(data),
		}
	);
	const result = await response.json();
	return result;
}

query({"inputs": "Can you please let us know more details about your ai?"}).then((response) => {
	console.log(JSON.stringify(response));
});
```

I am performing simple question answering task and still facing this unresolved issue.

{'error': 'Model deepset/roberta-base-squad2 is currently loading', 'estimated_time': 20.0}
{'error': 'Model deepset/roberta-base-squad2 is currently loading', 'estimated_time': 20.0}
{'error': 'Model deepset/roberta-base-squad2 is currently loading', 'estimated_time': 20.0}
continues...


Hi everyone, I tried generating some answers with different models...the last being llama and I also get the same response: 

```
{
    "error": "Model meta-llama/Llama-2-70b-chat is currently loading",
    "estimated_time": 20.0
}
```
I thought there must be something wrong with my API call until I made a `cURL` request and it happened again. That's what brought me here. 

If it helps, in the browser, I'm getting a `503 error`.

I hope there is a fix for this soon. At the moment, I am unable to use the API.



I'm currently getting this with various models as well and it never resolves.

```
{"error":"Model bartowski/Tiger-Gemma-9B-v1-GGUF is currently loading","estimated_time":20.0}
```

That 20 is a blatant lie unless it means 20 hours.

I've created a new issue #32665.