transformers offline model loading is not working from version 4.40.0 for models without safetensors
Author: sukumarburra
Pull request: no

### System Info

```
- `transformers` version: 4.40.1
- Platform: macOS-14.0-arm64-arm-64bit
- Python version: 3.10.14
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
```

### Who can help?

@ArthurZucker  
@younesbelkada 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Model : `Helsinki-NLP/opus-mt-en-fr`

I already have the model under hugging-face cache directory  at `~/.cache/huggingface/hub/models--Helsinki-NLP--opus-mt-en-fr/` . When tried to run following script in **offline mode without network connectivity intentionally **,  it is trying to reach `huggingface.co` and failing with error. 

I exported ENV `HF_HUB_OFFLINE=1` .

```
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, pipeline
tokenizer = AutoTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-fr')
model = AutoModelForSeq2SeqLM.from_pretrained('Helsinki-NLP/opus-mt-en-fr')
translator = pipeline("translation", model=model, tokenizer=tokenizer)

line="This is example"
print(translator(line)[0]['translation_text'])
```

Here is the error:
```
    model = AutoModelForSeq2SeqLM.from_pretrained('Helsinki-NLP/opus-mt-en-fr')
  File "/Users/sukumar.burra/.env/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained
    return model_class.from_pretrained(
  File "/Users/sukumar.burra/.env/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3370, in from_pretrained
    if not has_file(pretrained_model_name_or_path, safe_weights_name, **has_file_kwargs):
  File "/Users/sukumar.burra/.env/lib/python3.10/site-packages/transformers/utils/hub.py", line 627, in has_file
    r = requests.head(url, headers=headers, allow_redirects=False, proxies=proxies, timeout=10)
  File "/Users/sukumar.burra/.env/lib/python3.10/site-packages/requests/api.py", line 100, in head
    return request("head", url, **kwargs)
  File "/Users/sukumar.burra/.env/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/sukumar.burra/.env/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/sukumar.burra/.env/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/Users/sukumar.burra/.env/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /Helsinki-NLP/opus-mt-en-fr/resolve/main/model.safetensors (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x17f9ede10>: Failed to resolve 'huggingface.co' ([Errno 8] nodename nor servname provided, or not known)"))
```

Model loading and offline inference (without internet/network) used to work with transformers package till `4.39.3` for model `Helsinki-NLP/opus-mt-en-fr` . It started failing from `tranformers 4.40.0`

I also see the same issue with another model `cardiffnlp/twitter-roberta-base-sentiment-latest`
I noticed, this issue is only observed with models which do not have safetensors version of the model in hugging face repo.

Is this something intentional or due to issue in latest transformers version: 4.40.0 ?
Also, Is there any workaround for this ?


### Expected behavior

The expectation is offline loading and inference for huggingface models should with latest transformers packages 
 
With ENV HF_HUB_OFFLINE=1 , If you already have model available under huggingface cache, it should not try to reach the network. 

Comments:
cc @LysandreJik as it's hitting automatic safetensors weight conversion code

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This should now be fixed!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I still see this problem with transformers 4.41.2 version. The fix https://github.com/huggingface/transformers/pull/31016 is not part of 4.41.2 version. Can we expect it in next version ? Also, When will the next version available ?

@sukumarburra The fix will be included in the next release, which should be very soon, probably this week! In the meantime, if you want to get the latest changes on main, you can install directly from source with `pip install git+https://github.com/huggingface/transformers`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.