`AutoModelForCasualLM.from_pretrained()` exits without warning/error
Author: ccocks
Pull request: no
Labels: name: bug

### System Info

`transformers `4.49.0
`Python `3.11.9

### Who can help?

@ArthurZucker @gante 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

This is what I ran:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import os
current_dir = os.path.dirname(os.path.abspath(__file__))
tokenizer = AutoTokenizer.from_pretrained(
    current_dir,
    use_safetensors=True,
)
print("tokenizer loaded")
model = AutoModelForCausalLM.from_pretrained(
    current_dir,
    use_safetensors=True,
)
print("model loaded")
input_ids = tokenizer("Hello, whats 23*6/4?", return_tensors="pt").input_ids
response = model.generate(
    input_ids,
    temperature=0.3,
)
print("response generated")
print(tokenizer.decode(response[0]))
```

I get "tokenizer loaded" but not "model loaded". the python script exits (unerrored) before that.
The current directory of the file was produced by `git clone https://huggingface.co/deca-ai/2-mini-beta` which **does include** a model.safetensors.index.json.
Looking at Task Manager, memory never goes higher than 200-300MB, so I'm not sure if it's even getting there.


### Expected behavior

The model should run like normal (even if it's slow)

Comments:
Hi, we might need a bit more info to help diagnose this! Can you:

1) Try loading the model from the Hub directly instead of a local folder
2) Try loading another model of a similar size (to make sure it's not just a memory issue)
3) Try running this on a non-Windows machine

Ok, I can't do number 3, but for the other two,
1. Where will the model be stored if I load it directly from the Hub? Will it be cached so I can use it again without re-downloading.
2. I think I have enough RAM. By the way, I am not using a GPU. Also, as I mentioned, RAM usage never goes above 200-300MB. Even if it was a memory issue, shouldn't it error?

________________________________
From: Matt ***@***.***>
Sent: Tuesday, February 18, 2025 6:19 AM
To: huggingface/transformers ***@***.***>
Cc: blazgocompany ***@***.***>; Author ***@***.***>
Subject: Re: [huggingface/transformers] `AutoModelForCasualLM.from_pretrained()` exits without warning/error (Issue #36245)


Hi, we might need a bit more info to help diagnose this! Can you:

  1.  Try loading the model from the Hub directly instead of a local folder
  2.  Try loading another model of a similar size (to make sure it's not just a memory issue)
  3.  Try running this on a non-Windows machine

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/transformers/issues/36245#issuecomment-2665852846>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AXA4YJJF2DXSOBRRUAKTVGL2QM6PNAVCNFSM6AAAAABXKL2YGKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRVHA2TEOBUGY>.
You are receiving this because you authored the thread.Message ID: ***@***.***>

[Rocketknight1]Rocketknight1 left a comment (huggingface/transformers#36245)<https://github.com/huggingface/transformers/issues/36245#issuecomment-2665852846>

Hi, we might need a bit more info to help diagnose this! Can you:

  1.  Try loading the model from the Hub directly instead of a local folder
  2.  Try loading another model of a similar size (to make sure it's not just a memory issue)
  3.  Try running this on a non-Windows machine

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/transformers/issues/36245#issuecomment-2665852846>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AXA4YJJF2DXSOBRRUAKTVGL2QM6PNAVCNFSM6AAAAABXKL2YGKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRVHA2TEOBUGY>.
You are receiving this because you authored the thread.Message ID: ***@***.***>


The model is stored in the huggingface cache, when you load from the Hub, so you won't need to redownload it after the first time. On Linux/Mac the cache is in `~/.cache/huggingface`, on Windows I'm not sure but I suspect it lives in `%appdata%`.

Sorry for the very late reply. Same error occurs when using directly

Oh no. I closed it by accident. Can someone reopen?

@blazgocompany can you give us some reproducer code that we can run to demonstrate the bug? This means the code should load models from the Hub, because we don't have your saved models locally! 

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import os
current_dir = os.path.dirname(os.path.abspath(__file__))
tokenizer = AutoTokenizer.from_pretrained(
    "deca-ai/2-mini",
    use_safetensors=True,
)
print("tokenizer loaded")
model = AutoModelForCausalLM.from_pretrained(
    "deca-ai/2-mini",
    use_safetensors=True,
)
print("model loaded")
input_ids = tokenizer("Hello, whats 23*6/4?", return_tensors="pt").input_ids
response = model.generate(
    input_ids,
    temperature=0.3,
)
print("response generated")
print(tokenizer.decode(response[0]))

```

Deca 2 mini is on the Hub

Also, when trying from A100 GPU I get:

```
Failed to import transformers.models.qwen2.modeling_qwen2 because of the following error (look up to see its traceback):
operator torchvision::nms does not exist
```

Hi @blazgocompany 👋 

`deca-ai/2-mini` is a [qwen 2 model](https://huggingface.co/deca-ai/2-mini/blob/main/config.json#L4), so the snippet below should be equivalent -- same model architecture.

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "Qwen/Qwen2-0.5B-Instruct",
    use_safetensors=True,
)
print("tokenizer loaded")
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2-0.5B-Instruct",
    use_safetensors=True,
)
print("model loaded")
input_ids = tokenizer("Hello, whats 23*6/4?", return_tensors="pt").input_ids
response = model.generate(
    input_ids,
    temperature=0.3,
)
print("response generated")
print(tokenizer.decode(response[0]))
```

On my end, on `main`, I get no exceptions.

______________________________

Meanwhile, have you looked into [this issue](https://discuss.pytorch.org/t/runtimeerror-operator-torchvision-nms-does-not-exist/192829) from the `torch` forum? (same error message)

Yes that is working. But why doesn't deca 2 work? There is no debug info either!

I have 16gb ram... is that a problem??

From the issue I linked, it seems like you need to update torch-related installation in your machine :)

No, actually I lost access to that GPU (someone kindly had hosted a Jupyter notebook on HF spaces). Do you have any insights about running on CPU? Does it just randomly crash like this?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.