Tensor item cannot be called
Author: lumideez
Pull request: no
Labels: name: bug

### System Info

NVIDIA GeForce RTX 5060 Ti with CUDA capability sm_120 is not compatible with the current PyTorch installation.                                                                                                                                                                                                                                           The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_61 sm_70 sm_75 sm_80 sm_86 sm_90.                                                                                                                                                                                                                                                   If you want to use the NVIDIA GeForce RTX 5060 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         warnings.warn(                                                                                                                                                                                                                                                                                                                                          Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                          File "C:\HibarrTranslator\test_load.py", line 21, in <module>                                                                                                                                                                                                                                                                                               model = AutoModelForCausalLM.from_pretrained(                                                                                                                                                                                                                                                                                                           File "C:\HibarrTranslator\venv\lib\site-packages\transformers\models\auto\auto_factory.py", line 366, in from_pretrained                                                                                                                                                                                                                                    return model_class.from_pretrained(                                                                                                                                                                                                                                                                                                                     File "C:\HibarrTranslator\venv\lib\site-packages\transformers\modeling_utils.py", line 3929, in from_pretrained                                                                                                                                                                                                                                             model = cls(config, *model_args, **model_kwargs)                                                                                                                                                                                                                                                                                                        File "C:\Users\ADMIN\.cache\huggingface\modules\transformers_modules\microsoft\Phi_hyphen_4_hyphen_multimodal_hyphen_instruct\93f923e1a7727d1c4f446756212d9d3e8fcc5d81\modeling_phi4mm.py", line 1942, in __init__                                                                                                                                          self.model = Phi4MMModel(config)                                                                                                                                                                                                                                                                                                                        File "C:\Users\ADMIN\.cache\huggingface\modules\transformers_modules\microsoft\Phi_hyphen_4_hyphen_multimodal_hyphen_instruct\93f923e1a7727d1c4f446756212d9d3e8fcc5d81\modeling_phi4mm.py", line 1633, in __init__                                                                                                                                          self.embed_tokens_extend = Phi4MMImageAudioEmbedding(config, **embedding_config)                                                                                                                                                                                                                                                                        File "C:\Users\ADMIN\.cache\huggingface\modules\transformers_modules\microsoft\Phi_hyphen_4_hyphen_multimodal_hyphen_instruct\93f923e1a7727d1c4f446756212d9d3e8fcc5d81\modeling_phi4mm.py", line 673, in __init__                                                                                                                                           self.audio_embed = Phi4MMAudioEmbedding(config, **self.audio_embd_layer_kwargs)                                                                                                                                                                                                                                                                         File "C:\Users\ADMIN\.cache\huggingface\modules\transformers_modules\microsoft\Phi_hyphen_4_hyphen_multimodal_hyphen_instruct\93f923e1a7727d1c4f446756212d9d3e8fcc5d81\modeling_phi4mm.py", line 488, in __init__                                                                                                                                           self.encoder = ConformerEncoder(**encoder_config)                                                                                                                                                                                                                                                                                                       File "C:\Users\ADMIN\.cache\huggingface\modules\transformers_modules\microsoft\Phi_hyphen_4_hyphen_multimodal_hyphen_instruct\93f923e1a7727d1c4f446756212d9d3e8fcc5d81\speech_conformer_encoder.py", line 2746, in __init__                                                                                                                                 super().__init__(                                                                                                                                                                                                                                                                                                                                       File "C:\Users\ADMIN\.cache\huggingface\modules\transformers_modules\microsoft\Phi_hyphen_4_hyphen_multimodal_hyphen_instruct\93f923e1a7727d1c4f446756212d9d3e8fcc5d81\speech_conformer_encoder.py", line 2298, in __init__                                                                                                                                 self.embed = NemoConvSubsampling(                                                                                                                                                                                                                                                                                                                       File "C:\Users\ADMIN\.cache\huggingface\modules\transformers_modules\microsoft\Phi_hyphen_4_hyphen_multimodal_hyphen_instruct\93f923e1a7727d1c4f446756212d9d3e8fcc5d81\speech_conformer_encoder.py", line 1435, in __init__                                                                                                                                 self.out = torch.nn.Linear(conv_channels * int(out_length), feat_out)                                                                                                                                                                                                                                                                                   File "C:\HibarrTranslator\venv\lib\site-packages\torch\utils\_device.py", line 106, in __torch_function__                                                                                                                                                                                                                                                   return func(*args, **kwargs)                                                                                                                                                                                                                                                                                                                            File "C:\HibarrTranslator\venv\lib\site-packages\torch\_meta_registrations.py", line 6471, in meta_local_scalar_dense                                                                                                                                                                                                                                       raise RuntimeError("Tensor.item() cannot be called on meta tensors")                                                                                                                                                                                                                                                                                  RuntimeError: Tensor.item() cannot be called on meta tensors

### Who can help?

@lumideez

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

pip install bitsandbytes   
pip install git+https://github.com/huggingface/transformers.git
pip install git+https://github.com/huggingface/accelerate.git      

### Expected behavior

Was supposed to show 'Success' on the cli

Comments:
opening a PR shortly to fix this.


```
"C:\Users\ADMIN.cache\huggingface\modules\transformers_modules\microsoft\Phi_hyphen_4_hyphen_multimodal_hyphen_instruct\93f923e1a7727d1c4f446756212d9d3e8fcc5d81\speech_conformer_encoder.py", line 1435, in init self.out =
```

this does not look related to `transformers`no? more like remote code?

agree with @ArthurZucker 

I verified this behavior is driven by remote model code, not the transformer implementation.
With trust_remote_code=True, AutoConfig and AutoModel resolve to classes under ~/.cache/huggingface/modules/transformers_modules/microsoft/... and  not transformers.models.*.

the remote config/model code sets defaults that trigger the observed failures. In particular:
	•	Locally, the remote config enables FlashAttention2 by default, which fails if flash_attn is not installed.
	•	In the reported stack trace, the crash happens inside speech_conformer_encoder.py from the remote repository, due to a meta-tensor .item() call.

this suggests the issue should be handled in remote code rather than in transformers core.

closing!