current transformers library does not work with microsoft/Phi-3.5-mini-instruct
Author: datanizing
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.48.2
- Platform: Linux-6.8.0-52-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 0.28.1
- Safetensors version: 0.5.2
- Accelerate version: 1.3.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 4090

### Who can help?

The following Python program does not work:
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = 'microsoft/Phi-3.5-mini-instruct'
model = AutoModelForCausalLM.from_pretrained(model_name,
                                                   revision='main',
                                                   trust_remote_code=False,
                                                   attn_implementation='flash_attention_2',
                                                   torch_dtype=torch.bfloat16,
                                                   use_cache=False,
                                                   device_map="cuda",
                                                   low_cpu_mem_usage=True)
tokenizer = AutoTokenizer.from_pretrained(model_name)
model.eval()
messages = [{ "content": """You are an educated researcher and always answer in correct scientific terms.
                             You are very deep into AI and its methodologies. You are very creative.""",
              "role": "system" },
            { "content": "Write an abstract with the title 'New Training Methods for LLMs'",
              "role": "user" },
            ]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors = "pt").to("cuda")
outputs = model.generate(inputs, max_new_tokens = 512, use_cache = True,
                         do_sample=True, temperature=0.7, top_k=25, top_p=0.8)
```
I get the following error message:
```
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Traceback (most recent call last):
  File "phi3.py", line 21, in <module>
    outputs = model.generate(inputs, max_new_tokens = 512, use_cache = True,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$PYTHON_PATH/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "$PYTHON_PATH/lib/python3.12/site-packages/transformers/generation/utils.py", line 2255, in generate
    result = self._sample(
             ^^^^^^^^^^^^^
  File "$PYTHON_PATH/lib/python3.12/site-packages/transformers/generation/utils.py", line 3254, in _sample
    outputs = self(**model_inputs, return_dict=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$PYTHON_PATH/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$PYTHON_PATH/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$PYTHON_PATH/lib/python3.12/site-packages/transformers/models/phi3/modeling_phi3.py", line 899, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "$PYTHON_PATH/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$PYTHON_PATH/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$PYTHON_PATH/lib/python3.12/site-packages/transformers/models/phi3/modeling_phi3.py", line 605, in forward
    position_embeddings = self.rotary_emb(hidden_states, position_ids)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$PYTHON_PATH/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$PYTHON_PATH/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "$PYTHON_PATH/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "$PYTHON_PATH/lib/python3.12/site-packages/transformers/models/phi3/modeling_phi3.py", line 368, in forward
    freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
             ~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat2 in method wrapper_CUDA_bmm)
```

All other models that I have tried work with this code

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the Python program above.

### Expected behavior

The program should run without an error. It works for other models.

Comments:
Hmm, this seems like the RoPE code is generating tensors and not pushing them to the right device, but when I checked the code it does look like the tensors should be in the right place, so I'm not sure what's going on. cc @Cyrilvallez since the git blame says you're the most recent person to touch the RoPE code there!

Indeed, this was fixed in https://github.com/huggingface/transformers/pull/35608 but did not make it into the patch. Very sorry about that @datanizing! In the meantime, installing from main `pip install git+https://github.com/huggingface/transformers.git` will solve it!

Closing since this is fixed on `main`, but feel free to reopen it if you still have problems!