[Kosmos-2.5] RuntimeError: a leaf Variable that requires grad is being used in an in-place operation during training
Author: Neuro1729
Pull request: no
Labels: name: bug

### System Info

Environment Info
Transformers version: 5.0.0.dev0

Platform: Linux-6.6.87.1-microsoft-standard-WSL2-x86_64-with-glibc2.35

Python version: 3.11.10

PyTorch version (GPU?): 2.9.1+cu128 

Accelerate version: 1.12.0

PEFT version: 0.18.1

GPU models: NVIDIA RTX A5000

### Who can help?

@zucchini-nlp  @BenjaminBossan @yonigozlan

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

problem description : Fine-tuning microsoft/kosmos-2.5 with Trainer and LoRA (PEFT) fails at the first training step with a PyTorch autograd error.

During the forward pass, Kosmos2_5ForConditionalGeneration performs an in-place assignment on inputs_embeds, which is a leaf tensor that requires gradients when training with LoRA and gradient checkpointing enabled. This causes the following runtime error:

````markdown
### Code sample

```python
import os
import sys

# Force usage of local editable transformers
local_transformers_path = os.path.abspath("./transformers/src")
sys.path.insert(0, local_transformers_path)

os.environ["CUDA_VISIBLE_DEVICES"] = "0"
os.environ["NCCL_P2P_DISABLE"] = "1"
os.environ["NCCL_IB_DISABLE"] = "1"

import re
import torch
from datasets import load_dataset
from transformers import (
    AutoProcessor,
    Kosmos2_5ForConditionalGeneration,
    TrainingArguments,
    Trainer,
)
from peft import LoraConfig, get_peft_model

import transformers
print(f"Transformers version: {transformers.__version__}")
print(f"Loading from: {transformers.__file__}")

# Dataset
dataset = load_dataset("merve/doclaynet-small")["test"]
dataset = dataset.train_test_split(test_size=0.1)
train_ds = dataset["train"]
test_ds = dataset["test"]

# Model
model = Kosmos2_5ForConditionalGeneration.from_pretrained(
    "microsoft/kosmos-2.5",
    torch_dtype=torch.bfloat16,
)

model.config.use_cache = False
model.gradient_checkpointing_enable()

# LoRA config
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "v_proj"],
)

model = get_peft_model(model, lora_config)

processor = AutoProcessor.from_pretrained("microsoft/kosmos-2.5")

def collate_fn(examples):
    images = [ex["image"].convert("RGB") for ex in examples]
    texts = ["<ocr>"] * len(images)

    inputs = processor(
        images=images,
        text=texts,
        return_tensors="pt",
        padding=True,
    )

    labels = inputs["input_ids"].clone()
    labels[:] = -100
    inputs["labels"] = labels
    return inputs

training_args = TrainingArguments(
    output_dir="kosmos2_5_output",
    num_train_epochs=1,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    learning_rate=2e-5,
    bf16=True,
    logging_steps=1,
    save_strategy="no",
    report_to="none",
    remove_unused_columns=False,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_ds,
    eval_dataset=test_ds,
    data_collator=collate_fn,
)

trainer.train()
````

---

### Error / stack trace

```text
RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

Traceback (most recent call last):
  File "/workspace/kosmos.py", line 161, in <module>
    trainer.train()
  File "/workspace/transformers/src/transformers/trainer.py", line 2174, in train
    return inner_training_loop(
  File "/workspace/transformers/src/transformers/trainer.py", line 2536, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
  File "/workspace/transformers/src/transformers/trainer.py", line 3809, in training_step
    loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
  File "/workspace/transformers/src/transformers/trainer.py", line 3880, in compute_loss
    outputs = model(**inputs)
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.11/site-packages/peft/peft_model.py", line 1923, in forward
    return self.base_model(*args, **kwargs)
  File "/workspace/transformers/src/transformers/models/kosmos2_5/modeling_kosmos2_5.py", line 1094, in forward
    inputs_embeds[image_embeds_position_mask == 1] = image_embeds.to(inputs_embeds.device).view(...)
RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.
```

### Expected behavior

```markdown
Training should run successfully when fine-tuning `Kosmos2_5ForConditionalGeneration` with LoRA using `Trainer`, without crashing due to in-place operations on leaf tensors.

Comments: