GPT2 model shows inplace operation error when using input_embeds that come from a nn.Linear
Author: retepViolet
Pull request: no
Labels: name: Usage

name: bug

### Describe the bug

GPT2 model shows inplace operation error when using input_embeds that come from a nn.Linear.
```
embeds = self.decoder(embeds)
return self.gpt(inputs_embeds = embeds, labels = labels, attention_mask = attention_mask)
```
self.decoder is a nn.linear.
If I replace `inputs_embeds = embeds` with `inputs_embeds = self.layer_norm(embeds)` the error disappears.

### Reproduction

_No response_

### Logs

```shell
File "/content/gdrive/Othercomputers/retep's PC/tvae/trainer.py", line 96, in <module>
    trainer.train()
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2052, in train
    return inner_training_loop(
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2388, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 3485, in training_step
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 3532, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/utils/operations.py", line 820, in forward
    return model_forward(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/utils/operations.py", line 808, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/usr/local/lib/python3.10/dist-packages/torch/amp/autocast_mode.py", line 43, in decorate_autocast
    return func(*args, **kwargs)
  File "/content/gdrive/Othercomputers/retep's PC/tvae/tvae.py", line 101, in forward
    return self.decode(embeds, gpt_mask, labels)
  File "/content/gdrive/Othercomputers/retep's PC/tvae/tvae.py", line 95, in decode
    return self.gpt(inputs_embeds = embeds, labels = labels,
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/adapters/models/gpt2/adapter_model.py", line 72, in forward
    outputs, context = self.transformer(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/adapters/context.py", line 116, in wrapper_func
    results = f(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/adapters/model_mixin.py", line 1470, in forward
    return super().forward(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 1130, in forward
    outputs = block(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1603, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/adapters/models/gpt2/modeling_gpt2.py", line 206, in forward
    attn_outputs = self.attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/adapters/models/gpt2/modeling_gpt2.py", line 171, in forward
    attn_output = torch.nn.functional.scaled_dot_product_attention(
 (Triggered internally at ../torch/csrc/autograd/python_anomaly_mode.cpp:111.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "/content/gdrive/Othercomputers/retep's PC/tvae/trainer.py", line 96, in <module>
    trainer.train()
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2052, in train
    return inner_training_loop(
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2388, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 3518, in training_step
    self.accelerator.backward(loss, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 2192, in backward
    self.scaler.scale(loss).backward(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_tensor.py", line 521, in backward
    torch.autograd.backward(
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/__init__.py", line 289, in backward
    _engine_run_backward(
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/graph.py", line 769, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [32, 12, 128, 128]] is at version 6; expected version 5 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
```


### System info

```shell
- huggingface_hub version: 0.24.7
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Running in iPython ?: Yes
- iPython shell: Shell
- Running in notebook ?: Yes
- Running in Google Colab ?: Yes
- Token path ?: /root/.cache/huggingface/token
- Has saved token ?: False
- Configured git credential helpers: 
- FastAI: 2.7.17
- Tensorflow: 2.17.0
- Torch: 2.4.1+cu121
- Jinja2: 3.1.4
- Graphviz: 0.20.3
- keras: 3.4.1
- Pydot: 3.0.2
- Pillow: 10.4.0
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: 1.26.4
- pydantic: 2.9.2
- aiohttp: 3.10.8
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /root/.cache/huggingface/hub
- HF_ASSETS_CACHE: /root/.cache/huggingface/assets
- HF_TOKEN_PATH: /root/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10

{'huggingface_hub version': '0.24.7',
 'Platform': 'Linux-6.1.85+-x86_64-with-glibc2.35',
 'Python version': '3.10.12',
 'Running in iPython ?': 'Yes',
 'iPython shell': 'Shell',
 'Running in notebook ?': 'Yes',
 'Running in Google Colab ?': 'Yes',
 'Token path ?': '/root/.cache/huggingface/token',
 'Has saved token ?': False,
 'Configured git credential helpers': '',
 'FastAI': '2.7.17',
 'Tensorflow': '2.17.0',
 'Torch': '2.4.1+cu121',
 'Jinja2': '3.1.4',
 'Graphviz': '0.20.3',
 'keras': '3.4.1',
 'Pydot': '3.0.2',
 'Pillow': '10.4.0',
 'hf_transfer': 'N/A',
 'gradio': 'N/A',
 'tensorboard': 'N/A',
 'numpy': '1.26.4',
 'pydantic': '2.9.2',
 'aiohttp': '3.10.8',
 'ENDPOINT': 'https://huggingface.co',
 'HF_HUB_CACHE': '/root/.cache/huggingface/hub',
 'HF_ASSETS_CACHE': '/root/.cache/huggingface/assets',
 'HF_TOKEN_PATH': '/root/.cache/huggingface/token',
 'HF_HUB_OFFLINE': False,
 'HF_HUB_DISABLE_TELEMETRY': False,
 'HF_HUB_DISABLE_PROGRESS_BARS': None,
 'HF_HUB_DISABLE_SYMLINKS_WARNING': False,
 'HF_HUB_DISABLE_EXPERIMENTAL_WARNING': False,
 'HF_HUB_DISABLE_IMPLICIT_TOKEN': False,
 'HF_HUB_ENABLE_HF_TRANSFER': False,
 'HF_HUB_ETAG_TIMEOUT': 10,
 'HF_HUB_DOWNLOAD_TIMEOUT': 10}
```


Comments:
This looks like a `transformers` issue, not a `huggingface_hub` one. `huggingface_hub` is the underlying library that handles only upload/download files from the Hub. I'll transfer your issue and ping @ArthurZucker.  In the meantime, could you provide more details about what you are trying to achieve and especially a full example to reproduce the issue? Thanks!

cc @Rocketknight1 if you have the bandwidth to take a look?

Definitely willing to take a look, but seconding @wauplin's request for a full script to reproduce the issue first!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.