🐛 Bug Report: Accelerate config to disable torch dynamo is ignored by transformers automatic compilation
Author: leobianco
Pull request: no

### System Info

- transformers version: 4.53.0
- platform: Debian 6.1.128-1 x86_64 GNU/Linux
- python version: 3.11
- torch version: 2.7.1
- trl version: 0.19.0
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes, DeepSpeed ZeRO 3

### Who can help?

@SunMarc @gante

### Information

The problem arises when using:
- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

The tasks I am working on are:
- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (see below)

### Reproduction

Although my accelerate config was explicitly set to disable torch dynamo, I found that transformers (after commit `ee37bf0`) still enabled automatic compilation (e.g., torch.compile/torch._dynamo) during the forward pass. This led to unexpected compilation and the following error when using the RLOO trainer from the TRL library to finetune a Gemma 2 model:

```python
[rank5]: Traceback (most recent call last):
[rank5]:   File "/home/leo/perl_hallucination/perl.py", line 120, in <module>
[rank5]:     main()
[rank5]:   File "/home/leo/perl_hallucination/perl.py", line 109, in main
[rank5]:     trainer.train()
[rank5]:   File "/home/leo/perl_hallucination/.venv/lib/python3.11/site-packages/trl/trainer/rloo_trainer.py", line 337, in train
[rank5]:     query_responses, logitss = batch_generation(
[rank5]:                                ^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/leo/perl_hallucination/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank5]:     return func(*args, **kwargs)
[rank5]:            ^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/leo/perl_hallucination/.venv/lib/python3.11/site-packages/trl/trainer/utils.py", line 1430, in batch_generation
[rank5]:     query_response, logits = generate(
[rank5]:                              ^^^^^^^^^
[rank5]:   File "/home/leo/perl_hallucination/.venv/lib/python3.11/site-packages/trl/trainer/utils.py", line 1404, in generate
[rank5]:     output = lm_backbone.generate(
[rank5]:              ^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/leo/perl_hallucination/.venv/lib/python3.11/site-packages/peft/peft_model.py", line 1875, in generate
[rank5]:     outputs = self.base_model.generate(*args, **kwargs)
[rank5]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/leo/perl_hallucination/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank5]:     return func(*args, **kwargs)
[rank5]:            ^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/leo/perl_hallucination/.venv/lib/python3.11/site-packages/transformers/generation/utils.py", line 2623, in generate
[rank5]:     result = self._sample(
[rank5]:              ^^^^^^^^^^^^^
[rank5]:   File "/home/leo/perl_hallucination/.venv/lib/python3.11/site-packages/transformers/generation/utils.py", line 3607, in _sample
[rank5]:     outputs = model_forward(**model_inputs, return_dict=True)
[rank5]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/leo/perl_hallucination/.venv/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 655, in _fn
[rank5]:     return fn(*args, **kwargs)
[rank5]:            ^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/leo/perl_hallucination/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank5]:     return self._call_impl(*args, **kwargs)
[rank5]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/leo/perl_hallucination/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank5]:     return forward_call(*args, **kwargs)
[rank5]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/leo/perl_hallucination/.venv/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 1432, in __call__
[rank5]:     return self._torchdynamo_orig_callable(
[rank5]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]:   File "/home/leo/perl_hallucination/.venv/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 598, in __call__
[rank5]:     return _compile(
[rank5]:            ^^^^^^^^^
[rank5]:   File "/home/leo/perl_hallucination/.venv/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 981, in _compile
[rank5]:     raise FailOnRecompileLimitHit(
[rank5]: torch._dynamo.exc.FailOnRecompileLimitHit: recompile_limit reached with one_graph=True. Excessive recompilations can degrade performance due to the compilation overhead of each recompilation. To monitor recompilations, enable TORCH_LOGS=recompiles. If recompilations are expected, consider increasing torch._dynamo.config.cache_size_limit to an appropriate value. 
```

[The docstring](https://github.com/huggingface/transformers/blame/8178c43112295bf8c4ef04c667efbbbfd34b8bca/src/transformers/generation/configuration_utils.py#L386-L386C2) asks us to open an issue if one wishes to disable this.

This was a bit confusing to debug, as I expected my configuration to be respected. In my view, when the accelerate config disables torch dynamo, any form of automatic compilation in downstream libraries should also be disabled by default.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.