[torch.compile] Remove compiler related configs from model_kwargs
Author: anijain2305
Pull request: yes

# What does this PR do?

This allows us to add arguments `disable_compile` and `compile_config` while calling `from_pretrained` and control `torch.compile` from the user program.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc @gante @ArthurZucker 

cc @zucchini-nlp 

Another ping @gante 

@zucchini-nlp I am trying to pass it to GenerationConfig.from_pretrained here - https://github.com/anijain2305/transformers/blob/0b0ed032d0d5ba995595ba08d18f03dd8963ab57/src/transformers/modeling_utils.py#L4538

The idea is to control the disable_compile and compile_config from user code. This is what I am trying to achieve

```
rom transformers import AutoProcessor, Gemma3ForConditionalGeneration
from PIL import Image
import requests
import torch

from transformers.generation import CompileConfig


eager_compile_config = CompileConfig(backend="eager")


model_id = "google/gemma-3-4b-it"

# Override the compile_config
model = Gemma3ForConditionalGeneration.from_pretrained(
    model_id, device_map="auto", compile_config=eager_compile_config
).eval()

# Disable compilation

model = Gemma3ForConditionalGeneration.from_pretrained(
    model_id, device_map="auto", disable_compile=True
).eval()

```

However, If you have some other way to control the compile without changing the transformers source code, then please let me know. In addition to the obvious ease in controlling torch.compile, it can also help us write benchmarks to track compile + HF performance.

@anijain2305 Currently it is not possible to overwrite generation config params when loading from pretrained in that way. Generation config can be completely overriden by passing a new config `model = AutoModelForCausalLM.from_pretrained('meta-llama/Llama-3.2-1B-Instruct', generation_config=my_config)`

To overwrite certain keys only, you can workaround by changing them after the model is loaded

```python

model = Gemma3ForConditionalGeneration.from_pretrained(
    model_id, device_map="auto"
).eval()
model.generation_config.disable_compile=True
```

I realize this is not a perfect solution and n longer term @gante had planned to load model's own generation config separate from model config, which probably would allow overwriting only certain keys. I will let @gante share the plans or suggest a better workaround :)


In addition to what @zucchini-nlp wrote:

Most models don't compile by default, and only compile when a compilable cache is set (e.g. `model.generate(..., cache_implementation="static")`). The logic is "if a user is passing a compilable cache, then they want compilation".

However, some recent models (like `gemma3`) only work with compilable caches. This is messing our current interface, as it makes compilation trigger by default. 

This is currently a big source of problems. I'm considering disabling automatic compilation or finding a more controllable interface. For now, the suggesting above should work.

Ok, this sounds good. And thanks for more background. I did not know about `cache_implementation="static"`.