Automatic compilation in generate: do not rely on inner function
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As discussed, I don't think we should rely on defining an inner function and compiling it for every call to `generate`.

This moves the compiled forward to `PreTrainedModel` instead for reuse, which makes the most sense for me. That way, every `PreTrainedModel` effectively has 2 forwards, and we can dynamically choose between non-compiled (prefill) and compiled (iterative decoding). This is similar to what PyTorch does internally when calling `model.compile()` inplace, except in their case they force the use of the compiled one after the call was invoked.

Let me know what you think @ArthurZucker 

Also address https://github.com/huggingface/transformers/issues/34906

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34923). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey @BenjaminBossan, could you check if this PR solves the issue in `peft`, or point me to the failing tests please?  

Raised this point in issue #34906 as well, how can we pass custom arguments to compile in this case, say different backend or other parameters?
There should be a way to pass the user arguments for non-default cases.

Yeah we can use `generate_kwargs` for this. The original implem was super minimal for big gains, but down to include that!

another comments:

- (nit) I am not sure it's necessary to save the compile config (unlike our other config classes)
- I understand we might want to change compile options.  But now imagine a use case: I compile with option 1. Then with option 2. But now I want to check with option 1 again for some reason. **Question**: would  compiling with option 1 the second time take the same amount time of compiling with option 1 the first time?


Thanks for looking into this as well @ydshieh! Regarding your question, `torch` is able to cache different graphs for the exact same function, so no, it will actually not re-compile even after the switch. Not sure how many times you can do it before starting to loose cache entries though. But the following:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, CompileConfig
import torch
import time
device = 3
import warnings
warnings.filterwarnings("ignore")

model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B", torch_dtype=torch.float16).to(device)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")

sequence = "Hey what's the plan"

inputs = tokenizer.encode(sequence, return_tensors='pt').to(device)
model.generation_config.temperature = 1.0
model.generation_config.top_p = 1.0

# Compile default config
t0 = time.time()
out = model.generate(inputs, do_sample=False, max_new_tokens=500, cache_implementation="static")
out = tokenizer.batch_decode(out, skip_special_tokens=True)[0]
dt = time.time() - t0
print(f'dt: {dt}')
print(model._last_compile_config.to_dict())


# Compile new config
t0 = time.time()
out = model.generate(inputs, do_sample=False, max_new_tokens=500, cache_implementation="static", compile_config=CompileConfig(dynamic=True))
out = tokenizer.batch_decode(out, skip_special_tokens=True)[0]
dt = time.time() - t0
print(f'dt: {dt}')
print(model._last_compile_config.to_dict())

# Back to 1st config
t0 = time.time()
out = model.generate(inputs, do_sample=False, max_new_tokens=500, cache_implementation="static")
out = tokenizer.batch_decode(out, skip_special_tokens=True)[0]
dt = time.time() - t0
print(f'dt: {dt}')
print(model._last_compile_config.to_dict())


# Back to 2nd config
t0 = time.time()
out = model.generate(inputs, do_sample=False, max_new_tokens=500, cache_implementation="static", compile_config=CompileConfig(dynamic=True))
out = tokenizer.batch_decode(out, skip_special_tokens=True)[0]
dt = time.time() - t0
print(f'dt: {dt}')
print(model._last_compile_config.to_dict())
```

actually does not re-compile for the last 2 `generate` calls, and use already compiled graphs 

Confirming that [this script](https://gist.github.com/Cyrilvallez/aa8fc622d698f1425615a6651d8ff3a9) returns
```py
>>> Without static cache and compile: 27.079 s
>>> Using `torch.compile`.
>>> Compiling default config: 26.816 s
>>> Using compiled graph: 6.541 s
>>> Compiling new config: 24.327 s
>>> Using compiled new graph: 6.930 s
>>> Back to 1st config and graph: 6.528 s
```
on a machine with A100 GPU, which is the expected result