Stop requiring CacheConfig in GenerationConfig with StaticCache
Author: poedator
Pull request: no
Labels: name: bug

name: Generation

name: Cache

I have an observation that in some common use cases, configuring StaticCache during GenerationConfig initialisation is unnecessary.
it was introduced in https://github.com/huggingface/transformers/pull/32830

Specifically, when the model is used with `.generate`, only the `cache_implementation` config option is relevant. The rest of the cache config is determined inside `generate()`, specifically in `transformers.generation.utils::GenerationMixin._get_cache()`. In that function, StaticCache is created based on the requested generation parameters, ignoring `cache_config` entirely.

**Suggestion**:
- do not require StaticCache config in GenerationConfig.__init__
- have some custom logic for ExecuTorch, that does not affect other use cases
- have default arguments for StaticCache.__init__() so that it would quietly get created with some default parameters (not a good idea though).
- have tests in transformers that set just `cache_implementation="static"` and then call `.generate()`

### current workaround:
do not set cache_implementation in GenerationConfig constructor, but set it afterwards with:
`model.generation_config.cache_implementation = "static"`

**system info**
applies to transformers from september 2024 up to the current 4.46.3

### Who can help?
@gante

Comments:
cc @zucchini-nlp as well!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

up
@zucchini-nlp @gante 

Yeah, it shouldn't be required. Setting `model.generation_config.cache_implementation = "static"` should be enough for most cases. Having a look.

Hi @poedator 👋 

I'm trying to reproduce the issue, but I'm not sure how. Can you help me out? The script below works as expected.

```py
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2", device_map="auto")

inputs = tokenizer(["The quick brown"], return_tensors="pt").to(model.device)
gen_out = model.generate(**inputs, do_sample=False, cache_implementation="static")
print(tokenizer.batch_decode(gen_out, skip_special_tokens=True))

model.generation_config.cache_implementation = "static"
gen_out = model.generate(**inputs, do_sample=False)
print(tokenizer.batch_decode(gen_out, skip_special_tokens=True))
```

Hi, @gante !
your example works fine, but in my app I pack all generation args into `GenerationConfig` (because it can conveniently hold it all) and it fails when `cache_implementation="static"` is among the args.
The constructor requires 'batch_size' and 'max_cache_len', but I may not know them in advance. These are decided inside `.generate()`.

Try this example, please:
```
from transformers import GenerationConfig
generation_config = GenerationConfig(cache_implementation="static")

>> output:
TypeError                                 Traceback (most recent call last)
Cell In[1], line 2
      1 from transformers import GenerationConfig
----> 2 generation_config = GenerationConfig(
      3     cache_implementation="static"
      4 )

File [~/conda/envs/tts39/lib/python3.9/site-packages/transformers/generation/configuration_utils.py:415](http://terranova.vla.yp-c.yandex.net:6501/lab/workspaces/tts/tree/tts/conda/envs/tts39/lib/python3.9/site-packages/transformers/generation/configuration_utils.py#line=414), in GenerationConfig.__init__(self, **kwargs)
    413 cache_config_class = NEEDS_CACHE_CONFIG[self.cache_implementation]
    414 if self.cache_config is None:
--> 415     self.cache_config = cache_config_class()
    416 elif isinstance(self.cache_config, dict):
    417     self.cache_config = cache_config_class.from_dict(self.cache_config)

TypeError: __init__() missing 2 required positional arguments: 'batch_size' and 'max_cache_len'
```


Gotcha. That shouldn't happen, opening a PR to fix it.