Cache: new Cache format in decoder-only models
Author: zucchini-nlp
Pull request: yes
Labels: name: run-slow

# What does this PR do?

This PR adds to all decoder-only models (except for XLNet) support for dynamic cache. I am working on general support for cache class in all models, but I found that encoder-decoder models are trickier and we might want to make a tuple of DynamicCache for those. That's why I am splitting the task into multiple PRs for easier review :)

All models were testes with GenerationTesterMixin tests and passed on my end. For special Bloom case, I made a tiny change that cache will hold same tensor-shapes in key/values (we need to be able to concat at shape[-2]). Another special case Git seems to work from the tests.

For encoder-decoder models I will try to add first for one model and make a PR. If the format is okay, I'll open another PR with all the generative models left

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31421). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> For encoder-decoder models I will try to add first for one model and make a PR. 

@zucchini-nlp regarding encoder-decoder models: let's first finish the PR for Whisper (https://github.com/huggingface/transformers/pull/31166), and then copy the design all over the library! :D Your input in that PR would be appreciated :)

All slow tests are passing on my end

Ping me for another review when needed! 🤗 

@ArthurZucker yep, ready for final review, changed the deprecation to be same as in another PR. Git model is a bit peculiar and thus will not support `cache_position`

Done + added slow test commit but I don't know why it's skipped in CI

Noticed that VLMs do not get dynamic cache by default, so added cache flags to VLMs + ran tests. Unfortunately quantized and static cache will not be supported until #30962 is merged

Discovered just today that GPT-Neo and a few models are just like Gemma2, i.e. have hybrid cache format. It will take me some time to figure out how to make things work and still support old format

@ArthurZucker teh only model that is like Gemma2 is GPT-Neo. I started working on adding a HybridCache support on it but realized it will bloat up the code with if/elses to check on which cache type we're using now. So, I propose to review and merge this PR to deprecate cache as tuple.

For gpt-neo currently I just toggled static-cache support to `False`, and added a TODO for us. Might be quite breaking and imo should be part of compile compatibility, not deprecation PR.

@zucchini-nlp ping me when this gets merged -- with this PR merged, I can [uniformize RoPE](https://github.com/huggingface/transformers/pull/31999) on even more models 🙌 

Cool will do one last pass then!

> BTW we could also just add the class to the set _support_quantized_cache={""} in cache_utils, we don't pollute this here, and we can directly get all classes that support quantized / static etc.

Sounds good, but imo it should be another PR because this one is finalizing deprecation for a few models left. I'll make a PR for auto-mapping-cache to see how it goes

yep another pr for sure! 

Ran the tests and fixed in couple places.  Will merge later today/tomorrow

Btw @gante seems like compile isn't working on current `main` after making cache classes `nn.Module`. Fails to `register_buffer` for some reason. For ex: `tests/models/llama/test_modeling_llama.py::LlamaModelTest::test_generate_compile_fullgraph` fails

@zucchini-nlp will have a look, thank you for flagging 👍