Add a static cache that offloads to the CPU or other device
Author: gerbenvv
Pull request: yes

# What does this PR do?

This PR adds a static cache that offloads to another device.

Fixes #32179


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @gante @n17s

## Performance tests

Performance tested it with:

- L40S (48 GB)
- Llama 3 70B (4 bit quantized)
- sdpa attention
- `torch.compile(model)`
- batch size of 1000
- sequence size of 25 (no prompt, fully generative)

and I am getting a throughput of about 535 tokens/s (OOM static).

Also with:

- batch size of 1
- sequence size of 100 (no prompt, fully generative)

which gets 10.6 tokens/s (12.8 tokens/s static).

And with:

- batch size of 1
- sequence size of 100 (fully prompt)

which does 98.8 tokens/s (106.8 tokens/s static)

And with:

- batch size of 50
- sequence size of 100 (fully prompt)

which does 939.5 tokens/s (995.6 tokens/s static)

Comments:
Also would be interesting to test this / showcase potential for huge beamsearch ! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32161). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker @gante Thanks for reviewing! Made some fixes and added unit tests in a new commit. Also added the performance testing into the PR description.

@gerbenvv I can also merge like this and commit later if you are busy! 

> @gerbenvv I can also merge like this and commit later if you are busy!

Yeah trying to run the tests now `RUN_SLOW=1 pytest tests/utils/test_cache_utils.py` but it's taking quite a while.
I also tried to install the full dev environment but it failed to install `av` due to some compile error.

I'll try running just the file `pytest tests/utils/test_cache_utils.py` and if that doesn't work for me, the tests that I have changed/added. Since it is basically a new class, that should be fine.

Hmm, the tests have literally crashed the whole machine. Is it supposed to use all the GPUs on the machine?

I am struggling a bit to run this.

`RUN_SLOW=1 pytest tests/utils/test_cache_utils.py` gave me:

```
========================================================================================================= short test summary info ==========================================================================================================
FAILED tests/utils/test_cache_utils.py::CacheTest::test_static_cache_exportability - OSError: You are trying to access a gated repo.                                                                                                        
FAILED tests/utils/test_cache_utils.py::CacheIntegrationTest::test_dynamic_cache_batched - OSError: You are trying to access a gated repo.                                                                                                  
FAILED tests/utils/test_cache_utils.py::CacheIntegrationTest::test_dynamic_cache_beam_search - OSError: You are trying to access a gated repo.                                                                                              
FAILED tests/utils/test_cache_utils.py::CacheIntegrationTest::test_dynamic_cache_hard - OSError: You are trying to access a gated repo.                                                                                                     
FAILED tests/utils/test_cache_utils.py::CacheIntegrationTest::test_hybrid_cache_n_sequences - OSError: You are trying to access a gated repo.                                                                                               
FAILED tests/utils/test_cache_utils.py::CacheIntegrationTest::test_offloaded_cache_equivalent_to_dynamic_cache - ImportError: Using `low_cpu_mem_usage=True` or a `device_map` requires Accelerate: `pip install accelerate`                
FAILED tests/utils/test_cache_utils.py::CacheIntegrationTest::test_offloaded_cache_uses_less_memory_than_dynamic_cache - ImportError: Using `low_cpu_mem_usage=True` or a `device_map` requires Accelerate: `pip install accelerate`        
FAILED tests/utils/test_cache_utils.py::CacheIntegrationTest::test_sink_cache_iterative_prompts - ImportError: Using `low_cpu_mem_usage=True` or a `device_map` requires Accelerate: `pip install accelerate`                               
===================================================================================== 8 failed, 14 passed, 2 skipped, 24 warnings in 627.82s (0:10:27) ====================================================================================
```

Then I tried to fix those errors by passing the token & installing `accelerate` but then the machine crashed. But we at least know that:

```
test_static_cache_greedy_decoding_pad_left
test_static_cache_greedy_decoding_pad_right
test_static_cache_extra_left_padding
```

ran succesful and those were the ones that I have changed.

Okay, making progress ;-)

New output of `CUDA_VISIBLE_DEVICES=0 RUN_SLOW=1 pytest tests/utils/test_cache_utils.py` is:

```
========================================================================================================= short test summary info ==========================================================================================================
FAILED tests/utils/test_cache_utils.py::CacheTest::test_static_cache_exportability - torch._export.verifier.SpecViolationError: Node.meta _enter_autocast is missing val field.
FAILED tests/utils/test_cache_utils.py::CacheIntegrationTest::test_hybrid_cache_n_sequences - ValueError: Greedy methods without beam search do not support `num_return_sequences` different than 1 (got 2).
FAILED tests/utils/test_cache_utils.py::CacheIntegrationTest::test_sink_cache_iterative_prompts - AssertionError: False is not true
===================================================================================== 3 failed, 19 passed, 2 skipped, 28 warnings in 756.35s (0:12:36) =====================================================================================
```

So no tests regarding the static, dynamic or offloading caches are failing. I think this should be good enough to get this merged, right?

Yep let's go! 🔥