Sink cache: fix generate from cache
Author: zucchini-nlp
Pull request: yes
Labels: name: WIP

# What does this PR do?

Fixes #31381. The reason why the script was throwing an error is because input ids were not being cropped to the cache max length, causing size mismatch between inputs and attn mask.

The solution is to crop inputs to the max cache length, which will recompute kv for the whole cache and replace with new kv. Because we cannot infer anymore what was the actual length of the inputs in kv and what should be the length of unprocessed inputs. Yes, we could use '_seen_tokens' but that feature is already deprecated.

Also we need to crop `cache_position` the other way because otherwise the causal mask will not be inferred correctly. 


Tested with the below script. I will add a parameterized to `test_continue from cache` to test that all Cache classes can continue from past_kv and distribute this change to all models, if you think the proposed fix makes sense. IMO it's not the best, but is an option to overcome `cache_position` clash with SinkCache. 

Modifying `cache_position` in generate before forward is not an option, as it will cause cropping inputs using incorrect `cache_position`.

```python
import torch

from transformers import AutoTokenizer,AutoModelForCausalLM
from transformers.cache_utils import SinkCache, DynamicCache

model_id = "meta-llama/Llama-2-7b-chat-hf"

model = AutoModelForCausalLM.from_pretrained(
      model_id,
      low_cpu_mem_usage=True,
      device_map='auto',
      attn_implementation="eager", # eager to actually use causal mask and test it works
      torch_dtype=torch.bfloat16
)

tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side = 'left')
tokenizer.pad_token = tokenizer.eos_token
device = model.device

prefix_list = ["Hello, my name is Raushan and I am", "What's you name?"]
dialog_history = ""
cache = SinkCache(window_length=7, num_sink_tokens=4)

for prefix in prefix_list:
    dialog_history += prefix
    inputs = tokenizer(dialog_history, return_tensors='pt').to(device)
    input_length = inputs.input_ids.shape[-1]

    gen_out = model.generate(**inputs, do_sample=False, max_new_tokens=64,
                            use_cache=True,
                            past_key_values=cache,
                            pad_token_id=tokenizer.pad_token_id,
                            return_dict_in_generate=True,
                            # output_attentions=True,
                        )

    decoded = tokenizer.decode(gen_out.sequences[0, input_length:], skip_special_tokens=True)
    dialog_history += decoded
    cache = gen_out.past_key_values

    print(f"Output: {decoded}")

``` 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31537). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@gante ready for re-review. All the changes in other models are caused by 'fix-copies'. Let me know if you agree with the current state, as it's not the perfect solution but imo it's hard to get all edge cases work now. We already did so many changes to cache API and now we need to catch up with it

Also, I didn't know we had add `test_cache_utils.py`, so I'll be running it from now on whenever generation changes slightly 😅 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

close as sink cache will not be maintained anymore