Do not update cache when use_cache=False and past_key_values are provided?
Author: PheelaV
Pull request: no
Labels: name: Feature request

### Feature request

Hi, currently I am doing some training and I have a use case where I want to reuse the KV cache for the context without having to resolve to a deep copy or truncating the cache after each generation.

### Motivation

In the decoder, `.update` is called even in cases where use_cache = False, that is to concatenate the existing cache with new k/v projections, even though kv_cache is not returned (and my naive assumption is that the updated cache is not used after the decoders). I want to use cache, but I do not want it returned nor updated. In the case of Qwen2, the updated cache is never used within the same forward again. Hence this creates a side effect which is undesirable. I am creating a feature request and not a bug issue as I do not think it has negative effects for the current API usage. Getting rid of this side effect would benefit the scenario where you want to reuse the exact same cache.

This is not a suggestion, more of a clarification as to what side effect I am addressing here, I myself use this as a patch for the time being so I do not have to patch the model. After setting `self.update_cache` I will either return the concatenation of the current cache and the new k/v, or I will just return the new k/v pairs if there is no previous cache:

```python
    def update(
        self,
        key_states: torch.Tensor,
        value_states: torch.Tensor,
        layer_idx: int,
        cache_kwargs: Optional[Dict[str, Any]] = None,
    ) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        Updates the cache with the new `key_states` and `value_states` for the layer `layer_idx`.

        Parameters:
            key_states (`torch.Tensor`):
                The new key states to cache.
            value_states (`torch.Tensor`):
                The new value states to cache.
            layer_idx (`int`):
                The index of the layer to cache the states for.
            cache_kwargs (`Dict[str, Any]`, `optional`):
                Additional arguments for the cache subclass. No additional arguments are used in `DynamicCache`.

        Return:
            A tuple containing the updated key and value states.
        """
        # Update the number of seen tokens
        if layer_idx == 0:
            self._seen_tokens += key_states.shape[-2]

        # Update the cache
        if key_states is not None:
            if len(self.key_cache) <= layer_idx:
                # There may be skipped layers, fill them with empty lists
                for _ in range(len(self.key_cache), layer_idx):
                    self.key_cache.append([])
                    self.value_cache.append([])
                self.key_cache.append(key_states)
                self.value_cache.append(value_states)
            elif (
                len(self.key_cache[layer_idx]) == 0 and self.update_cache
            ):  # fills previously skipped layers; checking for tensor causes errors
                self.key_cache[layer_idx] = key_states
                self.value_cache[layer_idx] = value_states
            elif (
                len(self.key_cache[layer_idx]) == 0 and not self.update_cache
            ):  # fills previously skipped layers; checking for tensor causes errors
                return key_states, value_states
            elif self.update_cache:
                self.key_cache[layer_idx] = torch.cat([self.key_cache[layer_idx], key_states], dim=-2)
                self.value_cache[layer_idx] = torch.cat([self.value_cache[layer_idx], value_states], dim=-2)
            else:
                new_keys = torch.cat([self.key_cache[layer_idx], key_states], dim=-2)
                new_values = torch.cat([self.value_cache[layer_idx], value_states], dim=-2)
                return new_keys, new_values

        return self.key_cache[layer_idx], self.value_cache[layer_idx]
```

For the suggestion, the update could be split into update and retrieve, where update would be used in case of `use_cache == True` and `retrieve_cache` when `use_cache == False` and `past_key_values is not None`.

I only inspected the implementation of Qwen2, I am not using gradient checkpointing, and I am only using DynamicCache for the time being, that is to say the alternative combinations have not been considered. My question upon the community and HF team: Is there a good reason besides simplicity, not to do this?

Kindly tagging only @gante as not to oversubscribe the HF team.

### Your contribution

I would be happy to contribute, depending on the scope and after initial sanity check.

Comments:
Hi! @PheelaV 
I already tried your implementation, and wrote a test in the file test_cache_utils.py.
Seems it passes all the tests!
However, the function update() should be used as an ```@abstractmethod```, or it would not work.
I can submit a pull request for you.

@PheelaV thank you for opening the issue and for the suggestion 🤗 

Indeed, it requires custom code with our current codebase. For a long time, `use_cache` controls whether a cache is both used and updated, so using a cache and not updating it is not covered by any flag combination at the moment.

This is the first time I'm seeing this specific request. I'm deciding not to add this feature for now, to prevent bloating the codebase. However, if other folks are interested in it, let me know in the comments below -- if it gets traction, I will change my decision to enable this functionality in `transformers`.

Meanwhile, overwriting `update` (e.g. as in the snippet at the top) is my recommendation. We will soon add functionality to crop to a specific length, in case it helps (cc @manueldeprada)

Hi @gante. Thanks for the clarification! I also have a use case where I need to use the existing cache but avoid updating it during forward passes. If adding this feature is still not planned, could you please point me to where the “crop to a specific length” functionality is implemented or referenced?

Here is the pointer to the docs: https://huggingface.co/docs/transformers/main/en/internal/generation_utils#transformers.Cache.crop

I advise you to go carefully through the cache_utils.py file. To avoid updates while still using the cache, probably you need to modify the relevant CacheLayer, since it is a very niche use case.

> Here is the pointer to the docs: https://huggingface.co/docs/transformers/main/en/internal/generation_utils#transformers.Cache.crop
> 
> I advise you to go carefully through the cache_utils.py file. To avoid updates while still using the cache, probably you need to modify the relevant CacheLayer, since it is a very niche use case.

Thank you for the suggestions. Using `crop` can indeed solve the issue in a simple way, but it may still conflict with other APIs (for example, `torch.utils.checkpoint.checkpoint` does not support in-place modifications). For future readers: if you want `past_key_value` to remain read-only during `forward` with minimal code changes, you may consider using a [monkey-patch approach](https://www.geeksforgeeks.org/python/monkey-patching-in-python-dynamic-behavior/) to avoid modifying the `transformers` package. My own solution is to add a `read_only` property to the `Cache` class in `transformers/cache_utils.py` and check this flag inside the `update` method.