DynamicCache support in vsion llm: "microsoft/Phi-3.5-vision-instruct"
Author: dahwin
Pull request: no
Labels: name: bug

name: Vision

name: Cache

### System Info

### what i want
So I want a solution that can quickly generate AI output by efficiently using precompute kv caches of text and images of all previous prompts!




### by using DynamicCache llm or vision llm don't need to re compute the previous prompt. it can re use previous prompt kv cache!
### In gemini it's called context caching gemini & in anthropic it's called prompt caching
```text
DynamicCache
used to store and reuse the intermediate computations (key-value pairs) from previous iterations of the model's attention layers. This is particularly useful in scenarios where you're generating multiple responses in a conversation or processing a stream of related inputs
```


###  Down bellow Problem I'm getting while using DynamicCache in vision LLM like : "microsoft/Phi-3.5-vision-instruct" 
I have used thousands of prompt caching in only text llm! 
But here in vision llm 3 pronpt text+image prompt caching can be seen! ai  is not responding  prompt 3! 
Why is this happening?
See problem down bellow!



```python
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor
import torch
from transformers.cache_utils import DynamicCache
import time
model_id = "microsoft/Phi-3.5-vision-instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    trust_remote_code=True,
    torch_dtype="auto",
    _attn_implementation='eager'
)
processor = AutoProcessor.from_pretrained(model_id,
    trust_remote_code=True,
    num_crops=4
)
# Load images
image_files = [
    "llm-applications-main.jpg",
    "llm-applications-main.jpg",
    "llm-applications-main.jpg",
]
images = [Image.open(image_file) for image_file in image_files]
# Prepare image placeholders
placeholder = "".join([f"<|image_{i}|>\n" for i in range(1, len(images) + 1)])
user_prompts = [
    "Give me a short introduction to large language models. Under 20 words.",
    "Can you elaborate on that? Under 20 words! Answer always in English.",
    'This branch of ML is my favorite'
]
past_key_values = DynamicCache()
messages = []
for prompt in user_prompts:
    start_time = time.time()
    
    messages.append({"role": "user", "content": placeholder + prompt})
    full_prompt = processor.tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )
    
    inputs = processor(full_prompt, images, return_tensors="pt").to(model.device)
    
    input_length = inputs['input_ids'].shape[1]
    
    generation_args = {
        "max_new_tokens": 2560,
        "temperature": 0.0,
        "do_sample": False,
    }
    
    outputs = model.generate(
        **inputs,
        eos_token_id=processor.tokenizer.eos_token_id,
        past_key_values=past_key_values,
        **generation_args
    )
    
    completion = processor.batch_decode(outputs[:, input_length:],
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False)[0]
    
    messages.append({"role": "assistant", "content": completion})
    print(f"User: {prompt}")
    print(f"Assistant: {completion}\n")
    
    end_time = time.time()
    print(f"Total time: {end_time - start_time:.2f} seconds")
```
### why i'm not getting answer of last assistant? (prompt 3)
### OUTPUTS

```text

User: Give me a short introduction to large language models. Under 20 words.
Assistant: Large language models are AI systems designed to understand and generate human language.
Total time: 9.37 seconds
User: Can you elaborate on that? Under 20 words! Answer always in English.
Assistant: Large language models are AI systems that process and generate human language, often used for tasks like translation and conversation.
Total time: 10.48 seconds
User: This branch of ML is my favorite
Assistant: 
 
Total time: 9.79 seconds
```












### i have found an alternative solution (Clear cache every 2 interactions).
###  which is not good 
Here every two prompt system to clear the cache! But through it the previous information or context is deleted! Which is the complete opposite of my purpose! prompt caching My intention was to reuse pre computed kv caches of text +image for efficient generation of user ai communication! But if you clear the cache like this, the context is lost! So ai loses the context of the user!

### what i want
So I want a solution that can quickly generate AI output by efficiently using precompute kv caches of text and images of all previous prompts!







```python
from PIL import Image
from transformers import AutoModelForCausalLM, AutoProcessor
import torch
from transformers.cache_utils import DynamicCache
import time
model_id = "microsoft/Phi-3.5-vision-instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    trust_remote_code=True,
    torch_dtype="auto",
    _attn_implementation='eager'
)
processor = AutoProcessor.from_pretrained(model_id,
    trust_remote_code=True,
    num_crops=4
)
# Load images
image_files = [
    "llm-applications-main.jpg",
    "llm-applications-main.jpg",
    "llm-applications-main.jpg",
]
images = [Image.open(image_file) for image_file in image_files]
# Prepare image placeholders
placeholder = "".join([f"<|image_{i}|>\n" for i in range(1, len(images) + 1)])
user_prompts = [
    "Give me a short introduction to large language models. Under 20 words.",
    "Can you elaborate on that? Under 20 words! Answer always in English.",
    'This branch of ML is my favorite. Why might it be interesting?'
]
past_key_values = DynamicCache()
messages = []
for i, prompt in enumerate(user_prompts):
    start_time = time.time()
    
    messages.append({"role": "user", "content": placeholder + prompt})
    full_prompt = processor.tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )
    
    inputs = processor(full_prompt, images, return_tensors="pt").to(model.device)
    
    input_length = inputs['input_ids'].shape[1]
    
    generation_args = {
        "max_new_tokens": 2560,
        "temperature": 0.7 if i == len(user_prompts) - 1 else 0.0,  # Increase temperature for last prompt
        "do_sample": True if i == len(user_prompts) - 1 else False,  # Enable sampling for last prompt
    }
    
    outputs = model.generate(
        **inputs,
        eos_token_id=processor.tokenizer.eos_token_id,
        past_key_values=past_key_values,
        **generation_args
    )
    
    completion = processor.batch_decode(outputs[:, input_length:],
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False)[0]
    
    messages.append({"role": "assistant", "content": completion})
    print(f"User: {prompt}")
    print(f"Assistant: {completion}\n")
    
    end_time = time.time()
    print(f"Total time: {end_time - start_time:.2f} seconds")
    
    # Clear cache every 2 interactions
    if (i + 1) % 2 == 0:
        past_key_values = DynamicCache()
        messages = []  # Optionally clear message history as well


```






```text
User: Give me a short introduction to large language models. Under 20 words.
Assistant: Large language models are AI systems designed to understand and generate human language.

Total time: 9.31 seconds
User: Can you elaborate on that? Under 20 words! Answer always in English.
Assistant: Large language models are AI systems that process and generate human language, often used for tasks like translation and conversation.

Total time: 10.30 seconds
User: This branch of ML is my favorite. Why might it be interesting?
Assistant: I'm sorry, but as an AI, I don't have personal opinions or feelings, including preferences for branches of machine learning.

Total time: 10.10 seconds
```



i don't want this it loosing it's context
so i don't want to use *Clear cache every 2 interactions*
try to fix this without *Clear cache every 2 interactions*
### what i want
So I want a solution that can quickly generate AI output by efficiently using precompute kv caches of text and images of all previous prompts!




Comments:
Hey @dahwin !

I don't see anything wrong in the script you use and as you've mentioned it works perfectly for LLMs. 

The only difference between an LLM and a VLM is that VLMs require more context length as each input image is embedded and concatenated with `hidden_states`. I tried running the provided script with a different image and prompts, and got an answer in all three turns. So my guess is that Phi-vision is not very good at long-context generations or in multi-turn chatting, and the provided snippet easily gets to 5k+ tokens after the second turn (depending on your image resolution).

Also not very clear why the same image is passed in each turn, I'd recommend to rely on cache which should hold the very first passed image. In case your use-case doesn't require multiple different images ofc 😃 



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.