Slow Generation of Tokens when trying to run for the first time.
Author: dikshyantacharya
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.46.2
- Platform: Windows-11-10.0.22631-SP0
- Python version: 3.12.7
- Huggingface_hub version: 0.25.1
- Safetensors version: 0.4.5
- Accelerate version: 1.0.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.0+cu118 (True)
- Tensorflow version (GPU?): 2.18.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA RTX A5000 Laptop GPU

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
# Load model directly
from transformers import  AutoProcessor, AutoModelForImageTextToText, TextStreamer, TextIteratorStreamer
import requests
import torch
from PIL import Image
import warnings
from threading import Thread
import time
import io

# Model
model_id = "SeanScripts/Llama-3.2-11B-Vision-Instruct-nf4"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, use_safetensors=True, device_map="cuda:0")

# Prompt
chat_template = """
{% for message in messages %}
{{ message['role'] }}:
{% for item in message['content'] %}
{% if item['type'] == 'image' %}<image>{% endif %}
{% if item['type'] == 'text' %}{{ item['text'] }}{% endif %}
{% endfor %}
{% endfor %}
"""
system_prompt="""
You are an assistant that extracts answers from given image.
"""

#Downloading Image
url = "https://englishan.com/wp-content/uploads/2023/08/body.png"
response = requests.get(url)
image_content = response.content
image = Image.open(io.BytesIO(image_content))


question = """
Give me summary of the file.
"""
messages = [
    {"role": "system", "content": [{"type": "text", "text": f"{system_prompt}"}]},
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": f"{question}\nassistant:\n"}
    ]}
]

processor.chat_template = chat_template
processor.tokenizer.add_special_tokens({"additional_special_tokens": ["<image>"]})

input_text = processor.apply_chat_template(messages, add_generation_prompt=False)
input_ids = processor.tokenizer.encode(input_text, add_special_tokens=True)
image_token_id = processor.tokenizer.convert_tokens_to_ids("<image>")
input_ids.insert(input_ids.index(image_token_id), image_token_id)

# Process the image and get pixel values, aspect_ratio_ids, and aspect_ratio_mask
image_processing_output = processor.image_processor(image, return_tensors="pt")
pixel_values = image_processing_output.pixel_values.to(model.device)
aspect_ratio_ids = image_processing_output.aspect_ratio_ids.to(model.device)
aspect_ratio_mask = image_processing_output.aspect_ratio_mask.to(model.device)

# Prepare inputs for the model
inputs = {
    "input_ids": torch.tensor([input_ids], device=model.device),
    "pixel_values": pixel_values,
    "aspect_ratio_ids": aspect_ratio_ids,
    "aspect_ratio_mask": aspect_ratio_mask,
    "max_new_tokens": 10,
    "temperature": 0.1,
    "use_cache": True 
}

class CustomTextStreamer(TextStreamer):
    def on_finalized_text(self, text: str, stream_end: bool = False):
        # Custom logic: Modify the text before printing
        custom_text = f"{text}"

        # Print the modified text
        print(custom_text, flush=True, end="" if not stream_end else None)

custom_streamer = CustomTextStreamer(tokenizer=processor.tokenizer, skip_special_tokens=True)
inputs.update({"streamer": custom_streamer})


# Case -1: Running the model directly without using thread
# Start the timer
start_time = time.time()

# Generate tokens
model.generate(**inputs)

# Calculate the time taken
end_time = time.time()
print(f"Time taken with TextStreamer: {end_time - start_time:.2f} seconds")
```


### Expected behavior

I run it in jupyternotebook, where the models are already loaded in a different shell, and when i run the shell with model.generate(**inputs)
Then i get this: 
![image](https://github.com/user-attachments/assets/d22020d3-fac1-411a-adee-dbbb4c41be80)
Time taken with TextStreamer: 8.12 seconds to generate 10 tokens

But when i run same line without restarting the kernel, but for different image or different prompt, then the token generation is fast.
![image](https://github.com/user-attachments/assets/dcbb23de-c8a7-4907-b287-4613d3df65e5)
Here, it just takes maximum of 2 seconds to generate the token.

This is causing the issue when i have to use thread with TextIteratorStreamer to stream the generated tokens at the real time to client, because the model just keeps generating the token slow.
I want to have that fast token generation even at all time. 

Thanks for your help in advance. 

Comments:
cc @gante @zucchini-nlp 

Could it be related to auto-compile, or this feature not yet been released?
 - https://github.com/huggingface/transformers/pull/34247

@Rocketknight1 , @qubvel  Thank you for your response. I will close this issue.