Assistant Decoding for Llava-Onevision Does Not Work
Author: Brianzhengca
Pull request: no
Labels: name: Feature request

name: bug

### System Info


- `transformers` version: 4.51.2
- Platform: Linux-4.18.0-513.18.1.el8_9.x86_64-x86_64-with-glibc2.28
- Python version: 3.12.2
- Huggingface_hub version: 0.30.2
- Safetensors version: 0.4.4
- Accelerate version: 1.0.1
- Accelerate config:    not found
- DeepSpeed version: 0.15.3
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: yes
- GPU type: NVIDIA L40


### Who can help?

@zucchini-nlp 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi all, reproduction code is given below: 
```
from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image

import torch
import requests

img_urls =["https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png",
           "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"]
images = [Image.open(requests.get(img_urls[0], stream=True).raw),
          Image.open(requests.get(img_urls[1], stream=True).raw)]

target_processor = AutoProcessor.from_pretrained("llava-hf/llava-onevision-qwen2-7b-ov-hf")
target_processor.tokenizer.padding_side = "left"
draft_processor = AutoProcessor.from_pretrained("llava-hf/llava-onevision-qwen2-0.5b-ov-hf")
draft_processor.tokenizer.padding_side = "left"
target = AutoModelForImageTextToText.from_pretrained("llava-hf/llava-onevision-qwen2-7b-ov-hf").to('cuda')
draft = AutoModelForImageTextToText.from_pretrained("llava-hf/llava-onevision-qwen2-0.5b-ov-hf").to('cuda')

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Describe this image in 500 words."}
        ]
    }
]

prompt = target_processor.apply_chat_template(messages, add_generation_prompt=True)
print(prompt)
inputs = target_processor(text=prompt, images=[images[0]], return_tensors="pt").to("cuda")

with torch.no_grad():
    generated_ids = target.generate(**inputs, max_new_tokens=1000, assistant_model=draft, tokenizer=target_processor.tokenizer, assistant_tokenizer=draft_processor.tokenizer)
generated_texts = target_processor.batch_decode(generated_ids, skip_special_tokens=True)

print(generated_texts)
```
The error I am getting is as follows:
```
Traceback (most recent call last):                                                                                                                                       
  File "my_path/Llava/main.py", line 34, in <module>                                                                                     
    generated_ids = target.generate(**inputs, max_new_tokens=1000, assistant_model=draft, tokenizer=draft_processor.tokenizer, assistant_tokenizer=draft_processor.tokeniz
er)                                                                                                                                                                      
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^                                                                                                                                                                      
  File "my_path/miniconda3/envs/open-instruct-run/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context                  
    return func(*args, **kwargs)                                                                                                                                         
           ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                         
  File "my_path/miniconda3/envs/open-instruct-run/lib/python3.12/site-packages/transformers/generation/utils.py", line 2409, in generate                   
    result = self._assisted_decoding(                                                                                                                                    
             ^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                    
  File "my_path/miniconda3/envs/open-instruct-run/lib/python3.12/site-packages/transformers/generation/utils.py", line 4685, in _assisted_decoding         
    candidate_input_ids, candidate_logits = candidate_generator.get_candidates(input_ids)                                                                                
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                
  File "my_path/miniconda3/envs/open-instruct-run/lib/python3.12/site-packages/transformers/generation/candidate_generator.py", line 523, in get_candidates
    assistant_output = self.assistant_model.generate(**generation_args, **self.assistant_kwargs)                                                                         
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                         
  File "my_path/miniconda3/envs/open-instruct-run/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context                  
    return func(*args, **kwargs)                                                                                                                                         
           ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                         
  File "my_path/miniconda3/envs/open-instruct-run/lib/python3.12/site-packages/transformers/generation/utils.py", line 2465, in generate                   
    result = self._sample(                                                                                                                                               
             ^^^^^^^^^^^^^                                                                                                                                               
  File "my_path/miniconda3/envs/open-instruct-run/lib/python3.12/site-packages/transformers/generation/utils.py", line 3431, in _sample                    
    outputs = self(**model_inputs, return_dict=True)                                                                                                                     
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                     
  File "my_path/miniconda3/envs/open-instruct-run/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl               
    return self._call_impl(*args, **kwargs)                                                                                                                              
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                              
  File "my_path/miniconda3/envs/open-instruct-run/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl                       
    return forward_call(*args, **kwargs)                                                                                                                                 
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                 
  File "my_path/miniconda3/envs/open-instruct-run/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func               
    return func(*args, **kwargs)                                                                                                                                         
           ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                         
  File "my_path/miniconda3/envs/open-instruct-run/lib/python3.12/site-packages/transformers/models/llava_onevision/modeling_llava_onevision.py", line 721, i
n forward                                                                                                                                                                
    raise ValueError(                                                                                                                                                    
ValueError: Image features and image tokens do not match: tokens: 0, features 2709 
```
I am really not sure where it might be going wrong, thank you so much!!

### Expected behavior

Should generate normally. 

Comments:
We don't support assisted decoding for VLMs when tokenizers are different. For llava models, actually the tokenizers are same but seems like the model configs diverged as one was padded to 64 while the other wasn't

I will take a look if we can support assisted decoding with different tokenizer

Ohh I see, thank you so much for your help!!