Mismatch Between Image Tokens and Features in LLaVA Model Fine-Tuning
Author: Md-Nasif03
Pull request: no

**Model: llava-hf/llava-1.5-7b-hf**
**Issue Description**
When I try to generate a response using the fine-tuned model, I encounter the following error:
ValueError: Image features and image tokens do not match: tokens: 575, features: 576

This error occurs during the generate() call, indicating a mismatch between the number of image tokens and image features.

**Steps I’ve Taken:**

- Image Preprocessing:
- I resized the input image to dimensions that are multiples of the patch_size (14 for LLaVA models).
- The image is resized to 518x336, which is a multiple of 14.

**Processor Configuration:**

- I manually set the patch_size and vision_feature_select_strategy in the processor to match the model's configuration.
- I verified that the processor's configuration is correct.

**Debugging Inputs:**

- I printed the inputs (input_ids, pixel_values, etc.) to ensure they are correctly formatted.
- The inputs are moved to the GPU for processing.

**Model Loading:**
The model and processor are loaded from the fine-tuned directory, and the model is moved to the GPU.

Code Snippet
Here’s the relevant part of my code:

`
```
#Load the processor and model
processor1 = LlavaProcessor.from_pretrained(model_path)
new_model_v1 = LlavaForConditionalGeneration.from_pretrained(model_path).to("cuda:0")

#Resize the image
patch_size = new_model_v1.config.vision_config.patch_size
shortest_edge = processor1.image_processor.size.get("shortest_edge", 336)
original_width, original_height = raw_image.size
scale_factor = shortest_edge / min(original_width, original_height)
new_width = int(original_width * scale_factor)
new_height = int(original_height * scale_factor)

new_width = (new_width // patch_size) * patch_size
new_height = (new_height // patch_size) * patch_size
raw_image = raw_image.resize((new_width, new_height)) # Resized to multiples of patch_size

#Process inputs
inputs = processor1(images=raw_image, text=prompt, return_tensors='pt')
inputs = {k: v.to("cuda:0") for k, v in inputs.items()}

#Generate response
output = new_model_v1.generate(
input_ids=inputs["input_ids"],
pixel_values=inputs["pixel_values"],
max_new_tokens=200,
do_sample=False
)

```
**Questions:**

- Why is there a mismatch between the number of image tokens (575) and image features (576)?
- Is there a mistake in my image preprocessing or model configuration that could cause this issue?
- How can I ensure that the number of image tokens matches the number of image features?
- Are there any additional steps I need to take to align the image tokens and features correctly?

**Additional Information**

- I have tried the Hugging Face transformers library both version 4.48.2 and 4.48.2.
- The fine-tuned model is saved in Google Drive, and I’m loading it in a new Colab session.
- The base model is llava-hf/llava-1.5-7b-hf.

<img width="885" alt="Image" src="https://github.com/user-attachments/assets/444c2286-629d-4385-85df-6c485a27f076" />
<img width="418" alt="Image" src="https://github.com/user-attachments/assets/3a24ce8b-09ae-4a36-b54f-3de0946619b8" />

Comments:
I am getting a similar error:
ValueError: Image features and image tokens do not match: tokens: 0, features 576,

how did you solve this?

@Himanshunitrr   probably you need to add processor.num_additional_tokens = 1 or processor.num_additional_image_tokens = 1

@Md-Nasif03 In your case the issue was probably that your model was outdated (did not include [this commit](https://huggingface.co/llava-hf/llava-1.5-7b-hf/commit/ca4bea31bcbe6d887effc1aa34957161566f7033)). @Himanshunitrr You have a different issue, I don't know how you can have 0 tokens.

I had the same problem，ValueError: Image features and image tokens do not match: tokens: 575, features: 576
how did you solve it?

@WX3210 By updating the model. The fix is the processor_config.json file in the commit I linked above.

"What you said is  helpful, thank you."

> I am getting a similar error: ValueError: Image features and image tokens do not match: tokens: 0, features 576,
> 
> how did you solve this?

Hi, Himanshunitrr. Have you found any solutions? i also met similar error: ValueError: Image features and image tokens do not match: tokens: 1, features 576. Are there any suggestions on how to solve it? Thanks!

> [@Md-Nasif03](https://github.com/Md-Nasif03)就您而言，问题可能是您的模型已经过时（未包含[此提交](https://huggingface.co/llava-hf/llava-1.5-7b-hf/commit/ca4bea31bcbe6d887effc1aa34957161566f7033)）。[@Himanshunitrr](https://github.com/Himanshunitrr)您有一个不同的问题，我不知道您怎么会拥有 0 个令牌。

Thank you so much for sharing!

I have meet a problem - ```ValueError: Image features and image tokens do not match: tokens: 0, features 5967```.
And adding ```"num_additional_image_tokens": 1,``` in processor_config.json， the problem was solved.



> "num_additional_image_tokens": 1,

@NiYueLiuFeng I have the following code to initialize the processor and model. There is no processor_config.json file in the project. May I know where I should add this file in the project, and how I can use it to initialize the processor and model?

processor = AutoProcessor.from_pretrained("reducto/RolmOCR")
model = AutoModelForImageTextToText.from_pretrained("reducto/RolmOCR")

image = Image.open("./demo.jpg").convert("RGB")
prompt = "What is written in the document?"
inputs = processor(text=prompt, images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=4096)

text = processor.batch_decode(outputs, skip_special_tokens=True)[0]
print(text)

> > "num_additional_image_tokens": 1,
> 
> [@NiYueLiuFeng](https://github.com/NiYueLiuFeng) I have the following code to initialize the processor and model. There is no processor_config.json file in the project. May I know where I should add this file in the project, and how I can use it to initialize the processor and model?
> 
> processor = AutoProcessor.from_pretrained("reducto/RolmOCR") model = AutoModelForImageTextToText.from_pretrained("reducto/RolmOCR")
> 
> image = Image.open("./demo.jpg").convert("RGB") prompt = "What is written in the document?" inputs = processor(text=prompt, images=image, return_tensors="pt")
> 
> with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=4096)
> 
> text = processor.batch_decode(outputs, skip_special_tokens=True)[0] print(text)

I met the same problem. Have you solved the promblem?

> > 我收到类似的错误：ValueError：图像特征和图像令牌不匹配：tokens：0，特征 576，
> > 你是如何解决这个问题的？
> 
> 嗨，Himanshunitrr。你找到什么解决方案了吗？我还遇到了类似的错误：ValueError：图像特征和图像令牌不匹配：tokens：1，特征 576。有什么解决建议吗？谢谢！
 您好，请问你找到什么解决方案了吗，我也遇到了类似的错误ValueError: Image features and image tokens do not match: tokens: 1, features 297


I meet the same problem when using LiquidAI/LFM2-VL-1.6B as the lora's base model 
and I solve this problem by set the truncation of processor to False, although it takes more GPU memory usage. 
I don't know why it works just because GPT told me to do that.