Why can't `inputs_embeds` be used during the first generation in a multimodal model?
Author: aohenuo
Pull request: no
Labels: name: Feature request

name: Multimodal

When reading the source code of transformers related to multimodal models, I noticed a specific behavior: in the Idefics v2 and Idefics v3 models, if only inputs_embeds are provided without input_ids during the first input, the following error occurs: 
***"When first calling the model, if inputs_embeds are passed, input_ids should not be None."***
For example, raising value error's code is in 1384 line in modeling_idefics2.py
![image](https://github.com/user-attachments/assets/b037f6fe-7c18-42c9-8655-9a96941fd70e)

However, in Idefics v1, this exception does not appear.

I initially thought this might be due to differences between cross-attention-based multimodal models and adapter-based architectures. However, in other adapter-based multimodal models (e.g., LLAVA), this exception is also absent.

So, why was this exception introduced in Idefics v2 and v3? If you could provide an explanation, I would greatly appreciate it!

Comments:
@aohenuo hey! I agree this behavior can be unintuitive and currently VLMs are not consistent on what input combinations are acceptable or not. In Idefics 2 and 3 the embeds are not accepted because the model wants to know how to merge image embeddings with text embeddings, i.e. which positions in text embeds are meant to be replaced by image hidden states.

In other models like LLaVA what we do is just restrict passing embeds and pixel values together, but do not restrict passing only embeds. And it is the users' responsibility to make sure that the embeddings are already correctly formatted and can be forwarded to the LM.

Can you share your use-case and I'll try to find a quick solution for you? But as a long term we are planning to remove all the above restrictions and allow users to pass any inputs, that should be done in a 1-2 months

@zucchini-nlp Thank you very much for your reply! I’m currently a bit tied up with something. Could you wait for two to three days? I’ll provide you with a use-case code then.🙏

@zucchini-nlp 
This is a use-case code I created myself. I organized the code from the idefics2 source code that maps image features to text embedding space into this ***generate_merge_embedding*** function. Theoretically, it performs the operation of mapping images to the text embedding space and integrating them with text embedding.
```python
import requests
import torch
from PIL import Image
root_dir = "HuggingFaceM4/idefics2-8b"
from transformers import AutoProcessor, AutoModelForVision2Seq
from transformers.image_utils import load_image
def generate_merge_embedding(LMM_model, query_inputs):
    #input is the idev2 process generate query_inputs, includes pixel_values, pixel_attention_mask, input_ids
    #output is the merged embeddings 
    #the output shape is (bs,seq_len,hidden_dim)
    pixel_values = query_inputs["pixel_values"].to(torch.float16)
    batch_size, num_images, num_channels, height, width = pixel_values.shape
    pixel_values = pixel_values.view(batch_size * num_images, *pixel_values.shape[2:])
    
    # Remove padding images - padding images are full 0.
    nb_values_per_image = pixel_values.shape[1:].numel()
    real_images_inds = (pixel_values == 0.0).sum(dim=(-1, -2, -3)) != nb_values_per_image
    pixel_values = pixel_values[real_images_inds].contiguous()
    pixel_attention_mask = query_inputs["pixel_attention_mask"]
    pixel_attention_mask = pixel_attention_mask.view(batch_size * num_images, *pixel_attention_mask.shape[2:]
            )
    pixel_attention_mask = pixel_attention_mask[real_images_inds].contiguous()
    patch_size = LMM_model.model.config.vision_config.patch_size
    patches_subgrid = pixel_attention_mask.unfold(dimension=1, size=patch_size, step=patch_size)
    patches_subgrid = patches_subgrid.unfold(dimension=2, size=patch_size, step=patch_size)
    patch_attention_mask = (patches_subgrid.sum(dim=(-1, -2)) == patch_size * patch_size).bool()
    image_hidden_states = LMM_model.model.vision_model(
            pixel_values=pixel_values,
            patch_attention_mask=patch_attention_mask,
        ).last_hidden_state
    image_hidden_states = LMM_model.model.connector(
            image_hidden_states, attention_mask=patch_attention_mask.view(pixel_values.size(0), -1)
        )
    org_embedding = LMM_model.model.text_model.get_input_embeddings()(query_inputs["input_ids"])
    
    
    merge_embeddings = LMM_model.model.inputs_merger(input_ids=query_inputs["input_ids"], inputs_embeds=org_embedding, image_hidden_states=image_hidden_states)
    return merge_embeddings, image_hidden_states
DEVICE = "cuda:0"

# Note that passing the image urls (instead of the actual pil images) to the processor is also possible
image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg")



processor = AutoProcessor.from_pretrained(root_dir)
model = AutoModelForVision2Seq.from_pretrained(
   root_dir, torch_dtype=torch.float16
).to(DEVICE).eval()


# Create inputs
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What do we see in this image?"},
        ]
    },
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image1], return_tensors="pt")
inputs = {k: v.to(DEVICE) for k, v in inputs.items()}
#generate merge embeddings
model_embedding, _= generate_merge_embedding(model,inputs)
inputs["inputs_embeds"] = model_embedding
inputs["input_ids"] = None
# Generate
generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)

print(generated_texts)
```


@aohenuo I see, so you are merging the inputs yourself and it makes sense. That should be working after applying the planned changes on all VLMs. I will assign this issue to myself and keep it open until resolved

As a workaround to unblock you, you can create dummy "input_ids" and pass them to model's forward. When the model receives `embeds` and `ids` but no images , nothing happens because the whole block of ode is skipped. This is exactly your case in the snippet above