Fix qwen2_vl bug when in inference.
Author: muazhari
Pull request: yes

# What does this PR do?
Add missing validation.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)
![image](https://github.com/user-attachments/assets/bc9bdb6b-4e8a-499a-8684-65b43455b716)
Source: https://huggingface.co/MrLight/dse-qwen2-2b-mrl-v1

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hi @muazhari , what is exactly the bug being solved here? It would help to have a reproducer contained within the library!

> Hi @muazhari , what is exactly the bug being solved here? It would help to have a reproducer contained within the library!

Error exception bug with this reproducible:

https://github.com/muazhari/research-assistant-backend/blob/7703bf423ba96cebb763bd0881269dd5b41c7c67/playgrounds/multi-modal-playground.ipynb

You can run then compare it using before and after this pr commit.



I'm sorry but there is a misunderstanding on what a reproducer is: it seems there is a bug, what triggers it? What error does it raise exactly? The `cache_position` should not be `None` as it should be initialized in `generate`. 

> I'm sorry but there is a misunderstanding on what a reproducer is: it seems there is a bug, what triggers it? What error does it raise exactly? The `cache_position` should not be `None` as it should be initialized in `generate`.

These are the error logs when using the main branch (https://github.com/huggingface/transformers/commit/5f0c181f4e8faf88d5c186961eeca0779b6354daz) :
```python
TypeError                                 Traceback (most recent call last)
Cell In[7], line 1
----> 1 rag_2.index(
      2     input_path="./tests",  # The path to your documents
      3     index_name="1",  # The name you want to give to your index. It'll be saved at `index_root/index_name/`.
      4     store_collection_with_index=True,  # Whether the index should store the base64 encoded documents.
      5     # doc_ids=[0, 1, 2], # Optionally, you can specify a list of document IDs. They must be integers and match the number of documents you're passing. Otherwise, doc_ids will be automatically created.
      6     # metadata=[{"author": "John Doe", "date": "2021-01-01"}], # Optionally, you can specify a list of metadata for each document. They must be a list of dictionaries, with the same length as the number of documents you're passing.
      7     overwrite=True
      8     # Whether to overwrite an index if it already exists. If False, it'll return None and do nothing if `index_root/index_name` exists.
      9 )

File /usr/local/lib/python3.12/dist-packages/byaldi/RAGModel.py:111, in RAGMultiModalModel.index(self, input_path, index_name, doc_ids, store_collection_with_index, overwrite, metadata)
     82 def index(
     83     self,
     84     input_path: Union[str, Path],
   (...)
     94     ] = None,
     95 ):
     96     """Build an index from input documents.
     97 
     98     Parameters:
   (...)
    109         None
    110     """
--> 111     return self.model.index(
    112         input_path,
    113         index_name,
    114         doc_ids,
    115         store_collection_with_index,
    116         overwrite=overwrite,
    117         metadata=metadata,
    118     )

Cell In[3], line 292, in DseModel.index(self, input_path, index_name, doc_ids, store_collection_with_index, overwrite, metadata)
    290         doc_id = doc_ids[i] if doc_ids else self.highest_doc_id + 1
    291         doc_metadata = metadata[doc_id] if metadata else None
--> 292         self.add_to_index(item, store_collection_with_index, doc_id=doc_id, metadata=doc_metadata)
    293         self.doc_ids_to_file_names[doc_id] = str(item)
    294 else:

Cell In[3], line 347, in DseModel.add_to_index(self, input_item, store_collection_with_index, doc_id, metadata)
    345         self._process_directory(item_path, store_collection_with_index, current_doc_id, current_metadata)
    346     else:
--> 347         self._process_and_add_to_index(item_path, store_collection_with_index, current_doc_id,
    348                                        current_metadata)
    349     self.doc_ids_to_file_names[current_doc_id] = str(item_path)
    350 elif isinstance(item, Image.Image):

Cell In[3], line 379, in DseModel._process_and_add_to_index(self, item, store_collection_with_index, doc_id, metadata)
    377     images = convert_from_path(item)
    378     for i, image in enumerate(images):
--> 379         self._add_to_index(image, store_collection_with_index, doc_id, page_id=i + 1, metadata=metadata)
    380 elif item.suffix.lower() in [".jpg", ".jpeg", ".png", ".bmp"]:
    381     image = Image.open(item)

Cell In[3], line 406, in DseModel._add_to_index(self, image, store_collection_with_index, doc_id, page_id, metadata)
    403 if any(entry["doc_id"] == doc_id and entry["page_id"] == page_id for entry in self.embed_id_to_doc_id.values()):
    404     raise ValueError(f"Document ID {doc_id} with page ID {page_id} already exists in the index")
--> 406 doc_embeddings = self.encode_image(image)
    408 # Add to index
    409 embed_id = len(self.indexed_embeddings)

Cell In[3], line 552, in DseModel.encode_image(self, input_data)
    549 doc_image_inputs, doc_video_inputs = process_vision_info(doc_messages)
    550 doc_inputs = self.processor(text=doc_texts, images=doc_image_inputs, videos=doc_video_inputs, padding='longest',
    551                             return_tensors='pt').to('cuda:0')
--> 552 doc_inputs = self.model.prepare_inputs_for_generation(**doc_inputs, use_cache=False)
    553 with torch.no_grad():
    554     output = self.model(**doc_inputs, return_dict=True, output_hidden_states=True)

File /usr/local/lib/python3.12/dist-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py:1788, in Qwen2VLForConditionalGeneration.prepare_inputs_for_generation(self, input_ids, past_key_values, attention_mask, inputs_embeds, cache_position, position_ids, use_cache, pixel_values, pixel_values_videos, image_grid_thw, video_grid_thw, **kwargs)
   1785         position_ids = position_ids.add(delta)
   1786         position_ids = position_ids.unsqueeze(0).expand(3, -1, -1)
-> 1788 if cache_position[0] != 0:
   1789     pixel_values = None
   1790     pixel_values_videos = None

TypeError: 'NoneType' object is not subscriptable
```

If the exact comprehensive cause and reason are needed, we could wait for the author's answer @MXueguang (https://github.com/MXueguang).


Hi, I am a bit unfamilar with the bug report procedure here, feel free to correct me to the right way. I am here to share the context.

I was training qwen2_vl as a document embedding model and the qwen2 model is used as encoder, and not for generation.
https://huggingface.co/MrLight/dse-qwen2-2b-mrl-v1

However, in the line https://github.com/huggingface/transformers/blob/94f18cf23c128055a984ffbe9c57df133c1f6cc7/src/transformers/models/qwen2_vl/modeling_qwen2_vl.py#L1787-L1790
the if statement here not properly handels the case cache_position is None.

I am not the author of qwen2_vl, so I am not very sure the best way to fix this. I would be very appreciate if someone fix it in the correct way.

Thank you!

@muazhari actually the code snippet from https://huggingface.co/MrLight/dse-qwen2-2b-mrl-v1 is wrong and is missing the following: 
```
cache_position = torch.arange(0, len(query_texts))
query_inputs = model.prepare_inputs_for_generation(**query_inputs, cache_position=cache_position, use_cache=False)
```

Thanks @zucchini-nlp for the useful pointer !


closing the PR as resolved then :)