TypeError: 'NoneType' object is not iterable
Author: 0xD4rky
Pull request: no

I was trying to fine-tune the qwen-vl2-7b instruct model on a custom dataset. For that, I wrote the following collate function:

```
@dataclass
class Collator:
    processor: ProcessorMixin

    def __call__(self, batch: List[Dict[str, Any]]) -> Dict[str, torch.Tensor]:

        images = [item["image"] for item in batch]
        texts = [item["text"] for item in batch]
        
        conversations = []
        for txt in texts:
            conversation = [
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "image",
                        },
                        {
                            "type": "text",
                            "text": "Given this pallet load manifest image, provide the following information as JSON: " + txt
                        }
                    ]
                }
            ]
            conversations.append(conversation)
        
        text_prompts = [
            self.processor.apply_chat_template(conv, add_generation_prompt=True)
            for conv in conversations
        ]
        vision_inputs = self.processor.image_processor(
            images=images,
            return_tensors="pt",
            do_resize=True,
            size={'height': 448, 'width': 448}
        )

        inputs = self.processor(
            text=text_prompts,
            padding=True,
            return_tensors="pt"
        )
        
        processed_inputs = {
            "input_ids": inputs.input_ids,
            "attention_mask": inputs.attention_mask,
            "pixel_values": vision_inputs.pixel_values,
            "labels": inputs.input_ids.clone() 
        }
        
        return processed_inputs
```


When I am trying to run the training script, this error pops up no matter how I handle image processing:

```
[/usr/local/lib/python3.11/dist-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py](https://localhost:8080/#) in rot_pos_emb(self, grid_thw)
    994     def rot_pos_emb(self, grid_thw):
    995         pos_ids = []
--> 996         for t, h, w in grid_thw:
    997             hpos_ids = torch.arange(h).unsqueeze(1).expand(-1, w)
    998             hpos_ids = hpos_ids.reshape(

TypeError: 'NoneType' object is not iterable
```

Please help me out on this!

Comments:
Hey @0xD4rky !

I'd recommend to pass images and text to `inputs = self.processor(images, text)` so instead of doing a separate `image_processor` call. In case you need to overwrite any image processor attributes, you can directly pass them as `processor(do_resize=True, **other_kwargs_for_image)` 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Facing the same issue with video understanding. Everything works fine with single image. 

```
python -m mlx_vlm.video_generate --model mlx-community/Qwen2-VL-2B-Instruct-4bit --max-tokens 100 --prompt "Describe this video" --video video.mp4  --fps 1.0
This is a beta version of the video understanding. It may not work as expected.
<frozen runpy>:128: RuntimeWarning: 'mlx_vlm.video_generate' found in sys.modules after import of package 'mlx_vlm', but prior to execution of 'mlx_vlm.video_generate'; this may result in unpredictable behaviour
This is a beta version of the video understanding. It may not work as expected.
Loading model: mlx-community/Qwen2-VL-2B-Instruct-4bit
Fetching 11 files: 100%|█████████████████████████████████████████████████████████| 11/11 [00:00<00:00, 163607.60it/s]
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
max_pixels: 10000
numpy reader: video_path=video.mp4, total_frames=77, video_fps=25.0, time=0.000s
Generating response...
==========
Files: ['video.mp4'] 

Prompt: 
system
You are a helpful assistant.


user



Describe this video


assistant

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/parsa.morsal/.pyenv/versions/newenv/lib/python3.12/site-packages/mlx_vlm/video_generate.py", line 604, in <module>
    main()
  File "/Users/parsa.morsal/.pyenv/versions/newenv/lib/python3.12/site-packages/mlx_vlm/video_generate.py", line 591, in main
    response = generate(
               ^^^^^^^^^
  File "/Users/parsa.morsal/.pyenv/versions/newenv/lib/python3.12/site-packages/mlx_vlm/utils.py", line 1115, in generate
    for response in stream_generate(model, processor, prompt, image, **kwargs):
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/parsa.morsal/.pyenv/versions/newenv/lib/python3.12/site-packages/mlx_vlm/utils.py", line 1036, in stream_generate
    for n, (token, logprobs) in enumerate(
                                ^^^^^^^^^^
  File "/Users/parsa.morsal/.pyenv/versions/newenv/lib/python3.12/site-packages/mlx_vlm/utils.py", line 949, in generate_step
    outputs = model(input_ids, pixel_values, cache=cache, mask=mask, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/parsa.morsal/.pyenv/versions/newenv/lib/python3.12/site-packages/mlx_vlm/models/qwen2_vl/qwen2_vl.py", line 116, in __call__
    input_embddings = self.get_input_embeddings(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/parsa.morsal/.pyenv/versions/newenv/lib/python3.12/site-packages/mlx_vlm/models/qwen2_vl/qwen2_vl.py", line 70, in get_input_embeddings
    hidden_states = self.vision_tower(
                    ^^^^^^^^^^^^^^^^^^
  File "/Users/parsa.morsal/.pyenv/versions/newenv/lib/python3.12/site-packages/mlx_vlm/models/qwen2_vl/vision.py", line 292, in __call__
    rotary_pos_emb = self.rot_pos_emb(grid_thw)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/parsa.morsal/.pyenv/versions/newenv/lib/python3.12/site-packages/mlx_vlm/models/qwen2_vl/vision.py", line 249, in rot_pos_emb
    for t, h, w in grid_thw:
                   ^^^^^^^^
TypeError: 'NoneType' object is not iterable
```

@parsamorsal can you share a minimal reproducer?

Also you can make sure that the `grid_thw` and `pixel_values` are actually present in the input dict before generating