mllama_modeling.py file attention_mask = aspect_ratio_mask.reshape(batch_size * num_concurrent_media, -1) Got errored for Nonetype aspect_ratio_mask
Author: youky860423
Pull request: no
Labels: name: bug

### System Info

File "/data/anaconda3/envs/llama/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/llama/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/llama/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/anaconda3/envs/llama/lib/python3.12/site-packages/transformers/models/mllama/modeling_mllama.py", line 1500, in forward
    attention_mask = aspect_ratio_mask.reshape(batch_size * num_concurrent_media, -1)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'reshape'

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. given custom data of {
            'input_ids': input_ids,
            'attention_mask': attention_masks,
            'pixel_values': pixel_values,
            'aspect_ratio_ids': aspect_ratio_ids_stacked,
            'labels': labels
        }
 2. calling trainer.train() # model is MllamaForConditionalGeneration.from_pretrained
 3. Got an error for aspect_ratio_mask is Nonetype on forward function

### Expected behavior

would espect the model would train on the given data input

Comments:
hey @youky860423 !

Can you provide a small reproducer, that would make it easier for us to help you :)

Thanks, I think the main issue is aspect_ratio_mask is not supposed to be given by custom data input, right? But the forward function code in mllama reshape this aspect_ratio_mask before assigning it, and there is no check on whether it is None or not. I am giving 'pixel_values': pixel_values, and 'aspect_ratio_ids': aspect_ratio_ids_stacked as in the data_collator, there is no part that can auto-generate this  aspect_ratio_mask based on aspect_ratio_ids.

The aspect ratio mask should be returned by image processor and thus by the processor when you call it with image+text inputs. You can see in the below code where the mask is built and returned. Make sure you data collator returns all the keys that are present in the input dict, I guess the collator was selectively getting some keys only instead of passing all 

https://github.com/huggingface/transformers/blob/b0f0c61899019d316db17a493023828aa44db06d/src/transformers/models/mllama/image_processing_mllama.py#L805-L815

Thanks, the issue was resolved by me, I used Autoprocessor for my combined data inputs and give that to the model

closing as resolved then :)