Vision models don't work for non-square object
Author: liujilei2233
Pull request: no
Labels: name: Good First Issue

name: Vision

### System Info

- `transformers` version: 4.47.0
- Platform: Linux-6.8.0-50-generic-x86_64-with-glibc2.39
- Python version: 3.9.20
- Huggingface_hub version: 0.26.5
- Safetensors version: 0.4.5
- Accelerate version: 1.2.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 4060 Ti


### Who can help?

 @amyeroberts, @qubvel

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import ViTMAEConfig,ViTMAEForPreTraining
config = ViTMAEConfig(image_size=tuple((16,1024)),patch_size=tuple((4,256)))
model = ViTMAEForPreTraining(config)
```

```
TypeError                                 Traceback (most recent call last)
Cell In[40], line 3
      1 from transformers import ViTMAEConfig,ViTMAEForPreTraining
      2 config = ViTMAEConfig(image_size=tuple((16,1024)),patch_size=tuple((4,256)))
----> 3 model = ViTMAEForPreTraining(config)

File ~/miniconda3/lib/python3.9/site-packages/transformers/models/vit_mae/modeling_vit_mae.py:994, in ViTMAEForPreTraining.__init__(self, config)
    991 self.config = config
    993 self.vit = ViTMAEModel(config)
--> 994 self.decoder = ViTMAEDecoder(config, num_patches=self.vit.embeddings.num_patches)
    996 # Initialize weights and apply final processing
    997 self.post_init()

File ~/miniconda3/lib/python3.9/site-packages/transformers/models/vit_mae/modeling_vit_mae.py:852, in ViTMAEDecoder.__init__(self, config, num_patches)
    846 self.decoder_layers = nn.ModuleList(
    847     [ViTMAELayer(decoder_config) for _ in range(config.decoder_num_hidden_layers)]
    848 )
    850 self.decoder_norm = nn.LayerNorm(config.decoder_hidden_size, eps=config.layer_norm_eps)
    851 self.decoder_pred = nn.Linear(
--> 852     config.decoder_hidden_size, config.patch_size**2 * config.num_channels, bias=True
    853 )  # encoder to decoder
    854 self.gradient_checkpointing = False
    855 self.config = config

TypeError: unsupported operand type(s) for ** or pow(): 'tuple' and 'int'

```

### Expected behavior

I want ViTMAEForPreTraining to analyze my biological data, which should get a normal output even though it's not a real image. I've noticed that your model doesn't take into account some irregular inputs, such as non-square images, resulting in errors. So, hopefully the code can be improved to accommodate a wider range of inputs.

Comments:
Hi, @liujilei156231! Thanks for opening the issue! Indeed, some models are not working with square images, and we would appreciate a contribution to fix this behavior. Would you like to help us with this?

Reopening this issue to enable more vision models to support non-square images

@qubvel Hi! I'm interested in this issue. Do you have any recommended vision models that could help improve Hugging Face? If not, I can try to fix VitDet.