Beit image classification have different results compared from versions prior to 4.43.0
Author: KaraKaraWitch
Pull request: no
Labels: name: bug

name: Vision

### System Info

- `transformers` version: 4.43.0
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.9
- Huggingface_hub version: 0.26.1
- Safetensors version: 0.4.5
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA GeForce RTX 3060 Ti

### Who can help?

@amyeroberts

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Given the following image:
![image](https://github.com/user-attachments/assets/4c4cf99d-e5fc-40ff-adaf-13d3c1b3d337)

Running the following pipeline for versions prior to `4.43.0` (4.42.4)
```py
from PIL import Image
from transformers import pipeline
import transformers

pipeline_aesthetic = pipeline(
    "image-classification", "cafeai/cafe_aesthetic", device=0
)
with Image.open("F:\\Downloads\\Tower.jpg") as img:
    predictions = pipeline_aesthetic(img, top_k=2)
    predict_keyed = {}
    for p in predictions:
        # print(type(p))
        if not isinstance(p, dict):
            raise Exception("Prediction value is missing?")
        predict_keyed[p["label"]] = p["score"]
    print(predict_keyed,transformers.__version__)
```

For 4.42.4, it returns:
```
{'aesthetic': 0.651885986328125, 'not_aesthetic': 0.3481140434741974} 4.42.4
```
For 4.43.0:
```
{'aesthetic': 0.43069663643836975, 'not_aesthetic': 0.2877475321292877} 4.43.0
```

### Expected behavior

Expected results from 4.42.4 instead of 4.43.0.


### Addn Notes.

I narrowed it down to this commit being the cause: https://github.com/huggingface/transformers/blob/06fd7972acbc6a5e9cd75b4d482583c060ac2ed0/src/transformers/models/beit/modeling_beit.py but unsure where exactly it is changed.

Comments:
cc @molbap @qubvel 

Hi @KaraKaraWitch, thanks! looking into it. I have a hunch though, in your setup, can you run the pipeline with an input image resized to the model's default expected size? It might be due to the interpolation of positional embeddings acting up

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> Hi @KaraKaraWitch, thanks! looking into it. I have a hunch though, in your setup, can you run the pipeline with an input image resized to the model's default expected size? It might be due to the interpolation of positional embeddings acting up

Hi! Sorry I didn't reply. I got held up by other things. I'll get back to this in a bit.

Hey @molbap. Resizing the image to 384x384px doesn't change the result.
![Tower-384](https://github.com/user-attachments/assets/d478af66-61d4-4da8-96d3-02043b399655)


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi, any updates? @molbap @qubvel @LysandreJik

Hi @KaraKaraWitch, I can't pinpoint yet the error but can indeed confirm the issue. I think the interpolation changed during the update you found in `Beit`, shifting the results. If you have bandwidth to investigate more, I'm suspecting the `BeitRelativePositionBias` or the `window_size` usage to be responsible, but I haven't had time to look in-depth yet.

> Hi @KaraKaraWitch, I can't pinpoint yet the error but can indeed confirm the issue. I think the interpolation changed during the update you found in `Beit`, shifting the results. If you have bandwidth to investigate more, I'm suspecting the `BeitRelativePositionBias` or the `window_size` usage to be responsible, but I haven't had time to look in-depth yet.

Hi! Thanks for the update. Unfortunately deep code dive into ML code is not what I excel at... So aftering starting for a bit, nothing to my eyes appears out of the ordinary when comparing code.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi, sorry to reopen and bring this up but i believe this hasn't been resolved.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

not stale bump

...This has been open since **August of last year.**

I know there is a lot of issues to crawl through, but has no one really looked into this?

Hey @KaraKaraWitch, thanks for the ping and sorry for the delay. The issue was fixed quite a while ago, sorry for not letting you know

```
# {'aesthetic': 0.8415985107421875, 'not_aesthetic': 0.1584014743566513} 4.54.0.dev0
# {'aesthetic': 0.8415985107421875, 'not_aesthetic': 0.1584014743566513} 4.53.0
# {'aesthetic': 0.8415985107421875, 'not_aesthetic': 0.1584014743566513} 4.51.0
# {'aesthetic': 0.8415985107421875, 'not_aesthetic': 0.1584014743566513} 4.49.0
# {'aesthetic': 0.6708129644393921, 'not_aesthetic': 0.2772175073623657} 4.45.0
# {'aesthetic': 0.6708129644393921, 'not_aesthetic': 0.2772175073623657} 4.43.0
# {'aesthetic': 0.8415985107421875, 'not_aesthetic': 0.1584014892578125} 4.42.0
```