Verify interpolation of image processors
Author: NielsRogge
Pull request: no
Labels: name: Good First Issue

### Feature request

As pointed out in https://github.com/huggingface/transformers/pull/27742, some image processors might need a correction on the default interpolation method being used (resampling in Pillow). We could check this on a per-model basis.

### Motivation

Interpolation methods have a slight (often minimal) impact on performance. However it would be great to verify this on a per-model basis.

e.g. [ViT](https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit/image_processing_vit.py#L52)'s image processor defaults to BILINEAR but should use BICUBIC as seen [here](https://github.com/huggingface/pytorch-image-models/blob/main/timm/models/vision_transformer.py#L1062). We can update the default values of the image processors, but can't update the configs on the hub as this would break people's fine-tuned models.

### Your contribution

I could work on this, but this seems like a good first issue for first contributors.

To be checked (by comparing against original implementation):

- [ ] beit
- [ ] bit
- [ ] clip
- [ ] convnext
- [ ] convnextv2
- [ ] cvt
- [ ] data2vec-vision
- [ ] deit
- [ ] dinat
- [ ] dinov2
- [ ] efficientformer
- [ ] efficientnet
- [ ] focalnet
- [ ] imagegpt
- [ ] levit
- [ ] mobilenet_v1
- [ ] mobilenet_v2
- [ ] mobilevit
- [ ] mobilevitv2
- [ ] nat
- [ ] perceiver
- [ ] poolformer
- [ ] pvt
- [ ] regnet
- [ ] resnet
- [ ] segformer
- [ ] siglip
- [ ] swiftformer
- [ ] swin
- [ ] swinv2
- [ ] van
- [ ] vit
- [ ] vit_hybrid
- [ ] vit_msn

Comments:
@NielsRogge Thanks for opening the issue! 

It's fine to open up to the community but you'll need to add a checklist of the image processors so it's clear who is working on what and what's done as well as ideally some instructions on what it means for each one to be "done" e.g. making sure to run slow tests for models. 

@NielsRogge ,

If I understand it correctly, we need to match the interpolation:

For example for convnext: [convnext](https://github.com/huggingface/transformers/blob/main/src/transformers/models/convnext/image_processing_convnext.py#L93) should be changed to Bicubic as per [timm/convnext](https://github.com/huggingface/pytorch-image-models/blob/main/timm/models/convnext.py#L498) .

If that's correct , I can take this up for all the models. Let me know.

Yes that is correct, see also the [original implementation](https://github.com/facebookresearch/ConvNeXt/blob/048efcea897d999aed302f2639b6270aedf8d4c8/main.py#L107). Thanks for spotting that. Hence feel free to open a PR to update this, along with the image processor created in the [conversion script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/convnext/convert_convnext_to_pytorch.py#L149-L150). Ideally we assert the pixel values created by it against the original implementation, like done [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/dinov2/convert_dinov2_to_hf.py#L197-L220) for DINOv2.

Sure! thanks for the pointers, will work on it.

DieT and DPT default interpolation types matches with the original implementation types to BICUBIC . That's what I see it. Let me know  if  I overlooked.



@NielsRogge ,

would you have a look ?

@NielsRogge Can you please complete the checklist here?

@NielsRogge 
I would like to take this issue up.

@NielsRogge , @nileshkokane01  Can I work on this issue and help in completing the checklist ?

@NielsRogge , is it possible for me to contribute ? Thanks.

@NielsRogge , can I work on this issue?

Please don't ask to work on it, just open a PR for one of the models which are still open/which doesn't have a merged PR yet :)

Thanks.

#43017 Fix default interpolation for PVT to match original implementation [(Bicubic)] @NielsRogge 

Hi @NielsRogge,

I’d like to work on fixing the default interpolation for the SegFormer
image processor by verifying it against the original implementation
and adding a test similar to existing vision models.

Please let me know if that’s okay. Thanks!


I'm fixing MobileNet v1 and v2 - they currently default to BILINEAR but should use BICUBIC to match the timm implementation. Will submit a PR shortly.

Hi @NielsRogge, I'd like to work on fixing the default interpolation for the ResNet image processor by verifying it against the original torchvision/timm implementation and adding a pixel-value assertion test similar to existing vision models.


Closing this because it's attracting way too many code agents, and they're not handling it well!