Accuracy regression of ViT
Author: blzheng
Pull request: no

### System Info

- `transformers` version: 4.25.0.dev0
- Platform: Linux-5.15.0-53-generic-x86_64-with-glibc2.35
- Python version: 3.9.17
- Huggingface_hub version: 0.19.4
- PyTorch version (GPU?): 2.0.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

@amyeroberts

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


Accuracy regression caused by https://github.com/huggingface/transformers/pull/19796


Reproduce command:
python transformers/examples/pytorch/image-classification/run_image_classification.py --model_name_or_path google/vit-base-patch16-224 --do_eval --dataset_name imagenet-1k --per_device_eval_batch_size 1 --remove_unused_columns False --output_dir ./

### Expected behavior

Expected results:  
  eval_accuracy           =     0.8131
  eval_loss               =     0.7107
  eval_runtime            = 0:43:40.30
  eval_samples_per_second =     19.082
  eval_steps_per_second   =     19.082
Current results:
  eval_accuracy           =     0.8033
  eval_loss               =      0.755
  eval_runtime            = 0:34:05.81
  eval_samples_per_second =      24.44
  eval_steps_per_second   =      0.436

Comments:
Hi @blzheng, thanks for raising this issue! 

#19796 has been merged in for over a year now, and there have been a few subsequent updates to the image processing logic. Could you confirm how you narrowed it down to this commit? 

What performance do you get, running on main with different seeds? 

Hi @amyeroberts , we observed accuracy drop from 0.8131 (transformers==4.18.0) to 0.8033 (transformers==4.28.1), then I narrowed down to this commit with git bisect. This issue can be reproduced stably by running the following command. Even with the latest codebase, this issue still exists.
"python transformers/examples/pytorch/image-classification/run_image_classification.py --model_name_or_path google/vit-base-patch16-224 --do_eval --dataset_name imagenet-1k --per_device_eval_batch_size 1 --remove_unused_columns False --output_dir ./"

@blzheng Thanks for confirming. 

The reason for the change is because the processing logic in the image classification script was updated to reflect that of the model's image processor.

Previously, `size` could be an int, and was passed directly to `torchvision.transforms.Resize`. If `size` is an int (which it is for many model's e.g. [here for a vit checkpoint](https://huggingface.co/google/vit-base-patch16-224/blob/3f49326eb077187dfe1c2a2bb15fbd74e6ab91e3/preprocessor_config.json#L14)), then the [shortest edge of the image is resized to `size` and the other edge rescaled to keep the image's aspect ratio](https://pytorch.org/vision/main/generated/torchvision.transforms.Resize.html).

However, in the now-deprecated feature extractors (superceeded in #19796), the [default behaviour](https://github.com/huggingface/transformers/blob/aa4a0f8ef37eb5d42b4e3810f37e554585c90d41/src/transformers/image_utils.py#L543) if `size` was an int, was to resize the image to `(size, size)`. This [was the case of ViT](https://github.com/amyeroberts/transformers/blob/a23819ed6ab852df6d8f04815306440531418260/src/transformers/models/vit/feature_extraction_vit.py#L144). 

The script now reflects the behaviour of the image processor, even when using torchvision transforms. 

@amyeroberts Thanks for your information. 
Now that the changes in image processing logic are reasonable, does it mean the accuracy drop is expected?

@blzheng It depends what you mean by "expected". The change in the logic means that the aspect ratio of the input images is different, and so one would expect there to be a performance difference. Even though it's not in-line with the processing of the model's image processors, the previous processing might bring better performance because it preserves the true aspect of images and hence shape/dimensions of the subjects in the image (this is speculation).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.