Add DetrImageProcessorFast
Author: yonigozlan
Pull request: yes
Labels: name: Vision

name: optimization

name: Processing

# What does this PR do?
Adds a fast image processors for DETR. Follows issue #33810.
This image processor is a result of [this work](https://www.notion.so/huggingface2/OptimVision-Optimize-preprocessing-time-10f1384ebcac8091a12debb87fe5f591) on comparing different image processing method.

The processing methods use only [torchvision transforms](https://pytorch.org/vision/stable/transforms.html) (either v1 or v2, depending on the torchvision version) and torch tensors.
Just like the current DETR image processor, this processor can also process object detection or segmentation annotations. This processing also uses only torch tensors and torchvision transforms.
The post-processing methods have not been modified from the original image processor.

## Implementation
A previous fast image processor implementation for VIT ([link](https://github.com/huggingface/transformers/blob/c9afee539204f5e658d03e63a1df3aacb4cab305/src/transformers/models/vit/image_processing_vit_fast.py#L50)) uses torchvision transform classes and `Compose` to create a one step processing class. However this poses two problems:
- The torchvision v2 Transforms are only torch compile/scripting compatible in their functional forms and not in their Class form ([source](https://pytorch.org/vision/stable/transforms.html#torchscript-support)).
- A one step processing class is not possible when the processing depends on the input, like it's the case for DETR for resizing and padding.

So this implementation uses the functional forms of torchvision transforms, and it's structure is very similar to the current DETR image processor.

All the numpy/PIL operations have been converted to torch or torchvision operations, and like the VIT fast image processor, this processor only accept `return_tensors = "pt"`

The processor call function accept a `device` kwarg, as processing can be performed on both CPU and GPU, but is much faster on GPU. 
I wanted to add device as an `init` argument, but that would make the signatures of fast and slow processors different, which make some tests fails.

## Usage
Except for the fact that it only returns torch tensors, this fast processor is fully compatible with the current one.
It can be instantiated through AutoImageProcessor with use_fast=True, or through the Class directly:
```python
from transformers import AutoImageProcessor

processor = AutoImageProcessor.from_pretrained("facebook/detr-resnet-50", use_fast=True)
```
```python
from transformers import DetrImageProcessorFast

processor = DetrImageProcessorFast.from_pretrained("facebook/detr-resnet-50")
```

Usage is the same as the current processor, except for the `device` kwarg:
```python
from torchvision.io import read_image
images = torchvision.io.read_image(image_path)
processor = DetrImageProcessorFast.from_pretrained("facebook/detr-resnet-50")
images_processed = processor(images , return_tensors="pt", device="cuda")
```
If `device` is not specified:
- If the input images are tensors, the processing will be done on the device of the images.
- If the inputs are PIL or Numpy images, the processing is done on CPU.

## Performance gains

### Main Takeaways
#### Processing speedup
- **~60x faster processing on GPU (single image)**
- **~80x faster processing on GPU (batch_size=8)**
- **~5x faster processing on CPU (single image)**
- **~2.6x faster processing on CPU (batch_size=8)**
#### Inference pass speedup (GPU)
- **~2.2x speedup on whole model inference pass (single image, eager)**
- **~3.2x speedup on whole model inference pass (single image, compiled)**
- **~2.4x speedup on whole model inference pass (batch_size=8, eager)**

---

- Average over 100 runs on the same 480x640 image. No padding needed, as "all" the images have the same size.

![benchmark_results_full_pipeline_detr_fast](https://github.com/user-attachments/assets/46129c0c-02ac-485e-9211-afeb68e5fe22)

---

- Average over 10% of the COCO 2017 validation dataset, with  `batch_size=8`. Padding needed, as the images have different sizes, and the DETR processor resize them using "shortest_edge"/"longest_edge", resulting in different sized resized images.

![benchmark_results_full_pipeline_detr_fast_batched](https://github.com/user-attachments/assets/d3c7136d-3f43-45d6-b25f-52e28d36c3b8)

---

- Average over 10% of the COCO 2017 validation dataset, with `batch_size=8`. Forcing padding to 1333x1333 (="longest_edge"), as otherwise torch.compile needs to recompile if the different batches have different max sizes.
(I'm not sure what is going wrong when using the compiled model with the current processor)

![benchmark_results_full_pipeline_detr_fast_batched_compiled](https://github.com/user-attachments/assets/1a2b82a9-44e9-4084-b29f-c072137a1e59)

---

- Average over 10% of the COCO 2017 validation dataset, with `batch_size=1`. Forcing padding to 1333x1333 for comparison with batched inputs

![benchmark_results_full_pipeline_detr_fast_padded](https://github.com/user-attachments/assets/b80b0c77-d81e-40a1-a765-82827f6f24d7)

---

## Tests
- The new image processor is tested on all the tests of the current processor.
- I have also added two consistency tests (panoptic and detection) for processing on GPU vs CPU.


---

Looking forward to your feedback!
I was also wondering if we should adopt a more modular approach to the fast image processors, as there is quite a lot of repetition with the "slow" processor for now. It looks like something like this was done for Fast tokenizers? If someone that worked on Fast tokenizers has any advice on that I'll gladly hear them 🤗.
There will also be the question of how to advertise this "use_fast" option to users, and if we want to make it default eventually when torchvision is available?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34063). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@yonigozlan Hi, super excited to see this. just left two questions
1. Figure says it is `RTDetr` but this is `Detr` right?
2. I think this can be also adapted to all vision processor?
3. Can you also elaborate the computing device?

Hi @SangbumChoi ! 

  1. Oops, thanks for pointing this out, yep it's a typo, this is indeed Detr (but I'm also working on a fast image processor for RTDetr :) ).
  2. Yes absolutely this is the plan. There are quite a few other image processors based on the Detr one so their fast image processor should be out soon.
  3. This was on a A10 GPU! 

@yonigozlan thanks! I would love support this task if you need some help. Also if you can share the comparison script for compile + visualize plot etc... it would be very helpful!