Optim deformable detr
Author: yonigozlan
Pull request: yes
Labels: name: run-slow

# What does this PR do?
This PR is part of an ongoing effort to optimize the inference speed of Transformers' vision models. (along with #33412 for now).
For more info on the specific issues these optimizations target and how they help improve the inference speed of compiled models, you can check out this [Notion page](https://www.notion.so/huggingface2/Vision-models-compilation-and-OptimVision-first-results-e2b47dabd1804cdc9cac5537ae5e9363).

The following metrics are for model inference only! Pre/post-processing time are not measured here. Currently, Transformers image processors seem to be a big bottleneck for inference speed, so end-to-end inference will sadly be much slower than this.

A few things to note:
- Other models which use deformable attention were modified due to "copied from", but the modifications should be non breaking. I haven't measured it but those models could also be slightly faster due to it :).
- Grounding Dino in particular could really benefit from more changes than those that were simply copied, but this could be part of another PR.
- Contrary to the optimization done on RT-DETR, here  both eager and compiled models are improved :).
- I have no idea what is going on with the compiled fp16 model without custom kernel. The trace shows full use of the GPU, and similar changes were made on RT-DETR with considerable gain in the same settings. If anyone has a theory I would be glad to hear it :).
- Deformable DETR (unlike RT-DETR) uses the custom kernel for deformable attention by default. But the user needs to have `ninja-build` installed for it to work.

![benchmark_results_deformable_detr_nightly](https://github.com/user-attachments/assets/6b4c7051-5fa7-4e4d-96bc-e4616b470e3b)


<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@qubvel @molbap @amyeroberts @NielsRogge 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33600). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey! 🤗 Thanks for your contribution to the `transformers` library!

Before merging this pull request, slow tests CI should be triggered. To enable this:
- Add the `run-slow` label to the PR
- When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command `[run-slow]` followed by a comma separated list of all the models to be tested, i.e. `[run_slow] model_to_test_1, model_to_test_2`
  - If the pull request affects a lot of models, put at most 10 models in the commit message
- A `transformers` maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is [here](https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804).

Hi, I'm coming to this PR a month late, but I noticed there are some changes here that I think would be great to transfer over to other models that are down-stream of DeformableDetr, specifically Mask2Former.

I would love to contribute these changes, but to do this properly, I'd love if I could get more context on a few things that I couldn't discover from this PR, because the Notion doc is not public :)

1. How do you run the eager/compiled versions of these models? I tried a very naive pass at pulling a DeformableDetr model into `torch.compile` with no arguments, but that failed. Is there another path?
2. What is the underlying motivation here? Do you see users compile these models to run them faster in production environments?

For context, I'm trying to use the new-ish [torch.export() non-strict path](https://pytorch.org/tutorials/intermediate/torch_export_tutorial.html#non-strict-export) to export Mask2Former and I noticed that the changes in this PR include some of the same changes I would need to make to the Mask2Former modeling code. I would love to contribute these changes and have them align with the optimization objective you have in this PR!

Hi @philkuz!
Great to see some interest in propagating these changes and making Transformers models more torch compile compatible :).
Here is a [published version of the notion page](https://huggingface2.notion.site/Vision-models-compilation-and-OptimVision-first-results-e2b47dabd1804cdc9cac5537ae5e9363), I haven't had the time to polish it and it's not quite up to date, but hopefully it can help answer some of your questions.

As for the motivation, in general I think ensuring Transformers models are fully optimized for compilation sets a nice standard. There has been more and more issues and feature requests related to torch compilation, so there seems to be growing interest in using it in production. With torch.compile and torch.export gaining traction, especially for performance-critical applications like real-time vision, I can see these optimizations becoming increasingly valuable.