Add common test for `torch.export` and fix some vision models
Author: qubvel
Pull request: yes
Labels: name: Vision

name: run-slow

name: torch export

# What does this PR do?

Add a common **slow** test to check if a model can be exported with no issues using `torch.export.export`

1. Add an optional test, to enable it please set `test_torch_exportable = True` flag for model-specific test.
2. Enable test for vision and video models
3. Fix most of the vision models

The main fixes include:
 - Use a compile-compatible LRU cache for models.
 - Avoid modifying model parameters in the forward pass (e.g. self.param = self.param + x).
 - Avoid modifying leaf in-place tensors created in the forward pass.
 - Avoid creating tensors with `requires_grad=True` in the forward pass.
 
Testing is not complete, there might be code paths that can't be exported. I did additional testing with specific checkpoints. In most cases, we are safe. The only two situations I found where tests pass but checkpoint export does not pass are:
 - beit (fixed)
 - zoedepth (not fixed)

## Results

✅ - can be exported with `torch.export.export`
🔵 - export fixed in this PR
❌ - can't be exported

### Vision models

 - 🔵 beit
 - 🔵 bit
 - 🔵 conditional_detr
 - ✅ convnext
 - ✅ convnextv2
 - ✅ cvt
 - ✅ dab_detr
 - 🔵 deformable_detr
 - ✅ deit
 - ✅ depth_anything
 - ✅ depth_pro
 - 🔵 detr
 - ✅ dinat
 - ✅ dinov2
 - ✅ dinov2_with_registers
 - ✅ dit
 - ✅ dpt
 - ✅ efficientnet
 - 🔵 focalnet
 - ✅ glpn
 - ✅ hiera
 - ✅ ijepa
 - 🔵 imagegpt
 - ❌ levit (low usage, won't fix)
 - ✅ mask2former
 - 🔵 maskformer
 - ✅ mobilenet_v1
 - ✅ mobilenet_v2
 - ✅ mobilevit
 - ✅ mobilevitv2
 - ✅ poolformer
 - ✅ pvt
 - ✅ pvt_v2
 - ✅ regnet
 - ✅ resnet
 - ✅ rt_detr
 - 🔵 rt_detr_v2
 - ✅ segformer
 - 🔵 seggpt
 - ❌ superpoint (data-dependent [expression](https://github.com/huggingface/transformers/blob/main/src/transformers/models/superpoint/modeling_superpoint.py#L320))
 - ✅ swiftformer
 - ✅ swin
 - ✅ swinv2
 - 🔵 swin2sr
 - ✅ table_transformer
 - ✅ textnet
 - ✅ upernet
 - ✅ vit
 - ✅ vitdet
 - ✅ vit_mae
 - ✅ vitmatte
 - ✅ vit_msn
 - ✅ vitpose
 - ✅ vitpose_backbone
 - ✅ yolos
 - ❌  zoedept (data-dependent expression, test config pass but checkpoint not)

### Video models

 - ✅ timesformer
 - ✅ vivit
 - ✅ videomae

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35124). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@guangy10 please have a look if you have bandwidth! Do you have anything in mind that should be added to the common test on the side of the `transformers` to ensure the model is exportable and executorch compatible?

Ok, sure, what is a new way to trigger slow tests?

Thanks for review 🤗 

Second part in 

https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804

or my message in slack 

https://huggingface.slack.com/archives/C01NE71C4F7/p1734527174922859

run-slow rt_detr

run-slow: rt_detr

This comment contains run-slow, running the specified jobs: ['models/rt_detr'] ...

run-slow: rt_detr

This comment contains run-slow, running the specified jobs: ['models/rt_detr'] ...

run-slow: beit, bit, conditional_detr, data2vec, deformable_detr, detr, focalnet, imagegpt, maskformer, rt_detr, seggpt, swin2sr

This comment contains run-slow, running the specified jobs: ['models/beit', 'models/bit', 'models/conditional_detr', 'models/data2vec', 'models/deformable_detr', 'models/detr', 'models/focalnet', 'models/imagegpt', 'models/masformer', 'models/rt_detr', 'models/seggpt', 'models/swin2sr'] ...

@qubvel Let me know the timeline. I can start working on e2e enablement by connecting these models to [optimum-executorch](https://github.com/huggingface/optimum-executorch) once this PR merged.

Hey @guangy10, going to finish this week!

> Hey @guangy10, going to finish this week!

@qubvel Just a friendly reminder to merge this PR. Let mw know how I can help if there is any blocker.

Nice work in bringing the torch.export coverage up.

Are the export tests executed with torch==2.6.0?

Hey @chrsmcgrr, yes we have 2.6.0 in CI. I have also run these tests with torch 2.5.0 locally

run-slow: rt_detr_v2, dab_detr, beit, conditional_detr, deformable_detr, detr, swin2sr

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/beit', 'models/conditional_detr', 'models/dab_detr', 'models/deformable_detr', 'models/detr', 'models/rt_detr_v2', 'models/swin2sr']
quantizations: [] ...

Test failures are unrelated (fixed in https://github.com/huggingface/transformers/pull/35654). 

Merging this PR to unblock @guangy10 work. I will work on docs in the follow-up PR asap. cc @stevhliu in case you have suggestions on how to organize it better.

Depending on when the new docs ship (hopefully this week or the next), you can add them [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31757/en/executorch) I think :)

If it takes longer, then feel free to add them [here](https://huggingface.co/docs/transformers/main/en/main_classes/executorch) and I can move them later!

> Depending on when the new docs ship (hopefully this week or the next), you can add them [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31757/en/executorch) I think :)
> 
> If it takes longer, then feel free to add them [here](https://huggingface.co/docs/transformers/main/en/main_classes/executorch) and I can move them later!

Feel free to subscribe me on the new PR for the doc fix.

@qubvel While I'm working on lowering these vision models e2e to ExecuTorch in `optimum`, would you like to start expanding the export coverage over audio models as well? I’d really appreciate the effort!

@guangy10 I'm not sure if I will have bandwidth in the coming few weeks, but maybe someone from the audio team can have a look
cc @eustlb

@qubvel Thanks for looping in the audio team! 

👋 @eustlb, nice to e-meet you! I'm from PyTorch team at Meta. We've been collaborating with 🤗 to expand `torch.export` coverage on transformer models since last year *(FYI there is a parallel efforts focusing on `torch.compile` coverage)*. So far we have covered text and vision models, but not on audio models yet. It seems like your team is the right one to talk. I'd happy to share more context in slack.

Hey @guangy10, nice to e-meet you too! It would be a pleasure to help 🤗