[PoC] HF exporters
Author: IlyasMoutawwakil
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=41992)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=41992)
<!-- ci-dashboard-badge:end -->

# HfExporters: Native, Unified export for PyTorch / ONNX / ExecuTorch

<img alt="thumbnail" src="https://github.com/user-attachments/assets/3ba5751b-c99e-4945-b5a8-b2b29231f5df" />

> **Edit**: pieces of this PR were merged separately ([#42697](https://github.com/huggingface/transformers/pull/42697), [#42317](https://github.com/huggingface/transformers/pull/42317)), so this one is now focused on the **HfExporters** machinery itself.

## What this adds

A native, in-Transformers export pipeline — one base class (`HfExporter`), three subclasses for the runtimes we care about, one unified API:

| Exporter | Output | Runtime |
|---|---|---|
| `DynamoExporter` | `ExportedProgram` | Any PyTorch runtime, AOT compilation |
| `OnnxExporter` | `ONNXProgram` | Any ONNX runtime (ORT, TensorRT, OpenVINO, …) |
| `ExecutorchExporter` | `ExecutorchProgramManager` | Mobile and edge (ExecuTorch) |

Same call shape across all three. Dynamic shapes by default. Generation-style models split automatically into prefill + decode (+ vision/audio sub-encoders for VLMs).

```python
from transformers import AutoModelForMaskedLM, AutoTokenizer
from transformers.exporters import OnnxExporter, OnnxConfig

model_id = "hf-internal-testing/tiny-random-BertForMaskedLM"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id).eval()
inputs = tokenizer(["Hello, my dog is cute"] * 2, return_tensors="pt")
onnx_program = OnnxExporter().export(model, inputs, config=OnnxConfig(dynamic=True))

new_input = tokenizer("Hello, my cat is so adorable!", return_tensors="pt")
torch.testing.assert_close(
    onnx_program.call_reference(**new_input)[0],   # numpy reference
    onnx_program(**new_input)[0],                  # onnxruntime
    rtol=1e-4, atol=1e-4,
)
```

Swap one line for another runtime — `DynamoExporter()` / `DynamoConfig` or `ExecutorchExporter()` / `ExecutorchConfig(backend=...)`.

For generative models the prefill/decode split is captured automatically:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.exporters import OnnxExporter, OnnxConfig

model_id = "hf-internal-testing/tiny-random-LlamaForCausalLM"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).eval()
inputs = tokenizer(["Hello, my dog is cute"] * 2, return_tensors="pt")

artifacts = OnnxExporter().export_for_generation(model, inputs, config=OnnxConfig(dynamic=True))
# {"prefill": ONNXProgram, "decode": ONNXProgram}
# For VLMs: also vision_encoder, audio_encoder, multi_modal_projector, language_model, lm_head
```

## Why native, not via Optimum

`torch.export` is fast and clean but **strict** compared to TorchScript — data-dependent `if`s that TorchScript would just trace through (with a warning) become hard guard failures, hence the `if not torch.compiler.is_exporting()` opt-outs landing throughout modeling code. Living downstream in Optimum means every new architecture (a new attention pattern, a new cache type, a new modality) waits for someone to write export glue. By colocating the exporters with the models, **the export evolves with the model** and breakages surface in the model's own test file, not weeks later in a separate repo.

Optimum will keep building on this — generating inputs, dynamic axes, splitting components, wiring inference. This PR is the API and the registry it builds on.

## How it works

`export_for_generation` runs the model's real `generate(..., max_new_tokens=2)` once, hooks `forward` to capture the actual prefill and decode kwargs, then exports each independently. Multimodal models decompose further into per-component sub-graphs. Decoder-only, SSM, encoder-decoder — same machinery.

### Multi-stage patching

Every backend follows the same pipeline (5 stages for ONNX, 4 for the others). The principle: **fix exports without touching modeling code** wherever possible. Each fix lives at the lowest stage that can express it cleanly.

| Stage | Mechanism | When to use |
|---|---|---|
| 1. Torch patches | `@register_patch(backend, "torch.X.method")` reversible class-attribute swap during tracing | The op's graph structure needs to change (no ONNX equivalent, missing decomposition, data-dependent branch) |
| 2. FX node fixes | `@register_fx_node_fix(backend)` per-node rewriters on the post-export graph | Functional-graph violations (in-place ops, alias chains), dead symbolic-guard nodes |
| 3. FX program fixes | `@register_fx_program_fix(backend)` program-level passes on the `ExportedProgram` | Structural repairs (signature alignment, cross-graph rewrites) |
| 4. ONNX translations | `_ONNX_TRANSLATION_TABLE` custom onnxscript per aten op | Op traces correctly but lacks a correct ONNX lowering |
| 5. ONNX IR fixes | post-export mutation of the `ONNXProgram` | Backend-specific quirks (e.g. `TopK(sorted=1)` for ORT) |

`@register_export_input_preparer(*markers)` covers a related case: precompute data-dependent kwargs (`cu_seqlens`, position ids, window indices, bilinear interpolation grids, audio chunking) outside the traced graph.

**Adding support for a new model is a function and a decorator.** No new files, no plumbing, no per-backend boilerplate.

### Other moving parts

- **Dynamic shapes by default.** `dynamic=True` walks tensors / caches / dicts / lists / `ModelOutput`s and assigns `Dim.AUTO` everywhere — no per-model `dynamic_shapes` dict.
- **Cache pytree registration is reflection-driven.** `Cache.__subclasses__()` is walked at export time and each registered as a pytree node; model-specific cache types (anything ending in `Cache`) get picked up too. `past_key_values` becomes just another tensor structure to the exporter.
- **State cleanup after export.** Stateful tensor attrs (sliding-window buffers, etc.) that `torch.export` may leave as `FakeTensor`s are reset so a follow-up eager forward on the same module stays safe.
- **Forward-signature flattening.** `**kwargs: Unpack[TransformersKwargs]` is rewritten to a flat signature derived from the user's inputs — otherwise `torch.export` bundles `**kwargs` into one blob that doesn't line up with `dynamic_shapes`.

## What's tested

The export tests are wired into `ModelTesterMixin` directly, so **every model that uses the standard test mixin gets `test_torch_export_*`, `test_onnx_export_*`, and `test_executorch_export_*` for free** (forward + generation variants where relevant). That's the bulk of the model zoo — LLMs, VLMs, audio encoders, vision backbones, detection / segmentation.

Skip lists in `tests/exporters/test_utils.py` are short, scoped, and each entry points to a specific upstream `torch.export` / `onnxscript` issue or modeling bug we want to fix at the source — not paper over here.

## What's not perfect yet

- `export_for_generation` returns **independent graphs**, not a turnkey inference pipeline. Wiring them together (encoder → projector → language model → lm_head, decode loop) is downstream's job. We're actively working to reduce that glue.
- Some workarounds in the registries will disappear when PyTorch or onnxscript lands the equivalent fix. Each is marked with the issue it works around — keeping these visible is the point.

## Reproducing

```bash
pip install -e . torch onnxscript executorch onnxruntime
```

Then any of the snippets above; full per-model tests:

```bash
RUN_SLOW=1 pytest tests/models/<model>/test_modeling_<model>.py \
    -k "test_onnx_export or test_torch_export or test_executorch_export"
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41992). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Currently all models (except a select few) are tested and pass the tests successfully !

`389 passed, 87 skipped, 413 warnings in 143.73s (0:02:23)`

skipped tests either: 
- explicitly skipped with `test_torch_exportable = False`, this is for custom cache models and some MoEs (15).
- errors with an informative error `torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNod` (67).
- errors with a cryptic `Expected cond to be True, but got False..` (16).

run-slow: dia, efficientloftr, ernie4_5_vl_moe, falcon_mamba, flava, glm4v, glm4v_moe, glm_image, glm_moe_dsa, glm_ocr

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/24151893277)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/dia", "models/efficientloftr", "models/ernie4_5_vl_moe", "models/falcon_mamba", "models/flava", "models/glm4v", "models/glm4v_moe", "models/glm_image", "models/glm_moe_dsa", "models/glm_ocr"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/24151893277)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [c60b877d](https://github.com/huggingface/transformers/commit/c60b877d9dc9b4eaaf95e43bcc43c3fa0268713a) | workflow commit (merge commit) |
| PR | [7f9382c9](https://github.com/huggingface/transformers/commit/7f9382c9553a4a7619338df692f5646330b9b288) | branch commit (from PR) |
| main | [380e3cc5](https://github.com/huggingface/transformers/commit/380e3cc5d59912a48508cb6d4959a31cd460e12e) | base commit (on `main`) |

### Model CI Report

❌ **[6 new failed tests from this PR](https://huggingface.co/datasets/hf-internal-testing/transformers_pr_ci/raw/5d486d73d38e64c1ef3119047a544ee89d6715d7/2026-04-08/runs/32541-24151893277/ci_results_run_models_gpu/new_failures_with_bad_commit.json)** 😭

- [glm4v](https://github.com/huggingface/transformers/actions/runs/24151893277/job/70481170291):
    tests/models/glm4v/test_modeling_glm4v.py::Glm4vModelTest::test_get_video_features_attentions (✅ ⟹ ❌)
    tests/models/glm4v/test_modeling_glm4v.py::Glm4vModelTest::test_get_video_features_hidden_states (✅ ⟹ ❌)
    tests/models/glm4v/test_modeling_glm4v.py::Glm4vModelTest::test_get_video_features_output_0 (✅ ⟹ ❌)
    tests/models/glm4v/test_modeling_glm4v.py::Glm4vModelTest::test_get_video_features_output_1 (✅ ⟹ ❌)
    tests/models/glm4v/test_modeling_glm4v.py::Glm4vModelTest::test_get_video_features_output_2 (✅ ⟹ ❌)
    tests/models/glm4v/test_modeling_glm4v.py::Glm4vIntegrationTest::test_small_model_integration_test_with_video (✅ ⟹ ❌)



run-slow: dia, efficientloftr, ernie4_5_vl_moe, falcon_mamba, flava, glm46v, glm4v, glm4v_moe, glm_image, glm_moe_dsa

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/24230754520)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/dia", "models/efficientloftr", "models/ernie4_5_vl_moe", "models/falcon_mamba", "models/flava", "models/glm46v", "models/glm4v", "models/glm4v_moe", "models/glm_image", "models/glm_moe_dsa"]
quantizations: []

run-slow: dia, efficientloftr, ernie4_5_vl_moe, falcon_mamba, flava, glm46v, glm4v, glm4v_moe, glm_image, glm_moe_dsa, glm_ocr

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/24230754520)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [a4365fea](https://github.com/huggingface/transformers/commit/a4365feac25191207db2d65ad8367dadc89e1094) | workflow commit (merge commit) |
| PR | [f30fa969](https://github.com/huggingface/transformers/commit/f30fa969d9487e3470e809e9046cbe38ed848cba) | branch commit (from PR) |
| main | [1e931b8f](https://github.com/huggingface/transformers/commit/1e931b8fcafa19dc82b2a482898098e4d15aca81) | base commit (on `main`) |

⚠️ No test being reported (jobs are skipped or cancelled)!


[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/24233194008)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/dia", "models/efficientloftr", "models/ernie4_5_vl_moe", "models/falcon_mamba", "models/flava", "models/glm46v", "models/glm4v", "models/glm4v_moe", "models/glm_image", "models/glm_moe_dsa", "models/glm_ocr"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/24233194008)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [49ec14f9](https://github.com/huggingface/transformers/commit/49ec14f9d537713cbc77c8dbe6c44c8324e4dbe3) | workflow commit (merge commit) |
| PR | [e44c5aec](https://github.com/huggingface/transformers/commit/e44c5aec51b81ef2707a5ece60e764b54dd323e5) | branch commit (from PR) |
| main | [c585eeaa](https://github.com/huggingface/transformers/commit/c585eeaa65d6a299cb03455ecd33e30981572c62) | base commit (on `main`) |

⚠️ No test being reported (jobs are skipped or cancelled)!


@ArthurZucker yes definitely there's more room for standardizing the patching process, especially since we now know all kind of patches necessary for full support. standardizing right now !

in terms of maintenance we really need these issues/patches to be fixed upstream, ngl all this patching shouldn't be on our side and we are only doing it to help adoption, demonstrate capability and not block the effort. @justinchuby for onnx @jackzhxng for executorch, would love to get your help on this matter.

Happy to help upstream the onnx patches. Let me know which ones are still needed? @IlyasMoutawwakil 

thanks @justinchuby the patches/fixes are all listed in the onnx_exporter.py file. numbered by stages (torch patches, fx patches, onnx ir patches).

@arthur thanks for the directions ! i created a single generic patching registry, works for torch and backend-specific (executorch) patches. and an fx node fix registry. kept the onnx ir fixes as a list because there's only one and it should be fixed soon in onnxscript/onnx ir soon.

@ArthurZucker successfully unified all patching ; torch patches, model patches and library-specific (eg executorch) patches under a single api !

also no more test timeouts, all tests are passing
```
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========= 1552 passed, 124 skipped, 3731 warnings in 850.49s (0:14:10) =========
```

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41992&sha=05d20b

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: autoformer, bigbird_pegasus, bridgetower, canine, clvp, colqwen2, deepseek_ocr2, diffusion_gemma, emu3, encodec, eomt, eomt_dinov3, esm, evolla, fastspeech2_conformer, funnel

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=41992)
**Latest run:** [28610022278:1](https://github.com/huggingface/transformers/actions/runs/28610022278)
**Result:** `success` | **Jobs:** 14 | **Tests:** 47,273 | **Failures:** 0 | **Duration:** 16h 58m

<!-- ci-dashboard-recap:end -->

Congrats!!