Export to ExecuTorch
Author: guangy10
Pull request: no
Labels: name: Feature request

name: ExecuTorch

## Feature request

Unlock a new workflow for on-device use-cases via [**torch.export**](https://pytorch.org/tutorials/intermediate/torch_export_tutorial.html) and [**ExecuTorch**](https://pytorch.org/executorch/main/intro-overview.html).

So ideally the users can have an e2e experience by loading a pretrained transformer model from HuggingFace, export and lower it to `ExecuTorch` and get reasonable performance out-of-the-box. 

For example:

1. Load a model with StaticCache:
```
model = AutoModelForCausalLM.from_pretrained(
    hf_model_repo,
    config=config,
    attn_implementation="sdpa",
    cache_config={
        "use_cache": True, 
        "cache_implementation": "static", 
        "max_cache_length": 128,
    },  # Mandatory field to set ONLY for "Export to ExecuTorch" workflow, optional in other use-cases
)
```

2. Then export the model with StaticCache. 
```
exported_program = convert_and_export_with_cache(
    model, 
    args=(model_inputs,), 
    kwargs={"position_ids": <val>, "inputs_embeds": <val>, "cache_position": <val>}
```
and then further lower the exported program to `ExecuTorch` with delegates for performance:
```
executorch_m = lower_to_executorch(
    model, 
    recipes="xnnpack_fp32",  # Delegate to XNNPACK backend
)

# The lowered artifact can be saved into a `.pte` binary format for integration and distribution.
```
With that you may get a model for on-device with reasonable performance to start with. 

From there and still within `ExecuTorch` stack, you can easily tailor the experience for your use-cases, of course, with better performance! Note that `ExecuTorch` supports delegatation to [XNNPACK backend](https://pytorch.org/executorch/main/native-delegates-executorch-xnnpack-delegate.html), [Apple Core ML](https://pytorch.org/executorch/main/build-run-coreml.html) and [MPS](https://github.com/pytorch/executorch/tree/main/examples/apple/mps), [Qualcomm QNN](https://pytorch.org/executorch/main/build-run-qualcomm-ai-engine-direct-backend.html), [ARM Ethos-U](https://pytorch.org/executorch/stable/executorch-arm-delegate-tutorial.html), [Vulkan GPU](https://pytorch.org/executorch/main/build-run-vulkan.html) and more. You can learn more by reading our [tutorial](https://pytorch.org/executorch/main/examples-end-to-end-to-lower-model-to-delegate.html).


3. Use the exported/lowered artifact for inference:
```

# The lowered artifact can run on a local device in the ExecuTorch runtime in c++ or via pybind, providing the same experience as how users run inference with the eager model on server.

generate(model=executorch_m, prompt="Hello world")  # Will generate up to the maximal sequence length/cache length 
```

The example workflow above shows direct integration between `ExecuTorch` and HF `transformers` models. Eventually this workflow could be accessible via `optimum exporters-et`, `Transformers.js` or in [`ExecuTorch`](https://github.com/pytorch/executorch) and [`torchchat`](https://github.com/pytorch/torchchat).

## Motivation

Unlock a whole new on-device experience of using HuggingFace models w/o leaving the PyTorch ecosystem ([`ExecuTorch`](https://pytorch.org/executorch/main/intro-overview.html) is native PyTorch!)


##  Issues Tracker

### Fundamental
- [x] Make `StaticCache` compatible with `torch.export`: PR #32168
- [x] #32500: PR #32830
- [x] #32503
- [ ] Support dynamic length slicing in `StaticCache`: PR #30862
- [x] #32504: PR #33707
- [ ] Convert Hugging Face tokenizer files to be the c++ `llm_runner` consumable: https://github.com/pytorch/executorch/issues/6813 

### E2E workflow
- [x] Umbrella task for `Optimum` enablement: https://github.com/huggingface/optimum/issues/2128
- [ ] Umbrella task for `Tranformers.js` enablement: https://github.com/huggingface/transformers.js/issues/1039

### Optimization
- [ ] Support quantized models w/ ExecuTorch + TorchAO #34787

### Models
- [x] #33709: PR #33707
- [x] #32505: PR #34101
- [ ] #32506
- [x] #32507: PR #34424
- [ ] #32508
- [ ] #32509
- [x] #33833: PR #34102
- [x] #33834: PR #36486
- [x] #33835: PR #34475
- [x] #33836: PR #34476
- [ ] #33837
- [x] #33838
- [ ] #33839
- [x] #33840: PR #34181
- [x] #33841: PR #34425
- [x] #33843: PR #34473
- [x] #34879
- [ ] #35327
- [x] #37727: PR https://github.com/huggingface/transformers/pull/37728

And more! We're ambitious to expanding the model coverage massively. Please comment below if you are interested in a particular model for on-device use-case!

Even better, we warmly welcome direct contributions from the community to support more models in exporting to ExecuTorch!
- [x] Cohere2: #35224
- [x] OLMo2: #34551
- [x] DPT, DepthAnything & ZoeDepth: #34103
- [x] #33842: PR https://github.com/huggingface/optimum-executorch/pull/45
- [x] https://github.com/huggingface/transformers/issues/37844: https://github.com/huggingface/transformers/pull/36878


## Your contribution

1. Co-design the "Export to ExecuTorch" workflow.
2. Co-design the `generate` for exported model and the integration in `Optimum`
3. Identify and fill gaps in DevX and UX

Here is how ExecuTorch implements the `generate()` for llama2/3 in [eager python](https://github.com/pytorch/executorch/blob/main/examples/models/llama2/runner/generation.py) and [c++](https://github.com/pytorch/executorch/blob/main/examples/models/llama2/runner/runner.cpp).


cc: @amyeroberts @gante @ArthurZucker @michaelbenayoun 

Comments:
Thank you for detailing Executorch's goals 🤗 

Two follow-up questions:
1. In the snippet you shared at the top, you explicitly load the model config before loading the model with `.from_pretrained`. However, `.from_pretrained` handles loading the config and modification of the base config -- for instance, `model = AutoModelForCausalLM.from_pretrained("distilgpt2", use_cache=False)` will change `use_cache` in `model.config` from the default `True` to `False`. Am I correct in saying that we don't need to manually load the config then?
2. We have been separating the parameterization of everything specifically related to auto-regressive generation to `generation_config` (i.e. it is not only for `generate`, just like `config` is not only for our model classes). As such, we want to place the cache config in `generation_config`, as KV caching only exists in auto-regressive generation. `generation_config` is also loaded in `.from_pretrained`, just as `config`. However, updating parameters through `.from_pretrained` is not yet supported (e.g. `.from_pretrained(model_repo, cache_config={...})` would't work). If the answer to 1. is yes: would this API [passing the cache config to `.from_pretrained`] be useful to you?

@gante Thanks for the great follow-up questions:

For #1, yes if we can pass/override the config while loading the pretrained model, e.g. `model = AutoModelForCausalLM.from_pretrained("distilgpt2", use_cache=True, cache_implementation="static", max_seq_lenght=128, attn_implementation="sdpa", ...)`, or even better to consolidate all cache related configs into one field `cache_config` (something you and Arthur are suggesting?)

For #2, yes I understand there are use-cases where make cache config closer to auto-regressive generation is cleaner. KV cache config can still be passed through `generation_config` in my proposal, for any use case, and no conflict. It's required to passed to `.from_pretrained(cache_config={"use_cache": True, "cache_implementation": "static", "max_cache_length": 128, ...})`like this only when "Export to ExecuTorch". It's because `ExecuTorch` is handling the memory planning ahead-of-time *(during export and lowering to `ExecuTorch`)* so the Runtime doesn't deal with the dynamic memory allocation, that's where the fast inference comes from. And yes, the API [passing the cache config to .from_pretrained] will be useful to `ExecuTorch` use-case!

Hey, saw your comments from another PR and wanted to share that I was thinking to make cache-config savable/loadable same way as generation config. It will hold all the needed args for all cache types, and loading a model `from_pretrained` should also load cache config and assign `self.cache_config=cache_config`. @gante WDYT about it?

I'm quite biased towards keeping the cache config inside `generation_config`:
1. A separate file to hold a handful of fields seems overkill
2. Caching exists because of generation, it is not a stand-alone feature (contrarily to e.g. quantization, where it is not part of any existing configuration file)

But happy to reconsider if there are strong arguments to keep them separate :)

Wait, i just realized that we will save the cache config even if it's inside generation config. So it will be loadable from hub. Oke, that makes sense, thanks!

> I'm quite biased towards keeping the cache config inside `generation_config`:
> 
> 1. A separate file to hold a handful of fields seems overkill
> 2. Caching exists because of generation, it is not a stand-alone feature (contrarily to e.g. quantization, where it is not part of any existing configuration file)
> 
> But happy to reconsider if there are strong arguments to keep them separate :)


@gante, I see there are two orthogonal things from your and @zucchini-nlp 's comments. Let's get more clarify on it:

1. Enable the ability to pass/override the cache config via `PreTrainedModel.from_pretrained()`

It would take the cache config to construct the model. This is a new feature needed in order to support `torch.export` and `ExecuTorch`. I think we're on the same page on this?

2. Decide on where to load/read the cache config from 

So this is about whether `PreTrainedModel.from_pretrained()` will load cache config from a separate config, generation config, or other config. I won't have strong option where it should go. To me, there may be a use case to quantize the cache at some point, and for `torch.export` and `ExecuTorch`, the quantization process is independent from generation.

> It would take the cache config to construct the model. This is a new feature needed in order to support torch.export and ExecuTorch. I think we're on the same page on this?

Yes :) 

> To me, there may be a use case to quantize the cache at some point

We have indeed support for quantized caches! Their quantization configuration is set at initialization time, so it will belong in the cache config as well :) (we can have, e.g. a FP16 model and a quantized cache, to support very long generation)

This could eventually enable also AOTI compilation:
https://pytorch.org/tutorials/recipes/torch_export_aoti_python.html

> This could eventually enable also AOTI compilation: https://pytorch.org/tutorials/recipes/torch_export_aoti_python.html

Yeah, we are having a proof-of-concept where making AOTI a backend of ExecuTorch enabling users can utilize both desktop GPU, HTP, and CPU altogether on a desktop where all these accelerators are available.

@guangy10 
I have been trying to get whisper implemented but wanting to check my understanding.  Would you expect there to be 2 artifacts? The encoder that runs independently, and a decoder that has an additional input containing the input embeddings?

Or am I missing something with how encoder/decoder architectures should be best implemented in executorch?

> @guangy10 I have been trying to get whisper implemented but wanting to check my understanding. Would you expect there to be 2 artifacts? The encoder that runs independently, and a decoder that has an additional input containing the input embeddings?
> 
> Or am I missing something with how encoder/decoder architectures should be best implemented in executorch?

Yeah, you can export the model to multiple artifacts. Here is an example of how another encoder-decoder model (t5) is  supported: https://github.com/huggingface/transformers/blob/d9e6f307e71b5108a7882ec00ffcc0d0eb316cb7/tests/models/t5/test_modeling_t5.py#L1650-L1706https://github.com/huggingface/transformers/blob/d9e6f307e71b5108a7882ec00ffcc0d0eb316cb7/tests/models/t5/test_modeling_t5.py#L1650-L1706. The example is using torch.compile, the idea would be same for using torch.export to ExecuTorch.

Hi @guangy10 !

I have a question and I think you are the best person I can ask for help 🙏 .

With the torch 2.7 [RC](https://dev-discuss.pytorch.org/t/pytorch-2-7-rc1-produced-for-pytorch-audio-vision/2855), some export tests are failing.

For example, 

> tests/models/dpt/test_modeling_dpt_auto_backbone.py::DPTModelTest::test_torch_export FAILED

is failing with 

```
2025-04-05T11:34:14.9962130Z E               from user code:
2025-04-05T11:34:14.9963313Z E                  File "/transformers/src/transformers/models/dpt/modeling_dpt.py", line 1186, in forward
2025-04-05T11:34:14.9964744Z E                   outputs = self.backbone.forward_with_filtered_kwargs(
2025-04-05T11:34:14.9966356Z E                 File "/transformers/src/transformers/utils/backbone_utils.py", line 233, in forward_with_filtered_kwargs
2025-04-05T11:34:14.9968028Z E                   signature = dict(inspect.signature(self.forward).parameters)
2025-04-05T11:34:14.9969748Z E                 File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/polyfills/__init__.py", line 168, in construct_dict
2025-04-05T11:34:14.9971331Z E                   if isinstance(src, (dict, MutableMapping)):
2025-04-05T11:34:14.9972119Z E               
2025-04-05T11:34:14.9973908Z E               Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
```
and I am able to avoid this failure by [this change](https://github.com/huggingface/transformers/commit/28c3bc8a79d4a78c733b0929b8f55ed172aa2815), and have another fix [here](https://github.com/huggingface/transformers/commit/49e7bd3e406e20beedec3c5d6d8be54aeb51daf5) to avoid similar issue at a different line.

It seems to me that, torch 2.7's dynamo doesn't like the access of `self.config.text_confg` (see [for example](https://github.com/huggingface/transformers/commit/95eb065772215cab276678c45daf933599cfd337) ), although `self.config` works well, and other kinds like [here](https://github.com/huggingface/transformers/commit/28c3bc8a79d4a78c733b0929b8f55ed172aa2815).

My fix is to avoid such failures by either define an object in `__init__` or use `self.language_model.confg` insetead of `self.config.text_config`.

I am wondering if you know a better approach to avoid such failures in general, or it's really necessary to require the changes I made and we have to stick to such rules when we add new models to `transformers`?

Thank you in advance.

> Hi [@guangy10](https://github.com/guangy10) !
> 
> I have a question and I think you are the best person I can ask for help 🙏 .
> 
> With the torch 2.7 [RC](https://dev-discuss.pytorch.org/t/pytorch-2-7-rc1-produced-for-pytorch-audio-vision/2855), some export tests are failing.
> 
> For example,
> 
> > tests/models/dpt/test_modeling_dpt_auto_backbone.py::DPTModelTest::test_torch_export FAILED
> 
> is failing with
> 
> ```
> 2025-04-05T11:34:14.9962130Z E               from user code:
> 2025-04-05T11:34:14.9963313Z E                  File "/transformers/src/transformers/models/dpt/modeling_dpt.py", line 1186, in forward
> 2025-04-05T11:34:14.9964744Z E                   outputs = self.backbone.forward_with_filtered_kwargs(
> 2025-04-05T11:34:14.9966356Z E                 File "/transformers/src/transformers/utils/backbone_utils.py", line 233, in forward_with_filtered_kwargs
> 2025-04-05T11:34:14.9968028Z E                   signature = dict(inspect.signature(self.forward).parameters)
> 2025-04-05T11:34:14.9969748Z E                 File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/polyfills/__init__.py", line 168, in construct_dict
> 2025-04-05T11:34:14.9971331Z E                   if isinstance(src, (dict, MutableMapping)):
> 2025-04-05T11:34:14.9972119Z E               
> 2025-04-05T11:34:14.9973908Z E               Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
> ```
> 
> and I am able to avoid this failure by [this change](https://github.com/huggingface/transformers/commit/28c3bc8a79d4a78c733b0929b8f55ed172aa2815), and have another fix [here](https://github.com/huggingface/transformers/commit/49e7bd3e406e20beedec3c5d6d8be54aeb51daf5) to avoid similar issue at a different line.
> 
> It seems to me that, torch 2.7's dynamo doesn't like the access of `self.config.text_confg` (see [for example](https://github.com/huggingface/transformers/commit/95eb065772215cab276678c45daf933599cfd337) ), although `self.config` works well, and other kinds like [here](https://github.com/huggingface/transformers/commit/28c3bc8a79d4a78c733b0929b8f55ed172aa2815).
> 
> My fix is to avoid such failures by either define an object in `__init__` or use `self.language_model.confg` insetead of `self.config.text_config`.
> 
> I am wondering if you know a better approach to avoid such failures in general, or it's really necessary to require the changes I made and we have to stick to such rules when we add new models to `transformers`?
> 
> Thank you in advance.

@ydshieh Oh the signature inspector. I think @tugsbayasgalan added that a few weeks ago. TBW he is the expert from `torch.export` team, should be able to help with the compatibility issue with the upcoming 2.7 release.

Thank you @guangy10 a lot. 

Hi @tugsbayasgalan would you be help in [this](https://github.com/huggingface/transformers/issues/32253#issuecomment-2782523327)? I am able to avoid the issues in some way, but don't know if they are the best approach.

It's not just about `inspect.signature`. For example,

> self.config.text_confg

gives similar issue while

> self.language_model.config

avoid the issue, or if I define `self.text_config = self.config.text_confg` and use `self.text_config` in `forward` also works.

This looks like regression from torchdynamo. cc @anijain2305. In the meanwhile, could you try using torch.export.export(strict=False)? This option is actually default export option starting 2.7 and has much better pass rate. 

Hi @tugsbayasgalan The failing tests indeed pass now with `strict=False` now.

@guangy10 I remember previously we decided to go with `strict=True` in order to detect the issue instead of just passing the test, right?

I wrote the step to reproduce. I also opened [an issue](https://github.com/pytorch/pytorch/issues/150994) in torch repository too.

## To Reproduce

### 1. Install torch 2.7 RC: 
> pip install torch==2.7.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/test/cu126

### 2. Install transformers

> git clone https://github.com/huggingface/transformers.git && cd transformers && git fetch origin && git checkout ci_with_torch_2.7_commit_0ef339ff1b63bb03a388c79bfbebec9085e10564 && pip install -e .[torch,testing]

### 3. Running test regarding torch.compile: get the error shown at the end, i.e. `7.` (works if running with torch 2.6)

> git checkout bcd1da9580ba1c6c4db019a91b1de9d88966e1fa && RUN_SLOW=1 python3 -m pytest -v tests/models/gemma3/test_modeling_gemma3.py::Gemma3Vision2TextModelTest::test_generate_compile_model_forward

### 4. Running test with a change in `transformers`: it works

> git checkout 95eb065772215cab276678c45daf933599cfd337&& RUN_SLOW=1 python3 -m pytest -v tests/models/gemma3/test_modeling_gemma3.py::Gemma3Vision2TextModelTest::test_generate_compile_model_forward

The commit changes is [here](https://github.com/huggingface/transformers/commit/95eb065772215cab276678c45daf933599cfd337)

### 5. another test with `torch.export` fails with torch 2.7 RC

> git checkout bcd1da9580ba1c6c4db019a91b1de9d88966e1fa && RUN_SLOW=1 python3 -m pytest -v tests/models/dpt/test_modeling_dpt_hybrid.py::DPTModelTest::test_torch_export

### 6. a change that would work

> git checkout 49e7bd3e406e20beedec3c5d6d8be54aeb51daf5&& RUN_SLOW=1 python3 -m pytest -v tests/models/dpt/test_modeling_dpt_hybrid.py::DPTModelTest::test_torch_export

The commit changes is [here](https://github.com/huggingface/transformers/commit/49e7bd3e406e20beedec3c5d6d8be54aeb51daf5)


### 7. Error log from `3.`

```bash
E               torch._dynamo.exc.Unsupported: Unexpected type in sourceless builder transformers.models.gemma3.configuration_gemma3.Gemma3TextConfig
E
E               from user code:
E                  File "/transformers/src/transformers/utils/generic.py", line 965, in wrapper
E                   output = func(self, *args, **kwargs)
E                 File "/transformers/src/transformers/utils/deprecation.py", line 172, in wrapped_func
E                   return func(*args, **kwargs)
E                 File "/transformers/src/transformers/models/gemma3/modeling_gemma3.py", line 1323, in forward
E                   causal_mask = self._update_causal_mask(
E                 File "/transformers/src/transformers/models/gemma3/modeling_gemma3.py", line 1117, in _update_causal_mask
E                   if self.config.text_config._attn_implementation == "flash_attention_2":
E                 File "/transformers/src/transformers/configuration_utils.py", line 210, in __getattribute__
E                   return super().__getattribute__(key)
E
E               Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
```