Cannot export sdxl encoder to onnx when transformers[torch] >= 4.43.0 (Occurred when translating scaled_dot_product_attention).
Author: puppetpyt
Pull request: no
Labels: name: bug

### System Info

python3.9
```
Requirement already satisfied: filelock in /usr/local/lib/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (3.13.4)
Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /usr/local/lib/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (0.24.6)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib64/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (1.26.4)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (24.0)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib64/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (6.0.1)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib64/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (2023.12.25)
Requirement already satisfied: requests in /usr/local/lib/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (2.31.0)
Requirement already satisfied: tokenizers<0.20,>=0.19 in /home/yuntaopan/.local/lib/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (0.19.1)
Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib64/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (0.4.2)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.9/site-packages (from transformers==4.43.0->transformers[torch]==4.43.0) (4.66.2)
Requirement already satisfied: torch in /usr/local/lib64/python3.9/site-packages (from transformers[torch]==4.43.0) (2.1.2)
Requirement already satisfied: accelerate>=0.21.0 in /usr/local/lib/python3.9/site-packages (from transformers[torch]==4.43.0) (0.29.2)
Requirement already satisfied: psutil in /usr/local/lib64/python3.9/site-packages (from accelerate>=0.21.0->transformers[torch]==4.43.0) (5.9.8)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.9/site-packages (from huggingface-hub<1.0,>=0.23.2->transformers==4.43.0->transformers[torch]==4.43.0) (2024.2.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.9/site-packages (from huggingface-hub<1.0,>=0.23.2->transformers==4.43.0->transformers[torch]==4.43.0) (4.11.0)
Requirement already satisfied: sympy in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (1.12)
Requirement already satisfied: networkx in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (3.2.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (3.1.3)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (12.1.105)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (12.1.105)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (12.1.105)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (8.9.2.26)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (12.1.3.1)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (11.0.2.54)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (10.3.2.106)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (11.4.5.107)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (12.1.0.106)
Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in /home/yuntaopan/.local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (2.18.1)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (12.1.105)
Requirement already satisfied: triton==2.1.0 in /home/yuntaopan/.local/lib/python3.9/site-packages (from torch->transformers[torch]==4.43.0) (2.1.0)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.9/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch->transformers[torch]==4.43.0) (12.4.127)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib64/python3.9/site-packages (from requests->transformers==4.43.0->transformers[torch]==4.43.0) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/site-packages (from requests->transformers==4.43.0->transformers[torch]==4.43.0) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.9/site-packages (from requests->transformers==4.43.0->transformers[torch]==4.43.0) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/site-packages (from requests->transformers==4.43.0->transformers[torch]==4.43.0) (2024.2.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib64/python3.9/site-packages (from jinja2->torch->transformers[torch]==4.43.0) (2.1.5)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.9/site-packages (from sympy->torch->transformers[torch]==4.43.0) (1.3.0)
```


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I use version 4.42.4 transformers,  I can still export.Do I need to make any changes to adapt to version 4.43?
export_onnx.py
```
from dataclasses import dataclass
from diffusers import DiffusionPipeline
from diffusers.utils import BaseOutput
from diffusers.utils.torch_utils import randn_tensor
import torch
import os, sys
from pathlib import Path

from optimum.exporters.tasks import TasksManager
import transformers
import onnx
import shutil
# load both base & refiner
base = DiffusionPipeline.from_pretrained("sdxl/huggingface_model/", torch_dtype=torch.float32, variant = 'float16', use_safetensors=True)
base.to("cpu")

class text_encoder(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = base.text_encoder

    def forward(self, input_ids):
        prompt_embeds = self.encoder(input_ids, output_hidden_states=True)
        return prompt_embeds[0], prompt_embeds.hidden_states[-2]


def export_text_encoder():
    model = text_encoder()
    input = torch.tensor([range(77)], dtype=torch.long)
    output = 'encoder.onnx'

    torch.onnx.export(model, (input),
                        str(output),
                        verbose=False,
                        input_names=['input_ids'],
                        output_names=['pooled_prompt_embed', 'prompt_embed'],
                        #dynamic_axes={"input": { 1: "seq_len" } },
                        do_constant_folding=True,
                        opset_version=15)
if __name__ == '__main__':

    export_text_encoder()

```

Traceback (most recent call last):
  File "/home/export_onnx.py", line 59, in <module>
    export_text_encoder()
  File "/home/export_onnx.py", line 49, in export_text_encoder
    torch.onnx.export(model, (input),
  File "/usr/local/lib64/python3.9/site-packages/torch/onnx/utils.py", line 516, in export
    _export(
  File "/usr/local/lib64/python3.9/site-packages/torch/onnx/utils.py", line 1596, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/usr/local/lib64/python3.9/site-packages/torch/onnx/utils.py", line 1139, in _model_to_graph
    graph = _optimize_graph(
  File "/usr/local/lib64/python3.9/site-packages/torch/onnx/utils.py", line 677, in _optimize_graph
    graph = _C._jit_pass_onnx(graph, operator_export_type)
  File "/usr/local/lib64/python3.9/site-packages/torch/onnx/utils.py", line 1940, in _run_symbolic_function
    return symbolic_fn(graph_context, *inputs, **attrs)
  File "/usr/local/lib64/python3.9/site-packages/torch/onnx/symbolic_helper.py", line 306, in wrapper
    return fn(g, *args, **kwargs)
  File "/usr/local/lib64/python3.9/site-packages/torch/onnx/symbolic_opset14.py", line 176, in scaled_dot_product_attention
    query_scaled = g.op("Mul", query, g.op("Sqrt", scale))
  File "/usr/local/lib64/python3.9/site-packages/torch/onnx/_internal/jit_utils.py", line 87, in op
    return _add_op(self, opname, *raw_args, outputs=outputs, **kwargs)
  File "/usr/local/lib64/python3.9/site-packages/torch/onnx/_internal/jit_utils.py", line 238, in _add_op
    inputs = [_const_if_tensor(graph_context, arg) for arg in args]
  File "/usr/local/lib64/python3.9/site-packages/torch/onnx/_internal/jit_utils.py", line 238, in <listcomp>
    inputs = [_const_if_tensor(graph_context, arg) for arg in args]
  File "/usr/local/lib64/python3.9/site-packages/torch/onnx/_internal/jit_utils.py", line 269, in _const_if_tensor
    return _add_op(graph_context, "onnx::Constant", value_z=arg)
  File "/usr/local/lib64/python3.9/site-packages/torch/onnx/_internal/jit_utils.py", line 246, in _add_op
    node = _create_node(
  File "/usr/local/lib64/python3.9/site-packages/torch/onnx/_internal/jit_utils.py", line 305, in _create_node
    _add_attribute(node, key, value, aten=aten)
  File "/usr/local/lib64/python3.9/site-packages/torch/onnx/_internal/jit_utils.py", line 356, in _add_attribute
    return getattr(node, f"{kind}_")(name, value)
TypeError: z_(): incompatible function arguments. The following argument types are supported:
    1. (self: torch._C.Node, arg0: str, arg1: torch.Tensor) -> torch._C.Node

Invoked with: %354 : Tensor = onnx::Constant(), scope: __main__.text_encoder::/transformers.models.clip.modeling_clip.CLIPTextModel::encoder/transformers.models.clip.modeling_clip.CLIPTextTransformer::text_model/transformers.models.clip.modeling_clip.CLIPEncoder::encoder/transformers.models.clip.modeling_clip.CLIPEncoderLayer::layers.0/transformers.models.clip.modeling_clip.CLIPSdpaAttention::self_attn
, 'value', 0.125
(Occurred when translating scaled_dot_product_attention).


### Expected behavior

Successfully exported onnx

Comments:
cc @IlyasMoutawwakil 

the change I had to do in `optimum` https://github.com/huggingface/optimum/pull/1971 is to use the eager attention since you can't foward `attn_implementation="eager"` in a diffusion pipeline to the clip model https://github.com/huggingface/diffusers/issues/8957. I still haven't investigated why sdpa can't be exported in this case.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @IlyasMoutawwakil ! Please could this be re-opened? It's absolutely still an issue - I have the exact same problem on transformers 4.46.3, and I've just had to downgrade to work around it. I can't believe how much effort it has taken to just export an ONNX of a CLIP text encoder for use with SD - the optimum library doesn't support it out of the box either (or maybe that's just broken too).

(P.S. huge thanks to original poster @puppetpyt for the workaround, I was stuck on this for a couple hours and tried upgrading transformers but not downgrading)

@zzbuzzard could you try with a more recent version (4.48 for example ) if this still fails feel free to open an issue on the optimum repo!

@zzbuzzard  when I export sd3.5 onnx，I also encountered the same problem. Could you please tell me how you solved it?

I downgraded to v4.42.4 (see my comment and the OP's)

This issue was already fixed in pytorch, it is also possible to export SD (or 2), SDXL, SD3 (or 3.5) to ONNX and run ONNXRuntime inference using [optimum-onnx](https://github.com/huggingface/optimum-onnx)

docs ref : https://huggingface.co/docs/optimum-onnx/en/onnxruntime/package_reference/modeling_diffusion#optimum.onnxruntime.ORTStableDiffusion3Pipeline



> I can't believe how much effort it has taken to just export an ONNX of a CLIP text encoder

sorry to hear, it was a torch problem not being able to jit trace the sdpa operator, the only workaround was to use eager attention as I mentioned above but the problem is fixed starting from torch 2.5, here's the original issue  http://github.com/pytorch/pytorch/issues/135615