[FX] `symbolic_trace` yields a TraceError for `BertModel`
Author: fxmarty
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.16.0.dev0
- Platform: Linux
- Python version: 3.9.2
- PyTorch version (GPU?): 1.10.1+cpu
- Tensorflow version (GPU?): /
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@michaelbenayoun 

Models:
- BERT: @LysandreJik

## Information

Model I am using : Bert

The problem arises when using:
* [ x ] my own modified scripts

The tasks I am working on is:
* Convert a `BertModel` to `torch.fx.GraphModule`

## To reproduce

Steps to reproduce the behavior:
```
from transformers import BertModel
import torch
from transformers.utils.fx import symbolic_trace
import transformers

# see https://github.com/huggingface/transformers/issues/14632
transformers.utils.fx.is_torch_fx_available = lambda: True

model = BertModel.from_pretrained("bert-base-uncased")

model = model.eval()

g = symbolic_trace(model,
                   sequence_length=20)
```

Traceback message:
```
Traceback (most recent call last):
  File "<tmp 1>", line 20, in <module>
    g = symbolic_trace(model,
  File "/home/felix/.local/lib/python3.9/site-packages/transformers-4.16.0.dev0-py3.9.egg/transformers/utils/fx.py", line 581, in symbolic_trace
    traced_graph = tracer.trace(model, concrete_args=concrete_args)
  File "/home/felix/.local/lib/python3.9/site-packages/transformers-4.16.0.dev0-py3.9.egg/transformers/utils/fx.py", line 372, in trace
    graph = super().trace(root, concrete_args=concrete_args)
  File "/home/felix/.local/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 615, in trace
    self.create_node('output', 'output', (self.create_arg(fn(*args)),), {},
  File "/home/felix/.local/lib/python3.9/site-packages/transformers-4.16.0.dev0-py3.9.egg/transformers/models/albert/modeling_albert.py", line 720, in forward
    encoder_outputs = self.encoder(
  File "/home/felix/.local/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 604, in module_call_wrapper
    return self.call_module(mod, forward, args, kwargs)
  File "/home/felix/.local/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 422, in call_module
    return forward(*args, **kwargs)
  File "/home/felix/.local/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py", line 600, in forward
    return _orig_module_call(mod, *args, **kwargs)
  File "/home/felix/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/felix/.local/lib/python3.9/site-packages/transformers-4.16.0.dev0-py3.9.egg/transformers/models/albert/modeling_albert.py", line 484, in forward
    return BaseModelOutput(
  File "<string>", line 6, in __init__
  File "/home/felix/.local/lib/python3.9/site-packages/transformers-4.16.0.dev0-py3.9.egg/transformers/file_utils.py", line 2296, in __post_init__
    iterator = iter(first_field)
  File "/home/felix/.local/lib/python3.9/site-packages/torch/fx/proxy.py", line 248, in __iter__
    return self.tracer.iter(self)
  File "/home/felix/.local/lib/python3.9/site-packages/torch/fx/proxy.py", line 161, in iter
    raise TraceError('Proxy object cannot be iterated. This can be '
torch.fx.proxy.TraceError: Proxy object cannot be iterated. This can be attempted when the Proxy is used in a loop or as a *args or **kwargs function argument. See the torch.fx docs on pytorch.org for a more detailed explanation of what types of control flow can be traced, and check out the Proxy docstring for help troubleshooting Proxy iteration errors
```

Tweaking the parameters here https://github.com/huggingface/transformers/blob/master/src/transformers/utils/fx.py#L497 does not seem to help.

Maybe this is related, but I am not sure, I am not familiar with fx: https://github.com/pytorch/pytorch/issues/44665

Maybe Bert and other models are not yet fx-friendly?

## Expected behavior

No error.


Comments:
Hi,

In order to symbolic trace a model, you also need to provide the input_names:

```
from transformers import BertModel
from transformers.utils.fx import symbolic_trace

model = BertModel.from_pretrained("bert-base-uncased")

traced_model = symbolic_trace(
    model,
    input_names=["input_ids", "attention_mask", "token_type_ids"],
    batch_size=1,
    sequence_length=128,
)
```
This works for me on Transformers v4.15.0 and PyTorch 1.9.0 in Google Colab.

Running `print(traced_model.graph)` prints the following for me:

```
graph():
    %input_ids : [#users=1] = placeholder[target=input_ids]
    %embeddings_token_type_ids : [#users=1] = get_attr[target=embeddings.token_type_ids]
    %getitem : [#users=1] = call_function[target=operator.getitem](args = (%embeddings_token_type_ids, (slice(None, None, None), slice(None, 20, None))), kwargs = {})
    %expand : [#users=1] = call_method[target=expand](args = (%getitem, 1, 20), kwargs = {})
    %embeddings_position_ids : [#users=1] = get_attr[target=embeddings.position_ids]
    %getitem_1 : [#users=1] = call_function[target=operator.getitem](args = (%embeddings_position_ids, (slice(None, None, None), slice(0, 20, None))), kwargs = {})
    %embeddings_word_embeddings : [#users=1] = call_module[target=embeddings.word_embeddings](args = (%input_ids,), kwargs = {})
    %embeddings_token_type_embeddings : [#users=1] = call_module[target=embeddings.token_type_embeddings](args = (%expand,), kwargs = {})
    %add : [#users=1] = call_function[target=operator.add](args = (%embeddings_word_embeddings, %embeddings_token_type_embeddings), kwargs = {})
    %embeddings_position_embeddings : [#users=1] = call_module[target=embeddings.position_embeddings](args = (%getitem_1,), kwargs = {})
    %add_1 : [#users=1] = call_function[target=operator.add](args = (%add, %embeddings_position_embeddings), kwargs = {})
    %embeddings_layer_norm : [#users=1] = call_module[target=embeddings.LayerNorm](args = (%add_1,), kwargs = {})
    %embeddings_dropout : [#users=4] = call_module[target=embeddings.dropout](args = (%embeddings_layer_norm,), kwargs = {})
    %encoder_layer_0_attention_self_query : [#users=1] = call_module[target=encoder.layer.0.attention.self.query](args = (%embeddings_dropout,), kwargs = {})
    %encoder_layer_0_attention_self_key : [#users=1] = call_module[target=encoder.layer.0.attention.self.key](args = (%embeddings_dropout,), kwargs = {})
    %view : [#users=1] = call_method[target=view](args = (%encoder_layer_0_attention_self_key, 1, 20, 12, 64), kwargs = {})
    %permute : [#users=1] = call_method[target=permute](args = (%view, 0, 2, 1, 3), kwargs = {})
    %encoder_layer_0_attention_self_value : [#users=1] = call_module[target=encoder.layer.0.attention.self.value](args = (%embeddings_dropout,), kwargs = {})
    %view_1 : [#users=1] = call_method[target=view](args = (%encoder_layer_0_attention_self_value, 1, 20, 12, 64), kwargs = {})
    %permute_1 : [#users=1] = call_method[target=permute](args = (%view_1, 0, 2, 1, 3), kwargs = {})
    %view_2 : [#users=1] = call_method[target=view](args = (%encoder_layer_0_attention_self_query, 1, 20, 12, 64), kwargs = {})
    %permute_2 : [#users=1] = call_method[target=permute](args = (%view_2, 0, 2, 1, 3), kwargs = {})
    %transpose : [#users=1] = call_method[target=transpose](args = (%permute, -1, -2), kwargs = {})
    %matmul : [#users=1] = call_function[target=torch.matmul](args = (%permute_2, %transpose), kwargs = {})
    %truediv : [#users=1] = call_function[target=operator.truediv](args = (%matmul, 8.0), kwargs = {})
    %_tensor_constant0 : [#users=1] = get_attr[target=_tensor_constant0]
    %add_2 : [#users=1] = call_function[target=operator.add](args = (%truediv, %_tensor_constant0), kwargs = {})
    %softmax : [#users=1] = call_function[target=torch.nn.functional.softmax](args = (%add_2,), kwargs = {dim: -1, _stacklevel: 3, dtype: None})
    %encoder_layer_0_attention_self_dropout : [#users=1] = call_module[target=encoder.layer.0.attention.self.dropout](args = (%softmax,), kwargs = {})
    %matmul_1 : [#users=1] = call_function[target=torch.matmul](args = (%encoder_layer_0_attention_self_dropout, %permute_1), kwargs = {})
    %permute_3 : [#users=1] = call_method[target=permute](args = (%matmul_1, 0, 2, 1, 3), kwargs = {})
    %contiguous : [#users=1] = call_method[target=contiguous](args = (%permute_3,), kwargs = {})
    %view_3 : [#users=1] = call_method[target=view](args = (%contiguous, 1, 20, 768), kwargs = {})
    %encoder_layer_0_attention_output_dense : [#users=1] = call_module[target=encoder.layer.0.attention.output.dense](args = (%view_3,), kwargs = {})
    %encoder_layer_0_attention_output_dropout : [#users=1] = call_module[target=encoder.layer.0.attention.output.dropout](args = (%encoder_layer_0_attention_output_dense,), kwargs = {})
    %add_3 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_0_attention_output_dropout, %embeddings_dropout), kwargs = {})
    %encoder_layer_0_attention_output_layer_norm : [#users=2] = call_module[target=encoder.layer.0.attention.output.LayerNorm](args = (%add_3,), kwargs = {})
    %encoder_layer_0_intermediate_dense : [#users=1] = call_module[target=encoder.layer.0.intermediate.dense](args = (%encoder_layer_0_attention_output_layer_norm,), kwargs = {})
    %gelu : [#users=1] = call_function[target=torch.nn.functional.gelu](args = (%encoder_layer_0_intermediate_dense,), kwargs = {})
    %encoder_layer_0_output_dense : [#users=1] = call_module[target=encoder.layer.0.output.dense](args = (%gelu,), kwargs = {})
    %encoder_layer_0_output_dropout : [#users=1] = call_module[target=encoder.layer.0.output.dropout](args = (%encoder_layer_0_output_dense,), kwargs = {})
    %add_4 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_0_output_dropout, %encoder_layer_0_attention_output_layer_norm), kwargs = {})
    %encoder_layer_0_output_layer_norm : [#users=4] = call_module[target=encoder.layer.0.output.LayerNorm](args = (%add_4,), kwargs = {})
    %encoder_layer_1_attention_self_query : [#users=1] = call_module[target=encoder.layer.1.attention.self.query](args = (%encoder_layer_0_output_layer_norm,), kwargs = {})
    %encoder_layer_1_attention_self_key : [#users=1] = call_module[target=encoder.layer.1.attention.self.key](args = (%encoder_layer_0_output_layer_norm,), kwargs = {})
    %view_4 : [#users=1] = call_method[target=view](args = (%encoder_layer_1_attention_self_key, 1, 20, 12, 64), kwargs = {})
    %permute_4 : [#users=1] = call_method[target=permute](args = (%view_4, 0, 2, 1, 3), kwargs = {})
    %encoder_layer_1_attention_self_value : [#users=1] = call_module[target=encoder.layer.1.attention.self.value](args = (%encoder_layer_0_output_layer_norm,), kwargs = {})
    %view_5 : [#users=1] = call_method[target=view](args = (%encoder_layer_1_attention_self_value, 1, 20, 12, 64), kwargs = {})
    %permute_5 : [#users=1] = call_method[target=permute](args = (%view_5, 0, 2, 1, 3), kwargs = {})
    %view_6 : [#users=1] = call_method[target=view](args = (%encoder_layer_1_attention_self_query, 1, 20, 12, 64), kwargs = {})
    %permute_6 : [#users=1] = call_method[target=permute](args = (%view_6, 0, 2, 1, 3), kwargs = {})
    %transpose_1 : [#users=1] = call_method[target=transpose](args = (%permute_4, -1, -2), kwargs = {})
    %matmul_2 : [#users=1] = call_function[target=torch.matmul](args = (%permute_6, %transpose_1), kwargs = {})
    %truediv_1 : [#users=1] = call_function[target=operator.truediv](args = (%matmul_2, 8.0), kwargs = {})
    %_tensor_constant1 : [#users=1] = get_attr[target=_tensor_constant1]
    %add_5 : [#users=1] = call_function[target=operator.add](args = (%truediv_1, %_tensor_constant1), kwargs = {})
    %softmax_1 : [#users=1] = call_function[target=torch.nn.functional.softmax](args = (%add_5,), kwargs = {dim: -1, _stacklevel: 3, dtype: None})
    %encoder_layer_1_attention_self_dropout : [#users=1] = call_module[target=encoder.layer.1.attention.self.dropout](args = (%softmax_1,), kwargs = {})
    %matmul_3 : [#users=1] = call_function[target=torch.matmul](args = (%encoder_layer_1_attention_self_dropout, %permute_5), kwargs = {})
    %permute_7 : [#users=1] = call_method[target=permute](args = (%matmul_3, 0, 2, 1, 3), kwargs = {})
    %contiguous_1 : [#users=1] = call_method[target=contiguous](args = (%permute_7,), kwargs = {})
    %view_7 : [#users=1] = call_method[target=view](args = (%contiguous_1, 1, 20, 768), kwargs = {})
    %encoder_layer_1_attention_output_dense : [#users=1] = call_module[target=encoder.layer.1.attention.output.dense](args = (%view_7,), kwargs = {})
    %encoder_layer_1_attention_output_dropout : [#users=1] = call_module[target=encoder.layer.1.attention.output.dropout](args = (%encoder_layer_1_attention_output_dense,), kwargs = {})
    %add_6 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_1_attention_output_dropout, %encoder_layer_0_output_layer_norm), kwargs = {})
    %encoder_layer_1_attention_output_layer_norm : [#users=2] = call_module[target=encoder.layer.1.attention.output.LayerNorm](args = (%add_6,), kwargs = {})
    %encoder_layer_1_intermediate_dense : [#users=1] = call_module[target=encoder.layer.1.intermediate.dense](args = (%encoder_layer_1_attention_output_layer_norm,), kwargs = {})
    %gelu_1 : [#users=1] = call_function[target=torch.nn.functional.gelu](args = (%encoder_layer_1_intermediate_dense,), kwargs = {})
    %encoder_layer_1_output_dense : [#users=1] = call_module[target=encoder.layer.1.output.dense](args = (%gelu_1,), kwargs = {})
    %encoder_layer_1_output_dropout : [#users=1] = call_module[target=encoder.layer.1.output.dropout](args = (%encoder_layer_1_output_dense,), kwargs = {})
    %add_7 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_1_output_dropout, %encoder_layer_1_attention_output_layer_norm), kwargs = {})
    %encoder_layer_1_output_layer_norm : [#users=4] = call_module[target=encoder.layer.1.output.LayerNorm](args = (%add_7,), kwargs = {})
    %encoder_layer_2_attention_self_query : [#users=1] = call_module[target=encoder.layer.2.attention.self.query](args = (%encoder_layer_1_output_layer_norm,), kwargs = {})
    %encoder_layer_2_attention_self_key : [#users=1] = call_module[target=encoder.layer.2.attention.self.key](args = (%encoder_layer_1_output_layer_norm,), kwargs = {})
    %view_8 : [#users=1] = call_method[target=view](args = (%encoder_layer_2_attention_self_key, 1, 20, 12, 64), kwargs = {})
    %permute_8 : [#users=1] = call_method[target=permute](args = (%view_8, 0, 2, 1, 3), kwargs = {})
    %encoder_layer_2_attention_self_value : [#users=1] = call_module[target=encoder.layer.2.attention.self.value](args = (%encoder_layer_1_output_layer_norm,), kwargs = {})
    %view_9 : [#users=1] = call_method[target=view](args = (%encoder_layer_2_attention_self_value, 1, 20, 12, 64), kwargs = {})
    %permute_9 : [#users=1] = call_method[target=permute](args = (%view_9, 0, 2, 1, 3), kwargs = {})
    %view_10 : [#users=1] = call_method[target=view](args = (%encoder_layer_2_attention_self_query, 1, 20, 12, 64), kwargs = {})
    %permute_10 : [#users=1] = call_method[target=permute](args = (%view_10, 0, 2, 1, 3), kwargs = {})
    %transpose_2 : [#users=1] = call_method[target=transpose](args = (%permute_8, -1, -2), kwargs = {})
    %matmul_4 : [#users=1] = call_function[target=torch.matmul](args = (%permute_10, %transpose_2), kwargs = {})
    %truediv_2 : [#users=1] = call_function[target=operator.truediv](args = (%matmul_4, 8.0), kwargs = {})
    %_tensor_constant2 : [#users=1] = get_attr[target=_tensor_constant2]
    %add_8 : [#users=1] = call_function[target=operator.add](args = (%truediv_2, %_tensor_constant2), kwargs = {})
    %softmax_2 : [#users=1] = call_function[target=torch.nn.functional.softmax](args = (%add_8,), kwargs = {dim: -1, _stacklevel: 3, dtype: None})
    %encoder_layer_2_attention_self_dropout : [#users=1] = call_module[target=encoder.layer.2.attention.self.dropout](args = (%softmax_2,), kwargs = {})
    %matmul_5 : [#users=1] = call_function[target=torch.matmul](args = (%encoder_layer_2_attention_self_dropout, %permute_9), kwargs = {})
    %permute_11 : [#users=1] = call_method[target=permute](args = (%matmul_5, 0, 2, 1, 3), kwargs = {})
    %contiguous_2 : [#users=1] = call_method[target=contiguous](args = (%permute_11,), kwargs = {})
    %view_11 : [#users=1] = call_method[target=view](args = (%contiguous_2, 1, 20, 768), kwargs = {})
    %encoder_layer_2_attention_output_dense : [#users=1] = call_module[target=encoder.layer.2.attention.output.dense](args = (%view_11,), kwargs = {})
    %encoder_layer_2_attention_output_dropout : [#users=1] = call_module[target=encoder.layer.2.attention.output.dropout](args = (%encoder_layer_2_attention_output_dense,), kwargs = {})
    %add_9 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_2_attention_output_dropout, %encoder_layer_1_output_layer_norm), kwargs = {})
    %encoder_layer_2_attention_output_layer_norm : [#users=2] = call_module[target=encoder.layer.2.attention.output.LayerNorm](args = (%add_9,), kwargs = {})
    %encoder_layer_2_intermediate_dense : [#users=1] = call_module[target=encoder.layer.2.intermediate.dense](args = (%encoder_layer_2_attention_output_layer_norm,), kwargs = {})
    %gelu_2 : [#users=1] = call_function[target=torch.nn.functional.gelu](args = (%encoder_layer_2_intermediate_dense,), kwargs = {})
    %encoder_layer_2_output_dense : [#users=1] = call_module[target=encoder.layer.2.output.dense](args = (%gelu_2,), kwargs = {})
    %encoder_layer_2_output_dropout : [#users=1] = call_module[target=encoder.layer.2.output.dropout](args = (%encoder_layer_2_output_dense,), kwargs = {})
    %add_10 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_2_output_dropout, %encoder_layer_2_attention_output_layer_norm), kwargs = {})
    %encoder_layer_2_output_layer_norm : [#users=4] = call_module[target=encoder.layer.2.output.LayerNorm](args = (%add_10,), kwargs = {})
    %encoder_layer_3_attention_self_query : [#users=1] = call_module[target=encoder.layer.3.attention.self.query](args = (%encoder_layer_2_output_layer_norm,), kwargs = {})
    %encoder_layer_3_attention_self_key : [#users=1] = call_module[target=encoder.layer.3.attention.self.key](args = (%encoder_layer_2_output_layer_norm,), kwargs = {})
    %view_12 : [#users=1] = call_method[target=view](args = (%encoder_layer_3_attention_self_key, 1, 20, 12, 64), kwargs = {})
    %permute_12 : [#users=1] = call_method[target=permute](args = (%view_12, 0, 2, 1, 3), kwargs = {})
    %encoder_layer_3_attention_self_value : [#users=1] = call_module[target=encoder.layer.3.attention.self.value](args = (%encoder_layer_2_output_layer_norm,), kwargs = {})
    %view_13 : [#users=1] = call_method[target=view](args = (%encoder_layer_3_attention_self_value, 1, 20, 12, 64), kwargs = {})
    %permute_13 : [#users=1] = call_method[target=permute](args = (%view_13, 0, 2, 1, 3), kwargs = {})
    %view_14 : [#users=1] = call_method[target=view](args = (%encoder_layer_3_attention_self_query, 1, 20, 12, 64), kwargs = {})
    %permute_14 : [#users=1] = call_method[target=permute](args = (%view_14, 0, 2, 1, 3), kwargs = {})
    %transpose_3 : [#users=1] = call_method[target=transpose](args = (%permute_12, -1, -2), kwargs = {})
    %matmul_6 : [#users=1] = call_function[target=torch.matmul](args = (%permute_14, %transpose_3), kwargs = {})
    %truediv_3 : [#users=1] = call_function[target=operator.truediv](args = (%matmul_6, 8.0), kwargs = {})
    %_tensor_constant3 : [#users=1] = get_attr[target=_tensor_constant3]
    %add_11 : [#users=1] = call_function[target=operator.add](args = (%truediv_3, %_tensor_constant3), kwargs = {})
    %softmax_3 : [#users=1] = call_function[target=torch.nn.functional.softmax](args = (%add_11,), kwargs = {dim: -1, _stacklevel: 3, dtype: None})
    %encoder_layer_3_attention_self_dropout : [#users=1] = call_module[target=encoder.layer.3.attention.self.dropout](args = (%softmax_3,), kwargs = {})
    %matmul_7 : [#users=1] = call_function[target=torch.matmul](args = (%encoder_layer_3_attention_self_dropout, %permute_13), kwargs = {})
    %permute_15 : [#users=1] = call_method[target=permute](args = (%matmul_7, 0, 2, 1, 3), kwargs = {})
    %contiguous_3 : [#users=1] = call_method[target=contiguous](args = (%permute_15,), kwargs = {})
    %view_15 : [#users=1] = call_method[target=view](args = (%contiguous_3, 1, 20, 768), kwargs = {})
    %encoder_layer_3_attention_output_dense : [#users=1] = call_module[target=encoder.layer.3.attention.output.dense](args = (%view_15,), kwargs = {})
    %encoder_layer_3_attention_output_dropout : [#users=1] = call_module[target=encoder.layer.3.attention.output.dropout](args = (%encoder_layer_3_attention_output_dense,), kwargs = {})
    %add_12 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_3_attention_output_dropout, %encoder_layer_2_output_layer_norm), kwargs = {})
    %encoder_layer_3_attention_output_layer_norm : [#users=2] = call_module[target=encoder.layer.3.attention.output.LayerNorm](args = (%add_12,), kwargs = {})
    %encoder_layer_3_intermediate_dense : [#users=1] = call_module[target=encoder.layer.3.intermediate.dense](args = (%encoder_layer_3_attention_output_layer_norm,), kwargs = {})
    %gelu_3 : [#users=1] = call_function[target=torch.nn.functional.gelu](args = (%encoder_layer_3_intermediate_dense,), kwargs = {})
    %encoder_layer_3_output_dense : [#users=1] = call_module[target=encoder.layer.3.output.dense](args = (%gelu_3,), kwargs = {})
    %encoder_layer_3_output_dropout : [#users=1] = call_module[target=encoder.layer.3.output.dropout](args = (%encoder_layer_3_output_dense,), kwargs = {})
    %add_13 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_3_output_dropout, %encoder_layer_3_attention_output_layer_norm), kwargs = {})
    %encoder_layer_3_output_layer_norm : [#users=4] = call_module[target=encoder.layer.3.output.LayerNorm](args = (%add_13,), kwargs = {})
    %encoder_layer_4_attention_self_query : [#users=1] = call_module[target=encoder.layer.4.attention.self.query](args = (%encoder_layer_3_output_layer_norm,), kwargs = {})
    %encoder_layer_4_attention_self_key : [#users=1] = call_module[target=encoder.layer.4.attention.self.key](args = (%encoder_layer_3_output_layer_norm,), kwargs = {})
    %view_16 : [#users=1] = call_method[target=view](args = (%encoder_layer_4_attention_self_key, 1, 20, 12, 64), kwargs = {})
    %permute_16 : [#users=1] = call_method[target=permute](args = (%view_16, 0, 2, 1, 3), kwargs = {})
    %encoder_layer_4_attention_self_value : [#users=1] = call_module[target=encoder.layer.4.attention.self.value](args = (%encoder_layer_3_output_layer_norm,), kwargs = {})
    %view_17 : [#users=1] = call_method[target=view](args = (%encoder_layer_4_attention_self_value, 1, 20, 12, 64), kwargs = {})
    %permute_17 : [#users=1] = call_method[target=permute](args = (%view_17, 0, 2, 1, 3), kwargs = {})
    %view_18 : [#users=1] = call_method[target=view](args = (%encoder_layer_4_attention_self_query, 1, 20, 12, 64), kwargs = {})
    %permute_18 : [#users=1] = call_method[target=permute](args = (%view_18, 0, 2, 1, 3), kwargs = {})
    %transpose_4 : [#users=1] = call_method[target=transpose](args = (%permute_16, -1, -2), kwargs = {})
    %matmul_8 : [#users=1] = call_function[target=torch.matmul](args = (%permute_18, %transpose_4), kwargs = {})
    %truediv_4 : [#users=1] = call_function[target=operator.truediv](args = (%matmul_8, 8.0), kwargs = {})
    %_tensor_constant4 : [#users=1] = get_attr[target=_tensor_constant4]
    %add_14 : [#users=1] = call_function[target=operator.add](args = (%truediv_4, %_tensor_constant4), kwargs = {})
    %softmax_4 : [#users=1] = call_function[target=torch.nn.functional.softmax](args = (%add_14,), kwargs = {dim: -1, _stacklevel: 3, dtype: None})
    %encoder_layer_4_attention_self_dropout : [#users=1] = call_module[target=encoder.layer.4.attention.self.dropout](args = (%softmax_4,), kwargs = {})
    %matmul_9 : [#users=1] = call_function[target=torch.matmul](args = (%encoder_layer_4_attention_self_dropout, %permute_17), kwargs = {})
    %permute_19 : [#users=1] = call_method[target=permute](args = (%matmul_9, 0, 2, 1, 3), kwargs = {})
    %contiguous_4 : [#users=1] = call_method[target=contiguous](args = (%permute_19,), kwargs = {})
    %view_19 : [#users=1] = call_method[target=view](args = (%contiguous_4, 1, 20, 768), kwargs = {})
    %encoder_layer_4_attention_output_dense : [#users=1] = call_module[target=encoder.layer.4.attention.output.dense](args = (%view_19,), kwargs = {})
    %encoder_layer_4_attention_output_dropout : [#users=1] = call_module[target=encoder.layer.4.attention.output.dropout](args = (%encoder_layer_4_attention_output_dense,), kwargs = {})
    %add_15 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_4_attention_output_dropout, %encoder_layer_3_output_layer_norm), kwargs = {})
    %encoder_layer_4_attention_output_layer_norm : [#users=2] = call_module[target=encoder.layer.4.attention.output.LayerNorm](args = (%add_15,), kwargs = {})
    %encoder_layer_4_intermediate_dense : [#users=1] = call_module[target=encoder.layer.4.intermediate.dense](args = (%encoder_layer_4_attention_output_layer_norm,), kwargs = {})
    %gelu_4 : [#users=1] = call_function[target=torch.nn.functional.gelu](args = (%encoder_layer_4_intermediate_dense,), kwargs = {})
    %encoder_layer_4_output_dense : [#users=1] = call_module[target=encoder.layer.4.output.dense](args = (%gelu_4,), kwargs = {})
    %encoder_layer_4_output_dropout : [#users=1] = call_module[target=encoder.layer.4.output.dropout](args = (%encoder_layer_4_output_dense,), kwargs = {})
    %add_16 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_4_output_dropout, %encoder_layer_4_attention_output_layer_norm), kwargs = {})
    %encoder_layer_4_output_layer_norm : [#users=4] = call_module[target=encoder.layer.4.output.LayerNorm](args = (%add_16,), kwargs = {})
    %encoder_layer_5_attention_self_query : [#users=1] = call_module[target=encoder.layer.5.attention.self.query](args = (%encoder_layer_4_output_layer_norm,), kwargs = {})
    %encoder_layer_5_attention_self_key : [#users=1] = call_module[target=encoder.layer.5.attention.self.key](args = (%encoder_layer_4_output_layer_norm,), kwargs = {})
    %view_20 : [#users=1] = call_method[target=view](args = (%encoder_layer_5_attention_self_key, 1, 20, 12, 64), kwargs = {})
    %permute_20 : [#users=1] = call_method[target=permute](args = (%view_20, 0, 2, 1, 3), kwargs = {})
    %encoder_layer_5_attention_self_value : [#users=1] = call_module[target=encoder.layer.5.attention.self.value](args = (%encoder_layer_4_output_layer_norm,), kwargs = {})
    %view_21 : [#users=1] = call_method[target=view](args = (%encoder_layer_5_attention_self_value, 1, 20, 12, 64), kwargs = {})
    %permute_21 : [#users=1] = call_method[target=permute](args = (%view_21, 0, 2, 1, 3), kwargs = {})
    %view_22 : [#users=1] = call_method[target=view](args = (%encoder_layer_5_attention_self_query, 1, 20, 12, 64), kwargs = {})
    %permute_22 : [#users=1] = call_method[target=permute](args = (%view_22, 0, 2, 1, 3), kwargs = {})
    %transpose_5 : [#users=1] = call_method[target=transpose](args = (%permute_20, -1, -2), kwargs = {})
    %matmul_10 : [#users=1] = call_function[target=torch.matmul](args = (%permute_22, %transpose_5), kwargs = {})
    %truediv_5 : [#users=1] = call_function[target=operator.truediv](args = (%matmul_10, 8.0), kwargs = {})
    %_tensor_constant5 : [#users=1] = get_attr[target=_tensor_constant5]
    %add_17 : [#users=1] = call_function[target=operator.add](args = (%truediv_5, %_tensor_constant5), kwargs = {})
    %softmax_5 : [#users=1] = call_function[target=torch.nn.functional.softmax](args = (%add_17,), kwargs = {dim: -1, _stacklevel: 3, dtype: None})
    %encoder_layer_5_attention_self_dropout : [#users=1] = call_module[target=encoder.layer.5.attention.self.dropout](args = (%softmax_5,), kwargs = {})
    %matmul_11 : [#users=1] = call_function[target=torch.matmul](args = (%encoder_layer_5_attention_self_dropout, %permute_21), kwargs = {})
    %permute_23 : [#users=1] = call_method[target=permute](args = (%matmul_11, 0, 2, 1, 3), kwargs = {})
    %contiguous_5 : [#users=1] = call_method[target=contiguous](args = (%permute_23,), kwargs = {})
    %view_23 : [#users=1] = call_method[target=view](args = (%contiguous_5, 1, 20, 768), kwargs = {})
    %encoder_layer_5_attention_output_dense : [#users=1] = call_module[target=encoder.layer.5.attention.output.dense](args = (%view_23,), kwargs = {})
    %encoder_layer_5_attention_output_dropout : [#users=1] = call_module[target=encoder.layer.5.attention.output.dropout](args = (%encoder_layer_5_attention_output_dense,), kwargs = {})
    %add_18 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_5_attention_output_dropout, %encoder_layer_4_output_layer_norm), kwargs = {})
    %encoder_layer_5_attention_output_layer_norm : [#users=2] = call_module[target=encoder.layer.5.attention.output.LayerNorm](args = (%add_18,), kwargs = {})
    %encoder_layer_5_intermediate_dense : [#users=1] = call_module[target=encoder.layer.5.intermediate.dense](args = (%encoder_layer_5_attention_output_layer_norm,), kwargs = {})
    %gelu_5 : [#users=1] = call_function[target=torch.nn.functional.gelu](args = (%encoder_layer_5_intermediate_dense,), kwargs = {})
    %encoder_layer_5_output_dense : [#users=1] = call_module[target=encoder.layer.5.output.dense](args = (%gelu_5,), kwargs = {})
    %encoder_layer_5_output_dropout : [#users=1] = call_module[target=encoder.layer.5.output.dropout](args = (%encoder_layer_5_output_dense,), kwargs = {})
    %add_19 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_5_output_dropout, %encoder_layer_5_attention_output_layer_norm), kwargs = {})
    %encoder_layer_5_output_layer_norm : [#users=4] = call_module[target=encoder.layer.5.output.LayerNorm](args = (%add_19,), kwargs = {})
    %encoder_layer_6_attention_self_query : [#users=1] = call_module[target=encoder.layer.6.attention.self.query](args = (%encoder_layer_5_output_layer_norm,), kwargs = {})
    %encoder_layer_6_attention_self_key : [#users=1] = call_module[target=encoder.layer.6.attention.self.key](args = (%encoder_layer_5_output_layer_norm,), kwargs = {})
    %view_24 : [#users=1] = call_method[target=view](args = (%encoder_layer_6_attention_self_key, 1, 20, 12, 64), kwargs = {})
    %permute_24 : [#users=1] = call_method[target=permute](args = (%view_24, 0, 2, 1, 3), kwargs = {})
    %encoder_layer_6_attention_self_value : [#users=1] = call_module[target=encoder.layer.6.attention.self.value](args = (%encoder_layer_5_output_layer_norm,), kwargs = {})
    %view_25 : [#users=1] = call_method[target=view](args = (%encoder_layer_6_attention_self_value, 1, 20, 12, 64), kwargs = {})
    %permute_25 : [#users=1] = call_method[target=permute](args = (%view_25, 0, 2, 1, 3), kwargs = {})
    %view_26 : [#users=1] = call_method[target=view](args = (%encoder_layer_6_attention_self_query, 1, 20, 12, 64), kwargs = {})
    %permute_26 : [#users=1] = call_method[target=permute](args = (%view_26, 0, 2, 1, 3), kwargs = {})
    %transpose_6 : [#users=1] = call_method[target=transpose](args = (%permute_24, -1, -2), kwargs = {})
    %matmul_12 : [#users=1] = call_function[target=torch.matmul](args = (%permute_26, %transpose_6), kwargs = {})
    %truediv_6 : [#users=1] = call_function[target=operator.truediv](args = (%matmul_12, 8.0), kwargs = {})
    %_tensor_constant6 : [#users=1] = get_attr[target=_tensor_constant6]
    %add_20 : [#users=1] = call_function[target=operator.add](args = (%truediv_6, %_tensor_constant6), kwargs = {})
    %softmax_6 : [#users=1] = call_function[target=torch.nn.functional.softmax](args = (%add_20,), kwargs = {dim: -1, _stacklevel: 3, dtype: None})
    %encoder_layer_6_attention_self_dropout : [#users=1] = call_module[target=encoder.layer.6.attention.self.dropout](args = (%softmax_6,), kwargs = {})
    %matmul_13 : [#users=1] = call_function[target=torch.matmul](args = (%encoder_layer_6_attention_self_dropout, %permute_25), kwargs = {})
    %permute_27 : [#users=1] = call_method[target=permute](args = (%matmul_13, 0, 2, 1, 3), kwargs = {})
    %contiguous_6 : [#users=1] = call_method[target=contiguous](args = (%permute_27,), kwargs = {})
    %view_27 : [#users=1] = call_method[target=view](args = (%contiguous_6, 1, 20, 768), kwargs = {})
    %encoder_layer_6_attention_output_dense : [#users=1] = call_module[target=encoder.layer.6.attention.output.dense](args = (%view_27,), kwargs = {})
    %encoder_layer_6_attention_output_dropout : [#users=1] = call_module[target=encoder.layer.6.attention.output.dropout](args = (%encoder_layer_6_attention_output_dense,), kwargs = {})
    %add_21 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_6_attention_output_dropout, %encoder_layer_5_output_layer_norm), kwargs = {})
    %encoder_layer_6_attention_output_layer_norm : [#users=2] = call_module[target=encoder.layer.6.attention.output.LayerNorm](args = (%add_21,), kwargs = {})
    %encoder_layer_6_intermediate_dense : [#users=1] = call_module[target=encoder.layer.6.intermediate.dense](args = (%encoder_layer_6_attention_output_layer_norm,), kwargs = {})
    %gelu_6 : [#users=1] = call_function[target=torch.nn.functional.gelu](args = (%encoder_layer_6_intermediate_dense,), kwargs = {})
    %encoder_layer_6_output_dense : [#users=1] = call_module[target=encoder.layer.6.output.dense](args = (%gelu_6,), kwargs = {})
    %encoder_layer_6_output_dropout : [#users=1] = call_module[target=encoder.layer.6.output.dropout](args = (%encoder_layer_6_output_dense,), kwargs = {})
    %add_22 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_6_output_dropout, %encoder_layer_6_attention_output_layer_norm), kwargs = {})
    %encoder_layer_6_output_layer_norm : [#users=4] = call_module[target=encoder.layer.6.output.LayerNorm](args = (%add_22,), kwargs = {})
    %encoder_layer_7_attention_self_query : [#users=1] = call_module[target=encoder.layer.7.attention.self.query](args = (%encoder_layer_6_output_layer_norm,), kwargs = {})
    %encoder_layer_7_attention_self_key : [#users=1] = call_module[target=encoder.layer.7.attention.self.key](args = (%encoder_layer_6_output_layer_norm,), kwargs = {})
    %view_28 : [#users=1] = call_method[target=view](args = (%encoder_layer_7_attention_self_key, 1, 20, 12, 64), kwargs = {})
    %permute_28 : [#users=1] = call_method[target=permute](args = (%view_28, 0, 2, 1, 3), kwargs = {})
    %encoder_layer_7_attention_self_value : [#users=1] = call_module[target=encoder.layer.7.attention.self.value](args = (%encoder_layer_6_output_layer_norm,), kwargs = {})
    %view_29 : [#users=1] = call_method[target=view](args = (%encoder_layer_7_attention_self_value, 1, 20, 12, 64), kwargs = {})
    %permute_29 : [#users=1] = call_method[target=permute](args = (%view_29, 0, 2, 1, 3), kwargs = {})
    %view_30 : [#users=1] = call_method[target=view](args = (%encoder_layer_7_attention_self_query, 1, 20, 12, 64), kwargs = {})
    %permute_30 : [#users=1] = call_method[target=permute](args = (%view_30, 0, 2, 1, 3), kwargs = {})
    %transpose_7 : [#users=1] = call_method[target=transpose](args = (%permute_28, -1, -2), kwargs = {})
    %matmul_14 : [#users=1] = call_function[target=torch.matmul](args = (%permute_30, %transpose_7), kwargs = {})
    %truediv_7 : [#users=1] = call_function[target=operator.truediv](args = (%matmul_14, 8.0), kwargs = {})
    %_tensor_constant7 : [#users=1] = get_attr[target=_tensor_constant7]
    %add_23 : [#users=1] = call_function[target=operator.add](args = (%truediv_7, %_tensor_constant7), kwargs = {})
    %softmax_7 : [#users=1] = call_function[target=torch.nn.functional.softmax](args = (%add_23,), kwargs = {dim: -1, _stacklevel: 3, dtype: None})
    %encoder_layer_7_attention_self_dropout : [#users=1] = call_module[target=encoder.layer.7.attention.self.dropout](args = (%softmax_7,), kwargs = {})
    %matmul_15 : [#users=1] = call_function[target=torch.matmul](args = (%encoder_layer_7_attention_self_dropout, %permute_29), kwargs = {})
    %permute_31 : [#users=1] = call_method[target=permute](args = (%matmul_15, 0, 2, 1, 3), kwargs = {})
    %contiguous_7 : [#users=1] = call_method[target=contiguous](args = (%permute_31,), kwargs = {})
    %view_31 : [#users=1] = call_method[target=view](args = (%contiguous_7, 1, 20, 768), kwargs = {})
    %encoder_layer_7_attention_output_dense : [#users=1] = call_module[target=encoder.layer.7.attention.output.dense](args = (%view_31,), kwargs = {})
    %encoder_layer_7_attention_output_dropout : [#users=1] = call_module[target=encoder.layer.7.attention.output.dropout](args = (%encoder_layer_7_attention_output_dense,), kwargs = {})
    %add_24 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_7_attention_output_dropout, %encoder_layer_6_output_layer_norm), kwargs = {})
    %encoder_layer_7_attention_output_layer_norm : [#users=2] = call_module[target=encoder.layer.7.attention.output.LayerNorm](args = (%add_24,), kwargs = {})
    %encoder_layer_7_intermediate_dense : [#users=1] = call_module[target=encoder.layer.7.intermediate.dense](args = (%encoder_layer_7_attention_output_layer_norm,), kwargs = {})
    %gelu_7 : [#users=1] = call_function[target=torch.nn.functional.gelu](args = (%encoder_layer_7_intermediate_dense,), kwargs = {})
    %encoder_layer_7_output_dense : [#users=1] = call_module[target=encoder.layer.7.output.dense](args = (%gelu_7,), kwargs = {})
    %encoder_layer_7_output_dropout : [#users=1] = call_module[target=encoder.layer.7.output.dropout](args = (%encoder_layer_7_output_dense,), kwargs = {})
    %add_25 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_7_output_dropout, %encoder_layer_7_attention_output_layer_norm), kwargs = {})
    %encoder_layer_7_output_layer_norm : [#users=4] = call_module[target=encoder.layer.7.output.LayerNorm](args = (%add_25,), kwargs = {})
    %encoder_layer_8_attention_self_query : [#users=1] = call_module[target=encoder.layer.8.attention.self.query](args = (%encoder_layer_7_output_layer_norm,), kwargs = {})
    %encoder_layer_8_attention_self_key : [#users=1] = call_module[target=encoder.layer.8.attention.self.key](args = (%encoder_layer_7_output_layer_norm,), kwargs = {})
    %view_32 : [#users=1] = call_method[target=view](args = (%encoder_layer_8_attention_self_key, 1, 20, 12, 64), kwargs = {})
    %permute_32 : [#users=1] = call_method[target=permute](args = (%view_32, 0, 2, 1, 3), kwargs = {})
    %encoder_layer_8_attention_self_value : [#users=1] = call_module[target=encoder.layer.8.attention.self.value](args = (%encoder_layer_7_output_layer_norm,), kwargs = {})
    %view_33 : [#users=1] = call_method[target=view](args = (%encoder_layer_8_attention_self_value, 1, 20, 12, 64), kwargs = {})
    %permute_33 : [#users=1] = call_method[target=permute](args = (%view_33, 0, 2, 1, 3), kwargs = {})
    %view_34 : [#users=1] = call_method[target=view](args = (%encoder_layer_8_attention_self_query, 1, 20, 12, 64), kwargs = {})
    %permute_34 : [#users=1] = call_method[target=permute](args = (%view_34, 0, 2, 1, 3), kwargs = {})
    %transpose_8 : [#users=1] = call_method[target=transpose](args = (%permute_32, -1, -2), kwargs = {})
    %matmul_16 : [#users=1] = call_function[target=torch.matmul](args = (%permute_34, %transpose_8), kwargs = {})
    %truediv_8 : [#users=1] = call_function[target=operator.truediv](args = (%matmul_16, 8.0), kwargs = {})
    %_tensor_constant8 : [#users=1] = get_attr[target=_tensor_constant8]
    %add_26 : [#users=1] = call_function[target=operator.add](args = (%truediv_8, %_tensor_constant8), kwargs = {})
    %softmax_8 : [#users=1] = call_function[target=torch.nn.functional.softmax](args = (%add_26,), kwargs = {dim: -1, _stacklevel: 3, dtype: None})
    %encoder_layer_8_attention_self_dropout : [#users=1] = call_module[target=encoder.layer.8.attention.self.dropout](args = (%softmax_8,), kwargs = {})
    %matmul_17 : [#users=1] = call_function[target=torch.matmul](args = (%encoder_layer_8_attention_self_dropout, %permute_33), kwargs = {})
    %permute_35 : [#users=1] = call_method[target=permute](args = (%matmul_17, 0, 2, 1, 3), kwargs = {})
    %contiguous_8 : [#users=1] = call_method[target=contiguous](args = (%permute_35,), kwargs = {})
    %view_35 : [#users=1] = call_method[target=view](args = (%contiguous_8, 1, 20, 768), kwargs = {})
    %encoder_layer_8_attention_output_dense : [#users=1] = call_module[target=encoder.layer.8.attention.output.dense](args = (%view_35,), kwargs = {})
    %encoder_layer_8_attention_output_dropout : [#users=1] = call_module[target=encoder.layer.8.attention.output.dropout](args = (%encoder_layer_8_attention_output_dense,), kwargs = {})
    %add_27 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_8_attention_output_dropout, %encoder_layer_7_output_layer_norm), kwargs = {})
    %encoder_layer_8_attention_output_layer_norm : [#users=2] = call_module[target=encoder.layer.8.attention.output.LayerNorm](args = (%add_27,), kwargs = {})
    %encoder_layer_8_intermediate_dense : [#users=1] = call_module[target=encoder.layer.8.intermediate.dense](args = (%encoder_layer_8_attention_output_layer_norm,), kwargs = {})
    %gelu_8 : [#users=1] = call_function[target=torch.nn.functional.gelu](args = (%encoder_layer_8_intermediate_dense,), kwargs = {})
    %encoder_layer_8_output_dense : [#users=1] = call_module[target=encoder.layer.8.output.dense](args = (%gelu_8,), kwargs = {})
    %encoder_layer_8_output_dropout : [#users=1] = call_module[target=encoder.layer.8.output.dropout](args = (%encoder_layer_8_output_dense,), kwargs = {})
    %add_28 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_8_output_dropout, %encoder_layer_8_attention_output_layer_norm), kwargs = {})
    %encoder_layer_8_output_layer_norm : [#users=4] = call_module[target=encoder.layer.8.output.LayerNorm](args = (%add_28,), kwargs = {})
    %encoder_layer_9_attention_self_query : [#users=1] = call_module[target=encoder.layer.9.attention.self.query](args = (%encoder_layer_8_output_layer_norm,), kwargs = {})
    %encoder_layer_9_attention_self_key : [#users=1] = call_module[target=encoder.layer.9.attention.self.key](args = (%encoder_layer_8_output_layer_norm,), kwargs = {})
    %view_36 : [#users=1] = call_method[target=view](args = (%encoder_layer_9_attention_self_key, 1, 20, 12, 64), kwargs = {})
    %permute_36 : [#users=1] = call_method[target=permute](args = (%view_36, 0, 2, 1, 3), kwargs = {})
    %encoder_layer_9_attention_self_value : [#users=1] = call_module[target=encoder.layer.9.attention.self.value](args = (%encoder_layer_8_output_layer_norm,), kwargs = {})
    %view_37 : [#users=1] = call_method[target=view](args = (%encoder_layer_9_attention_self_value, 1, 20, 12, 64), kwargs = {})
    %permute_37 : [#users=1] = call_method[target=permute](args = (%view_37, 0, 2, 1, 3), kwargs = {})
    %view_38 : [#users=1] = call_method[target=view](args = (%encoder_layer_9_attention_self_query, 1, 20, 12, 64), kwargs = {})
    %permute_38 : [#users=1] = call_method[target=permute](args = (%view_38, 0, 2, 1, 3), kwargs = {})
    %transpose_9 : [#users=1] = call_method[target=transpose](args = (%permute_36, -1, -2), kwargs = {})
    %matmul_18 : [#users=1] = call_function[target=torch.matmul](args = (%permute_38, %transpose_9), kwargs = {})
    %truediv_9 : [#users=1] = call_function[target=operator.truediv](args = (%matmul_18, 8.0), kwargs = {})
    %_tensor_constant9 : [#users=1] = get_attr[target=_tensor_constant9]
    %add_29 : [#users=1] = call_function[target=operator.add](args = (%truediv_9, %_tensor_constant9), kwargs = {})
    %softmax_9 : [#users=1] = call_function[target=torch.nn.functional.softmax](args = (%add_29,), kwargs = {dim: -1, _stacklevel: 3, dtype: None})
    %encoder_layer_9_attention_self_dropout : [#users=1] = call_module[target=encoder.layer.9.attention.self.dropout](args = (%softmax_9,), kwargs = {})
    %matmul_19 : [#users=1] = call_function[target=torch.matmul](args = (%encoder_layer_9_attention_self_dropout, %permute_37), kwargs = {})
    %permute_39 : [#users=1] = call_method[target=permute](args = (%matmul_19, 0, 2, 1, 3), kwargs = {})
    %contiguous_9 : [#users=1] = call_method[target=contiguous](args = (%permute_39,), kwargs = {})
    %view_39 : [#users=1] = call_method[target=view](args = (%contiguous_9, 1, 20, 768), kwargs = {})
    %encoder_layer_9_attention_output_dense : [#users=1] = call_module[target=encoder.layer.9.attention.output.dense](args = (%view_39,), kwargs = {})
    %encoder_layer_9_attention_output_dropout : [#users=1] = call_module[target=encoder.layer.9.attention.output.dropout](args = (%encoder_layer_9_attention_output_dense,), kwargs = {})
    %add_30 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_9_attention_output_dropout, %encoder_layer_8_output_layer_norm), kwargs = {})
    %encoder_layer_9_attention_output_layer_norm : [#users=2] = call_module[target=encoder.layer.9.attention.output.LayerNorm](args = (%add_30,), kwargs = {})
    %encoder_layer_9_intermediate_dense : [#users=1] = call_module[target=encoder.layer.9.intermediate.dense](args = (%encoder_layer_9_attention_output_layer_norm,), kwargs = {})
    %gelu_9 : [#users=1] = call_function[target=torch.nn.functional.gelu](args = (%encoder_layer_9_intermediate_dense,), kwargs = {})
    %encoder_layer_9_output_dense : [#users=1] = call_module[target=encoder.layer.9.output.dense](args = (%gelu_9,), kwargs = {})
    %encoder_layer_9_output_dropout : [#users=1] = call_module[target=encoder.layer.9.output.dropout](args = (%encoder_layer_9_output_dense,), kwargs = {})
    %add_31 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_9_output_dropout, %encoder_layer_9_attention_output_layer_norm), kwargs = {})
    %encoder_layer_9_output_layer_norm : [#users=4] = call_module[target=encoder.layer.9.output.LayerNorm](args = (%add_31,), kwargs = {})
    %encoder_layer_10_attention_self_query : [#users=1] = call_module[target=encoder.layer.10.attention.self.query](args = (%encoder_layer_9_output_layer_norm,), kwargs = {})
    %encoder_layer_10_attention_self_key : [#users=1] = call_module[target=encoder.layer.10.attention.self.key](args = (%encoder_layer_9_output_layer_norm,), kwargs = {})
    %view_40 : [#users=1] = call_method[target=view](args = (%encoder_layer_10_attention_self_key, 1, 20, 12, 64), kwargs = {})
    %permute_40 : [#users=1] = call_method[target=permute](args = (%view_40, 0, 2, 1, 3), kwargs = {})
    %encoder_layer_10_attention_self_value : [#users=1] = call_module[target=encoder.layer.10.attention.self.value](args = (%encoder_layer_9_output_layer_norm,), kwargs = {})
    %view_41 : [#users=1] = call_method[target=view](args = (%encoder_layer_10_attention_self_value, 1, 20, 12, 64), kwargs = {})
    %permute_41 : [#users=1] = call_method[target=permute](args = (%view_41, 0, 2, 1, 3), kwargs = {})
    %view_42 : [#users=1] = call_method[target=view](args = (%encoder_layer_10_attention_self_query, 1, 20, 12, 64), kwargs = {})
    %permute_42 : [#users=1] = call_method[target=permute](args = (%view_42, 0, 2, 1, 3), kwargs = {})
    %transpose_10 : [#users=1] = call_method[target=transpose](args = (%permute_40, -1, -2), kwargs = {})
    %matmul_20 : [#users=1] = call_function[target=torch.matmul](args = (%permute_42, %transpose_10), kwargs = {})
    %truediv_10 : [#users=1] = call_function[target=operator.truediv](args = (%matmul_20, 8.0), kwargs = {})
    %_tensor_constant10 : [#users=1] = get_attr[target=_tensor_constant10]
    %add_32 : [#users=1] = call_function[target=operator.add](args = (%truediv_10, %_tensor_constant10), kwargs = {})
    %softmax_10 : [#users=1] = call_function[target=torch.nn.functional.softmax](args = (%add_32,), kwargs = {dim: -1, _stacklevel: 3, dtype: None})
    %encoder_layer_10_attention_self_dropout : [#users=1] = call_module[target=encoder.layer.10.attention.self.dropout](args = (%softmax_10,), kwargs = {})
    %matmul_21 : [#users=1] = call_function[target=torch.matmul](args = (%encoder_layer_10_attention_self_dropout, %permute_41), kwargs = {})
    %permute_43 : [#users=1] = call_method[target=permute](args = (%matmul_21, 0, 2, 1, 3), kwargs = {})
    %contiguous_10 : [#users=1] = call_method[target=contiguous](args = (%permute_43,), kwargs = {})
    %view_43 : [#users=1] = call_method[target=view](args = (%contiguous_10, 1, 20, 768), kwargs = {})
    %encoder_layer_10_attention_output_dense : [#users=1] = call_module[target=encoder.layer.10.attention.output.dense](args = (%view_43,), kwargs = {})
    %encoder_layer_10_attention_output_dropout : [#users=1] = call_module[target=encoder.layer.10.attention.output.dropout](args = (%encoder_layer_10_attention_output_dense,), kwargs = {})
    %add_33 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_10_attention_output_dropout, %encoder_layer_9_output_layer_norm), kwargs = {})
    %encoder_layer_10_attention_output_layer_norm : [#users=2] = call_module[target=encoder.layer.10.attention.output.LayerNorm](args = (%add_33,), kwargs = {})
    %encoder_layer_10_intermediate_dense : [#users=1] = call_module[target=encoder.layer.10.intermediate.dense](args = (%encoder_layer_10_attention_output_layer_norm,), kwargs = {})
    %gelu_10 : [#users=1] = call_function[target=torch.nn.functional.gelu](args = (%encoder_layer_10_intermediate_dense,), kwargs = {})
    %encoder_layer_10_output_dense : [#users=1] = call_module[target=encoder.layer.10.output.dense](args = (%gelu_10,), kwargs = {})
    %encoder_layer_10_output_dropout : [#users=1] = call_module[target=encoder.layer.10.output.dropout](args = (%encoder_layer_10_output_dense,), kwargs = {})
    %add_34 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_10_output_dropout, %encoder_layer_10_attention_output_layer_norm), kwargs = {})
    %encoder_layer_10_output_layer_norm : [#users=4] = call_module[target=encoder.layer.10.output.LayerNorm](args = (%add_34,), kwargs = {})
    %encoder_layer_11_attention_self_query : [#users=1] = call_module[target=encoder.layer.11.attention.self.query](args = (%encoder_layer_10_output_layer_norm,), kwargs = {})
    %encoder_layer_11_attention_self_key : [#users=1] = call_module[target=encoder.layer.11.attention.self.key](args = (%encoder_layer_10_output_layer_norm,), kwargs = {})
    %view_44 : [#users=1] = call_method[target=view](args = (%encoder_layer_11_attention_self_key, 1, 20, 12, 64), kwargs = {})
    %permute_44 : [#users=1] = call_method[target=permute](args = (%view_44, 0, 2, 1, 3), kwargs = {})
    %encoder_layer_11_attention_self_value : [#users=1] = call_module[target=encoder.layer.11.attention.self.value](args = (%encoder_layer_10_output_layer_norm,), kwargs = {})
    %view_45 : [#users=1] = call_method[target=view](args = (%encoder_layer_11_attention_self_value, 1, 20, 12, 64), kwargs = {})
    %permute_45 : [#users=1] = call_method[target=permute](args = (%view_45, 0, 2, 1, 3), kwargs = {})
    %view_46 : [#users=1] = call_method[target=view](args = (%encoder_layer_11_attention_self_query, 1, 20, 12, 64), kwargs = {})
    %permute_46 : [#users=1] = call_method[target=permute](args = (%view_46, 0, 2, 1, 3), kwargs = {})
    %transpose_11 : [#users=1] = call_method[target=transpose](args = (%permute_44, -1, -2), kwargs = {})
    %matmul_22 : [#users=1] = call_function[target=torch.matmul](args = (%permute_46, %transpose_11), kwargs = {})
    %truediv_11 : [#users=1] = call_function[target=operator.truediv](args = (%matmul_22, 8.0), kwargs = {})
    %_tensor_constant11 : [#users=1] = get_attr[target=_tensor_constant11]
    %add_35 : [#users=1] = call_function[target=operator.add](args = (%truediv_11, %_tensor_constant11), kwargs = {})
    %softmax_11 : [#users=1] = call_function[target=torch.nn.functional.softmax](args = (%add_35,), kwargs = {dim: -1, _stacklevel: 3, dtype: None})
    %encoder_layer_11_attention_self_dropout : [#users=1] = call_module[target=encoder.layer.11.attention.self.dropout](args = (%softmax_11,), kwargs = {})
    %matmul_23 : [#users=1] = call_function[target=torch.matmul](args = (%encoder_layer_11_attention_self_dropout, %permute_45), kwargs = {})
    %permute_47 : [#users=1] = call_method[target=permute](args = (%matmul_23, 0, 2, 1, 3), kwargs = {})
    %contiguous_11 : [#users=1] = call_method[target=contiguous](args = (%permute_47,), kwargs = {})
    %view_47 : [#users=1] = call_method[target=view](args = (%contiguous_11, 1, 20, 768), kwargs = {})
    %encoder_layer_11_attention_output_dense : [#users=1] = call_module[target=encoder.layer.11.attention.output.dense](args = (%view_47,), kwargs = {})
    %encoder_layer_11_attention_output_dropout : [#users=1] = call_module[target=encoder.layer.11.attention.output.dropout](args = (%encoder_layer_11_attention_output_dense,), kwargs = {})
    %add_36 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_11_attention_output_dropout, %encoder_layer_10_output_layer_norm), kwargs = {})
    %encoder_layer_11_attention_output_layer_norm : [#users=2] = call_module[target=encoder.layer.11.attention.output.LayerNorm](args = (%add_36,), kwargs = {})
    %encoder_layer_11_intermediate_dense : [#users=1] = call_module[target=encoder.layer.11.intermediate.dense](args = (%encoder_layer_11_attention_output_layer_norm,), kwargs = {})
    %gelu_11 : [#users=1] = call_function[target=torch.nn.functional.gelu](args = (%encoder_layer_11_intermediate_dense,), kwargs = {})
    %encoder_layer_11_output_dense : [#users=1] = call_module[target=encoder.layer.11.output.dense](args = (%gelu_11,), kwargs = {})
    %encoder_layer_11_output_dropout : [#users=1] = call_module[target=encoder.layer.11.output.dropout](args = (%encoder_layer_11_output_dense,), kwargs = {})
    %add_37 : [#users=1] = call_function[target=operator.add](args = (%encoder_layer_11_output_dropout, %encoder_layer_11_attention_output_layer_norm), kwargs = {})
    %encoder_layer_11_output_layer_norm : [#users=2] = call_module[target=encoder.layer.11.output.LayerNorm](args = (%add_37,), kwargs = {})
    %getitem_2 : [#users=1] = call_function[target=operator.getitem](args = (%encoder_layer_11_output_layer_norm, (slice(None, None, None), 0)), kwargs = {})
    %pooler_dense : [#users=1] = call_module[target=pooler.dense](args = (%getitem_2,), kwargs = {})
    %pooler_activation : [#users=1] = call_module[target=pooler.activation](args = (%pooler_dense,), kwargs = {})
    return {'last_hidden_state': encoder_layer_11_output_layer_norm, 'pooler_output': pooler_activation}
```

Thank you!

So it is a version problem. I let you know this code with

transformers version: 4.16.0.dev0
Python version: 3.9.2
PyTorch version (GPU?): 1.10.1+cpu

do yield the error I gave:

```
from transformers import BertModel
from transformers.utils.fx import symbolic_trace
import transformers

model = BertModel.from_pretrained("bert-base-uncased")

transformers.utils.fx.is_torch_fx_available = lambda: True

traced_model = symbolic_trace(
    model,
    input_names=["input_ids", "attention_mask", "token_type_ids"],
    batch_size=1,
    sequence_length=128,
)
```

I will downgrade PyTorch and transformers, thank you.

Hi @fxmarty,
I am working on a PR (#14321) that should allow you to use both 1.9 and 1.10 version if everything goes well. It should be merged by the end of the week.

It's all good, thanks a lot!