Convert LongT5 to ONNX
Author: jplu
Pull request: no
Labels: name: ONNX

### System Info

transformers-cli env

- `transformers` version: 4.24.0
- Platform: Linux-5.4.0-99-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.1+cu102 (True)
- onnxruntime-gpu: 1.13.1
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

ONNX model conversion: @morgan

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

This command line:
```
python -m transformers.onnx --model pszemraj/long-t5-tglobal-base-16384-book-summary --feature seq2seq-lm-with-past --preprocessor tokenizer --framework pt .
```
Gives me the following error during export validation:
```
Validating ONNX model...
Floating point exception (core dumped)
```

### Expected behavior

Having a usable and validated ONNX model.

Comments:
cc @lewtun 

Same silent issue on longformer when `global_attention_mask` is zero everywhere and running with ONNX Runtime. When at least one value is non-zero, it's fine.

Anything I can do on my side to help fixing this issue? :)

@jplu Do you have any warning during the onnx conversion? Things like `TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!`? In my case there were control flows in the model, leading to the issue.

@fxmarty Yes I do have several of these `TracerWarning`:
```
/data/conda/beir/lib/python3.8/site-packages/transformers/models/longt5/modeling_longt5.py:180: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  _global_block_ids_lower_bound = torch.tensor(-1.0, dtype=global_block_ids.dtype, device=global_block_ids.device)
/data/conda/beir/lib/python3.8/site-packages/transformers/models/longt5/modeling_longt5.py:188: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  num_globals = seq_len // global_block_size
/data/conda/beir/lib/python3.8/site-packages/transformers/models/longt5/modeling_longt5.py:190: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_globals > 0:
/data/conda/beir/lib/python3.8/site-packages/transformers/models/longt5/modeling_longt5.py:217: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  block_ids >= 0, torch.tensor(global_seq_len, dtype=block_ids.dtype, device=block_ids.device)
/data/conda/beir/lib/python3.8/site-packages/transformers/models/longt5/modeling_longt5.py:217: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  block_ids >= 0, torch.tensor(global_seq_len, dtype=block_ids.dtype, device=block_ids.device)
/data/conda/beir/lib/python3.8/site-packages/transformers/models/longt5/modeling_longt5.py:84: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if x.shape[dim] % block_len != 0:
/data/conda/beir/lib/python3.8/site-packages/transformers/models/longt5/modeling_longt5.py:67: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if not all(x.shape):
/data/conda/beir/lib/python3.8/site-packages/transformers/models/longt5/modeling_longt5.py:86: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  num_blocks = x.shape[dim] // block_len
/data/conda/beir/lib/python3.8/site-packages/transformers/models/longt5/modeling_longt5.py:89: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if 0 in output_shape:
/data/conda/beir/lib/python3.8/site-packages/transformers/modeling_utils.py:769: FutureWarning: The `device` argument is deprecated and will be removed in v5 of Transformers.
  warnings.warn(
/data/conda/beir/lib/python3.8/site-packages/transformers/modeling_utils.py:781: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if causal_mask.shape[1] < attention_mask.shape[1]:
In-place op on output of tensor.shape. See https://pytorch.org/docs/master/onnx.html#avoid-inplace-operations-when-using-tensor-shape-in-tracing-mode
In-place op on output of tensor.shape. See https://pytorch.org/docs/master/onnx.html#avoid-inplace-operations-when-using-tensor-shape-in-tracing-mode
In-place op on output of tensor.shape. See https://pytorch.org/docs/master/onnx.html#avoid-inplace-operations-when-using-tensor-shape-in-tracing-mode
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
```

Could be a duplicate https://github.com/huggingface/transformers/issues/19297

In my opinion, you are falling into the issue that the example input provided during the conversion for `torch.onnx.convert` takes a path different than the one you wish to use during inference with the exported ONNX.

A good reference is https://pytorch.org/docs/stable/generated/torch.jit.trace.html .

We should export along the `model.onnx` probably an `onnx_config.json` detailing which cases are supported by the exported ONNX.

OK I see. Should-I test to change the input in the export code in order to see if it goes better with a more appropriate input? How did you manage to convert and validate this model as it appears in the list of available models to be exported? You have tested on the official Google one?

Pinging @echarlaix , is longt5 the model you were dealing with? To me given the trace warnings, especially `num_globals = seq_len // global_block_size` that is a constant,if you want a quick and dirty fix you can indeed change the input length in the export code to match your input length for your use case using the .onnx.

The validation for onnx conversion is currently lacking, as it tests only on a very similar sequence length (typically 9 while the export is done with a sequence length = 8, see https://github.com/huggingface/transformers/blob/v4.24-release/src/transformers/onnx/convert.py#L382-L397 ). So if later on there is controlflow dependent on the sequence length, a single path is recorded during the export and you are screwed. We are looking for a clean solution for this.

edit: although here `global_block_size = 16`, and `seq_len = 8` then `seq_len = 9` so I would expect the path to be the same in the two warnings concerning `num_globals`. The issue could come from an other of the following warnings.

> Pinging @echarlaix , is longt5 the model you were dealing with? 

Yes we had the same [issue](https://github.com/huggingface/optimum/issues/285#issuecomment-1191409005) for LongT5 model with transient-global attention (models with local attention were not causing problem). Now transferred to huggingface/transformers#18243.


I have tried multiple `seq_len` for the input. As long as the value is not too lower than the "original" value it seems working, but if it goes too lower, I start to get a big "max absolute difference" and the validation doesn't pass. So indeed, it is not really usable and seems too unstable as you said @fxmarty. Thanks a lot anyway for your lights on this, I let the issue open, don't hesitate to ping me here if I can do something to help fixing on my side.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

not stale

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

not stale

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

still not

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

not

cc @IlyasMoutawwakil 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.