Onnx Runtime Errors With LongT5
Author: reelmath
Pull request: no
Labels: name: Good Second Issue

name: bug

name: ONNX

### System Info

- `optimum` version: 1.2.3 (installed via Github installation)
- `transformers` version: 4.20.1
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.11.0+cu113 (False)
- Tensorflow version (GPU?): 2.8.2 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@stancld @echarlaix @LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

LongT5 with TGlobal Attention isn't able to run sequences longer than **global_block_size * 2**. This is because during the model tracing [num_globals > 0](https://github.com/huggingface/transformers/blob/main/src/transformers/models/longt5/modeling_longt5.py#L191) is being converted to False. I originally posted the error in Optimum (https://github.com/huggingface/optimum/issues/285) but @echarlaix asked me to open an issue here because this error concerns the ONNX export.

Code to reproduce is below:

```
!pip install transformers
!pip install transformers[onnx]
!python -m pip install git+https://github.com/huggingface/optimum.git
!python -m pip install git+[https://github.com/huggingface/optimum.git#egg=optimum[onnxruntime]](https://github.com/huggingface/optimum.git#egg=optimum%5Bonnxruntime%5D)
!pip install datasets
```

```py
from optimum.onnxruntime import ORTModelForSeq2SeqLM

model = ORTModelForSeq2SeqLM.from_pretrained("longt5-tglobal-base", from_transformers=True)
from transformers import AutoTokenizer, pipeline

tokenizer = AutoTokenizer.from_pretrained('google/long-t5-tglobal-base')

onnx_summarization = pipeline("summarization", model=model, tokenizer=tokenizer)

text = # Something longer than 32 tokens if I don't change the number of global blocks
pred = onnx_summarization(text)`
```
```
RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running LessOrEqual node. Name:'LessOrEqual_648' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/math/element_wise_ops.h:603 onnxruntime::Broadcaster::Broadcaster(gsl::span, gsl::span) largest <= 1 was false. Can broadcast 0 by 0 or 1. 16 is invalid.
```
### Expected behavior

Should work for very large seq lens on default global block size without error

Comments:
Hey @reelmath, thanks for opening an issue, it seems you and @echarlaix managed to find the source of the problem.

We unfortunately don't have a lot of bandwidth to dive into solving that code, so I'll add an `onnx` tag and a `Good second issue` tag so that experienced users know that this is an issue that could be fixed. If you'd like to try your hand at it, please go ahead!

Hi, I would like to work on this if it has not been assigned to anyone, but could take some time if that is ok?

Hey @yhl48, this would be great indeed :-)

Hello @reelmath , I was trying to mimic your error with my setting as follows:

- transformers version 4.23.1
- Python version: 3.10.5
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


but I faced the same errors with you.

![abcds](https://user-images.githubusercontent.com/35699839/195714963-171a83dd-d204-4464-b14a-b9449dbbdd58.png)


It looks like the pretrained model is not available anymore?

Upon running the following line

```
model = ORTModelForSeq2SeqLM.from_pretrained("longt5-tglobal-base", from_transformers=True)
```

The following error was raised

```
OSError: longt5-tglobal-base is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.
```

@yhl48 I think you need to use `google/long-t5-tglobal-base` name

Thanks @stancld!

Has this issue been resolved? I can no longer replicate the error.