core dumps run_onnx_exporter.py in gpu.
Author: lonelydancer
Pull request: no

Hi, 
when i use the scrips in  https://github.com/huggingface/transformers/tree/master/examples/onnx/pytorch/summarization/run_onnx_exporter.py; 
i change the code
ort_sess = onnxruntime.InferenceSession(new_onnx_file_path, providers=['CUDAExecutionProvider'])
i get core dumps. 
![image](https://user-images.githubusercontent.com/548443/147087899-047c75b3-f128-44d0-8e84-43b93dc4bd98.png)
![image](https://user-images.githubusercontent.com/548443/147522053-0532a0cf-9136-4b75-aa48-00024ae69487.png)

And the scripts is ok on cpu.
My purpose is to get speed up in GPU for bart model.

environment:
Collecting environment information...
PyTorch version: 1.10.1+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (GCC) 8.2.0
Clang version: 3.8.0 (tags/RELEASE_380/final)
CMake version: version 3.16.0
Libc version: glibc-2.26

Python version: 3.7.12 (default, Sep 10 2021, 00:21:48)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.15.0-163-generic-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: True
CUDA runtime version: 11.2.142
GPU models and configuration: 
GPU 0: Tesla K80
GPU 1: Tesla K80
GPU 2: Tesla K80
GPU 3: Tesla K80

Nvidia driver version: 470.82.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.4
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.5
[pip3] torch==1.10.1+cu113
@lewtun  @NielsRogge 


Comments:
Hi @lonelydancer thank you for raising this issue!

If I understand correctly, you changed [this line](https://github.com/huggingface/transformers/blob/13504dcbea231d2cae701d1ffdeb0810d62aff81/examples/onnx/pytorch/summarization/run_onnx_exporter.py#L147) in the `run_onnx_exporter.py` script as follows:

```python
ort_sess = onnxruntime.InferenceSession(new_onnx_file_path, providers=['CUDAExecutionProvider'])
```

I ran this but was not able to reproduce your error - could it be a problem with your CUDA version? (I was only able to test this on CUDA Version: 10.2)

One gotcha that I've experienced before is having both `onnxruntime` _and_ `onnxruntime-gpu` installed. Is it possible you still have the former installed in your environment? You could try running 

```
pip uninstall onnxruntime
```

and then re-running the export script.

@lewtun 
hello, i change the cuda version to 10.2, and i'm using onnxruntime-gpu. The scripts is ok.
but i calculate the time of the generate function( 0.10198545455932617s) and the ort_sess.run(0.12904071807861328s)
is it normal? I expect the onnx version will faster.



Good to hear that it works on CUDA Version 10.2 @lonelydancer! 

Regarding the runtime performance between the non-optimised and optimised models on GPU, I too would have expected the optimised version to run faster. One possibility is that the implementation of beam search in the `generate()` of `transformers` is somehow more efficient.

Perhaps @fatcat-z can comment here on whether he ever tested his BART implementation on a GPU and saw a performance gain?

@lewtun 
i think i still have problem.
2021-12-23 08:53:56.216073982 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:535 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.

Could you please provide a minimal reproducible example @lonelydancer so I can test it myself?

Ideally, it would be good to have:

* The modified `run_onnx_exporter.py` script
* The CLI command used to export the model
* A small script where you run the model with ONNX Runtime

Hi, @lewtun 
https://github.com/lonelydancer/algorithm/blob/master/run_onnx_exporter.py
transformers/examples/onnx/pytorch/summarization
python run_onnx_exporter.py --model_name_or_path facebook/bart-base --device=cuda

![image](https://user-images.githubusercontent.com/548443/147306479-517f3321-dc8d-4a96-9073-39de9049f573.png)

transformers       4.16.0.dev0
onnxruntime-gpu    1.10.0
onnx               1.10.2
torch              1.10.0+cu102

Hi @lewtun, I wanted to jump into the discussion here as I'm experiencing similar issues with an MBart model. CPU inference time is comparable for Pytorch and ONNXRuntime, with Pytorch being slightly faster. Additionally, I experienced similar core dumps when I tried to run the CPU-generated ONNX model on GPU (GTX3060 CUDA 11.4) and generating directly on GPU requires more than 12GB memory which is more than my gtx3060 has. 

I saw you made it work on CUDA 10.2 and I wanted to ask if you or @lonelydancer managed to run this implementation with the TensorRT backend to still potentially gain that edge over the Pytorch GPU implementation? Attempting TensorRT backend on the CPU-generated model (with `onnx.shape_inference.infer_shapes_path` ran on it) currently results in this error for me:

`[E:onnxruntime:, inference_session.cc:1448 operator()] Exception during initialization: /onnxruntime_src/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc:925 SubGraphCollection_t onnxruntime::TensorrtExecutionProvider::GetSupportedList(SubGraphCollection_t, int, int, const onnxruntime::GraphViewer&, bool*) const [ONNXRuntimeError] : 1 : FAIL : TensorRT input: 3419 has no shape specified. Please run shape inference on the onnx model first. Details can be found in https://www.onnxruntime.ai/docs/reference/execution-providers/TensorRT-ExecutionProvider.html#shape-inference-for-tensorrt-subgraphs`

Is this something to be expected due to the beam search implementation requiring empty output nodes (?) or could this be related to the CUDA version?

Thanks!

@JeroendenBoef 
i try the config of cuda10.2/onnxruntime-gpu 1.6 and cuda11.4/onnxruntime-gpu 10.0 in https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html;  it doesn't work on gpu. i update my driver, it still not work.

@lewtun @LysandreJik @fatcat-z   is there any solution? i get stuck by the problem. 

Sorry for the slow reply @lonelydancer - I’m on leave for a few more days and will take a look at this issue next week.

@lewtun do you have time to see this issue? i really need your help.

Hi @lonelydancer, unfortunately I'm still not able to reproduce your error. Using your script and export command, the following test works for me:

```python
import numpy as np
from transformers import AutoTokenizer, AutoConfig
from onnxruntime import InferenceSession, SessionOptions

num_beams = 2
max_length = 5
model_ckpt = "facebook/bart-base"
onnx_file_path = "BART.onnx"
ARTICLE_TO_SUMMARIZE = "My friends are cool but they eat too many carbs."

def main():
    # Prepare inputs
    tokenizer = AutoTokenizer.from_pretrained(model_ckpt)
    config = AutoConfig.from_pretrained(model_ckpt)
    inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors="pt")
    # Create ORT session
    options = SessionOptions()
    ort_sess = InferenceSession(onnx_file_path, options, providers=["CUDAExecutionProvider"])
    # Run inference
    ort_out = ort_sess.run(
        None,
        {
            "input_ids": inputs["input_ids"].cpu().numpy(),
            "attention_mask": inputs["attention_mask"].cpu().numpy(),
            "num_beams": np.array(num_beams),
            "max_length": np.array(max_length),
            "decoder_start_token_id": np.array(config.decoder_start_token_id),
        },
    )

    print(f"ORT outputs: {ort_out}")
    print("Success!")

if __name__ == "__main__":
    main()
```

For reference, this is the output of `transformers-cli env`:

```
- `transformers` version: 4.15.0.dev0
- Platform: Linux-5.0.0-1020-gcp-x86_64-with-debian-buster-sid
- Python version: 3.7.11
- PyTorch version (GPU?): 1.10.1+cu102 (True)
```

I'm also running the following versions of the ONNX libraries:

```
onnx==1.10.2
onnxruntime-gpu==1.10.0
onnxruntime-tools==1.7.0
```

Finally, here's the output from `nvidia-smi`

```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.26       Driver Version: 430.26       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   35C    P0    37W / 300W |      0MiB / 16160MiB |      0%      Default |
```

My suggestion would be to share a minimal reproducible example of the code you're using to run inference with ONNX Runtime. Without that, it is very hard to understand what is causing the issue.

@lewtun thank you very much.
may i know the time of the pytorch scripts and onnx?
is there "Failed to create CUDAExecutionProvider" in your log?


Hi @lonelydancer here's the average latencies I get on CPU vs GPU with the example inputs I shared above:

```
CPU
Average latency (ms) - 122.95 +\- 6.40

GPU
Average latency (ms) - 153.27 +\- 1.55
```

So indeed, one gets slightly faster inference on CPU vs GPU. To get these numbers, I first exported the models as follows:

```bash
# Export CPU model
python run_onnx_exporter.py --model_name_or_path facebook/bart-base --device=cpu --output_file_path=bart_cpu.onnx
# Export GPU model
python run_onnx_exporter.py --model_name_or_path facebook/bart-base --device=cuda --output_file_path=bart_gpu.onnx
```

And then ran them through the following script:

```python
from time import perf_counter

import numpy as np
from tqdm.auto import tqdm

from onnxruntime import InferenceSession, SessionOptions
from transformers import AutoConfig, AutoTokenizer


num_beams = 2
max_length = 5
model_ckpt = "facebook/bart-base"
onnx_file_path = "bart_cpu.onnx" # Change to `bart_gpu.onnx` for CUDA inference
ARTICLE_TO_SUMMARIZE = "My friends are cool but they eat too many carbs."


def main():
    # Prepare inputs
    tokenizer = AutoTokenizer.from_pretrained(model_ckpt)
    config = AutoConfig.from_pretrained(model_ckpt)
    inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors="pt")
    # Create ORT session
    options = SessionOptions()
    ort_sess = InferenceSession(onnx_file_path, options, providers=["CPUExecutionProvider"]) # Change to `CUDAExecutionProvider` for CUDA inference
    # Run inference
    latencies = []
    for _ in tqdm(range(100)):
        start_time = perf_counter()
        ort_out = ort_sess.run(
            None,
            {
                "input_ids": inputs["input_ids"].cpu().numpy(),
                "attention_mask": inputs["attention_mask"].cpu().numpy(),
                "num_beams": np.array(num_beams),
                "max_length": np.array(max_length),
                "decoder_start_token_id": np.array(config.decoder_start_token_id),
            },
        )
        latency = perf_counter() - start_time
        latencies.append(latency)
    # Compute run statistics
    time_avg_ms = 1000 * np.mean(latencies)
    time_std_ms = 1000 * np.std(latencies)
    print(f"Average latency (ms) - {time_avg_ms:.2f} +\- {time_std_ms:.2f}")
    print(f"ORT outputs: {ort_out}")
    print("Success!")


if __name__ == "__main__":
    main()
```

I do not see any `Failed to create CUDAExecutionProvider` errors in my logs, which suggests there is an environment issue with your ONNX Runtime installation not finding CUDA. 

If you wish to run fast summarisation with `transformers` and ONNX Runtime, I suggest opening a feature request in our `optimum` [library](https://github.com/huggingface/optimum). The `transformers` library is only concerned with _exporting_ models to ONNX, while `optimum` is responsible for _optimising_ these exports. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.