onnx run error at translation model
Author: xyx361100238
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.17.0
- Platform: Linux-5.4.0-122-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.11
- PyTorch version (GPU?): 1.8.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@patrickvonplaten
@patil-suraj

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1、convert model  to onnx：
python3 -m transformers.onnx  --model opus-mt-en-zh --atol=2e-04 --feature=seq2seq-lm opus-mt-en-zh-onnx-301
tips：
Validating ONNX model...
        -[✓] ONNX model output names match reference model ({'logits'})
        - Validating ONNX Model output "logits":
                -[✓] (2, 8, 65001) matches (2, 8, 65001)
                -[✓] all values close (atol: 0.0002)
All good, model saved at: opus-mt-en-zh-onnx-301/model.onnx

2、translation：
```py
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from onnxruntime import InferenceSession

tokenizer=AutoTokenizer.from_pretrained("opus-mt-en-zh")
session = InferenceSession("opus-mt-en-zh-onnx-301/model.onnx")
inputs = tokenizer("Using DistilBERT with ONNX Runtime!", return_tensors="pt")
outputs = session.run(output_names=["last_hidden_state"], input_feed=dict(inputs))
```
tips：

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xieyouxi/anaconda3/envs/HuggingFace-torch-gpu/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 196, in run
    raise ValueError("Model requires {} inputs. Input Feed contains {}".format(num_required_inputs, num_inputs))
ValueError: Model requires 4 inputs. Input Feed contains 2
```

### Expected behavior

Unable to translate from en to zh, Am I using the wrong interface？


Comments:
Hey @regisss @JingyaHuang @michaelbenayoun, do you have ideas about what might be happening there? I never used onnxruntime's `InferenceSession`.

@xyx361100238  The error message says that the model requires 4 inputs but you are providing only 2 of them. Either you need to provide the missing inputs, or you need to modify the `OnnxConfig` associated to your model to specify only 2 inputs.

The architecture of *opus-mt-en-zh* seems to be *MarianMTModel*. According to what I see in the `OnnxConfig` [here](https://github.com/huggingface/transformers/blob/8cb5ecd912e09301be126c6ce6e9a22ca7153da4/src/transformers/models/marian/configuration_marian.py#L176), the 4 expected inputs are:
- `input_ids`,
- `attention_mask`,
- `decoder_input_ids`,
- `decoder_attention_mask`.

So I think you are only providing `input_ids` and `attention_mask` here. To generate the missing inputs, you can take a look at [how dummy inputs used for exporting the model are generated](https://github.com/huggingface/transformers/blob/8cb5ecd912e09301be126c6ce6e9a22ca7153da4/src/transformers/models/marian/configuration_marian.py#L233).

Thanks for your reply!
I'm sorry,still understand to generate decoder_input_ids & decoder_attention_mask,could you please give a example,
or onnxruntime  example with marian model!

So you also need to provide the inputs for the decoder side, something along the lines:

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from onnxruntime import InferenceSession

tokenizer=AutoTokenizer.from_pretrained("opus-mt-en-zh")
session = InferenceSession("opus-mt-en-zh-onnx-301/model.onnx")
inputs = tokenizer("Using DistilBERT with ONNX Runtime!", return_tensors="pt")
inputs["decoder_input_ids"] = torch.tensor([tokenizer.bos_token_id], dtype=torch.long)
inputs["decoder_attention_mask"] = torch.tensor([1], dtype=torch.long)
outputs = session.run(output_names=["last_hidden_state"], input_feed=inputs)
```

What is true is that it would be easier if the `decoder_attention_mask` was automatically generated, but we you currently need to provide it manually.

got error：
![image](https://user-images.githubusercontent.com/19569322/183870045-562624c5-ee9c-400f-95a8-cafcf969cdc9.png)


Basically, if you want to your ONNX model to predict the next token, provide the start of sentence token as first token, maybe you do not have `tokenizer.bos_token_id` but you know the value? Or maybe you do not have a start of sentence token?
How are you running things on the `transformers` side?

![image](https://user-images.githubusercontent.com/19569322/183874977-e7dbe217-861d-4a94-b05b-abbc58f188ad.png)


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Has this problem been solved?

Not Yet！

From the [marian tokenizer](https://github.com/huggingface/transformers/blob/e342ac7e0390d157e8c14a8a7c5cf2e715341cee/src/transformers/models/marian/tokenization_marian.py#L146), the bos_token_id is not initialized. Instead it recommends using the decoder_start_token_id from the config. For this model, the decoder_start_token_id is [65000](https://huggingface.co/Helsinki-NLP/opus-mt-en-zh/blob/main/config.json#L24).

Example:
```
session = InferenceSession("opus-mt-en-zh-onnx-301/model.onnx")
inputs = tokenizer("Using DistilBERT with ONNX Runtime!", return_tensors="np")
inputs["decoder_input_ids"] = np.array([[65000]])
inputs["decoder_attention_mask"] = np.array([[1]])
outputs = session.run(None, input_feed=dict(inputs))
```

Alternatively, I found that the optimum library makes working with seq2seq models in ONNX much easier.

```
from transformers import AutoTokenizer, pipeline
from optimum.onnxruntime import ORTModelForSeq2SeqLM

model_path = "Helsinki-NLP/opus-mt-en-zh"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = ORTModelForSeq2SeqLM.from_pretrained(model_path, from_transformers=True)
onnx_translation = pipeline("translation", model=model, tokenizer=tokenizer)

pred = onnx_translation("Hello")
```