MarianMT - ONNX only accepts fixed input despite setting dynamic axes
Author: 10-zin
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.1.0
- Platform: Linux-5.8.0-43-generic-x86_64-with-glibc2.29 (Ubuntu 20.04.2)
- Python version: 3.8.3
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes (No too, error persists in both cases)
- Using distributed or parallel set-up in script?: No

### Who can help
marian: @patrickvonplaten, @patil-suraj 

## Information

Model I am using (Bert, XLNet ...): Helsinki-NLP/opus-mt-en-hi (bug persists for other languages too)

The problem arises when using:
* [ ] my own modified scripts: (give details below)
 I slightly modified the  `convert_graph_to_onnx.py` script with the code snippet given below(call to export is exactly the same). Apparently torch.triu() is not supported for onnx conversion, so following prior issues in PyTorch [#32968](https://github.com/pytorch/pytorch/issues/32968) I modified the script, resulting in successful onnx conversion of models.


        torch_triu = torch.triu

        def triu_onnx(x, diagonal=0):
            l = x.shape[0]
            arange = torch.arange(l, device=x.device)
            mask = arange.expand(l, l)
            arange = arange.unsqueeze(-1)
            if diagonal:
                arange = arange + diagonal
            mask = mask >= arange
            return x.masked_fill(mask == 0, 0)

        torch.triu = triu_onnx
        export(
            nlp.model,
            model_args,
            f=output.as_posix(),
            input_names=ordered_input_names,
            output_names=output_names,
            dynamic_axes=dynamic_axes,
            do_constant_folding=True,
            use_external_data_format=use_external_format,
            enable_onnx_checker=True,
            opset_version=opset,
        )
        torch.triu = torch_triu
The tasks I am working on is:
* Simple Machine Translation

## To reproduce

Steps to reproduce the behavior:

1. ```python convert_graph_to_onnx.py --framework pt --model Helsinki-NLP/opus-mt-en-hi onnx-models/opus-mt-en-hi.onnx```
2.  
```
from transformers import AutoTokenizer
from onnxruntime import ExecutionMode, InferenceSession, SessionOptions
import numpy as np 

tok_name = 'Helsinki-NLP/opus-mt-en-hi'
model_name = 'onnx-models/opus-mt-en-hi.onnx'

tokenizer = AutoTokenizer.from_pretrained(tok_name)
options = SessionOptions()
options.intra_op_num_threads = 1
options.execution_mode = ExecutionMode.ORT_SEQUENTIAL
session = InferenceSession(model_name, options)

tokens = tokenizer.encode_plus('Testing onnx conversion through a sample input for machine translation.')
tokens = {name: np.atleast_2d(value) for name, value in tokens.items()}
op = session.run(None, tokens)
```
3. Stack Trace

```
---------------------------------------------------------------------------
RuntimeException                          Traceback (most recent call last)
<ipython-input-96-d76889a45083> in <module>
     14 tokens = tokenizer.encode_plus('Testing onnx conversion through a sample input for machine translation.')
     15 tokens = {name: np.atleast_2d(value) for name, value in tokens.items()}
---> 16 op = session.run(None, tokens)

~/.pyenv/versions/3.8.3/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py in run(self, output_names, input_feed, run_options)
    122             output_names = [output.name for output in self._outputs_meta]
    123         try:
--> 124             return self._sess.run(output_names, input_feed, run_options)
    125         except C.EPFail as err:
    126             if self._enable_fallback:
```
```
RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_62' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:42 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, std::vector<long int>&) gsl::narrow_cast<int64_t>(input_shape.Size()) == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{1,13}, requested shape:{5}
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Ideally the input should have smoothly passed through the onnx converted model during inference, but it doesn't.

Possible useful information
```
Dynamic Axes:
 {'input_ids': {0: 'batch', 1: 'sequence'},
 'attention_mask': {0: 'batch', 1: 'sequence'},
 'output_0': {0: 'batch', 1: 'sequence'},
 'output_1': {0: 'batch', 1: 'sequence'}}
```
```
Generated inputs order: ['input_ids', 'attention_mask']
```
**My lead is that** - 
Exporting is not taking into account the dynamic axes for some reason, when any Marian Mt model is being used.
The error also notes - requested shape size - 5, which is the sequence length of the dummy input (line 196 in `convert_graph_to_onnx.py` while converting to onnx.
Notably, passing an input with sequence length 5 works perfectly fine.
Moreover, this script works perfectly for standard models like distilbert for both model conversion, and inference. So it's surely some model-specific problem.

Will be really helpful to get a fix for this, especially since there are numerous Marian mt models so it can have a larger impact!

Comments:
pinging @mfuntowicz, our `onnx` expert.

Any lead on the solution? @mfuntowicz, @patrickvonplaten, @patil-suraj 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.