Repository: huggingface/transformers

Issue:
Converting to ONNX doesn't apply to all models
Is it possible to list or highlight which models are ONNX convertible? 

On a fresh python environment: 

```
python -m pip install -U transformers
python -m  pip install mosestokenizer
```

The transformer version on the machine is `2.11.0` 

When trying to convert the https://huggingface.co/transformers/model_doc/marian.html to ONNX, it throws the following error:


```
$ python convert_graph_to_onnx.py --framework pt --model Helsinki-NLP/opus-mt-en-ROMANCE opus-mt-en-romance.onnx

Neither PyTorch nor TensorFlow >= 2.0 have been found. Models won't be available and only tokenizers, configurationand file/data utilities can be used.
ONNX opset version set to: 11

Loading pipeline (model: Helsinki-NLP/opus-mt-en-ROMANCE, tokenizer: Helsinki-NLP/opus-mt-en-ROMANCE)
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/tokenization_utils.py:828: FutureWarning: Parameter max_len is deprecated and will be removed in a future release. Use model_max_length instead.
  warnings.warn(
stdbuf was not found; communication with perl may hang due to stdio buffering.
Error while converting the model: 'NoneType' object has no attribute 'from_pretrained'
```

Then after installing pytorch

```
$ python -m pip install -U pytorch

$ python convert_graph_to_onnx.py --framework pt --model Helsinki-NLP/opus-mt-en-ROMANCE opus-mt-en-romance.onnx

ONNX opset version set to: 11
Loading pipeline (model: Helsinki-NLP/opus-mt-en-ROMANCE, tokenizer: Helsinki-NLP/opus-mt-en-ROMANCE)
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/tokenization_utils.py:828: FutureWarning: Parameter max_len is deprecated and will be removed in a future release. Use model_max_length instead.
  warnings.warn(
stdbuf was not found; communication with perl may hang due to stdio buffering.

Downloading: 100%|██████████████████████████████| 312M/312M [01:59<00:00, 2.61MB/s]

Error while converting the model: Folder /Users/username/git-stuff/transformers/src/transformers is not empty, aborting conversion
```

Then after creating a new directory: 

```

$ python ../convert_graph_to_onnx.py --framework pt --model Helsinki-NLP/opus-mt-en-ROMANCE opus-mt-en-romance.onnx

ONNX opset version set to: 11
Loading pipeline (model: Helsinki-NLP/opus-mt-en-ROMANCE, tokenizer: Helsinki-NLP/opus-mt-en-ROMANCE)

/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/tokenization_utils.py:828: FutureWarning: Parameter max_len is deprecated and will be removed in a future release. Use model_max_length instead.
  warnings.warn(
stdbuf was not found; communication with perl may hang due to stdio buffering.

Using framework PyTorch: 1.5.0
Found input input_ids with shape: {0: 'batch', 1: 'sequence'}
Found input attention_mask with shape: {0: 'batch', 1: 'sequence'}
Found output output_0 with shape: {0: 'batch', 1: 'sequence'}
Found output output_1 with shape: {0: 'batch', 1: 'sequence'}
Ensuring inputs are in correct order
decoder_input_ids is not present in the generated input list.
Generated inputs order: ['input_ids', 'attention_mask']
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/modeling_bart.py:173: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if not padding_mask.any():
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/modeling_bart.py:590: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert embed_dim == self.embed_dim
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/modeling_bart.py:591: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert list(query.size()) == [tgt_len, bsz, embed_dim]
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/modeling_bart.py:633: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert attn_weights.size() == (bsz * self.num_heads, tgt_len, src_len)
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/modeling_bart.py:642: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert key_padding_mask is None or key_padding_mask.size()[:2] == (bsz, src_len,)
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/transformers/modeling_bart.py:654: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert attn_output.size() == (bsz * self.num_heads, tgt_len, self.head_dim)
/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/torch/onnx/utils.py:736: UserWarning: ONNX export failed on ATen operator triu because torch.onnx.symbolic_opset11.triu does not exist
  warnings.warn("ONNX export failed on ATen operator {} because "
Error while converting the model: Exporting the operator triu to ONNX opset version 11 is not supported. Please open a bug to request ONNX export support for the missing operator.
```

It is understandable that ONNX might not be supporting all models that are available through Huggingface's transformers. 

**But is there a way to know which model(s) are ONNX convertible and which aren't?**


Discussion:
Here is a related PyTorch issue: https://github.com/pytorch/pytorch/issues/32968.

Here is a list of pretraineds model that can be exported to ONNX:
bert-base-cased, distilbert-base-uncased, roberta-base, gpt2, distilgpt2, openai-gpt, albert-base-v2, xlnet-base-cased

So far, the following models have problem in exporting to ONNX:
bart-large, transfo-xl-wt103, t5-base, xlm-mlm-en-2048




Anyone had any luck resolving this issue? The Pytorch issue linked above has a couple of potential workarounds. I notice it affects Pegasus as well
This pull request was closed by its author but doesn't raise the issue when I run convert on a Bart model. https://github.com/huggingface/transformers/pull/6334


Resolving patch:
diff --git a/src/transformers/modeling_bart.py b/src/transformers/modeling_bart.py
--- a/src/transformers/modeling_bart.py
+++ b/src/transformers/modeling_bart.py
@@ -154,9 +154,10 @@ def _prepare_bart_decoder_inputs(
     if decoder_padding_mask is not None and decoder_padding_mask.shape[1] > 1:
         # never mask leading token, even if it is pad
         decoder_padding_mask[:, 0] = decoder_padding_mask[:, 1]
-    causal_mask = torch.triu(fill_with_neg_inf(torch.zeros(tgt_len, tgt_len)), 1).to(
-        dtype=causal_mask_dtype, device=decoder_input_ids.device
-    )
+    tmp = fill_with_neg_inf(torch.zeros(tgt_len, tgt_len))
+    mask = torch.arange(tmp.size(-1))
+    tmp.masked_fill_(mask < (mask + 1).view(tmp.size(-1), 1), 0)
+    causal_mask = tmp.to(dtype=causal_mask_dtype, device=decoder_input_ids.device)
     return decoder_input_ids, decoder_padding_mask, causal_mask
 
 


Test patch: