convert_graph_to_onnx.py and associated example notebook are broken for TensorFlow
Author: amaiya
Pull request: no
Labels: name: wontfix

## Information
The `convert_graph_to_onnx.py` file and the associated [example notebook](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb) appear to be broken for TensorFlow.


For ONNX-exported TensorFlow models, **only input tokens of length 5 are accepted**.  Other inputs (e.g., `len(tokens)>5`) result in an error:

```
InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: input_ids for the following indices
index: 1 Got: 6 Expected: 5
```
Also, If you run `session.get_inputs()` on ONNX-exported TensorFlow model, only `input_ids` key is listed as inputs (i.e., no `attention_mask`) while ONNX PyTorch behaves differently:
```python
# ONNX TensorFlow inputs for BERT model
print([input.name for input in cpu_model.get_inputs()])
# only prints 'input_ids' - no 'attention_mask'

# ONNX PyTorch inputs for BERT model
print([input.name for input in cpu_model.get_inputs()])
# prints ['input_ids', 'attention_mask', 'token_type_ids']
```


## How to Reproduce
In the [example notebook](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb), uncomment this TensorFlow `convert` line:
```
convert(framework="tf", model="bert-base-cased", output="onnx-test-tf/bert-base-cased.onnx", opset=11)
```
I have also posted [this Google Colab notebook](https://colab.research.google.com/drive/1A2frWgfRlL5Ysf7xVVifx58NmEoxeYmu?usp=sharing) that more concisely reproduces this issue.


Comments:
## Environment Info
- `transformers` version: 3.4.0
- Platform: Linux-4.15.0-108-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cpu (False)
- Tensorflow version (GPU?): 2.3.1 (False)

Also, reproduced on Google Colab, as indicated above.

## Who can help
TensorFlow: @jplu 
ONNX: @mfuntowicz @sgugger  @LysandreJik


You have to create your own model with the size you need and then use the script to convert it. All the TF models are by default initialized with input sequence of 5 tokens.

@jplu  Thanks.  Would you mind clarifying what you mean by "create your own model with the size you need"?  I'm creating and fine-tuning a model with `TFBertForSequenceClassification.from_pretrained` and was trying to use the example notebook to convert it.  

What I mean is that you have to update the input shape of your model. When you do:

```
TFBertForSequenceClassification.from_pretrained("name")
```

You model is initialized with a [dummy input](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_utils.py#L330) of [5 tokens](https://github.com/huggingface/transformers/blob/master/src/transformers/file_utils.py#L215). Then when you use your model the max length allowed is 5 by default.

If you want to use a model with a larger max length you have to update your input shape with:
```
from transformers import TFBertForSequenceClassification, BertTokenizerFast
import tensorflow as tf
my_model_name = "bert-base-cased" # replace here by the name of your model
tokenizer = BertTokenizerFast.from_pretrained(my_model_name )
model = TFBertForSequenceClassification.from_pretrained(my_model_name )
size = 510 # the max length you expect for your model. Don't forget the two extra tokens of start and end, here 510 + 2 to make 512 which is the max length allowed for all the models (except longformer).
inputs_dict = tokenizer("hello" * [size], return_tensors="tf")
model._saved_model_inputs_spec = None
model._set_save_spec(inputs_dict)
tf.saved_model.save(model, "path")
``` 

And then you can create your ONNX model afterward from your saved model that will take as input your proper input length.

@jplu  Thank you!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> 
NotEncodableError: No encoder for object {'input_ids': TensorSpec(shape=(None, 128), dtype=tf.int32, name='input_ids/input_ids'), 'token_type_ids': TensorSpec(shape=(None, 128), dtype=tf.int32, name='token_type_ids'), 'attention_mask': TensorSpec(shape=(None, 128), dtype=tf.int32, name='attention_mask')} of type <class 'transformers.tokenization_utils_base.BatchEncoding'>. 

I am using the mobileBert model. But when I follow this procedure to save model in Saved Model fromat, it gives the error above. Any suggestions? Thanks!