Check TF ops for ONNX compliance
Author: jplu
Pull request: yes

# What does this PR do?

This PR aims to check if a model is compliant with ONNX Opset12 by adding a quick test and a script. The script is only for testing a saved model while the quick test aims to be run over a manually built graph. For now, only BERT is forced to be compliant with ONNX, but the test can be unlocked for any other model.

The logic can also be extended to any other framework/SDK we might think of, such as TFLite or NNAPI.


Comments:
That's the moment to create that list :)

Really like the idea of improving our ONNX compatibility in a more reliable way. 

In this sense, I'm not sure this is the easiest way for approaching the problem. 
May be it would be more suitable to just attempt to export the model through `keras2onnx` and report errors. This can also allow us to more easily test compatibility with various ONNX opset (10 is the minimum required).

We already have the `keras2onnx` dependency when using the optional requirements `onnxruntime`

Also, regarding the list of model to be supported I think we would like to have: 
- BERT
- GPT2
- BART (cc @Narsil wdyt?)

> In this sense, I'm not sure this is the easiest way for approaching the problem. 
May be it would be more suitable to just attempt to export the model through `keras2onnx` and report errors. This can also allow us to more easily test compatibility with various ONNX opset (10 is the minimum required).

I'm not in favour of running the export directly in the tests, as it is less flexible and not a compatible solution with other frameworks/SDKs. We can add any other opsets without problems with the proposed approach, but I don't think that going below the 12 is good idea. Only a few of the current models are compliant with opset < 12. Also, the default in the convert script is 11, not 10, so maybe we can propose opset 11 to be aligned. I think the proposed models below are compliant but we will largely reduce the number of compliant models with ONNX.

> Also, regarding the list of model to be supported I think we would like to have:
BERT
GPT2
BART (cc @Narsil wdyt?)

These three are ok on my side!

Add BART and GPT2 as a mandatory green test.

Testing ONNX operator support might be more complicated than this. 

Each operator in itself supports a set of different shape(s) as input/output combined with different data types and various dynamic axis support ... 

I would go for the easiest solution, well tested of using the official converter to report incompatibilities.

The problem with the solution to use the converter is that we cannot have the full list of incompatible operators, it will stop at the first encounter one, which would be too much annoying IMO. I think we can also assume that as long as the operator belongs to this list https://github.com/onnx/tensorflow-onnx/blob/master/support_status.md it is compliant. Until now, this assumption is true for all of our models.

Unless you know a case for which it is not true?

Also, I'm afraid to add a dependency to the onnxruntime would switch the quick test into a slow test, which reduces the traceability of a potential change that will break it.

If @LysandreJik, @sgugger and @patrickvonplaten agree on making the TF tests dependent on the two keras2onnx and onnxruntime packages, I can add a slow test that will run the following pipeline:

1. Create a SavedModel
2. Convert this SavedModel into ONNX with keras2onnx
3. Run the converted model with onnxruntime

We can add a test depending on `keras2onnx` or `onnxruntime` with a `@require_onnx` decorator. If you decide to go down this road, according to the time spent doing those tests, we'll probably put them in the slow suite (which is okay, no need to test that the model opsets on each PR)

I  like the idea to add a decorator. I will add a slow test doing this in addition to the quick test.

I have reworked the quick test. Now, we can easily specify against which opset we want to test a model to be compliant. In the `onnx.json` file, all the operators are split in multiple opset, where each of them corresponds to the list of operators implemented in it. This should be way much easier to maintain and more flexible to use.

In addition to this I have added slow test that runs a complete pipeline of "keras model -> ONNX model -> optimized ONNX model -> quantized ONNX model".

As proposed by @mfuntowicz I switched the min required opset version from 12 to 10 for BERT, GPT2 and BART.

> Do you have an idea of how long the slow tests take ?

Depending of the model between 1 and 5min.

> According to the information gathered, would it be possible (in a next PR) to have a doc referencing the opset compliancy/onnx support for each model?

Do you mean to have an entire page about ONNX? Or just to add a paragraph in the doc of every model about it?

I think it is also important to mention that the model `TFGPT2ForSequenceClassification` cannot be converted into ONNX for now. The reason is because of the `tf.map_fn` function, that internally creates a `tf.while` with an iterator of type `tf.variant` which is not allowed in ONNX.

LGTM on my side!

@LysandreJik I have fixed the issue with `TFGPT2ForSequenceClassification`, so now it is compliant with ONNX.

@mfuntowicz I should have addressed your comments, please double check ^^

LGTM 👍🏻 

@LysandreJik Feel free to merge if the recent changes look ok for you!

@LysandreJik Yes, this is exactly that :) I plan to apply this update to the other causal models one by one 😉