Bert and Roberta models cannot be converted to TFLite
Author: neonbjb
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert / Roberta

Language I am using the model on (English, Chinese ...): N/A

The problem arises when using:
* [ ] the official example scripts: (give details below)
Sort of. Using the tflite conversion script provided here:
https://github.com/huggingface/tflite-android-transformers/blob/master/models_generation/distilbert.py

The tasks I am working on is: Converting models to tflite format

## To reproduce

Steps to reproduce the behavior:
I first tried the example script provided above to convert a distilbert model to tflite, and it worked fine. The GPT conversion also works great.

Next, I modified the above script to the following:
```
import tensorflow as tf
from transformers import TFRobertaModel

model = TFRobertaModel.from_pretrained('roberta-base')

input_spec = [tf.TensorSpec([1, 128], tf.int32), tf.TensorSpec([1, 128], tf.int32)]
model._set_inputs(input_spec, training=False)

print(model.inputs)
print(model.outputs)

converter = tf.lite.TFLiteConverter.from_keras_model(model)

# For conversion with hybrid quantization:
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]
converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
converter.experimental_new_converter = True

tflite_model = converter.convert()
```
Note that the above can be replaced with "TFBertModel" and "bert-base-cased" with 3 input tensors with the same result below.

## Expected behavior

No errors, creates tflite model.

## Actual behavior
Error for both BERT and Roberta:
```
[<tf.Tensor 'input_1_11:0' shape=(None, 128) dtype=int32>, <tf.Tensor 'input_2_9:0' shape=(None, 128) dtype=int32>]
[<tf.Tensor 'tf_roberta_model_1/Identity:0' shape=(None, 128, 768) dtype=float32>, <tf.Tensor 'tf_roberta_model_1/Identity_1:0' shape=(None, 768) dtype=float32>]
---------------------------------------------------------------------------
ConverterError                            Traceback (most recent call last)
<ipython-input-15-1f63532e8b87> in <module>
     26 converter.experimental_new_converter = True
     27 
---> 28 tflite_model = converter.convert()
     29 
     30 open("distilbert-squad-384.tflite", "wb").write(tflite_model)

c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\tensorflow_core\lite\python\lite.py in convert(self)
    444         input_tensors=input_tensors,
    445         output_tensors=output_tensors,
--> 446         **converter_kwargs)
    447 
    448     if self._is_calibration_quantize():

c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\tensorflow_core\lite\python\convert.py in toco_convert_impl(input_data, input_tensors, output_tensors, enable_mlir_converter, *args, **kwargs)
    447       input_data.SerializeToString(),
    448       debug_info_str=debug_info_str,
--> 449       enable_mlir_converter=enable_mlir_converter)
    450   return data
    451 

c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\tensorflow_core\lite\python\convert.py in toco_convert_protos(model_flags_str, toco_flags_str, input_data_str, debug_info_str, enable_mlir_converter)
    198       stdout = _try_convert_to_unicode(stdout)
    199       stderr = _try_convert_to_unicode(stderr)
--> 200       raise ConverterError("See console for info.\n%s\n%s\n" % (stdout, stderr))
    201   finally:
    202     # Must manually cleanup files.

ConverterError: See console for info.
2020-02-03 14:16:20.869205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
2020-02-03 14:16:25.853657: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Cumsum
2020-02-03 14:16:25.854123: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.854715: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.855259: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.855869: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.856324: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.856863: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.857394: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.857914: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.858543: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.859107: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.859552: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:25.860084: I tensorflow/lite/toco/import_tensorflow.cc:659] Converting unsupported operation: Erf
2020-02-03 14:16:26.060782: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] Before Removing unused ops: 1517 operators, 2651 arrays (0 quantized)
2020-02-03 14:16:26.149298: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 1517 operators, 2651 arrays (0 quantized)
2020-02-03 14:16:26.149831: F tensorflow/lite/toco/graph_transformations/resolve_strided_slice_attributes.cc:95] Check failed: start_indices_size <= num_input_axes (4 vs. 2)StridedSlice op requires no more than 2 start indices
Fatal Python error: Aborted

Current thread 0x000013c4 (most recent call first):
  File "c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\tensorflow_core\lite\toco\python\toco_from_protos.py", line 52 in execute
  File "c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\absl\app.py", line 250 in _run_main
  File "c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\absl\app.py", line 299 in run
  File "c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\tensorflow_core\python\platform\app.py", line 40 in run
  File "c:\drive\projects\ml-notebooks\pycharm-venv\lib\site-packages\tensorflow_core\lite\toco\python\toco_from_protos.py", line 89 in main
  File "C:\drive\projects\ml-notebooks\pycharm-venv\Scripts\toco_from_protos.exe\__main__.py", line 9 in <module>
  File "C:\python\lib\runpy.py", line 85 in _run_code
  File "C:\python\lib\runpy.py", line 193 in _run_module_as_main
```

## Environment info
     
- `transformers` version:
- Platform: Windows 10
- Python version: 3.6.8
- PyTorch version (GPU?): N/A
- Tensorflow version (GPU?): 2.0.0-dev20191002 (gpu=yes)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: N/A


Comments:
cc @Pierrci 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.