Can i covert a MarianMT Model to tensorflow lite model?
Author: iDustPan
Pull request: no


I'd liked to run MarianMT model on tensorflow lite,  how to covert the en-zh model to a .pb model or a .tflight model?


Comments:
i have tried to covert the .h5 to .pb, but there is something weird.

My tensorflow version is 2.6.0.

this is my dir info:

(venv) ➜  Documents tree opus-mt-en-zh

opus-mt-en-zh
├── README.md
├── config.json
├── flax_model.msgpack
├── metadata.json
├── pytorch_model.bin
├── rust_model.ot
├── source.spm
├── target.spm
├── tf_model.h5
├── tokenizer_config.json
└── vocab.json

when i load the model with tensorflow,  it crashed:


```
import tensorflow as tf
model = tf.keras.models.load_model("./opus-mt-en-zh",compile=False)
```


Traceback (most recent call last):
  File "translate.py", line 27, in <module>
    model = tf.keras.models.load_model("./opus-mt-en-zh",compile=False)
  File "/Users/xuanyue/venv/lib/python3.8/site-packages/keras/saving/save.py", line 205, in load_model
    return saved_model_load.load(filepath, compile, options)
  File "/Users/xuanyue/venv/lib/python3.8/site-packages/keras/saving/saved_model/load.py", line 108, in load
    meta_graph_def = tf.__internal__.saved_model.parse_saved_model(path).meta_graphs[0]
  File "/Users/xuanyue/venv/lib/python3.8/site-packages/tensorflow/python/saved_model/loader_impl.py", line 118, in parse_saved_model
    raise IOError(
OSError: SavedModel file does not exist at: ./opus-mt-en-zh/{saved_model.pbtxt|saved_model.pb}




i also tried this:

```
import tensorflow as tf
from transformers import MarianMTModel, MarianTokenizer
print("loading model...")
model_name = 'Helsinki-NLP/opus-mt-en-zh'
tokenizer = MarianTokenizer.from_pretrained(model_name)
model = MarianMTModel.from_pretrained(model_name)
print("finished loading model")

converter = tf.lite.TFLiteConverter.from_keras_model(model);
tflite_model = converter.convert();
```

but it died with:

(venv) ➜  Documents python translate.py
loading model...
finished loading model
Traceback (most recent call last):
  File "translate.py", line 10, in <module>
    tflite_model = converter.convert();
  File "/Users/xuanyue/venv/lib/python3.8/site-packages/tensorflow/lite/python/lite.py", line 729, in wrapper
    return self._convert_and_export_metrics(convert_func, *args, **kwargs)
  File "/Users/xuanyue/venv/lib/python3.8/site-packages/tensorflow/lite/python/lite.py", line 715, in _convert_and_export_metrics
    result = convert_func(self, *args, **kwargs)
  File "/Users/xuanyue/venv/lib/python3.8/site-packages/tensorflow/lite/python/lite.py", line 1123, in convert
    self._freeze_keras_model())
  File "/Users/xuanyue/venv/lib/python3.8/site-packages/tensorflow/lite/python/convert_phase.py", line 218, in wrapper
    raise error from None  # Re-throws the exception.
  File "/Users/xuanyue/venv/lib/python3.8/site-packages/tensorflow/lite/python/convert_phase.py", line 208, in wrapper
    return func(*args, **kwargs)
  File "/Users/xuanyue/venv/lib/python3.8/site-packages/tensorflow/lite/python/lite.py", line 1066, in _freeze_keras_model
    if not isinstance(self._keras_model.call, _def_function.Function):
  File "/Users/xuanyue/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'MarianMTModel' object has no attribute 'call'


Any suggestions?


Maybe @Rocketknight1 has some experience with TF Lite

Hi! In general, we don't support conversions to TFLite - it has a very limited set of operations and that makes it hard to automatically convert a lot of models.

You can try this yourself if you want, but we probably can't offer detailed assistance. The bug you encountered, however, is most likely caused by the fact that you used `MarianMTModel`, which is the Pytorch model instead of `TFMarianMTModel`. If you try converting the `TFMarianMTModel` it might work, but I suspect there will still be other problems!

Sorry to tag @Rocketknight1 in this again - but I had another question on this topic: I was able to get a rudimentary yet successful quantized conversion of the TFMarianMTModel ( Sample script provided [here](https://colab.research.google.com/drive/1Vsinmju5FsbiEtfD9BqBHWeKxM3LES-U?usp=sharing)) for tflite. I was even able to ( not mentioned in this script ) generate nearly identical encoder outputs with the tflite interpreter on edge - but I am uncertain of how to invoke the decoder for the decoding process. 

I am not sure if there is any way I could even port this model as a whole ( since the decoder will have to be invoked repeatedly for the decoding process on edge ).
More specifically -  Can you please give any tips on how I can go about doing any of the following ?

a. Is there any way I could use [Tensorflow Signatures](https://www.tensorflow.org/lite/guide/signatures#:~:text=Signatures%20can%20be%20specified%20when%20building%20a%20SavedModel,TensorFlow%20Lite%20model%20to%20support%20multiple%20entry%20points.) to separately invoke the interpreter for the encoder and decoder ? 

b. Is there any way I can port the TFMarianMTModels encoder and decoder separately ? ( so that I may invoke them as two different graphs )

Thanks in advance for any help. 

Hello @harshitadd,

I'll try to help. I see you converted like this:
```
print('Converting Model to tflite')
converter = tf.lite.TFLiteConverter.from_saved_model('config_model')

print('Optimizing for Size Reduction')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
```

Can you try the code snippets below? Call the encoder and decoder inferences under separate functions decorated under @tf.function() when you're defining the Model (as encode and decode), if you want to do it with signatures. Is your question more about how to separate an encoder decoder model and put them separately? 

```
model = Model()
tf.saved_model.save(
    model, SAVED_MODEL_PATH,
    signatures={
      'encode': model.encode.get_concrete_function(),
      'decode': model.decode.get_concrete_function()
    })
converter = tf.lite.TFLiteConverter.from_saved_model(SAVED_MODEL_PATH)
converter.target_spec.supported_ops = [
    tf.lite.OpsSet.TFLITE_BUILTINS,  # TensorFlow Lite ops.
    tf.lite.OpsSet.SELECT_TF_OPS  # TensorFlow ops.
]
tflite_model = converter.convert()
```

> Sorry to tag @Rocketknight1 in this again - but I had another question on this topic: I was able to get a rudimentary yet successful quantized conversion of the TFMarianMTModel ( Sample script provided [here](https://colab.research.google.com/drive/1Vsinmju5FsbiEtfD9BqBHWeKxM3LES-U?usp=sharing)) for tflite. I was even able to ( not mentioned in this script ) generate nearly identical encoder outputs with the tflite interpreter on edge - but I am uncertain of how to invoke the decoder for the decoding process.
> 
> I am not sure if there is any way I could even port this model as a whole ( since the decoder will have to be invoked repeatedly for the decoding process on edge ).
> More specifically - Can you please give any tips on how I can go about doing any of the following ?
> 
> a. Is there any way I could use [Tensorflow Signatures](https://www.tensorflow.org/lite/guide/signatures#:~:text=Signatures%20can%20be%20specified%20when%20building%20a%20SavedModel,TensorFlow%20Lite%20model%20to%20support%20multiple%20entry%20points.) to separately invoke the interpreter for the encoder and decoder ?
> 
> b. Is there any way I can port the TFMarianMTModels encoder and decoder separately ? ( so that I may invoke them as two different graphs )
> 
> Thanks in advance for any help.



Thanks for the help @merveenoyan; I was able to port the encoder/decoder separately; Using signatures just seemed like a more elegant solution; Thanks for your help!