Model inputs and outputs are ``None`` when converting fine-tuned gpt2 to Tensorflow?
Author: farazk86
Pull request: no
Labels: name: wontfix

Hi,

I've fine tuned a distilgpt2 model using my own text using ``run_language_modeling.py`` and its working fine after training and ``run_generation.py`` script produces the expected results.

Now I want to convert this to a Tensorflow Lite model and did so by using the following 

```python
from transformers import *

CHECKPOINT_PATH = '/content/drive/My Drive/gpt2_finetuned_models/checkpoint-2500'

model = GPT2LMHeadModel.from_pretrained("distilgpt2")
model.save_pretrained(CHECKPOINT_PATH)
model = TFGPT2LMHeadModel.from_pretrained(CHECKPOINT_PATH, from_pt=True) 
```

But I dont think I'm doing this right as after conversion, when I write

```python
print(model.inputs)
print(model.outputs)
```

I get 

```
None
None
```

But I still went ahead with the TFLite conversion using :

```python
import tensorflow as tf

input_spec = tf.TensorSpec([1, 64], tf.int32)
model._set_inputs(input_spec, training=False)

converter = tf.lite.TFLiteConverter.from_keras_model(model)

# FP16 quantization:
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_types = [tf.float16]

tflite_model = converter.convert()

open("/content/gpt2-fp16.tflite", "wb").write(tflite_model)
```

But does not work and when using the generated ``tflite`` model I get the error:

> tensorflow/lite/kernels/kernel_util.cc:249 d1 == d2 || d1 == 1 || d2 == 1 was not true.

Which I'm sure has something to to with my model not converting properly and getting ``None`` for input/output.


Does anyone have any idea how to fix this?

Thanks

Comments:
Hey @farazk86,

thanks for your issue! I've sadly never worked with tflite so not really sure how to best help you here. Maybe @jplu (or @LysandreJik) ? 

> Hey @farazk86,
> 
> thanks for your issue! I've sadly never worked with tflite so not really sure how to best help you here. Maybe @jplu (or @LysandreJik) ?

Thanks @patrickvonplaten but the ``print(model.inputs)`` line is before the tflite conversion. For starters, I wanted to convert the fine-tuned distilgpt2 from pytorch to tensorflow and then to tensorflow lite.


Hello @farazk86 

It is normal that the inputs/outputs are not set when using from_pretrained because they are not explicitely given when the model is built. You have to create yourself a model by setting them.

> Hello @farazk86
> 
> It is normal that the inputs/outputs are not set when using from_pretrained because they are not explicitely given when the model is built. You have to create yourself a model by setting them.

Thanks for your reply.

Is my method for converting the pretrained model to tflite wrong? I followed the code and explanation mentioned here: https://towardsdatascience.com/on-device-machine-learning-text-generation-on-android-6ad940c00911

I would also like to add that when I convert the model to tflite, using the above code I get the following warnings

```
WARNING:absl:Found untraced functions such as wte_layer_call_fn, wte_layer_call_and_return_conditional_losses, wpe_layer_call_fn, wpe_layer_call_and_return_conditional_losses, dropout_layer_call_fn while saving (showing 5 of 380). These functions will not be directly callable after loading.
WARNING:absl:Found untraced functions such as wte_layer_call_fn, wte_layer_call_and_return_conditional_losses, wpe_layer_call_fn, wpe_layer_call_and_return_conditional_losses, dropout_layer_call_fn while saving (showing 5 of 380). These functions will not be directly callable after loading.
```
 does this help identify the issue?

These warnings are not directly related to your issue, you can safely ignore them for now and the tutorial you linked is wrong for the TFLite creation part.

Unfortunately, the current state of the TF models in transformers are not fully compliant with TFLite so, I suggest to do not push to far the conversion. It is in our plans to have a better compliancy, but we don't know when yet.

You use the following piece of code to create your TFLite model:
```python
from transformers import TFGPT2LMHeadModel
import tensorflow as tf

base_model = TFGPT2LMHeadModel.from_pretrained("gpt2")
input_ids = tf.keras.layers.Input((128, ), batch_size=1, dtype=tf.int32, name='input_ids')
attention_mask = tf.keras.layers.Input((128, ), batch_size=1, dtype=tf.int32, name='attention_mask')
inputs = {"input_ids": input_ids, "attention_mask": attention_mask}
output = base_model(inputs)
model = tf.keras.models.Model(inputs=inputs, outputs=output)
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.experimental_new_converter = True
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.inference_input_type = tf.float32
converter.target_spec.supported_ops = [tf.lite.OpsSet.SELECT_TF_OPS]
tflite_quant_model = converter.convert()

with open("model.tflite", "wb") as f:
        f.write(tflite_quant_model)
```

With this piece of code you should be able to convert your model into a TFLite one. Note also that the current TF models are not compliant with float16 so you have to keep with float32.

Thank you, but with the code above I am getting the following error: 

```
Regular TensorFlow ops are not supported by this interpreter. Make sure you apply/link the Flex delegate before inference. 
Node number 0 (FlexIdentity) failed to prepare.
```

I even tried with ``tf-nightly`` as a google search of this error suggested that the nightly has flex delegate support. But still got the above error.

The android tflite interpreter I am using works fine with all the models presented here: https://github.com/huggingface/tflite-android-transformers/tree/master/gpt2#change-the-model . They are even quantized.

Would it be possible for me to train the models using a previous version of transformers? Which version was used at the time of writing the article and for providing the above tflite models?  Would it help to change to that version?

P.S. I would like to add that I am currently using the April 21st 2020 git version: ``!git checkout b1ff0b2ae7d368b7db3a8a8472a29cc195d278d8`` as I needed the ``line-by-line`` parameter during training.

Thank you


To make this work you have to use the current release of TF and Transformers, not below.

> To make this work you have to use the current release of TF and Transformers, not below.

hmm, that would make sense why your code did not work. The reason I was using the April 21st version is because I needed the ``line-by-line`` parameter during fine tuning. Does the current version and ``run_clm.py`` have ``line-by-line`` support?

I don't think the new `run_clm.py` still supports `line-by-line`. Better to open a new issue to discuss of this.

> To make this work you have to use the current release of TF and Transformers, not below.

So I am now using the current version of transformers and tf and fine-tuned a model using ``run_clm.py`` and used your above code to convert that model to tflite but still got the same error:

```
Regular TensorFlow ops are not supported by this interpreter. Make sure you apply/link the Flex delegate before inference. 
Node number 0 (FlexIdentity) failed to prepare.
```

When converting this model, I got a lot of messages in console, 

```
Tensorflow version: 2.4.0
WARNING:tensorflow:AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f2df19a0660>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: <cyfunction Socket.send at 0x7f2e091e6e58> is not a module, class, method, function, traceback, frame, or code object
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
WARNING: AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f2df19a0660>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: <cyfunction Socket.send at 0x7f2e091e6e58> is not a module, class, method, function, traceback, frame, or code object
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
WARNING:tensorflow:AutoGraph could not transform <function wrap at 0x7f2e06b7a8c8> and will run it as-is.
Cause: while/else statement not yet supported
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
WARNING: AutoGraph could not transform <function wrap at 0x7f2e06b7a8c8> and will run it as-is.
Cause: while/else statement not yet supported
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
WARNING:absl:Found untraced functions such as wte_layer_call_and_return_conditional_losses, wte_layer_call_fn, wpe_layer_call_and_return_conditional_losses, wpe_layer_call_fn, dropout_layer_call_and_return_conditional_losses while saving (showing 5 of 385). These functions will not be directly callable after loading.
WARNING:absl:Found untraced functions such as wte_layer_call_and_return_conditional_losses, wte_layer_call_fn, wpe_layer_call_and_return_conditional_losses, wpe_layer_call_fn, dropout_layer_call_and_return_conditional_losses while saving (showing 5 of 385). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: /tmp/tmpus6vmwet/assets
INFO:tensorflow:Assets written to: /tmp/tmpus6vmwet/assets
The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).
The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.
INFO:absl:Using new converter: If you encounter a problem please file a bug. You can opt-out by setting experimental_new_converter=False
```
comparatively, a model fine tuned using the old v2.4 version of transformers did not generate any such messages:

Those are just warnings and expected messages, this is ok. Can you try with usual `gpt2` models? If it works, the issue is certainly coming from your model. Otherwise, we will check deeper what is going wrong. Nevertheless, TFLite compliancy is not our priority for now, so if we have to fix something it will certainly be postponed to a bit later in the coming months.

Hi, I'm getting the same error as above when using the default pretrained gpt2.

Humm with Transformers from source and TF 2.4 I get no errors. What is your env?

> Humm with Transformers from source and TF 2.4 I get no errors. What is your env?

I'm trying to run on android using a flutter tflite interpreter. and yes, I was also considering that maybe the fault is not in the converted model but the interpreter. 

To confirm this I wanted to use the python interpreter, if I can get an output here then that means that the converted models are fine and its an issue with the flutter interpreter.

But when using the python tflite interpreter, everytime I invoke the model tflite generated model using your above provided code, my colab runtime crashes. 

I'm using current version of transformers and TF2.4

The same interpreter works for the tflite models provided here and produces output in the expected shape : https://github.com/huggingface/tflite-android-transformers/tree/master/gpt2#change-the-model

Below is the code I am using for tflite interpreter

```python
from transformers import *
import tensorflow as tf
import numpy as np


tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

# start with their provided model, the -O will change downloaded file name to model.tflite

# !wget -O model.tflite https://s3.amazonaws.com/models.huggingface.co/bert/distilgpt2-64.tflite

# Encode random strings
sentance = ("""You are John, a punk living in the futuristic city of Zail. You have a small xore blaster hidden in you jacket and a holoband on your wrist. You are John, a punk living in the futuristic city of Zail. You have a small xore blaster hidden in you jacket and a holoband on your wrist. You are John, a punk living in the futuristic city of Zail. You have a small xore blaster hidden in you jacket and a holoband on your wrist. You are John, a punk living in the futuristic city of Zail. You have a small xore blaster hidden in you jacket and a holoband on your wrist. You are John, a punk living in the futuristic city of Zail. You have a small xore blaster hidden in you jacket and a holoband on your wrist. You are John, a punk living in the futuristic city of Zail. You have a small xore blaster hidden in you jacket and a holoband on your wrist.""")
review_token = tokenizer.encode(sentance)
print(len(review_token))
review_token = np.array(review_token, dtype=np.int32)
review_token = review_token[:128]
review_token = np.expand_dims(review_token, axis=0)           # unsqueeze to add the batch dimension
print(sentance)
print(review_token)

tflite_interpreter = tf.lite.Interpreter(model_path='/content/model.tflite')
tflite_interpreter.allocate_tensors()

input_details = tflite_interpreter.get_input_details()
output_details = tflite_interpreter.get_output_details()

print("== Input details ==")
print("name:", input_details[0]['name'])
print("shape:", input_details[0]['shape'])
print("type:", input_details[0]['dtype'])

print("\n== Output details ==")
print("name:", output_details[0]['name'])
print("shape:", output_details[0]['shape'])
print("type:", output_details[0]['dtype'])

tflite_interpreter.set_tensor(input_details[0]['index'], review_token)

tflite_interpreter.invoke()

tflite_model_predictions = tflite_interpreter.get_tensor(output_details[0]['index'])
print("Prediction results shape:", tflite_model_predictions.shape)
```


Ok, thanks for sharing, I will check this once I can dedicate some time.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.