How to convert the bert model in tf1 ckpt format which converted by the pytorch version to tf1/tf2 SavedModel?
Author: catqaq
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.2.1
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-debian-buster-sid
- Python version: 3.6.13
- PyTorch version (GPU?): 1.10.0+cu102 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: <yes>
- Using distributed or parallel set-up in script?: <no>
```


### Who can help?

@LysandreJik

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1.follow the official demo convert_bert_pytorch_checkpoint_to_original_tf.py to **convert pytorch bert model to tensorflow1 ckpt model**
2.then try to **convert bert.ckpt to SavedModel** format,:
`
def convert_tf1_ckpt_to_saved_model(ckpt_dir):
    # ckpt to SavedModel
    saver = tf.compat.v1.train.import_meta_graph(ckpt_dir + "/bert.ckpt.meta")
    input_ids = tf.compat.v1.placeholder(tf.int32, [None, 64], name='input_ids')
    attention_mask = tf.compat.v1.placeholder(tf.int32, [None, 64], name='attention_mask')
    inputs = {"input_ids": input_ids, "attention_mask": attention_mask}
    session = tf.compat.v1.Session()
    saver.restore(session, ckpt_dir + "/bert.ckpt")
    output = tf.compat.v1.placeholder(tf.float32, [None, 64], name='output')
    # output = session.run(output, feed_dict=inputs)
    tf.compat.v1.saved_model.simple_save(session, ckpt_dir + "/savedmodel",
                                         inputs=inputs,
                                         outputs={"output": output})
`
The code in step 2 is probably wrong because **the inputs and outputs are not specified correctly**. I followed this script [https://stackoverflow.com/questions/44251328/tensorflow-print-all-placeholder-variable-names-from-meta-graph](url) to **get inputs and outputs of the meta graph**. But, it did not work.

3. load the savedmodel format bert, i got an error:
`tf_model = tf.saved_model.load("./savedmodel")`
error info:
tensorflow.python.ops.op_selector.UnliftableError: A SavedModel signature needs an input for each placeholder the signature's outputs use. An output for signature 'serving_default' depends on a placeholder which is not an input (i.e. the placeholder is not fed a value).

Unable to lift tensor <tf.Tensor 'output:0' shape=(None, 64) dtype=float32> because it depends transitively on placeholder <tf.Operation 'output' type=Placeholder> via at least one path, e.g.: output (Placeholder)

### Expected behavior

```shell
The bert model in tf1 ckpt format converted by the pytorch version can be correctly converted to tf1/tf2 SavedModel format, and the SavedModel version model can be loaded and deployed correctly.
```


Comments:
cc @Rocketknight1 @gante 

Hi @catqaq 👋  Thank you for posting a detailed description. Before I dig into the technical problem, I'd like to ask you a more general question. 

Is it your final goal to obtain a TF `SavedModel` containing `BERT`? If you don't have requirements regarding how it's done, I believe you can also achieve it from our `from_pretrained` model method. See the following example

```python
# save the model with native TF functions and a signature ready for the maximum length
import tensorflow as tf
from transformers import BertTokenizer, TFBertModel

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = TFBertModel.from_pretrained('bert-base-uncased')

text = "This is a test input."
encoded_input = tokenizer(text, return_tensors='tf', padding="max_length")
model._saved_model_inputs_spec = None
model._set_save_spec(dict(encoded_input))  # <-- resizes the expected input to max model length
tf.saved_model.save(model, '/tmp/bert')

# load the saved model with native TF functions
model_tf = tf.saved_model.load("/tmp/bert")

# confirm that they both work and have the same output
hf_model_output = model(encoded_input)
tf_model_output = model_tf(encoded_input)
assert tf.experimental.numpy.allclose(
    hf_model_output["last_hidden_state"],
    tf_model_output["last_hidden_state"],
    atol=1e-5
)
```

Let me know if this solves your problem :)

@gante Hi, thanks for your reply. **I've trained a series of models based on bert by pytorch**. So, **my final goal is to convert the pretrained pytorch version bert model to tensorflow savedmodel for deployment**. So, i tried to follow your demo(convert_bert_pytorch_checkpoint_to_original_tf.py) to convert my pytorch model to tensorflow1 ckpt model. But ckpt model is not convenient to deploy. 

@catqaq I see. That script is a bit outdated, and TF 1 is indeed inconvenient. The following script should work for your case:

```python
# save the model with native TF functions and a signature ready for the maximum length
import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModel

model_path = "/path/to/your/pytorch/model/saved/locally"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = TFAutoModel.from_pretrained(model_path, from_pt=True)

text = "This is a test input."
encoded_input = tokenizer(text, return_tensors='tf', padding="max_length")
model._saved_model_inputs_spec = None
model._set_save_spec(dict(encoded_input))  # <-- resizes the expected input to max model length
tf.saved_model.save(model, '/tmp/bert')

# load the saved model with native TF functions
model_tf = tf.saved_model.load("/tmp/bert")

# confirm that they both work and have the same output
hf_model_output = model(encoded_input)
tf_model_output = model_tf(encoded_input)
assert tf.experimental.numpy.allclose(
    hf_model_output["last_hidden_state"],
    tf_model_output["last_hidden_state"],
    atol=1e-5
)
```

Notice the `model_path` variable, which should be your local pytorch model (or the path to the pytorch model on the hub) and the `from_pt` argument in `from_pretrained`. This script should load your PT model, convert it to TF 2, and store it in the saved model format.

@gante Thanks! It works well for original pytorch bert model.  But i also have some **customized models based on bert with task related layers**. I found that from_pt related to load_pytorch_checkpoint_in_tf2_model function. So i followed this function to directly convert customized pytorch bert style model. In this case, do I have to rewrite the model (especially the task related parts) with tf2 and then load the pytorch model weights?

@catqaq if you have task-specific layers that are not defined in the `transformers` repository, I'm afraid you'll also have to create a custom TF2 model architecture, as you mentioned above -- tools like our `from_pt` work because we have the architecture definition in both frameworks, in such a way that we can cross-load weights :) 

Since the issue you're having is not a bug in the repository, but rather a challenging use-case, I'd like to suggest using [our forum](https://discuss.huggingface.co/).

@gante Yep, i will to create a custom TF2 model architecture as close to pytorch version as possible. Thanks for your advice.

Awesome. I'm closing this issue, but feel free to reopen if you run into new `transformers`-related problems 👍 

@gante Hi, I rewrite my model in tf2 and convert pt_model to tf_model sucessfully. But the difference between the pt_model and tf_model is huge. At first I thought there was something wrong with my tf_model implementation, but I **double-checked the mapping of every variable and made sure there was nothing wrong**. So, i tried to check the original bert model and **the difference between pt_bert and tf_bert(or tf_bert_from_pt) is also huge**!
Here is the code:
`from transformers import AutoTokenizer,TFAutoModel,AutoModel
import torch
import numpy
bert_path="./models/pretrained_models/**bert-base-chinese**"
pt_bert=AutoModel.from_pretrained(bert_path)
tf_bert=TFAutoModel.from_pretrained(bert_path)
tf_bert_from_pt=TFAutoModel.from_pretrained(bert_path,from_pt=True)
def compare_tf_pt_model_consistency(pt_model, tf_model, pt_inputs=None, tf_inputs=None, tokenizer=None, texts=None,
                                    threshold=2e-2):
    pt_inputs = pt_model.dummy_inputs if pt_inputs is None else pt_inputs
    tf_inputs = tf_model.dummy_inputs if tf_inputs is None else tf_inputs
    tf_outputs = tf_model(tf_inputs, training=False)  # build the network

    pt_model.eval()  # 新增：必须是eval模式
    with torch.no_grad():
        pt_outputs = pt_model(**pt_inputs)

    np_pt = pt_outputs[0].numpy()
    np_tf = tf_outputs[0].numpy()
    diff = numpy.amax(numpy.abs(np_pt - np_tf))
    print("Max absolute difference between models outputs {}".format(diff))
    assert diff <= threshold, "Error, model absolute difference is >{}: {}".format(threshold, diff)
compare_tf_pt_model_consistency(pt_bert, tf_bert_from_pt, pt_inputs=None, tf_inputs=None)
compare_tf_pt_model_consistency(pt_bert, tf_bert, pt_inputs=None, tf_inputs=None)`

Then i got the error caused by the huge diff between pt_bert and tf_bert_from_pt/bert:
AssertionError: Error, model absolute difference is >0.02: **13.102652549743652**
If we change the model to **bert-base-uncased**, the diff will be 6.858981132507324, also huge!
AssertionError: Error, model absolute difference is >0.02: 6.858981132507324

And tf_model.summary() seems a bit strange with Output Shape=multiple:
tf_bert.summary()
Model: "tf_bert_model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
bert (TFBertMainLayer)       multiple                  102267648 
=================================================================
Total params: 102,267,648
Trainable params: 102,267,648
Non-trainable params: 0

Hi @catqaq 👋  I'm afraid I'm unable to directly help you with your custom model -- as per our "[issues guidelines](https://github.com/huggingface/transformers/blob/main/ISSUES.md#the-github-issues)", we simply don't have the capacity to support custom code :)

I can, however, tell you what I would do in your situation. TF-PT mismatches often arise in two places: 
1. weights that were not properly loaded 👉 check for warnings in the console 
2. non-trainable parameters that have different default values in PT and TF 👉 add a debugger inside your model and check where the variables start diverging

@gante Hi, the point now is that not only does my custom model have a difference between tf_model and pt_model, **the original bert model also has a huge difference between tf_model and pt_model**. As mention above, **there is a diff up to 13.102652549743652 for bert-base-chinese and 6.858981132507324 for bert-base-uncased**. Maybe it is similar to https://github.com/huggingface/transformers/issues/3386

@catqaq -- that indeed seems like a relevant bug, thanks for pointing that out! I will have a look and let you know of my findings.

@catqaq the weights and the conversion script seem fine. Have a look at the script below:

```python
import numpy as np
from transformers import AutoModel, AutoTokenizer, TFAutoModel

model_name = "bert-base-uncased"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model_pt = AutoModel.from_pretrained(model_name)
model_tf = TFAutoModel.from_pretrained(model_name)
model_tf_from_pt = TFAutoModel.from_pretrained(model_name, from_pt=True)

text = "Replace me by any text you'd like."
encoded_input_pt = tokenizer([text], return_tensors="pt")
encoded_input_tf = tokenizer([text], return_tensors="tf")

output_pt = model_pt(**encoded_input_pt)["last_hidden_state"].detach().numpy()
output_tf = model_tf(**encoded_input_tf)["last_hidden_state"].numpy()
output_tf_from_pt = model_tf_from_pt(**encoded_input_tf)["last_hidden_state"].numpy()

print("TF - PT max diff:", np.max(np.abs(output_pt - output_tf)))
print("TF - TF (loaded from PT) max diff:", np.max(np.abs(output_tf - output_tf_from_pt)))
```

The output for this script on `main` is 
```
TF - PT max diff: 9.059906e-06
TF - TF (loaded from PT) max diff: 0.0
```

Replacing `model_name` to `"bert-base-chinese"` and `text` to `"这是一个随机测试"`, the output is
```
TF - PT max diff: 8.583069e-06
TF - TF (loaded from PT) max diff: 0.0
```

@gante Hi, thanks for the experimental verification. What caused our experimental results to be completely different? Transformers version? I use 4.2.1, which version of your transformers?

Hi @catqaq -- I'm testing on the current version of transformers' code, which is the unreleased `4.19.0.dev0`. Perhaps it is an error related to an older version, but the latest release (`4.18.0`) should also behave correctly. My advice here would be to update your version of transformers, since we don't fix older versions :)

@gante Thanks! Our production environment uses an older version, and I didn't find a similar problem in the issues, so at first I thought the diff was caused by incorrect parameter loading. But **after checking all the parameters one by one, I started to suspect that it was a version problem**. **Then I started trying to use the new version of the conversion code, and the results were the same as the old version of the conversion code**. Therefore, I guess that there may be **a little problem with the old version of TFBertModel, which may be similar to the situation of TFALBERT**https://github.com/huggingface/transformers/issues/3386. 
I will continue the experiment tomorrow:
1. Experiment in the environment of the new version to determine that the diff is caused by the old version. This might be a good opportunity to upgrade our transformers!
2. Take a look at the **differences in the implementation of TFBertModel between the old and new versions**, and hope to find out **the source of the huge diff of the old version**. If you have time, any help would be greatly appreciated.

Hi@gante, It works! In transformers 4.18, max absolute difference between models (tf_custom_model_from_pt and pt_custom_model) outputs is 1.7881393432617188e-07. Then I compared TFBert implementations in 4.2.1 and 4.18. For 
TFBertModel and TFBertMainLayer, the most obvious difference is that new version removed **input_processing** function. Could this be the reason?



@catqaq probably not -- the `input_processing` got replaced by the `@unpack_inputs` decorator on top of `call()`. They have the same purpose, but the decorator results in cleaner code and in direct variable use (as opposed to `inputs[variable_name]`)

Possibly something got changed in the tool that converts PT weights into TF. I can't confirm, as that predates my existence at HuggingFace :)

@gante Yep, i didn't notice the @unpack_inputs decorator which plays a similar role as input_processing at first. But as mentioned above,  new version convertion scripts combined with old transformers leads to the same results as old convertion scripts. Anyway, let's put this question aside for now. One last question is that for my custom model based on bert, is it necessary to use @unpack_inputs decorator before `call()`?

It is not :) The decorator does things like converting deprecated arguments into the new arguments, unpacking tensor dictionaries, etc. If you pass the expected inputs with the correct type, it shouldn't be needed.

@gante Got it. thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.