convert tf ckpt to pytorch_model.bin, load back model(TFBertModel), will loss params
Author: zwqjoy
Pull request: no
Labels: name: wontfix

```
import os
pretrained_path = 'Models/chinese_L-12_H-768_A-12'
config_path = os.path.join(pretrained_path, 'bert_config.json')
checkpoint_path = os.path.join(pretrained_path, 'bert_model.ckpt.index')

config = BertConfig.from_pretrained(config_path)
model = BertForPreTraining.from_pretrained(checkpoint_path, from_tf=True, config=config)
model.save_pretrained('Models/chinese')
```
INFO:transformers.configuration_utils:Configuration saved in Models/chinese/config.json
INFO:transformers.modeling_utils:Model weights saved in Models/chinese/pytorch_model.bin



The load the save model:

```
# 加载模型
config = BertConfig.from_json_file("Models/chinese/config.json")
tfmodel = TFBertModel.from_pretrained('Models/chinese/',from_pt=True, config=config)
```
INFO:transformers.modeling_tf_utils:loading weights file Models/chinese/pytorch_model.bin
INFO:transformers.modeling_tf_pytorch_utils:Loading PyTorch weights from /home/work/Bert/Models/chinese/pytorch_model.bin
INFO:transformers.modeling_tf_pytorch_utils:PyTorch checkpoint contains 119,108,746 parameters
INFO:transformers.modeling_tf_pytorch_utils:Loaded 102,267,648 parameters in the TF 2.0 model.
INFO:transformers.modeling_tf_pytorch_utils:Weights or buffers not loaded from PyTorch model: {'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias'}



PyTorch checkpoint contains 119,108,746 parameters  To  Loaded 102,267,648 parameters in the TF 2.0 model.




Comments:
Try changing it from TFBERT model to BertModel. Since you already converted it to a pytorch checkpoint. 

@zanderkent 
If have any tool can convert tf ckpt to tf_model.h5. So I can use TFBert Class to load.

Because I with use tf2  model.fit to train , and use the tf2 strategy to train dist.

If I am not mistaken transformers would allow you to use a tf CKPT using TFbert. In your first part of the code you converted it to pytorch. When you initially load the model save it as tensorflow model. 

Anyone else have any other ideas?

@zanderkent 
Can you give a demo for that? I still cannot know how to do
Thanks


If I use:
  model = TFBertForPreTraining.from_pretrained(checkpoint_path, config=config)

will get the log info
```
INFO:transformers.modeling_tf_utils:loading weights file Models/chinese_L-12_H-768_A-12/bert_model.ckpt.index
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/util.py:1249: NameBasedSaverStatus.__init__ (from tensorflow.python.training.tracking.util) is deprecated and will be removed in a future version.
Instructions for updating:
Restoring a name-based tf.train.Saver checkpoint using the object-based restore API. This mode uses global names to match variables, and so is somewhat fragile. It also adds new restore ops to the graph each time it is called when graph building. Prefer re-encoding training checkpoints in the object-based format: run save() on the object-based saver (the same one this message is coming from) and use that checkpoint in the future.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/util.py:1249: NameBasedSaverStatus.__init__ (from tensorflow.python.training.tracking.util) is deprecated and will be removed in a future version.
Instructions for updating:
Restoring a name-based tf.train.Saver checkpoint using the object-based restore API. This mode uses global names to match variables, and so is somewhat fragile. It also adds new restore ops to the graph each time it is called when graph building. Prefer re-encoding training checkpoints in the object-based format: run save() on the object-based saver (the same one this message is coming from) and use that checkpoint in the future.
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-6-3e9de93c3943> in <module>
----> 1 model = TFBertForPreTraining.from_pretrained(checkpoint_path, config=config)

/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    315         # see https://github.com/tensorflow/tensorflow/blob/00fad90125b18b80fe054de1055770cfb8fe4ba3/tensorflow/python/keras/engine/network.py#L1339-L1357
    316         try:
--> 317             model.load_weights(resolved_archive_file, by_name=True)
    318         except OSError:
    319             raise OSError("Unable to load weights from h5 file. "

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py in load_weights(self, filepath, by_name)
    179         raise ValueError('Load weights is not yet supported with TPUStrategy '
    180                          'with steps_per_run greater than 1.')
--> 181     return super(Model, self).load_weights(filepath, by_name)
    182 
    183   @trackable.no_automatic_dependency_tracking

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py in load_weights(self, filepath, by_name)
   1150       if by_name:
   1151         raise NotImplementedError(
-> 1152             'Weights may only be loaded based on topology into Models when '
   1153             'loading TensorFlow-formatted weights (got by_name=True to '
   1154             'load_weights).')

NotImplementedError: Weights may only be loaded based on topology into Models when loading TensorFlow-formatted weights (got by_name=True to load_weights).
```

Sorry, someone else will have to help you with that. I am only vaguely familiar with this library. 

I get the same error `NotImplementedError: Weights may only be loaded based on topology into Models when loading TensorFlow-formatted weights (got by_name=True to load_weights).` and have no idea what it means!

I'm trying to:
```
model_dir = 'my/dir/to/bert/model'
config = BertConfig.from_json_file(model_dir + '/bert_config.json')
config.num_labels = 14
model = TFBertForSequenceClassification.from_pretrained(model_dir + '/bert_model.ckpt.index', config = config)
```

But this gives me the following error:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mydir/share/virtualenvs/pdBERTlm-giwpujkO/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 317, in from_pretrained
    model.load_weights(resolved_archive_file, by_name=True)
  File "/mydir/share/virtualenvs/pdBERTlm-giwpujkO/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 234, in load_weights
    return super(Model, self).load_weights(filepath, by_name, skip_mismatch)
  File "/mydir/share/virtualenvs/pdBERTlm-giwpujkO/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 1196, in load_weights
    'Weights may only be loaded based on topology into Models when '
NotImplementedError: Weights may only be loaded based on topology into Models when loading TensorFlow-formatted weights (got by_name=True to load_weights).
```

If I try to run `BertForSequenceClassification` (with from_tf = True), this error shows:
```
>>> model = BertForSequenceClassification.from_pretrained(model_dir + '/bert_model.ckpt.index', from_tf = True, config = config)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mydir/virtualenvs/pdBERTlm-giwpujkO/lib/python3.7/site-packages/transformers/modeling_utils.py", line 427, in from_pretrained
    model = cls.load_tf_weights(model, config, resolved_archive_file[:-6])  # Remove the '.index'
  File "/mydir/virtualenvs/pdBERTlm-giwpujkO/lib/python3.7/site-packages/transformers/modeling_bert.py", line 99, in load_tf_weights_in_bert
    pointer = getattr(pointer, 'bias')
  File "/mydir/virtualenvs/pdBERTlm-giwpujkO/lib/python3.7/site-packages/torch/nn/modules/module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'BertForSequenceClassification' object has no attribute 'bias'
```


However if I run the `transformers-cli convert` and then load the pytorch model, it all works fine...
```
export BERT_BASE_DIR=my/dir/to/bert/model

transformers-cli convert --model_type bert \
  --tf_checkpoint $BERT_BASE_DIR/bert_model.ckpt \
  --config $BERT_BASE_DIR/bert_config.json \
  --pytorch_dump_output $BERT_BASE_DIR/pytorch_model.bin

```


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Does someone know how to fix it?

` NotImplementedError: Weights may only be loaded based on topology into Models when loading TensorFlow-formatted weights (got by_name=True to load_weights). `