Loading a TF pretrained model into BertForSequenceClassification module
Author: dennymarcels
Pull request: no

Hi, there might be something I am doing wrong, but I cannot figure out what that is, then any help would be welcome.

After downloading a TF checkpoint (containing model.index, model.data, model.meta, config.json and vocab.txt files), I used it to perform pretraining using some more text, more relevant to the task I would have ahead. Pretraining was performed using the API from BERT's official github. This generated other model.index, model.data and model.meta files. I am now trying to load them into the BertForSequenceClassification module, using the `from_pretrained` method. I figured I should include a config instance as well, so I used that config.json file that was attached to that very first one TF checkpoint I mentioned. But when passing the index file to `from_pretrained`, I get the error:

`AttributeError: 'BertForSequenceClassification' object has no attribute 'bias'`

Any help would be much appreciated.



Comments:
Hi! You can't directly load an official TensorFlow checkpoint in the PyTorch model, you first need to convert it. You can use this [script](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py) to convert it to a PyTorch checkpoint.

If you want to use our TensorFlow interface in order to do this, you would still need to use this script to convert it to our interface, and then use the `TFBertForSequenceClassification` while specifying the `from_pt` option (as the result would be a PyTorch checkpoint):

```py
model = TFBertForSequenceClassification.from_pretrained("directory", from_pt=True)
```

 I had understood that parameter `from_tf` would take care of this, did I
get it wrong? Your suggestion worked though, thank you!

Em qui., 23 de abr. de 2020 às 16:55, Lysandre Debut <
notifications@github.com> escreveu:

> Hi! You can't directly load an official TensorFlow checkpoint in the
> PyTorch model, you first need to convert it. You can use this script
> <https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py>
> to convert it to a PyTorch checkpoint.
>
> If you want to use our TensorFlow interface in order to do this, you would
> still need to use this script to convert it to our interface, and then use
> the TFBertForSequenceClassification while specifying the from_pt option
> (as the result would be a PyTorch checkpoint):
>
> model = TFBertForSequenceClassification.from_pretrained("directory", from_pt=True)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/3931#issuecomment-618629373>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ADBVWZB5OSLJ2X6WEC63COTROCMKPANCNFSM4MPLTASQ>
> .
>


`from_tf` specified we're loading a TensorFlow checkpoint that is already in the HuggingFace format (TF2), which is usually different than the original implementation (TF1).

The original implementation's checkpoints should first be converted in order to be usable.

> `from_tf` specified we're loading a TensorFlow checkpoint that is already in the HuggingFace format (TF2), which is usually different than the original implementation (TF1).
> 
> The original implementation's checkpoints should first be converted in order to be usable.

how can I convert ckpt(TF1) to h5(TF2)?
I only found convert_bert_original_tf_checkpoint_to_pytorch.py

You could use the script you mention to convert the model to PyTorch; then this PyTorch checkpoint can be seamlessly loaded in a TensorFlow implementation, see comment above: https://github.com/huggingface/transformers/issues/3931#issuecomment-618629373

After that you can just do 

```py
model.save_pretrained("here")
```

and you should have a `tf_model.h5` under `here`.