Issue with converting my own BERT TF2 checkpoint to PyTorch and loading the converted PyTorch checkpoint for training
Author: jalajthanaki
Pull request: no

Hi,

I’m using huggingface to train my own bert model. I have checkpoints which are in TensorFlow2 and I have converted them successfully in PyTorch. The checkpoint conversion script has created **_.bin_** file which is having following subdirectories and pkl file.

![dfa9d353b4e25888726ebae99c19c048aaa9d4e4](https://user-images.githubusercontent.com/12840374/109604765-167cb500-7b4a-11eb-9e52-013762d02608.png)


I wanted to gain some more information on the following points:

- Am’I doing the conversation of TensorFlow checkpoint to PyTorch checkpoint correct? I can see the pre-trained bert-base-uncased model which is hosted on hugging face model repository is just having single [pytorch_model.bin](https://huggingface.co/bert-base-uncased/blob/main/pytorch_model.bin) file.
- How I can load this custom BERT-PyTorch model’s checkpoint using [modeling_bert.py](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py) script?

Any help or suggestions will be helpful.

Thanks

Comments:
You've converted them successfully in PyTorch, so you should be left with a `pytorch_model.bin` alongside a `config.json`, is that right?

I recommend reading the [quicktour entry related to using models](https://huggingface.co/transformers/quicktour.html#using-the-model) in order to get a sense of how one can load a model in PyTorch.

Hi @LysandreJik,  Thank you for replying on this thread. 

> You've converted them successfully in PyTorch, so you should be left with a pytorch_model.bin alongside a config.json, is that right? 

Yes, I have pytorch_model.bin file but the point is: it is not single serialised file like [this](https://huggingface.co/bert-base-uncased/blob/main/pytorch_model.bin) but  I can unzip that exported bin file which I have generated using [convert_bert_original_tf2_checkpoint_to_pytorch.py](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/convert_bert_original_tf2_checkpoint_to_pytorch.py) script and I can see the three main components which I posted in snapshot earlier. Ideally,  I should have single serialised pytorch_model.bin, right? 

I have checked the same with original BERT model and tried to convert it from TF2 to PyTorch there also I have bert-model.bin file but I can unzip that bert-model.bin and it also have three components present inside the bin. See the snapshots. 

Snapshot 1: My exported bin file
![Screenshot from 2021-03-08 10-18-44](https://user-images.githubusercontent.com/12840374/110276112-7d8de400-7ff8-11eb-956e-a72e13cf68b0.png)

Snapshot 2: I can unzip this bin file
![Screenshot from 2021-03-08 10-18-52](https://user-images.githubusercontent.com/12840374/110276111-7cf54d80-7ff8-11eb-9a0e-f004106b3a5c.png)

Snapshot 3: I can see the three components 
![Screenshot from 2021-03-08 10-18-58](https://user-images.githubusercontent.com/12840374/110276106-7bc42080-7ff8-11eb-90a1-cf64f0ca58d6.png)

Is this expected or Am I missing something?

Basically my goal here is to make sure that I'm converting the my own BERT model which is trained using TF2 to PyTorch in bug-free manner.

Thanks you again for sharing the documentation link I will try that and post my further questions. 

Thanks,

That is the way PyTorch works. See the documentation for [torch.save](https://pytorch.org/docs/stable/generated/torch.save.html).

It states:

> The 1.6 release of PyTorch switched torch.save to use a new zipfile-based file format. torch.load still retains the ability to load files in the old format. If for any reason you want torch.save to use the old format, pass the kwarg _use_new_zipfile_serialization=False.

Thank you so much @LysandreJik for your help now I have one single .bin file. Closing this issue. I will reopen if needed.