Small question about BertForMaskedLM usage on TF model
Author: rmxkyz
Pull request: no

Hi everyone, 
I was using BertForMaskedLM to predict possible candidate words in the content. 

For example:
**cat like to drink [MASK], so am I.**

on [tf1 bert-pretrained model](https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip)
output would be 
`[[{'sequence': '[CLS] cat like to drink it, so am i. [SEP]', 'score': 0.22914603352546692, 'token': 2009, 'token_str': 'i t'}, {'sequence': '[CLS] cat like to drink water, so am i. [SEP]', 'score': 0.1088637188076973, 'token': 2300, 'token_str': 'w a t e r'}, {'sequence': '[CLS] cat like to drink blood, so am i. [SEP]', 'score': 0.1075243279337883, 'token': 2668, 'token_str': 'b l o o d'}]]`

However, in [tf2 bert-pretrained model](https://storage.googleapis.com/cloud-tpu-checkpoints/bert/keras_bert/uncased_L-12_H-768_A-12.tar.gz)
output gives, 
`[[{'sequence': '[CLS] cat like to drink∘, so am i. [SEP]', 'score': 0.0002078865800285712, 'token': 30126, 'token_str': '# # ∘'}, {'sequence': '[CLS] cat like to drink zinc, so am i. [SEP]', 'score': 0.00020266805950086564, 'token': 15813, 'token_str': 'z i n c'}, {'sequence': '[CLS] cat like to drink organic, so am i. [SEP]', 'score': 0.00019718357361853123, 'token': 7554, 'token_str': 'o r g a n i c'}]]
`
It seems tf2 will randomly give unreasonable words, but I have no idea why causing this.

I notice the implementation of BERT on TF1 and TF2 are different (I think so? since tf2 use keras as main core implementation), however, does the architecture vary from TF1 pretrained model to TF2 that cause BertForMaskedLM give different embedding?  Does BertForMaskedLM support reading weight on TF2 model?

The code I am using are something like this
```
config = BertConfig.from_pretrained("./pretrained/tf2")
config.is_decoder=False

tokenizer = BertTokenizer.from_pretrained("./pretrained/tf2")
model = BertForMaskedLM.from_pretrained("./pretrained/tf2",config=config,from_tf=True)
```
tf1 checkpoint in FileZilla
![tf1](https://user-images.githubusercontent.com/56808566/110295862-96b08800-802c-11eb-985e-90b12449a21a.PNG)
tf2
![tf2](https://user-images.githubusercontent.com/56808566/110295918-a760fe00-802c-11eb-84a5-b35dd101da7f.PNG)

Am I using wrong API(like should I use other API instead other than BertForMaskedLM) or any place I have to change to make it work? 
Any suggestion or reflection are sincerely grateful, thanks in advance!

Comments:
Hi! In order to use either of those weights you'll have to convert them to a HuggingFace format.

For that you have two available scripts:
- From [TF1](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/convert_bert_original_tf_checkpoint_to_pytorch.py)
- From [TF2](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/convert_bert_original_tf2_checkpoint_to_pytorch.py)

Hi @LysandreJik! Thanks for always being kind and active to solve our questions, I have checked both TF1 and TF2 convert script which TF1 works perfectly to convert bert_model.ckpt to pytorch_model.bin, while the TF2 will always give error message as picture here:
![1](https://user-images.githubusercontent.com/56808566/110410789-6caaa480-80c4-11eb-9814-228961224342.PNG)
To add on, in my task I keep training checkpoint model on my dataset by running [run_pretraining.py ](https://github.com/tensorflow/models/blob/master/official/nlp/bert/run_pretraining.py), so according to the script description on TF2 convert script, I believe the problem is mlm head which I have to add something above the convert script to convert model right? or keep running run_pretraining.py won't give such header on it?
![2](https://user-images.githubusercontent.com/56808566/110411072-e0e54800-80c4-11eb-9ef1-38cf0a1a144e.PNG)
If the header exist, how can I solve this problem? Even though I tried BERT for months but I am not that confident to say I understand how it works. In #9941 I have tried to add elif condition on m_name but seems it's a bad approach on this question. 

Again, thanks for the reply! Really appreciated for giving me such direction to it!



Hmmm, I see this is an issue indeed! Could you let me know how you obtained your TF2 checkpoint so that I may check it on my side?

You're welcome, happy to help :)

Sure! Due to the [repository](https://github.com/tensorflow/models/tree/master/official/nlp/bert#access-to-pretrained-checkpoints) haven't release chinese model yet, so the way I obtained this model is by these steps, 
1. Download [bert-base-chinese](https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip) from google-bert
2. Use this [script](https://github.com/tensorflow/models/blob/master/official/nlp/bert/tf2_encoder_checkpoint_converter.py) to convert tf1 checkpoint to tf2 with following args
`python tf2_encoder_checkpoint_converter.py --checkpoint_to_convert=$BASE_DIR/bert_model.ckpt --converted_checkpoint_path=tmp/ --bert_config_file=$BASE_DIR/bert_config.json`
note: $BASE_DIR is the path to the model directory (which is the model done by step 1 & 2)
3. Using [create_pretraining_data.py](https://github.com/tensorflow/models/blob/master/official/nlp/data/create_pretraining_data.py) to create the dataset that I want to keep training by its domain, the simple data I use can be found [here](https://drive.google.com/file/d/163IAWfgQZ1TIN6WYzWdoRwmFXDzGIRxj/view?usp=sharing).
The args I use are like this
`python create_pretraining_data.py --input_file=./sample.txt --output_file=ModelRecord --vocab_file=./$BASE_DIR/vocab.txt --max_seq_length=128 --max_predictions_per_seq=19 --masked_lm_prob=0.15 --random_seed=46 --dupe_factor=1`
4. Now we have ModelRecord, use tf2 version [run_pretraining.py](https://github.com/tensorflow/models/blob/master/official/nlp/bert/run_pretraining.py) to training model on these instances in ModelRecord. The args I use look like this
`python run_pretraining.py --input_file=ModelRecord --output_dir=tmp/Model_L128_B32 --bert_config_file=bert_config.json --max_seq_length=128 --max_predictions_per_seq=19 --do_train=True --train_batch_size=32 --num_train_steps=2000000 --num_warmup_steps=2000 --learning_rate=1e-5 --save_checkpoints_step=60000 --keep_checkpoint_max=4`
5. After training, the directory in Model_L128_B32(where model saved) containing these checkpoint,
![4](https://user-images.githubusercontent.com/56808566/110595525-0fdce600-81b9-11eb-918a-4a418b049d02.PNG) 
I load the checkpoint in directory "pretrained", and rename the checkpoint I want to evaluate to model.ckpt.data-00000-of-00001, ckpt_index to model.ckpt.index, with config and vocab in the directory. 

### System Info
My environments are: (using pipreqs)
Python version: 3.7.5
CUDA used to build PyTorch: cuda_11.0_bu.TC445_37.28845127_0
OS: Ubuntu 18.04.5 LTS
GCC version: 7.5.0

### Versions of relevant libraries: (using pipreqs)
numpy==1.19.5
transformers==4.2.2
six==1.15.0
torch==1.7.1+cu110
tensorflow_gpu==2.2.0
gin_config==0.1.1
absl_py==0.11.0
tensorflow_hub==0.11.0
sentencepiece==0.1.94
absl==0.0
torchvision==0.8.2+cu110
bert4keras==0.9.9
gin==0.1.006
tensorflow==2.4.1


I think that's all of it, if I miss any step please inform me with no hesitate, I might get wrong :P 
Best regards!


I see, the conversion script should work for that use-case. Is there a way for you to share the checkpoints you have obtained so that I can take a look? You can share them through the hub under your username.

I test for a while and also follow what #8504 did, but now sure why not success
![5](https://user-images.githubusercontent.com/56808566/110792383-47798a00-82ae-11eb-8be3-b73ea4f615bd.PNG)

instead, I upload to [mydrive](https://drive.google.com/drive/folders/1e1xHXZQSEpHBI0YF6xLpUY2Ne8Zi7Cjl?usp=sharing), I will try it again tomorrow see if I miss something.

Ah, it's probably because you didn't install git-lfs/didn't track the files! Doing this in the repo should help:
```
git-lfs install
git-lfs track <name_of_your_large_file>
```
You can check it's being correctly tracked:
```
git-lfs track
```
check for the name of your large file and ensure it's being tracked by git-lfs:
```
Objects to be committed:

	.gitattributes (Git: 31aaf10)
	README.md (Git: 358442a)
	config.json (Git: 57a54a8)
                 
[...]                       
	pytorch_model.bin (LFS: 6a9a9a5)
                           ^^^
[...]
	special_tokens_map.json (Git: e3ec7ab)
	tokenizer_config.json (Git: ab033df)
	vocab.txt (Git: 4d96f93)

```
Then you should be able to push without any issue

LGTM! The only issue I encounter is I don't actually know how these file being add. 

```                    
	pytorch_model.bin (LFS: 6a9a9a5)
                           ^^^
```
My occasion for track and commit is
![2](https://user-images.githubusercontent.com/56808566/110880117-12565180-8319-11eb-8877-3a1c928966e5.PNG)
![1](https://user-images.githubusercontent.com/56808566/110880101-0bc7da00-8319-11eb-87ad-5ee6686a2d01.PNG)
 
I assume this will be shown at the git commit -m "comment" phase? Anyway I finally upload my [model](https://huggingface.co/rmxkyz/zh_tf2/tree/main), weeee! 


Fantastic! I don't have time to look at it today, but I'll try and do that on Monday. Thanks!

Thanks for your kindly support! I am not rush with this experiment, so do it ease when you're free :D

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.