How to load google-research/bert .ckpt weight
Author: Smile-L-up
Pull request: no

# 🌟 New model addition
Hello，I'm going to convert.ckpt file to h5 file. Compare the two sides of the parameter information is the same, but through the TFBertForMaskedLM load, check the weight found a lot of programming 1 and 0.
## Model description

<!-- Important information -->

## Open source status

* [ ] the model implementation is available: (give details)
* [ ] the model weights are available: (give details)
* [ ] who are the authors: (mention them, if possible by @gh-username)


Comments:
Hi @Smile-L-up 👋  From your message, I'm understanding that you converted the weights and found differences. Is this correct? If so, would you be able to share a script (or a collab) where I could reproduce the issue?

Thank you for your reply. I read the converted H5 file and the correct H5 file, and roughly compared the weight value is consistent. But load the weights with TFBertForMaskedLM.from_pretrained () and check the weights with  {for layer in model.layers: layer.get_weights()}. It is found that the weight value of the converted H5 file is a lot of programming 1 or 0。
The conversion script is as follows：

import tensorflow as tf
import h5py
import numpy as np
import re

def ckpt_to_h5(cpktFileName='',h5FileName=''):
    cpktFileName = 'model/pre-trained/bert-base-chinese/bert_model.ckpt'  #cpkt 文件路径
    h5FileName = 'model/fine-tuned/tf_model111.h5'

    reader = tf.compat.v1.train.NewCheckpointReader(cpktFileName)
    f = h5py.File(h5FileName, 'w')
    t_g = None
    for key in sorted(reader.get_variable_to_shape_map()):        
        print(key)
        prefix_list=[]        
        if re.match(r'^bert/embeddings',key):
            prefix_list.append('bert/tf_bert_for_pre_training_6/')
            prefix_list.append('mlm___cls/tf_bert_for_pre_training_6/')            
        elif re.match(r'^bert',key):
            prefix='bert/tf_bert_for_pre_training_6/'
            prefix_list.append(prefix)
        elif re.match(r'^cls/predictions',key):
            prefix='mlm___cls/tf_bert_for_pre_training_6/mlm___'
            prefix_list.append(prefix)
        elif re.match(r'^cls/seq_relationship',key):
            prefix='nsp___cls/tf_bert_for_pre_training_6/nsp___'
            prefix_list.append(prefix)
        else:
            continue
        for prefix in prefix_list:
            tmp_key=key
            if tmp_key=='cls/seq_relationship/output_bias':
                tmp_key='cls/seq_relationship/bias'
            elif tmp_key=='cls/predictions/output_bias':
                tmp_key='cls/predictions/bias'
            elif tmp_key=='cls/seq_relationship/output_weights':
                tmp_key='cls/seq_relationship/kernel'
  
            elif re.search(r'embeddings$',tmp_key):
                if re.search(r'word_embeddings$',tmp_key):
                    tmp_key=tmp_key+'/weight'
                else:
                    tmp_key=tmp_key+'/embeddings'
            h5_key=prefix+re.sub(r'layer_','layer_._',tmp_key,flags=re.I)+':0'
            tensor=reader.get_tensor(key)
            if tmp_key=='cls/seq_relationship/kernel':
                tensor=tf.transpose(tensor)
            print(tensor)
            f[h5_key] = tensor
        
    f.close()
def read_ckpt(cpktFileName=''):
    cpktFileName = 'model/pre-trained/bert-base-chinese/bert_model.ckpt'  #cpkt 文件路径
    reader = tf.compat.v1.train.NewCheckpointReader(cpktFileName)
    for key in sorted(reader.get_variable_to_shape_map()):            
        print(key)
def out_h5(f,group):
    group_read=f[group]
    if isinstance(group_read,h5py._hl.dataset.Dataset):
        print(group_read.name)
        data=np.array(group_read)
        print(data.shape)
        print(data)
        pass
    else:
        for subgroup in group_read.keys():
            key=group+'/'+subgroup
            out_h5(f,key)
def read_h5(h5FileName=''):
    h5FileName = 'model/fine-tuned/tf_model111.h5'
    f=h5py.File(h5FileName,'r')
    print(f)
    for group in f.keys():
        out_h5(f,group)
        '''
        print(group)
        group_read=f[group]
        for subgroup in group_read.keys():
            key=group+'/'+subgroup
            print(key)
            dset_read = f[group+'/'+subgroup]
            for dset in dset_read.keys():
                key=group+'/'+subgroup+'/'+dset
                print(key)
        '''
                
if __name__ == '__main__':
    ckpt_to_h5()
    #read_h5()
    #read_ckpt()


Hey @Smile-L-up 👋 

I am unable to reproduce your code, as it relies on local files (`cpktFileName `). 

However, from your description, I have yet to understand the problem, which makes helping difficult :) Are you:
1. Trying to convert the weights from a checkpoint that you created, and they do not load properly into `TFBertForMaskedLM `?
2. Converting weights from an existing checkpoint on TF Hub, and they do not load properly into `TFBertForMaskedLM `?
3. Converting weights from an existing checkpoint on TF Hub, and they do not match a checkpoint hosted on Hugging Face?
4. (other issue?)

Finally, keep in mind that `transformer` do not support TF v1, which may explain some mismatches :) 

Thank you for your reply. I have realized the conversion of CKPT of Bert base Chinese into H5. However, I encountered another problem. Due to the needs of the project, the H5 file is required. I didn't seem to find the H5 file of Albert base Chinese model. I felt that it would take a lot of time to convert the CKPT of Albert base Chinese into H5. I tried to convert the bin of Albert base Chinese into H5, but there was a problem.

I see what you mean -- Albert models like [this one](https://huggingface.co/ckiplab/albert-tiny-chinese/tree/main) don't have TF weights. 

You can still load them and they will be a TensorFlow model if you load them with the `from_pt=True` flag. E.g.
```
model = TFAlbertModel.from_pretrained("ckiplab/albert-tiny-chinese", from_pt=True)
```

If you want to save the corresponding TF2 weights as a  local`.h5` file, you can also do it by calling
```
model.save_weights("local/path/model.h5")
```

If you have local `h5` files, you can reload the model doing
```
model = TFAlbertModel.from_pretrained("local/path/model.h5")
```

Alternatively, you can also push these weights into the HF hub -- see docs [here](https://huggingface.co/docs/transformers/master/en/model_sharing#push-a-model-with-pushtohubcallback)

__________
Does this solve your problem?

Thank you very much for your reply, you solved me a big favor!

Awesome, glad I could help 👍  Closing this issue