converting REALM tensorflow checkpoints to pytorch
Author: mchari
Pull request: no
Labels: name: wontfix

## Environment info
- `transformers` version: 3.3.1
- Platform: Linux-3.10.0-1062.12.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core
- Python version: 3.6.3
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: <Yes>
- Using distributed or parallel set-up in script?: <No>  


### Who can help
@jplu, @sgugger 

 -->

## Information

Model I am using (Bert, XLNet ...):  Bert


## To reproduce

Steps to reproduce the behavior:

1. Download checkpoint from the http link https://console.cloud.google.com/storage/browser/realm-data/cc_news_pretrained/embedded/.
2. python convert_bert_original_tf2_checkpoint_to_pytorch.py \
       --tf_checkpoint_path="./cc_news_pretrained/embedder/encoded" \
       --bert_config_file="./bert_config.json" \
       --pytorch_dump_path="./pytorch"

The checkpoint file has the following entries which are probably internal developer files(?):
model_checkpoint_path: "/cns/li-d/home/lumiere/public/models/gatoatigrado/ner-with-dates/10923195/1-active_losses=mlm_loss/export/temp/1580364602/retriever/encoded/encoded.ckpt"
all_model_checkpoint_paths: "/cns/li-d/home/lumiere/public/models/gatoatigrado/ner-with-dates/10923195/1-active_losses=mlm_loss/export/temp/1580364602/retriever/encoded/encoded.ckpt"

1)When I set tf_checkpoint_path to the directory containing the checkpoint, I get the error :
tensorflow.python.framework.errors_impl.NotFoundError: /cns/li-d/home/lumiere/public/models/gatoatigrado/ner-with-dates/10923195/1-active_losses=mlm_loss/export/temp/1580364602/retriever/encoded; No such file or directory

2)When I set tf_checkpoint_path to the checkpoint file encode.ckpt.data-00000-of-00001, I get the error:
env/lib/python3.6/site-packages/tensorflow/python/training/py_checkpoint_reader.py", line 95, in NewCheckpointReader
return CheckpointReader(compat.as_bytes(filepattern))
RuntimeError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for ./cc_news_pretrained/embedder/encoded/encode.ckpt.data-00000-of-00001

Thanks!



Comments:
I changed --tf_checkpoint_path="./cc_news_pretrained/embedder/encoded to 
--tf_checkpoint_path="./cc_news_pretrained/embedder/encoded/encoded.ckpt.

and used _convert_bert_original_tf_checkpoint_to_pytorch.py_ .
I see the following messages :
Loading TF weight block_emb with shape [13353718, 128]
Skipping block_emb

Building PyTorch model from configuration: BertConfig {
"attention_probs_dropout_prob": 0.1,
"gradient_checkpointing": false,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 1024,
"initializer_range": 0.02,
"intermediate_size": 4096,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 512,
"model_type": "bert",
"num_attention_heads": 16,
"num_hidden_layers": 24,
"pad_token_id": 0,
"type_vocab_size": 2,
"vocab_size": 30522
}

Traceback (most recent call last):
File "convert_tf_checkpoint_to_pytorch.py", line 78, in
args.tf_checkpoint_path, args.bert_config_file, args.pytorch_dump_path
File "convert_tf_checkpoint_to_pytorch.py", line 44, in convert_tf_checkpoint_to_pytorch
load_tf_weights_in_bert(model, config, tf_checkpoint_path)
File "/gstore/home/madabhuc/hayEnv/env/lib/python3.6/site-packages/transformers/modeling_bert.py", line 155, in load_tf_weights_in_bert
pointer.shape == array.shape
File "/gstore/home/madabhuc/hayEnv/env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 772, in getattr
type(self).name, name))
torch.nn.modules.module.ModuleAttributeError: 'BertForPreTraining' object has no attribute 'shape'



Following the suggestion #393, I I hacked transformers/src/transformers/modeling_bert.py, and I now see the following :
Converting TensorFlow checkpoint from ./cc_news_pretrained/embedder/encoded/encoded.ckpt
Loading TF weight block_emb with shape [13353718, 128]
Skipping block_emb
Initialize PyTorch weight ['block_emb']

Building PyTorch model from configuration: BertConfig {
"attention_probs_dropout_prob": 0.1,
"gradient_checkpointing": false,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 1024,
"initializer_range": 0.02,
"intermediate_size": 4096,
"layer_norm_eps": 1e-12,
"max_position_embeddings": 512,
"model_type": "bert",
"num_attention_heads": 16,
"num_hidden_layers": 24,
"pad_token_id": 0,
"type_vocab_size": 2,
"vocab_size": 30522
}

Save PyTorch model to /gstore/home/madabhuc/hayEnv/pytorch/pytorch.bin

Skipping and intializing 'block_emb' tells me that I have lost the weights info from the checkpoint. Don't believe this is correct.

Initializing did copy the weights from the checkpoint.

When I try to open another REALM tensorflow checkpoint, I get the following error message : 
transformers/modeling_bert.py", line 135, in load_tf_weights_in_bert
    pointer = getattr(pointer, "bias")
  File "/gstore/home/madabhuc/hayEnv/env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 772, in __getattr__
    type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'BertForPreTraining' object has no attribute 'bias'

@sgugger , @jplu , @LysandreJik :   Any ideas ?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

@mchari , do you manage to convert the REALM pre-trained TF models into pytorch models?

No, I didn't....

On Tue, Jun 29, 2021 at 10:54 PM Wei-Cheng Chang ***@***.***>
wrote:

> @mchari <https://github.com/mchari> , do you manage to convert the REALM
> pre-trained TF models into pytorch models?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8606#issuecomment-871116705>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHIXZJ4KXQQFNMXSPSAKWRDTVKWPZANCNFSM4TZGZWCQ>
> .
>