ModuleAttributeError occurs during Converting TensorFlow Checkpoints (BERT)
Author: forest1988
Pull request: no

## Environment info

- `transformers` version: 4.1.1
- Platform: Linux-4.15.0-129-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
 albert, bert, GPT2, XLM: @LysandreJik

## Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* Convert TF v1 ckpt to PyTorch

## To reproduce

I tried to convert a TensorFlow checkpoint, but `ModuleAttributeError` occurred.

What I run:
```
****@**** $ transformers-cli convert --model_type bert \
>   --tf_checkpoint $MODEL_DIR/model.ckpt \
>   --config ****/bert_config.json \
>   --pytorch_dump_output $MODEL_DIR/pytorch_model.bin
```

(In this time, `bert_config.json` is in a separate folder, but it corresponds to the `ckpt`.)

Output is:
```
Traceback (most recent call last):
  File "/****/.pyenv/versions/anaconda3-2020.07/bin/transformers-cli", line 8, in <module>
    sys.exit(main())
  File "/****/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/transformers/commands/transformers_cli.py", line 51, in main
    service.run()
  File "/****/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/transformers/commands/convert.py", line 105, in run
    convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
  File "/****/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/transformers/models/bert/convert_bert_original_tf_checkpoint_to_pytorch.py", line 36, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model, config, tf_checkpoint_path)
  File "/****/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 155, in load_tf_weights_in_bert
    pointer.shape == array.shape
  File "/****/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/torch/nn/modules/module.py", line 778, in __getattr__
    raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
torch.nn.modules.module.ModuleAttributeError: 'BertEmbeddings' object has no attribute 'shape'
```

## Expected behavior

I think it is not strange that `BertEmbeddings` (`nn.Module`) doesn't have `shape.`

Is it possible to get such an error depending on the original TensorFlow checkpoint?　
In such a case, is there any tips to deal with it?

I really appreciate any help you can provide.


Comments:
Hi, how did you obtain your TensorFlow checkpoint? Was it trained with http://github.com/google-research/bert?

Hi @LysandreJik,

Thank you for giving me the comment.

The TensorFlow checkpoint is not my own but is provided by a researcher. There may be my misreading about the related paper, but in the paper the researcher says:

- They fine-tune all the parameters including the BERT and the two additional linear layers.
- They directly used public pretrained parameters of BERT from https://github.com/google-research/bert

From the information in the paper, I think the checkpoint is trained with the URL you showed me.

I'm having a hard time reproducing the issue, the following works:

```
transformers-cli convert --model_type bert \
    --tf_checkpoint bert_model.ckpt \
    --config bert_config.json \
    --pytorch_dump_output pytorch_model.bin

```

on both of these:

```
BERT-Base, Uncased: 12-layer, 768-hidden, 12-heads, 110M parameters
BERT-Large, Uncased: 24-layer, 1024-hidden, 16-heads, 340M parameters
```

Do you know of any difference between those architectures and the one you have?

Thank you for taking your time to reproduce this issue.

The checkpoint is using `uncased_L-12_H-768_A-12/bert_model.ckpt` as an initial checkpoint for fine-tuning.
Hence, the checkpoint seems to be an `Uncased` model.

The `BertConfig` of the checkpoint says the architecture has the following parameters:

```
12-layer, 768-hidden, 12-heads
```

For your reference, the items below in the fine-tuned checkpoint folder (which I referred to as`$MODEL_DIR`).
``` sh
****@**** $ ls
checkpoint  model.ckpt.data-00000-of-00001  model.ckpt.index  model.ckpt.meta
```

I would like to try converting on another checkpoint as well and see if I get the same problem.


```
****@**** $ pwd
/****/uncased_L-12_H-768_A-12
****@**** $ ls
bert_config.json  bert_model.ckpt.data-00000-of-00001  bert_model.ckpt.index  bert_model.ckpt.meta  vocab.txt
****@**** $ transformers-cli convert --model_type bert \
>     --tf_checkpoint bert_model.ckpt \
>     --config bert_config.json \
>     --pytorch_dump_output pytorch_model.bin
```

It worked without any error, and showed `Save PyTorch model to pytorch_model.bin`.

Is it possible that `vocab.txt` needs to be in the same folder as `ckpt`, not in the same folder as `bert_config.json`?
I'm sorry if I'm missing the point.

So it worked with the second BERT model but not with the first? Do you know of any difference between the first and second?

The `vocab.txt` shouldn't have an impact; this is for the tokenizer and it can be automatically loaded by the `BertTokenizer`

Yes, it worked with the second one but not with the first one.
- the first BERT model: fine-tuned and provided by a third-party, using the official pre-trained model as an initial point.
- the second BERT model: official pre-trained model provided in https://github.com/google-research/bert

Thank you for telling me that `vocab.txt` is not the cause.
Before I saw your comment, I had tried putting vocab.txt in the same folder, but I still got the same error.

The output during the conversion of the first model is as below.
It seems `bert/embeddings` is skipped.

``` sh
2021-01-19 15:00:49.880931: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Building PyTorch model from configuration: BertConfig {
  "attention_probs_dropout_prob": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "type_vocab_size": 2,
  "vocab_size": 30522
}

Converting TensorFlow checkpoint from /****/model.ckpt
Loading TF weight bert/embeddings/LayerNorm/beta with shape [768]
Loading TF weight bert/embeddings/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/embeddings/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/embeddings/LayerNorm/gamma with shape [768]
Loading TF weight bert/embeddings/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/embeddings/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/embeddings/position_embeddings with shape [512, 768]
Loading TF weight bert/embeddings/position_embeddings/adam_m with shape [512, 768]
Loading TF weight bert/embeddings/position_embeddings/adam_v with shape [512, 768]
Loading TF weight bert/embeddings/relation_embedding with shape [47, 768]
Loading TF weight bert/embeddings/token_type_embeddings with shape [2, 768]
Loading TF weight bert/embeddings/token_type_embeddings/adam_m with shape [2, 768]
Loading TF weight bert/embeddings/token_type_embeddings/adam_v with shape [2, 768]
Loading TF weight bert/embeddings/word_embeddings with shape [30522, 768]
2021-01-19 15:00:58.056232: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 93763584 exceeds 10% of free system memory.
Loading TF weight bert/embeddings/word_embeddings/adam_m with shape [30522, 768]
2021-01-19 15:00:58.912772: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 93763584 exceeds 10% of free system memory.
Loading TF weight bert/embeddings/word_embeddings/adam_v with shape [30522, 768]
2021-01-19 15:00:59.755334: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 93763584 exceeds 10% of free system memory.
Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/LayerNorm/beta/adam_v with shape [768] 
Loading TF weight bert/encoder/layer_0/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_0/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_0/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_0/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_0/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_0/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_0/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_0/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_0/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_0/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_0/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_1/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_1/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_1/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_1/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_1/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_1/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_1/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_1/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_1/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_1/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_10/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_10/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_10/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_10/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_10/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_10/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_10/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_10/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_10/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_10/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_11/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_11/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_11/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_11/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_11/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_11/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_11/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_11/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_11/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_11/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_2/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_2/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_2/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_2/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_2/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_2/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_2/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_2/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_2/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_2/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_3/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_3/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_3/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_3/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_3/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_3/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_3/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_3/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_3/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_3/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_4/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_4/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_4/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_4/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_4/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_4/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_4/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_4/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_4/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_4/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_5/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_5/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_5/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_5/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_5/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_5/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_5/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_5/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_5/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_5/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_6/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_6/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_6/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_6/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_6/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_6/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_6/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_6/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_6/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_6/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_7/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_7/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_7/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_7/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_7/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_7/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_7/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_7/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_7/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_7/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_8/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_8/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_8/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_8/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_8/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_8/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_8/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_8/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_8/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_8/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/output/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/key/bias with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/key/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/key/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/key/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/key/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/key/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/query/bias with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/query/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/query/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/query/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/query/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/query/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/value/bias with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/value/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/value/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/attention/self/value/kernel with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/value/kernel/adam_m with shape [768, 768]
Loading TF weight bert/encoder/layer_9/attention/self/value/kernel/adam_v with shape [768, 768]
Loading TF weight bert/encoder/layer_9/intermediate/dense/bias with shape [3072]
Loading TF weight bert/encoder/layer_9/intermediate/dense/bias/adam_m with shape [3072]
Loading TF weight bert/encoder/layer_9/intermediate/dense/bias/adam_v with shape [3072]
Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel with shape [768, 3072]
Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel/adam_m with shape [768, 3072]
Loading TF weight bert/encoder/layer_9/intermediate/dense/kernel/adam_v with shape [768, 3072]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta with shape [768]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/beta/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma with shape [768]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/output/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/output/dense/bias with shape [768]
Loading TF weight bert/encoder/layer_9/output/dense/bias/adam_m with shape [768]
Loading TF weight bert/encoder/layer_9/output/dense/bias/adam_v with shape [768]
Loading TF weight bert/encoder/layer_9/output/dense/kernel with shape [3072, 768]
Loading TF weight bert/encoder/layer_9/output/dense/kernel/adam_m with shape [3072, 768]
Loading TF weight bert/encoder/layer_9/output/dense/kernel/adam_v with shape [3072, 768]
Loading TF weight bert/final_output/token_score/bias with shape [3]
Loading TF weight bert/final_output/token_score/kernel with shape [768, 3]
Loading TF weight bert/pooler/dense/bias with shape [768]
Loading TF weight bert/pooler/dense/bias/adam_m with shape [768]
Loading TF weight bert/pooler/dense/bias/adam_v with shape [768]
Loading TF weight bert/pooler/dense/kernel with shape [768, 768]
Loading TF weight bert/pooler/dense/kernel/adam_m with shape [768, 768]
Loading TF weight bert/pooler/dense/kernel/adam_v with shape [768, 768]
Loading TF weight bert/relation/bias with shape [768]
Loading TF weight bert/relation/kernel with shape [768, 768]
Loading TF weight global_step with shape []
Loading TF weight loss/cls/predictions/output_bias with shape [30522]
Loading TF weight loss/cls/predictions/output_bias/adam_m with shape [30522]
Loading TF weight loss/cls/predictions/output_bias/adam_v with shape [30522]
Loading TF weight loss/cls/predictions/transform/LayerNorm/beta with shape [768]
Loading TF weight loss/cls/predictions/transform/LayerNorm/beta/adam_m with shape [768]
Loading TF weight loss/cls/predictions/transform/LayerNorm/beta/adam_v with shape [768]
Loading TF weight loss/cls/predictions/transform/LayerNorm/gamma with shape [768]
Loading TF weight loss/cls/predictions/transform/LayerNorm/gamma/adam_m with shape [768]
Loading TF weight loss/cls/predictions/transform/LayerNorm/gamma/adam_v with shape [768]
Loading TF weight loss/cls/predictions/transform/dense/bias with shape [768]
Loading TF weight loss/cls/predictions/transform/dense/bias/adam_m with shape [768]
Loading TF weight loss/cls/predictions/transform/dense/bias/adam_v with shape [768]
Loading TF weight loss/cls/predictions/transform/dense/kernel with shape [768, 768]
Loading TF weight loss/cls/predictions/transform/dense/kernel/adam_m with shape [768, 768]
Loading TF weight loss/cls/predictions/transform/dense/kernel/adam_v with shape [768, 768]
Loading TF weight output_bias with shape [2]
Loading TF weight output_bias/adam_m with shape [2]
Loading TF weight output_bias/adam_v with shape [2]
Loading TF weight output_weights with shape [2, 768]
Loading TF weight output_weights/adam_m with shape [2, 768]
Loading TF weight output_weights/adam_v with shape [2, 768]
Initialize PyTorch weight ['bert', 'embeddings', 'LayerNorm', 'beta']
Skipping bert/embeddings/LayerNorm/beta/adam_m
Skipping bert/embeddings/LayerNorm/beta/adam_v
Initialize PyTorch weight ['bert', 'embeddings', 'LayerNorm', 'gamma']
Skipping bert/embeddings/LayerNorm/gamma/adam_m
Skipping bert/embeddings/LayerNorm/gamma/adam_v
Initialize PyTorch weight ['bert', 'embeddings', 'position_embeddings']
Skipping bert/embeddings/position_embeddings/adam_m
Skipping bert/embeddings/position_embeddings/adam_v
Skipping bert/embeddings/relation_embedding
Traceback (most recent call last):
  File "/****/.pyenv/versions/anaconda3-2020.07/bin/transformers-cli", line 8, in <module>
    sys.exit(main())
  File "/****/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/transformers/commands/transformers_cli.py", line 51, in main
    service.run()
  File "/****/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/transformers/commands/convert.py", line 105, in run
    convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
  File "/****/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/transformers/models/bert/convert_bert_original_tf_checkpoint_to_pytorch.py", line 36, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model, config, tf_checkpoint_path)
  File "/****/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 155, in load_tf_weights_in_bert
    pointer.shape == array.shape
  File "/****/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/torch/nn/modules/module.py", line 778, in __getattr__
    raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
torch.nn.modules.module.ModuleAttributeError: 'BertEmbeddings' object has no attribute 'shape'
```



Hmmm it seems your model contains an additional weight? `bert/embeddings/relation_embedding` is not in the PyTorch model.

I'd like to say thank you for considering this matter together.

Referring to your comment, I've just now checked the open-sourced code of the model I'd like to convert to PyTorch.
However, I cannot find `relation_embedding` there.

Maybe the version of the fine-tuned model provided by the author is different from the published implementation.

As a test, I tried to convert the model which I had fine-tuned by myself using the author's published implementation.
In this case, the `relation_embedding` error did not occur, but the `Skipping global_step` caused an error shown below:

``` sh
ModuleAttributeError: 'BertForPreTraining' object has no attribute 'bias'.
```

This `global_step` is included in the author's published implementation, and I think it is defined by the author.

I think I was able to load the model provided by the author with the code published by the author, but it may be my misunderstanding.
I would like to verify that point as well.


Hmmm I understand.

I don't think it's the `global_step`, as this gets skipped here:

https://github.com/huggingface/transformers/blob/b020a736c374460af1b34267283f957988350630/src/transformers/models/bert/modeling_bert.py#L120-L125

As a way to debug what's happening here, could you add the following log statement:
```py
logger.info(f"Trying to assign {name}")
```

right after the following line:
https://github.com/huggingface/transformers/blob/b020a736c374460af1b34267283f957988350630/src/transformers/models/bert/modeling_bert.py#L116

It would then look like:

```py
    for name, array in zip(names, arrays):
        logger.info(f"Trying to assign {name}")
        name = name.split("/")
        # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
        # which are not required for using pretrained model
        if any(
            n in ["adam_v", "adam_m", "AdamWeightDecayOptimizer", "AdamWeightDecayOptimizer_1", "global_step"]
            for n in name
        ):
```

we can then try to identify what's happening with the checkpoint.

Thanks! I was also just checking modeling_bert.py#L116-L125 now.

It seems the author's code skips a variable if its name is not in the model for loading.

I've just read the `load_tf_weights_in_bert` used in `transformers-cli convert  --model_type bert`, and understood how items such as `adam_v` and `adam_m` (which are not required for use pretrained model) are skipped.

https://github.com/huggingface/transformers/blob/b020a736c374460af1b34267283f957988350630/src/transformers/models/bert/modeling_bert.py#L116-L125

At this time, I think I can skip the `relation_embedding` for my usage.
Hence, I'll try to modify the `convert` code to skip for this time. 

Also, I'll try the snippet you kindly wrote!

I inserted `logger.info(f"Trying to assign {name}")` and got the following outputs.

When try to convert the author provided fine-tuned model, the output is as below:

```
Trying to assign bert/embeddings/LayerNorm/beta
Initialize PyTorch weight ['bert', 'embeddings', 'LayerNorm', 'beta']
Trying to assign bert/embeddings/LayerNorm/beta/adam_m
Skipping bert/embeddings/LayerNorm/beta/adam_m
Trying to assign bert/embeddings/LayerNorm/beta/adam_v
Skipping bert/embeddings/LayerNorm/beta/adam_v
Trying to assign bert/embeddings/LayerNorm/gamma
Initialize PyTorch weight ['bert', 'embeddings', 'LayerNorm', 'gamma']
Trying to assign bert/embeddings/LayerNorm/gamma/adam_m
Skipping bert/embeddings/LayerNorm/gamma/adam_m
Trying to assign bert/embeddings/LayerNorm/gamma/adam_v
Skipping bert/embeddings/LayerNorm/gamma/adam_v
Trying to assign bert/embeddings/position_embeddings
Initialize PyTorch weight ['bert', 'embeddings', 'position_embeddings']
Trying to assign bert/embeddings/position_embeddings/adam_m
Skipping bert/embeddings/position_embeddings/adam_m
Trying to assign bert/embeddings/position_embeddings/adam_v
Skipping bert/embeddings/position_embeddings/adam_v
Trying to assign bert/embeddings/relation_embedding
Skipping bert/embeddings/relation_embedding
Traceback (most recent call last):
```

When try to convert my own fine-tuned model with the author's code, the output is as below:

```
...
Trying to assign bert/encoder/layer_9/output/dense/bias/adam_v
Skipping bert/encoder/layer_9/output/dense/bias/adam_v
Trying to assign bert/encoder/layer_9/output/dense/kernel
Initialize PyTorch weight ['bert', 'encoder', 'layer_9', 'output', 'dense', 'kernel']
Trying to assign bert/encoder/layer_9/output/dense/kernel/adam_m
Skipping bert/encoder/layer_9/output/dense/kernel/adam_m
Trying to assign bert/encoder/layer_9/output/dense/kernel/adam_v
Skipping bert/encoder/layer_9/output/dense/kernel/adam_v
Trying to assign bert/pooler/dense/bias
Initialize PyTorch weight ['bert', 'pooler', 'dense', 'bias']
Trying to assign bert/pooler/dense/bias/adam_m
Skipping bert/pooler/dense/bias/adam_m
Trying to assign bert/pooler/dense/bias/adam_v
Skipping bert/pooler/dense/bias/adam_v
Trying to assign bert/pooler/dense/kernel
Initialize PyTorch weight ['bert', 'pooler', 'dense', 'kernel']
Trying to assign bert/pooler/dense/kernel/adam_m
Skipping bert/pooler/dense/kernel/adam_m
Trying to assign bert/pooler/dense/kernel/adam_v
Skipping bert/pooler/dense/kernel/adam_v
Trying to assign global_step
Skipping global_step
Trying to assign output_bias
Traceback (most recent call last):
```

As you have pointed out, what caused the error is not `global_step`, but `output_bias`.

It seems that `output_bias` is not the part of BERT, but of the linear layer, as the related paper said that the authors fine-tune all the parameters including the BERT and the two additional linear layers.

```
Loading TF weight output_bias with shape [2]
Loading TF weight output_bias/adam_m with shape [2]
Loading TF weight output_bias/adam_v with shape [2]
Loading TF weight output_weights with shape [2, 768]
Loading TF weight output_weights/adam_m with shape [2, 768]
Loading TF weight output_weights/adam_v with shape [2, 768]
```

Hmmm indeed it seems that the model doesn't fit one-to-one to our architecture. You might need to slightly tweak the architecture and conversion script to load it, but you're probably the most expert on the matter. If you want me to take a deeper look, feel free to send me the weights/config so I can take a look locally.

Thank you for your kind and encouraging comment!
Thanks to your advice, it seems that what is a problem I should solve becomes clear.
I'll do my best to solve it!

Hi, 

Sorry it's been a few days because I had another issue, but I am working on this issue again.

I would like to ask one question about the relationship between `m_name` and `name`.
I'm assuming that `name` is split into the parts of the name hierarchy and that `m_name` handles each part.
`m_name` is referred after the `for` statement, is it safe to consider it as the same as `name[-1]`?

It seems that it is judged whether `m_name` is `_embedding` or `kernel`, is that correct?
Is there any reason why `m_name` is used instead of `name[-1]` (after the end of the `for` statement)?

https://github.com/huggingface/transformers/blob/9152f16023b59d262b51573714b40325c8e49370/src/transformers/models/bert/modeling_bert.py#L116-L152

I added two log statements to check if there are differences in `m_name` (after `for` statement) and `name[-1]`, but cannot found them.

`added`

```python
        logger.info(f"name: {name}")
        logger.info(f"m_name: {m_name}")
        if m_name[-11:] == "_embeddings":
            pointer = getattr(pointer, "weight")
        elif m_name == "kernel":
            array = np.transpose(array)
```

`output`

```
2021-01-24 08:49:34,227 | INFO : Skipping bert/embeddings/LayerNorm/gamma/adam_m
2021-01-24 08:49:34,227 | INFO : Skipping bert/embeddings/LayerNorm/gamma/adam_v
2021-01-24 08:49:34,228 | INFO : name: ['bert', 'embeddings', 'position_embeddings']
2021-01-24 08:49:34,229 | INFO : m_name: position_embeddings
2021-01-24 08:49:34,230 | INFO : Initialize PyTorch weight ['bert', 'embeddings', 'position_embeddings']
2021-01-24 08:49:34,231 | INFO : Skipping bert/embeddings/position_embeddings/adam_m
2021-01-24 08:49:34,232 | INFO : Skipping bert/embeddings/position_embeddings/adam_v
2021-01-24 08:49:34,233 | INFO : Skipping bert/embeddings/relation_embedding
2021-01-24 08:49:34,233 | INFO : name: ['bert', 'embeddings', 'relation_embedding']
2021-01-24 08:49:34,234 | INFO : m_name: relation_embedding
```



Thanks to your advice, I think I've almost achieved the conversion I'm aiming for.

I added a `force` option to force skip the unrelated items and save them separately as `npy.`
I split the loop part form `load_tf_weights_in_bert`, and defined new function `getpointer`.

If `force` is `True,` some items that cannot be found in BERT is skipped and saved separately. 

Here is my code snippet.

```python
def getpointer(pointer, m_name, name):
    if re.fullmatch(r"[A-Za-z]+_\d+", m_name):
        scope_names = re.split(r"_(\d+)", m_name)
    else:
        scope_names = [m_name]
    if scope_names[0] == "kernel" or scope_names[0] == "gamma":
        pointer = getattr(pointer, "weight")
    elif scope_names[0] == "output_bias" or scope_names[0] == "beta":
        pointer = getattr(pointer, "bias")
    elif scope_names[0] == "output_weights":
        pointer = getattr(pointer, "weight")
    elif scope_names[0] == "squad":
        pointer = getattr(pointer, "classifier")
    else:
        try:
            pointer = getattr(pointer, scope_names[0])
        except AttributeError:
            logger.info("Skipping {}".format("/".join(name)))
            # continue
            return pointer
    if len(scope_names) >= 2:
        num = int(scope_names[1])
        pointer = pointer[num]

    return pointer


def load_tf_weights_in_bert(model, config, tf_checkpoint_path, force=True, skipped_save_path="./skipped"):
    """Load tf checkpoints in a pytorch model."""

    if force:
        logger.warning("The 'force' option is set to be True. It will force conversion even if the model types do not match.")
        os.makedirs(os.path.join(skipped_save_path, "skipped"), exist_ok=True)
        skipped_names = []
        skipped_arrays = []

    try:
        import tensorflow as tf
    except ImportError:
        logger.error(
            "Loading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see "
            "https://www.tensorflow.org/install/ for installation instructions."
        )
        raise
    tf_path = os.path.abspath(tf_checkpoint_path)
    logger.info("Converting TensorFlow checkpoint from {}".format(tf_path))
    # Load weights from TF model
    init_vars = tf.train.list_variables(tf_path)
    names = []
    arrays = []
    for name, shape in init_vars:
        logger.info("Loading TF weight {} with shape {}".format(name, shape))
        array = tf.train.load_variable(tf_path, name)
        names.append(name)
        arrays.append(array)

    for name, array in zip(names, arrays):
        # logger.info(f"Trying to assign {name}")
        name = name.split("/")
        # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
        # which are not required for using pretrained model
        if any(
            n in ["adam_v", "adam_m", "AdamWeightDecayOptimizer", "AdamWeightDecayOptimizer_1", "global_step"]
            for n in name
        ):
            logger.info("Skipping {}".format("/".join(name)))
            continue
        pointer = model
        for m_name in name:
            if force:
                # Skip unrelated items and save them separately.
                try:
                    pointer = getpointer(pointer, m_name, name)
                except AttributeError:
                    logger.info("Skipping {}".format("/".join(name)))
                    skipped_names.append(name)
                    skipped_arrays.append(array)
            else:
                pointer = getpointer(pointer, m_name, name)
        if m_name[-11:] == "_embeddings":
            pointer = getattr(pointer, "weight")
        elif m_name == "kernel":
            array = np.transpose(array)
        try:
            if force:
                try:
                    pointer.shape
                except AttributeError:
                    logger.info("Skipping {}".format("/".join(name)))
                    skipped_names.append(name)
                    skipped_arrays.append(array)
            else:
                assert (
                    pointer.shape == array.shape
                ), f"Pointer shape {pointer.shape} and array shape {array.shape} mismatched"
        except AssertionError as e:
            e.args += (pointer.shape, array.shape)
            raise
        logger.info("Initialize PyTorch weight {}".format(name))
        pointer.data = torch.from_numpy(array)

    if force:
        logger.info("Save force skipped files")
        for name, array in zip(skipped_names, skipped_arrays):
            skipped_to_save = os.path.join(skipped_save_path, "skipped", "-".join(name) + ".npy")
            logger.info("Save force skipped {} to {}".format("/".join(name), skipped_to_save))
            np.save(skipped_to_save, array)

    return model
```

The conversion is done!

In my case, the skipped items are saved as follows.

``` sh
$ ls converted/skipped/
bert-embeddings-relation_embedding.npy  bert-final_output-token_score-kernel.npy  bert-relation-kernel.npy  output_weights.npy
bert-final_output-token_score-bias.npy  bert-relation-bias.npy                    output_bias.npy
```

I should manage these "unrelated" `np.array` files by making appropriate layers and write the arrays as weight and bias of the layer. 
Moreover, I have trouble loading the generated `pytorch_model.bin`. It is said that there is no `config.json` file. I'd like to check how to generate the correct one (just copying TF `bert_config.json` doesn't work) with converted `pytorch_model.bin`.

Thank you very much for your help!

Excuse me for my frequent posting.

To get the appropriate `config.json`, I've changed the last part of the conversion function, where the model is saved, to as follows (changed from `torch.save()` to `model.save_pretrained()`):

```python
    # Save pytorch-model
    print("Save the PyTorch model and the config file to {}".format(pytorch_dump_dir))
    # torch.save(model.state_dict(), pytorch_dump_path)
    model.save_pretrained(pytorch_dump_dir)
```

(The original code is in 
https://github.com/huggingface/transformers/blob/9152f16023b59d262b51573714b40325c8e49370/src/transformers/models/bert/convert_bert_original_tf_checkpoint_to_pytorch.py#L38-L40)

Then, the output of my modified script is:

```python
...
2021-01-25 04:27:00,442 | INFO : Initialize PyTorch weight ['output_bias']
2021-01-25 04:27:00,442 | INFO : Skipping output_bias/adam_m
2021-01-25 04:27:00,442 | INFO : Skipping output_bias/adam_v
2021-01-25 04:27:00,443 | INFO : Skipping output_weights
2021-01-25 04:27:00,443 | INFO : Skipping output_weights
2021-01-25 04:27:00,444 | INFO : Initialize PyTorch weight ['output_weights']
2021-01-25 04:27:00,444 | INFO : Skipping output_weights/adam_m
2021-01-25 04:27:00,444 | INFO : Skipping output_weights/adam_v
2021-01-25 04:27:00,445 | INFO : Save force skipped files
2021-01-25 04:27:00,445 | INFO : Save force skipped bert/embeddings/relation_embedding to ./converted/skipped/bert-embeddings-relation_embedding.npy
2021-01-25 04:27:00,456 | INFO : Save force skipped bert/final_output/token_score/bias to ./converted/skipped/bert-final_output-token_score-bias.npy
2021-01-25 04:27:00,458 | INFO : Save force skipped bert/final_output/token_score/kernel to ./converted/skipped/bert-final_output-token_score-kernel.npy
2021-01-25 04:27:00,461 | INFO : Save force skipped bert/final_output/token_score/kernel to ./converted/skipped/bert-final_output-token_score-kernel.npy
2021-01-25 04:27:00,463 | INFO : Save force skipped bert/relation/bias to ./converted/skipped/bert-relation-bias.npy
2021-01-25 04:27:00,466 | INFO : Save force skipped bert/relation/kernel to ./converted/skipped/bert-relation-kernel.npy
2021-01-25 04:27:00,492 | INFO : Save force skipped bert/relation/kernel to ./converted/skipped/bert-relation-kernel.npy
2021-01-25 04:27:00,519 | INFO : Save force skipped output_bias to ./converted/skipped/output_bias.npy
2021-01-25 04:27:00,521 | INFO : Save force skipped output_bias to ./converted/skipped/output_bias.npy
2021-01-25 04:27:00,523 | INFO : Save force skipped output_weights to ./converted/skipped/output_weights.npy
2021-01-25 04:27:00,525 | INFO : Save force skipped output_weights to ./converted/skipped/output_weights.npy
Save the PyTorch model and the config file to ./converted/
Configuration saved in ./converted/config.json
Model weights saved in ./converted/pytorch_model.bin
```

Now I can load the converted model!

About the skipped items (saved as `npy`), I think I can convert them to `nn.Module` by referring to:

https://github.com/huggingface/transformers/blob/9152f16023b59d262b51573714b40325c8e49370/src/transformers/models/bert/modeling_bert.py#L161

Thank you again!

Fantastic! Great job, thank you for sharing your progress!

I greatly appreciate your help on this issue.
It's my pleasure if someone who will come across a similar problem can look at this issue and solve the problem.

I think my `force` convert script is not adequately simple and is a bit hard to apply to the all models, 
but changing from `torch.save()` to `model.save_pretrained()` may help some users.
If you don’t mind, could you please tell me what do you think about this change?

```python
    # Save pytorch-model
    print("Save the PyTorch model and the config file to {}".format(pytorch_dump_dir))
    # torch.save(model.state_dict(), pytorch_dump_path)
    model.save_pretrained(pytorch_dump_dir)
```

It seems for me that creating `config.json` with the converted model `pytorch_model.bin` would be useful, or, for models where the convert command works correctly, is the `config.json` generated elsewhere?

If this point can be changed, the main code changes I assume are as follows:
- The save statement shown above.
- The option `--pytorch_dump_output` of convert command will be changed to have `/path/to/directory/` instead of `/path/to/directory/pytorch_model.bin`. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.