AttributeError: 'BertForPreTraining' object has no attribute 'shape'
Author: leejason
Pull request: no
Labels: name: Need more information

name: wontfix

Is there any suggestion for fixing the following? I was trying "convert_tf_checkpoint_to_pytorch.py" to convert a model trained from scratch but the conversion didn't work out....
```bash
Skipping cls/seq_relationship/output_weights/adam_v
Traceback (most recent call last):
  File "pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py", line 66, in <module>
    args.pytorch_dump_path)
  File "pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py", line 37, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model, tf_checkpoint_path)
  File "/content/my_pytorch-pretrained-BERT/pytorch_pretrained_bert/modeling.py", line 117, in load_tf_weights_in_bert
    assert pointer.shape == array.shape
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 535, in __getattr__
    type(self).__name__, name))
AttributeError: 'BertForPreTraining' object has no attribute 'shape'
```

Comments:
Hi,
Is it a model trained from the original Google BERT Tensorflow implementation?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I get the same error as @leejason. I used the pre-trained BERT base uncased model from the original TF implementation and fine-tuned it on my own training set. I am now trying to use the fine-tuned model to do masked LM. 

I also have similar issue. I pretrained a bert **from scratch** using **nvidia implementation** with **customized config file and vocab**.
Then I use
```
convert_tf_checkpoint_to_pytorch.convert_tf_checkpoint_to_pytorch(BERT_MODEL_PATH + 'model.ckpt',
                                                                 BERT_MODEL_PATH  + 'bert_config.json',
                                                                 BERT_MODEL_PATH + 'pytorch_model.bin')
```
```
...
Loading TF weight cls/predictions/transform/dense/bias with shape [512]
Loading TF weight cls/predictions/transform/dense/bias/adam_m with shape [512]
Loading TF weight cls/predictions/transform/dense/bias/adam_v with shape [512]
Loading TF weight cls/predictions/transform/dense/kernel with shape [512, 512]
Loading TF weight cls/predictions/transform/dense/kernel/adam_m with shape [512, 512]
Loading TF weight cls/predictions/transform/dense/kernel/adam_v with shape [512, 512]
Loading TF weight cls/seq_relationship/output_bias with shape [2]
Loading TF weight cls/seq_relationship/output_weights with shape [2, 512]
Loading TF weight global_step with shape []
Loading TF weight good_steps with shape []
Loading TF weight loss_scale with shape []
Skipping bad_steps
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-20-f28c5910adfc> in <module>
----> 1 bert = BertModel.from_pretrained(BERT_MODEL_PATH, from_tf=True).bert()

~/InEx/input/huggingface/pytorch-pretrained-BERT-master/pytorch_pretrained_bert/modeling.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    611             # Directly load from a TensorFlow checkpoint
    612             weights_path = os.path.join(serialization_dir, TF_WEIGHTS_NAME)
--> 613             return load_tf_weights_in_bert(model, weights_path)
    614         # Load from a PyTorch state_dict
    615         old_keys = []

~/InEx/input/huggingface/pytorch-pretrained-BERT-master/pytorch_pretrained_bert/modeling.py in load_tf_weights_in_bert(model, tf_checkpoint_path)
    107             array = np.transpose(array)
    108         try:
--> 109             assert pointer.shape == array.shape
    110         except AssertionError as e:
    111             e.args += (pointer.shape, array.shape)

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    537                 return modules[name]
    538         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 539             type(self).__name__, name))
    540 
    541     def __setattr__(self, name, value):

AttributeError: 'BertModel' object has no attribute 'shape'```

I solve this problem by bypass some variables in the model, such as "bad_steps", “global_step", "good_steps", "loss_scale". They don't have attribute 'shape‘ and I don't need them when fineturning the model.

In modeling.py, line 121, replace it with        
 if any(n in ["adam_v", "adam_m", "global_step", "bad_steps", "global_step", "good_steps", "loss_scale"] for n in name):
and delete line 151-156.

> I solve this problem by bypass some variables in the model, such as "bad_steps", “global_step", "good_steps", "loss_scale". They don't have attribute 'shape‘ and I don't need them when fineturning the model.
> 
> In modeling.py, line 121, replace it with
> if any(n in ["adam_v", "adam_m", "global_step", "bad_steps", "global_step", "good_steps", "loss_scale"] for n in name):
> and delete line 151-156.

It works. Thanks very much !

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I'm getting a similar error when trying to convert the newer BERT models released at
[tensorflow/models/tree/master/official/nlp/](https://github.com/tensorflow/models/tree/master/official/nlp/bert#pre-trained-models).

These models are either BERT models trained with Keras or else checkpoints converted from 
the original [google-research/bert](https://github.com/google-research/bert) repository. I also get the same error when I convert the TF1 to TF2 checkpoints myself using the [tf2_encoder_checkpoint_converter.py](https://github.com/tensorflow/models/blob/master/official/nlp/bert/tf2_encoder_checkpoint_converter.py) script: 

What I have tried:

First, I have downloaded a model:
```
wget https://storage.googleapis.com/cloud-tpu-checkpoints/bert/keras_bert/cased_L-12_H-768_A-12.tar.gz
# or
wget https://storage.googleapis.com/cloud-tpu-checkpoints/bert/tf_20/cased_L-12_H-768_A-12.tar.gz 
```
After unpacking:
```
export BERT_BASE_DIR=cased_L-12_H-768_A-12

transformers-cli convert --model_type bert \
    --tf_checkpoint $BERT_BASE_DIR/bert_model.ckpt \
    --config $BERT_BASE_DIR/bert_config.json \
    --pytorch_dump_output $BERT_BASE_DIR/pytorch_model.bin
```

The command prints the configuration but throws the following error: 

```
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_attention_layer/_value_dense/kernel/.ATTRIBUTES/VARIABLE_VALUE with shape [768, 12, 64]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_attention_layer_norm/beta/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_attention_layer_norm/gamma/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_attention_output_dense/bias/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_attention_output_dense/kernel/.ATTRIBUTES/VARIABLE_VALUE with shape [12, 64, 768]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_intermediate_dense/bias/.ATTRIBUTES/VARIABLE_VALUE with shape [3072]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_intermediate_dense/kernel/.ATTRIBUTES/VARIABLE_VALUE with shape [768, 3072]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_output_dense/bias/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_output_dense/kernel/.ATTRIBUTES/VARIABLE_VALUE with shape [3072, 768]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_output_layer_norm/beta/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_output_layer_norm/gamma/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
INFO:transformers.modeling_bert:Loading TF weight save_counter/.ATTRIBUTES/VARIABLE_VALUE with shape []
INFO:transformers.modeling_bert:Skipping _CHECKPOINTABLE_OBJECT_GRAPH
Traceback (most recent call last):
  File "/home/jbarry/anaconda3/envs/transformers/bin/transformers-cli", line 30, in <module>
    service.run()
  File "/home/jbarry/anaconda3/envs/transformers/lib/python3.6/site-packages/transformers/commands/convert.py", line 62, in run
    convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
  File "/home/jbarry/anaconda3/envs/transformers/lib/python3.6/site-packages/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py", line 36, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model, config, tf_checkpoint_path)
  File "/home/jbarry/anaconda3/envs/transformers/lib/python3.6/site-packages/transformers/modeling_bert.py", line 118, in load_tf_weights_in_bert
    assert pointer.shape == array.shape
  File "/home/jbarry/anaconda3/envs/transformers/lib/python3.6/site-packages/torch/nn/modules/module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'BertForPreTraining' object has no attribute 'shape'
```

This is happening in a fresh environment with PyTorch 1.3 installed in Anaconda (Linux), as well as pip-installing `tf-nightly` and `transformers` (2.3.0).

Has anyone else been able to successfully convert the TF 2.0 version models to PyTorch or know where I'm going wrong? Thanks!

> I'm getting a similar error when trying to convert the newer BERT models released at
> [tensorflow/models/tree/master/official/nlp/](https://github.com/tensorflow/models/tree/master/official/nlp/bert#pre-trained-models).
> 
> These models are either BERT models trained with Keras or else checkpoints converted from
> the original [google-research/bert](https://github.com/google-research/bert) repository. I also get the same error when I convert the TF1 to TF2 checkpoints myself using the [tf2_encoder_checkpoint_converter.py](https://github.com/tensorflow/models/blob/master/official/nlp/bert/tf2_encoder_checkpoint_converter.py) script:
> 
> What I have tried:
> 
> First, I have downloaded a model:
> 
> ```
> wget https://storage.googleapis.com/cloud-tpu-checkpoints/bert/keras_bert/cased_L-12_H-768_A-12.tar.gz
> # or
> wget https://storage.googleapis.com/cloud-tpu-checkpoints/bert/tf_20/cased_L-12_H-768_A-12.tar.gz 
> ```
> 
> After unpacking:
> 
> ```
> export BERT_BASE_DIR=cased_L-12_H-768_A-12
> 
> transformers-cli convert --model_type bert \
>     --tf_checkpoint $BERT_BASE_DIR/bert_model.ckpt \
>     --config $BERT_BASE_DIR/bert_config.json \
>     --pytorch_dump_output $BERT_BASE_DIR/pytorch_model.bin
> ```
> 
> The command prints the configuration but throws the following error:
> 
> ```
> INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_attention_layer/_value_dense/kernel/.ATTRIBUTES/VARIABLE_VALUE with shape [768, 12, 64]
> INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_attention_layer_norm/beta/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
> INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_attention_layer_norm/gamma/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
> INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_attention_output_dense/bias/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
> INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_attention_output_dense/kernel/.ATTRIBUTES/VARIABLE_VALUE with shape [12, 64, 768]
> INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_intermediate_dense/bias/.ATTRIBUTES/VARIABLE_VALUE with shape [3072]
> INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_intermediate_dense/kernel/.ATTRIBUTES/VARIABLE_VALUE with shape [768, 3072]
> INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_output_dense/bias/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
> INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_output_dense/kernel/.ATTRIBUTES/VARIABLE_VALUE with shape [3072, 768]
> INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_output_layer_norm/beta/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
> INFO:transformers.modeling_bert:Loading TF weight model/layer_with_weights-9/_output_layer_norm/gamma/.ATTRIBUTES/VARIABLE_VALUE with shape [768]
> INFO:transformers.modeling_bert:Loading TF weight save_counter/.ATTRIBUTES/VARIABLE_VALUE with shape []
> INFO:transformers.modeling_bert:Skipping _CHECKPOINTABLE_OBJECT_GRAPH
> Traceback (most recent call last):
>   File "/home/jbarry/anaconda3/envs/transformers/bin/transformers-cli", line 30, in <module>
>     service.run()
>   File "/home/jbarry/anaconda3/envs/transformers/lib/python3.6/site-packages/transformers/commands/convert.py", line 62, in run
>     convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
>   File "/home/jbarry/anaconda3/envs/transformers/lib/python3.6/site-packages/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py", line 36, in convert_tf_checkpoint_to_pytorch
>     load_tf_weights_in_bert(model, config, tf_checkpoint_path)
>   File "/home/jbarry/anaconda3/envs/transformers/lib/python3.6/site-packages/transformers/modeling_bert.py", line 118, in load_tf_weights_in_bert
>     assert pointer.shape == array.shape
>   File "/home/jbarry/anaconda3/envs/transformers/lib/python3.6/site-packages/torch/nn/modules/module.py", line 585, in __getattr__
>     type(self).__name__, name))
> AttributeError: 'BertForPreTraining' object has no attribute 'shape'
> ```
> 
> This is happening in a fresh environment with PyTorch 1.3 installed in Anaconda (Linux), as well as pip-installing `tf-nightly` and `transformers` (2.3.0).
> 
> Has anyone else been able to successfully convert the TF 2.0 version models to PyTorch or know where I'm going wrong? Thanks!

ignore those lines causing erros by changing
https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L123-L127 to 
        try:
            assert pointer.shape == array.shape
        except:
            pass

same thing for https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L129


> 
> 
> I solve this problem by bypass some variables in the model, such as "bad_steps", “global_step", "good_steps", "loss_scale". They don't have attribute 'shape‘ and I don't need them when fineturning the model.
> 
> In modeling.py, line 121, replace it with
> if any(n in ["adam_v", "adam_m", "global_step", "bad_steps", "global_step", "good_steps", "loss_scale"] for n in name):
> and delete line 151-156.

It helps! Thx u so much <3

I'm running into the same problem, I tried the solution proposed by @yzhang123 but it only makes me run in another error 

```
Traceback (most recent call last):
  File "C:\Users\lrizzello\AppData\Local\JetBrains\PyCharm 2019.3.4\plugins\python\helpers\pydev\pydevd.py", line 1434, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Users\lrizzello\AppData\Local\JetBrains\PyCharm 2019.3.4\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/source/repos/DeduplicationSiamese/PythonDeduper/eval/eval_matcher_tf.py", line 91, in <module>
    load_tf_weights_in_bert(model, config, tf_path)
  File "C:\Users\lrizzello\Anaconda3\envs\dedupe_transformer\lib\site-packages\transformers\modeling_bert.py", line 129, in load_tf_weights_in_bert
    pointer.data = torch.from_numpy(array)
TypeError: expected np.ndarray (got bytes)

Process finished with exit code -1

```

I got those checkpoints by training an existing huggingface model (namely 'google/bert_uncased_L-12_H-256_A-4') via the TFTrainer/TFTrainingArguments method

I have tried many other hacks to get this working, such as [this one](https://stackoverflow.com/questions/60539758/biobert-for-keras-version-of-huggingface-transformers) or [this one](https://github.com/huggingface/transformers/issues/676#issuecomment-502101078) but unsuccesfully but nothing worked. I keep running into error after error.

Has anyone managed to get this working any other way?

I managed to get it working by going through the pointers in debug mode and checking what variable name corresponded to what. This is the function I ended up using.

```
def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, bert_config_file, pytorch_dump_path):
    config_path = os.path.abspath(bert_config_file)
    tf_path = os.path.abspath(tf_checkpoint_path)
    print("Converting TensorFlow checkpoint from {} with config at {}".format(tf_path, config_path))
    # Load weights from TF model
    init_vars = tf.train.list_variables(tf_path)
    excluded = ["BERTAdam", "_power", "global_step", "_CHECKPOINTABLE_OBJECT_GRAPH"]
    init_vars = list(filter(lambda x: all([True if e not in x[0] else False for e in excluded]), init_vars))
    names = []
    arrays = []
    for name, shape in init_vars:
        print("Loading TF weight {} with shape {}".format(name, shape))
        array = tf.train.load_variable(tf_path, name)
        names.append(name)
        arrays.append(array)

    config = BertConfig.from_json_file(bert_config_file)
    print("Building PyTorch model from configuration: {}".format(str(config)))
    # Initialise PyTorch model
    model = BertForSequenceClassification(config)

    for name, array in zip(names, arrays):
        name = name.split("/")
        # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
        # which are not required for using pretrained model
        if any(n in ["adam_v", "adam_m", "global_step", "bad_steps", "global_step", "good_steps", "loss_scale",
                     "AdamWeightDecayOptimizer", "AdamWeightDecayOptimizer_1", "save_counter", ".OPTIMIZER_SLOT"] for n in name) or \
                name[0] == "optimizer":
            print("Skipping {}".format("/".join(name)))
            continue
        if ".OPTIMIZER_SLOT" in name:
            idx = name.index(".OPTIMIZER_SLOT")
            name = name[:idx]
        elif ".ATTRIBUTES" in name:
            idx = name.index(".ATTRIBUTES")
            name = name[:idx]
        print(name)
        pointer = model
        for m_name in name:
            if re.fullmatch(r"[A-Za-z]+_\d+", m_name):
                scope_names = re.split(r"_(\d+)", m_name)
            else:
                scope_names = [m_name]
            if scope_names[0] == "kernel" or scope_names[0] == "gamma":
                pointer = getattr(pointer, "weight")
            elif scope_names[0] == "output_bias" or scope_names[0] == "beta":
                pointer = getattr(pointer, "bias")
            elif scope_names[0] == "output_weights":
                pointer = getattr(pointer, "weight")
            elif scope_names[0] == "squad":
                pointer = getattr(pointer, "classifier")
            elif scope_names[0] == "dense_output" or scope_names[0] == "bert_output":
                pointer = getattr(pointer, "output")
            elif scope_names[0] == "self_attention":
                pointer = getattr(pointer, "self")
            else:
                try:
                    pointer = getattr(pointer, scope_names[0])
                except AttributeError:
                    logger.info("Skipping {}".format("/".join(name)))
                    continue
            if len(scope_names) >= 2:
                num = int(scope_names[1])
                pointer = pointer[num]
        if m_name[-11:] == "_embeddings":
            pointer = getattr(pointer, "weight")
        elif m_name == "kernel" or m_name == "gamma" or m_name == "output_weights":
            array = np.transpose(array)
        # print("Initialize PyTorch weight {}".format(name))
        pointer.data = torch.from_numpy(array)

    # Save pytorch-model
    print("Save PyTorch model to {}".format(pytorch_dump_path))
    torch.save(model.state_dict(), pytorch_dump_path)


convert_tf_checkpoint_to_pytorch(tf_path, config_path, pytorch_dump_path)
```



Hi, this is an actual programming error in modeling_bert.py.  If you look at line 145 it's pretty obvious that the code should be continuing to the next iteration of the *outer* loop (over name, array) rather than the inner one (over the path components of name) - otherwise why would the error messages say "skipping {name}":

https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py#L145

To fix this, simply extract the try/except block so that it wraps the entire loop (lines 127-148).  I would supply a patch but I have to work with transformers 3.5.1 for the moment since I'm using sentence-transformers which hasn't been updated to the latest version.

@thomwolf If the above fix will be added to the master branch this will be great
https://github.com/smartshark/transformers/pull/1


> 

I revised `modeling_bert.by` following @lrizzello 's code and could save tf1 checkpoint I personally trained into pytorch. I first changed tf1 checkpoint to tf2, and then used the below code. Here is the code I revised in `modeling_bert.py`
 
```
def load_tf_weights_in_bert(model, config, tf_checkpoint_path):
    try:
        import re
        import numpy as np
        import tensorflow as tf
    except ImportError:
        logger.error(
            "Loading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see "
            "https://www.tensorflow.org/install/ for installation instructions."
        )
        raise
    tf_path = os.path.abspath(tf_checkpoint_path)
    logger.info(f"Converting TensorFlow checkpoint from {tf_path}")
    # Load weights from TF model
    init_vars = tf.train.list_variables(tf_path)
    names = []
    arrays = []
    for name, shape in init_vars:
        logger.info(f"Loading TF weight {name} with shape {shape}")
        array = tf.train.load_variable(tf_path, name)
        names.append(name)
        arrays.append(array)
    for name, array in zip(names, arrays):
        name = name.split("/")
        # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
        # which are not required for using pretrained model
        if any(
            ["adam_v", "adam_m", "global_step", "bad_steps", "global_step", "good_steps", "loss_scale",
                     "AdamWeightDecayOptimizer", "AdamWeightDecayOptimizer_1", "save_counter", ".OPTIMIZER_SLOT"] for n in name) or \
                name[0] == "optimizer":
        #     n in ["adam_v", "adam_m", "AdamWeightDecayOptimizer", "AdamWeightDecayOptimizer_1", "global_step"]
        #     for n in name
        # ):
            logger.info(f"Skipping {'/'.join(name)}")
            continue
        if ".OPTIMIZER_SLOT" in name:
            idx = name.index(".OPTIMIZER_SLOT")
            name = name[:idx]
        elif ".ATTRIBUTES" in name:
            idx = name.index(".ATTRIBUTES")
            name = name[:idx]
        print(name)
        pointer = model
        for m_name in name:
            if re.fullmatch(r"[A-Za-z]+_\d+", m_name):
                scope_names = re.split(r"_(\d+)", m_name)
            else:
                scope_names = [m_name]
            if scope_names[0] == "kernel" or scope_names[0] == "gamma":
                pointer = getattr(pointer, "weight")
            elif scope_names[0] == "output_bias" or scope_names[0] == "beta":
                pointer = getattr(pointer, "bias")
            elif scope_names[0] == "output_weights":
                pointer = getattr(pointer, "weight")
            elif scope_names[0] == "squad":
                pointer = getattr(pointer, "classifier")
            elif scope_names[0] == "dense_output" or scope_names[0] == "bert_output":
                pointer = getattr(pointer, "output")
            elif scope_names[0] == "self_attention":
                pointer = getattr(pointer, "self")
            else:
                try:
                    pointer = getattr(pointer, scope_names[0])
                except AttributeError:
                    logger.info("Skipping {}".format("/".join(name)))
                    continue
            if len(scope_names) >= 2:
                num = int(scope_names[1])
                pointer = pointer[num]
        if m_name[-11:] == "_embeddings":
            pointer = getattr(pointer, "weight")
        elif m_name == "kernel" or m_name == "gamma" or m_name == "output_weights":
            array = np.transpose(array)
        # try:
        #     if pointer.shape != array.shape:
        #         raise ValueError(f"Pointer shape {pointer.shape} and array shape {array.shape} mismatched")
        # except AssertionError as e:
        #     e.args += (pointer.shape, array.shape)
        #     raise
        logger.info(f"Initialize PyTorch weight {name}")
        pointer.data = torch.from_numpy(array)
    return model
```

For convert_tf_to_pytorch function, I used below.

```
import argparse
import os
import torch

from transformers import BertConfig, BertForPreTraining, load_tf_weights_in_bert
from transformers.utils import logging

logging.set_verbosity_info()
def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, bert_config_file, pytorch_dump_path):
    # Initialise PyTorch model
    config = BertConfig.from_json_file(bert_config_file)
    print(f"Building PyTorch model from configuration: {config}")
    model = BertForPreTraining(config)
    # Load weights from tf checkpoint
    load_tf_weights_in_bert(model, config, tf_checkpoint_path)
    # Save pytorch-model
    os.makedirs(pytorch_dump_path)
    pytorch_dump_path = os.path.join(pytorch_dump_path, '0')
    print(f"Save PyTorch model to {pytorch_dump_path}")
    torch.save(model.state_dict(), pytorch_dump_path)
if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    # Required parameters
    parser.add_argument(
        "--tf_checkpoint_path", default=None, type=str, required=True, help="Path to the TensorFlow checkpoint path."
    )
    parser.add_argument(
        "--bert_config_file",
        default=None,
        type=str,
        required=True,
        help="The config json file corresponding to the pre-trained BERT model. \n"
        "This specifies the model architecture.",
    )
    parser.add_argument(
        "--pytorch_dump_path", default=None, type=str, required=True, help="Path to the output PyTorch model."
    )
    args = parser.parse_args()
    convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.bert_config_file, args.pytorch_dump_path)
```

Hope this help!

I use @Jwmc999 's method and successfully  converted tensoeflow 2.x ckpt file to Pytorch.bin.
Check the below jupyterbook if needed.
[Converting-TF-ckpt-to-Pytorch-model](https://github.com/suchunxie/Converting-TF-ckpt-to-Pytorch-model)