Embedding' object has no attribute 'shape' 
Author: Dhanachandra
Pull request: no
Labels: name: wontfix

While running the script to convert the Tensorflow checkpoints to Pytorch Model. 
Model path: https://github.com/naver/biobert-pretrained/releases/download/v1.0-pubmed-pmc/biobert_pubmed_pmc.tar.gz
python pytorch_pretrained_BERT/pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py \
    --tf_checkpoint_path pubmed_pmc_470k/biobert_model.ckpt \
    --bert_config_file pubmed_pmc_470k/bert_config.json \
    --pytorch_dump_path pytorch_model

Comments:
@Dhanachandra Same issue with another pre-trained BERT model.
Have you managed to solve that?

> @Dhanachandra Same issue with another pre-trained BERT model.
> Have you managed to solve that?

@Dhanachandra I've just found the solution: this exception occurs at "modeling.py" module. It's because of tensorflow->pytorch transformation. You need to find rows where 'shape' is used in modeling.py (you'll see its path in error logs) and delete it (it's somewhere in try... assert ... except..., just delete it).

You can use the following code: 

tf_path = 'pubmed_pmc_470k/biobert_model.ckpt'
config_path = 'pubmed_pmc_470k/bert_config.json'
pytorch_dump_path = 'pytorch_model/pytorch_model.bin'
# Save pytorch-model

import os
import re
import argparse
import tensorflow as tf
import torch
import numpy as np

from pytorch_pretrained_bert import BertConfig, BertForPreTraining

def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, bert_config_file, pytorch_dump_path):
    config_path = os.path.abspath(bert_config_file)
    tf_path = os.path.abspath(tf_checkpoint_path)
    print("Converting TensorFlow checkpoint from {} with config at {}".format(tf_path, config_path))
    # Load weights from TF model
    init_vars = tf.train.list_variables(tf_path)
    excluded = ['BERTAdam','_power','global_step']
    init_vars = list(filter(lambda x:all([True if e not in x[0] else False for e in excluded]),init_vars))
    names = []
    arrays = []
    for name, shape in init_vars:
        print("Loading TF weight {} with shape {}".format(name, shape))
        array = tf.train.load_variable(tf_path, name)
        names.append(name)
        arrays.append(array)

    # Initialise PyTorch model
    config = BertConfig.from_json_file(bert_config_file)
    print("Building PyTorch model from configuration: {}".format(str(config)))
    model = BertForPreTraining(config)

    for name, array in zip(names, arrays):
        name = name.split('/')
        # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
        # which are not required for using pretrained model
        if any(n in ["adam_v", "adam_m", "global_step"] for n in name):
            print("Skipping {}".format("/".join(name)))
            continue
        pointer = model
        for m_name in name:
            if re.fullmatch(r'[A-Za-z]+_\d+', m_name):
                l = re.split(r'_(\d+)', m_name)
            else:
                l = [m_name]
            if l[0] == 'kernel' or l[0] == 'gamma':
                pointer = getattr(pointer, 'weight')
            elif l[0] == 'output_bias' or l[0] == 'beta':
                pointer = getattr(pointer, 'bias')
            elif l[0] == 'output_weights':
                pointer = getattr(pointer, 'weight')
            else:
                pointer = getattr(pointer, l[0])
            if len(l) >= 2:
                num = int(l[1])
                pointer = pointer[num]
        if m_name[-11:] == '_embeddings':
            pointer = getattr(pointer, 'weight')
        elif m_name == 'kernel':
            array = np.transpose(array)
        try:
            assert pointer.shape == array.shape
        except AssertionError as e:
            e.args += (pointer.shape, array.shape)
            raise
        print("Initialize PyTorch weight {}".format(name))
        pointer.data = torch.from_numpy(array)

    # Save pytorch-model
    print("Save PyTorch model to {}".format(pytorch_dump_path))
    torch.save(model.state_dict(), pytorch_dump_path)

convert_tf_checkpoint_to_pytorch(tf_path, config_path, pytorch_dump_path) `

@DmLitov4 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.