Would you like to add convert the generator script by ConvBert
Author: RyanHuangNLP
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

I excite to find that transformer add the support fort ConvBert, but I found that it just provide the script about convert the discriminator, would you like to support for convert the generator of Convbert like the Electra, I train both electra convbert and masklm convbert.

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
Pinging @abhishekkrthakur 

@abhishekkrthakur I have try to convert a mlm convbert to transformers one, this is the convert code

```
import torch
import os

import tensorflow as tf

from transformers import ConvBertConfig, ConvBertForMaskedLM, ConvBertPreTrainedModel
from transformers.utils import logging
from operator import attrgetter

logger = logging.get_logger(__name__)

config_file = "weights/convbert_base_mlm/config.json"
tf_path = "tf_weights/ft_local/model.ckpt-490000"
pytorch_dump_path = "weights/convbert_base_mlm"
config = ConvBertConfig.from_json_file(config_file)

#model = ConvBertPreTrainedModel(config)
model = ConvBertForMaskedLM(config)

def load_tf_weights_in_convbert(model, config, tf_checkpoint_path):
    """Load tf checkpoints in a pytorch model."""
    try:
        import tensorflow as tf
    except ImportError:
        logger.error(
            "Loading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see "
            "https://www.tensorflow.org/install/ for installation instructions."
        )
        raise
    tf_path = os.path.abspath(tf_checkpoint_path)
    logger.info("Converting TensorFlow checkpoint from {}".format(tf_path))
    # Load weights from TF model
    init_vars = tf.train.list_variables(tf_path)
    tf_data = {}
    for name, shape in init_vars:
        logger.info("Loading TF weight {} with shape {}".format(name, shape))
        array = tf.train.load_variable(tf_path, name)
        tf_data[name] = array

    param_mapping = {
        "convbert.embeddings.word_embeddings.weight": "electra/embeddings/word_embeddings",
        "convbert.embeddings.position_embeddings.weight": "electra/embeddings/position_embeddings",
        "convbert.embeddings.token_type_embeddings.weight": "electra/embeddings/token_type_embeddings",
        "convbert.embeddings.LayerNorm.weight": "electra/embeddings/LayerNorm/gamma",
        "convbert.embeddings.LayerNorm.bias": "electra/embeddings/LayerNorm/beta",
        "convbert.embeddings_project.weight": "electra/embeddings_project/kernel",
        "convbert.embeddings_project.bias": "electra/embeddings_project/bias",
        "generator_predictions.LayerNorm.weight": "generator_predictions/LayerNorm/gamma",
        "generator_predictions.LayerNorm.bias": "generator_predictions/LayerNorm/beta",
        "generator_predictions.dense.weight": "generator_predictions/dense/kernel",
        "generator_predictions.dense.bias": "generator_predictions/dense/bias",
        "generator_lm_head.bias": "generator_predictions/output_bias"
    }
    if config.num_groups > 1:
        group_dense_name = "g_dense"
    else:
        group_dense_name = "dense"

    for j in range(config.num_hidden_layers):
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.self.query.weight"
        ] = f"electra/encoder/layer_{j}/attention/self/query/kernel"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.self.query.bias"
        ] = f"electra/encoder/layer_{j}/attention/self/query/bias"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.self.key.weight"
        ] = f"electra/encoder/layer_{j}/attention/self/key/kernel"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.self.key.bias"
        ] = f"electra/encoder/layer_{j}/attention/self/key/bias"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.self.value.weight"
        ] = f"electra/encoder/layer_{j}/attention/self/value/kernel"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.self.value.bias"
        ] = f"electra/encoder/layer_{j}/attention/self/value/bias"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.self.key_conv_attn_layer.depthwise.weight"
        ] = f"electra/encoder/layer_{j}/attention/self/conv_attn_key/depthwise_kernel"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.self.key_conv_attn_layer.pointwise.weight"
        ] = f"electra/encoder/layer_{j}/attention/self/conv_attn_key/pointwise_kernel"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.self.key_conv_attn_layer.bias"
        ] = f"electra/encoder/layer_{j}/attention/self/conv_attn_key/bias"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.self.conv_kernel_layer.weight"
        ] = f"electra/encoder/layer_{j}/attention/self/conv_attn_kernel/kernel"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.self.conv_kernel_layer.bias"
        ] = f"electra/encoder/layer_{j}/attention/self/conv_attn_kernel/bias"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.self.conv_out_layer.weight"
        ] = f"electra/encoder/layer_{j}/attention/self/conv_attn_point/kernel"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.self.conv_out_layer.bias"
        ] = f"electra/encoder/layer_{j}/attention/self/conv_attn_point/bias"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.output.dense.weight"
        ] = f"electra/encoder/layer_{j}/attention/output/dense/kernel"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.output.LayerNorm.weight"
        ] = f"electra/encoder/layer_{j}/attention/output/LayerNorm/gamma"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.output.dense.bias"
        ] = f"electra/encoder/layer_{j}/attention/output/dense/bias"
        param_mapping[
            f"convbert.encoder.layer.{j}.attention.output.LayerNorm.bias"
        ] = f"electra/encoder/layer_{j}/attention/output/LayerNorm/beta"
        param_mapping[
            f"convbert.encoder.layer.{j}.intermediate.dense.weight"
        ] = f"electra/encoder/layer_{j}/intermediate/{group_dense_name}/kernel"
        param_mapping[
            f"convbert.encoder.layer.{j}.intermediate.dense.bias"
        ] = f"electra/encoder/layer_{j}/intermediate/{group_dense_name}/bias"
        param_mapping[
            f"convbert.encoder.layer.{j}.output.dense.weight"
        ] = f"electra/encoder/layer_{j}/output/{group_dense_name}/kernel"
        param_mapping[
            f"convbert.encoder.layer.{j}.output.dense.bias"
        ] = f"electra/encoder/layer_{j}/output/{group_dense_name}/bias"
        param_mapping[
            f"convbert.encoder.layer.{j}.output.LayerNorm.weight"
        ] = f"electra/encoder/layer_{j}/output/LayerNorm/gamma"
        param_mapping[f"convbert.encoder.layer.{j}.output.LayerNorm.bias"] = f"electra/encoder/layer_{j}/output/LayerNorm/beta"

    for param in model.named_parameters():
        param_name = param[0]
        retriever = attrgetter(param_name)
        result = retriever(model)
        tf_name = param_mapping[param_name]
        value = torch.from_numpy(tf_data[tf_name])
        logger.info(f"TF: {tf_name}, PT: {param_name} ")
        if tf_name.endswith("/kernel"):
            if not tf_name.endswith("/intermediate/g_dense/kernel"):
                if not tf_name.endswith("/output/g_dense/kernel"):
                    value = value.T
        if tf_name.endswith("/depthwise_kernel"):
            value = value.permute(1, 2, 0)  # 2, 0, 1
        if tf_name.endswith("/pointwise_kernel"):
            value = value.permute(2, 1, 0)  # 2, 1, 0
        if tf_name.endswith("/conv_attn_key/bias"):
            value = value.unsqueeze(-1)
        result.data = value
    return model

model = load_tf_weights_in_convbert(model, config, tf_path)
model.save_pretrained(pytorch_dump_path)
```

@RyanHuangNLP good idea! do you want to make a PR? Or should I fix it? 

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

@RyanHuangNLP I have a question regarding your script. I tried to extract the generator from the tf checkpoint, but it seems the size mismatch. I reduce the hidden_size by 4 (25%), as in the electra config file, and num_attention_head to 4. Does your script is currently converting the discriminator instead?

@Shiro-LK my script is not for the electra one, that is for mlm one, may be you should first convert the generator parameters names to the discriminator. It is important to check the parameter name

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.