Load Pretrained Model Error in Inherit Class
Author: yangzhch6
Pull request: no
Labels: name: wontfix

name: PyTorch

name: Core: Modeling

# ❓ Questions & Help
I wrote a new class based on RoBerta Model(pytorch)
## Details
The code is shown below

```
import torch 
import numpy as np 
import logging

import torch
from torch import nn
from torch.autograd import Variable
from torch.nn import CrossEntropyLoss
import torch.nn.functional as F

from transformers import BertPreTrainedModel
from transformers import RobertaConfig, RobertaTokenizer, RobertaModel
# from transformers.modeling_bert import BertEmbeddings

logging.basicConfig(format='%(asctime)s - %(levelname)s - %(name)s -   %(message)s',
                    datefmt='%m/%d/%Y %H:%M:%S',
                    level=logging.INFO)
logger = logging.getLogger(__name__)

class RoBertaMultiwayMatch(BertPreTrainedModel):
    def __init__(self, config, num_choices=4):
        super(RoBertaMultiwayMatch, self).__init__(config)
        self.num_choices = num_choices
        self.RoBerta = RobertaModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.linear_trans = nn.Linear(config.hidden_size, config.hidden_size)
        self.linear_fuse_p = nn.Linear(config.hidden_size*2, config.hidden_size)
        self.linear_fuse_q = nn.Linear(config.hidden_size*2, config.hidden_size)
        self.linear_fuse_a = nn.Linear(config.hidden_size * 2, config.hidden_size)
        self.classifier = nn.Linear(config.hidden_size*3, 1)
        self.init_weights()

    def matching(self, passage_encoded, question_encoded, passage_attention_mask, question_attention_mask): ...
    def fusing_mlp(self, passage_encoded, mp_q, mp_a, mp_qa, question_encoded, ...
    def forward(self, input_ids, token_type_ids=None, attention_mask=None, doc_len=None, ...


if __name__ == "__main__":
    # tokenizer = RobertaTokenizer.from_pretrained('roberta-large', do_lower_case=True)
    model = RoBertaMultiwayMatch.from_pretrained('/data3/yangzhicheng/Data/RoBerta/roberta-large/', num_choices=4)
```

But the logger indicate that some weights are not initialized:
```

02/18/2020 15:24:00 - INFO - transformers.modeling_utils -   loading weights file /data3/yangzhicheng/Data/RoBerta/roberta-large/pytorch_model.bin 
02/18/2020 15:24:29 - INFO - transformers.modeling_utils -   Weights of RoBertaMultiwayMatch not initialized from pretrained model: ['roberta.RoBerta.embeddings.word_embeddings.weight', 'roberta.RoBerta.embeddings.position_embeddings.weight', 'roberta.RoBerta.embeddings.token_type_embeddings.weight', 'roberta.RoBerta.embeddings.LayerNorm.weight', 'roberta.RoBerta.embeddings.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.0.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.0.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.0.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.0.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.0.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.0.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.0.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.0.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.0.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.0.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.0.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.0.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.0.output.dense.weight', 'roberta.RoBerta.encoder.layer.0.output.dense.bias', 'roberta.RoBerta.encoder.layer.0.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.0.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.1.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.1.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.1.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.1.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.1.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.1.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.1.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.1.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.1.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.1.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.1.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.1.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.1.output.dense.weight', 'roberta.RoBerta.encoder.layer.1.output.dense.bias', 'roberta.RoBerta.encoder.layer.1.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.1.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.2.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.2.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.2.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.2.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.2.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.2.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.2.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.2.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.2.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.2.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.2.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.2.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.2.output.dense.weight', 'roberta.RoBerta.encoder.layer.2.output.dense.bias', 'roberta.RoBerta.encoder.layer.2.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.2.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.3.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.3.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.3.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.3.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.3.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.3.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.3.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.3.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.3.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.3.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.3.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.3.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.3.output.dense.weight', 'roberta.RoBerta.encoder.layer.3.output.dense.bias', 'roberta.RoBerta.encoder.layer.3.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.3.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.4.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.4.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.4.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.4.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.4.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.4.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.4.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.4.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.4.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.4.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.4.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.4.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.4.output.dense.weight', 'roberta.RoBerta.encoder.layer.4.output.dense.bias', 'roberta.RoBerta.encoder.layer.4.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.4.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.5.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.5.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.5.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.5.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.5.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.5.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.5.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.5.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.5.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.5.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.5.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.5.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.5.output.dense.weight', 'roberta.RoBerta.encoder.layer.5.output.dense.bias', 'roberta.RoBerta.encoder.layer.5.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.5.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.6.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.6.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.6.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.6.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.6.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.6.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.6.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.6.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.6.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.6.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.6.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.6.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.6.output.dense.weight', 'roberta.RoBerta.encoder.layer.6.output.dense.bias', 'roberta.RoBerta.encoder.layer.6.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.6.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.7.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.7.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.7.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.7.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.7.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.7.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.7.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.7.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.7.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.7.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.7.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.7.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.7.output.dense.weight', 'roberta.RoBerta.encoder.layer.7.output.dense.bias', 'roberta.RoBerta.encoder.layer.7.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.7.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.8.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.8.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.8.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.8.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.8.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.8.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.8.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.8.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.8.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.8.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.8.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.8.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.8.output.dense.weight', 'roberta.RoBerta.encoder.layer.8.output.dense.bias', 'roberta.RoBerta.encoder.layer.8.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.8.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.9.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.9.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.9.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.9.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.9.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.9.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.9.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.9.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.9.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.9.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.9.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.9.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.9.output.dense.weight', 'roberta.RoBerta.encoder.layer.9.output.dense.bias', 'roberta.RoBerta.encoder.layer.9.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.9.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.10.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.10.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.10.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.10.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.10.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.10.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.10.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.10.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.10.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.10.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.10.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.10.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.10.output.dense.weight', 'roberta.RoBerta.encoder.layer.10.output.dense.bias', 'roberta.RoBerta.encoder.layer.10.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.10.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.11.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.11.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.11.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.11.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.11.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.11.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.11.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.11.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.11.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.11.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.11.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.11.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.11.output.dense.weight', 'roberta.RoBerta.encoder.layer.11.output.dense.bias', 'roberta.RoBerta.encoder.layer.11.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.11.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.12.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.12.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.12.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.12.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.12.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.12.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.12.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.12.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.12.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.12.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.12.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.12.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.12.output.dense.weight', 'roberta.RoBerta.encoder.layer.12.output.dense.bias', 'roberta.RoBerta.encoder.layer.12.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.12.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.13.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.13.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.13.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.13.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.13.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.13.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.13.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.13.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.13.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.13.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.13.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.13.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.13.output.dense.weight', 'roberta.RoBerta.encoder.layer.13.output.dense.bias', 'roberta.RoBerta.encoder.layer.13.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.13.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.14.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.14.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.14.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.14.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.14.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.14.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.14.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.14.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.14.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.14.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.14.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.14.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.14.output.dense.weight', 'roberta.RoBerta.encoder.layer.14.output.dense.bias', 'roberta.RoBerta.encoder.layer.14.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.14.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.15.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.15.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.15.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.15.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.15.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.15.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.15.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.15.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.15.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.15.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.15.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.15.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.15.output.dense.weight', 'roberta.RoBerta.encoder.layer.15.output.dense.bias', 'roberta.RoBerta.encoder.layer.15.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.15.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.16.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.16.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.16.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.16.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.16.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.16.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.16.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.16.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.16.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.16.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.16.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.16.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.16.output.dense.weight', 'roberta.RoBerta.encoder.layer.16.output.dense.bias', 'roberta.RoBerta.encoder.layer.16.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.16.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.17.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.17.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.17.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.17.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.17.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.17.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.17.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.17.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.17.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.17.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.17.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.17.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.17.output.dense.weight', 'roberta.RoBerta.encoder.layer.17.output.dense.bias', 'roberta.RoBerta.encoder.layer.17.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.17.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.18.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.18.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.18.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.18.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.18.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.18.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.18.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.18.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.18.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.18.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.18.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.18.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.18.output.dense.weight', 'roberta.RoBerta.encoder.layer.18.output.dense.bias', 'roberta.RoBerta.encoder.layer.18.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.18.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.19.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.19.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.19.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.19.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.19.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.19.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.19.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.19.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.19.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.19.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.19.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.19.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.19.output.dense.weight', 'roberta.RoBerta.encoder.layer.19.output.dense.bias', 'roberta.RoBerta.encoder.layer.19.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.19.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.20.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.20.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.20.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.20.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.20.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.20.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.20.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.20.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.20.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.20.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.20.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.20.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.20.output.dense.weight', 'roberta.RoBerta.encoder.layer.20.output.dense.bias', 'roberta.RoBerta.encoder.layer.20.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.20.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.21.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.21.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.21.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.21.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.21.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.21.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.21.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.21.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.21.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.21.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.21.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.21.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.21.output.dense.weight', 'roberta.RoBerta.encoder.layer.21.output.dense.bias', 'roberta.RoBerta.encoder.layer.21.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.21.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.22.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.22.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.22.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.22.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.22.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.22.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.22.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.22.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.22.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.22.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.22.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.22.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.22.output.dense.weight', 'roberta.RoBerta.encoder.layer.22.output.dense.bias', 'roberta.RoBerta.encoder.layer.22.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.22.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.23.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.23.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.23.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.23.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.23.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.23.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.23.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.23.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.23.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.23.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.23.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.23.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.23.output.dense.weight', 'roberta.RoBerta.encoder.layer.23.output.dense.bias', 'roberta.RoBerta.encoder.layer.23.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.23.output.LayerNorm.bias', 'roberta.RoBerta.pooler.dense.weight', 'roberta.RoBerta.pooler.dense.bias', 'roberta.linear_trans.weight', 'roberta.linear_trans.bias', 'roberta.linear_fuse_p.weight', 'roberta.linear_fuse_p.bias', 'roberta.linear_fuse_q.weight', 'roberta.linear_fuse_q.bias', 'roberta.linear_fuse_a.weight', 'roberta.linear_fuse_a.bias', 'roberta.classifier.weight', 'roberta.classifier.bias']
02/18/2020 15:24:29 - INFO - transformers.modeling_utils -   Weights from pretrained model not used in RoBertaMultiwayMatch: ['roberta.embeddings.word_embeddings.weight', 'roberta.embeddings.position_embeddings.weight', 'roberta.embeddings.token_type_embeddings.weight', 'roberta.embeddings.LayerNorm.weight', 'roberta.embeddings.LayerNorm.bias', 'roberta.encoder.layer.0.attention.self.query.weight', 'roberta.encoder.layer.0.attention.self.query.bias', 'roberta.encoder.layer.0.attention.self.key.weight', 'roberta.encoder.layer.0.attention.self.key.bias', 'roberta.encoder.layer.0.attention.self.value.weight', 'roberta.encoder.layer.0.attention.self.value.bias', 'roberta.encoder.layer.0.attention.output.dense.weight', 'roberta.encoder.layer.0.attention.output.dense.bias', 'roberta.encoder.layer.0.attention.output.LayerNorm.weight', 'roberta.encoder.layer.0.attention.output.LayerNorm.bias', 'roberta.encoder.layer.0.intermediate.dense.weight', 'roberta.encoder.layer.0.intermediate.dense.bias', 'roberta.encoder.layer.0.output.dense.weight', 'roberta.encoder.layer.0.output.dense.bias', 'roberta.encoder.layer.0.output.LayerNorm.weight', 'roberta.encoder.layer.0.output.LayerNorm.bias', 'roberta.encoder.layer.1.attention.self.query.weight', 'roberta.encoder.layer.1.attention.self.query.bias', 'roberta.encoder.layer.1.attention.self.key.weight', 'roberta.encoder.layer.1.attention.self.key.bias', 'roberta.encoder.layer.1.attention.self.value.weight', 'roberta.encoder.layer.1.attention.self.value.bias', 'roberta.encoder.layer.1.attention.output.dense.weight', 'roberta.encoder.layer.1.attention.output.dense.bias', 'roberta.encoder.layer.1.attention.output.LayerNorm.weight', 'roberta.encoder.layer.1.attention.output.LayerNorm.bias', 'roberta.encoder.layer.1.intermediate.dense.weight', 'roberta.encoder.layer.1.intermediate.dense.bias', 'roberta.encoder.layer.1.output.dense.weight', 'roberta.encoder.layer.1.output.dense.bias', 'roberta.encoder.layer.1.output.LayerNorm.weight', 'roberta.encoder.layer.1.output.LayerNorm.bias', 'roberta.encoder.layer.2.attention.self.query.weight', 'roberta.encoder.layer.2.attention.self.query.bias', 'roberta.encoder.layer.2.attention.self.key.weight', 'roberta.encoder.layer.2.attention.self.key.bias', 'roberta.encoder.layer.2.attention.self.value.weight', 'roberta.encoder.layer.2.attention.self.value.bias', 'roberta.encoder.layer.2.attention.output.dense.weight', 'roberta.encoder.layer.2.attention.output.dense.bias', 'roberta.encoder.layer.2.attention.output.LayerNorm.weight', 'roberta.encoder.layer.2.attention.output.LayerNorm.bias', 'roberta.encoder.layer.2.intermediate.dense.weight', 'roberta.encoder.layer.2.intermediate.dense.bias', 'roberta.encoder.layer.2.output.dense.weight', 'roberta.encoder.layer.2.output.dense.bias', 'roberta.encoder.layer.2.output.LayerNorm.weight', 'roberta.encoder.layer.2.output.LayerNorm.bias', 'roberta.encoder.layer.3.attention.self.query.weight', 'roberta.encoder.layer.3.attention.self.query.bias', 'roberta.encoder.layer.3.attention.self.key.weight', 'roberta.encoder.layer.3.attention.self.key.bias', 'roberta.encoder.layer.3.attention.self.value.weight', 'roberta.encoder.layer.3.attention.self.value.bias', 'roberta.encoder.layer.3.attention.output.dense.weight', 'roberta.encoder.layer.3.attention.output.dense.bias', 'roberta.encoder.layer.3.attention.output.LayerNorm.weight', 'roberta.encoder.layer.3.attention.output.LayerNorm.bias', 'roberta.encoder.layer.3.intermediate.dense.weight', 'roberta.encoder.layer.3.intermediate.dense.bias', 'roberta.encoder.layer.3.output.dense.weight', 'roberta.encoder.layer.3.output.dense.bias', 'roberta.encoder.layer.3.output.LayerNorm.weight', 'roberta.encoder.layer.3.output.LayerNorm.bias', 'roberta.encoder.layer.4.attention.self.query.weight', 'roberta.encoder.layer.4.attention.self.query.bias', 'roberta.encoder.layer.4.attention.self.key.weight', 'roberta.encoder.layer.4.attention.self.key.bias', 'roberta.encoder.layer.4.attention.self.value.weight', 'roberta.encoder.layer.4.attention.self.value.bias', 'roberta.encoder.layer.4.attention.output.dense.weight', 'roberta.encoder.layer.4.attention.output.dense.bias', 'roberta.encoder.layer.4.attention.output.LayerNorm.weight', 'roberta.encoder.layer.4.attention.output.LayerNorm.bias', 'roberta.encoder.layer.4.intermediate.dense.weight', 'roberta.encoder.layer.4.intermediate.dense.bias', 'roberta.encoder.layer.4.output.dense.weight', 'roberta.encoder.layer.4.output.dense.bias', 'roberta.encoder.layer.4.output.LayerNorm.weight', 'roberta.encoder.layer.4.output.LayerNorm.bias', 'roberta.encoder.layer.5.attention.self.query.weight', 'roberta.encoder.layer.5.attention.self.query.bias', 'roberta.encoder.layer.5.attention.self.key.weight', 'roberta.encoder.layer.5.attention.self.key.bias', 'roberta.encoder.layer.5.attention.self.value.weight', 'roberta.encoder.layer.5.attention.self.value.bias', 'roberta.encoder.layer.5.attention.output.dense.weight', 'roberta.encoder.layer.5.attention.output.dense.bias', 'roberta.encoder.layer.5.attention.output.LayerNorm.weight', 'roberta.encoder.layer.5.attention.output.LayerNorm.bias', 'roberta.encoder.layer.5.intermediate.dense.weight', 'roberta.encoder.layer.5.intermediate.dense.bias', 'roberta.encoder.layer.5.output.dense.weight', 'roberta.encoder.layer.5.output.dense.bias', 'roberta.encoder.layer.5.output.LayerNorm.weight', 'roberta.encoder.layer.5.output.LayerNorm.bias', 'roberta.encoder.layer.6.attention.self.query.weight', 'roberta.encoder.layer.6.attention.self.query.bias', 'roberta.encoder.layer.6.attention.self.key.weight', 'roberta.encoder.layer.6.attention.self.key.bias', 'roberta.encoder.layer.6.attention.self.value.weight', 'roberta.encoder.layer.6.attention.self.value.bias', 'roberta.encoder.layer.6.attention.output.dense.weight', 'roberta.encoder.layer.6.attention.output.dense.bias', 'roberta.encoder.layer.6.attention.output.LayerNorm.weight', 'roberta.encoder.layer.6.attention.output.LayerNorm.bias', 'roberta.encoder.layer.6.intermediate.dense.weight', 'roberta.encoder.layer.6.intermediate.dense.bias', 'roberta.encoder.layer.6.output.dense.weight', 'roberta.encoder.layer.6.output.dense.bias', 'roberta.encoder.layer.6.output.LayerNorm.weight', 'roberta.encoder.layer.6.output.LayerNorm.bias', 'roberta.encoder.layer.7.attention.self.query.weight', 'roberta.encoder.layer.7.attention.self.query.bias', 'roberta.encoder.layer.7.attention.self.key.weight', 'roberta.encoder.layer.7.attention.self.key.bias', 'roberta.encoder.layer.7.attention.self.value.weight', 'roberta.encoder.layer.7.attention.self.value.bias', 'roberta.encoder.layer.7.attention.output.dense.weight', 'roberta.encoder.layer.7.attention.output.dense.bias', 'roberta.encoder.layer.7.attention.output.LayerNorm.weight', 'roberta.encoder.layer.7.attention.output.LayerNorm.bias', 'roberta.encoder.layer.7.intermediate.dense.weight', 'roberta.encoder.layer.7.intermediate.dense.bias', 'roberta.encoder.layer.7.output.dense.weight', 'roberta.encoder.layer.7.output.dense.bias', 'roberta.encoder.layer.7.output.LayerNorm.weight', 'roberta.encoder.layer.7.output.LayerNorm.bias', 'roberta.encoder.layer.8.attention.self.query.weight', 'roberta.encoder.layer.8.attention.self.query.bias', 'roberta.encoder.layer.8.attention.self.key.weight', 'roberta.encoder.layer.8.attention.self.key.bias', 'roberta.encoder.layer.8.attention.self.value.weight', 'roberta.encoder.layer.8.attention.self.value.bias', 'roberta.encoder.layer.8.attention.output.dense.weight', 'roberta.encoder.layer.8.attention.output.dense.bias', 'roberta.encoder.layer.8.attention.output.LayerNorm.weight', 'roberta.encoder.layer.8.attention.output.LayerNorm.bias', 'roberta.encoder.layer.8.intermediate.dense.weight', 'roberta.encoder.layer.8.intermediate.dense.bias', 'roberta.encoder.layer.8.output.dense.weight', 'roberta.encoder.layer.8.output.dense.bias', 'roberta.encoder.layer.8.output.LayerNorm.weight', 'roberta.encoder.layer.8.output.LayerNorm.bias', 'roberta.encoder.layer.9.attention.self.query.weight', 'roberta.encoder.layer.9.attention.self.query.bias', 'roberta.encoder.layer.9.attention.self.key.weight', 'roberta.encoder.layer.9.attention.self.key.bias', 'roberta.encoder.layer.9.attention.self.value.weight', 'roberta.encoder.layer.9.attention.self.value.bias', 'roberta.encoder.layer.9.attention.output.dense.weight', 'roberta.encoder.layer.9.attention.output.dense.bias', 'roberta.encoder.layer.9.attention.output.LayerNorm.weight', 'roberta.encoder.layer.9.attention.output.LayerNorm.bias', 'roberta.encoder.layer.9.intermediate.dense.weight', 'roberta.encoder.layer.9.intermediate.dense.bias', 'roberta.encoder.layer.9.output.dense.weight', 'roberta.encoder.layer.9.output.dense.bias', 'roberta.encoder.layer.9.output.LayerNorm.weight', 'roberta.encoder.layer.9.output.LayerNorm.bias', 'roberta.encoder.layer.10.attention.self.query.weight', 'roberta.encoder.layer.10.attention.self.query.bias', 'roberta.encoder.layer.10.attention.self.key.weight', 'roberta.encoder.layer.10.attention.self.key.bias', 'roberta.encoder.layer.10.attention.self.value.weight', 'roberta.encoder.layer.10.attention.self.value.bias', 'roberta.encoder.layer.10.attention.output.dense.weight', 'roberta.encoder.layer.10.attention.output.dense.bias', 'roberta.encoder.layer.10.attention.output.LayerNorm.weight', 'roberta.encoder.layer.10.attention.output.LayerNorm.bias', 'roberta.encoder.layer.10.intermediate.dense.weight', 'roberta.encoder.layer.10.intermediate.dense.bias', 'roberta.encoder.layer.10.output.dense.weight', 'roberta.encoder.layer.10.output.dense.bias', 'roberta.encoder.layer.10.output.LayerNorm.weight', 'roberta.encoder.layer.10.output.LayerNorm.bias', 'roberta.encoder.layer.11.attention.self.query.weight', 'roberta.encoder.layer.11.attention.self.query.bias', 'roberta.encoder.layer.11.attention.self.key.weight', 'roberta.encoder.layer.11.attention.self.key.bias', 'roberta.encoder.layer.11.attention.self.value.weight', 'roberta.encoder.layer.11.attention.self.value.bias', 'roberta.encoder.layer.11.attention.output.dense.weight', 'roberta.encoder.layer.11.attention.output.dense.bias', 'roberta.encoder.layer.11.attention.output.LayerNorm.weight', 'roberta.encoder.layer.11.attention.output.LayerNorm.bias', 'roberta.encoder.layer.11.intermediate.dense.weight', 'roberta.encoder.layer.11.intermediate.dense.bias', 'roberta.encoder.layer.11.output.dense.weight', 'roberta.encoder.layer.11.output.dense.bias', 'roberta.encoder.layer.11.output.LayerNorm.weight', 'roberta.encoder.layer.11.output.LayerNorm.bias', 'roberta.encoder.layer.12.attention.self.query.weight', 'roberta.encoder.layer.12.attention.self.query.bias', 'roberta.encoder.layer.12.attention.self.key.weight', 'roberta.encoder.layer.12.attention.self.key.bias', 'roberta.encoder.layer.12.attention.self.value.weight', 'roberta.encoder.layer.12.attention.self.value.bias', 'roberta.encoder.layer.12.attention.output.dense.weight', 'roberta.encoder.layer.12.attention.output.dense.bias', 'roberta.encoder.layer.12.attention.output.LayerNorm.weight', 'roberta.encoder.layer.12.attention.output.LayerNorm.bias', 'roberta.encoder.layer.12.intermediate.dense.weight', 'roberta.encoder.layer.12.intermediate.dense.bias', 'roberta.encoder.layer.12.output.dense.weight', 'roberta.encoder.layer.12.output.dense.bias', 'roberta.encoder.layer.12.output.LayerNorm.weight', 'roberta.encoder.layer.12.output.LayerNorm.bias', 'roberta.encoder.layer.13.attention.self.query.weight', 'roberta.encoder.layer.13.attention.self.query.bias', 'roberta.encoder.layer.13.attention.self.key.weight', 'roberta.encoder.layer.13.attention.self.key.bias', 'roberta.encoder.layer.13.attention.self.value.weight', 'roberta.encoder.layer.13.attention.self.value.bias', 'roberta.encoder.layer.13.attention.output.dense.weight', 'roberta.encoder.layer.13.attention.output.dense.bias', 'roberta.encoder.layer.13.attention.output.LayerNorm.weight', 'roberta.encoder.layer.13.attention.output.LayerNorm.bias', 'roberta.encoder.layer.13.intermediate.dense.weight', 'roberta.encoder.layer.13.intermediate.dense.bias', 'roberta.encoder.layer.13.output.dense.weight', 'roberta.encoder.layer.13.output.dense.bias', 'roberta.encoder.layer.13.output.LayerNorm.weight', 'roberta.encoder.layer.13.output.LayerNorm.bias', 'roberta.encoder.layer.14.attention.self.query.weight', 'roberta.encoder.layer.14.attention.self.query.bias', 'roberta.encoder.layer.14.attention.self.key.weight', 'roberta.encoder.layer.14.attention.self.key.bias', 'roberta.encoder.layer.14.attention.self.value.weight', 'roberta.encoder.layer.14.attention.self.value.bias', 'roberta.encoder.layer.14.attention.output.dense.weight', 'roberta.encoder.layer.14.attention.output.dense.bias', 'roberta.encoder.layer.14.attention.output.LayerNorm.weight', 'roberta.encoder.layer.14.attention.output.LayerNorm.bias', 'roberta.encoder.layer.14.intermediate.dense.weight', 'roberta.encoder.layer.14.intermediate.dense.bias', 'roberta.encoder.layer.14.output.dense.weight', 'roberta.encoder.layer.14.output.dense.bias', 'roberta.encoder.layer.14.output.LayerNorm.weight', 'roberta.encoder.layer.14.output.LayerNorm.bias', 'roberta.encoder.layer.15.attention.self.query.weight', 'roberta.encoder.layer.15.attention.self.query.bias', 'roberta.encoder.layer.15.attention.self.key.weight', 'roberta.encoder.layer.15.attention.self.key.bias', 'roberta.encoder.layer.15.attention.self.value.weight', 'roberta.encoder.layer.15.attention.self.value.bias', 'roberta.encoder.layer.15.attention.output.dense.weight', 'roberta.encoder.layer.15.attention.output.dense.bias', 'roberta.encoder.layer.15.attention.output.LayerNorm.weight', 'roberta.encoder.layer.15.attention.output.LayerNorm.bias', 'roberta.encoder.layer.15.intermediate.dense.weight', 'roberta.encoder.layer.15.intermediate.dense.bias', 'roberta.encoder.layer.15.output.dense.weight', 'roberta.encoder.layer.15.output.dense.bias', 'roberta.encoder.layer.15.output.LayerNorm.weight', 'roberta.encoder.layer.15.output.LayerNorm.bias', 'roberta.encoder.layer.16.attention.self.query.weight', 'roberta.encoder.layer.16.attention.self.query.bias', 'roberta.encoder.layer.16.attention.self.key.weight', 'roberta.encoder.layer.16.attention.self.key.bias', 'roberta.encoder.layer.16.attention.self.value.weight', 'roberta.encoder.layer.16.attention.self.value.bias', 'roberta.encoder.layer.16.attention.output.dense.weight', 'roberta.encoder.layer.16.attention.output.dense.bias', 'roberta.encoder.layer.16.attention.output.LayerNorm.weight', 'roberta.encoder.layer.16.attention.output.LayerNorm.bias', 'roberta.encoder.layer.16.intermediate.dense.weight', 'roberta.encoder.layer.16.intermediate.dense.bias', 'roberta.encoder.layer.16.output.dense.weight', 'roberta.encoder.layer.16.output.dense.bias', 'roberta.encoder.layer.16.output.LayerNorm.weight', 'roberta.encoder.layer.16.output.LayerNorm.bias', 'roberta.encoder.layer.17.attention.self.query.weight', 'roberta.encoder.layer.17.attention.self.query.bias', 'roberta.encoder.layer.17.attention.self.key.weight', 'roberta.encoder.layer.17.attention.self.key.bias', 'roberta.encoder.layer.17.attention.self.value.weight', 'roberta.encoder.layer.17.attention.self.value.bias', 'roberta.encoder.layer.17.attention.output.dense.weight', 'roberta.encoder.layer.17.attention.output.dense.bias', 'roberta.encoder.layer.17.attention.output.LayerNorm.weight', 'roberta.encoder.layer.17.attention.output.LayerNorm.bias', 'roberta.encoder.layer.17.intermediate.dense.weight', 'roberta.encoder.layer.17.intermediate.dense.bias', 'roberta.encoder.layer.17.output.dense.weight', 'roberta.encoder.layer.17.output.dense.bias', 'roberta.encoder.layer.17.output.LayerNorm.weight', 'roberta.encoder.layer.17.output.LayerNorm.bias', 'roberta.encoder.layer.18.attention.self.query.weight', 'roberta.encoder.layer.18.attention.self.query.bias', 'roberta.encoder.layer.18.attention.self.key.weight', 'roberta.encoder.layer.18.attention.self.key.bias', 'roberta.encoder.layer.18.attention.self.value.weight', 'roberta.encoder.layer.18.attention.self.value.bias', 'roberta.encoder.layer.18.attention.output.dense.weight', 'roberta.encoder.layer.18.attention.output.dense.bias', 'roberta.encoder.layer.18.attention.output.LayerNorm.weight', 'roberta.encoder.layer.18.attention.output.LayerNorm.bias', 'roberta.encoder.layer.18.intermediate.dense.weight', 'roberta.encoder.layer.18.intermediate.dense.bias', 'roberta.encoder.layer.18.output.dense.weight', 'roberta.encoder.layer.18.output.dense.bias', 'roberta.encoder.layer.18.output.LayerNorm.weight', 'roberta.encoder.layer.18.output.LayerNorm.bias', 'roberta.encoder.layer.19.attention.self.query.weight', 'roberta.encoder.layer.19.attention.self.query.bias', 'roberta.encoder.layer.19.attention.self.key.weight', 'roberta.encoder.layer.19.attention.self.key.bias', 'roberta.encoder.layer.19.attention.self.value.weight', 'roberta.encoder.layer.19.attention.self.value.bias', 'roberta.encoder.layer.19.attention.output.dense.weight', 'roberta.encoder.layer.19.attention.output.dense.bias', 'roberta.encoder.layer.19.attention.output.LayerNorm.weight', 'roberta.encoder.layer.19.attention.output.LayerNorm.bias', 'roberta.encoder.layer.19.intermediate.dense.weight', 'roberta.encoder.layer.19.intermediate.dense.bias', 'roberta.encoder.layer.19.output.dense.weight', 'roberta.encoder.layer.19.output.dense.bias', 'roberta.encoder.layer.19.output.LayerNorm.weight', 'roberta.encoder.layer.19.output.LayerNorm.bias', 'roberta.encoder.layer.20.attention.self.query.weight', 'roberta.encoder.layer.20.attention.self.query.bias', 'roberta.encoder.layer.20.attention.self.key.weight', 'roberta.encoder.layer.20.attention.self.key.bias', 'roberta.encoder.layer.20.attention.self.value.weight', 'roberta.encoder.layer.20.attention.self.value.bias', 'roberta.encoder.layer.20.attention.output.dense.weight', 'roberta.encoder.layer.20.attention.output.dense.bias', 'roberta.encoder.layer.20.attention.output.LayerNorm.weight', 'roberta.encoder.layer.20.attention.output.LayerNorm.bias', 'roberta.encoder.layer.20.intermediate.dense.weight', 'roberta.encoder.layer.20.intermediate.dense.bias', 'roberta.encoder.layer.20.output.dense.weight', 'roberta.encoder.layer.20.output.dense.bias', 'roberta.encoder.layer.20.output.LayerNorm.weight', 'roberta.encoder.layer.20.output.LayerNorm.bias', 'roberta.encoder.layer.21.attention.self.query.weight', 'roberta.encoder.layer.21.attention.self.query.bias', 'roberta.encoder.layer.21.attention.self.key.weight', 'roberta.encoder.layer.21.attention.self.key.bias', 'roberta.encoder.layer.21.attention.self.value.weight', 'roberta.encoder.layer.21.attention.self.value.bias', 'roberta.encoder.layer.21.attention.output.dense.weight', 'roberta.encoder.layer.21.attention.output.dense.bias', 'roberta.encoder.layer.21.attention.output.LayerNorm.weight', 'roberta.encoder.layer.21.attention.output.LayerNorm.bias', 'roberta.encoder.layer.21.intermediate.dense.weight', 'roberta.encoder.layer.21.intermediate.dense.bias', 'roberta.encoder.layer.21.output.dense.weight', 'roberta.encoder.layer.21.output.dense.bias', 'roberta.encoder.layer.21.output.LayerNorm.weight', 'roberta.encoder.layer.21.output.LayerNorm.bias', 'roberta.encoder.layer.22.attention.self.query.weight', 'roberta.encoder.layer.22.attention.self.query.bias', 'roberta.encoder.layer.22.attention.self.key.weight', 'roberta.encoder.layer.22.attention.self.key.bias', 'roberta.encoder.layer.22.attention.self.value.weight', 'roberta.encoder.layer.22.attention.self.value.bias', 'roberta.encoder.layer.22.attention.output.dense.weight', 'roberta.encoder.layer.22.attention.output.dense.bias', 'roberta.encoder.layer.22.attention.output.LayerNorm.weight', 'roberta.encoder.layer.22.attention.output.LayerNorm.bias', 'roberta.encoder.layer.22.intermediate.dense.weight', 'roberta.encoder.layer.22.intermediate.dense.bias', 'roberta.encoder.layer.22.output.dense.weight', 'roberta.encoder.layer.22.output.dense.bias', 'roberta.encoder.layer.22.output.LayerNorm.weight', 'roberta.encoder.layer.22.output.LayerNorm.bias', 'roberta.encoder.layer.23.attention.self.query.weight', 'roberta.encoder.layer.23.attention.self.query.bias', 'roberta.encoder.layer.23.attention.self.key.weight', 'roberta.encoder.layer.23.attention.self.key.bias', 'roberta.encoder.layer.23.attention.self.value.weight', 'roberta.encoder.layer.23.attention.self.value.bias', 'roberta.encoder.layer.23.attention.output.dense.weight', 'roberta.encoder.layer.23.attention.output.dense.bias', 'roberta.encoder.layer.23.attention.output.LayerNorm.weight', 'roberta.encoder.layer.23.attention.output.LayerNorm.bias', 'roberta.encoder.layer.23.intermediate.dense.weight', 'roberta.encoder.layer.23.intermediate.dense.bias', 'roberta.encoder.layer.23.output.dense.weight', 'roberta.encoder.layer.23.output.dense.bias', 'roberta.encoder.layer.23.output.LayerNorm.weight', 'roberta.encoder.layer.23.output.LayerNorm.bias', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']
```

How can I solve this problem? 
Thanks in advance!

Comments:
```
class RoBertaMultiwayMatch(nn.Module):
    def __init__(self, pretrainedConfigName, num_choices=4):
        super(RoBertaMultiwayMatch, self).__init__()
        self.num_choices = num_choices
        self.RoBerta = RobertaModel.from_pretrained(pretrainedConfigName)
        config = self.RoBerta.config
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.linear_trans = nn.Linear(config.hidden_size, config.hidden_size)
        self.linear_fuse_p = nn.Linear(config.hidden_size*2, config.hidden_size)
        self.linear_fuse_q = nn.Linear(config.hidden_size*2, config.hidden_size)
        self.linear_fuse_a = nn.Linear(config.hidden_size * 2, config.hidden_size)
        self.classifier = nn.Linear(config.hidden_size*3, 1)

    #def matching(self, passage_encoded, question_encoded, passage_attention_mask, question_attention_mask): ...
    #def fusing_mlp(self, passage_encoded, mp_q, mp_a, mp_qa, question_encoded, ...
    #def forward(self, input_ids, token_type_ids=None, attention_mask=None, doc_len=None, ...
```


You're inheriting from `BertPreTrainedModel` in your `RoBertaMultiwayMatch`, with an attribute `RoBerta` which contains a `RobertaModel`.

As I see it, you want to load your roberta model from a given set of weights, but by calling `from_pretrained` on your class, he's looking to load those weights directly on your model.

I believe you could override the `from_pretrained` method as such:

```py
def from_pretrained(...):
    self.RoBerta.from_pretrained(...)
```

by specifying the correct arguments to your method and to your `RoBerta`'s method. This way when you call `from_pretrained`, it only loads the data for `RoBerta`. You'd have to find a way to save/load your data for your own layers though.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.