Load Pretrained Model Error in Inherit Class

Core: Modeling

Internals of the library; Models.

PyTorch

Anything PyTorch

# ❓ Questions & Help
I wrote a new class based on RoBerta Model(pytorch)
## Details
The code is shown below

```
import torch
import numpy as np
import logging

import torch
from torch import nn
from torch.autograd import Variable
from torch.nn import CrossEntropyLoss
import torch.nn.functional as F

from transformers import BertPreTrainedModel
from transformers import RobertaConfig, RobertaTokenizer, RobertaModel
# from transformers.modeling_bert import BertEmbeddings

logging.basicConfig(format='%(asctime)s - %(levelname)s - %(name)s -   %(message)s',
                    datefmt='%m/%d/%Y %H:%M:%S',
                    level=logging.INFO)
logger = logging.getLogger(__name__)

class RoBertaMultiwayMatch(BertPreTrainedModel):
    def __init__(self, config, num_choices=4):
        super(RoBertaMultiwayMatch, self).__init__(config)
        self.num_choices = num_choices
        self.RoBerta = RobertaModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.linear_trans = nn.Linear(config.hidden_size, config.hidden_size)
        self.linear_fuse_p = nn.Linear(config.hidden_size*2, config.hidden_size)
        self.linear_fuse_q = nn.Linear(config.hidden_size*2, config.hidden_size)
        self.linear_fuse_a = nn.Linear(config.hidden_size * 2, config.hidden_size)
        self.classifier = nn.Linear(config.hidden_size*3, 1)
        self.init_weights()

    def matching(self, passage_encoded, question_encoded, passage_attention_mask, question_attention_mask): ...
    def fusing_mlp(self, passage_encoded, mp_q, mp_a, mp_qa, question_encoded, ...
    def forward(self, input_ids, token_type_ids=None, attention_mask=None, doc_len=None, ...

if __name__ == "__main__":
    # tokenizer = RobertaTokenizer.from_pretrained('roberta-large', do_lower_case=True)
    model = RoBertaMultiwayMatch.from_pretrained('/data3/yangzhicheng/Data/RoBerta/roberta-large/', num_choices=4)
```

But the logger indicate that some weights are not initialized:
```

02/18/2020 15:24:00 - INFO - transformers.modeling_utils -   loading weights file /data3/yangzhicheng/Data/RoBerta/roberta-large/pytorch_model.bin
02/18/2020 15:24:29 - INFO - transformers.modeling_utils -   Weights of RoBertaMultiwayMatch not initialized from pretrained model: ['roberta.RoBerta.embeddings.word_embeddings.weight', 'roberta.RoBerta.embeddings.position_embeddings.weight', 'roberta.RoBerta.embeddings.token_type_embeddings.weight', 'roberta.RoBerta.embeddings.LayerNorm.weight', 'roberta.RoBerta.embeddings.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.0.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.0.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.0.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.0.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.0.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.0.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.0.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.0.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.0.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.0.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.0.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.0.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.0.output.dense.weight', 'roberta.RoBerta.encoder.layer.0.output.dense.bias', 'roberta.RoBerta.encoder.layer.0.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.0.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.1.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.1.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.1.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.1.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.1.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.1.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.1.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.1.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.1.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.1.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.1.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.1.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.1.output.dense.weight', 'roberta.RoBerta.encoder.layer.1.output.dense.bias', 'roberta.RoBerta.encoder.layer.1.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.1.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.2.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.2.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.2.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.2.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.2.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.2.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.2.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.2.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.2.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.2.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.2.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.2.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.2.output.dense.weight', 'roberta.RoBerta.encoder.layer.2.output.dense.bias', 'roberta.RoBerta.encoder.layer.2.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.2.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.3.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.3.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.3.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.3.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.3.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.3.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.3.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.3.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.3.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.3.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.3.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.3.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.3.output.dense.weight', 'roberta.RoBerta.encoder.layer.3.output.dense.bias', 'roberta.RoBerta.encoder.layer.3.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.3.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.4.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.4.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.4.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.4.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.4.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.4.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.4.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.4.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.4.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.4.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.4.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.4.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.4.output.dense.weight', 'roberta.RoBerta.encoder.layer.4.output.dense.bias', 'roberta.RoBerta.encoder.layer.4.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.4.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.5.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.5.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.5.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.5.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.5.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.5.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.5.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.5.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.5.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.5.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.5.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.5.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.5.output.dense.weight', 'roberta.RoBerta.encoder.layer.5.output.dense.bias', 'roberta.RoBerta.encoder.layer.5.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.5.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.6.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.6.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.6.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.6.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.6.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.6.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.6.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.6.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.6.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.6.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.6.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.6.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.6.output.dense.weight', 'roberta.RoBerta.encoder.layer.6.output.dense.bias', 'roberta.RoBerta.encoder.layer.6.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.6.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.7.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.7.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.7.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.7.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.7.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.7.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.7.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.7.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.7.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.7.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.7.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.7.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.7.output.dense.weight', 'roberta.RoBerta.encoder.layer.7.output.dense.bias', 'roberta.RoBerta.encoder.layer.7.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.7.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.8.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.8.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.8.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.8.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.8.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.8.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.8.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.8.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.8.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.8.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.8.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.8.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.8.output.dense.weight', 'roberta.RoBerta.encoder.layer.8.output.dense.bias', 'roberta.RoBerta.encoder.layer.8.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.8.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.9.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.9.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.9.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.9.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.9.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.9.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.9.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.9.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.9.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.9.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.9.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.9.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.9.output.dense.weight', 'roberta.RoBerta.encoder.layer.9.output.dense.bias', 'roberta.RoBerta.encoder.layer.9.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.9.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.10.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.10.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.10.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.10.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.10.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.10.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.10.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.10.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.10.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.10.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.10.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.10.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.10.output.dense.weight', 'roberta.RoBerta.encoder.layer.10.output.dense.bias', 'roberta.RoBerta.encoder.layer.10.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.10.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.11.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.11.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.11.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.11.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.11.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.11.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.11.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.11.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.11.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.11.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.11.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.11.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.11.output.dense.weight', 'roberta.RoBerta.encoder.layer.11.output.dense.bias', 'roberta.RoBerta.encoder.layer.11.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.11.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.12.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.12.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.12.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.12.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.12.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.12.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.12.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.12.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.12.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.12.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.12.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.12.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.12.output.dense.weight', 'roberta.RoBerta.encoder.layer.12.output.dense.bias', 'roberta.RoBerta.encoder.layer.12.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.12.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.13.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.13.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.13.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.13.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.13.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.13.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.13.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.13.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.13.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.13.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.13.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.13.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.13.output.dense.weight', 'roberta.RoBerta.encoder.layer.13.output.dense.bias', 'roberta.RoBerta.encoder.layer.13.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.13.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.14.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.14.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.14.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.14.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.14.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.14.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.14.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.14.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.14.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.14.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.14.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.14.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.14.output.dense.weight', 'roberta.RoBerta.encoder.layer.14.output.dense.bias', 'roberta.RoBerta.encoder.layer.14.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.14.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.15.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.15.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.15.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.15.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.15.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.15.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.15.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.15.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.15.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.15.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.15.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.15.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.15.output.dense.weight', 'roberta.RoBerta.encoder.layer.15.output.dense.bias', 'roberta.RoBerta.encoder.layer.15.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.15.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.16.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.16.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.16.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.16.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.16.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.16.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.16.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.16.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.16.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.16.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.16.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.16.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.16.output.dense.weight', 'roberta.RoBerta.encoder.layer.16.output.dense.bias', 'roberta.RoBerta.encoder.layer.16.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.16.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.17.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.17.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.17.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.17.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.17.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.17.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.17.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.17.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.17.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.17.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.17.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.17.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.17.output.dense.weight', 'roberta.RoBerta.encoder.layer.17.output.dense.bias', 'roberta.RoBerta.encoder.layer.17.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.17.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.18.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.18.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.18.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.18.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.18.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.18.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.18.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.18.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.18.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.18.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.18.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.18.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.18.output.dense.weight', 'roberta.RoBerta.encoder.layer.18.output.dense.bias', 'roberta.RoBerta.encoder.layer.18.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.18.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.19.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.19.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.19.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.19.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.19.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.19.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.19.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.19.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.19.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.19.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.19.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.19.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.19.output.dense.weight', 'roberta.RoBerta.encoder.layer.19.output.dense.bias', 'roberta.RoBerta.encoder.layer.19.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.19.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.20.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.20.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.20.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.20.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.20.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.20.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.20.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.20.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.20.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.20.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.20.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.20.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.20.output.dense.weight', 'roberta.RoBerta.encoder.layer.20.output.dense.bias', 'roberta.RoBerta.encoder.layer.20.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.20.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.21.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.21.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.21.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.21.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.21.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.21.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.21.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.21.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.21.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.21.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.21.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.21.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.21.output.dense.weight', 'roberta.RoBerta.encoder.layer.21.output.dense.bias', 'roberta.RoBerta.encoder.layer.21.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.21.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.22.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.22.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.22.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.22.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.22.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.22.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.22.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.22.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.22.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.22.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.22.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.22.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.22.output.dense.weight', 'roberta.RoBerta.encoder.layer.22.output.dense.bias', 'roberta.RoBerta.encoder.layer.22.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.22.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.23.attention.self.query.weight', 'roberta.RoBerta.encoder.layer.23.attention.self.query.bias', 'roberta.RoBerta.encoder.layer.23.attention.self.key.weight', 'roberta.RoBerta.encoder.layer.23.attention.self.key.bias', 'roberta.RoBerta.encoder.layer.23.attention.self.value.weight', 'roberta.RoBerta.encoder.layer.23.attention.self.value.bias', 'roberta.RoBerta.encoder.layer.23.attention.output.dense.weight', 'roberta.RoBerta.encoder.layer.23.attention.output.dense.bias', 'roberta.RoBerta.encoder.layer.23.attention.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.23.attention.output.LayerNorm.bias', 'roberta.RoBerta.encoder.layer.23.intermediate.dense.weight', 'roberta.RoBerta.encoder.layer.23.intermediate.dense.bias', 'roberta.RoBerta.encoder.layer.23.output.dense.weight', 'roberta.RoBerta.encoder.layer.23.output.dense.bias', 'roberta.RoBerta.encoder.layer.23.output.LayerNorm.weight', 'roberta.RoBerta.encoder.layer.23.output.LayerNorm.bias', 'roberta.RoBerta.pooler.dense.weight', 'roberta.RoBerta.pooler.dense.bias', 'roberta.linear_trans.weight', 'roberta.linear_trans.bias', 'roberta.linear_fuse_p.weight', 'roberta.linear_fuse_p.bias', 'roberta.linear_fuse_q.weight', 'roberta.linear_fuse_q.bias', 'roberta.linear_fuse_a.weight', 'roberta.linear_fuse_a.bias', 'roberta.classifier.weight', 'roberta.classifier.bias']
02/18/2020 15:24:29 - INFO - transformers.modeling_utils -   Weights from pretrained model not used in RoBertaMultiwayMatch: ['roberta.embeddings.word_embeddings.weight', 'roberta.embeddings.position_embeddings.weight', 'roberta.embeddings.token_type_embeddings.weight', 'roberta.embeddings.LayerNorm.weight', 'roberta.embeddings.LayerNorm.bias', 'roberta.encoder.layer.0.attention.self.query.weight', 'roberta.encoder.layer.0.attention.self.query.bias', 'roberta.encoder.layer.0.attention.self.key.weight', 'roberta.encoder.layer.0.attention.self.key.bias', 'roberta.encoder.layer.0.attention.self.value.weight', 'roberta.encoder.layer.0.attention.self.value.bias', 'roberta.encoder.layer.0.attention.output.dense.weight', 'roberta.encoder.layer.0.attention.output.dense.bias', 'roberta.encoder.layer.0.attention.output.LayerNorm.weight', 'roberta.encoder.layer.0.attention.output.LayerNorm.bias', 'roberta.encoder.layer.0.intermediate.dense.weight', 'roberta.encoder.layer.0.intermediate.dense.bias', 'roberta.encoder.layer.0.output.dense.weight', 'roberta.encoder.layer.0.output.dense.bias', 'roberta.encoder.layer.0.output.LayerNorm.weight', 'roberta.encoder.layer.0.output.LayerNorm.bias', 'roberta.encoder.layer.1.attention.self.query.weight', 'roberta.encoder.layer.1.attention.self.query.bias', 'roberta.encoder.layer.1.attention.self.key.weight', 'roberta.encoder.layer.1.attention.self.key.bias', 'roberta.encoder.layer.1.attention.self.value.weight', 'roberta.encoder.layer.1.attention.self.value.bias', 'roberta.encoder.layer.1.attention.output.dense.weight', 'roberta.encoder.layer.1.attention.output.dense.bias', 'roberta.encoder.layer.1.attention.output.LayerNorm.weight', 'roberta.encoder.layer.1.attention.output.LayerNorm.bias', 'roberta.encoder.layer.1.intermediate.dense.weight', 'roberta.encoder.layer.1.intermediate.dense.bias', 'roberta.encoder.layer.1.output.dense.weight', 'roberta.encoder.layer.1.output.dense.bias', 'roberta.encoder.layer.1.output.LayerNorm.weight', 'roberta.encoder.layer.1.output.LayerNorm.bias', 'roberta.encoder.layer.2.attention.self.query.weight', 'roberta.encoder.layer.2.attention.self.query.bias', 'roberta.encoder.layer.2.attention.self.key.weight', 'roberta.encoder.layer.2.attention.self.key.bias', 'roberta.encoder.layer.2.attention.self.value.weight', 'roberta.encoder.layer.2.attention.self.value.bias', 'roberta.encoder.layer.2.attention.output.dense.weight', 'roberta.encoder.layer.2.attention.output.dense.bias', 'roberta.encoder.layer.2.attention.output.LayerNorm.weight', 'roberta.encoder.layer.2.attention.output.LayerNorm.bias', 'roberta.encoder.layer.2.intermediate.dense.weight', 'roberta.encoder.layer.2.intermediate.dense.bias', 'roberta.encoder.layer.2.output.dense.weight', 'roberta.encoder.layer.2.output.dense.bias', 'roberta.encoder.layer.2.output.LayerNorm.weight', 'roberta.encoder.layer.2.output.LayerNorm.bias', 'roberta.encoder.layer.3.attention.self.query.weight', 'roberta.encoder.layer.3.attention.self.query.bias', 'roberta.encoder.layer.3.attention.self.key.weight', 'roberta.encoder.layer.3.attention.self.key.bias', 'roberta.encoder.layer.3.attention.self.value.weight', 'roberta.encoder.layer.3.attention.self.value.bias', 'roberta.encoder.layer.3.attention.output.dense.weight', 'roberta.encoder.layer.3.attention.output.dense.bias', 'roberta.encoder.layer.3.attention.output.LayerNorm.weight', 'roberta.encoder.layer.3.attention.output.LayerNorm.bias', 'roberta.encoder.layer.3.intermediate.dense.weight', 'roberta.encoder.layer.3.intermediate.dense.bias', 'roberta.encoder.layer.3.output.dense.weight', 'roberta.encoder.layer.3.output.dense.bias', 'roberta.encoder.layer.3.output.LayerNorm.weight', 'roberta.encoder.layer.3.output.LayerNorm.bias', 'roberta.encoder.layer.4.attention.self.query.weight', 'roberta.encoder.layer.4.attention.self.query.bias', 'roberta.encoder.layer.4.attention.self.key.weight', 'roberta.encoder.layer.4.attention.self.key.bias', 'roberta.encoder.layer.4.attention.self.value.weight', 'roberta.encoder.layer.4.attention.self.value.bias', 'roberta.encoder.layer.4.attention.output.dense.weight', 'roberta.encoder.layer.4.attention.output.dense.bias', 'roberta.encoder.layer.4.attention.output.LayerNorm.weight', 'roberta.encoder.layer.4.attention.output.LayerNorm.bias', 'roberta.encoder.layer.4.intermediate.dense.weight', 'roberta.encoder.layer.4.intermediate.dense.bias', 'roberta.encoder.layer.4.output.dense.weight', 'roberta.encoder.layer.4.output.dense.bias', 'roberta.encoder.layer.4.output.LayerNorm.weight', 'roberta.encoder.layer.4.output.LayerNorm.bias', 'roberta.encoder.layer.5.attention.self.query.weight', 'roberta.encoder.layer.5.attention.self.query.bias', 'roberta.encoder.layer.5.attention.self.key.weight', 'roberta.encoder.layer.5.attention.self.key.bias', 'roberta.encoder.layer.5.attention.self.value.weight', 'roberta.encoder.layer.5.attention.self.value.bias', 'roberta.encoder.layer.5.attention.output.dense.weight', 'roberta.encoder.layer.5.attention.output.dense.bias', 'roberta.encoder.layer.5.attention.output.LayerNorm.weight', 'roberta.encoder.layer.5.attention.output.LayerNorm.bias', 'roberta.encoder.layer.5.intermediate.dense.weight', 'roberta.encoder.layer.5.intermediate.dense.bias', 'roberta.encoder.layer.5.output.dense.weight', 'roberta.encoder.layer.5.output.dense.bias', 'roberta.encoder.layer.5.output.LayerNorm.weight', 'roberta.encoder.layer.5.output.LayerNorm.bias', 'roberta.encoder.layer.6.attention.self.query.weight', 'roberta.encoder.layer.6.attention.self.query.bias', 'roberta.encoder.layer.6.attention.self.key.weight', 'roberta.encoder.layer.6.attention.self.key.bias', 'roberta.encoder.layer.6.attention.self.value.weight', 'roberta.encoder.layer.6.attention.self.value.bias', 'roberta.encoder.layer.6.attention.output.dense.weight', 'roberta.encoder.layer.6.attention.output.dense.bias', 'roberta.encoder.layer.6.attention.output.LayerNorm.weight', 'roberta.encoder.layer.6.attention.output.LayerNorm.bias', 'roberta.encoder.layer.6.intermediate.dense.weight', 'roberta.encoder.layer.6.intermediate.dense.bias', 'roberta.encoder.layer.6.output.dense.weight', 'roberta.encoder.layer.6.output.dense.bias', 'roberta.encoder.layer.6.output.LayerNorm.weight', 'roberta.encoder.layer.6.output.LayerNorm.bias', 'roberta.encoder.layer.7.attention.self.query.weight', 'roberta.encoder.layer.7.attention.self.query.bias', 'roberta.encoder.layer.7.attention.self.key.weight', 'roberta.encoder.layer.7.attention.self.key.bias', 'roberta.encoder.layer.7.attention.self.value.weight', 'roberta.encoder.layer.7.attention.self.value.bias', 'roberta.encoder.layer.7.attention.output.dense.weight', 'roberta.encoder.layer.7.attention.output.dense.bias', 'roberta.encoder.layer.7.attention.output.LayerNorm.weight', 'roberta.encoder.layer.7.attention.output.LayerNorm.bias', 'roberta.encoder.layer.7.intermediate.dense.weight', 'roberta.encoder.layer.7.intermediate.dense.bias', 'roberta.encoder.layer.7.output.dense.weight', 'roberta.encoder.layer.7.output.dense.bias', 'roberta.encoder.layer.7.output.LayerNorm.weight', 'roberta.encoder.layer.7.output.LayerNorm.bias', 'roberta.encoder.layer.8.attention.self.query.weight', 'roberta.encoder.layer.8.attention.self.query.bias', 'roberta.encoder.layer.8.attention.self.key.weight', 'roberta.encoder.layer.8.attention.self.key.bias', 'roberta.encoder.layer.8.attention.self.value.weight', 'roberta.encoder.layer.8.attention.self.value.bias', 'roberta.encoder.layer.8.attention.output.dense.weight', 'roberta.encoder.layer.8.attention.output.dense.bias', 'roberta.encoder.layer.8.attention.output.LayerNorm.weight', 'roberta.encoder.layer.8.attention.output.LayerNorm.bias', 'roberta.encoder.layer.8.intermediate.dense.weight', 'roberta.encoder.layer.8.intermediate.dense.bias', 'roberta.encoder.layer.8.output.dense.weight', 'roberta.encoder.layer.8.output.dense.bias', 'roberta.encoder.layer.8.output.LayerNorm.weight', 'roberta.encoder.layer.8.output.LayerNorm.bias', 'roberta.encoder.layer.9.attention.self.query.weight', 'roberta.encoder.layer.9.attention.self.query.bias', 'roberta.encoder.layer.9.attention.self.key.weight', 'roberta.encoder.layer.9.attention.self.key.bias', 'roberta.encoder.layer.9.attention.self.value.weight', 'roberta.encoder.layer.9.attention.self.value.bias', 'roberta.encoder.layer.9.attention.output.dense.weight', 'roberta.encoder.layer.9.attention.output.dense.bias', 'roberta.encoder.layer.9.attention.output.LayerNorm.weight', 'roberta.encoder.layer.9.attention.output.LayerNorm.bias', 'roberta.encoder.layer.9.intermediate.dense.weight', 'roberta.encoder.layer.9.intermediate.dense.bias', 'roberta.encoder.layer.9.output.dense.weight', 'roberta.encoder.layer.9.output.dense.bias', 'roberta.encoder.layer.9.output.LayerNorm.weight', 'roberta.encoder.layer.9.output.LayerNorm.bias', 'roberta.encoder.layer.10.attention.self.query.weight', 'roberta.encoder.layer.10.attention.self.query.bias', 'roberta.encoder.layer.10.attention.self.key.weight', 'roberta.encoder.layer.10.attention.self.key.bias', 'roberta.encoder.layer.10.attention.self.value.weight', 'roberta.encoder.layer.10.attention.self.value.bias', 'roberta.encoder.layer.10.attention.output.dense.weight', 'roberta.encoder.layer.10.attention.output.dense.bias', 'roberta.encoder.layer.10.attention.output.LayerNorm.weight', 'roberta.encoder.layer.10.attention.output.LayerNorm.bias', 'roberta.encoder.layer.10.intermediate.dense.weight', 'roberta.encoder.layer.10.intermediate.dense.bias', 'roberta.encoder.layer.10.output.dense.weight', 'roberta.encoder.layer.10.output.dense.bias', 'roberta.encoder.layer.10.output.LayerNorm.weight', 'roberta.encoder.layer.10.output.LayerNorm.bias', 'roberta.encoder.layer.11.attention.self.query.weight', 'roberta.encoder.layer.11.attention.self.query.bias', 'roberta.encoder.layer.11.attention.self.key.weight', 'roberta.encoder.layer.11.attention.self.key.bias', 'roberta.encoder.layer.11.attention.self.value.weight', 'roberta.encoder.layer.11.attention.self.value.bias', 'roberta.encoder.layer.11.attention.output.dense.weight', 'roberta.encoder.layer.11.attention.output.dense.bias', 'roberta.encoder.layer.11.attention.output.LayerNorm.weight', 'roberta.encoder.layer.11.attention.output.LayerNorm.bias', 'roberta.encoder.layer.11.intermediate.dense.weight', 'roberta.encoder.layer.11.intermediate.dense.bias', 'roberta.encoder.layer.11.output.dense.weight', 'roberta.encoder.layer.11.output.dense.bias', 'roberta.encoder.layer.11.output.LayerNorm.weight', 'roberta.encoder.layer.11.output.LayerNorm.bias', 'roberta.encoder.layer.12.attention.self.query.weight', 'roberta.encoder.layer.12.attention.self.query.bias', 'roberta.encoder.layer.12.attention.self.key.weight', 'roberta.encoder.layer.12.attention.self.key.bias', 'roberta.encoder.layer.12.attention.self.value.weight', 'roberta.encoder.layer.12.attention.self.value.bias', 'roberta.encoder.layer.12.attention.output.dense.weight', 'roberta.encoder.layer.12.attention.output.dense.bias', 'roberta.encoder.layer.12.attention.output.LayerNorm.weight', 'roberta.encoder.layer.12.attention.output.LayerNorm.bias', 'roberta.encoder.layer.12.intermediate.dense.weight', 'roberta.encoder.layer.12.intermediate.dense.bias', 'roberta.encoder.layer.12.output.dense.weight', 'roberta.encoder.layer.12.output.dense.bias', 'roberta.encoder.layer.12.output.LayerNorm.weight', 'roberta.encoder.layer.12.output.LayerNorm.bias', 'roberta.encoder.layer.13.attention.self.query.weight', 'roberta.encoder.layer.13.attention.self.query.bias', 'roberta.encoder.layer.13.attention.self.key.weight', 'roberta.encoder.layer.13.attention.self.key.bias', 'roberta.encoder.layer.13.attention.self.value.weight', 'roberta.encoder.layer.13.attention.self.value.bias', 'roberta.encoder.layer.13.attention.output.dense.weight', 'roberta.encoder.layer.13.attention.output.dense.bias', 'roberta.encoder.layer.13.attention.output.LayerNorm.weight', 'roberta.encoder.layer.13.attention.output.LayerNorm.bias', 'roberta.encoder.layer.13.intermediate.dense.weight', 'roberta.encoder.layer.13.intermediate.dense.bias', 'roberta.encoder.layer.13.output.dense.weight', 'roberta.encoder.layer.13.output.dense.bias', 'roberta.encoder.layer.13.output.LayerNorm.weight', 'roberta.encoder.layer.13.output.LayerNorm.bias', 'roberta.encoder.layer.14.attention.self.query.weight', 'roberta.encoder.layer.14.attention.self.query.bias', 'roberta.encoder.layer.14.attention.self.key.weight', 'roberta.encoder.layer.14.attention.self.key.bias', 'roberta.encoder.layer.14.attention.self.value.weight', 'roberta.encoder.layer.14.attention.self.value.bias', 'roberta.encoder.layer.14.attention.output.dense.weight', 'roberta.encoder.layer.14.attention.output.dense.bias', 'roberta.encoder.layer.14.attention.output.LayerNorm.weight', 'roberta.encoder.layer.14.attention.output.LayerNorm.bias', 'roberta.encoder.layer.14.intermediate.dense.weight', 'roberta.encoder.layer.14.intermediate.dense.bias', 'roberta.encoder.layer.14.output.dense.weight', 'roberta.encoder.layer.14.output.dense.bias', 'roberta.encoder.layer.14.output.LayerNorm.weight', 'roberta.encoder.layer.14.output.LayerNorm.bias', 'roberta.encoder.layer.15.attention.self.query.weight', 'roberta.encoder.layer.15.attention.self.query.bias', 'roberta.encoder.layer.15.attention.self.key.weight', 'roberta.encoder.layer.15.attention.self.key.bias', 'roberta.encoder.layer.15.attention.self.value.weight', 'roberta.encoder.layer.15.attention.self.value.bias', 'roberta.encoder.layer.15.attention.output.dense.weight', 'roberta.encoder.layer.15.attention.output.dense.bias', 'roberta.encoder.layer.15.attention.output.LayerNorm.weight', 'roberta.encoder.layer.15.attention.output.LayerNorm.bias', 'roberta.encoder.layer.15.intermediate.dense.weight', 'roberta.encoder.layer.15.intermediate.dense.bias', 'roberta.encoder.layer.15.output.dense.weight', 'roberta.encoder.layer.15.output.dense.bias', 'roberta.encoder.layer.15.output.LayerNorm.weight', 'roberta.encoder.layer.15.output.LayerNorm.bias', 'roberta.encoder.layer.16.attention.self.query.weight', 'roberta.encoder.layer.16.attention.self.query.bias', 'roberta.encoder.layer.16.attention.self.key.weight', 'roberta.encoder.layer.16.attention.self.key.bias', 'roberta.encoder.layer.16.attention.self.value.weight', 'roberta.encoder.layer.16.attention.self.value.bias', 'roberta.encoder.layer.16.attention.output.dense.weight', 'roberta.encoder.layer.16.attention.output.dense.bias', 'roberta.encoder.layer.16.attention.output.LayerNorm.weight', 'roberta.encoder.layer.16.attention.output.LayerNorm.bias', 'roberta.encoder.layer.16.intermediate.dense.weight', 'roberta.encoder.layer.16.intermediate.dense.bias', 'roberta.encoder.layer.16.output.dense.weight', 'roberta.encoder.layer.16.output.dense.bias', 'roberta.encoder.layer.16.output.LayerNorm.weight', 'roberta.encoder.layer.16.output.LayerNorm.bias', 'roberta.encoder.layer.17.attention.self.query.weight', 'roberta.encoder.layer.17.attention.self.query.bias', 'roberta.encoder.layer.17.attention.self.key.weight', 'roberta.encoder.layer.17.attention.self.key.bias', 'roberta.encoder.layer.17.attention.self.value.weight', 'roberta.encoder.layer.17.attention.self.value.bias', 'roberta.encoder.layer.17.attention.output.dense.weight', 'roberta.encoder.layer.17.attention.output.dense.bias', 'roberta.encoder.layer.17.attention.output.LayerNorm.weight', 'roberta.encoder.layer.17.attention.output.LayerNorm.bias', 'roberta.encoder.layer.17.intermediate.dense.weight', 'roberta.encoder.layer.17.intermediate.dense.bias', 'roberta.encoder.layer.17.output.dense.weight', 'roberta.encoder.layer.17.output.dense.bias', 'roberta.encoder.layer.17.output.LayerNorm.weight', 'roberta.encoder.layer.17.output.LayerNorm.bias', 'roberta.encoder.layer.18.attention.self.query.weight', 'roberta.encoder.layer.18.attention.self.query.bias', 'roberta.encoder.layer.18.attention.self.key.weight', 'roberta.encoder.layer.18.attention.self.key.bias', 'roberta.encoder.layer.18.attention.self.value.weight', 'roberta.encoder.layer.18.attention.self.value.bias', 'roberta.encoder.layer.18.attention.output.dense.weight', 'roberta.encoder.layer.18.attention.output.dense.bias', 'roberta.encoder.layer.18.attention.output.LayerNorm.weight', 'roberta.encoder.layer.18.attention.output.LayerNorm.bias', 'roberta.encoder.layer.18.intermediate.dense.weight', 'roberta.encoder.layer.18.intermediate.dense.bias', 'roberta.encoder.layer.18.output.dense.weight', 'roberta.encoder.layer.18.output.dense.bias', 'roberta.encoder.layer.18.output.LayerNorm.weight', 'roberta.encoder.layer.18.output.LayerNorm.bias', 'roberta.encoder.layer.19.attention.self.query.weight', 'roberta.encoder.layer.19.attention.self.query.bias', 'roberta.encoder.layer.19.attention.self.key.weight', 'roberta.encoder.layer.19.attention.self.key.bias', 'roberta.encoder.layer.19.attention.self.value.weight', 'roberta.encoder.layer.19.attention.self.value.bias', 'roberta.encoder.layer.19.attention.output.dense.weight', 'roberta.encoder.layer.19.attention.output.dense.bias', 'roberta.encoder.layer.19.attention.output.LayerNorm.weight', 'roberta.encoder.layer.19.attention.output.LayerNorm.bias', 'roberta.encoder.layer.19.intermediate.dense.weight', 'roberta.encoder.layer.19.intermediate.dense.bias', 'roberta.encoder.layer.19.output.dense.weight', 'roberta.encoder.layer.19.output.dense.bias', 'roberta.encoder.layer.19.output.LayerNorm.weight', 'roberta.encoder.layer.19.output.LayerNorm.bias', 'roberta.encoder.layer.20.attention.self.query.weight', 'roberta.encoder.layer.20.attention.self.query.bias', 'roberta.encoder.layer.20.attention.self.key.weight', 'roberta.encoder.layer.20.attention.self.key.bias', 'roberta.encoder.layer.20.attention.self.value.weight', 'roberta.encoder.layer.20.attention.self.value.bias', 'roberta.encoder.layer.20.attention.output.dense.weight', 'roberta.encoder.layer.20.attention.output.dense.bias', 'roberta.encoder.layer.20.attention.output.LayerNorm.weight', 'roberta.encoder.layer.20.attention.output.LayerNorm.bias', 'roberta.encoder.layer.20.intermediate.dense.weight', 'roberta.encoder.layer.20.intermediate.dense.bias', 'roberta.encoder.layer.20.output.dense.weight', 'roberta.encoder.layer.20.output.dense.bias', 'roberta.encoder.layer.20.output.LayerNorm.weight', 'roberta.encoder.layer.20.output.LayerNorm.bias', 'roberta.encoder.layer.21.attention.self.query.weight', 'roberta.encoder.layer.21.attention.self.query.bias', 'roberta.encoder.layer.21.attention.self.key.weight', 'roberta.encoder.layer.21.attention.self.key.bias', 'roberta.encoder.layer.21.attention.self.value.weight', 'roberta.encoder.layer.21.attention.self.value.bias', 'roberta.encoder.layer.21.attention.output.dense.weight', 'roberta.encoder.layer.21.attention.output.dense.bias', 'roberta.encoder.layer.21.attention.output.LayerNorm.weight', 'roberta.encoder.layer.21.attention.output.LayerNorm.bias', 'roberta.encoder.layer.21.intermediate.dense.weight', 'roberta.encoder.layer.21.intermediate.dense.bias', 'roberta.encoder.layer.21.output.dense.weight', 'roberta.encoder.layer.21.output.dense.bias', 'roberta.encoder.layer.21.output.LayerNorm.weight', 'roberta.encoder.layer.21.output.LayerNorm.bias', 'roberta.encoder.layer.22.attention.self.query.weight', 'roberta.encoder.layer.22.attention.self.query.bias', 'roberta.encoder.layer.22.attention.self.key.weight', 'roberta.encoder.layer.22.attention.self.key.bias', 'roberta.encoder.layer.22.attention.self.value.weight', 'roberta.encoder.layer.22.attention.self.value.bias', 'roberta.encoder.layer.22.attention.output.dense.weight', 'roberta.encoder.layer.22.attention.output.dense.bias', 'roberta.encoder.layer.22.attention.output.LayerNorm.weight', 'roberta.encoder.layer.22.attention.output.LayerNorm.bias', 'roberta.encoder.layer.22.intermediate.dense.weight', 'roberta.encoder.layer.22.intermediate.dense.bias', 'roberta.encoder.layer.22.output.dense.weight', 'roberta.encoder.layer.22.output.dense.bias', 'roberta.encoder.layer.22.output.LayerNorm.weight', 'roberta.encoder.layer.22.output.LayerNorm.bias', 'roberta.encoder.layer.23.attention.self.query.weight', 'roberta.encoder.layer.23.attention.self.query.bias', 'roberta.encoder.layer.23.attention.self.key.weight', 'roberta.encoder.layer.23.attention.self.key.bias', 'roberta.encoder.layer.23.attention.self.value.weight', 'roberta.encoder.layer.23.attention.self.value.bias', 'roberta.encoder.layer.23.attention.output.dense.weight', 'roberta.encoder.layer.23.attention.output.dense.bias', 'roberta.encoder.layer.23.attention.output.LayerNorm.weight', 'roberta.encoder.layer.23.attention.output.LayerNorm.bias', 'roberta.encoder.layer.23.intermediate.dense.weight', 'roberta.encoder.layer.23.intermediate.dense.bias', 'roberta.encoder.layer.23.output.dense.weight', 'roberta.encoder.layer.23.output.dense.bias', 'roberta.encoder.layer.23.output.LayerNorm.weight', 'roberta.encoder.layer.23.output.LayerNorm.bias', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']
```

How can I solve this problem?
Thanks in advance!


Improve special_token_id logic in run_generation.py and add tests

This PR finally implements the following `bos_token_id, pad_token_id, eos_token_ids` logic for lm model generation.

1. If `bos_token_id` is None, then the input_ids must be defined otherwise, the model cannot generate text, which is checked by the asserts in the beginning. The `bos_token_id` is only relevant for starting a new sentence.

2. If `eos_token_id` is None, then the length of the generated text will always equal max_length,
no matter how the pad_token_id is defined. Since there is no `eos_token_id` the text will also not "end".

3. If `pad_token_id` is None and `eos_token_ids` is defined (as it is the case for gpt2), then the pad_token_id will be set to the `eos_token_ids[0]` tensor `batches_len` is used to keep track of the first time the sequence generated an eos_token and will later set all tokens following this token to the `pad_token_id`, which is `eos_token_ids[0]` and can thus be handled by the tokenizer (whereas the -1 cannot be handled by the tokenizer).

4. **No** eos_token_id is appended to sentences that finish due to `max_length`. Instead those sentences are returned with the last token being the last token produced by the model until `max_length` was hit.

As an overview, here a table showing which LMModel Tokenizer have which of the tokens `bos_token_id`, `pad_token_id` and `eos_token_ids` is defined:

| LM Model  | bos_token_id | pad_token_id | eos_token_ids |
| ------------- | ------------- | ------------- | ------------- |
| XLNet  | x | x | x |
| OpenAIGPT  | o | o | o |
| CTRL  | o | o | o |
| GPT2  | x | o | x |
| Transfo-XL  | o | o | x |
| XLM  | x | x | o |

Testing times are increased by the following times (measured on a local machine):

| LM Model  | Increase in test time |
| ------------- | ------------- |
| XLNet  | 8.0s -> 9.7s |
| OpenAIGPT  | 7.1s -> 8.3s |
| CTRL  | 2.5s -> 4.3s |
| GPT2  | 7.3s -> 8.0s |
| Transfo-XL  | 7.5s -> 8.0s |
| XLM  | 7.4s -> 7.7s |

-> So overall mostly around 10% increase in testing time

## Future PRs:

- [x] [WIP] adding hard-coded slow tests for pretrained lms in PR #2909
- [x] [WIP] adapting the `generate` function for Seq-2-Seq and DoubleHeads or other special LM models in PR #2888
- [x] checking and possibly adapting behavior of `generate_beam_search`
- [x] treat Issues: #2482 and #2415


Evaluation and Inference added to run_glue.py

# 🚀 Feature request

It would be useful to have the following arguments added to `run_glue.py`, as well as probably the other task example scripts: `--eval_only` and `--inference_only`.

## Motivation

This will allow users to either provide a `.tsv` or `.csv` with either sentences and labels or just sentences and then perform evaluation or inference on the model without having to train. This may already exist, but I was unable to find it. Consequently, I'm using https://github.com/kaushaltrivedi/fast-bert for these tasks. It would be helpful to not have to stray from the main huggingface transformers library.


Create README.md in the right path for bert-spanish-cased-finetuned-ner

model card

Related to pretrained model cards


No prediction for some words (BERT NER) when run on GPU

PyTorch

Anything PyTorch

I have tried to make predictions over the test data using GPU, but ended up having no predictions for some words.  Any help would be appreciated.

Following is the shell script used for prediction.

export MAX_LENGTH=128
export BERT_MODEL=bert-base-multilingual-cased

DATA_DIR=../DATA/after_preprocess
OUTPUT_DIR=../CHECKPOINTS/GPU/CONLL_25L
LABELS_FILE_25=../DATA/after_preprocess/labels.txt

export BATCH_SIZE=32
export NUM_EPOCHS=3
export SAVE_STEPS=750
export SEED=1

cd ../examples

python3 -m torch.distributed.launch run_ner.py --data_dir $DATA_DIR/ \
--model_type bert \
--labels $LABELS_FILE_25 \
--model_name_or_path $BERT_MODEL \
--output_dir $OUTPUT_DIR \
--max_seq_length  $MAX_LENGTH \
--num_train_epochs $NUM_EPOCHS \
--per_gpu_train_batch_size $BATCH_SIZE \
--save_steps $SAVE_STEPS \
--seed $SEED \
--do_predict

Following is the error message:

02/17/2020 12:07:51 - INFO - transformers.tokenization_utils -   Model name '../CHECKPOINTS/GPU/CONLL_25L' not found in model shortcut name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, bert-base-finnish-cased-v1, bert-base-finnish-uncased-v1, bert-base-dutch-cased). Assuming '../CHECKPOINTS/GPU/CONLL_25L' is a path, a model identifier, or url to a directory containing tokenizer files.
02/17/2020 12:07:51 - INFO - transformers.tokenization_utils -   Didn't find file ../CHECKPOINTS/GPU/CONLL_25L/added_tokens.json. We won't load it.
02/17/2020 12:07:51 - INFO - transformers.tokenization_utils -   loading file ../CHECKPOINTS/GPU/CONLL_25L/vocab.txt
02/17/2020 12:07:51 - INFO - transformers.tokenization_utils -   loading file None
02/17/2020 12:07:51 - INFO - transformers.tokenization_utils -   loading file ../CHECKPOINTS/GPU/CONLL_25L/special_tokens_map.json
02/17/2020 12:07:51 - INFO - transformers.tokenization_utils -   loading file ../CHECKPOINTS/GPU/CONLL_25L/tokenizer_config.json
02/17/2020 12:07:51 - INFO - transformers.configuration_utils -   loading configuration file ../CHECKPOINTS/GPU/CONLL_25L/config.json
02/17/2020 12:07:51 - INFO - transformers.configuration_utils -   Model config BertConfig {
  "architectures": [
    "BertForTokenClassification"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "directionality": "bidi",
  "do_sample": false,
  "eos_token_ids": 0,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_eps": 1e-12,
  "length_penalty": 1.0,
  "max_length": 20,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_beams": 1,
  "num_hidden_layers": 12,
  "num_labels": 25,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 0,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 119547
}

02/17/2020 12:07:51 - INFO - transformers.modeling_utils -   loading weights file ../CHECKPOINTS/GPU/CONLL_25L/pytorch_model.bin
02/17/2020 12:07:54 - INFO - __main__ -   Creating features from dataset file at ../DATA/after_preprocess/
02/17/2020 12:07:54 - INFO - utils_ner -   Writing example 0 of 5100
02/17/2020 12:07:54 - INFO - utils_ner -   *** Example ***
02/17/2020 12:07:54 - INFO - utils_ner -   guid: test-1
02/17/2020 12:07:54 - INFO - utils_ner -   tokens: [CLS] 1951 bis 1953 wurde der nÃ¶rdlich ##e Teil als Jugend ##burg des Ko ##lp ##ing ##werke ##s gebaut . [SEP]
02/17/2020 12:07:54 - INFO - utils_ner -   input_ids: 101 11200 10467 11087 10283 10118 28253 10112 13043 10223 32790 12248 10139 30186 35451 10230 32827 10107 25760 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
02/17/2020 12:07:54 - INFO - utils_ner -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
02/17/2020 12:07:54 - INFO - utils_ner -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
02/17/2020 12:07:54 - INFO - utils_ner -   label_ids: -100 24 24 24 24 24 24 -100 24 24 24 -100 24 6 -100 -100 -100 -100 24 24 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100
02/17/2020 12:07:54 - INFO - utils_ner -   *** Example ***
02/17/2020 12:07:54 - INFO - utils_ner -   guid: test-2
02/17/2020 12:07:54 - INFO - utils_ner -   tokens: [CLS] Da Mu ##ck das Krieg ##ss ##chreiben nicht Ã¼ber ##bra ##cht hat , wird er als Re ##tter des Landes ausgezeichnet und soll zum Sc ##hat ##zm ##eister ernannt werden . [SEP]
02/17/2020 12:07:54 - INFO - utils_ner -   input_ids: 101 11818 49056 11263 10242 20587 13420 82089 10726 10848 13581 11640 11250 117 10790 10163 10223 20304 18413 10139 23244 32149 10130 17375 10580 55260 19180 37661 45940 27093 10615 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
02/17/2020 12:07:54 - INFO - utils_ner -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
02/17/2020 12:07:54 - INFO - utils_ner -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
02/17/2020 12:07:54 - INFO - utils_ner -   label_ids: -100 24 9 -100 24 24 -100 -100 24 24 -100 -100 24 24 24 24 24 24 -100 24 24 24 24 24 24 24 -100 -100 -100 24 24 24 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100
02/17/2020 12:07:54 - INFO - utils_ner -   *** Example ***
02/17/2020 12:07:54 - INFO - utils_ner -   guid: test-3
02/17/2020 12:07:54 - INFO - utils_ner -   tokens: [CLS] Mit 1 . JÃ¤nner 2007 wurde Robert Sc ##h ##Ã¶r ##gen ##hof ##er , als Nachfolger des aus ##ges ##chie ##dene ##n Dietmar Dr ##abe ##k , in die Kader ##liste der FIFA - Sc ##hie ##ds ##richter aufgenommen . [SEP]
02/17/2020 12:07:54 - INFO - utils_ner -   input_ids: 101 12699 122 119 105531 10202 10283 10820 55260 10237 15020 11280 20202 10165 117 10223 27968 10139 10441 13156 50784 49906 10115 102411 11612 40929 10174 117 10106 10128 53361 26719 10118 13707 118 55260 72287 13268 59410 25919 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
02/17/2020 12:07:54 - INFO - utils_ner -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
02/17/2020 12:07:54 - INFO - utils_ner -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
02/17/2020 12:07:54 - INFO - utils_ner -   label_ids: -100 24 24 -100 24 24 24 9 21 -100 -100 -100 -100 -100 24 24 24 24 24 -100 -100 -100 -100 9 21 -100 -100 24 24 24 24 -100 24 5 -100 -100 -100 -100 -100 24 24 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100
02/17/2020 12:07:54 - INFO - utils_ner -   *** Example ***
02/17/2020 12:07:54 - INFO - utils_ner -   guid: test-4
02/17/2020 12:07:54 - INFO - utils_ner -   tokens: [CLS] Die These , Sc ##hla ##tter sei Anti ##sem ##it gewesen , wurde seither in der theo ##logischen Fach ##lite ##ratu ##r nicht mehr vertreten . [SEP]
02/17/2020 12:07:54 - INFO - utils_ner -   input_ids: 101 10236 13252 117 55260 74935 18413 13868 26267 38443 10486 27044 117 10283 85983 10106 10118 13951 57325 100705 66289 50088 10129 10726 12471 41852 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
02/17/2020 12:07:54 - INFO - utils_ner -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
02/17/2020 12:07:54 - INFO - utils_ner -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
02/17/2020 12:07:54 - INFO - utils_ner -   label_ids: -100 24 24 24 9 -100 -100 24 24 -100 -100 24 24 24 24 24 24 24 -100 24 -100 -100 -100 24 24 24 24 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100
02/17/2020 12:07:54 - INFO - utils_ner -   *** Example ***
02/17/2020 12:07:54 - INFO - utils_ner -   guid: test-5
02/17/2020 12:07:54 - INFO - utils_ner -   tokens: [CLS] " Le ##hm ##bru ##ck - Be ##uy ##s . Zeichnungen " lautet der Titel der gerade erÃ¶ffnete ##n Ausstellung , die Kur ##atori ##n Dr . Marion Born ##sche ##uer bis zum 11 . Januar im Le ##hm ##bru ##ck - Museum prÃ¤sentiert . [SEP]
02/17/2020 12:07:54 - INFO - utils_ner -   input_ids: 101 107 10281 29389 40309 11263 118 14321 53452 10107 119 96784 107 77566 10118 16076 10118 43234 61469 10115 41972 117 10128 61912 45804 10115 11612 119 27276 18021 12279 19047 10467 10580 10193 119 12468 10211 10281 29389 40309 11263 118 11325 91619 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
02/17/2020 12:07:54 - INFO - utils_ner -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
02/17/2020 12:07:54 - INFO - utils_ner -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
02/17/2020 12:07:54 - INFO - utils_ner -   label_ids: -100 24 6 -100 -100 -100 18 18 -100 -100 18 -100 24 24 24 24 24 24 24 -100 24 24 24 24 -100 -100 24 24 9 21 -100 -100 24 24 24 -100 24 24 3 -100 -100 -100 -100 -100 24 24 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100 -100
02/17/2020 12:07:57 - INFO - __main__ -   Saving features into cached file ../DATA/after_preprocess/cached_test_bert-base-multilingual-cased_128
02/17/2020 12:07:58 - INFO - __main__ -   ***** Running evaluation  *****
02/17/2020 12:07:58 - INFO - __main__ -     Num examples = 5100
02/17/2020 12:07:58 - INFO - __main__ -     Batch size = 8

Evaluating:   0%|          | 0/638 [00:00<?, ?it/s]
Evaluating: 100%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ| 638/638 [00:27<00:00, 23.15it/s]

02/17/2020 12:08:27 - INFO - __main__ -   ***** Eval results  *****
02/17/2020 12:08:27 - INFO - __main__ -     f1 = 0.8600886024969794
02/17/2020 12:08:27 - INFO - __main__ -     loss = 0.070130527494103
02/17/2020 12:08:27 - INFO - __main__ -     precision = 0.8560205226871893
02/17/2020 12:08:27 - INFO - __main__ -     recall = 0.8641955325348009
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'wird'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'er'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'als'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Retter'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'des'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Landes'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'ausgezeichnet'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'und'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'soll'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'zum'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Schatzmeister'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'ernannt'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'werden'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for '.'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Nachfolger'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'des'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'ausgeschiedenen'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Dietmar'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Drabek'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for ','.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'in'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'die'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Kaderliste'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'der'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'FIFA-Schiedsrichter'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'aufgenommen'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for '.'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for '"'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'lautet'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'der'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Titel'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'der'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'gerade'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'erÃ¶ffneten'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Ausstellung'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for ','.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'die'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Kuratorin'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Dr'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for '.'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Marion'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Bornscheuer'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'bis'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'zum'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for '11.'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Januar'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'im'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Lehmbruck-Museum'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'prÃ¤sentiert'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for '.'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'an'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'der'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'SÃ¼dseite'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'des'.
02/17/2020 12:08:27 - WARNING - __main__ -   Maximum sequence length exceeded: No prediction for 'Saals'.


update .gitignore to ignore .swp files created when using vim

adds one line to .gitignore


Transformers と Simpletransfomrersを使ったAlebertでのNERの対応

AlbertでNERが出来るようになっていなかったので、その対応をしました。
主に、modeling_albert.pyにAlbertForTokenClassificationを実装したのと、それと整合性をとるために、その他のコードの変更を行いました。


[model_cards] 🇹🇷 Add new (cased) BERTurk model

model card

Related to pretrained model cards

Hi,

this PR adds the model card for the (cased) community-driven 🇹🇷 BERTurk model.

Uncased model is coming soon!


# 🐛 Bug

FileNotFoundError when python runs setup.py for sentencepiece
I am Running Python 3.7, Tensorflow 2.1, Buster

Model I am using (Bert, XLNet ...):  Would be using gpt-2 if I can install it...

Language I am using the model on is English

The problem arises when installing using

    pip install transformers

## To reproduce

Steps to reproduce the behavior:

Run on a Raspberry Pi 4B (4Gig) running Python 3.7, Tensorflow 2.1 and Buster

1.  pip install transformers

Wait for other downloads and installing to complete and this will eventually arise:

```
Collecting sentencepiece (from transformers)
  Downloading https://files.pythonhosted.org/packages/1b/87/c3c2fa8cbec61fffe031ca9f0da512747520bec9be7f886f748457daac31/sentencepiece-0.1.83.tar.gz (497kB)
    100% |████████████████████████████████| 501kB 225kB/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-xohx1aio/sentencepiece/setup.py", line 29, in <module>
        with codecs.open(os.path.join('..', 'VERSION'), 'r', 'utf-8') as f:
      File "/usr/lib/python3.7/codecs.py", line 898, in open
        file = builtins.open(filename, mode, buffering)
    FileNotFoundError: [Errno 2] No such file or directory: '../VERSION'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-xohx1aio/sentencepiece/

```

SO I TRIED download the wheel file from https://github.com/google/sentencepiece/releases for my python version and installing it with pip install sentencepiece-xxx-cpxx-xx.whl

However I see only Mac, x86, and "manylinux" wheels but the manylinux wheels specifically reference iOS or x86, nothing I can see for Arm Core 71 (linux_armv7l).

Also tried a straight install on sentencepiece with identical failure::

```
$ pip install sentencepiece
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting sentencepiece
  Using cached https://files.pythonhosted.org/packages/1b/87/c3c2fa8cbec61fffe031ca9f0da512747520bec9be7f886f748457daac31/sentencepiece-0.1.83.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-6tdniw95/sentencepiece/setup.py", line 29, in <module>
        with codecs.open(os.path.join('..', 'VERSION'), 'r', 'utf-8') as f:
      File "/usr/lib/python3.7/codecs.py", line 898, in open
        file = builtins.open(filename, mode, buffering)
    FileNotFoundError: [Errno 2] No such file or directory: '../VERSION'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-6tdniw95/sentencepiece/

```

## Expected behavior

I was expecting installation to complete successfully.

## Environment info

$ python transformers-cli env
python: can't open file 'transformers-cli': [Errno 2] No such file or directory

This makes sense since transformers never finishes installing

- `transformers` version: Newest available with pip install
- Platform:  Raspbian Buster
- Python version:  3.7.3
- PyTorch version (GPU?):
- Tensorflow version (GPU?):  2.1 cpu
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

Thanks for your assistance.


Error with run_language_modeling.py training from scratch

Ex: LM (Pretraining)

Related to language modeling pre-training

PyTorch

Anything PyTorch

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Training from scratch

Language I am using the model on (English, Chinese ...): Training from scratch with Esperanto (per tutorial)

The problem arises when using:
* [ ] the official example scripts: (give details below) run_language_model.py
* [x] my own modified scripts: (give details below)

Added the class per the tutorial (https://huggingface.co/blog/how-to-train) and call it instead of TextDataset

```
class EsperantoDataset(Dataset):
    def __init__(self, evaluate: bool = false):
        tokenizer = ByteLevelBPETokenizer(
            "./models/EsperBERTo-small/vocab.json",
            "./models/EsperBERTo-small/merges.txt",
        )
        tokenizer._tokenizer.post_processor = BertProcessing(
            ("</s>", tokenizer.token_to_id("</s>")),
            ("<s>", tokenizer.token_to_id("<s>")),
        )
        tokenizer.enable_truncation(max_length=512)
        # or use the RobertaTokenizer from `transformers` directly.

        self.examples = []

        src_files = Path("./data/").glob("*-eval.txt") if evaluate else Path("./data/").glob("*-train.txt")
        for src_file in src_files:
            print("🔥", src_file)
        lines = src_file.read_text(encoding="utf-8").splitlines()
            self.examples += [x.ids for x in tokenizer.encode_batch(lines)]

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i):
        # We’ll pad at the batch level.
        return torch.tensor(self.examples[i])
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name) Training from scratch
* [x] my own task or dataset: (give details below)
 eo_dedup.txt.gz (https://traces1.inria.fr/oscar/)

## To reproduce

Steps to reproduce the behavior:

1. file structure:
project
│   run_language_modeling.py
│
└───models
│    │
│   └───EsperBERTo-small
│            │   merges.txt
│            │   vocab.json
|
└───datasets
             │   eo-dedup-train.txt

2.
```
python run_language_model.py
	--output_dir ./models/EsperBERTo-small-v1
	--model_type roberta
	--mlm
	--tokenizer_name ./models/EsperBERTo-small
	--do_train
	--learning_rate 1e-4
	--num_train_epochs 5
	--save_total_limit 2
	--save_steps 2000
	--per_gpu_train_batch_size 4
	--evaluate_during_training
	--seed 42
	--train_data_file eo-dedup-train.txt
```
Results in this stack trace with CUDA_LAUNCH_BLOCKING=1:

```
/opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [53,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [53,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "test.py", line 832, in <module>
    main()
  File "test.py", line 782, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "test.py", line 386, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/modeling_roberta.py", line 243, in forward
    inputs_embeds=inputs_embeds,
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/modeling_bert.py", line 799, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/modeling_roberta.py", line 64, in forward
    input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
  File "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/modeling_bert.py", line 193, in forward
    embeddings = inputs_embeds + position_embeddings + token_type_embeddings
RuntimeError: CUDA error: device-side assert triggered
```

## Expected behavior

## Environment info

- `transformers` version: 2.4.1
- Platform: AWS p2.xlarge ubuntu
- Python version: 3.6.5
- PyTorch version (GPU?): 1.3.1
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Create bert-spanish-cased-finedtuned-ner.md

model card

Related to pretrained model cards


Update README.md

Related to pretrained model cards

I trained the model for more epochs so I improved the results. This commit will update the results of the model and add a gif using it with **transformers/pipelines**


How  to run TFBERT model in disable_eager_execution() mode

How  to run `TFBERT `model in `disable_eager_execution()` mode.

If it is possible please let me know, thanks!


how to get "xlnet-base-cased-pytorch_model.bin" original 'last modified' date?

I'd like to test my finetuning on some wikipedia articles that have not been seen by the model.
For that, I can find the date of creation on the wikipedia article, but I'd also like to verify that this is after the 'last modified' date of the model.
How can I get the answer to when was the model parameters' last modified?


Explanation of the results derived from fine tuning

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Hi,

It would be super nice if you can add a visualization util that can show why the model inferred a particular result such that which words of the "sentencex" made it labelled as positive?

Thank you.


RoBERTa has a token_type layer (just a cosmetic issue)

Core: Modeling

Internals of the library; Models.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): RoBERTa

Language I am using the model on (English, Chinese ...): Language-independent

The cosmetic problem:
The fairseq RoBERTa doesn't has a token_type layer:
```
TransformerSentenceEncoder(
  (embed_tokens): Embedding(50265, 768, padding_idx=1)
  (embed_positions): LearnedPositionalEmbedding(514, 768, padding_idx=1)
```
The huggingface implementation of RoBERTa accepts token typ ids because RobertaModel inherits from BertModel and the layer is inherited by RobertaEmbeddings from BertEmbeddings:
```
RobertaEmbeddings(
  (word_embeddings): Embedding(50265, 768, padding_idx=1)
  (position_embeddings): Embedding(514, 768, padding_idx=1)
  (token_type_embeddings): Embedding(1, 768)
  (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
  (dropout): Dropout(p=0.1, inplace=False)
)
```

@julien-c wrote in #2702:
> On the fact that our RoBERTa implem(tation) takes (inoperant by default) `token_type_ids`, maybe we should actually remove them from the implem. If you want to train some, you can always subclass RoBERTa and add them back (but I'm not 100% sure a lot of people use them). Thoughts?

## Expected behavior
I think the huggingface models should be as close to original as possible and therefore RoBERTA should not have a token_type_embeddings layer and not accept token_type_ids. I know this is just a cosmetic issue, but I think it causes some confusion. I would like to use this issue to collect some opinions.

When there are no others opinions, I would like to work on this. This also affects #2727


distilberttokenizer.encode_plus() token_type_ids are non-default

DistilBert doesn't use token_type_ids. Therefore the encode_plus() method of the DistilBertTokenizer should generate them per default. This fix sets the default value of return_token_type_ids to False.

Closes #2702


ValueError: too many dimensions 'str'

# 🐛 Bug

**To Reproduce**
Steps to reproduce the behavior:

Here is my Colab Notebook you can run to to see the error
https://colab.research.google.com/drive/1ESyf46RNBvrg-7DDQ5l8zhlKZjWGdqUv#scrollTo=MqlsdjFVMmMZ

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-9-0b9dcdf94c77> in <module>()
     71
     72 # Train the model
---> 73 model.train_model(train_df)
     74
     75 # Evaluate the model

1 frames
/usr/local/lib/python3.6/dist-packages/simpletransformers/classification/classification_model.py in train_model(self, train_df, multi_label, output_dir, show_running_loss, args, eval_df, verbose, **kwargs)
    261             ]
    262
--> 263         train_dataset = self.load_and_cache_examples(train_examples, verbose=verbose)
    264
    265         os.makedirs(output_dir, exist_ok=True)

/usr/local/lib/python3.6/dist-packages/simpletransformers/classification/classification_model.py in load_and_cache_examples(self, examples, evaluate, no_cache, multi_label, verbose, silent)
    757
    758         if output_mode == "classification":
--> 759             all_label_ids = torch.tensor([f.label_id for f in features], dtype=torch.long)
    760         elif output_mode == "regression":
    761             all_label_ids = torch.tensor([f.label_id for f in features], dtype=torch.float)

ValueError: too many dimensions 'str'

```

The problem arises when using

```
from simpletransformers.classification import ClassificationModel

import pandas as pd
prefix = '/content/'
train_df = pd.read_csv(prefix + 'train.csv', header=None)
train_df=train_df.drop(index=0)

model = ClassificationModel('roberta', 'roberta-base')

model.train_model(train_df)

```


How can I run NER on ALBERT?

I what to run NER on ALBERT, so I checked the [run_ner.py](https://github.com/huggingface/transformers/blob/master/examples/run_ner.py), but it seems like no ALBERT support.
So can I simply import `AlbertTokenizer`, `AlbertForTokenClassification` and
`AlbertConfig` in the script and add them to `MODEL_CLASSES` and `ALL_MODELS`(Or need other config)?


from_pretrained making internet connection if internet turned on

I'd like to ask why model.from_pretrained makes ssl connection event though I provide cache_dir? If I turn off the internet everything works just fine.

```
│     └─ 0.726 from_pretrained  transformers/tokenization_utils.py:256
│        └─ 0.726 _from_pretrained  transformers/tokenization_utils.py:311
│           ├─ 0.570 cached_path  transformers/file_utils.py:205
│           │  └─ 0.570 get_from_cache  transformers/file_utils.py:333
│           │     └─ 0.570 head  requests/api.py:91
│           │        └─ 0.570 request  requests/api.py:16
│           │           └─ 0.565 request  requests/sessions.py:466
│           │              └─ 0.558 send  requests/sessions.py:617
│           │                 └─ 0.558 send  requests/adapters.py:394
│           │                    └─ 0.557 urlopen  urllib3/connectionpool.py:494
│           │                       └─ 0.557 _make_request  urllib3/connectionpool.py:351
│           │                          ├─ 0.413 _validate_conn  urllib3/connectionpool.py:986
│           │                          │  └─ 0.413 connect  urllib3/connection.py:298
│           │                          │     ├─ 0.281 ssl_wrap_socket  urllib3/util/ssl_.py:296
│           │                          │     │  ├─ 0.263 wrap_socket  ssl.py:410
│           │                          │     │  │  └─ 0.263 _create  ssl.py:813
│           │                          │     │  │     └─ 0.263 do_handshake  ssl.py:1132
│           │                          │     │  └─ 0.018 [self]
│           │                          │     └─ 0.132 _new_conn  urllib3/connection.py:143
│           │                          │        └─ 0.132 create_connection  urllib3/util/connection.py:33
│           │                          │           └─ 0.130 [self]
│           │                          └─ 0.144 getresponse  http/client.py:1300
│           │                             └─ 0.144 begin  http/client.py:299
│           │                                └─ 0.144 _read_status  http/client.py:266
│           │                                   └─ 0.144 readinto  socket.py:575
│           │                                      └─ 0.144 recv_into  ssl.py:1060
│           │                                         └─ 0.144 read  ssl.py:920
```

and here's the output with internet turned off

```
     └─ 0.358 from_pretrained  transformers/tokenization_utils.py:256
│        └─ 0.358 _from_pretrained  transformers/tokenization_utils.py:311
│           ├─ 0.255 __init__  transformers/tokenization_bert.py:138
│           │  ├─ 0.163 load_vocab  transformers/tokenization_bert.py:98
│           │  │  └─ 0.160 [self]
│           │  ├─ 0.056 <listcomp>  transformers/tokenization_bert.py:186
│           │  └─ 0.036 [self]
│           └─ 0.102 cached_path  transformers/file_utils.py:205
│              └─ 0.101 get_from_cache  transformers/file_utils.py:333
│                 ├─ 0.083 head  requests/api.py:91
│                 │  └─ 0.083 request  requests/api.py:16
│                 │     └─ 0.080 request  requests/sessions.py:466
│                 │        ├─ 0.066 send  requests/sessions.py:617
│                 │        │  └─ 0.066 send  requests/adapters.py:394
│                 │        │     ├─ 0.046 urlopen  urllib3/connectionpool.py:494
│                 │        │     │  ├─ 0.035 _make_request  urllib3/connectionpool.py:351
│                 │        │     │  │  └─ 0.035 _validate_conn  urllib3/connectionpool.py:986
│                 │        │     │  │     └─ 0.035 connect  urllib3/connection.py:298
│                 │        │     │  │        └─ 0.035 _new_conn  urllib3/connection.py:143
│                 │        │     │  │           ├─ 0.015 create_connection  urllib3/util/connection.py:33
│                 │        │     │  │           │  └─ 0.014 getaddrinfo  socket.py:735
│                 │        │     │  │           ├─ 0.012 [self]
│                 │        │     │  │           └─ 0.008 __init__  urllib3/exceptions.py:20
│                 │        │     │  └─ 0.006 increment  urllib3/util/retry.py:355
│                 │        │     ├─ 0.008 [self]
│                 │        │     └─ 0.008 __init__  requests/exceptions.py:17
│                 │        ├─ 0.006 merge_environment_settings  requests/sessions.py:690
│                 │        │  └─ 0.005 get_environ_proxies  requests/utils.py:755
│                 │        └─ 0.006 [self]
│                 └─ 0.014 filter  fnmatch.py:48
│                    └─ 0.009 _compile_pattern  fnmatch.py:38
│                       └─ 0.005 compile  re.py:232
│                          └─ 0.005 _compile  re.py:271
│                             └─ 0.005 compile  sre_compile.py:759
```


How to get the matrix that is used to combine output from multiple number of attention heads?

Help wanted

Extra attention is needed, help appreciated

Hello,

if I am understanding transformers correctly, right before the feedforward layer, output of individual attention head are concatenated and multiplied by a matrix **H**, so that the outputs from the multiple number of heads will be combined into one output which will then be an input to the subsequent feedforward block within the same layer.

Is there any way that I can retrieve the matrix **H** from the Hugging Face GPT2 model?

Thank you,


UserWarning: The number of elements in the out tensor of shape [1] is 1

PyTorch

Anything PyTorch

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I am using HuggingFace pytorch-transformers and one of my pre-trained models refuse to fine tune giving me those UserWarnings for every torch.utils.data.DataLoader call.

I have described the details in  https://stackoverflow.com/questions/60218634/userwarning-the-number-of-elements-in-the-out-tensor-of-shape-1-is-1

Here is my Notebook so you can run and see the results:
https://colab.research.google.com/drive/1mq9RZ_BX1O5vgxCM0CvPzAm9YVKnq4DQ

But someone downgraded my question for some  reason. What am I missing?

Thanks for your help!
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->

**A link to original question on Stack Overflow**:  https://stackoverflow.com/questions/60218634/userwarning-the-number-of-elements-in-the-out-tensor-of-shape-1-is-1


Update model card: new performance chart

We found a bug in our German conll03 data and fixed it. See deepset-ai/FARM#235

We reran the eval scripts on the new data and updated our charts accordingly.


What does the variable 'present' represent?

Hello,

does the variable 'present' shown in [this](https://github.com/huggingface/transformers/blob/4e69104a1fba717026d6909d06288788e684c749/src/transformers/modeling_gpt2.py#L187) line of Hugging Face GPT-2 code represent final output of a single attention-head? (i.e. **not** the final output of the _output head_ , but the final output of the individual _attention head_, which is placed right before the feedforward block of the same layer).

If yes, is there anyway that I can retrieve the value of the variable 'present'?
Would it be possible that Hugging Face will make the value available for everyone?

Thank you,


PreTrainedTokenizer returns potentially incorrect attention mask

Core: Tokenization

Internals of the library; Tokenization.

Discussion

Discussion on a topic (keep it focused or open a new issue though)

# 🐛 Bug

## Information

When deriving the `attention_mask` `PreTrainedTokenizer` makes an assumption in `prepare_for_model` that the input hasn't been padded prior, this assumption can be false. For example, in the case where one precomputes padded token ids for sentences separately and then uses `BertTokenizer.encode_plus` to join them.

I'm submitting this issue, in order to find out whether this assumption has been made on purpose and if it hasn't I can easily submit a PR fixing it.

In the `PreTrainedTokenizer`, the  `attention_mask` is obtained in two places:

- line `1175`: `encoded_inputs["attention_mask"] = [0] * difference + [1] * len(encoded_inputs["attention_mask"] = [0] * difference + [1] * len(encoded_inputs["input_ids"]) `

- line `1188`: `encoded_inputs["attention_mask"] = [1] * len(encoded_inputs["input_ids"])`.

I suggest that instead of making the assumption attention mask is derived as:

`encoded_inputs["attention_mask"] = encoded_inputs["input_ids"] != 0`


DistilBERT distilbert-base-cased failed to load

**Issue**
DistilBERT **distilbert-base-cased** failed to load. _Please note, 'distilbert-base-uncased' works perfectly fine._

**Error Message**
OSError: Model name 'distilbert-base-cased' was not found in tokenizers model name list (distilbert-base-uncased, distilbert-base-uncased-distilled-squad, distilbert-base-german-cased, distilbert-base-multilingual-cased). We assumed 'distilbert-base-cased' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.

**Model I am using** : distilbert-base-cased

**Language** : English

**The problem arises when using below code**
```
MODELS = [(DistilBertModel, DistilBertTokenizer, 'distilbert-base-cased')]

for model_class, tokenizer_class, pretrained_weights in MODELS:
    # Load pretrained model/tokenizer
    tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
    model = model_class.from_pretrained(pretrained_weights)
```

**Environment info**
Python 3.6.9
ipykernel==5.1.3
ipython==7.11.1
ipython-genutils==0.2.0
ipywidgets==7.5.1
jupyter==1.0.0
jupyter-client==5.3.4
jupyter-console==6.0.0
jupyter-core==4.6.1
jupyter-http-over-ws==0.0.7
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
matplotlib==3.1.2
numpy==1.18.1
scipy==1.4.1
tensorboard==2.1.0
tensorflow-estimator==2.1.0
tensorflow-gpu==2.1.0
tokenizers==0.0.11
torch==1.4.0
tornado==6.0.3
tqdm==4.42.1
traitlets==4.3.3
transformers==2.4.1